The purpose of this request is to discover if there is a "captive portal" in the way. A captive portal is when, after connecting to the WiFi, any web request you makes gets redirected to a login/ToS page. In order to continue, you must either login with a username/password (or sign up, then login), and/or access the Terms of Service.
The reason Apple does this is because you may be using an app other than the web browser. For example, the only thing you might be doing is syncing your e-mail. In such situations, you would never see the portal page, and your app will mysteriously fail to connect to the Internet.
Therefore, before your app has a chance to access the network, Apple does this for you. It sends out a request to the above URL. If the request gets redirected, then Apple knows there is a portal. It then launches a dialog box, containing Safari, to give you a chance to login.
The following is the sniffed version of the HTTP request:
GET /library/test/success.html HTTP/1.0 Host: www.apple.com User-Agent: CaptiveNetworkSupport/1.0 wispr Connection: close
One of the questions people had was whether this was a privacy issue. the answer is "largely no". It sends no personally identifiable information. In particular, it doesn't send any cookies. The request is made from the WiFi software, not Safari. Therefore, any cookies you have in Safari will not be sent via this request. I verified this myself, by accessing Apple.com via Safari and watching cookies being sent, but verifying this did not send cookies.
Another question is whether this is an attack vector. The answer is "probably yes". There is more to the functionality than a simple HTTP request. If you look up the keyword "wispr" from the User-Agent string, you'll find out why.
The idea is that smart WiFi portals will detect that this is a WISPr-supporting device, and send back a WISPr message in XML. This allows the iPhone to then login with cached credentials via another XML message. This means, for example, you might be able to grab somebody's credentials with a properly configured WiFi access-point.
I've seen the iPhone deal with such an intelligent WiFi access-point, but at the time, I did not have the presence of mind to sniff the exchange, so I'm not sure what happened.
Another odd bit of behavior was loging onto an "attwifi" access-point at Starbucks. As you might have heard, using the iPhone on their network is free. The way this works is that the iPhone sends out a request to "http://attwifi.apple.com/library/test/success.html: the same URL as before, but with the "attwifi" in front.
At my local Starbucks, all web surfing is free. But, Windows presents a captive logon page where you must accept the Terms of Service, but the iPhone doesn't. I assume the portal detects this URL, and automatically opens up the access-point without doing a redirection. I need to test witha Linux distro in order to figure out what's going on.
No personally identifiable information is sent, so there isn't much of a privacy breach.
There is more complexity to this feature than the simple HTTP request; there is probably a way to attack it.
You can probably configure your machine to emulate this request, and get free WiFi that is intended for iPhones.
UPDATE 2012-Sep-19: On the day of iPhone 5 release, this webpage went down, breaking everyone's WiFi.