I mainly want to write this down somewhere so I can refer back to it. HTTP never ceases to surprise me and I never know when I'll need this info again.

Yesterday I started looking into a bug with our Web Macro Recorder (WMR) tool. In the world of web scanners being able to record login macros and knowing when to replay them is essential to quality coverage during a scan.

The WMR would hang when attempting go through the login process on some sites, e.g. Yahoo.com. After receiving a 301 redirect to the HTTPS login page it would just quit, give up, throw in the proverbial towel. I watched this happen from WireShark, there was no RST there was nothing.

The response form Yahoo is:

HTTP/1.0 301 Moved Permanently
Location: https://login.yahoo.com/config/login?.src=fpctx&.done=http://www.yahoo.com
Cache-Control: no-cache
Set-Cookie: D=; expires=Thu, 01-Jan-1970 00:00:00 GMT; path=/; domain=.yahoo.com

I decided to try and recreate this on a server I control so that I could mess around. After I noticed it was using HTTP/1.0 I tried using the headers() function in PHP to give responses as http proto ver 1.0. No luck... There is a bug with the headers() function in PHP that requires you to add some additional voodoo.

?php
apache_setenv("downgrade-1.0", "true");
apache_setenv("force-response-1.0", "true");
header("HTTP/1.0 301 Moved Permanently");
header("Location: https://login.yahoo.com/config/login?.src=fpctx&.done=http://$
header("Cache-Control: no-cache");
header("Set-Cookie: D=; expires=Thu, 01-Jan-1970 00:00:00 GMT path=/; domain=.y$
exit("Redirect Success");
?

After implementing this I wasn't able to reproduce the bug in WMR that was occurring on Yahoo.com. The Apache server I was using was also configured to send additional headers that Yahoo wasn't including. The one that caught my suspicion was the Connection: Keep-Alive header.

Using interactive mode I tried removing the Connection header sent by my server and SUCCESS was able to reproduce the same behavior that occurs with Yahoo.com. So the one issue is that the WMR is not properly handling HTTP/1.0 redirects.

In other testing it was determined that WMR would work sometimes but not every time. e.g. We saw it work about 66% of the time on Citigroup.com. After the revelation with Yahoo.com I decided to go back and look at Citigroup's HTTP responses.

I noticed this:

HTTP/1.1 302 Moved Temporarily
Date: Thu, 12 Feb 2009 16:11:20 GMT
Server: Hitbox Gateway 9.3.6-rc1
P3P: policyref="/w3c/p3p.xml", CP="NOI DSP LAW NID PSA ADM OUR IND NAV COM"
Set-Cookie: CTG=1234455080; path=/; domain=vendorweb.citibank.com; expires=Thu, 19-Feb-2009 16:11:20 GMT; max-age=604800
nnCoection: close
Pragma: no-c... [ Read More (0.1k in body) ]