HttpWebResponse.GetResponse() fiddler says "Response Header parsing failed. "

Tagged:

I'm making a request to a website via the HttpWebRequest/HttpWebResponse objects.

I'm making several successful calls to the web site and every other call to the same dynamic page is failing.

In the debugger I'm getting a "Internal server error 500" fiddler also shows a 500 response and contains:

[Fiddler] Response Header parsing failed.
This can be caused by an illegal HTTP response earlier on this reused server socket--     for instance, a HTTP/304 response which illegally contains a body.
Response Data:
<plaintext> 
0D 0A 3C 21 44 4F 43 54 59 50 45 20 48 54 4D 4C 20 50 55 42 4C 49 43 20  ..<!DOCTYPE      HTML PUBLIC 
22 2D 2F 2F 57 33 43 2F 2F 44 54 44 20 48 54 4D 4C 20 34 2E 30 20 54 72  "-//W3C//DTD HTML 4.0 Tr
61 6E 73 69 74 69 6F 6E 61 6C 2F 2F 45 4E 22 3E 0D 0A 3C 48 54 4D 4C 3E  ansitional//EN">..<HTML>
0D 0A 09 3C 48 45 41 44 3E 0D 0A 09 09 3C 74 69 74 6C 65 3E 56 69 65 77  ...<HEAD>....<title>View

I've removed all the hex and viewed the page and is what I expect to be returned but for some reason the server is reporting a 500 and the HttpWebRequest object throws an exception on this.

I've tried all the other "fixes" for this issue and none work. It might just be malformed data sent from the server but is there a lower level object to use than HttpWebRequest that's not a pita to work with?

EDIT: I didn't include the entire hex/entire html block in the above example.
EDIT: Turning off fiddler I get this in the debugger

EDIT: So, from what I've seen the HttpWebResponse object is acting accordingly. The server is just flaky and sometime returns the same exact data with different http status codes. For a quick fix I just wrapped each call in a try/catch and in the catch block just retying the exact same call. So far it works great and semi-proves that it's the sites fault and not the HttpWebResponse object.

The server committed a protocol violation. Section=ResponseStatusLine

user1231231412H
2012-01-09 00:00:32 Scores:2
Do you have control over the website you're trying to hit?
M.BabcockH
2012-01-09 00:04:36 Scores:1
@M.Babcock I do not. So far it looks like I can strip the html out of the 500 response and parse that, but that would stink!
user1231231412H
2012-01-09 00:06:45 Scores:0
@Amadan It's a winforms app.
user1231231412H
2012-01-09 00:20:12 Scores:0
The output from Fiddler indicates that the server isn't returning HTTP Headers. You could use Netmon to determine whether the problem is that the server isn't sending them at all, or that the previous HTTP response had a bad Content-Length header that caused the next response's headers to get eaten.
EricLaw -MSFT-H
2012-01-24 01:23:41 Scores:0
@EricLaw-MSFT- I'll check the content-length because it *is* an issue for every other request but only on the "bad" pages, all other pages work fine. PS. Thanks for Fiddler!
user1231231412H
2012-01-26 18:09:29 Scores:0

2 answers

Answer 1
Scores:1

An HTTP 304 response means that the page content has not changed since the last time you hit the page (they're likely using caching). Hit the page less often or cache the response for use when this is encountered.

EDIT

The server is sending an invalid 304 response which contains data. This violates the HTTP spec and the HttpWebResponse/Fiddler is validly transforming it into a 500 as such.

EDIT

You may be able to keep using the HttpWebRequest/HttpWebResponse if you use the following setting in your app.config:

<configuration>
    <system.net>
        <settings>
            <httpWebRequest useUnsafeHeaderParsing="true" />
        </settings>
    </system.net>
</configuration> 
M.Babcock H
2012-01-09 00:10:32
On the 1st page hit I'm collecting a handful of links to files I'm downloading, each subsequent hit to the site is to the same "download" url but with different querystring params. I would think that would prevent the caching but that may only be for specific configurations.
user1231231412 H
2012-01-09 00:18:49 Scores:0
That would depend on how sophisticated their caching system is, which apparently it isn't very if it is generating malformed HTTP responses.
M.Babcock H
2012-01-09 00:21:45 Scores:0
It's a .NET 1.1 site and has major malformed html so probably not :)
user1231231412 H
2012-01-09 00:24:31 Scores:0
Then you'll need to increase the interval at which you pull down pages so that it doesn't hit the site caching bugs. Alternatively you could take the approach you talked about in your comment above by trying to read from the response anyway, but that is less than optimal as well.
M.Babcock H
2012-01-09 00:26:53 Scores:0
I wrapped a try catch around it and in the catch running the same line of code. Like a poor mans auto-retry. This seems to work for some of them but, the site is returning a 500, fiddler says in the error message it could be caused by a 304. There any way to force a return status of 200 with HttpWebRequest? or other?
user1231231412 H
2012-01-09 00:33:35 Scores:0
Not that I'm aware of short of implementing your own HTTP parser. How familiar are you with the RFCs? ;)
M.Babcock H
2012-01-09 00:35:43 Scores:0
You might be able to use something like [this](https://github.com/bvanderveen/httpmachine) as a starting point.
M.Babcock H
2012-01-09 00:43:08 Scores:0
That looks like a good lib I'll have to try out. I've updated the post with more info.
user1231231412 H
2012-01-09 00:53:51 Scores:0
@JonC - See my edit. I did a little googling of the error from your edit and this seems to be a common solution.
M.Babcock H
2012-01-09 01:08:06 Scores:0
Thanks for the research. I've tried that settings in the config and programmatically. Also tried changing the http protocol versions. No Luck.
user1231231412 H
2012-01-09 01:09:46 Scores:0
Answer 2
Scores:0

Hei Jon,

Is the same outcome without Fiddler? (just printing the exception on screen). I had trouble sometimes with debuggers misbehavior.

If the server is returning intermittent error and you do not have control over it, I am afraid you can not do much. The message seems clear that is a standard for 304 that the response should not have a body, but the server can do whatever it wants, so it is pretty much its problem to address to.

See W3C: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

If the client has performed a conditional GET request and access is allowed, but the document has not been modified, the server SHOULD respond with this status code. The 304 response MUST NOT contain a message-body, and thus is always terminated by the first empty line after the header fields.

Regarding the HttpWebRequest, I never had any problems, and never heard of any cases it can not handle HTTP communication. But if the case is that you want to go nuts and handle the packets yourself, Google how to build your own HttpWebRequest with sockets.

This project could be a start: http://www.codeproject.com/Articles/13486/A-Simple-Crawler-Using-C-Sockets

bruno.braga H
2012-01-24 01:06:41
Thanks for the link, that looks like a good project.
user1231231412 H
2012-01-26 18:08:11 Scores:0
Share |
View original post at stackoverflow.com

Related topics

There may be a place for this type of question, but only if the answer is correct. While isn't a keyword in C#. while is. Also, the space between the ! and = isn't valid. Try: int i=0; while (i != 10) { Console.WriteLine(i); ...

c# HttpWebResponse Header encoding

I have the following problem. I contact an address which I know employs a 301 redirect. using HttpWebRequest loHttp = (HttpWebRequest)WebRequest.Create(lcUrl); and loHttp.AllowAutoRedirect = false; so that I am not redirected. Now I get the header of the response in order to identify the new url. using loWebResponse.GetResponseHeader("Location"); The problem is that since this url ...

Fiddler/C#: search content of request/response for special phrases

this is my first visit to stackoverflow and right now I feel very comfortable with this site. It already helped me to get the FiddlerCore embedded into MS Visual C# 2008 Express Edition. Just needed to create a reference(Verweis) to the fiddlercore dll from Project Explorer(Projektmappenexplorer) in MS Visual C# ...

See HttpWebRequest as string before GetResponse without using fiddler

How can i see HttpWebRequest object as string before calling GetResponse method? I want to see raw format of request something like this as in fiddler: Content-Type: multipart/form-data; boundary=---------------------------2600251021003 Content-Length: 338 -----------------------------2600251021003 Content-Disposition: form-data; name="UPLOAD_FILEName"; filename="Searchlight062210 w price.csv" Content-Type: application/vnd.ms-excel ,,,,, -----------------------------2600251021003 Content-Disposition: form-data; name="submit" submit -----------------------------2600251021003-- I tried following code, ...

PHP: fopen failed "HTTP Request Failed", but response header has a status code 200

I have a PHP script, that should connect to a proxy, chosen from a proxy list and download a file. Some of the proxies (out of 200-400 working ones) work perfectly, but others don't, and I cannot find out why. Here the code that connects through the proxy: ...

GetResponse throws WebException and ex.Response is null

I have found example of how to treat WebException on GetResponse call, and puzzling on how the response can be extracted from WebException Response. The second puzzle is why null response is treated as throw; Any suggestion? HttpWebResponse response = null; try { response = (HttpWebResponse) request.GetResponse(); } catch (WebException ex) { ...

Fiddler - ReadResponse failed: The server did not return a response for this request

This was the first time I encountered this kind of error after dealing with RESTful web service in couple of times. I find it hard to trace the cause of error, hope you could help me. I have this attribute for Login service [WebGet(UriTemplate = "Login?username={username}&password={password}&ip={ip}", ResponseFormat = ...

HttpWebRequest GetResponse. How to wait for that response?

I'm sending some SMS using HttpWebRequest. But some times the messages appear to send double, three times and more reviewing all the proccess appear to be ok. The only thing i think is HttpWebRequest is working asynchronus. here is my code, public ...

How do I prevent an https response from throwing an AuthenticationException with Fiddler running?

Relative newbie to C# here :) I'm currently creating a web link scraper and having issues with the responses I'm getting when trying to login to the website via my program. I'm trying to use Fiddler to see if my program is sending the proper data, but my program is throwing ...

C# - HttpWebResponse Header Not Being Set

I'm attempting to set the content-length of my outgoing response headers but my code seems to have no effect on the headers. web_op_context.OutgoingResponse.Headers.Set("Content-Type", http_req.content_type); web_op_context.OutgoingResponse.Headers.Add("Content-Length", "10"); web_op_context.OutgoingResponse.Headers.Add("Access-Control-Allow-Origin", "*"); web_op_context.OutgoingResponse.Headers.Remove("Server"); web_op_context.OutgoingResponse.Headers.Remove("X-Powered-By"); The line for adding content-length ...