wpull uses the following code to check whether an HTTP response is chunked:
|
chunked_match = re.match( |
|
r'chunked($|;)', |
|
response.fields.get('Transfer-Encoding', '') |
|
) |
This would match something like foochunked, which shouldn't trigger chunk decoding, but that's probably not a significant issue. But more importantly, the matching is case-sensitive, while RFC 2616 mandates that
All transfer-coding values are case-insensitive.
ArchiveBot job 5cg5phpdt64ikpaabpfa4hdh2 just ran into this because the server sends Transfer-Encoding: Chunked. It resulted in chunk sizes being included in extracted links.
wpull uses the following code to check whether an HTTP response is chunked:
wpull/wpull/protocol/http/stream.py
Lines 374 to 377 in cfa5bcc
This would match something like
foochunked, which shouldn't trigger chunk decoding, but that's probably not a significant issue. But more importantly, the matching is case-sensitive, while RFC 2616 mandates thatArchiveBot job 5cg5phpdt64ikpaabpfa4hdh2 just ran into this because the server sends
Transfer-Encoding: Chunked. It resulted in chunk sizes being included in extracted links.