respmod_copy.py errors on files > 65535 bytes

[cyberxml]

setup a squid+pyicap server using respmod_copy.py from the examples directory

on an ftp or http server, prepare two files
head -c 65535 < /dev/urandom > file-65535.txt
head -c 65536 < /dev/urandom > file-65536.txt

on the ftp client, attempt to retrieve each file
curl -m 10 -x squidproxy:3128 ftp://username:userpass@netserver:21/file-65535.txt (pass)
curl -m 10 -x squidproxy:3128 ftp://username:userpass@netserver:21/file-65536.txt (fail)
curl -m 10 -x squidproxy:3128 http://netserver/file-65535.txt (pass)
curl -m 10 -x squidproxy:3128 http://netserver/file-65536.txt (fail)

Error generated is

Exception happened during processing of request from ('127.0.0.1', 50874)
Traceback (most recent call last)
File "/usr/lib64/python2.7/SocketServer.py", line 593, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib64/python2.7/SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib64/python2.7/SocketServer.py", line 649, in init
self.handle()
File "/usr/lib/python2.7/site-packages/pyicap.py", line 443, in handle
self.handle_one_request()
File "/usr/lib/python2.7/site-packages/pyicap.py", line 481, in handle_one_request
self.raw_requestline = self.rfile.readline(65537)
File "/usr/lib64/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
error: [Errno 104] Connection reset by peer

[cyberxml]

icap portion of squid.conf (recall this works for all files < 65536 bytes)

#icap config for pyicap
icap_enable on
icap_preview_enable on
icap_preview_size 1024
icap_service service_resp respmod_precache bypass=1 icap://127.0.0.1:1344/example
adaptation_access service_resp allow all

[cyberxml]

Looking at the tcpdump trace for the loopback (respmod_copy pyicap service is on same server as squid), the full file is being returned to squid in both instances. When the final continuation for the 65535 byte file is transferred from ICAP to SQUID, squid replies with FIN,ACK. When the final continuation for the 65536 byte file is transferred from ICAP to SQUID, squid replies with a RST, ACK.

[cyberxml]

Examining the pcaps more closely, I see that an ICAP header with "Allow: 204" is sent in the case of a file with 65535 bytes but not sent when the file is 65536 or more. So this might be a squid thing. Can override in OPTIONS?

[cyberxml]

This may be relevant

/My icap conf activated preview and preview size as follow : />/icap_preview_enable on />/icap_preview_size 1024 /
IIRC, Squid ignores icap_preview_size in squid.conf (a bug). The ICAP
service OPTIONS response determines the Preview size (subject to an
internal limit of 64KB).

http://squid-web-proxy-cache.1019090.n4.nabble.com/ICAP-and-Allow-204-Header-td4675762.html

[cyberxml]

Confirmed the 64KB limit is due to Squid handling of Preview.
Updated limit in squid source code, compiled, and retested.
Larger files could transfer with the modified code.

But how can you pass files outside of the Preview feature?
For example squid+squidclamav+c_icap passes large files for ClamAV scanning.