Re: [squid-users] newbie question about large post requests

On 29/05/2012 13:33, Mahmoud Abdel-Fattah wrote:
> Thanks Amo for your fast reply,
>
> I meant it ignores the POST requests, I can access the requested page
> normally but with no post parameters. So, If I want to navigate
> through the website using POST, I can't and I get only the first page.
>
> Actually I didn't change any configuration !!
>
> Squid Version and Configuration options:
> Squid Cache: Version 3.0.STABLE16
> configure options: '--build=i386-redhat-linux-gnu'
> '--host=i386-redhat-linux-gnu' '--target=i386-redhat-linux-gnu'
> '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr'
> '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc'
> '--includedir=/usr/include' '--libdir=/usr/lib'
> '--libexecdir=/usr/libexec' '--sharedstatedir=/usr/com'
> '--mandir=/usr/share/man' '--infodir=/usr/share/info'
> '--exec_prefix=/usr' '--bindir=/usr/sbin'
> '--libexecdir=/usr/lib/squid' '--localstatedir=/var'
> '--datadir=/usr/share' '--sysconfdir=/etc/squid'
> '--disable-dependency-tracking' '--enable-arp-acl'
> '--enable-auth=basic,digest,ntlm,negotiate'
> '--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SMB,YP,getpwnam,multi-domain-NTLM,SASL'
> '--enable-negotiate-auth-helpers=squid_kerb_auth'
> '--enable-cache-digests' '--enable-cachemgr-hostname=localhost'
> '--enable-delay-pools' '--enable-digest-auth-helpers=password'
> '--enable-epoll'
> '--enable-external-acl-helpers=ip_user,ldap_group,unix_group,wbinfo_group'
> '--enable-icap-client' '--enable-ident-lookups' '--with-large-files'
> '--enable-linux-netfilter' '--enable-ntlm-auth-helpers=SMB,fakeauth'
> '--enable-referer-log' '--enable-removal-policies=heap,lru'
> '--enable-snmp' '--enable-ssl' '--enable-storeio=aufs,diskd,null,ufs'
> '--enable-useragent-log' '--enable-wccpv2' '--with-aio'
> '--with-default-user=squid' '--with-filedescriptors=16384' '--with-dl'
> '--with-openssl=/usr/kerberos' '--with-pthreads'
> 'build_alias=i386-redhat-linux-gnu' 'host_alias=i386-redhat-linux-gnu'
> 'target_alias=i386-redhat-linux-gnu' 'CFLAGS=-fPIE -Os -g -pipe
> -fsigned-char -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
> -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386
> -mtune=generic -fasynchronous-unwind-tables' 'LDFLAGS=-pie'
> 'CXXFLAGS=-fPIE -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
> -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32
> -march=i386 -mtune=generic -fasynchronous-unwind-tables' 'FFLAGS=-O2
> -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
> --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic
> -fasynchronous-unwind-tables'
>
>
>
> But I don't know how to get the HTTP headers of the traffic between
> squid and the spider. Can squid log such details ?
>
> Finally, the spider is trying to access this website, navigate on it
> using post requests and parse the data. When I use the spider without
> squid, it works like a charm, once I use squid, the post problem
> occurs.
>
>
> Thanks again
>

you can use tcpdump on the squidbox to capture all the traffic between
the proxy and the client but you will need to put some patterns as for
the client name\ip and maybe destination ip\name.

you can use the POST method only for pages that are built for that.
what do you get in your access.log when the spider tries to access the
page with post?

Eliezer

>
>
>
>
> On Tue, May 29, 2012 at 2:29 AM, Amos Jeffries<squid3_at_treenet.co.nz> wrote:
>> On 29.05.2012 07:47, Mahmoud Abdel-Fattah wrote:
>>>
>>> Hello,
>>>
>>> I'm totally new to squid and using it as a proxy for a scrapy (python
>>> spider ). it works great with all get requests, but it doesn't work
>>> with large post requests like the requests is generated by ASP.net in
>>> the following site : http://www.e3050.com/
>>>
>>> So, is there any configuration should I adjust to enable / enlarge
>>> post requests ?
>>
>>
>> Maybe yes, Maybe no. The answer to this question depends on your definition
>> of "doesn't work".
>>
>> What kind of limitation did you configure to prevent POST working like they
>> do out of the box?
>>
>> We need to know specifics about what is going wrong...
>>
>> * software version numbers.
>> -> check against the latest current release, if its old try with the
>> current one(s).
>>
>> * HTTP headers of the traffic between Squid and the spider if you can get
>> them.
>> -> what exactly is the spider trying to do? and how?
>> --> what does the site do? how is the spider reacting to the response?
>>
>> Amos
>>

-- 
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il