Re: [squid-users] newbie question about large post requests

Thanks Amo for your fast reply,

I meant it ignores the POST requests, I can access the requested page
normally but with no post parameters. So, If I want to navigate
through the website using POST, I can't and I get only the first page.

Actually I didn't change any configuration !!

Squid Version and Configuration options:
Squid Cache: Version 3.0.STABLE16
configure options: '--build=i386-redhat-linux-gnu'
'--host=i386-redhat-linux-gnu' '--target=i386-redhat-linux-gnu'
'--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr'
'--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc'
'--includedir=/usr/include' '--libdir=/usr/lib'
'--libexecdir=/usr/libexec' '--sharedstatedir=/usr/com'
'--mandir=/usr/share/man' '--infodir=/usr/share/info'
'--exec_prefix=/usr' '--bindir=/usr/sbin'
'--libexecdir=/usr/lib/squid' '--localstatedir=/var'
'--datadir=/usr/share' '--sysconfdir=/etc/squid'
'--disable-dependency-tracking' '--enable-arp-acl'
'--enable-auth=basic,digest,ntlm,negotiate'
'--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SMB,YP,getpwnam,multi-domain-NTLM,SASL'
'--enable-negotiate-auth-helpers=squid_kerb_auth'
'--enable-cache-digests' '--enable-cachemgr-hostname=localhost'
'--enable-delay-pools' '--enable-digest-auth-helpers=password'
'--enable-epoll'
'--enable-external-acl-helpers=ip_user,ldap_group,unix_group,wbinfo_group'
'--enable-icap-client' '--enable-ident-lookups' '--with-large-files'
'--enable-linux-netfilter' '--enable-ntlm-auth-helpers=SMB,fakeauth'
'--enable-referer-log' '--enable-removal-policies=heap,lru'
'--enable-snmp' '--enable-ssl' '--enable-storeio=aufs,diskd,null,ufs'
'--enable-useragent-log' '--enable-wccpv2' '--with-aio'
'--with-default-user=squid' '--with-filedescriptors=16384' '--with-dl'
'--with-openssl=/usr/kerberos' '--with-pthreads'
'build_alias=i386-redhat-linux-gnu' 'host_alias=i386-redhat-linux-gnu'
'target_alias=i386-redhat-linux-gnu' 'CFLAGS=-fPIE -Os -g -pipe
-fsigned-char -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386
-mtune=generic -fasynchronous-unwind-tables' 'LDFLAGS=-pie'
'CXXFLAGS=-fPIE -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32
-march=i386 -mtune=generic -fasynchronous-unwind-tables' 'FFLAGS=-O2
-g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic
-fasynchronous-unwind-tables'

But I don't know how to get the HTTP headers of the traffic between
squid and the spider. Can squid log such details ?

Finally, the spider is trying to access this website, navigate on it
using post requests and parse the data. When I use the spider without
squid, it works like a charm, once I use squid, the post problem
occurs.

Thanks again

On Tue, May 29, 2012 at 2:29 AM, Amos Jeffries <squid3_at_treenet.co.nz> wrote:
> On 29.05.2012 07:47, Mahmoud Abdel-Fattah wrote:
>>
>> Hello,
>>
>> I'm totally new to squid and using it as a proxy for a scrapy (python
>> spider ). it works great with all get requests, but it doesn't work
>> with large post requests like the requests is generated by ASP.net in
>> the following site : http://www.e3050.com/
>>
>> So, is there any configuration should I adjust to enable / enlarge
>> post requests ?
>
>
> Maybe yes, Maybe no. The answer to this question depends on your definition
> of "doesn't work".
>
> What kind of limitation did you configure to prevent POST working like they
> do out of the box?
>
> We need to know specifics about what is going wrong...
>
> * software version numbers.
>  -> check against the latest current release, if its old try with the
> current one(s).
>
> * HTTP headers of the traffic between Squid and the spider if you can get
> them.
>  -> what exactly is the spider trying to do? and how?
>  --> what does the site do? how is the spider reacting to the response?
>
> Amos
>
Received on Tue May 29 2012 - 10:34:15 MDT