Re: [squid-users] WARNING: An error inside Squid has caused an HTTP reply without Date

On 7/7/2011 1:25 AM, Amos Jeffries wrote:
> On 07/07/11 05:32, Wilson Hernandez wrote:
>> On 7/6/2011 1:08 PM, Wilson Hernandez wrote:
>>> On 7/6/2011 12:28 PM, Wilson Hernandez wrote:
>>>> Hello.
>>>>
>>>> I have a partition for squid that is almost full (I'm getting a
>>>> message):
>>>> WARNING: An error inside Squid has caused an HTTP reply without
>>>> Date:. Please report this
>>>> 2011/07/06 12:23:35| DiskThreadsDiskFile::openDone: (2) No such file
>>>> or directory
>>>> 2011/07/06 12:23:35| /var2/squid/cache/18/A6/0057F5D8
>>>> 2011/07/06 12:23:35| DiskThreadsDiskFile::openDone: (2) No such file
>>>> or directory
>>>> 2011/07/06 12:23:35| /var2/squid/cache/18/A6/0057F5E4
>>>> 2011/07/06 12:23:35| clientProcessRequest: Invalid Request
>>>> 2011/07/06 12:23:40| WARNING: Disk space over limit: 3615331284 KB >
>>>> 204800000 KB
>>>> 2011/07/06 12:23:41| DiskThreadsDiskFile::openDone: (2) No such file
>>>> or directory
>>>> 2011/07/06 12:23:41| /var2/squid/cache/18/A9/0057F8BD
>>>> 2011/07/06 12:23:42| DiskThreadsDiskFile::openDone: (2) No such file
>>>> or directory
>>>> 2011/07/06 12:23:42| /var2/squid/cache/18/A9/0057F8DC
>>>> 2011/07/06 12:23:43| could not parse headers from on disk structure!
>>>> 2011/07/06 12:23:43| WARNING: An error inside Squid has caused an
>>>> HTTP reply without Date:. Please report this
>>>> 2011/07/06 12:23:44| DiskThreadsDiskFile::openDone: (2) No such file
>>>> or directory
>>>> 2011/07/06 12:23:44| /var2/squid/cache/18/A9/0057F8E8
>>>> 2011/07/06 12:23:46| DiskThreadsDiskFile::openDone: (2) No such file
>>>> or directory
>>>> 2011/07/06 12:23:46| /var2/squid/cache/18/A9/0057F923
>>>> 2011/07/06 12:23:51| WARNING: Disk space over limit: 3615319720 KB >
>>>> 204800000 KB
>>>> 2011/07/06 12:24:02| WARNING: Disk space over limit: 3615308076 KB >
>>>> 204800000 KB
>>>> 2011/07/06 12:24:07| could not parse headers from on disk structure!
>>>> 2011/07/06 12:24:07| WARNING: An error inside Squid has caused an
>>>> HTTP reply without Date:. Please report this
>>>> 2011/07/06 12:24:09| DiskThreadsDiskFile::openDone: (2) No such file
>>>> or directory
>>>> 2011/07/06 12:24:09| /var2/squid/cache/3C/0B/003B93E7
>>>> 2011/07/06 12:24:09| DiskThreadsDiskFile::openDone: (2) No such file
>>>> or directory
>>>> 2011/07/06 12:24:09| /var2/squid/cache/13/B8/0053126B
>>>> 2011/07/06 12:24:10| DiskThreadsDiskFile::openDone: (2) No such file
>>>> or directory
>>>> 2011/07/06 12:24:10| /var2/squid/cache/00/68/00006868
>>>> 2011/07/06 12:24:10| could not parse headers from on disk structure!
>>>> 2011/07/06 12:24:10| WARNING: An error inside Squid has caused an
>>>> HTTP reply without Date:. Please report this
>>>> 2011/07/06 12:24:13| WARNING: Disk space over limit: 3615298144 KB >
>>>> 204800000 KB
>>>> 2011/07/06 12:24:14| could not parse headers from on disk structure!
>>>> 2011/07/06 12:24:14| WARNING: An error inside Squid has caused an
>>>> HTTP reply without Date:. Please report this
>>>> 2011/07/06 12:24:14| could not parse headers from on disk structure!
>>>> 2011/07/06 12:24:14| WARNING: An error inside Squid has caused an
>>>> HTTP reply without Date:. Please report this
>>>> 2011/07/06 12:24:14| could not parse headers from on disk structure!
>>>> 2011/07/06 12:24:14| WARNING: An error inside Squid has caused an
>>>> HTTP reply without Date:. Please report this
>>>> 2011/07/06 12:24:14| DiskThreadsDiskFile::openDone: (2) No such file
>>>> or directory
>>>> 2011/07/06 12:24:14| /var2/squid/cache/11/0D/00506C25
>>>> 2011/07/06 12:24:16| could not parse headers from on disk structure!
>>>> 2011/07/06 12:24:16| WARNING: An error inside Squid has caused an
>>>> HTTP reply without Date:. Please report this
>>>> 2011/07/06 12:24:23| could not parse headers from on disk structure!
>>>> 2011/07/06 12:24:23| WARNING: An error inside Squid has caused an
>>>> HTTP reply without Date:. Please report this
>>>> 2011/07/06 12:24:24| WARNING: Disk space over limit: 3615285372 KB >
>>>> 204800000 KB
>>>>
>>>>
>>>> I don't know if this is caused by disk space or what. If is caused by
>>>> disk space how can I clean old cached files from the squid's cache?
>
> Looks like something other than Squid is playing with the cache files
> on disk.
> "No such file or directory" - something erased a file without
> informing Squid. Though it could be that Squid scheduled the file for
> erase and the OS did not get around to it until after Squid was
> re-using the same fileno/name for another object. (rare condition but
> possible).
>
> "Disk space over limit" - seriously nasty. The object size accounting
> in Squid has been screwed up somehow.
> Squid will divert resources to erasing that extra 3.4 _TB_ of
> unwanted cache content ASAP. I'm a little surprised to see numbers
> that far out in 3.1.11, AFAIK the cache accounting was fixed in an
> earlier 3.1 release.
>
>
> "could not parse headers from on disk structure!" and "WARNING: An
> error inside Squid has caused an HTTP reply without Date:."
> - strong signs of either disk corruption, a bug, or objects stored by
> some older Squid version without Date: headers. We are still trying to
> track down which. If you can move to 3.1.12 there are some extra
> details logged about those objects to help figure this out.
>
>
>>>>
>>>> Thanks.
>>>
>>> /usr/local/squid/sbin/squid -v
>>> Squid Cache: Version 3.1.11
>>> configure options: '--prefix=/usr/local/squid'
>>> '--sysconfdir=/etc/squid' '--enable-delay-pools'
>>> '--enable-kill-parent-hack' '--disable-htcp'
>>> '--enable-default-err-language=es' '--enable-linux-netfilter'
>>> '--disable-ident-lookups' '--localstatedir=/var/log/squid'
>>> '--enable-stacktraces' '--with-default-user=proxy'
>>> '--with-large-files' '--enable-icap-client' '--enable-async-io'
>>> '--enable-storeio=aufs' '--enable-removal-policies=heap,lru'
>>> '--with-maxfd=65536' --with-squid=/workingdir/squid-3.1.11
>>> --enable-ltdl-convenience
>>>
>>>
>> Hello.
>>
>> I deleted the swap.state file and re-run squid -z and get the following:
>
> _just_ the swap.state erase does nothing but trigger a full disk scan
> (DIRTY re-build) if Squid was stopped at the time. Otherwise it has no
> effect.
>
> Erasing individual cache data files is pretty safe. Squid will log
> that "openDone: (2) No such file or directory" message next time it
> tries to use the object and drop the details out of its index
> automatically.
>
> -z is for rebuilding the folder structure if you erase any of them.
>
>>
>> 2011/07/06 13:29:29| Rebuilding storage in /var2/squid/cache (DIRTY)
>> 2011/07/06 13:29:29| Using Round Robin store dir selection
>> 2011/07/06 13:29:29| Current Directory is /
>> 2011/07/06 13:29:29| Loaded Icons.
>> 2011/07/06 13:29:29| Accepting intercepted HTTP connections at
>> 172.16.0.1:3128, FD 34.
>> 2011/07/06 13:29:29| HTCP Disabled.
>> 2011/07/06 13:29:29| Squid plugin modules loaded: 0
>> 2011/07/06 13:29:29| Adaptation support is off.
>> 2011/07/06 13:29:29| Ready to serve requests.
>>
>>
>> but no access to internet.
>>
>> What's wrong now?
>
> Normally service is down for the duration of the index rebuild.
> "Rebuilding storage in /var2/squid/cache (DIRTY)" is just the start
> message. Depending on your cache size and disk speed it may take some
> hours to complete.
>
> Not sure why its saying "Ready to serve requests" when the disk cache
> is not available yet. You override the waiting with -F command-line
> option?
>
> Amos

Thanks Amos for replying.

I did not override the waiting with -F. This cache directory is 200GB
not in the TB. I have this system running on a 3.2Ghz, 4GB RAM machine
and is taking for ever to rebuild the cache (almost 20hrs), don't know
why is taking so looong. I am thinking of deleting the entire cache but,
don't want to loose it all.

I will upgrade to the latest squid version and retry with that.
Received on Thu Jul 07 2011 - 14:36:58 MDT