Re: [squid-users] WARNING: An error inside Squid has caused an HTTP reply without Date

On 07/07/11 05:32, Wilson Hernandez wrote:
> On 7/6/2011 1:08 PM, Wilson Hernandez wrote:
>> On 7/6/2011 12:28 PM, Wilson Hernandez wrote:
>>> Hello.
>>>
>>> I have a partition for squid that is almost full (I'm getting a
>>> message):
>>> WARNING: An error inside Squid has caused an HTTP reply without
>>> Date:. Please report this
>>> 2011/07/06 12:23:35| DiskThreadsDiskFile::openDone: (2) No such file
>>> or directory
>>> 2011/07/06 12:23:35| /var2/squid/cache/18/A6/0057F5D8
>>> 2011/07/06 12:23:35| DiskThreadsDiskFile::openDone: (2) No such file
>>> or directory
>>> 2011/07/06 12:23:35| /var2/squid/cache/18/A6/0057F5E4
>>> 2011/07/06 12:23:35| clientProcessRequest: Invalid Request
>>> 2011/07/06 12:23:40| WARNING: Disk space over limit: 3615331284 KB >
>>> 204800000 KB
>>> 2011/07/06 12:23:41| DiskThreadsDiskFile::openDone: (2) No such file
>>> or directory
>>> 2011/07/06 12:23:41| /var2/squid/cache/18/A9/0057F8BD
>>> 2011/07/06 12:23:42| DiskThreadsDiskFile::openDone: (2) No such file
>>> or directory
>>> 2011/07/06 12:23:42| /var2/squid/cache/18/A9/0057F8DC
>>> 2011/07/06 12:23:43| could not parse headers from on disk structure!
>>> 2011/07/06 12:23:43| WARNING: An error inside Squid has caused an
>>> HTTP reply without Date:. Please report this
>>> 2011/07/06 12:23:44| DiskThreadsDiskFile::openDone: (2) No such file
>>> or directory
>>> 2011/07/06 12:23:44| /var2/squid/cache/18/A9/0057F8E8
>>> 2011/07/06 12:23:46| DiskThreadsDiskFile::openDone: (2) No such file
>>> or directory
>>> 2011/07/06 12:23:46| /var2/squid/cache/18/A9/0057F923
>>> 2011/07/06 12:23:51| WARNING: Disk space over limit: 3615319720 KB >
>>> 204800000 KB
>>> 2011/07/06 12:24:02| WARNING: Disk space over limit: 3615308076 KB >
>>> 204800000 KB
>>> 2011/07/06 12:24:07| could not parse headers from on disk structure!
>>> 2011/07/06 12:24:07| WARNING: An error inside Squid has caused an
>>> HTTP reply without Date:. Please report this
>>> 2011/07/06 12:24:09| DiskThreadsDiskFile::openDone: (2) No such file
>>> or directory
>>> 2011/07/06 12:24:09| /var2/squid/cache/3C/0B/003B93E7
>>> 2011/07/06 12:24:09| DiskThreadsDiskFile::openDone: (2) No such file
>>> or directory
>>> 2011/07/06 12:24:09| /var2/squid/cache/13/B8/0053126B
>>> 2011/07/06 12:24:10| DiskThreadsDiskFile::openDone: (2) No such file
>>> or directory
>>> 2011/07/06 12:24:10| /var2/squid/cache/00/68/00006868
>>> 2011/07/06 12:24:10| could not parse headers from on disk structure!
>>> 2011/07/06 12:24:10| WARNING: An error inside Squid has caused an
>>> HTTP reply without Date:. Please report this
>>> 2011/07/06 12:24:13| WARNING: Disk space over limit: 3615298144 KB >
>>> 204800000 KB
>>> 2011/07/06 12:24:14| could not parse headers from on disk structure!
>>> 2011/07/06 12:24:14| WARNING: An error inside Squid has caused an
>>> HTTP reply without Date:. Please report this
>>> 2011/07/06 12:24:14| could not parse headers from on disk structure!
>>> 2011/07/06 12:24:14| WARNING: An error inside Squid has caused an
>>> HTTP reply without Date:. Please report this
>>> 2011/07/06 12:24:14| could not parse headers from on disk structure!
>>> 2011/07/06 12:24:14| WARNING: An error inside Squid has caused an
>>> HTTP reply without Date:. Please report this
>>> 2011/07/06 12:24:14| DiskThreadsDiskFile::openDone: (2) No such file
>>> or directory
>>> 2011/07/06 12:24:14| /var2/squid/cache/11/0D/00506C25
>>> 2011/07/06 12:24:16| could not parse headers from on disk structure!
>>> 2011/07/06 12:24:16| WARNING: An error inside Squid has caused an
>>> HTTP reply without Date:. Please report this
>>> 2011/07/06 12:24:23| could not parse headers from on disk structure!
>>> 2011/07/06 12:24:23| WARNING: An error inside Squid has caused an
>>> HTTP reply without Date:. Please report this
>>> 2011/07/06 12:24:24| WARNING: Disk space over limit: 3615285372 KB >
>>> 204800000 KB
>>>
>>>
>>> I don't know if this is caused by disk space or what. If is caused by
>>> disk space how can I clean old cached files from the squid's cache?

Looks like something other than Squid is playing with the cache files on
disk.
  "No such file or directory" - something erased a file without
informing Squid. Though it could be that Squid scheduled the file for
erase and the OS did not get around to it until after Squid was re-using
the same fileno/name for another object. (rare condition but possible).

"Disk space over limit" - seriously nasty. The object size accounting in
Squid has been screwed up somehow.
  Squid will divert resources to erasing that extra 3.4 _TB_ of unwanted
cache content ASAP. I'm a little surprised to see numbers that far out
in 3.1.11, AFAIK the cache accounting was fixed in an earlier 3.1 release.

"could not parse headers from on disk structure!" and "WARNING: An error
inside Squid has caused an HTTP reply without Date:."
  - strong signs of either disk corruption, a bug, or objects stored by
some older Squid version without Date: headers. We are still trying to
track down which. If you can move to 3.1.12 there are some extra details
logged about those objects to help figure this out.

>>>
>>> Thanks.
>>
>> /usr/local/squid/sbin/squid -v
>> Squid Cache: Version 3.1.11
>> configure options: '--prefix=/usr/local/squid'
>> '--sysconfdir=/etc/squid' '--enable-delay-pools'
>> '--enable-kill-parent-hack' '--disable-htcp'
>> '--enable-default-err-language=es' '--enable-linux-netfilter'
>> '--disable-ident-lookups' '--localstatedir=/var/log/squid'
>> '--enable-stacktraces' '--with-default-user=proxy'
>> '--with-large-files' '--enable-icap-client' '--enable-async-io'
>> '--enable-storeio=aufs' '--enable-removal-policies=heap,lru'
>> '--with-maxfd=65536' --with-squid=/workingdir/squid-3.1.11
>> --enable-ltdl-convenience
>>
>>
> Hello.
>
> I deleted the swap.state file and re-run squid -z and get the following:

_just_ the swap.state erase does nothing but trigger a full disk scan
(DIRTY re-build) if Squid was stopped at the time. Otherwise it has no
effect.

Erasing individual cache data files is pretty safe. Squid will log that
"openDone: (2) No such file or directory" message next time it tries to
use the object and drop the details out of its index automatically.

-z is for rebuilding the folder structure if you erase any of them.

>
> 2011/07/06 13:29:29| Rebuilding storage in /var2/squid/cache (DIRTY)
> 2011/07/06 13:29:29| Using Round Robin store dir selection
> 2011/07/06 13:29:29| Current Directory is /
> 2011/07/06 13:29:29| Loaded Icons.
> 2011/07/06 13:29:29| Accepting intercepted HTTP connections at
> 172.16.0.1:3128, FD 34.
> 2011/07/06 13:29:29| HTCP Disabled.
> 2011/07/06 13:29:29| Squid plugin modules loaded: 0
> 2011/07/06 13:29:29| Adaptation support is off.
> 2011/07/06 13:29:29| Ready to serve requests.
>
>
> but no access to internet.
>
> What's wrong now?

Normally service is down for the duration of the index rebuild.
"Rebuilding storage in /var2/squid/cache (DIRTY)" is just the start
message. Depending on your cache size and disk speed it may take some
hours to complete.

Not sure why its saying "Ready to serve requests" when the disk cache is
not available yet. You override the waiting with -F command-line option?

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.14
   Beta testers wanted for 3.2.0.9