Re: [squid-users] WARNING: An error inside Squid has caused an HTTP reply without Date

On 7/7/2011 10:42 AM, Wilson Hernandez wrote:
> On 7/7/2011 1:25 AM, Amos Jeffries wrote:
>> On 07/07/11 05:32, Wilson Hernandez wrote:
>>> On 7/6/2011 1:08 PM, Wilson Hernandez wrote:
>>>> On 7/6/2011 12:28 PM, Wilson Hernandez wrote:
>>>>> Hello.
>>>>>
>>>>> I have a partition for squid that is almost full (I'm getting a
>>>>> message):
>>>>> WARNING: An error inside Squid has caused an HTTP reply without
>>>>> Date:. Please report this
>>>>> 2011/07/06 12:23:35| DiskThreadsDiskFile::openDone: (2) No such file
>>>>> or directory
>>>>> 2011/07/06 12:23:35| /var2/squid/cache/18/A6/0057F5D8
>>>>> 2011/07/06 12:23:35| DiskThreadsDiskFile::openDone: (2) No such file
>>>>> or directory
>>>>> 2011/07/06 12:23:35| /var2/squid/cache/18/A6/0057F5E4
>>>>> 2011/07/06 12:23:35| clientProcessRequest: Invalid Request
>>>>> 2011/07/06 12:23:40| WARNING: Disk space over limit: 3615331284 KB >
>>>>> 204800000 KB
>>>>> 2011/07/06 12:23:41| DiskThreadsDiskFile::openDone: (2) No such file
>>>>> or directory
>>>>> 2011/07/06 12:23:41| /var2/squid/cache/18/A9/0057F8BD
>>>>> 2011/07/06 12:23:42| DiskThreadsDiskFile::openDone: (2) No such file
>>>>> or directory
>>>>> 2011/07/06 12:23:42| /var2/squid/cache/18/A9/0057F8DC
>>>>> 2011/07/06 12:23:43| could not parse headers from on disk structure!
>>>>> 2011/07/06 12:23:43| WARNING: An error inside Squid has caused an
>>>>> HTTP reply without Date:. Please report this
>>>>> 2011/07/06 12:23:44| DiskThreadsDiskFile::openDone: (2) No such file
>>>>> or directory
>>>>> 2011/07/06 12:23:44| /var2/squid/cache/18/A9/0057F8E8
>>>>> 2011/07/06 12:23:46| DiskThreadsDiskFile::openDone: (2) No such file
>>>>> or directory
>>>>> 2011/07/06 12:23:46| /var2/squid/cache/18/A9/0057F923
>>>>> 2011/07/06 12:23:51| WARNING: Disk space over limit: 3615319720 KB >
>>>>> 204800000 KB
>>>>> 2011/07/06 12:24:02| WARNING: Disk space over limit: 3615308076 KB >
>>>>> 204800000 KB
>>>>> 2011/07/06 12:24:07| could not parse headers from on disk structure!
>>>>> 2011/07/06 12:24:07| WARNING: An error inside Squid has caused an
>>>>> HTTP reply without Date:. Please report this
>>>>> 2011/07/06 12:24:09| DiskThreadsDiskFile::openDone: (2) No such file
>>>>> or directory
>>>>> 2011/07/06 12:24:09| /var2/squid/cache/3C/0B/003B93E7
>>>>> 2011/07/06 12:24:09| DiskThreadsDiskFile::openDone: (2) No such file
>>>>> or directory
>>>>> 2011/07/06 12:24:09| /var2/squid/cache/13/B8/0053126B
>>>>> 2011/07/06 12:24:10| DiskThreadsDiskFile::openDone: (2) No such file
>>>>> or directory
>>>>> 2011/07/06 12:24:10| /var2/squid/cache/00/68/00006868
>>>>> 2011/07/06 12:24:10| could not parse headers from on disk structure!
>>>>> 2011/07/06 12:24:10| WARNING: An error inside Squid has caused an
>>>>> HTTP reply without Date:. Please report this
>>>>> 2011/07/06 12:24:13| WARNING: Disk space over limit: 3615298144 KB >
>>>>> 204800000 KB
>>>>> 2011/07/06 12:24:14| could not parse headers from on disk structure!
>>>>> 2011/07/06 12:24:14| WARNING: An error inside Squid has caused an
>>>>> HTTP reply without Date:. Please report this
>>>>> 2011/07/06 12:24:14| could not parse headers from on disk structure!
>>>>> 2011/07/06 12:24:14| WARNING: An error inside Squid has caused an
>>>>> HTTP reply without Date:. Please report this
>>>>> 2011/07/06 12:24:14| could not parse headers from on disk structure!
>>>>> 2011/07/06 12:24:14| WARNING: An error inside Squid has caused an
>>>>> HTTP reply without Date:. Please report this
>>>>> 2011/07/06 12:24:14| DiskThreadsDiskFile::openDone: (2) No such file
>>>>> or directory
>>>>> 2011/07/06 12:24:14| /var2/squid/cache/11/0D/00506C25
>>>>> 2011/07/06 12:24:16| could not parse headers from on disk structure!
>>>>> 2011/07/06 12:24:16| WARNING: An error inside Squid has caused an
>>>>> HTTP reply without Date:. Please report this
>>>>> 2011/07/06 12:24:23| could not parse headers from on disk structure!
>>>>> 2011/07/06 12:24:23| WARNING: An error inside Squid has caused an
>>>>> HTTP reply without Date:. Please report this
>>>>> 2011/07/06 12:24:24| WARNING: Disk space over limit: 3615285372 KB >
>>>>> 204800000 KB
>>>>>
>>>>>
>>>>> I don't know if this is caused by disk space or what. If is caused by
>>>>> disk space how can I clean old cached files from the squid's cache?
>>
>> Looks like something other than Squid is playing with the cache files
>> on disk.
>> "No such file or directory" - something erased a file without
>> informing Squid. Though it could be that Squid scheduled the file for
>> erase and the OS did not get around to it until after Squid was
>> re-using the same fileno/name for another object. (rare condition but
>> possible).
>>
>> "Disk space over limit" - seriously nasty. The object size accounting
>> in Squid has been screwed up somehow.
>> Squid will divert resources to erasing that extra 3.4 _TB_ of
>> unwanted cache content ASAP. I'm a little surprised to see numbers
>> that far out in 3.1.11, AFAIK the cache accounting was fixed in an
>> earlier 3.1 release.
>>
>>
>> "could not parse headers from on disk structure!" and "WARNING: An
>> error inside Squid has caused an HTTP reply without Date:."
>> - strong signs of either disk corruption, a bug, or objects stored
>> by some older Squid version without Date: headers. We are still
>> trying to track down which. If you can move to 3.1.12 there are some
>> extra details logged about those objects to help figure this out.
>>
>>
>>>>>
>>>>> Thanks.
>>>>
>>>> /usr/local/squid/sbin/squid -v
>>>> Squid Cache: Version 3.1.11
>>>> configure options: '--prefix=/usr/local/squid'
>>>> '--sysconfdir=/etc/squid' '--enable-delay-pools'
>>>> '--enable-kill-parent-hack' '--disable-htcp'
>>>> '--enable-default-err-language=es' '--enable-linux-netfilter'
>>>> '--disable-ident-lookups' '--localstatedir=/var/log/squid'
>>>> '--enable-stacktraces' '--with-default-user=proxy'
>>>> '--with-large-files' '--enable-icap-client' '--enable-async-io'
>>>> '--enable-storeio=aufs' '--enable-removal-policies=heap,lru'
>>>> '--with-maxfd=65536' --with-squid=/workingdir/squid-3.1.11
>>>> --enable-ltdl-convenience
>>>>
>>>>
>>> Hello.
>>>
>>> I deleted the swap.state file and re-run squid -z and get the
>>> following:
>>
>> _just_ the swap.state erase does nothing but trigger a full disk scan
>> (DIRTY re-build) if Squid was stopped at the time. Otherwise it has
>> no effect.
>>
>> Erasing individual cache data files is pretty safe. Squid will log
>> that "openDone: (2) No such file or directory" message next time it
>> tries to use the object and drop the details out of its index
>> automatically.
>>
>> -z is for rebuilding the folder structure if you erase any of them.
>>
>>>
>>> 2011/07/06 13:29:29| Rebuilding storage in /var2/squid/cache (DIRTY)
>>> 2011/07/06 13:29:29| Using Round Robin store dir selection
>>> 2011/07/06 13:29:29| Current Directory is /
>>> 2011/07/06 13:29:29| Loaded Icons.
>>> 2011/07/06 13:29:29| Accepting intercepted HTTP connections at
>>> 172.16.0.1:3128, FD 34.
>>> 2011/07/06 13:29:29| HTCP Disabled.
>>> 2011/07/06 13:29:29| Squid plugin modules loaded: 0
>>> 2011/07/06 13:29:29| Adaptation support is off.
>>> 2011/07/06 13:29:29| Ready to serve requests.
>>>
>>>
>>> but no access to internet.
>>>
>>> What's wrong now?
>>
>> Normally service is down for the duration of the index rebuild.
>> "Rebuilding storage in /var2/squid/cache (DIRTY)" is just the start
>> message. Depending on your cache size and disk speed it may take some
>> hours to complete.
>>
>> Not sure why its saying "Ready to serve requests" when the disk cache
>> is not available yet. You override the waiting with -F command-line
>> option?
>>
>> Amos
> Forgot to mention:
>
> I am running squid along with squidguard and I don't think it messes
> with squid's cache directory... This cache was used by an older squid
> version (3.1.9) that maybe messed it up.
>
> Any recommendation?
>
>
Just installed ver 3.1.14 and get the same results, no internet and is
taking a looong time for swap.state to complete.

2011/07/07 10:52:27| Starting Squid Cache version 3.1.14 for
i686-pc-linux-gnu...
2011/07/07 10:52:27| Process ID 13441
2011/07/07 10:52:27| With 32768 file descriptors available
2011/07/07 10:52:27| Initializing IP Cache...
2011/07/07 10:52:27| DNS Socket created at [::], FD 7
2011/07/07 10:52:27| DNS Socket created at 0.0.0.0, FD 8
2011/07/07 10:52:27| Adding nameserver 172.16.0.2 from squid.conf
2011/07/07 10:52:27| Adding nameserver 172.16.0.1 from squid.conf
2011/07/07 10:52:27| helperOpenServers: Starting 10/10 'zapchain' processes
2011/07/07 10:52:27| Unlinkd pipe opened on FD 32
2011/07/07 10:52:27| Store logging disabled
2011/07/07 10:52:27| Swap maxSize 204800000 + 1048576 KB, estimated
3216384 objects
2011/07/07 10:52:27| Target number of buckets: 160819
2011/07/07 10:52:27| Using 262144 Store buckets
2011/07/07 10:52:27| Max Mem size: 1048576 KB
2011/07/07 10:52:27| Max Swap size: 204800000 KB
2011/07/07 10:52:27| Rebuilding storage in /var2/squid/cache (DIRTY)
2011/07/07 10:52:27| Using Round Robin store dir selection
2011/07/07 10:52:27| Current Directory is /workingdir/squid-3.1.14
2011/07/07 10:52:27| Loaded Icons.
2011/07/07 10:52:27| Accepting intercepted HTTP connections at
172.16.0.1:3128, FD 34.
2011/07/07 10:52:27| HTCP Disabled.
2011/07/07 10:52:27| Squid plugin modules loaded: 0
2011/07/07 10:52:27| Adaptation support is off.
2011/07/07 10:52:27| Ready to serve requests.
Received on Thu Jul 07 2011 - 14:58:54 MDT