Re: [squid-users] Squid 2.0 PATCH 2 - Falsely reporting no space on device.

On Fri, Oct 23, 1998 at 09:48:09AM +0100, John Sloan wrote:

> 1998/10/22 17:00:01| diskHandleWrite: FD 63: disk write error: (28) No
> space left on device
> 1998/10/22 17:00:01| storeSwapOutHandle: SwapOut failure (err code = -6).
> 1998/10/22 17:00:01| WARNING: Shrinking cache_dir #0 to 10804393 KB
> 1998/10/22 17:00:02| diskHandleWrite: FD 16: disk write error: (28) No
> space left on device
> 1998/10/22 17:00:02| storeDirWriteCleanLogs: Starting...
> 1998/10/22 17:00:02| WARNING: Closing open FD 38
> 1998/10/22 17:00:02| storeDirWriteCleanLogs:
> /squid/cache/swap.state.clean: write: (28) No space left on device
> 1998/10/22 17:00:02| storeDirWriteCleanLogs: Current swap logfile not
> replaced.
> 1998/10/22 17:00:02| Finished. Wrote 2727 entries.
> 1998/10/22 17:00:02| Took 0 seconds (2727.0 entries/sec).
> FATAL: Write failure -- check your disk space and cache.log
> Squid Cache (Version 2.0.PATCH2): Terminated abnormally.
> CPU Usage: 5310.950 seconds
> Maximum Resident Size: 0 KB
> Page faults with physical i/o: 87759
>
> And yet the disk was neither out of disk space nor of inodes. Is there
> another error condition which can cause this?

Let me guess? Solaris? Perhaps 2.6? Using standard UFS? If so, read on.

UFS can't handle the load squid puts on it very well. You'll want to, at
minimum, tunefs the filesystem to set minfree to 3% or less, and set
optimization to space rather than time.

The problems are caused by the severe fragmentation that UFS suffers from
under the workload squid imposes on it. During a quiet moment, run an "fsck -n"
on the raw device for the filesystem (you don't need to unmount it). I
bet you'll see very high fragmentation (ie 16% or so). Healthy fragmentation
is less than 1% :-(

It will take a couple of days for the fragmentation to decrease, it seems to
stablise at around 3-4%. You may need a couple of reconfigures in the
intermediate period to make squid try to use that additional space again.

Alternatively, you could newfs the filesystem with these parameters from the
start. Note that newfs under 2.6 is severely bust. Unless sun have released
a patch yet, it will almost certainly set maxcontig wrong, it will almost
certainlk set the speed (in rpm) wrong and it will almost certainly pick the
wrong number of cylinder groups. You'll want to correct these manually on
the newfs commandline.

Sun won't fix this (or are unwilling to at the moment). The only solution
they offer, beyond the workarounds above, is to use VxFS, which allows
online defragmentation to be carried out.

With minfree set to 1% and optimization for space being used, I've managed
to get the space lost down from around 20% to 6% on the 7 2Gb drives I'm
using. Not brilliant, but much better.

Oh, you probably want to put your swap log on a separate filesystem that
isn't going to randomly and prematurely fill up like this :-)

Cheers,

Chris

-- 
Chris Tilbury, UNIX Systems Administrator, IT Services, University of Warwick
EMAIL: cudch+s@csv.warwick.ac.uk PHONE: +44 1203 523365(V)/+44 1203 523267(F)
                            URL: http://www.warwick.ac.uk/staff/Chris.Tilbury