Re: [squid-users] COSS causing squid Segment Violation on FreeBSD 6.2S (store_io_coss.c)

On Thu, 14 Aug 2008, Mark Powell wrote:
> On Thu, 14 Aug 2008, Adrian Chadd wrote:
>
>> Don't load aio.ko; that way if squid tries using the POSIX AIO
>> routines it'll crash.
>>
>> I should get around to removing those from Squid-2.HEAD (as the
>> implementation isn't "good enough" for the current way Squid does disk
>> IO) and reintroduce it later in a more sensibly abstracted disk IO
>> layer.
>>
>> Yes, compile in aufs and coss together.

Well I found out what was causing the crashes. It was using the libthr
library instead of libpthread.
   Now I have my first FreeBSD 7 squid server. FreeBSD 7 has done away with
libpthread and libthr is all that seems to exist. Consequently the crashes
are back.
   Any ideas how I can get around this in FreeBSD 7?
   Many thanks.

>
> Ok. Thanks.
> I have both compiled in:
>
> # squid -v | xargs -n 1 echo | grep coss
> --enable-storeio=ufs diskd null aufs coss
>
> My COSS storage is definately less than disk size:
>
> # diskinfo amrd1
> amrd1 512 73274490880 143114240 8908 255 63
>
> Or 69.880MB.
>
> cache_dir coss /dev/amrd1 65000 max-size=16384 block-size=4096
>
> And AUFS on ufs on two more identical disks:
>
> cache_dir aufs /2 56000 16 256
> cache_dir aufs /3 56000 16 256
>
> That seems right?
> one last question.
> I've currently got /etc/libmap.conf to map threads libraries to libthr, as
> I've found them more reliable for other servers:
>
> # ldd /usr/local/sbin/squid
> /usr/local/sbin/squid:
> libcrypt.so.3 => /lib/libcrypt.so.3 (0xa8163000)
> libmd.so.3 => /lib/libmd.so.3 (0xa817c000)
> libm.so.4 => /lib/libm.so.4 (0xa818b000)
> libpthread.so.2 => /usr/lib/libthr.so.2 (0xa81a1000)
> libc.so.6 => /lib/libc.so.6 (0xa81b4000)
>
> Is that ok or should I just leave it to use libpthread?
> Cheers.
>
>>
>>
>> Adrian
>>
>>
>> 2008/8/14 Mark Powell <M.S.Powell_at_salford.ac.uk>:
>>> On Thu, 14 Aug 2008, Adrian Chadd wrote:
>>>
>>>> I'm not sure how much time I'm going to have to look at this code right
>>>> now.
>>>>
>>>> My priorities are elsewhere and you're the only person who is
>>>> currently reporting this sort of pending relocation issues with COSS
>>>> and FreeBSD-6.3.
>>>>
>>>> I'd suggest patching the source to propagate upward the async IO write
>>>> failure (ie - the errno) and log the reason in cache.log. You may find
>>>> its something simple like the disk size being different to your
>>>> configured cache_dir size.. :)
>>>
>>> NP. If it's just me then I'll have a play around and see if I can get a
>>> config that's stable.
>>> Can you confirm that the KLM aio.ko should or should not be loaded?
>>> But AUFS and COSS must be both be compiled into squid?
>>> Cheers.
>>>
>>>>
>>>>
>>>>
>>>> 2008/8/13 Mark Powell <M.S.Powell_at_salford.ac.uk>:
>>>>>
>>>>> Hi,
>>>>> Updated to 2.7.STABLE4. Curiosity got the better of me and I re-enabled
>>>>> COSS.
>>>>> The same failure still seems to be occuring i.e.
>>>>> 'storeCossCompletePendingReloc: got failure (-1)' and a segfault on
>>>>> FreeBSD
>>>>> 6.3-p4.
>>>>> Not sure the bug has been looked at since I filed it?
>>>>>
>>>>> http://www.squid-cache.org/bugs/show_bug.cgi?id=1944
>>>>>
>>>>> Does this mean COSS always going to be a no-go on FreeBSD?
>>>>> Many thanks.
>>>>>
>>>>> On Thu, 26 Apr 2007, Adrian Chadd wrote:
>>>>>>
>>>>>> On Thu, Apr 26, 2007, Mark Powell wrote:
>>>>>>
>>>>>>> It does stop the crashes, but I'm getting a lot of errors, so of which
>>>>>>> appear to be new ones:
>>>>>>
>>>>>> Very eww. Now we need to figure out why the hell its spitting out those
>>>>>> pending relocate
>>>>>> IO errors as its probably playing havoc with things.
>>>>>>
>>>>>> Could you do me a favour and stuff all of this into a Bugzilla bug
>>>>>> report?
>>>>>> Bugzilla is linked off the main squid website front page.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Adrian
>>>>>>
>>>>>>>
>>>>>>> 2007/04/26 15:12:17| COSS: /dev/amrd1: Rebuild Completed
>>>>>>> 2007/04/26 15:12:17| 81219 objects scanned, 144 objects relocated,
>>>>>>> 555
>>>>>>> objects fresher, 9346 objects ignored
>>>>>>> 2007/04/26 15:12:18| Completed Validation Procedure
>>>>>>> 2007/04/26 15:12:18| Validated 143160 Entries
>>>>>>> 2007/04/26 15:12:18| store_swap_size = 2576724k
>>>>>>> 2007/04/26 15:12:19| storeLateRelease: released 4996 objects
>>>>>>> 2007/04/26 15:12:21| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:29| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:29| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:29| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:29| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:36| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:38| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:38| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:38| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:38| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:38| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:38| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:38| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:38| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:38| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:38| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:38| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:38| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:38| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:39| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:39| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:39| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:39| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:39| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:39| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:12:39| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:13:18| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:13:31| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:13:31| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:13:32| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:13:32| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:13:32| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:13:32| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:13:32| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:13:32| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:13:32| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:13:33| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:13:41| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:14:23| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:14:35| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:14:35| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:15:00| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:15:24| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:15:33| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:15:45| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:15:45| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:15:56| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:15:56| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:15:56| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:15:56| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:15:56| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:15:56| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:15:57| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:15:57| squidaio_queue_request: WARNING - Queue
>>>>>>> congestion
>>>>>>> 2007/04/26 15:16:04| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:07| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:07| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:07| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:07| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:07| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:07| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:07| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:07| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:08| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:08| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:08| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:08| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:08| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:08| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:08| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:08| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:08| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:14| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:19| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:19| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:19| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:19| ctx: enter level 0:
>>>>>>> 'http://www.theaa.com/css/product_styles.css'
>>>>>>> 2007/04/26 15:16:19| WARNING: failed to unpack meta data
>>>>>>> 2007/04/26 15:16:19| ctx: exit level 0
>>>>>>> 2007/04/26 15:16:19| ctx: enter level 0:
>>>>>>> 'http://www.theaa.com/css/print.css'
>>>>>>> 2007/04/26 15:16:19| WARNING: failed to unpack meta data
>>>>>>> 2007/04/26 15:16:19| ctx: exit level 0
>>>>>>> 2007/04/26 15:16:19| ctx: enter level 0:
>>>>>>> 'http://www.theaa.com/css/aa_homepage_styles.css'
>>>>>>> 2007/04/26 15:16:19| WARNING: failed to unpack meta data
>>>>>>> 2007/04/26 15:16:19| ctx: exit level 0
>>>>>>> 2007/04/26 15:16:19| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:19| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:19| ctx: enter level 0:
>>>>>>> 'http://www.theaa.com/css/tools.css'
>>>>>>> 2007/04/26 15:16:19| WARNING: failed to unpack meta data
>>>>>>> 2007/04/26 15:16:19| ctx: exit level 0
>>>>>>> 2007/04/26 15:16:19| ctx: enter level 0:
>>>>>>> 'http://www.theaa.com/css/leftNav.css'
>>>>>>> 2007/04/26 15:16:19| WARNING: failed to unpack meta data
>>>>>>> 2007/04/26 15:16:19| ctx: exit level 0
>>>>>>> 2007/04/26 15:16:19| ctx: enter level 0:
>>>>>>> 'http://www.theaa.com/css/topnav.css'
>>>>>>> 2007/04/26 15:16:19| WARNING: failed to unpack meta data
>>>>>>> 2007/04/26 15:16:19| ctx: exit level 0
>>>>>>> 2007/04/26 15:16:19| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:19| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:19| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:19| ctx: enter level 0:
>>>>>>> 'http://www.theaa.com/javascript/metafaqs.js'
>>>>>>> 2007/04/26 15:16:19| WARNING: failed to unpack meta data
>>>>>>> 2007/04/26 15:16:19| ctx: exit level 0
>>>>>>> 2007/04/26 15:16:19| ctx: enter level 0:
>>>>>>> 'http://www.theaa.com/images/topnav/mainlogo.gif'
>>>>>>> 2007/04/26 15:16:19| WARNING: failed to unpack meta data
>>>>>>> 2007/04/26 15:16:19| ctx: exit level 0
>>>>>>> 2007/04/26 15:16:19| ctx: enter level 0:
>>>>>>> 'http://www.theaa.com/images/common/bg_body.png'
>>>>>>> 2007/04/26 15:16:19| WARNING: failed to unpack meta data
>>>>>>> 2007/04/26 15:16:19| ctx: exit level 0
>>>>>>> 2007/04/26 15:16:19| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:19| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:19| ctx: enter level 0:
>>>>>>> 'http://www.theaa.com/images/topnav/meta_faq_button.gif'
>>>>>>> 2007/04/26 15:16:19| WARNING: failed to unpack meta data
>>>>>>> 2007/04/26 15:16:19| ctx: exit level 0
>>>>>>> 2007/04/26 15:16:19| ctx: enter level 0:
>>>>>>> 'http://www.theaa.com/touchclarity/logging.js'
>>>>>>> 2007/04/26 15:16:19| WARNING: failed to unpack meta data
>>>>>>> 2007/04/26 15:16:19| ctx: exit level 0
>>>>>>> 2007/04/26 15:16:19| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:20| ctx: enter level 0:
>>>>>>> 'http://www.theaa.com/touchclarity/logging-code.js'
>>>>>>> 2007/04/26 15:16:20| WARNING: failed to unpack meta data
>>>>>>> 2007/04/26 15:16:20| ctx: exit level 0
>>>>>>> 2007/04/26 15:16:20| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:20| ctx: enter level 0:
>>>>>>> 'http://www.theaa.com/touchclarity/optimise.js'
>>>>>>> 2007/04/26 15:16:20| WARNING: failed to unpack meta data
>>>>>>> 2007/04/26 15:16:20| ctx: exit level 0
>>>>>>> 2007/04/26 15:16:20| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:20| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:20| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:20| storeCossCompletePendingReloc: got failure (-1)
>>>>>>> 2007/04/26 15:16:20| ctx: enter level 0:
>>>>>>> 'http://www.theaa.com/touchclarity/logging.html'
>>>>>>> 2007/04/26 15:16:20| WARNING: failed to unpack meta data
>>>>>>> 2007/04/26 15:16:20| ctx: exit level 0
>>>>>>> 2007/04/26 15:16:20| ctx: enter level 0:
>>>>>>>
>>>>>>> 'http://www.theaa.com/images/homepage/travel/travel_newsflash_title.gif'
>>>>>>> 2007/04/26 15:16:20| WARNING: failed to unpack meta data
>>>>>>> 2007/04/26 15:16:20| ctx: exit level 0
>>>>>>> 2007/04/26 15:16:20| ctx: enter level 0:
>>>>>>>
>>>>>>> 'http://www.theaa.com/images/homepage/travel/travel_newflash_button.gif'
>>>>>>> 2007/04/26 15:16:20| WARNING: failed to unpack meta data
>>>>>>> 2007/04/26 15:16:20| ctx: exit level 0
>>>>>>> 2007/04/26 15:16:20| ctx: enter level 0:
>>>>>>> 'http://www.theaa.com/images/homepage/travel/travel_newflash_bg.jpg'
>>>>>>> 2007/04/26 15:16:20| WARNING: failed to unpack meta data
>>>>>>>
>>>>>>> Cheers.
>>>>>>>
>>>>>>> --
>>>>>>> Mark Powell - UNIX System Administrator - The University of Salford
>>>>>>> Information Services Division, Clifford Whitworth Building,
>>>>>>> Salford University, Manchester, M5 4WT, UK.
>>>>>>> Tel: +44 161 295 4837 Fax: +44 161 295 5888 www.pgp.com for PGP key
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Mark Powell - UNIX System Administrator - The University of Salford
>>>>> Information Services Division, Clifford Whitworth Building,
>>>>> Salford University, Manchester, M5 4WT, UK.
>>>>> Tel: +44 161 295 6843 Fax: +44 161 295 5888 www.pgp.com for PGP key
>>>>>
>>>>>
>>>>
>>>
>>> --
>>> Mark Powell - UNIX System Administrator - The University of Salford
>>> Information Services Division, Clifford Whitworth Building,
>>> Salford University, Manchester, M5 4WT, UK.
>>> Tel: +44 161 295 6843 Fax: +44 161 295 5888 www.pgp.com for PGP key
>>>
>>>
>>
>>
>
>

-- 
Mark Powell - UNIX System Administrator - The University of Salford
Information Services Division, Clifford Whitworth Building,
Salford University, Manchester, M5 4WT, UK.
Tel: +44 161 295 6843  Fax: +44 161 295 5888  www.pgp.com for PGP key