Re: memory-mapped files in Squid

On 29 Jan 99, at 0:49, Henrik Nordstrom <hno@hem.passagen.se> wrote:

> Andres Kroonmaa wrote:
>
> > Yes, of course, mentioned that. But this would be too simplistic.
> > It is hard to believe that fifo as Replacement Policy could be very
> > effective in both reducing network traffic and disk utilisation.
>
 [..]
 agree with everything.

> > This basically means that we do NOT want to blindly overwrite objects
> > that could be useful and longterm. And this leads us to unavoidable
>
> I am not exactly proposing to blindly overwrite objects but at the same
> time I don't agree in that a blind FIFO is to simplistic. There are
> however several approaches which can be used to preserve useful
> information without a serious I/O impact. You could say that what I am
> trying to do is to have a FIFO mimic some of the properties of LRU, and
> if neccesary with some unused space optimization, but my epasis is on
> optimized write performance.
 
 In fact, to mimic LRU should be easy. Whenever we fetch object from disk
 to service client hit, we can append that object to the write queue, that
 way moving all accessed objects to the front of LRU. This would make fifo
 into real LRU. With small objects it could even add little overhead.
 
 How are you planning to implement the metadata?
 We can integrate IO routines quite tightly with squid. This allows us to
 do tricks ordinary FS's can't, like let squidFS decide the file ID, based
 on (currently) optimal location. We could also allow squidFS to change the
 file ID if practical and notify squid about that fact.

 In fact, we could pack objects much tighter together by using allocation
 granularity of say 512 bytes. Into file ID code the chunk number and
 offset, from Squid object map we know the size, and the object itself
 should have some crosscheck prepended.
 Hmm, this way we could always write almost any-size object sequentially,
 assign it a number, and from this number alone we can always find the
 object on disk, and we need to know only a start of it. Fragmentation
 totally eliminated.
 We can manage many chunks concurrently in memory, to try place related
 objects together, eg. by URL hostname, this way also optimisting read
 performance. With sufficient ram we can manage hundreds of chunks
 concurrently. But then we'd need to cope with partially filled chunks..
 Anyway, we could just write them all out in sequence, in whatever order,
 they are still all pretty close to each other, at least within a cylinder
 or few.

 I'm starting to like the beauty of its simplicity...

> It would be nice if someone with some real life logs could calculate the
> ratio of TCP_REFRESH_HIT to TCP_REFRESH_MISS to guide me in my future
> thoughts.

      Bytes Request type Resolve Method Count Count-% Bytes-%
 5995707919 TCP_MISS DIRECT 674345 32.68 % 47.71 %
 2873310741 TCP_HIT NONE 367157 17.79 % 22.86 %
 1348208061 TCP_MISS TIMEOUT_DIRECT 94084 4.56 % 10.73 %
  992042854 TCP_MISS SIBLING_HIT 143092 6.93 % 7.89 %
  398727919 ERR_CLIENT_ABORT DIRECT 43125 2.09 % 3.17 %
  311738853 TCP_CLIENT_REFRESH DIRECT 68694 3.33 % 2.48 %
  191591763 TCP_IMS_HIT NONE 147684 7.16 % 1.52 %
  152717453 TCP_REFRESH_MISS DIRECT 13799 0.67 % 1.22 %
  120455743 TCP_REFRESH_HIT DIRECT 16608 0.80 % 0.96 %
   47846534 TCP_MISS NONE 11374 0.55 % 0.38 %

 But this cache is hardly able to cache a 1-days worth of objects. So the
 amount of refreshes is expected low.

 ----------------------------------------------------------------------
  Andres Kroonmaa mail: andre@online.ee
  Network Manager
  Organization: MicroLink Online Tel: 6308 909
  Tallinn, Sakala 19 Pho: +372 6308 909
  Estonia, EE0001 http://www.online.ee Fax: +372 6308 901
 ----------------------------------------------------------------------
Received on Tue Jul 29 2003 - 13:15:56 MDT