Re: Squid performance wish-list

> This I find dangerously low. Think everyone having 34Mb/s connections
> and everyone pulling in MPEG movies.

If 1GB per file is too low we can can a double indirect pointer, ie. a pointer
to a block of pointers to blocks of pointers to data. That will cover the
extremely large file case. Tell you what, in my code I'll leave this open
to be coded up later if and when needed.

> On another note, what is to be gained by using direct and indirect
> block pointers?
> How about a much simpler scheme, a linked list like:
>
> Say, 1KB block sizes, chunks consist of one or more blocks,
> whereas every chunk starts with:
>
> [ 1-byte: misc flags (reserved) ]
> [ 3-bytes: length of this chunk in bytes ]
> [ 4-bytes: blocknumber of the next chunk in the chain ]
> [ 1016-bytes: data for this chunk ]
> [ 1024-bytes: optional data to extend this chunk (no header) ]

The reason is for file deletions!!
All the blocks of a file have to marked as free. If we can't quickly index
all the blocks by having simple pointers to chunks of data then we will have
to traverse the entire file to find all the blocks that belog to that file so
we can mark them as free. Linked-lists equals VERY slow deletes. Also it
means we can't do lseeks fast either (although this may be not used very
often for Squid - if at all).

> I'd say this is simpler, more space efficient, more extensible, and
> faster than the FS you're proposing?

You've proposed a more complex filesystem to maintain. It may be more space
efficient. With 512byte chunks there's 8% internal block fragmentation,
with 1K chunks there's 11% internal frag, with 4K blocks there's 17%.

As for faster, having small blocks will be slower in EVERY reespect. It will
mean the possible need to move the heads for every access of a 512 byte block
instead of for 8K.

Tell you what, go read the Berkely demon book on BSD design and the FFS (UFS)
filesystem to understand the concepts I'm trying to resolve here.

        Stew.

-- 
Stewart Forster (Snr. Development Engineer)
connect.com.au pty ltd, Level 9, 114 Albert Rd, Sth Melbourne, VIC 3205, Aust.
Email: slf@connect.com.au   Phone: +61 3 9251-3684   Fax: +61 3 9251-3666