RE: Store Object length

Hi Alex,

Sorry I did miss your post.

I just updated it on the squid-users website.
But here it goes again.

I had integrated a HTML patch from the squid devel site which will help the squid to do a proactive fetching of web pages in the background.
So I am storing the list of all the prefetched urls and go through them one by one and retrieve the object from the store/cache and send it over to the child squid when it is requested. This will help to save some time if there is a request to one of the links in near future.( It helps for our case as child and parent squids are sitting thousands of miles away).

On the whole, I will need to send the objects serially one after the other. So it would not be following a normal http request parse -> object fetch -> object sent back with http reply.

Instead my API will directly fetch from the cache given the url and send it across.
But I seem to be missing some logic in the original flow that causes an iterative pull from the cache until the entire object is retrieved. Which is what i want to know so that I can add that in my API.
Since I am setting the default size as 4k, for an object more than that the contents are truncated.

We are already into thick of our development that we do not want to move to higher versions as it will be a lot of work for us though we may consider for a version change in future.

Regards,
Anita

-----Original Message-----
From: Alex Rousskov [mailto:rousskov_at_measurement-factory.com]
Sent: 27 February 2013 00:34
To: Anita Sivakumar (WT01 - GMT-Telecom Equipment)
Cc: squid-dev_at_squid-cache.org
Subject: Re: Store Object length

Quoting earlier responses to your question in case you missed them:

>> On 25/02/2013 8:00 p.m., anita wrote:
>>> I am trying to write an API that will fetch objects from the squid cache
>>> given the url and http method.
>>> When I analyzed the code, I found that this can be achieved with a
>>> storeGetPublic(url, http_method) to see if the url has an entry in the
>>> cache.
>>> This followed by storeClientCopy() call.
>>> However, the StoreIOBuffer used here has been initialized with a
>>> lenght of HTTP_REQBUF_SZ (4096 or 4 k) by default.

> On 02/25/2013 04:45 AM, Amos Jeffries wrote:
>> Err. Yes storeGetPublic() is the API to do this with.
>>
>> Why are you writing a new one? what are you trying to do with Squid?

On 02/25/2013, Alex Rousskov wrote:
> I agree with Amos that these questions are important, for many reasons,
> ranging from compatibility with official code to being able to get
> support on squid-dev.
>
>
> To answer your buffer size question: The API does not restrict what
> buffer sizes the caller should use. Some client side code uses
> HTTP_REQBUF_SZ, but your code may use a different size. The store_client
> subscriber will receive callbacks when more data is available. It is a
> complex, iterative process; not a single "give me the whole response
> data" call.
>
> In theory, even a 1-byte buffer should work, but, besides being too
> slow, I suspect some Store code assumes that HTTP response headers will
> fit into the first buffer given to Store by store_client. Thus, it would
> be unwise to use smallish buffers without fixing Store first.
>
>
> Also, please note that storeGetPublic() returns an entry that is not
> necessarily fully cached. In the extreme case of collapsed forwarding,
> it returns an entry that is only expected to be cached when received
> from the origin server.

>>> I would like to know how larger objects are stored and retrieved from
>>> cache? How can I determine the length of the object read in that case?

> In general, you should not rely on knowing the response size a priori
> because some store modules may allow storage of objects where the size
> is not known when swap out begins (that is a good thing) and do not
> update the recorded size when swap out ends (that would be a performance
> optimization). I doubt Squid fully supports that today, but eventually
> it should, so it is better not to make a priori size knowledge a
> critical precondition in your code (if possible).
>
> Please describe what problem you are trying to solve and why you think
> you need a new API for what is already supported. With some luck,
> squid-dev folks will be able to help you along the way.
>
>
> Thank you,
>
> Alex.

>>> Version used: Squid ver 3.1.16
>>> squid.conf : default configuration file with cache_dir directive
>>> uncommented.
>>>
>>> Any pointers are appreciated.
>>
>> When playing with the Squid code please always upgrade to the most
>> recent you can before starting. Preferrably that would be 3.HEAD which
>> is our rolling development version.
>> 3.1 Squid series has been officially deprecated for about five months
>> now, and even 3.2 is nearing end of life.
>>
>> The Squid-3 series is one very long process of converting the code from
>> C code constructs to C++ objects APIs and pluggable mechanisms. So you
>> may find it much easier to code later releases, or that your fix has
>> already been done.
>>
>> Amos

Please do not print this email unless it is absolutely necessary.

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments.

WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.

www.wipro.com
Received on Fri Mar 01 2013 - 11:05:09 MST