Re: Store Object length

On 03/01/2013 04:04 AM, anita.sivakumar_at_wipro.com wrote:

> I am storing the list of all the prefetched urls and
> go through them one by one

OK.

> and retrieve the object from the store/cache

That part is puzzling. See below.

> On the whole, I will need to send the objects serially one after the
> other. So it would not be following a normal http request parse ->
> object fetch -> object sent back with http reply.

I am not sure I understand the "send serially" part of this. My
understanding is is your prefetching Squid will:

  1. Find a URL to prefetch in the response.
  2. Create a fake HTTP request for that URL.
  3. Fetch that URL using regular server-side code,
     which may also cache the response.

The above should work without many Store changes, similar to quick_abort
handling. Your prefetching code should not need to get anything from the
Store (until the client requests the prefetched object, but that
transaction will just use the existing code).

> Instead my API will directly fetch from the cache given the url and send it across.

That part I do not understand. It sounds like you might be trying to do
two separate things:

  1. Prefetch into the parent cache.
  2. Push prefetched objects into the child cache.

If that is the case, can you explain why not just make the child cache
(or both caches) prefetch? Why the push?

> We are already into thick of our development that we do not want to
> move to higher versions as it will be a lot of work for us though we
> may consider for a version change in future.

Are you planning on submitting your changes for the inclusion into the
official sources?

Thank you,

Alex.

> -----Original Message-----
> From: Alex Rousskov [mailto:rousskov_at_measurement-factory.com]
> Sent: 27 February 2013 00:34
> To: Anita Sivakumar (WT01 - GMT-Telecom Equipment)
> Cc: squid-dev_at_squid-cache.org
> Subject: Re: Store Object length
>
> Quoting earlier responses to your question in case you missed them:
>
>>> On 25/02/2013 8:00 p.m., anita wrote:
>>>> I am trying to write an API that will fetch objects from the squid cache
>>>> given the url and http method.
>>>> When I analyzed the code, I found that this can be achieved with a
>>>> storeGetPublic(url, http_method) to see if the url has an entry in the
>>>> cache.
>>>> This followed by storeClientCopy() call.
>>>> However, the StoreIOBuffer used here has been initialized with a
>>>> lenght of HTTP_REQBUF_SZ (4096 or 4 k) by default.
>
>> On 02/25/2013 04:45 AM, Amos Jeffries wrote:
>>> Err. Yes storeGetPublic() is the API to do this with.
>>>
>>> Why are you writing a new one? what are you trying to do with Squid?
>
> On 02/25/2013, Alex Rousskov wrote:
>> I agree with Amos that these questions are important, for many reasons,
>> ranging from compatibility with official code to being able to get
>> support on squid-dev.
>>
>>
>> To answer your buffer size question: The API does not restrict what
>> buffer sizes the caller should use. Some client side code uses
>> HTTP_REQBUF_SZ, but your code may use a different size. The store_client
>> subscriber will receive callbacks when more data is available. It is a
>> complex, iterative process; not a single "give me the whole response
>> data" call.
>>
>> In theory, even a 1-byte buffer should work, but, besides being too
>> slow, I suspect some Store code assumes that HTTP response headers will
>> fit into the first buffer given to Store by store_client. Thus, it would
>> be unwise to use smallish buffers without fixing Store first.
>>
>>
>> Also, please note that storeGetPublic() returns an entry that is not
>> necessarily fully cached. In the extreme case of collapsed forwarding,
>> it returns an entry that is only expected to be cached when received
>> from the origin server.
>
>>>> I would like to know how larger objects are stored and retrieved from
>>>> cache? How can I determine the length of the object read in that case?
>
>> In general, you should not rely on knowing the response size a priori
>> because some store modules may allow storage of objects where the size
>> is not known when swap out begins (that is a good thing) and do not
>> update the recorded size when swap out ends (that would be a performance
>> optimization). I doubt Squid fully supports that today, but eventually
>> it should, so it is better not to make a priori size knowledge a
>> critical precondition in your code (if possible).
>>
>> Please describe what problem you are trying to solve and why you think
>> you need a new API for what is already supported. With some luck,
>> squid-dev folks will be able to help you along the way.
>>
>>
>> Thank you,
>>
>> Alex.
>
>
>>>> Version used: Squid ver 3.1.16
>>>> squid.conf : default configuration file with cache_dir directive
>>>> uncommented.
>>>>
>>>> Any pointers are appreciated.
>>>
>>> When playing with the Squid code please always upgrade to the most
>>> recent you can before starting. Preferrably that would be 3.HEAD which
>>> is our rolling development version.
>>> 3.1 Squid series has been officially deprecated for about five months
>>> now, and even 3.2 is nearing end of life.
>>>
>>> The Squid-3 series is one very long process of converting the code from
>>> C code constructs to C++ objects APIs and pluggable mechanisms. So you
>>> may find it much easier to code later releases, or that your fix has
>>> already been done.
>>>
>>> Amos
>
>
>
> Please do not print this email unless it is absolutely necessary.
>
> The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments.
>
> WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.
>
> www.wipro.com
>
Received on Fri Mar 01 2013 - 15:25:14 MST