Re: [squid-users] Jobs

On 29/06/11 19:28, Mohsen Pahlevanzadeh wrote:
> On Wed, 2011-06-29 at 02:32 +1200, Amos Jeffries wrote:
>> On 29/06/11 01:37, Mohsen Pahlevanzadeh wrote:
>>> On Tue, 2011-06-28 at 06:01 -0700, John Doe wrote:
>>>> From: Mohsen Pahlevanzadeh<mohsen_at_pahlevanzadeh.org>
>>>>
>>>>>>> We must write a program that along with normal tasks, it had do a
>>>>>>> variety of jobs,But i need to PURGE and insert cache.
>>>>> I must a write a web appl thet it manages squid with many extra job.
>>>>> normal job: every work that squid can do.
>>>>> Variety of job: a range of task that my boss ordered.
>>>>
>>>> Amos question: What are these "normal tasks" and "variety of jobs"?
>>>> Your answer: extra job, normal job, a variety of job, a range of tasks...
>>>> Which does not answer the question at all...
>>>> Can you name the main tasks/jobs you need to do?
>>>> By example: start/stop/restart/reload squid, reset cache, purge/cache url?
>>>> Graph statistics, etc...
>>>> I believe that for most of these, you do not need to play with the squid code...
>>>>
>>>> JD
>>> 1. PURGE from my program, but i can't call squidclient PURGE -m
>>> "blahblah" from my code.
>>
>> Sure you can. Several prefetchers just run exec("squidclient blah")
>>
>>
>> But now that it is clear you are building a whole management app, not
>> just a prefetcher an HTTP library would probably be the better way to
>> go. libcurl or whatever the equivalent is for your chosen language.
> It has problem. Because you suppose server has 2000 or more request
> concurrent, Then 2000 times squidclinet had run, it's wrong......
> So i must write a function same as purge and do it from code.
>>
>>> 2.Insert into cache.
>>
>> HTTP GET request. Tricky. Since you will have to figure out whether the
>> clients will be asking for plain or compressed copies.
>>
>> By far and away the best way to do this is simply not to bother doing it
>> at all. Squid is designed to do the work of figuring out where objects
>> are and how to get them to the client fastest.
>>
>> Inserting objects into the cache may _seem_ to be a good idea. But HTTP
>> is very complicated and there is a very good chance you will push the
>> wrong variants of each object into the cache.
>>
> Your suggestion is very very nisce and i'll start HTTP Definitive Guide
> of O'Reilly.I know libcurl, it's good idea that we request to squid for
> push and squid itself push.

My point earlier was to use libcurl for PURGE as well. No difference to
you between PURGE and GET other than the name.

>
>>> 3.concurrent receive of site(minimum 100 sites)
>>
>> If by "site" you mean website. Squid is used by ISP. They have
>> accessible site numbers ranging in the high millions or billions. These
>> are all concurrently available to an ISP situation, so safe bet on that
>> requirement.
>>
>> If by "site" you mean visitor. One Squid routinely handles hundreds or
>> thousands of clients depending on your hardware specs. Or it may
>> overload the network on _one_ client requesting TB sized objects.
>>
>> You need to figure out a request/time-unit metric or a concurrent
>> connections metric and test that is achievable with the desired
>> configuration. The squid config file is a mix between simple
>> on/off/value settings and a big script which tells Squid how to operate
>> on a request. Seemingly simple changes can easily raise or lower the
>> response speed by whole orders of magnitude.
>>
> We must do following task:
> 1. Possibility to download concurrent 100 sites (website) from
> internet.
> 2.Possibility to filter them that i know squid uses url_regex.
> 3.Possibility to answer as proxy for minimum 1000 concurrent request.
> 4.Time of answering to each request must be 100 mili second.

Possible trouble.

  If you were setting up a reverse-proxy situation with controlled
network between the proxy and web servers that could be a realistic
goal. In those cases it is relatively easy to get a lot of requests in
the low dozens of millisec.

However your earlier requirement (1), indicates that your traffic will
be from the Internet. You will be 100% at the mercy of external
administrators when it comes to timing.

> 5.Space and algorithm should be chosen that can expand to store 10000000
> pages.

Sounds like you are tasked with creating a whole new proxy system. That
was all tasks _your app_ has to do was it not?

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.12
   Beta testers wanted for 3.2.0.9 and 3.1.12.3