Re: pseudo-specs for a String class: tokenization

On Thu, Sep 4, 2008 at 7:30 PM, Alex Rousskov
<rousskov_at_measurement-factory.com> wrote:
> On Thu, 2008-09-04 at 18:00 +0200, Kinkie wrote:
>
>> > BTW, this is yet another case where a Tokenizer class would be better
>> > than let-String-do-everything approach because a tokenizer object can at
>> > any time return the current token, the current delimiter, and/or both,
>> > without performance overhead or design complications.
>>
>> There is no such thing as "current delimiter"; it's supplied by the
>> caller each time.
>
> According to your documentation the caller supplies a _set_ of delimiter
> characters. Thus, the current or actual delimiter (i.e., the actual
> character at the end of the returned token, if any) is unknown to the
> caller if the caller used a multi-character delimiter set:

Yes.

You convinced me, but for a different reason.
If the Tokenizer is a separate object, it must hold a reference to the
KBuf it's parsing.
If rather than a reference it holds a copy, this will have the
practical effect of making the KBuf being parsed immutable.
The current design would allow some modifications to the KBuf being
scanned (appends, but some modifications are planned.
This would in turn make race conditions a distinct possibility.

Any preferences for the Tokenizer interface?

-- 
 /kinkie