[PATCH] Generated SBuf::find() test cases

From: Alex Rousskov <rousskov_at_measurement-factory.com>
Date: Mon, 25 Feb 2013 15:06:11 -0700

Hello,

    The attached patch adds about 700'000 test cases for eight
SBuf::find() methods. Needless to say, they are generated rather than
manually written. They take a couple of seconds to run. Configuration
options allow us to increase or decrease the number of test cases if needed.

The code attempts to avoid reporting the same kind of find() bugs over
and over again by suppressing reports about similar failures. The
definition of "similar" is imperfect and the corresponding
categorization code can probably be significantly improved, but it
already results in reporting just 28 out of 22949 failures.

Here are a few reported test case failures (stringng r9612):

> case196: SBuf("a").rfind("a", 0) returns npos instead of 0
> std::string("a").rfind("a", 0) returns 0
> category: hay1.rfind(needle1_at_B,posB)

> case5796: SBuf("2").rfind("", 0) returns 1 instead of 0
> std::string("2").rfind("", 0) returns 0
> category: hay1.rfind(needle0_at_E,posL)

> case5799: SBuf("2").find("", 1) returns 0 instead of 1
> std::string("2").find("", 1) returns 1
> category: hay1.find(needle0_at_B,posP)

> case6236: SBuf("2Q").rfind("Q", 1) returns npos instead of 1
> std::string("2Q").rfind("Q", 1) returns 1
> category: hayN.rfind(needle1_at_E,posB)

> case22048: SBuf("CQGGR4").rfind('G', 2) returns npos instead of 2
> std::string("CQGGR4").rfind('G', 2) returns 2
> category: hayN.rfind(char_at_E,posL)

> case30524: SBuf("UtRGRR").rfind("RR", 3) returns 4 instead of npos
> std::string("UtRGRR").rfind("RR", 3) returns npos
> category: hayN.rfind(needleN_at_E,posL)

...

> Generated SBuf test cases: 716672
> failed cases: 22949
> reported cases: 28
> Asserting because some cases failed...

Needless to say, all of the above could be coming from the same bug or
two in SBuf code. 28 test case failures do not imply that there are 28
different bugs (our the categorization code is not that clever!).

The large number of generated test cases, the logic to suppress similar
reports, and lack of CppUnit knowledge resulted in very limited CppUnit
integration. All the generated test cases are treated as one from
CppUnit point of view. Perhaps this can be improved in the future.

New generated test cases for SBuf methods other than find() and rfind()
should be added as well, of course. Class and file names should be
polished. There are a few other TODOs in the patch.

As you know, SBuf in the StringNG branch has been under Squid Project
review for a very long time. While each review iteration did find some
bugs, the overall progress has been very slow. I suspect one of the
reasons behind these delays is that it take a long time to find the time
for review and then to find the time to fix bugs.

By adding these test cases to the branch, I hope we can eliminate some
of the delays because at least find() bugs can be found automatically,
without yet another review iteration. Similar generated test cases for
other areas should be added as well, IMO.

Kinkie, would you be interested in taking over this code, polishing it
as needed, committing it to stringng, and using it for your work?

Thank you,

Alex.

Received on Mon Feb 25 2013 - 22:06:30 MST

This archive was generated by hypermail 2.2.0 : Tue Feb 26 2013 - 12:00:07 MST