MemeStreams | MemeStreams Discussion

Create an Account

This page contains all of the posts and discussion on MemeStreams referencing the following web page: Ory and the kicking of ass and taking of names. You can find discussions on MemeStreams as you surf the web, even if you aren't a MemeStreams member, using the Threads Bookmarklet.

Ory and the kicking of ass and taking of names
by Acidus at 3:19 pm EST, Dec 6, 2007

Ory over at IBM/Watchfire does a good job attempting to sort the wheat from the chaff in regards to Larry Suto's comparison report of web scanners. Couple it with HP/SPI's Jeff Forristal's report and you have a good idea about the difficulties of having a true apples to apples comparison of any type of security product, not just web scanners.

If only WASC or OWASP or somebody has some guidelines for evaluating web scanner results :-).

The Web Application Security Evaluation Criteria is a set of guidelines to evaluate web application security scanners on their identification of web application vulnerabilities and its completeness. It will cover things like crawling, parsing, session handling, types of vulnerabilities and information about those vulnerabilities.

Hopefully this will raise awareness about how confusing accurate product comparisons in the security space must be to product reviewers, prospective customers, academics, and even lay people and foster more participation in this WASC project.

But back to Ory:

In addition, I am concerned by the web application security industry - an industry filled with gifted security experts and practitioners, who embraced Suto's whitepaper warmly, without questioning its results or the methodology by which it was conducted for a single moment.
Suto, having good intentions published what he thought was in the best interest of the industry, and my biggest complaint to him was that his experiment methodology was never fully disclosed to the public, therefore could never be confirmed nor rebutted.
On the other hand, one would expect security experts to use a little more judgment when reading technical whitepapers, and be skeptical of results from experiments that are not well documented. Putting numbers into a table doesn't make them meaningful.

Ory, bravo for calling us all out for accepting things without fact checking. It seems even web professionals suffer from improper input validation for time to time! :-)

RE: Ory and the kicking of ass and taking of names
by dre at 5:11 pm EST, Dec 6, 2007

Acidus wrote:
If only WASC or OWASP or somebody has some guidelines for evaluating web scanner results :-).

Sensitivity is a great measurement for scanner evaluation.

Were you able to read this thread on the webappsec-l mailing-list?

I also make reference to the Brian Chess Metric, available in this presentation from MetriCon 1.0 - http://www.securitymetrics.org/content/attach/Welcome_blogentry_010806_1/software_chess.ppt

The Web Application Security Evaluation Criteria is a set of guidelines to evaluate web application security scanners on their identification of web application vulnerabilities and its completeness. It will cover things like crawling, parsing, session handling, types of vulnerabilities and information about those vulnerabilities.

Nah, there just writing up some definitions. It will never get very advanced.

In addition, I am concerned by the web application security industry - an industry filled with gifted security experts and practitioners, who embraced Suto's whitepaper warmly, without questioning its results or the methodology by which it was conducted for a single moment.

This is B.S. Jeremiah Grossman and RSnake (Robert Hansen) both quoted Larry Suto's paper - but myself, Dave Aitel, Matthew Wollenweber, Charlie Miller, J.M. Seitz, Adam Muntner, and several others were quick to jump in and complain about Larry's results. I'm talking about people with real-world experience and that don't have any ties to IBM, HP, Cenzic, or another web application vulnerability scanner vendor.

Did you see the outcome of the Larry Suto paper to the wassec-l, securitymetrics-l, samate-l, fuzzing-l and dailydave-l mailing-lists? Go back and check your work. Unfortunately, it was only recently discussed on the webappsec-l mailing-list, via a link to Ory's paper.

Suto, having good intentions published what he thought was in the best interest of the industry, and my biggest complaint to him was that his experiment methodology was never fully disclosed to the public, therefore could never be confirmed nor rebutted.
On the other hand, one would expect security experts to use a little more judgment when reading technical whitepapers, and be skeptical of results from experiments that are not well documented. Putting numbers into a table doesn't make them meaningful.

I do find it surprising, but you have to realize that there may be more at work here. While Larry Suto may not have had a ... [ Read More (0.8k in body) ]

RE: Ory and the kicking of ass and taking of names
by Acidus at 8:27 pm EST, Dec 6, 2007

dre wrote:... [snip]

Damn it Dre! Stop being so good. I've got like 3 half written emails to you in my drafts folders and then you have to go and write something else thats thought provoking! Grrrrrrrrrrrrrrrr :-)