Matasano gives some love, which is nice.

Some of my favorite reads (there are others) have recently written about about Scrawlr and some of what I have read has been critical. Critical enough? Depending on your level of pedantry with respect to webapp security and/or free software, probably not.

Stop that. Right now. Overlook the limitations of the tool that was released, realize that this is a closely targeted thing designed to help alleviate a specific problem. Go back and think a little harder about what is going on and why this is actually A Good Thing(tm).

[snip]

The scanner is built to look for things being indexed by search engines. If those sites are fixed, 99.999% of the problem should go away.

Trying to compare Scrawlr to a full blown SQL Injection scanning tool is like comparing a letter opener to a Swiss Army Knife. Sure, you can do other things with a letter opener (and some of you probably want to slit my throat for that simile. That’s fine, use the knife) —- but its stated purpose is to open letters.

The feedback we've been getting from developers has been "Thanks for the tool, I didn't understand [other tool]/couldn't make it work." Not surprising. These are people 5 years behind the security curve, with only a passing understanding of SQL injection and still believing XSS is all alert boxes and cookie theft. You average classic ASP dev can no more use Burp than my mom can use a methane digester. In both cases the fundamental concept of what the tool does is lost on the end user.

The feedback I've gotten from security folks is "why isn't this WI Lite. I'm sick of paying you guys $30k a year." Well, not exactly, but the subtext is there. :-)

Believe me, I really wish I could talk about the challenges of writing modern web crawlers. The fact I got to do it once was a bit of a fluk and was extremely limited in scope. So if I cannot even talk about it publicly, do you really think I would be allowed to manage a team to write a free one?