There has been an almost unbelievable amount of hubbub lately about the research that Mike Lynn gave a demonstration of at the BlackHat conference last week, and there's been a positively dizzying amount of "spin" applied to the media. Let me say one thing to everyone reading this, right up front. What Lynn uncovered is a serious issue, probably actually more serious than what the media is making it out to be. While coverage on the issue is good (and useful to both "sides") the lack of actual accurate reporting on the issue isn't helpful to anyone.

Part of the problem is that apparently, outside of the list of BlackHat attendees, there's not that many people running around who truly understand what Lynn's research uncovered. Lynn did not reveal an "exploit" in the usual sense. In fact, Lynn of his own volition has been playing his cards fairly close to his chest on this, and omitted most of the technical details of the problem from his presentation in order to assure that no one would be able to easily "follow in his footsteps". Lynn, it can safely be said, was scared by what he discovered--scared enough that he has risked his livelihood not once but twice in order to be sure that should the technical aspects of what he's found not be resolved before someone with less respect for the continuation of the Internet figures it out for themselves, the network and security administrators of the world will have had time to take some steps to reduce the amount of damage done. It can no longer be thought of as a sure thing that just because a particular vulnerability could "break the Internet" that no one's going to try it just to see if it's really true. We have a rather excellent example in recent history that pretty much everyone is aware of by now... the MS Blaster worm which raged around the Internet wreaking rather unprecedented havok. Pretty much everyone on the Internet was either personally affected by this, or knows someone who was. Blaster made use of a vulnerability that had become rather common knowledge by the time it was released, but had already been known to many security professionals for months. The real problem that made things so painful and propagation of Blaster so widespread, was that for those months, Microsoft had been actively denying that there was ever a problem until Blaster forced them to admit it. Had system administrators been made aware of the issue and the meager steps needed to impede the spread of Blaster (which everyone implemented in a white-hot hurry once their networks were figuratively ablaze) the damage could have been much less indeed.

Cisco is not helping the issue, or I should say, Cisco's lawyers are not helping the issue. Cisco makes some really awesome products, and their technical people can't really be faulted for this one technical flaw. The problem is that Cisco's lawyers are convinced that public knowledge of a serious issue with Cisco's products would in some way harm their stock valuation. ...and so they are trying to squelch the information. This is misguided at best. Everyone's products, and I mean everyone's products are discovered to be flawed at some point or another. The real thing that matters is how those manufacturers handle these problems and that they fix them in a timely fashion. Had Cisco's lawyers not did what they did, there is every expecation that this would have been sorted out by Cisco rather quickly (I actually expected them to have a solution ready in time for BlackHat), administrators would begin patching their equipment, and things would continue to be business as usual.

Now, on to what is really the thing that makes the technical aspect of Lynn's research very dire indeed. I've already said that Lynn did not find an "exploit" in the classic sense of the term, and this is quite true. The code that is the result of the research that Lynn demonstrated at BlackHat, used by itself, can't be used to do anything to a router. It's more accurate to say that it works as an "exploit enhancement". There is also a fundamental misconception about the root reason for Lynn's presentation. The message Lynn was attempting to get out had very little to do with any particular vulnerability in Cisco's products--the message was that one of the fundamental assumptions (that hardware-based firewalls are invulnerable to remote compromise attacks) that the majority of the internet uses to form their network availability policies is wrong. Lynn demonstrated rather clearly to the BlackHat conference attendees just how wrong it is.

Some background is necessary to make sure the laymen can keep up before we go further. In recent years, more and more of the "exploits" that have been released have come down to requiring the presence of a particular type software glitch to take advantage of. This type of glitch is called an "overflow". What it amounts to is that the attacker sends some information to the affected equipment that is in some way malformed enough so that the equipment gets confused as to what it's doing and winds up treating part of the data that was sent as if it were it's own executeable code. This allows the attacker to substitute some small amount of their own programming for what the equipment is supposed to be doing, and usually what the attacker does is give the equipment instructions to the effect of "you take orders from me now". From that point on, the legitimate administrators have lost control of things, and the attacker can have the equipment do whatever they want it to.

Vendors of "hardware" firewall products (like Cisco, Foundry, Juniper, etc) have been playing to the advantage that normally their systems are designed in such a way as to make it extraordinarily complex for an attacker to actually create a set of instructions that the targeted equipment can actually make sense of once the attacker has confused the equipment, or that if the equipment can make sense of it, it's simply not capable of carrying out the instructions it was given. Cisco had gone one step farther (and if anything they should be proud of this, it's a very smart thing to have done) in their equipment design and had incorporated software that acts as a kind of watchdog, that once it sees something going on that shouldn't be happening, it puts a stop to it. So, under ordinary circumstances, when someone finds an overflow glitch in Cisco equipment and tries to exploit it, the worst that can happen is the the part of the equipment that was attacked gets killed by the watchdog routine. This might mean that a VPN service stops and disconnects some users, or maybe that the router stops working altogether, but these aren't the sort of events that will go unnoticed by the administrator. The administrator can rather easily resolve things in just a few minutes by merely restarting the affected parts of the equipment, and business goes on as usual.

Mike Lynn, in short, figured out how to make the watchdog believe that nothing is actually wrong, so that the attacker's own code gets executed and isn't stopped. Ultimately, this means that an attacker who has also figured out this trick (because Lynn sure as hell wasn't going to tell anyone), can take code to exploit an existing, known vulnerability and couple it with this technique, and actually get the router to do what they want. Since the watchdog didn't shut anything down, it's also rather likely that after the attack has successfully completed that no one knows anything actually happened, and the attacker now has administrative control of the equipment.

Now, once you've got administrative control of some equipment, you can do pretty much whatever you want with it. A really vicious person could go ahead and turn it off, which is going to raise some eyebrows, but there's nothing to stop the administrators from just going into their office and turning it back on. A truly malicious bastard can quickly mess around with the equipment's operating system to a degree that pushing the power button isn't going to accomplish anything--the equipment is for all intents and purposes dead, and will require some moderately advanced technical knowledge and a bit of time to fix. Now, if this were someone's workstation, it would be pretty bad for them, but we're talking about routers--the equipment that interconnects lots and lots of people's workstations and servers. It's a piece of equipment that once it falls off the network, no one's computers can talk to each other any more. Someone who takes the time to tie a few existing exploits together and utilize a technique similar to what Lynn discovered to make a worm that infects equipment, spends a small amount of time trying to infect other equipment, and then viciously puts the equipment out of commission in the aforementioned fashion, could in a very real sense turn off large chunks of the Internet.

No, I was not joking about the last sentence. If you work in an IT (Information Technology shop) take a moment to look around your office at all the very important equipment you have that just happens to have the Cisco logo on it. (I say "just happens to have the Cisco logo" because the root problem here has nothing to do with Cisco in particular, they're just the first company who have had this weakness uncovered--and as I said earlier, they were already in better shape than most.) Now imagine what would happen if that all that equipment just shut off, and you couldn't get it back up and running any time in the next twelve hours or so. You might think, "well, I will just go to their website and get the updates" but no, no... the Internet connection ran through one of the pieces of equipment that is now down so you can't do that. ...and even if it's not, there's a good chance that the people who your company connects to in order to reach the Internet has equipment that's has been effected, so you still can't get to the website with the updates you need. So you pick up the phone and call the manufacturer, and get to wait on hold for a very long time indeed, because many thousands of other people are just as stuck as you are. FedEx can get things out fast, but they're not nearly instantaneous, and hundreds of thousands of packages all marked "Red Tag, Highest Priority" at once are going to give them fits. Unless you know someone with magic powers of teleportation, you're looking at a very long wait for a package to be delivered by a truck that can fix your problem, and you're going to have to deal with all the upper-management types freaking out in the meantime. (Mind you, if you're lucky, your inter-office email system will also have been shut down by this, so they can only get to you through your cell phone and pager, which limits the number of panicked managers who can get to you at once.)

Now, the reason why it's so terribly important that this possible scenario be made aware to people that Mike Lynn quit his job to get the word out, is that there are things that can be done to limit the damage done by a worm of this sort, but unless people understand that this type of nightmare scenario is not a once in a thousand years thing, but in actuality a very real danger to them, they're simply not going to take those steps. Lynn and others can say with some certainty that administrators are not going to take those steps, because they're the security people who on a daily basis see that administrators simply aren't taking those steps because until now all they've seen is that they occasionally have to make a late night trip to the office to restart a piece of malfunctioning equipment. The equipment manfacturers, in the interests of their stockholders, have been telling them all along that this is the worst they can expect of have to deal with.

(Sidebar: If you look at the announcement that Cisco made shortly after Lynn's presentation that pertained to the ready availability of the patch that Cisco had already made to fix the particular IPv6 exploit that Lynn used during his BlackHat presentation, you will see that there are ways of keeping the vulnerabilities out of the reach of attackers, even without the patch. Read it carefully because these are the types of things you should have been doing all along. This new discovery does not mean you are defenseless.)

I could go on for several paragraphs more about why administrators aren't taking those steps, but I figure I've probably given enough arguments at this point. My main reason for posting this is so that people will understand that what Cisco's lawyers are telling people is not necessarily true--that ignorance is not bliss

Normally, what we've seen with Mike Lynn is what security experts have grown to expect--that people intelligent enough to figure out these things are generally responsible enough people to not be inclined to do anything spectacularly destructive with this information. Blaster showed us that this is actually not always true, and considering the ramifications of the one lone nut that it would take, well... people need to know. They can prevent this issue from blowing up in their face, and it would all be part of their normal job descriptions.

Here's to hoping things change for the better before some nutjob tries to go down in infamy.

Mike Lynn's 'exploit', in plain (non-technical) English