flynn23 wrote:

Despite the fact that the inFlow DC was taken down because an *external* transformer to the building caught on fire is sufficient enough to worry about, but then I think to myself that it could've been worse - the sprinklers(!) inside the DC could've went off too!

I would love to get the full story behind the handling of this outage. Coming from the perspective of trade craft, I'm highly interested. From what I was able to figure out by talking to people, the IDC's backup systems all functioned after the transformer exploded, but the Nashville Fire Chief told them to pull the plug on the facility. The one fellow I spoke to at Inflow didn't want to give me any details, and I didn't really press him on it. After I got the information I required, I just wished him luck and told him I hope the rest of their day goes better.

I'm guessing that Inflow does not have much equipment/sq footage. In some cases, taking a machine room filled with equipment and pulling the plug is about the best way you could come up with to create a fire hazard. Computer, storage, and networking equipment, when tightly packed, have an amazing quality for holding energy in the form of heat. Once the air stops flowing, regardless of if the equipment is powered down, the temperature can rise rapidly. From what I've been told in my past DC experience, exceeding 120F isn't that unheard of. This has caused DCs to burn down before. This is the type of thing I wonder if the Fire Chief was aware of.

Do you know if their fire suppression system is water or gas? That matters when determining how/if to pull the plug. The equipment can sit and cool down if the environment is (or can be) flooded with gas. If its water, the best way to handle it is to kill the equipment and keep the cooling going, unless of course there is an actual fire within the facility.

I'm guessing Inflow's Disaster Plan does not connect with the city in the way it should. A transformer blowing up should not lead to a DC being shutdown unless its pouring smoke into the facility or something that poses a direct life safety threat.

The Fire Chief's job is to be paranoid, so I can understand the call he made. It just begs a larger question: What do Fire Chiefs need to know about Data-Centers when it comes to these types of decisions? If every time something goes wrong such as an external transformer blowing up, it requires the facility to be shut down, it would be impossible to build a carrier class facility capable of getting past 4 nines without its success grounded on luck rather than design..

While pondering that question, you could watch this music video of the Nashville FDP-EMS doing a training exercise.

What do Fire Chiefs need to know about Data Centers?

MemeStreams experienced an outage today for eight hours due to a transformer explosion at our primary data-center located next to the landmark Bell-South "bat-building" in Nashville, Tennessee. Pictures taken by Industrial Memetics staff of NES workers responding to the situation can be seen here and here.

For my generation, the Hiroshima and Nagasaki bombings and the war in general now represent the equivalent of a cultural "game over" or "reset" button. Through a combination of conscious policy and unconscious culture, the painful memories and images of the war have lost their context, surfacing only as twisted echoes in our subculture. The result, for better and worse, is that, 60 years after Hiroshima, we dwell more on the future than the past.

Joi Ito has an editorial in the New York Times today about the 60th anniversary of the atomic bomb being dropped.

(Yes, I'm going to start blogging about things other than Mike again..)

An Anniversary to Forget - New York Times

The next installment of Jennifer's story about representing Mike is up.

This post has one key piece of information that explains definitively why Jennifer kicks so much ass. She is a Jersey Girl!

I also find it somewhat intriguing that both her cat and dog look very serious.

Update: The last installment is up now as well. Wired has picked up the story from Jennifer's blog and is running it.

Jennifer Granick | The Shout | Reverse Engineering Lawyer Code

I guess that's the big question.

Cisco can best be considered "high risk, high return". Lets hope they adjust their security culture and we see those returns. Even the media following the financial markets has noted Cisco is taking a vacation at Club Microsoft. I don't think anyone even had to connect the dots for them. There have been some changes in Cisco's Chinese management, which I'm sure have nothing to do with this.. No dots here. No sir. Just things that look like nodes, and a general neglect for all things American that are not American Business. At least they are not Enron.

As the days roll on, Mike will not be sitting in the hot-seat any longer. I expect ISS to take his place, and rightfully so, they deserve every black-eye they get. Right now the Cisco legal team is doing the equivalent of a pre-fight pump-up. I'm sure of it.

Ed Felten has a good post over at the Freedom To Tinker blog that goes into a number of the legal issues this presents:

Any discussion of this argument has to start with the obvious: Cisco is claiming that part of its product is a trade secret. The software is key to the product’s function, and Cisco sells the product to essentially anybody who wants it. It’s hard to think of any reasonable sense in which this can be called a secret. (I know that legal definitions of terms like “trade secret” aren’t always intuitive, but still, this seems a bit much.)

Clearly an issue we are concerned with. The most stressful parts of the coming Cisco vs. ISS battle are going to surround this. Many bullets will fly. Some might strike the innocent, but they will fly for awhile and strike them far off in the distant future. We will be listening for the fire and keeping our heads down.

So what about Mike? Ira Winkler at the IT Defense Patrol blog offers this:

Let’s stop chastising Michael Lynn. He may have violated is employee agreement, but that is not really an issue for us. He may have technically violated Cisco licensing, and that is the whole point - any bad guy would probably do the same. However, he did it within a regulated environment, which where it should happen. And where it happens all the time. And the result is often publication of a security alert. Lynn's actions are no different.

Sure. Too bad it wasn't a well regulated environment, although its not like we have a goo... [ Read More (0.4k in body) ]

Jennifer Granick has posted the latest installment of her story about defending Mike. I blogged the previous installment yesterday. Again, its a really rare and great thing to hear the story told from the perspective of one of the lawyers fighting the battle. And to continue repeating myself, Jennifer has been doing an excellent job, continue to shower this woman with complements.

There are numerous sections of her post worth of quoting. Just go and read the entire thing. It contains some of the best legal blow-by-blog (sticking with that typo) we are going to get..

After you are done, reflect on how when it rains, it pours and check out this post on TaoSecurity.

Jennifer Granick | The Shout | Luckily, there were bagels.

The discussion I've been having over at the Cargo Cult is ongoing. Head over there to pick up the full context..

Here is my latest response:

All your points on Intellectual Property are valid. Hence, I opted not to attack your argument from that perspective. I am also not in the anit-IP crowd. I think to get the carrier firmly up, we need to establish more facts. It appears that incorrect assumptions lie at the heart of the differences in our views.

Mike never had the IOS source code. Cisco never gave it to him; he was not under NDA to have it. He did not violate any of their rights to protect their source code. Mike did his research by disassembling the publicly available IOS images. This was a case of reverse engineering. If anything, Mike's legal liability lied in exposing trade secrets, but not due to any access to (Cisco) proprietary information. Both you and I could easily have access in a "virgin" manor to what Mike based his research on. I assume that both Cisco and ISS (and their respective legal teams) did not feel they were standing on firm enough ground to go after Mike based on this tactic. The dogs smelled blood and were barking, but they were kept on the leash. The reason certainly wasn't a feeling of mercy on the part of Cisco.

Once we get here, we come full circle again. And its a big circle. The ends meet at places like the old DeCSS situation. If you own the chattel (odd context to use that term, I know), do you have the right to take it apart and figure out how it works? Can you share that information? If the law is vague, where do ethics supplement the pre-legal argument? If this situation winds up being framed in that light, its the DMCA we have to look to. If you want a laugh, think about if coming up with an exploit qualifies as "necessary to achieve interoperability of an independently created computer program."

I also do firmly believe that Mike's statements are grounded in fact when it comes to matters like "China has this." The big picture could best be described as "huge" if you acknowledge that. Thinking about the problem honestly feels like wargaming to me.

The general option of the hacker (read: security, not criminal) community is that Cisco has a broken security culture. They need a wake-up call like this to bring positive change to it. I think they will be successful with it in the long term too. I'm also not a member of the "all corps are dumb" crowd. At worst, I prefer to think of them (us?) as "slow". ISS on the other hand, I don't have such kind words for. Please take a look at this post on my blog which contains a link to and some excerpts from Mike's interview with Wired.

That made me outright angry. It also raised some serious questions, which I am not going to put forth in a public forum. Bonus points if you can figure them out...

What follows is my take on “Ciscogate”, the uproar over researcher Michael Lynn’s presentation at this year’s Black Hat conference, in which he revealed that he was able to remotely execute code on Cisco routers. I have been representing Mike during this crisis, so I’m clearly partisan, and what I can say is limited by attorney-client responsibilities. But while many people are speculating about the facts, there hasn’t been much on the law, which turns out to be really interesting.

Jennifer Granick has posted the first installment of the story about her representing Mike. Its very rare you get to hear the take of a case like this directly from the lawyers involved, so this is a treat.

Earlier I suggested that everyone leave a comment on Jennifer's blog thanking her for representing Mike. I'd like to renew that suggestion. Thanks Jennifer!

After reading this, you might want to check out this collection of comments on Cryptome about the situation. It includes links to pictures of the presentation Mike actually gave, as opposed to the one that is floating around.

And seriously don't miss the truly excellent video floating around of the Cisco temp-workers slicing the materials out of the conference booklets. You can get it here or here.

The Shout | Jennifer Granick | ISS and Cisco v. Granick’s Gambling Plans

A post on The Lazy Genius blog points out that US-CERT believes we should be on the lookout for usage of bugs in the wild based on what Lynn discovered. Basically, network administrators should be vigilant and prepared.

US−CERT Operations Center Synopsis: A presentation at Defcon entitled "Live penetration Test of the Backbone" was scheduled to include use of an exploit disclosed by Michael Lynn earlier this week. The exploit is NOT the weak version demo'd by Lynn, but a fully working version that is capable of re−routing traffic, man in the middle and / or dropping the router. EFF lawyers toned down the presentation to avoid ISS and/or Cisco lawsuits. Analysis: There is an exploit. It will fall into the wrong hands. Prepare your Networks.

RECOMMENDATIONS AND COUNTERMEASURES If your network doesn’t need IPv6, disable it. This will eliminate exposure to this vulnerability. On a router which supports IPv6, disable it by issuing the command "no ipv6 enable" and "no ipv6 address" on each interface.

There seems to be a fair amount of concern over this situation present in the pharmaceutical and health care industry, although few are talking publicly. Over at the HIPAA Blog there is the following post advising what most are advising:

What does this all mean? Beats the hell out of me. But it is a good lesson for everyone who is subject to HIPAA (and even those who aren't) that you need to keep track of your systems and software, find out about security issues ASAP, and make sure you patch up any security issues as soon as you find out about them. That may mean making sure your IT staff knows what's up, or leaning on your vendors to make sure they're taking the right steps to keep your backside covered.

Everyone should have their guard up. The keyword of the day is vigilance.

Dept. of Homeland Security Raises the Red Flag

Kudos to MemeStreams user Dagmar for putting together a post with breaks the technical aspects of Lynn's disclosure down in a way that non-technical people can understand. Be sure to click through and read his entire post.

Someone who takes the time to tie a few existing exploits together and utilize a technique similar to what Lynn discovered to make a worm that infects equipment, spends a small amount of time trying to infect other equipment, and then viciously puts the equipment out of commission in the aforementioned fashion, could in a very real sense turn off large chunks of the Internet.

No, I was not joking about the last sentence. If you work in an IT (Information Technology shop) take a moment to look around your office at all the very important equipment you have that just happens to have the Cisco logo on it. (I say "just happens to have the Cisco logo" because the root problem here has nothing to do with Cisco in particular, they're just the first company who have had this weakness uncovered--and as I said earlier, they were already in better shape than most.) Now imagine what would happen if that all that equipment just shut off, and you couldn't get it back up and running any time in the next twelve hours or so. You might think, "well, I will just go to their website and get the updates" but no, no... the Internet connection ran through one of the pieces of equipment that is now down so you can't do that. ...and even if it's not, there's a good chance that the people who your company connects to in order to reach the Internet has equipment that's has been effected, so you still can't get to the website with the updates you need. So you pick up the phone and call the manufacturer, and get to wait on hold for a very long time indeed, because many thousands of other people are just as stuck as you are. FedEx can get things out fast, but they're not nearly instantaneous, and hundreds of thousands of packages all marked "Red Tag, Highest Priority" at once are going to give them fits. Unless you know someone with magic powers of teleportation, you're looking at a very long wait for a package to be delivered by a truck that can fix your problem, and you're going to have to deal with all the upper-management types freaking out in the meantime. (Mind you, if you're lucky, your inter-office email system will also have been shut down by this, so they can only get to you through your cell phone and pager, which limits the number of panicked managers who can get to you at once.)

One message that Dagmar tries to get across in this, that should be spread and embraced, is that equipment (and software) mono-cultures are inherently dangerous. A post on the blog Art Of Noh... [ Read More (0.1k in body) ]

Non-Technical Explanation of Mike Lynn's Disclosure