http://www.wired.com/news/privacy/0,1848,68365,00.html By Kim Zetter Aug. 01, 2005 LAS VEGAS -- Security researcher Mike Lynn roiled the Black Hat conference Wednesday when he resigned from his job at Internet Security Systems to deliver a talk about a serious vulnerability in Cisco IOS, the operating system powering its routers, defying efforts by the router manufacturer and his former employer to block the presentation. In the aftermath, Lynn reached a legal settlement with Cisco and ISS in which he agreed to erase his research material on the vulnerability, to keep secret the details of the attack, and to refrain from distributing copies of his presentation, among other concessions. Now facing an FBI investigation -- and sudden celebrity status in the tech world -- Lynn discusses the events leading up to this week's disclosure, and what he thinks it means for the security of the internet in an exclusive interview with Wired News. Wired News: Can you tell me how all of this started? You were asked by your employer, ISS, to reverse-engineer the Cisco operating system, weren't you? Michael Lynn: I was very specifically told.... It was January 26th and Cisco had just announced a totally different vulnerability than the one I demonstrated. They'd announced a vulnerability for something called "Multiple Crafted IPv6 Packets Cause Router Reload" (as they worded it in their patch message). But that's a very vague term. It just says, "Hey, something is wrong in IP6 with the router reload" ... but it didn't say you could be in control of it. ISS wanted to get protection in their products (against this problem) so that their customers wouldn't be as affected by it. So they called up Cisco to try to get some more details for it ... and Cisco wouldn't give (the information) to them. So (ISS managers) came to me and said, "Can you reverse-engineer ... can you disassemble IOS ... to find out what their vulnerability is?" WN: So this was a different vulnerability from the one you demonstrated at the conference this week? Lynn: Yes, but (Cisco) had (also) found the vulnerability that I demonstrated on stage about two weeks before I (found it). WN: Then what happened? Lynn: So on January 27th, ISS comes out with their response to this vulnerability -- the advice to their customers based on my analysis.... I stayed up all night basically (to research it). I realized in looking at this (that the program) is actually way worse than Cisco said.... So (our guy) calls up ... Cisco and says, "OK, we aren't 100 percent sure that we found the same bug that you're talking about, but it's important we find out because the one we found has much, much greater impact. You said there's (the possibility) of a denial-of-service attack. But the one we found is fully exploitable." Cisco said, "You guys are lying. It is impossible to execute shell code on Cisco IOS." At that point (ISS) management was annoyed.... They were like, "Mike, your new research project is Cisco IOS. Go find out how to exploit bugs on Cisco IOS so we can prove these people wrong." WN: In your speech you said you worked on the reverse engineering with cooperation from Cisco. Lynn: We did, in fact. The cooperation came later. They didn't start that way, and they were not happy to begin with.... They didn't cooperate in the actual reverse engineering itself. They cooperated in the research effort, I would say, in finding vulnerabilities and confirming (them). WN: They didn't stop you. Lynn: They didn't stop us, and at this point there was some back-and-forth communication. (Lynn spent the next month researching the program.) WN: After you came to them with the serious flaw and said, "This is the bug we found...." Lynn: They said, "We don't believe you." And (ISS managers) said ... "come down to Atlanta and we'll show you." And that's never happened, by the way, at ISS. They've never brought somebody, let alone a competitor, into the office just to show them (something).... Mike Caudill, (Cisco's) customer advocate, came out. And they also sent out an engineer ... who described himself as an IOS architect.... I was told he helped design parts of the source code.... And his jaw hit the ground. He was very impressed, he was just (saying), "Wow, that's cool." That was June 14th. WN: Cisco saw your Black Hat presentation long before they decided to pull it. When did they see it? Lynn: Probably June 14th, the day that they came out (to Atlanta). We told them about the vulnerabilities well before (that). WN: So at what point did they get nervous about the talk? Lynn: When they saw the listing of the presentation on the Black Hat site is when they actually called us back and said, "Wait, you guys were serious?" And we said, "Yes, we were serious." Incidentally, it was ISS who submitted (the talk) for Black Hat. I was told (by ISS), "Hey, you want to go to Black Hat? We'd like you to do it." WN: So ISS knew the seriousness of the bug. Lynn: Yes, they did. In fact, at one point ... they apparently didn't get it, and they actually wanted to distribute the full working exploit very widely inside the company.... I was told ... "Give this to all the sales engineers and to all the pen testers." WN: Why would they want you to do that? Lynn: Well, because it bruises Cisco, remember? Mind you, this was something that Cisco hadn't gone public with yet and that's not useful to pen testers because what do they advise their customers to do (to protect themselves if no information about the vulnerability has been released yet)? I told them, "You do realize if you do that, it's going to leak?" And (one of the ISS guys) says, "That's Cisco's problem." And then (another ISS guy) turns to me and says that they need to understand this could be their Witty worm. I was like, Whoa, what meeting did I walk into? (The Witty worm was a particularly aggressive and destructive code released by someone last year that targeted computer systems running a security program made by Internet Security Systems and even more specifically targeted military bases using the software. It infected more than 12,000 servers and computer systems in about an hour. Because of the worm's speed in spreading and its creators' apparent knowledge of who ISS' customers were, some security experts speculated that someone working for or connected to ISS might have been responsible for writing and releasing it.) At that point, I told them all no, and they fought it and I resigned right there on the spot. And this was about a month ago. I thought they were handling this in a non-ethical manner. Because it was just way too fast and loose with who can see this.... I mean, I don't even want people to see it now. (ISS talked him out of the resignation by agreeing to give him control over who could see or have the exploit.) So we start moving forward with the talk and we're working with Cisco, and Cisco seems OK with it. WN: They had already released information about what you found before your speech, right? Lynn: Yes, and the fix. The fix was about six months before the message. WN: So they already knew how serious the problem was. Lynn: If they didn't know, they should have. WN: But they didn't indicate to their customers how serious it was. Lynn: No, they did not. WN: And Cisco saw your Black Hat presentation long before they decided to pull it, right? Lynn: Probably June 14, the day that they came out (to Atlanta). (Then) it was two weeks ago, I was first told that Cisco might want to come onto (the) stage with me and say a couple words. And I said, provided the words aren't something to the effect that "he's a liar," I'm OK with it.... It didn't really matter. It lent credence to my talk. And it's good because I felt my talk really needed to be taken seriously. (However, the plan changed even more and Lynn was told to remove any mention of reverse engineering from his talk or cancel the presentation. If he did neither, he would be fired.) Mind you this is a complete reversal. Like a week or so prior, the night of the close of the fiscal quarter, and they were all celebrating that they hit the numbers, the CEO invited me out for a beer, and he just couldn't say enough awesome things about this talk. WN: Was Cisco threatening them? Lynn: I asked point-blank, "Are you being threatened by Cisco?" They said no.... To be perfectly honest, I don't think there was any legal threat. I think that it was more of a "scratch our back and we'll scratch yours." (Cisco asked him to wait a year until it could release a new version of its operating system. When he didn't back down, Cisco threatened a lawsuit against Lynn and Black Hat. Then with Black Hat's cooperation, Cisco arranged to tear out pages with images of Lynn's slides from the conference book.) WN: You met with the feds after your talk, and someone gave you a challenge coin (a special coin created for members of the military to commemorate challenging missions)? Lynn: Yes, they did, actually. And I didn't know what it was, so I didn't thank him properly.... This was a really funny story. (Right after my talk, this) guy walks up with a very, very impressive badge ... and says, "I need to speak with you. Now." WN: What agency was it? Lynn: Air Force (Office of Special Investigations). NSA, is what I'm told, but he wouldn't show me his credentials. There were a lot of flashy badges around from lots of three-letter agencies. So they take me to a maintenance area and I'm surrounded by people ... and one of them says (to another guy), "You've got the van ready?" I'm going, "Oh my god." And they go, "Just kidding!... Oh, man, you rock! We can't thank you enough." And I'm just sitting there, like still pale white. They all shook my hand. I get the feeling that they were in the audience because they were told that there was a good chance that I was about to do something that would cause a serious problem. And when they realized that I was actually there to pretty much clue them in on ... the storm that's coming ... they just couldn't say enough nice things about me.... Also, US-CERT (Computer Emergency Response Team) asked me if I would come up to D.C. in a week or two and help them formulate the nation's strategy for cybersecurity. WN: So this new version of the operating system that they're coming out with, that's in beta testing. Lynn: It's actually a better architecture ... but it will be less secure.... That's why I felt it was important to make the point now rather than sweep it under the rug. I think it's something that we can fix.... The problem now ... is that if you want to attack something ... you're going to (have to) hack one machine (at a time) and take control of the part of the network (it's on). If you had (the exploit) up running against the new version that's in beta now, you can take everything. That's the difference between something you can make a worm out of and something you can't make a worm out of. (Right now) nobody patches Cisco routers because there's been this culture (that) there's just never anything that can go wrong (with them). So, unless there's some really critical thing that's making it crash, people don't install the patches.... We have to change the public perception about patching now, and that cause is not best served by pretending that there's not a problem and saying maybe you can talk about this next year.... The time to talk about this is before the critical problem comes around. WN: Cisco has said this is not a critical flaw that you found. Lynn: I would agree with them in part and disagree with them. In a way I would say, yes, it's actually not all that exceptional in that all it proved is it's just like any other computer -- they're all hackable. Because in any complicated system, people make mistakes. It's our very nature. But in the sense that the potential impact of something like a router worm (attacking the routers) is no big deal, I would strongly disagree. Unlike most other vulnerabilities or exploits, when you ... take control of another machine, it's very difficult, if at all possible, for you to ... destroy the hardware.... But on a router? This is (a scenario in which) the network is down, and it's down in a way that it's not getting up again. How do you ship the patch when the network won't (be up so you can distribute it)? Are you going to mail out a CD? But there's no CD drive. The real point is there's a ticking clock but we still have plenty of time. I wanted people to be afraid a little bit ... because I needed people to act. But at the same time, now that I think they already are, I will say it's not as bad as you probably think it is. Not yet ... because the version that makes this an unstoppable critical problem is not out yet. _________________________________________ Attend ToorCon Sept 16-18th, 2005 Convention Center San Diego, California www.toorcon.org
This archive was generated by hypermail 2.1.3 : Wed Aug 03 2005 - 20:56:44 PDT