After some disturbing conversations with IDS software vendors, we've come to the conclusion that there is quite a good deal of misunderstanding surrounding our paper and it's relevance to intrusion detection. We understand and appreciate the fact that the information we're providing is wrapped up in a long and detailed technical report, and feel that it'd be best to clarify the issues publically. The first and most important point that we feel a need to convey is that we REALLY ARE saying that sniffer-driven intrusion detection systems are fundementally flawed, and (from what we can tell at this point) cannot work in the manner they are advertised to. The next important statement we're making is that ALL the systems we reviewed have bugs that will currently allow an attacker to silently slip past them. The bugs we conclusively identified and tested in our paper are not due to fundemental flaws in protocol analysis, but rather to an apparently remarkable lack of software testing applied to these products. Some of the problems we identified can be fixed. For instance, one way to evade all of the ID systems we tested was to send traffic in IP-fragmented packets. This is due to the fact that (with the exception of Network Flight Recorder), none of the vendors we tested thought to implement IP fragmentation reassembly in their products. We expect that, in light of the seriousness of these issues, most IDS vendors will have working fragmentation reassembly implemented soon. However, we do not see a way in which sniffer-driven ID systems can accurately detect SPECIFIC TYPES of attacks in IP traffic. We are not contesting the fact that it is possible to detect traffic that is likely "malicious", and we are not saying that it is impossible to detect the fact that a network is being attacked. The issue is that sniffers cannot isolate (most types of) specific attacks from any other type of attack. The reasoning behind this (clearly bold) statement is that there are ambiguities stemming from the meaning of individual packets that can only be resolved with secondary sources of information. For instance, there are situations in which you can only determine conclusively what a packet means if you know what operating system is running on the destination host. The technical basis for this (particular) problem is that different operating systems have different, slightly nonstandard TCP/IP implementations. The inconsistancies between different operating system TCP/IP implementations and the implementation of the IDS can be exploited. This is one of the fundemental technical points of our paper. Another misconception being propagated by one of the vendors mentioned in our paper is that the paper is either a "commercial review" of ID systems (with rankings, point scores, and purchasing recommendations), or a "comprehensive assessment" of the tested ID systems. Neither of these statements are accurate, and both are immensely damaging to the security community. The reason we mentioned specific ID systems in the paper was because we felt our points would be driven home best by providing empirical evidence of the types of bugs we demonstrated. Because the paper is a technical report, meant to provide information to the community about intrusion detection technology, we provided some comparitive analysis between different systems (ie, "it is interesting that system X implemented this function in this manner, as opposed to function Y"). However, one thing we specifically did NOT do was recommend any one system over another. The only technically credible statement that can be made about the results of our (cursory) comparitive analysis is that we recommend Network Flight Recorder as a result of their (excellent) policy of providing free source code to the public. In terms of actual performance and reliability, no system did "better" or "worse" than any other. All had basic, devastating bugs, and all systems (when used as network intrusion detection systems) have fundemental problems that we don't expect to see fixed any time soon. Another thing we did NOT do was provide a comprehensive security analysis of different ID systems. This is always an important point to make when discussing the publication of security flaws in software, because it is easy to think that the release of security vulnerability information means that there are no other problems that we could have identified in the product other than those we explained. This was a technical paper, not an audit report. There are likely numerous other problems with all the systems we tested, just as it is likely that there are numerous problems in any piece of network software. What the industry needs to see happen is comprehensive testing of products; when this happens, some claims about the overall reliability and security of these intrusion detection systems will be credible. Until then, we can only continue to point out the specific problems. Thanks for listening to this. The feedback we've received has been encouraging and appreciated, and we welcome as much input as you can possibly provide us with. One thing the security community certainly cannot be harmed by is public discussion, scrutiny, and peer review. Just as (we feel) the community has benefited from our contributions to the review of security products, it will benefit from your contributions to the review of the validity of our results. Have a great weekend. ----------------------------------------------------------------------------- Thomas H. Ptacek Secure Networks, Inc. ----------------------------------------------------------------------------- http://www.enteract.com/~tqbf "mmm... sacrilicious"
This archive was generated by hypermail 2b30 : Fri Apr 13 2001 - 13:42:37 PDT