Folks, We are honored to have the founder of our CRIME list, and valued CRIME member at large, Dr. John McHugh from CERT, as our Valentine's Day speaker. This is a 'must' attend--a fitting Valentine's treat for our group. See you there! Geo _____________________ Title: Evaluating IDS Systems (Why testing Security Software is Hard) Topic: In 1998 (and again in 1999), the Lincoln Laboratory of MIT conducted a comparative evaluation of Intrusion Detection Systems (IDSs) developed under DARPA funding. While this evaluation represents a significant and monumental undertaking, there are a number of issues associated with its design and execution that remain questionable. The difficulties associated with this evaluation have been the subject of several papers and a number of presentations. As a result of our investigations of Lincoln's efforts, we have been attempting to develop an appropriate framework in which similar, but meaningful and useful, evaluations can be performed. This talk will contrast our proposed approach with the work that Lincoln performed (and is continuing to perform). Our primary conclusion for signature based systems are that the we simply do not know enough to generate appropriate artificial background data for false alarm evaluation, but that there are a systematic approaches to measuring true positive and negative performance, under both ideal and appropriate environmental stress conditions. The situation is much less clear for with respect to anomaly based systems since the relationships between anomalous and intrusive behavior are poorly understood. In both areas, there is a paucity of theory that can be applied to the problem and we feel that the ad hoc and intuitive approaches that characterize today's efforts may be nearing their limits. Dr. McHugh's CV: John McHugh is a member of the technical staff at CERT, part of the SEI at CMU. He was a professor and former chairman of the Computer Science Department at Portland State University in Portland, Oregon where he held a Tektronix Professorship. His research interests include computer security, software engineering, and programming languages. He has previously taught at The University of North Carolina and at Duke University. He has been an active researcher in the application of formal methods to the construction of dependable and secure systems for many years. He was the architect of the Gypsy code optimizer and the Gypsy Covert Channel Analysis tool. Dr. McHugh received his PhD degree in computer science from the University of Texas at Austin. He has a MS degree in computer science from the University of Maryland, and a BS degree in physics from Duke University. He grew up in Durham, North Carolina, leaving when he graduated from Duke. Twenty years later, he returned, demonstrating that Thomas Wolfe was wrong. After another ten years in Durham, he moved to Portland, demonstrating, perhaps, that Wolfe knew what he was talking about after all.
This archive was generated by hypermail 2b30 : Wed Feb 05 2003 - 11:12:37 PST