Ok, so ideally we want something that a court can use straight out of the box (Presumably with sysadmin's testimony that they didn't tamper with it). What requirements would the court have? (Reliable timestamping, ...?) What do we we need to do? Edward todd glassey <todd.glasseyat_private> on 16/08/2001 12:07:16 To: Chris Calabrese <chris_calabreseat_private> cc: loganalysisat_private Subject: Re: [loganalysis] Logging standards and such Just putting Syslog Data into XML is not enough. What Syslog data needs to be constrained such that the Syslog daemon can act as an evidentiary agent or reception agent and that means that it will do the testimony not the systems admin that operates it. With that said there are many holes that have to be filled to deal with the legal testimony generator requirements. Todd ----- Original Message ----- From: Chris Calabrese To: edward.j.sargissonat_private ; loganalysisat_private Sent: Wednesday, August 15, 2001 11:49 AM Subject: RE: [loganalysis] Logging standards and such > However, I have gotten some interest in it from time > to time, and I offer it here as a good starting > point... Oh bother, it looks like my attachments didn't make it through some step of the way. Tina, are you stripping attachments? Hmm, here's HTML. If you want this in ASCII, send me an e-mail... (it's easier to do the HTML because the Yahoo mail client reformats the text bits if they're not attachments) -------------------------- .title { FONT-WEIGHT: bold; FONT-SIZE: 22px; COLOR: #990000; LINE-HEIGHT: 22px; FONT-FAMILY: helvetica, arial, sans-serif; TEXT-ALIGN: right } .filename { FONT-WEIGHT: bold; FONT-SIZE: 18px; COLOR: #666666; LINE-HEIGHT: 28px; FONT-FAMILY: helvetica, arial, sans-serif; TEXT-ALIGN: right } P.copyright { FONT-SIZE: 10px; COLOR: #000000; FONT-FAMILY: verdana, charcoal, helvetica, arial, sans-serif } P { MARGIN-LEFT: 2em; MARGIN-RIGHT: 2em } OL { MARGIN-LEFT: 2em; MARGIN-RIGHT: 2em } UL.text { MARGIN-LEFT: 2em; MARGIN-RIGHT: 2em } PRE { MARGIN-LEFT: 3em; COLOR: #333333 } UL.toc { COLOR: #000000; LINE-HEIGHT: 16px; FONT-FAMILY: verdana, charcoal, helvetica, arial, sans-serif } H3 { FONT-SIZE: 16px; COLOR: #333333; LINE-HEIGHT: 16px; FONT-FAMILY: helvetica, arial, sans-serif } H4 { FONT-SIZE: 14px; COLOR: #000000; FONT-FAMILY: helvetica, arial, sans-serif } TD.header { FONT-SIZE: 10px; COLOR: #ffffff; FONT-FAMILY: arial, helvetica, san-serif; valign: top } TD.author-text { FONT-SIZE: 10px; COLOR: #000000; FONT-FAMILY: verdana, charcoal, helvetica, arial, sans-serif } TD.author { FONT-WEIGHT: bold; FONT-SIZE: 10px; MARGIN-LEFT: 4em; COLOR: #000000; FONT-FAMILY: verdana, charcoal, helvetica, arial, sans-serif } A:link { FONT-WEIGHT: bold; FONT-SIZE: 10px; TEXT-TRANSFORM: uppercase; COLOR: #990000; FONT-FAMILY: MS Sans Serif, verdana, charcoal, helvetica, arial, sans-serif } A:visited { FONT-WEIGHT: bold; FONT-SIZE: 10px; TEXT-TRANSFORM: uppercase; COLOR: #333333; FONT-FAMILY: MS Sans Serif, verdana, charcoal, helvetica, arial, sans-serif } A:unknown { FONT-WEIGHT: bold; FONT-SIZE: 10px; TEXT-TRANSFORM: uppercase; COLOR: #333333; FONT-FAMILY: MS Sans Serif, verdana, charcoal, helvetica, arial, sans-serif } .link2 { FONT-WEIGHT: bold; FONT-SIZE: 9px; COLOR: #ffffff; FONT-FAMILY: monaco, charcoal, geneva, MS Sans Serif, helvetica, monotype, verdana, sans-serif; TEXT-DECORATION: none } .RFC { FONT-WEIGHT: bold; FONT-SIZE: 9px; COLOR: #666666; FONT-FAMILY: monaco, charcoal, geneva, MS Sans Serif, helvetica, monotype, verdana, sans-serif; TEXT-DECORATION: none } .hotText { FONT-WEIGHT: normal; FONT-SIZE: 9px; COLOR: #ffffff; FONT-FAMILY: charcoal, monaco, geneva, MS Sans Serif, helvetica, monotype, verdana, sans-serif; TEXT-DECORATION: none } TOC Network Working Group C. J. Calabrese Internet-Draft The SANS Institute Expires: July 16, 2001 January 15, 2001 XML for Secure System Logs draft-calabrese-xml-log-00 Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on July 16, 2001. Copyright Notice Copyright (C) The Internet Society (2001). All Rights Reserved. Abstract This memo presents an Extensible Markup Language (XML)[16] encoding that applications may use to express logging information in a structured manner. Thus making these logs easier for computer programs to parse the the free-form data traditionally associated with Syslog[3]-based logs. TOC Table of Contents 1. Introduction 2. Schema Overview 3. Examples 3.1 Nested `log:EVNT' Entries 4. Deviation from Pure XML 5. Implementation Considerations 5.1 Event Filtering 5.2 Event Viewing/Reporting 5.3 Logging API's 5.4 Backward Compatibility 5.4.1 Syslog 5.4.2 NT Event Logger 5.5 Network Communications 6. Security Considerations 6.1 Correctness and Completeness of Identifying Information 6.2 Confidentiality, Integrity, and Availability of Log Messages § References § Author's Address A. XML DTD for the `log:EVNT' Element B. XML DTD for Data Types Used in the `log:EVNT' Element § Full Copyright Statement TOC 1. Introduction One of the most important aspects of system, application, and security administration is event logging. Such logging involves capturing information about important events, possibly transmitting that information to a centralized logging facility, timestamping/filtering/manipulating the information, recording the information to persistent storage, and finally filtering, retrieval and analysis by events by an off-line program or person. This memo presents an Extensible Markup Language (XML)[16] encoding that applications may use to express logging information in a structured manner. Thus making these logs easier for computer programs to parse the the free-form data traditionally associated with Syslog[3]-based logs. This scheme is, for the most part, an "XML-ization" of the Abela and Debeaupuis' Universal Format for Logger Messages (ULM)[11]. A new version of their ULM tool has been written by Nicolas Jombart to deal with an earlier version of this XML-ized format[12]. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119[1]. TOC 2. Schema Overview The basic idea is very straight forward. Each log entry is represented by a 'log:EVNT' element. Inside this element are the following attributes: log:HOST The host on which the log entry was originally generated. log:DATE The date/time at which the log was generated in ' ISO.8601:1998_Date_Format'. [2] log:SEV Severity of the event using the standard Syslog[3] as enumerated in 'log:SEV_Format' (`LOG_DEBUG', `LOG_INFO', `LOG_NOTICE', `LOG_WARNING', `LOG_ERR', `LOG_CRIT', `LOG_ALERT', `LOG_EMERG',). When operating in Windows NT environments, the native NT Event Types (`Information', `Warning', `Error', `Success Audit', and `Falure Audit') should map into `LOG_INFO', `LOG_WARNING',, `LOG_ERR', `LOG_INFO', and `LOG_INFO' respecitively. Similarly, the standard SNMP Trap Severities (`Normal', `Warning', `Minor', `Major', and `Critical') should map into `LOG_INFO', `LOG_WARNING', `LOG_ERR', `LOG_ERR', and `LOG_CRIT' respecitively. NOTE: Need references for NT and SNMP log:PROC.NAME The name of the program generating the event. Can either be a simple name (`Sendmail'), or a hierarchy identifying the program and it's Facility (`Mail/MTA/Sendmail', `Audit/Tripwire', `Kernel/FS/Buffer_Cache'). NOTE: Standard hiercharies need to be specified. log:PROC.ID An identification of the instance of the program/process that generated the event. On a Unix system, this would by the process identification (`PID') plus the thread identifier for multi-threaded processes (`2207.3'). log:PROC.PRIV A list of any security ID's/groups/labels associated with the process generating the log entry (`USR-chris, UID-748, GRP-adm, GRP-sys, GRP-users, EUID-root, EGID-kmem, DAC_READ, SYSTEM_HIGH') NOTE: Need to specify standard nonclemature for Unix, NT, and IEEE 1003.1e/2c priveleges. Also, do we really want to do it this way, or would it be better done as different attribute types, or even nested elements or something? log:PROC.MAIL An e-mail address associated with the usr/login/account the process is running as. log:PROC.TTY The TTY or other description of the user's physical connection to the host generating the event. log:EVENT.TYPE Gives a further taxonomy of the type of event that occurred. Allowed values given under 'log:Event_Type_Enumeration'. XML:LANG The language used for any textual messages in the standard XML `LangCode' format[16]. log:DUR Duration of the event being logged in ' Seconds.Decimel' format. log:SRC.ADDR (log:DST.ADDR) The network source (destination) address of any network traffic represented by this event in 'log:Address_Format'. log:SRC.PROT (log:DST.PROT) The network protocols/ports used by the source (destination) of network traffic in ' log:Protocol_Format' (`IP/TCP/25/ESMTP', `IP/ICMP/Source_Quench'). log:SRC.FQDN (log:DST.FQDN) The fully qualified domain name for the traffic source (destination). log:SRC.NAME (log:DST.NAME) Some other unique identifier for the traffic source (destination). log:SRC.USR (log:DST.USR) A user/login/account associated with the traffic source (destination). log:SRC.MAIL (log.DST.MAIL) An e-mail address associated with the traffic source (destination). log:REL.IN.ADDR, log:REL.IN.PROT, log:REL.IN.FQDN, log:REL.IN.NAME, log:REL.IN.USR, log:REL.IN.MAIL, log:REL.OUT.ADDR, log:REL.OUT.PROT, log:REL.OUT.FQDN, log:REL.OUT.NAME, log:REL.OUT.USR, log:REL.OUT.MAIL Similar to the `log:SRC.' and `log:DST.' attributes, but representing inbound and outbound traffic through a relay or proxy. log:VOL, VOL.SENT, VOL.RCVD Total data volume in octets, volume sent, and volume received. log:CNT, log:CNT.SENT, log:CNT.RCVD Total data volume in records, records sent, and records received. log:PROG.FILE, log:PROG.LINE The name of the program source file and line number within the source file. Typically used for debugging and assert messages. log:DOC Name of an accessed document/file, such as an FTP file, a newsgroup, or a URL. log:CMD Issued command that generated the event. For example: <log:EVNT ... log:CMD="/bin/rm -rf /"> log:MSG Free form message text. TOC 3. Examples 3.1 Nested `log:EVNT' Entries At first blush, it's not obvious why we need a MSG attribute when we could use something like: <log:EVNT ...>this is the text</log:EVNT> The reason is to allow `log:EVNT' elements to nest. One possible interpretation of nested log entries is to be able to represent complex relationships in the data. Instead, here we use nesting to collapse redundant data. For example, the following are considered equivalent: <log:EVNT log:SEV=80 log:PROC.NAME="Audit/Tripwire" log:PROC.ID=1234 log:MSG="unexpected file" log:DOC="/a/b/c"/> <log:EVNT log:SEV=80 log:PROC.NAME="Audit/Tripwire" log:PROC.ID=1234 log:MSG="unexpected file" log:DOC="/a/b/d"/> <log:EVNT log:SEV=80 log:PROC.NAME="Audit/Tripwire" log:PROC.ID=1234 log:MSG="unexpected file" log:DOC="/a/b/e"/> <log:EVNT log:SEV=80 log:PROC.NAME="Audit/Tripwire" log:PROC.ID=1234 log:MSG="unexpected file"> <log:EVNT log:DOC="/a/b/c"/> <log:EVNT log:DOC="/a/b/d"/> <log:EVNT log:DOC="/a/b/e"/> </log:EVNT> As a result, only inner-most `log:EVNT' elements are log entries. Outer `log:EVNT' elements set attributes common to the contained entries. TOC 4. Deviation from Pure XML Although XML is a good fit for the structure needed to describe system logs, it is not a perfect fit. In particular: In the interest of compactness, logs will be considered complete even if they do not contain the standard XML prolog containing the XML version, element declarations, etc. Also in the interest of compactness, log message streams not beginning with `<' will assume to be bracketed by `<log:' and `>'. This way, the existing syslog behavior of sending a single log entry inside a single UDP packet may be preserved. Some might argue that having the free-form messages as `log:MSG' attribute values is somewhat messy from an XML standpoint and that "proper" XML would be to unravel the nesting and put the message text in as the "payload" of the `log:EVNT' elements. Implementations may contain a tool to generate 100% XML compliant "documents" from log files so that the logs may be processed by standard XML tools. TOC 5. Implementation Considerations 5.1 Event Filtering NOTE: Need info about filtering... 5.2 Event Viewing/Reporting NOTE: Need info viewing/reporting... 5.3 Logging API's NOTE: Need info on API's, especially making sure that the TCB generates whatever information it can. Refer to sec section. 5.4 Backward Compatibility 5.4.1 Syslog NOTE: Need info on this..., especially issue of including level/facility info in both XML and native syslog levels. 5.4.2 NT Event Logger NOTE: Need info on this... 5.5 Network Communications NOTE: Need info on this..., including references to Syslog WG work. TOC 6. Security Considerations 6.1 Correctness and Completeness of Identifying Information NOTE: TCB should/must generate log:HOST log:DATE log:SEV log:PROC.NAME log:PROC.ID log:PROC.PRIV 6.2 Confidentiality, Integrity, and Availability of Log Messages The confidentiality, integrity, and availability of log messages or logging mechanisms is beyond the scope of this memo. However, these issues are addressed in the following places. NOTE: need references here... TOC References [1] Bradner, S., Key words for use in RFCs to Indicate Requirement Levels", RFC 2119, BCP 14, March 1997. [2] International Standards Organization, "Data elements and interchange formats - Information interchange - Representation of dates and times", International Standard ISO 8601:1988, available from http://www.iso.ch/markete/8601.pdf , 1988. [3] University of California Computer Systems Research Group, "Unix Programmer's Manual, 4.3 Berkeley Software Distribution, Virtual VAX-11 Version", April 1986. [4] U.S. Department of Defense, Assistant Secretary of Defense for Command, Control, Communications, and Intelligence, "Department of Defense Trusted Computer System Evaluation Criteria", DoD 5200.28-STD, available from http://www.radium.ncsc.mil/tpep/li brary/rainbow/5200.28-STD.html, December 1985. [5] International Standards Organization, "Common Criteria for Information Technology Security Evaluation Version 2.1", International Standard ISO/IEC 15408:1999, available from http://www.radium.ncsc.mil/tpep/li brary/ccitse/ccitse.html, August 1999. [6] U.S. National Security Agency, Information Systems Security Organization, "Labeled Security Protection Profile", available from http://www.radium.ncsc.mil/tpep/li brary/protection_profiles/index.ht ml, October 1999. [7] U.S. National Security Agency, Information Systems Security Organization, "Controlled Access Protection Profile", available from http://www.radium.ncsc.mil/tpep/li brary/protection_profiles/index.ht ml, October 1999. [8] Lonvick, C.M., "syslog Protocol ", draft-ietf-syslog-syslog-03 (work in progress), January 2001. [9] Schneier, B. and J. Kelsey, "Cryptographic Support for Secure Logs on Untrusted Machines", The Seventh USENIX Security Symposium Proceedings, USENIX Press pp. 53-62, available from http://www.counterpane.com/secure- logs.html, January 1998. [10] Schneier, B. and J. Kelsey, "Minimizing Bandwidth for Remote Access to Cryptographically Protected Audit Logs", Second International Workshop on the Recent Advances inIntrusion Detection (RAID '99), available from http://www.counterpane.com/auditlo g2.html, September 1999. [11] Abela, J. and T. Debeaupuis, "Universal Format for Logger Messages", May 1999. [12] Jombart, N., "Private correspondence", May 2000. [13] New, D. and M.T. Rose, "Reliable Delivery for Syslog", draft-ietf-syslog-reliable-02 (work in progress), November 2000. [14] Kelsey, J., "Syslog-Auth Protocol", draft-ietf-syslog-auth-00 (work in progress), December 2000. [15] Kelsey, J., "Syslog-Sign Protocol", draft-ietf-syslog-sign-00 (work in progress), December 2000. [16] World Wide Web Consortium, "Extensible Markup Language (XML) 1.0.", available from http://www.w3.org/TR/REC-xml, February 1998. TOC Author's Address Christopher J. Calabrese The SANS Institute 26 Wellesley Road Upper Montclair, NJ 07043 US Phone: +1 973 233 1464 EMail: chris_calabreseat_private TOC Appendix A. XML DTD for the `log:EVNT' Element <!-- Note that attributes are marked REQUIRED -- here if it is implied by draft-calabrese-requir-loggprot-01. -- This assumes this information is not being provided by -- another source. --> <!ELEMENT log:ENTRY (log:ENTRY)*> <!ATTLIST log:ENTRY log:HOST CDATA #REQUIRED log:DATE ISO.8601:1998_Date_Format #REQUIRED log:SEV log:Severity_Type #REQUIRED log:PROC.NAME CDATA #REQUIRED log:PROC.ID CDATA #REQUIRED log:PROC.PRIV CDATA #REQUIRED log:PROC.MAIL CDATA #IMPLIED log:PROC.TTY CDATA #IMPLIED log:EVENT.TYPE log:Event_Type_Enumeration #IMPLIED log:XML:LANG LangCode "EN" log:DUR Seconds.Decimal #IMPLIED log:SRC.ADDR log:Address_Format #IMPLIED log:SRC.PROT log:Protocol_Format #IMPLIED log:SRC.FQDN CDATA #IMPLIED log:SRC.NAME CDATA #IMPLIED log:SRC.USR CDATA #IMPLIED log:SRC.MAIL CDATA #IMPLIED log:DST.ADDR log:Address_Format #IMPLIED log:DST.PROT log:Protocol_Format #IMPLIED log:DST.FQDN CDATA #IMPLIED log:DST.NAME CDATA #IMPLIED log:DST.USR CDATA #IMPLIED log:DST.MAIL CDATA #IMPLIED log:REL.IN.ADDR log:Address_Format #IMPLIED log:REL.IN.PROT log:Protocol_Format #IMPLIED log:REL.IN.FQDN CDATA #IMPLIED log:REL.IN.NAME CDATA #IMPLIED log:REL.IN.USR CDATA #IMPLIED log:REL.IN.MAIL CDATA #IMPLIED log:REL.OUT.ADDR log:Address_Format #IMPLIED log:REL.OUT.PROT log:Protocol_Format #IMPLIED log:REL.OUT.FQDN CDATA #IMPLIED log:REL.OUT.NAME CDATA #IMPLIED log:REL.OUT.USR CDATA #IMPLIED log:REL.OUT.MAIL CDATA #IMPLIED log:VOL Integer #IMPLIED log:VOL.SENT Integer #IMPLIED log:VOL.RCVD Integer #IMPLIED log:CNT Integer #IMPLIED log:CNT.SENT Integer #IMPLIED log:CNT.RCVD Integer #IMPLIED log:PROG.FILE CDATA #IMPLIED log:PROG.LINE Integer #IMPLIED log:DOC CDATA #IMPLIED log:CMD CDATA #IMPLIED log:MSG CDATA #IMPLIED > TOC Appendix B. XML DTD for Data Types Used in the `log:EVNT' Element <!NOTATION log:Severity_Type ('0'|'1'|...|'99')> <!NOTATION ISO.8601:1998_Date_Format (YYYY '-' MM '-' DD 'T' hh ':' mm [':' ss ['.' f+]]('+' | '-') zzzz)> <!-- NOTE: Need reference and further explanation of date format --> <!NOTATION Seconds.Decimal (ss ['.' f+]) <!NOTATION log:Event_Type_Enumeration ('AUTH.FAIL'|'AUTH.SUCCESS'|'PROG.FAIL'|'PROG.START'|...)> <!-- NOTE: Need more complete enumeration. --> <!NOTATION log:Address_Format (IPv4_Address|IPv6_Address|IPX_Address|...)> <!-- NOTE: Do we need more a complete enumeration? of address types? --> <!NOTATION IPv4_Address CDATA > <!-- NOTE: Need proper definition and reference for IPv4 addresses --> <!NOTATION IPv6_Address CDATA> <!-- NOTE: Need proper definition and reference for IPv6 addresses --> <!NOTATION IPX_Address CDATA> <!-- NOTE: Need proper definition and reference for IPX addresses --> <!NOTATION log:Protocol_Format ( ('IPv4/' ('UDP'|'TCP') '/' IPv4_Port_Number ['/' CDATA]) | ('IPv4/ICMP/' IPv4_ICMP_Message_Type) | ( ('IPv6/' ('UDP'|'TCP') '/' IPv6_Port_Number ['/' CDATA]) | ('IPv6/ICMP/' IP64_ICMP_Message_Type) | ('IPX' ['/SPX/'] IPX_Port_Number ['/' CDATA]) ) <!-- NOTE: Need further definitions of referenced types... --> TOC Full Copyright Statement Copyright (C) The Internet Society (2001). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Acknowledgement Funding for the RFC editor function is currently provided by the Internet Society. Do You Yahoo!? Make international calls for as low as $0.04/minute with Yahoo! Messenger. ---------------------------------------------------------------- The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. --------------------------------------------------------------------- To unsubscribe, e-mail: loganalysis-unsubscribeat_private For additional commands, e-mail: loganalysis-helpat_private
This archive was generated by hypermail 2b30 : Fri Aug 17 2001 - 08:38:53 PDT