RE: [loganalysis] Logging standards and such

From: Chris Calabrese (chris_calabreseat_private)
Date: Wed Aug 15 2001 - 11:49:16 PDT

  • Next message: Corey Steele: "Re: [loganalysis] Logging standards and such"

    > However, I have gotten some interest in it from time> to time, and I offer it here as a good starting> point...Oh bother, it looks like my attachments didn't make it through some step of the way.Tina, are you stripping attachments?Hmm, here's HTML.  If you want this in ASCII, send me an e-mail... (it's easier to do the HTML because the Yahoo mail client reformats the text bits if they're not attachments)--------------------------XML for Secure System Logs.title { color: #990000; font-size: 22px; line-height: 22px; font-weight: bold; text-align: right;             font-family: helvetica, arial, sans-serif }    .filename { color: #666666; font-size: 18px; line-height: 28px; font-weight: bold; text-align: right;                  font-family: helvetica, arial, sans-serif }    p.copyright { color: #000000; font-size: 10px;                  font-family: verdana, charcoal, helvetica, arial, sans-serif }    p { margin-left: 2em; margin-right: 2em; }    ol { margin-left: 2em; margin-right: 2em; }    ul.text { margin-left: 2em; margin-right: 2em; }    pre { margin-left: 3em; color: #333333 }    ul.toc { color: #000000; line-height: 16px;             font-family: verdana, charcoal, helvetica, arial, sans-serif }    H3 { color: #333333; font-size: 16px; line-height: 16px; font-family: helvetica, arial, sans-serif }    H4 { color: #000000; font-size: 14px; font-family: helvetica, arial, sans-serif }    TD.header { color: #ffffff; font-size: 10px; font-family: arial, helvetica, san-serif; valign: top }    TD.author-text { color: #000000; font-size: 10px;                     font-family: verdana, charcoal, helvetica, arial, sans-serif }    TD.author { color: #000000; font-weight: bold; margin-left: 4em; font-size: 10px; font-family: verdana, charcoal, helvetica, arial, sans-serif }    A:link { color: #990000; font-size: 10px; text-transform: uppercase; font-weight: bold;             font-family: MS Sans Serif, verdana, charcoal, helvetica, arial, sans-serif }    A:visited { color: #333333; font-weight: bold; font-size: 10px; 
    text-transform: uppercase;                font-family: MS Sans Serif, verdana, charcoal, helvetica, arial, sans-serif }    A:name { color: #333333; font-weight: bold; font-size: 10px; text-transform: uppercase;             font-family: MS Sans Serif, verdana, charcoal, helvetica, arial, sans-serif }    .link2 { color:#ffffff; font-weight: bold; text-decoration: none;             font-family: monaco, charcoal, geneva, MS Sans Serif, helvetica, monotype, verdana, sans-serif;             font-size: 9px }    .RFC { color:#666666; font-weight: bold; text-decoration: none;           font-family: monaco, charcoal, geneva, MS Sans Serif, helvetica, monotype, verdana, sans-serif;           font-size: 9px }    .hotText { color:#ffffff; font-weight: normal; text-decoration: none;               font-family: charcoal, monaco, geneva, MS Sans Serif, helvetica, monotype, verdana, sans-serif;               font-size: 9px } TOC 
    Network Working GroupC. J. CalabreseInternet-DraftThe SANS InstituteExpires: July 16, 2001January 15, 2001
    XML for Secure System Logsdraft-calabrese-xml-log-00Status of this Memo
    This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026.
    
    Internet-Drafts are working documents of the Internet EngineeringTask Force (IETF), its areas, and its working groups.Note that other groups may also distribute working documents asInternet-Drafts.
    
    Internet-Drafts are draft documents valid for a maximum of six monthsand may be updated, replaced, or obsoleted by other documents at any time.It is inappropriate to use Internet-Drafts as reference material or to citethem other than as "work in progress."
    
    The list of current Internet-Drafts can be accessed athttp://www.ietf.org/ietf/1id-abstracts.txt.
    
    The list of Internet-Draft Shadow Directories can be accessed athttp://www.ietf.org/shadow.html.
    
    This Internet-Draft will expire on July 16, 2001.
    Copyright Notice
    Copyright (C) The Internet Society (2001). All Rights Reserved.
    Abstract
    This memo presents an Extensible Markup Language		(XML)[16]		encoding that applications may use to express logging		information in a structured manner.  Thus making these logs		easier for computer programs to parse the the free-form		data traditionally associated with		Syslog[3]-based logs.		
    
    
    ---------------------------------
     TOC 
    Table of Contents1. Introduction
    2. Schema Overview
    3. Examples
    3.1 Nested `log:EVNT' Entries
    4. Deviation from Pure XML
    5. Implementation Considerations
    5.1 Event Filtering
    5.2 Event Viewing/Reporting
    5.3 Logging API's
    5.4 Backward Compatibility
    5.4.1 Syslog
    5.4.2 NT Event Logger
    5.5 Network Communications
    6. Security Considerations
    6.1 Correctness and Completeness of Identifying Information
    6.2 Confidentiality, Integrity, and Availability of Log Messages
    #167; References
    #167; Author's Address
    A. XML DTD for the `log:EVNT' Element
    B. XML DTD for Data Types Used in the `log:EVNT' Element
    #167; Full Copyright Statement
    
    
    
    
    ---------------------------------
     TOC 
    1. Introduction
    One of the most important aspects of system, application,		and security administration is event logging.		Such logging involves capturing information about		important events, possibly transmitting that information		to a centralized logging facility,		timestamping/filtering/manipulating the information,		recording the information to persistent storage,		and finally filtering, retrieval and analysis by events		by an off-line program or person.		  
    
    This memo presents an Extensible Markup Language		(XML)[16]		encoding that applications may use to express logging		information in a structured manner.  Thus making these logs		easier for computer programs to parse the the free-form		data traditionally associated with		Syslog[3]-based logs.		  
    
    This  scheme is, for the most part, an		"XML-ization" of the Abela and Debeaupuis'		Universal Format for Logger Messages		(ULM)[11].		A new version of their ULM tool		has been written by Nicolas Jombart to deal with an		earlier version of this XML-ized		format[12].		  
    
    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL",	        "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED",		"MAY", and "OPTIONAL" in this document are to be		interpreted as described in		RFC 2119[1].		
    
    
    ---------------------------------
     TOC 
    2. Schema Overview
    The basic idea is very straight forward.  Each log entry		is represented by a 'log:EVNT' element.  Inside this		element are the following attributes:		
    log:HOST
       The host on which the log entry was originally			generated.			
    log:DATE
       The date/time at which the log was generated in			'ISO.8601:1998_Date_Format'.			[2]
    log:SEV
       Severity of the event using the standard			Syslog[3]			as enumerated in			'log:SEV_Format'			(`LOG_DEBUG',			 `LOG_INFO',			 `LOG_NOTICE',			 `LOG_WARNING',			 `LOG_ERR',			 `LOG_CRIT',			 `LOG_ALERT',			 `LOG_EMERG',).			  
    When operating in Windows NT environments,			the native NT Event Types (`Information', `Warning',			`Error', `Success Audit', and `Falure Audit')			should map into `LOG_INFO', `LOG_WARNING',,			`LOG_ERR', `LOG_INFO', and `LOG_INFO'			respecitively.			  
    
    Similarly, the standard SNMP Trap Severities			(`Normal', `Warning', `Minor', `Major',			and `Critical') should map into `LOG_INFO',			`LOG_WARNING', `LOG_ERR', `LOG_ERR', and			`LOG_CRIT' respecitively.			  
    
    NOTE: Need references for NT and SNMP			
    log:PROC.NAME
       The name of the program generating the event.			Can either be a simple name (`Sendmail'),			or a hierarchy identifying the program and			it's Facility (`Mail/MTA/Sendmail',			`Audit/Tripwire', `Kernel/FS/Buffer_Cache').			  			NOTE: Standard hiercharies need to be specified.			
    log:PROC.ID
       An identification of the instance of the			program/process that generated the event.			On a Unix system, this would by the			process identification (`PID')			plus the thread identifier for multi-threaded			processes (`2207.3').			
    log:PROC.PRIV
       A list of any security ID's/groups/labels			associated with the process generating the log			entry (`USR-chris, UID-748, GRP-adm, GRP-sys,			GRP-users, EUID-root, EGID-kmem, DAC_READ,			SYSTEM_HIGH')			  			NOTE: Need to specify standard nonclemature for			Unix, NT, and IEEE 1003.1e/2c priveleges.			Also, do we really want to do it this way, or			would it be better done as different attribute			types, or even nested elements or something?			
    log:PROC.MAIL
       An e-mail address associated with the			usr/login/account the process is running as.			
    log:PROC.TTY
       The TTY or other description of the			user's physical connection to the			host generating the event.			
    log:EVENT.TYPE
       Gives a further taxonomy of the type of event that			occurred.  Allowed values given under			'log:Event_Type_Enumeration'.			
    XML:LANG
       The language used for any textual messages in the			standard XML `LangCode'			format[16].			
    log:DUR
       Duration of the event being logged in			'Seconds.Decimel' format.			
    log:SRC.ADDR (log:DST.ADDR)
       The network source (destination)			address of any network traffic represented			by this event in			'log:Address_Format'.			
    log:SRC.PROT (log:DST.PROT)
       The network protocols/ports used by the			source (destination) of network traffic in			'log:Protocol_Format'			(`IP/TCP/25/ESMTP', `IP/ICMP/Source_Quench').			
    log:SRC.FQDN (log:DST.FQDN)
       The fully qualified domain name for			the traffic source (destination).			
    log:SRC.NAME (log:DST.NAME)
       Some other unique identifier for the			traffic source (destination).			
    log:SRC.USR (log:DST.USR)
       A user/login/account associated with the			traffic source (destination).			
    log:SRC.MAIL (log.DST.MAIL)
       An e-mail address associated with the			traffic source (destination).			
    log:REL.IN.ADDR, log:REL.IN.PROT, log:REL.IN.FQDN, log:REL.IN.NAME, log:REL.IN.USR, log:REL.IN.MAIL, log:REL.OUT.ADDR, log:REL.OUT.PROT, log:REL.OUT.FQDN, log:REL.OUT.NAME, log:REL.OUT.USR, log:REL.OUT.MAIL
       Similar to the `log:SRC.' and `log:DST.'			attributes, but representing inbound and outbound			traffic through a relay or proxy.			
    log:VOL, VOL.SENT, VOL.RCVD
       Total data volume in octets, volume sent, and			volume received.			
    log:CNT, log:CNT.SENT, log:CNT.RCVD
       Total data volume in records, records sent, and			records received.			
    log:PROG.FILE, log:PROG.LINE
       The name of the program source file and line			number within the source file.  Typically used for			debugging and assert messages.			
    log:DOC
       Name of an accessed document/file, such as an FTP			file, a newsgroup, or a URL.			
    log:CMD
       Issued command that generated the event.			For example:			
    <log:EVNT ... log:CMD="/bin/rm -rf /">
    
    log:MSG
       Free form message text.			
    
    
    ---------------------------------
     TOC 
    3. Examples3.1 Nested `log:EVNT' Entries
    At first blush, it's not obvious why we need a MSG		    attribute when we could use something like:		    
    <log:EVNT ...>this is the text</log:EVNT>
    The reason is to allow `log:EVNT' elements to nest.		    One possible interpretation of nested log entries is to		    be able to represent complex relationships in the data.		    Instead, here we use nesting to collapse redundant data.		    For example, the following are considered equivalent:		    
    <log:EVNT log:SEV=80 log:PROC.NAME="Audit/Tripwire" log:PROC.ID=1234 log:MSG="unexpected file" log:DOC="/a/b/c"/><log:EVNT log:SEV=80 log:PROC.NAME="Audit/Tripwire" log:PROC.ID=1234 log:MSG="unexpected file" log:DOC="/a/b/d"/><log:EVNT log:SEV=80 log:PROC.NAME="Audit/Tripwire" log:PROC.ID=1234 log:MSG="unexpected file" log:DOC="/a/b/e"/><log:EVNT log:SEV=80 log:PROC.NAME="Audit/Tripwire"log:PROC.ID=1234 log:MSG="unexpected file"><log:EVNT log:DOC="/a/b/c"/><log:EVNT log:DOC="/a/b/d"/><log:EVNT log:DOC="/a/b/e"/></log:EVNT>
    As a result, only inner-most `log:EVNT'		    elements are log entries.		    Outer `log:EVNT' elements set		    attributes common to the contained entries.		    
    
    
    ---------------------------------
     TOC 
    4. Deviation from Pure XML
    Although XML is a good fit for the structure needed to describe		system logs, it is not a perfect fit.  In particular:		
       In the interest of compactness, logs will be			considered complete even if they do not contain			the standard XML prolog containing the XML version,			element declarations, etc.			
       Also in the interest of compactness, log message streams			not beginning with `<' will assume to be			bracketed by `<log:' and `>'.			This way, the existing syslog behavior of sending a			single log entry inside a single UDP packet may be			preserved.			
       Some might argue that having the free-form messages as			`log:MSG' attribute values is somewhat messy from an XML			standpoint and that "proper" XML would be to unravel the			nesting and put the message text in as the "payload"			of the `log:EVNT' elements.			
    Implementations may contain a tool to generate 100% XML		compliant "documents" from log files so that the logs may		be processed by standard XML tools.		
    
    
    ---------------------------------
     TOC 
    5. Implementation Considerations5.1 Event Filtering
    NOTE: Need info about filtering...		    
    5.2 Event Viewing/Reporting
    NOTE: Need info viewing/reporting...		    
    5.3 Logging API's
    NOTE: Need info on API's, especially making sure that		    the TCB generates whatever information it can.		    Refer to sec section.		    
    5.4 Backward Compatibility5.4.1 Syslog
    NOTE: Need info on this..., especially issue of			including level/facility info in both XML and			native syslog levels.			
    5.4.2 NT Event Logger
    NOTE: Need info on this...			
    5.5 Network Communications
    NOTE: Need info on this..., including references			to Syslog WG work.			
    
    
    ---------------------------------
     TOC 
    6. Security Considerations6.1 Correctness and Completeness of Identifying Information
    NOTE: TCB should/must generate		    log:HOST log:DATE log:SEV		    log:PROC.NAME log:PROC.ID log:PROC.PRIV		    
    6.2 Confidentiality, Integrity, and Availability of Log Messages
    The confidentiality, integrity, and availability of log		    messages or logging mechanisms is beyond the scope of		    this memo.  However, these issues are addressed in the		    following places.		      
    NOTE: need references here...		    
    
    
    ---------------------------------
     TOC 
    References[1]Bradner, S., "Key words for use in RFCs to			Indicate Requirement Levels", RFC 2119, BCP 14, March 1997.[2]International Standards Organization, "Data elements and interchange formats -			Information interchange - Representation of dates			and times", International Standard ISO 8601:1988, available from http://www.iso.ch/markete/8601.pdf, 1988.[3]University of California Computer Systems			    Research Group, "Unix Programmer's Manual, 4.3 Berkeley			Software Distribution, Virtual VAX-11 Version", April 1986.[4]U.S. Department of Defense,			    Assistant Secretary of Defense for			    Command, Control, Communications,			    and Intelligence, "Department of Defense Trusted Computer System			Evaluation Criteria", DoD 5200.28-STD, available from http://www.radium.ncsc.mil/tpep/library/rainbow/5200.28-STD.html, December 1985.[5]International Standards Organization, "Common Criteria for Information Technology			Security Evaluation Version 2.1", International Standard ISO/IEC 15408:1999, available from http://www.radium.ncsc.mil/tpep/library/ccitse/ccitse.html, August 1999.[6]U.S. National Security Agency,			    Information Systems Security Organization, "Labeled Security Protection Profile", available from http://www.radium.ncsc.mil/tpep/library/protection_profiles/index.html, October 1999.[7]U.S. National Security Agency,			    Information Systems Security Organization, "Controlled Access Protection Profile", available from http://www.radium.ncsc.mil/tpep/library/protection_profiles/index.html, October 1999.[8]Lonvick, C.M., "syslog Protocol", draft-ietf-syslog-syslog-03 (work in progress), January 2001.[9]Schneier, B. and J. Kelsey, "Cryptographic Support for Secure			Logs on Untrusted Machines", The Seventh USENIX Security Symposium Proceedings, USENIX Press pp. 53-62, available from http://www.counterpane.com/secure-logs.html, January 1998.[10]Schneier, B. and J. Kelsey, "Minimizing Bandwidth for Remote Access to			Cryptographically Protected Audit Logs", Second International Workshop on the Recent Ad
    vances inIntrusion Detection (RAID '99), available from http://www.counterpane.com/auditlog2.html, September 1999.[11]Abela, J. and T. Debeaupuis, "Universal Format for Logger Messages", May 1999.[12]Jombart, N., "Private correspondence", May 2000.[13]New, D. and M.T. Rose, "Reliable Delivery for Syslog", draft-ietf-syslog-reliable-02 (work in progress), November 2000.[14]Kelsey, J., "Syslog-Auth Protocol", draft-ietf-syslog-auth-00 (work in progress), December 2000.[15]Kelsey, J., "Syslog-Sign Protocol", draft-ietf-syslog-sign-00 (work in progress), December 2000.[16]World Wide Web Consortium, "Extensible Markup Language (XML) 1.0.", available from http://www.w3.org/TR/REC-xml, February 1998.
    
    ---------------------------------
     TOC 
    Author's Address Christopher J. Calabrese The SANS Institute 26 Wellesley Road Upper Montclair, NJ  07043 USPhone: +1 973 233 1464EMail: chris_calabreseat_private
    
    ---------------------------------
     TOC 
    Appendix A. XML DTD for the `log:EVNT' Element
    
    <!-- Note that attributes are marked REQUIRED  -- here if it is implied by draft-calabrese-requir-loggprot-01.  -- This assumes this information is not being provided by  -- another source.  --><!ELEMENT log:ENTRY (log:ENTRY)*><!ATTLIST log:ENTRY    log:HOST CDATA #REQUIRED    log:DATE ISO.8601:1998_Date_Format #REQUIRED    log:SEV log:Severity_Type #REQUIRED    log:PROC.NAME CDATA #REQUIRED    log:PROC.ID CDATA #REQUIRED    log:PROC.PRIV CDATA #REQUIRED    log:PROC.MAIL CDATA #IMPLIED    log:PROC.TTY CDATA #IMPLIED    log:EVENT.TYPE log:Event_Type_Enumeration #IMPLIED    log:XML:LANG LangCode "EN"    log:DUR Seconds.Decimal #IMPLIED    log:SRC.ADDR log:Address_Format #IMPLIED    log:SRC.PROT log:Protocol_Format #IMPLIED    log:SRC.FQDN CDATA #IMPLIED    log:SRC.NAME CDATA #IMPLIED    log:SRC.USR CDATA #IMPLIED    log:SRC.MAIL CDATA #IMPLIED    log:DST.ADDR log:Address_Format #IMPLIED    log:DST.PROT log:Protocol_Format #IMPLIED    log:DST.FQDN CDATA #IMPLIED    log:DST.NAME CDATA #IMPLIED    log:DST.USR CDATA #IMPLIED    log:DST.MAIL CDATA #IMPLIED    log:REL.IN.ADDR log:Address_Format #IMPLIED    log:REL.IN.PROT log:Protocol_Format #IMPLIED    log:REL.IN.FQDN CDATA #IMPLIED    log:REL.IN.NAME CDATA #IMPLIED    log:REL.IN.USR CDATA #IMPLIED    log:REL.IN.MAIL CDATA #IMPLIED    log:REL.OUT.ADDR log:Address_Format #IMPLIED    log:REL.OUT.PROT log:Protocol_Format #IMPLIED    log:REL.OUT.FQDN CDATA #IMPLIED    log:REL.OUT.NAME CDATA #IMPLIED    log:REL.OUT.USR CDATA #IMPLIED    log:REL.OUT.MAIL CDATA #IMPLIED    log:VOL Integer #IMPLIED    log:VOL.SENT Integer #IMPLIED    log:VOL.RCVD Integer #IMPLIED    log:CNT Integer #IMPLIED    log:CNT.SENT Integer #IMPLIED    log:CNT.RCVD Integer #IMPLIED    log:PROG.FILE CDATA #IMPLIED    log:PROG.LINE Integer #IMPLIED    log:DOC CDATA #IMPLIED    log:CMD CDATA #IMPLIED    log:MSG CDATA #IMPLIED    >
    
    
    
    ---------------------------------
     TOC 
    Appendix B. XML DTD for Data Types Used in the `log:EVNT' Element
    
    <!NOTATION log:Severity_Type ('0'|'1'|...|'99')><!NOTATION ISO.8601:1998_Date_Format    (YYYY '-' MM '-' DD 'T' hh ':'     mm [':' ss ['.' f+]]('+' | '-') zzzz)><!-- NOTE: Need reference and further     explanation of date format --><!NOTATION Seconds.Decimal    (ss ['.' f+])<!NOTATION log:Event_Type_Enumeration     ('AUTH.FAIL'|'AUTH.SUCCESS'|'PROG.FAIL'|'PROG.START'|...)><!-- NOTE: Need more complete enumeration. --><!NOTATION log:Address_Format    (IPv4_Address|IPv6_Address|IPX_Address|...)><!-- NOTE: Do we need more a complete     enumeration? of address types? --><!NOTATION IPv4_Address CDATA ><!-- NOTE: Need proper definition     and reference for IPv4 addresses --><!NOTATION IPv6_Address CDATA><!-- NOTE: Need proper definition     and reference for IPv6 addresses --><!NOTATION IPX_Address CDATA><!-- NOTE: Need proper definition     and reference for IPX addresses --><!NOTATION log:Protocol_Format    ( ('IPv4/' ('UDP'|'TCP') '/' IPv4_Port_Number ['/' CDATA])    | ('IPv4/ICMP/' IPv4_ICMP_Message_Type)    | ( ('IPv6/' ('UDP'|'TCP') '/' IPv6_Port_Number ['/' CDATA])    | ('IPv6/ICMP/' IP64_ICMP_Message_Type)    | ('IPX' ['/SPX/'] IPX_Port_Number ['/' CDATA])    )    <!-- NOTE: Need further definitions of     referenced types... -->
    
    
    
    ---------------------------------
     TOC 
    Full Copyright Statement
    Copyright (C) The Internet Society (2001). All Rights Reserved.
    
    This document and translations of it may be copied and furnished toothers, and derivative works that comment on or otherwise explain itor assist in its implementation may be prepared, copied, published anddistributed, in whole or in part, without restriction of any kind,provided that the above copyright notice and this paragraph areincluded on all such copies and derivative works. However, thisdocument itself may not be modified in any way, such as by removingthe copyright notice or references to the Internet Society or otherInternet organizations, except as needed for the purpose ofdeveloping Internet standards in which case the procedures forcopyrights defined in the Internet Standards process must befollowed, or as required to translate it into languages other thanEnglish.
    
    The limited permissions granted above are perpetual and will not berevoked by the Internet Society or its successors or assigns.
    
    This document and the information contained herein is provided on an"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERINGTASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDINGBUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATIONHEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OFMERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
    Acknowledgement
    Funding for the RFC editor function is currently provided by theInternet Society.
    
    
    
    ---------------------------------
    Do You Yahoo!?
    Make international calls for as low as $0.04/minute with Yahoo! Messenger.
    



    This archive was generated by hypermail 2b30 : Wed Aug 15 2001 - 15:40:13 PDT