Re: [logs] Syslog payload format

From: Chris Calabrese (chris_calabreseat_private)
Date: Thu Dec 19 2002 - 07:03:10 PST

  • Next message: Mikael Olsson: "Re: [logs] Syslog payload format" 

    --- Rainer Gerhards <rgerhardsat_private> wrote:
> [...]
> - A (now expired) Internet Draft on the issue
>   http://www.hsc.fr/gulp/draft-abela-ulm-05.txt
>   Though it obviously did not receive too much support, I still find
> it interesting and useful.

FYI, I was was working on an XML-ization of the Abela and Debeaupuis
ULM format with some other folks from the IETF WG for a while.

Some of the ideas on extensions for log signing made it into the
syslog-sign RFC, but the rest was dropped because the WG charter
doesn't cover payload format.

This work might be a very good starting point, hoever, since it's
partly baked, there's already code that handles it (Nicolas Jombart of
HSC, the company Abela and Debeaupuis work form modified the ULM/syslog
code they have to handle it), and there was some definite vendor
interest in the idea at the time.

I'm too busy to pick this back up now (all my time allocated for this
type activity is being taken up by the Center for Internet Security
benchmark programs these days). But everyone should feel free to build
on what I've got as long as they credit me.

I'm attaching the doc in HTML and TXT format. I've also got the
original RFC 2629 XML source (yes, you can write RFC's in XML and
publish them in HTML!) if anyone wants to start tinkering with it.


__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com
    XML for Secure System Logs
     TOC 
    Network Working GroupC. J. Calabrese
    Internet-DraftThe SANS Institute
    Expires: July 16, 2001January 15, 2001

    XML for Secure System Logs
    draft-calabrese-xml-log-00

    Status of this Memo

    This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026.

    Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

    Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

    The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt.

    The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.

    This Internet-Draft will expire on July 16, 2001.

    Copyright Notice

    Copyright (C) The Internet Society (2001). All Rights Reserved.

    Abstract

    This memo presents an Extensible Markup Language (XML)[16] encoding that applications may use to express logging information in a structured manner. Thus making these logs easier for computer programs to parse the the free-form data traditionally associated with Syslog[3]-based logs.


     TOC 

    Table of Contents
     TOC 

    1. Introduction

    One of the most important aspects of system, application, and security administration is event logging. Such logging involves capturing information about important events, possibly transmitting that information to a centralized logging facility, timestamping/filtering/manipulating the information, recording the information to persistent storage, and finally filtering, retrieval and analysis by events by an off-line program or person.

    This memo presents an Extensible Markup Language (XML)[16] encoding that applications may use to express logging information in a structured manner. Thus making these logs easier for computer programs to parse the the free-form data traditionally associated with Syslog[3]-based logs.

    This scheme is, for the most part, an "XML-ization" of the Abela and Debeaupuis' Universal Format for Logger Messages (ULM)[11]. A new version of their ULM tool has been written by Nicolas Jombart to deal with an earlier version of this XML-ized format[12].

    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119[1].


     TOC 

    2. Schema Overview

    The basic idea is very straight forward. Each log entry is represented by a 'log:EVNT' element. Inside this element are the following attributes:

    log:HOST
    The host on which the log entry was originally generated.
    log:DATE
    The date/time at which the log was generated in 'ISO.8601:1998_Date_Format'. [2]
    log:SEV
    Severity of the event using the standard Syslog[3] as enumerated in 'log:SEV_Format' (`LOG_DEBUG', `LOG_INFO', `LOG_NOTICE', `LOG_WARNING', `LOG_ERR', `LOG_CRIT', `LOG_ALERT', `LOG_EMERG',).
    When operating in Windows NT environments, the native NT Event Types (`Information', `Warning', `Error', `Success Audit', and `Falure Audit') should map into `LOG_INFO', `LOG_WARNING',, `LOG_ERR', `LOG_INFO', and `LOG_INFO' respecitively.

    Similarly, the standard SNMP Trap Severities (`Normal', `Warning', `Minor', `Major', and `Critical') should map into `LOG_INFO', `LOG_WARNING', `LOG_ERR', `LOG_ERR', and `LOG_CRIT' respecitively.

    NOTE: Need references for NT and SNMP
    log:PROC.NAME
    The name of the program generating the event. Can either be a simple name (`Sendmail'), or a hierarchy identifying the program and it's Facility (`Mail/MTA/Sendmail', `Audit/Tripwire', `Kernel/FS/Buffer_Cache'). NOTE: Standard hiercharies need to be specified.
    log:PROC.ID
    An identification of the instance of the program/process that generated the event. On a Unix system, this would by the process identification (`PID') plus the thread identifier for multi-threaded processes (`2207.3').
    log:PROC.PRIV
    A list of any security ID's/groups/labels associated with the process generating the log entry (`USR-chris, UID-748, GRP-adm, GRP-sys, GRP-users, EUID-root, EGID-kmem, DAC_READ, SYSTEM_HIGH') NOTE: Need to specify standard nonclemature for Unix, NT, and IEEE 1003.1e/2c priveleges. Also, do we really want to do it this way, or would it be better done as different attribute types, or even nested elements or something?
    log:PROC.MAIL
    An e-mail address associated with the usr/login/account the process is running as.
    log:PROC.TTY
    The TTY or other description of the user's physical connection to the host generating the event.
    log:EVENT.TYPE
    Gives a further taxonomy of the type of event that occurred. Allowed values given under 'log:Event_Type_Enumeration'.
    XML:LANG
    The language used for any textual messages in the standard XML `LangCode' format[16].
    log:DUR
    Duration of the event being logged in 'Seconds.Decimel' format.
    log:SRC.ADDR (log:DST.ADDR)
    The network source (destination) address of any network traffic represented by this event in 'log:Address_Format'.
    log:SRC.PROT (log:DST.PROT)
    The network protocols/ports used by the source (destination) of network traffic in 'log:Protocol_Format' (`IP/TCP/25/ESMTP', `IP/ICMP/Source_Quench').
    log:SRC.FQDN (log:DST.FQDN)
    The fully qualified domain name for the traffic source (destination).
    log:SRC.NAME (log:DST.NAME)
    Some other unique identifier for the traffic source (destination).
    log:SRC.USR (log:DST.USR)
    A user/login/account associated with the traffic source (destination).
    log:SRC.MAIL (log.DST.MAIL)
    An e-mail address associated with the traffic source (destination).
    log:REL.IN.ADDR, log:REL.IN.PROT, log:REL.IN.FQDN, log:REL.IN.NAME, log:REL.IN.USR, log:REL.IN.MAIL, log:REL.OUT.ADDR, log:REL.OUT.PROT, log:REL.OUT.FQDN, log:REL.OUT.NAME, log:REL.OUT.USR, log:REL.OUT.MAIL
    Similar to the `log:SRC.' and `log:DST.' attributes, but representing inbound and outbound traffic through a relay or proxy.
    log:VOL, VOL.SENT, VOL.RCVD
    Total data volume in octets, volume sent, and volume received.
    log:CNT, log:CNT.SENT, log:CNT.RCVD
    Total data volume in records, records sent, and records received.
    log:PROG.FILE, log:PROG.LINE
    The name of the program source file and line number within the source file. Typically used for debugging and assert messages.
    log:DOC
    Name of an accessed document/file, such as an FTP file, a newsgroup, or a URL.
    log:CMD
    Issued command that generated the event. For example: 
    <log:EVNT ... log:CMD="/bin/rm -rf /">
    log:MSG
    Free form message text.


     TOC 

    3. Examples

    3.1 Nested `log:EVNT' Entries

    At first blush, it's not obvious why we need a MSG attribute when we could use something like: 

    <log:EVNT ...>this is the text</log:EVNT>
    The reason is to allow `log:EVNT' elements to nest. One possible interpretation of nested log entries is to be able to represent complex relationships in the data. Instead, here we use nesting to collapse redundant data. For example, the following are considered equivalent: 
    <log:EVNT log:SEV=80 log:PROC.NAME="Audit/Tripwire"
 log:PROC.ID=1234 log:MSG="unexpected file"
 log:DOC="/a/b/c"/>
<log:EVNT log:SEV=80 log:PROC.NAME="Audit/Tripwire"
 log:PROC.ID=1234 log:MSG="unexpected file"
 log:DOC="/a/b/d"/>
<log:EVNT log:SEV=80 log:PROC.NAME="Audit/Tripwire"
 log:PROC.ID=1234 log:MSG="unexpected file"
 log:DOC="/a/b/e"/>

<log:EVNT log:SEV=80 log:PROC.NAME="Audit/Tripwire"
log:PROC.ID=1234 log:MSG="unexpected file">
    <log:EVNT log:DOC="/a/b/c"/>
    <log:EVNT log:DOC="/a/b/d"/>
    <log:EVNT log:DOC="/a/b/e"/>
    </log:EVNT>
    As a result, only inner-most `log:EVNT' elements are log entries. Outer `log:EVNT' elements set attributes common to the contained entries.


     TOC 

    4. Deviation from Pure XML

    Although XML is a good fit for the structure needed to describe system logs, it is not a perfect fit. In particular:

    1. In the interest of compactness, logs will be considered complete even if they do not contain the standard XML prolog containing the XML version, element declarations, etc.
    2. Also in the interest of compactness, log message streams not beginning with `<' will assume to be bracketed by `<log:' and `>'. This way, the existing syslog behavior of sending a single log entry inside a single UDP packet may be preserved.
    3. Some might argue that having the free-form messages as `log:MSG' attribute values is somewhat messy from an XML standpoint and that "proper" XML would be to unravel the nesting and put the message text in as the "payload" of the `log:EVNT' elements.
    Implementations may contain a tool to generate 100% XML compliant "documents" from log files so that the logs may be processed by standard XML tools.


     TOC 

    5. Implementation Considerations

    5.1 Event Filtering

    NOTE: Need info about filtering...

    5.2 Event Viewing/Reporting

    NOTE: Need info viewing/reporting...

    5.3 Logging API's

    NOTE: Need info on API's, especially making sure that the TCB generates whatever information it can. Refer to sec section.

    5.4 Backward Compatibility

    5.4.1 Syslog

    NOTE: Need info on this..., especially issue of including level/facility info in both XML and native syslog levels.

    5.4.2 NT Event Logger

    NOTE: Need info on this...

    5.5 Network Communications

    NOTE: Need info on this..., including references to Syslog WG work.


     TOC 

    6. Security Considerations

    6.1 Correctness and Completeness of Identifying Information

    NOTE: TCB should/must generate log:HOST log:DATE log:SEV log:PROC.NAME log:PROC.ID log:PROC.PRIV

    6.2 Confidentiality, Integrity, and Availability of Log Messages

    The confidentiality, integrity, and availability of log messages or logging mechanisms is beyond the scope of this memo. However, these issues are addressed in the following places.
    NOTE: need references here...


     TOC 

    References

    [1] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", RFC 2119, BCP 14, March 1997.
    [2] International Standards Organization, "Data elements and interchange formats - Information interchange - Representation of dates and times", International Standard ISO 8601:1988, available from http://www.iso.ch/markete/8601.pdf, 1988.
    [3] University of California Computer Systems Research Group, "Unix Programmer's Manual, 4.3 Berkeley Software Distribution, Virtual VAX-11 Version", April 1986.
    [4] U.S. Department of Defense, Assistant Secretary of Defense for Command, Control, Communications, and Intelligence, "Department of Defense Trusted Computer System Evaluation Criteria", DoD 5200.28-STD, available from http://www.radium.ncsc.mil/tpep/library/rainbow/5200.28-STD.html, December 1985.
    [5] International Standards Organization, "Common Criteria for Information Technology Security Evaluation Version 2.1", International Standard ISO/IEC 15408:1999, available from http://www.radium.ncsc.mil/tpep/library/ccitse/ccitse.html, August 1999.
    [6] U.S. National Security Agency, Information Systems Security Organization, "Labeled Security Protection Profile", available from http://www.radium.ncsc.mil/tpep/library/protection_profiles/index.html, October 1999.
    [7] U.S. National Security Agency, Information Systems Security Organization, "Controlled Access Protection Profile", available from http://www.radium.ncsc.mil/tpep/library/protection_profiles/index.html, October 1999.
    [8] Lonvick, C.M., "syslog Protocol", draft-ietf-syslog-syslog-03 (work in progress), January 2001.
    [9] Schneier, B. and J. Kelsey, "Cryptographic Support for Secure Logs on Untrusted Machines", The Seventh USENIX Security Symposium Proceedings, USENIX Press pp. 53-62, available from http://www.counterpane.com/secure-logs.html, January 1998.
    [10] Schneier, B. and J. Kelsey, "Minimizing Bandwidth for Remote Access to Cryptographically Protected Audit Logs", Second International Workshop on the Recent Advances inIntrusion Detection (RAID '99), available from http://www.counterpane.com/auditlog2.html, September 1999.
    [11] Abela, J. and T. Debeaupuis, "Universal Format for Logger Messages", May 1999.
    [12] Jombart, N., "Private correspondence", May 2000.
    [13] New, D. and M.T. Rose, "Reliable Delivery for Syslog", draft-ietf-syslog-reliable-02 (work in progress), November 2000.
    [14] Kelsey, J., "Syslog-Auth Protocol", draft-ietf-syslog-auth-00 (work in progress), December 2000.
    [15] Kelsey, J., "Syslog-Sign Protocol", draft-ietf-syslog-sign-00 (work in progress), December 2000.
    [16] World Wide Web Consortium, "Extensible Markup Language (XML) 1.0.", available from http://www.w3.org/TR/REC-xml, February 1998.
     TOC 

    Author's Address

      Christopher J. Calabrese
      The SANS Institute
      26 Wellesley Road
      Upper Montclair, NJ 07043
      US
    Phone:  +1 973 233 1464
    EMail:  chris_calabreseat_private
     TOC 

    Appendix A. XML DTD for the `log:EVNT' Element

    <!-- Note that attributes are marked REQUIRED
  -- here if it is implied by draft-calabrese-requir-loggprot-01.
  -- This assumes this information is not being provided by
  -- another source.
  -->
<!ELEMENT log:ENTRY (log:ENTRY)*>
<!ATTLIST log:ENTRY
    log:HOST CDATA #REQUIRED
    log:DATE ISO.8601:1998_Date_Format #REQUIRED
    log:SEV log:Severity_Type #REQUIRED
    log:PROC.NAME CDATA #REQUIRED
    log:PROC.ID CDATA #REQUIRED
    log:PROC.PRIV CDATA #REQUIRED
    log:PROC.MAIL CDATA #IMPLIED
    log:PROC.TTY CDATA #IMPLIED
    log:EVENT.TYPE log:Event_Type_Enumeration #IMPLIED
    log:XML:LANG LangCode "EN"
    log:DUR Seconds.Decimal #IMPLIED
    log:SRC.ADDR log:Address_Format #IMPLIED
    log:SRC.PROT log:Protocol_Format #IMPLIED
    log:SRC.FQDN CDATA #IMPLIED
    log:SRC.NAME CDATA #IMPLIED
    log:SRC.USR CDATA #IMPLIED
    log:SRC.MAIL CDATA #IMPLIED
    log:DST.ADDR log:Address_Format #IMPLIED
    log:DST.PROT log:Protocol_Format #IMPLIED
    log:DST.FQDN CDATA #IMPLIED
    log:DST.NAME CDATA #IMPLIED
    log:DST.USR CDATA #IMPLIED
    log:DST.MAIL CDATA #IMPLIED
    log:REL.IN.ADDR log:Address_Format #IMPLIED
    log:REL.IN.PROT log:Protocol_Format #IMPLIED
    log:REL.IN.FQDN CDATA #IMPLIED
    log:REL.IN.NAME CDATA #IMPLIED
    log:REL.IN.USR CDATA #IMPLIED
    log:REL.IN.MAIL CDATA #IMPLIED
    log:REL.OUT.ADDR log:Address_Format #IMPLIED
    log:REL.OUT.PROT log:Protocol_Format #IMPLIED
    log:REL.OUT.FQDN CDATA #IMPLIED
    log:REL.OUT.NAME CDATA #IMPLIED
    log:REL.OUT.USR CDATA #IMPLIED
    log:REL.OUT.MAIL CDATA #IMPLIED
    log:VOL Integer #IMPLIED
    log:VOL.SENT Integer #IMPLIED
    log:VOL.RCVD Integer #IMPLIED
    log:CNT Integer #IMPLIED
    log:CNT.SENT Integer #IMPLIED
    log:CNT.RCVD Integer #IMPLIED
    log:PROG.FILE CDATA #IMPLIED
    log:PROG.LINE Integer #IMPLIED
    log:DOC CDATA #IMPLIED
    log:CMD CDATA #IMPLIED
    log:MSG CDATA #IMPLIED
    >


     TOC 

    Appendix B. XML DTD for Data Types Used in the `log:EVNT' Element

    <!NOTATION log:Severity_Type ('0'|'1'|...|'99')>
<!NOTATION ISO.8601:1998_Date_Format
    (YYYY '-' MM '-' DD 'T' hh ':'
     mm [':' ss ['.' f+]]('+' | '-') zzzz)>
    <!-- NOTE: Need reference and further
     explanation of date format -->
<!NOTATION Seconds.Decimal
    (ss ['.' f+])
<!NOTATION log:Event_Type_Enumeration 
    ('AUTH.FAIL'|'AUTH.SUCCESS'|'PROG.FAIL'|'PROG.START'|...)>
    <!-- NOTE: Need more complete enumeration. -->
<!NOTATION log:Address_Format
    (IPv4_Address|IPv6_Address|IPX_Address|...)>
    <!-- NOTE: Do we need more a complete
     enumeration? of address types? -->
<!NOTATION IPv4_Address CDATA >
    <!-- NOTE: Need proper definition
     and reference for IPv4 addresses -->
<!NOTATION IPv6_Address CDATA>
    <!-- NOTE: Need proper definition
     and reference for IPv6 addresses -->
<!NOTATION IPX_Address CDATA>
    <!-- NOTE: Need proper definition
     and reference for IPX addresses -->
<!NOTATION log:Protocol_Format
    ( ('IPv4/' ('UDP'|'TCP') '/' IPv4_Port_Number ['/' CDATA])
    | ('IPv4/ICMP/' IPv4_ICMP_Message_Type)
    | ( ('IPv6/' ('UDP'|'TCP') '/' IPv6_Port_Number ['/' CDATA])
    | ('IPv6/ICMP/' IP64_ICMP_Message_Type)
    | ('IPX' ['/SPX/'] IPX_Port_Number ['/' CDATA])
    )
    <!-- NOTE: Need further definitions of
     referenced types... -->


     TOC 

    Full Copyright Statement

    Copyright (C) The Internet Society (2001). All Rights Reserved.

    This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.

    The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

    This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

    Acknowledgement

    Funding for the RFC editor function is currently provided by the Internet Society.

     Network Working Group C. J. Calabrese Internet-Draft The SANS Institute Expires: July 16, 2001 January 15, 2001 XML for Secure System Logs draft-calabrese-xml-log-00 Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on July 16, 2001. Copyright Notice Copyright (C) The Internet Society (2001). All Rights Reserved. Abstract This memo presents an Extensible Markup Language (XML)[16] encoding that applications may use to express logging information in a structured manner. Thus making these logs easier for computer programs to parse the the free-form data traditionally associated with Syslog[3]-based logs. Calabrese Expires July 16, 2001 [Page 1] Internet-Draft XML for Secure System Logs January 2001 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Schema Overview . . . . . . . . . . . . . . . . . . . . . . 4 3. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3.1 Nested `log:EVNT' Entries . . . . . . . . . . . . . . . . . 7 4. Deviation from Pure XML . . . . . . . . . . . . . . . . . . 8 5. Implementation Considerations . . . . . . . . . . . . . . . 9 5.1 Event Filtering . . . . . . . . . . . . . . . . . . . . . . 9 5.2 Event Viewing/Reporting . . . . . . . . . . . . . . . . . . 9 5.3 Logging API's . . . . . . . . . . . . . . . . . . . . . . . 9 5.4 Backward Compatibility . . . . . . . . . . . . . . . . . . . 9 5.4.1 Syslog . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 5.4.2 NT Event Logger . . . . . . . . . . . . . . . . . . . . . . 9 5.5 Network Communications . . . . . . . . . . . . . . . . . . . 9 6. Security Considerations . . . . . . . . . . . . . . . . . . 10 6.1 Correctness and Completeness of Identifying Information . . 10 6.2 Confidentiality, Integrity, and Availability of Log Messages 10 References . . . . . . . . . . . . . . . . . . . . . . . . . 11 Author's Address . . . . . . . . . . . . . . . . . . . . . . 12 A. XML DTD for the `log:EVNT' Element . . . . . . . . . . . . . 13 B. XML DTD for Data Types Used in the `log:EVNT' Element . . . 15 Full Copyright Statement . . . . . . . . . . . . . . . . . . 16 Calabrese Expires July 16, 2001 [Page 2] Internet-Draft XML for Secure System Logs January 2001 1. Introduction One of the most important aspects of system, application, and security administration is event logging. Such logging involves capturing information about important events, possibly transmitting that information to a centralized logging facility, timestamping/filtering/manipulating the information, recording the information to persistent storage, and finally filtering, retrieval and analysis by events by an off-line program or person. This memo presents an Extensible Markup Language (XML)[16] encoding that applications may use to express logging information in a structured manner. Thus making these logs easier for computer programs to parse the the free-form data traditionally associated with Syslog[3]-based logs. This scheme is, for the most part, an "XML-ization" of the Abela and Debeaupuis' Universal Format for Logger Messages (ULM)[11]. A new version of their ULM tool has been written by Nicolas Jombart to deal with an earlier version of this XML-ized format[12]. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119[1]. Calabrese Expires July 16, 2001 [Page 3] Internet-Draft XML for Secure System Logs January 2001 2. Schema Overview The basic idea is very straight forward. Each log entry is represented by a 'log:EVNT' element. Inside this element are the following attributes: log:HOST The host on which the log entry was originally generated. log:DATE The date/time at which the log was generated in 'ISO.8601:1998_Date_Format (Appendix B)'. [2] log:SEV Severity of the event using the standard Syslog[3] as enumerated in 'log:SEV_Format (Appendix B)' (`LOG_DEBUG', `LOG_INFO', `LOG_NOTICE', `LOG_WARNING', `LOG_ERR', `LOG_CRIT', `LOG_ALERT', `LOG_EMERG',). When operating in Windows NT environments, the native NT Event Types (`Information', `Warning', `Error', `Success Audit', and `Falure Audit') should map into `LOG_INFO', `LOG_WARNING',, `LOG_ERR', `LOG_INFO', and `LOG_INFO' respecitively. Similarly, the standard SNMP Trap Severities (`Normal', `Warning', `Minor', `Major', and `Critical') should map into `LOG_INFO', `LOG_WARNING', `LOG_ERR', `LOG_ERR', and `LOG_CRIT' respecitively. NOTE: Need references for NT and SNMP log:PROC.NAME The name of the program generating the event. Can either be a simple name (`Sendmail'), or a hierarchy identifying the program and it's Facility (`Mail/MTA/Sendmail', `Audit/Tripwire', `Kernel/FS/Buffer_Cache'). NOTE: Standard hiercharies need to be specified. log:PROC.ID An identification of the instance of the program/process that generated the event. On a Unix system, this would by the process identification (`PID') plus the thread identifier for multi-threaded processes (`2207.3'). log:PROC.PRIV A list of any security ID's/groups/labels associated with the process generating the log entry (`USR-chris, UID-748, GRP-adm, GRP-sys, GRP-users, EUID-root, EGID-kmem, DAC_READ, SYSTEM_HIGH') NOTE: Need to specify standard nonclemature for Unix, NT, and IEEE 1003.1e/2c priveleges. Also, do we really want to do it this way, or would it be better done as different attribute types, or even nested elements or something? log:PROC.MAIL An e-mail address associated with the Calabrese Expires July 16, 2001 [Page 4] Internet-Draft XML for Secure System Logs January 2001 usr/login/account the process is running as. log:PROC.TTY The TTY or other description of the user's physical connection to the host generating the event. log:EVENT.TYPE Gives a further taxonomy of the type of event that occurred. Allowed values given under 'log:Event_Type_Enumeration (Appendix B)'. XML:LANG The language used for any textual messages in the standard XML `LangCode' format[16]. log:DUR Duration of the event being logged in 'Seconds.Decimel (Appendix B)' format. log:SRC.ADDR (log:DST.ADDR) The network source (destination) address of any network traffic represented by this event in 'log:Address_Format (Appendix B)'. log:SRC.PROT (log:DST.PROT) The network protocols/ports used by the source (destination) of network traffic in 'log:Protocol_Format (Appendix B)' (`IP/TCP/25/ESMTP', `IP/ICMP/Source_Quench'). log:SRC.FQDN (log:DST.FQDN) The fully qualified domain name for the traffic source (destination). log:SRC.NAME (log:DST.NAME) Some other unique identifier for the traffic source (destination). log:SRC.USR (log:DST.USR) A user/login/account associated with the traffic source (destination). log:SRC.MAIL (log.DST.MAIL) An e-mail address associated with the traffic source (destination). log:REL.IN.ADDR, log:REL.IN.PROT, log:REL.IN.FQDN, log:REL.IN.NAME, log:REL.IN.USR, log:REL.IN.MAIL, log:REL.OUT.ADDR, log:REL.OUT.PROT, log:REL.OUT.FQDN, log:REL.OUT.NAME, log:REL.OUT.USR, log:REL.OUT.MAIL Similar to the `log:SRC.' and `log:DST.' attributes, but representing inbound and outbound traffic through a relay or proxy. log:VOL, VOL.SENT, VOL.RCVD Total data volume in octets, volume sent, and volume received. log:CNT, log:CNT.SENT, log:CNT.RCVD Total data volume in records, records sent, and records received. log:PROG.FILE, log:PROG.LINE The name of the program source file and Calabrese Expires July 16, 2001 [Page 5] Internet-Draft XML for Secure System Logs January 2001 line number within the source file. Typically used for debugging and assert messages. log:DOC Name of an accessed document/file, such as an FTP file, a newsgroup, or a URL. log:CMD Issued command that generated the event. For example: <log:EVNT ... log:CMD="/bin/rm -rf /"> log:MSG Free form message text. Calabrese Expires July 16, 2001 [Page 6] Internet-Draft XML for Secure System Logs January 2001 3. Examples 3.1 Nested `log:EVNT' Entries At first blush, it's not obvious why we need a MSG attribute when we could use something like: <log:EVNT ...>this is the text</log:EVNT> The reason is to allow `log:EVNT' elements to nest. One possible interpretation of nested log entries is to be able to represent complex relationships in the data. Instead, here we use nesting to collapse redundant data. For example, the following are considered equivalent: <log:EVNT log:SEV=80 log:PROC.NAME="Audit/Tripwire" log:PROC.ID=1234 log:MSG="unexpected file" log:DOC="/a/b/c"/> <log:EVNT log:SEV=80 log:PROC.NAME="Audit/Tripwire" log:PROC.ID=1234 log:MSG="unexpected file" log:DOC="/a/b/d"/> <log:EVNT log:SEV=80 log:PROC.NAME="Audit/Tripwire" log:PROC.ID=1234 log:MSG="unexpected file" log:DOC="/a/b/e"/> <log:EVNT log:SEV=80 log:PROC.NAME="Audit/Tripwire" log:PROC.ID=1234 log:MSG="unexpected file"> <log:EVNT log:DOC="/a/b/c"/> <log:EVNT log:DOC="/a/b/d"/> <log:EVNT log:DOC="/a/b/e"/> </log:EVNT> As a result, only inner-most `log:EVNT' elements are log entries. Outer `log:EVNT' elements set attributes common to the contained entries. Calabrese Expires July 16, 2001 [Page 7] Internet-Draft XML for Secure System Logs January 2001 4. Deviation from Pure XML Although XML is a good fit for the structure needed to describe system logs, it is not a perfect fit. In particular: 1. In the interest of compactness, logs will be considered complete even if they do not contain the standard XML prolog containing the XML version, element declarations, etc. 2. Also in the interest of compactness, log message streams not beginning with `<' will assume to be bracketed by `<log:' and `>'. This way, the existing syslog behavior of sending a single log entry inside a single UDP packet may be preserved. 3. Some might argue that having the free-form messages as `log:MSG' attribute values is somewhat messy from an XML standpoint and that "proper" XML would be to unravel the nesting and put the message text in as the "payload" of the `log:EVNT' elements. Implementations may contain a tool to generate 100% XML compliant "documents" from log files so that the logs may be processed by standard XML tools. Calabrese Expires July 16, 2001 [Page 8] Internet-Draft XML for Secure System Logs January 2001 5. Implementation Considerations 5.1 Event Filtering NOTE: Need info about filtering... 5.2 Event Viewing/Reporting NOTE: Need info viewing/reporting... 5.3 Logging API's NOTE: Need info on API's, especially making sure that the TCB generates whatever information it can. Refer to sec section. 5.4 Backward Compatibility 5.4.1 Syslog NOTE: Need info on this..., especially issue of including level/facility info in both XML and native syslog levels. 5.4.2 NT Event Logger NOTE: Need info on this... 5.5 Network Communications NOTE: Need info on this..., including references to Syslog WG work. Calabrese Expires July 16, 2001 [Page 9] Internet-Draft XML for Secure System Logs January 2001 6. Security Considerations 6.1 Correctness and Completeness of Identifying Information NOTE: TCB should/must generate log:HOST log:DATE log:SEV log:PROC.NAME log:PROC.ID log:PROC.PRIV 6.2 Confidentiality, Integrity, and Availability of Log Messages The confidentiality, integrity, and availability of log messages or logging mechanisms is beyond the scope of this memo. However, these issues are addressed in the following places. NOTE: need references here... Calabrese Expires July 16, 2001 [Page 10] Internet-Draft XML for Secure System Logs January 2001 References [1] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", RFC 2119, BCP 14, March 1997. [2] International Standards Organization, "Data elements and interchange formats - Information interchange - Representation of dates and times", International Standard ISO 8601:1988, available from http://www.iso.ch/markete/8601.pdf, 1988. [3] University of California Computer Systems Research Group, "Unix Programmer's Manual, 4.3 Berkeley Software Distribution, Virtual VAX-11 Version", April 1986. [4] U.S. Department of Defense, Assistant Secretary of Defense for Command, Control, Communications, and Intelligence, "Department of Defense Trusted Computer System Evaluation Criteria", DoD 5200.28-STD, available from http://www.radium.ncsc.mil/tpep/library/rainbow/5200.28-STD.html , December 1985. [5] International Standards Organization, "Common Criteria for Information Technology Security Evaluation Version 2.1", International Standard ISO/IEC 15408:1999, available from http://www.radium.ncsc.mil/tpep/library/ccitse/ccitse.html, August 1999. [6] U.S. National Security Agency, Information Systems Security Organization, "Labeled Security Protection Profile", available from http://www.radium.ncsc.mil/tpep/library/protection_profiles/index.html , October 1999. [7] U.S. National Security Agency, Information Systems Security Organization, "Controlled Access Protection Profile", available from http://www.radium.ncsc.mil/tpep/library/protection_profiles/index.html , October 1999. [8] Lonvick, C.M., "syslog Protocol", draft-ietf-syslog-syslog-03 (work in progress), January 2001. [9] Schneier, B. and J. Kelsey, "Cryptographic Support for Secure Logs on Untrusted Machines", The Seventh USENIX Security Symposium Proceedings, USENIX Press pp. 53-62, available from http://www.counterpane.com/secure-logs.html, January 1998. [10] Schneier, B. and J. Kelsey, "Minimizing Bandwidth for Remote Access to Cryptographically Protected Audit Logs", Second Calabrese Expires July 16, 2001 [Page 11] Internet-Draft XML for Secure System Logs January 2001 International Workshop on the Recent Advances inIntrusion Detection (RAID '99), available from http://www.counterpane.com/auditlog2.html, September 1999. [11] Abela, J. and T. Debeaupuis, "Universal Format for Logger Messages", May 1999. [12] Jombart, N., "Private correspondence", May 2000. [13] New, D. and M.T. Rose, "Reliable Delivery for Syslog", draft-ietf-syslog-reliable-02 (work in progress), November 2000. [14] Kelsey, J., "Syslog-Auth Protocol", draft-ietf-syslog-auth-00 (work in progress), December 2000. [15] Kelsey, J., "Syslog-Sign Protocol", draft-ietf-syslog-sign-00 (work in progress), December 2000. [16] World Wide Web Consortium, "Extensible Markup Language (XML) 1.0.", available from http://www.w3.org/TR/REC-xml, February 1998. Author's Address Christopher J. Calabrese The SANS Institute 26 Wellesley Road Upper Montclair, NJ 07043 US Phone: +1 973 233 1464 EMail: chris_calabreseat_private Calabrese Expires July 16, 2001 [Page 12] Internet-Draft XML for Secure System Logs January 2001 Appendix A. XML DTD for the `log:EVNT' Element <!-- Note that attributes are marked REQUIRED -- here if it is implied by draft-calabrese-requir-loggprot-01. -- This assumes this information is not being provided by -- another source. --> <!ELEMENT log:ENTRY (log:ENTRY)*> <!ATTLIST log:ENTRY log:HOST CDATA #REQUIRED log:DATE ISO.8601:1998_Date_Format #REQUIRED log:SEV log:Severity_Type #REQUIRED log:PROC.NAME CDATA #REQUIRED log:PROC.ID CDATA #REQUIRED log:PROC.PRIV CDATA #REQUIRED log:PROC.MAIL CDATA #IMPLIED log:PROC.TTY CDATA #IMPLIED log:EVENT.TYPE log:Event_Type_Enumeration #IMPLIED log:XML:LANG LangCode "EN" log:DUR Seconds.Decimal #IMPLIED log:SRC.ADDR log:Address_Format #IMPLIED log:SRC.PROT log:Protocol_Format #IMPLIED log:SRC.FQDN CDATA #IMPLIED log:SRC.NAME CDATA #IMPLIED log:SRC.USR CDATA #IMPLIED log:SRC.MAIL CDATA #IMPLIED log:DST.ADDR log:Address_Format #IMPLIED log:DST.PROT log:Protocol_Format #IMPLIED log:DST.FQDN CDATA #IMPLIED log:DST.NAME CDATA #IMPLIED log:DST.USR CDATA #IMPLIED log:DST.MAIL CDATA #IMPLIED log:REL.IN.ADDR log:Address_Format #IMPLIED log:REL.IN.PROT log:Protocol_Format #IMPLIED log:REL.IN.FQDN CDATA #IMPLIED log:REL.IN.NAME CDATA #IMPLIED log:REL.IN.USR CDATA #IMPLIED log:REL.IN.MAIL CDATA #IMPLIED log:REL.OUT.ADDR log:Address_Format #IMPLIED log:REL.OUT.PROT log:Protocol_Format #IMPLIED log:REL.OUT.FQDN CDATA #IMPLIED log:REL.OUT.NAME CDATA #IMPLIED log:REL.OUT.USR CDATA #IMPLIED log:REL.OUT.MAIL CDATA #IMPLIED log:VOL Integer #IMPLIED log:VOL.SENT Integer #IMPLIED log:VOL.RCVD Integer #IMPLIED log:CNT Integer #IMPLIED Calabrese Expires July 16, 2001 [Page 13] Internet-Draft XML for Secure System Logs January 2001 log:CNT.SENT Integer #IMPLIED log:CNT.RCVD Integer #IMPLIED log:PROG.FILE CDATA #IMPLIED log:PROG.LINE Integer #IMPLIED log:DOC CDATA #IMPLIED log:CMD CDATA #IMPLIED log:MSG CDATA #IMPLIED > Calabrese Expires July 16, 2001 [Page 14] Internet-Draft XML for Secure System Logs January 2001 Appendix B. XML DTD for Data Types Used in the `log:EVNT' Element <!NOTATION log:Severity_Type ('0'|'1'|...|'99')> <!NOTATION ISO.8601:1998_Date_Format (YYYY '-' MM '-' DD 'T' hh ':' mm [':' ss ['.' f+]]('+' | '-') zzzz)> <!-- NOTE: Need reference and further explanation of date format --> <!NOTATION Seconds.Decimal (ss ['.' f+]) <!NOTATION log:Event_Type_Enumeration ('AUTH.FAIL'|'AUTH.SUCCESS'|'PROG.FAIL'|'PROG.START'|...)> <!-- NOTE: Need more complete enumeration. --> <!NOTATION log:Address_Format (IPv4_Address|IPv6_Address|IPX_Address|...)> <!-- NOTE: Do we need more a complete enumeration? of address types? --> <!NOTATION IPv4_Address CDATA > <!-- NOTE: Need proper definition and reference for IPv4 addresses --> <!NOTATION IPv6_Address CDATA> <!-- NOTE: Need proper definition and reference for IPv6 addresses --> <!NOTATION IPX_Address CDATA> <!-- NOTE: Need proper definition and reference for IPX addresses --> <!NOTATION log:Protocol_Format ( ('IPv4/' ('UDP'|'TCP') '/' IPv4_Port_Number ['/' CDATA]) | ('IPv4/ICMP/' IPv4_ICMP_Message_Type) | ( ('IPv6/' ('UDP'|'TCP') '/' IPv6_Port_Number ['/' CDATA]) | ('IPv6/ICMP/' IP64_ICMP_Message_Type) | ('IPX' ['/SPX/'] IPX_Port_Number ['/' CDATA]) ) <!-- NOTE: Need further definitions of referenced types... --> Calabrese Expires July 16, 2001 [Page 15] Internet-Draft XML for Secure System Logs January 2001 Full Copyright Statement Copyright (C) The Internet Society (2001). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Acknowledgement Funding for the RFC editor function is currently provided by the Internet Society. Calabrese Expires July 16, 2001 [Page 16] _______________________________________________ LogAnalysis mailing list LogAnalysisat_private http://lists.shmoo.com/mailman/listinfo/loganalysis

    This archive was generated by hypermail 2b30 : Thu Dec 19 2002 - 19:28:32 PST