RE: [logs] Generic Log Message Parsing Tool

From: Dale.Drewat_private
Date: Mon Jun 10 2002 - 09:26:06 PDT

  • Next message: Marcus J. Ranum: "Re: [logs] Generic Log Message Parsing Tool"

    APE (http://www.hackertracker.org/cst/ape.tar
    <http://www.hackertracker.org/cst/ape.tar> ) supports multiple line parsing.
    Basically, you create dependency rules to parse multiple lines.
     
    Rule 1: look for X, when you see it, jump to rule 1.1
    Rule 1.1; look for Y, when you see it, evaluate it against a security policy
     
    Dale
     
    -----Original Message-----
    From: Rajkumar S. [mailto:listuserat_private] 
    Sent: Monday, June 10, 2002 9:25 AM
    To: Log Analysis Mailing List
    Subject: Re: [logs] Generic Log Message Parsing Tool
     
    Hello, 
    I have been following this discussion with great interest. Here are some 
    random questions and ideas that came up in my mind. All this might be 
    absolute rubbish, so please be gentle ;) 
    How are we going to parse logs that represent a single event in multiple 
    lines. For example the case of qmail where for an event "an email 
    send" generates multiple entries in logs. This gets more interesting when 
    we have a central log collector for multiple qmail servers in the network. 
    I imagine this parser tool to convert all the logs (the actual text 
    strings) in the network to some sort of events, that can be later 
    analyzed. I imagine an event as, well an event, with some parameters :) 
    For example, the event "an email send" will have parameters like from 
    addr, to addr and size. An event "a web page accessed" will have 
    parameters size, url and who accessed it, etc... 
    The events have to be some sort of normalized log messages which are 
    platform and daemon independent. I am waiting for Tina'a mail for more 
    info on this. These events can be fed into database or xml etc for further 
    processing like anomaly detection or mundane analysis like the MB of data 
    transmitted via email. 
    Can we have a configuration file, rather a collection of files like the 
    logrotate.d where we can drop in the config file for each daemon. This 
    will enable us to mix and match the set of daemons that we want to parse 
    according to our setup rather than to have a single file. Each file in the 
    logparse.d will have complete info for a single daemon. 
    As a corollary to this will it be possible to assign the config file and 
    the corresponding engine using a main syslog.conf style conf file. 
    For example 
    host1.mail.*                    qmail.parser 
    host2.mail.*                    sendmail.parser 
     
    Btw, I have some students who are interested to work on some thing like 
    this. If we can hammer out a neat spec for the log tool I can assign this 
    to them. 
    raj 
    
    
    
    --------------------------------------------------------------------- 
    To unsubscribe, e-mail: loganalysis-unsubscribeat_private 
    For additional commands, e-mail: loganalysis-helpat_private 
    



    This archive was generated by hypermail 2b30 : Mon Jun 10 2002 - 09:54:54 PDT