Re: [logs] Generic Log Message Parsing Tool

From: Sweth Chandramouli (loganalysisat_private)
Date: Mon Jun 10 2002 - 23:34:37 PDT

  • Next message: Nick Vargish: "Re: [logs] Generic Log Message Parsing Tool" 

    On Mon, Jun 10, 2002 at 05:47:07PM -0400, Marcus J. Ranum wrote:
> The algorithm to do this without using tons of CPU and memory is
> non-trivial but not awful. What you need to be able to do is parse
> a log message into abstract values (host=   queue-id=   etc) and
> then match for matching values of those abstract values.
	The trick to this is that the abstract values usually
have to be context-dependent--the values that you extract from, say,
a sendmail message are different the ones you extract from a BIND
message.  Trying to force all of your messages to fit the same arbitrary
data structure will just cause you headaches.  (As an example that I'm
sure will make Tina wince, try deciding whether the value you should plug
into the "Source IP" field of your predefined data structure for a
message from an IDS machine that has seen a response message from a
machine on your network to a trojan scan from some other machine should
be the IP address of the IDS box (it's the source of the actual message),
the IP address of the machine that responded (it's the source of the
response that triggered the message), or the IP address of the machine
that sent the scan to which the machine on your net responded (it's the
source of the "event" of which the IDS message is alerting you)...)
	The BIG pain is when you start trying to extract values
when you don't know a priori what they represent, where they'll occur,
or how many of them there might be (e.g. an app that scans all of your
log messages, finding ones that contain IP addresses, and then flagging
as related all messages that have the same IP address, regardless of the
application that generated them).  Not pretty at all, especially if the
things on which you're trying to correlate aren't as well-defined as IP
addresses.

	-- Sweth.

-- 
Sweth Chandramouli      Idiopathic Systems Consulting
svcat_private      http://www.idiopathic.net/

---------------------------------------------------------------------
To unsubscribe, e-mail: loganalysis-unsubscribeat_private
For additional commands, e-mail: loganalysis-helpat_private

    This archive was generated by hypermail 2b30 : Tue Jun 11 2002 - 08:33:30 PDT