[logs] Re: regex-less parsing of messages

frank@private

Okay, Todd, I'll take a byte of your bait.

I looked over the SPLUNK web site.

Are you using the free version or the professional version?

How big/how many log files, servers etc?  How long have you used it?
The license on the free version has a definite limit, if I read it
right, so I'm guessing you're using the paid-for version.

How do you maintain a steady-state, is it easy to cleanse the index
system of the old logs?  Is that somehow automagic?  How do you archive?

Am I reading this right?  Is it basically "Google" for your logs?  Are
the logs actually collected in a single place?  Or, do the web queries
somehow reference the detailed log information "in place?"  What sort of
database technology is employed?  Is this a Unix hash (dbm-based) system
or is there a relational engine used in there somewhere or something
entirely different?

On a more fundamental level, does this satisfy your needs when it comes
to analysis; what happens when the word you're querying for is very
common?  How do you recognize patterns?  The time graph of the query
hits looks like a nice feature, are there other similar analysis aids?
Have you found that useful?

I take it when you say "exactly this" you mean searching the mail logs.
I can see how that might work if there's a common keyword or queue
number or whatever.  Sometimes that gets complicated in our environment
where we have multiple mail routers sharing the load, passing the mail
through an AV gateway, anti-spam filtering, re-writing headers using
directory lookup and so forth; each router may be using a different
number for the same piece of mail.  Do you have a similar situation?
How did/do you cope with it?

Thanks in advance.

Frank

Frank Solomon
University of Kentucky
Lead Systems Programmer, Enterprise Systems
http://www.franksolomon.net
"If you give someone a program, you will frustrate them for a day; if
you teach them how to program, you will frustrate them for a lifetime."
--Anonymous

-----Original Message-----
[mailto:loganalysis-bounces+frank=email.uky.edu@private] On
Behalf Of todd.glassey@private
Sent: Tuesday, December 06, 2005 11:13 AM
To: Solomon, Frank; LogAnalysis@private
Subject: [logs] Re: regex-less parsing of messages

We use SPLUNK for exactly this.

Todd
_______________________________________________
LogAnalysis mailing list
LogAnalysis@private
http://lists.shmoo.com/mailman/listinfo/loganalysis