[logs] Re: regex-less parsing of messages

todd.glassey@private

Frank the win is in a pre-built set of LOG SPLUNKS to look for certain well defined issues in logging, and a practice procedure that codifies this for SOX work.

Todd
 -------------- Original message ----------------------
From: Christina Noren <cfrln@private>
> answering a few of the straightforward technical questions...
> 
> On Dec 6, 2005, at 6:17 PM, Solomon, Frank wrote:
> 
> > Okay, Todd, I'll take a byte of your bait.
> >
> > I looked over the SPLUNK web site.
> >
> > Are you using the free version or the professional version?
> >
> > How big/how many log files, servers etc?  How long have you used it?
> > The license on the free version has a definite limit, if I read it
> > right, so I'm guessing you're using the paid-for version.
> The free version does not time out. There are feature differences  
> like multiple user accounts, role based security, multiple indexes,  
> automated scheduled "live splunks" etc. You can get a free trial of  
> the pro product but that is time limited.
> >
> > How do you maintain a steady-state, is it easy to cleanse the index
> > system of the old logs?  Is that somehow automagic?  How do you  
> > archive?
> There is a built in rotation system that ensures that old data is  
> rotated out. see: http://www.splunk.com/index.php/docs?doc=admin.html#92
> >
> > Am I reading this right?  Is it basically "Google" for your logs?  Are
> > the logs actually collected in a single place?
> Splunk makes a copy of the log files and adds an index, both in its  
> local central datastore.
> > Or, do the web queries
> > somehow reference the detailed log information "in place?"  What  
> > sort of
> > database technology is employed?  Is this a Unix hash (dbm-based)  
> > system
> > or is there a relational engine used in there somewhere or something
> > entirely different?
> Something entirely different - an advanced purpose built dense data  
> index.
> >
> > On a more fundamental level, does this satisfy your needs when it  
> > comes
> > to analysis; what happens when the word you're querying for is very
> > common?
> You refine your search based on other parameters by pointing and  
> clicking on other terms, source type, host, etc.
> > How do you recognize patterns?  The time graph of the query
> > hits looks like a nice feature, are there other similar analysis aids?
> > Have you found that useful?
> >
> > I take it when you say "exactly this" you mean searching the mail  
> > logs.
> > I can see how that might work if there's a common keyword or queue
> > number or whatever.  Sometimes that gets complicated in our  
> > environment
> > where we have multiple mail routers sharing the load, passing the mail
> > through an AV gateway, anti-spam filtering, re-writing headers using
> > directory lookup and so forth; each router may be using a different
> > number for the same piece of mail.  Do you have a similar situation?
> > How did/do you cope with it?
> >
> > Thanks in advance.
> >
> > Frank
> >
> > Frank Solomon
> > University of Kentucky
> > Lead Systems Programmer, Enterprise Systems
> > http://www.franksolomon.net
> > "If you give someone a program, you will frustrate them for a day; if
> > you teach them how to program, you will frustrate them for a  
> > lifetime."
> > --Anonymous
> >
> >
> > -----Original Message-----
> > [mailto:loganalysis-bounces+frank=email.uky.edu@private] On
> > Behalf Of todd.glassey@private
> > Sent: Tuesday, December 06, 2005 11:13 AM
> > To: Solomon, Frank; LogAnalysis@private
> > Subject: [logs] Re: regex-less parsing of messages
> >
> > We use SPLUNK for exactly this.
> >
> > Todd
> > _______________________________________________
> > LogAnalysis mailing list
> > LogAnalysis@private
> > http://lists.shmoo.com/mailman/listinfo/loganalysis
> >
> 
> _______________________________________________
> LogAnalysis mailing list
> LogAnalysis@private
> http://lists.shmoo.com/mailman/listinfo/loganalysis

_______________________________________________
LogAnalysis mailing list
LogAnalysis@private
http://lists.shmoo.com/mailman/listinfo/loganalysis