[logs] Re: regex-less parsing of messages

cfrln@private

answering a few of the straightforward technical questions...

On Dec 6, 2005, at 6:17 PM, Solomon, Frank wrote:

> Okay, Todd, I'll take a byte of your bait.
>
> I looked over the SPLUNK web site.
>
> Are you using the free version or the professional version?
>
> How big/how many log files, servers etc?  How long have you used it?
> The license on the free version has a definite limit, if I read it
> right, so I'm guessing you're using the paid-for version.
The free version does not time out. There are feature differences  
like multiple user accounts, role based security, multiple indexes,  
automated scheduled "live splunks" etc. You can get a free trial of  
the pro product but that is time limited.
>
> How do you maintain a steady-state, is it easy to cleanse the index
> system of the old logs?  Is that somehow automagic?  How do you  
> archive?
There is a built in rotation system that ensures that old data is  
rotated out. see: http://www.splunk.com/index.php/docs?doc=admin.html#92
>
> Am I reading this right?  Is it basically "Google" for your logs?  Are
> the logs actually collected in a single place?
Splunk makes a copy of the log files and adds an index, both in its  
local central datastore.
> Or, do the web queries
> somehow reference the detailed log information "in place?"  What  
> sort of
> database technology is employed?  Is this a Unix hash (dbm-based)  
> system
> or is there a relational engine used in there somewhere or something
> entirely different?
Something entirely different - an advanced purpose built dense data  
index.
>
> On a more fundamental level, does this satisfy your needs when it  
> comes
> to analysis; what happens when the word you're querying for is very
> common?
You refine your search based on other parameters by pointing and  
clicking on other terms, source type, host, etc.
> How do you recognize patterns?  The time graph of the query
> hits looks like a nice feature, are there other similar analysis aids?
> Have you found that useful?
>
> I take it when you say "exactly this" you mean searching the mail  
> logs.
> I can see how that might work if there's a common keyword or queue
> number or whatever.  Sometimes that gets complicated in our  
> environment
> where we have multiple mail routers sharing the load, passing the mail
> through an AV gateway, anti-spam filtering, re-writing headers using
> directory lookup and so forth; each router may be using a different
> number for the same piece of mail.  Do you have a similar situation?
> How did/do you cope with it?
>
> Thanks in advance.
>
> Frank
>
> Frank Solomon
> University of Kentucky
> Lead Systems Programmer, Enterprise Systems
> http://www.franksolomon.net
> "If you give someone a program, you will frustrate them for a day; if
> you teach them how to program, you will frustrate them for a  
> lifetime."
> --Anonymous
>
>
> -----Original Message-----
> [mailto:loganalysis-bounces+frank=email.uky.edu@private] On
> Behalf Of todd.glassey@private
> Sent: Tuesday, December 06, 2005 11:13 AM
> To: Solomon, Frank; LogAnalysis@private
> Subject: [logs] Re: regex-less parsing of messages
>
> We use SPLUNK for exactly this.
>
> Todd
> _______________________________________________
> LogAnalysis mailing list
> LogAnalysis@private
> http://lists.shmoo.com/mailman/listinfo/loganalysis
>

_______________________________________________
LogAnalysis mailing list
LogAnalysis@private
http://lists.shmoo.com/mailman/listinfo/loganalysis