Re: AW: [logs] Webserver logs to database - Toward data mining

From: Dennis Jenkins (djenkinsat_private)
Date: Thu Sep 20 2001 - 08:52:51 PDT

  • Next message: Alexander Gretencord: "Re: AW: [logs] Webserver logs to database - Toward data mining"

    	I would suggest that before you insert the data into any database that
    you find some way to remove log entries that are un-important, like
    images and such.
    
    Nistor.Lubomir@Star-21.De wrote:
    > 
    > I was thinking as well about log processing..
    > You just have to design a script, that will take over logs from apache,
    > split it into fields and put it into SQL, that can be sitting on one
    > server...
    > 
    > But, I'm not that good in scripting to split one line into fields..
    > everything else is fine..
    > 
    > anyway you could do it in C, or in perl or whatever.. I need some binary for
    > it..
    > 
    > anybody got ideas? examples?
    > 
    > -----Ursprüngliche Nachricht-----
    > Von: Nicolai Rasmussen [mailto:nicolaiat_private]
    > Gesendet: Mittwoch, 19. September 2001 21:48
    > An: loganalysisat_private
    > Betreff: [logs] Webserver logs to database - Toward data mining
    > 
    > Hi!
    > 
    > We run some websites that generates more than 5 gb logs pr. day on approx.
    > 50 different sites and we would like to put them into a database so we could
    > do some data mining on them.
    > 
    > Does anyone have any idears, input, thoughts or anything on how we should do
    > this ?
    > 
    > We thought about making a optimized table definition and then dump each line
    > into the database. From there we would make some summary reports..
    > 
    > Please write us if you have any idears :-)
    > 
    > We really appreciate it!
    > 
    > Nicolai Rasmussen
    > Software Engineer
    > 
    > ---------------------------------------------------------------------
    > To unsubscribe, e-mail: loganalysis-unsubscribeat_private
    > For additional commands, e-mail: loganalysis-helpat_private
    > 
    > ---------------------------------------------------------------------
    > To unsubscribe, e-mail: loganalysis-unsubscribeat_private
    > For additional commands, e-mail: loganalysis-helpat_private
    
    -- 
    djenkinsat_private                           Universal Savings Bank.
    Security Administrator, Unix Administrator, Alpha Geek
    
    The three most dangerous things are a programmer with a soldering
    iron, a manager who codes, and a user who gets ideas.
    
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: loganalysis-unsubscribeat_private
    For additional commands, e-mail: loganalysis-helpat_private
    



    This archive was generated by hypermail 2b30 : Thu Sep 20 2001 - 09:55:28 PDT