We do quite a large set of mail logs mostly but also apache logs and other various application logs. We have a perl script that rotates out the apache logs (without restart) and then drops them onto an NFS share (NetApp). This share is mounted on our log crunching boxes which spit out our daily reports. We've got 100+ machines transferring logs this way. Our main issue currently is that we can't store logs for more than 3 days - as 3 days logs eats up at least 150GB+. For example we generate probably about 2GB/hour of mail logs (qmail is quite verbose, plus our own additions). So you might consider a solution similar to this. If you needed more bandwidth you could even setup a private network just for the NFS traffic. This solution is nice since we can add various log crunching boxes at will and haven't noticed any performance problems with the NetApp. We're currently checking out a product by a startup - Addamark - www.addamark.com - which is a distributed log management system. We're testing a 3 node system and so far have been impressed with the feature set and folks who work there. They are just about to release version 1.0 I believe. In some of our logs we're seeing up to 50-to-1 compression when imported into their system. So we're hopeful we'll be able to start storing a few months of logs... Anyone else here looked at the product? If you're dealing with large sets of logs you should give them a look see. Great bunch of people working there - just down the street from my office in SF. thx, Scott At 10:08 AM -0800 1/29/02, Bill Burge wrote: >Since the call to logger sends it to the local syslog daemon, it is >by default logged to the local host. Your configuration of syslog >will keep it local, send it elsewhere or both. > >I "send it elsewhere". As someone pointed out, a busy web server >will make big logs fast. > >An added advantage of using syslog is that you can rotate logs >without restarting the Apache processes. > >Another advantage is that I have multiple, load balanced, servers >all sending to the same syslog server. This allows me a single log >to process instead of multiple logs. The next issue would be >running stat reports on "the site" from multiple files that would >need to be merged or processed individually and the results combined >(the latter seems to be beyond my company's choice of analysis >products). > -- -------------------------------------------------------------- Topica - http://www.topica.com/ Scott Nelson - drenalinat_private Director of Operations -------------------------------------------------------------- --------------------------------------------------------------------- To unsubscribe, e-mail: loganalysis-unsubscribeat_private For additional commands, e-mail: loganalysis-helpat_private
This archive was generated by hypermail 2b30 : Tue Jan 29 2002 - 21:49:48 PST