[logs] Re: hosts to central logging servers efficiency: syslog or syslog-ng

From: Marcus J. Ranum (mjr@private)
Date: Fri May 26 2006 - 13:24:37 PDT

ScottO wrote:
>I guess some things that I have been thinking about are:  is possibly 
>slightly less data sent over tcp more or less efficient network 
>bandwidth-wise, than sending all the data over udp?  

If that's what's worrying you, you should definitely do UDP at the
edge and do first-order analysis and event compaction at the edge,
compress the logfiles at the edge then just use something like rsync
over SSH to get the data back to your central.

Syslogs compress _really_ well - on the order of 90% or so (your
mileage may vary)

All that said, in general it's not a good idea to proceed with a design
before you've done some back of the envelope measurement and
determined if it's possible or not. You might find out that the data
rates you're dealing with are insignificant, anyhow. Last time I saw
someone go into a syslog design that hadn't thought it through was
buying and fielding big bad-ass machines and my back of the envelope
estimates showed that a 30 gig iPod would have had about the
right amount of processing and storage for his syslog load...


LogAnalysis mailing list

This archive was generated by hypermail 2.1.3 : Sun May 28 2006 - 00:11:29 PDT