[logs] Re: Centralized Logging + large number of active hosts

From: Marcus J. Ranum (mjr@private)
Date: Wed May 10 2006 - 10:17:12 PDT


>The LVS setup seems appealing to me for the scalability potential, but 
>not sure if it is overkill.  What I am currently most concerned with is 
>the amount of traffic over the network.

In the long run it'll be cheaper in terms of hardware AND network
costs if you use multiple systems; that way you're effectively getting
multiprocessor performance (and private busses!)  Send everything
to a local aggregator - use rsync or whatever to push config
files to all the aggregators so that they're all using the same
white/black lists against the traffic, and rsync the greylist
results back to your central. That way you can use the blacklists
to prune traffic at the edge nodes and simply send count summaries
back to the central. If you do the rsync over SSH you get "free"
compression on the stream data. Or rsync compressed files.
If you want to do push-only processing you don't even need to
use rsync you can just do it all with a shell script on the central
that calls scp.

Depending on your data-mix you can cut your traffic by 50% to
90% at the edges and if you need to, you can use the hard
drives on the edge aggregators as rotating storage for the data
you blacklist out.

mjr. 

_______________________________________________
LogAnalysis mailing list
LogAnalysis@private
http://lists.shmoo.com/mailman/listinfo/loganalysis



This archive was generated by hypermail 2.1.3 : Wed May 10 2006 - 16:00:38 PDT