Re: [logs] how much memory can I have?

Wolfgang.Ley@private

Hi,

>Hi
>
>I'm working on a log analysis application.  For now, i'm wondering what 
>sort of log analysis machine people have?  (realistically so I can ask my 
>boss for a new log analysis box :)
>
>I've read Tina's "Building a scalable logging infrastructure" stuff on
>loganlysis.org (thanks Tina!), but there seemed to be little discussion of
>memory.
>
>For those of you who generate some sort of high-level report offline, how
>much Memory/CPU does that machine have?

Well... the question itself already contains part of the answer. Talking 
about sizing issues implies that you've a common understanding of your log
analysis needs (which won't work).

Of course you'll get some replies from people telling you what they are
using for *their* log analysis. Unfortunatly few of them will tell you some
other important paramters (such as log volume, complexitity of rules, derived
actions etc.). Since this is a policy (and not a technical) decision it's
nearly impossible to get "real world numbers".

Let's make a few examples:
1) My own logging infrastructure at home...
   ...a simple DSL connection (2Mbis/sec) and logging parsing is done on a 
   (dedicated) Sun IPX which usualy can cope with the load but has problems
   to deal with floods, (extreme) port scans etc.
2) Some small company i've worked before...
   ...a 34Mbit/sec connection and about 10 machines to watch. In this case
   you've only a few machines but enough traffic to keep an Ultra 2 busy (CPU 
   technically). But: this was logsurfer with about 2000+ lines config and CPU
   was the bottleneck (256MB was still sufficient for a 24 hour window of logs
   even though a new context was started for each new connection).
3) A larger company...
   ...with several connections and a *lot* of machines. In this case the real
   problem starts way before log analysis. In fact it's already a problem to
   configure your routers with so many rules (required to get input for your
   analysis program) and still be performant. And of course your routers are
   only one sensor - so you'll get the same problems on your other sensors.

Ok - these examples are very extreme. The real answer to such a generic question
has to be "...this depends". Most readers of this list would classify their needs
somewhat between example 2) and 3), but exactly this classifaction is the problem.
Once you know what to expect (form your logs) you should be able to determine the
required ressources to deal with automated processing. Unfortunatly people outside
of your busineess/infrastructure can hardly help you with this sizing issues (unlesss
they know a *lot* more about your needs).

Building your own log anaylisis framework shouldn't start at the hardware requirements
(although the upper limit on this is imposed by fincal limitations).

>From my own experience a few questions:
a) how much incoming log traffic (such as audit logs, http logs, syslog logs etc.)
   must be processed (in Kb/sec)?
b) how long does your log framework (software) needs to keep state of certain situations?
c) what kind of analyis (and reactions) is activated (only counters for some statistics
   or some individual response)?

Bye,
  Wolfgang.

_______________________________________________
LogAnalysis mailing list
LogAnalysis@private
http://lists.shmoo.com/mailman/listinfo/loganalysis