Re: [logs] Secure Central Log Host

tepat_private

>>>>> On Sat, 26 Oct 2002 13:04:31 -0400, "Marcus J. Ranum" <mjrat_private> said:

    MJR> Rainer Gerhards wrote:
    >> Obviously, my central log
    >> server will be a hardened machine placed in a secure part of my network.
    >> So I feel bad about placing the database engine on that same machine -
    >> after all, the DB might be queried by security admins to work with it.
    >> So I would need to open up a number of ports that I do not really like
    >> to open....

    MJR> That's what VPNs are for, really. Why not use a couple of
    MJR> machines running IPSEC and have them centralize things
    MJR> that way?

I suspect that in a larger environment, putting the database server on
the same machine as the final collector may run into performance
issues.  Database insertions are always more expensive than file
writes.  If you are planning to have a machine accept syslog traffic
from hundreds of hosts, that might not be the best place for the
database itself.

But yeah, there's absolutely no excuse for unencrypted traffic these
days, unless you are moving terabytes of "public" data.

    >> Ok, next approach: the collector forwards events to the repository via
    >> syslog protocol.

    MJR> Syslog protocol is unreliable and, unless you use a version
    MJR> that has been made decent, will lose messages if the recipient
    MJR> is down. I don't recommend using syslog unless you can possibly
    MJR> help it.

Old syslog - Bad.  "New syslog", when it arrives -Good.  The new
syslog protocol allows syslog via (BEEP over TCP) which gives you
reliability, (optional) security and (optional) integrity.  At the
expense of added protocol and implementation complexity.

    MJR> How about:
    MJR> collector
    MJR>         - stores syslog events to log file and rotates them/archives
    MJR>         them on schedule
    MJR>         - applies "stop list" (list of records that are known to be
    MJR>         OK to discard to log file and generates a temporary view
    MJR>         file
    MJR> repository machine:
    MJR>         - scp (or FTP's over VPN or whatever) copies temporary view
    MJR>         files from collector and deletes them when successful
    MJR>         repository machine

    MJR> I've left the whole "...and stick it in a database" part out
    MJR> because that's a HARD problem to tackle right and I think that
    MJR> will be the bulk of your pain. Hint: before you think about
    MJR> putting it into a database, ask yourself "what queries will I
    MJR> want to make?" and see if it's even possible to make a data
    MJR> model that will allow them...

Yeah, again, everyone wants to "put it all in a database", but often
without having a reason to do so.  Databases trade performance and
scalability for superior data handling (e.g. searching and
correlation).  What kind of database engine (HW and SW) is it going to
take to handle transaction streams from a few hundred machines that
are each generating 10K logs records/sec (think e-commerce web
servers)?  (Hint: Your Sun salesperson will *love* that new E-server
sale...)

-- 
Tom E. Perrine <tepat_private> | San Diego Supercomputer Center 
http://www.sdsc.edu/~tep/     | 
_______________________________________________
LogAnalysis mailing list
LogAnalysisat_private
http://lists.shmoo.com/mailman/listinfo/loganalysis