Rsync (http://rsync.samba.org/ is your friend. It syncs up files using block level deltas so its VERY efficient on low bandwidth networks. Also supports bandwidth limiter option to prevent filling up the pipe. Supports ssh and compresses the data stream in transit. If you don't want to do filtering at the satellite sites, you could just rsync the log files to your central server and process the file by "tail -f" You can use "gzip" on the central server to append the incoming logs files to long term logs files, ie "cat realtimelog | gzip >> storagelog.gz" (YES, you can incrementally append gzip data this way, it work!). The only problem is "gunzip -l" will lie about the compression ratios. One of the things we realized early on is having a well defined directory hier on the central server help in the long run. We went with the scheme of /SOMEROOT/yyyy/mm/server-x/service/fnamex_yyyy-mm-dd.txt . This allowed us to blow away and restore any year's data from tape. It also help is partitioning your drives for growth. Ashish Fidelity Investments Corporate Security Investigations > -----Original Message----- > From: Andreas Siegert [SMTP:afxat_private] > Sent: Wednesday, September 05, 2001 6:03 AM > To: loganalysisat_private > Subject: [loganalysis] Larg scale log architecture > > Hi, > > I am looking for any hints on how others have solved large scale log > architecture problems. > > Central sites with several hundred GB of log data per day plus remote > sistes > on slow or already full links. Logging of accepted and denied traffic from > firewalls (raptor, fw-1, ...) plus syslog. > > Goal is to have real time alerts and long term analysis of all the data. > > Any experience with SLR from NFR or others? > > --------------------------------------------------------------------- To unsubscribe, e-mail: loganalysis-unsubscribeat_private For additional commands, e-mail: loganalysis-helpat_private
This archive was generated by hypermail 2b30 : Thu Sep 06 2001 - 15:19:31 PDT