Is anyone aware of a logging tool that will allow the following. I want to centrally manage log files for a number of hosts and display statistics in real time for access. I would like to be able to generate stats also on some of the frequently accessed sites etc and customise reports on this basis. Any help or suggestions would be greatly accepted. regards Niall -----Original Message----- From: loganalysis-requestat_private [mailto:loganalysis-requestat_private] Sent: Thursday, December 05, 2002 12:00 PM To: loganalysisat_private Subject: LogAnalysis digest, Vol 1 #61 - 9 msgs Send LogAnalysis mailing list submissions to loganalysisat_private To subscribe or unsubscribe via the World Wide Web, visit http://lists.shmoo.com/mailman/listinfo/loganalysis or, via email, send a message with subject or body 'help' to loganalysis-requestat_private You can reach the person managing the list at loganalysis-adminat_private When replying, please edit your Subject line so it is more specific than "Re: Contents of LogAnalysis digest..." Today's Topics: 1. Re: Secure Central Log Host (Jason Royes) 2. Re: Secure Central Log Host (Matt Bing) 3. Re: Secure Central Log Host (Tevfik Karagulle) 4. syslog data from nessus scan (Tina Bird) 5. Re: Secure Central Log Host (Tom Perrine) 6. reinventing syslog [was: Secure Central Log Host] (Florin Andrei) 7. SDSC Secure Syslog (Tom Perrine) 8. Re: SDSC Secure Syslog (Darren Reed) 9. Re: reinventing syslog [was: Secure Central Log Host] (tevfik) --__--__-- Message: 1 Date: Wed, 4 Dec 2002 06:39:27 -0500 (EST) Subject: Re: [logs] Secure Central Log Host From: "Jason Royes" <jroyesat_private> To: <loganalysisat_private> > On Tue, 2002-12-03 at 09:37, Marcus J. Ranum wrote: >> Jason Royes wrote: >> >Databases (w/ good schema) excel when complex >> >analysis is required. >> >> Databases also require index inserts, support for transaction >> rollback, and all kinda crazy stuff that makes them completely >> unsuitable as logging systems. We (the collective unconscious "we") >> keep using them, though, because they're available and can be >> made to suit the purpose by throwing a bunch of hardware at the >> problem - which is cheaper, really, than understanding the problem I don't think the problem is understanding as much as it is integration and administration. Using a typical RDBMS has it's advantages: - It is easier to administer a single database system. - No need to integrate ranumdb with an SQL backend - Less learning required for log monkeys and developers - Generate reports with SQL aware tools - Large number of SQL ready packages exist - Moore's law consoles the bloat For ex., what happens when you want your inventory database to be factored into threat analysis? > > There's also the convenience of having programming interfaces to them > in your other tools of choice, like PHP, Perl, etc. > (also read below) > >> lot of simplifying assumptions you can make about logs: >> - they are inserted in event-sequence >> - they are approximately clustered by time >> - you seldom (if ever) will need to seek back 20 minutes >> and delete a single log record >> - the fields you'll want to search on are either bounded >> fairly tightly (priority, source, time) or are >> free-form (regexp or string fragment) - so you'll >> either want a very compact primary index for >> the bounded values and a patricia tree or inverted >> index for the strings > > I like your analysis, and in fact it's pretty close to my own > conclusions a while ago. But then, when deciding which tools i was > going to use to implement a logging DB, guess what? It was the > convenience of having an SQL programming interface in PHP that won the > battle. :-) > > Now, sure, if you can afford the resources, coming up with your own > thing is the best way. I guess that's not far from what Addamark did > (although on a completely different scale). > >> mjr. ("once a database guy - always a database guy.") > > Yeah. :-) > > -- > Florin Andrei > > It's ok to use the names of your pets or children as passwords > as long as they contain several non-alphanumeric characters. > > _______________________________________________ > LogAnalysis mailing list > LogAnalysisat_private > http://lists.shmoo.com/mailman/listinfo/loganalysis -- Jason Royes Data Access Experts, LLC --__--__-- Message: 2 Date: Wed, 4 Dec 2002 12:59:53 -0500 From: Matt Bing <mbingat_private> To: loganalysisat_private Subject: Re: [logs] Secure Central Log Host mjr said: > lot of simplifying assumptions you can make about logs: > - they are inserted in event-sequence I think this is an important point to stress. syslogd records entries with the timestamp from the originator. In a networked environment this can be misleading when correlating logs based on that timestamp due to clock ambiguities or drifts. Log correlation should take into account the strict sequencing [1] that an "append-to-disk" centralized log system gives you. [1] As "strict" as you can get when taking into account packet loss, network congestion, kernel buffering and other extenuating factors. -- Matt Bing NFR Security Rapid Response Team --__--__-- Message: 3 From: "Tevfik Karagulle" <tevfikat_private> To: <loganalysisat_private> Subject: Re: [logs] Secure Central Log Host Date: Wed, 4 Dec 2002 23:00:40 +0100 Hi, Wouldn't it be enough to configure your central log host as an NTP server for machines generating syslogs or other logs ? As long as all those machines are placed in a high capacity network, drift problems would be minimal and under control. If this is really a big issue for your network and environment, you might also consider out-of-band NTP sync. In that way, It is also possible timestamping all syslog entries (or other logs!) centrally by taking time related issues into consideration. Best regards, Tevfik Karagulle ITEFIX Consulting mail tevfikat_private, web http://www.itefix.no Logrep - a logfile extraction and reporting system - http://logrep.sourceforge.net ----- Original Message ----- From: "Matt Bing" <mbingat_private> To: <loganalysisat_private> Sent: Wednesday, December 04, 2002 6:59 PM Subject: Re: [logs] Secure Central Log Host > mjr said: > > lot of simplifying assumptions you can make about logs: > > - they are inserted in event-sequence > > I think this is an important point to stress. syslogd records entries with > the timestamp from the originator. In a networked environment this can > be misleading when correlating logs based on that timestamp due to clock > ambiguities or drifts. Log correlation should take into account the > strict sequencing [1] that an "append-to-disk" centralized log system gives > you. > > [1] As "strict" as you can get when taking into account packet loss, > network congestion, kernel buffering and other extenuating factors. > > -- > Matt Bing > NFR Security > Rapid Response Team > _______________________________________________ > LogAnalysis mailing list > LogAnalysisat_private > http://lists.shmoo.com/mailman/listinfo/loganalysis > --__--__-- Message: 4 Date: Thu, 5 Dec 2002 01:30:06 +0000 (GMT) From: Tina Bird <tbird@precision-guesswork.com> To: "loganalysisat_private" <loganalysisat_private> Subject: [logs] syslog data from nessus scan I'm getting ready to put an OpenBSD box on the net. I've just installed OBSD 3.2, enabled Apache, and done very little else to it. We ran a Nessus scan and were somewhat surprised to discover that it's running ident. I was even more surprised -- and >>completely<< delighted -- so see that inetd and ident recorded a whole lot of unauthorized connection attempt messages in syslog when it got hit with the scan: Dec 3 18:38:16 bettiepage inetd[20239]: accept (for time): Software caused connection abort Dec 3 18:38:16 bettiepage inetd[20239]: accept (for ftp): Software caused connection abort Dec 3 18:38:16 bettiepage inetd[20239]: accept (for ident): Software caused connection abort Dec 3 18:39:12 bettiepage identd[5128]: scanf: invalid-port(s): 0 , 0 from netops-26.Stanford.EDU Dec 3 18:39:12 bettiepage identd[29996]: scanf: invalid-port(s): 0 , 0 from netops-26.Stanford.EDU Dec 3 18:39:12 bettiepage identd[13377]: scanf: invalid-port(s): 0 , 0 from netops-26.Stanford.EDU Dec 3 18:39:12 bettiepage identd[1804]: scanf: invalid-port(s): 0 , 0 from netops-26.Stanford.EDU Dec 3 18:39:21 bettiepage identd[15689]: scanf: invalid-port(s): 0 , 0 from netops-26.Stanford.EDU Dec 3 18:39:23 bettiepage identd[31692]: read from netops-26.Stanford.EDU: EOF Dec 3 18:39:41 bettiepage identd[7624]: scanf: invalid-port(s): 0 , 0 from netops-26.Stanford.EDU Dec 3 18:39:41 bettiepage identd[4088]: scanf: invalid-port(s): 0 , 0 from netops-26.Stanford.EDU Dec 3 18:39:41 bettiepage identd[18061]: scanf: invalid-port(s): 0 , 0 from netops-26.Stanford.EDU Dec 4 17:12:53 bettiepage inetd[20239]: accept (for time): Software caused connection abort Dec 4 17:12:53 bettiepage inetd[20239]: accept (for ftp): Software caused connection abort Dec 4 17:12:53 bettiepage inetd[20239]: accept (for ident): Software caused connection abort Dec 4 17:13:55 bettiepage identd[4717]: scanf: invalid-port(s): 0 , 0 from netops-26.Stanford.EDU Dec 4 17:13:55 bettiepage identd[15209]: scanf: invalid-port(s): 0 , 0 from netops-26.Stanford.EDU Dec 4 17:14:00 bettiepage identd[16074]: read from netops-26.Stanford.EDU: EOF Dec 4 17:14:00 bettiepage identd[13534]: scanf: invalid-port(s): 0 , 0 from netops-26.Stanford.EDU Dec 4 17:14:10 bettiepage identd[25838]: scanf: invalid-port(s): 0 , 0 from netops-26.Stanford.EDU Dec 4 17:14:11 bettiepage identd[9691]: read from netops-26.Stanford.EDU: EOF Dec 4 17:14:26 bettiepage identd[29770]: scanf: invalid-port(s): 0 , 0 from netops-26.Stanford.EDU Dec 4 17:14:26 bettiepage identd[18091]: scanf: invalid-port(s): 0 , 0 from netops-26.Stanford.EDU Dec 4 17:14:26 bettiepage identd[3823]: scanf: invalid-port(s): 0 , 0 from netops-26.Stanford.EDU Okay, so I'm easily amused. But I'm pretty used to >not< seeing log data that I expect to see. Having data show up that I don't expect is a lovely change of pace! cheers -- tbird --__--__-- Message: 5 Date: Wed, 4 Dec 2002 17:48:54 -0800 From: Tom Perrine <tepat_private> To: tevfikat_private Cc: loganalysisat_private Subject: Re: [logs] Secure Central Log Host >>>>> On Wed, 4 Dec 2002 23:00:40 +0100, "Tevfik Karagulle" <tevfikat_private> said: TK> Hi, TK> Wouldn't it be enough to configure your central log host as an NTP server TK> for machines generating syslogs or other logs ? Well, NTP is probably the right answer, but I don't think that the log server is the right place to put a low-stratum NTP server. Time service needs to be viewed as just one more critical service that is relied upon for security and auditing. We rolled out NTP way back when for Kerberos and NFS services. Having time-synced logs on the various independent boxes while we built the central log infrastructure was really helpful. NTP is the right answer, and your log services should use it, but you may want one or more dedicated NTP servers, depending on whether you use your own real clock or someone else's. --tep -- Tom E. Perrine <tepat_private> | San Diego Supercomputer Center http://www.sdsc.edu/~tep/ | --__--__-- Message: 6 From: Florin Andrei <florinat_private> To: loganalysisat_private Date: 04 Dec 2002 18:39:14 -0800 Subject: [logs] reinventing syslog [was: Secure Central Log Host] On Wed, 2002-12-04 at 14:00, Tevfik Karagulle wrote: > > Wouldn't it be enough to configure your central log host as an NTP server > for machines generating syslogs or other logs ? Sometimes you cannot do that; think, for example, of the cases when you need to "poke yet another hole" through some firewall to allow a host to send syslog datagrams to the logging server. In that case, poking two holes (syslog and NTP) instead of one (syslog) might make a big difference. (If you don't believe one more protocol poked through the firewall can be the cause for a major fuss, go ask the closest security analyst :-P) The true solution is to modify syslogd. This software is so old and outdated, i'm perpetually amazed so very few people notice. Changing the code to use local timestamps instead of the ones provided by the datagrams should be no big deal. Some external configuration flag could change the behaviour between the default (use the datagrams' timestamps, or use local time). I don't care what the RFC says (if there really is any mention in it to which time should be used); sometimes you just need to do it in a different way. And logging, in the Unix world, is broken anyway. Just look at how every daemon has to reinvent the wheel and come up with its own logging thing. In an ideal world, there would be a unique syslog daemon, and everything would be "aware" of it: the kernel, every daemon, different apps that need to use syslog, etc. That means a set of standardized syslog message formats, generic enough to be used by all aforementioned entities, flexible enough to be easily extented, clever enough to satisfy everyone's needs without compromises. That means a rewritten syslog network protocol, to provide support for these message formats, and some other neat features (some of them already mentioned recently on this mailing list: encryption, reliability, etc.). That means a completely new architecture for the syslog daemon, to support all these things, but also to scale gracefully in demanding enterprise environments, without forgetting the smallest embedded systems. <plug> A proposal for such a syslog daemon architecture can be found here: http://florin.myip.org/syslog/ I wrote it for the Msyslog project (check SourceForge). This document contains quite a few things that i learned while i designed and implemented a logging database in a big enterprise environment. I'm not sure if the msyslog team (Alejo Sanchez, Fredrick Paul Eisele) will use it in future versions (it's up to them, really), but that would be a good thing, i suppose. The document, anyway, is open to anyone who finds these ideas interesting. </plug> -- Florin Andrei It's ok to use the names of your pets or children as passwords as long as they contain several non-alphanumeric characters. --__--__-- Message: 7 Date: Wed, 4 Dec 2002 22:31:44 -0800 From: Tom Perrine <tepat_private> To: Log Analysis Mailing List <loganalysisat_private> Subject: [logs] SDSC Secure Syslog -----BEGIN PGP SIGNED MESSAGE----- Announcing SDSC Secure Syslog (Release Candidate) The Security Technologies group at the San Diego Supercomputer Center (SDSC) is pleased to announce the early availability of "SDSC secure syslog", a replacement for the standard Linux/UNIX syslog daemon that adds security and performance features, while retaining backwards compatibility. We believe it is the first syslog implementation to target "syslog-reliable" (RFC 3195) functionality and it is the first syslog targeted at very high performance and forensically-sound auditing. The project home page is at: http://security.sdsc.edu/software/sdsc-syslog Authors of other RFC3195-compliant software, please contact us at sdscsyslogat_private, so we can explore inter-operability testing with you. This is a release candidate for version 1.0. SDSC syslog is intended as a high-performance and high-security replacement for "syslog classic". It is intended for sites with high volumes of syslog transactions that also want security and integrity features and compatibility with new audit standards. SDSC syslog is a complete new design incorporating new features and capabilities, including: *modular *input modules for socket, UDP network connections, TCP/BEEP, etc. * a message switch to perform log message routing * multiple output modules for UDP, TCP/BEEP, "syslog classic" files, structured files *multi-processing - handles more input syslog steams, provides better scalability *support for draft standards such as "syslog-reliable" (RFC 3195, syslog messages over BEEP) This Release Candidate does not yet have complete support for RFC 3195, but is fully backwards compatible with "syslog classic" using 514/UDP. Note that this software ***currently*** carries the standard University of California copyright statement, permitting free use for educational and non-profit activities. We are exploring a transition to an Open Source license, such as the "BSD license". This requires completion of the pending approval of the University of California. - -- Tom E. Perrine <tepat_private> | San Diego Supercomputer Center http://www.sdsc.edu/~tep/ | -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: Processed by Mailcrypt 3.5.7 <http://mailcrypt.sourceforge.net/> iQCVAwUBPe7yxhTSxpWcaAFRAQF3QwP/VHEvjQLhyNCb3mWmTHsu/OW7AMmltDmt t8gqRSMokjOMJti03/4/1Oji3u6YBiyZ+oKP/YsaqGjeFKjhV1qZcqagjZi48pvD OthgG+J9BHAaexILVNb0+rLQeWseeJDJyyYHIui8po5Fnexfh8jp4Am8HG2lbVIv IoekSFrExIc= =lO5H -----END PGP SIGNATURE----- --__--__-- Message: 8 From: Darren Reed <avalonat_private> Subject: Re: [logs] SDSC Secure Syslog To: tepat_private (Tom Perrine) Date: Thu, 5 Dec 2002 17:59:52 +1100 (Australia/ACT) Cc: loganalysisat_private (Log Analysis Mailing List) In some mail from Tom Perrine, sie said: > > The Security Technologies group at the San Diego Supercomputer Center > (SDSC) is pleased to announce the early availability of "SDSC secure > syslog", a replacement for the standard Linux/UNIX syslog daemon that > adds security and performance features, while retaining backwards > compatibility. > > We believe it is the first syslog implementation to target > "syslog-reliable" (RFC 3195) functionality and it is the first syslog > targeted at very high performance and forensically-sound auditing. I have yet to see this "product" mentioned on the syslog-secure IETF list that has been the home for online discussion about syslog-reliable, so I would advise you to mention it there with perhaps a little more modesty before making bold claims like you have above, because I for one would challenge your "first" claims about "very high performance and forensically-sound auditing" unless they're markedly different to current/traditional methods for providing this. Darren --__--__-- Message: 9 From: "tevfik" <tevfikat_private> To: loganalysisat_private Subject: Re: [logs] reinventing syslog [was: Secure Central Log Host] Date: Thu, 5 Dec 2002 08:33:51 +0600 > On Wed, 2002-12-04 at 14:00, Tevfik Karagulle wrote: > > > > Wouldn't it be enough to configure your central log host as an NTP server > > for machines generating syslogs or other logs ? > > Sometimes you cannot do that; think, for example, of the cases when you > need to "poke yet another hole" through some firewall to allow a > host to send syslog datagrams to the logging server. In that case, > poking two holes (syslog and NTP) instead of one (syslog) might make > a big difference. > (If you don't believe one more protocol poked through the firewall > can be the cause for a major fuss, go ask the closest security > analyst :-P) > I think that the idea about syncing time is a good one. If you need to poke one well-defined port in your firewall to get that advantage, just do it! I haven't seen too much security trouble around those NTP daemons ( I remember an old one in xntpd !!). As you suggest, the real problem is syslog architecture and lacking security features. Actually, I would be more cautious to open my defenses for syslog :- )). It is natural that intermediary solutions pops up !!. Hopefully we can see some movements there. When it comes to dedicated NTP servers, that should be feasible for large scale and high performance solutions. However, I don't see any problem integrating this feature on a central log server when you have a small/midsized environment. It all depends on your priorities. Regards Tevfik Karagulle ITEFIX Consulting --__--__-- _______________________________________________ LogAnalysis mailing list LogAnalysisat_private http://lists.shmoo.com/mailman/listinfo/loganalysis End of LogAnalysis Digest _______________________________________________ LogAnalysis mailing list LogAnalysisat_private http://lists.shmoo.com/mailman/listinfo/loganalysis
This archive was generated by hypermail 2b30 : Thu Dec 05 2002 - 08:15:07 PST