On Tue, Jan 29, 2002 at 02:14:47PM -0700, William D. Colburn (aka Schlake) wrote: > > Since the beginning of the year (not counting today) it has sysloged a > total of 10394522 messages from about 100 different machines via UDP > (average 4 per second). Every message is written to an "everything" file, > and usually one other file. Some messages are written to two or three > different files. My machine is a linux-2.2.19 with 1 733 Mhz P3. These numbers are probably close to what my syslog server sees. The problem is that many of us have web logs that go into a data warehouse, it's such a huge amount of logs. Just now I did a tail -f on an apache log for one of a five server cluster for Wired News, and saw 25 hits in one second, on just that one box. Wired News is a low traffic site compared to many of the sites that fall under my company's umbrella, too. Our reporting people would laugh at me if I even mentioned syslog and webserver logs in the same sentence. We have looked at mod-log-spread, and it looks promising for networked logging with apache (but that's another topic). I really doubt that anyone has pulled it off with a conventional syslog setup. UDP based solutions could work well if sequencing and acks are built into the app itself (like BEEP I guess), but most solutions will probably be built on top of TCP. I happen to know that some commercial data warehousing vendors monitor this list, too. I wonder what they have cooked up. Some have network-striped storage, but I don't know if any have network-striping at the collection stage. It would be killer if they did (something like spread maybe). Even if we're talking about a syslog daemon that can do TCP, I don't know how much throughput you can get from a Solaris door or a Linux named pipe while feeding the messages into syslog via logger. I'm interested to hear more on that (Marcus? anyone?). Might be best to write directly to a TCP socket then to go though the actual local syslog daemon, though I'm only a casual programmer. For Marcus or others who want to torture test a syslog server, this post fo the syslog-ng list looks helpful: http://lists.balabit.hu/pipermail/syslog-ng/2001-May/001512.html The person who wrote that did it for syslog-ng vs. native AIX syslogd, but it seems useful for general performance testing too. -- Nate Campi | Terra Lycos DNS | WiReD UNIX Operations Eat right, exercise regularly, die anyway. --------------------------------------------------------------------- To unsubscribe, e-mail: loganalysis-unsubscribeat_private For additional commands, e-mail: loganalysis-helpat_private
This archive was generated by hypermail 2b30 : Tue Jan 29 2002 - 14:23:16 PST