I wrote: | "Marcus J. Ranum" wrote: | | | My client took 98 seconds to send a million records, which is to say | | 10,000+ records/second. So we know that syslog (on my box, with | | my O/S) can't handle 10,000+ records/second. | | [and loses over 50% of the data] | | | Conclusions: | | syslogd is decidedly not OK for loads in the 10,000 messages/second range on | | my machine | | Somehow, I don't think syslogd is suitable for high loads at | all; certainly UDP-based logging is a waste of time if you want | to get all the log data. I then went on to show some times for some simple examples of generating log data and sending it to multilog via various paths. Since I had a spare hour this afternoon, I thought I'd do a bit more research. I wrote a trivial daemon that listens to some TCP port for incoming connections from multiple clients (up to 1,024 clients in this case, but who's counting?). To make it at least look like a logging tool, it takes some care to read complete lines from the clients and outputs them with one call to write(2), to produce useful logs without dropped data or partial lines interleaved. I sent its output to multilog as before and changed my genlog client to put host and pid info in its output so that I could dismantle the resulting logs and make sure they had no errors. My test run had three simultaneous clients running, one on the log host, and one each on two hosts on the LAN (writing via a 100baseTX switch). Each client generated 1 million lines of data (one printf(3) per line) and used netcat to send it to the log host. The final logs on disk contained 201 million bytes of (completely correct) data. The run took 52.43 seconds in total (with the three clients running simultaneously). That's over 57,000 log records per second being handled correctly on the log host which was the same machine as before: a Celeron-366 with 384 MB of RAM and a really cheap IDE disk. The point of this? Well, it's clear that it's quite possible to get high levels of logging, even from multiple clients to the one log daemon, provided we're not using syslogd and UDP. My logger would need a bit of work before it could go into service in a production environment, but I think it serves to show that solving the logging problem is not really all that hard. What would be needed to bring this toy program up to a level where it could replace syslogd? Since I'd be happy to use multilog to do the actual log writing and rotation, as it already does that really well, the daemon would only need a bit of code for handling potential errors more robustly and it would pay to profile the code that collects the complete lines, as that was a really quick hack. The other part of the job would be to re-implement syslog(3) and friends to do talk TCP with this daemon -- I'd say less than a day's work for a competent programmer. I won't include the daemon's code here -- it's only 135 lines of C, but it's probably of limited interest to the majority of list subscribers. If anybody wants to look at it, just ask and I'll put it up some place handy. Greg --------------------------------------------------------------------- To unsubscribe, e-mail: loganalysis-unsubscribeat_private For additional commands, e-mail: loganalysis-helpat_private
This archive was generated by hypermail 2b30 : Fri Feb 01 2002 - 09:56:29 PST