Re: [logs] Re: syslogd / some analysis

From: Greg Black (gjbat_private)
Date: Tue Jan 29 2002 - 18:58:51 PST

  • Next message: Rich Salz: "Re: [logs] Re: syslogd / some analysis"

    "Marcus J. Ranum" wrote:
    | My client took 98 seconds to send a million records, which is to say
    | 10,000+ records/second. So we know that syslog (on my box, with
    | my O/S) can't handle 10,000+ records/second.
    [and loses over 50% of the data]
    | Conclusions:
    | syslogd is decidedly not OK for loads in the 10,000 messages/second range on
    |         my machine
    Somehow, I don't think syslogd is suitable for high loads at
    all; certainly UDP-based logging is a waste of time if you want
    to get all the log data.
    Just for comparison, I wrote a very trivial log generator and
    aimed it at Dan Bernstein's multilog via a pipe, a fifo and over
    a TCP link.  It generates 1 million records of 18 bytes; and
    multilog adds a 26-byte timestamp and handles log rotation.  I
    don't show it below, but I checked that all the output logs did
    contain all the data in the correct order.
    First, a baseline with output to a file and no timestamps:
        $ time ./genlog > log.out
        2.910s real     2.593s user     0.241s system
        $ wc log.out 
         1000000 3000000 18000000 log.out
    Now, pipe it into multilog and get timestamps and log rotation:
        $ time ./genlog | multilog t s16000000 ./pipelog
        10.290s real    5.993s user     3.757s system
    Next, create a FIFO, send output there for multilog to read:
        $ mkfifo fifo
        $ multilog t s16000000 ./fifolog < fifo &
        [1] 56562
        $ time ./genlog > fifo
        11.080s real    2.650s user     0.448s system
    Now, generate the output on another machine and read it here
    with netcat and multilog:
        $ nc -l -p 1234 | multilog t s16000000 ./ncatlog
    Other machine:
        $ time ./genlog | nc -w2 bambi 1234
        13.052s real    1.553s user     0.241s system
    So we get 1 million records over the network in 13 seconds on
    some pretty average hardware and no data loss or corruption.
    And, just for fun, genlog2 generates 1 million 50 byte records
    (instead of 18 byte records):
        $ time ./genlog2 | nc -w2 bambi 1234
        22.349s real    1.921s user     0.786s system
    For those unfamiliar with multilog, the `t' argument tells it to
    add timestamps; the `s16000000' says to limit log files to this
    many bytes (resulting in 2 full and one partial log in this
    case); and the `./xxx' is a directory to put the logs in.
    The tests were all run on a Celeron 366 with 384 MB of RAM; the
    TCP test was generated on a Celeron 600 with 512 MB and the data
    travelled over a 100baseTX switch.
    Clearly, 45,000 to 75,000+ records/second with full reliability
    is much better than the syslog figures we have been given.  It's
    time to build better network logging tools.  In the meantime,
    I'll keep using netcat and multilog for network logging and
    fifos or pipes with multilog for local logging.
    To unsubscribe, e-mail: loganalysis-unsubscribeat_private
    For additional commands, e-mail: loganalysis-helpat_private

    This archive was generated by hypermail 2b30 : Tue Jan 29 2002 - 21:47:13 PST