Re: [logs] High Network Load

From: Paul Robertson (probertsat_private)
Date: Fri Sep 19 2003 - 13:51:57 PDT

  • Next message: Brown, James (Jim): "RE: [logs] High Network Load"

    On Fri, 19 Sep 2003, Marcus J. Ranum wrote:
    > On UNIX boxen most client sides write to /dev/log, which behaves
    > differently (since it's a pseudo-device) than an actual UDP send.
    That only helps for local logging, which wasn't the proposed scenerio...
    Also, there were reports earlier this year of loss if /dev/log was opened 
    as a UNIX_STREAM verus a UNIX_DGRAM socket under glibc with syslog-ng.
    I also know there were issues with glibc blocking on full /dev/log buffers 
    at one point under Linux, not sure if it's a libc issue, or a linux 
    /dev/log issue.
    In either case, we get pretty quickly to the "enough going on that 
    centralizing logging on this is a bad idea."
    > I've never measured the kernel dropping log messages between
    > the application, /dev/log, and the kernel. I have, however, measured
    > some horrible log traffic loss...  (see below)
    Do you mean "never tried to," or "tried and couldn't?"
    > >With the right volume, the OS won't even get the message, it'll be dropped 
    > >at the router if its buffers get full...
    > With the right volume, it'll never leave the machine. I did some
    > testing (and posted it to loganalysis ages ago) and discovered
    > that the UDP output queue is maintained per network interface
    > and is relatively "shallow" -- lots of outgoing UDP packets results
    > in them simply being tossed before they leave the box. I tested
    > this by running tcpdump on one machine while I syslogged in a
    > tight loop on the other. I counted something like 10,000 packets
    > sent as a result of 1,000,000 syslog() calls. Syslog over TCP
    > will, of course, not exhibit this issue. It'll suck in other ways.
    > Syslog is very, very badly designed.
    Did you do any tuning to see if upping udp.sendspace on the sender helped, 
    and if so, how much?  (assuming, of course that raw.rcvspace and/or 
    udp.rcvspace wasn't an issue on the measuring box...)
    Paul D. Robertson      "My statements in this message are personal opinions
    probertsat_private      which may have no basis whatsoever in fact."
    probertsonat_private Director of Risk Assessment TruSecure Corporation
    LogAnalysis mailing list

    This archive was generated by hypermail 2b30 : Fri Sep 19 2003 - 17:46:33 PDT