Re: [logs] High Network Load

From: Devin Kowatch (devink@private)
Date: Mon Sep 22 2003 - 10:46:01 PDT

  • Next message: Florin Andrei: "Re: [logs] High Network Load"

    On Fri, Sep 19, 2003 at 04:51:57PM -0400, Paul Robertson wrote:
    > On Fri, 19 Sep 2003, Marcus J. Ranum wrote:
    > > On UNIX boxen most client sides write to /dev/log, which behaves
    > > differently (since it's a pseudo-device) than an actual UDP send.
    > That only helps for local logging, which wasn't the proposed scenerio...
    > Also, there were reports earlier this year of loss if /dev/log was opened 
    > as a UNIX_STREAM verus a UNIX_DGRAM socket under glibc with syslog-ng.
    > I also know there were issues with glibc blocking on full /dev/log buffers 
    > at one point under Linux, not sure if it's a libc issue, or a linux 
    > /dev/log issue.
    I've seen this happen when /dev/log is opened as a UNIX_STREAM.  The
    system isn't quite unusable ... but the only way I've found to recover
    is to have a root shell open to kill syslogd.
    IIRC, linux's stock syslogd does not open /dev/log as a STREAM.  
    > In either case, we get pretty quickly to the "enough going on that 
    > centralizing logging on this is a bad idea."
    > > I've never measured the kernel dropping log messages between
    > > the application, /dev/log, and the kernel. I have, however, measured
    > > some horrible log traffic loss...  (see below)
    > Do you mean "never tried to," or "tried and couldn't?"
    > > 
    > > >With the right volume, the OS won't even get the message, it'll be dropped 
    > > >at the router if its buffers get full...
    > > 
    > > With the right volume, it'll never leave the machine. I did some
    > > testing (and posted it to loganalysis ages ago) and discovered
    > > that the UDP output queue is maintained per network interface
    > > and is relatively "shallow" -- lots of outgoing UDP packets results
    > > in them simply being tossed before they leave the box. I tested
    > > this by running tcpdump on one machine while I syslogged in a
    > > tight loop on the other. I counted something like 10,000 packets
    > > sent as a result of 1,000,000 syslog() calls. Syslog over TCP
    > > will, of course, not exhibit this issue. It'll suck in other ways.
    > > Syslog is very, very badly designed.
    > Did you do any tuning to see if upping udp.sendspace on the sender helped, 
    > and if so, how much?  (assuming, of course that raw.rcvspace and/or 
    > udp.rcvspace wasn't an issue on the measuring box...)
    > Paul
    > -----------------------------------------------------------------------------
    > Paul D. Robertson      "My statements in this message are personal opinions
    > proberts@private      which may have no basis whatsoever in fact."
    > probertson@private Director of Risk Assessment TruSecure Corporation
    > _______________________________________________
    > LogAnalysis mailing list
    > LogAnalysis@private
    Devin Kowatch
    LogAnalysis mailing list

    This archive was generated by hypermail 2b30 : Mon Sep 22 2003 - 16:37:07 PDT