Re: [logs] Apache Logs

From: Nate Campi (nateat_private)
Date: Tue Jan 29 2002 - 14:14:43 PST

  • Next message: Marcus J. Ranum: "[logs] Re: syslogd / some analysis"

    On Tue, Jan 29, 2002 at 02:14:47PM -0700, William D. Colburn (aka Schlake) wrote:
    > Since the beginning of the year (not counting today) it has sysloged a
    > total of 10394522 messages from about 100 different machines via UDP
    > (average 4 per second).  Every message is written to an "everything" file,
    > and usually one other file. Some messages are written to two or three
    > different files.  My machine is a linux-2.2.19 with 1 733 Mhz P3.
    These numbers are probably close to what my syslog server sees. The
    problem is that many of us have web logs that go into a data warehouse,
    it's such a huge amount of logs. Just now I did a tail -f on an apache
    log for one of a five server cluster for Wired News, and saw 25 hits in
    one second, on just that one box. Wired News is a low traffic site
    compared to many of the sites that fall under my company's umbrella, 
    Our reporting people would laugh at me if I even mentioned syslog and
    webserver logs in the same sentence. We have looked at mod-log-spread,
    and it looks promising for networked logging with apache (but that's 
    another topic).
    I really doubt that anyone has pulled it off with a conventional syslog
    setup. UDP based solutions could work well if sequencing and acks are
    built into the app itself (like BEEP I guess), but most solutions will
    probably be built on top of TCP. I happen to know that some commercial
    data warehousing vendors monitor this list, too. I wonder what they have
    cooked up. Some have network-striped storage, but I don't know if any
    have network-striping at the collection stage. It would be killer if 
    they did (something like spread maybe).
    Even if we're talking about a syslog daemon that can do TCP, I don't
    know how much throughput you can get from a Solaris door or a Linux
    named pipe while feeding the messages into syslog via logger. I'm
    interested to hear more on that (Marcus? anyone?). Might be best to
    write directly to a TCP socket then to go though the actual local syslog
    daemon, though I'm only a casual programmer.
    For Marcus or others who want to torture test a syslog server, this post
    fo the syslog-ng list looks helpful:
    The person who wrote that did it for syslog-ng vs. native AIX syslogd,
    but it seems useful for general performance testing too.
    Nate Campi | Terra Lycos DNS | WiReD UNIX Operations
    Eat right, exercise regularly, die anyway.
    To unsubscribe, e-mail: loganalysis-unsubscribeat_private
    For additional commands, e-mail: loganalysis-helpat_private

    This archive was generated by hypermail 2b30 : Tue Jan 29 2002 - 14:23:16 PST