[logs] Re: Looking for toolkits and products that support RFC3195-- COOKED

rgerhards@private

I am currently tackling this dilema within my rsyslog project
(http://www.rsyslog.com/). I agree with Bennett that in almost all cases
it is not worth stopping the application just because the log system
does block for some reason. However, udp-based syslog is also lossy
where it must not necessarily be - with TCP, a simple (tcp stack level)
retry (and a bit luck combined with low message volume at that time) can
buy you much. Also, tcp-based syslog can enable you to use encryption
very easily.

I am currently working with plain tcp syslog (non-standardized), because
that provides me the freedom to try out several approaches. And while it
is non-standard, there is still a sufficiently good understanding in the
syslog industry how tcp syslog shall work, so it is also fairly
interoperable. What I currently do in rsyslog is to create tcp
connections, but only send messages via them if that would not block. If
it blocks, I prefer to keep the app running and loose the message. Even
if I loose the message, a big plus is that the sender *knows* it has
lost it. So it might log a message or initiate some other action.

The current design seems to work well, but I see room for further
improvement. Today, if I have a short-term congestion, I will loose a
message. Even if it would be safe to send it in 5 seconds from now. So
the next design will introduce a send buffer which can hold a
(configurable) number of messages before starting to drop them. If you
think a little bit about this, probably the next approach after that
would be to keep an on-disk queue that would spool messages while the
sender is not capable of actually sending. The key is that tcp lets you
know you can not send the message at this time, thus allowing the
application to act accordingly.

So I think there is actual benefit in using reliable transports for
syslog. To be honest, I have not yet thought about how this can be
integrated in RFC 3195 syslog.  It's simply too early, my efforts are
not yet mature enough to have any solid base of data. But I think it can
be integrated and its worth it.

Just my 2ct's...

Rainer

> -----Original Message-----
> From: 
> loganalysis-bounces+rgerhards=hq.adiscon.com@private 
> [mailto:loganalysis-bounces+rgerhards=hq.adiscon.com@private
> oo.com] On Behalf Of Edward Sargisson
> Sent: Wednesday, August 03, 2005 11:49 PM
> To: Bennett Todd
> Cc: 
> loganalysis-bounces+edward.j.sargisson=nz1.ibm.com@private
> .com; loganalysis@private
> Subject: [logs] Re: Looking for toolkits and products that 
> support RFC3195-- COOKED
> 
> 
> Actually I came across a very similar situation myself 
> recently. I've setup an reliable logging system using message 
> queuing infrastructure and a database. 
> A configuration change stopped the queue being emptied and 
> eventually the source of the logs couldn't log and was blocked. 
> 
> An alternative to the lossy logging you propose, Bennett, 
> would be to back off to a different logging solution. i.e. 
> I'm planning to add an enhancement such that if the log 
> message cannot be submitted to the queue then it is written 
> to a file and the administrator is notified in some way. Then 
> the administrator can fix the problem and then resubmit the 
> messages from the file to the queue. 
> 
> Note: I speak for myself, not my employer. 
> 
> Cheers, 
> Edward
> 
> 
> Edward Sargisson BSc, BCom
> Consultant
> IBM Business Consulting Services
> Wellington, New Zealand
> DDI: + 64-4-462-3586, Mob: + 64-21-254-8927
> P O Box 38 993, Wellington, NEW ZEALAND
> edward.j.sargisson@private 	 
> <http://www.ibm.com/ibm/values/> 
> 
> 
> 
> 
> Bennett Todd <bet@private> 
> Sent by: 
> loganalysis-bounces+edward.j.sargisson=nz1.ibm.com@private 
> 
> 04/08/2005 07:24 To
> "Moehrke, John (GE Healthcare)" <John.Moehrke@private> 
> cc
> loganalysis@private 
> Subject
> [logs] Re: Looking for toolkits and products that support 
> RFC3195 --        COOKED
> 
> 	
> 
> 
> 
> 
> It's funny; I used to think I wanted my syslog infrastructure to be
> reliable and to not lose messages. Now that I've gotten the ability
> to do that, I've discovered that I usually would rather have a
> logging outage (or slowdown) lose messages, rather than hanging (or
> slowing) the apps that are logging to it.
> 
> Reliable logging says, the completeness and correctness of your log
> capture is more important than the reliability and availability of
> your service. While there may be cases where that's true, I find
> them to be the exception more than the norm.
> 
> So I use unix-dgram and udp, in preference to unix-stream and tcp.
> 
> What's really appropriate, I think, is to leave syslog alone,
> unreliable but loosely coupled between logging client app and
> logging server, and introduce a new, distinctly separate logging
> service that offers reliable logging --- and write client apps to
> use it only when they've got something to say whose logging is more
> important than the app continuing to run. Critical audit events.
> 
> -Bennett
> _______________________________________________
> LogAnalysis mailing list
> LogAnalysis@private
> http://lists.shmoo.com/mailman/listinfo/loganalysis
> 
> 
> 
_______________________________________________
LogAnalysis mailing list
LogAnalysis@private
http://lists.shmoo.com/mailman/listinfo/loganalysis