>>>>> On Fri, 6 Dec 2002 02:43:18 +1100 (Australia/ACT), Darren Reed <avalonat_private> said: DR> In some mail from Tom Perrine, sie said: DR> I suppose you mean "open source" solutions/products, here... DR> I would hope there is at least one commercial product supporting DR> it given the vendor interest in that list! (No, I haven't checked DR> myself...) Yup, should have been more clear. Open source or at least not fully proprietary. DR> [...] DR> Hmmm, BEEP sounded like a bad idea when I heard of it, now I'm sure it DR> is a bad idea ;-) It was a bad idea when you heard of it, and its still a bad idea, AT LEAST FOR SYSLOG. I can easily see places where multiple virtual channels over a single TCP connection could be useful. The "stackable" session profiles is kinda a nice idea. If I was going to do a fourth generation TELNET, BEEP would be a good answer, if there were some libraries... But the implementations are, well, "interesting". Why one BEEP lib group decided that their library HAD to impose its own thread model on the application, is just not understandable to me. I want a BEEP lib, not a BEEP+thread-police library. DR> As a matter of fact, yes :-) (I'm not sure if you're asking that DR> in jest wnad you did see my name in syslog-ng and/or elsewhere or DR> you're thinking that I haven't...) Mostly in jest. But, more, I was hoping that you had RFC 3195 code that we could test with, or perhaps had written some super-secret-but-it-works BEEP library. DR> I wrote nsyslogd which was the "inspiration" for syslog-ng. My goal DR> wasn't pattern matching, so much as flexibility in mux'ing data from DR> inputs to outputs and a secure implementation that performed well. Aha! Yes, I suspect that our ideas about input channels, switch logic and output channels are *very* similar. Our config files are probably very reminiscent as well. And your implmentation predates syslog-reliable, right? We also looked at syslog-signed and ... DR> I don't give a rats a*** about SQL and database insertion. The DR> other goal was to make the data become tamper evident (tamper proof DR> requires immeadiate protected storage and that's not something that DR> is solved with a few lines of code.) The problem with doing that is DR> to make it all work properly imposed a bunch of operational problems DR> that require compromises for ease of use. That problem kind of got DR> the better of me and I never really got back to doing something about DR> it and producing good tools to work with its encrypted/mac'd logs. Way back when, we (Andrew Gross, mostly) played with crypto hash chains, with per-host keys for everything from the original client, through all the relays into the final storage system. Yup, it was ugly, and the crypto-signing overhead was "amausing". But you had a list of timestamps and hash chains all the way back to the source, IIRC. DR> If truth be known, IMHO you would have to try hard to write syslogd DR> in a manner that is as simple as it is today that performs worse. Oh, you noticed :-) However, as bad as it is I suspect that many sites have "solved" the performance problem. Either they just don't bother turning logging on, or log selectively, or have spent some time working on syslog relay trees, or other methods. However, I must point out, that on more modern hardware (Sun Ultra-1, or modern PC) we just haven't seen classic syslog performance as a problem, because we are only logging (UNIX) system log data through it. To avoid some of those problems ourselves, we shoot the web logs through other means, and the Windows world does its own thing. Until we get the new syslog deployed everywhere, that is. We've only got about 300+ machines (not counting supercomputers and clusters) that do "*.debug -> loghost". Add in the clusters and supers and its probably closer to 500 or so. I consider us a "small" site in terms of host counts. These days, it still seems that when we lose syslog data, its always due to router congestion, and not to problems in the syslog daemon code itself. But as I said, we are already limiting our log generation rate. DR> Can you even do that, theoretically, with modern hardware given the DR> constraints placed on you by hardware ? PCI is typically 32bits at DR> 33MHz, which is barely 1Gbit and that's if you're going flat out. DR> Then you've got to move the data around buses inside the computer, DR> spend cycles doing computations on it, etc. PCIx running at 166 Mhz does hit about 1 G bytes/sec. We have 10 Gig E hardware in house, as well as the 10 Gig E testing lab. People are seeing "useful percentages of theoretical" (that's all they'll say :-( ) They won't let us play with it yet, though :-( There's also IBM Infiniband which can run at 2.5 G bits. If you don't mind AIX on Power3 or Power4. DR> I can see why you're "not there" yet, quite easily. I did some DR> testing years ago on ultra5's on trying to switch ethernet packets DR> very quickly and was optimizing branches out for gains of around DR> 250kb/s, per branch. Hmmm, I'll keep that in mind. Good to know. But we ditched all the Ultra-5s and went back to Ultra 1 and 2s. The 1s and 2s seemed to outperform the U5 by *quite* a bit. DR> btw, thanks for the credentials brief - they impressed me :) Oops! Sorry 'bout that. Late night, final exams, crotchety-ness. DR> Darren --tep _______________________________________________ LogAnalysis mailing list LogAnalysisat_private http://lists.shmoo.com/mailman/listinfo/loganalysis
This archive was generated by hypermail 2b30 : Thu Dec 05 2002 - 12:38:20 PST