On Fri, 2004-09-17 at 05:59, Mike Blomgren wrote: > Thanks for the tip - I'll have to try that one with perl doing the sort > instead of gnu sort. I have been somewhat reluctant to use perl since I find > it has a severe performance impact in some cases - but that may be related > to my regexp's and not the sorting. For a fact though, I do know that using > associative arrays is a good way to consume memory in a hurry. And thus > causing the os to start swapping memory to disk, which is not very > beneficial for speed, to say the least... If you are short of memory sort may be swapping stuff out to disk and hence your performance problems. It depends on the implementations but some sorts are smart enough to work out how much memory is really available and then do sort & merges with in this. This is much better than sorts that simply assume that virtual memory is endless and cause the OS to thrash madly but is much slower than doing the whole thing in memory. This will not show up as OS level swapping though, just as lots of disk activity during the sort. -- Russell Fulton, Information Security Officer, The University of Auckland New Zealand _______________________________________________ LogAnalysis mailing list LogAnalysis@private http://lists.shmoo.com/mailman/listinfo/loganalysis
This archive was generated by hypermail 2.1.3 : Sun Sep 19 2004 - 23:42:47 PDT