RE: [logs] Faster unix 'sort' replacement?

From: Russell Fulton (r.fulton@private)
Date: Sun Sep 19 2004 - 17:13:59 PDT

On Fri, 2004-09-17 at 05:59, Mike Blomgren wrote:
> Thanks for the tip -  I'll have to try that one with perl doing the sort
> instead of gnu sort. I have been somewhat reluctant to use perl since I find
> it has a severe performance impact in some cases - but that may be related
> to my regexp's and not the sorting. For a fact though, I do know that using
> associative arrays is a good way to consume memory in a hurry. And thus
> causing the os to start swapping memory to disk, which is not very
> beneficial for speed, to say the least... 

If you are short of memory sort may be swapping stuff out to disk and
hence your performance problems.  It depends on the implementations but
some sorts are smart enough to work out how much memory is really
available and then do sort & merges with in this. This is much better
than sorts that simply assume that virtual memory is endless and cause
the OS to thrash madly but is much slower than doing the whole thing in

This will not show up as OS level swapping though, just as lots of disk
activity during the sort.

Russell Fulton, Information Security Officer, The University of Auckland
New Zealand

LogAnalysis mailing list

This archive was generated by hypermail 2.1.3 : Sun Sep 19 2004 - 23:42:47 PDT