Re: [logs] most popular reports...?

spb@private

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Marcus J. Ranum writes:

>Jose Nazario wrote:
>>is no one doing any trend analysis (ie fasting rising ports being hit)?
>>all i see listed so far are static "top N" reports.

>Most of the folks who are looking at that problem are intensely
>focused on visualization. Take a look at "Therminator" and
>"The spinning cube of potential doom" etc.
>http://www.nersc.gov/nusers/security/TheSpinningCube.php

This is fairly similar to (and, as near as I can tell, more limited
than) something I wrote about a year earlier:  the shoki packet hustler
(or hustler(1)).  Screenshots, source, and even a little documentation
is available at:

	http://shoki.sourceforge.net/hustler/

Short _precis_:  it's a 3D GUI that lets you plot arbitrary values from
packet data (here in the form of libpcap dumpfiles).  There's also some
clustering code that's probably only of interest to statistical intrusion
detection loons.

>I'm not convinced of the value of such systems outside of the
>cool-factor but it's mostly because I keep seeing them as
>just different ways of accessing the same underlying metaphors
>and presenting them in new ways.

I agree, but I'm not at all dismissive of gaining different ways to
access the same underlying data.  If I'm looking at a visualisation
of a whole mess (and I use that word advisedly) of network data and
my eyes can identify something that looks `interesting' even if I can't
immediately enunciate what that is, then I stil consider that a Big Win.
Granted, if a substantial portion of the things that look interesting
to the casual visual grep are, upon further examination, revealed to be
uninteresting, then this is just an elaborate way to waste your time.
Certainly wouldn't be the first time a IDS technology turned out to have
this as its primary effect.

But my point is that if we get `something odd happened around 23:30'
out of looking at a plot -and no more-, then this can be enough to
justify the exercise.  Ideally we'd like a widget to tell us -what-
happened, and (better still) -why- it happened, but merely indicating
that it did in fact happen is still a Win.

It has been my (admittedly biased, since I really dig this kind of thing)
experience that visualisation widgets frequently earn their keep this
way.

>The underlying metaphors
>are really moving averages, runs tests, and distances from the
>mean. What we haven't figured out how to do is use them in a
>way that helps, so visualizing is really just a cool way of
>graphically twiddling the "gain" "bass" and "treble" to see
>what comes out.

Sure, but there are lots of fairly complex relationships that your
eye can pick out of a plot that are -hugely- computationally expensive
to test for[0].  Particularly when you're looking at plots of n variables,
and n starts getting fairly large.

Anyway, anyone who is skeptical about the utility of such systems (and,
really, everyone should be.  I am, and I've written 'em) should take a
look at similar work in other fields.  I.e., phone misuse detection systems
and the like.  I think it is the case that there aren't any systems for
doing this sort of thing for network data that are `there' yet, but it's
worth noting that none of them[0] are as sophisticated[1] as the traffic
analysis systems routinely used by Allied intelligence in early '40's, for
example.

- -spb

- -----
0	Indeed, there are many relationships of this sort (easily seen,
	only enunciated with difficulty) for which no generalised
	algorithm for n variables is known to exist.
1	At least none in the private sector.
2	In terms of technique rather than implementation.  We've got more
	elaborate machines, but our tradecraft, on the whole, stinks.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (OpenBSD)

iD8DBQFBJFMCG3kIaxeRZl8RAmZNAJ9kcmPOc7BjeJXcmnjZfzlqMMJ7KACguqYD
G2luoEEkksMASCTp+16bJJU=
=po+2
-----END PGP SIGNATURE-----
_______________________________________________
LogAnalysis mailing list
LogAnalysis@private
http://lists.shmoo.com/mailman/listinfo/loganalysis