RE: [logs] Visual Event Analysis WAS: most popular reports...?

rob.jenkins@private

Hi All

I must confess to lurking for a while on this thread, it is a fascinating
subject which has rightly generated a lively discussion.

I come with the perspective of a developer of general data analysis tools
and in fact many of the principles of this discussion whilst it's network
log analysis specific does actually hold true for any type of data analysis.
After all data mining is often defined as 'finding patterns and anomalies in
data' which is precisely what we're talking about here.

I would add my support for the value of visualisation in the data analysis
and data mining process. I recently presented on this at the SGI User Group,
the slides are at ftp://ftp.purpleinsight.com/slides/ username:
purpleinsight password: purple if anyone is interested.

To analyze data of any size one typically goes through several common stages
for example: understanding, filtering, transforming, modeling the data,
presenting findings etc. Visualisation can be valuable at every stage of
that process. One thing that is also generally true and applies in the
specific case of log analysis is that for best results one needs to combine
data analysis tool/process expertise with domain expertise. For example I
can successfully process log data and pull useful insight from it but I'm no
network security expert, adding that expertise really produces exciting
results.

Much of the discussion to date has been about analysis (visual or otherwise)
of the 'raw'data i.e. without adding anything to data. There has been some
talk of statistical processing but there is scope for further processing for
example 'clustering' of events to aid in anomaly detection or modeling of
data to determine which attributes in the data are most significant in
determining another attribute.

As data/traffic keeps on growing we're all going to have to find new ways to
get our hands round it all, visualisation is bound to be an important part
of that.

Cheers
Rob

Rob Jenkins
p u r p l e | i n s i g h t  
rob.jenkins@private

mobile: +44(0)7815 777686
direct:  +44(0)1189 484604
fax:      +44(0)1189 483517

www.purpleinsight.com

> -----Original Message-----
> From: 
> loganalysis-bounces+rob.jenkins=purpleinsight.com@private
> com 
> [mailto:loganalysis-bounces+rob.jenkins=purpleinsight.com@list
> s.shmoo.com] On Behalf Of Raffael Marty
> Sent: 19 August 2004 17:27
> To: Marcus J. Ranum
> Cc: loganalysis@private; Jose Nazario
> Subject: [logs] Visual Event Analysis WAS: most popular reports...?
> 
> Marcus,
> 
> > I'm not convinced of the value of such systems outside of the 
> > cool-factor but it's mostly because I keep seeing them as just 
> > different ways of accessing the same underlying metaphors and 
> > presenting them in new ways.
> 
> So far I agree with you. Visualization techniques cannot 
> produce information. We all are operating on the same sets of 
> data, either events from some kind of a system or raw packet 
> dumps. Visualization does certainly not generate new information.
> 
> > The underlying metaphors are really moving averages, runs 
> tests, and 
> > distances from the mean.
> 
> I don't quite understand what you mean by this. There are 
> more factors that you can visualize. It's not all about 
> statistical analysis and graphing. What about event-graphs 
> (or link-graphs)? They don't have anything to do with moving 
> averages, runs or distances from the mean.
> Maybe I am missing your point here.
> 
> > What we haven't figured out how to do is use them in a way 
> that helps, 
> > so visualizing is really just a cool way of graphically 
> twiddling the 
> > "gain" "bass" and "treble" to see what comes out.
> 
> Here I vastly disagree. I don't think it's just a "cool" way 
> of twiddling data. I think it's a very powerful way of 
> quickly analyzing big amounts of data and getting a feeling 
> for what is going on in a dataset. No report can show you the 
> amount of information that a graph can. A visual 
> representation of several thousend events can give you a very 
> good understanding on what's going on in the data and even 
> uncover anomalies.
> 
> Cheers
> 
> 	-Raffy
> 
> Disclaimer: Raffy's opinions might not be ArcSight's policy.
> 
> -- 
> 
> Raffael Marty, CISSP                          
> raffael.marty@private
> Senior Security Engineer                    Content Team @ 
> ArcSight Inc.
> 5 Results Way            Cupertino, CA  95014             
> (408) 864-2662
> _______________________________________________
> LogAnalysis mailing list
> LogAnalysis@private
> http://lists.shmoo.com/mailman/listinfo/loganalysis

_______________________________________________
LogAnalysis mailing list
LogAnalysis@private
http://lists.shmoo.com/mailman/listinfo/loganalysis