"Desai, Ashish" wrote:
> One of the things we did is to write a script/program
> [ in language of your choice, we chose perl] for every data source we have.
> All the scripts support at least the following arguments,
> start date
> end date
> keyword regex (OPTIONAL)
> Example
> pdata -s 12/15/2001 -e 12/20/2001 -k "128.*"
> # this would spit out to stdout all the data between those dates for
> the matched pattern
>
First off, thanks for a response. Actually, what you've mentioned right here
is quite similar to what I had thought out for writing myself. The basis of it
all would be a more or less 'framework' cgi that accepted plugin 'modules' such
as what you described that would do the individual searching and parsing
necessary; all fitting together to make a mostly dynamic searching/viewing
interface. I'd do this all in perl.
>
> This satisfied our power/intermediate users as they could continue to use
> the standard
> unix tools, grep, pipe etc to play with the results. Note, we output data in
> tab delimited format so
> the standard IFS work fine with the output.
>
The main problem we have, is that we do not necessarily want to grant user
accounts on the viewing machine to anyone and everyone who would fit (or claim
to fit) into the power/intermediate range of users.
Being that we're a university, there are many people from various departments
and colleges who could have a legitimate reason to view portions of data, but
we're sort of cautious of granting direct machine access, thus our desire to
create a web frontend.
>
> Next we wrote small cgi's that call these scripts and save the output to a
> file. We then
> send the requester email with the URL to the results (compressed with gzip).
> This allows the
> naive users to request data and fetch it via a browers. Some of these users
> put the data in Excel
> for analysis while others just use a text editor.
> The advantage of this is once you implement the scripts you can trivally
> reuse them for the web interface
> Also some of our data sets are large, can take upto 48 hrs to search through
> them. So the web interface
> just dumps the request into a file and a shell scripts queues up the
> request. This way the requestor
> does not have to wait at the web interface.
>
I like what you're saying here, I think something along these lines would
probably work for us as a beginning, at least to get us started with something
for the higher-end users. Like you said, a backend 'searching' script would be
simple to tie into a frontend cgi, and thus allow us to easily generate search
results via the web. I guess the real problem is really how to cater to the
varying levels of expertise versus what data they need available without handing
the keys to the kingdom to the expert users, and ignoring the novices.
For example, in my design, it would be possible for a user to select a
filter (or have it forcefully selected for them) that would parse/translate
certain sets of data depending on what they were looking for. For example, a
helpdesk technician trying to troubleshoot a user's RAS (dial-in) problem, could
look at a parsed log of only login attempts/failures/successes w/ only necessary
details without having to understand the 'meat' of what's being logged back from
any of the involved components -- not to mention, in some cases, some details of
the logs might be inappropriate for certain folks to view.
Go figure-- this is probably more of a 'moral' dilemma than not having the
technology available to do the job.
Do you (or anyone else) know offhand if anyone maintains a list of some type of
the various options for syslog consoles and monitors? .. or maybe can make a
recommendation based on what they're using? I guess what I'm really trying to
avoid is to have to write the system myself if something close enough to it
already exists; you know -- no sense reinventing the wheel.
Thanks a lot for your help!
..Sean.
---------------------------------------------------------------------
To unsubscribe, e-mail: loganalysis-unsubscribeat_private
For additional commands, e-mail: loganalysis-helpat_private
This archive was generated by hypermail 2b30 : Mon Nov 12 2001 - 13:35:18 PST