In deference to Tina's request that vendors provide detailed technical information regarding their product's capabilities, this post runs a little long. Addamark Technologies developed the Scalable Log Server specifically to meet the requirements driven by large volume log data management. The Scalable Log Server * Scalable log server is a Linux based software solution which runs on a cluster of Linux/Intel servers (typically 5) * Log files loaded into a Scalable Log Server are distributed equally among the servers in the cluster, so that each log record is sent to two separate servers - one as primary, one as secondary * The log records are parsed locally on each server into columns for storage in a table structure. * table data are stored in a well defined directory structure on each server, which implements a B-tree index on the timestamps of the log records. * Each column's data is stored in a separate file in the appropriate leaf-node of the B-tree * Each column file is compressed with gzip, but the overall compression of the data is usually superior to gzipping the raw original file, since multiple log records in a short time range tend to repeat certain fields (IP address, port, etc). The end result of this architecture provides a very scalable architecture, since parsing, compression, and uncompression are CPU and memory-intensive processes which run well in parallel on multiple machines. Because the CPU and memory overhead far outweighs communications, scalability is essentially linear as the number of servers in the cluster grows. Because of the efficient compression, an entry-level 5-server cluster is frequently capable of managing terabytes of logs on local disk. Unlike relational databases, there is no rollback, index or temporary partitions needed on the disk - all space is used for data storage. Since all data is stored on two servers, and all servers in the cluster are peers, the system is fault-tolerant. Total failure of any server will result in no data loss, and only proportional performance loss. All data is stored in regular files in the regular filesystem, so that backup, restore and maintenance is accomplished by system administrators with their usual tools. Data stored in the Scalable Log Server is accessed via SQL, which can optionally be extended with Perl functions. Interfaces include ODBC, Perl DBI::DBD, Addamark's command-line utilities and web based Analyzer product. To complement the Scalable Log Server, Addamark provides a Collector, which manages collection and loading of logs from around an enterprise, and a Security Analytics Subscription, which provides a variety of tools and reports for log generating devices and applications. best regards, -- Kevin Hanrahan 415-281-1900 x105 Director, Security Strategy 707-342-2037 (cell) Addamark Technologies kevinh@private _______________________________________________ LogAnalysis mailing list LogAnalysis@private http://lists.shmoo.com/mailman/listinfo/loganalysis
This archive was generated by hypermail 2b30 : Fri Feb 13 2004 - 12:08:14 PST