"The advantage of this approach is that it's dumb, it's cheap -- and it catches stuff you don't know about already." --mjr (who wasn't really talking about this code.. but..)

Description

There are at least two ways to scan your syslogs for security problems or other errors. One is to grep for data you know is indicative of a problem .. "refused connect from", "REPEATED LOGIN FAILURES", "STOR .rhosts". This method requires you to generate huge lists of things you think are "suspicious behaviour" -- lists which are by their nature incomplete; you need to know that a problem exists in order to detect it.

The other way is to make a list of all the "normal" system behaviour -- connections from "trusted" systems, ordinary mail delivery, cron logs, named stats messages, and so on-- and ignore it. Everything else is considered "out of the ordinary" and hence should be paid close attention to. This program attempts to do this. I didn't come up with this idea on my own; in late 1996 I'd been reading about it in the IDS list (Intrusion Detection Systems) for some time, and on 4 December 1996, decided to implement it.

This program is designed to analyze Unix syslog files, the concept applies to logfiles with largely predictable content-- web server error logs, for instance.

The rule file

The checksyslog rule file contains regular expressions which describe normal behaviour. The simplest case to describe is this: assume you're using tcp wrappers on your telnet daemon and while you want to see who telnets to you, you know that connections from linux.friendly.org and pc2.foo.edu are "safe", but you definitely want to know if tcp wrappers refused the connection for some reason.

So you spend some time looking through your syslog, and you come up with entries such as:

Apr 15 12:34:45 mybox in.telnetd[9484]: connect from shell.isp.com 
Apr 22 01:25:26 mybox in.telnetd[16501]: connect from linux.friendly.org
Apr 23 14:27:39 mybox in.telnetd[2492]: connect from linux.friendly.org
Apr 27 20:13:02 mybox in.telnetd[36035]: connect from pc2.foo.edu
Apr 29 03:31:29 mybox in.telnetd[635]: REFUSED CONNECT FROM 0wned.broken.com
 

You would then construct a rule such as:

in.telnetd\[\d+\]: connect from (friendly\.org|pc2\.foo\.edu)

This essentially says "match the string 'in.telnetd[a number ]: connect from either friendly.org or pc2.foo.edu'"

Note that regexp metacharacters such as [, ], and . are escaped by putting a backslash in front of them. Also note the use of the perl regexp \d+, which matches 1 or more digits.

Now, when you run checksyslog against this syslog file, only two entries will stand out:

Apr 15 12:34:45 mybox in.telnetd[9484]: connect from shell.isp.com 
Apr 29 03:31:29 mybox in.telnetd[635]: REFUSED CONNECT FROM 0wned.broken.com

The rule file also supports definitions, so instead of spelling out the friendly.org and pc2.foo.edu hostnames multiple times in the rule file (which can get tedious, particularly if you want to add or remove things from the list), you can do something like:

%SECURE_HOSTS=(friendly\.org|pc2\.foo\.edu)
. . .

# ignore telnet or ftp sessions from %SECURE_HOSTS

in.telnetd\[\d+\]: connect from %SECURE_HOSTS
 

The construction of rules is the most painful part of configuring checksyslog. There is a sample rule file which hopefully makes things clearer.

Setup

This program should be running on your master loghost -- your other systems should have a line in syslog.conf like

        *.debug                 @loghost.my.domain
 

so "Everything" gets logged to a single file; this is how I like to do things. If you're one of these people with 700 different lines in your syslog.conf pointing to different files and log levels, then you'll have to run a separate instance of checksyslog to analyze those files. (My philosophy: "Log 'em all and let Grep sort 'em out." :-)

In some cases (observed when logging from Solaris 2.5 to Linux), the end of line will be padded with extra (invisible to the nekkid eye) characters. This will affect your usage of '$' to indicate the end-of-line in the rules file.

Edit the example.rules file to suit your particular setup -- you may need to run checksyslog several times in order to separate the wheat from the chaff. Copy the rulefile to an appropriate location, and chmod it to 400 (why give out more information when you don't need to?)

Usage

checksyslog takes the following options:
• --rules rulefilename -- Specifies the location of the rule file. This is a required argument.
• --log logfilename -- Specifies the location of the logfile (i.e., /var/log/syslog). If not set, input is taken from stdin.
and ONE of:
• --today -- Only show messages from today
• --filter filterstring -- Only extract message matching this particular filter (make sure you quote "things that might look like arguments to the shell")

In an effort to make the resulting output more palatable, another program named 'resort' breaks up the output of checksyslog on process name, instead of by date:

 ==
 == nimue -- in.ftpd (2 entries)
 ==
 May  3 01:23:02 nimue in.ftpd[27156]: connect from 172.16.195.3
 May  3 01:23:04 nimue in.ftpd[27157]: connect from 134.114.84.1

 ==
 == nimue -- ftpd (4 entries)
 ==
 May  3 01:23:03 nimue ftpd[27156]: FTP LOGIN FAILED (cannot set guest privileges) for nimue.int.jammed.com [172.16.195.3], ftp
 May  3 01:23:03 nimue ftpd[27156]: FTP session closed
 May  3 01:23:04 nimue ftpd[27157]: FTP LOGIN FAILED (cannot set guest privileges) for nimue.int.jammed.com [134.114.84.1], ftp
 May  3 01:23:04 nimue ftpd[27157]: FTP session closed

 ==
 == nimue -- PAM_pwdb (19 entries)
 ==
 May  1 01:13:08 nimue PAM_pwdb[1836]: (su) session closed for user root
 May  1 13:22:23 nimue PAM_pwdb[12883]: (su) session opened for user root by jwa(uid=100)
 May  1 13:23:02 nimue PAM_pwdb[12883]: (su) session closed for user root
 May  1 14:59:11 nimue PAM_pwdb[13324]: (su) session opened for user root by jwa(uid=100)
 
 [ .. snipped for brevity .. ]
 

Suggested usage:

  checksyslog --today --rules /usr/lib/checksyslog.rules --log /var/log/syslog | resort 

I run a variant of this out of cron at 23:59 (so --today works. I really otta add a --yesterday .. :-)

Notes

If you're logging information from multiple systems, you must implement some sort of time synchronization -- at the minimum, rdate, but ideally ntpdate/xntpd. Logfile information is worthless unless it reflects accurate times.

If your rules are sufficently vague or if an intruder can peek at your rulefile, in theory he or she could launch an attack on your system without triggering checksyslog -- as long as the intruder imbeds a rule entry in each system log file message he creates. In practice, this is difficult to accomplish, but it is something that should be taken into account.

You can't grep dead trees. But hopefully, this program will reduce the amount of paper :)

Please contact me (jwa@jammed.com) if you have any comments, suggestions, questions, bugs, or useful rules.