[Plugins-writers] Better false positive detection for CGI scanning (Modified by charl van der walt)

roelof@private

Hi there,

This is a re-post of a submission we sent just before Christmas last 
year. We received ... 0 ... response at the time, which either means 
that this is an idiotic submission, or that you were all simply too 
busy with other holiday-related things. ;) In the hope that it's the 
latter, we're posting this again. Please forgive the re-posting. We 
really wouldn't do it if we didn't think it was worthwhile. We've also 
posted to two lists. You'll have to forgive us for that also. The 
point...

Some time ago we released a tool called Wikto. Wikto used the Nikto 
scan database. It's a .NET tool. The main difference between Nikto and 
Wikto was that you could set Wikto to ignore error codes and just rely 
on message content to decide if the CGI is there or not. This was 
achieved by building a database of responses for non-existent files and 
doing a content compare between that and the response for the actual 
request (this is done in real time BTW).

The method greatly decreases the occurance of false positives. It also 
adds a little bandwidth (and time) overhead.

Currently Nessus uses no404.nasl for the detection of "friendly" 404 
messages. This works fairly well..except when the server has different
behaviour for certain extentions, or responds differntly for different 
locations. Wikto tries to solve this by storing responses according to
location and extention.

Extracting locations and extentions from attacks are not always easy. 
Clearly /scripts/login.asp is an ASP file and located in /scripts/, but
how about /%c0%af/.../test?go=/cgi-bin/\\admin.pl ? Because of this 
nastiness these parameters can be specified manually. If they are not
specified (as all NASLs will be currently) best effort is done to 
extract it correctly. If the extention cannot be determined (or if the 
request has
no extention) the value of "default" is given.

The response is saved in the KB (which dramatically increases the size 
of the KB [with about 120k]) with a key of www/<port>/<path>/<ext>. A 
typical key would be "www/80//webcart/config/txt". This means that the 
checker does not need to replay the dummy request if it is already in 
the KB.

So - how well does it work? Tested it against 2 servers:

1) Server comes back with a friendly 200 Message if the resource cannot 
be found.
------------------------------------------------
Without no404.nasl and without wikto style is_cgi: 55 holes, 27 
warnings [all false positives]

With no404.nasl and no wikto style is_cgi: 1 hole, 5 warnings [no false 
positives]

Without no404.nasl and with wikto style is_cgi: 1 hole, 5 warnings [no 
false positives]

2) Server responds with 200s on EXEs but normal 404 on resources it 
cannot find
-------------------------------------------------
Without no404.nasl and without wikto style is_cgi:
18 holes, 3 infos [all false positives]

With no404.nasl and no wikto style is_cgi:
18 holes, 2 infos [all false positives]

Without no404.nasl and wikto style is_cgi:
0 holes, 1 info [no false positives]

During the same tests we also includes some vulnerabilities on the 
servers to make sure that these will be detected (and it did so 
nicely).

When the two messages are compared (VERY primative at the moment, but 
works) a index is given - this index shows how close the messages 
match. Because of changing server time the messages never quite match 
perfectly. The value of the index range from 0-100 (and sometimes >100, 
I said it was primative). At the moment the nasl triggers for anything 
below 50. If the community likes the concept I guess we could have this 
as a tweakable value in the config. Keep in mind that the lower you 
have the value, the less false positives you would get...but the 
greater the chance of missing stuff.

The script is a "drop-in" replacement for is_cgi_installed_ka. It takes 
4
parameters:

function is_cgi_installed_ka(item, port, location, ext)

item is backwards compatible item...
port is .. well..the port
location is the location of the scripts - e.g. /scripts/
ext is the extension (without a dot) - e.g. dll

The last two parameters is optional - as said before - if you dont 
trust the ext and location extraction tools please populate this.

If you want to play with this do the following:

1. Add the new functions to the end of http_keepalive.inc

2. Change the current is_cgi_installed_ka to is_cgi_installed_ka_old 
within http_keepalive.inc.

3. If you want to see it work without no404.nasl then just do an 
exit(0) early on in the script. There's no need for this tho..

If you have a mission critical server..make sure you test it properly 
before you use it. It might not be everyone's cup of tea.

PS: I suspect the badblue_null.nasl is broken (you will see it show up 
every time with this tech). Line 57 reads:

r = string("/ext.ini.% 00.txt");

which is exactly how it appears on Bugtraq. Wikto sees the path as / 
and the ext as txt. It now request /SP_never_SP.txt, and gets the page 
(let's for now assume it's a 404 Not Found). If you request /ext.ini.% 
00.txt you almost certainly get back a 400 Bad Request. Because the 
requests dont match it is treated as a positive. So..shouldn't it be 
/ext.ini%00.txt? Makes more sense to me, unless I am missing the point?

Anyhow..enjoy and please send flames and comments to 
research@private

Regards,
SensePost.