[Plugins-writers] Better false positive detection for CGI scanning

roelof@private

Hi there,

Nessus (and just about all of the scanners we have seen) have problems
with false positives against sites that return non-standard responses to
certain requests. no404.nasl definately cuts down on some of them but
still leaves you with a fair amount of a chance to get it wrong..

Some time ago we released a tool called Wikto. Wikto used the Nikto scan
database, _but_ you can set Wikto to ignore error/return codes and just
rely on message content to decide if the CGI is there or not. This is
achieved by comparing the results (in real time) of an actual CGI request
with a database of results from known non-existant requests.

This method adds some overhead but greatly decreases the occurance of
false positives. We have wrapped some of its logic into the following
wikto.inc

Currently Nessus uses no404.nasl for the detection of "friendly" 404
messages. This works fairly well..except when the server has different
behaviour for certain extensions, or responds differntly for different
locations. Wikto tries to solve this by storing responses according to
location and extension.

Extracting locations and extentions from attacks is not always possible.
Clearly /scripts/login.asp is an ASP file and located in /scripts/, but
how about /%c0%af/.../test?go=/cgi-bin/\\admin.pl ?

Because of this nastiness these parameters can be passed as arguments to
functions within wikto.inc (If they are not specified (as all NASLs will
be currently) best effort is done to extract it correctly. If the
extension cannot be determined (or if the request has no extension) the
value of "default" is given.

The response is saved in the KB (which dramatically increases the size of
the KB [averages about 120k on most sites we tested]) with a key of
www/<port>/<path>/<ext>. A typical key would be
"www/80//webcart/config/txt". This means that the checker does not need to
replay the dummy request for subsequent tests.

The results:

1) Server comes back with a friendly 200 Message if the resource cannot
be found.

Without no404.nasl and without wikto :
55 holes, 27 warnings [all false positives]

With no404.nasl and no wikto :
1 hole, 5 warnings [no false positives]

Without no404.nasl and with wikto :
1 hole, 5 warnings [no false positives]

---------------------------------------------------------------------------------------

2) Server responds with 200s on EXEs but normal 404 on resources it
cannot find

Without no404.nasl and without wikto :
18 holes, 3 infos [all false positives]

With no404.nasl and no wikto :
18 holes, 2 infos [all false positives]

Without no404.nasl and with wikto :
0 holes, 1 info [no false positives]

During the same tests we also includes some vulnerabilities on the servers
to make sure that these will be detected (and it did so nicely).

When the two messages are compared (VERY primitive at the moment, but
works) an index is given - this index shows how close the messages match.
Because of changing server time the messages never quite match perfectly.
The value of the index range from 0-100 (and sometimes >100, I said it was
primitive). At the moment the nasl triggers for anything below 50. If the
community likes the concept I guess we could have this as a tweakable
value in the config. Keep in mind that the lower you have the value, the
less false positives you would get...but the greater the chance of missing
real problems on the server.

The script is a "drop-in" replacement for is_cgi_installed_ka. It takes
4 parameters:

function is_cgi_installed_ka(item, port, location, ext)

  item is backwards compatible item...
  port is .. well..the port
  location is the location of the scripts - e.g. /scripts/
  ext is the extension (without a dot) - e.g. dll

  The last two parameters are optional - as said before - if you dont trust
  the extension and location extraction tools please populate this.

  If you want to play with this do the following:

  1. Add the new functions to the end of http_keepalive.inc

  2. In http_keepalive.inc, change the current is_cgi_installed_ka to
is_cgi_installed_ka_old. That way you can switch back to the old style
checking easily if need be.

  3. You can probably just make no404.nasl do an "exit(0)" if you want to
test the results of a full scan without no404. (There's no need for this
tho..)

---snip--

PS: I suspect the badblue_null.nasl is broken (you will see it show up
on every scan if you using wikto tech).

Line 57 reads:

r = string("/ext.ini.% 00.txt");

which is exactly how it appears on Bugtraq. Wikto sees the path as / and
the ext as txt. It now requests /SP_never_SP.txt, and gets the page (let's
for now assume it's a 404 Not Found). If you request /ext.ini.% 00.txt you
almost certainly get back a 400 Bad Request. Because the requests dont
match it is treated as a positive. Shouldn't it be /ext.ini%00.txt? Makes
more sense to me, unless I am missing the point?

Anyhow..enjoy!! Please send flames/comments to research@private

Regards,
Roelof.

=====================
Roelof Temmingh
+ 27 12 460 0880
GMT+2
=====================