* Cleaned up and function-ized the code. Planning to move the functions to http.lua or similar to let other scripts (like http-userdir-enum.nse) use them.
* Better support for different HTTP error codes
* Significantly improved detection of 404 pages to prevent false positives. Some of the techniques used:
- Request a non-existent page and check the status code
- If it's 200, try to remove anything that may change (times, URI, filenames, etc), md5 it if ssl is available, and store it. Request a second 404 page and do the same. If they don't match, abort.
- If it's any other error code, store it, and look for it in addition to 404 Not Found
- Request the root folder, /
- If it returns a 301 Move Permanently or 401 Authentication Required, abort
* Abstracted the fingerprints into files in the nselib/data directory:
- http-fingerprints: common files/folders
- yokoso-fingerprints: common applications (from the Yokoso project, by InGuardians -- used under the Nmap license with pemission from Kevin Johnson -- http://seclists.org/nmap-dev/2009/q3/0685.html)
* Added a script-arg that can display all output (except known 404s), instead of just 200 OK and 401 Authentication Required
* Added a lot of debugging messages
