Lua 5.3 adds several awesome features of particular interest to nmap including
bitwise operators and integers, a utf8 library, and standard binary pack/unpack
functions.
In addition to adding Lua 5.3, this branch changes:
o Complete removal of the NSE bit library (in C), It has been replaced with
a new Lua library wrapping Lua 5.3's bit-wise operators.
o Complete removal of the NSE bin library (in C). It has been replaced with a
new Lua library wrapping Lua 5.3's string.pack|unpack functions.
o The bin.pack "B" format specifier (which has never worked correctly) is
unimplemented. All scripts/libraries which use it have been updated. Most
usage of this option was to allow string based bit-wise operations which are no
longer necessary now that Lua 5.3 provides integers and bit-wise operators.
o The base32/base64 libraries have been reimplemented using Lua 5.3's new
bitwise operators. (This library was the main user of the bin.pack "B" format
specifier.)
o A new "bits" library has been added for common bit hacks. Currently only has
a reverse function.
Thanks to David Fifield, Daniel Miller, Jacek Wielemborek, and Paulino
Calderon for testing this branch.
proposal. This only affects Nmap's root directory. We might also need to
modify the code which autogenerates Nmap's source code files such as IPv6
fingerprinting code.
see http://seclists.org/nmap-dev/2013/q4/168
Move some includes out of nmap.h: nmap.h gets included lots of places,
and unconditionally included math.h, ctype.h, errno.h, stdio.h,
sys/stat.h, fcntl.h, sys/types.h, and stdarg.h. This commit moves those
includes into the .cc files where they are necessary and out of nmap.h
Remove redundant include global_structures.h, included from nmap.h
Removed redundant code included from nmap.h
Removing #include nbase.h when nmap.h is included (redundant)
Remove duplicate #include lines
Add ifndef guards to a few .h files
Scripts may now return a key–value table, or such a table in addition to
a string. The table will be automatically formatted for normal output
and will appear as a hierarchy of elements in XML output.
Some history and discussion of this development can be found at
https://secwiki.org/w/Nmap/Structured_Script_Output.
This is a merge of r29484:29569 from /nmap-exp/david/xml-output.
referencing deallocated memory.
The class was defined basically as follows:
class ScriptResult
{
private:
std::string output;
public:
std::string get_output() const
{
return this->output;
}
};
The problem was when it was used like this, as in our script output
routines:
const char *s = sr.get_output().c_str();
printf("%s\n", s);
The reason is that the temporary std::string returned by get_output goes
out of scope after the line containing it, which invalidates the memory
pointed to by c_str(). By the time of the printf, s may be pointing to
deallocated memory.
This could have been fixed by returning a const reference that would
remain valid as long as the ScriptResult's output member is valid:
const std::string& get_output() const
{
return this->output;
}
However I noticed that get_output() was always immediately followed by a
c_str(), so I just had get_output return that instead, which has the
same period of validity.
This problem became visiable when compiling with Visual C++ 2010. The
first four bytes of script output in normal output would be garbage
(probably some kind of free list pointer). It didn't happen in XML
output, because the get_output-returned string happened to remain in
scope during that.
This is a maintenance fix for the NSE Nsock library binding. The patch focuses
on code correctness and simplicity. The patch also brings some initial updates
with an eye towards the upcoming Lua 5.2 release. See [1] for a post concerning
this branch.
[1] http://seclists.org/nmap-dev/2010/q3/710
o Add two new Script scan phases:
Script Pre-scanning phase: before any Nmap scan operation, activated by the new "prerule".
Script Post-scanning phase: after all Nmap scan operations, activated by the new "postrule".
o New environment variables:
SCRIPT_PATH
SCRIPT_NAME
SCRIPT_TYPE: the type of the rule that activated the script.
changes. The first is that Port objects don't allocate memory for
service and RPC results unless that information is set. This reduces the
size of a bare Port from 92 to 40 bytes on my machine. The second change
is that PortList now has the notion of a "default port state," which is
the state of any ports that didn't receive a response. These ports don't
need an allocated Port object, which saves a lot of memory in scans
where most ports didn't get a response.
nse_gettarget (lua_State *L, int index);
to replace the current_hosts static variable shared between nse_main.cc
and nse_nmaplib.cc. This improves locality and offers a clearer interface.
again. Here are the items which were merged:
------------------------------------------------------------------------
r13971 | jah | 2009-06-29 14:30:27 -0700 (Mon, 29 Jun 2009) | 2 lines
Improved a pattern for matching HTTP status-line, tidy away some variables and
fix a typo.
------------------------------------------------------------------------
r13967 | daniel | 2009-06-29 13:47:04 -0700 (Mon, 29 Jun 2009) | 5 lines
o Added a convenience top-level BSD makefile redirecting BSD make
to GNU make on BSD systems. This should help prevent bogus
error reports when users run "make" instead of "gmake" on BSD
systems. [Daniel Roethlisberger]
------------------------------------------------------------------------
r13965 | batrick | 2009-06-29 06:50:11 -0700 (Mon, 29 Jun 2009) | 14 lines
[NSE] The NSE Nsock Library binding no longer relies on garbage collection to
monitor the use of socket "slots". A thread (script) attempting to connect must
first obtain one of a limited number of available socket locks (usually 10 or
--max-parallelism). The binding would use garbage collection of sockets to
determine when a thread has finished using its allocated sockets. This is
unfortunately slow and requires us to constantly run the garbage collector to
cause timely reallocation. I have changed the binding to now regularly inspect
allocated sockets in the nsock_loop function. Available sockets slots are now
immediately reallocated and done with far less execution time.
See [1] for benchmarks and further explanation.
[1] http://seclists.org/nmap-dev/2009/q2/0624.html
------------------------------------------------------------------------
r13964 | batrick | 2009-06-29 06:37:49 -0700 (Mon, 29 Jun 2009) | 10 lines
[NSE] Fixed a rare (and usually undetectable) bug that can cause a SEGFAULT.
The NSE nsock library binding may attempt to push values on the stack of
a thread that ended due to an error. It is possible that the internal
Lua stack was completely full and any further pushed values would result
in a segmentation memory violation.
This bug is very hard to reproduce with a SEGFAULT but is usually visible
when Lua assertion checks are turned on. A socket handler routine must be
called AFTER a thread has ended in error.
------------------------------------------------------------------------
r13963 | batrick | 2009-06-29 05:51:20 -0700 (Mon, 29 Jun 2009) | 3 lines
Fixed some global scoped variables to be local. This caused a many scripts to
overwrite each others' sockets, options, etc.
------------------------------------------------------------------------
r13939 | joao | 2009-06-27 16:07:35 -0700 (Sat, 27 Jun 2009) | 2 lines
Fixed port rule to include ssl pop3 port, now that pop3.lua supports SSL connections in function capabilities
------------------------------------------------------------------------
r13938 | joao | 2009-06-27 16:06:28 -0700 (Sat, 27 Jun 2009) | 2 lines
Added transparent SSL support using comm.tryssl
------------------------------------------------------------------------
r13937 | joao | 2009-06-27 16:05:19 -0700 (Sat, 27 Jun 2009) | 2 lines
Added transparent SSL support using comm.tryssl
------------------------------------------------------------------------
r13936 | joao | 2009-06-27 16:03:50 -0700 (Sat, 27 Jun 2009) | 2 lines
Added SSL transparent support using comm.tryssl
------------------------------------------------------------------------
r13935 | joao | 2009-06-27 16:02:39 -0700 (Sat, 27 Jun 2009) | 2 lines
Added SSL transparent support using comm.tryssl
------------------------------------------------------------------------
r13934 | joao | 2009-06-27 16:01:38 -0700 (Sat, 27 Jun 2009) | 2 lines
Added SSL transparent support using comm.tryssl
------------------------------------------------------------------------
r13933 | joao | 2009-06-27 16:00:27 -0700 (Sat, 27 Jun 2009) | 2 lines
SSL transparent support using comm.tryssl
------------------------------------------------------------------------
r13932 | joao | 2009-06-27 15:19:58 -0700 (Sat, 27 Jun 2009) | 2 lines
Included transparent ssl support to function pop3.capabilities using comm.tryssl
------------------------------------------------------------------------
r13931 | joao | 2009-06-27 15:19:06 -0700 (Sat, 27 Jun 2009) | 3 lines
New version of comm.lua with function tryssl, that transparently adds support to ssl connections
------------------------------------------------------------------------
r13930 | joao | 2009-06-27 14:50:38 -0700 (Sat, 27 Jun 2009) | 6 lines
Fixed buffering problem exposed by david on nmap-dev list.
The problem was solved using a buffer to receive the data, making the script work fine in cases where the ssh packets are fragmented.
A very similar solution was applied to ssh1.lua.
------------------------------------------------------------------------
r13928 | batrick | 2009-06-27 04:43:12 -0700 (Sat, 27 Jun 2009) | 18 lines
[NSE] We now propogate a NSE initiated yield on a script through all user
coroutines so that NSE may resume control. Previously, scripts that would yield
in a child coroutine (e.g. a script's child coroutine generated by Lua's
coroutine.create function) would give control back to the script. A script
would yield in this way by making a blocking socket operation. NSE would be
unable to correctly resume child coroutine when the socket operation is
finished processing.
By yielding the chain of coroutines a script has operating, we allow to NSE to
handle the socket operation properly. NSE would then resume the entire chain so
execution may correctly resume at the coroutine which initiated the socket
operation. This restores the "illusion" that a script executes without
interruption.
See [1] for more information, further explanation, and some use cases.
[1] http://seclists.org/nmap-dev/2009/q2/0586.html
------------------------------------------------------------------------
r13817 | david | 2009-06-18 15:57:29 -0700 (Thu, 18 Jun 2009) | 3 lines
Improve an OS fingerprint with a model number and broader matching.
Based on a follow-up report from a submitter.
------------------------------------------------------------------------
r13814 | josh | 2009-06-17 21:34:15 -0700 (Wed, 17 Jun 2009) | 3 lines
[zenmap] Added support to zenmap for the new SCTP options: -PY, -sY and -sZ
------------------------------------------------------------------------
r13797 | ron | 2009-06-17 11:02:18 -0700 (Wed, 17 Jun 2009) | 1 line
Applied a patch from Mak Kolibabi that enhances the output of smb-enum-processes. The output is now modeled after the output of the 'ps' tool for higher verbosity levels.
------------------------------------------------------------------------
r13795 | david | 2009-06-17 09:05:21 -0700 (Wed, 17 Jun 2009) | 6 lines
The configure script now allows cross-compiling by assuming that
libpcap is recent enough. Previously it would quit because a test
program could not be run. libpcap will always be recent enough when
the included copy is used. The patch was contributed by Mike
Frysinger.
Here is a mostly exhaustive list of the changes:
o Removes the SCRIPT_ENGINE_* status defines and replaces the
instances with regular boolean integer returns or changes the
procedure to return void. The latter case is better generally because
the caller ignores any status return (e.g. nmap.cc calling open_nse)
and/or the procedure raises a fatal error when unsuccessful.
o Moves the SCRIPT_ENGINE_LUA_DIR and the like to the nse_main.h header file.
o Removes the use of the SCRIPT_ENGINE_TRY (there was only one left)
and thus changes the call to l_dnet_open to a void function called
directly by luaopen_nsock (in nse_nsock.cc) instead of luaopen_nmap
(in nse_nmaplib.cc). I felt moving the function was also an
appropriate (but somewhat unrelated to the intent of the patch) change
as opening the dnet metatable is very related to opening up the
nsock library. This confines errors in opening the nsock library, including
opening the dnet metatable, to the call to luaopen_nsock.
o The FILES and DIRS defines are moved in to nse_fs.h where they are
more appropriate and localalized.
Some scripts would fail due to an error (whois.nse) causing other
scripts to become deadlocked on a mutex that would never unlock. This
patch fixes this problem. See [1] for more information.
[1] http://seclists.org/nmap-dev/2009/q2/0533.html
Moved the includes for Lua headers to the .cc files so they are
not needlessly, repeatedly included.
Similarly, moved some standard headers to the .cc files and reorganized
includes to be uniform for all nse_* source files.
Fixed whitespace (removed tabs).
