Lua 5.3 adds several awesome features of particular interest to nmap including
bitwise operators and integers, a utf8 library, and standard binary pack/unpack
functions.
In addition to adding Lua 5.3, this branch changes:
o Complete removal of the NSE bit library (in C), It has been replaced with
a new Lua library wrapping Lua 5.3's bit-wise operators.
o Complete removal of the NSE bin library (in C). It has been replaced with a
new Lua library wrapping Lua 5.3's string.pack|unpack functions.
o The bin.pack "B" format specifier (which has never worked correctly) is
unimplemented. All scripts/libraries which use it have been updated. Most
usage of this option was to allow string based bit-wise operations which are no
longer necessary now that Lua 5.3 provides integers and bit-wise operators.
o The base32/base64 libraries have been reimplemented using Lua 5.3's new
bitwise operators. (This library was the main user of the bin.pack "B" format
specifier.)
o A new "bits" library has been added for common bit hacks. Currently only has
a reverse function.
Thanks to David Fifield, Daniel Miller, Jacek Wielemborek, and Paulino
Calderon for testing this branch.
This was previously done for --script-args-file with a blind gsub of all
newlines with commas (","), which meant that quoted strings could not
contain newlines. Now they can.
The keyword patterns like "categories" or K "true" were consuming the
first part of patterns like --script 'broadcast-*' resulting in the
error "'broadcast-*' did not match a category, filename, or directory"
Changed to add a lookahead match for space, parentheses, or end-of-line
before considering a keyword to have matched.
Update the patterns used to extract hex chars from the string representation of a coroutine. It seems the string has changed in Lua 5.3 from "0xAB..." to "thread: AB..."; this was before:-
NSE: Starting http-feed M:nil against ...
NSE: http-feed M:nil spawning new thread (thread: 02C63A78).
NSE: Finished http-feed W:nil against ...
this is now:-
NSE: Starting http-feed M:02D6EAF0 against ...
NSE: http-feed M:02D6EAF0 spawning new thread (thread: 02D6E640).
NSE: Finished http-feed W:02D6E640 against ...
This allows usage of verbose/debug in portrule/hostrule and access to all
functionality of NSE, including sockets. So for example, we can now do:
function portrule (host, port)
local response = http.get(host, port, "/");
stdnse.debug1(response.body)
...
end
The verbose/debug function did not work in rule functions because the
introspection API (getid, gettid, etc.) only work when NSE is in the main loop.
The main loop sets the required internal variable current needed by the API.
List of changes:
stdnse.lua:
o debug/verbose check the debugging/verbosity level much earlier to allow
returning if nothing will be printed.
o Simplified debug/verbose logic to handle the optional first argument
better.
o made debug/verbose local functions to avoid using globals and allow self
tail calls
nse_main.lua:
o The logic for adding threads via a rule function is simplified. So long as
the script has the desired rule function, a thread is always returned.
Evaluation of the rule function is done while NSE is in the main loop (i.e.
not in script:new_thread()). The rule function only determines if the action
function is run.
o [Not a change:] If the action function will be run or was run then we see
the usual "Starting X" and "Finished X" messages from NSE.
o Use Lua 5.2's pack function instead of the slightly more expensive
{n = select("#", ...), ...} idiom.
o New stdnse.getinfo introspection function which is used by stdnse.debug.
string.gsub returns 2 values, the new string and the number of
replacements made. It also has a 4th argument, the number of
replacements to make. So when you use the return value of gsub as the
3rd argument, and no replacements were made, it instructs the next call
to not make any replacements. Thanks to Ron Bowes for reporting this
issue.
A user reported this crash when scanning a target whose name contained
the '%' character:
NSE: Script Engine Scan Aborted.
An error was thrown by the engine: nse_main.lua:322: invalid capture index
stack traceback:
[C]: in function 'gsub'
nse_main.lua:322: in function 'd'
nse_main.lua:377: in function 'start'
nse_main.lua:912: in function 'run'
nse_main.lua:1390: in function <nse_main.lua:1293>
[C]: in ?
I'm not sure how a name with '%' got resolved, but I was able to
reproduce the crash by adding this line to /etc/hosts:
127.0.0.1 a%40b
and then running
./nmap --script=banner a%40b -d --top-ports 5
The gsub function recognizes "%d", where d is a digit, as a capture
index. The constructed string is then passed to print_debug, which is
like printf. Therefore we escape every occurrence of "%" twice, to get
"%%%%".
final script-args table. The rationale is, unfortunately shells interpret
quotes differently and so it can be hard to tell exactly what NSE ends up
seeing/producing. [Some discussion in #nmap on Freenode resulted in this
addition.]
o Moved some specific behavior for resuming a thread to Thread:resume().
o Cleaned up the inheritance mechanism to use a static metatable (the Class
table itself).
o Worker main functions are no longer wrapped with a function to truncate to 0
results. Instead, we just check if a thread is a worker in Thread:set_output()
to prevent adding output by worker threads.
NSE: Script Engine Scan Aborted.
An error was thrown by the engine: attempt to get length of a number
value
stack traceback:
[C]: in function 'port_set_output'
/home/rt/builds/nmap/nse_main.lua:314: in function 'set_output'
/home/rt/builds/nmap/nse_main.lua:947: in function 'run'
/home/rt/builds/nmap/nse_main.lua:1330: in function
</home/rt/builds/nmap/nse_main.lua:1237>
[C]: in ?
This now allows return of anything stringifiable as second return value
without crashing (though this may not be what the script author intends,
better to not crash)
