PenTest/nmap
1
0
Fork 0
mirror of https://github.com/nmap/nmap.git synced 2025-12-08 13:41:29 +00:00
Code Issues Projects Releases Wiki Activity
Files
00b8d455c300da4a3b18b47374bec30559f5dd8d
nmap/docs/scripting.xml

2451 lines
108 KiB
XML
Raw Blame History

<indexterm class="startofrange" id="nse-indexterm"><primary>Nmap Scripting Engine (NSE)</primary></indexterm>
<indexterm><primary>scripting</primary><see>Nmap Scripting Engine</see></indexterm>
<indexterm><primary>NSE</primary><see>Nmap Scripting Engine</see></indexterm>
<sect1 id="nse-intro">
<title>Introduction</title>
<para>The Nmap Scripting Engine (NSE) is one of Nmap's most
powerful and flexible features. It allows users to write (and
share) simple scripts to automate a wide variety of networking
tasks. Those scripts are then executed in parallel with the speed
and efficiency you expect from Nmap. Users can rely on the
growing and diverse set of scripts distributed with Nmap, or write
their own to meet custom needs.</para>
<para>We designed NSE to be versatile, with the following tasks in mind:</para>
<variablelist>
<varlistentry>
<term>Network discovery</term>
<listitem>
<para>This is Nmap's bread and butter. Examples include
looking up whois data based on the target domain,
querying ARIN, RIPE, or APNIC for the target IP to determine ownership,
performing identd lookups on open ports, SNMP queries, and
listing available NFS/SMB/RPC shares and services.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
<indexterm><primary>version detection</primary><secondary>using NSE</secondary></indexterm>
More sophisticated version detection</term>
<listitem>
<para>The Nmap version detection system (<xref linkend="vscan"/>)
is able to recognize thousands of different services through
its probe and regular expression based matching system, but it
cannot recognize everything. For example, identifying the Skype v2 service requires two
independent probes, which version detection isn't flexible enough to handle. Nmap could also recognize more SNMP services
if it tried a few hundred different community names by brute
force. Neither of these tasks are well suited to traditional
Nmap version detection, but both are easily accomplished with
NSE. For these reasons, version detection now calls NSE by
default to handle some tricky services. This is described in
<xref linkend="nse-vscan"/>.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
<indexterm><primary>vulnerability detection</primary></indexterm>
Vulnerability detection</term>
<listitem>
<para>When a new vulnerability is discovered, you often want
to scan your networks quickly to identify vulnerable systems
before the bad guys do. While Nmap isn't a
comprehensive <ulink role="hidepdf" url="http://sectools.org/vuln-scanners.html">vulnerability scanner</ulink>,
NSE is powerful enough to handle even demanding vulnerability
checks. Many vulnerability detection scripts have already been
written and we plan to distribute more as they are written.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>Backdoor detection</term>
<listitem>
<para>
Many attackers and some automated worms leave
backdoors to enable later reentry. Some of these can be
detected by Nmap's regular expression based version detection.
For example, within hours of the MyDoom worm hitting the
Internet,
Jay Moran<indexterm><primary>Moran, Jay</primary></indexterm>
posted an Nmap version detection probe and
signature so that others could quickly scan their networks.
For more complex worms and backdoors, NSE is needed
for reliable detection.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>Vulnerability exploitation</term>
<listitem>
<para>
As a general scripting language, NSE can even
be used to exploit vulnerabilities rather than just find them.
The capability to add custom exploit scripts may be valuable
for some people (particularly
penetration testers),<indexterm><primary>penetration testing</primary></indexterm>
though we aren't
planning to turn Nmap into an exploitation framework like
<ulink url="http://www.metasploit.com">Metasploit</ulink>.<indexterm><primary><application>Metasploit</application></primary></indexterm>
</para>
</listitem>
</varlistentry>
</variablelist>
<para>
The listed items were our initial goals, but we expect that Nmap
users will come up with inventive uses for NSE.
</para>
<print><note><para>The Nmap Scripting Engine is a new Nmap feature
which already works well, but is under active development. To
provide the latest NSE news and updates, this chapter has been
updated and posted for free online at
<ulink url="http://nmap.org/book/nse.html"/>.</para></note></print>
<para>
Scripts are written in the
embedded
<ulink url="http://www.lua.org/">Lua programming language</ulink>.<indexterm><primary>Lua programming language</primary><seealso>Nmap Scripting Engine</seealso></indexterm>
The language itself is well documented in the books
<web>
<citetitle><ulink url="http://www.amazon.com/exec/obidos/ASIN/8590379825/secbks-20">Programming
in Lua, Second Edition</ulink></citetitle> and
<citetitle><ulink url="http://www.amazon.com/exec/obidos/ASIN/8590379825/secbks-20">Lua
5.1 Reference Manual</ulink></citetitle>.
</web>
<print>
<citetitle>Programming in Lua, Second Edition</citetitle> and
<citetitle>Lua 5.1 Reference Manual</citetitle>.
</print>
The reference manual is also
<ulink url="http://www.lua.org/manual/5.1/">freely available
online</ulink>, as is the
<ulink url="http://www.lua.org/pil/">first edition of <citetitle>Programming in
Lua</citetitle></ulink>. Given the availability of these excellent general
Lua programming references, this document only covers aspects and
extensions specific to Nmap's scripting engine.
</para>
<para>
NSE is activated with the <option>-sC</option> option (or
<option>--script</option> if you wish to specify a custom set of
scripts) and results are integrated into Nmap
normal<indexterm><primary>normal output</primary></indexterm>
and XML output.<indexterm><primary>XML output</primary></indexterm>
Two types of scripts are supported: service and host
scripts. Service scripts relate to a certain open port
(service) on the target host, and any results they produce are included
next to that port in the Nmap output port table. Host scripts,
on the other hand, run no more than once against each target IP
and produce results below the port table. <xref
linkend="nse-ex1" xrefstyle="select: label nopage"/> shows a typical script scan. Examples of
service scripts producing output are: <literal>Stealth SSH
version</literal>, which tricks some SSH servers into divulging
version information without logging the attempt as they normally
would; <literal>Service Owner</literal>, which connects to open
ports, then performs a reverse-identd query to determine what
username each is running under; and <literal>HTML Title</literal>,
which simply grabs the title of the root path of any web servers
found. A sample host script is <literal>RIPE Query</literal>,
which looks up and reports target IP ownership
information.<indexterm><primary>script names, examples of</primary></indexterm>
</para>
<example id="nse-ex1">
<title>Typical NSE output</title>
<indexterm><primary><option>-sC</option></primary><secondary>example of</secondary></indexterm>
<screen>
$ nmap -sC localhost -p 22,23,80,113
Starting Nmap ( http://nmap.org )
Interesting ports on localhost (127.0.0.1):
PORT STATE SERVICE
22/tcp open ssh
|_ Stealth SSH version: SSH-1.99-OpenSSH_4.2
|_ SSH protocol version 1: Server supports SSHv1
23/tcp closed telnet
80/tcp open http
|_ HTML title:Test Page for Apache Installation
113/tcp closed auth
Host script results:
|_ RIPE Query: IP belongs to: Internet Assigned Numbers Authority
Nmap done: 1 IP address (1 host up) scanned in 0.91 seconds
</screen>
</example>
</sect1>
<sect1 id="nse-usage">
<title>Usage and Examples</title>
<para>
While NSE has a complex implementation for efficiency, it is
strikingly easy to use. Simply specify
<option>-sC</option><indexterm><primary><option>-sC</option></primary></indexterm>
to enable the most common scripts. Or specify the
<option>--script</option><indexterm><primary><option>--script</option></primary></indexterm>
option to choose your own scripts to
execute by providing categories, script file names, or the name of
directories full of scripts you wish to execute. You can customize
some scripts by providing arguments to them via the
<option>--script-args</option><indexterm><primary><option>--script-args</option></primary></indexterm>
option. The two remaining options,
<option>--script-trace</option><indexterm><primary><option>--script-trace</option></primary></indexterm>
and <option>--script-updatedb</option>,<indexterm><primary><option>--script-updatedb</option></primary></indexterm>
are generally only used for script debugging and development.
</para>
<sect2 id="nse-categories"><title>Script Categories</title>
<indexterm><primary>script categories</primary></indexterm>
<para>NSE scripts define a list of categories they belong to.
Currently defined categories are <literal>safe</literal>,
<literal>intrusive</literal>, <literal>malware</literal>,
<literal>version</literal>, <literal>discovery</literal>,
<literal>vuln</literal>, <literal>auth</literal>,
<literal>external</literal> and <literal>default</literal>.
Category names are not case
sensitive. The following list describes each category.</para>
<variablelist>
<varlistentry>
<term>
<indexterm><primary sortas="safe script category">&ldquo;<literal>safe</literal>&rdquo; script category</primary></indexterm>
<option>safe</option>
</term>
<listitem>
<para>Scripts
which weren't designed to crash services, use large
amounts of network bandwidth or other resources, or
exploit security holes. These are less likely to offend
remote sysadmins. Of course (as with all other Nmap
features) we cannot guarantee that they won't ever cause
adverse reactions. Most of these perform general
network discovery. Examples are SSH-hostkey (gets an SSH
host key) and showHTMLTitle (grabs the
title from a web page).</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
<indexterm><primary sortas="intrusive script category">&ldquo;<literal>intrusive</literal>&rdquo; script category</primary></indexterm>
<option>intrusive</option>
</term>
<listitem>
<para>These are scripts that cannot be classified in the
<literal>safe</literal> category because the risks are too high that they
will crash the target system, use up significant resources
on the target host (such as bandwidth or CPU time), or
otherwise be perceived as malicious by the target's
system administrators.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
<indexterm><primary sortas="malware script category">&ldquo;<literal>malware</literal>&rdquo; script category</primary></indexterm>
<option>malware</option>
</term>
<listitem>
<para>These scripts test whether the target platform is
infected by malware or backdoors.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
<indexterm><primary sortas="version script category">&ldquo;<literal>version</literal>&rdquo; script category</primary></indexterm>
<indexterm><primary>version detection</primary><seealso>&ldquo;<literal>version</literal>&rdquo; script category</seealso></indexterm>
<option>version</option>
</term>
<listitem>
<para>The scripts in this category are an extension to the
version detection option and cannot be selected
explicitly. They are selected to run only if version
detection (<option>-sV</option>) was requested. Their
output cannot be distinguished from version detection
output and they do not produce service or host script
results.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
<indexterm><primary sortas="discovery script category">&ldquo;<literal>discovery</literal>&rdquo; script category</primary></indexterm>
<option>discovery</option>
</term>
<listitem>
<para>These scripts try to actively learn more about the
network by querying public registries, SNMP-enabled
devices, directory services, and the like.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
<indexterm><primary sortas="vuln script category">&ldquo;<literal>vuln</literal>&rdquo; script category</primary></indexterm>
<option>vuln</option>
</term>
<listitem>
<para>These scripts check for specific known vulnerabilities and
generally only report results if they are found.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
<indexterm><primary sortas="auth script category">&ldquo;<literal>auth</literal>&rdquo; script category</primary></indexterm>
<option>auth</option>
</term>
<listitem>
<para>These scripts try to determine authentication credentials
on the target system, often through a brute-force attack.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
<indexterm><primary sortas="external script category">&ldquo;<literal>external</literal>&rdquo; script category</primary></indexterm>
<option>external</option>
</term>
<listitem>
<para>Scripts in this category may send data to a
third-party database or other network resource. An example
of this is <filename>whois.nse</filename>, which makes a
connection to a
whois<indexterm><primary>whois</primary></indexterm> server
to learn about the address of the target. There is always
the possibility that the operators of the third-party
database will record anything you send to them, which in
many cases will include your IP address and the address of
the target. Most scripts involve traffic strictly between
the scanning computer and the client; any that do not are
placed in this category.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
<indexterm><primary sortas="default script category">&ldquo;<literal>default</literal>&rdquo; script category</primary></indexterm>
<option>default</option>
</term>
<listitem>
<para>These scripts are the default set and are run when
using <option>-sC</option> or <option>-A</option> without
listing scripts with <option>--script</option>. This
category can also be specified explicitly like any other
using <option>--script=default</option>.</para>
</listitem>
</varlistentry>
</variablelist>
</sect2>
<sect2 id="nse-cmd-line-args">
<title>Command-line Arguments</title>
<para>
These are the five command line arguments specific to script-scanning:
</para>
<variablelist>
<varlistentry>
<term>
<indexterm><primary><option>-sC</option></primary></indexterm>
<option>-sC</option>
</term>
<listitem>
<para>Performs a script scan using the default set of scripts. It is
equivalent to <option>--script=default</option>. Some of the
scripts in this category are considered intrusive and should
not be run against a target network without permission. </para>
</listitem>
</varlistentry>
<varlistentry>
<term>
<indexterm><primary><option>--script</option></primary></indexterm>
<option>--script <replaceable>script-categories</replaceable>|<replaceable>directory</replaceable>|<replaceable>filename</replaceable>|all</option></term>
<listitem>
<para>Runs a script scan (like <option>-sC</option>) using the comma-separated list of
script categories, individual scripts, or directories containing
scripts, rather than the default set. Nmap first tries to interpret the
arguments as categories, then (if that fails) as files or
directories. A script or directory of scripts may be specified as an
absolute or relative path. Absolute paths are used as
supplied. Relative paths are searched for in the following places
until found:<indexterm><primary>data files</primary><secondary>directory search order</secondary></indexterm><indexterm><primary>scripts, location of</primary></indexterm>
<filename>--datadir/</filename>;
<filename>$NMAPDIR/</filename>;<indexterm><primary><envar>NMAPDIR</envar> environment variable</primary></indexterm>
<filename>~/.nmap/</filename> (not searched on Windows);<indexterm><primary sortas="nmap directory"><filename>.nmap</filename> directory</primary></indexterm>
NMAPDATADIR/ or<indexterm><primary>NMAPDATADIR</primary></indexterm>
<filename>./</filename>. A <filename>scripts/</filename> subdirectory
is also tried in each of these.</para>
<para>If a directory is specified and found, Nmap loads all NSE
scripts (any filenames ending with <literal>.nse</literal>) from that
directory. Filenames without the <literal>nse</literal> extension are
ignored. Nmap does not search recursively into subdirectories to find
scripts. If individual file names are specified, the file extension
does not have to be <literal>nse</literal>.</para>
<para>Nmap scripts are stored in a <filename>scripts</filename>
subdirectory of the Nmap data directory by default (see
<xref linkend="data-files"/>). For efficiency, scripts are indexed in
a database stored
in <filename>scripts/script.db</filename>.<indexterm><primary><filename>script.db</filename></primary></indexterm>
which lists the category or categories in which each script belongs.
Give the argument <literal>all</literal> to execute all scripts in the
Nmap script database.</para>
<para>Malicious scripts are not run in a sandbox and thus could damage your system or invade your privacy. Never run scripts from third parties unless you trust the authors or have carefully audited the scripts yourself.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
<indexterm><primary><option>--script-args</option></primary></indexterm>
<option>--script-args</option>
</term>
<listitem>
<para>provides arguments to the scripts. See <xref
linkend="nse-args"/> for a detailed explanation.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
<indexterm><primary><option>--script-trace</option></primary></indexterm>
<option>--script-trace</option>
</term>
<listitem>
<para>
This option is similar to
<option>--packet-trace</option>, but works at the
application level rather than packet by packet. If this
option is specified, all incoming and outgoing
communication performed by scripts is printed. The
displayed information includes the communication
protocol, source and target addresses, and the
transmitted data. If more than 5% of transmitted data is
unprintable, hex dumps are given instead.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>
<indexterm><primary><option>--script-updatedb</option></primary></indexterm>
<option>--script-updatedb</option>
</term>
<listitem>
<para>This option updates the script database found
in <filename>scripts/script.db</filename> which is used by
Nmap to determine the available default scripts and
categories. It is only necessary to update the database if
you have added or removed NSE scripts from the
default <filename>scripts</filename> directory or if you
have changed the categories of any script. This option is
generally used by
itself: <command>nmap --script-updatedb</command>.</para>
</listitem>
</varlistentry>
</variablelist>
<para>
Some of the Nmap options have effects on script scans. The most
prominent of these is
<option>-sV</option>.<indexterm><primary><option>-sV</option></primary></indexterm>
A version scan executes
the scripts in the
<literal>version</literal> category.<indexterm><primary sortas="version script category">&ldquo;<literal>version</literal>&rdquo; script category</primary></indexterm>
The scripts
in this category are slightly different than other scripts. Their
output blends in with the version scan and they do not produce any
script scan output.
</para>
<para>
Another option which has effect on the scripting engine is
<option>-A</option>.<indexterm><primary><option>-A</option></primary><secondary>features enabled by</secondary></indexterm>
The advanced/aggressive mode of Nmap implies
the option <option>-sC</option>.
</para>
<para>
</para>
</sect2>
<sect2 id="nse-args">
<title>Arguments to Scripts</title>
<indexterm><primary>script arguments</primary></indexterm>
<para>
You can pass arguments to NSE scripts via the
<option>--script-args</option> option. The script-arguments generally are
name-value pairs, which are provided to the script as a Lua table called
<literal>args</literal> inside the <literal><link
linkend="nse-api-registry">nmap.registry</link></literal> with
the names as keys for the corresponding values. The values can either be
strings or tables. Subtables can be used to pass arguments to
scripts with a finer granularity (e.g. pass different usernames for
different scripts). A typical nmap invocation with script arguments may
look like:
</para>
<para>
<indexterm><primary><option>-sC</option></primary><secondary>example of</secondary></indexterm>
<indexterm><primary><option>--script-args</option></primary><secondary>example of</secondary></indexterm>
<userinput>
$ nmap -sC --script-args user=foo,pass=bar,anonFTP={pass=ftp@foobar.com}
</userinput>
</para>
<para>
which would result in the Lua table:
</para>
<programlisting>
{user="foo",pass="bar",anonFTP={pass="nobody@foobar.com"}}
</programlisting>
<para>You could therefore access the username (<literal>"foo"</literal>)
inside your script as
<literal>local username= nmap.registry.args.user</literal>.
As a general rule the subtables used to override
options for scripts should be named as the script's
<literal>id</literal>, otherwise scripts won't know where to
retrieve their arguments.
</para>
</sect2>
<sect2 id="nse-usage-examples">
<title>Usage Examples</title>
<para>
A simple script scan using the default set of scripts
</para>
<para>
<indexterm><primary><option>-sC</option></primary><secondary>example of</secondary></indexterm>
<userinput>
$ nmap -sC example.com
</userinput>
</para>
<para>
Tracing a specific script.
</para>
<para>
<indexterm><primary><option>--script</option></primary><secondary>example of</secondary></indexterm>
<indexterm><primary><option>--script-trace</option></primary><secondary>example of</secondary></indexterm>
<userinput>
$ nmap --script=./showSSHVersion.nse --script-trace example.com
</userinput>
</para>
<para>
All scripts in a subdirectory named <filename>mycustomscripts</filename> in addition to all of Nmap's included scripts which are in the <literal>safe</literal> category.
</para>
<para>
<userinput>
$ nmap --script=mycustomscripts,safe example.com
</userinput>
</para>
</sect2>
</sect1>
<sect1 id="nse-script-format">
<title>Script Format</title>
<para>NSE scripts consist of six descriptive fields along with either a port or host rule defining when the script should be executed and an action block containing the actual script instructions. Values can be assigned to these fields just as you would assign any other Lua variables. Their names must be lowercase as shown here.</para>
<sect2 id="nse-format-id">
<title><literal>id</literal> Field</title>
<indexterm><primary sortas="id script variable">&ldquo;<varname>id</varname>&rdquo; script variable</primary></indexterm>
<para>
The script's <literal>id</literal> field is displayed in the Nmap output
table if the script produces any output. It should be unique so users
can identify exactly which script file produced a message. IDs
should be kept short to conserve space in Nmap output, while
still being meaningful enough for users to recognize. Some
good examples are <literal>RIPE query</literal>, <literal>HTML
title</literal>, and <literal>Kibuv worm</literal>.<indexterm><primary>script names, examples of</primary></indexterm>
</para>
</sect2>
<sect2 id="nse-format-description">
<title><literal>description</literal> Field</title>
<indexterm><primary sortas="description script variable">&ldquo;<varname>description</varname>&rdquo; script variable</primary></indexterm>
<para>The <literal>description</literal> field describes what the script is testing
for and any critical notes the user must be aware of. A good
example is this description from a user-contributed recursive
DNS script: <quote>Checks whether a nameserver on UDP port 53
allows queries for third party names. It is expected that
recursion will be enabled on your own internal
nameserver.</quote></para>
</sect2>
<sect2 id="nse-format-categories">
<title><literal>categories</literal> Field</title>
<indexterm><primary sortas="categories script variable">&ldquo;<varname>categories</varname>&rdquo; script variable</primary></indexterm>
<para>The <literal>categories</literal> field defines one or
more categories to which a script belongs (see
<xref linkend="nse-categories"/>). The categories are case-insensitive and may be specified in any order. They are listed in an array-like Lua table as in this example:</para>
<programlisting>
categories = {"default", "discovery", "safe"}
</programlisting>
</sect2>
<sect2 id="nse-format-author">
<title><literal>author</literal> Field </title>
<indexterm><primary sortas="author script variable">&ldquo;<varname>author</varname>&rdquo; script variable</primary></indexterm>
<para>
The <literal>author</literal> field contains the script authors name and contact information. If you are worried about spam, you might want to omit or obscure your email address, or give your home page URL instead. This optional field is not used by NSE, but is important for giving script authors due credit or blame.
</para>
</sect2>
<sect2 id="nse-format-license">
<title><literal>license</literal> Field </title>
<indexterm><primary sortas="license script variable">&ldquo;<varname>license</varname>&rdquo; script variable</primary></indexterm>
<indexterm><primary>copyright</primary><secondary>of scripts</secondary></indexterm>
<para>Nmap is a community project and we welcome all sorts of
code contributions, including NSE scripts. So if you write a
valuable script, don't keep it to yourself!
The <literal>license</literal> field helps ensure that we have
legal permission to distribute all the scripts which come with Nmap. All of those scripts
currently use the standard Nmap license
(described in <xref linkend="nmap-copyright"/>). They include
the following line:</para>
<programlisting>
license = "Same as Nmap--See http://nmap.org/book/man-legal.html"
</programlisting>
<para>The Nmap license is similar to the GNU GPL. Script authors may
use a BSD-style license (no advertising clause) instead if they prefer
that.</para>
</sect2>
<sect2 id="nse-format-runlevel">
<title><literal>runlevel</literal> Field</title>
<indexterm><primary sortas="runlevel script variable">&ldquo;<varname>runlevel</varname>&rdquo; script variable</primary></indexterm>
<indexterm><primary>run level of scripts</primary></indexterm>
<para>
This optional field determines script execution order. When
this section is absent the run level defaults to 1.0. A script
with the run level 1.0 is run before any scripts with <literal>runlevel</literal> set to
<literal>2.5</literal>, which in turn runs before any scripts
with <literal>runlevel 2.55</literal>. Scripts with the same run level are run
concurrently. One
application of run levels is allowing scripts to depend on
each other. If <literal>script A</literal> relies on some
information gathered by <literal>script B</literal>, give
<literal>B</literal> a lower run level than
<literal>A</literal>. <literal>Script B</literal> can store
information in the NSE registry for <literal>A</literal> to
retrieve later. For information on the NSE registry see to
<xref linkend="nse-api-registry"/>.
</para>
</sect2>
<sect2 id="nse-format-rules">
<title>Port and Host Rules</title>
<indexterm><primary sortas="portrule script variable">&ldquo;<varname>portrule</varname>&rdquo; script variable</primary></indexterm>
<indexterm><primary sortas="hostrule script variable">&ldquo;<varname>hostrule</varname>&rdquo; script variable</primary></indexterm>
<indexterm><primary>rules in NSE</primary><see>&ldquo;<varname>portrule</varname>&rdquo; and &ldquo;<varname>hostrule</varname>&rdquo;</see></indexterm>
<para>
Nmap uses the script rules to determine whether a script should be run
against a target. A script contains either a <emphasis>port
rule</emphasis>, which governs which ports of a target the scripts may
run against, or a <emphasis>host rule</emphasis>, which specifies that
the script should be run only once against a target IP and only if
certain conditions are met. A rule is a Lua function that returns
either <literal>true</literal> or <literal>false</literal>. The
script <emphasis>action</emphasis> is only performed if the rule
evaluates to <literal>true</literal>. The host rule accepts a host
table as an argument and may test, for example, the IP address or
hostname of the target. A port rule accepts both host and port tables
as arguments for any TCP or UDP port in either the
<literal>open</literal><indexterm><primary><literal>open</literal> port state</primary></indexterm>,
<literal>open|filtered</literal><indexterm><primary><literal>open|filtered</literal> port state</primary></indexterm>,
or <literal>unfiltered</literal><indexterm><primary><literal>unfiltered</literal> port state</primary></indexterm> port states. Port rules generally test factors such as the port number, port state, or listening service name in deciding whether to run against a port. Example rules are shown in <xref linkend="nse-tutorial-rule"/>.</para>
</sect2>
<sect2 id="nse-format-action"><title>Action</title>
<indexterm><primary sortas="action script variable">&ldquo;<varname>action</varname>&rdquo; script variable</primary></indexterm>
<para>
The action is the heart of an NSE script. It contains all of the
instructions to be executed when the script's port or host rule
triggers. It is a Lua function which accepts the same arguments as the
rule and can return either <literal>nil</literal> or a string. If a string is returned by a service script, the string and script ID are printed in the Nmap port table output. A string returned by a host script is printed below the port table. No output is produced if the
script returns <literal>nil</literal>. For an example of an NSE
action refer to <xref linkend="nse-tutorial-action"/>.
</para>
</sect2>
</sect1>
<sect1 id="nse-language">
<title>Script Language</title>
<indexterm><primary>Nmap Scripting Engine (NSE)</primary><secondary>parts of</secondary></indexterm>
<para>
The core of the Nmap Scripting Engine is an embeddable Lua
interpreter. Lua is a lightweight language designed for
extensibility. It offers a powerful and well documented API for
interfacing with other software such as Nmap.
</para>
<indexterm><primary>Nmap Scripting Engine (NSE)</primary><secondary>library</secondary></indexterm>
<para>
The second part of the Nmap Scripting Engine is the NSE Library, which
connects Lua and Nmap. This layer
handles issues such as initialization of the Lua interpreter,
scheduling of parallel script execution, script retrieval and
more. It is also the heart of the NSE network I/O framework and the
exception handling mechanism. It also includes utility libraries to make scripts more powerful and convenient. The utility library modules and extensions are described in <xref linkend="nse-library"/>.</para>
<sect2 id="nse-lua">
<title>Lua Base Language</title>
<indexterm><primary>Lua programming language</primary></indexterm>
<para>
The Nmap scripting language is an embedded <ulink
url="http://www.lua.org/">Lua</ulink> interpreter which was
extended with libraries for interfacing with Nmap. The Nmap
API is in the Lua namespace <literal>nmap</literal>. This
means that all calls to resources provided by Nmap have an
<literal>nmap</literal> prefix.<indexterm><primary><varname>nmap</varname> NSE module</primary></indexterm>
<literal>nmap.new_socket()</literal>, for example, returns a
new socket wrapper object. The Nmap library layer also takes
care of initializing the Lua context, scheduling parallel
scripts and collecting the output produced by completed
scripts.
</para>
<para>
During the planning stages, we considered several programming
languages as the bases for Nmap scripting. One option was to
implement a completely new programming language. The criteria
imposed on the options were strict, NSE needed to be easy to
use, small in size, compatible with the Nmap license,
scalable, fast and parallelizable. There have been several
efforts to design a security auditing language from scratch
which have resulted in well-known awkward solutions. It was
clear from the beginning that we would not go down this
road. For a while the Guile Scheme interpreter was considered
but the preference drifted towards Elk in favor of its more
liberal license. But parallelizing Elk scripts would have been
difficult. In addition, the subset of Nmap users familiar with
functional programming is regarded too small to consider
Scheme as an option. Larger interpreters like Perl, Python or
Ruby are well-known and loved, but are difficult to embed
efficiently. In the end, Lua exceeded in all criteria for
NSE. It is small, distributed under the MIT license, has
coroutines for efficient parallel script
execution, was designed with embeddability in mind, has
excellent documentation, and is actively developed by a large
and committed community.
Lua is even embedded in popular applications including
<application>Wireshark</application> and <application>Second Life</application>.
</para>
</sect2>
</sect1>
<sect1 id="nse-scripts">
<indexterm><primary>Nmap Scripting Engine (NSE)</primary><secondary>list of scripts</secondary></indexterm>
<title>NSE Scripts</title>
<para>
This is a list of the scripts packaged with Nmap as of this writing.
This documentation comes straight from the source code of the
scripts thanks to the NSEDoc documentation system, described in
<xref linkend="nse-documentation"/>.
<print>
Of course no paper documentation can hope to stay current with
software that is developed as actively as NSE is.
</print>
For the latest documentation see the online NSE documentation portal
at <ulink url="http://nmap.org/nsedoc/"/>.
</para>
&nse-scripts;
</sect1>
<sect1 id="nse-library">
<indexterm><primary>Nmap Scripting Engine (NSE)</primary><secondary>list of modules</secondary></indexterm>
<title>NSE Libraries</title>
<para>In addition to the significant built-in capabilities of
Lua, we have written or integrated many extension libraries which make
script writing more powerful and convenient. These libraries (sometimes called modules) are compiled and installed along with
Nmap. They have their own directory, <filename>nselib</filename>, which
is installed in the configured datadir. Scripts need only
<ulink url="http://www.lua.org/manual/5.1/manual.html#pdf-require">
<literal>require</literal>
</ulink> the default libraries in order to use them.
</para>
<para>
This list is just an overview to give an idea of what libraries
are available. Developers will want to consult the complete
documentation at <ulink url="http://nmap.org/nsedoc/"/>.
</para>
&nse-modules;
</sect1>
<sect1 id="nse-api">
<title>Nmap API</title>
<indexterm class="startofrange" id="nse-nmap-indexterm"><primary><varname>nmap</varname> NSE module</primary></indexterm>
<indexterm><primary>Nmap Scripting Engine (NSE)</primary><secondary>API</secondary></indexterm>
<para>
NSE scripts have access to several Nmap facilities for writing
flexible and elegant scripts. The API provides target host
details such as port states and version detection results. It
also offers an interface to the Nsock<indexterm><primary>Nsock</primary></indexterm>
library
for efficient network I/O.
</para>
<sect2 id="nse-api-arguments">
<title>Information Passed to a Script</title>
<para>
An effective Nmap scripting engine requires more than just a
Lua interpreter. Users need easy access to the information
Nmap has learned about the target hosts. This data is passed
as arguments to the NSE
<literal>action</literal> method.<indexterm><primary sortas="action script variable">&ldquo;<varname>action</varname>&rdquo; script variable</primary></indexterm>
The arguments, <literal>host</literal> and
<literal>port</literal>, are Lua tables which contain
information on the target against which the script is
executed. If a script matched a hostrule, it gets only the
<literal>host</literal> table, and if it matched a portrule it
gets both <literal>host</literal> and <literal>port</literal>.
The following list describes each variable in these two tables.
</para>
<para>
<variablelist>
<varlistentry>
<term><option>host</option>
</term>
<listitem>
<para>
This table is passed as a parameter to the rule and action
functions. It contains information on the operating system run by
the host (if the <option>-O</option> switch was supplied), the
IP address and the host name of the scanned target.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>host.os</option>
</term>
<listitem>
<para>
The <literal>os</literal> entry in the host table is
an array of strings. The strings (as many as eight) are the
names of the operating systems the target is possibly
running. Strings are only entered in this array if the
target machine is a perfect match for one or more OS
database entries. If Nmap was run without the
<option>-O</option> option, then
<literal>host.os</literal> is <literal>nil</literal>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>host.ip</option>
</term>
<listitem>
<para>Contains a string representation of the IP address of the
target host. If the scan was run against a host name and the
reverse DNS query returned more than one IP addresses then the
same IP address is used as the one chosen for the scan.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>host.name</option>
</term>
<listitem>
<para>Contains the reverse DNS entry of the scanned target host
represented as a string. If the host has no reverse DNS entry,
the value of the field is an empty string.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>host.targetname</option>
</term>
<listitem>
<para>Contains the name of the host as specified on the command line.
If the target given on the command line contains a netmask or is an IP
address the value of the field is <literal>nil</literal>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>host.directly_connected</option>
</term>
<listitem>
<para> A Boolean value indicating whether or not the target host is
directly connected (i.e. on the same network segment).
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>host.mac_addr</option>
</term>
<listitem>
<para>MAC address<indexterm><primary>MAC address</primary></indexterm>
of the destination host (6-byte long binary
string) or <literal>nil</literal>, if the host is not directly connected.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>host.mac_addr_src</option>
</term>
<listitem>
<para>Our own MAC address, which was used to connect to the
host (either our network card's, or (with
<option>--spoof-mac</option>)<indexterm><primary><option>--spoof-mac</option></primary></indexterm>
the spoofed address).
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>host.interface</option>
</term>
<listitem>
<para>A string containing the interface name
(dnet-style)<indexterm><primary>libdnet</primary></indexterm>
through
which packets to the host are sent.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>host.bin_ip</option>
</term>
<listitem>
<para>The target host's IPv4 address as a 32-bit binary value.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>host.bin_ip_src</option>
</term>
<listitem>
<para>Our host's (running Nmap) source IPv4 address as a 32-bit binary value.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>port</option>
</term>
<listitem>
<para>
The port table is passed to the Lua script in the same
fashion as the host table. It contains information about the port
against which the script is running. If the script is run
according to a host rule, then no port table is passed to the
script. Port states on the target can still be requested from Nmap
using the <literal>nmap.get_port_state()</literal> call.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>port.number</option>
</term>
<listitem>
<para>
Contains the number of the currently scanned port.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>port.protocol</option>
</term>
<listitem>
<para>
Defines the protocol of the port. Valid values are
<literal>"tcp"</literal> and <literal>"udp"</literal>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>port.service</option>
</term>
<listitem>
<para>
Contains a string representation of the service running on
<literal>port.number</literal> as detected by the Nmap service
detection. If the <literal>port.version</literal> field is
<literal>nil</literal> then Nmap has guessed the service based
only on the port number. Otherwise this field is equal to
<literal>port.version.name</literal>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>port.version</option>
</term>
<listitem>
<para>
This entry is a table which contains information
retrieved by the Nmap version scanning engine. Some
of the values (like service name, service type
confidence, RPC related values) may be retrieved by
Nmap even if a version scan was not required. Values
which were not retrieved default to
<literal>nil</literal>. The meaning of each value is given in the following table:</para>
<table id="scripting-tbl-port-version-values">
<title><literal>port.version</literal> values</title>
<tgroup cols="2">
<colspec colwidth="2*" />
<colspec colwidth="5*" />
<thead><row>
<entry>Name</entry>
<entry>Description</entry>
</row></thead>
<tbody>
<row>
<entry><literal>name</literal></entry>
<entry>Contains the service name Nmap will use for the port.</entry>
</row>
<row>
<entry><literal>name_confidence</literal></entry>
<entry>Evaluates how confident the version detection is about the accuracy of <literal>name</literal>, from 1 (least confident) to 10.</entry>
</row>
<row>
<entry><literal>product</literal>, <literal>version</literal>, <literal>extrainfo</literal>, <literal>hostname</literal>, <literal>ostype</literal>, <literal>devicetype</literal></entry>
<entry>These five variables are described in <xref linkend="vscan-versioninfo"/>.
</entry>
</row>
<row>
<entry><literal>service_tunnel</literal></entry>
<entry>Contains the string <literal>"none"</literal> or <literal>"ssl"</literal> based on whether or not Nmap used SSL tunneling to detect the service.</entry>
</row>
<row>
<entry><literal>service_fp</literal></entry>
<entry>The service fingerprint, if any, is provided in this value. This is described in
<xref linkend="vscan-community"/>.
</entry>
</row>
<row>
<entry><literal>rpc_status</literal></entry>
<entry>Contains a string value of <literal>good_prog</literal> if
we were able to determine the program number of an RPC service
listening on the port, <literal>unknown</literal> if the port
appears to be RPC but we couldn't determine the program
number, <literal>not_rpc</literal> if the port doesn't appear be
RPC, or <literal>untested</literal> if we haven't checked for RPC
status.</entry>
</row>
<row>
<entry><literal>rpc_program</literal>, <literal>rpc_lowver</literal>, <literal>rpc_highver</literal></entry>
<entry>The detected RPC program number and the range of version
numbers supported by that program. These will be
<literal>nil</literal> if <literal>rpc_status</literal> is
anything other than <literal>good_prog</literal>.</entry>
</row>
</tbody></tgroup></table>
</listitem>
</varlistentry>
<varlistentry>
<term><option>port.state</option>
</term>
<listitem>
<para>
Contains information on the state of the port.
Service scripts are only run against ports in the
<literal>open</literal> or
<literal>open|filtered</literal> states, so
<literal>port.state</literal> generally contains one
of those values. Other values might appear if the port
table is a result of the
<literal>get_port_state</literal> function. You can
adjust the port state using the
<literal>nmap.set_port_state()</literal> call. This is
normally done when an <literal>open|filtered</literal>
port is determined to be <literal>open</literal>.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</sect2>
<sect2 id="nse-api-networkio">
<title>Network I/O API</title>
<para>
To allow for efficient and parallelizable network I/O, NSE
provides an interface to Nsock, the Nmap socket library. The
smart callback mechanism Nsock uses is fully transparent to
NSE scripts. The main benefit of NSE's sockets is that they
never block on I/O operations, allowing many scripts to be run in parallel.
The I/O parallelism is fully transparent to authors of NSE scripts.
In NSE you can either program as if you were using a single
non-blocking socket or you can program as if your connection is
blocking. Seemingly blocking I/O calls still return once a
specified timeout has been exceeded. Two flavors of Network I/O are
supported: connect-style and raw packet.
</para>
<sect3 id="nse-api-networkio-connect">
<title>Connect-style network I/O</title>
<indexterm><primary>sockets in NSE</primary></indexterm>
<para>This part of the network API should be suitable for most
classical network uses: Users create a socket, connect it to a
remote address, send and receive data and close the socket again.
Everything up to the Transport layer (which is either TCP, UDP or
SSL) is handled by the library.
</para>
<para>
An NSE socket is created by calling
<function>nmap.new_socket()</function>, which returns a socket object.
The socket object supports the usual <function>connect</function>,
<function>send</function>, <function>receive</function>, and
<function>close</function> methods. Additionally the functions
<function>receive_bytes</function>,
<function>receive_lines</function>, and
<function>receive_buf</function> allow greater control of the
receiving of data.
<xref linkend="nse-api-networkio-connect-example" xrefstyle="select: label nopage"/>
shows the use of connect-style network operations. The
<function>try</function> function is for error handling; see
<xref linkend="nse-exceptions"/>.
</para>
<example id="nse-api-networkio-connect-example">
<title>Connect-style I/O</title>
<programlisting>
require("nmap")
local socket = nmap.new_socket()
socket:set_timeout(1000)
try = nmap.new_try(function() socket:close() end)
try(socket:connect(host.ip, port.number))
try(socket:send("login"))
response = try(socket:receive())
socket:close()
</programlisting>
</example>
</sect3>
<sect3 id="nse-api-networkio-raw">
<title>Raw packet network I/O</title>
<indexterm><primary>raw packets</primary><secondary>in NSE</secondary></indexterm>
<para>For those cases where the connection-oriented approach is too inflexible,
NSE provides script developers with a more powerful option:
raw packet network I/O. The greater flexibility comes, however, at
the cost of a slightly more complex API. Receiving raw packets is
accomplished via a wrapper around
Libpcap<indexterm><primary>libpcap</primary></indexterm>
inside the
Nsock library.<indexterm><primary>Nsock</primary></indexterm></para>
<para>For efficiency, the interface for raw packet capturing
works in three steps. First, a capture device is opened.
Second, listeners are registered with the interface. Third,
packets are received.</para>
<para>A handle for raw socket reads is created from an
ordinary socket object using the
<function>pcap_open()</function> method. This method takes a
callback function, which computes a so-called packet hash from
a packet along with its headers. This hash can return any
binary string, which is later compared to the strings
registered with the <function>pcap_register()</function>
function. Normally the packet hash callback will extract some
portion of the packet, such as its source address.</para>
<para>The pcap reader is instructed to listen for certain
packets using the <function>pcap_register()</function> function.
The function takes a binary string which is compared against
the hash value of every packet received. Those packets whose
hashes match any registered strings will be returned by the
<function>pcap_receive()</function> method. Register the empty
string to receive all packets.</para>
<para>A script then receives packets for which a listener has
been registered by calling the
<function>pcap_receive()</function> method. The method blocks
until a packet is received or a timeout occurs.</para>
<para>The more general the packet hash computing function is
kept, the more scripts may receive the packet and proceed with
their execution. To use the packet capturing inside your
script you have to create a socket with
<function>nmap.new_socket()</function> and later close the socket
with <function>socket_object:close()</function>&mdash;just like
with the connection-based network I/O.</para>
<para>
Receiving raw packets is a great feature, but it is also only half
the job. Now for sending raw packets: To accomplish this NSE has
access to a wrapper around the
<literal>dnet</literal> library.<indexterm><primary>libdnet</primary></indexterm></para>
<para>Unlike raw socket reads, raw packet writes are not
through a standard socket object. Instead, the function
<function>nmap.new_dnet()</function> creates a dnet object
with ethernet sending methods. Open an interface with the
<function>ethernet_open()</function> method. Send raw ethernet
frames with <function>ethernet_send()</function>. Close the
ethernet handle with <function>ethernet_close()</function> when
you're done.</para>
</sect3>
</sect2>
<sect2 id="nse-mutex">
<title>Thread Mutexes</title>
<indexterm><primary>threads in NSE</primary></indexterm>
<indexterm><primary>mutexes in NSE</primary></indexterm>
<para>
Each thread made for a script (e.g. <filename>anonFTP.nse</filename>) will yield to other
scripts whenever it makes a call on network objects (sending/receiving
data). Some scripts need finer control over threads' execution. An
example is the <literal>whois.nse</literal> script which queries
whois<indexterm><primary>whois</primary></indexterm>
servers for each target. Because many concurrent queries often result in
getting one's IP banned for abuse and a query may return additional
information for targets other threads are running against, it is useful
to have other threads pause while one thread is conducting a query.
</para>
<para>
To solve this problem, there is an nmap function,
<literal>mutex</literal>, that provides a
<ulink url="http://en.wikipedia.org/wiki/Mutual_exclusion">mutex</ulink>
usable by scripts. The mutex allows for only one thread to be working
on an object. Threads waiting to work on this object are put in the
waiting queue until they can get a "lock" on the mutex. A solution for
the <literal>whois.nse</literal> problem above is to have each thread
block on a mutex for <xref linkend="nse-format-id">script's ID field
</xref>, thus ensuring only one thread is working so its results can
be shared with other scripts which may not need to run and so queries
to the whois servers are staggered.
</para>
<variablelist>
<varlistentry>
<term>
<option>mutex = nmap.mutex(object)</option>
</term>
<listitem>
<para>
Returns a function that works on a mutex for the object passed.
This object can be any
<ulink role="hidepdf" url="http://www.lua.org/manual/5.1/manual.html#2.2">
Lua data type
</ulink> except <literal>nil</literal>,
<literal>booleans</literal>, and <literal>numbers</literal>.
The returned function allows you to lock, try to lock, and
release the mutex. Its first and only parameter must be one of the following:
</para>
<itemizedlist>
<listitem>
<para>
<literal>"lock"</literal>&mdash;Make a blocking lock on the
mutex. If the mutex is busy
(another thread has a lock on it), then the thread will
yield and wait. The function returns with the mutex
locked.
</para>
</listitem>
<listitem>
<para>
<literal>"trylock"</literal>&mdash;Makes a non-blocking lock
on the mutex. If the mutex is
busy then it immediately returns with a return value of
<literal>false</literal>. Otherwise the mutex locks the
mutex and returns <literal>true</literal>.
</para>
</listitem>
<listitem>
<para>
<literal>"done"</literal>&mdash;Releases the mutex and allows
another thread to lock it.
If the thread does not have a lock on the mutex, an
error will be raised.
</para>
</listitem>
<listitem>
<para>
<literal>"running"</literal>&mdash;Returns the thread locked
on the mutex or <literal>nil</literal> if the mutex is not
locked. This should only be used for debugging as it
interferes with finished threads from being
collected.
</para>
</listitem>
</itemizedlist>
</listitem>
</varlistentry>
</variablelist>
<example id="nse-mutex-handling">
<title>Mutex manipulation</title>
<programlisting>
id = "My Script's Unique ID";
local mutex = nmap.mutex(id);
function action(host, port)
mutex "lock";
-- do stuff
mutex "done";
return script_output;
end
</programlisting>
</example>
</sect2>
<sect2 id="nse-exceptions">
<title>Exception Handling</title>
<indexterm><primary>exceptions in NSE</primary></indexterm>
<para>
NSE provides an exception handling mechanism not present in
the plain Lua language. The exception handling is tailored
specifically for network I/O operations. The mechanism
follows a functional programming paradigm rather than an
object oriented programming paradigm. To create an exception
handler the <literal>nmap.new_try()</literal> API method is
used. This method returns a function, which takes a variable
number of arguments, assumed to be the return values of
another function. If an exception is detected in the return
values (the first return value is false),
then the script execution is aborted and no
output is produced. Optionally you can pass a function to
the <literal>new_try()</literal> method which will be called
if an exception is caught. In this function you can perform
required clean up operations.</para>
<para>
<xref linkend="nse-exception-handling" xrefstyle="select: label nopage"/> shows cleanup
exception handling at work. A new function named
<literal>catch</literal> is defined to simply close the
newly created socket in case of an error. It is then used
to protect connection and communication attempts on that
socket. If no catch function is specified, execution of the
script aborts without further ado&mdash;open sockets
will remain open. If the verbosity level is at least one
or if the scan is performed in debugging mode a description
of the uncaught error condition is printed on standard output.
Note that it is currently not easily possible to group several
statements in one try block. It is also important to remember
that if the socket is not closed it will occupy memory
until the next run of Lua's garbage collector.
</para>
<example id="nse-exception-handling">
<title>Exception handling example</title>
<programlisting>
local result, socket, try, catch
result = ""
socket = nmap.new_socket()
catch = function()
socket:close()
end
try = nmap.new_try(catch)
try(socket:connect(host.ip, port.number))
result = try(socket:receive_lines(1))
try(socket:send(result))
</programlisting>
</example>
<para>
Writing a function which is treated properly by the
try/catch mechanism is straightforward. The function should
return multiple values. The first value should be a Boolean
which is <literal>true</literal> upon successful completion of the function and
<literal>false</literal> otherwise. If the function completed successfully the try
construct consumes the indicator value and returns the
remaining values. If the function failed then the second
returned value must be a string describing the error
condition. Note that if the value is not
<literal>nil</literal> or <literal>false</literal> it is
treated as <literal>true</literal> so you can return your
value in the normal case and return <literal>nil, <replaceable>error description</replaceable></literal>
if an error occurs.
</para>
</sect2>
<sect2 id="nse-api-registry">
<title>The Registry</title>
<indexterm><primary>registry (NSE)</primary></indexterm>
<para>
The registry is a normal Lua table. What is special about it
is that it is visible by all scripts and it retains its state
between script executions. Nmap does not scan every host
specified on the command line at the same time, it puts them
in smaller groups and these groups are scanned in parallel. The
registry is rebuilt for every group, so information stored
there is only deleted after NSE finishes processing the
current target group. This implies of course that the registry
is transient&mdash;it is not stored between Nmap executions. Every
script can read the registry and write to it. If a script is
running after another script, it can read some information in
the registry which was left by the first script. This feature
is particularly powerful in combination with the run level
concept. A script with a higher run level can rely on entries
left behind for it by scripts with lower run levels.
<indexterm><primary>run level of scripts</primary></indexterm>
Remember
however that the registry can be written by all scripts
equally, so choose the keys for your entries wisely. The
registry is stored in <literal>nmap.registry</literal>. The
behavior of the registry allows caching of already calculated
data. The cache can be seen by all scripts until the registry
is rebuilt with the next target group. <!-- If for example you have
compiled a regular expression, you can store the compiled
expression in the registry so that scripts which need the same
pattern do not have to recompile it. -->
</para>
</sect2>
<indexterm class="endofrange" startref="nse-nmap-indexterm"/>
</sect1>
<sect1 id="nse-tutorial">
<title>Script Writing Tutorial</title>
<indexterm class="startofrange" id="nse-tutorial-indexterm"><primary>Nmap Scripting Engine (NSE)</primary><secondary>tutorial</secondary></indexterm>
<para>
Suppose that you are convinced of the power of NSE. How do you
go about writing your own script? Let's say
that you want to extract information from an identification
server.<indexterm><primary>auth service</primary></indexterm>
Nmap used to have this functionality but it was removed
because of inconsistencies in the code base. Fortunately, the
protocol identd uses is pretty simple. Unfortunately, it is too
complicated to be expressible in Nmap's version detection
language. Let's look at how the identification protocol
works. First you connect to the identification server. Next you
send a query of the form <literal><replaceable>port-on-server</replaceable>,
<replaceable>port-on-client</replaceable></literal> terminated with a new line
character. The server should then respond with a string of the
form <literal><replaceable>port-on-server</replaceable>, <replaceable>port-on-client</replaceable>:<replaceable>response-type</replaceable>:<replaceable>address-information</replaceable></literal>. In case of an error the address
information is omitted. This description is sufficient for our
purposes, for more details refer to <ulink role="hidepdf" url="http://www.rfc-editor.org/rfc/rfc1413.txt">RFC 1413</ulink>. The protocol cannot be modeled in Nmap's version
detection language for two reasons. The first is that you need
to know both the local and the remote port of a
connection. Version detection does not provide this data. The
second, more severe obstacle, is that you need two open
connections to the target&mdash;one to the identification server and
one to the port you want to query. Both obstacles are easily
overcome with NSE. </para>
<para>
The anatomy of a script is described in <xref linkend="nse-script-format"/>.
In this section we will show how the described structure is utilized.
</para>
<sect2 id="nse-tutorial-head">
<title>The Head</title>
<para>
The head of the script is essentially its meta information. This
includes the
fields: <literal>id</literal>, <literal>description</literal>, <literal>categories</literal>, <literal>runlevel</literal>, <literal>author</literal>
and <literal>license</literal>. We are not going to change the
run level, or worry about the author and license fields for now.
The <literal>id</literal> of a script should uniquely identify
it. If it is absent, the path to the script will be used as an
id. We recommend to choose an id which concisely identifies the
purpose of the script, since the ID is printed before the
script's results in Nmap output.
</para>
<para>
<indexterm><primary sortas="Service Owner script">&ldquo;<literal>Service Owner</literal>&rdquo; script</primary></indexterm>
<indexterm><primary sortas="id script variable">&ldquo;<varname>id</varname>&rdquo; script variable</primary></indexterm>
<programlisting>
id = "Service owner"
</programlisting>
</para>
<para>
The description field should contain a sentence or two describing what the script does. If anything about the script results might confuse or mislead users, and you can't eliminate the issue by improving the script or results text, it should be documented in the <literal>description</literal> string.
</para>
<para>
<indexterm><primary sortas="description script variable">&ldquo;<varname>description</varname>&rdquo; script variable</primary></indexterm>
<programlisting>
description = [[
Attempts to find the owner of a scanned port.
The script makes a connection to the auth port (113) and queries the owner of
an open port.
]]
</programlisting>
</para>
<para>
The author of a script must decide what categories it belongs
to. This script is
<literal>safe</literal><indexterm><primary><literal>safe</literal>
script category</primary></indexterm> because we are not using
the service for anything it was not intended for. Because this
script is one that should run by default it is also in the
<literal>default</literal><indexterm><primary><literal>default</literal>
script category</primary></indexterm>
category.
</para>
<indexterm><primary sortas="categories script variable">&ldquo;<varname>categories</varname>&rdquo; script variable</primary></indexterm>
<programlisting>
categories = {"default", "safe"}
</programlisting>
</sect2>
<sect2 id="nse-tutorial-rule">
<title>The Rule</title>
<para>
The rule section is a Lua method which decides when the
script's action should be performed and when it should be
skipped. Usually this decision is based on the host and port
information passed to the rule function. In the case of the
identification script it is slightly more complicated than
that. To decide whether to run the identification script on a
given port we need to know if there is an identification
server running on the target machine. Or more formally: the
script should be run only if the currently scanned TCP port is open and
TCP port 113 is also open. For now we will rely on the fact that
identification servers listen on TCP port 113. Unfortunately NSE
only gives us information about the currently scanned port.
To find out if port 113 is open we are going to use the
<literal>nmap.get_port_state()</literal> function. If the identd
port was not scanned, the <literal>get_port_state</literal>
function returns <literal>nil</literal>. So we need to make
sure that the table is not <literal>nil</literal>. We also
check if both ports are in the <literal>open</literal> state.
If this is the case, the action is executed, otherwise we skip
the action.
</para>
<para>
<indexterm><primary sortas="portrule script variable">&ldquo;<varname>portrule</varname>&rdquo; script variable</primary></indexterm>
<programlisting>
portrule = function(host, port)
local auth_port = { number=113, protocol="tcp" }
local identd = nmap.get_port_state(host, auth_port)
if
identd ~= nil
and identd.state == "open"
and port.protocol == "tcp"
and port.state == "open"
then
return true
else
return false
end
end
</programlisting>
</para>
</sect2>
<sect2 id="nse-tutorial-action">
<title>The Mechanism</title>
<para>
At last we implement the actual functionality. The script will
first connect to the port on which we expect to find the
identification server, then it will connect to the port we
want information about. Afterward we construct a query string
and parse the response. If we received a satisfactory
response, we return the retrieved information.
</para>
<para>
First we need to create two socket objects. These objects
represent the sockets we are going to use. By using object methods
like
<literal>open()</literal>,
<literal>close()</literal>,
<literal>send()</literal> or
<literal>receive()</literal> we can operate on the network
socket. To avoid excessive error checking code we use NSE's
exception handling mechanism.<indexterm><primary>exceptions in NSE</primary></indexterm>
We create a function which will
be executed if an error occurs and call this function
<literal>catch</literal>. Using this function we generate
a <literal>try</literal> function. The <literal>try</literal>
function will call the <literal>catch</literal> function
whenever there is an error condition in the tried block.
</para>
<para>
<indexterm><primary sortas="action script variable">&ldquo;<varname>action</varname>&rdquo; script variable</primary></indexterm>
<programlisting>
action = function(host, port)
local owner = ""
local client_ident = nmap.new_socket()
local client_service = nmap.new_socket()
local catch = function()
client_ident:close()
client_service:close()
end
local try = nmap.new_try(catch)
try(client_ident:connect(host.ip, 113))
try(client_service:connect(host.ip, port.number))
local localip, localport, remoteip, remoteport =
try(client_service:get_info())
local request = port.number .. ", " .. localport .. "\n"
try(client_ident:send(request))
owner = try(client_ident:receive_lines(1))
if string.match(owner, "ERROR") then
owner = nil
else
owner = string.match(owner, "USERID : .+ : (.+)\n", 1)
end
try(client_ident:close())
try(client_service:close())
return owner
end
</programlisting>
</para>
<para>Note that because we know that the remote port is stored
in <literal>port.number</literal>, we could have ignored the last two
return values of <literal>client_service:get_info()</literal> like
this:</para>
<programlisting>
local localip, localport = try(client_service:get_info())
</programlisting>
<para>In this example we avoided telling the user if the service responded with an error. Instead we assigned <literal>nil</literal> to the <varname>owner</varname> variable. NSE scripts generally only return messages when they succeed.</para>
</sect2>
<indexterm class="endofrange" startref="nse-tutorial-indexterm"/>
</sect1>
<sect1 id="nse-documentation">
<title>Script Documentation Writing</title>
<indexterm class="startofrange" id="nse-documentation-indexterm"><primary>Nmap Scripting Engine (NSE)</primary><secondary>documentation in</secondary></indexterm>
<indexterm class="startofrange" id="nse-nsedoc-indexterm"><primary>NSEDoc</primary></indexterm>
<para>
Scripts are used by more than just their author, so scripts must
have documentation. NSE modules need documentation so developers can
use them in their scripts. NSE's documentation system, described in
this section, aims to meet both these needs. While reading this
section, you may want to browse NSE's online documentation, which is
generated using this documentation. It is at
<ulink url="http://nmap.org/nsedoc/"/>.
</para>
<para>
NSE uses a customized version of the
<ulink url="http://luadoc.luaforge.net/">LuaDoc</ulink><indexterm><primary>LuaDoc</primary></indexterm>
documentation system called NSEDoc.
The documentation for scripts
and modules is contained in their source code, in the form of
comments with a special form.
<xref linkend="nse-documentation-comment" xrefstyle="select: label nopage"/>
is an NSEDoc comment taken from the
<function>stdnse.print_debug()</function> function.
</para>
<!-- From stdnse.lua. -->
<!-- Be careful to change <code> to &lt;code&gt; when you copy code.
<code> is a DocBook tag so it will disappear within a programlisting! -->
<example id="nse-documentation-comment">
<title>An NSEDoc comment for a function</title>
<programlisting>
--- Prints a formatted debug message if the current verbosity level is greater
-- than or equal to a given level.
--
-- This is a convenience wrapper around
-- &lt;code&gt;nmap.print_debug_unformatted()&lt;/code&gt;. The first optional numeric
-- argument, &lt;code&gt;verbosity&lt;/code&gt;, is used as the verbosity level necessary
-- to print the message (it defaults to 1 if omitted). All remaining arguments
-- are processed with Lua's &lt;code&gt;string.format()&lt;/code&gt; function.
-- @param level Optional verbosity level.
-- @param fmt Format string.
-- @param ... Arguments to format.
</programlisting>
</example>
<para>
Documentation comments start with three dashes:
<literal>---</literal>. The body of the comment is the description
of the following code. The first paragraph of the description should
be a brief summary, with the following paragraphs giving more
detail. Special tags starting with <literal>@</literal> mark off
other parts of the documentation. In the above example you see
<literal>@param</literal>, which is used to describe each parameter
of the function. A complete list of the documentation tags is found
in <xref linkend="nse-documentation-tags"/>.
</para>
<para>
Text enclosed in the HTML-like <literal>&lt;code&gt;</literal> and
<literal>&lt;/code&gt;</literal> tags will be rendered in a
monospace font. This should be used for variable and function names,
as well as multi-line code examples. When a sequence of lines start
with the characters <quote><literal>* </literal></quote>, they will
be rendered as a bulleted list.
</para>
<para>
It is good practice to document every public function and table in a
script or module. Additionally every script and module should have
its own file-level documentation. A documentation comment at the
beginning of a file (one that is not followed by a function or table
definition) applies to the entire file. File-level documentation can
and should be several paragraphs long, with all the high-level
information useful to a developer using a module or a user running a
script.
<xref linkend="nse-documentation-module" xrefstyle="select: label nopage"/>
shows documenatation for the <literal>comm</literal> module (with a
few paragraphs removed to save space).
</para>
<example id="nse-documentation-module">
<title>An NSEDoc comment for a module</title>
<programlisting>
--- Common communication functions for network discovery tasks like
-- banner grabbing and data exchange.
--
-- These functions may be passed a table of options, but it's not required. The
-- keys for the options table are <code>"bytes"</code>, <code>"lines"</code>,
-- <code>"proto"</code>, and <code>"timeout"</code>. <code>"bytes"</code> sets
-- a minimum number of bytes to read. <code>"lines"</code> does the same for
-- lines. <code>"proto"</code> sets the protocol to communicate with,
-- defaulting to <code>"tcp"</code> if not provided. <code>"timeout"</code>
-- sets the socket timeout (see the socket function <code>set_timeout()</code>
-- for details).
-- @author Kris Katterjohn 04/2008
-- @copyright Same as Nmap--See http://nmap.org/book/man-legal.html
</programlisting>
</example>
<para>
There are some special considerations when documenting scripts as
opposed to functions and modules. Some information that might be put
in an <literal>@</literal>-tag in a comment should go in one of the
special script variables instead. (Script variables are described in
<xref linkend="nse-script-format"/>.) Specifically, the script's
description should be in the <varname>description</varname> variable
rather than in a documentation comment, and the information that
would go in <literal>@author</literal> and
<literal>@copyright</literal> should go in the variables
<varname>author</varname> and <varname>license</varname> instead.
NSEDoc knows about these variables and will use them in preference
to fields in the comments. Scripts should also have an
<varname>@output</varname> tag showing sample output.
<xref linkend="nse-documentation-script" xrefstyle="select: label nopage"/>
shows proper form for script-level documentation, using a
combination of documentation comments and NSE variables.
</para>
<!-- From ASN.nse. -->
<example id="nse-documentation-script">
<title>An NSEDoc comment for a script</title>
<programlisting>
id = "AS Numbers"
description = [[
Maps IP addresses to autonomous system (AS) numbers.
The script works by sending DNS TXT queries to a DNS server which in
turn queries a third-party service provided by Team Cymru
(team-cymru.org) using an in-addr.arpa style zone set up especially for
use by Nmap.
]]
---
-- @usage
-- nmap --script ASN.nse [--script-args dns=&lt;DNS server&gt;] &lt;target&gt;
-- @args dns The address of a recursive nameserver to use (optional).
-- @output
-- Host script results:
-- | AS Numbers:
-- | BGP: 64.13.128.0/21 | Country: US
-- | Origin AS: 10565 SVCOLO-AS - Silicon Valley Colocation, Inc.
-- | Peer AS: 3561 6461
-- | BGP: 64.13.128.0/18 | Country: US
-- | Origin AS: 10565 SVCOLO-AS - Silicon Valley Colocation, Inc.
-- |_ Peer AS: 174 2914 6461
author = "jah, Michael"
license = "Same as Nmap--See http://nmap.org/book/man-legal.html"
categories = {"discovery", "external"}
</programlisting>
</example>
<indexterm><primary>NSEDoc</primary><secondary>for C modules</secondary></indexterm>
<para>
Compiled NSE modules are also documented with NSEDoc, even though
they have no Lua source code. Each compiled module has a file
<filename><replaceable>modulename</replaceable>.luadoc</filename><indexterm><primary sortas="luadoc filename extension"><filename>.luadoc</filename> filename extension</primary></indexterm>
that is kept in the <filename>nselib</filename> directory alongside
the Lua modules. This file lists and documents the functions and
tables in the compiled module as though they were written in Lua.
Only the name of each function is required, not its definition (not
even <literal>end</literal>). You must use the
<literal>@name</literal> and <literal>@class</literal> tags when
documenting a table to assist the documentation parser in
identifying it. There are several examples of this method of
documentation in the Nmap source distribution.
</para>
<sect2 id="nse-documentation-tags">
<title>NSE Documentation Tags</title>
<para>
This is a list of tags understood by NSEDoc and their purpose.
</para>
<variablelist>
<varlistentry>
<term><option>@param</option></term>
<listitem>
<para>
Describes a function parameter. The first word following
<literal>@param</literal> is the name of the parameter
being described. The tag should appear once for each
parameter of the function.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>@see</option></term>
<listitem>
<para>
Adds a cross-reference to another function or table.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>@return</option></term>
<listitem>
<para>
Describes a return value of a function.
<literal>@return</literal> may be used multiple times for
multiple return values.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>@usage</option></term>
<listitem>
<para>
Gives an example of the usage of a function or script. In
the case of a function, the example is Lua code; for a
script it is an Nmap command line.
<literal>@usage</literal> may be given more than once.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>@name</option></term>
<listitem>
<para>
Defines a name for the function or table being documented.
This tag is normally not necessary, as NSEDoc infers the
name through code analysis.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>@class</option></term>
<listitem>
<para>
Defines the <quote>class</quote> of the thing being
modified: <literal>function</literal>,
<literal>table</literal>, or <literal>module</literal>.
Like <literal>@name</literal>, this is normally inferred
automatically.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>@field</option></term>
<listitem>
<para>
In the documentation of a table, describes the value of a
named field.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>@args</option></term>
<listitem>
<para>
Describes a script argument, as used with the
<option>--script-args</option> option (see
<xref linkend="nse-args"/>). The first word after
<literal>@args</literal> is the name of the argument, and
everything following that is the description. This tag is
special to script-level comments.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>@output</option></term>
<listitem>
<para>
Shows sample output of a script. This tag is special to
script-level comments.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>@author</option></term>
<listitem>
<para>
Lists an author of a module. It may be given more than
once. Don't use this tag in script documentation; use the
<varname>author</varname> variable instead.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>@copyright</option></term>
<listitem>
<para>
Describes the copyright of a module. Don't use this tag in
script documentation; use the <varname>license</varname>
variable instead.
</para>
</listitem>
</varlistentry>
</variablelist>
<!-- These tags are undocumented here: @description, @summary, and
@release. @documentation and @summary are automatically extracted
from the contents of a comment. @release has not been used with
NSEDoc. -->
</sect2>
<indexterm class="endofrange" startref="nse-documentation-indexterm"/>
<indexterm class="endofrange" startref="nse-nsedoc-indexterm"/>
</sect1>
<sect1 id="nse-vscan">
<title>Version Detection Using NSE</title>
<indexterm class="startofrange" id="nse-sample-indexterm"><primary>Nmap Scripting Engine (NSE)</primary><secondary>sample scripts</secondary></indexterm>
<indexterm><primary>version detection</primary><secondary>using NSE</secondary></indexterm>
<para>
The version detection system built into Nmap was designed to
efficiently recognize the vast majority of protocols with a simple
pattern matching syntax. Some protocols require a more complex
approach though, and a generalized scripting language is perfect for
this.
</para>
<para>
NSE's <literal>version</literal><indexterm><primary><varname>version</varname> script category</primary></indexterm>
category contains the scripts that enhance standard version
detection. Scripts in this category are run whenever you request
version detection with <option>-sV</option>; you don't need to use
<option>-sC</option> to get version-detection scripts. (This cuts
the other way too: if you use <option>-sC</option> you won't get
<literal>version</literal> scripts unless you also use
<option>-sV</option>.)
</para>
<para>
This script detects version 2 of the Skype VoIP protocol, one which
is difficult to identify with version detection alone. If Skype gets
an HTTP GET request, it pretends to be an HTTP server and sends back
a 404. But for any other request it sends back a chunk of
random-looking data. Proper identification requires sending two
probes and comparing the two responses&mdash;an ideal task for NSE.
</para>
<programlisting>
id = "Skype v2"
description = [[
Detects the Skype version 2 service.
]]
author = "Brandon Enright &lt;bmenrigh@ucsd.edu&gt;"
license = "Same as Nmap--See http://nmap.org/book/man-legal.html"
categories = {"version"}
require "comm"
portrule = function(host, port)
if (port.number == 80 or
port.number == 443 or
port.service == nil or
port.service == "" or
port.service == "unknown")
and port.protocol == "tcp"
and port.state == "open"
and port.service ~= "http"
and port.service ~= "ssl/http"
then
return true
else
return false
end
end
action = function(host, port)
local status, result = comm.exchange(host, port,
"GET / HTTP/1.0\r\n\r\n", {bytes=26, proto=port.protocol})
if (not status) then
return
end
if (result ~= "HTTP/1.0 404 Not Found\r\n\r\n") then
return
end
-- So far so good, now see if we get random data for another request
status, result = comm.exchange(host, port,
"random data\r\n\r\n", {bytes=15, proto=port.protocol})
if (not status) then
return
end
if string.match(result, "[^%s!-~].*[^%s!-~].*[^%s!-~]") then
-- Detected
port.version.name = "skype2"
port.version.product = "Skype"
nmap.set_port_version(host, port, "hardmatched")
return
end
return
end
</programlisting>
<para>
If the script detects Skype, it augments its <varname>port</varname>
table with now-known <varname>name</varname> and
<varname>product</varname> fields. It then sends this new
information to Nmap by calling
<function>nmap.set_port_version()</function>. Several other version
fields are available to be set if they are known, but in this case
we only have the name and product. For the full list of version
fields refer to the documentation of
<function>nmap.set_port_version()</function>.
</para>
<para>
Notice that if the script does not detect the protocol, it does
nothing. This is considered good practice; a script shouldn't
produce output (other than debug output) just to say it didn't learn
anything.
</para>
</sect1>
<sect1 id="nse-example-scripts">
<title>Example Script</title>
<sect2 id="nse-example-script-finger">
<title>Finger-Test Script</title>
<indexterm><primary sortas="Finger Results script">&ldquo;<literal>Finger Results</literal>&rdquo; script</primary></indexterm>
<para>The finger script (<filename>finger.nse</filename>) is a perfect
example of how short typical NSE scripts are.
</para>
<para>first the information fields are filled out, note that the
<literal>id</literal> field is kept short, this is important since it is
printed in Nmap's output. A detailed description of what the script
actually does should go in the <literal>description</literal> field.</para>
<programlisting>
id="Finger Results"<indexterm><primary sortas="id script variable">&ldquo;<varname>id</varname>&rdquo; script variable</primary></indexterm>
description="attempts to get a list of usernames via the finger service"<indexterm><primary sortas="description script variable">&ldquo;<varname>description</varname>&rdquo; script variable</primary></indexterm>
author = "Eddie Bell &lt;ejlbell@gmail.com&gt;"<indexterm><primary>Bell, Eddie</primary></indexterm><indexterm><primary sortas="author script variable">&ldquo;<varname>author</varname>&rdquo; script variable</primary></indexterm>
license = "Same as Nmap--See http://nmap.org/book/man-legal.html"<indexterm><primary sortas="license script variable">&ldquo;<varname>license</varname>&rdquo; script variable</primary></indexterm>
</programlisting>
<para>The <literal>categories</literal> field is a table
containing all the categories the script belongs to&mdash;These are used for
script selection through the <option>--script</option> option.</para>
<programlisting>
categories = {"discovery"}
</programlisting>
<para>You can use the facilities provided by the nselib (<xref
linkend="nse-library"/>) with <literal>require</literal>. Here
we want to use shorter port rules.</para>
<programlisting>
require "shortport"
</programlisting>
<para>We want to run the script against the finger service. So we
test whether it is using the well-known finger port (<literal>79/tcp</literal>), or
whether the service is named <quote>finger</quote> based on version
detection results or in the port number's listing
in <filename>nmap-services</filename>.</para>
<para>We want to check whether the service behind the port is finger,
or whether it runs on finger's well-known port 79. Through this we can
use the information gathered during the version scan (if finger runs
on a non-standard port) or still run against at least the port we
expect it, should the version detection information not be available.</para>
<programlisting>
portrule = shortport.port_or_service(79, "finger")<indexterm><primary sortas="portrule script variable">&ldquo;<varname>portrule</varname>&rdquo; script variable</primary></indexterm>
action = function(host, port)<indexterm><primary sortas="action script variable">&ldquo;<varname>action</varname>&rdquo; script variable</primary></indexterm>
local socket = nmap.new_socket()
local results = ""
local status = true
</programlisting>
<para>The function <literal>err_catch()</literal> will be called for
clean up, through NSE's exception handling mechanism. Here it only
closes the previously opened socket (which should be enough in most
cases).</para>
<programlisting>
local err_catch = function()
socket:close()
end
</programlisting>
<para>The clean up function gets registered for exception handling via
a call to <literal>nmap.new_try()</literal></para>
<programlisting>
local try = nmap.new_try(err_catch())
</programlisting>
<para>The script sets a timeout of 5000 (five seconds).
Should any operation require more time we'll receive a
<literal>TIMEOUT</literal> error message.</para>
<programlisting>
socket:set_timeout(5000)
</programlisting>
<para>To make use of the exception handling we need to wrap calls to those functions which might return an error, inside <literal>try()</literal></para>
<programlisting>
try(socket:connect(host.ip, port.number, port.protocol))
try(socket:send("\n\r"))
</programlisting>
<para>The call to <literal>receive_lines()</literal> is not wrapped
in <literal>try()</literal>, because we don't want to abort the script
just because we didn't receive the data we expected. Note that if
there is less data than requested (100 lines), we will still receive
it and the status will be <literal>true</literal>&mdash;subsequent
calls would yield a <literal>false</literal> status.</para>
<programlisting>
status, results = socket:receive_lines(100)
socket:close()
</programlisting>
<para>The script returns a string if the call to <literal>receive_lines()</literal> was successful, otherwise it returns <literal>nil</literal>.</para>
<programlisting>
return results
end
</programlisting>
</sect2>
<indexterm class="endofrange" startref="nse-sample-indexterm"/>
</sect1>
<sect1 id="nse-implementation">
<title>Implementation</title>
<indexterm><primary>Nmap Scripting Engine (NSE)</primary><secondary>implementation</secondary></indexterm>
<para>
Now how does all this work? The following section describes
some interesting aspects of NSE. While the focus primarily lies on
giving script writers a better feeling of what happens with scripts, it
should also provide a starting point for understanding (and extending) the
NSE sources.
</para>
<sect2 id="nse-implementation-init">
<title>Initialization Phase</title>
<para>
During its initialization stage, Nmap loads the Lua interpreter, including its provided libraries. These libraries are documented in the <ulink url="http://www.lua.org/manual/5.1/manual.html">Lua Reference Manual</ulink>. Here is a summary:</para>
<itemizedlist>
<listitem>
<para>The <emphasis>package</emphasis> library (namespace:
<literal>package</literal>)&mdash;Lua's
<ulink role="hidepdf" url="http://www.lua.org/manual/5.1/manual.html#5.3">package-lib</ulink> provides (among others) the <literal>require</literal> function, used to load modules from the
nselib.
</para>
</listitem>
<listitem>
<para>The <emphasis>table</emphasis> library (namespace:
<literal>table</literal>)&mdash;The
<ulink role="hidepdf" url="http://www.lua.org/manual/5.1/manual.html#5.5">table manipulation library</ulink> contains many functions used
to operate on <literal>tables</literal>&mdash;Lua's central data
structure.
</para>
</listitem>
<listitem>
<para>The <emphasis>I/O</emphasis> library (namespace:
<literal>io</literal>)&mdash;The
<ulink role="hidepdf" url="http://www.lua.org/manual/5.1/manual.html#5.7">Input/Output library</ulink> offers functions such as reading files and reading the output from programs you execute.
</para>
</listitem>
<listitem>
<para>The <emphasis>OS</emphasis> library (namespace:
<literal>os</literal>)&mdash;The
<ulink role="hidepdf" url="http://www.lua.org/manual/5.1/manual.html#5.8">Operating System library</ulink> provides facilities of the operating system, including filesystem operations (renaming/removing files, temporary file creation) and access to the environment.</para>
</listitem>
<listitem>
<para>The <emphasis>string</emphasis> library (namespace:
<literal>string</literal>)&mdash;The
<ulink role="hidepdf" url="http://www.lua.org/manual/5.1/manual.html#5.4">
string library </ulink> helps you with functions used to manipulate
strings inside Lua. Functions include: printf-style
string formatting, pattern matching using Lua-style patterns,
substring extraction, etc.
</para>
</listitem>
<listitem>
<para>The <emphasis>math</emphasis> library (namespace:
<literal>math</literal>)&mdash;Numbers in Lua usually correspond to the <literal>double</literal> C type, so the <ulink role="hidepdf" url="http://www.lua.org/manual/5.1/manual.html#5.6">math library</ulink> provides access to rounding functions, trigonometric functions, random number generation, and more.</para>
</listitem>
<listitem>
<para>The <emphasis>debug</emphasis> library (namespace:
<literal>debug</literal>)&mdash;The
<ulink role="hidepdf" url="http://www.lua.org/manual/5.1/manual.html#5.9">debug library</ulink> provides you with a somewhat lower level API
to the Lua interpreter. Through it you can access functions along
the execution stack, get function closures and object metatables,
etc.
</para>
</listitem>
</itemizedlist>
<para>In addition to loading the libraries provided by Lua, the functions in the <literal>nmap</literal> namespace are loaded. The search paths are the same directories that Nmap searches for its data files and scripts, except that the <literal>nselib</literal> directory is appended to each. In this step the provided script arguments are stored inside the registry.<indexterm><primary>registry (NSE)</primary></indexterm></para>
<para>
The next phase of NSE initialization is loading the chosen
scripts, which are the arguments provided to the
<option>--script</option><indexterm><primary><option>--script</option></primary></indexterm>
option or <literal>default</literal>, in
case of a default script scan. The string
<literal>version</literal><indexterm><primary><varname>version</varname> script category</primary></indexterm>
is appended, if version detection was enabled.
The arguments afterwards are tried to be
interpreted as script categories. This is done via a Lua C function
in <filename>nse_init.cc</filename> called <literal>entry</literal>.
Inside <filename>script.db</filename>,<indexterm><primary><filename>script.db</filename></primary><seealso><option>--script-updatedb</option></seealso></indexterm>
for each category of a script,
there is a call to <literal>Entry</literal>. If the category was chosen
then the script is loaded. Every argument of
<option>--script</option> that could not be interpreted as a category
is loaded as a file or directory. If the file or directory could not
be located, then an error is raised and the Script Engine aborts.
</para>
<para>
All of the <literal>.nse</literal> files inside a loaded directory are
loaded as files. Each file loaded is executed by Lua. If a
<emphasis>portrule</emphasis> is present, then it is saved in the
<emphasis>porttests</emphasis> table with a portrule key and file
closure value. Otherwise, if the script has a
<emphasis>hostrule</emphasis>, then it is saved in the <emphasis>hosttests</emphasis> table
in the same manner.
</para>
</sect2>
<sect2 id="nse-implementation-match">
<title>Matching of Scripts to Targets</title>
<para>
After the initialization is finished the
<literal>hostrules</literal><indexterm><primary sortas="hostrule script variable">&ldquo;<varname>hostrule</varname>&rdquo; script variable</primary></indexterm>
and <literal>portrules</literal><indexterm><primary sortas="portrule script variable">&ldquo;<varname>portrule</varname>&rdquo; script variable</primary></indexterm>
are evaluated for each host in the current
target group. At this check a list is built which contains the combinations of scripts and the hosts they will run against.
It should be noted that the rules of all chosen scripts are
checked against all hosts and their
<literal>open</literal><indexterm><primary><literal>open</literal> port state</primary></indexterm>
and <literal>open|filtered</literal><indexterm><primary><literal>open|filtered</literal> port state</primary></indexterm>
ports.
Therefore it is advisable to leave the rules as simple as possible and
to do all the computation inside the <literal>action</literal>, as a script will only be
executed if it is run against a specific target. After the check those script-target combinations
get their own <ulink url="http://www.lua.org/manual/5.1/manual.html#2.11">Lua thread</ulink>. A
thread running against a host will have only a hostrule passed to the action closure whereas
a thread running against a port will have both a hostrule and portrule passed. Each thread
is stored with information relevant to the thread. This information
includes the runlevel, target, target port (if applicable), host and port tables
(passed to action), its type (running against a host or port), and its id.
The mainloop function will work on each runlevel grouping of threads in order.
</para>
</sect2>
<sect2 id="nse-implementation-run">
<title>Running Scripts</title>
<para>
Nmap is able to perform NSE script scanning in
parallel<indexterm><primary>parallelism</primary><secondary>in NSE</secondary></indexterm>
by making use of Lua language features. In particular,
<ulink url="http://www.lua.org/manual/5.1/manual.html#2.11">coroutines
</ulink> offer collaborative multi-threading so scripts can suspend themselves at defined points, and allow other coroutines
to execute. Since network I/O, especially waiting for responses from
remote host, is the part of scripts which would consume most time with
waiting, this is the point where scripts suspend themselves and let
others execute. Each call to some of the functions of the Nsock wrapper
causes the calling script to yield (pause). Once the request is
processed by the Nsock library, the
callback causes the script to be pushed from the waiting queue to the
running queue, which will eventually let it resume its operation.</para>
<para>
The mainloop function will maintain two sets of threads, running and
waiting. Threads will be
moved back and forth between the sets; when a thread yields, it
is moved to the waiting group. Threads run in the running set will either
yield, complete, or error. After all scripts are resumed in the running
set, mainloop will place all yielded threads ready to be
run in the running set. Threads are made "ready" by calling
<literal>process_waiting2running</literal>. This process of running
threads and moving paused threads to the waiting and running sets is
repeated until no threads exist in either waiting or running.
</para>
</sect2>
<sect2 id="nse-implementation-c-modules">
<title>Adding C Modules to Nselib</title>
<indexterm><primary>Nmap Scripting Engine (NSE)</primary><secondary>C modules</secondary></indexterm>
<para>
Some of the modules included in nselib are not written in Lua but
in C or C++. <literal>bit</literal> and <literal>pcre</literal>
are two examples. This section describes how to write your own
compiled extensions to nselib.
</para>
<para>
The C API of Lua is described at length in
<web><ulink url="http://www.amazon.com/exec/obidos/ASIN/8590379825/secbks-20"><citetitle>Programming in Lua, Second Edition</citetitle></ulink>,</web>
<print><citetitle>Programming in Lua, Second Edition</citetitle>,</print>
so this is a short summary. C modules consist of functions that
follow the protocol of the
<ulink url="http://www.lua.org/manual/5.1/manual.html#lua_CFunction"><type>lua_CFunction</type></ulink>
type. The functions are registered with Lua and assembled into a
library by calling the
<function>luaL_register</function><indexterm><primary><function>luaL_register</function></primary></indexterm>
function. A special initialization function provides the interface
between the module and the rest of the NSE code. By convention the
initialization function has a name of the form
<function>luaopen_<replaceable>module</replaceable></function>.
</para>
<para>
The smallest compiled modules that comes with NSE is
<literal>bit</literal>,<indexterm><primary><varname>bit</varname> NSE module</primary></indexterm>
and one of the most straightforward is
<literal>openssl</literal>.<indexterm><primary><varname>openssl</varname> NSE module</primary></indexterm>
These modules serve as good examples for a beginning module
writer. The
source code for <literal>bit</literal> is in the files
<filename>nse_bit.cc</filename> and
<filename>nse_bit.h</filename>, and likewise the source for
<literal>openssl</literal> is in <filename>nse_openssl.cc</filename> and
<filename>nse_openssl.h</filename>. The other compiled modules
usually follow this naming convention.
</para>
<para>
Let us look at the <literal>openssl</literal> module. One of the
functions in <filename>nse_openssl.cc</filename> is
<function>l_md5</function>, which calculates an MD5 digest. Its
function prototype is
<programlisting>
static int l_md5(lua_State *L);
</programlisting>
The prototype shows that <function>l_md5</function> matches the
<type>lua_CFunction</type> type. The function is static because it
does not have to be visible to other compiled code, it just needs
an address so it can be registered with Lua. Later in the file we
see <function>l_md5</function> entered into an array of type
<type>luaL_reg</type> and associated with the name
<function>md5</function>, the name it will be known by to NSE:
<programlisting>
static const struct luaL_reg openssllib[] = {
{ "md5", l_md5 },
{ NULL, NULL }
};
</programlisting>
Then the library is registered with a call to
<function>luaL_register</function> inside the initialization
function <function>luaopen_openssl</function>. Some lines relating
to the registration of OpenSSL <type>BIGNUM</type> types have been
omitted.
<programlisting>
LUALIB_API int luaopen_openssl(lua_State *L) {
luaL_register(L, OPENSSLLIBNAME, openssllib);
return 1;
}
</programlisting>
(<varname>NSE_OPENSSLLIBNAME</varname> is just the string
<literal>"openssl"</literal>, the name of the module.)
<function>luaopen_openssl</function>
is the only function in the file that is exposed in
<filename>nse_openssl.h</filename>.
</para>
<para>
Once a compiled module is written, it is added to NSE by including
it in the list of standard libraries in
<filename>nse_init.cc</filename>. Just follow the example of the
modules that are already there. Then the names of the module's
source files of the must be added to
<filename>Makefile.in</filename> in the appropriate places. Again
it is easiest to follow the example of the other modules. For the
Windows build the new source files must be added to the
<filename>mswin32/nmap.vcproj</filename> project file.
</para>
</sect2>
</sect1>
<indexterm class="endofrange" startref="nse-indexterm"/>
Reference in New Issue View Git Blame Copy Permalink