Nmap Scripting Engine The Nmap Scripting Engine (NSE) is one of Nmap's most powerful and flexible features. It allows users to write (and share) simple scripts to automate a wide variety of networking tasks. Those scripts are then executed in parallel with the speed and efficiency you expect from Nmap. Users can rely on the growing and diverse set of scripts distributed with Nmap, or write their own to meet custom needs. The Nmap project would like to thank Diman Todorov for his excellent work building the initial NSE implementation and writing much of this documentation. Stoiko Ivanov also contributed greatly. The tasks we had in mind when creating the system are: Network discovery This is Nmap's bread and butter. Examples include looking up whois data based on the target domain, querying ARIN, RIPE, or APNIC for the target IP to determine ownership, performing identd lookups on open ports, SNMP queries, and listing available NFS/SMB/RPC shares and services. More sophisticated version detection The Nmap version detection system () is able to recognize thousands of different services through its probe and regular expression based matching system, but it cannot recognize everything. For example, identifying the Skype v2 service requires two independent probes. Nmap could also recognize more SNMP services if it tried a few hundred different community names by brute force. Neither of these tasks are well suited to traditional Nmap version detection, but both are easily accomplished with NSE. For these reasons, version detection now calls NSE by default to handle some tricky services. This is described in . Vulnerability detection When a new vulnerability is discovered, you often want to scan your networks quickly to identify vulnerable systems before the bad guys do. While Nmap isn't a comprehensive vulnerability scanner, we plan to distribute scripts for some very severe or common vulnerabilities and misconfigurations. Backdoor detection Many attackers and some automated worms leave backdoors to enable later reentry. Some of these can be detected by Nmap's regular expression based version detection. For example, within hours of the MyDoom worm hitting the Internet, Jay Moran posted an Nmap version detection probe and signature so that others could quickly scan their networks. For more complex worms and backdoors, NSE is needed instead. Vulnerability exploitation As a general scripting language, NSE could even be used to exploit vulnerabilities rather than just find them. The capability to add custom exploit scripts may be valuable for some people (particularly penetration testers), though we aren't planning to turn Nmap into an exploitation framework like Metasploit. The listed items are just the initial script classes. It is likely that Nmap users will come up with even more inventive uses for NSE. Scripts are written in the embedded Lua programming language. The language itself is well documented in the books Programming in Lua, Second Edition and Lua 5.1 Reference Manual. Programming in Lua, Second Edition and Lua 5.1 Reference Manual. The reference manual is also freely available online, as is the first edition of Programming in Lua. Given the availability of these excellent general Lua programming references, this document only covers aspects and extensions specific to Nmap's scripting engine. NSE is activated with the -sC option (or --script if you wish to specify a custom set of scripts) and results are integrated into Nmap normal and XML output. Two types of scripts are supported: service and host scripts. Service scripts relate to a certain open port (service) on the target host, and any results they produce are included next to that port in the Nmap output port table. Host scripts, on the other hand, run no more than once against each target IP and produce results below the port table. shows a typical script scan. Examples of service scripts producing output are Stealth SSH Version, which tricks some SSH servers into divulging version information without logging the attempt as they normally would, Service Owner, which connects to open ports, then performs a reverse-identd query to determine what username it is running under, and HTML Title, which simply grabs the title of the root path of any web servers found. A sample host script is RIPE Query, which looks up and reports target IP ownership information. $ ./nmap -sC localhost -p 22,23,80,113 Starting Nmap ( http://nmap.org ) Interesting ports on localhost (127.0.0.1): PORT STATE SERVICE 22/tcp open ssh |_ Stealth SSH version: SSH-1.99-OpenSSH_4.2 |_ SSH protocol version 1: Server supports SSHv1 23/tcp closed telnet 80/tcp open http |_ HTML title:Test Page for Apache Installation 113/tcp closed auth Host script results: |_ RIPE Query: IP belongs to: Internet Assigned Numbers Authority Nmap finished: 1 IP address (1 host up) scanned in 0.907 seconds While NSE has a complex implementation for efficiency, it is strikingly easy to use. Simply specify -sC to enable the most common scripts. Or specify the --script option to choose your own scripts to execute by providing categories, script file names, or the name of directories full of scripts you wish to execute. You can customize some scripts by providing arguments to them via the --script-args option. The two remaining options, --script-trace and --script-updatedb, are generally only used for script debugging and development. NSE scripts define a list of categories they belong to. Currently defined categories are safe, intrusive, malware, version, discovery, vulnerability and default. Categories are not case sensitive. The following list describes each category. safe Scripts which weren't designed to crash services, use large amounts of network bandwidth or other resources, or exploit security holes. These are less likely to offend remote sysadmins. Of course (as with all other Nmap features) we cannot guarantee that they won't ever cause adverse reactions. Most of these perform general network discovery. Examples are echoTest (sends a string to the UDP echo service) and showHTMLTitle (grabs the title from a web page). intrusive These are not intended to crash or damage anything, but are more likely to leave suspicious logs or otherwise arouse sysadmin ire. Scripts which attempt to login to services with default passwords fall into this class. malware These scripts test if the target platform is infected by malware or backdoors. version This category cannot be selected explicitly. It is only run if -sV was supplied. The scripts in this category are an extension to the version detection service. Their output cannot be distinguished from version detection output and they do not produce script scanning output. discovery These scripts try to actively learn more about the network by querying public registries, SNMP-enabled devices, directory services, and the like. vulnerability These scripts check for a specific vulnerability and report results only if it is found. default These scripts are the default set and are run when using -sC. This category can also be specified like any other with --script. Don't be fooled into thinking that just because these scripts are run by default that they are all completely unobtrusive: these scripts should not be run against target networks without permission. You can pass arguments to NSE scripts via the --script-args option. The script-arguments generally are name-value pairs, which are provided to the script as a Lua table called args inside the nmap.registry with the names as keys for the corresponding values. The values can either be strings or tables. Subtables can be used to pass arguments to scripts with a finer granularity (e.g. pass different usernames for different scripts). A typical nmap invocation with script arguments may look like: $ nmap -sC --script-args user=foo,pass=bar,anonFTP={pass=ftp@foobar.com} which would result in the Lua table: {user="foo",pass="bar",anonFTP={pass="nobody@foobar.com"}} You could therefore access the username ("foo") inside your script as local username= nmap.registry.args.user . As a general rule the subtables used to override options for scripts should be named as the script's id, since otherwise scripts can't know where to search for their arguments. These are the five command line arguments specific to script-scanning: -sC -sC Performs a script scan using the default set of scripts. It is equivalent to --script=default. Some of the scripts in this category are considered intrusive and should not be run against a target network without permission. --script <script-categories|directory|filename|all>--script Runs a script scan (like -sC) with the comma separated list of scripts you have chosen rather than the defaults. Specifically, the list can contain script categories, single scripts or directories with scripts which are to be run against the target hosts instead of the default set. Nmap will try to interpret the arguments at first as categories and afterwards as files or directories. Absolute paths are used as is, relative paths are searched in the following places until found: --datadir/; $(NMAPDIR)/; ~user/nmap/ (not searched on Windows); NMAPDATADIR/ or ./. A scripts/ subdirectory is also tried in each of these. Give the argument all to execute all scripts in the Nmap script database. If a directory is specified and found, Nmap loads all NSE scripts (any filenames ending with .nse) from that directory. They must have the filename extension nse. Nmap does not recurse into subdirectories to find scripts. When individual file names are specified, the file extension does not have to be nse. Nmap scripts are stored in a scripts subdirectory of the Nmap data directory (see ) by default. Scripts are indexed in a database stored in scripts/script.db. The database lists all of the scripts in each category. A single script may be in several categories. --script-args --script-args provides arguments to the scripts. See for a detailed explanation. --script-trace --script-trace This option is similar to --packet-trace, but works at the application level rather than packet by packet. If this option is specified, all incoming and outgoing communication performed by scripts is printed. The displayed information includes the communication protocol, source and target addresses, and the transmitted data. If more than 5% of transmitted data is unprintable, hex dumps are given instead. --script-updatedb --script-updatedb This option is only useful if you have added or removed NSE scripts from the default scripts directory, or if you have changed any of the scripts' categories fields. This field contains categories such as safe and discovery which the script belongs to. Categories may be specified with the --script option. For efficiency reasons, NSE generates a script.db file which maps categories to the scripts they contain. If you changed tag directives or added/removed scripts, run nmap --script-updatedb. Some of the Nmap options have effects on script scans. The most prominent of these is -sV. A version scan executes the scripts in the version category. The scripts in this category are slightly different than other scripts. Their output blends in with the version scan and they do not produce any script scan output. Another option which has effect on the scripting engine is -A. The aggressive mode of Nmap implies the option -sC. Simple script scan. $ nmap -sC hostname Tracing a specific script. $ nmap --script=./showSSHVersion.nse --script-trace hostname NSE scripts consist of four descriptive fields, a port or host rule defining when the script should be executed, and an action block containing the actual script instructions. All six of these are Lua variables that are assigned to. Their names must be lowercase as shown here. The script's id field is displayed in the Nmap output table if the script produces any output. It should be unique so users can identify exactly which script file produced a message. IDs should be kept short to conserve space in Nmap output, while still being meaningful enough for users to recognize. Some good examples are RIPE query, HTML title, and Kibuv worm. The description describes what the script is testing for and any critical notes the user must be aware of. A good example is this user contributed recursive DNS script description Checks whether a nameserver on UDP port 53 allows queries for third party names. It is expected that recursion will be enabled on your own internal nameserver. The author field contains the script authors name and contact information. If you are worried about spam, you might want to omit or obscure your email address, or give your home page URL instead. This optional field is not used by NSE, but is important for giving script authors due credit or blame. Nmap is a community project and we welcome all sorts of code contributions, including NSE scripts. So if you write a valuable script, don't keep it to yourself! The license field helps ensure that we have legal permission to distribute all the scripts which come with Nmap. All of those scripts currently use the standard Nmap license (described in ). They the following line: license = "Same as Nmap--See http://nmap.org/book/man-legal.html" The Nmap license is similar to the GNU GPL. Script authors may use a BSD-style license (no advertising clause) instead if they prefer that. This optional field determines script execution order. When this section is absent the run level defaults to 1.0. A script with the run level 1.0 is run before any scripts with runlevel set to 2.5, which in turn runs before any scripts with runlevel 2.55. Scripts with the same run level are run concurrently. One application of run levels is allowing scripts to depend on each other. If script A relies on some information gathered by script B, give B a lower run level than A. Script B can store information in the NSE registry for A to retrieve later. For information on the NSE registry see to . There are two types of rules: host rules which run only once against a target IP and port rules which run against individual ports on a target. A rule is a Lua function which takes a host and a port table as arguments and must returns a boolean value. If the rule evaluates to true, the script action is performed. Otherwise the action is skipped. Port rules are only matched against TCP or UDP ports in the open, open|filtered or unfiltered states. Host rules are matched exactly once against every scanned host. The action, like the rule, is a Lua function, which takes a host and port table as arguments. If the script is matched using a host rule, then the port table is absent (nil). Example rules are shown in . The action is the heart of an NSE script. It contains all of the instructions to be executed when the script's port or host rule triggers. It is a Lua function which can return either nil or a string. If a string is returned, it is printed along with the script ID in (if it is a service script) or below (if it is a host script) the Nmap port table. If the script returns nil, no output is produced. For an example of an NSE action refer to . Nmap's scripting engine consists of three more or less distinct parts. The largest part is the embeddable Lua interpreter. This is a lightweight language designed for extensibility. It offers a powerful and well documented API for interfacing with other software (such as Nmap). The second part of the Nmap scripting engine is the NSE library, which connects Lua and Nmap. This layer handles issues such as initialization of the Lua interpreter, scheduling of parallel script execution, script retrieval and more. It is also the heart of the NSE network I/O framework and the exception handling mechanism. Lua was designed with a small feature set to ease embedding. So we have added extensions to support more specialized functionality. These are basically Lua modules written either in Lua itself, or in C. This NSE library is the third part of the NSE. The Nmap scripting language is an embedded Lua interpreter which was extended with libraries for interfacing with Nmap. The Nmap API is in the Lua namespace nmap. This means that all calls to resources provided by Nmap have an nmap prefix. nmap.new_socket(), for example, returns a new socket wrapper object. The Nmap library layer also takes care of initializing the Lua context, scheduling parallel scripts and collecting the output produced by completed scripts. During the planning stages, we considered several programming languages as the bases for Nmap scripting. One option was to implement a completely new programming language. The criteria imposed on the options were strict, NSE needed to be easy to use, small in size, compatible with the Nmap license, scalable, fast and parallelizable. There have been several efforts to design a security auditing language from scratch which have resulted in well known awkward solutions. It was clear from the beginning that we would not go down this road. For a while the Guile scheme interpreter was considered but the preference drifted towards Elk in favor of its more liberal license. But parallelizing Elk scripts would have been difficult. In addition, the subset of Nmap users familiar with functional programming is regarded too small to consider Scheme as an option. Larger interpreters like Perl, Python or Ruby are well known and loved, but are difficult to embed efficiently. In the end, Lua exceeded in all criteria for NSE. It is small, distributed under the MIT license, has coroutines for efficient parallel script execution, was designed with embeddability in mind, has excellent documentation, and is actively developed by a large and committed community. In addition to the significant built-in capabilities of Lua, we have written or integrated several extensions to make NSE scripts more powerful and convenient to write. These modules are compiled and installed along with Nmap. They have their own directory, nselib, which is installed in the configured datadir. Scripts need only require the default modules in order to use them. The default modules are described in the following sections. Lua does not provide bitwise logical operations. Since they are often useful for low-level network communication, Reuben Thomas' bitwise operation library for Lua has integrated into NSE. The arguments to the bitwise operation functions should be integers. The number of bits available for logical operations depends on the data type used to represent Lua numbers—this is typically 8-byte IEEE floats (double), which give 53 bits (the size of the mantissa). This implies that the bitwise operations won't work (as expected) for numbers larger than 1014. You can use them with 32-bit wide numbers without any problems. Operations involving 64-bit wide numbers, however, may not return the expected result. The logical operations start with b (for bit) to avoid clashing with reserved words; although xor isn't a reserved word, it seemed better to use bxor for consistency. In NSE the bitwise functions are in the bit namespace. bit.bnot(a) bit.bnot(a) Returns the one's complement of a. bit.band(w1,...) bit.band(w1,...) Returns the bitwise and of the w's. bit.bor(w1,...) bit.bor(w1,...) Returns the bitwise or of the w's. bit.bxor(w1,...) bit.bxor(w1,...) Returns the bitwise xor of the w's. bit.lshift(a,b) bit.lshift(a,b) Returns a shifted left b places—padded with zeros. bit.rshift(a,b) bit.rshift(a,b) Returns a shifted logically right b places. bit.arshift(a,b) bit.arshift(a,b) Returns a shifted arithmetically right b places. bit.mod(a,b) bit.mod(a,b) Returns the integer remainder of a divided by b. One of Lua's quirks is its string patterns. While they have great performance and are tightly integrated into the Lua interpreter, they are very different in syntax and not as powerful as standard regular expressions. So we have integrated Perl compatible regular expressions into Lua using libPCRE and a modified version of the Lua PCRE library written by Reuben Thomas and Shmuel Zeigerman. These are the same sort of regular expressions used by Nmap version detection. The main modification to their library is that the NSE version only supports PCRExpressions instead of both PCRE and POSIX patterns. In order to maintain a high script execution speed, the library interfacing with libPCRE is kept very thin. It is not integrated as seamlessly as the Lua string pattern API. This allows script authors to decide when to use PCRE expressions versus Lua patterns. PCRE involves a separate pattern compilation step, which saves execution time when patterns are reused. Compiled patterns can be cached in the NSE registry and reused by other scripts. The PCRE functions reside inside the pcre namespace. LibPCRE has a history of security vulnerabilities allowing attackers who are able to compile arbitrary regular expressions to execute arbitrary code. More such vulnerabilities may be discovered in the future. These have never affected Nmap because it doesn't give attackers any control over the regular expressions it uses. Similarly, NSE scripts should never build regular expressions with untrusted network input. Matching hardcoded regular expressions against the untrusted input is fine. The following documentation is derived from that supplied by the PCRE Lua lib. pcre.new(pattern, flags, locale) pcre.new Returns a compiled regular expression. The first argument is a string describing the pattern, such as ^foo$. The second argument is a number describing which compilation flags are set. The compilation flags are set bitwise. If you want to set the 3rd (corresponding to the number 4) and the 1st (corresponding to 1) bit for example you would pass the number 5 as a second argument. The compilation flags accepted are those of the PCRE C library. These include flags for case insensitive matching (1), matching line beginnings (^) and endings ($) even in multiline strings (i.e. strings containing \n) (2) and a flag for matching across line boundaries (4). No compilation flags yield a default value of 0. The third (optional) argument is a string describing the locale which should be used to compile the regular expression. The variable is a string which is passed to the C standard library function setlocale. For more information on this argument refer to the documentation of setlocale. The resulting compiled regular expression is ready to be matched against strings. Compiled regular expressions are subject to Lua's garbage collection. Generally speaking, my_regex = pcre.new("pcre-pattern",0,"C") should do the job most of the time. pcre.flags() pcre.flags Returns a table of the available PCRE option flags (numbers) keyed by their names (strings). Possible names of the available strings can be retrieved from the documentation of the PCRE library used to link against Nmap. The key is the option name in the manual minus the PCRE prefix. PCRE_CASELESS becomes CASELESS for example. pcre.version() pcre.version Returns the version of the PCRE library in use as a string. For example 6.4 05-Sep-2005. pcre_obj:match(string, start, flags) pcre.match Returns the start point and the end point of the first match of the compiled regular expression pcre_obj in the string. A third returned value is a table which contains false in the positions where the pattern did not match. If named sub-patterns were used the table also contains substring matches keyed by their sub-pattern name. Should no match be found the function returns nil. The second and third arguments are optional. The second argument is a number specifying where the engine should start trying to apply the pattern. The third argument specifies execution flags for the pattern. If you want to see if a given string matches a certain expression you could use: s = pcre_obj:match("string to be searched", 0,0); if(s) code_to_be_done_on_match end pcre_obj:exec(string, start, flags) pcre.exec This function is like match() except that a table returned as a third result contains offsets of substring matches rather than substring matches themselves. That table will not contain string keys, even if named sub-patterns are used. For example, if the whole match is at offsets 10, 20 and substring matches are at offsets 12, 14 and 16, 19 then the function returns the following: 10, 20, {12,14,16,19} pcre_obj:gmatch(string, func, n, ef) pcre.gmatch Tries to match the regular expression pcre_obj against string up to n times (or as many as possible if n is either not given or is not a positive number), subject to execution flags ef. Each time there is a match, func is called as func(m, t), where m is the matched string and t is a table of substring matches. This table contains false in the positions where the corresponding sub-pattern did not match. If named sub-patterns are used then the table also contains substring matches keyed by their correspondent sub-pattern names (strings). If func returns a true value, then gmatch immediately returns; gmatch returns the number of matches made. The ipOps module provides some functions for manipulating IPv4 addresses. The functions reside inside the ipOps namespace. bool = ipOps.isPrivate("ip-string") isPrivate checks whether an IP address, provided as a string in dotted-quad notation, is part of the non-routed private IP address space, as described in RFC 1918. These addresses are the well known 10.0.0.0/8,192.168.0.0/16 and 172.16.0.0/12 networks. DWORD = ipOps.todword("ip-string") todword returns the IP address as DWORD value (i.e. the IP a.b.c.d becomes (((a*256+b)*256+c)*256+d) ) a,b,c,d = ipOps.get_parts_as_number("ip-string") get_parts_as_number returns 4 numbers corresponding to the fields in dotted-quad notation. For example, ipOps.get_parts_as_number("192.168.1.1") returns 192,168,1,1. Since portrules are mostly the same for many scripts, the shortport module provides functions for the most common tests. The arguments in brackets ([]) are optional. If no proto is provided, tcp is used. The default state is open shortport.portnumber(port,[proto],[state]) portnumber The port argument is either a number or a table of numbers which are interpreted as port numbers, against which the script should run. shortport.service(service,[proto],[state]) service The service argument is either a string or a table of strings which are interpreted as service names (e.g. "http", "https", "smtp" or "ftp") against which the script should run. These service names are determined by Nmap's version scan or (if no version scan information is available) the service assigned to the port in nmap-services (i.e. "http" for TCP port 80). shortport.port_or_service(port,service,[proto],[state]) port_or_service This is a combination of the above functions, since many scripts explicitly try to run against the well known ports, but want also to run against any other port which was discovered to run the named service. A typical example for this function is: portrule = shortport.port_or_service(22,"ssh"). People used to programming in functional languages, such as Lisp or Haskell, appreciate their handling of lists very much. The listop module tries to bring much of the functionality from functional languages to Lua using Lua's central data structure, the table, as a base for its list operations. Highlights include a map function applying a given function to each element of a list. bool = listop.is_empty(list) is_empty Returns true if the given list is empty. bool = listop.is_list(value) is_list Returns true if the given value is a list (or rather a table). list = listop.map(function, list) map The provided function is applied to each element of the list separately. The returned list contains the results of each function call. For example listop.map(tostring,{1,2,true}) returns {"1","2","true"}. value = listop.apply(function, list) apply All of the elements in the list are passed to a call of function. The result is then returned. For example listop.apply(math.max,{1,5,6,7,50000}) yields 50000. list = listop.filter(predicate, list) filter Returns a list containing only those elements for which the predicate returns true. The predicate has to be a function, which takes an element of the list as argument and the result of which is interpreted as boolean value. If it returns true (or rather anything besides false and nil) the argument is appended to the return value of filter. For example: listop.filter(isnumber,{1,2,3,"foo",4,"bar"}) returns {1,2,3,4}. list = listop.flatten(list) flatten Since a list can itself contain lists as elements, flatten returns a list which only contains values that are not themselves lists. For example: listop.flatten({1,2,3,"foo",{4,5,{"bar"}}}) returns {1,2,3,"foo",4,5,"bar"}. list = listop.append(list1, list2) append Returns a list containing all elements of list1 appended by all elements of list2. list = listop.cons(value1, value2) cons Returns a list containing value1 appended by value2, which may be of any type. list = listop.reverse(list) reverse Returns a list containing all elements of the given list in inverted order. value = listop.car(list) car Returns the first element of the given list. value = listop.ncar(list,n) ncar Returns the nth (or first if n is omitted) element of the given list. value = listop.cdr(list) cdr Returns a list containing all elements but the first of the given list. value = listop.ncdr(list, n) ncdr Returns a list containing all elements but the first n of the given list, where n is 2 if it is omitted. Lua's string operations are very flexible and offer an easy-to-use way to manipulate strings. Concatenation using the .. operator is such an operation. The drawback of the built-in API however is the way it handles concatenation of many string values. Since strings in Lua are immutable values, each time you concatenate two strings both get copied into the result string. The strbuf module offers a workaround for this problem, while maintaining the nice syntax. This is accomplished by overloading the concatenation operator (..) the equality operator (==) and the tostring operator. By overloading these operators, we reduce the overhead of using a string buffer instead of a plain string to wrap the first literal string assigned to a variable inside a strbuf.new() call. Afterwards you can append to the string buffer, or compare two string buffers for equality just as you would do with normal strings. When looking at the details there are some more restrictions/oddities: The concatenation operator requires its left-hand value to be a string buffer. Therefore, if you want to prepend a string to a given string buffer you have to create a new string buffer out of the string you want to prepend. The string buffer's tostring operator concatenates the strings inside the buffer using newlines by default, since this appears to be the separator used most often. buffer = strbuf.new(...) new Creates a new string buffer. The optional arguments are added to the string buffer. Attempting to add non-strings will result in undefined behavior. buffer = strbuf.concat(strbuf1, value) concat Concatenates the value (which has to be either a string or a string buffer) to strbuf1. This is also the function serving as the string buffer's concatenation operator. The above function call can thus also be expressed as: buffer = strbuf1 .. value bool = strbuf.eqbuf(strbuf1, strbuf2) eqbuf Compares strbuf1 and strbuf2 for equality. For the function to return true, both values must be string buffers containing exactly the same strings. The eqbuf function is called to compare two strings for equality. strbuf.clear(strbuf) clear Deletes all strings in strbuf. string = strbuf.dump(strbuf, "delimiter") dump Dumps strbuf's contents as string. The second parameter is used as a delimiter between the strings stored inside strbuf. dump(strbuf, "\n") is used as the tostring function of string buffers. URL manipulation functions have obvious uses. Fortunately there is already an implementation of URL generation functions inside the Lua-socket package, which is fairly complete and well documented. For NSE, Lua-socket's URL module was extended with two functions: table = url.parse_query("query-string") parse_query This function takes a query-string of the form name1=value1&name2=value2... and returns a table containing the name-value pairs, with the name as the key and the value as its associated value. The table corresponding to the above query-string would have two entries: table["name1"]="value1" and table["name2"]="value2". query_string = url.build_query(table) build_query This is the inverse function to parse_query(). The match module was written to provide functions which can be used for delimiting data received by the receive_buf() function from the Network I/O API: start,end = match.regex("regexpattern") regex This is actually a wrapper around NSE' PCRE library exec function (see , thus giving script developers the possibility to use regular expressions for delimiting instead of Lua's string patterns. If you want to get the data in chunks separated by regex (which has to be a valid regular expression), you would write status, val = sockobj:receive_buf(match.lua("regex")). start,end = match.numbytes(number) numbytes Takes a number as argument and returns that many bytes. It can be used to get a buffered version of sockobj:receive_bytes(n) in case a script requires more than one fixed-size chunk, as the unbuffered version may return more bytes than requested and thus would require you to do the parsing on your own. The http module provides functions for dealing with the client side of the http protocol. The functions reside inside the http namespace. The return value of each function in this module is a table with the following keys: status, header and body. status is the status code of the http request In case of an error status is nil. header is a table with the headers received from the server. The header names are lower-cased and multiple headers of the same name are concatenated with comma. body holds a string with the request body. table = http.get(host,port,path,[options]) get Fetches a resource with a GET request. The first argument is either a string with the hostname or a table like the host table passed by nmap. The second argument is either the port number or a table like the port table passed by nmap. The third argument is the path of the resource. The fourth argument is a table for further options. The table may have 2 keys: timeout and header. timeout is the timeout used for the socket operations. header is a table with additional headers to be used for the request. The function builds the request and calls http.request table = http.request(host,port,request,[options]) request Sends request to host:port and parses the answer. The first argument is either a string with the hostname or a table like the host table passed by nmap. The second argument is either the port number or a table like the port table passed by nmap. SSL is used for the request if either port.service equals "https" or port.version.service_tunnel equals "ssl". The third argument is the request. The fourth argument is a table for further options. You can specify a timeout for the socket operations with the timeout key. table = http.get_url(url,[options]) get_url Parses url and calls http.get with the result. The second argument is a table for further options. The table may have 2 keys: timeout and header. timeout is the timeout used for the socket operations. header is a table with additional headers to be used for the request. The comm module provides functions for common network discovery tasks such as banner-grabbing and making a quick exchange of data. These functions' return values are setup for use with exception handling via nmap.new_try(). These functions can all be passed a table of options, but it's not required. The relevant indexes for this table are bytes, lines, proto and timeout. bytes is used to provide the minimum number of bytes required for a read. lines does the same, but for the minimum number of lines. proto is used to set the protocol to communicate with, defaulting to "tcp" if not provided. timeout is used to set the socket timeout (see the socket function set_timeout() for details). bool, response = comm.get_banner(host, port, [options]) get_banner This function simply connects to the specified port number on the specified host and returns any data received. bool is a boolean value indicating success. If bool is true, then the second returned value is the response from the target host. If bool is false, an error message is returned as the second value instead of a response. bool, response = comm.exchange(host, port, data, [options]) exchange This function connects to the specified port number on the specified host, sends data, then waits for and returns the response, if any. bool is a boolean value indicating success. If bool is true, then the second returned value is the response from the target host. If bool is false, an error message is returned as the second value instead of a response. The datafiles module provides functions for reading and parsing Nmap's data files (e.g. nmap-protocol, nmap-rpc, etc.). These functions' return values are setup for use with exception handling via nmap.new_try(). bool, table = datafiles.parse_protocols() parse_protocols This function reads and parses Nmap's nmap-protocols file. bool is a boolean value indicating success. If bool is true, then the second returned value is a table with protocol numbers indexing the protocol names. If bool is false, an error message is returned as the second value instead of the table. bool, table = datafiles.parse_rpc() parse_rpc This function reads and parses Nmap's nmap-rpc file. bool is a boolean value indicating success. If bool is true, then the second returned value is a table with RPC numbers indexing the RPC names. If bool is false, an error message is returned as the second value instead of the table. bool, table = datafiles.parse_services([protocol]) parse_services This function reads and parses Nmap's nmap-services file. bool is a boolean value indicating success. If bool is true, then the second returned value is a table containing two other tables: tcp{} and udp{}. tcp{} contains services indexed by TCP port numbers. udp{} is the same, but for UDP. You can pass "tcp" or "udp" as an argument to parse_services() to only get the corresponding table. If bool is false, an error message is returned as the second value instead of the table. The stdnse library contains various handy functions which are too small to justify modules of their own: stdnse.print_debug(...) print_debug Wrapper function around print_debug_unformatted() in the nmap namespace. The first argument, if numeric, is used as the necessary debug level to print the message (it defaults to 1 if omitted). All remaining arguments are processed with Lua's string.format() function, which provides a C-style printf interface. table = stdnse.strsplit("delimiter","text") strsplit This function will certainly be appreciated by Perl programmers. It takes two strings as arguments and splits the second one around all occurrences of the first one, returning a table, which contains the substrings without the delimiting string. string = stdnse.strjoin("delimiter", table) strjoin Inverse function to strsplit(). Basically this is Lua's table.concat() function with the parameters swapped for coherence. NSE scripts have access to several Nmap facilities for writing flexible and elegant scripts. The API provides target host details such as port states and version detection results. It also offers an interface to the Nsock library for efficient network I/O. An effective Nmap scripting engine requires more than just a Lua interpreter. Users need easy access to the information Nmap has learned about the target hosts. This data is passed as arguments to the NSE action method. The arguments, host and port, are Lua tables which contain information on the target against which the script is executed. The following list describes each variable in the host and port tables. host host This table is passed as a parameter to the rule and action functions. It contains information on the operating system run by the host (if the -O switch was supplied), the IP address and the host name of the scanned target. host.os host.os The os entry in the host table is an array of strings. The strings (maximally 8) are the names of the operating systems the target is possibly running. Strings are only entered in this array if the target machine is a perfect match for one or more OS database entries. If Nmap was run without the -O option, then host.os is nil. host.ip host.ip Contains a string representation of the IP address of the target host. If the scan was run against a host name and the reverse DNS query returned more than one IP addresses then the same IP address is used as the one chosen for the scan. host.name host.name Contains the reverse DNS entry of the scanned target host represented as a string. If the host has no reverse DNS entry, the value of the field is an empty string. host.targetname host.targetname Contains the name of the host as specified on the command line. If the target given on the command line contains a netmask or is an IP address the value of the field is nil. host.directly_connected host.directly_connected A boolean value indicating whether or not the target host is directly connected (i.e. on the same network segment). host.mac_addr host.mac_addr MAC address of the destination host (6-byte long binary string) or nil, if the host is not directly connected. host.mac_addr_src host.mac_addr_src Our own MAC address, which was used to connect to the host (either our network card's, or (with --spoof-mac) the spoofed address). host.interface host.interface A string containing the interface name (dnet-style) through which packets to the host are sent. host.bin_ip host.bin_ip The hosts IP as 4 byte long binary value. host.bin_ip_src host.bin_ip_src Our hosts IP as 4 byte long binary value. port port The port table is passed to the Lua script in the same fashion as the host table. It contains information about the port against which the script is running. If the script is run according to a host rule, then no port table is passed to the script. Port states on the target can still be requested from Nmap using the nmap.get_port_state() call. port.number port.number Contains the number of the currently scanned port. port.protocol port.protocol Defines the protocol of the port. Valid values are tcp and udp. port.service port.service Contains a string representation of the service running on port.number as detected by the Nmap service detection. If the port.version field is nil then Nmap has guessed the service based only on the port number. Otherwise this field is equal to port.version.name. port.version port.version This entry is a table which contains information retrieved by the Nmap version scanning engine. Some of the values (like service name, service type confidence, RPC related values) may be retrieved by Nmap even if a version scan was not required. Values which were not retrieved default to nil. The meaning of each value is given in the following table: Name Description name Contains the service name Nmap will use for the port. name_confidence Evaluates how confident the version detection is about the accuracy of name, from one (least confident) to 10. product, version, extrainfo, hostname, ostype, devicetype These five variables are described in . service_tunnel Contains the string none or ssl based on whether or not Nmap used SSL tunnelling to detect the service. service_fp The service fingerprint, if any, is provided in this value. This is described in . rpc_status Contains a string value of good_prog if we were able to determine the program number of an RPC service listening on the port, unknown if the port appears to be RPC but we couldn't determine the program number, not_rpc if the port doesn't appear be RPC, or untested if we haven't checked for RPC status. The rpc_program, rpc_lowver, and rpc_highver variables are nil unless rpc_status is good_prog. rpc_program, rpc_lowver, rpc_highver The detected RPC program number and the range of version numbers supported by that program. These will be nil if rpc_status is anything other than good_prog. port.state port.state Contains information on the state of the port. Service scripts are only run against ports in the open or open|filtered states, so port.state generally contains one of those values. Other values might appear if the port table is a result of the get_port_state function. You can adjust the port state using the nmap.set_port_state() call. This is normally done when an open|filtered port is determined to be open. Scripts also have access to some of Nmap’s functions and state variables that are exposed through functions in the nmap table. nmap.debugging() debuggingnmap.debugging Returns the debugging level as a non-negative integer. The debugging level can be set with the -d option (see ). nmap.have_ssl() have_ssl Returns true if Nmap was compiled with SSL support, false otherwise. This can be used to avoid sending SSL probes when SSL is not available. nmap.verbosity()verbositynmap.verbosity Returns the verbosity level as a non-negative integer. The verbosity level can be set with the -v option (see ). nmap.fetchfile(filename) fetchfile Allows access to Nmap's data files. fetchfile() searches for the specified file and returns a string containing it's path if it is found and readable (to the process). If the file is not found, not readable, or is a directory, nil is returned. The call nmap.fetchfile("nmap-rpc") will search for the data file nmap-rpc and, assuming it's found (which it should be), return a location like /usr/local/share/nmap/nmap-rpc. Often the information passed to the script is not enough. Sometimes a script might want to correct target information or set it in the first place. The following API methods handle this. nmap.get_port_state(host, port, protocol) get_port_state The get_port_state() call takes a host table, a port table and a protocol (tcp or udp) and returns a port table for the queried port. The host and port table are similar in structure to the ones passed to the rule and action functions. The host table should have an IP address field. The port table needs a port number and a protocol field. A call could look like this: nmap.get_port_state({ip="127.0.0.1"}, {number="80", protocol="tcp"}) You can of course reuse the host and port tables passed to the port rule function. The purpose of this call is to be able to match scripts against more than one open port. For example if the target host has an open port 22 and a running identd server, then you can write a script which will only fire if both ports are open and there is an identification server on port 113. While it is possible to specify IP addresses different to the currently scanned target, the result will only be correct if the target is in the currently scanned group of hosts. nmap.set_port_state(host, port, state) set_port_state The set_port_state() call takes a host table, a port table, and a port state (open or closed). With this method the final port state can be changed. This is useful when Nmap detects a port as open|filtered but the script successfully connects to it. In this case the port state can be set to open. Note that the port.state value, which was passed to the script's action function will not be changed by this call. nmap.set_port_version(host, port, probestate) set_port_version To provide a flexible extension to Nmap's version detection NSE scripts can set the version and service variables of a port. The method takes a host and a port table as arguments. The third argument describes the state in which the script completed. It is a string which is one of: hardmatched, softmatched, nomatch, tcpwrapped, or incomplete. A hard match will almost always be used, as it means that the script was able to determine the protocol. You can pass in nomatch if the script fails to match the target port, but it is probably already set that way anyway. One of the other states should only be used if you know exactly what you are doing. The host and port arguments to this function should either be the tables passed to the action method or they should have the same structure. The version detection fields this function looks at are name, product, version, extrainfo, hostname, ostype, devicetype, and service_tunnel. All values in this table are optional. It is possible to pass a table in which all these values are set to nil or not to set the values at all. NSE has support for sending raw ethernet frames and capturing packets. The following two functions may be handy in this context: nmap.clock_ms() nmap.clock_ms() Returns a number representing the current time as milliseconds since the start of the epoch (on most systems this is 01/01/1970). nmap.get_interface_link("interface_name") nmap.get_interface_link(interface_name) For the provided dnet-style interface_name, nmap.get_interface_link() returns what kind of link level hardware the interface belongs. Return values are: ethernet, loopback or p2p. If the provided interface_name is not one of those types, nil is returned. To allow for efficient and parallelizable network I/O, NSE provides an interface to Nsock, the Nmap socket library. The smart callback mechanism Nsock uses is fully transparent to NSE scripts. The main benefit of Nmap-NSE sockets is that they never block on I/O operations, allowing many scripts to be run in parallel. The I/O parallelism is fully transparent to authors of NSE scripts. In NSE you can either program as if you were using a single non blocking socket or you can program as if your connection is blocking. Seemingly blocking I/O calls still return once a specified timeout has been exceeded. Two flavors of Network I/O are supported: This part of the network API should be suitable for most classical network uses: Users create a socket, connect it to a remote address, send and receive data and close the socket again. Everything up to the Transport layer (which is either TCP, UDP or SSL) is handled by the library. The following socket API methods are supported: nmap.new_socket() nmap.new_socket() The new_socket() Nmap call returns an Nmap-NSE socket object which is the recommended method for network I/O. It provides facilities to perform communication using the UDP, TCP and SSL protocol in a uniform manner. status, error = socket_object:connect(hostid, port, [protocol]) connect The connect method of Nmap-NSE socket objects will put the socket in a state ready for communication. It takes as arguments a host descriptor (either an IP address or a host name), a port number and optionally a protocol. The protocol must be one of "tcp", "udp" or "ssl". By default the connect call will attempt to open a TCP connection. On success the returned value of status is true. If the connection attempt has failed, the error value contains a description of the error condition stored as a string. Those strings are taken from the gai_strerror() C function. They are (with the errorcode in parentheses): Address family for hostname not supported (EAI_ADDRFAMILY) Temporary failure in name resolution (EAI_AGAIN) Bad value for ai_flags (EAI_BADFLAGS) Non-recoverable failure in name resolution (EAI_FAIL) ai_family not supported (EAI_FAMILY) Memory allocation failure (EAI_MEMORY) No address associated with hostname (EAI_NODATA) Name or service not known (EAI_NONAME) Servname not supported for ai_socktype (EAI_SERVICE) ai_socktype not supported (EAI_SOCKTYPE) System error (EAI_SYSTEM) In addition to these standard system error based messages are the following two NSE-specific errors: Sorry, you don't have OpenSSL. occurs if ssl is passed as third argument, but Nmap was compiled without OpenSSL support. invalid connection method occurs if the second parameter is not one of tcp, udp, ssl. status, error = socket_object:send(data) send The send method sends the data contained in the data string through an open connection. On success the returned value of status is true. If the send operation has failed, the error value contains a description of the error condition stored as a string. The error strings are: Trying to send through a closed socket—if there was no call to socket_object:connect before the send operation. TIMEOUT—if the operation took longer than the specified timeout for the socket. ERROR—if an error occurred inside the underlying Nsock library. CANCELLED—if the operation was cancelled. KILL—if for example the script scan is aborted due to a faulty script. EOF—if an EOF was read—will probably not occur for a send operation. status, value = socket_object:receive() receive The receive method does a non-blocking receive operation on an open socket. On success the returned value of status is true and the received data is stored in value. If receiving data has failed, value contains a description of the error condition stored as a string. A failure occurs for example if receive is called on a closed socket. The receive call returns to the NSE script all the data currently stored in the receive buffer of the socket. Error conditions are the same as for the send operation. status, value = socket_object:receive_lines(n) receive_lines Tries to receive at least n lines from an open connection. A line is a string delimited with \n characters. If it was not possible to receive at least n lines before the operation times out a TIMEOUT error occurs. On the other hand, if more than n lines were received, all are returned, not just n. On success the returned value of status is true and the received data is stored in value. If the connection attempt has failed, value contains a description of the error condition stored as string. Error conditions are the same as for the send operation. status, value = socket_object:receive_bytes(n) receive_bytes Tries to receive at least n bytes from an open connection. On success the returned value of status is true and the received data is stored in value. If operation fails, value contains a description of the error condition stored as a string. Similarly to receive_lines() n is the minimum amount of characters we would like to receive. If more arrive, we get all of them. If less than n characters arrive before the operation times out, a TIMEOUT error occurs. Other error conditions are the same as for the send operation. status, value = socket_object:receive_buf(func/"string", keeppattern) receive_buf receive_buf tries to circumvent several limitations in the other receive* functions. receive_line(n), for example, tries to ensure that there are at least n lines received and returns everything it has already read from the connection (even though there may be much more data than requested). It also leaves line-parsing to the user. receive_buf on the other hand returns only the part of the received data until the first match of a delimiter, with the rest being saved inside a buffer for later calls to receive_buf. This buffer gets cleared on calls to other functions inside the Network I/O API. Should the data not contain the delimiter another read request is sent and the buffer is checked again when more data is present. receive_buf takes two arguments. The first one is either a string or a function. If it is a string it gets passed to Lua's string.find function as the (second) pattern parameter, with the buffer data being searched. If it is a function it is expected to take exactly one parameter (the buffer) and its return values have to be like those of string.find (i.e. offsets of the start and the end of the delimiter inside the buffer, or nil, if the delimiter is not found). The second argument is a boolean value which indicates whether the delimiting pattern should be returned along with the received data or discarded. A module inside the nselib match.lua () provides functions for matching received data against regular expressions or for receiving a defined number of bytes. receive_buf's return values behave exactly as the return values of the other receive* functions. Two values are returned (status,val)— the first indicating whether the request was successful, the other containing the returned data (or the case of a failure, an error message). Possible error messages are those of the other receive* functions and, in addition, the following: Error inside splitting-function—if the first argument was a function which caused an error while being called. Error in string.find (nsockobj:receive_buf)!—if a string was provided as the first argument, and string.find() yielded an error while being called. Expected either a function or a string!—if the first argument was neither a function nor a string. Delimiter has negative size!—if the returned start offset is greater than the end offset. status, err = socket_object:close() close Closes an open connection. On success the returned value of status is true. If the connection attempt has failed, value contains a description of the error condition stored as a string. Currently the only error message is: Trying to close a closed socket, which is issued if the socket has already been closed. Sockets are subject to garbage collection. Should you forget to close a socket, it will get closed before it gets deleted (on the next occasion Lua's garbage collector is run). However since garbage collection cycles are difficult to predict, it is considered good practice to close opened sockets. status,localip,localport,remoteip,remoteport=socket_object:get_info() get_info This function returns information about the socket object. It returns 5 values. If an error occurred, the first value is nil and the second value describes the error condition. Otherwise the first value describes the success of the operation and the remaining 4 values describe both endpoints of the TCP connection. If you put the call in a try() statement the status value is consumed. The call can be used for example if you want to query an authentication server. socket_object:set_timeout(t) set_timeout Sets the time, in milliseconds, after which input and output operations on a socket should time out and return. The default value is 30,000 (30 seconds). The lowest allowed value is 10ms, since this is the granularity of NSE network I/O. For those cases where the connection oriented approach is too inflexible, NSE provides script developers with a more powerful option: raw packet network I/O. The greater flexibility comes, however, at the cost of a slightly more complex API. Receiving raw packets is accomplished via a wrapper around Libpcap inside the Nsock library. In order to keep the capturing efficient it works in a three tiered approach: Opening a device for capturing, registering listeners to it and receiving packets. With each call to pcap_open() you have to provide a callback function, which receives the packet (along with it's layer 2 and 3 headers) and is used to compute a so-called packet hash. Each call to pcap_register() takes a binary string as argument. For every packet captured the computed hash is matched against all registered strings. Those scripts for which the compare yields true are then provided with the packet as a return value to pcap_receive(). The more general the packet hash computing function is kept, the more scripts may receive the packet and proceed with their execution. To use the packet capturing inside your script you have to create (and afterwards close) a socket with nmap.newsocket() (or socket_object:close() respectively)—just like with the connection-based network I/O. A more detailed description of the functions for packet capturing follows: socket_object:pcap_open(device, snaplen, promisc, test_function, bpf) pcap_open The pcap_open() call opens the socket for packet capturing. The parameters are: device—the dnet-style interface name of the device you want to capture from. snaplen—defines the length of each packet you want to capture (similar to the -s option to tcpdump) promisc—should be set to 1 if the interface should activate promiscuous mode, and zero otherwise. test_function—callback function used to compute the packet-hash bpf—a string describing a Berkeley packet filter expression (like those provided to tcpdump) socket_object:pcap_register(packet-hash) pcap_register Starts the listening for incoming packages. The provided packet-hash is a binary string which has to match the hash returned by the test_function parameter provided to pcap_open(). If you want to receive all packets, just provide the empty string (""). There has to be a call to pcap_register() before a call to pcap_receive(). status, packet_len, l2_data, l3_data = socket_object:pcap_receive() pcap_receive Receives a captured packet. If successful, the return values are: status—a boolean with the value true. packet_len—the length of the captured packet (note, that you could have received less data if the snaplen parameter was smaller than the packet length) l2_data—data from the second OSI layer (e.g. ethernet headers) l3_data—data from the third OSI layer (e.g. IPv4 headers). Should an error or timeout occur, while waiting for a packet the return values are: nil,error_message,nil,nil, where error_message describes the occurred error. socket_object:pcap_close() pcap_close() Closes the pcap device. Receiving raw packets is a great feature, but it is also only the half job. Now for sending raw packets: To accomplish this NSE has access to a wrapper around the dnet library. Currently NSE has the ability to send raw ethernet frames via the following API: dnet_object=nmap.new_dnet() new_dnet() Creates and returns a new dnet_object, which can be used to send raw packets. dnet_object:ethernet_open(interface_name) ethernet_open Opens the interface defined by the provided interface_name for sending ethernet frames through it. An error (device is not valid ethernet interface) is thrown in case the provided argument is not valid. dnet_object:ethernet_send(packet) ethernet_send Sends the provided data as ethernet frame across the previously opened interface. Note that you have to provide the packet including IP header and ethernet header. If there was no previous valid call to ethernet_open() an error is thrown (dnet is not valid opened ethernet interface). dnet_object:ethernet_close() ethernet_close Closes the interface. The only error which may be thrown is the same as for the ethernet_send() operation. Each thread made for a script (e.g. anonFTP.nse) will yield to other scripts whenever it makes a call on network objects (sending/receiving data). Some scripts need finer control over threads' execution. An example is the whois.nse script which queries whois servers for each target. Because many concurrent queries often result in getting one's IP banned for abuse and a query may return additional information for targets other threads are running against, it is useful to have other threads pause while one thread is conducting a query. To solve this problem, there is an nmap function, mutex, that provides a mutex usable by scripts. The mutex allows for only one thread to be working on an object. Threads waiting to work on this object are put in the waiting queue until they can get a "lock" on the mutex. A solution for the whois.nse problem above is to have each thread block on a mutex for script's ID field , thus ensuring only one thread is working so its results can be shared with other scripts which may not need to run and so queries to the whois servers are staggered. nmap.mutex(object) mutex Returns a function that works on a mutex for the object passed. This object can be any Lua data type except nil, booleans, and numbers. The returned function allows you to lock, try to lock, and release the mutex. It's first and only parameter is either: "lock"—Make a blocking lock on the mutex. If the mutex is busy (another thread has a lock on it), then the thread will yield and wait. The function returns with the mutex locked. "trylock"—Makes a non-blocking lock on the mutex. If the mutex is busy then it immediately returns with a return value of false. Otherwise the mutex locks the mutex and returns true. "done"—Releases the mutex and allows another thread to lock it. If the thread does not have a lock on the mutex, an error will be raised. "running"—Returns the thread locked on the mutex or nil. This should only be used for debugging as it interferes with finished threads from being collected. id = "My Script's Unique ID"; local mutex = nmap.mutex(id); function action(host, port) mutex "lock"; -- do stuff mutex"done"; return script_output; end NSE provides an exception handling mechanism not present in the plain Lua language. The exception handling is tailored specifically for network I/O operations. The mechanism follows a functional programming paradigm rather than an object oriented programming paradigm. To create an exception handler the nmap.new_try() API method is used. This method returns a function, which takes a function as an argument. If the function passed as an argument raises an exception, then the script execution is aborted and no output is produced. Optionally you can pass a function to the new_try() method which will be called if an exception is caught. In this function you can perform required clean up operations. shows cleanup exception handling at work. A new function named catch is defined to simply close the newly created socket in case of an error. It is then used to protect connection and communication attempts on that socket. If no catch function is specified, execution of the script aborts without further ado—open sockets will remain open. If the verbosity level is at least one or if the scan is performed in debugging mode a description of the uncaught error condition is printed on standard output. Note that it is currently not easily possible to group several statements in one try block. It is also important to remember that if the socket is not closed it will occupy memory until the next run of Lua's garbage collector. local result, socket, try, catch result = "" socket = nmap.new_socket() catch = function() socket:close() end try = nmap.newtry(catch) try(socket:connect(host.ip, port.number)) result = try(socket:receive_lines(1)); try(socket:send(result)) Writing a function which is treated properly by the try/catch mechanism is straightforward. The function should return multiple values. The first value should be a boolean which is true upon successful completion of the function and false otherwise. If the function completed successfully the try construct consumes the indicator value and returns the remaining values. If the function failed then the second returned value must be a string describing the error condition. Note that if the value is not nil it is treated as true so you can return your value in the normal case and return nil, error description if an error occurs. The registry is a normal Lua table. What is special about it is that it is visible by all scripts and it retains its state between script executions. Nmap does not scan every host specified on the command line at the same time, it puts them in smaller groups and these groups are scanned in parallel. The registry is rebuilt for every group, so information stored there is only deleted after NSE finishes processing the current target group. This implies of course that the registry is transient—it is not stored between Nmap executions. Every script can read the registry and write to it. If a script is running after another script, it can read some information in the registry which was left by the first script. This feature is particularly powerful in combination with the run level concept. A script with a higher run level can rely on entries left behind for it by scripts with lower run levels. Remember however that the registry can be written by all scripts equally, so choose the keys for your entries wisely. The registry is stored in nmap.registry. The behavior of the registry allows caching of already calculated data. The cache can be seen by all scripts until the registry is rebuilt with the next target group. Suppose that you are convinced of the power of NSE. How do you go about writing your own script? Let's say that you want to extract information from an identification server. Nmap used to have this functionality but it was removed because of inconsistencies in the code base. Fortunately, the protocol identd uses is pretty simple. Unfortunately, it is too complicated to be expressible in Nmap's version detection language. Let's look at how the identification protocol works. First you connect to the identification server. Next you send a query of the form port-on-server, port-on-client terminated with a new line character. The server should then respond with a string of the form port-on-server, port-on-client:response-type:address-information. In case of an error the address information is omitted. This description is sufficient for our purposes, for more details refer to RFC 1413. The protocol cannot be modeled in Nmap's version detection language for two reasons. The first is that you need to know both the local and the remote port of a connection. Version detection does not provide this data. The second, more severe obstacle, is that you need two open connections to the target—one to the identification server and one to the port you want to query. Both obstacles are easily overcome with NSE. The anatomy of a script is described in . In this section we will show how the described structure is utilized. The head of the script is essentially its meta information. This includes the fields id, description, author, license and categories. We are not going to change the run level for now. The id of a script should uniquely identify it. If it is absent, the path to the script will be used as an id. We recommend to choose an id which concisely identifies the purpose of the script, since the ID is printed before the script's results in Nmap output. id = "Service Owner" The description field should contain a sentence or two describing what the script does. If anything about the script results might confuse or mislead users, and you can't eliminate the issue by improving the script or results text, it should be documented in the description string. description = "Opens a connection to the scanned port, opens a connection to \ port 113, queries the owner of the service on the scanned port and prints it." Users must tell the Lua interpreter that the string continues on the following line by ending the line with a backslash (‘\’). They must also decide what categories the script belongs to. This script is a good example of a script which cannot be categorized clearly. It is safe because we are not using the service for anything it was not intended for. On the other hand, it is intrusive because we connect to a service on the target and therefore potentially give out information about us. To solve this dilemma we will place our script in two categories: categories = {"safe", "intrusive"} The rule section is a Lua method which decides when the script's action should be performed and when it should be skipped. Usually this decision is based on the host and port information passed to the rule function. In the case of the identification script it is slightly more complicated than that. To decide whether to run the identification script on a given port we need to know if there is an identification server running on the target machine. Or more formally: the script should be run if (and only if) the currently scanned TCP port is open and TCP port 113 is also open. For now we will rely on the fact that identification servers listen on TCP port 113. Unfortunately NSE only gives us information about the currently scanned port. To find out if port 113 is open we are going to use the nmap.get_port_state() method. If the identd port was not scanned, the get_port_state function returns nil. So we need to make sure that the table is not nil. We also check if both ports are in the open state. If this is the case, the action is executed, otherwise we skip the action. portrule = function(host, port) local ident_port = { number=113, protocol="tcp" } local identd = nmap.get_port_state(host, ident_port) if identd ~= nil and identd.state == "open" and port.state == "open" then return true else return false end end This rule is almost correct, but still slightly buggy. Can you find the bug? It is a pretty subtle one. The problem is that this script fires on any kind of open port, TCP or UDP. The connect() method on the other hand assumes a TCP protocol unless it is explicitly told to use another protocol. Since the identification service is only defined for TCP connections, we need to narrow down the range of ports which fire our script. Our new rule only runs the script if the port is open, we are looking at a TCP port, and TCP port 113 is open. Writing the new and improved port rule is left as an exercise to the reader (or peek at the script in the latest Nmap distribution). At last we implement the actual functionality. The script will first connect to the port on which we expect to find the identification server, then it will connect to the port we want information about. Afterward we construct a query string and parse the response. If we received a satisfactory response, we return the retrieved information. First we need to create two socket objects. These objects represent the sockets we are going to use. By using object methods like open(), close(), send() or receive() we can operate on the network socket. To avoid excessive error checking code we use NSE's exception handling mechanism. We create a function which will be executed if an error occurs and call this function catch. Using this function we generate a try function. The try function will call the catch function whenever there is an error condition in the tried block. Note that we could have ignored the last two return values of client_service:get_info() like this: local localip, localport = client_service:get_info() This would have sufficed because we know that the remote port is stored in port.number. In this example we prefer not to tell the user if the query resulted in an error. To inform users of failed identification queries, simply uncomment the corresponding line. It is necessary that we assign the variable owner a nil value because returning nil is the only way to tell the script engine to suppress script output. action = function(host, port) local owner = "" local client_ident = nmap.new_socket() local client_service = nmap.new_socket() local catch = function() client_ident:close() client_service:close() end local try = nmap.newtry(catch) try(client_ident:connect(host.ip, 113)) try(client_service:connect(host.ip, port.number)) local localip, localport, remoteip, remoteport = client_service:get_info() local request = port.number .. ", " .. localport .. "\n" try(client_ident:send(request)) owner = try(client_ident:receive_lines(1)) if string.match(owner, "ERROR") then owner = nil -- owner = "Service owner could not be determined: " .. owner else owner = string.match(owner, "USERID : .+ : (.+)\n", 1) end try(client_ident:close()) try(client_service:close()) return owner end The version detection system built into Nmap was designed to efficiently recognize the vast majority of protocols with a simple pattern matching syntax. Some protocols require a more complex approach, and a generalized scripting language is perfect for this. Skype2 is one such protocol. It pretends to be an http server, requiring multiple queries to determine its true nature. NSE has been integrated into Nmap's version detection framework to handle these cases. The scripts which extend the version scanner belong to the reserved category version. This category cannot be run from the command line. It is only executed if the user has required a version scan. The following listing shows a simple script which demonstrates the use of the NSE version detection API. If either the TCP port 80 is open or the service has been determined to be http, the script is triggered. Although it could be extended to recognize different http servers, its only purpose is to show off the version detection API. It is not advisable to use NSE for version detection in the simple case of http servers. The version detection variables have been filled with dummy entries to illustrate their effect on the Nmap output. description = "Demonstration of a version detection NSE script. It checks \ and reports the version of a remote web server. For real life purposes it is \ better to use Nmap version detection (-sV)." author = "Diman Todorov <diman.todorov@gmail.at>" license = "Same as Nmap--See http://nmap.org/book/man-legal.html" id = "HTTP version" categories = {"version"} runlevel = 1.0 portrule = function(host, port) if (port.number == 80 or port.service == "http" ) and port.protocol == "tcp" then return true else return false end end action = function(host, port) local query = "GET / HTTP/2.1\r\n" query = query .. "Accept: */*\r\n" query = query .. "Accept-Language: en\r\n" query = query .. "User-Agent: Nmap NSE\r\n" query = query .. "Host: " .. host.ip .. ":" .. port.number .. "\r\n\r\n" local socket = nmap.new_socket() local catch = function() socket:close() end local try = nmap.new_try(catch) try(socket:connect(host.ip, port.number)) try(socket:send(query)) local response = "" local lines local status local value while true do status, lines = socket:receive_lines(1) if not status or value then break end response = response .. lines value = string.match(response, "Server: (.-)\n") end try(socket:close()) if value then port.version.name = "[Name]" port.version.name_confidence = 10 port.version.product = "[Product]" port.version.version = "[Version]" port.version.extrainfo = "[ExtraInfo]" port.version.hostname = "[HostName]" port.version.ostype = "[OSType]" port.version.devicetype = "[DeviceType]" port.version.service_tunnel = "none" port.version.fingerprint = nil nmap.setPortVersion(host, port, "hardmatched") end end This is what the output of this script looks like: $ ./nmap -sV localhost -p 80 Starting Nmap ( http://nmap.org ) Interesting ports on localhost (127.0.0.1): PORT STATE SERVICE VERSION 80/tcp open [Name] [Product] [Version] ([ExtraInfo]) Service Info: Host: [HostName]; OS: [OSType]; Device: [DeviceType] Nmap finished: 1 IP address (1 host up) scanned in 9.317 seconds The name variable denotes the detected protocol name. The product, version and extrainfo variables are used to produce a human readable description of the server version. The remaining variables provide information deduced from the output of the server concerning the target host. The finger script (finger.nse) is a perfect example of how short typical NSE scripts are. first the information fields are filled out, note that the id field is kept short, this is important since it is printed in Nmap's output. A detailed description of what the script actually does should go in the description field. id="Finger Results" description="attempts to get a list of usernames via the finger service" author = "Eddie Bell <ejlbell@gmail.com>" license = "Same as Nmap--See http://nmap.org/book/man-legal.html" The categories field is a table containing all the categories the script belongs to—These are used for script selection through the --script option. categories = {"discovery"} You can use the facilities provided by the nselib () by requiring them. Here we want to use shorter port rules. require "shortport" We want to check whether the service behind the port is finger, or whether it runs on finger's well known port 79. Through this we can use the information gathered during the version scan (if finger runs on a non-standard port) or still run against at least the port we expect it, should the version detection information not be available. portrule = shortport.port_or_service(79, "finger") action = function(host, port) local socket = nmap.new_socket() local results = "" local status = true The function err_catch() will be called for clean up, through NSE's exception handling mechanism. Here it only closes the previously opened socket (which should be enough in most cases). local err_catch = function() socket:close() end The clean up function gets registered for exception handling via a call to nmap.new_try() local try = nmap.new_try(err_catch()) The script sets a timeout of 5000, which is equivalent to 50 seconds. Should any operation require more time we'll receive a TIMEOUT error message. socket:set_timeout(5000) For actually using exception handling we need to wrap calls to functions, which may return an error inside try() try(socket:connect(host.ip, port.number, port.protocol)) try(socket:send("\n\r")) The call to receive_lines() is not wrapped in try(), because we don't want to abort the script just because we didn't receive the data we expected. Note that there is less data than requested (100 lines), we still receive it and the status is true —consequent calls would yield a false status. status, results = socket:receive_lines(100) socket:close() The script returns a string only if we got the data we wanted, otherwise nil is returned (automatically, since scripts return one result). if not(status) then return results end end showOwner.nse demonstrates the flexibility of the NSE, which is unmatched by other parts of Nmap. If the target is running an identd daemon it connects to it for each running service and tries to identify its owner. id = "Service owner" description = "Opens a connection to the scanned port, opens a connection to \ port 113, queries the owner of the service on the scanned port and prints it." author = "Diman Todorov <diman.todorov@gmail.com>" license = "Same as Nmap--See http://nmap.org/book/man-legal.html" categories = {"default", "safe"} Portrules are not restricted to those provided by the short-port module (). They can be any function taking a host- and a port table as argument and returning a boolean. portrule = function(host, port) local auth_port = { number=113, protocol="tcp" } In order to determine the state of a port, which is not provided as argument we just have to construct a table describing the port (i.e. its number and the protocol it's using) and pass it to nmap.get_port_state() which returns a table filled with the information Nmap has about the port. local identd = nmap.get_port_state(host, auth_port) if identd ~= nil and identd.state == "open" and port.protocol == "tcp" and port.state == "open" then return true else return false end end action = function(host, port) local owner = "" Scripts can open any number of connection they want. local client_ident = nmap.new_socket() local client_service = nmap.new_socket() local catch = function() client_ident:close() client_service:close() end local try = nmap.new_try(catch) try(client_ident:connect(host.ip, 113)) try(client_service:connect(host.ip, port.number)) local localip,localport,remoteip,remoteport = try(client_service:get_info()) local request = port.number .. ", " .. localport .. "\n" try(client_ident:send(request)) owner = try(client_ident:receive_lines(1)) if string.match(owner, "ERROR") then owner = nil else owner = string.match(owner, "USERID : .+ : (.+)\n", 1) end try(client_ident:close()) try(client_service:close()) return owner end Now how does all this work? The following section describes some interesting aspects of the NSE. While the focus primarily lies on giving script writers a better feeling of what happens with scripts, it should also provide a starting point for understanding (and extending) the NSE sources. During its initialization stage, Nmap loads the Lua interpreter and its provided libraries get loaded. These libraries are: The package library (namespace: package)—Lua's package-lib provides (among others) the require function, used to load modules from the nselib. The table library (namespace: table)—The table manipulation library contains many functions used to operate on tables—Lua's central data structure. The I/O library (namespace: io)—The Input/Output library offers functions such as reading files and reading the output from programs you execute. The OS library (namespace: os)—The Operating System lib provides facilities of the operating system, including filesystem operations (renaming/removing files, creating of temporary filenames) and access to the environment. The string library (namespace: string)—The string library helps you with functions used to manipulate strings inside Lua. Functions include: printf-style string formating, pattern matching using Lua-style patterns, substring extraction, etc. The math library (namespace: math)—Since usually numbers in Lua correspond to the double C-type, the math library gives you access to rounding functions, trigonometric functions, random number generation, and many more. The debug library (namespace: debug)—The debug library provides you with a somewhat lower level API to the Lua-interpreter. Through it you can access functions along the execution stack, get function closures and object metatables, etc. In addition to loading the libraries provided with Lua, the functions in the nmap namespace also get loaded. and search path for modules is set to the default one prepended by the nselib directory (which is searched in the locations Nmap searches for its data files and scripts). In this step the provided script arguments also get stored inside the registry. The next phase of NSE initialization is loading the chosen scripts, which are the arguments provided to the --script option or default, in case of a default script scan. The string version is appended, if version detection was enabled. The arguments afterwards are tried to be interpreted as script categories. This is done via a Lua C function in nse_init.cc called entry. Inside script.db, for each category of a script, there is a call to Entry. If the category was chosen then the script is loaded. Every argument of --script that could not be interpretted as a category is loaded as a file or directory. If the file or directory could not be located, then an error is raised and the Script Engine aborts. All the .nse files inside a loaded directory are loaded as files. Each file loaded is exectuted in Lua. If a portrule is present, then it is saved in the porttests table with a portrule key and file closure value. Otherwise, if the script has a hostrule , then it is saved in the hosttests in the same manner. After the initialization is finished the hostrules and portrules are evaluated for each host in the current target group. At this check a list is built which contains the combinations of scripts and the hosts they will run against. It should be noted that the rules of all chosen scripts are checked against all hosts and their open and open|filtered ports. Therefore it is advisable to leave the rules as simple as possible and to do all the computation inside the action, as a script will only be executed if it is run against a specific target. After the check those script-target combinations get their own Lua-thread. A thread running against a host will have only a hostrule passed to the action closure whereas a thread running against a port will have both a hostrule and portrule passed. Each thread is stored in a runlevel table with a table of information for the thread. This information includes the runlevel, target, target port (if applicable), host and port tables (passed to action), its type (running against a host or port), and its id. When script scanning begins, these runlevel tables that store the threads will be passed to mainloop where the real work begins. Now to the actual script scanning, and the way NSE accomplishes parallelization. Lua, through its concept of coroutines offers collaborative multi-threading, which means that scripts can suspend themselves, at defined points, and let other coroutines execute. Since network I/O, especially waiting for responses from remote host, is the part of scripts which would consume most time with waiting, this is the point where scripts suspend themselves and let others execute. Each call to some of the functions of the Nsock wrapper causes the calling script to yield (pause). Once the request is processed by the Nsock library, the callback causes the script to be pushed from the waiting queue to the running queue, which will eventually let it resume its operation. The running queue is the runlevel table passed to mainloop (see nse_main.cc). Mainloop will create a table for waiting scripts which will have the same form as the running queue. Threads will be moved back and forth between the tables; when a thread yields, it is moved to the waiting queue. After all scripts are run in the running queue, mainloop will place all threads ready to be run in the running queue. Threads are made "ready" by calling process_waiting2running. This process of running threads and moving paused threads to the waiting and running queues is repeated until no threads exist in the waiting or running queues. This section tries to give a short walk-through to adding nselib modules written in C (or C++) to Nmap's build system, since this has shown to be sometimes tedious. Writing C modules is described at length in Programming in Lua, Second Edition. Programming in Lua, Second Edition. Basically C modules consist of the functions they provide to Lua, which have to be of type lua_CFunction. Additionally they have to contain a function which is used to actually open the module. By convention these function names are luaopen_modulename. A good starting point for writing such modules is provided with bit.c and pcre.c inside the nselib/ subdirectory of Nmap's source tree, which are two C modules already provided by the nselib. C modules basically are shared libraries which get loaded at runtime by Lua. The Unix build system uses libtool for compilation in a platform independent way. As long as the new module does not depend on foreign libraries, you should only need to add modulename.so to the all and clean targets in Makefile.in and copy and adapt the lines from bit.so. If your module does have dependencies you will most probably have to add checks for those libraries to configure.ac and put the dependencies inside the libtool commands in Makefile.in—here you should take a look at how pcre.so handles this. So much for the way it should work. Now for some pitfalls we've come across so far: These are results from building shared libraries in general and not really specific to nselib. Linking with static libraries (e.g. libnbase) sometimes leads to problems with exporting symbols on some platforms (in our case this was a x86_64-linux platform). To our knowledge no such problems occur when linking against already existing shared libraries. The Windows build system requires C module developers to create a MS Visual Studio Project file for their module (<modulename>.vcproj) inside the nselib subdirectory. On Windows you have to include the liblua/ subdirectory as an additional include path as well as a library search path. In addition you have to tell the project to link against the liblua.lib static library provided with Nmap. Other properties of the project should be the same as for other nselib C modules (e.g. see nse_bitlib.vcproj). Afterwards you have to include the newly created project file in Nmap's Visual Studio solution file (mswin32\nmap.sln) and make sure that nse_bitlib.vcproj depends on your project, because it is there nselib-modules get copied to their final destinations and are included in Nmap.