Nmap Scripting Engine (NSE) scriptingNmap Scripting Engine NSENmap Scripting Engine The Nmap Scripting Engine (NSE) is one of Nmap's most powerful and flexible features. It allows users to write (and share) simple scripts to automate a wide variety of networking tasks. Those scripts are then executed in parallel with the speed and efficiency you expect from Nmap. Users can rely on the growing and diverse set of scripts distributed with Nmap, or write their own to meet custom needs. We designed NSE to be versatile, with the following tasks in mind: Network discovery This is Nmap's bread and butter. Examples include looking up whois data based on the target domain, querying ARIN, RIPE, or APNIC for the target IP to determine ownership, performing identd lookups on open ports, SNMP queries, and listing available NFS/SMB/RPC shares and services. version detectionusing NSE More sophisticated version detection The Nmap version detection system () is able to recognize thousands of different services through its probe and regular expression signature based matching system, but it cannot recognize everything. For example, identifying the Skype v2 service requires two independent probes, which version detection isn't flexible enough to handle. Nmap could also recognize more SNMP services if it tried a few hundred different community names by brute force. Neither of these tasks are well suited to traditional Nmap version detection, but both are easily accomplished with NSE. For these reasons, version detection now calls NSE by default to handle some tricky services. This is described in . vulnerability detection Vulnerability detection When a new vulnerability is discovered, you often want to scan your networks quickly to identify vulnerable systems before the bad guys do. While Nmap isn't a comprehensive vulnerability scanner, NSE is powerful enough to handle even demanding vulnerability checks. Many vulnerability detection scripts are already available and we plan to distribute more as they are written. Backdoor detection Many attackers and some automated worms leave backdoors to enable later reentry. Some of these can be detected by Nmap's regular expression based version detection. For example, within hours of the MyDoom worm hitting the Internet, Jay MoranMoran, Jay posted an Nmap version detection probe and signature so that others could quickly scan their networks for MyDoom infections. NSE is needed to reliably detect more complex worms and backdoors. Vulnerability exploitation As a general scripting language, NSE can even be used to exploit vulnerabilities rather than just find them. The capability to add custom exploit scripts may be valuable for some people (particularly penetration testers),penetration testing though we aren't planning to turn Nmap into an exploitation framework such as Metasploit.Metasploit These listed items were our initial goals, and we expect Nmap users to come up with even more inventive uses for NSE. Scripts are written in the embedded Lua programming language.Lua programming languageNmap Scripting Engine The language itself is well documented in the books Programming in Lua, Second Edition and Lua 5.1 Reference Manual. Programming in Lua, Second Edition and Lua 5.1 Reference Manual. The reference manual is also freely available online, as is the first edition of Programming in Lua. Given the availability of these excellent general Lua programming references, this document only covers aspects and extensions specific to Nmap's scripting engine. NSE is activated with the -sC option (or --script if you wish to specify a custom set of scripts) and results are integrated into Nmap normalnormal output and XML output.XML output Two types of scripts are supported: service and host scripts. Service scripts relate to a certain open port (service) on the target host, and any results they produce are included next to that port in the Nmap output port table. Host scripts, on the other hand, run no more than once against each target IP and produce results below the port table. shows a typical script scan. Examples of service scripts producing output are: sshv1, saying that SSH-1 is supported; and html-title, which simply grabs the title of the root path of any web servers found. A sample host script is ripeQuery, which looks up and reports target IP ownership information.script names, examples of -sCexample of $ nmap -sC localhost -p 22,23,80,113 Starting Nmap ( http://nmap.org ) Interesting ports on localhost (127.0.0.1): PORT STATE SERVICE 22/tcp open ssh |_ sshv1: Server supports SSHv1 23/tcp closed telnet 80/tcp open http |_ html-title: Test Page for Apache Installation 113/tcp closed auth Host script results: |_ ripeQuery: IP belongs to: Internet Assigned Numbers Authority Nmap done: 1 IP address (1 host up) scanned in 0.91 seconds While NSE has a complex implementation for efficiency, it is strikingly easy to use. Simply specify -sC-sC to enable the most common scripts. Or specify the --script--script option to choose your own scripts to execute by providing categories, script file names, or the name of directories full of scripts you wish to execute. You can customize some scripts by providing arguments to them via the --script-args--script-args option. The two remaining options, --script-trace--script-trace and --script-updatedb,--script-updatedb are generally only used for script debugging and development. Script scanning is also included as part of the -A (aggressive scan) option. script categories NSE scripts define a list of categories they belong to. Currently defined categories are auth, default, discovery, external, intrusive, malware, safe, version, and vuln. Category names are not case sensitive. The following list describes each category. “auth” script category auth These scripts try to determine authentication credentials on the target system, often through a brute-force attack. Examples include snmp-brute, http-auth, and ftp-anon. “default” script category default These scripts are the default set and are run when using the -sC or -A options rather than listing scripts with --script. This category can also be specified explicitly like any other using --script=default. Many factors are considered in deciding whether a script should be run by default: Speed A default scan must finish quickly, which excludes brute force authentication crackers, web spiders, and any other scripts which can take minutes or hours to scan a single service. Usefulness Default scans need to produce valuable and actionable information. If even the script author has trouble explaining why an average networking or security professional would find the output valuable, the script should not run by default. The script may still be worth including in Nmap so that administrators can run for those occasions when they do need the extra information. Verbosity Nmap output is used for a wide variety of purposes and needs to be readable and concise. A script which frequently produces pages full of output should not be added to the default category. When there is no important information to report, NSE scripts (particularly default ones) should return nothing. Checking for an obscure vulnerability may be OK by default as long as it only produces output when that vulnerability discovered. Reliability Many scripts use heuristics and fuzzy signature matching to reach conclusions about the target host or service. Examples include sniffer-detect and sql-injection. If the script is often wrong, it doesn't belong in the default category where it may confuse or mislead casual users. Users who specify a script or category directly are generally more advanced and likely know how the script works or at least where to find its documentation. Intrusiveness Some scripts are very intrusive because they use significant resources on the remote system, are likely to crash the system or service, or are likely to be perceived as an attack by the remote administrators. The more intrusive a script is, the less suitable it is for the default category. Privacy Some scripts, particularly those in the external category described later, divulge information to third parties by their very nature. For example, the whois script must divulge the target IP address to regional whois registries. We have also considered (and decided against) adding scripts which check target SSH and SSL key fingerprints against Internet weak key databases. The more privacy-invasive a script is, the less suitable it is for default category inclusion. We don't have exact thresholds for each of these criteria, and many of them are subjective. All of these factors are considered together when making a decision whether to promote a script into the default category. A few default scripts are identd-owners (determines the username running remote services using identd), http-auth (obtains authentication scheme and realm of web sites requiring authentication), and ftp-anon (tests whether an FTP server allows anonymous access). “discovery” script category discovery These scripts try to actively discover more about the network by querying public registries, SNMP-enabled devices, directory services, and the like. Examples include html-title (obtains the title of the root path of web sites), smb-enum-shares (enumerates Windows shares), and snmp-sysdescr (extracts system details via SNMP). “external” script category external Scripts in this category may send data to a third-party database or other network resource. An example of this is whois, which makes a connection to whoiswhois servers to learn about the address of the target. There is always the possibility that operators of the third-party database will record anything you send to them, which in many cases will include your IP address and the address of the target. Most scripts involve traffic strictly between the scanning computer and the client; any that do not are placed in this category. “intrusive” script category intrusive These are scripts that cannot be classified in the safe category because the risks are too high that they will crash the target system, use up significant resources on the target host (such as bandwidth or CPU time), or otherwise be perceived as malicious by the target's system administrators. Examples are http-open-proxy (which attempts to use the target server as an HTTP proxy) and snmp-brute (which tries to guess a device's SNMP community string by sending common values such as public, private, and cisco). “malware” script category malware These scripts test whether the target platform is infected by malware or backdoors. Examples include smtp-strangeport, which watches for SMTP servers running on unusual port numbers, and auth-spoof, which detects identd spoofing daemons which provide a fake answer before even receiving a query. Both of these behaviors are commonly associated with malware infections. “safe” script category safe Scripts which weren't designed to crash services, use large amounts of network bandwidth or other resources, or exploit security holes are categorized as safe. These are less likely to offend remote administrators, though (as with all other Nmap features) we cannot guarantee that they won't ever cause adverse reactions. Most of these perform general network discovery. Examples are ssh-hostkey (retrieves an SSH host key) and html-title (grabs the title from a web page). “version” script category version detection“version” script category version The scripts in this special category are an extension to the version detection feature and cannot be selected explicitly. They are selected to run only if version detection (-sV) was requested. Their output cannot be distinguished from version detection output and they do not produce service or host script results. Examples are skypev2-version, pptp-version, and iax2-version. “vuln” script category vuln These scripts check for specific known vulnerabilities and generally only report results if they are found. Examples include realvnc-auth-bypass and xampp-default-auth. These are the five command line arguments specific to script-scanning: -sC -sC Performs a script scan using the default set of scripts. It is equivalent to --script=default. Some of the scripts in this default category are considered intrusive and should not be run against a target network without permission. --script --script script-categories|directory|filename|all Runs a script scan (like -sC) using the comma-separated list of script categories, individual scripts, or directories containing scripts, rather than the default set. Nmap first tries to interpret the arguments as categories, then (if that fails) as files or directories. A script or directory of scripts may be specified as an absolute or relative path. Absolute paths are used as supplied. Relative paths are searched for in the following places until found:data filesdirectory search orderscripts, location of --datadir/; $NMAPDIR/;NMAPDIR environment variable ~/.nmap/ (not searched on Windows);.nmap directory NMAPDATADIR/ orNMAPDATADIR ./. A scripts/ subdirectory is also tried in each of these. If a directory is specified and found, Nmap loads all NSE scripts (any filenames ending with .nse) from that directory. Filenames without the nse extension are ignored. Nmap does not search recursively into subdirectories to find scripts. If individual file names are specified, the file extension does not have to be nse. Nmap scripts are stored in a scripts subdirectory of the Nmap data directory by default (see ). For efficiency, scripts are indexed in a database stored in scripts/script.db.script.db which lists the category or categories in which each script belongs. Give the argument all to execute all scripts in the Nmap script database. Scripts are not run in a sandbox and thus could accidentally or maliciously damage your system or invade your privacy. Never run scripts from third parties unless you trust the authors or have carefully audited the scripts yourself. --script-args --script-args provides arguments to the scripts. See for a detailed explanation. --script-trace --script-trace This option is similar to --packet-trace, but works at the application level rather than packet by packet. If this option is specified, all incoming and outgoing communication performed by scripts is printed. The displayed information includes the communication protocol, source and target addresses, and the transmitted data. If more than 5% of transmitted data is unprintable, hex dumps are given instead. Specifying --packet-trace enables script tracing too. --script-updatedb --script-updatedb This option updates the script database found in scripts/script.db which is used by Nmap to determine the available default scripts and categories. It is only necessary to update the database if you have added or removed NSE scripts from the default scripts directory or if you have changed the categories of any script. This option is used by itself without arguments: nmap --script-updatedb. Some other Nmap options have effects on script scans. The most prominent of these is -sV.-sV A version scan automatically executes the scripts in the version category.“version” script category The scripts in this category are slightly different than other scripts because their output blends in with the version scan results and they do not produce any script scan output. Another option which affects the scripting engine is -A.-Afeatures enabled by The aggressive Nmap mode implies the -sC option. script arguments Arguments may be passed to NSE scripts using the --script-args option. The script arguments are generally name-value pairs. They are provided to scripts as a Lua table named args inside nmap.registry. The argument names are keys for the corresponding values. The values can be either strings or tables. Subtables can be used to pass arguments to scripts with finer granularity, such as passing different usernames for different scripts. Here is a typical Nmap invocation with script arguments: -sCexample of --script-argsexample of $ nmap -sC --script-args user=foo,pass=bar,whois={whodb=nofollow+ripe} The aforementioned command results in this Lua table: {user="foo",pass="bar",whois={whodb="nofollow+ripe"}} You could therefore access the username (foo) inside your script with this statement: local username = nmap.registry.args.user Subtables used to override options for scripts are usually named after the script to ease retrieval. A simple script scan using the default set of scripts: -sCexample of $ nmap -sC example.com Executing a specific script with tracing enabled: --scriptexample of --script-traceexample of $ nmap --script=./showSSHVersion.nse --script-trace example.com Execute all scripts in the mycustomscripts directory as well as all default scripts in the safe category: $ nmap --script=mycustomscripts,safe example.com NSE scripts consist of two–five descriptive fields along with either a port or host rule defining when the script should be executed and an action block containing the actual script instructions. Values can be assigned to the descriptive fields just as you would assign any other Lua variables. Their names must be lowercase as shown in this section. “description” script variable The description field describes what a script is testing for and any important notes the user should be aware of. Depending on script complexity, the description may vary from a few sentences to a few paragraphs. The first paragraph should be a brief synopsis of the script function suitable for stand-alone presentation to the user. Further paragraphs may provide much more script detail. “categories” script variable The categories field defines one or more categories to which a script belongs (see ). The categories are case-insensitive and may be specified in any order. They are listed in an array-style Lua table as in this example: categories = {"default", "discovery", "safe"} “author” script variable The author field contains the script authors' names and contact information. If you are worried about spam, feel free to omit or obscure your email address, or give your home page URL instead. This optional field is not used by NSE, but gives script authors due credit or blame. “license” script variable copyrightof scripts Nmap is a community project and we welcome all sorts of code contributions, including NSE scripts. So if you write a valuable script, don't keep it to yourself! The optional license field helps ensure that we have legal permission to distribute all the scripts which come with Nmap. All of those scripts currently use the standard Nmap license (described in ). They include the following line: license = "Same as Nmap--See http://nmap.org/book/man-legal.html" The Nmap license is similar to the GNU GPL. Script authors may use a BSD-style license (no advertising clause) instead if they prefer that. “runlevel” script variable run level of scripts This optional field determines script execution order. When this section is absent, the run level defaults to 1.0. Scripts with a given runlevel execute after any with a lower runlevel and before any scripts with a higher runlevel against a single target machine. The order of scripts with the same runlevel is undefined and they often run concurrently. One application of run levels is allowing scripts to depend on each other. If script A relies on some information gathered by script B, give B a lower run level than A. Script B can store information in the NSE registry for A to retrieve later. For information on the NSE registry, see . “portrule” script variable “hostrule” script variable rules in NSE“portrule” and “hostrule” Nmap uses the script rules to determine whether a script should be run against a target. A script contains either a port rule, which governs which ports of a target the scripts may run against, or a host rule, which specifies that the script should be run only once against a target IP and only if the given conditions are met. A rule is a Lua function that returns either true or false. The script action is only performed if its rule evaluates to true. Host rules accept a host table as their argument and may test, for example, the IP address or hostname of the target. A port rule accepts both host and port tables as arguments for any TCP or UDP port in the openopen port state, open|filteredopen|filtered port state, or unfilteredunfiltered port state port states. Port rules generally test factors such as the port number, port state, or listening service name in deciding whether to run against a port. Example rules are shown in . “action” script variable The action is the heart of an NSE script. It contains all of the instructions to be executed when the script's port or host rule triggers. It is a Lua function which accepts the same arguments as the rule and can return either nil or a string. If a string is returned by a service script, the string and script's filename are printed in the Nmap port table output. A string returned by a host script is printed below the port table. No output is produced if the script returns nil. For an example of an NSE action refer to . Nmap Scripting Engine (NSE)parts of The core of the Nmap Scripting Engine is an embeddable Lua interpreter. Lua is a lightweight language designed for extensibility. It offers a powerful and well documented API for interfacing with other software such as Nmap. Nmap Scripting Engine (NSE)library The second part of the Nmap Scripting Engine is the NSE Library, which connects Lua and Nmap. This layer handles issues such as initialization of the Lua interpreter, scheduling of parallel script execution, script retrieval and more. It is also the heart of the NSE network I/O framework and the exception handling mechanism. It also includes utility libraries to make scripts more powerful and convenient. The utility library modules and extensions are described in . Lua programming language The Nmap scripting language is an embedded Lua interpreter which was extended with libraries for interfacing with Nmap. The Nmap API is in the Lua namespace nmap. This means that all calls to resources provided by Nmap have an nmap prefix.nmap NSE module nmap.new_socket(), for example, returns a new socket wrapper object. The Nmap library layer also takes care of initializing the Lua context, scheduling parallel scripts and collecting the output produced by completed scripts. During the planning stages, we considered several programming languages as the base for Nmap scripting. Another option was to implement a completely new programming language. Our criteria were strict: NSE had to be easy to use, small in size, compatible with the Nmap license, scalable, fast and parallelizable. Several previous efforts (by other projects) to design their own security auditing language from scratch resulted in awkward solutions, so we decided early not to follow that route. First the Guile Scheme interpreter was considered, but the preference drifted towards the Elk interpreter due to its more favorable license. But parallelizing Elk scripts would have been difficult. In addition, we expect that most Nmap users prefer procedural programming over functional languages such as Scheme. Larger interpreters such as Perl, Python, and Ruby are well-known and loved, but are difficult to embed efficiently. In the end, Lua excelled in all of our criteria. It is small, distributed under the liberal MIT open source license, has coroutines for efficient parallel script execution, was designed with embeddability in mind, has excellent documentation, and is actively developed by a large and committed community. Lua is now even embedded in other popular open source security tools including the Wireshark sniffer and Snort IDS. Nmap Scripting Engine (NSE)list of scripts This section lists (alphabetically) all NSE scripts packaged with Nmap at the time of this writing. It comes straight from the script source code thanks to the NSEDoc documentation system described in . Of course no paper documentation can stay current with software developed as actively as NSE is. For the most comprehensive and up-to-date documentation, see the online NSE Documentation Portal at . &nse-scripts; This section (a long list of NSE scripts with brief summaries) is only provided in the printed edition of this book because we already provide a better online interface to the information at the NSE Documentation Portal. Nmap Scripting Engine (NSE)list of modules In addition to the significant built-in capabilities of Lua, we have written or integrated many extension libraries which make script writing more powerful and convenient. These libraries (sometimes called modules) are compiled if necessary and installed along with Nmap. They have their own directory, nselib, which is installed in the configured datadir. Scripts need only require the default libraries in order to use them. This list is just an overview to give an idea of what libraries are available. Developers will want to consult the complete documentation at . &nse-modules; nmap NSE module Nmap Scripting Engine (NSE)API NSE scripts have access to several Nmap facilities for writing flexible and elegant scripts. The API provides target host details such as port states and version detection results. It also offers an interface to the NsockNsock library for efficient network I/O. An effective Nmap scripting engine requires more than just a Lua interpreter. Users need easy access to the information Nmap has learned about the target hosts. This data is passed as arguments to the NSE script's action method.“action” script variable The arguments, host and port, are Lua tables which contain information on the target against which the script is executed. If a script matched a hostrule, it gets only the host table, and if it matched a portrule it gets both host and port. The following list describes each variable in these two tables. host This table is passed as a parameter to the rule and action functions. It contains information on the operating system run by the host (if the -O switch was supplied), the IP address and the host name of the scanned target. host.os The os entry in the host table is an array of strings. The strings (as many as eight) are the names of the operating systems the target is possibly running. Strings are only entered in this array if the target machine is a perfect match for one or more OS database entries. If Nmap was run without the -O option, then host.os is nil. host.ip Contains a string representation of the IP address of the target host. If the scan was run against a host name and the reverse DNS query returned more than one IP addresses then the same IP address is used as the one chosen for the scan. host.name Contains the reverse DNS entry of the scanned target host represented as a string. If the host has no reverse DNS entry, the value of the field is an empty string. host.targetname Contains the name of the host as specified on the command line. If the target given on the command line contains a netmask or is an IP address the value of the field is nil. host.directly_connected A Boolean value indicating whether or not the target host is directly connected to (i.e. on the same network segment as) the host running Nmap. host.mac_addr MAC addressMAC address of the destination host (six-byte long binary string) or nil, if the host is not directly connected. host.mac_addr_src Our own MAC address, which was used to connect to the host (either our network card's, or (with --spoof-mac)--spoof-mac the spoofed address). host.interface A string containing the interface name (dnet-style)libdnet through which packets to the host are sent. host.bin_ip The target host's IPv4 address as a 32-bit binary value. host.bin_ip_src Our host's (running Nmap) source IPv4 address as a 32-bit binary value. port The port table is passed to an NSE service script (i.e. only those with a portrule rather than a hostrule) in the same fashion as the host table. It contains information about the port against which the script is running. While this table is not passed to host scripts, port states on the target can still be requested from Nmap using the nmap.get_port_state() call. port.number Contains the port number of the target port. port.protocol Defines the protocol of the target port. Valid values are "tcp" and "udp". port.service Contains a string representation of the service running on port.number as detected by the Nmap service detection. If the port.version field is nil, Nmap has guessed the service based on the port number. Otherwise version detection was able to determine the listening service and this field is equal to port.version.name. port.version This entry is a table which contains information retrieved by the Nmap version scanning engine. Some of the values (such as service name, service type confidence, and the RPC-related values) may be retrieved by Nmap even if a version scan was not performed. Values which were not determined default to nil. The meaning of each value is given in the following table: Name Description name Contains the service name Nmap decided on for the port. name_confidence Evaluates how confident Nmap is about the accuracy of name, from 1 (least confident) to 10. product, version, extrainfo, hostname, ostype, devicetype These five variables are described in . service_tunnel Contains the string "none" or "ssl" based on whether or not Nmap used SSL tunneling to detect the service. service_fp The service fingerprint, if any, is provided in this value. This is described in . rpc_status Contains a string value of good_prog if we were able to determine the program number of an RPC service listening on the port, unknown if the port appears to be RPC but we couldn't determine the program number, not_rpc if the port doesn't appear be RPC, or untested if we haven't checked for RPC status. rpc_program, rpc_lowver, rpc_highver The detected RPC program number and the range of version numbers supported by that program. These will be nil if rpc_status is anything other than good_prog. port.state Contains information on the state of the port. Service scripts are only run against ports in the open or open|filtered states, so port.state generally contains one of those values. Other values might appear if the port table is a result of the get_port_state function. You can adjust the port state using the nmap.set_port_state() call. This is normally done when an open|filtered port is determined to be open. To allow for efficient and parallelizable network I/O, NSE provides an interface to Nsock, the Nmap socket library. The smart callback mechanism Nsock uses is fully transparent to NSE scripts. The main benefit of NSE's sockets is that they never block on I/O operations, allowing many scripts to be run in parallel. The I/O parallelism is fully transparent to authors of NSE scripts. In NSE you can either program as if you were using a single non-blocking socket or you can program as if your connection is blocking. Even blocking I/O calls return once a specified timeout has been exceeded. Two flavors of Network I/O are supported: connect-style and raw packet. sockets in NSE This part of the network API should be suitable for most classical network uses: Users create a socket, connect it to a remote address, send and receive data and finally close the socket. Everything up to the Transport layer (which is either TCP, UDP or SSL) is handled by the library. An NSE socket is created by calling nmap.new_socket, which returns a socket object. The socket object supports the usual connect, send, receive, and close methods. Additionally the functions receive_bytes, receive_lines, and receive_buf allow greater control over data reception. shows the use of connect-style network operations. The try function is used for error handling, as described in . require("nmap") local socket = nmap.new_socket() socket:set_timeout(1000) try = nmap.new_try(function() socket:close() end) try(socket:connect(host.ip, port.number)) try(socket:send("login")) response = try(socket:receive()) socket:close() raw packetsin NSE For those cases where the connection-oriented approach is too high-level, NSE provides script developers with the option of raw packet network I/O. Raw packet reception is handled through a a Libpcaplibpcap wrapper inside the Nsock library.Nsock The steps are to open a capture device, register listeners with the device, and then process packets as they are received. The pcap_open method creates a handle for raw socket reads from an ordinary socket object. This method takes a callback function, which computes a packet hash from a packet (including its headers). This hash can return any binary string, which is later compared to the strings registered with the pcap_register function. The packet hash callback will normally extract some portion of the packet, such as its source address. The pcap reader is instructed to listen for certain packets using the pcap_register function. The function takes a binary string which is compared against the hash value of every packet received. Those packets whose hashes match any registered strings will be returned by the pcap_receive method. Register the empty string to receive all packets. A script receives all packets for which a listener has been registered by calling the pcap_receive method. The method blocks until a packet is received or a timeout occurs. The more general the packet hash computing function is kept, the more scripts may receive the packet and proceed with their execution. To handle packet capture inside your script you first have to create a socket with nmap.new_socket and later close the socket with socket_object:close—just like with the connection-based network I/O. Receiving raw packets is important, but sending them is a key feature as well. To accomplish this, NSE can access a wrapper around the libdnet library.libdnet Raw packet writes do not use a standard socket object like reads do. Instead, call the function nmap.new_dnet to create a dnet object with ethernet sending methods. Then open an interface with the ethernet_open method. Raw ethernet frames can then be sent with ethernet_send. When you're done, close the ethernet handle with ethernet_close. Sometimes the easiest ways to understand complex APIs is by example. The sniffer-detect.nse script included with Nmap uses raw packet capture and sending in an attempt to detect promiscuous-mode machines on the network (those running sniffers). threads in NSE mutexes in NSE Each script execution thread (e.g. ftp-anon running against an FTP server on the target host) yields to other scripts whenever it makes a call on network objects (sending or receiving data). Some scripts require finer concurrency control over thread execution. An example is the whois script which queries whoiswhois servers for each target IP address. Because many concurrent queries often result in getting one's IP banned for abuse, and because a single query may return additional information for targets other threads are running against, it is useful to have other threads pause while one thread performs a query. To solve this problem, NSE includes a mutex function which provides a mutex (mutual exclusion object) usable by scripts. The mutex allows for only one thread to be working on an object. Competing threads waiting to work on this object are put in the waiting queue until they can get a "lock" on the mutex. A solution for the whois problem above is to have each thread block on a mutex using a common string, thus ensuring that only one thread is querying whois servers at once. That thread can store the results in the NSE Registry before releasing unlocking the mutex. The next script in the waiting queue can then run. It will first check the Registry and only query whois servers if the previous results were insufficient. The first step is to create a mutex object using a statement such as: mutexfn = nmap.mutex(object) The mutexfn returned is a function which works as a mutex for the object passed in. This object can be any Lua data type except nil, booleans, and numbers. The returned function allows you to lock, try to lock, and release the mutex. Its first and only parameter must be one of the following: "lock" Make a blocking lock on the mutex. If the mutex is busy (another thread has a lock on it), then the thread will yield and wait. The function returns with the mutex locked. "trylock" Makes a non-blocking lock on the mutex. If the mutex is busy then it immediately returns with a return value of false. Otherwise the mutex locks the mutex and returns true. "done" Releases the mutex and allows another thread to lock it. If the thread does not have a lock on the mutex, an error will be raised. "running" Returns the thread locked on the mutex or nil if the mutex is not locked. This should only be used for debugging as it interferes with garbage collection of finished threads. A simple example of using the API is provided in . For real-life examples, read the asn-query.nse and whois.nse scripts in the Nmap distribution. local mutex = nmap.mutex("My Script's Unique ID"); function action(host, port) mutex "lock"; -- Do critical section work - only one thread at a time executes this. mutex "done"; return script_output; end exceptions in NSE NSE provides an exception handling mechanism which is not present in the base Lua language. It is tailored specifically for network I/O operations, and follows a functional programming paradigm rather than an object oriented one. The nmap.new_try API method is used to create an exception handler. This method returns a function which takes a variable number of arguments that are assumed to be the return values of another function. If an exception is detected in the return values (the first return value is false), then the script execution is aborted and no output is produced. Optionally, you can pass a function to new_try which will be called if an exception is caught. The function would generally perform any required cleanup operations. shows cleanup exception handling at work. A new function named catch is defined to simply close the newly created socket in case of an error. It is then used to protect connection and communication attempts on that socket. If no catch function is specified, execution of the script aborts without further ado—open sockets will remain open until the next run of Lua's garbage collector. If the verbosity level is at least one or if the scan is performed in debugging mode a description of the uncaught error condition is printed on standard output. Note that it is currently not easily possible to group several statements in one try block. local result, socket, try, catch result = "" socket = nmap.new_socket() catch = function() socket:close() end try = nmap.new_try(catch) try(socket:connect(host.ip, port.number)) result = try(socket:receive_lines(1)) try(socket:send(result)) Writing a function which is treated properly by the try/catch mechanism is straightforward. The function should return multiple values. The first value should be a Boolean which is true upon successful completion of the function and false otherwise. If the function completed successfully, the try construct consumes the indicator value and returns the remaining values. If the function failed then the second returned value must be a string describing the error condition. Note that if the value is not nil or false it is treated as true so you can return your value in the normal case and return nil, error description if an error occurs. registry (NSE) The registry is a normal Lua table. What is special about it is that it is visible by all scripts and it retains its state between script executions. The registry is transient—it is not stored between Nmap executions. Every script can read the registry and write to it. If a script is running after another script, it can read some information in the registry which was left by the first script. This feature is particularly powerful in combination with the run level concept. A script with a higher run level can rely on entries left behind for it by scripts with lower run levels. run level of scripts Remember however that the registry can be written by all scripts equally, so choose the keys for your entries wisely. The registry is stored in nmap.registry. The behavior of the registry allows caching of already calculated data. The cache can be seen by all scripts until the registry is rebuilt with the next target group. Nmap Scripting Engine (NSE)tutorial Suppose that you are convinced of the power of NSE. How do you go about writing your own script? Let's say that you want to extract information from an identification server.auth service Nmap used to have this functionality but it was removed because of inconsistencies in the code base. Fortunately, the protocol identd uses is pretty simple. Unfortunately, it is too complicated to be expressible in Nmap's version detection language. Let's look at how the identification protocol works. First you connect to the identification server. Next you send a query of the form port-on-server, port-on-client terminated with a new line character. The server should then respond with a string of the form port-on-server, port-on-client:response-type:address-information. In case of an error the address information is omitted. This description is sufficient for our purposes, for more details refer to RFC 1413. The protocol cannot be modeled in Nmap's version detection language for two reasons. The first is that you need to know both the local and the remote port of a connection. Version detection does not provide this data. The second, more severe obstacle, is that you need two open connections to the target—one to the identification server and one to the port you want to query. Both obstacles are easily overcome with NSE. The anatomy of a script is described in . In this section we will show how the described structure is utilized. The head of the script is essentially its meta information. This includes the fields: description, categories, runlevel, author and license. We are not going to change the run level, or worry about the author and license fields for now. The description field should contain a sentence or two describing what the script does. If anything about the script results might confuse or mislead users, and you can't eliminate the issue by improving the script or results text, it should be documented in the description string. identd-owners script “description” script variable description = [[ Attempts to find the owner of a scanned port. The script makes a connection to the auth port (113) and queries the owner of an open port. ]] The author of a script must decide what categories it belongs to. This script is safesafe script category because we are not using the service for anything it was not intended for. Because this script is one that should run by default it is also in the defaultdefault script category category. “categories” script variable categories = {"default", "safe"} The rule section is a Lua method which decides when the script's action should be performed and when it should be skipped. Usually this decision is based on the host and port information passed to the rule function. In the case of the identification script it is slightly more complicated than that. To decide whether to run the identification script on a given port we need to know if there is an identification server running on the target machine. Or more formally: the script should be run only if the currently scanned TCP port is open and TCP port 113 is also open. For now we will rely on the fact that identification servers listen on TCP port 113. Unfortunately NSE only gives us information about the currently scanned port. To find out if port 113 is open we are going to use the nmap.get_port_state() function. If the identd port was not scanned, the get_port_state function returns nil. So we need to make sure that the table is not nil. We also check if both ports are in the open state. If this is the case, the action is executed, otherwise we skip the action. “portrule” script variable portrule = function(host, port) local auth_port = { number=113, protocol="tcp" } local identd = nmap.get_port_state(host, auth_port) if identd ~= nil and identd.state == "open" and port.protocol == "tcp" and port.state == "open" then return true else return false end end At last we implement the actual functionality. The script will first connect to the port on which we expect to find the identification server, then it will connect to the port we want information about. Afterward we construct a query string and parse the response. If we received a satisfactory response, we return the retrieved information. First we need to create two socket objects. These objects represent the sockets we are going to use. By using object methods like open(), close(), send() or receive() we can operate on the network socket. To avoid excessive error checking code we use NSE's exception handling mechanism.exceptions in NSE We create a function which will be executed if an error occurs and call this function catch. Using this function we generate a try function. The try function will call the catch function whenever there is an error condition in the tried block. “action” script variable action = function(host, port) local owner = "" local client_ident = nmap.new_socket() local client_service = nmap.new_socket() local catch = function() client_ident:close() client_service:close() end local try = nmap.new_try(catch) try(client_ident:connect(host.ip, 113)) try(client_service:connect(host.ip, port.number)) local localip, localport, remoteip, remoteport = try(client_service:get_info()) local request = port.number .. ", " .. localport .. "\n" try(client_ident:send(request)) owner = try(client_ident:receive_lines(1)) if string.match(owner, "ERROR") then owner = nil else owner = string.match(owner, "USERID : .+ : (.+)\n", 1) end try(client_ident:close()) try(client_service:close()) return owner end Note that because we know that the remote port is stored in port.number, we could have ignored the last two return values of client_service:get_info() like this: local localip, localport = try(client_service:get_info()) In this example we avoided telling the user if the service responded with an error. Instead we assigned nil to the owner variable. NSE scripts generally only return messages when they succeed. Nmap Scripting Engine (NSE)documentation in NSEDoc Scripts are used by more than just their author, so scripts must have documentation. NSE modules need documentation so developers can use them in their scripts. NSE's documentation system, described in this section, aims to meet both these needs. While reading this section, you may want to browse NSE's online documentation, which is generated using this system. It is at . NSE uses a customized version of the LuaDocLuaDoc documentation system called NSEDoc. The documentation for scripts and modules is contained in their source code, as comments with a special form. is an NSEDoc comment taken from the stdnse.print_debug() function. --- Prints a formatted debug message if the current verbosity level is greater -- than or equal to a given level. -- -- This is a convenience wrapper around -- <code>nmap.print_debug_unformatted()</code>. The first optional numeric -- argument, <code>verbosity</code>, is used as the verbosity level necessary -- to print the message (it defaults to 1 if omitted). All remaining arguments -- are processed with Lua's <code>string.format()</code> function. -- @param level Optional verbosity level. -- @param fmt Format string. -- @param ... Arguments to format. Documentation comments start with three dashes: ---. The body of the comment is the description of the following code. The first paragraph of the description should be a brief summary, with the following paragraphs giving more detail. Special tags starting with @ mark off other parts of the documentation. In the above example you see @param, which is used to describe each parameter of the function. A complete list of the documentation tags is found in . Text enclosed in the HTML-like <code> and </code> tags will be rendered in a monospace font. This should be used for variable and function names, as well as multi-line code examples. When a sequence of lines start with the characters * , they will be rendered as a bulleted list. It is good practice to document every public function and table in a script or module. Additionally every script and module should have its own file-level documentation. A documentation comment at the beginning of a file (one that is not followed by a function or table definition) applies to the entire file. File-level documentation can and should be several paragraphs long, with all the high-level information useful to a developer using a module or a user running a script. shows documentation for the comm module (with a few paragraphs removed to save space). --- Common communication functions for network discovery tasks like -- banner grabbing and data exchange. -- -- These functions may be passed a table of options, but it's not required. The -- keys for the options table are <code>"bytes"</code>, <code>"lines"</code>, -- <code>"proto"</code>, and <code>"timeout"</code>. <code>"bytes"</code> sets -- a minimum number of bytes to read. <code>"lines"</code> does the same for -- lines. <code>"proto"</code> sets the protocol to communicate with, -- defaulting to <code>"tcp"</code> if not provided. <code>"timeout"</code> -- sets the socket timeout (see the socket function <code>set_timeout()</code> -- for details). -- @author Kris Katterjohn 04/2008 -- @copyright Same as Nmap--See http://nmap.org/book/man-legal.html There are some special considerations when documenting scripts as opposed to functions and modules. Some information that might be put in an @-tag in a comment should go in one of the special script variables instead. (Script variables are described in .) Specifically, the script's description should be in the description variable rather than in a documentation comment, and the information that would go in @author and @copyright should go in the variables author and license instead. NSEDoc knows about these variables and will use them in preference to fields in the comments. Scripts should also have an @output tag showing sample output. shows proper form for script-level documentation, using a combination of documentation comments and NSE variables. description = [[ Maps IP addresses to autonomous system (AS) numbers. The script works by sending DNS TXT queries to a DNS server which in turn queries a third-party service provided by Team Cymru (team-cymru.org) using an in-addr.arpa style zone set up especially for use by Nmap. ]] --- -- @usage -- nmap --script asn-query.nse [--script-args dns=<DNS server>] <target> -- @args dns The address of a recursive nameserver to use (optional). -- @output -- Host script results: -- | AS Numbers: -- | BGP: 64.13.128.0/21 | Country: US -- | Origin AS: 10565 SVCOLO-AS - Silicon Valley Colocation, Inc. -- | Peer AS: 3561 6461 -- | BGP: 64.13.128.0/18 | Country: US -- | Origin AS: 10565 SVCOLO-AS - Silicon Valley Colocation, Inc. -- |_ Peer AS: 174 2914 6461 author = "jah, Michael" license = "Same as Nmap--See http://nmap.org/book/man-legal.html" categories = {"discovery", "external"} NSEDocfor C modules Compiled NSE modules are also documented with NSEDoc, even though they have no Lua source code. Each compiled module has a file modulename.luadoc.luadoc filename extension that is kept in the nselib directory alongside the Lua modules. This file lists and documents the functions and tables in the compiled module as though they were written in Lua. Only the name of each function is required, not its definition (not even end). You must use the @name and @class tags when documenting a table to assist the documentation parser in identifying it. There are several examples of this method of documentation in the Nmap source distribution. This is a list of tags understood by NSEDoc and their purpose. @param Describes a function parameter. The first word following @param is the name of the parameter being described. The tag should appear once for each parameter of the function. @see Adds a cross-reference to another function or table. @return Describes a return value of a function. @return may be used multiple times for multiple return values. @usage Gives an example of the usage of a function or script. In the case of a function, the example is Lua code; for a script it is an Nmap command line. @usage may be given more than once. @name Defines a name for the function or table being documented. This tag is normally not necessary, as NSEDoc infers the name through code analysis. @class Defines the class of the thing being modified: function, table, or module. Like @name, this is normally inferred automatically. @field In the documentation of a table, describes the value of a named field. @args Describes a script argument, as used with the --script-args option (see ). The first word after @args is the name of the argument, and everything following that is the description. This tag is special to script-level comments. @output Shows sample output of a script. This tag is special to script-level comments. @author Lists an author of a module. It may be given more than once. Don't use this tag in script documentation; use the author variable instead. @copyright Describes the copyright of a module. Don't use this tag in script documentation; use the license variable instead. Nmap Scripting Engine (NSE)sample scripts version detectionusing NSE The version detection system built into Nmap was designed to efficiently recognize the vast majority of protocols with a simple pattern matching syntax. Some protocols require a more complex approach though, and a generalized scripting language is perfect for this. NSE's versionversion script category category contains the scripts that enhance standard version detection. Scripts in this category are run whenever you request version detection with -sV; you don't need to use -sC to get version-detection scripts. (This cuts the other way too: if you use -sC you won't get version scripts unless you also use -sV.) This script detects version 2 of the Skype VoIP protocol, one which is difficult to identify with version detection alone. If Skype gets an HTTP GET request, it pretends to be an HTTP server and sends back a 404. But for any other request it sends back a chunk of random-looking data. Proper identification requires sending two probes and comparing the two responses—an ideal task for NSE. description = [[ Detects the Skype version 2 service. ]] author = "Brandon Enright <bmenrigh@ucsd.edu>" license = "Same as Nmap--See http://nmap.org/book/man-legal.html" categories = {"version"} require "comm" portrule = function(host, port) if (port.number == 80 or port.number == 443 or port.service == nil or port.service == "" or port.service == "unknown") and port.protocol == "tcp" and port.state == "open" and port.service ~= "http" and port.service ~= "ssl/http" then return true else return false end end action = function(host, port) local status, result = comm.exchange(host, port, "GET / HTTP/1.0\r\n\r\n", {bytes=26, proto=port.protocol}) if (not status) then return end if (result ~= "HTTP/1.0 404 Not Found\r\n\r\n") then return end -- So far so good, now see if we get random data for another request status, result = comm.exchange(host, port, "random data\r\n\r\n", {bytes=15, proto=port.protocol}) if (not status) then return end if string.match(result, "[^%s!-~].*[^%s!-~].*[^%s!-~]") then -- Detected port.version.name = "skype2" port.version.product = "Skype" nmap.set_port_version(host, port, "hardmatched") return end return end If the script detects Skype, it augments its port table with now-known name and product fields. It then sends this new information to Nmap by calling nmap.set_port_version(). Several other version fields are available to be set if they are known, but in this case we only have the name and product. For the full list of version fields refer to the documentation of nmap.set_port_version(). Notice that if the script does not detect the protocol, it does nothing. This is considered good practice; a script shouldn't produce output (other than debug output) just to say it didn't learn anything. finger script The finger script (finger.nse) is a perfect example of how short typical NSE scripts are. First the information fields are filled out. A detailed description of what the script actually does should go in the description field. description = [[ Attempts to get a list of usernames via the finger service. ]]“description” script variable author = "Eddie Bell <ejlbell@gmail.com>"Bell, Eddie“author” script variable license = "Same as Nmap--See http://nmap.org/book/man-legal.html"“license” script variable The categories field is a table containing all the categories the script belongs to—These are used for script selection through the --script option. categories = {"default", "discovery"} You can use the facilities provided by the nselib () with require. Here we want to use common communication functions and shorter port rules. require "comm" require "shortport" We want to run the script against the finger service. So we test whether it is using the well-known finger port (79/tcp), or whether the service is named finger based on version detection results or in the port number's listing in nmap-services. We want to check whether the service behind the port is finger, or whether it runs on finger's well-known port 79. Through this we can use the information gathered during the version scan (if finger runs on a non-standard port) or still run against at least the port we expect it, should the version detection information not be available. portrule = shortport.port_or_service(79, "finger")“portrule” script variable First, the script uses nmap.new_try() to create an exception handler that will quit the script in case of an error. Next, it passes control to comm.exchange(), which handles the network transaction. Here we have asked to receive no more than around 100 lines, with a timeout of five seconds (5000 ms). Any errors will be handled by the try exception handler. The script returns a string if the call to comm.exchange() was successful. action = function(host, port) local try = nmap.new_try() return try(comm.exchange(host, port, "\r\n", {lines=100, proto=port.protocol, timeout=5000})) end Nmap Scripting Engine (NSE)implementation Now how does all this work? The following section describes some interesting aspects of NSE. While the focus primarily lies on giving script writers a better feeling of what happens with scripts, it should also provide a starting point for understanding (and extending) the NSE sources. During its initialization stage, Nmap loads the Lua interpreter, including its provided libraries. These libraries are documented in the Lua Reference Manual. Here is a summary: The package library (namespace: package)—Lua's package-lib provides (among others) the require function, used to load modules from the nselib. The table library (namespace: table)—The table manipulation library contains many functions used to operate on tables—Lua's central data structure. The I/O library (namespace: io)—The Input/Output library offers functions such as reading files and reading the output from programs you execute. The OS library (namespace: os)—The Operating System library provides facilities of the operating system, including filesystem operations (renaming/removing files, temporary file creation) and access to the environment. The string library (namespace: string)—The string library helps you with functions used to manipulate strings inside Lua. Functions include: printf-style string formatting, pattern matching using Lua-style patterns, substring extraction, etc. The math library (namespace: math)—Numbers in Lua usually correspond to the double C type, so the math library provides access to rounding functions, trigonometric functions, random number generation, and more. The debug library (namespace: debug)—The debug library provides you with a somewhat lower level API to the Lua interpreter. Through it you can access functions along the execution stack, get function closures and object metatables, etc. In addition to loading the libraries provided by Lua, the functions in the nmap namespace are loaded. The search paths are the same directories that Nmap searches for its data files and scripts, except that the nselib directory is appended to each. In this step the provided script arguments are stored inside the registry.registry (NSE) The next phase of NSE initialization is loading the chosen scripts, which are the arguments provided to the --script--script option or default, in case of a default script scan. The string versionversion script category is appended, if version detection was enabled. The arguments afterwards are tried to be interpreted as script categories. This is done via a Lua C function in nse_init.cc called entry. Inside script.db,script.db--script-updatedb for each category of a script, there is a call to Entry. If the category was chosen then the script is loaded. Every argument of --script that could not be interpreted as a category is loaded as a file or directory. If the file or directory could not be located, then an error is raised and the Script Engine aborts. All of the .nse files inside a loaded directory are loaded as files. Each file loaded is executed by Lua. If a portrule is present, then it is saved in the porttests table with a portrule key and file closure value. Otherwise, if the script has a hostrule, then it is saved in the hosttests table in the same manner. After the initialization is finished the hostrules“hostrule” script variable and portrules“portrule” script variable are evaluated for each host in the current target group. At this check a list is built which contains the combinations of scripts and the hosts they will run against. It should be noted that the rules of all chosen scripts are checked against all hosts and their openopen port state and open|filteredopen|filtered port state ports. Therefore it is advisable to leave the rules as simple as possible and to do all the computation inside the action, as a script will only be executed if it is run against a specific target. After the check those script-target combinations get their own Lua thread. A thread running against a host will have only a hostrule passed to the action closure whereas a thread running against a port will have both a hostrule and portrule passed. Each thread is stored with information relevant to the thread. This information includes the runlevel, target, target port (if applicable), host and port tables (passed to action), and its type (running against a host or port). The mainloop function will work on each runlevel grouping of threads in order. Nmap is able to perform NSE script scanning in parallelparallelismin NSE by making use of Lua language features. In particular, coroutines offer collaborative multi-threading so scripts can suspend themselves at defined points, and allow other coroutines to execute. Since network I/O, especially waiting for responses from remote host, is the part of scripts which would consume most time with waiting, this is the point where scripts suspend themselves and let others execute. Each call to some of the functions of the Nsock wrapper causes the calling script to yield (pause). Once the request is processed by the Nsock library, the callback causes the script to be pushed from the waiting queue to the running queue, which will eventually let it resume its operation. The mainloop function will maintain two sets of threads, running and waiting. Threads will be moved back and forth between the sets; when a thread yields, it is moved to the waiting group. Threads run in the running set will either yield, complete, or error. After all scripts are resumed in the running set, mainloop will place all yielded threads ready to be run in the running set. Threads are made "ready" by calling process_waiting2running. This process of running threads and moving paused threads to the waiting and running sets is repeated until no threads exist in either waiting or running. Nmap Scripting Engine (NSE)C modules Some of the modules included in nselib are not written in Lua but in C or C++. bit and pcre are two examples. This section describes how to write your own compiled extensions to nselib. The C API of Lua is described at length in Programming in Lua, Second Edition, Programming in Lua, Second Edition, so this is a short summary. C modules consist of functions that follow the protocol of the lua_CFunction type. The functions are registered with Lua and assembled into a library by calling the luaL_registerluaL_register function. A special initialization function provides the interface between the module and the rest of the NSE code. By convention the initialization function has a name of the form luaopen_module. The smallest compiled module that comes with NSE is bit,bit NSE module and one of the most straightforward is openssl.openssl NSE module These modules serve as good examples for a beginning module writer. The source code for bit is in the files nse_bit.cc and nse_bit.h, and likewise the source for openssl is in nse_openssl.cc and nse_openssl.h. The other compiled modules usually follow this naming convention. Let us look at the openssl module. One of the functions in nse_openssl.cc is l_md5, which calculates an MD5 digest. Its function prototype is static int l_md5(lua_State *L); The prototype shows that l_md5 matches the lua_CFunction type. The function is static because it does not have to be visible to other compiled code, it just needs an address so it can be registered with Lua. Later in the file we see l_md5 entered into an array of type luaL_reg and associated with the name md5, the name it will be known by to NSE: static const struct luaL_reg openssllib[] = { { "md5", l_md5 }, { NULL, NULL } }; Then the library is registered with a call to luaL_register inside the initialization function luaopen_openssl. Some lines relating to the registration of OpenSSL BIGNUM types have been omitted. LUALIB_API int luaopen_openssl(lua_State *L) { luaL_register(L, OPENSSLLIBNAME, openssllib); return 1; } (OPENSSLLIBNAME is just the string "openssl", the name of the module.) luaopen_openssl is the only function in the file that is exposed in nse_openssl.h. Once a compiled module is written, it is added to NSE by including it in the list of standard libraries in nse_init.cc. Just follow the example of the modules that are already there. Then the names of the module's source files of the must be added to Makefile.in in the appropriate places. Again it is easiest to follow the example of the other modules. For the Windows build the new source files must be added to the mswin32/nmap.vcproj project file.