diff --git a/docs/scripting.xml b/docs/scripting.xml index d3300d1b4..42a6bb350 100644 --- a/docs/scripting.xml +++ b/docs/scripting.xml @@ -221,10 +221,20 @@ Nmap done: 1 IP address (1 host up) scanned in 0.33 seconds script categories NSE scripts define a list of categories they belong to. - Currently defined categories - are auth, default, discovery, external, fuzzer, -intrusive, malware, safe, version, -and vuln. Category names are not case sensitive. The following list describes each category. + Currently defined categories are + auth, + default. + discovery, + dos, + exploit, + external, + fuzzer, + intrusive, + malware, + safe, + version, and + vuln. + Category names are not case sensitive. The following list describes each category. @@ -314,6 +324,28 @@ and vuln. Category names are not case sensitive. The follow + + + “dos” script category + + + + Scripts in this category may cause denial of service, usually + because they crash a service as a side effect of testing it for a + vulnerability. + + + + + + “exploit” script category + + + + These scripts aim to actively exploit some vulnerability. + + + “external” script category @@ -431,7 +463,7 @@ and vuln. Category names are not case sensitive. The follow These scripts check for specific known vulnerabilities and - generally only report results if they are found. Examples include realvnc-auth-bypass and xampp-default-auth. + generally only report results if they are found. Examples include realvnc-auth-bypass and afp-path-vuln. @@ -441,7 +473,7 @@ and vuln. Category names are not case sensitive. The follow Command-line Arguments - These are the five command line arguments specific to script-scanning: + These are the five command-line arguments specific to script scanning: @@ -754,7 +786,7 @@ Nmap script database, but should be used cautiously since Nmap may contain explo example of -nmap -sC --script-args 'user=foo,pass=",{}=bar",whois={whodb=nofollow+ripe},userdb=C:\Some\Path\To\File' +nmap -sC --script-args 'user=foo,pass=",{}=bar",whois={whodb=nofollow+ripe},userdb=custom' Notice that the script arguments are surrounded in single quotes. For the @@ -763,13 +795,32 @@ Nmap script database, but should be used cautiously since Nmap may contain explo require you to escape quotes or to use different quotes. See your relevant manual. The command results in this Lua table: -{user="foo",pass=",{}=bar",whois={whodb="nofollow+ripe"},userdb="C:\\Some\\Path\\To\\File"} +nmap.registry.args = { + user = "foo", + pass = ",{}=bar", + whois = { + whodb = "nofollow+ripe" + }, + userdb="custom" +} You could then access the username "foo" inside your script with this statement: local username = nmap.registry.args.user + + + All script arguments share a global namespace, the + nmap.registry.args table. For this reason, short or + ambiguous names like user are not recommended. Some + scripts prefix their arguments with their script name, like + smtp-open-relay.domain. Others, like + whois in the example above, take their arguments in a + table named after the script. Arguments used by libraries, which can + affect many scripts, usually have names beginning with the name of the + library, like smbuser and + snmpcommunity. The online NSE Documentation Portal at Script Format - NSE scripts consist of two–five descriptive fields along with either a port or host rule defining when the script should be executed and an action block containing the actual script instructions. Values can be assigned to the descriptive fields just as you would assign any other Lua variables. Their names must be lowercase as shown in this section. + NSE scripts consist of a handful of descriptive fields, either a portrule or a hostrule defining when the script should be executed, and an action function containing the actual script instructions. Values can be assigned to the descriptive fields just as you would assign any other Lua variables. Their names must be lowercase as shown in this section. <literal>description</literal> Field “description” script variable The description field describes what a script is testing - for and any important notes the user should be aware of. Depending on script complexity, the description may vary from a few sentences to a few paragraphs. The first paragraph should be a brief synopsis of the script function suitable for stand-alone presentation to the user. Further paragraphs may provide much more script detail. + for and any important notes the user should be aware of. Depending on script complexity, descriptions may vary in length from a few sentences to a few paragraphs. The first paragraph should be a brief synopsis of the script function suitable for stand-alone presentation to the user. Further paragraphs may provide much more script detail. @@ -864,7 +915,7 @@ categories = {"default", "discovery", "safe"} <literal>author</literal> Field “author” script variable - The author field contains the script authors' names and can also contain contact information (such as home page URLs). We no longer recommend including email addresses because spammers might scrape them from the nsedoc web site. This optional field is not used by NSE, but gives script authors their due credit or blame. + The author field contains the script authors' names and can also contain contact information (such as home page URLs). We no longer recommend including email addresses because spammers might scrape them from the NSEDoc web site. This optional field is not used by NSE, but gives script authors their due credit or blame. @@ -898,20 +949,18 @@ that. script dependencies The dependencies field is an array containing the - names of scripts that should run before this script. This is used when + names of scripts that should run before this script, if they are also selected. This is used when one script can make use of the results of another. For example, most of the smb-* scripts depend on smb-brute,smb-brute.nse because the accounts found by smb-brute may allow - the other scripts to get more information. - - - When we say depend on, we mean it in a loose sense. That is, a - script will still run despite missing dependencies. Given the - dependencies, the script will run after all the scripts listed in the - dependencies array. This is an example of the - dependencies table from - smb-os-discovery:smb-os-discovery.nse + the other scripts to get more information. Listing a script in + dependencies doesn't cause that script to be run; it + still has to be selected through the option + or otherwise. dependencies merely forces an ordering + among the scripts that are selected. This is an + example of a dependencies table, from + smb-os-discovery:smb-os-discovery.nse dependencies = {"smb-brute"} @@ -950,16 +999,16 @@ NSE: Script Scanning completed. Nmap uses the script rules to determine whether a script should be run -against a target. A script contains either a port -rule, which governs which ports of a target the scripts may -run against, or a host rule, which specifies that +against a target. A script contains either a portrule +function, which governs which ports of a target the scripts may +run against, or a hostrule function, which specifies that the script should be run only once against a target IP and only if the given conditions are met. A rule is a Lua function that returns either true or false. The -script action is only performed if its rule -evaluates to true. Host rules accept a host +script action function is only performed if the rule +evaluates to true. Hostrules accept a host table as their argument and may test, for example, the IP address or -hostname of the target. A port rule accepts both host and port tables +hostname of the target. Portrules accept both host and port tables as arguments for any TCP or UDP port in the openopen port state, open|filteredopen|filtered port state, @@ -971,7 +1020,7 @@ or unfilteredunfiltered The action is the heart of an NSE script. It contains all of the -instructions to be executed when the script's port or host rule +instructions to be executed when the script's portrule or hostrule triggers. It is a Lua function which accepts the same arguments as the rule and can return either nil or a string. If a string is returned by a service script, the string and script's filename are printed in the Nmap port table output. A string returned by a host script is printed below the port table. No output is produced if the script returns nil. For an example of an NSE @@ -987,7 +1036,7 @@ action refer to . The core of the Nmap Scripting Engine is an embeddable Lua interpreter. Lua is a lightweight language designed for - extensibility. It offers a powerful and well documented API for + extensibility. It offers a powerful and well-documented API for interfacing with other software such as Nmap. Nmap Scripting Engine (NSE)library @@ -1005,7 +1054,7 @@ action refer to . The Nmap scripting language is an embedded Lua interpreter which was + url="http://www.lua.org/">Lua interpreter which is extended with libraries for interfacing with Nmap. The Nmap API is in the Lua namespace nmap. This means that all calls to resources provided by Nmap have an @@ -1081,9 +1130,7 @@ action refer to . script writing more powerful and convenient. These libraries (sometimes called modules) are compiled if necessary and installed along with Nmap. They have their own directory, nselib, which is installed in the configured Nmap data directory. Scripts need only - - require - the default libraries in order to use them. + require the default libraries in order to use them. @@ -1100,26 +1147,15 @@ action refer to . Hacking NSE Libraries - Libraries often accidentally make use of globals variables when local - scope was intended. Two or more scripts that make use of library - functions which unintentionally use the same global variable will - find that variable constantly rewritten. This is a serious bug that - can cause NSE to stall or a correct script to spectacularly fail, - and, because Lua uses global-by-default scope assignment when it - encounters a variable, this is also a common bug. + A common mistake when editing libraries is to accidentally use a + global variable instead of a local one. Different libraries using the + same global variable can be the cause of mysterious bugs. Lua's scope + assignment is global by default, so this mistake is easy to make. - Consider a global variable being used by two different scripts, - within the library, to hold sockets or data. When one script is - yielded after storing data in the variable, another script awakens - only to replace that data. In contrast, a local variable would store - the information on the stack of the running script separate from - others. - - - To help correct this problem, NSE now uses an adapted library from + To help correct this problem, NSE uses a library adapted from the standard Lua distribution called - strict.lua.strict.lua + strict.lua.strict NSE library The library will raise a runtime error on any access or modification of a global variable which was undeclared in the file scope. A global variable is @@ -1237,7 +1273,7 @@ LUALIB_API int luaopen_openssl(lua_State *L) { NSE scripts have access to several Nmap facilities for writing flexible and elegant scripts. The API provides target host details such as port states and version detection results. It - also offers an interface to the NsockNsock + also offers an interface to the NsockNsockin NSE library for efficient network I/O. @@ -1296,8 +1332,8 @@ LUALIB_API int luaopen_openssl(lua_State *L) { Contains a string representation of the IP address of the - target host. If the scan was run against a host name and the - reverse DNS query returned more than one IP addresses then the + target host. If the scan was run against a host name and its + DNS lookup returned more than one IP addresses, then the same IP address is used as the one chosen for the scan. @@ -1465,35 +1501,35 @@ LUALIB_API int luaopen_openssl(lua_State *L) { - name + name Contains the service name Nmap decided on for the port. - name_confidence + name_confidence Evaluates how confident Nmap is about the accuracy of name, from 1 (least confident) to 10. - product, version, extrainfo, hostname, ostype, devicetype + product, version, extrainfo, hostname, ostype, devicetype These five variables are the same as those described under versioninfo in . - service_tunnel + service_tunnel Contains the string "none" or "ssl" based on whether or not Nmap used SSL tunneling to detect the service. - service_fp + service_fp The service fingerprint, if any, is provided in this value. This is described in . - rpc_status + rpc_status Contains a string value of good_prog if we were able to determine the program number of an RPC service @@ -1505,7 +1541,7 @@ LUALIB_API int luaopen_openssl(lua_State *L) { - rpc_program, rpc_lowver, rpc_highver + rpc_program, rpc_lowver, rpc_highver The detected RPC program number and the range of version numbers supported by that program. These will be nil if rpc_status is @@ -1682,7 +1718,7 @@ socket:close() the base Lua language. It is tailored specifically for network I/O operations, and follows a functional programming paradigm rather than an - object oriented one. The nmap.new_try API method is used to + object-oriented one. The nmap.new_try API method is used to create an exception handler. This method returns a function which takes a variable number of arguments that are assumed to be the return values of another function. If an exception is detected in the return @@ -1703,7 +1739,7 @@ socket:close() script aborts without further ado—open sockets will remain open until the next run of Lua's garbage collector. If the verbosity level is at least one or if the - scan is performed in debugging mode a description of the + scan is performed in debugging mode, a description of the uncaught error condition is printed on standard output. Note that it is currently not easily possible to group several statements in one try block. @@ -1733,7 +1769,7 @@ try(socket:send(result)) try/catch mechanism is straightforward. The function should return multiple values. The first value should be a Boolean which is true upon successful completion of the function and - false otherwise. If the function completed successfully, the try + false (or nil) otherwise. If the function completed successfully, the try construct consumes the indicator value and returns the remaining values. If the function failed then the second returned value must be a string describing the error @@ -1918,19 +1954,13 @@ categories = {"default", "safe"} “portrule” script variable portrule = function(host, port) - local auth_port = { number=113, protocol="tcp" } - local identd = nmap.get_port_state(host, auth_port) + local auth_port = { number=113, protocol="tcp" } + local identd = nmap.get_port_state(host, auth_port) - if - identd ~= nil - and identd.state == "open" - and port.protocol == "tcp" - and port.state == "open" - then - return true - else - return false - end + return identd ~= nil + and identd.state == "open" + and port.protocol == "tcp" + and port.state == "open" end @@ -1938,7 +1968,7 @@ end - The Mechanism + The Action At last we implement the actual functionality! The script first connects to the port on which we expect to find the @@ -1977,7 +2007,7 @@ action = function(host, port) local localip, localport, remoteip, remoteport = try(client_service:get_info()) - local request = port.number .. ", " .. localport .. "\n" + local request = port.number .. ", " .. localport .. "\r\n" try(client_ident:send(request)) @@ -1986,7 +2016,8 @@ action = function(host, port) if string.match(owner, "ERROR") then owner = nil else - owner = string.match(owner, "USERID : .+ : (.+)\n", 1) + owner = string.match(owner, + "%d+%s*,%s*%d+%s*:%s*USERID%s*:%s*.+%s*:%s*(.+)\r?\n") end try(client_ident:close()) @@ -2045,15 +2076,16 @@ local localip, localport = try(client_service:get_info()) An NSEDoc comment for a function ---- Prints a formatted debug message if the current verbosity level is greater +--- +-- Prints a formatted debug message if the current verbosity level is greater -- than or equal to a given level. -- -- This is a convenience wrapper around --- <code>nmap.print_debug_unformatted()</code>. The first optional numeric --- argument, <code>verbosity</code>, is used as the verbosity level necessary +-- <code>nmap.log_write</code>. The first optional numeric +-- argument, <code>level</code>, is used as the debugging level necessary -- to print the message (it defaults to 1 if omitted). All remaining arguments --- are processed with Lua's <code>string.format()</code> function. --- @param level Optional verbosity level. +-- are processed with Lua's <code>string.format</code> function. +-- @param level Optional debugging level. -- @param fmt Format string. -- @param ... Arguments to format. @@ -2077,7 +2109,8 @@ local localip, localport = try(client_service:get_info()) monospace font. This should be used for variable and function names, as well as multi-line code examples. When a sequence of lines start with the characters * , they will - be rendered as a bulleted list. + be rendered as a bulleted list. Each list item must be entirely on + one pysical line. @@ -2097,7 +2130,8 @@ local localip, localport = try(client_service:get_info()) An NSEDoc comment for a module ---- Common communication functions for network discovery tasks like +--- +-- Common communication functions for network discovery tasks like -- banner grabbing and data exchange. -- -- These functions may be passed a table of options, but it's not required. The @@ -2106,8 +2140,9 @@ local localip, localport = try(client_service:get_info()) -- a minimum number of bytes to read. <code>"lines"</code> does the same for -- lines. <code>"proto"</code> sets the protocol to communicate with, -- defaulting to <code>"tcp"</code> if not provided. <code>"timeout"</code> --- sets the socket timeout (see the socket function <code>set_timeout()</code> +-- sets the socket timeout (see the socket function <code>set_timeout</code> -- for details). +-- -- @author Kris Katterjohn 04/2008 -- @copyright Same as Nmap--See http://nmap.org/book/man-legal.html @@ -2141,7 +2176,17 @@ Maps IP addresses to autonomous system (AS) numbers. The script works by sending DNS TXT queries to a DNS server which in turn queries a third-party service provided by Team Cymru (team-cymru.org) using an in-addr.arpa style zone set up especially for -use by Nmap. +use by Nmap. The responses to these queries contain both Origin and Peer +ASNs and their descriptions, displayed along with the BGP Prefix and +Country Code. The script caches results to reduce the number of queries +and should perform a single query for all scanned targets in a BGP +Prefix present in Team Cymru's database. + +Be aware that any targets against which this script is run will be sent +to and potentially recorded by one or more DNS servers and Team Cymru. +In addition your IP address will be sent along with the ASN to a DNS +server (your default DNS server, or whichever one you specified with the +<code>dns</code> script argument). ]] --- @@ -2150,7 +2195,7 @@ use by Nmap. -- @args dns The address of a recursive nameserver to use (optional). -- @output -- Host script results: --- | AS Numbers: +-- | asn-query: -- | BGP: 64.13.128.0/21 | Country: US -- | Origin AS: 10565 SVCOLO-AS - Silicon Valley Colocation, Inc. -- | Peer AS: 3561 6461 @@ -2160,7 +2205,7 @@ use by Nmap. author = "jah, Michael" license = "Same as Nmap--See http://nmap.org/book/man-legal.html" -categories = {"discovery", "external"} +categories = {"discovery", "external", "safe"} @@ -2315,52 +2360,51 @@ categories = {"discovery", "external"} Script Parallelism in NSE - Before now, we have only lightly touched on the steps NSE takes to allow - multiple scripts to execute in parallel. Usually, the author need not - concern himself with how any of this is implemented; however, there are a - couple cases that warrant discussion that we will cover in this section. - As a script writer, you may need to control how multiple scripts interact - in a library; you may require multiple threads to work in parallel; or - perhaps you need to serialize access to a remote resource. + In , it was mentioned that NSE + automatically parallelizes network operations. Usually this process is + transparent to a script author, but there are some advanced techniques + that require knowledge of how it works. The techniques covered in this + section are controlling how multiple scripts interact in a library, using + multiple threads in parallel, and disabling parallelism for special + cases. The standard mechanism for parallel execution is a thread. A thread - encapsulates execution flow and data of a script using the Lua - thread or coroutine. A Lua thread - allows us to yield the current script at arbitrary points to continue - work on another script. Typically, these yield points are blocking calls - to the NSE Socket library. The yield back to NSE is also transparent; the - script is unaware of the transition and views each socket method as a - blocking call. + encapsulates the execution flow and data of a script. + Lua thread may be yielded at arbitrary locations to continue + work on another script. Typically, these yield locations are blocking + socket operations in the + nmapnmap NSE library + library. The yield back to the script is also transparent, a side effect + of the socket operation. Let's go over some common terminology. A script is analogous to a binary executable; it holds the information necessary to - execute our script. A thread (a Lua coroutine) is + execute a script. A thread (a Lua coroutine) is analogous to a process; it runs a script against a host and possibly - port. We sometimes abuse our terminology throughout the book by referring - to a thread as a running script. We are really saying the "instantiation - of the script", in the same sense that a process is the instantiation of - an executable. + port. Sometimes we abuse terminology and refer to a running thread + as a running script, but what this really means is an + instantiation of a script, in the same way that a process is the + instantiation of an executable. - NSE provides the bare-bone essentials you need to expand your degree - of parallelism beyond the basic script thread: new independent threads, - Mutexes, and Condition Variables. We will go into depth on each of - these mechanisms in the following sections. + NSE provides the bare-bone essentials needed to expand parallelism + basic model of one thread per script: new independent threads, + mutexes, and condition variables. Worker Threads There are several instances where a script needs finer control with respect to parallel execution beyond what is offered by default with a - generic script. The common reason for this need is the inability for a - script to read from multiple sockets concurrently. For example, an HTTP + generic script. A common need is to read from multiple sockets + concurrently. For example, an HTTP spidering script may want to have multiple Lua threads querying web - server resources in parallel. To solve this problem, NSE offers the + server resources in parallel. To answer this need, NSE offers the function stdnse.new_thread to create worker threads. These worker threads have all the power of independent scripts with the - only restriction that they may not report Script Output. + only restriction that they may not report script output. Each worker thread launched by a script is given a main function and @@ -2371,51 +2415,50 @@ categories = {"discovery", "external"} worker_thread, status_function = stdnse.new_thread(main, ...) - You are given back the Lua thread (coroutine) that uniquely identifies - your worker thread and a status query function that queries the status - of your new worker. - - + stdnse.new_thread returns two values: the Lua thread + (coroutine) that uniquely identifies your worker thread, and a status + query function that queries the status of your new worker. The status query function returns two values: status, error_object = status_function() - The first return value, status, is simply the return - value of coroutine.status on the worker thread - coroutine (more precisely, the base coroutine, read + The first return value is simply the return + value of coroutine.status run on the worker thread + coroutine. (More precisely, the base coroutine. Read more about base coroutine in ). The second return value contains - the error object thrown that ended the worker thread or + linkend="nse-parallelism-base"/>.) The second return value contains + an error object that caused the termination of the worker thread, or nil if no error was thrown. This object is typically - a string, like most Lua errors. However, recall that any Lua type can - be an error object, even nil! You should + a string, like most Lua errors. However, any Lua type can + be an error object, even nil. Therefore inspect the error object, the second return value, only if the status - of your worker is "dead". + of the worker is "dead". NSE discards all return values from the main function when the worker thread finishes execution. You should communicate with your worker through the use of main function parameters, - upvalues, or function environments. You will see how to do this in - . + upvalues, or function environments. See + + for an example. Finally, when using worker threads you should always use condition - variables and Mutexes to coordinate with your worker threads. Keep in - mind that Nmap is single threaded so there are no (memory) issues in - synchronization to worry about; however, there is resource - contention. Your resources are usually network bandwidth, network - sockets, etc. Condition variables are also useful if the work for any + variables or mutexes to coordinate them. Nmap is single-threaded so + there are no memory synchronization issues to worry about; but there + is is contention for resources. + These resources include usually network bandwidth and sockets. + Condition variables are also useful if the work for any single thread is dynamic. For example, a web server spider script with - a pool of workers will initially have a single root html document. + a pool of workers will initially have a single root HTML document. Following the retrieval of the root document, the set of resources to - be retrieved (the worker's work) will become very large (an html - document adds many new hyperlinks (resources) to fetch). + be retrieved (the worker's work) may become very large as each new + document adds new URLs to fetch. - Worker Thread Example + Worker threads local requests = {"/", "/index.html", --[[ long list of objects ]]} @@ -2455,25 +2498,24 @@ end For brevity, this example omits typical behavior of a traditional web - spider. The requests table is assumed to contain a number of objects - (hundreds or thousands) to warrant the use of worker threads. Our - example will dispatch a new thread with 11 relative - Uniform Resource Identifiers (URI) to request, up to the length of the - requests table. Worker threads are very cheap so we - are not afraid to create a lot of them. After we dispatch this large - number of threads, we wait on our Condition Variable until every thread - has finished then finally return the responses table. + spider. The requests table is assumed to contain enough objects + to warrant the use of worker threads. The code in this example + example dispatches a new thread with as many as 11 relative URLs. + Worker threads are cheap, so don't be afraid to create a lot of them. + After dispatching all these threads, the code waits on a + condition variable until every thread + has finished, then finally return the responses table. You may have noticed that we did not use the status function returned by stdnse.new_thread. You will typically use this for debugging or if your program must stop based on the error thrown by one of your worker threads. Our simple example did not require this but - a fault tolerant library may. + a more fault-tolerant library may. - Thread Mutexes + Mutexes threads in NSE mutexes in NSE @@ -2482,48 +2524,51 @@ end on a target host) yields to other scripts whenever it makes a call on network objects (sending or receiving data). Some scripts require finer concurrency control over thread execution. An example is the - whois script which queries + whoiswhois script + script which queries whoiswhois servers for each - target IP address. Because many concurrent queries often result in - getting one's IP banned for abuse, and because a single query may - return additional information for targets other threads are running - against, it is useful to have other threads pause while one thread + target IP address. Because many concurrent queries can get your + IP banned for abuse, and because a single query may + return the same information another instance of the script is about to + request, it is useful to have other threads pause while one thread performs a query. To solve this problem, NSE includes a mutex function which provides a mutex - (mutual exclusion object) usable by scripts. The Mutex allows for only - one thread to be working on an object. Competing threads waiting to + (mutual exclusion object) usable by scripts. The mutex allows for only + one thread to be working on an object at a time. Competing threads + waiting to work on this object are put in the waiting queue until they can get a - "lock" on the Mutex. A solution for the whois - problem above is to have each thread block on a Mutex using a common - string, thus ensuring that only one thread is querying whois servers at - once. When finished querying the remote servers, the thread can store - results in the NSE registry and unlock the Mutex. Other scripts waiting - to query the remote server can then obtain a lock, check for usable - results retrieved from previous queries, make their own queries, and - unlock the Mutex. This is a good example of serializing access to a + lock on the mutex. A solution for the whois + problem above is to have each thread block on a mutex using a common + string, ensuring that only one thread at a time is querying a server. + When finished querying the remote servers, the thread can store + results in the NSE registry and unlock the mutex. Other scripts waiting + to query the remote server can then obtain a lock, check for the cache + for a usable result from a previous query, make their own queries, and + unlock the mutex. This is a good example of serializing access to a remote resource. - The first step in using a Mutex is to create one via a call to the - nmap library: + The first step in using a mutex is to create one with a call to + nmap.mutex. - - mutexfn = nmap.mutex(object) + +mutexfn = nmap.mutex(object) + The mutexfn returned is a function which works as a - Mutex for the object passed in. This object can be + mutex for the object passed in. This object can be any Lua data type except nil, - booleans, and numbers. The + Boolean, and number. The returned function allows you to lock, try to lock, and release the - Mutex. Its first and only parameter must be one of the + mutex. Its sole argument must be one of the following: @@ -2532,9 +2577,9 @@ end "lock" - Make a blocking lock on the Mutex. If the Mutex is busy (another + Makes a blocking lock on the mutex. If the mutex is busy (another thread has a lock on it), then the thread will yield and - wait. The function returns with the Mutex locked. + wait. The function returns with the mutex locked. @@ -2543,9 +2588,9 @@ end "trylock" - Makes a non-blocking lock on the Mutex. If the Mutex is busy then + Makes a non-blocking lock on the mutex. If the mutex is busy then it immediately returns with a return value of - false. Otherwise the Mutex locks the Mutex and + false. Otherwise, locks the mutex and returns true. @@ -2555,8 +2600,8 @@ end "done" - Releases the Mutex and allows another thread to lock it. If the - thread does not have a lock on the Mutex, an error will be + Releases the mutex and allows another thread to lock it. If the + thread does not have a lock on the mutex, an error will be raised. @@ -2566,8 +2611,8 @@ end "running" - Returns the thread locked on the Mutex or nil - if the Mutex is not locked. This should only be used for + Returns the thread locked on the mutex or nil + if the mutex is not locked. This should only be used for debugging as it interferes with garbage collection of finished threads. @@ -2576,13 +2621,13 @@ end - NSE maintains a weak reference to the Mutex so other calls to + NSE maintains a weak reference to the mutex so other calls to nmap.mutex with the same object will return the same - function (Mutex); however, if you discard your reference to the Mutex - then it may be collected; and, subsequent calls to + mutex function. However, if you discard your reference to the mutex + then it may be collected and subsequent calls to nmap.mutex with the object will return a different - Mutex function! Thus you should save your Mutex to a (local) variable - that persists for the entire time you require. + function. Therefore save your mutex to a (local) variable + that persists as long as you need it. @@ -2609,46 +2654,44 @@ end Condition Variables - Condition Variables arose out of a need to coordinate with worker - threads created using the stdnse.new_thread - function. A Condition Variable allows one or more threads to wait on - an object and one or more threads to awaken one or all threads waiting - on the object. Said differently, multiple threads may unconditionally - block on the Condition Variable by - waiting. Other threads may wake up one or all of - the waiting threads via signalling the Condition - Variable. + Condition variables arose out of a need to coordinate with worker + threads created by the stdnse.new_thread + function. A condition variable allows many threads to wait on + one object, and one or all of them to be awakened when some condition + is met. Said differently, multiple threads may unconditionally + block on the condition variable by + waiting. Other threads may use the condition + variable to wake up the waiting threads. - As an example, we may dispatch multiple worker threads that will - produce results for us to use, like our earlier . Until all - the workers finish, our master thread must sleep. Note that we cannot - poll for results like in a traditional Operating - System thread because NSE does not preempt Lua threads. Instead, - we use a Condition Variable that the master thread + For example, consider the earlier . Until all + the workers finish, the master thread must sleep. Note that we cannot + poll for results like in a traditional operating + system thread because NSE does not preempt Lua threads. Instead, + we use a condition variable that the master thread waits on until awakened by a worker. The master will continually wait until all workers have terminated. - The first step in using a Condition Variable is to create one via a - call to the nmap library: + The first step in using a condition variable is to create one + with a call to nmap.condvar. condvarfn = nmap.condvar(object) - The semantics for Condition Variables are similar to Mutexes. The + The semantics for condition variables are similar to those of mutexes. The condvarfn returned is a function which works as a - Condition Variable for the object passed in. This + condition variable for the object passed in. This object can be any Lua data type except nil, - booleans, and numbers. The + Boolean, and number. The returned function allows you to wait, signal, and broadcast on the - Condition Variable. Its first and only parameter must be one of the + condition variable. Its sole argument must be one of the following: @@ -2657,10 +2700,10 @@ end "wait" - Wait on the Condition Variable. This adds your thread to the - waiting queue for the Condition Variable. You will resume + Wait on the condition variable. This adds the current thread to the + waiting queue for the condition variable. It will resume execution when another thread signals or broadcasts on the - Condition Variable. + condition variable. @@ -2668,8 +2711,8 @@ end "signal" - Signal the Condition Variable. A thread in the Condition - Variable's waiting queue will be resumed. + Signal the condition variable. One of the threads in the condition + variable's waiting queue will be resumed. @@ -2677,78 +2720,75 @@ end "broadcast" - Signal all threads in the Condition Variable's waiting - queue. + Resume all the threads in the condition variable's waiting queue. - Like with Mutexes, NSE maintains a weak reference to the Condition - Variable so other calls to nmap.condvar with the - same object will return the same function (Condition Variable); - however, if you discard your reference to the Condition Variable then - it may be collected; and, subsequent calls to + Like with mutexes, NSE maintains a weak reference to the condition + variable so other calls to nmap.condvar with the + same object will return the same function. However, if you discard your reference to the condition variable then + it may be collected and subsequent calls to nmap.condvar with the object will return a different - Condition Variable function! Thus you should save your Condition - Variable to a (local) variable that persists for the entire time you - require. + function. Therefore save your condition + variable to a (local) variable that persists as long as you need it. - When using Condition Variables, it is important to check the predicate + When using condition variables, it is important to check the predicate before and after waiting. A predicate is a test on whether to continue - doing work within your worker or master thread. For your worker + doing work within a worker or master thread. For worker threads, this will at the very least include a test to see if the master thread is still alive. You do not want to continue doing work - when no thread will use your results. A typical test before waiting - may be: check whether the master is still running, if not then quit; - check that there is work to be done; if not then wait. + when there's no thread to use your results. A typical test before waiting + may be: Check whether the master is still running; if not, then quit. + Check if is work to be done; if not, then wait. - NSE does not guarantee spurious wakeups will not occur; that is, there - is no guarantee your thread will not be awakened when no thread called - "signal" or "broadcast" on the - Condition Variable. The typical, but not only, reason for a spurious - wakeup is the termination of a thread using a Condition Variable. This + A thread waiting on a condition variable may be resumed without any + other thread having called "signal" or + "broadcast" on the condition variable (a spurious + wakeup). + The usual, but not only, reason that this may happen + is the termination of one of the threads using the condition variable. This is an important guarantee NSE makes that allows you to avoid deadlock where a worker or master waits for a thread to wake them up that ended - without signaling the Condition Variable. + without signaling the condition variable. Collaborative Multithreading - One of Lua's least known features is collaborative multithreading + One of Lua's least-known features is collaborative multithreading through coroutines. A coroutine provides an - independent execution stack that is resumable. - The standard coroutine provides access to the + independent execution stack that can be yielded and resumed. + The standard coroutine table provides access to the creation and manipulation of coroutines. Lua's online first - edition of Programming in - Lua contains an excellent introduction to - coroutines. We will provide an overview of the - use of coroutines here for completeness but this is no replacement for - reviewing PiL. + edition of Programming in + Lua contains an excellent introduction to + coroutines. What follows is an overview of the + use of coroutines here for completeness, but this is no replacement for + the definitive reference. We have mentioned coroutines throughout this section as threads. This is the type - (thread) of a coroutine in Lua. Users of NSE that - have any parallel programming experience with Operating System threads - may be confused by this. As a reminder, Nmap is single threaded. Lua - threads provide the basis for parallel scripting but only one thread is - ever running at a time. + ("thread") of a coroutine in Lua. They are not the + preemptive threads that programmers may be expecting. Lua threads + provide the basis for parallel scripting but only one thread is ever + running at a time. - A Lua function executes on top of a Lua - thread. The thread maintains a stack of active - functions, local variables, and the current instruction. We can switch - between coroutines by explicitly yielding the - running thread. The coroutine which resumed the + A Lua function executes on top of a Lua + thread. The thread maintains a stack of active + functions, local variables, and the current instruction pointer. We can switch + between coroutines by explicitly yielding the + running thread. The coroutine which resumed the yielded thread resumes operation. shows a brief use of coroutines to print numbers. @@ -2772,21 +2812,19 @@ end - What you should take from this example is the ability to transfer - between flows of control extremely easily through the use of - coroutine.yield. This is an extremely powerful - concept that enables NSE to run scripts in parallel. All scripts are + Coroutines are the facility + that enables NSE to run scripts in parallel. All scripts are run as coroutines that yield whenever they make a blocking socket function call. This enables NSE to run other scripts and later resume the blocked script when its I/O operation has completed. - As a script writer, there are times when coroutines are the best - tool for a job. One common use in socket programming is to filter - data. You may produce a function that generates all the links from an + Sometimes coroutines are the best + tool for a job within a single script. One common use in socket programming is filtering + data. You may write a function that generates all the links from an HTML document. An iterator using string.gmatch - only catchs a single pattern. Because some complex matches may take + can catches only a single pattern. Because some complex matches may take many different Lua patterns, it is more appropriate to use a coroutine. @@ -2821,24 +2859,18 @@ end - - There are many other instances where coroutines may provide an - easier solution to a problem. It takes experience from use to help - identify those cases. - - - The Base Thread + The base thread Because scripts may use coroutines for their own multithreading, - it is important to be able to identify an owner + it is important to be able to identify the owner of a resource or to establish whether the script is still alive. NSE provides the function stdnse.base for this purpose. Particularly when writing a library that attributes - ownership of a cache or socket to a script, you may use the + ownership of a cache or socket to a script, you can use the base thread to establish whether the script is still running. coroutine.status on the base thread will give the current state of the script. In cases where the script is @@ -2880,7 +2912,8 @@ end One protocol which we were unable to detect with normal version - detection is Skype version 2. The protocol was likely designed to + detection is SkypeSkype + version 2. The protocol was likely designed to frustrate detection out of a fear that telecom-affiliated Internet service providers might consider Skype competition and interfere with the traffic. Yet we did find one way to detect it. If Skype @@ -2898,12 +2931,19 @@ end description = [[ Detects the Skype version 2 service. -]] -author = "Brandon Enright" -license = "Same as Nmap--See http://nmap.org/book/man-legal.html" +]]“description” script variable + +--- +-- @output +-- PORT STATE SERVICE VERSION +-- 80/tcp open skype2 Skype + +author = "Brandon Enright"Enright, Brandon“author” script variable +license = "Same as Nmap--See http://nmap.org/book/man-legal.html"“license” script variable categories = {"version"} require "comm" +require "shortport" portrule = function(host, port) return (port.number == 80 or port.number == 443 or @@ -2911,6 +2951,7 @@ portrule = function(host, port) port.service == "unknown") and port.protocol == "tcp" and port.state == "open" and port.service ~= "http" and port.service ~= "ssl/http" + and not(shortport.port_is_excluded(port.number,port.protocol)) end action = function(host, port) @@ -2975,16 +3016,38 @@ actually does goes in the description field. description = [[ Attempts to get a list of usernames via the finger service. ]]“description” script variable + author = "Eddie Bell"Bell, Eddie“author” script variable + license = "Same as Nmap--See http://nmap.org/book/man-legal.html"“license” script variable The categories field is a table -containing all the categories the script belongs to—These are used for +containing all the categories the script belongs to. These are used for script selection with the option: -categories = {"default", "discovery"} +categories = {"default", "discovery", "safe"}“categories” script variable + + + +Every good script comes with a sample of its output in an NSEDoc comment. + + + +--- +-- @output +-- PORT STATE SERVICE +-- 79/tcp open finger +-- | finger: +-- | Welcome to Linux version 2.6.31.12-0.2-default at linux-pb94.site ! +-- | 01:14am up 18:54, 4 users, load average: 0.14, 0.08, 0.01 +-- | +-- | Login Name Tty Idle Login Time Where +-- | Gutek Ange Gutek *:0 - Wed 06:19 console +-- | Gutek Ange Gutek pts/1 18:54 Wed 06:20 +-- | Gutek Ange Gutek *pts/0 - Thu 00:41 +-- |_Gutek Ange Gutek *pts/4 3 Thu 01:06 You can use the facilities provided by the nselib ( Initialization Phase - NSE is initialized before any scanning when Nmap first starts. We start - this initialization through a call to open_nse. This - procedure starts by creating a fresh Lua state that will persist for the - scans against all host groups. We next load the standard Lua libraries - and all statically compiled NSE libraries. The standard Lua libraries are - fully documented in the open_nse function. open_nse + creates a fresh Lua state that will persist across + host groups,host groupspersistence of NSE through + until the program exits. + It then loads the standard Lua libraries and compiled NSE libraries. + The standard Lua libraries are + documented in the Lua Reference - Manual. Here is a summary of the libraries, listed - alphabetically by their namespace name: + Manual. The standard Lua libraries available to NSE are + debug, + io, + math, + os, + package, + string, and + table. + Compiled NSE libraries are those that are defiend in a C++ file instead + of a Lua file. They include + nmap, + pcre, + bin, + bit, + openssl (if available), and + stdnse.c (C functions for the stdnse library). - - - debug - - The debug library provides a low-level API to the Lua interpreter, allowing you to access functions along - the execution stack, retrieve function closures and object metatables, - and more. - - - - - - io - - The Input/Output library offers functions such as reading from files or from the output from programs you execute. - - - - - - math - - Numbers in Lua usually correspond to the double C type, so the math library provides access to rounding functions, trigonometric functions, random number generation, and more. - - - - - - os - - The - Operating System library provides system facilities such as filesystem operations (including file renaming or removal and temporary file creation) and system environment access. - - - - - - package - - Among the functions provided by Lua's - package-lib is require, which is used to load nselib modules. - - - - - - string - - The - string library provides functions for manipulating - Lua strings, including printf-style - string formatting, pattern matching using Lua-style patterns, - substring extraction, and more. - - - - - - table - - The - table manipulation library is essential for operating on Lua's central data structure (tables). - - - - - The libraries included with NSE are documented in NSEDoc. They include: - nmap, - pcre, - bin, - bit, - ssl (if available), and - stdnse.c (C functions for the - stdnse library). - - - - Following loading basic libraries, NSE loads the file - nse_main.lua. The majority of NSE is written in - Lua -- Lua code manages scripts and sets up the appropriate + After loading the basic libraries, open_nse loads + the file + nse_main.lua. The NSE core is in this + file—Lua code manages scripts and sets up the appropriate environment. In this situation Lua really shines as a glue language. - We use C to provide our network framework and low-level libraries. - We use Lua to structure our data, determine which scripts to load, - and, of course, schedule and execute our scripts. + C++ is used to provide the network framework and low-level libraries. + Lua is used to structure data, determine which scripts to load, + and schedule and execute scripts. - A key feature of Lua we use in NSE is coroutines. Coroutines allow for - collaborative multi-threading so that scripts can suspend themselves at - defined points and allow other coroutines to execute. Network I/O, - particularly waiting for responses from remote hosts, often involves - long wait times, so this is when scripts yield to others. Key functions - of the Nsock wrapper cause scripts to yield (pause). When Nsock - finishes processing such a request, it makes a callback which causes - the script to be pushed from the waiting queue back into the running - queue so it can resume operations when its turn comes up again. Keep - in mind that scripts must explicitly yield (usually within a network - function) to relinquish control. Yielding is never asynchronous. + nse_main.lua sets up the Lua + environment to be ready for script scanning later on. It + loads all the scripts the user has chosen and returns a function + that does the actual script scanning to + to open_nse. - When nse_main.lua is loaded, it sets up the Lua - environment to be ready for script scanning later on. Ultimately, - it will load all scripts the user has chosen and return a function - to nse_main.cc that can be executed to script - scan a host group. - - - - We prepare the Lua environment by adding the nselib - directory the Lua path. This allows NSE Libraries to be required - by scripts. Next NSE loads replacements for the standard - coroutine functions so yields initiated by NSE are caught and - propagated back to the NSE scheduler. + The nselib directory is added to the Lua path to + give scripts access to the standard NSE library. NSE loads replacements + for the standard coroutine functions so that yields initiated by NSE are + caught and propagated back to the NSE scheduler. nse_main.lua next defines classes and functions - to be used during setup (we go over these later). Next, the script - arguments (--script-args) are loaded into - nmap.registry.args. This includes a custom parser - as Lua's patterns are currently insufficient for the task. After - arguments are loaded, we create a new script database if a - pre-existing script database does not exist or a new one was requested. - Our final task during initialization is to load Scripts chosen on - the command line. + to be used during setup. The script arguments + (--script-args) are loaded into + nmap.registry.args. + A script database is created if one doesn't already exist or if this + was requested with . - Our get_chosen_scripts function works to find - the scripts a user chose via categories, file names, and directories. - These scripts will be loaded into memory for later use. The - --script argument is changed to valid Lua code - that is executed. The code generated will dynamically check whether - a script in the database satisfies the boolean equation given on the - command line (recall that --script may take a - boolean expression). Simple categories and filenames will match - immediately causing the script to be "chosen". Other complex expressions - will be determined using Lua's boolean operators. Specifications - given using --script that do not match in this - way are instead checked to be a regular file or directory. If - the specification is a regular file, we load it. If the specification - is a directory, we load all the *.nse files it - contains. Otherwise, we throw an error. + Finally, the scripts listed on the command line are loaded. + The get_chosen_scripts function works to find + chosen scripts by comparing categories, filenames, and directoriy names. + The scripts are loaded into memory for later use. + get_chosen_scripts works by transforming the + argument to --script into a block of Lua code and + then executing it. (This is how the and, + or, and not operators are + supported.) Any specifications that don't directly match a category or + a filename from + script.dbscript.db + are checked against file and directory names. If the specification is a + regular file, it's loaded. If a directory, all the + *.nse files within it are loaded. Otherwise, the + engine raises an error. get_chosen_scripts finishes by arranging the - scripts to run in an ordered way. We do this by sorting the scripts - into runlevels. These runlevels are determined by the dependencies - a script has. Script that have no dependency will run at level - 1 while a script that depends on a runlevel - 1 script will run at level 2. + selected scripts according to their dependencies (see + ). + Scripts that have no dependencies are in runlevel 1. Scripts that + directly depend on these are in runlevel 2, and so on. When a script scan is run, each runlevel is run separately and in order. @@ -3211,12 +3195,12 @@ end nse_main.lua defines two classes: Script and Thread. These classes - are the objects that represent NSE scripts and the script threads we - run. When a script is loaded, we call Script.new - that creates a new Script object. The script file is loaded into Lua + are the objects that represent NSE scripts and their script threads. + When a script is loaded, Script.new + creates a new Script object. The script file is loaded into Lua and saved for later use. These classes and their methods are intended - for encapsulating the data needed for each script and its threads. The - Script.new also contains sanity checks to ensure the + to encapsulate the data needed for each script and its threads. + Script.new also contains sanity checks to ensure that the script has required fields such as the action function. @@ -3226,39 +3210,36 @@ end Scanning a Host Group When NSE runs a script scan, script_scan is called - in nse_main.cc with a Vector of targets to scan. - These targets will be passed to our nse_main.lua + in nse_main.cc with a list of targets to scan. + These targets will be passed to the nse_main.lua main function for scanning. - The main function for a script scan will generate a number of script - threads based on whether a hostrule or - portrule return true for a given host and port. - The threads generated are stored in a list of runlevel lists. Each - runlevel list of threads (starting with runlevel 1) - will be passed to the run function. The - run function is the main worker function for - NSE where all the magic happens. + The main function for a script scan generates a number of script + threads based on whether the hostrule or + portrule returns true for a given host and port. + The generated threads are stored in a list of runlevel lists. Each + runlevel list of threads is passed separately to the + run function. The run function + is the main worker function for NSE where all the magic happens. The run function's purpose is run all the threads - in a runlevel until all finish executing. Before doing this however, - the function starts by redefining some Lua Registry values that + in a runlevel until they all finish. Before doing this however, + run redefines some Lua registry values that help C code function. One such function, _R[WAITING_TO_RUNNING], allows the network library binding written in C to move a thread from the waiting queue to the - running queue. After these functions are defined we begin running - script threads. This involves running script threads until the + running queue. Scripts are run until the running and waiting queues are both empty. Threads that yield are moved to the waiting queue; threads that are ready to continue - are moved back to the running queue. We continue this cycle of - moving a thread between the running and waiting queues until - the thread quits or ends in error. Note that a pending queue exists - as well. It serves as a temporary location for threads moving from - the waiting queue to the running queue before a new iteration of - the running queue begins. + are moved back to the running queue. The cycle continues until + the thread quits or ends in error. Along with the waiting and running + queues, there is a pending queue. It serves as a temporary location + for threads moving from the waiting queue to the running queue before + a new iteration of the running queue begins.