From 1f74b2361cd6093a7c45045d5b508eb14e69960c Mon Sep 17 00:00:00 2001 From: fyodor Date: Sat, 8 Nov 2008 11:39:48 +0000 Subject: [PATCH] Woohoo! Doen editing Ch9. --- docs/scripting.xml | 248 +++++++++++++++++++++++---------------------- 1 file changed, 128 insertions(+), 120 deletions(-) diff --git a/docs/scripting.xml b/docs/scripting.xml index c952f9721..34357b467 100644 --- a/docs/scripting.xml +++ b/docs/scripting.xml @@ -2279,159 +2279,167 @@ end Implementation Details Nmap Scripting Engine (NSE)implementation - Now how does all this work? The following section describes - some interesting aspects of NSE. While the focus primarily lies on - giving script writers a better feeling of what happens with scripts, it - should also provide a starting point for understanding (and extending) the - NSE sources. + Now it is time to get into the nitty gritty details of the NSE + implementation. Understanding how it works is useful for + designing efficient scripts and libraries. The canonical + reference to the NSE implementation is the source code, but + this section provides an overview of key details. It should + be valuable to folks trying to understand and extend the NSE + source code, as well as to script authors who want to + better-understand how their scripts are executed. Initialization Phase - During its initialization stage, Nmap loads the Lua interpreter, including its provided libraries. These libraries are documented in the Lua Reference Manual. Here is a summary: - - - The package library (namespace: - package)—Lua's - package-lib provides (among others) the require function, used to load modules from the - nselib. - - - - The table library (namespace: - table)—The - table manipulation library contains many functions used - to operate on tables—Lua's central data - structure. - - - - The I/O library (namespace: - io)—The - Input/Output library offers functions such as reading files and reading the output from programs you execute. - - - - The OS library (namespace: - os)—The - Operating System library provides facilities of the operating system, including filesystem operations (renaming/removing files, temporary file creation) and access to the environment. - - - The string library (namespace: - string)—The - - string library helps you with functions used to manipulate - strings inside Lua. Functions include: printf-style - string formatting, pattern matching using Lua-style patterns, - substring extraction, etc. - - - - The math library (namespace: - math)—Numbers in Lua usually correspond to the double C type, so the math library provides access to rounding functions, trigonometric functions, random number generation, and more. - - - The debug library (namespace: - debug)—The - debug library provides you with a somewhat lower level API - to the Lua interpreter. Through it you can access functions along - the execution stack, get function closures and object metatables, - etc. - - - + During its initialization stage, Nmap loads the Lua interpreter and its provided libraries. These libraries are fully documented in the Lua Reference Manual. Here is a summary of the libraries, listed alphabetically by their namespace name: + + + + debug + + The debug library provides a low-level API to the Lua interpreter, allowing you to access functions along + the execution stack, retrieve function closures and object metatables, + and more. + + + -In addition to loading the libraries provided by Lua, the functions in the nmap namespace are loaded. The search paths are the same directories that Nmap searches for its data files and scripts, except that the nselib directory is appended to each. In this step the provided script arguments are stored inside the registry.registry (NSE) + + io + + The Input/Output library offers functions such as reading from files or from the output from programs you execute. + + + + + + math + + Numbers in Lua usually correspond to the double C type, so the math library provides access to rounding functions, trigonometric functions, random number generation, and more. + + + + + + os + + The + Operating System library provides system facilities such as filesystem operations (including file renaming or removal and temporary file creation) and system environment access. + + + + + + package + + Among the functions provided by Lua's + package-lib is require, which is used to load nselib modules. + + + + + + string + + The + string library provides functions for manipulating + Lua strings, including printf-style + string formatting, pattern matching using Lua-style patterns, + substring extraction, and more. + + + + + + table + + The + table manipulation library is essential for operating on Lua's central data structure (tables). + + + + + + +In addition to loading the libraries provided by Lua, +the nmap namespace functions are loaded. The +search paths are the same directories that Nmap searches for its data +files, except that the nselib directory +is appended to each. At this stage any provided script arguments are +stored inside the registry.registry +(NSE) - The next phase of NSE initialization is loading the chosen - scripts, which are the arguments provided to the + The next phase of NSE initialization is loading the selected + scripts, based on the defaults or arguments provided to the - option or default, in - case of a default script scan. The string + option. The versionversion script category - is appended, if version detection was enabled. - The arguments afterwards are tried to be - interpreted as script categories. This is done via a Lua C function - in nse_init.cc called entry. - Inside script.db,script.db - for each category of a script, - there is a call to Entry. If the category was chosen - then the script is loaded. Every argument of - that could not be interpreted as a category - is loaded as a file or directory. If the file or directory could not - be located, then an error is raised and the Script Engine aborts. - + category scripts are loaded as well if version detection was enabled. + NSE first tries to interpret each argument as a category. + This is done with a Lua C function + in nse_init.cc named entry based on data from + the script.db script categorization database.script.db + If the category is found, those scripts are loaded. + Otherwise Nmap tries to interpret arguments as + files or directories. If no files or directories with a given name are found in Nmap's search path, + an error is raised and the Script Engine aborts. + - All of the .nse files inside a loaded directory are - loaded as files. Each file loaded is executed by Lua. If a - portrule is present, then it is saved in the + If a directory is specified, all of the .nse files inside it are + loaded. Each loaded file is executed by Lua. If a + portrule is present, it is saved in the porttests table with a portrule key and file closure value. Otherwise, if the script has a - hostrule, then it is saved in the hosttests table + hostrule, it is saved in the hosttests table in the same manner. - Matching of Scripts to Targets + Matching Scripts with Targets - After the initialization is finished the + After initialization is finished, the hostrules“hostrule” script variable and portrules“portrule” script variable are evaluated for each host in the current - target group. At this check a list is built which contains the combinations of scripts and the hosts they will run against. - -It should be noted that the rules of all chosen scripts are -checked against all hosts and their -openopen port state + target group. + The rules of every chosen script is tested against every host and (in the case of service scripts) each openopen port state and open|filteredopen|filtered port state -ports. -Therefore it is advisable to leave the rules as simple as possible and -to do all the computation inside the action, as a script will only be -executed if it is run against a specific target. After the check those script-target combinations -get their own Lua thread. A -thread running against a host will have only a hostrule passed to the action closure whereas -a thread running against a port will have both a hostrule and portrule passed. Each thread -is stored with information relevant to the thread. This information -includes the runlevel, target, target port (if applicable), host and port tables -(passed to action), and its type (running against a host or port). -The mainloop function will work on each runlevel grouping of threads in order. +port on the hosts. The combination can grow quite large, so portrules should be kept as simple as possible. Save any heavy computation for the script's action. + +Next, a Lua thread is created for each of the matching script-target combinations. Each thread +is stored with pertinent information such as the runlevel, target, target port (if applicable), host and port tables +(passed to the action), and the script type (service or host script). +The mainloop function then processes each runlevel grouping of threads in order. - - Running Scripts + + Script Execution - Nmap is able to perform NSE script scanning in + Nmap performs NSE script scanning in parallelparallelismin NSE - by making use of Lua language features. In particular, + by taking advantage of Nmap's Nsock parallel I/O library and the Lua coroutines - offer collaborative multi-threading so scripts can suspend themselves at defined points, and allow other coroutines - to execute. Since network I/O, especially waiting for responses from - remote host, is the part of scripts which would consume most time with - waiting, this is the point where scripts suspend themselves and let - others execute. Each call to some of the functions of the Nsock wrapper - causes the calling script to yield (pause). Once the request is - processed by the Nsock library, the - callback causes the script to be pushed from the waiting queue to the - running queue, which will eventually let it resume its operation. + language feature. Coroutines offer collaborative multi-threading so that scripts can suspend themselves at defined points and allow other coroutines to execute. Network I/O, particularly waiting for responses from + remote hosts, often involves long wait times, so + this is when scripts yield to others. + Key functions of the Nsock wrapper + cause scripts to yield (pause). When Nsock finishes processing such a request, it makes a callback + which causes the script to be pushed from the waiting queue back into the + running queue so it can resume operations when its turn comes up again. - The mainloop function will maintain two sets of threads, running and - waiting. Threads will be - moved back and forth between the sets; when a thread yields, it - is moved to the waiting group. Threads run in the running set will either - yield, complete, or error. After all scripts are resumed in the running - set, mainloop will place all yielded threads ready to be - run in the running set. Threads are made "ready" by calling - process_waiting2running. This process of running - threads and moving paused threads to the waiting and running sets is - repeated until no threads exist in either waiting or running. - + The mainloop function moves threads between the waiting and running queues as needed. + A thread which yields is moved from the running queu into the waiting list. Running threads execute until they either + yield, complete, or fail with an error. Threads are made ready to run (placed in the running queue) by calling + process_waiting2running. This process of scheduling running + threads and moving threads between queues continues + until no threads exist in either queue.