probes that will be sent and calculate from that. This makes the 100%
completion match up closely with the end of the scan, as the estimate gets
better as the scan gets closer to finishing. It also works against filtered
hosts. There is still the related problem that the completion time estimate
uses a global average to estimate completion rate, so it tends to
underestimate.
estimates, and don't consider the number of outstanding probes or number of
retries. The old code would overestimate completion, causing a period of 99.99%
completion at the end of a scan (which caused earlier estimates to be too
soon). For a long UDP scan this puts the completion estimate right on after a
little time to allow for convergence. See
http://www.bamsoftware.com/wiki/Nmap/CompletionTimeEstimates.
the present. This allows the sending rate to temporarily exceed the
maximum to keep from being slowed too much by other delays in the scan
engine. See the discussion at http://seclists.org/nmap-dev/2008/q3/0236.html.
probes, especially IP protocol probes.
Previously if IP protocol ping (-PO) was used anywhere in a host
discovery scan, any response was treated as a protocol response. (The
handlers for other response types had an explicit check for this.) This
means that if you did
nmap -PS -PO
and got back a SYN/ACK in response to the -PS probe, it would be marked
with a reason of proto-response rather than syn-ack. Now, because the IP
protocol response handler matches so broadly, it is given the last
chance at handling a response, only if no interpretation makes sense.
Now the aforementioned scan will give a reason of syn-ack.
The old behavior was not only misleading with respect to reasons, it had
a minor and subtle bug. Consider the following packet trace:
SENT (2.0990s) TCP 192.168.0.21:42205 > target:25 S ttl=40 id=39342 iplen=44 seq=114128202 win=1024 <mss 1460>
SENT (2.2560s) TCP 192.168.0.21:42205 > target:53 S ttl=40 id=51247 iplen=44 seq=114128202 win=1024 <mss 1460>
SENT (2.3280s) TCP 192.168.0.21:42206 > target:25 S ttl=37 id=31111 iplen=44 seq=114062667 win=2048 <mss 1460>
RCVD (2.3530s) TCP target:53 > 192.168.0.21:42205 SA ttl=51 id=0 iplen=44 seq=4159224453 win=5840 ack=114128203 <mss 1460>
ultrascan_host_probe_update called for machine target state UNKNOWN -> HOST_UP (trynum 1 time: 25123)
Ultrascan DROPPED probe packet to target detected
Changing ping technique for target to tcp to port 25; flags: S
Why is the received packet marked as a drop? And why is the ping
technique change to SYN to port 25 when the response came back from port
53? The reason is that the IP protocol response handler caught the probe
and decided it was in response to one of the sent TCP probes--any of the
TCP probes. It selected the probe to port 25 essentially at random and
used that as the relevant probe. The result is that a drop is wrongly
recorded (slowing down the scan), and a worse than useless ping probe is
used (worse than useless because it will cause another drop any time
it's used).
I found this while trying to emulate PortBunny's default ping scan,
which is
-PS80,25,22,443,21,113,23,53,554,3389,445 -PA3333,11 -PE -PP -PU161,162 -PO51
though not in the same order Nmap uses.
o Fixed host discovery probe matching when looking at the returned TCP data in
an ICMP error message. This could lead to incorrectly discarded responses
and the debugging error message: "Bogus trynum or sequence number in ICMP
error message" [Kris]
Fyodor was getting the error message "Got ICMP error with a TCP header that was
too short" while scanning, and looked at the code to see a comment I made about
requiring 12 bytes of TCP data in an ICMP error message instead of the minimum
RFC requirement of 8 bytes.
I made this comment and requirement because tcp_trynum_pingseq_decode() was
being called on the TCP data, and was using the ACK field (which is just past
the 8 byte range). However, upon further inspection, we came to the conclusion
that this code was broken because examining the ACK field should only be done
on a TCP response, not on our own probe (which is what we're looking at in the
ICMP data).
This assumes that -g is used (the only reason that the SEQ/ACK is checked since
the source port number is used otherwise), but the code is also broken without
it because the *_decode() function checks the destination port number rather
than the source port (which should be checked since it's our own probe we're
looking at).
So I've removed the 12-byte requirement and pingseq checking calls, and just
check that the received SEQ number matches the probe SEQ number.
Should we just work with the SEQ/ACK matching when using TCP and leave the
pingseq/trynum port number encoding to UDP? This means behavior won't change
with the use of -g, and it should be guaranteed to be there since we'll only
be looking at whole TCP headers rather than any smaller chunks. Plus, the SEQ
number is already getting encoded with the pingseq/trynum info, we're just not
decoding the ACK responses unless -g is used.
recorded. This applies during a TCP connect scan, where we have a count of
packets sent but not of bytes (which we cannot measure, and will vary from
platform to platform).
-Raw TCP (not SYN to an open port)
-ICMP information queries (echo request, timestamp request, netmask req)
-ARP
-Raw TCP (SYN to an open port)
-UDP, IP protocol, or other ICMP
-TCP connect
-Anything else
The order used to be
-ARP
-Raw TCP (not SYN to an open port)
-UDP, IP protocol, or ICMP
-Raw TCP (SYN to an open port)
-TCP connect
-Anything else
inconvenient to change separately.
The first change fixes a logical error in the storage of timing ping
probes. Each target contains a description of a timing ping probe, which
is stored in the two members
probespec pingprobe;
int pingprobe_state;
pingprobe is the probe itself, and pingprobe_state is the state of the
port that the probe was sent to (PORT_OPEN, PORT_CLOSED, etc.). A change
in the state of the port was a criterion used in deciding whether to
replace the current ping probe.
The problem with this was that pingprobe_state was used to hold a host
state, not a port state, during host discovery. Therefore it held a
value like HOST_DOWN or HOST_UP. This was fine as long as host discovery
and port scanning were separate, but now that timing pings are shared
between those phases the states were in confict: HOST_UP = 1 = PORT_CLOSED.
THis was fixed by using a value of PORT_UNKNOWN during host discovery.
The second change redoes how timing ping probes are replaced. There is
now an order of preference for timing ping probe types, defined by the
function pingprobe_score (and pingprobe_is_better, which calls it). The
order I have defined, from highest preference to lowest, is
ARP
Raw TCP (not SYN to an open port)
UDP, IP protocol, or ICMP
Raw TCP (SYN to an open port)
TCP connect
Anything else
The port state is considered only in raw TCP SYN to an open port, which
is given a lower preference because of the possibility of SYN flooding.
Better ping probes supersede worse ping probes. So in
nmap -PS -sA scanme.nmap.org
the ping probe will be SYN to port 80 after host discovery, but then
will change to ACK to an unfiltered port during port scanning. In
nmap -PA -sS scanme.nmap.org
the ping probe will be ACK to port 80 after host discovery and will
remain that way during port scanning because SYN to an open port is a
worse ping probe. Run with -d2 to see when timing pings change.
o Added the undocumented (except here) --nogcc option which disables
global/group congestion control algorithms and so each member of a
scan group of machines is treated separately. This is just an
experimental option for now. [Fyodor]
to reuse an ACK ping probe from host detection during a SYN port scan. This can
greatly speed up a scan if the SYN scan finds only filtered ports.
One difficulty with implementing this is that not all ping probes are
appropriate for all scan types.
nmap -PA -sU scanme.nmap.org
would cache the ACK ping probe and send ACK pings during the UDP scan. But the
pcap filter for the UDP scan doesn't catch TCP packets, so the replies would
not be noticed and they would show up as dropped pings. Likewise,
nmap -PR -sS 192.168.0.1
would segfault when it tried to use an uninitialized Ethernet descriptor to
send an ARP ping during the SYN scan, which would use raw sockets.
To fix this I added a function pingprobe_is_appropriate that determines whether
a given ping probe is appropriate for the current scan type. If not, the
constructor for HostScanStats just erases the ping probe.
More types of ping probes could be made "appropriate." TCP timing pings work
during a UDP scan if only the pcap filter is expanded to include TCP packets.
In r8541 readip_pcap was given the ability to validate packets, and it also
returns a different length in some cases than it used to:
+ /* OK, since the IP header has been validated, we don't want to tell
+ * the caller they have more packet than they really have. This can
+ * be caused by the Ethernet CRC trailer being counted, for example.
+ */
+ if (*len > ntohs(iphdr->ip_len))
+ *len = ntohs(iphdr->ip_len);
which made some tests having to do with packet length invalid. They were
removed but this one was missed.
from ICMP probes during a protocol scan (protoscanicmphack). I don't know why
it was NULL before, but that's wrong. It was probably never noticed because in
the case of a port update, all that happens is a failure to update the timing.
In the case of a ping probe, it would look like a dropped ping probe, but that
woudl be unlikely because protocol scans usually don't take very long. I
discovered it while testing code to allow ping probes to persist between host
discovery and port scanning.
packet is OK from the get-go rather than running basic checks of it's own.
In a nutshell this patch checks to make sure:
1) there is enough room for an IP header in the amount of bytes read
2) the IP version number is correct
3) the IP length fields are at least as big as the standard header
4) the IP packet received isn't a fragment, or is the initial fragment
5) that next level headers seem reasonable
For TCP, this checks that there is enough room for the header in the number
of bytes read, and that any option lengths are correct. The options checked
are MSS, WScale, SackOK, Sack, and Timestamp.
This also fixes a bug I discovered while testing. Since the Ethernet CRC
(and other datalink-layer data) could be read and counted, it was being
returned that there was more IP packet than there really was. This didn't
cause an overrun of the buffer or anything, just that garbage data could have
easily been read instead of real packet data. Now, if validity is checked for
and the number of total bytes read is larger than the IP's length, the length
is set to the IP header's total length field.
This seems to work great after doing what testing I could. It's been out on
nmap-dev for a couple of weeks without any bad reports (none at all for that
matter). I reviewed this patch again before committing and it looks good as
well.
doAnyOutstandingProbes performance improvements. Here is the log message from
r7914 in nmap-fixed-rate.
Keep a cache of the most recently processed probe for each host in
doAnyOutstandingRetransmits. This greatly reduces the amount of CPU used by
that function when the lists of outstanding probes grow long, such as when a
high scan rate is specified with --min-rate.
This is not most efficient possible way this could be done, but it is a pretty
big win, and it's very non-invasive. The changes are limited entirely to
doAnyOutstandingRetransmits, with no new global state in ultra_scan.
# nmap -d --min-rate 50000 -n -PN -p1-65535 --max-rtt-timeout 500 --max-retries 1 scanme.nmap.org
gprof before:
% cumulative self self total
time seconds seconds calls s/call s/call name
49.74 30.96 30.96 2709 0.01 0.02 doAnyOutstandingRetransmits(UltraScanInfo*)
10.51 37.50 6.54 127256413 0.00 0.00 std::_List_iterator<UltraProbe*>::operator--(int)
gprof after:
% cumulative self self total
time seconds seconds calls s/call s/call name
20.48 3.36 3.36 2667 0.00 0.00 doAnyOutstandingRetransmits(UltraScanInfo*)
16.21 6.02 2.66 2667 0.00 0.00 processData(UltraScanInfo*)
Note that 50000 packets per second is way excessive. I really only get about
6000 in practice. But the point is there is no huge CPU penalty for giving an
excessive rate.
Previously the ping probe data structures were stored in NmapOps,
now they will be stored in the scan_lists struct. All other changes
auxiliary to this reorganization.
only code left in Nmap that still uses rand() is in the Lua math
library. Perhaps at some point we'll need to expose high-quality random
numbers to Lua via our custom nmap library.
guide. They don't honor scan delay and may violate congestion control.
Both this things should be fixed. I was going to do it by having
get_next_target_probe just return the same probe multiple times, and
then either extend struct probespec to include a source address or have
sendIPScanProbe keep track of the decoy index and fill in source
addresses. But I was stopped by timing pings. Those should certainly be
decoyed, but in the code they are just sent as they are needed, and
don't have a dispatching function to modify. What would be good is a
global queue of probes waiting to be sent you could just insert all your
spoofed probes into, and then let the rest of the code take care of
scheduling them.
This change keeps a list of probes awaiting retransmit so that
doAnyOutstandingRetransmits doesn't have to search for them. At high
scan rates this function could take 100 ms or more. Now I have measured
it to take 2 ms or less.
The variable num_probes_waiting_retransmit has been renamed
num_probes_timed_out to better explain its purpose. This list of probes
that can be retransmitted immediately is called
probes_waiting_retransmits, but not all timed-out probes can be
retransmitted immediately. I've done my best to explain the distinction
in comments.
I thought long and hard about how to address this issue, and this is
what I decided on. But of course, every little optimization brings some
complexity and the chance of making a mistake. I'd appreciate someone
taking a look at this change.
foudn that five files can be open on Mac OS X: stdin, stdout, stderr, /dev/tty,
and /private/var/run/utmpx. This could cause a non-root scan at a high scan
rateto fail with the message "Too many open files". I was able to cause this
with "nmap --min-rate 5000 localhost -p-".
That command still fails with the same error message, but for an entirely
different reason. After a while, one of the connect calls fails with an errno of
22 = EINVAL, Invalid argument. Whatever this means, the socket doesn't get
closed, Nmap just reports a "Strange error from connect". The socket is still
open but Nmap doesn't include it in its count of open sockets, so it's off by
one (or more, conceivably). This allows it to try to open one too many sockets
and bomb with an error message.
Note that running as non-root is important both because it uses a connect scan
and because non-root users have a lower limit on open files.
I've tried just closing the socket when EINVAL is returned, and that fixes the
problem. But that's likely to differ on different systems. Plus I don't know why
EINVAL is returned; maybe it's an OS bug. This only affects localhost scans and
only at high scan rates, so I'm leaving it alone.
