i.e., the probe didn't just time out. This was the case before r11253 when I
mistakenly removed it. I meant only to allow ultrascan_adjust_timing to be
called when rcvdtime == NULL, but as a side effect I also allowed
max_successful_tryno to increase in that case too. I was able to get that bit
of code to run with rcvdtime == NULL in a normal scan, but I don't think it
happens often.
I also improved the comments in that block of code.
GroupScanStats::probeSent. This will allow updating timing variables for a
per-host rate limiter.
This fixes a bug, which was that decoy probes were not recorded by the scan
rate meter. Decoy scans would show a lower scan rate than the actual: with four
decoys the rate printed would be 1/5 of actual. This only affects printed
output, not the actual scan rate.
replied to or times out. Previously the scaling factor was increased as soon a
s a probe was sent. So if you sent 10 probes right away at the beginning of the
scan, the first reply that came back would be scaled by 10 (= 10/1, and the
next by 9/2, etc.). Now if every probe is replied to then the scaling factor
will be exactly 1 throughout the scan.
completed hosts from the lists after two minutes. These hosts are
kept around in case there is a late response, but this draws the
line on how long we wait and hence keep this information in memory.
See http://seclists.org/nmap-dev/2008/q3/0902.html for more.
whether a probe response counts as a drop for scan delay purposes. This avoids
an integer overflow in TIMEVAL_SUBTRACT that caused all responses to probes
retransmitted from the retry stack to be counted as drops. This would cause the
scan to grind to a near-halt, with the scan delay at 1000 ms, if even a few
hundred probes were retransmitted from the bench.
Increased max_successful_tryno for 192.168.0.190 to 1 (packet drop)
Increased max_successful_tryno for 192.168.0.190 to 2 (packet drop)
Increasing send delay for 192.168.0.190 from 0 to 5 due to 216 out of 718 dropped probes since last increase.
Increased max_successful_tryno for 192.168.0.190 to 3 (packet drop)
Increasing send delay for 192.168.0.190 from 5 to 10 due to 92 out of 305 dropped probes since last increase.
Increasing send delay for 192.168.0.190 from 10 to 20 due to 11 out of 11 dropped probes since last increase.
Increasing send delay for 192.168.0.190 from 20 to 40 due to 11 out of 11 dropped probes since last increase.
Increasing send delay for 192.168.0.190 from 40 to 80 due to 11 out of 11 dropped probes since last increase.
Increasing send delay for 192.168.0.190 from 80 to 160 due to 11 out of 11 dropped probes since last increase.
Increasing send delay for 192.168.0.190 from 160 to 320 due to 11 out of 11 dropped probes since last increase.
...
The problem was in this bit of code:
if ((!rcvdtime && TIMEVAL_SUBTRACT(probe->sent, hss->sdn.last_boost) > 0) ||
(probe->tryno > 0 && TIMEVAL_SUBTRACT(probe->prevSent, hss->sdn.last_boost) > 0)) {
the TIMEVAL_SUBTRACT(probe->prevSent, hss->sdn.last_boost) > 0) to be specific.
When a probe is retransmitted, the time it was sent is recorded in the prevSent
member of the retransmit probe. prevSent is properly set in retransmitProbe,
but it is not set in sendNextRetryStackProbe, which sends probes that have been
moved from the bench to the retry stack. The problem is that when probes are
moved to the bench they are compressed to probespecs and lose most of their
auxiliary information, like the send time. When they are retransmitted as real
UltraProbe objects, their prevSent message is left initialized to { 0, 0 }.
That led to the integer overflow, with TIMEVAL_SUBTRACT returning a nonsense
(but positive) value.
I fixed it by using TIMEVAL_AFTER(...), which works like
TIMEVAL_SUBTRACT(...) > 0 except that it is immune to integer overflows. Every
other timeval is after { 0, 0 }, so the condition is false for probes
retransmitted from the bench, as it should be. However this is not the most
correct solution. Better would be to somehow store each probe's send time with
it on the bench so it could be restored when it is retransmitted. The way the
bench and the retry stack work makes that cumbersome though, and this is the
only place prevSent is used, so I think this solution is acceptable.
formula algebraically reduced to
1.0 - gstats->numprobes / host->freshPortsLeft(), which doesn't regard the
number of sent probes, leading to long stalls in the completion when, for
example, maxtries is increased.
probes that will be sent and calculate from that. This makes the 100%
completion match up closely with the end of the scan, as the estimate gets
better as the scan gets closer to finishing. It also works against filtered
hosts. There is still the related problem that the completion time estimate
uses a global average to estimate completion rate, so it tends to
underestimate.
estimates, and don't consider the number of outstanding probes or number of
retries. The old code would overestimate completion, causing a period of 99.99%
completion at the end of a scan (which caused earlier estimates to be too
soon). For a long UDP scan this puts the completion estimate right on after a
little time to allow for convergence. See
http://www.bamsoftware.com/wiki/Nmap/CompletionTimeEstimates.
the present. This allows the sending rate to temporarily exceed the
maximum to keep from being slowed too much by other delays in the scan
engine. See the discussion at http://seclists.org/nmap-dev/2008/q3/0236.html.
probes, especially IP protocol probes.
Previously if IP protocol ping (-PO) was used anywhere in a host
discovery scan, any response was treated as a protocol response. (The
handlers for other response types had an explicit check for this.) This
means that if you did
nmap -PS -PO
and got back a SYN/ACK in response to the -PS probe, it would be marked
with a reason of proto-response rather than syn-ack. Now, because the IP
protocol response handler matches so broadly, it is given the last
chance at handling a response, only if no interpretation makes sense.
Now the aforementioned scan will give a reason of syn-ack.
The old behavior was not only misleading with respect to reasons, it had
a minor and subtle bug. Consider the following packet trace:
SENT (2.0990s) TCP 192.168.0.21:42205 > target:25 S ttl=40 id=39342 iplen=44 seq=114128202 win=1024 <mss 1460>
SENT (2.2560s) TCP 192.168.0.21:42205 > target:53 S ttl=40 id=51247 iplen=44 seq=114128202 win=1024 <mss 1460>
SENT (2.3280s) TCP 192.168.0.21:42206 > target:25 S ttl=37 id=31111 iplen=44 seq=114062667 win=2048 <mss 1460>
RCVD (2.3530s) TCP target:53 > 192.168.0.21:42205 SA ttl=51 id=0 iplen=44 seq=4159224453 win=5840 ack=114128203 <mss 1460>
ultrascan_host_probe_update called for machine target state UNKNOWN -> HOST_UP (trynum 1 time: 25123)
Ultrascan DROPPED probe packet to target detected
Changing ping technique for target to tcp to port 25; flags: S
Why is the received packet marked as a drop? And why is the ping
technique change to SYN to port 25 when the response came back from port
53? The reason is that the IP protocol response handler caught the probe
and decided it was in response to one of the sent TCP probes--any of the
TCP probes. It selected the probe to port 25 essentially at random and
used that as the relevant probe. The result is that a drop is wrongly
recorded (slowing down the scan), and a worse than useless ping probe is
used (worse than useless because it will cause another drop any time
it's used).
I found this while trying to emulate PortBunny's default ping scan,
which is
-PS80,25,22,443,21,113,23,53,554,3389,445 -PA3333,11 -PE -PP -PU161,162 -PO51
though not in the same order Nmap uses.
o Fixed host discovery probe matching when looking at the returned TCP data in
an ICMP error message. This could lead to incorrectly discarded responses
and the debugging error message: "Bogus trynum or sequence number in ICMP
error message" [Kris]
Fyodor was getting the error message "Got ICMP error with a TCP header that was
too short" while scanning, and looked at the code to see a comment I made about
requiring 12 bytes of TCP data in an ICMP error message instead of the minimum
RFC requirement of 8 bytes.
I made this comment and requirement because tcp_trynum_pingseq_decode() was
being called on the TCP data, and was using the ACK field (which is just past
the 8 byte range). However, upon further inspection, we came to the conclusion
that this code was broken because examining the ACK field should only be done
on a TCP response, not on our own probe (which is what we're looking at in the
ICMP data).
This assumes that -g is used (the only reason that the SEQ/ACK is checked since
the source port number is used otherwise), but the code is also broken without
it because the *_decode() function checks the destination port number rather
than the source port (which should be checked since it's our own probe we're
looking at).
So I've removed the 12-byte requirement and pingseq checking calls, and just
check that the received SEQ number matches the probe SEQ number.
Should we just work with the SEQ/ACK matching when using TCP and leave the
pingseq/trynum port number encoding to UDP? This means behavior won't change
with the use of -g, and it should be guaranteed to be there since we'll only
be looking at whole TCP headers rather than any smaller chunks. Plus, the SEQ
number is already getting encoded with the pingseq/trynum info, we're just not
decoding the ACK responses unless -g is used.
recorded. This applies during a TCP connect scan, where we have a count of
packets sent but not of bytes (which we cannot measure, and will vary from
platform to platform).
-Raw TCP (not SYN to an open port)
-ICMP information queries (echo request, timestamp request, netmask req)
-ARP
-Raw TCP (SYN to an open port)
-UDP, IP protocol, or other ICMP
-TCP connect
-Anything else
The order used to be
-ARP
-Raw TCP (not SYN to an open port)
-UDP, IP protocol, or ICMP
-Raw TCP (SYN to an open port)
-TCP connect
-Anything else
inconvenient to change separately.
The first change fixes a logical error in the storage of timing ping
probes. Each target contains a description of a timing ping probe, which
is stored in the two members
probespec pingprobe;
int pingprobe_state;
pingprobe is the probe itself, and pingprobe_state is the state of the
port that the probe was sent to (PORT_OPEN, PORT_CLOSED, etc.). A change
in the state of the port was a criterion used in deciding whether to
replace the current ping probe.
The problem with this was that pingprobe_state was used to hold a host
state, not a port state, during host discovery. Therefore it held a
value like HOST_DOWN or HOST_UP. This was fine as long as host discovery
and port scanning were separate, but now that timing pings are shared
between those phases the states were in confict: HOST_UP = 1 = PORT_CLOSED.
THis was fixed by using a value of PORT_UNKNOWN during host discovery.
The second change redoes how timing ping probes are replaced. There is
now an order of preference for timing ping probe types, defined by the
function pingprobe_score (and pingprobe_is_better, which calls it). The
order I have defined, from highest preference to lowest, is
ARP
Raw TCP (not SYN to an open port)
UDP, IP protocol, or ICMP
Raw TCP (SYN to an open port)
TCP connect
Anything else
The port state is considered only in raw TCP SYN to an open port, which
is given a lower preference because of the possibility of SYN flooding.
Better ping probes supersede worse ping probes. So in
nmap -PS -sA scanme.nmap.org
the ping probe will be SYN to port 80 after host discovery, but then
will change to ACK to an unfiltered port during port scanning. In
nmap -PA -sS scanme.nmap.org
the ping probe will be ACK to port 80 after host discovery and will
remain that way during port scanning because SYN to an open port is a
worse ping probe. Run with -d2 to see when timing pings change.
o Added the undocumented (except here) --nogcc option which disables
global/group congestion control algorithms and so each member of a
scan group of machines is treated separately. This is just an
experimental option for now. [Fyodor]
to reuse an ACK ping probe from host detection during a SYN port scan. This can
greatly speed up a scan if the SYN scan finds only filtered ports.
One difficulty with implementing this is that not all ping probes are
appropriate for all scan types.
nmap -PA -sU scanme.nmap.org
would cache the ACK ping probe and send ACK pings during the UDP scan. But the
pcap filter for the UDP scan doesn't catch TCP packets, so the replies would
not be noticed and they would show up as dropped pings. Likewise,
nmap -PR -sS 192.168.0.1
would segfault when it tried to use an uninitialized Ethernet descriptor to
send an ARP ping during the SYN scan, which would use raw sockets.
To fix this I added a function pingprobe_is_appropriate that determines whether
a given ping probe is appropriate for the current scan type. If not, the
constructor for HostScanStats just erases the ping probe.
More types of ping probes could be made "appropriate." TCP timing pings work
during a UDP scan if only the pcap filter is expanded to include TCP packets.
In r8541 readip_pcap was given the ability to validate packets, and it also
returns a different length in some cases than it used to:
+ /* OK, since the IP header has been validated, we don't want to tell
+ * the caller they have more packet than they really have. This can
+ * be caused by the Ethernet CRC trailer being counted, for example.
+ */
+ if (*len > ntohs(iphdr->ip_len))
+ *len = ntohs(iphdr->ip_len);
which made some tests having to do with packet length invalid. They were
removed but this one was missed.
from ICMP probes during a protocol scan (protoscanicmphack). I don't know why
it was NULL before, but that's wrong. It was probably never noticed because in
the case of a port update, all that happens is a failure to update the timing.
In the case of a ping probe, it would look like a dropped ping probe, but that
woudl be unlikely because protocol scans usually don't take very long. I
discovered it while testing code to allow ping probes to persist between host
discovery and port scanning.
packet is OK from the get-go rather than running basic checks of it's own.
In a nutshell this patch checks to make sure:
1) there is enough room for an IP header in the amount of bytes read
2) the IP version number is correct
3) the IP length fields are at least as big as the standard header
4) the IP packet received isn't a fragment, or is the initial fragment
5) that next level headers seem reasonable
For TCP, this checks that there is enough room for the header in the number
of bytes read, and that any option lengths are correct. The options checked
are MSS, WScale, SackOK, Sack, and Timestamp.
This also fixes a bug I discovered while testing. Since the Ethernet CRC
(and other datalink-layer data) could be read and counted, it was being
returned that there was more IP packet than there really was. This didn't
cause an overrun of the buffer or anything, just that garbage data could have
easily been read instead of real packet data. Now, if validity is checked for
and the number of total bytes read is larger than the IP's length, the length
is set to the IP header's total length field.
This seems to work great after doing what testing I could. It's been out on
nmap-dev for a couple of weeks without any bad reports (none at all for that
matter). I reviewed this patch again before committing and it looks good as
well.
doAnyOutstandingProbes performance improvements. Here is the log message from
r7914 in nmap-fixed-rate.
Keep a cache of the most recently processed probe for each host in
doAnyOutstandingRetransmits. This greatly reduces the amount of CPU used by
that function when the lists of outstanding probes grow long, such as when a
high scan rate is specified with --min-rate.
This is not most efficient possible way this could be done, but it is a pretty
big win, and it's very non-invasive. The changes are limited entirely to
doAnyOutstandingRetransmits, with no new global state in ultra_scan.
# nmap -d --min-rate 50000 -n -PN -p1-65535 --max-rtt-timeout 500 --max-retries 1 scanme.nmap.org
gprof before:
% cumulative self self total
time seconds seconds calls s/call s/call name
49.74 30.96 30.96 2709 0.01 0.02 doAnyOutstandingRetransmits(UltraScanInfo*)
10.51 37.50 6.54 127256413 0.00 0.00 std::_List_iterator<UltraProbe*>::operator--(int)
gprof after:
% cumulative self self total
time seconds seconds calls s/call s/call name
20.48 3.36 3.36 2667 0.00 0.00 doAnyOutstandingRetransmits(UltraScanInfo*)
16.21 6.02 2.66 2667 0.00 0.00 processData(UltraScanInfo*)
Note that 50000 packets per second is way excessive. I really only get about
6000 in practice. But the point is there is no huge CPU penalty for giving an
excessive rate.
Previously the ping probe data structures were stored in NmapOps,
now they will be stored in the scan_lists struct. All other changes
auxiliary to this reorganization.
only code left in Nmap that still uses rand() is in the Lua math
library. Perhaps at some point we'll need to expose high-quality random
numbers to Lua via our custom nmap library.
