checking if a pingprobe has been set. We always use PORT_UNKNOWN during host
discovery (rather than HOST_UP or HOST_DOWN) to avoid conflicts with other
PORT_* constants. See the log for r8784.
num_probes_outstanding() == 0. The active probes are probes that haven't timed
out; the outstanding probes includes those and some timed-out probes. When a
host makes it to the completed list with outstanding probes, it is unlikely
they will ever be removed, so we weren't sending any global pings in some
cases. A host only moves to the completed list with it has no active probes,
and any active probes it has after that are global pings.
Also remove only timed-out ping probes from the ping host. All other probes are
timed out but waiting for a response.
me) in r8784 when I made the change to retain ping probes between ping scanning
and port scanning.
It only gets set during host discovery scans, because that's how it was before,
but it might be advantageous to set it during port scans as well.
target. In my testing, TCP scanning certain hosts, other hosts send back ICMP
destination unreachables (admin prohibited) for some ports (the Microsoft
ports). If that's the only response we got from a host, we would make it the
timing ping probe. But the admin-prohibiteds come back at some very slow rate,
much slower than the 1.25/s ping interval. So most of them are dropped, keeping
the congestion window perpetually at 1 and slowly increasing the scan delay. In
a -F scan these hosts could take over twice as long as any other host.
I also put in some missing adjust_timing arguments to the host and ping update
functions whenever adjust_timing could be false (the default is true). I think
those were just oversights. I also made it so that we don't select a new ping
probe when adjust_timing is false; there's no point sending ping probes whose
responses we're going to ignore.
positive response that results in being marked filtered, the reply was an ICMP
error. Such probes are now ranked a 2 along with UDP, IP proto, and other
probes that get ICMP replies that are likely to be rate limited. Previously
they got the highest rank, 6, because they were TCP probes other than SYN to an
open port. SYN to an open port retains its rank of 3.
socket). It's possible to get an EHOSTUNREACH or EHOSTDOWN (and possibly other
errors) against a nonexistent LAN host and formerly the socket wasn't closed.
This allowed Nmap's internal count of sockets to be less that the true count of
open sockets. It would eventually try to open too many sockets and quit with
the message
Socket creation in sendConnectScanProbe: Too many open files (24)
Be aware that in this message 24 is the errno, not the number of open files.
i.e., the probe didn't just time out. This was the case before r11253 when I
mistakenly removed it. I meant only to allow ultrascan_adjust_timing to be
called when rcvdtime == NULL, but as a side effect I also allowed
max_successful_tryno to increase in that case too. I was able to get that bit
of code to run with rcvdtime == NULL in a normal scan, but I don't think it
happens often.
I also improved the comments in that block of code.
GroupScanStats::probeSent. This will allow updating timing variables for a
per-host rate limiter.
This fixes a bug, which was that decoy probes were not recorded by the scan
rate meter. Decoy scans would show a lower scan rate than the actual: with four
decoys the rate printed would be 1/5 of actual. This only affects printed
output, not the actual scan rate.
replied to or times out. Previously the scaling factor was increased as soon a
s a probe was sent. So if you sent 10 probes right away at the beginning of the
scan, the first reply that came back would be scaled by 10 (= 10/1, and the
next by 9/2, etc.). Now if every probe is replied to then the scaling factor
will be exactly 1 throughout the scan.
completed hosts from the lists after two minutes. These hosts are
kept around in case there is a late response, but this draws the
line on how long we wait and hence keep this information in memory.
See http://seclists.org/nmap-dev/2008/q3/0902.html for more.
whether a probe response counts as a drop for scan delay purposes. This avoids
an integer overflow in TIMEVAL_SUBTRACT that caused all responses to probes
retransmitted from the retry stack to be counted as drops. This would cause the
scan to grind to a near-halt, with the scan delay at 1000 ms, if even a few
hundred probes were retransmitted from the bench.
Increased max_successful_tryno for 192.168.0.190 to 1 (packet drop)
Increased max_successful_tryno for 192.168.0.190 to 2 (packet drop)
Increasing send delay for 192.168.0.190 from 0 to 5 due to 216 out of 718 dropped probes since last increase.
Increased max_successful_tryno for 192.168.0.190 to 3 (packet drop)
Increasing send delay for 192.168.0.190 from 5 to 10 due to 92 out of 305 dropped probes since last increase.
Increasing send delay for 192.168.0.190 from 10 to 20 due to 11 out of 11 dropped probes since last increase.
Increasing send delay for 192.168.0.190 from 20 to 40 due to 11 out of 11 dropped probes since last increase.
Increasing send delay for 192.168.0.190 from 40 to 80 due to 11 out of 11 dropped probes since last increase.
Increasing send delay for 192.168.0.190 from 80 to 160 due to 11 out of 11 dropped probes since last increase.
Increasing send delay for 192.168.0.190 from 160 to 320 due to 11 out of 11 dropped probes since last increase.
...
The problem was in this bit of code:
if ((!rcvdtime && TIMEVAL_SUBTRACT(probe->sent, hss->sdn.last_boost) > 0) ||
(probe->tryno > 0 && TIMEVAL_SUBTRACT(probe->prevSent, hss->sdn.last_boost) > 0)) {
the TIMEVAL_SUBTRACT(probe->prevSent, hss->sdn.last_boost) > 0) to be specific.
When a probe is retransmitted, the time it was sent is recorded in the prevSent
member of the retransmit probe. prevSent is properly set in retransmitProbe,
but it is not set in sendNextRetryStackProbe, which sends probes that have been
moved from the bench to the retry stack. The problem is that when probes are
moved to the bench they are compressed to probespecs and lose most of their
auxiliary information, like the send time. When they are retransmitted as real
UltraProbe objects, their prevSent message is left initialized to { 0, 0 }.
That led to the integer overflow, with TIMEVAL_SUBTRACT returning a nonsense
(but positive) value.
I fixed it by using TIMEVAL_AFTER(...), which works like
TIMEVAL_SUBTRACT(...) > 0 except that it is immune to integer overflows. Every
other timeval is after { 0, 0 }, so the condition is false for probes
retransmitted from the bench, as it should be. However this is not the most
correct solution. Better would be to somehow store each probe's send time with
it on the bench so it could be restored when it is retransmitted. The way the
bench and the retry stack work makes that cumbersome though, and this is the
only place prevSent is used, so I think this solution is acceptable.
formula algebraically reduced to
1.0 - gstats->numprobes / host->freshPortsLeft(), which doesn't regard the
number of sent probes, leading to long stalls in the completion when, for
example, maxtries is increased.
probes that will be sent and calculate from that. This makes the 100%
completion match up closely with the end of the scan, as the estimate gets
better as the scan gets closer to finishing. It also works against filtered
hosts. There is still the related problem that the completion time estimate
uses a global average to estimate completion rate, so it tends to
underestimate.
estimates, and don't consider the number of outstanding probes or number of
retries. The old code would overestimate completion, causing a period of 99.99%
completion at the end of a scan (which caused earlier estimates to be too
soon). For a long UDP scan this puts the completion estimate right on after a
little time to allow for convergence. See
http://www.bamsoftware.com/wiki/Nmap/CompletionTimeEstimates.
the present. This allows the sending rate to temporarily exceed the
maximum to keep from being slowed too much by other delays in the scan
engine. See the discussion at http://seclists.org/nmap-dev/2008/q3/0236.html.
probes, especially IP protocol probes.
Previously if IP protocol ping (-PO) was used anywhere in a host
discovery scan, any response was treated as a protocol response. (The
handlers for other response types had an explicit check for this.) This
means that if you did
nmap -PS -PO
and got back a SYN/ACK in response to the -PS probe, it would be marked
with a reason of proto-response rather than syn-ack. Now, because the IP
protocol response handler matches so broadly, it is given the last
chance at handling a response, only if no interpretation makes sense.
Now the aforementioned scan will give a reason of syn-ack.
The old behavior was not only misleading with respect to reasons, it had
a minor and subtle bug. Consider the following packet trace:
SENT (2.0990s) TCP 192.168.0.21:42205 > target:25 S ttl=40 id=39342 iplen=44 seq=114128202 win=1024 <mss 1460>
SENT (2.2560s) TCP 192.168.0.21:42205 > target:53 S ttl=40 id=51247 iplen=44 seq=114128202 win=1024 <mss 1460>
SENT (2.3280s) TCP 192.168.0.21:42206 > target:25 S ttl=37 id=31111 iplen=44 seq=114062667 win=2048 <mss 1460>
RCVD (2.3530s) TCP target:53 > 192.168.0.21:42205 SA ttl=51 id=0 iplen=44 seq=4159224453 win=5840 ack=114128203 <mss 1460>
ultrascan_host_probe_update called for machine target state UNKNOWN -> HOST_UP (trynum 1 time: 25123)
Ultrascan DROPPED probe packet to target detected
Changing ping technique for target to tcp to port 25; flags: S
Why is the received packet marked as a drop? And why is the ping
technique change to SYN to port 25 when the response came back from port
53? The reason is that the IP protocol response handler caught the probe
and decided it was in response to one of the sent TCP probes--any of the
TCP probes. It selected the probe to port 25 essentially at random and
used that as the relevant probe. The result is that a drop is wrongly
recorded (slowing down the scan), and a worse than useless ping probe is
used (worse than useless because it will cause another drop any time
it's used).
I found this while trying to emulate PortBunny's default ping scan,
which is
-PS80,25,22,443,21,113,23,53,554,3389,445 -PA3333,11 -PE -PP -PU161,162 -PO51
though not in the same order Nmap uses.
o Fixed host discovery probe matching when looking at the returned TCP data in
an ICMP error message. This could lead to incorrectly discarded responses
and the debugging error message: "Bogus trynum or sequence number in ICMP
error message" [Kris]
Fyodor was getting the error message "Got ICMP error with a TCP header that was
too short" while scanning, and looked at the code to see a comment I made about
requiring 12 bytes of TCP data in an ICMP error message instead of the minimum
RFC requirement of 8 bytes.
I made this comment and requirement because tcp_trynum_pingseq_decode() was
being called on the TCP data, and was using the ACK field (which is just past
the 8 byte range). However, upon further inspection, we came to the conclusion
that this code was broken because examining the ACK field should only be done
on a TCP response, not on our own probe (which is what we're looking at in the
ICMP data).
This assumes that -g is used (the only reason that the SEQ/ACK is checked since
the source port number is used otherwise), but the code is also broken without
it because the *_decode() function checks the destination port number rather
than the source port (which should be checked since it's our own probe we're
looking at).
So I've removed the 12-byte requirement and pingseq checking calls, and just
check that the received SEQ number matches the probe SEQ number.
Should we just work with the SEQ/ACK matching when using TCP and leave the
pingseq/trynum port number encoding to UDP? This means behavior won't change
with the use of -g, and it should be guaranteed to be there since we'll only
be looking at whole TCP headers rather than any smaller chunks. Plus, the SEQ
number is already getting encoded with the pingseq/trynum info, we're just not
decoding the ACK responses unless -g is used.
recorded. This applies during a TCP connect scan, where we have a count of
packets sent but not of bytes (which we cannot measure, and will vary from
platform to platform).
-Raw TCP (not SYN to an open port)
-ICMP information queries (echo request, timestamp request, netmask req)
-ARP
-Raw TCP (SYN to an open port)
-UDP, IP protocol, or other ICMP
-TCP connect
-Anything else
The order used to be
-ARP
-Raw TCP (not SYN to an open port)
-UDP, IP protocol, or ICMP
-Raw TCP (SYN to an open port)
-TCP connect
-Anything else
inconvenient to change separately.
The first change fixes a logical error in the storage of timing ping
probes. Each target contains a description of a timing ping probe, which
is stored in the two members
probespec pingprobe;
int pingprobe_state;
pingprobe is the probe itself, and pingprobe_state is the state of the
port that the probe was sent to (PORT_OPEN, PORT_CLOSED, etc.). A change
in the state of the port was a criterion used in deciding whether to
replace the current ping probe.
The problem with this was that pingprobe_state was used to hold a host
state, not a port state, during host discovery. Therefore it held a
value like HOST_DOWN or HOST_UP. This was fine as long as host discovery
and port scanning were separate, but now that timing pings are shared
between those phases the states were in confict: HOST_UP = 1 = PORT_CLOSED.
THis was fixed by using a value of PORT_UNKNOWN during host discovery.
The second change redoes how timing ping probes are replaced. There is
now an order of preference for timing ping probe types, defined by the
function pingprobe_score (and pingprobe_is_better, which calls it). The
order I have defined, from highest preference to lowest, is
ARP
Raw TCP (not SYN to an open port)
UDP, IP protocol, or ICMP
Raw TCP (SYN to an open port)
TCP connect
Anything else
The port state is considered only in raw TCP SYN to an open port, which
is given a lower preference because of the possibility of SYN flooding.
Better ping probes supersede worse ping probes. So in
nmap -PS -sA scanme.nmap.org
the ping probe will be SYN to port 80 after host discovery, but then
will change to ACK to an unfiltered port during port scanning. In
nmap -PA -sS scanme.nmap.org
the ping probe will be ACK to port 80 after host discovery and will
remain that way during port scanning because SYN to an open port is a
worse ping probe. Run with -d2 to see when timing pings change.
o Added the undocumented (except here) --nogcc option which disables
global/group congestion control algorithms and so each member of a
scan group of machines is treated separately. This is just an
experimental option for now. [Fyodor]
to reuse an ACK ping probe from host detection during a SYN port scan. This can
greatly speed up a scan if the SYN scan finds only filtered ports.
One difficulty with implementing this is that not all ping probes are
appropriate for all scan types.
nmap -PA -sU scanme.nmap.org
would cache the ACK ping probe and send ACK pings during the UDP scan. But the
pcap filter for the UDP scan doesn't catch TCP packets, so the replies would
not be noticed and they would show up as dropped pings. Likewise,
nmap -PR -sS 192.168.0.1
would segfault when it tried to use an uninitialized Ethernet descriptor to
send an ARP ping during the SYN scan, which would use raw sockets.
To fix this I added a function pingprobe_is_appropriate that determines whether
a given ping probe is appropriate for the current scan type. If not, the
constructor for HostScanStats just erases the ping probe.
More types of ping probes could be made "appropriate." TCP timing pings work
during a UDP scan if only the pcap filter is expanded to include TCP packets.
In r8541 readip_pcap was given the ability to validate packets, and it also
returns a different length in some cases than it used to:
+ /* OK, since the IP header has been validated, we don't want to tell
+ * the caller they have more packet than they really have. This can
+ * be caused by the Ethernet CRC trailer being counted, for example.
+ */
+ if (*len > ntohs(iphdr->ip_len))
+ *len = ntohs(iphdr->ip_len);
which made some tests having to do with packet length invalid. They were
removed but this one was missed.
from ICMP probes during a protocol scan (protoscanicmphack). I don't know why
it was NULL before, but that's wrong. It was probably never noticed because in
the case of a port update, all that happens is a failure to update the timing.
In the case of a ping probe, it would look like a dropped ping probe, but that
woudl be unlikely because protocol scans usually don't take very long. I
discovered it while testing code to allow ping probes to persist between host
discovery and port scanning.
