num_probes_active == 0 in HostScanStats::completed. The reason for this is
fairly subtle and I didn't realize it at first: We have to make sure there are
no active probes because once in the completed list, probes don't time out.
Probes that are active stay active in the count. If the congestion window ever
falls below the number of these active probes, the program will hang waiting
for them to time out.
We could get away with this in the case of up hosts, because we call
HostScanStats::destroyAllOutstandingProbes in that case. We could do that in
the down case too, but that would prohibit a down host from being found up
later on. That's currently a matter of some luck; we don't keep sending probes
after a host is down but will accept replies to any other probes that have
already been sent.
source address didn't match the target address. Fyodor correctly pointed out
that this is wrong for UDP scans, when we need to slow down for a firewall
sending unreachables to know which probes don't elicit one. I'm going to try
something a little different in nmap-perf.
is up or down, we can move it to the completed list, regardless of any active
probes. However I can imagine changing this so that we move it when it is found
up, or when it is found down and there are no probes left to send. That would
give a down host a chance to become up with a different probe later on.
checking if a pingprobe has been set. We always use PORT_UNKNOWN during host
discovery (rather than HOST_UP or HOST_DOWN) to avoid conflicts with other
PORT_* constants. See the log for r8784.
num_probes_outstanding() == 0. The active probes are probes that haven't timed
out; the outstanding probes includes those and some timed-out probes. When a
host makes it to the completed list with outstanding probes, it is unlikely
they will ever be removed, so we weren't sending any global pings in some
cases. A host only moves to the completed list with it has no active probes,
and any active probes it has after that are global pings.
Also remove only timed-out ping probes from the ping host. All other probes are
timed out but waiting for a response.
me) in r8784 when I made the change to retain ping probes between ping scanning
and port scanning.
It only gets set during host discovery scans, because that's how it was before,
but it might be advantageous to set it during port scans as well.
target. In my testing, TCP scanning certain hosts, other hosts send back ICMP
destination unreachables (admin prohibited) for some ports (the Microsoft
ports). If that's the only response we got from a host, we would make it the
timing ping probe. But the admin-prohibiteds come back at some very slow rate,
much slower than the 1.25/s ping interval. So most of them are dropped, keeping
the congestion window perpetually at 1 and slowly increasing the scan delay. In
a -F scan these hosts could take over twice as long as any other host.
I also put in some missing adjust_timing arguments to the host and ping update
functions whenever adjust_timing could be false (the default is true). I think
those were just oversights. I also made it so that we don't select a new ping
probe when adjust_timing is false; there's no point sending ping probes whose
responses we're going to ignore.
o When a system route can't be matched up directly with an interface
by comparing addresses, Nmap now tries to match the route through
another route. This helps for instance with a PPP connection where
the default route's gateway address is routed through a different
route, the one associated with the address of the PPP device. The
problem would show itself as an inability to scan through the
default route and the error message
WARNING: Unable to find appropriate interface for system route to ...
need to have all the routes available before assigning interfaces to handle a
situation like the following:
# nmap --iflist
************************INTERFACES************************
DEV (SHORT) IP/MASK TYPE UP MAC
lo0 (lo0) 127.0.0.1/8 loopback up
ppp0 (ppp0) YY.YY.YY.YY/16 point2point up
# netstat -rn
Destination Gateway Flags Refs Use Netif Expire
default XX.XX.XX.XX UGSc 7 16 ppp0
XX.XX.XX.XX YY.YY.YY.YY UH 8 0 ppp0
Here the second route can be matched up immediately with the ppp0 device,
because the YY.YY.YY.YY gateway address matches the YY.YY.YY.YY/16 IP and mask
of ppp0. But the first route (default) needs to go through the second route in
order to find its interface. So we don't reject the default route immediately,
but save it an try to match it with an interface after all routes have been
found. See http://seclists.org/nmap-dev/2008/q4/0576.html for more information.
positive response that results in being marked filtered, the reply was an ICMP
error. Such probes are now ranked a 2 along with UDP, IP proto, and other
probes that get ICMP replies that are likely to be rate limited. Previously
they got the highest rank, 6, because they were TCP probes other than SYN to an
open port. SYN to an open port retains its rank of 3.
socket). It's possible to get an EHOSTUNREACH or EHOSTDOWN (and possibly other
errors) against a nonexistent LAN host and formerly the socket wasn't closed.
This allowed Nmap's internal count of sockets to be less that the true count of
open sockets. It would eventually try to open too many sockets and quit with
the message
Socket creation in sendConnectScanProbe: Too many open files (24)
Be aware that in this message 24 is the errno, not the number of open files.
The script's filename cannot be acessed when a thread's target times out
because we do not have access to the thread's environment when it is yielded.
All we can do is print out which target timed out. This output can
unfortunately be printed out many times for each thread (it is currently
difficult to track).
i.e., the probe didn't just time out. This was the case before r11253 when I
mistakenly removed it. I meant only to allow ultrascan_adjust_timing to be
called when rcvdtime == NULL, but as a side effect I also allowed
max_successful_tryno to increase in that case too. I was able to get that bit
of code to run with rcvdtime == NULL in a normal scan, but I don't think it
happens often.
I also improved the comments in that block of code.
o A problem that caused OS detection to fail for most hosts in a
certain was fixed. It happened when sending raw Ethernet frames
(by default on Windows or on other platforms with --send-eth) to
hosts on a switched LAN. The destination MAC address was wrong for
most targets. The symptom was that only one out of each scan group
of 20 or 30 hosts would have a meaningful OS fingerprint. Thanks go
to Michael Head for running tests and especially Trent Snyder for
testing and finding the cause of the problem. [David]
This fixes the following bug: When scanning with an Ethernet handle (as
opposed to raw sockets), only the first host in an OS scan group would
get a result. All others would be blank fingerprints with R=N for every
probe. This was first noticed on Windows because Ethernet is the default
sending method, but it affects other platforms with --send-eth.
OS scan initialized an Ethernet handle once for each group, and recorded
the first-hop MAC address of the first target at that time. That
first-hop address was used for all targets. This failed on a switched
LAN, when the first-hop address for every host is different (it's the
MAC address of each target).
All the various high-level probe sending functions now do their work
through three low-level sending functions: one each for TCP, UDP, and
ICMP. Those low-level functions take care of setting the MAC addresses
before each send.
I checked and the other places where Ethernet sends are used do not have
this problem. ultra_scan, idle scan, and traceroute all set the
addresses before every send.
measurements. These commonly responsive ports are moved to the front of the
list when randomizing ports to help us quickly get some timing feedback and
find a timing ping probe.
The previous list was
{ 21, 22, 23, 25, 53, 80, 113, 256, 389, 443, 554, 636, 1723, 3389 }
The new list is
{ 80, 23, 443, 21, 22, 25, 3389, 110, 445, 139, 143, 53, 135, 113 }
The ports that were removed are
256 fw1-secureremote
389 ldap
554 rtsp
636 ldapssl
1723 pptp
The ports that were added are
110 pop3
135 msrpc
139 netbios-ssn
143 imap
445 microsoft-ds
Like ultra_scan, OS scan has global and host-based congestion control
mechanisms like those in TCP. Part of global congestion control is
keeping track of how many probes are outstanding in the network; OS scan
keeps the number in a member variable called num_probes_active.
num_probes_active is meant to be the sum of the sizes of each host's
list of outstanding probes. It was correctly being decremented whenever
a probe was removed from an active list, but it was never incremented.
num_probes_active was always zero or negative, and therefore never
exceeded the global congestion window. This almost completely disabled
global congestion control.
With this fix OS scan will send a maximum of ten probes immediately at
the beginning of the scan. Previously it was limited only by the number
of hosts being scanned (20 or 30).
