PenTest/nmap
1
0
Fork 0
mirror of https://github.com/nmap/nmap.git synced 2025-12-10 17:59:04 +00:00
Code Issues Projects Releases Wiki Activity

Move /nmap-exp/david/ndiff to /nmap/ndiff.

Browse Source
This commit is contained in:
david
2008-09-18 23:31:19 +00:00
parent b556051021
commit 903e91a48b
17 changed files with 2231 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
ndiff/COPYING Normal file
View File

@@ -0,0 +1,4 @@
Copyright 2008 Insecure.Com LLC
Ndiff is distributed under the same license as Nmap. See the file COPYING in
the Nmap source distribution or http://nmap.org/data/COPYING. See
http://nmap.org/book/man-legal.html for more details.

78
ndiff/README Normal file
View File

@@ -0,0 +1,78 @@
Ndiff
Ndiff is a tool to aid in the comparison of Nmap scans. Specifically, it
takes two Nmap XML output files and prints the differences between them:
hosts coming up and down, ports becoming open or closed, and things like
that.
To install, run (as root)
python setup.py install
It's also possible to run the program from within the distribution
without installing it.
Use "ndiff --help" for usage instructions. Output can be in
human-readable text format ("ndiff --text") or machine-readable XML
format ("ndiff --xml").
Here is a sample of the text output:
$ ./ndiff test-scans/random-1.xml test-scans/random-2.xml
Thu Sep 11 11:39:32 2008 -> Tue Sep 16 13:59:22 2008
cuvtdnray-504.example.com (10.214.143.33):
Host is up, was unknown.
Add ipv4 address 10.214.143.33.
Add hostname cuvtdnray-504.example.com.
3389/tcp is open.
999 other tcp ports are filtered.
scnqxez-842.example.com (10.189.71.117):
Remove hostname scnqxez-842.example.com.
10.226.19.80:
21/tcp is open, was filtered.
23/tcp is open, was filtered.
80/tcp is open, was filtered.
8701/tcp is filtered, was open.
ywnleu-108.example.com (10.242.160.155):
Host is up, was unknown.
Add ipv4 address 10.242.160.155.
Add hostname ywnleu-108.example.com.
1000 other tcp ports are filtered.
Here is an abbreviated sample of the XML output:
$ ./ndiff --xml test-scans/random-1.xml test-scans/random-2.xml
<?xml version="1.0" encoding="UTF-8"?>
<nmapdiff>
<scandiff a-start="1221154772" b-start="1221595162">
<host>
<address addr="10.214.143.33" addrtype="ipv4"/>
<hostname name="cuvtdnray-504.example.com"/>
<host-state-change a-state="unknown" b-state="up"/>
<host-address-add>
<address addr="10.214.143.33" addrtype="ipv4"/>
</host-address-add>
<host-hostname-add>
<hostname name="cuvtdnray-504.example.com"/>
</host-hostname-add>
<port-state-change a-state="unknown" b-state="filtered" portid="1" protocol="tcp"/>
<port-state-change a-state="unknown" b-state="filtered" portid="3" protocol="tcp"/>
<port-state-change a-state="unknown" b-state="filtered" portid="4" protocol="tcp"/>
</host>
<host>
<address addr="10.189.71.117" addrtype="ipv4"/>
<hostname name="scnqxez-842.example.com"/>
<host-hostname-remove>
<hostname name="scnqxez-842.example.com"/>
</host-hostname-remove>
</host>
</scandiff>
</nmapdiff>
Ndiff started as a project by Michael Pattrick <mpattrick@rhinovirus.org>
during the 2008 Google Summer of Code. Michael designed the program and
led the discussion of its output formats. He wrote versions of the
program in Perl and C++, but the summer ended shortly after it was
decided to rewrite the program in Python for the sake of Windows
compatibility. This Python version is written by David Fifield
<david@bamsoftware.com>.
Ndiff web site: http://nmap.org/ndiff/

103
ndiff/docs/ndiff.1 Normal file
View File

@@ -0,0 +1,103 @@
.\" Title: ndiff
.\" Author:
.\" Generator: DocBook XSL Stylesheets v1.73.2 <http://docbook.sf.net/>
.\" Date: 09/17/2008
.\" Manual:
.\" Source:
.\"
.TH "NDIFF" "1" "09/17/2008" "" ""
.\" disable hyphenation
.nh
.\" disable justification (adjust text to left margin only)
.ad l
.SH "NAME"
ndiff - Utility to compare the results of Nmap scans
.SH "SYNOPSIS"
.HP 6
\fBndiff\fR [\fIoptions\fR] {\fI\fIa\.xml\fR\fR} {\fI\fIb\.xml\fR\fR}
.SH "DESCRIPTION"
.PP
Ndiff is a tool to aid in the comparison of Nmap scans\. Specifically, it takes two Nmap XML output files and prints the differences between them: hosts coming up and down, ports becoming open or closed, and things like that\.
.PP
Ndiff compares two scans at a time\. The
\(lqbefore\(rq
scan is called the A scan and the
\(lqafter\(rq
scan is the B scan\. The letters A and B are used to avoid giving the impression that scans must be given in time order\. They do not; it\'s possible to get a
\(lqbackward\(rq
diff from a newer scan to an older scan\.
.PP
Ndiff can produce output in human\-readable text or machine\-readable XML formats\. Use the
\fB\-\-text\fR
and
\fB\-\-xml\fR
options to control which\. Output goes to standard output\.
.SH "OPTIONS SUMMARY"
.PP
\fB\-h\fR, \fB\-\-help\fR
.RS 4
Show a help message and exit\.
.RE
.PP
\fB\-v\fR, \fB\-\-verbose\fR
.RS 4
Do not consolidate long port lists into a simple count\. When a host is up in the B scan that was not present in the A scan, commonly most of its ports will change from the state "unknown" to "closed" or "filtered"\. If the port list is very long, it will be consolidated into a line like
.sp
.RS 4
.nf
994 other tcp ports changed state from unknown to filtered\.
.fi
.RE
.sp
With
\fB\-\-verbose\fR, all 994 ports will be listed:
.sp
.RS 4
.nf
The following tcp ports changed state from unknown to filtered:
1,3,4,6,7,9,13,17,19\-21,23,24,26,30,32,
33,37,42,43,49,79,81\-85,88\-90,99,100,106,109\-11
1,119,125,135,139,143,144,146,161,163,179,199,2
.fi
.RE
.sp
and so on\.
.sp
In XML output, every port is always listed explictly\.
\fB\-\-verbose\fR
has no effect\.
.RE
.PP
\fB\-\-text\fR
.RS 4
Write output in human\-readable text format\.
.RE
.PP
\fB\-\-xml\fR
.RS 4
Write output in machine\-readable text format\. For a description of the XML format see the
\fInmap\.dtd\fR
file in the Ndiff distribution\.
.RE
.PP
Any other arguments are taken to be the names of Nmap XML output files\. There must be exactly two\. The first one listed is the A scan and the second is the B scan\.
.SH "BUGS"
.PP
Report bugs to the
nmap\-dev
mailing list at
<nmap\-dev@insecure\.org>\.
.SH "HISTORY"
.PP
Ndiff started as a project by Michael Pattrick during the 2008 Google Summer of Code\. Michael designed the program and led the discussion of its output formats\. He wrote versions of the program in Perl and C++, but the summer ended shortly after it was decided to rewrite the program in Python for the sake of Windows compatibility\. This Python version is written by David Fifield\.
.SH "AUTHORS"
.PP
David Fifield
<david@bamsoftware\.com>
.PP
Michael Pattrick
<mpattrick@rhinovirus\.org>
.SH "WEB SITE"
.PP
\fI\%http://nmap.org/ndiff/\fR

142
ndiff/docs/ndiff.dtd Normal file
View File

@@ -0,0 +1,142 @@
<!--
DTD for the Ndiff XML output format.
David Fifield <david@bamsoftware.com>
Ndiff compares two scans at a time. The "before" and "after" scans are
called the A and B scans, respectively. Some of the XML output uses this
convention, for example the a-start and b-start attributes of the
scandiff element.
The scandiff element represents a single diff of an A and B scan. Within
it are zero or more host elements. At the beginning of each host element
is any number of address and hostname elements, used to identify it. The
addresses and hostnames are taken from the A scan, unless the host was
not present in the A scan, in which case they come from the B scan.
Therefore they may not represent the final status of the host "after"
the diff; the addresses and hostnames may have changed between the A and
B scans.
Following the address and hostname elements is an ordered list of
elements, each representing one diff "hunk." A hunk is an atomic
difference operation. For example, the host-state-change element
represents a host changing its state, perhaps from "unknown" to "up".
See the comments above each diff hunk element for a precise description
of what they mean.
The order of diff hunks can matter. For example,
<port-state-change protocol="tcp" portid="100" a-state="open" b-state="closed"/>
<port-id-change a-protocol="tcp" a-portid="100" b-protocol="tcp" b-portid="200"/>
is different than the opposite order
<port-state-change protocol="tcp" portid="100" a-state="open" b-state="closed"/>
<port-id-change a-protocol="tcp" a-portid="100" b-protocol="tcp" b-portid="200"/>
The first order means, "Change the state of port 100/tcp from open to
closed, then swap ports 100/tcp and 200/tcp." If port 200/tcp was
initially filtered, this results in
PORT STATE
100/tcp filtered
200/tcp closed
The second order means, "Swap ports 100/tcp and 200/tcp, then change the
state of port 100/tcp from open to closed." In this case, port 200/tcp
must have originally been open. If port 100/tcp was initially filtered,
this results in
PORT STATE
100/tcp closed
200/tcp filtered
-->
<!-- Parameter entities defining "data types" used in the rest of the
DTD. -->
<!ENTITY % protocol "(ip | tcp | udp)">
<!ENTITY % host-state "(unknown | up | down)">
<!ENTITY % port-state "CDATA">
<!-- The diff-hunk parameter entity is any element that represents a
diff hunk. -->
<!ENTITY % diff-hunk
"(host-state-change | host-address-add | host-address-remove
| host-hostname-add | host-hostname-remove
| port-id-change | port-state-change)"
>
<!ELEMENT nmapdiff (scandiff)>
<!ELEMENT scandiff (host*)>
<!-- a-start and b-start are the start times of the A and B scans,
expressed as a decimal number of seconds since the epoch. -->
<!ATTLIST scandiff a-start CDATA #IMPLIED
b-start CDATA #IMPLIED>
<!ELEMENT host ((address | hostname)*, (%diff-hunk;)*)>
<!ELEMENT address EMPTY>
<!ATTLIST address addrtype (mac | ipv4 | ipv6) "ipv4"
addr CDATA #REQUIRED>
<!ELEMENT hostname EMPTY>
<!ATTLIST hostname name CDATA #REQUIRED>
<!-- Diff hunk elements. Each of these represents an atomic difference
operation. -->
<!--
The host changed its state, for example from "unknown" to "up". a-state
is the state of the host in the A scan and b-state is the state of the
host in the B scan.
-->
<!ELEMENT host-state-change EMPTY>
<!ATTLIST host-state-change a-state %host-state; #REQUIRED
b-state %host-state; #REQUIRED>
<!--
The host gained an address in the B scan that it didn't have in the A
scan.
-->
<!ELEMENT host-address-add (address)>
<!--
The host had an address in the A scan that it didn't have in the B scan.
-->
<!ELEMENT host-address-remove (address)>
<!--
The host gained a hostname in the B scan that it didn't have in the A
scan.
-->
<!ELEMENT host-hostname-add (hostname)>
<!--
The host had a hostname in the A scan that it didn't have in the B scan.
-->
<!ELEMENT host-hostname-remove (hostname)>
<!--
The services that were running on two ports were swapped between the A
and B scans. The portid and protocol attributes give the A and B port
specifications. The portid attributes are just decimal port numbers and
the protocol attributes are something like "ip", "tcp", or "udp".
For example: If, in the A scan port 100/tcp was filtered and port
200/tcp was open running OpenSSH, then the hunk
<port-id-change a-protocol="tcp" a-portid="100" b-protocol="tcp" b-portid="200"/>
means that in the B scan port 100/tcp is open running OpenSSH and port
200/tcp is filtered.
Later hunks may further modify the ports that were swapped by this hunk.
See the not about order in the comment at the top.
-->
<!ELEMENT port-id-change EMPTY>
<!ATTLIST port-id-change a-portid CDATA #REQUIRED
a-protocol %protocol; #REQUIRED
b-portid CDATA #REQUIRED
b-protocol %protocol; #REQUIRED>
<!--
The port identified by the portid and protocol attributes changed state
from that given by the a-state attribute to that given by the b-state
attribute.
-->
<!ELEMENT port-state-change EMPTY>
<!ATTLIST port-state-change portid CDATA #REQUIRED
protocol %protocol; #REQUIRED
a-state %port-state; #REQUIRED
b-state %port-state; #REQUIRED>

161
ndiff/docs/ndiff.xml Normal file
View File

@@ -0,0 +1,161 @@
<!-- This is the DocBook XML source for the Ndiff manual page. -->
<refentry>
<refmeta>
<refentrytitle>ndiff</refentrytitle>
<manvolnum>1</manvolnum>
</refmeta>
<refnamediv>
<refname>ndiff</refname>
<refpurpose>Utility to compare the results of Nmap scans</refpurpose>
</refnamediv>
<refsynopsisdiv>
<cmdsynopsis>
<command>ndiff</command>
<arg choice='opt'>
<replaceable>options</replaceable>
</arg>
<arg choice='req'>
<replaceable><filename>a.xml</filename></replaceable>
</arg>
<arg choice='req'>
<replaceable><filename>b.xml</filename></replaceable>
</arg>
</cmdsynopsis>
</refsynopsisdiv>
<refsect1>
<title>Description</title>
<para>
Ndiff is a tool to aid in the comparison of Nmap scans. Specifically, it
takes two Nmap XML output files and prints the differences between them:
hosts coming up and down, ports becoming open or closed, and things like
that.
</para>
<para>
Ndiff compares two scans at a time. The <quote>before</quote> scan
is called the A scan and the <quote>after</quote> scan is the B
scan. The letters A and B are used to avoid giving the impression
that scans must be given in time order. They do not; it's possible
to get a <quote>backward</quote> diff from a newer scan to an older
scan.
</para>
<para>
Ndiff can produce output in human-readable text or machine-readable
XML formats. Use the <option>--text</option> and
<option>--xml</option> options to control which. Output goes to
standard output.
</para>
</refsect1>
<refsect1>
<title>Options Summary</title>
<variablelist>
<varlistentry>
<term><option>-h</option></term>
<term><option>--help</option></term>
<listitem>
<para>
Show a help message and exit.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-v</option></term>
<term><option>--verbose</option></term>
<listitem>
<para>
Do not consolidate long port lists into a simple count. When
a host is up in the B scan that was not present in the A scan,
commonly most of its ports will change from the state
"unknown" to "closed" or "filtered". If the port list is very
long, it will be consolidated into a line like
<screen>994 other tcp ports changed state from unknown to filtered.
</screen>
With <option>--verbose</option>, all 994 ports will be listed:
<screen>The following tcp ports changed state from unknown to filtered:
1,3,4,6,7,9,13,17,19-21,23,24,26,30,32,
33,37,42,43,49,79,81-85,88-90,99,100,106,109-11
1,119,125,135,139,143,144,146,161,163,179,199,2
</screen>
and so on.
</para>
<para>
In XML output, every port is always listed explictly.
<option>--verbose</option> has no effect.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>--text</option></term>
<listitem>
<para>
Write output in human-readable text format.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>--xml</option></term>
<listitem>
<para>
Write output in machine-readable text format. For a
description of the XML format see the
<filename>nmap.dtd</filename> file in the Ndiff distribution.
</para>
</listitem>
</varlistentry>
</variablelist>
<para>
Any other arguments are taken to be the names of Nmap XML output
files. There must be exactly two. The first one listed is the A scan
and the second is the B scan.
</para>
</refsect1>
<refsect1>
<title>Bugs</title>
<para>
Report bugs to the <citetitle>nmap-dev</citetitle> mailing list at
<email>nmap-dev@insecure.org</email>.
</para>
</refsect1>
<refsect1>
<title>History</title>
<para>
Ndiff started as a project by Michael Pattrick during the 2008
Google Summer of Code. Michael designed the program and led the
discussion of its output formats. He wrote versions of the program
in Perl and C++, but the summer ended shortly after it was decided
to rewrite the program in Python for the sake of Windows
compatibility. This Python version is written by David Fifield.
</para>
</refsect1>
<refsect1>
<title>Authors</title>
<para>
David Fifield <email>david@bamsoftware.com</email>
</para>
<para>
Michael Pattrick <email>mpattrick@rhinovirus.org</email>
</para>
</refsect1>
<refsect1>
<title>Web site</title>
<para>
<ulink url="http://nmap.org/ndiff/"/>
</para>
</refsect1>
</refentry>

736
ndiff/ndiff Executable file
View File

@@ -0,0 +1,736 @@
#!/usr/bin/env python
# ndiff
#
# This programs reads two Nmap XML files and displays a list of their
# differences.
#
# Copyright 2008 Insecure.Com LLC
# Ndiff is distributed under the same license as Nmap. See the file COPYING or
# http://nmap.org/data/COPYING. See http://nmap.org/book/man-legal.html for more
# details.
#
# David Fifield
# based on a design by Michael Pattrick
import datetime
import getopt
import sys
import time
import xml.sax
import xml.dom.minidom
# Port state transitions with more than this many occurrences are consolidated
# into one text output entry.
PORT_STATE_CHANGE_CONSOLIDATION_THRESHOLD = 10
# Consolidated port state ranges whose *character length* is greater than this
# will be doubly consolidated into just a count of ports in text output.
PORT_STATE_CHANGE_DOUBLE_CONSOLIDATION_CHAR_THRESHOLD = 80
class Port(object):
"""A single port, consisting of a port specification and a state. A
specification, or "spec," is the 2-tuple (number, protocol). So (10, "tcp")
corresponds to the port 10/tcp. Port states are strings."""
# This represents an "unknown" port state, the state a port is in when it
# has not been scanned. It must not compare equal with any real Nmap port
# state like "open", "closed", etc. For future compatibility's sake, always
# compare against Port.UNKNOWN, not the literal string "unknown".
UNKNOWN = "unknown"
def __init__(self, spec):
self.spec = spec
self.state = Port.UNKNOWN
def get_state_string(self):
return Port.state_to_string(self.state)
def state_to_string(state):
return unicode(state)
state_to_string = staticmethod(state_to_string)
def spec_to_string(spec):
return u"%d/%s" % spec
spec_to_string = staticmethod(spec_to_string)
class PortDict(dict):
"""This is a dict that creates and inserts a new Port on demand when a
looked-up value isn't found."""
def __getitem__(self, key):
try:
port = dict.__getitem__(self, key)
except KeyError:
port = Port(key)
self[key] = port
return port
def __len__(self):
raise ValueError(u"__len__ is not defined for objects of type PortDict.")
class Host(object):
"""A single host, with a stated (unknown, up, or down), addresses, and a
dict mapping port specs to Ports."""
# This represents an "unknown" host state, the state a host is in when it
# hasn't been scanned. It must not compare equal with the real Nmap host
# states "up" and "down". For future compatibility's sake, always compare
# against Host.UNKNOWN, not the literal string "unknown".
UNKNOWN = "unknown"
def __init__(self):
self.state = self.UNKNOWN
# Addresses are represented as (address_type, address) tuples.
self.addresses = []
self.hostnames = []
self.ports = PortDict()
def get_id(self):
"""Return an id that is used to determine if hosts are "the same" across
scans."""
if len(self.addresses) > 0:
return sorted(self.addresses)[0]
if len(self.hostnames) > 0:
return sorted(self.hostnames)[0]
return id(self)
def format_name(self):
"""Return a human-readable identifier for this host."""
address = None
for address_type in (u"ipv4", u"ipv6", u"mac"):
addrs = [addr for addr in self.addresses if addr[0] == address_type]
if len(addrs) > 0:
address = addrs[0][1]
hostname = None
if len(self.hostnames) > 0:
hostname = self.hostnames[0]
if hostname is not None:
if address is not None:
return u"%s (%s)" % (hostname, address)
else:
return hostname
elif address is not None:
return address
return unicode(id(self))
def add_port(self, spec, state):
"""Add a port in the given state."""
port = self.ports[spec]
port.state = state
def swap_ports(self, spec_a, spec_b):
"""Swap the ports given by the two specs. This is used when a service is
moved from one port to another."""
port_a = self.ports[spec_a]
port_b = self.ports[spec_b]
assert port_a.spec == spec_a
assert port_b.spec == spec_b
port_a.spec, port_b.spec = port_b.spec, port_a.spec
self.ports[spec_a], self.ports[spec_b] = \
self.ports[spec_b], self.ports[spec_a]
def get_known_ports(self):
"""Return a list of all this host's Ports that are not in the unknown
state."""
return [p for p in self.ports.values() if p.state != Port.UNKNOWN]
def add_address(self, address_type, address):
if address not in self.addresses:
self.addresses.append((address_type, address))
def remove_address(self, address_type, address):
try:
self.addresses.remove((address_type, address))
except ValueError:
pass
def add_hostname(self, hostname):
if hostname not in self.hostnames:
self.hostnames.append(hostname)
def remove_hostname(self, hostname):
try:
self.hostnames.remove(hostname)
except ValueError:
pass
class Scan(object):
"""A single Nmap scan, corresponding to a single invocation of Nmap. It is a
container for a list of hosts. It also has utility methods to load itself
from an Nmap XML file."""
def __init__(self):
self.start_date = None
self.end_date = None
self.hosts = []
def load(self, f):
"""Load a scan from the Nmap XML in the file-like object f."""
parser = xml.sax.make_parser()
handler = NmapContentHandler(self)
parser.setContentHandler(handler)
parser.parse(f)
def load_from_file(self, filename):
"""Load a scan from the Nmap XML file with the given filename."""
f = open(filename, "r")
try:
self.load(f)
finally:
f.close()
# The types of each possible diff hunk.
(HOST_STATE_CHANGE, HOST_ADDRESS_ADD, HOST_ADDRESS_REMOVE,
HOST_HOSTNAME_ADD, HOST_HOSTNAME_REMOVE,
PORT_ID_CHANGE, PORT_STATE_CHANGE) = range(0, 7)
class DiffHunk(object):
"""A DiffHunk is a single unit of a diff. Each one represents one atomic
change: a host state change, port state change, and so on."""
def __init__(self, type):
self.type = type
def to_string(self):
"""Return a human-readable string representation of this hunk."""
if self.type == HOST_STATE_CHANGE:
return u"Host is %s, was %s." % (self.b_state, self.a_state)
elif self.type == HOST_ADDRESS_ADD:
return u"Add %s address %s." % (self.address_type, self.address)
elif self.type == HOST_ADDRESS_REMOVE:
return u"Remove %s address %s." % (self.address_type, self.address)
elif self.type == HOST_HOSTNAME_ADD:
return u"Add hostname %s." % self.hostname
elif self.type == HOST_HOSTNAME_REMOVE:
return u"Remove hostname %s." % self.hostname
elif self.type == PORT_ID_CHANGE:
return u"Service on %s is now running on %s." % (Port.spec_to_string(self.a_spec), Port.spec_to_string(self.b_spec))
elif self.type == PORT_STATE_CHANGE:
if self.a_state == Port.UNKNOWN:
return u"%s is %s." % (Port.spec_to_string(self.spec), self.b_state)
else:
return u"%s is %s, was %s." % (Port.spec_to_string(self.spec), self.b_state, self.a_state)
else:
assert False
def to_dom_fragment(self, document):
"""Return an XML DocumentFragment representation of this hunk."""
frag = document.createDocumentFragment()
if self.type == HOST_STATE_CHANGE:
elem = document.createElement(u"host-state-change")
elem.setAttribute(u"a-state", self.a_state)
elem.setAttribute(u"b-state", self.b_state)
frag.appendChild(elem)
elif self.type == HOST_ADDRESS_ADD:
elem = document.createElement(u"host-address-add")
frag.appendChild(elem)
address_elem = document.createElement("address")
address_elem.setAttribute(u"addrtype", self.address_type)
address_elem.setAttribute(u"addr", self.address)
elem.appendChild(address_elem)
elif self.type == HOST_ADDRESS_REMOVE:
elem = document.createElement(u"host-address-remove")
frag.appendChild(elem)
address_elem = document.createElement("address")
address_elem.setAttribute(u"addrtype", self.address_type)
address_elem.setAttribute(u"addr", self.address)
elem.appendChild(address_elem)
elif self.type == HOST_HOSTNAME_ADD:
elem = document.createElement(u"host-hostname-add")
frag.appendChild(elem)
address_elem = document.createElement("hostname")
address_elem.setAttribute(u"name", self.hostname)
elem.appendChild(address_elem)
elif self.type == HOST_HOSTNAME_REMOVE:
elem = document.createElement(u"host-hostname-remove")
frag.appendChild(elem)
address_elem = document.createElement("hostname")
address_elem.setAttribute(u"name", self.hostname)
elem.appendChild(address_elem)
elif self.type == PORT_ID_CHANGE:
elem = document.createElement(u"port-id-change")
elem.setAttribute(u"a-portid", unicode(self.a_spec[0]))
elem.setAttribute(u"a-protocol", self.a_spec[1])
elem.setAttribute(u"b-portid", unicode(self.b_spec[0]))
elem.setAttribute(u"b-protocol", self.b_spec[1])
frag.appendChild(elem)
elif self.type == PORT_STATE_CHANGE:
elem = document.createElement(u"port-state-change")
elem.setAttribute(u"portid", unicode(self.spec[0]))
elem.setAttribute(u"protocol", self.spec[1])
elem.setAttribute(u"a-state", self.a_state)
elem.setAttribute(u"b-state", self.b_state)
frag.appendChild(elem)
return frag
def partition_port_state_changes(diff):
"""Partition a list of PORT_STATE_CHANGE diff hunks into equivalence classes
based on the tuple (protocol, a_state, b_state). The partition is returned
as a list of lists of hunks."""
transitions = {}
for hunk in diff:
if hunk.type != PORT_STATE_CHANGE:
continue
a_state = hunk.a_state
b_state = hunk.b_state
protocol = hunk.spec[1]
transitions.setdefault((protocol, a_state, b_state), []).append(hunk)
return transitions.values()
def consolidate_port_state_changes(diff, threshold = 0):
"""Return a list of list of PORT_STATE_CHANGE diff hunks, where each list
contains hunks with the same partition and state change. A group of hunks is
returned in the list of lists only when its length exceeds threshold. Any
hunks moved to the list of lists are removed from diff in place. This is to
avoid overwhelming the output with a flood of "is filtered, was unknown"
messages."""
partition = partition_port_state_changes(diff)
consolidated = []
for group in partition:
if len(group) > threshold:
for hunk in group:
diff.remove(hunk)
consolidated.append(group)
return consolidated
class ScanDiff(object):
"""A complete diff of two scans. It is a container for two scans and the
diff between them, which is a list of (scan, host_diff) tuples as returned
by scan_diff."""
def __init__(self, scan_a, scan_b):
"""Create a ScanDiff from the "before" scan_a and the "after" scan_b."""
self.scan_a = scan_a
self.scan_b = scan_b
self.diff = scan_diff(scan_a, scan_b)
def print_text(self, f = sys.stdout, verbose = False):
"""Print this diff in a human-readable text form."""
if self.scan_a.start_date is not None:
start_date_a_str = self.scan_a.start_date.ctime()
else:
start_date_a_str = u"<unknown date>"
if self.scan_b.start_date is not None:
start_date_b_str = self.scan_b.start_date.ctime()
else:
start_date_b_str = u"<unknown date>"
print >> f, "%s -> %s" % (start_date_a_str, start_date_b_str)
for host, h_diff in self.diff:
print >> f, "%s:" % host.format_name()
h_diff_copy = h_diff[:]
cons_port_state_changes = consolidate_port_state_changes(h_diff_copy, PORT_STATE_CHANGE_CONSOLIDATION_THRESHOLD)
for hunk in h_diff_copy:
print >> f, u"\t" + hunk.to_string();
for group in cons_port_state_changes:
a_state = group[0].a_state
b_state = group[0].b_state
protocol = group[0].spec[1]
port_list = [hunk.spec[0] for hunk in group]
port_list_string = render_port_list(port_list)
if verbose or len(port_list_string) <= \
PORT_STATE_CHANGE_DOUBLE_CONSOLIDATION_CHAR_THRESHOLD:
if a_state == Port.UNKNOWN:
print >> f, u"\tThe following %d %s ports are %s:" % (len(port_list), protocol, b_state)
else:
print >> f, u"\tThe following %d %s ports changed state from %s to %s:" % (len(port_list), protocol, a_state, b_state)
print >> f, u"\t\t" + port_list_string
else:
if a_state == Port.UNKNOWN:
print >> f, u"\t%d other %s ports are %s." % (len(port_list), protocol, b_state)
else:
print >> f, u"\t%d other %s ports changed state from %s to %s." % (len(port_list), protocol, a_state, b_state)
def print_xml(self, f = sys.stdout):
impl = xml.dom.minidom.getDOMImplementation()
document = impl.createDocument(None, u"nmapdiff", None)
root = document.documentElement
scandiff_elem = document.createElement(u"scandiff")
if self.scan_a.start_date is not None:
a_start_date_str = unicode(int(time.mktime(self.scan_a.start_date.timetuple())))
scandiff_elem.setAttribute(u"a-start", a_start_date_str)
if self.scan_b.start_date is not None:
b_start_date_str = unicode(int(time.mktime(self.scan_b.start_date.timetuple())))
scandiff_elem.setAttribute(u"b-start", b_start_date_str)
root.appendChild(scandiff_elem)
for host, h_diff in self.diff:
host_elem = document.createElement(u"host")
scandiff_elem.appendChild(host_elem)
for (address_type, address) in host.addresses:
address_elem = document.createElement(u"address")
address_elem.setAttribute(u"addrtype", address_type)
address_elem.setAttribute(u"addr", address)
host_elem.appendChild(address_elem)
for hostname in host.hostnames:
hostname_elem = document.createElement(u"hostname")
hostname_elem.setAttribute(u"name", hostname)
host_elem.appendChild(hostname_elem)
for hunk in h_diff:
host_elem.appendChild(hunk.to_dom_fragment(document))
document.writexml(f, addindent = u" ", newl = u"\n", encoding = "UTF-8")
document.unlink()
def port_diff(a, b):
"""Diff two Ports. The return value is a list of DiffHunks."""
diff = []
if a.spec != b.spec:
hunk = DiffHunk(PORT_ID_CHANGE)
hunk.a_spec = a.spec
hunk.b_spec = b.spec
diff.append(hunk)
if a.state != b.state:
hunk = DiffHunk(PORT_STATE_CHANGE)
hunk.spec = b.spec
hunk.a_state = a.state
hunk.b_state = b.state
diff.append(hunk)
return diff
def addresses_diff(a, b):
"""Diff two lists of addresses. The return value is a list of DiffHunks."""
diff = []
a_addresses = set(a)
b_addresses = set(b)
for addrtype, addr in a_addresses - b_addresses:
hunk = DiffHunk(HOST_ADDRESS_REMOVE)
hunk.address_type = addrtype
hunk.address = addr
diff.append(hunk)
for addrtype, addr in b_addresses - a_addresses:
hunk = DiffHunk(HOST_ADDRESS_ADD)
hunk.address_type = addrtype
hunk.address = addr
diff.append(hunk)
return diff
def hostnames_diff(a, b):
"""Diff two lists of hostnames. The return value is a list of DiffHunks."""
diff = []
a_hostnames = set(a)
b_hostnames = set(b)
for hostname in a_hostnames - b_hostnames:
hunk = DiffHunk(HOST_HOSTNAME_REMOVE)
hunk.hostname = hostname
diff.append(hunk)
for hostname in b_hostnames - a_hostnames:
hunk = DiffHunk(HOST_HOSTNAME_ADD)
hunk.hostname = hostname
diff.append(hunk)
return diff
def host_diff(a, b):
"""Diff two Hosts. The return value is a list of DiffHunks."""
diff = []
if a.state != b.state:
hunk = DiffHunk(HOST_STATE_CHANGE)
hunk.id = a.get_id()
hunk.a_state = a.state
hunk.b_state = b.state
diff.append(hunk)
diff.extend(addresses_diff(a.addresses, b.addresses))
diff.extend(hostnames_diff(a.hostnames, b.hostnames))
all_specs = list(set(a.ports.keys()).union(set(b.ports.keys())))
all_specs.sort()
for spec in all_specs:
if b.ports[spec].state != Port.UNKNOWN:
diff.extend(port_diff(a.ports[spec], b.ports[spec]))
return diff
def scan_diff(a, b):
"""Diff two scans. The return value is a list of tuples. Each tuple has the
form (scan, host_diff), where scan is either a or b, whichever is present in
one of the scans (preferring a if both are present), and host_diff is the
host diff as returned by host_diff."""
diff = []
a_hosts = dict((host.get_id(), host) for host in a.hosts)
b_hosts = dict((host.get_id(), host) for host in b.hosts)
all_host_ids = set(a_hosts.keys()).union(set(b_hosts.keys()))
for id in all_host_ids:
host_a = a_hosts.get(id)
host_b = b_hosts.get(id)
if host_b is not None and host_b.state != Host.UNKNOWN:
h_diff = host_diff(host_a or Host(), host_b or Host())
if len(h_diff) > 0:
diff.append((host_a or host_b, h_diff))
return diff
def warn(str):
"""Print a warning to stderr."""
print >> sys.stderr, str
def parse_port_list(port_list):
"""Parse a port list like
"1-1027,1029-1033,1040,1043,1050,1058-1059,1067-1068,1076,1080,1083-1084"
into a list of numbers. Raises ValueError if the port list is somehow
invalid. This function and parse_port_list are rough inverses."""
result = set()
if port_list == u"":
return list(result)
chunks = port_list.split(u",")
for chunk in chunks:
if u"-" in chunk:
start, end = chunk.split(u"-")
start = int(start)
end = int(end)
if start >= end:
raise ValueError(u"In range %s, start must be less than end." % chunk)
else:
start = int(chunk)
end = start
for p in range(start, end + 1):
result.add(p)
result = list(result)
result.sort()
return result
def render_port_list(ports):
"""Render a list of numbers into a string like
"1-1027,1029-1033,1040,1043,1050,1058-1059,1067-1068,1076,1080,1083-1084".
This function and parse_port_list are rough inverses."""
if len(ports) == 0:
return u""
ports = ports[:]
ports.sort()
chunks = []
range_start = ports[0]
i = 0
while i < len(ports):
x = ports[i]
range_start = ports[i]
prev = ports[i]
i += 1
while i < len(ports) and ports[i] - prev <= 1:
prev = ports[i]
i += 1
if prev - range_start == 0:
chunks.append(u"%d" % range_start)
elif prev - range_start == 1:
chunks.append(u"%d" % range_start)
chunks.append(u"%d" % ports[i - 1])
else:
chunks.append(u"%d-%d" % (range_start, prev))
return u",".join(chunks)
class NmapContentHandler(xml.sax.handler.ContentHandler):
"""The xml.sax ContentHandler for the XML parser. It contains a Scan object
that is filled in and can be read back again once the parse method is
finished."""
def __init__(self, scan):
self.scan = scan
# We keep a stack of the elements we've seen, pushing on start and
# popping on end.
self.element_stack = []
self.scanned_ports = {}
self.current_host = None
self.current_extraports = []
self.current_spec = None
def parent_element(self):
"""Return the name of the element containing the current one, or None if
this is the root element."""
if len(self.element_stack) == 0:
return None
return self.element_stack[-1]
def startElement(self, name, attrs):
"""This method keeps track of element_stack. The real parsing work is
done in startElementAux."""
self.startElementAux(name, attrs)
self.element_stack.append(name)
def endElement(self, name):
"""This method keeps track of element_stack. The real parsing work is
done in endElementAux."""
self.element_stack.pop()
self.endElementAux(name)
def startElementAux(self, name, attrs):
if name == u"nmaprun":
assert self.parent_element() == None
if u"start" in attrs:
start_timestamp = int(attrs.get(u"start"))
self.scan.start_date = datetime.datetime.fromtimestamp(start_timestamp)
elif name == u"scaninfo":
assert self.parent_element() == u"nmaprun"
try:
protocol = attrs[u"protocol"]
except KeyError:
warn(u"scaninfo element missing the \"protocol\" attribute; skipping.")
return
try:
services_str = attrs[u"services"]
except KeyError:
warn(u"scaninfo element missing the \"services\" attribute; skipping.")
return
try:
services = parse_port_list(services_str)
except ValueError:
warn(u"Services list \"%s\" cannot be parsed; skipping.")
return
assert protocol not in self.scanned_ports
self.scanned_ports[protocol] = services
elif name == u"host":
assert self.parent_element() == u"nmaprun"
self.current_host = Host()
self.scan.hosts.append(self.current_host)
elif name == u"status":
assert self.parent_element() == u"host"
assert self.current_host is not None
try:
state = attrs[u"state"]
except KeyError:
warn("extraports element of host %s is missing the \"state\" attribute; assuming \"unknown\"." % self.current_host.format_name())
return
self.current_host.state = state
elif name == u"address":
assert self.parent_element() == u"host"
assert self.current_host is not None
try:
addr = attrs[u"addr"]
except KeyError:
warn("address element of host %s is missing the \"addr\" attribute; skipping." % (self.current_host.format_name()))
return
addrtype = attrs.get(u"addrtype", u"ipv4")
self.current_host.add_address(addrtype, addr)
elif name == u"hostname":
assert self.parent_element() == u"hostnames"
assert self.current_host is not None
try:
hostname = attrs[u"name"]
except KeyError:
warn("hostname element of host %s is missing the \"name\" attribute; skipping." % (self.current_host.format_name()))
return
self.current_host.add_hostname(hostname)
elif name == u"extraports":
assert self.parent_element() == u"ports"
assert self.current_host is not None
try:
state = attrs[u"state"]
except KeyError:
warn("extraports element of host %s is missing the \"state\" attribute; assuming \"unknown\"." % self.current_host.format_name())
state = Port.UNKNOWN
if state in self.current_extraports:
warn(u"Duplicate extraports state \"%s\" in host %s." % (state, self.current_host.format_name()))
# Perhaps check for count == 0.
self.current_extraports.append(state)
elif name == u"port":
assert self.parent_element() == u"ports"
assert self.current_host is not None
try:
portid_str = attrs[u"portid"]
except KeyError:
warn(u"port element of host %s missing the \"portid\" attribute; skipping." % self.current_host.format_name())
return
try:
portid = int(portid_str)
except ValueError:
warn(u"Can't convert portid \"%s\" to an integer in host %s; skipping port." % (portid_str, self.current_host.format_name()))
return
try:
protocol = attrs[u"protocol"]
except KeyError:
warn(u"port element of host %s missing the \"protocol\" attribute; skipping." % self.current_host.format_name())
return
self.current_spec = portid, protocol
elif name == u"state":
assert self.parent_element() == u"port"
assert self.current_host is not None
if self.current_spec is None:
return
if u"state" not in attrs:
warn("state element of port %s is missing the \"state\" attribute; assuming \"unknown\"." % Port.spec_to_string(self.current_spec))
return
state = attrs[u"state"]
self.current_host.add_port(self.current_spec, state)
elif name == u"finished":
assert self.parent_element() == u"runstats"
if u"time" in attrs:
end_timestamp = int(attrs.get(u"time"))
self.scan.end_date = datetime.datetime.fromtimestamp(end_timestamp)
def endElementAux(self, name):
if name == u"nmaprun":
self.scanned_ports = None
elif name == u"host":
if len(self.current_extraports) == 1:
extraports_state = self.current_extraports[0]
known_ports = self.current_host.get_known_ports()
known_specs = set(port.spec for port in known_ports)
for protocol in self.scanned_ports:
for portid in self.scanned_ports[protocol]:
spec = portid, protocol
if spec in known_specs:
continue
assert self.current_host.ports[spec].state == Port.UNKNOWN
self.current_host.add_port(spec, extraports_state)
self.current_host = None
self.current_extraports = []
elif name == u"port":
self.current_spec = None
def usage():
print u"""\
Usage: %s [option] FILE1 FILE2
Compare two Nmap XML files and display a list of their differences.
Differences include host state changes and port state changes.
-h, --help display this help
-v, --verbose don't consolidate long port lists into just a count
--text display output in text format (default)
--xml display output in XML format\
""" % sys.argv[0]
def usage_error(msg):
print >> sys.stderr, u"%s: %s" % (sys.argv[0], msg)
print >> sys.stderr, u"Try '%s -h' for help." % sys.argv[0]
sys.exit(1)
def main():
output_format = None
verbose = False
try:
opts, input_filenames = getopt.gnu_getopt(sys.argv[1:], "hv", ["help", "text", "verbose", "xml"])
except getopt.GetoptError, e:
print >> sys.stderr, u"%s: %s" % (sys.argv[0], unicode(e))
sys.exit(1)
for o, a in opts:
if o == "-h" or o == "--help":
usage()
sys.exit(0)
elif o == "-v" or o == "--verbose":
verbose = True
elif o == "--text":
if output_format is not None and output_format != "text":
usage_error("contradictory output format options.")
output_format = "text"
elif o == "--xml":
if output_format is not None and output_format != "xml":
usage_error("contradictory output format options.")
output_format = "xml"
if len(input_filenames) != 2:
usage_error("need exactly two input filenames.")
if output_format is None:
output_format = "text"
filename_a = input_filenames[0]
filename_b = input_filenames[1]
scan_a = Scan()
scan_a.load_from_file(filename_a)
scan_b = Scan()
scan_b.load_from_file(filename_b)
diff = ScanDiff(scan_a, scan_b)
if output_format == "text":
diff.print_text(verbose = verbose)
elif output_format == "xml":
diff.print_xml()
if __name__ == "__main__":
main()

1
ndiff/ndiff.py Symbolic link
View File

@@ -0,0 +1 @@
ndiff

536
ndiff/ndifftest.py Executable file
View File

@@ -0,0 +1,536 @@
#!/usr/bin/env python
# Unit tests for ndiff.
import unittest
import xml.dom.minidom
import StringIO
from ndiff import *
class parse_port_list_test(unittest.TestCase):
"""Test the parse_port_list function."""
def test_empty(self):
ports = parse_port_list(u"")
self.assertTrue(len(ports) == 0)
def test_single(self):
ports = parse_port_list(u"1,10,100")
self.assertTrue(len(ports) == 3)
self.assertTrue(set(ports) == set([1, 10, 100]))
def test_range(self):
ports = parse_port_list(u"10-20")
self.assertTrue(len(ports) == 11)
self.assertTrue(set(ports) == set(range(10, 21)))
def test_combo(self):
ports = parse_port_list(u"1,10,100-102,150")
self.assertTrue(set(ports) == set([1, 10, 100, 101, 102, 150]))
def test_dups(self):
ports = parse_port_list(u"5,1-10")
self.assertTrue(len(ports) == 10)
self.assertTrue(set(ports) == set(range(1, 11)))
def test_invalid(self):
self.assertRaises(ValueError, parse_port_list, u"a")
self.assertRaises(ValueError, parse_port_list, u",1")
self.assertRaises(ValueError, parse_port_list, u"1,,2")
self.assertRaises(ValueError, parse_port_list, u"1,")
self.assertRaises(ValueError, parse_port_list, u"1-2-3")
self.assertRaises(ValueError, parse_port_list, u"10-1")
class render_port_list_test(unittest.TestCase):
"""Test the render_port_list function."""
def test_roundtrip(self):
TESTS = ([],
[1],
[1,1],
[1,2,3,4,10,11,12],
[1,2,3,4,5,9,8,7,6],
[1,2,3,4,5,3]
)
for test in TESTS:
s = render_port_list(test)
result = parse_port_list(s)
self.assertTrue(list(set(test)) == result, u"Expected %s, got %s." % (list(set(test)), result))
class partition_port_state_changes_test(unittest.TestCase):
"""Test the partition_port_state_changes function."""
def setUp(self):
a = Scan()
a.load_from_file("test-scans/empty.xml")
b = Scan()
b.load_from_file("test-scans/simple.xml")
self.diff = scan_diff(a, b)
def test_port_state_change_only(self):
for host, h_diff in self.diff:
partition = partition_port_state_changes(h_diff)
for group in partition:
for hunk in group:
self.assertTrue(hunk.type == PORT_STATE_CHANGE)
def test_equivalence(self):
for host, h_diff in self.diff:
partition = partition_port_state_changes(h_diff)
for group in partition:
key = (group[0].spec[1], group[0].a_state, group[0].b_state)
for hunk in group:
self.assertTrue(key == (hunk.spec[1], hunk.a_state, hunk.b_state))
class consolidate_port_state_changes_test(unittest.TestCase):
"""Test the consolidate_port_state_changes function."""
def setUp(self):
a = Scan()
a.load_from_file("test-scans/empty.xml")
b = Scan()
b.load_from_file("test-scans/simple.xml")
self.diff = scan_diff(a, b)
def test_removal(self):
for host, h_diff in self.diff:
consolidated = consolidate_port_state_changes(h_diff, 0)
for hunk in h_diff:
self.assertTrue(hunk.type != PORT_STATE_CHANGE)
def test_conservation(self):
pre_length = 0
for host, h_diff in self.diff:
pre_length = len(h_diff)
consolidated = consolidate_port_state_changes(h_diff, 0)
post_length = len(h_diff) + sum(len(group) for group in consolidated)
self.assertTrue(pre_length == post_length)
def test_threshold(self):
for host, h_diff in self.diff:
for threshold in (0, 1, 2, 4, 8):
h_diff_copy = h_diff[:]
consolidated = consolidate_port_state_changes(h_diff_copy, threshold)
for group in consolidated:
self.assertTrue(len(group) > threshold, u"Length is %d, should be > %d." % (len(group), threshold))
class port_diff_test(unittest.TestCase):
"""Test the port_diff function."""
def test_equal(self):
spec = (10, "tcp")
a = Port(spec)
b = Port(spec)
diff = port_diff(a, b)
self.assertTrue(len(diff) == 0)
def test_self(self):
p = Port((10, "tcp"))
diff = port_diff(p, p)
self.assertTrue(len(diff) == 0)
def test_id_change(self):
a = Port((10, "tcp"))
b = Port((20, "tcp"))
diff = port_diff(a, b)
self.assertTrue(len(diff) == 1)
self.assertTrue(diff[0].type == PORT_ID_CHANGE)
def test_state_change(self):
spec = (10, "tcp")
a = Port(spec)
a.state = "open"
b = Port(spec)
b.state = "closed"
diff = port_diff(a, b)
self.assertTrue(len(diff) == 1)
self.assertTrue(diff[0].type == PORT_STATE_CHANGE)
def test_id_state_change(self):
a = Port((10, "tcp"))
a.state = "open"
b = Port((20, "tcp"))
b.state = "closed"
diff = port_diff(a, b)
self.assertTrue(len(diff) > 1)
class host_test(unittest.TestCase):
"""Test the Host class."""
def test_empty(self):
h = Host()
self.assertTrue(len(h.get_known_ports()) == 0)
def test_format_name(self):
h = Host()
self.assertTrue(isinstance(h.format_name(), basestring))
h.add_address("ipv4", "127.0.0.1")
self.assertTrue(isinstance(h.format_name(), basestring))
h.add_hostname("localhost")
self.assertTrue(isinstance(h.format_name(), basestring))
h.remove_address("ipv4", "127.0.0.1")
def test_empty_get_port(self):
h = Host()
for num in 10, 100, 1000, 10000:
for proto in ("tcp", "udp", "ip"):
port = h.ports[(num, proto)]
self.assertTrue(port.state == Port.UNKNOWN)
def test_add_port(self):
h = Host()
spec = (10, "tcp")
port = h.ports[spec]
self.assertTrue(port.state == Port.UNKNOWN, "Port state is %s, expected %s." % (port.get_state_string(), "unknown"))
h.add_port(spec, "open")
self.assertTrue(len(h.get_known_ports()) == 1)
port = h.ports[spec]
self.assertTrue(port.state == "open", "Port state is %s, expected %s." % (port.get_state_string(), "open"))
h.add_port(spec, "closed")
self.assertTrue(len(h.get_known_ports()) == 1)
port = h.ports[spec]
self.assertTrue(port.state == "closed", "Port state is %s, expected %s." % (port.get_state_string(), "closed"))
def test_swap_ports(self):
h = Host()
spec_a = (10, "tcp")
spec_b = (20, "tcp")
h.swap_ports(spec_a, spec_b)
self.assertTrue(h.ports[spec_a].state == Port.UNKNOWN)
self.assertTrue(h.ports[spec_b].state == Port.UNKNOWN)
self.assertTrue(h.ports[spec_a].spec == spec_a)
self.assertTrue(h.ports[spec_b].spec == spec_b)
h.add_port(spec_a, "open")
h.swap_ports(spec_a, spec_b)
self.assertTrue(h.ports[spec_a].state == Port.UNKNOWN)
self.assertTrue(h.ports[spec_b].state == "open")
self.assertTrue(h.ports[spec_a].spec == spec_a)
self.assertTrue(h.ports[spec_b].spec == spec_b)
h.add_port(spec_a, "closed")
h.swap_ports(spec_a, spec_b)
self.assertTrue(h.ports[spec_a].state == "open")
self.assertTrue(h.ports[spec_b].state == "closed")
self.assertTrue(h.ports[spec_a].spec == spec_a)
self.assertTrue(h.ports[spec_b].spec == spec_b)
def host_apply_diff(host, diff):
"""Apply a host diff to the given host."""
for hunk in diff:
if hunk.type == HOST_STATE_CHANGE:
assert host.state == hunk.a_state
host.state = hunk.b_state
elif hunk.type == HOST_ADDRESS_ADD:
host.add_address(hunk.address_type, hunk.address)
elif hunk.type == HOST_ADDRESS_REMOVE:
host.remove_address(hunk.address_type, hunk.address)
elif hunk.type == HOST_HOSTNAME_ADD:
host.add_hostname(hunk.hostname)
elif hunk.type == HOST_HOSTNAME_REMOVE:
host.remove_hostname(hunk.hostname)
elif hunk.type == PORT_ID_CHANGE:
host.swap_ports(hunk.a_spec, hunk.b_spec)
elif hunk.type == PORT_STATE_CHANGE:
port = host.ports[hunk.spec]
assert port.state == hunk.a_state
host.add_port(hunk.spec, hunk.b_state)
else:
assert False
class host_diff_test(unittest.TestCase):
"""Test the host_diff function."""
PORT_DIFF_HUNK_TYPES = (PORT_ID_CHANGE, PORT_STATE_CHANGE)
HOST_DIFF_HUNK_TYPES = (HOST_STATE_CHANGE,) + PORT_DIFF_HUNK_TYPES
def test_empty(self):
a = Host()
b = Host()
diff = host_diff(a, b)
self.assertTrue(len(diff) == 0)
def test_self(self):
h = Host()
h.add_port((10, "tcp"), "open")
h.add_port((22, "tcp"), "closed")
diff = host_diff(h, h)
self.assertTrue(len(diff) == 0)
def test_state_change(self):
a = Host()
b = Host()
a.state = "up"
b.state = "down"
diff = host_diff(a, b)
self.assertTrue(len(diff) > 0)
for hunk in diff:
self.assertTrue(hunk.type in self.HOST_DIFF_HUNK_TYPES)
def test_state_change_unknown(self):
a = Host()
b = Host()
a.state = "up"
diff = host_diff(a, b)
self.assertTrue(len(diff) > 0)
for hunk in diff:
self.assertTrue(hunk.type in self.HOST_DIFF_HUNK_TYPES)
diff = host_diff(b, a)
self.assertTrue(len(diff) > 0)
for hunk in diff:
self.assertTrue(hunk.type in self.HOST_DIFF_HUNK_TYPES)
def test_port_state_change(self):
a = Host()
b = Host()
spec = (10, "tcp")
a.add_port(spec, "open")
b.add_port(spec, "closed")
diff = host_diff(a, b)
self.assertTrue(len(diff) > 0)
for hunk in diff:
self.assertTrue(hunk.type in self.PORT_DIFF_HUNK_TYPES)
def test_port_state_change_unknown(self):
a = Host()
b = Host()
b.add_port((10, "tcp"), "open")
diff = host_diff(a, b)
self.assertTrue(len(diff) > 0)
for hunk in diff:
self.assertTrue(hunk.type in self.PORT_DIFF_HUNK_TYPES)
diff = host_diff(b, a)
self.assertTrue(len(diff) == 0)
for hunk in diff:
self.assertTrue(hunk.type in self.PORT_DIFF_HUNK_TYPES)
def test_port_state_change_multi(self):
a = Host()
b = Host()
a.add_port((10, "tcp"), "open")
a.add_port((20, "tcp"), "closed")
a.add_port((30, "tcp"), "open")
b.add_port((10, "tcp"), "open")
b.add_port((20, "tcp"), "open")
b.add_port((30, "tcp"), "open")
diff = host_diff(a, b)
self.assertTrue(len(diff) > 0)
for hunk in diff:
self.assertTrue(hunk.type in self.PORT_DIFF_HUNK_TYPES)
def test_address_add(self):
a = Host()
b = Host()
a.addresses = []
b.addresses = [("ipv4", "127.0.0.2")]
diff = host_diff(a, b)
self.assertTrue(len(diff) > 0)
for hunk in diff:
self.assertTrue(hunk.type == HOST_ADDRESS_ADD)
def test_address_add(self):
a = Host()
b = Host()
a.addresses = [("ipv4", "127.0.0.1")]
b.addresses = []
diff = host_diff(a, b)
self.assertTrue(len(diff) > 0)
for hunk in diff:
self.assertTrue(hunk.type == HOST_ADDRESS_REMOVE)
def test_address_add(self):
a = Host()
b = Host()
a.addresses = [("ipv4", "127.0.0.1")]
b.addresses = [("ipv4", "127.0.0.2")]
diff = host_diff(a, b)
self.assertTrue(len(diff) > 0)
for hunk in diff:
self.assertTrue(hunk.type in (HOST_ADDRESS_ADD, HOST_ADDRESS_REMOVE))
def test_hostname_add(self):
a = Host()
b = Host()
a.hostnames = []
b.hostnames = ["b"]
diff = host_diff(a, b)
self.assertTrue(len(diff) > 0)
for hunk in diff:
self.assertTrue(hunk.type == HOST_HOSTNAME_ADD)
def test_hostname_remove(self):
a = Host()
b = Host()
a.hostnames = ["a"]
b.hostnames = []
diff = host_diff(a, b)
self.assertTrue(len(diff) > 0)
for hunk in diff:
self.assertTrue(hunk.type == HOST_HOSTNAME_REMOVE)
def test_hostname_change(self):
a = Host()
b = Host()
a.hostnames = ["a"]
b.hostnames = ["b"]
diff = host_diff(a, b)
self.assertTrue(len(diff) > 0)
for hunk in diff:
self.assertTrue(hunk.type in (HOST_HOSTNAME_ADD, HOST_HOSTNAME_REMOVE))
def test_diff_is_effective(self):
"""Test that a host diff is effective.
This means that if the recommended changes are applied to the first host
the hosts become the same."""
a = Host()
b = Host()
a.add_port((10, "tcp"), "open")
a.add_port((20, "tcp"), "closed")
a.add_port((40, "udp"), "open|filtered")
b.add_port((10, "tcp"), "open")
b.add_port((30, "tcp"), "open")
a.add_port((40, "udp"), "open")
a.hostnames = ["a", "localhost"]
a.hostnames = ["b", "localhost", "b.example.com"]
diff = host_diff(a, b)
host_apply_diff(a, diff)
diff = host_diff(a, b)
self.assertTrue(len(diff) == 0)
class scan_test(unittest.TestCase):
"""Test the Scan class."""
def test_empty(self):
scan = Scan()
scan.load_from_file("test-scans/empty.xml")
self.assertTrue(len(scan.hosts) == 0)
self.assertTrue(scan.start_date is not None)
self.assertTrue(scan.end_date is not None)
def test_single(self):
scan = Scan()
scan.load_from_file("test-scans/single.xml")
self.assertTrue(len(scan.hosts) == 1)
def test_simple(self):
"""Test that the correct number of known ports is returned when there
are no extraports."""
scan = Scan()
scan.load_from_file("test-scans/simple.xml")
host = scan.hosts[0]
self.assertTrue(len(host.get_known_ports()) == 2,
u"Expected %d known ports, got %d." % (2, len(host.get_known_ports())))
def test_extraports(self):
"""Test that the correct number of known ports is returned when there
are extraports in only one state."""
scan = Scan()
scan.load_from_file("test-scans/single.xml")
host = scan.hosts[0]
self.assertTrue(len(host.get_known_ports()) == 100,
u"Expected %d known ports, got %d." % (100, len(host.get_known_ports())))
def test_extraports_multi(self):
"""Test that the correct number of known ports is returned when there
are extraports in more than one state."""
scan = Scan()
scan.load_from_file("test-scans/complex.xml")
host = scan.hosts[0]
self.assertTrue(len(host.get_known_ports()) == 5,
u"Expected %d known ports, got %d." % (5, len(host.get_known_ports())))
def test_addresses(self):
"""Test that addresses are recorded."""
scan = Scan()
scan.load_from_file("test-scans/simple.xml")
host = scan.hosts[0]
self.assertTrue(len(host.addresses) == 1)
def test_hostname(self):
"""Test that hostnames are recorded."""
scan = Scan()
scan.load_from_file("test-scans/simple.xml")
host = scan.hosts[0]
self.assertTrue(len(host.hostnames) == 1)
self.assertTrue(host.hostnames[0] == u"scanme.nmap.org")
# This test is commented out because doesn't store any information about down
# hosts, not even the fact that they are down. Recovering the list of scanned
# hosts to infer which ones are down would involve parsing the targets out of
# the /nmaprun/@args attribute (which is non-trivial) and possibly looking up
# their addresses.
# def test_down_state(self):
# """Test that hosts that are not marked "up" are in the "down" state."""
# scan = Scan()
# scan.load_from_file("test-scans/down.xml")
# self.assertTrue(len(scan.hosts) == 1)
# host = scan.hosts[0]
# self.assertTrue(host.state == "down")
def scan_apply_diff(scan, diff):
"""Apply a scan diff to the given scan."""
for host, h_diff in diff:
for h in scan.hosts:
if h == host:
break
else:
h = Host()
scan.hosts.append(h)
host_apply_diff(h, h_diff)
class scan_diff_test(unittest.TestCase):
"""Test the scan_diff function."""
def test_self(self):
scan = Scan()
scan.load_from_file("test-scans/complex.xml")
diff = scan_diff(scan, scan)
self.assertTrue(len(diff) == 0)
def test_unknown_up(self):
a = Scan()
a.load_from_file("test-scans/empty.xml")
b = Scan()
b.load_from_file("test-scans/simple.xml")
diff = scan_diff(a, b)
for host, h_diff in diff:
for hunk in h_diff:
if hunk.type == HOST_STATE_CHANGE:
self.assertTrue(hunk.a_state == Port.UNKNOWN)
self.assertTrue(hunk.b_state == u"up")
break
else:
fail("No host state change found.")
def test_up_unknown(self):
a = Scan()
a.load_from_file("test-scans/simple.xml")
b = Scan()
b.load_from_file("test-scans/empty.xml")
diff = scan_diff(a, b)
self.assertTrue(len(diff) == 0)
def test_diff_is_effective(self):
"""Test that a scan diff is effective.
This means that if the recommended changes are applied to the first scan
the scans become the same."""
a = Scan()
a.load_from_file("test-scans/empty.xml")
b = Scan()
b.load_from_file("test-scans/simple.xml")
diff = scan_diff(a, b)
self.assertTrue(len(diff) > 0)
scan_apply_diff(a, diff)
diff = scan_diff(a, b)
self.assertTrue(len(diff) == 0)
class scan_diff_xml_test(unittest.TestCase):
def setUp(self):
a = Scan()
a.load_from_file("test-scans/empty.xml")
b = Scan()
b.load_from_file("test-scans/simple.xml")
self.scan_diff = ScanDiff(a, b)
f = StringIO.StringIO()
self.scan_diff.print_xml(f)
self.xml = f.getvalue()
f.close()
def test_well_formed(self):
try:
document = xml.dom.minidom.parseString(self.xml)
except Exception, e:
fail(u"Parsing XML diff output caused the exception: %s" % str(e))
unittest.main()

19
ndiff/setup.py Normal file
View File

@@ -0,0 +1,19 @@
#!/usr/bin/env python
from distutils.core import setup
from distutils.cmd import Command
class null_command(Command):
"""This is a dummy distutils command that does nothing. We use it to replace
the install_egg_info and avoid installing a .egg-info file, because there's
no option to disable that."""
def initialize_options(self):
pass
def finalize_options(self):
pass
def run(self):
pass
setup(name = u"ndiff", scripts = [u"ndiff"],
data_files = [(u"share/man/man1", [u"docs/ndiff.1"])],
cmdclass = {"install_egg_info": null_command})

112
ndiff/test-scans/anonymize.py Executable file
View File

@@ -0,0 +1,112 @@
#!/usr/bin/env python
# Anonymize an Nmap XML file, replacing host name and IP addresses with random
# anonymous ones. Anonymized names will be consistent between runs of the
# program. Give a file name as an argument. The anonymized file is written to
# stdout.
#
# The anonymization is not rigorous. This program just matches regular
# expressions against things that look like address and host names. It is
# possible that it will leave some identifying information, for example a host
# name split across lines in a service fingerprint.
import hashlib
import random
import re
import sys
VERBOSE = True
r = random.Random()
def hash(s):
digest = hashlib.sha512(s).hexdigest()
return int(digest, 16)
def anonymize_mac_address(addr):
r.seed(hash(addr))
nums = (0, 0, 0) + tuple(r.randrange(256) for i in range(3))
return u":".join(u"%02X" % x for x in nums)
def anonymize_ipv4_address(addr):
r.seed(hash(addr))
nums = (10,) + tuple(r.randrange(256) for i in range(3))
return u".".join(unicode(x) for x in nums)
def anonymize_ipv6_address(addr):
r.seed(hash(addr))
# RFC 4193.
nums = (0xFD00 + r.randrange(256),)
nums = nums + tuple(r.randrange(65536) for i in range(7))
return u":".join("%04X" % x for x in nums)
# Maps to memoize address and host name conversions.
hostname_map = {}
address_map = {}
def anonymize_hostname(name):
if name in hostname_map:
return hostname_map[name]
LETTERS = "acbdefghijklmnopqrstuvwxyz"
r.seed(hash(name))
length = r.randrange(5, 10)
prefix = u"".join(r.sample(LETTERS, length))
num = r.randrange(1000)
hostname_map[name] = u"%s-%d.example.com" % (prefix, num)
if VERBOSE:
print >> sys.stderr, "Replace %s with %s" % (name, hostname_map[name])
return hostname_map[name]
mac_re = re.compile(r'\b([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}\b')
ipv4_re = re.compile(r'\b([0-9]{1,3}\.){3}[0-9]{1,3}\b')
ipv6_re = re.compile(r'\b([0-9a-fA-F]{1,4}::?){3,}[0-9a-fA-F]{1,4}\b')
def anonymize_address(addr):
if addr in address_map:
return address_map[addr]
if mac_re.match(addr):
address_map[addr] = anonymize_mac_address(addr)
elif ipv4_re.match(addr):
address_map[addr] = anonymize_ipv4_address(addr)
elif ipv6_re.match(addr):
address_map[addr] = anonymize_ipv6_address(addr)
else:
assert False
if VERBOSE:
print >> sys.stderr, "Replace %s with %s" % (addr, address_map[addr])
return address_map[addr]
def repl_addr(match):
addr = match.group(0)
anon_addr = anonymize_address(addr)
return anon_addr
def repl_hostname_name(match):
name = match.group(1)
anon_name = anonymize_hostname(name)
return r'<hostname name="%s"' % anon_name
def repl_hostname(match):
name = match.group(1)
anon_name = anonymize_hostname(name)
return r'hostname="%s"' % anon_name
def anonymize_file(f):
for line in f:
repls = []
line = re.sub(mac_re, repl_addr, line)
line = re.sub(ipv4_re, repl_addr, line)
line = re.sub(ipv6_re, repl_addr, line)
line = re.sub(r'<hostname name="([^"]*)"', repl_hostname_name, line)
line = re.sub(r'\bhostname="([^"]*)"', repl_hostname, line)
yield line
def main():
filename = sys.argv[1]
f = open(filename, "r")
for line in anonymize_file(f):
sys.stdout.write(line)
f.close()
if __name__ == "__main__":
main()

29
ndiff/test-scans/complex.xml Normal file
View File

@@ -0,0 +1,29 @@
<?xml version="1.0" ?>
<?xml-stylesheet href="/usr/share/nmap/nmap.xsl" type="text/xsl"?>
<!-- This scan has two scaninfos and two extraports. -->
<!-- Nmap 4.68 scan initiated Thu Sep 4 20:02:52 2008 as: nmap -oX complex.xml -sS -sU -p T:1-100,U:1-100 scanme.nmap.org -->
<nmaprun scanner="nmap" args="nmap -oX complex.xml -sS -sU -p T:1-100,U:1-100 scanme.nmap.org" start="1220580172" startstr="Thu Sep 4 20:02:52 2008" version="4.68" xmloutputversion="1.02">
<scaninfo type="syn" protocol="tcp" numservices="100" services="1-100" />
<scaninfo type="udp" protocol="udp" numservices="100" services="1-100" />
<verbose level="0" />
<debugging level="0" />
<host starttime="1220580172" endtime="1220580176"><status state="up" reason="echo-reply"/>
<address addr="64.13.134.52" addrtype="ipv4" />
<hostnames><hostname name="scanme.nmap.org" type="PTR" /></hostnames>
<ports><extraports state="open|filtered" count="100">
<extrareasons reason="no-responses" count="100"/>
</extraports>
<extraports state="filtered" count="95">
<extrareasons reason="no-responses" count="95"/>
</extraports>
<port protocol="tcp" portid="22"><state state="open" reason="syn-ack" reason_ttl="51"/><service name="ssh" method="table" conf="3" /></port>
<port protocol="tcp" portid="25"><state state="closed" reason="reset" reason_ttl="51"/><service name="smtp" method="table" conf="3" /></port>
<port protocol="tcp" portid="53"><state state="open" reason="syn-ack" reason_ttl="51"/><service name="domain" method="table" conf="3" /></port>
<port protocol="tcp" portid="70"><state state="closed" reason="reset" reason_ttl="51"/><service name="gopher" method="table" conf="3" /></port>
<port protocol="tcp" portid="80"><state state="open" reason="syn-ack" reason_ttl="51"/><service name="http" method="table" conf="3" /></port>
</ports>
<times srtt="85526" rttvar="14010" to="141566" />
</host>
<runstats><finished time="1220580176" timestr="Thu Sep 4 20:02:56 2008"/><hosts up="1" down="0" total="1" />
<!-- Nmap done at Thu Sep 4 20:02:56 2008; 1 IP address (1 host up) scanned in 4.00 seconds -->
</runstats></nmaprun>

11
ndiff/test-scans/down.xml Normal file
View File

@@ -0,0 +1,11 @@
<?xml version="1.0" ?>
<?xml-stylesheet href="/usr/share/nmap/nmap.xsl" type="text/xsl"?>
<!-- This scan has a down host. -->
<!-- Nmap 4.75 scan initiated Thu Sep 11 14:13:09 2008 as: nmap -oX down.xml -p 1-100 -PS100 scanme.nmap.org -->
<nmaprun scanner="nmap" args="nmap -oX down.xml -p 1-100 -PS100 scanme.nmap.org" start="1221163989" startstr="Thu Sep 11 14:13:09 2008" version="4.75" xmloutputversion="1.02">
<scaninfo type="syn" protocol="tcp" numservices="100" services="1-100" />
<verbose level="0" />
<debugging level="0" />
<runstats><finished time="1221163991" timestr="Thu Sep 11 14:13:11 2008"/><hosts up="0" down="1" total="1" />
<!-- Nmap done at Thu Sep 11 14:13:11 2008; 1 IP address (0 hosts up) scanned in 2.35 seconds -->
</runstats></nmaprun>

10
ndiff/test-scans/empty.xml Normal file
View File

@@ -0,0 +1,10 @@
<?xml version="1.0" ?>
<?xml-stylesheet href="/usr/share/nmap/nmap.xsl" type="text/xsl"?>
<!-- Nmap 4.68 scan initiated Wed Sep 3 20:03:57 2008 as: nmap -oX empty.xml -p 1-100 -->
<nmaprun scanner="nmap" args="nmap -oX empty.xml -p 1-100" start="1220493837" startstr="Wed Sep 3 20:03:57 2008" version="4.68" xmloutputversion="1.02">
<scaninfo type="syn" protocol="tcp" numservices="100" services="1-100" />
<verbose level="0" />
<debugging level="0" />
<runstats><finished time="1220493837" timestr="Wed Sep 3 20:03:57 2008"/><hosts up="0" down="0" total="0" />
<!-- Nmap done at Wed Sep 3 20:03:57 2008; 0 IP addresses (0 hosts up) scanned in 0.03 seconds -->
</runstats></nmaprun>

118
ndiff/test-scans/random-1.xml Normal file
View File

File diff suppressed because one or more lines are too long

127
ndiff/test-scans/random-2.xml Normal file
View File

File diff suppressed because one or more lines are too long

19
ndiff/test-scans/simple.xml Normal file
View File

@@ -0,0 +1,19 @@
<?xml version="1.0" ?>
<?xml-stylesheet href="/usr/share/nmap/nmap.xsl" type="text/xsl"?>
<!-- This simple scan has one scaninfo and no extraports. -->
<!-- Nmap 4.68 scan initiated Wed Sep 3 19:49:34 2008 as: nmap -oX simple.xml -p 22,113 scanme.nmap.org -->
<nmaprun scanner="nmap" args="nmap -oX simple.xml -p 22,113 scanme.nmap.org" start="1220492974" startstr="Wed Sep 3 19:49:34 2008" version="4.68" xmloutputversion="1.02">
<scaninfo type="syn" protocol="tcp" numservices="2" services="22,113" />
<verbose level="0" />
<debugging level="0" />
<host starttime="1220492974" endtime="1220492975"><status state="up" reason="echo-reply"/>
<address addr="64.13.134.52" addrtype="ipv4" />
<hostnames><hostname name="scanme.nmap.org" type="PTR" /></hostnames>
<ports><port protocol="tcp" portid="22"><state state="open" reason="syn-ack" reason_ttl="51"/><service name="ssh" method="table" conf="3" /></port>
<port protocol="tcp" portid="113"><state state="closed" reason="reset" reason_ttl="51"/><service name="auth" method="table" conf="3" /></port>
</ports>
<times srtt="85540" rttvar="49241" to="282504" />
</host>
<runstats><finished time="1220492975" timestr="Wed Sep 3 19:49:35 2008"/><hosts up="1" down="0" total="1" />
<!-- Nmap done at Wed Sep 3 19:49:35 2008; 1 IP address (1 host up) scanned in 0.48 seconds -->
</runstats></nmaprun>

25
ndiff/test-scans/single.xml Normal file
View File

@@ -0,0 +1,25 @@
<?xml version="1.0" ?>
<?xml-stylesheet href="/usr/share/nmap/nmap.xsl" type="text/xsl"?>
<!-- This scan has one scaninfo and one extraports. -->
<!-- Nmap 4.68 scan initiated Wed Sep 3 20:04:31 2008 as: nmap -oX single.xml -p 1-100 scanme.nmap.org -->
<nmaprun scanner="nmap" args="nmap -oX single.xml -p 1-100 scanme.nmap.org" start="1220493871" startstr="Wed Sep 3 20:04:31 2008" version="4.68" xmloutputversion="1.02">
<scaninfo type="syn" protocol="tcp" numservices="100" services="1-100" />
<verbose level="0" />
<debugging level="0" />
<host starttime="1220493871" endtime="1220493872"><status state="up" reason="echo-reply"/>
<address addr="64.13.134.52" addrtype="ipv4" />
<hostnames><hostname name="scanme.nmap.org" type="PTR" /></hostnames>
<ports><extraports state="filtered" count="95">
<extrareasons reason="no-responses" count="95"/>
</extraports>
<port protocol="tcp" portid="22"><state state="open" reason="syn-ack" reason_ttl="51"/><service name="ssh" method="table" conf="3" /></port>
<port protocol="tcp" portid="25"><state state="closed" reason="reset" reason_ttl="51"/><service name="smtp" method="table" conf="3" /></port>
<port protocol="tcp" portid="53"><state state="open" reason="syn-ack" reason_ttl="51"/><service name="domain" method="table" conf="3" /></port>
<port protocol="tcp" portid="70"><state state="closed" reason="reset" reason_ttl="51"/><service name="gopher" method="table" conf="3" /></port>
<port protocol="tcp" portid="80"><state state="open" reason="syn-ack" reason_ttl="51"/><service name="http" method="table" conf="3" /></port>
</ports>
<times srtt="88828" rttvar="25643" to="191400" />
</host>
<runstats><finished time="1220493873" timestr="Wed Sep 3 20:04:33 2008"/><hosts up="1" down="0" total="1" />
<!-- Nmap done at Wed Sep 3 20:04:33 2008; 1 IP address (1 host up) scanned in 2.02 seconds -->
</runstats></nmaprun>