* Instead of printing all disallowed entries it now only prints the
first 20. In verbose and debug modes it prints more entries, the exact
number depends on the debug and vebose level
* Prints out the number of disallowed entries
* Prints a message if a robot.txt file exists but has not entries.
* More resilient parsing and duplicate removal code
* Extended the line length to 50 characters so less vertical space used
7cf7259f9d