tomsellers b82c819afb Update to add additional blacklist entries the httpspider library. The goal is to avoid downloading and processing certain additional video, audio and binary formats. ...

This should speed up crawling certain sites.  In the case of http-email-harvest it should reduce some of the false positives generated by running the RegEx against binary data. The only script that this appears likely to have affected the results of would have been http-sitemap-generator and that script specifically disables the blacklist.

2012-07-10 00:23:02 +00:00

28 KiB

Raw Blame History

View Raw