Fixes #3560

Trivial update
Adding SQLi vulnserver (for testing purposes)
2025-12-09 06:01:29 +00:00 · 2019-04-01 09:47:36 +02:00 · 2019-03-29 16:23:39 +01:00 · 2019-03-29 11:04:58 +01:00 · 2019-03-29 02:28:16 +01:00 · 2019-03-29 01:25:34 +01:00
450 changed files with 6111 additions and 5934 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -3,6 +3,8 @@
 *.md5 text eol=lf
 *.py text eol=lf
 *.xml text eol=lf
+LICENSE text eol=lf
+COMMITMENT text eol=lf

 *_ binary
 *.dll binary
--- a/.github/ISSUE_TEMPLATE.md
+++ b/.github/ISSUE_TEMPLATE.md
@@ -19,7 +19,7 @@
 * Client OS (e.g. `Microsoft Windows 10`)
 * Program version (`python sqlmap.py --version` or `sqlmap --version` depending on installation): 
 * Target DBMS (e.g. `Microsoft SQL Server`): 
-* Detected WAF/IDS/IPS protection (e.g. `ModSecurity` or `unknown`):
+* Detected WAF/IPS protection (e.g. `ModSecurity` or `unknown`):
 * SQLi techniques found by sqlmap (e.g. `error-based` and `boolean-based blind`):
 * Results of manual target assessment (e.g. found that the payload `query=test' AND 4113 IN ((SELECT 'foobar'))-- qKLV` works):
 * Relevant console output (if any):
--- a/46
+++ b/46
@@ -0,0 +1,46 @@
+GPL Cooperation Commitment
+Version 1.0
+
+Before filing or continuing to prosecute any legal proceeding or claim
+(other than a Defensive Action) arising from termination of a Covered
+License, we commit to extend to the person or entity ('you') accused
+of violating the Covered License the following provisions regarding
+cure and reinstatement, taken from GPL version 3. As used here, the
+term 'this License' refers to the specific Covered License being
+enforced.
+
+    However, if you cease all violation of this License, then your
+    license from a particular copyright holder is reinstated (a)
+    provisionally, unless and until the copyright holder explicitly
+    and finally terminates your license, and (b) permanently, if the
+    copyright holder fails to notify you of the violation by some
+    reasonable means prior to 60 days after the cessation.
+
+    Moreover, your license from a particular copyright holder is
+    reinstated permanently if the copyright holder notifies you of the
+    violation by some reasonable means, this is the first time you
+    have received notice of violation of this License (for any work)
+    from that copyright holder, and you cure the violation prior to 30
+    days after your receipt of the notice.
+
+We intend this Commitment to be irrevocable, and binding and
+enforceable against us and assignees of or successors to our
+copyrights.
+
+Definitions
+
+'Covered License' means the GNU General Public License, version 2
+(GPLv2), the GNU Lesser General Public License, version 2.1
+(LGPLv2.1), or the GNU Library General Public License, version 2
+(LGPLv2), all as published by the Free Software Foundation.
+
+'Defensive Action' means a legal proceeding or claim that We bring
+against you in response to a prior proceeding or claim initiated by
+you or your affiliate.
+
+'We' means each contributor to this repository as of the date of
+inclusion of this file, including subsidiaries of a corporate
+contributor.
+
+This work is available under a Creative Commons Attribution-ShareAlike
+4.0 International license (https://creativecommons.org/licenses/by-sa/4.0/).
--- a/2
+++ b/2
@@ -1,7 +1,7 @@
 COPYING -- Describes the terms under which sqlmap is distributed. A copy
 of the GNU General Public License (GPL) is appended to this file.

-sqlmap is (C) 2006-2018 Bernardo Damele Assumpcao Guimaraes, Miroslav Stampar.
+sqlmap is (C) 2006-2019 Bernardo Damele Assumpcao Guimaraes, Miroslav Stampar.

 This program is free software; you may redistribute and/or modify it under
 the terms of the GNU General Public License as published by the Free
--- a/README.md
+++ b/README.md
@@ -1,6 +1,6 @@
 # sqlmap

-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)

 sqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers. It comes with a powerful detection engine, many niche features for the ultimate penetration tester and a broad range of switches lasting from database fingerprinting, over data fetching from the database, to accessing the underlying file system and executing commands on the operating system via out-of-band connections.

@@ -67,3 +67,4 @@ Translations
 * [Russian](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-ru-RUS.md)
 * [Spanish](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-es-MX.md)
 * [Turkish](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-tr-TR.md)
+* [Ukrainian](https://github.com/sqlmapproject/sqlmap/blob/master/doc/translations/README-uk-UA.md)
--- a/doc/THANKS.md
+++ b/doc/THANKS.md
@@ -764,6 +764,9 @@ ultramegaman, <seclists(at)ultramegaman.com>
 Vinicius, <viniciusmaxdaloop(at)gmail.com>
 * for reporting a minor bug

+w8ay
+* for contributing an implementation for chunked transfer-encoding (switch --chunked)
+
 wanglei, <wanglei(at)17uxi.cn>
 * for reporting a minor bug

--- a/doc/THIRD-PARTY.md
+++ b/doc/THIRD-PARTY.md
@@ -15,11 +15,6 @@ This file lists bundled packages and their associated licensing terms.
  Copyright (C) 2013, Jonathan Hartley.
 * The Fcrypt library located under thirdparty/fcrypt/.
  Copyright (C) 2000, 2001, 2004 Carey Evans.
-* The Odict library located under thirdparty/odict/.
-  Copyright (C) 2005, Nicola Larosa, Michael Foord.
-* The Oset library located under thirdparty/oset/.
-  Copyright (C) 2010, BlueDynamics Alliance, Austria.
-  Copyright (C) 2009, Raymond Hettinger, and others.
 * The PrettyPrint library located under thirdparty/prettyprint/.
  Copyright (C) 2010, Chris Hall.
 * The SocksiPy library located under thirdparty/socks/.
@@ -281,6 +276,10 @@ be bound by the terms and conditions of this License Agreement.

 * The bottle web framework library located under thirdparty/bottle/.
  Copyright (C) 2012, Marcel Hellkamp.
+* The ordereddict library located under thirdparty/odict/.
+  Copyright (C) 2009, Raymond Hettinger.
+* The six Python 2 and 3 compatibility library located under thirdparty/six/.
+  Copyright (C) 2010-2018, Benjamin Peterson.
 * The Termcolor library located under thirdparty/termcolor/.
  Copyright (C) 2008-2011, Volvox Development Team.

--- a/doc/translations/README-bg-BG.md
+++ b/doc/translations/README-bg-BG.md
@@ -1,6 +1,6 @@
 # sqlmap

-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![Лиценз](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![Лиценз](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)

 sqlmap e инструмент за тестване и проникване, с отворен код, който автоматизира процеса на откриване и използване на недостатъците на SQL база данните чрез SQL инжекция, която ги взима от сървъра. Снабден е с мощен детектор, множество специални функции за най-добрия тестер и широк спектър от функции, които могат да се използват за множество цели - извличане на данни от базата данни, достъп до основната файлова система и изпълняване на команди на операционната система.

--- a/doc/translations/README-es-MX.md
+++ b/doc/translations/README-es-MX.md
@@ -1,6 +1,6 @@
 # sqlmap

-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)

 sqlmap es una herramienta para pruebas de penetración "penetration testing" de software libre que automatiza el proceso de detección y explotación de fallos mediante inyección de SQL además de tomar el control de servidores de bases de datos. Contiene un poderoso motor de detección, así como muchas de las funcionalidades escenciales para el "pentester" y una amplia gama de opciones desde la recopilación de información para identificar el objetivo conocido como "fingerprinting" mediante la extracción de información de la base de datos, hasta el acceso al sistema de archivos subyacente para ejecutar comandos en el sistema operativo a través de conexiones alternativas conocidas como "Out-of-band".

--- a/doc/translations/README-fr-FR.md
+++ b/doc/translations/README-fr-FR.md
@@ -1,6 +1,6 @@
 # sqlmap

-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)

 **sqlmap** est un outil Open Source de test d'intrusion. Cet outil permet d'automatiser le processus de détection et d'exploitation des failles d'injection SQL afin de prendre le contrôle des serveurs de base de données. __sqlmap__ dispose d'un puissant moteur de détection utilisant les techniques les plus récentes et les plus dévastatrices de tests d'intrusion comme L'Injection SQL, qui permet d'accéder à la base de données, au système de fichiers sous-jacent et permet aussi l'exécution des commandes sur le système d'exploitation.

--- a/doc/translations/README-gr-GR.md
+++ b/doc/translations/README-gr-GR.md
@@ -1,6 +1,6 @@
 # sqlmap

-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)

 Το sqlmap είναι πρόγραμμα ανοιχτού κώδικα, που αυτοματοποιεί την εύρεση και εκμετάλλευση ευπαθειών τύπου SQL Injection σε βάσεις δεδομένων. Έρχεται με μια δυνατή μηχανή αναγνώρισης ευπαθειών, πολλά εξειδικευμένα χαρακτηριστικά για τον απόλυτο penetration tester όπως και με ένα μεγάλο εύρος επιλογών αρχίζοντας από την αναγνώριση της βάσης δεδομένων, κατέβασμα δεδομένων της βάσης, μέχρι και πρόσβαση στο βαθύτερο σύστημα αρχείων και εκτέλεση εντολών στο απευθείας στο λειτουργικό μέσω εκτός ζώνης συνδέσεων.

--- a/doc/translations/README-hr-HR.md
+++ b/doc/translations/README-hr-HR.md
@@ -1,6 +1,6 @@
 # sqlmap

-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)

 sqlmap je alat namijenjen za penetracijsko testiranje koji automatizira proces detekcije i eksploatacije sigurnosnih propusta SQL injekcije te preuzimanje poslužitelja baze podataka. Dolazi s moćnim mehanizmom za detekciju, mnoštvom korisnih opcija za napredno penetracijsko testiranje te široki spektar opcija od onih za prepoznavanja baze podataka, preko dohvaćanja podataka iz baze, do pristupa zahvaćenom datotečnom sustavu i izvršavanja komandi na operacijskom sustavu korištenjem tzv. "out-of-band" veza.

--- a/doc/translations/README-id-ID.md
+++ b/doc/translations/README-id-ID.md
@@ -1,6 +1,6 @@
 # sqlmap

-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)

 sqlmap merupakan alat _(tool)_ bantu _open source_ dalam melakukan tes penetrasi yang mengotomasi proses deteksi dan eksploitasi kelemahan _SQL injection_ dan pengambil-alihan server basisdata. sqlmap dilengkapi dengan pendeteksi canggih, fitur-fitur hanal bagi _penetration tester_, beragam cara untuk mendeteksi basisdata, hingga mengakses _file system_ dan mengeksekusi perintah dalam sistem operasi melalui koneksi _out-of-band_. 

--- a/doc/translations/README-it-IT.md
+++ b/doc/translations/README-it-IT.md
@@ -1,6 +1,6 @@
 # sqlmap

-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)

 sqlmap è uno strumento open source per il penetration testing. Il suo scopo è quello di rendere automatico il processo di scoperta ed exploit di vulnerabilità di tipo SQL injection al fine di compromettere database online. Dispone di un potente motore per la ricerca di vulnerabilità, molti strumenti di nicchia anche per il più esperto penetration tester ed un'ampia gamma di controlli che vanno dal fingerprinting di database allo scaricamento di dati, fino all'accesso al file system sottostante e l'esecuzione di comandi nel sistema operativo attraverso connessioni out-of-band.

--- a/doc/translations/README-ja-JP.md
+++ b/doc/translations/README-ja-JP.md
@@ -1,6 +1,6 @@
 # sqlmap

-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)

 sqlmapはオープンソースのペネトレーションテスティングツールです。SQLインジェクションの脆弱性の検出、活用、そしてデータベースサーバ奪取のプロセスを自動化します。
 強力な検出エンジン、ペネトレーションテスターのための多くのニッチ機能、持続的なデータベースのフィンガープリンティングから、データベースのデータ取得やアウトオブバンド接続を介したオペレーティング・システム上でのコマンド実行、ファイルシステムへのアクセスなどの広範囲に及ぶスイッチを提供します。
--- a/doc/translations/README-pl-PL.md
+++ b/doc/translations/README-pl-PL.md
@@ -1,6 +1,6 @@
 # sqlmap

-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)

 sqlmap to open sourceowe narzędzie do testów penetracyjnych, które automatyzuje procesy detekcji, przejmowania i testowania odporności serwerów SQL na podatność na iniekcję niechcianego kodu. Zawiera potężny mechanizm detekcji, wiele niszowych funkcji dla zaawansowanych testów penetracyjnych oraz szeroki wachlarz opcji począwszy od identyfikacji bazy danych, poprzez wydobywanie z nich danych, a nawet pozwalającuch na dostęp do systemu plików o uruchamianie poleceń w systemie operacyjnym serwera poprzez niestandardowe połączenia.

--- a/doc/translations/README-pt-BR.md
+++ b/doc/translations/README-pt-BR.md
@@ -1,6 +1,6 @@
 # sqlmap

-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)

 sqlmap é uma ferramenta de teste de penetração de código aberto que automatiza o processo de detecção e exploração de falhas de injeção SQL. Com essa ferramenta é possível assumir total controle de servidores de banco de dados em páginas web vulneráveis, inclusive de base de dados fora do sistema invadido. Ele possui um motor de detecção poderoso, empregando as últimas e mais devastadoras técnicas de teste de penetração por SQL Injection, que permite acessar a base de dados, o sistema de arquivos subjacente e executar comandos no sistema operacional.

--- a/doc/translations/README-ru-RUS.md
+++ b/doc/translations/README-ru-RUS.md
@@ -1,6 +1,6 @@
 # sqlmap

-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)

 sqlmap - это инструмент для тестирования уязвимостей с открытым исходным кодом, который автоматизирует процесс обнаружения и использования ошибок SQL-инъекций и захвата серверов баз данных. Он оснащен мощным механизмом обнаружения, множеством приятных функций для профессионального тестера уязвимостей и широким спектром скриптов, которые упрощают работу с базами данных, от сбора данных из базы данных, до доступа к базовой файловой системе и выполнения команд в операционной системе через out-of-band соединение.

--- a/doc/translations/README-tr-TR.md
+++ b/doc/translations/README-tr-TR.md
@@ -1,6 +1,6 @@
 # sqlmap

-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)

 sqlmap sql injection açıklarını otomatik olarak tespit ve istismar etmeye yarayan açık kaynak bir penetrasyon aracıdır. sqlmap gelişmiş tespit özelliğinin yanı sıra penetrasyon testleri sırasında gerekli olabilecek bir çok aracı, -uzak veritabınınından, veri indirmek, dosya sistemine erişmek, dosya çalıştırmak gibi - işlevleri de barındırmaktadır.

--- a/doc/translations/README-uk-UA.md
+++ b/doc/translations/README-uk-UA.md
@@ -0,0 +1,50 @@
+# sqlmap
+
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![PyPI version](https://badge.fury.io/py/sqlmap.svg)](https://badge.fury.io/py/sqlmap) [![GitHub closed issues](https://img.shields.io/github/issues-closed-raw/sqlmapproject/sqlmap.svg?colorB=ff69b4)](https://github.com/sqlmapproject/sqlmap/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)
+
+sqlmap - це інструмент для тестування вразливостей з відкритим сирцевим кодом, який автоматизує процес виявлення і використання дефектів SQL-ін'єкцій, а також захоплення серверів баз даних. Він оснащений потужним механізмом виявлення, безліччю приємних функцій для професійного тестувальника вразливостей і широким спектром скриптів, які спрощують роботу з базами даних - від відбитка бази даних до доступу до базової файлової системи та виконання команд в операційній системі через out-of-band з'єднання.
+
+Скриншоти
+----
+
+![Screenshot](https://raw.github.com/wiki/sqlmapproject/sqlmap/images/sqlmap_screenshot.png)
+
+Ви можете ознайомитися з [колекцією скриншотів](https://github.com/sqlmapproject/sqlmap/wiki/Screenshots), які демонструють деякі функції в wiki.
+
+Встановлення
+----
+
+Ви можете завантажити останню версію tarball натиснувши [сюди](https://github.com/sqlmapproject/sqlmap/tarball/master) або останню версію zipball натиснувши [сюди](https://github.com/sqlmapproject/sqlmap/zipball/master).
+
+Найкраще завантажити sqlmap шляхом клонування [Git](https://github.com/sqlmapproject/sqlmap) репозиторію:
+
+    git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
+
+sqlmap «працює з коробки» з [Python](http://www.python.org/download/) версії **2.6.x** та **2.7.x** на будь-якій платформі.
+
+Використання
+----
+
+Щоб отримати список основних опцій і перемикачів, використовуйте:
+
+    python sqlmap.py -h
+
+Щоб отримати список всіх опцій і перемикачів, використовуйте:
+
+    python sqlmap.py -hh
+
+Ви можете знайти приклад виконання [тут](https://asciinema.org/a/46601).
+Для того, щоб ознайомитися з можливостями sqlmap, списком підтримуваних функцій та описом всіх параметрів і перемикачів, а також прикладами, вам рекомендується скористатися [інструкцією користувача](https://github.com/sqlmapproject/sqlmap/wiki/Usage).
+
+Посилання
+----
+
+* Основний сайт: http://sqlmap.org
+* Завантаження: [.tar.gz](https://github.com/sqlmapproject/sqlmap/tarball/master) або [.zip](https://github.com/sqlmapproject/sqlmap/zipball/master)
+* Канал новин RSS: https://github.com/sqlmapproject/sqlmap/commits/master.atom
+* Відстеження проблем: https://github.com/sqlmapproject/sqlmap/issues
+* Інструкція користувача: https://github.com/sqlmapproject/sqlmap/wiki
+* Поширенні питання (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
+* Twitter: [@sqlmap](https://twitter.com/sqlmap)
+* Демо: [http://www.youtube.com/user/inquisb/videos](http://www.youtube.com/user/inquisb/videos)
+* Скриншоти: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots
--- a/doc/translations/README-zh-CN.md
+++ b/doc/translations/README-zh-CN.md
@@ -1,6 +1,6 @@
 # sqlmap

-[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://api.travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)
+[![Build Status](https://api.travis-ci.org/sqlmapproject/sqlmap.svg?branch=master)](https://travis-ci.org/sqlmapproject/sqlmap) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-GPLv2-red.svg)](https://raw.githubusercontent.com/sqlmapproject/sqlmap/master/LICENSE) [![Twitter](https://img.shields.io/badge/twitter-@sqlmap-blue.svg)](https://twitter.com/sqlmap)

 sqlmap 是一个开源的渗透测试工具，可以用来自动化的检测，利用SQL注入漏洞，获取数据库服务器的权限。它具有功能强大的检测引擎,针对各种不同类型数据库的渗透测试的功能选项，包括获取数据库中存储的数据，访问操作系统文件甚至可以通过外带数据连接的方式执行操作系统命令。

--- a/extra/init.py
+++ b/extra/init.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/extra/beep/init.py
+++ b/extra/beep/init.py
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/extra/beep/beep.py
+++ b/extra/beep/beep.py
@@ -3,12 +3,11 @@
 """
 beep.py - Make a beep sound

-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

 import os
-import subprocess
 import sys
 import wave

@@ -16,11 +15,11 @@ BEEP_WAV_FILENAME = os.path.join(os.path.dirname(__file__), "beep.wav")

 def beep():
    try:
-        if subprocess.mswindows:
+        if sys.platform == "nt":
            _win_wav_play(BEEP_WAV_FILENAME)
        elif sys.platform == "darwin":
            _mac_beep()
-        elif sys.platform == "linux2":
+        elif sys.platform.startswith("linux"):
            _linux_wav_play(BEEP_WAV_FILENAME)
        else:
            _speaker_beep()
--- a/extra/cloak/init.py
+++ b/extra/cloak/init.py
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/extra/cloak/cloak.py
+++ b/extra/cloak/cloak.py
@@ -3,24 +3,28 @@
 """
 cloak.py - Simple file encryption/compression utility

-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

+from __future__ import print_function
+
 import os
+import struct
 import sys
 import zlib

 from optparse import OptionError
 from optparse import OptionParser

+if sys.version_info >= (3, 0):
+    xrange = range
+
 def hideAscii(data):
-    retVal = ""
+    retVal = b""
    for i in xrange(len(data)):
-        if ord(data[i]) < 128:
-            retVal += chr(ord(data[i]) ^ 127)
-        else:
-            retVal += data[i]
+        value = data[i] if isinstance(data[i], int) else ord(data[i])
+        retVal += struct.pack('B', value ^ (127 if value < 128 else 0))

    return retVal

@@ -37,8 +41,9 @@ def decloak(inputFile=None, data=None):
            data = f.read()
    try:
        data = zlib.decompress(hideAscii(data))
-    except:
-        print 'ERROR: the provided input file \'%s\' does not contain valid cloaked content' % inputFile
+    except Exception as ex:
+        print(ex)
+        print('ERROR: the provided input file \'%s\' does not contain valid cloaked content' % inputFile)
        sys.exit(1)
    finally:
        f.close()
@@ -59,11 +64,11 @@ def main():
        if not args.inputFile:
            parser.error('Missing the input file, -h for help')

-    except (OptionError, TypeError), e:
-        parser.error(e)
+    except (OptionError, TypeError) as ex:
+        parser.error(ex)

    if not os.path.isfile(args.inputFile):
-        print 'ERROR: the provided input file \'%s\' is non existent' % args.inputFile
+        print('ERROR: the provided input file \'%s\' is non existent' % args.inputFile)
        sys.exit(1)

    if not args.decrypt:
--- a/extra/dbgtool/init.py
+++ b/extra/dbgtool/init.py
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/extra/dbgtool/dbgtool.py
+++ b/extra/dbgtool/dbgtool.py
@@ -3,13 +3,14 @@
 """
 dbgtool.py - Portable executable to ASCII debug script converter

-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

+from __future__ import print_function
+
 import os
 import sys
-import struct

 from optparse import OptionError
 from optparse import OptionParser
@@ -19,7 +20,7 @@ def convert(inputFile):
    fileSize = fileStat.st_size

    if fileSize > 65280:
-        print "ERROR: the provided input file '%s' is too big for debug.exe" % inputFile
+        print("ERROR: the provided input file '%s' is too big for debug.exe" % inputFile)
        sys.exit(1)

    script = "n %s\nr cx\n" % os.path.basename(inputFile.replace(".", "_"))
@@ -32,7 +33,7 @@ def convert(inputFile):
    fileContent = fp.read()

    for fileChar in fileContent:
-        unsignedFileChar = struct.unpack("B", fileChar)[0]
+        unsignedFileChar = fileChar if sys.version_info >= (3, 0) else ord(fileChar)

        if unsignedFileChar != 0:
            counter2 += 1
@@ -59,7 +60,7 @@ def convert(inputFile):

 def main(inputFile, outputFile):
    if not os.path.isfile(inputFile):
-        print "ERROR: the provided input file '%s' is not a regular file" % inputFile
+        print("ERROR: the provided input file '%s' is not a regular file" % inputFile)
        sys.exit(1)

    script = convert(inputFile)
@@ -70,7 +71,7 @@ def main(inputFile, outputFile):
        sys.stdout.write(script)
        sys.stdout.close()
    else:
-        print script
+        print(script)

 if __name__ == "__main__":
    usage = "%s -i <input file> [-o <output file>]" % sys.argv[0]
@@ -86,8 +87,8 @@ if __name__ == "__main__":
        if not args.inputFile:
            parser.error("Missing the input file, -h for help")

-    except (OptionError, TypeError), e:
-        parser.error(e)
+    except (OptionError, TypeError) as ex:
+        parser.error(ex)

    inputFile = args.inputFile
    outputFile = args.outputFile
--- a/extra/icmpsh/icmpsh_m.py
+++ b/extra/icmpsh/icmpsh_m.py
@@ -22,7 +22,6 @@
 import os
 import select
 import socket
-import subprocess
 import sys

 def setNonBlocking(fd):
@@ -37,7 +36,7 @@ def setNonBlocking(fd):
    fcntl.fcntl(fd, fcntl.F_SETFL, flags)

 def main(src, dst):
-    if subprocess.mswindows:
+    if sys.platform == "nt":
        sys.stderr.write('icmpsh master can only run on Posix systems\n')
        sys.exit(255)

@@ -128,7 +127,7 @@ def main(src, dst):
                try:
                    # Send it to the target host
                    sock.sendto(ip.get_packet(), (dst, 0))
-                except socket.error, ex:
+                except socket.error as ex:
                    sys.stderr.write("'%s'\n" % ex)
                    sys.stderr.flush()

--- a/extra/safe2bin/init.py
+++ b/extra/safe2bin/init.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/extra/safe2bin/safe2bin.py
+++ b/extra/safe2bin/safe2bin.py
@@ -1,12 +1,14 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
 safe2bin.py - Simple safe(hex) to binary format converter

-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

+from __future__ import print_function
+
 import binascii
 import re
 import string
@@ -16,6 +18,9 @@ import sys
 from optparse import OptionError
 from optparse import OptionParser

+if sys.version_info >= (3, 0):
+    xrange = range
+
 # Regex used for recognition of hex encoded characters
 HEX_ENCODED_CHAR_REGEX = r"(?P<result>\\x[0-9A-Fa-f]{2})"

@@ -108,11 +113,11 @@ def main():
        if not args.inputFile:
            parser.error('Missing the input file, -h for help')

-    except (OptionError, TypeError), e:
-        parser.error(e)
+    except (OptionError, TypeError) as ex:
+        parser.error(ex)

    if not os.path.isfile(args.inputFile):
-        print 'ERROR: the provided input file \'%s\' is not a regular file' % args.inputFile
+        print('ERROR: the provided input file \'%s\' is not a regular file' % args.inputFile)
        sys.exit(1)

    f = open(args.inputFile, 'r')
--- a/extra/shutils/drei.sh
+++ b/extra/shutils/drei.sh
@@ -0,0 +1,7 @@
+#!/bin/bash
+
+# Copyright (c) 2006-2013 sqlmap developers (http://sqlmap.org/)
+# See the file 'LICENSE' for copying permission
+
+# Stress test against Python3
+for i in $(find . -iname "*.py" | grep -v __init__); do python3 -c 'import '`echo $i | cut -d '.' -f 2 | cut -d '/' -f 2- | sed 's/\//./g'`''; done
--- a/extra/shutils/duplicates.py
+++ b/extra/shutils/duplicates.py
@@ -1,13 +1,16 @@
 #!/usr/bin/env python

-# Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+# Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 # See the file 'LICENSE' for copying permission

 # Removes duplicate entries in wordlist like files

+from __future__ import print_function
+
 import sys

-if len(sys.argv) > 0:
+if __name__ == "__main__":
+    if len(sys.argv) > 1:
        items = list()

        with open(sys.argv[1], 'r') as f:
@@ -17,7 +20,7 @@ if len(sys.argv) > 0:
                    str.encode(item)
                    if item in items:
                        if item:
-                        print item
+                            print(item)
                    else:
                        items.append(item)
                except:
--- a/extra/shutils/newlines.py
+++ b/extra/shutils/newlines.py
@@ -1,7 +1,6 @@
 #! /usr/bin/env python

-# Runs pylint on all python scripts found in a directory tree
-# Reference: http://rowinggolfer.blogspot.com/2009/08/pylint-recursively.html
+from __future__ import print_function

 import os
 import sys
@@ -9,19 +8,20 @@ import sys
 def check(filepath):
    if filepath.endswith(".py"):
        content = open(filepath, "rb").read()
+        pattern = "\n\n\n".encode("ascii")

-        if "\n\n\n" in content:
-            index = content.find("\n\n\n")
-            print filepath, repr(content[index - 30:index + 30])
+        if pattern in content:
+            index = content.find(pattern)
+            print(filepath, repr(content[index - 30:index + 30]))

 if __name__ == "__main__":
    try:
        BASE_DIRECTORY = sys.argv[1]
    except IndexError:
-        print "no directory specified, defaulting to current working directory"
+        print("no directory specified, defaulting to current working directory")
        BASE_DIRECTORY = os.getcwd()

-    print "looking for *.py scripts in subdirectories of ", BASE_DIRECTORY
+    print("looking for *.py scripts in subdirectories of '%s'" % BASE_DIRECTORY)
    for root, dirs, files in os.walk(BASE_DIRECTORY):
        if any(_ in root for _ in ("extra", "thirdparty")):
            continue
--- a/extra/shutils/postcommit-hook.sh
+++ b/extra/shutils/postcommit-hook.sh
@@ -1,6 +1,17 @@
 #!/bin/bash

+: '
+cat > .git/hooks/post-commit << EOF
+#!/bin/bash
+
+source ./extra/shutils/postcommit-hook.sh
+EOF
+
+chmod +x .git/hooks/post-commit
+'
+
 SETTINGS="../../lib/core/settings.py"
+PYPI="../../extra/shutils/pypi.sh"

 declare -x SCRIPTPATH="${0}"

@@ -18,6 +29,6 @@ then
        git tag $NEW_TAG
        git push origin $NEW_TAG
        echo "Going to push PyPI package"
-        /bin/bash ${SCRIPTPATH%/*}/pypi.sh
+        /bin/bash ${SCRIPTPATH%/*}/$PYPI
    fi
 fi
--- a/extra/shutils/precommit-hook.sh
+++ b/extra/shutils/precommit-hook.sh
@@ -1,14 +1,22 @@
 #!/bin/bash

+: '
+cat > .git/hooks/pre-commit << EOF
+#!/bin/bash
+
+source ./extra/shutils/precommit-hook.sh
+EOF
+
+chmod +x .git/hooks/pre-commit
+'
+
 PROJECT="../../"
 SETTINGS="../../lib/core/settings.py"
-CHECKSUM="../../txt/checksum.md5"

 declare -x SCRIPTPATH="${0}"

 PROJECT_FULLPATH=${SCRIPTPATH%/*}/$PROJECT
 SETTINGS_FULLPATH=${SCRIPTPATH%/*}/$SETTINGS
-CHECKSUM_FULLPATH=${SCRIPTPATH%/*}/$CHECKSUM

 git diff $SETTINGS_FULLPATH | grep "VERSION =" > /dev/null && exit 0

@@ -16,7 +24,7 @@ if [ -f $SETTINGS_FULLPATH ]
 then
    LINE=$(grep -o ${SETTINGS_FULLPATH} -e 'VERSION = "[0-9.]*"')
    declare -a LINE
-    INCREMENTED=$(python -c "import re, sys, time; version = re.search('\"([0-9.]*)\"', sys.argv[1]).group(1); _ = version.split('.'); _.append(0) if len(_) < 3 else _; _[-1] = str(int(_[-1]) + 1); month = str(time.gmtime().tm_mon); _[-1] = '0' if _[-2] != month else _[-1]; _[-2] = month; print sys.argv[1].replace(version, '.'.join(_))" "$LINE")
+    INCREMENTED=$(python -c "import re, sys, time; version = re.search('\"([0-9.]*)\"', sys.argv[1]).group(1); _ = version.split('.'); _.extend([0] * (4 - len(_))); _[-1] = str(int(_[-1]) + 1); month = str(time.gmtime().tm_mon); _[-1] = '0' if _[-2] != month else _[-1]; _[-2] = month; print sys.argv[1].replace(version, '.'.join(_))" "$LINE")
    if [ -n "$INCREMENTED" ]
    then
        sed -i "s/${LINE}/${INCREMENTED}/" $SETTINGS_FULLPATH
@@ -27,6 +35,3 @@ then
    fi
    git add "$SETTINGS_FULLPATH"
 fi
-
-truncate -s 0 "$CHECKSUM_FULLPATH"
-cd $PROJECT_FULLPATH && for i in $(find . -name "*.py" -o -name "*.xml" -o -iname "*_" | sort); do git ls-files $i --error-unmatch &>/dev/null && md5sum $i | stdbuf -i0 -o0 -e0 sed 's/\.\///' >> "$CHECKSUM_FULLPATH"; git add "$CHECKSUM_FULLPATH"; done
--- a/extra/shutils/pycodestyle.sh
+++ b/extra/shutils/pycodestyle.sh
@@ -1,6 +1,6 @@
 #!/bin/bash

-# Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+# Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 # See the file 'LICENSE' for copying permission

 # Runs pycodestyle on all python files (prerequisite: pip install pycodestyle)
--- a/extra/shutils/pylint.py
+++ b/extra/shutils/pylint.py
@@ -3,6 +3,8 @@
 # Runs pylint on all python scripts found in a directory tree
 # Reference: http://rowinggolfer.blogspot.com/2009/08/pylint-recursively.html

+from __future__ import print_function
+
 import os
 import re
 import sys
@@ -17,26 +19,26 @@ def check(module):

    if module[-3:] == ".py":

-        print "CHECKING ", module
+        print("CHECKING ", module)
        pout = os.popen("pylint --rcfile=/dev/null %s" % module, 'r')
        for line in pout:
            if re.match(r"\AE:", line):
-                print line.strip()
+                print(line.strip())
            if __RATING__ and "Your code has been rated at" in line:
-                print line
+                print(line)
                score = re.findall(r"\d.\d\d", line)[0]
                total += float(score)
                count += 1

 if __name__ == "__main__":
    try:
-        print sys.argv
+        print(sys.argv)
        BASE_DIRECTORY = sys.argv[1]
    except IndexError:
-        print "no directory specified, defaulting to current working directory"
+        print("no directory specified, defaulting to current working directory")
        BASE_DIRECTORY = os.getcwd()

-    print "looking for *.py scripts in subdirectories of ", BASE_DIRECTORY
+    print("looking for *.py scripts in subdirectories of ", BASE_DIRECTORY)
    for root, dirs, files in os.walk(BASE_DIRECTORY):
        if any(_ in root for _ in ("extra", "thirdparty")):
            continue
@@ -45,6 +47,6 @@ if __name__ == "__main__":
            check(filepath)

    if __RATING__:
-        print "==" * 50
-        print "%d modules found" % count
-        print "AVERAGE SCORE = %.02f" % (total / count)
+        print("==" * 50)
+        print("%d modules found" % count)
+        print("AVERAGE SCORE = %.02f" % (total / count))
--- a/extra/shutils/pypi.sh
+++ b/extra/shutils/pypi.sh
@@ -16,7 +16,7 @@ cat > $TMP_DIR/setup.py << EOF
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -26,10 +26,16 @@ setup(
    name='sqlmap',
    version='$VERSION',
    description='Automatic SQL injection and database takeover tool',
-    long_description='sqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers. It comes with a powerful detection engine, many niche features for the ultimate penetration tester and a broad range of switches lasting from database fingerprinting, over data fetching from the database, to accessing the underlying file system and executing commands on the operating system via out-of-band connections.',
+    long_description=open('README.rst').read(),
+    long_description_content_type='text/x-rst',
    author='Bernardo Damele Assumpcao Guimaraes, Miroslav Stampar',
    author_email='bernardo@sqlmap.org, miroslav@sqlmap.org',
    url='http://sqlmap.org',
+    project_urls={
+        'Documentation': 'https://github.com/sqlmapproject/sqlmap/wiki',
+        'Source': 'https://github.com/sqlmapproject/sqlmap/',
+        'Tracker': 'https://github.com/sqlmapproject/sqlmap/issues',
+    },
    download_url='https://github.com/sqlmapproject/sqlmap/archive/$VERSION.zip',
    license='GNU General Public License v2 (GPLv2)',
    packages=find_packages(),
@@ -61,7 +67,7 @@ cat > sqlmap/__init__.py << EOF
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -153,7 +159,7 @@ Links
 -  User's manual: https://github.com/sqlmapproject/sqlmap/wiki
 -  Frequently Asked Questions (FAQ):
   https://github.com/sqlmapproject/sqlmap/wiki/FAQ
-  Twitter: [@sqlmap](https://twitter.com/sqlmap)
+-  Twitter: https://twitter.com/sqlmap
 -  Demos: http://www.youtube.com/user/inquisb/videos
 -  Screenshots: https://github.com/sqlmapproject/sqlmap/wiki/Screenshots

@@ -171,7 +177,6 @@ Links
 EOF
 sed -i "s/^VERSION =.*/VERSION = \"$VERSION\"/g" sqlmap/lib/core/settings.py
 sed -i "s/^TYPE =.*/TYPE = \"$TYPE\"/g" sqlmap/lib/core/settings.py
-sed -i "s/.*lib\/core\/settings\.py/`md5sum sqlmap/lib/core/settings.py | cut -d ' ' -f 1`  lib\/core\/settings\.py/g" sqlmap/txt/checksum.md5
 for file in $(find sqlmap -type f | grep -v -E "\.(git|yml)"); do echo include $file >> MANIFEST.in; done
 python setup.py sdist upload
 rm -rf $TMP_DIR
--- a/extra/shutils/regressiontest.py
+++ b/extra/shutils/regressiontest.py
@@ -1,164 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
-# See the file 'LICENSE' for copying permission
-
-import codecs
-import inspect
-import os
-import re
-import smtplib
-import subprocess
-import sys
-import time
-import traceback
-
-from email.mime.multipart import MIMEMultipart
-from email.mime.text import MIMEText
-
-sys.path.append(os.path.normpath("%s/../../" % os.path.dirname(inspect.getfile(inspect.currentframe()))))
-
-from lib.core.revision import getRevisionNumber
-
-START_TIME = time.strftime("%H:%M:%S %d-%m-%Y", time.gmtime())
-SQLMAP_HOME = "/opt/sqlmap"
-
-SMTP_SERVER = "127.0.0.1"
-SMTP_PORT = 25
-SMTP_TIMEOUT = 30
-FROM = "regressiontest@sqlmap.org"
-# TO = "dev@sqlmap.org"
-TO = ["bernardo.damele@gmail.com", "miroslav.stampar@gmail.com"]
-SUBJECT = "regression test started on %s using revision %s" % (START_TIME, getRevisionNumber())
-TARGET = "debian"
-
-def prepare_email(content):
-    global FROM
-    global TO
-    global SUBJECT
-
-    msg = MIMEMultipart()
-    msg["Subject"] = SUBJECT
-    msg["From"] = FROM
-    msg["To"] = TO if isinstance(TO, basestring) else ','.join(TO)
-
-    msg.attach(MIMEText(content))
-
-    return msg
-
-def send_email(msg):
-    global SMTP_SERVER
-    global SMTP_PORT
-    global SMTP_TIMEOUT
-
-    try:
-        s = smtplib.SMTP(host=SMTP_SERVER, port=SMTP_PORT, timeout=SMTP_TIMEOUT)
-        s.sendmail(FROM, TO, msg.as_string())
-        s.quit()
-    # Catch all for SMTP exceptions
-    except smtplib.SMTPException, e:
-        print "Failure to send email: %s" % str(e)
-
-def failure_email(msg):
-    msg = prepare_email(msg)
-    send_email(msg)
-    sys.exit(1)
-
-def main():
-    global SUBJECT
-
-    content = ""
-    test_counts = []
-    attachments = {}
-
-    updateproc = subprocess.Popen("cd /opt/sqlmap/ ; python /opt/sqlmap/sqlmap.py --update", shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-    stdout, stderr = updateproc.communicate()
-
-    if stderr:
-        failure_email("Update of sqlmap failed with error:\n\n%s" % stderr)
-
-    regressionproc = subprocess.Popen("python /opt/sqlmap/sqlmap.py --live-test", shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=False)
-    stdout, stderr = regressionproc.communicate()
-
-    if stderr:
-        failure_email("Execution of regression test failed with error:\n\n%s" % stderr)
-
-    failed_tests = re.findall(r"running live test case: (.+?) \((\d+)\/\d+\)[\r]*\n.+test failed (at parsing items: (.+))?\s*\- scan folder: (\/.+) \- traceback: (.*?)( - SQL injection not detected)?[\r]*\n", stdout)
-
-    for failed_test in failed_tests:
-        title = failed_test[0]
-        test_count = int(failed_test[1])
-        parse = failed_test[3] if failed_test[3] else None
-        output_folder = failed_test[4]
-        traceback = False if failed_test[5] == "False" else bool(failed_test[5])
-        detected = False if failed_test[6] else True
-
-        test_counts.append(test_count)
-
-        console_output_file = os.path.join(output_folder, "console_output")
-        log_file = os.path.join(output_folder, TARGET, "log")
-        traceback_file = os.path.join(output_folder, "traceback")
-
-        if os.path.exists(console_output_file):
-            console_output_fd = codecs.open(console_output_file, "rb", "utf8")
-            console_output = console_output_fd.read()
-            console_output_fd.close()
-            attachments[test_count] = str(console_output)
-
-        if os.path.exists(log_file):
-            log_fd = codecs.open(log_file, "rb", "utf8")
-            log = log_fd.read()
-            log_fd.close()
-
-        if os.path.exists(traceback_file):
-            traceback_fd = codecs.open(traceback_file, "rb", "utf8")
-            traceback = traceback_fd.read()
-            traceback_fd.close()
-
-        content += "Failed test case '%s' (#%d)" % (title, test_count)
-
-        if parse:
-            content += " at parsing: %s:\n\n" % parse
-            content += "### Log file:\n\n"
-            content += "%s\n\n" % log
-        elif not detected:
-            content += " - SQL injection not detected\n\n"
-        else:
-            content += "\n\n"
-
-        if traceback:
-            content += "### Traceback:\n\n"
-            content += "%s\n\n" % str(traceback)
-
-        content += "#######################################################################\n\n"
-
-    end_string = "Regression test finished at %s" % time.strftime("%H:%M:%S %d-%m-%Y", time.gmtime())
-
-    if content:
-        content += end_string
-        SUBJECT = "Failed %s (%s)" % (SUBJECT, ", ".join("#%d" % count for count in test_counts))
-
-        msg = prepare_email(content)
-
-        for test_count, attachment in attachments.items():
-            attachment = MIMEText(attachment)
-            attachment.add_header("Content-Disposition", "attachment", filename="test_case_%d_console_output.txt" % test_count)
-            msg.attach(attachment)
-
-        send_email(msg)
-    else:
-        SUBJECT = "Successful %s" % SUBJECT
-        msg = prepare_email("All test cases were successful\n\n%s" % end_string)
-        send_email(msg)
-
-if __name__ == "__main__":
-    log_fd = open("/tmp/sqlmapregressiontest.log", "wb")
-    log_fd.write("Regression test started at %s\n" % START_TIME)
-
-    try:
-        main()
-    except Exception, e:
-        log_fd.write("An exception has occurred:\n%s" % str(traceback.format_exc()))
-
-    log_fd.write("Regression test finished at %s\n\n" % time.strftime("%H:%M:%S %d-%m-%Y", time.gmtime()))
-    log_fd.close()
--- a/extra/sqlharvest/sqlharvest.py
+++ b/extra/sqlharvest/sqlharvest.py
@@ -1,141 +0,0 @@
-#!/usr/bin/env python
-
-"""
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
-See the file 'LICENSE' for copying permission
-"""
-
-import cookielib
-import re
-import socket
-import sys
-import urllib
-import urllib2
-import ConfigParser
-
-from operator import itemgetter
-
-TIMEOUT = 10
-CONFIG_FILE = 'sqlharvest.cfg'
-TABLES_FILE = 'tables.txt'
-USER_AGENT = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; AskTB5.3)'
-SEARCH_URL = 'http://www.google.com/m?source=mobileproducts&dc=gorganic'
-MAX_FILE_SIZE = 2 * 1024 * 1024  # if a result (.sql) file for downloading is more than 2MB in size just skip it
-QUERY = 'CREATE TABLE ext:sql'
-REGEX_URLS = r';u=([^"]+?)&amp;q='
-REGEX_RESULT = r'(?i)CREATE TABLE\s*(/\*.*\*/)?\s*(IF NOT EXISTS)?\s*(?P<result>[^\(;]+)'
-
-def main():
-    tables = dict()
-    cookies = cookielib.CookieJar()
-    cookie_processor = urllib2.HTTPCookieProcessor(cookies)
-    opener = urllib2.build_opener(cookie_processor)
-    opener.addheaders = [("User-Agent", USER_AGENT)]
-
-    conn = opener.open(SEARCH_URL)
-    page = conn.read()  # set initial cookie values
-
-    config = ConfigParser.ConfigParser()
-    config.read(CONFIG_FILE)
-
-    if not config.has_section("options"):
-        config.add_section("options")
-    if not config.has_option("options", "index"):
-        config.set("options", "index", "0")
-
-    i = int(config.get("options", "index"))
-
-    try:
-        with open(TABLES_FILE, 'r') as f:
-            for line in f.xreadlines():
-                if len(line) > 0 and ',' in line:
-                    temp = line.split(',')
-                    tables[temp[0]] = int(temp[1])
-    except:
-        pass
-
-    socket.setdefaulttimeout(TIMEOUT)
-
-    files, old_files = None, None
-    try:
-        while True:
-            abort = False
-            old_files = files
-            files = []
-
-            try:
-                conn = opener.open("%s&q=%s&start=%d&sa=N" % (SEARCH_URL, QUERY.replace(' ', '+'), i * 10))
-                page = conn.read()
-                for match in re.finditer(REGEX_URLS, page):
-                    files.append(urllib.unquote(match.group(1)))
-                    if len(files) >= 10:
-                        break
-                abort = (files == old_files)
-
-            except KeyboardInterrupt:
-                raise
-
-            except Exception, msg:
-                print msg
-
-            if abort:
-                break
-
-            sys.stdout.write("\n---------------\n")
-            sys.stdout.write("Result page #%d\n" % (i + 1))
-            sys.stdout.write("---------------\n")
-
-            for sqlfile in files:
-                print sqlfile
-
-                try:
-                    req = urllib2.Request(sqlfile)
-                    response = urllib2.urlopen(req)
-
-                    if "Content-Length" in response.headers:
-                        if int(response.headers.get("Content-Length")) > MAX_FILE_SIZE:
-                            continue
-
-                    page = response.read()
-                    found = False
-                    counter = 0
-
-                    for match in re.finditer(REGEX_RESULT, page):
-                        counter += 1
-                        table = match.group("result").strip().strip("`\"'").replace('"."', ".").replace("].[", ".").strip('[]')
-
-                        if table and not any(_ in table for _ in ('>', '<', '--', ' ')):
-                            found = True
-                            sys.stdout.write('*')
-
-                            if table in tables:
-                                tables[table] += 1
-                            else:
-                                tables[table] = 1
-                    if found:
-                        sys.stdout.write("\n")
-
-                except KeyboardInterrupt:
-                    raise
-
-                except Exception, msg:
-                    print msg
-
-            else:
-                i += 1
-
-    except KeyboardInterrupt:
-        pass
-
-    finally:
-        with open(TABLES_FILE, 'w+') as f:
-            tables = sorted(tables.items(), key=itemgetter(1), reverse=True)
-            for table, count in tables:
-                f.write("%s,%d\n" % (table, count))
-
-        config.set("options", "index", str(i + 1))
-        with open(CONFIG_FILE, 'w+') as f:
-            config.write(f)
-
-if __name__ == "__main__":
-    main()
--- a/extra/vulnserver/init.py
+++ b/extra/vulnserver/init.py
@@ -1,7 +1,7 @@
 #!/usr/bin/env python

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/extra/vulnserver/vulnserver.py
+++ b/extra/vulnserver/vulnserver.py
@@ -0,0 +1,148 @@
+#!/usr/bin/env python
+
+"""
+vulnserver.py - Trivial SQLi vulnerable HTTP server (Note: for testing purposes)
+
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+from __future__ import print_function
+
+import re
+import sqlite3
+import sys
+import traceback
+
+if sys.version_info >= (3, 0):
+    from http.client import FOUND
+    from http.client import NOT_FOUND
+    from http.client import OK
+    from http.server import BaseHTTPRequestHandler
+    from http.server import HTTPServer
+    from socketserver import ThreadingMixIn
+    from urllib.parse import parse_qs
+    from urllib.parse import unquote_plus
+else:
+    from BaseHTTPServer import BaseHTTPRequestHandler
+    from BaseHTTPServer import HTTPServer
+    from httplib import FOUND
+    from httplib import NOT_FOUND
+    from httplib import OK
+    from SocketServer import ThreadingMixIn
+    from urlparse import parse_qs
+    from urllib import unquote_plus
+
+SCHEMA = """
+    CREATE TABLE users (
+        id INTEGER,
+        name TEXT,
+        surname TEXT
+    );
+    INSERT INTO users (id, name, surname) VALUES (1, 'luther', 'blisset');
+    INSERT INTO users (id, name, surname) VALUES (2, 'fluffy', 'bunny');
+    INSERT INTO users (id, name, surname) VALUES (3, 'wu', 'ming');
+    INSERT INTO users (id, name, surname) VALUES (4, 'sqlmap/1.0-dev (http://sqlmap.org)', 'user agent header');
+    INSERT INTO users (id, name, surname) VALUES (5, NULL, 'nameisnull');
+"""
+
+LISTEN_ADDRESS = "localhost"
+LISTEN_PORT = 8440
+
+_conn = None
+_cursor = None
+_server = None
+
+def init():
+    global _conn
+    global _cursor
+
+    _conn = sqlite3.connect(":memory:", isolation_level=None, check_same_thread=False)
+    _cursor = _conn.cursor()
+
+    _cursor.executescript(SCHEMA)
+
+class ThreadingServer(ThreadingMixIn, HTTPServer):
+    def finish_request(self, *args, **kwargs):
+        try:
+            HTTPServer.finish_request(self, *args, **kwargs)
+        except Exception:
+            traceback.print_exc()
+
+class ReqHandler(BaseHTTPRequestHandler):
+    def do_REQUEST(self):
+        path, query = self.path.split('?', 1) if '?' in self.path else (self.path, "")
+        params = {}
+
+        if query:
+            params.update(parse_qs(query))
+
+        if hasattr(self, "data"):
+            params.update(parse_qs(self.data))
+
+        for key in params:
+            if params[key]:
+                params[key] = params[key][-1]
+
+        self.url, self.params = path, params
+
+        if self.url == '/':
+            if "id" not in params:
+                self.send_response(FOUND)
+                self.send_header("Connection", "close")
+                self.send_header("Location", "/?id=1")
+                self.end_headers()
+            else:
+                self.send_response(OK)
+                self.send_header("Content-type", "text/html")
+                self.send_header("Connection", "close")
+                self.end_headers()
+
+                try:
+                    _cursor.execute("SELECT * FROM users WHERE id=%s LIMIT 0, 1" % self.params.get("id", ""))
+
+                    output = "<b>SQL results:</b>\n"
+                    output += "<table border=\"1\">\n"
+                    for row in _cursor.fetchall():
+                        output += "<tr>"
+                        for value in row:
+                            output += "<td>%s</td>" % value
+                        output += "</tr>\n"
+                    output += "</table>\n"
+                    output += "</body></html>";
+                except Exception as ex:
+                    output = "%s: %s" % (re.search(r"'([^']+)'", str(type(ex))).group(1), ex)
+
+                self.wfile.write(output.encode("utf8"))
+        else:
+            self.send_response(NOT_FOUND)
+            self.send_header("Connection", "close")
+            self.end_headers()
+
+    def do_GET(self):
+        self.do_REQUEST()
+
+    def do_POST(self):
+        length = int(self.headers.get("Content-length", 0))
+        if length:
+            data = self.rfile.read(length)
+            data = unquote_plus(data.decode("utf8"))
+            self.data = data
+        self.do_REQUEST()
+
+def run(address=LISTEN_ADDRESS, port=LISTEN_PORT):
+    global _server
+    try:
+        _server = ThreadingServer((address, port), ReqHandler)
+        print("[i] running HTTP server at '%s:%d'" % (address, port))
+        _server.serve_forever()
+    except KeyboardInterrupt:
+        _server.socket.close()
+        raise
+
+if __name__ == "__main__":
+    try:
+        init()
+        run(sys.argv[1] if len(sys.argv) > 1 else LISTEN_ADDRESS, int(sys.argv[2] if len(sys.argv) > 2 else LISTEN_PORT))
+    except KeyboardInterrupt:
+        print("\r[x] Ctrl-C received")
--- a/extra/wafdetectify/init.py
+++ b/extra/wafdetectify/init.py
@@ -1,8 +0,0 @@
-#!/usr/bin/env python
-
-"""
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
-See the file 'LICENSE' for copying permission
-"""
-
-pass
--- a/extra/wafdetectify/wafdetectify.py
+++ b/extra/wafdetectify/wafdetectify.py
@@ -1,119 +0,0 @@
-#!/usr/bin/env python
-
-"""
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
-See the file 'LICENSE' for copying permission
-"""
-
-import cookielib
-import glob
-import httplib
-import inspect
-import os
-import re
-import subprocess
-import sys
-import urllib2
-
-sys.dont_write_bytecode = True
-
-NAME, VERSION, AUTHOR = "WAF Detectify", "0.1", "sqlmap developers (@sqlmap)"
-TIMEOUT = 10
-HEADERS = {"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-US,en;q=0.5", "Accept-Encoding": "gzip, deflate", "Cache-Control": "max-age=0"}
-SQLMAP_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
-SCRIPTS_DIR = os.path.join(SQLMAP_DIR, "waf")
-LEVEL_COLORS = {"o": "\033[00;94m", "x": "\033[00;91m", "!": "\033[00;93m", "i": "\033[00;92m"}
-CACHE = {}
-WAF_FUNCTIONS = []
-
-def get_page(get=None, url=None, host=None, data=None):
-    key = (get, url, host, data)
-
-    if key in CACHE:
-        return CACHE[key]
-
-    page, headers, code = None, {}, httplib.OK
-
-    url = url or ("%s%s%s" % (sys.argv[1], '?' if '?' not in sys.argv[1] else '&', get) if get else sys.argv[1])
-    if not url.startswith("http"):
-        url = "http://%s" % url
-
-    try:
-        req = urllib2.Request("".join(url[_].replace(' ', "%20") if _ > url.find('?') else url[_] for _ in xrange(len(url))), data, HEADERS)
-        conn = urllib2.urlopen(req, timeout=TIMEOUT)
-        page = conn.read()
-        headers = conn.info()
-    except Exception, ex:
-        code = getattr(ex, "code", None)
-        page = ex.read() if hasattr(ex, "read") else getattr(ex, "msg", "")
-        headers = ex.info() if hasattr(ex, "info") else {}
-
-    result = CACHE[key] = page, headers, code
-
-    return result
-
-def colorize(message):
-    if not subprocess.mswindows and sys.stdout.isatty():
-        message = re.sub(r"\[(.)\]", lambda match: "[%s%s\033[00;49m]" % (LEVEL_COLORS[match.group(1)], match.group(1)), message)
-        message = message.replace("@sqlmap", "\033[00;96m@sqlmap\033[00;49m")
-        message = message.replace(NAME, "\033[00;93m%s\033[00;49m" % NAME)
-
-    return message
-
-def main():
-    global WAF_FUNCTIONS
-
-    print colorize("%s #v%s\n by: %s\n" % (NAME, VERSION, AUTHOR))
-
-    if len(sys.argv) < 2:
-        exit(colorize("[x] usage: python %s <hostname>" % os.path.split(__file__)[-1]))
-
-    cookie_jar = cookielib.CookieJar()
-    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookie_jar))
-    urllib2.install_opener(opener)
-
-    sys.path.insert(0, SQLMAP_DIR)
-
-    for found in glob.glob(os.path.join(SCRIPTS_DIR, "*.py")):
-        dirname, filename = os.path.split(found)
-        dirname = os.path.abspath(dirname)
-
-        if filename == "__init__.py":
-            continue
-
-        if dirname not in sys.path:
-            sys.path.insert(0, dirname)
-
-        try:
-            if filename[:-3] in sys.modules:
-                del sys.modules[filename[:-3]]
-            module = __import__(filename[:-3].encode(sys.getfilesystemencoding() or "utf8"))
-        except ImportError, msg:
-            exit(colorize("[x] cannot import WAF script '%s' (%s)" % (filename[:-3], msg)))
-
-        _ = dict(inspect.getmembers(module))
-        if "detect" not in _:
-            exit(colorize("[x] missing function 'detect(get_page)' in WAF script '%s'" % found))
-        else:
-            WAF_FUNCTIONS.append((_["detect"], _.get("__product__", filename[:-3])))
-
-    WAF_FUNCTIONS = sorted(WAF_FUNCTIONS, key=lambda _: "generic" in _[1].lower())
-
-    print colorize("[i] %d WAF scripts loaded" % len(WAF_FUNCTIONS))
-
-    found = False
-    for function, product in WAF_FUNCTIONS:
-        if found and "unknown" in product.lower():
-            continue
-
-        if function(get_page):
-            print colorize("[!] WAF/IPS identified as '%s'" % product)
-            found = True
-
-    if not found:
-        print colorize("[o] nothing found")
-
-    print
-
-if __name__ == "__main__":
-    main()
--- a/lib/init.py
+++ b/lib/init.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/lib/controller/init.py
+++ b/lib/controller/init.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/lib/controller/action.py
+++ b/lib/controller/action.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -75,7 +75,7 @@ def action():
    if conf.getPasswordHashes:
        try:
            conf.dumper.userSettings("database management system users password hashes", conf.dbmsHandler.getPasswordHashes(), "password hash", CONTENT_TYPE.PASSWORDS)
-        except SqlmapNoneDataException, ex:
+        except SqlmapNoneDataException as ex:
            logger.critical(ex)
        except:
            raise
@@ -83,7 +83,7 @@ def action():
    if conf.getPrivileges:
        try:
            conf.dumper.userSettings("database management system users privileges", conf.dbmsHandler.getPrivileges(), "privilege", CONTENT_TYPE.PRIVILEGES)
-        except SqlmapNoneDataException, ex:
+        except SqlmapNoneDataException as ex:
            logger.critical(ex)
        except:
            raise
@@ -91,40 +91,90 @@ def action():
    if conf.getRoles:
        try:
            conf.dumper.userSettings("database management system users roles", conf.dbmsHandler.getRoles(), "role", CONTENT_TYPE.ROLES)
-        except SqlmapNoneDataException, ex:
+        except SqlmapNoneDataException as ex:
            logger.critical(ex)
        except:
            raise

    if conf.getDbs:
+        try:
            conf.dumper.dbs(conf.dbmsHandler.getDbs())
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise

    if conf.getTables:
+        try:
            conf.dumper.dbTables(conf.dbmsHandler.getTables())
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise

    if conf.commonTables:
+        try:
            conf.dumper.dbTables(tableExists(paths.COMMON_TABLES))
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise

    if conf.getSchema:
+        try:
            conf.dumper.dbTableColumns(conf.dbmsHandler.getSchema(), CONTENT_TYPE.SCHEMA)
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise

    if conf.getColumns:
+        try:
            conf.dumper.dbTableColumns(conf.dbmsHandler.getColumns(), CONTENT_TYPE.COLUMNS)
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise

    if conf.getCount:
+        try:
            conf.dumper.dbTablesCount(conf.dbmsHandler.getCount())
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise

    if conf.commonColumns:
+        try:
            conf.dumper.dbTableColumns(columnExists(paths.COMMON_COLUMNS))
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise

    if conf.dumpTable:
+        try:
            conf.dbmsHandler.dumpTable()
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise

    if conf.dumpAll:
+        try:
            conf.dbmsHandler.dumpAll()
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise

    if conf.search:
+        try:
            conf.dbmsHandler.search()
+        except SqlmapNoneDataException as ex:
+            logger.critical(ex)
+        except:
+            raise

    if conf.query:
        conf.dumper.query(conf.query, conf.dbmsHandler.sqlQuery(conf.query))
--- a/lib/controller/checks.py
+++ b/lib/controller/checks.py
@@ -1,12 +1,11 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

 import copy
-import httplib
 import logging
 import os
 import random
@@ -22,6 +21,7 @@ from lib.core.agent import agent
 from lib.core.common import Backend
 from lib.core.common import extractRegexResult
 from lib.core.common import extractTextTagContent
+from lib.core.common import filterNone
 from lib.core.common import findDynamicContent
 from lib.core.common import Format
 from lib.core.common import getFilteredPageContent
@@ -48,6 +48,7 @@ from lib.core.common import unArrayizeValue
 from lib.core.common import urlencode
 from lib.core.common import wasLastResponseDBMSError
 from lib.core.common import wasLastResponseHTTPError
+from lib.core.compat import xrange
 from lib.core.convert import unicodeencode
 from lib.core.defaults import defaults
 from lib.core.data import conf
@@ -69,6 +70,7 @@ from lib.core.enums import NULLCONNECTION
 from lib.core.enums import PAYLOAD
 from lib.core.enums import PLACE
 from lib.core.enums import REDIRECTION
+from lib.core.enums import WEB_PLATFORM
 from lib.core.exception import SqlmapConnectionException
 from lib.core.exception import SqlmapDataException
 from lib.core.exception import SqlmapNoneDataException
@@ -85,7 +87,7 @@ from lib.core.settings import DUMMY_NON_SQLI_CHECK_APPENDIX
 from lib.core.settings import FI_ERROR_REGEX
 from lib.core.settings import FORMAT_EXCEPTION_STRINGS
 from lib.core.settings import HEURISTIC_CHECK_ALPHABET
-from lib.core.settings import IDS_WAF_CHECK_PAYLOAD
+from lib.core.settings import IPS_WAF_CHECK_PAYLOAD
 from lib.core.settings import IDS_WAF_CHECK_RATIO
 from lib.core.settings import IDS_WAF_CHECK_TIMEOUT
 from lib.core.settings import MAX_DIFFLIB_SEQUENCE_LENGTH
@@ -105,6 +107,8 @@ from lib.request.inject import checkBooleanExpression
 from lib.request.templates import getPageTemplate
 from lib.techniques.union.test import unionTest
 from lib.techniques.union.use import configUnion
+from thirdparty import six
+from thirdparty.six.moves import http_client as _http_client

 def checkSqlInjection(place, parameter, value):
    # Store here the details about boundaries and payload used to
@@ -425,11 +429,14 @@ def checkSqlInjection(place, parameter, value):
                    templatePayload = None
                    vector = None

+                    origValue = value
+                    if kb.customInjectionMark in origValue:
+                        origValue = origValue.split(kb.customInjectionMark)[0]
+                        origValue = re.search(r"(\w*)\Z", origValue).group(1)
+
                    # Threat the parameter original value according to the
                    # test's <where> tag
                    if where == PAYLOAD.WHERE.ORIGINAL or conf.prefix:
-                        origValue = value
-
                        if kb.tamperFunctions:
                            templatePayload = agent.payload(place, parameter, value="", newValue=origValue, where=where)
                    elif where == PAYLOAD.WHERE.NEGATIVE:
@@ -439,7 +446,7 @@ def checkSqlInjection(place, parameter, value):

                        if conf.invalidLogical:
                            _ = int(kb.data.randomInt[:2])
-                            origValue = "%s AND %s LIKE %s" % (value, _, _ + 1)
+                            origValue = "%s AND %s LIKE %s" % (origValue, _, _ + 1)
                        elif conf.invalidBignum:
                            origValue = kb.data.randomInt[:6]
                        elif conf.invalidString:
@@ -575,7 +582,7 @@ def checkSqlInjection(place, parameter, value):
                                        else:
                                            errorSet = set()

-                                        candidates = filter(None, (_.strip() if _.strip() in trueRawResponse and _.strip() not in falseRawResponse else None for _ in (trueSet - falseSet - errorSet)))
+                                        candidates = filterNone(_.strip() if _.strip() in trueRawResponse and _.strip() not in falseRawResponse else None for _ in (trueSet - falseSet - errorSet))

                                        if candidates:
                                            candidates = sorted(candidates, key=lambda _: len(_))
@@ -589,7 +596,7 @@ def checkSqlInjection(place, parameter, value):
                                            logger.info(infoMsg)

                                        if not any((conf.string, conf.notString)):
-                                            candidates = filter(None, (_.strip() if _.strip() in falseRawResponse and _.strip() not in trueRawResponse else None for _ in (falseSet - trueSet)))
+                                            candidates = filterNone(_.strip() if _.strip() in falseRawResponse and _.strip() not in trueRawResponse else None for _ in (falseSet - trueSet))

                                            if candidates:
                                                candidates = sorted(candidates, key=lambda _: len(_))
@@ -614,7 +621,7 @@ def checkSqlInjection(place, parameter, value):
                                page, headers, _ = Request.queryPage(reqPayload, place, content=True, raise404=False)
                                output = extractRegexResult(check, page, re.DOTALL | re.IGNORECASE)
                                output = output or extractRegexResult(check, threadData.lastHTTPError[2] if wasLastResponseHTTPError() else None, re.DOTALL | re.IGNORECASE)
-                                output = output or extractRegexResult(check, listToStrValue((headers[key] for key in headers.keys() if key.lower() != URI_HTTP_HEADER.lower()) if headers else None), re.DOTALL | re.IGNORECASE)
+                                output = output or extractRegexResult(check, listToStrValue((headers[key] for key in headers if key.lower() != URI_HTTP_HEADER.lower()) if headers else None), re.DOTALL | re.IGNORECASE)
                                output = output or extractRegexResult(check, threadData.lastRedirectMsg[1] if threadData.lastRedirectMsg and threadData.lastRedirectMsg[0] == threadData.lastRequestUID else None, re.DOTALL | re.IGNORECASE)

                                if output:
@@ -626,10 +633,10 @@ def checkSqlInjection(place, parameter, value):

                                        injectable = True

-                            except SqlmapConnectionException, msg:
+                            except SqlmapConnectionException as ex:
                                debugMsg = "problem occurred most likely because the "
                                debugMsg += "server hasn't recovered as expected from the "
-                                debugMsg += "error-based payload used ('%s')" % msg
+                                debugMsg += "error-based payload used ('%s')" % getSafeExString(ex)
                                logger.debug(debugMsg)

                        # In case of time-based blind or stacked queries
@@ -688,7 +695,7 @@ def checkSqlInjection(place, parameter, value):
                            # Test for UNION query SQL injection
                            reqPayload, vector = unionTest(comment, place, parameter, value, prefix, suffix)

-                            if isinstance(reqPayload, basestring):
+                            if isinstance(reqPayload, six.string_types):
                                infoMsg = "%s parameter '%s' is '%s' injectable" % (paramType, parameter, title)
                                logger.info(infoMsg)

@@ -1052,9 +1059,19 @@ def heuristicCheckSqlInjection(place, parameter):
    kb.heuristicTest = HEURISTIC_TEST.CASTED if casting else HEURISTIC_TEST.NEGATIVE if not result else HEURISTIC_TEST.POSITIVE

    if casting:
-        errMsg = "possible %s casting " % ("integer" if origValue.isdigit() else "type")
-        errMsg += "detected (e.g. \"$%s=intval($_REQUEST['%s'])\") " % (parameter, parameter)
-        errMsg += "at the back-end web application"
+        errMsg = "possible %s casting detected (e.g. '" % ("integer" if origValue.isdigit() else "type")
+
+        platform = conf.url.split('.')[-1].lower()
+        if platform == WEB_PLATFORM.ASP:
+            errMsg += "%s=CInt(request.querystring(\"%s\"))" % (parameter, parameter)
+        elif platform == WEB_PLATFORM.ASPX:
+            errMsg += "int.TryParse(Request.QueryString[\"%s\"], out %s)" % (parameter, parameter)
+        elif platform == WEB_PLATFORM.JSP:
+            errMsg += "%s=Integer.parseInt(request.getParameter(\"%s\"))" % (parameter, parameter)
+        else:
+            errMsg += "$%s=intval($_REQUEST[\"%s\"])" % (parameter, parameter)
+
+        errMsg += "') at the back-end web application"
        logger.error(errMsg)

        if kb.ignoreCasted is None:
@@ -1323,6 +1340,9 @@ def checkWaf():
    if any((conf.string, conf.notString, conf.regexp, conf.dummy, conf.offline, conf.skipWaf)):
        return None

+    if kb.originalCode == _http_client.NOT_FOUND:
+        return None
+
    _ = hashDBRetrieve(HASHDB_KEYS.CHECK_WAF_RESULT, True)
    if _ is not None:
        if _:
@@ -1339,7 +1359,7 @@ def checkWaf():
    logger.info(infoMsg)

    retVal = False
-    payload = "%d %s" % (randomInt(), IDS_WAF_CHECK_PAYLOAD)
+    payload = "%d %s" % (randomInt(), IPS_WAF_CHECK_PAYLOAD)

    if PLACE.URI in conf.parameters:
        place = PLACE.POST
@@ -1407,16 +1427,24 @@ def identifyWaf():
        page, headers, code = None, None, None
        try:
            pushValue(kb.redirectChoice)
-            kb.redirectChoice = REDIRECTION.NO
+            pushValue(kb.resendPostOnRedirect)
+
+            kb.redirectChoice = REDIRECTION.YES
+            kb.resendPostOnRedirect = True
+
            if kwargs.get("get"):
                kwargs["get"] = urlencode(kwargs["get"])
            kwargs["raise404"] = False
            kwargs["silent"] = True
+            kwargs["finalCode"] = True
+
            page, headers, code = Request.getPage(*args, **kwargs)
        except Exception:
            pass
        finally:
+            kb.resendPostOnRedirect = popValue()
            kb.redirectChoice = popValue()
+
        return page or "", headers or {}, code

    retVal = []
@@ -1428,7 +1456,7 @@ def identifyWaf():
        try:
            logger.debug("checking for WAF/IPS product '%s'" % product)
            found = function(_)
-        except Exception, ex:
+        except Exception as ex:
            errMsg = "exception occurred while running "
            errMsg += "WAF script for '%s' ('%s')" % (product, getSafeExString(ex))
            logger.critical(errMsg)
@@ -1482,6 +1510,15 @@ def checkNullConnection():
    if conf.data:
        return False

+    _ = hashDBRetrieve(HASHDB_KEYS.CHECK_NULL_CONNECTION_RESULT, True)
+    if _ is not None:
+        kb.nullConnection = _
+
+        if _:
+            dbgMsg = "resuming NULL connection method '%s'" % _
+            logger.debug(dbgMsg)
+
+    else:
        infoMsg = "testing NULL connection to the target URL"
        logger.info(infoMsg)

@@ -1518,10 +1555,14 @@ def checkNullConnection():

        finally:
            kb.pageCompress = popValue()
+            kb.nullConnection = False if kb.nullConnection is None else kb.nullConnection
+            hashDBWrite(HASHDB_KEYS.CHECK_NULL_CONNECTION_RESULT, kb.nullConnection, True)

-    return kb.nullConnection is not None
+    return kb.nullConnection in getPublicTypeMembers(NULLCONNECTION, True)

 def checkConnection(suppressOutput=False):
+    threadData = getCurrentThreadData()
+
    if not re.search(r"\A\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\Z", conf.hostname):
        if not any((conf.proxy, conf.tor, conf.dummy, conf.offline)):
            try:
@@ -1531,11 +1572,11 @@ def checkConnection(suppressOutput=False):
            except socket.gaierror:
                errMsg = "host '%s' does not exist" % conf.hostname
                raise SqlmapConnectionException(errMsg)
-            except socket.error, ex:
+            except socket.error as ex:
                errMsg = "problem occurred while "
                errMsg += "resolving a host name '%s' ('%s')" % (conf.hostname, getSafeExString(ex))
                raise SqlmapConnectionException(errMsg)
-            except UnicodeError, ex:
+            except UnicodeError as ex:
                errMsg = "problem occurred while "
                errMsg += "handling a host name '%s' ('%s')" % (conf.hostname, getSafeExString(ex))
                raise SqlmapDataException(errMsg)
@@ -1546,8 +1587,7 @@ def checkConnection(suppressOutput=False):

    try:
        kb.originalPageTime = time.time()
-        page, headers, _ = Request.queryPage(content=True, noteResponseTime=False)
-        kb.originalPage = kb.pageTemplate = page
+        Request.queryPage(content=True, noteResponseTime=False)

        kb.errorIsNone = False

@@ -1570,16 +1610,14 @@ def checkConnection(suppressOutput=False):
        else:
            kb.errorIsNone = True

-        threadData = getCurrentThreadData()
-
        if kb.redirectChoice == REDIRECTION.YES and threadData.lastRedirectURL and threadData.lastRedirectURL[0] == threadData.lastRequestUID:
            if (threadData.lastRedirectURL[1] or "").startswith("https://") and unicodeencode(conf.hostname) in threadData.lastRedirectURL[1]:
                conf.url = re.sub(r"https?://", "https://", conf.url)
                match = re.search(r":(\d+)", threadData.lastRedirectURL[1])
                port = match.group(1) if match else 443
-                conf.url = re.sub(r":\d+/", ":%s/" % port, conf.url)
+                conf.url = re.sub(r":\d+(/|\Z)", ":%s\g<1>" % port, conf.url)

-    except SqlmapConnectionException, ex:
+    except SqlmapConnectionException as ex:
        if conf.ipv6:
            warnMsg = "check connection to a provided "
            warnMsg += "IPv6 address with a tool like ping6 "
@@ -1588,7 +1626,7 @@ def checkConnection(suppressOutput=False):
            warnMsg += "any addressing issues"
            singleTimeWarnMessage(warnMsg)

-        if any(code in kb.httpErrorCodes for code in (httplib.NOT_FOUND, )):
+        if any(code in kb.httpErrorCodes for code in (_http_client.NOT_FOUND, )):
            errMsg = getSafeExString(ex)
            logger.critical(errMsg)

@@ -1602,6 +1640,9 @@ def checkConnection(suppressOutput=False):
                kb.ignoreNotFound = True
        else:
            raise
+    finally:
+        kb.originalPage = kb.pageTemplate = threadData.lastPage
+        kb.originalCode = threadData.lastCode

    return True

--- a/lib/controller/controller.py
+++ b/lib/controller/controller.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -40,6 +40,7 @@ from lib.core.common import safeCSValue
 from lib.core.common import showHttpErrorCodes
 from lib.core.common import urlencode
 from lib.core.common import urldecode
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -71,6 +72,7 @@ from lib.core.settings import REFERER_ALIASES
 from lib.core.settings import USER_AGENT_ALIASES
 from lib.core.target import initTargetEnv
 from lib.core.target import setupTargetEnv
+from lib.utils.hash import crackHashFile

 def _selectInjection():
    """
@@ -89,7 +91,7 @@ def _selectInjection():
        if point not in points:
            points[point] = injection
        else:
-            for key in points[point].keys():
+            for key in points[point]:
                if key != 'data':
                    points[point][key] = points[point][key] or injection[key]
            points[point]['data'].update(injection['data'])
@@ -243,7 +245,7 @@ def _saveToResultsFile():
        if key not in results:
            results[key] = []

-        results[key].extend(injection.data.keys())
+        results[key].extend(list(injection.data.keys()))

    try:
        for key, value in results.items():
@@ -256,7 +258,7 @@ def _saveToResultsFile():
            conf.resultsFP.write(line)

        conf.resultsFP.flush()
-    except IOError, ex:
+    except IOError as ex:
        errMsg = "unable to write to the results file '%s' ('%s'). " % (conf.resultsFilename, getSafeExString(ex))
        raise SqlmapSystemException(errMsg)

@@ -268,6 +270,9 @@ def start():
    check if they are dynamic and SQL injection affected
    """

+    if conf.hashFile:
+        crackHashFile(conf.hashFile)
+
    if conf.direct:
        initTargetEnv()
        setupTargetEnv()
@@ -423,7 +428,7 @@ def start():
                    checkStability()

                # Do a little prioritization reorder of a testable parameter list
-                parameters = conf.parameters.keys()
+                parameters = list(conf.parameters.keys())

                # Order of testing list (first to last)
                orderList = (PLACE.CUSTOM_POST, PLACE.CUSTOM_HEADER, PLACE.URI, PLACE.POST, PLACE.GET)
@@ -487,7 +492,7 @@ def start():
                        elif parameter in conf.testParameter:
                            pass

-                        elif parameter == conf.rParam:
+                        elif parameter in conf.rParam:
                            testSqlInj = False

                            infoMsg = "skipping randomizing %s parameter '%s'" % (paramType, parameter)
@@ -505,7 +510,7 @@ def start():
                            infoMsg = "skipping %s parameter '%s'" % (paramType, parameter)
                            logger.info(infoMsg)

-                        elif parameter == conf.csrfToken:
+                        elif conf.csrfToken and re.search(conf.csrfToken, parameter, re.I):
                            testSqlInj = False

                            infoMsg = "skipping anti-CSRF token parameter '%s'" % parameter
@@ -638,6 +643,9 @@ def start():
                        errMsg += "involved (e.g. WAF) maybe you could try to use "
                        errMsg += "option '--tamper' (e.g. '--tamper=space2comment')"

+                        if not conf.randomAgent:
+                            errMsg += " and/or switch '--random-agent'"
+
                    raise SqlmapNotVulnerableException(errMsg.rstrip('.'))
            else:
                # Flush the flag
@@ -682,7 +690,7 @@ def start():
        except SqlmapSilentQuitException:
            raise

-        except SqlmapBaseException, ex:
+        except SqlmapBaseException as ex:
            errMsg = getSafeExString(ex)

            if conf.multipleTargets:
--- a/lib/controller/handler.py
+++ b/lib/controller/handler.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -105,13 +105,13 @@ def setHandler():

                    if sqlalchemy.connector:
                        conf.dbmsConnector = sqlalchemy
-                except Exception, ex:
+                except Exception as ex:
                    exception = ex

            if not dialect or exception:
                try:
                    conf.dbmsConnector.connect()
-                except Exception, ex:
+                except Exception as ex:
                    if exception:
                        raise exception
                    else:
--- a/lib/core/init.py
+++ b/lib/core/init.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/lib/core/agent.py
+++ b/lib/core/agent.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -9,6 +9,7 @@ import re

 from lib.core.common import Backend
 from lib.core.common import extractRegexResult
+from lib.core.common import filterNone
 from lib.core.common import getSQLSnippet
 from lib.core.common import getUnicode
 from lib.core.common import isDBMSVersionAtLeast
@@ -23,6 +24,7 @@ from lib.core.common import splitFields
 from lib.core.common import unArrayizeValue
 from lib.core.common import urlencode
 from lib.core.common import zeroDepthSearch
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import queries
@@ -105,9 +107,9 @@ class Agent(object):
            if place == PLACE.URI:
                origValue = origValue.split(kb.customInjectionMark)[0]
            else:
-                origValue = filter(None, (re.search(_, origValue.split(BOUNDED_INJECTION_MARKER)[0]) for _ in (r"\w+\Z", r"[^\"'><]+\Z", r"[^ ]+\Z")))[0].group(0)
+                origValue = filterNone(re.search(_, origValue.split(BOUNDED_INJECTION_MARKER)[0]) for _ in (r"\w+\Z", r"[^\"'><]+\Z", r"[^ ]+\Z"))[0].group(0)
            origValue = origValue[origValue.rfind('/') + 1:]
-            for char in ('?', '=', ':', ','):
+            for char in ('?', '=', ':', ',', '&'):
                if char in origValue:
                    origValue = origValue[origValue.rfind(char) + 1:]
        elif place == PLACE.CUSTOM_POST:
--- a/lib/core/bigarray.py
+++ b/lib/core/bigarray.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -16,6 +16,7 @@ import os
 import sys
 import tempfile

+from lib.core.compat import xrange
 from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.exception import SqlmapSystemException
 from lib.core.settings import BIGARRAY_CHUNK_SIZE
@@ -87,7 +88,7 @@ class BigArray(list):
            try:
                with open(self.chunks[-1], "rb") as f:
                    self.chunks[-1] = pickle.loads(bz2.decompress(f.read()))
-            except IOError, ex:
+            except IOError as ex:
                errMsg = "exception occurred while retrieving data "
                errMsg += "from a temporary file ('%s')" % ex.message
                raise SqlmapSystemException(errMsg)
@@ -109,7 +110,7 @@ class BigArray(list):
            with open(filename, "w+b") as f:
                f.write(bz2.compress(pickle.dumps(chunk, pickle.HIGHEST_PROTOCOL), BIGARRAY_COMPRESS_LEVEL))
            return filename
-        except (OSError, IOError), ex:
+        except (OSError, IOError) as ex:
            errMsg = "exception occurred while storing data "
            errMsg += "to a temporary file ('%s'). Please " % ex.message
            errMsg += "make sure that there is enough disk space left. If problem persists, "
@@ -126,7 +127,7 @@ class BigArray(list):
            try:
                with open(self.chunks[index], "rb") as f:
                    self.cache = Cache(index, pickle.loads(bz2.decompress(f.read())), False)
-            except Exception, ex:
+            except Exception as ex:
                errMsg = "exception occurred while retrieving data "
                errMsg += "from a temporary file ('%s')" % ex.message
                raise SqlmapSystemException(errMsg)
@@ -148,7 +149,7 @@ class BigArray(list):
        if y < 0:
            y += len(self)

-        index = y / self.chunk_length
+        index = y // self.chunk_length
        offset = y % self.chunk_length
        chunk = self.chunks[index]

@@ -159,7 +160,7 @@ class BigArray(list):
            return self.cache.data[offset]

    def __setitem__(self, y, value):
-        index = y / self.chunk_length
+        index = y // self.chunk_length
        offset = y % self.chunk_length
        chunk = self.chunks[index]

--- a/lib/core/common.py
+++ b/lib/core/common.py
--- a/lib/core/compat.py
+++ b/lib/core/compat.py
@@ -0,0 +1,172 @@
+#!/usr/bin/env python2
+
+"""
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+import binascii
+import os
+import random
+import uuid
+import sys
+
+class WichmannHill(random.Random):
+    """
+    Reference: https://svn.python.org/projects/python/trunk/Lib/random.py
+    """
+
+    VERSION = 1     # used by getstate/setstate
+
+    def seed(self, a=None):
+        """Initialize internal state from hashable object.
+
+        None or no argument seeds from current time or from an operating
+        system specific randomness source if available.
+
+        If a is not None or an int or long, hash(a) is used instead.
+
+        If a is an int or long, a is used directly.  Distinct values between
+        0 and 27814431486575L inclusive are guaranteed to yield distinct
+        internal states (this guarantee is specific to the default
+        Wichmann-Hill generator).
+        """
+
+        if a is None:
+            try:
+                a = int(binascii.hexlify(os.urandom(16)), 16)
+            except NotImplementedError:
+                import time
+                a = int(time.time() * 256) # use fractional seconds
+
+        if not isinstance(a, int):
+            a = hash(a)
+
+        a, x = divmod(a, 30268)
+        a, y = divmod(a, 30306)
+        a, z = divmod(a, 30322)
+        self._seed = int(x)+1, int(y)+1, int(z)+1
+
+        self.gauss_next = None
+
+    def random(self):
+        """Get the next random number in the range [0.0, 1.0)."""
+
+        # Wichman-Hill random number generator.
+        #
+        # Wichmann, B. A. & Hill, I. D. (1982)
+        # Algorithm AS 183:
+        # An efficient and portable pseudo-random number generator
+        # Applied Statistics 31 (1982) 188-190
+        #
+        # see also:
+        #        Correction to Algorithm AS 183
+        #        Applied Statistics 33 (1984) 123
+        #
+        #        McLeod, A. I. (1985)
+        #        A remark on Algorithm AS 183
+        #        Applied Statistics 34 (1985),198-200
+
+        # This part is thread-unsafe:
+        # BEGIN CRITICAL SECTION
+        x, y, z = self._seed
+        x = (171 * x) % 30269
+        y = (172 * y) % 30307
+        z = (170 * z) % 30323
+        self._seed = x, y, z
+        # END CRITICAL SECTION
+
+        # Note:  on a platform using IEEE-754 double arithmetic, this can
+        # never return 0.0 (asserted by Tim; proof too long for a comment).
+        return (x/30269.0 + y/30307.0 + z/30323.0) % 1.0
+
+    def getstate(self):
+        """Return internal state; can be passed to setstate() later."""
+        return self.VERSION, self._seed, self.gauss_next
+
+    def setstate(self, state):
+        """Restore internal state from object returned by getstate()."""
+        version = state[0]
+        if version == 1:
+            version, self._seed, self.gauss_next = state
+        else:
+            raise ValueError("state with version %s passed to "
+                             "Random.setstate() of version %s" %
+                             (version, self.VERSION))
+
+    def jumpahead(self, n):
+        """Act as if n calls to random() were made, but quickly.
+
+        n is an int, greater than or equal to 0.
+
+        Example use:  If you have 2 threads and know that each will
+        consume no more than a million random numbers, create two Random
+        objects r1 and r2, then do
+            r2.setstate(r1.getstate())
+            r2.jumpahead(1000000)
+        Then r1 and r2 will use guaranteed-disjoint segments of the full
+        period.
+        """
+
+        if not n >= 0:
+            raise ValueError("n must be >= 0")
+        x, y, z = self._seed
+        x = int(x * pow(171, n, 30269)) % 30269
+        y = int(y * pow(172, n, 30307)) % 30307
+        z = int(z * pow(170, n, 30323)) % 30323
+        self._seed = x, y, z
+
+    def __whseed(self, x=0, y=0, z=0):
+        """Set the Wichmann-Hill seed from (x, y, z).
+
+        These must be integers in the range [0, 256).
+        """
+
+        if not type(x) == type(y) == type(z) == int:
+            raise TypeError('seeds must be integers')
+        if not (0 <= x < 256 and 0 <= y < 256 and 0 <= z < 256):
+            raise ValueError('seeds must be in range(0, 256)')
+        if 0 == x == y == z:
+            # Initialize from current time
+            import time
+            t = int(time.time() * 256)
+            t = int((t&0xffffff) ^ (t>>24))
+            t, x = divmod(t, 256)
+            t, y = divmod(t, 256)
+            t, z = divmod(t, 256)
+        # Zero is a poor seed, so substitute 1
+        self._seed = (x or 1, y or 1, z or 1)
+
+        self.gauss_next = None
+
+    def whseed(self, a=None):
+        """Seed from hashable object's hash code.
+
+        None or no argument seeds from current time.  It is not guaranteed
+        that objects with distinct hash codes lead to distinct internal
+        states.
+
+        This is obsolete, provided for compatibility with the seed routine
+        used prior to Python 2.1.  Use the .seed() method instead.
+        """
+
+        if a is None:
+            self.__whseed()
+            return
+        a = hash(a)
+        a, x = divmod(a, 256)
+        a, y = divmod(a, 256)
+        a, z = divmod(a, 256)
+        x = (x + a) % 256 or 1
+        y = (y + a) % 256 or 1
+        z = (z + a) % 256 or 1
+        self.__whseed(x, y, z)
+
+# Reference: https://github.com/urllib3/urllib3/blob/master/src/urllib3/filepost.py
+def choose_boundary():
+    return uuid.uuid4().hex
+
+if sys.version_info >= (3, 0):
+    xrange = range
+else:
+    xrange = xrange
--- a/lib/core/convert.py
+++ b/lib/core/convert.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -9,18 +9,15 @@ try:
    import cPickle as pickle
 except:
    import pickle
-finally:
-    import pickle as picklePy

 import base64
 import json
 import re
-import StringIO
 import sys

 from lib.core.settings import IS_WIN
 from lib.core.settings import UNICODE_ENCODING
-from lib.core.settings import PICKLE_REDUCE_WHITELIST
+from thirdparty import six

 def base64decode(value):
    """
@@ -30,7 +27,7 @@ def base64decode(value):
    'foobar'
    """

-    return base64.b64decode(value)
+    return base64.b64decode(unicodeencode(value))

 def base64encode(value):
    """
@@ -40,7 +37,7 @@ def base64encode(value):
    'Zm9vYmFy'
    """

-    return base64.b64encode(value)
+    return base64.b64encode(unicodeencode(value))

 def base64pickle(value):
    """
@@ -66,7 +63,7 @@ def base64pickle(value):

    return retVal

-def base64unpickle(value, unsafe=False):
+def base64unpickle(value):
    """
    Decodes value from Base64 to plain format and deserializes (with pickle) its content

@@ -76,26 +73,10 @@ def base64unpickle(value, unsafe=False):

    retVal = None

-    def _(self):
-        if len(self.stack) > 1:
-            func = self.stack[-2]
-            if func not in PICKLE_REDUCE_WHITELIST:
-                raise Exception("abusing reduce() is bad, Mkay!")
-        self.load_reduce()
-
-    def loads(str):
-        f = StringIO.StringIO(str)
-        if unsafe:
-            unpickler = picklePy.Unpickler(f)
-            unpickler.dispatch[picklePy.REDUCE] = _
-        else:
-            unpickler = pickle.Unpickler(f)
-        return unpickler.load()
-
    try:
-        retVal = loads(base64decode(value))
+        retVal = pickle.loads(base64decode(value))
    except TypeError:
-        retVal = loads(base64decode(bytes(value)))
+        retVal = pickle.loads(base64decode(bytes(value)))

    return retVal

@@ -108,7 +89,14 @@ def hexdecode(value):
    """

    value = value.lower()
-    return (value[2:] if value.startswith("0x") else value).decode("hex")
+    value = value[2:] if value.startswith("0x") else value
+
+    if six.PY2:
+        retVal = value.decode("hex")
+    else:
+        retVal = bytes.fromhex(value)
+
+    return retVal

 def hexencode(value, encoding=None):
    """
@@ -118,7 +106,14 @@ def hexencode(value, encoding=None):
    '666f6f626172'
    """

-    return unicodeencode(value, encoding).encode("hex")
+    retVal = unicodeencode(value, encoding)
+
+    if six.PY2:
+        retVal = retVal.encode("hex")
+    else:
+        retVal = retVal.hex()
+
+    return retVal

 def unicodeencode(value, encoding=None):
    """
@@ -129,11 +124,13 @@ def unicodeencode(value, encoding=None):
    """

    retVal = value
-    if isinstance(value, unicode):
+
+    if isinstance(value, six.text_type):
        try:
            retVal = value.encode(encoding or UNICODE_ENCODING)
        except UnicodeEncodeError:
-            retVal = value.encode(UNICODE_ENCODING, "replace")
+            retVal = value.encode(encoding or UNICODE_ENCODING, "replace")
+
    return retVal

 def utf8encode(value):
@@ -165,7 +162,7 @@ def htmlunescape(value):
    """

    retVal = value
-    if value and isinstance(value, basestring):
+    if value and isinstance(value, six.string_types):
        codes = (("&lt;", '<'), ("&gt;", '>'), ("&quot;", '"'), ("&nbsp;", ' '), ("&amp;", '&'), ("&apos;", "'"))
        retVal = reduce(lambda x, y: x.replace(y[0], y[1]), codes, retVal)
        try:
@@ -183,13 +180,11 @@ def stdoutencode(data):
    retVal = None

    try:
-        data = data or ""
+        retVal = unicodeencode(data or "", sys.stdout.encoding)

        # Reference: http://bugs.python.org/issue1602
        if IS_WIN:
-            output = data.encode(sys.stdout.encoding, "replace")
-
-            if '?' in output and '?' not in data:
+            if '?' in retVal and '?' not in retVal:
                warnMsg = "cannot properly display Unicode characters "
                warnMsg += "inside Windows OS command prompt "
                warnMsg += "(http://bugs.python.org/issue1602). All "
@@ -199,11 +194,8 @@ def stdoutencode(data):
                warnMsg += "corresponding output files. "
                singleTimeWarnMessage(warnMsg)

-            retVal = output
-        else:
-            retVal = data.encode(sys.stdout.encoding)
    except:
-        retVal = data.encode(UNICODE_ENCODING) if isinstance(data, unicode) else data
+        retVal = unicodeencode(data or "")

    return retVal

--- a/lib/core/data.py
+++ b/lib/core/data.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/lib/core/datatype.py
+++ b/lib/core/datatype.py
@@ -1,13 +1,16 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

+import collections
 import copy
 import types

+from thirdparty.odict import OrderedDict
+
 class AttribDict(dict):
    """
    This class defines the sqlmap object, inheriting from Python data
@@ -104,3 +107,95 @@ class InjectionDict(AttribDict):
        self.dbms = None
        self.dbms_version = None
        self.os = None
+
+# Reference: https://www.kunxi.org/2014/05/lru-cache-in-python
+class LRUDict(object):
+    def __init__(self, capacity):
+        self.capacity = capacity
+        self.cache = OrderedDict()
+
+    def __len__(self):
+        return len(self.cache)
+
+    def __contains__(self, key):
+        return key in self.cache
+
+    def __getitem__(self, key):
+        value = self.cache.pop(key)
+        self.cache[key] = value
+        return value
+
+    def get(self, key):
+        return self.__getitem__(key)
+
+    def __setitem__(self, key, value):
+        try:
+            self.cache.pop(key)
+        except KeyError:
+            if len(self.cache) >= self.capacity:
+                self.cache.popitem(last=False)
+        self.cache[key] = value
+
+    def set(self, key, value):
+        self.__setitem__(key, value)
+
+    def keys(self):
+        return self.cache.keys()
+
+# Reference: https://code.activestate.com/recipes/576694/
+class OrderedSet(collections.MutableSet):
+    def __init__(self, iterable=None):
+        self.end = end = []
+        end += [None, end, end]         # sentinel node for doubly linked list
+        self.map = {}                   # key --> [key, prev, next]
+        if iterable is not None:
+            self |= iterable
+
+    def __len__(self):
+        return len(self.map)
+
+    def __contains__(self, key):
+        return key in self.map
+
+    def add(self, key):
+        if key not in self.map:
+            end = self.end
+            curr = end[1]
+            curr[2] = end[1] = self.map[key] = [key, curr, end]
+
+    def discard(self, key):
+        if key in self.map:
+            key, prev, next = self.map.pop(key)
+            prev[2] = next
+            next[1] = prev
+
+    def __iter__(self):
+        end = self.end
+        curr = end[2]
+        while curr is not end:
+            yield curr[0]
+            curr = curr[2]
+
+    def __reversed__(self):
+        end = self.end
+        curr = end[1]
+        while curr is not end:
+            yield curr[0]
+            curr = curr[1]
+
+    def pop(self, last=True):
+        if not self:
+            raise KeyError('set is empty')
+        key = self.end[1][0] if last else self.end[2][0]
+        self.discard(key)
+        return key
+
+    def __repr__(self):
+        if not self:
+            return '%s()' % (self.__class__.__name__,)
+        return '%s(%r)' % (self.__class__.__name__, list(self))
+
+    def __eq__(self, other):
+        if isinstance(other, OrderedSet):
+            return len(self) == len(other) and list(self) == list(other)
+        return set(self) == set(other)
--- a/lib/core/decorators.py
+++ b/lib/core/decorators.py
@@ -1,31 +1,50 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

+import functools
 import hashlib
+import threading

+from lib.core.settings import MAX_CACHE_ITEMS
+from lib.core.datatype import LRUDict
 from lib.core.threads import getCurrentThreadData

-def cachedmethod(f, cache={}):
+_lock = threading.Lock()
+
+def cachedmethod(f, cache=LRUDict(capacity=MAX_CACHE_ITEMS)):
    """
    Method with a cached content

    Reference: http://code.activestate.com/recipes/325205-cache-decorator-in-python-24/
    """

+    @functools.wraps(f)
    def _(*args, **kwargs):
        key = int(hashlib.md5("|".join(str(_) for _ in (f, args, kwargs))).hexdigest(), 16) & 0x7fffffffffffffff
-        if key not in cache:
-            cache[key] = f(*args, **kwargs)

-        return cache[key]
+        try:
+            with _lock:
+                result = cache[key]
+        except KeyError:
+            result = f(*args, **kwargs)
+
+            with _lock:
+                cache[key] = result
+
+        return result

    return _

 def stackedmethod(f):
+    """
+    Method using pushValue/popValue functions (fallback function for stack realignment)
+    """
+
+    @functools.wraps(f)
    def _(*args, **kwargs):
        threadData = getCurrentThreadData()
        originalLevel = len(threadData.valueStack)
--- a/lib/core/defaults.py
+++ b/lib/core/defaults.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/lib/core/dicts.py
+++ b/lib/core/dicts.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/lib/core/dump.py
+++ b/lib/core/dump.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -20,6 +20,7 @@ from lib.core.common import dataToStdout
 from lib.core.common import getSafeExString
 from lib.core.common import getUnicode
 from lib.core.common import isListLike
+from lib.core.common import isMultiThreadMode
 from lib.core.common import normalizeUnicode
 from lib.core.common import openFile
 from lib.core.common import prioritySortColumns
@@ -27,6 +28,7 @@ from lib.core.common import randomInt
 from lib.core.common import safeCSValue
 from lib.core.common import unicodeencode
 from lib.core.common import unsafeSQLIdentificatorNaming
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -49,6 +51,7 @@ from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import UNSAFE_DUMP_FILEPATH_REPLACEMENT
 from lib.core.settings import VERSION_STRING
 from lib.core.settings import WINDOWS_RESERVED_NAMES
+from thirdparty import six
 from thirdparty.magic import magic

 from extra.safe2bin.safe2bin import safechardecode
@@ -74,16 +77,17 @@ class Dump(object):
        if console:
            dataToStdout(text)

-        if kb.get("multiThreadMode"):
+        multiThreadMode = isMultiThreadMode()
+        if multiThreadMode:
            self._lock.acquire()

        try:
            self._outputFP.write(text)
-        except IOError, ex:
+        except IOError as ex:
            errMsg = "error occurred while writing to log file ('%s')" % getSafeExString(ex)
            raise SqlmapGenericException(errMsg)

-        if kb.get("multiThreadMode"):
+        if multiThreadMode:
            self._lock.release()

        kb.dataOutputFlag = True
@@ -99,7 +103,7 @@ class Dump(object):
        self._outputFile = os.path.join(conf.outputPath, "log")
        try:
            self._outputFP = openFile(self._outputFile, "ab" if not conf.flushSession else "wb")
-        except IOError, ex:
+        except IOError as ex:
            errMsg = "error occurred while opening log file ('%s')" % getSafeExString(ex)
            raise SqlmapGenericException(errMsg)

@@ -110,8 +114,6 @@ class Dump(object):
        self._write(data, content_type=content_type)

    def string(self, header, data, content_type=None, sort=True):
-        kb.stickyLevel = None
-
        if conf.api:
            self._write(data, content_type=content_type)
            return
@@ -133,7 +135,7 @@ class Dump(object):
            if "\n" in _:
                self._write("%s:\n---\n%s\n---" % (header, _))
            else:
-                self._write("%s:    %s" % (header, ("'%s'" % _) if isinstance(data, basestring) else _))
+                self._write("%s: %s" % (header, ("'%s'" % _) if isinstance(data, six.string_types) else _))
        else:
            self._write("%s:\tNone" % header)

@@ -142,7 +144,7 @@ class Dump(object):
            try:
                elements = set(elements)
                elements = list(elements)
-                elements.sort(key=lambda _: _.lower() if isinstance(_, basestring) else _)
+                elements.sort(key=lambda _: _.lower() if hasattr(_, "lower") else _)
            except:
                pass

@@ -154,7 +156,7 @@ class Dump(object):
            self._write("%s [%d]:" % (header, len(elements)))

        for element in elements:
-            if isinstance(element, basestring):
+            if isinstance(element, six.string_types):
                self._write("[*] %s" % element)
            elif isListLike(element):
                self._write("[*] " + ", ".join(getUnicode(e) for e in element))
@@ -193,7 +195,7 @@ class Dump(object):
            userSettings = userSettings[0]

        users = userSettings.keys()
-        users.sort(key=lambda _: _.lower() if isinstance(_, basestring) else _)
+        users.sort(key=lambda _: _.lower() if hasattr(_, "lower") else _)

        if conf.api:
            self._write(userSettings, content_type=content_type)
@@ -287,7 +289,7 @@ class Dump(object):
                    colType = None

                    colList = columns.keys()
-                    colList.sort(key=lambda _: _.lower() if isinstance(_, basestring) else _)
+                    colList.sort(key=lambda _: _.lower() if hasattr(_, "lower") else _)

                    for column in colList:
                        colType = columns[column]
@@ -379,7 +381,7 @@ class Dump(object):
                    if count is None:
                        count = "Unknown"

-                    tables.sort(key=lambda _: _.lower() if isinstance(_, basestring) else _)
+                    tables.sort(key=lambda _: _.lower() if hasattr(_, "lower") else _)

                    for table in tables:
                        blank1 = " " * (maxlength1 - len(normalizeUnicode(table) or unicode(table)))
@@ -426,10 +428,10 @@ class Dump(object):
                    if not os.path.isdir(dumpDbPath):
                        try:
                            os.makedirs(dumpDbPath)
-                        except Exception, ex:
+                        except Exception as ex:
                            try:
                                tempDir = tempfile.mkdtemp(prefix="sqlmapdb")
-                            except IOError, _:
+                            except IOError as _:
                                errMsg = "unable to write to the temporary directory ('%s'). " % _
                                errMsg += "Please make sure that your disk is not full and "
                                errMsg += "that you have sufficient write permissions to "
@@ -624,8 +626,8 @@ class Dump(object):
                                with open(filepath, "wb") as f:
                                    _ = safechardecode(value, True)
                                    f.write(_)
-                        except magic.MagicException, err:
-                            logger.debug(str(err))
+                        except magic.MagicException as ex:
+                            logger.debug(getSafeExString(ex))

                    if conf.dumpFormat == DUMP_FORMAT.CSV:
                        if field == fields:
--- a/lib/core/enums.py
+++ b/lib/core/enums.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -156,13 +156,17 @@ class HASH:

 # Reference: http://www.zytrax.com/tech/web/mobile_ids.html
 class MOBILES:
-    BLACKBERRY = ("BlackBerry 9900", "Mozilla/5.0 (BlackBerry; U; BlackBerry 9900; en) AppleWebKit/534.11+ (KHTML, like Gecko) Version/7.1.0.346 Mobile Safari/534.11+")
-    GALAXY = ("Samsung Galaxy S", "Mozilla/5.0 (Linux; U; Android 2.2; en-US; SGH-T959D Build/FROYO) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1")
+    BLACKBERRY = ("BlackBerry Z10", "Mozilla/5.0 (BB10; Kbd) AppleWebKit/537.35+ (KHTML, like Gecko) Version/10.3.3.2205 Mobile Safari/537.35+")
+    GALAXY = ("Samsung Galaxy S7", "Mozilla/5.0 (Linux; Android 7.0; SM-G930V Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36")
    HP = ("HP iPAQ 6365", "Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; HP iPAQ h6300)")
-    HTC = ("HTC Sensation", "Mozilla/5.0 (Linux; U; Android 4.0.3; de-ch; HTC Sensation Build/IML74K) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30")
-    IPHONE = ("Apple iPhone 4s", "Mozilla/5.0 (iPhone; CPU iPhone OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B179 Safari/7534.48.3")
+    HTC = ("HTC 10", "Mozilla/5.0 (Linux; Android 8.0.0; HTC 10 Build/OPR1.170623.027) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Mobile Safari/537.36")
+    HUAWEI = ("Huawei P8", "Mozilla/5.0 (Linux; Android 4.4.4; HUAWEI H891L Build/HuaweiH891L) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/33.0.0.0 Mobile Safari/537.36")
+    IPHONE = ("Apple iPhone 8", "Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1")
+    LUMIA = ("Microsoft Lumia 950", "Mozilla/5.0 (Windows Phone 10.0; Android 6.0.1; Microsoft; Lumia 950) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Mobile Safari/537.36 Edge/15.14977")
    NEXUS = ("Google Nexus 7", "Mozilla/5.0 (Linux; Android 4.1.1; Nexus 7 Build/JRO03D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Safari/535.19")
    NOKIA = ("Nokia N97", "Mozilla/5.0 (SymbianOS/9.4; Series60/5.0 NokiaN97-1/10.0.012; Profile/MIDP-2.1 Configuration/CLDC-1.1; en-us) AppleWebKit/525 (KHTML, like Gecko) WicKed/7.1.12344")
+    PIXEL = ("Google Pixel", "Mozilla/5.0 (Linux; Android 8.0.0; Pixel Build/OPR3.170623.013) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.111 Mobile Safari/537.36")
+    XIAOMI = ("Xiaomi Mi 3", "Mozilla/5.0 (Linux; U; Android 4.4.4; en-gb; MI 3W Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/39.0.0.0 Mobile Safari/537.36 XiaoMi/MiuiBrowser/2.1.1")

 class PROXY_TYPE:
    HTTP = "HTTP"
@@ -227,6 +231,7 @@ class HASHDB_KEYS:
    DBMS = "DBMS"
    DBMS_FORK = "DBMS_FORK"
    CHECK_WAF_RESULT = "CHECK_WAF_RESULT"
+    CHECK_NULL_CONNECTION_RESULT = "CHECK_NULL_CONNECTION_RESULT"
    CONF_TMP_PATH = "CONF_TMP_PATH"
    KB_ABS_FILE_PATHS = "KB_ABS_FILE_PATHS"
    KB_BRUTE_COLUMNS = "KB_BRUTE_COLUMNS"
@@ -248,7 +253,7 @@ class PAYLOAD:
        2: "error-based",
        3: "inline query",
        4: "stacked queries",
-        5: "AND/OR time-based blind",
+        5: "time-based blind",
        6: "UNION query",
    }

@@ -310,7 +315,7 @@ class ADJUST_TIME_DELAY:
    NO = 0
    YES = 1

-class WEB_API:
+class WEB_PLATFORM:
    PHP = "php"
    ASP = "asp"
    ASPX = "aspx"
@@ -373,6 +378,7 @@ class MKSTEMP_PREFIX:
    COOKIE_JAR = "sqlmapcookiejar-"
    BIG_ARRAY = "sqlmapbigarray-"
    SPECIFIC_RESPONSE = "sqlmapresponse-"
+    PREPROCESS = "sqlmappreprocess-"

 class TIMEOUT_STATE:
    NORMAL = 0
--- a/lib/core/exception.py
+++ b/lib/core/exception.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/lib/core/log.py
+++ b/lib/core/log.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/lib/core/option.py
+++ b/lib/core/option.py
@@ -1,11 +1,10 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

-import cookielib
 import glob
 import inspect
 import logging
@@ -13,13 +12,10 @@ import os
 import random
 import re
 import socket
-import string
 import sys
 import tempfile
 import threading
 import time
-import urllib2
-import urlparse

 import lib.controller.checks
 import lib.core.common
@@ -33,13 +29,17 @@ from lib.core.common import Backend
 from lib.core.common import boldifyMessage
 from lib.core.common import checkFile
 from lib.core.common import dataToStdout
+from lib.core.common import decodeStringEscape
 from lib.core.common import getPublicTypeMembers
 from lib.core.common import getSafeExString
+from lib.core.common import getUnicode
+from lib.core.common import filterNone
 from lib.core.common import findLocalPort
 from lib.core.common import findPageForms
 from lib.core.common import getConsoleWidth
 from lib.core.common import getFileItems
 from lib.core.common import getFileType
+from lib.core.common import intersect
 from lib.core.common import normalizePath
 from lib.core.common import ntToPosixSlashes
 from lib.core.common import openFile
@@ -52,12 +52,14 @@ from lib.core.common import readInput
 from lib.core.common import resetCookieJar
 from lib.core.common import runningAsAdmin
 from lib.core.common import safeExpandUser
+from lib.core.common import safeFilepathEncode
 from lib.core.common import saveConfig
 from lib.core.common import setColor
 from lib.core.common import setOptimize
 from lib.core.common import setPaths
 from lib.core.common import singleTimeWarnMessage
 from lib.core.common import urldecode
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -65,6 +67,7 @@ from lib.core.data import mergedOptions
 from lib.core.data import queries
 from lib.core.datatype import AttribDict
 from lib.core.datatype import InjectionDict
+from lib.core.datatype import OrderedSet
 from lib.core.defaults import defaults
 from lib.core.dicts import DBMS_DICT
 from lib.core.dicts import DUMP_REPLACEMENTS
@@ -74,6 +77,7 @@ from lib.core.enums import CUSTOM_LOGGING
 from lib.core.enums import DUMP_FORMAT
 from lib.core.enums import HTTP_HEADER
 from lib.core.enums import HTTPMETHOD
+from lib.core.enums import MKSTEMP_PREFIX
 from lib.core.enums import MOBILES
 from lib.core.enums import OPTION_TYPE
 from lib.core.enums import PAYLOAD
@@ -100,6 +104,7 @@ from lib.core.optiondict import optDict
 from lib.core.settings import CODECS_LIST_PAGE
 from lib.core.settings import CUSTOM_INJECTION_MARK_CHAR
 from lib.core.settings import DBMS_ALIASES
+from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import DEFAULT_PAGE_ENCODING
 from lib.core.settings import DEFAULT_TOR_HTTP_PORTS
 from lib.core.settings import DEFAULT_TOR_SOCKS_PORTS
@@ -119,7 +124,6 @@ from lib.core.settings import SQLMAP_ENVIRONMENT_PREFIX
 from lib.core.settings import SUPPORTED_DBMS
 from lib.core.settings import SUPPORTED_OS
 from lib.core.settings import TIME_DELAY_CANDIDATES
-from lib.core.settings import UNICODE_ENCODING
 from lib.core.settings import UNION_CHAR_REGEX
 from lib.core.settings import UNKNOWN_DBMS_VERSION
 from lib.core.settings import URI_INJECTABLE_REGEX
@@ -134,6 +138,7 @@ from lib.request.basic import checkCharEncoding
 from lib.request.connect import Connect as Request
 from lib.request.dns import DNSServer
 from lib.request.basicauthhandler import SmartHTTPBasicAuthHandler
+from lib.request.chunkedhandler import ChunkedHandler
 from lib.request.httpshandler import HTTPSHandler
 from lib.request.pkihandler import HTTPSPKIAuthHandler
 from lib.request.rangehandler import HTTPRangeHandler
@@ -144,16 +149,20 @@ from lib.utils.crawler import crawl
 from lib.utils.deps import checkDependencies
 from lib.utils.search import search
 from lib.utils.purge import purge
+from thirdparty import six
 from thirdparty.keepalive import keepalive
 from thirdparty.multipart import multipartpost
-from thirdparty.oset.pyoset import oset
+from thirdparty.six.moves import http_client as _http_client
+from thirdparty.six.moves import http_cookiejar as _http_cookiejar
+from thirdparty.six.moves import urllib as _urllib
 from thirdparty.socks import socks
 from xml.etree.ElementTree import ElementTree

-authHandler = urllib2.BaseHandler()
+authHandler = _urllib.request.BaseHandler()
+chunkedHandler = ChunkedHandler()
 httpsHandler = HTTPSHandler()
 keepAliveHandler = keepalive.HTTPHandler()
-proxyHandler = urllib2.ProxyHandler()
+proxyHandler = _urllib.request.ProxyHandler()
 redirectHandler = SmartRedirectHandler()
 rangeHandler = HTTPRangeHandler()
 multipartPostHandler = multipartpost.MultipartPostHandler()
@@ -193,7 +202,7 @@ def _loadQueries():
    tree = ElementTree()
    try:
        tree.parse(paths.QUERIES_XML)
-    except Exception, ex:
+    except Exception as ex:
        errMsg = "something appears to be wrong with "
        errMsg += "the file '%s' ('%s'). Please make " % (paths.QUERIES_XML, getSafeExString(ex))
        errMsg += "sure that you haven't made any changes to it"
@@ -221,12 +230,13 @@ def _setMultipleTargets():
        errMsg = "the specified list of targets does not exist"
        raise SqlmapFilePathException(errMsg)

-    if os.path.isfile(conf.logFile):
+    if checkFile(conf.logFile, False):
        for target in parseRequestFile(conf.logFile):
-            url = target[0]
-            if url not in seen:
+            url, _, data, _, _ = target
+            key = re.sub(r"(\w+=)[^%s ]*" % (conf.paramDel or DEFAULT_GET_POST_DELIMITER), r"\g<1>", "%s %s" % (url, data))
+            if key not in seen:
                kb.targets.add(target)
-                seen.add(url)
+                seen.add(key)

    elif os.path.isdir(conf.logFile):
        files = os.listdir(conf.logFile)
@@ -237,10 +247,11 @@ def _setMultipleTargets():
                continue

            for target in parseRequestFile(os.path.join(conf.logFile, reqFile)):
-                url = target[0]
-                if url not in seen:
+                url, _, data, _, _ = target
+                key = re.sub(r"(\w+=)[^%s ]*" % (conf.paramDel or DEFAULT_GET_POST_DELIMITER), r"\g<1>", "%s %s" % (url, data))
+                if key not in seen:
                    kb.targets.add(target)
-                    seen.add(url)
+                    seen.add(key)

    else:
        errMsg = "the specified list of targets is not a file "
@@ -285,7 +296,7 @@ def _setRequestFromFile():
        conf.requestFile = safeExpandUser(conf.requestFile)
        seen = set()

-        if not os.path.isfile(conf.requestFile):
+        if not checkFile(conf.requestFile, False):
            errMsg = "specified HTTP request file '%s' " % conf.requestFile
            errMsg += "does not exist"
            raise SqlmapFilePathException(errMsg)
@@ -302,7 +313,7 @@ def _setRequestFromFile():
    if conf.secondReq:
        conf.secondReq = safeExpandUser(conf.secondReq)

-        if not os.path.isfile(conf.secondReq):
+        if not checkFile(conf.secondReq, False):
            errMsg = "specified second-order HTTP request file '%s' " % conf.secondReq
            errMsg += "does not exist"
            raise SqlmapFilePathException(errMsg)
@@ -310,7 +321,7 @@ def _setRequestFromFile():
        infoMsg = "parsing second-order HTTP request from '%s'" % conf.secondReq
        logger.info(infoMsg)

-        target = parseRequestFile(conf.secondReq, False).next()
+        target = next(parseRequestFile(conf.secondReq, False))
        kb.secondReq = target

 def _setCrawler():
@@ -332,7 +343,7 @@ def _setCrawler():
                if conf.verbose in (1, 2):
                    status = "%d/%d links visited (%d%%)" % (i + 1, len(targets), round(100.0 * (i + 1) / len(targets)))
                    dataToStdout("\r[%s] [INFO] %s" % (time.strftime("%X"), status), True)
-            except Exception, ex:
+            except Exception as ex:
                errMsg = "problem occurred while crawling at '%s' ('%s')" % (target, getSafeExString(ex))
                logger.error(errMsg)

@@ -404,7 +415,7 @@ def _setBulkMultipleTargets():
    infoMsg = "parsing multiple targets list from '%s'" % conf.bulkFile
    logger.info(infoMsg)

-    if not os.path.isfile(conf.bulkFile):
+    if not checkFile(conf.bulkFile, False):
        errMsg = "the specified bulk file "
        errMsg += "does not exist"
        raise SqlmapFilePathException(errMsg)
@@ -468,7 +479,7 @@ def _findPageForms():
                    dataToStdout("\r[%s] [INFO] %s" % (time.strftime("%X"), status), True)
            except KeyboardInterrupt:
                break
-            except Exception, ex:
+            except Exception as ex:
                errMsg = "problem occurred while searching for forms at '%s' ('%s')" % (target, getSafeExString(ex))
                logger.error(errMsg)

@@ -543,11 +554,11 @@ def _setMetasploit():

    if conf.msfPath:
        for path in (conf.msfPath, os.path.join(conf.msfPath, "bin")):
-            if any(os.path.exists(normalizePath(os.path.join(path, _))) for _ in ("msfcli", "msfconsole")):
+            if any(os.path.exists(normalizePath(os.path.join(path, "%s%s" % (_, ".bat" if IS_WIN else "")))) for _ in ("msfcli", "msfconsole")):
                msfEnvPathExists = True
-                if all(os.path.exists(normalizePath(os.path.join(path, _))) for _ in ("msfvenom",)):
+                if all(os.path.exists(normalizePath(os.path.join(path, "%s%s" % (_, ".bat" if IS_WIN else "")))) for _ in ("msfvenom",)):
                    kb.oldMsf = False
-                elif all(os.path.exists(normalizePath(os.path.join(path, _))) for _ in ("msfencode", "msfpayload")):
+                elif all(os.path.exists(normalizePath(os.path.join(path, "%s%s" % (_, ".bat" if IS_WIN else "")))) for _ in ("msfencode", "msfpayload")):
                    kb.oldMsf = True
                else:
                    msfEnvPathExists = False
@@ -582,11 +593,11 @@ def _setMetasploit():
        for envPath in envPaths:
            envPath = envPath.replace(";", "")

-            if any(os.path.exists(normalizePath(os.path.join(envPath, _))) for _ in ("msfcli", "msfconsole")):
+            if any(os.path.exists(normalizePath(os.path.join(envPath, "%s%s" % (_, ".bat" if IS_WIN else "")))) for _ in ("msfcli", "msfconsole")):
                msfEnvPathExists = True
-                if all(os.path.exists(normalizePath(os.path.join(envPath, _))) for _ in ("msfvenom",)):
+                if all(os.path.exists(normalizePath(os.path.join(envPath, "%s%s" % (_, ".bat" if IS_WIN else "")))) for _ in ("msfvenom",)):
                    kb.oldMsf = False
-                elif all(os.path.exists(normalizePath(os.path.join(envPath, _))) for _ in ("msfencode", "msfpayload")):
+                elif all(os.path.exists(normalizePath(os.path.join(envPath, "%s%s" % (_, ".bat" if IS_WIN else "")))) for _ in ("msfencode", "msfpayload")):
                    kb.oldMsf = True
                else:
                    msfEnvPathExists = False
@@ -650,7 +661,7 @@ def _setTechnique():
    validTechniques = sorted(getPublicTypeMembers(PAYLOAD.TECHNIQUE), key=lambda x: x[1])
    validLetters = [_[0][0].upper() for _ in validTechniques]

-    if conf.tech and isinstance(conf.tech, basestring):
+    if conf.tech and isinstance(conf.tech, six.string_types):
        _ = []

        for letter in conf.tech.upper():
@@ -728,8 +739,8 @@ def _setTamperingFunctions():
        for script in re.split(PARAMETER_SPLITTING_REGEX, conf.tamper):
            found = False

-            path = paths.SQLMAP_TAMPER_PATH.encode(sys.getfilesystemencoding() or UNICODE_ENCODING)
-            script = script.strip().encode(sys.getfilesystemencoding() or UNICODE_ENCODING)
+            path = safeFilepathEncode(paths.SQLMAP_TAMPER_PATH)
+            script = safeFilepathEncode(script.strip())

            try:
                if not script:
@@ -764,9 +775,9 @@ def _setTamperingFunctions():
                sys.path.insert(0, dirname)

            try:
-                module = __import__(filename[:-3].encode(sys.getfilesystemencoding() or UNICODE_ENCODING))
-            except Exception, ex:
-                raise SqlmapSyntaxException("cannot import tamper module '%s' (%s)" % (filename[:-3], getSafeExString(ex)))
+                module = __import__(safeFilepathEncode(filename[:-3]))
+            except Exception as ex:
+                raise SqlmapSyntaxException("cannot import tamper module '%s' (%s)" % (getUnicode(filename[:-3]), getSafeExString(ex)))

            priority = PRIORITY.NORMAL if not hasattr(module, "__priority__") else module.__priority__

@@ -774,7 +785,7 @@ def _setTamperingFunctions():
                if name == "tamper" and inspect.getargspec(function).args and inspect.getargspec(function).keywords == "kwargs":
                    found = True
                    kb.tamperFunctions.append(function)
-                    function.func_name = module.__name__
+                    function.__name__ = module.__name__

                    if check_priority and priority > last_priority:
                        message = "it appears that you might have mixed "
@@ -798,9 +809,9 @@ def _setTamperingFunctions():
                elif name == "dependencies":
                    try:
                        function()
-                    except Exception, ex:
+                    except Exception as ex:
                        errMsg = "error occurred while checking dependencies "
-                        errMsg += "for tamper module '%s' ('%s')" % (filename[:-3], getSafeExString(ex))
+                        errMsg += "for tamper module '%s' ('%s')" % (getUnicode(filename[:-3]), getSafeExString(ex))
                        raise SqlmapGenericException(errMsg)

            if not found:
@@ -820,6 +831,80 @@ def _setTamperingFunctions():
            for _, function in priorities:
                kb.tamperFunctions.append(function)

+def _setPreprocessFunctions():
+    """
+    Loads preprocess functions from given script(s)
+    """
+
+    if conf.preprocess:
+        for script in re.split(PARAMETER_SPLITTING_REGEX, conf.preprocess):
+            found = False
+
+            script = safeFilepathEncode(script.strip())
+
+            try:
+                if not script:
+                    continue
+
+                if not os.path.exists(script):
+                    errMsg = "preprocess script '%s' does not exist" % script
+                    raise SqlmapFilePathException(errMsg)
+
+                elif not script.endswith(".py"):
+                    errMsg = "preprocess script '%s' should have an extension '.py'" % script
+                    raise SqlmapSyntaxException(errMsg)
+            except UnicodeDecodeError:
+                errMsg = "invalid character provided in option '--preprocess'"
+                raise SqlmapSyntaxException(errMsg)
+
+            dirname, filename = os.path.split(script)
+            dirname = os.path.abspath(dirname)
+
+            infoMsg = "loading preprocess module '%s'" % filename[:-3]
+            logger.info(infoMsg)
+
+            if not os.path.exists(os.path.join(dirname, "__init__.py")):
+                errMsg = "make sure that there is an empty file '__init__.py' "
+                errMsg += "inside of preprocess scripts directory '%s'" % dirname
+                raise SqlmapGenericException(errMsg)
+
+            if dirname not in sys.path:
+                sys.path.insert(0, dirname)
+
+            try:
+                module = __import__(safeFilepathEncode(filename[:-3]))
+            except Exception as ex:
+                raise SqlmapSyntaxException("cannot import preprocess module '%s' (%s)" % (getUnicode(filename[:-3]), getSafeExString(ex)))
+
+            for name, function in inspect.getmembers(module, inspect.isfunction):
+                if name == "preprocess" and inspect.getargspec(function).args and all(_ in inspect.getargspec(function).args for _ in ("page", "headers", "code")):
+                    found = True
+
+                    kb.preprocessFunctions.append(function)
+                    function.__name__ = module.__name__
+
+                    break
+
+            if not found:
+                errMsg = "missing function 'preprocess(page, headers=None, code=None)' "
+                errMsg += "in preprocess script '%s'" % script
+                raise SqlmapGenericException(errMsg)
+            else:
+                try:
+                    _, _, _ = function("", {}, None)
+                except:
+                    handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.PREPROCESS, suffix=".py")
+                    os.close(handle)
+
+                    open(filename, "w+b").write("#!/usr/bin/env\n\ndef preprocess(page, headers=None, code=None):\n    return page, headers, code\n")
+                    open(os.path.join(os.path.dirname(filename), "__init__.py"), "w+b").write("pass")
+
+                    errMsg = "function 'preprocess(page, headers=None, code=None)' "
+                    errMsg += "in preprocess script '%s' " % script
+                    errMsg += "should return a tuple '(page, headers, code)' "
+                    errMsg += "(Note: find template script at '%s')" % filename
+                    raise SqlmapGenericException(errMsg)
+
 def _setWafFunctions():
    """
    Loads WAF/IPS detecting functions from script(s)
@@ -842,9 +927,9 @@ def _setWafFunctions():
            try:
                if filename[:-3] in sys.modules:
                    del sys.modules[filename[:-3]]
-                module = __import__(filename[:-3].encode(sys.getfilesystemencoding() or UNICODE_ENCODING))
-            except ImportError, msg:
-                raise SqlmapSyntaxException("cannot import WAF script '%s' (%s)" % (filename[:-3], msg))
+                module = __import__(safeFilepathEncode(filename[:-3]))
+            except ImportError as ex:
+                raise SqlmapSyntaxException("cannot import WAF script '%s' (%s)" % (getUnicode(filename[:-3]), getSafeExString(ex)))

            _ = dict(inspect.getmembers(module))
            if "detect" not in _:
@@ -893,6 +978,12 @@ def _setSocketPreConnect():
                        family, type, proto, address = key
                        s = socket.socket(family, type, proto)
                        s._connect(address)
+                        try:
+                            if type == socket.SOCK_STREAM:
+                                # Reference: https://www.techrepublic.com/article/tcp-ip-options-for-high-performance-data-transmission/
+                                s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
+                        except:
+                            pass
                        with kb.locks.socket:
                            socket._ready[key].append((s._sock, time.time()))
            except KeyboardInterrupt:
@@ -964,8 +1055,8 @@ def _setHTTPHandlers():
        logger.debug(debugMsg)

        try:
-            _ = urlparse.urlsplit(conf.proxy)
-        except Exception, ex:
+            _ = _urllib.parse.urlsplit(conf.proxy)
+        except Exception as ex:
            errMsg = "invalid proxy address '%s' ('%s')" % (conf.proxy, getSafeExString(ex))
            raise SqlmapSyntaxException(errMsg)

@@ -1000,10 +1091,14 @@ def _setHTTPHandlers():
        if scheme in (PROXY_TYPE.SOCKS4, PROXY_TYPE.SOCKS5):
            proxyHandler.proxies = {}

+            if scheme == PROXY_TYPE.SOCKS4:
+                warnMsg = "SOCKS4 does not support resolving (DNS) names (i.e. causing DNS leakage)"
+                singleTimeWarnMessage(warnMsg)
+
            socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5 if scheme == PROXY_TYPE.SOCKS5 else socks.PROXY_TYPE_SOCKS4, hostname, port, username=username, password=password)
-            socks.wrapmodule(urllib2)
+            socks.wrapmodule(_http_client)
        else:
-            socks.unwrapmodule(urllib2)
+            socks.unwrapmodule(_http_client)

            if conf.proxyCred:
                # Reference: http://stackoverflow.com/questions/34079/how-to-specify-an-authenticated-proxy-for-a-python-http-connection
@@ -1019,16 +1114,16 @@ def _setHTTPHandlers():
    debugMsg = "creating HTTP requests opener object"
    logger.debug(debugMsg)

-    handlers = filter(None, [multipartPostHandler, proxyHandler if proxyHandler.proxies else None, authHandler, redirectHandler, rangeHandler, httpsHandler])
+    handlers = filterNone([multipartPostHandler, proxyHandler if proxyHandler.proxies else None, authHandler, redirectHandler, rangeHandler, chunkedHandler if conf.chunked else None, httpsHandler])

    if not conf.dropSetCookie:
        if not conf.loadCookies:
-            conf.cj = cookielib.CookieJar()
+            conf.cj = _http_cookiejar.CookieJar()
        else:
-            conf.cj = cookielib.MozillaCookieJar()
+            conf.cj = _http_cookiejar.MozillaCookieJar()
            resetCookieJar(conf.cj)

-        handlers.append(urllib2.HTTPCookieProcessor(conf.cj))
+        handlers.append(_urllib.request.HTTPCookieProcessor(conf.cj))

    # Reference: http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html
    if conf.keepAlive:
@@ -1044,8 +1139,8 @@ def _setHTTPHandlers():
        else:
            handlers.append(keepAliveHandler)

-    opener = urllib2.build_opener(*handlers)
-    urllib2.install_opener(opener)
+    opener = _urllib.request.build_opener(*handlers)
+    _urllib.request.install_opener(opener)

 def _setSafeVisit():
    """
@@ -1058,14 +1153,14 @@ def _setSafeVisit():
        checkFile(conf.safeReqFile)

        raw = readCachedFileContent(conf.safeReqFile)
-        match = re.search(r"\A([A-Z]+) ([^ ]+) HTTP/[0-9.]+\Z", raw[:raw.find('\n')])
+        match = re.search(r"\A([A-Z]+) ([^ ]+) HTTP/[0-9.]+\Z", raw.split('\n')[0].strip())

        if match:
            kb.safeReq.method = match.group(1)
            kb.safeReq.url = match.group(2)
            kb.safeReq.headers = {}

-            for line in raw[raw.find('\n') + 1:].split('\n'):
+            for line in raw.split('\n')[1:]:
                line = line.strip()
                if line and ':' in line:
                    key, value = line.split(':', 1)
@@ -1077,7 +1172,7 @@ def _setSafeVisit():
                            if value.endswith(":443"):
                                scheme = "https"
                            value = "%s://%s" % (scheme, value)
-                        kb.safeReq.url = urlparse.urljoin(value, kb.safeReq.url)
+                        kb.safeReq.url = _urllib.parse.urljoin(value, kb.safeReq.url)
                else:
                    break

@@ -1186,7 +1281,7 @@ def _setHTTPAuthentication():
        elif authType == AUTH_TYPE.NTLM:
            regExp = "^(.*\\\\.*):(.*?)$"
            errMsg = "HTTP NTLM authentication credentials value must "
-            errMsg += "be in format 'DOMAIN\username:password'"
+            errMsg += "be in format 'DOMAIN\\username:password'"
        elif authType == AUTH_TYPE.PKI:
            errMsg = "HTTP PKI authentication require "
            errMsg += "usage of option `--auth-pki`"
@@ -1200,7 +1295,7 @@ def _setHTTPAuthentication():
        conf.authUsername = aCredRegExp.group(1)
        conf.authPassword = aCredRegExp.group(2)

-        kb.passwordMgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
+        kb.passwordMgr = _urllib.request.HTTPPasswordMgrWithDefaultRealm()

        _setAuthCred()

@@ -1208,7 +1303,7 @@ def _setHTTPAuthentication():
            authHandler = SmartHTTPBasicAuthHandler(kb.passwordMgr)

        elif authType == AUTH_TYPE.DIGEST:
-            authHandler = urllib2.HTTPDigestAuthHandler(kb.passwordMgr)
+            authHandler = _urllib.request.HTTPDigestAuthHandler(kb.passwordMgr)

        elif authType == AUTH_TYPE.NTLM:
            try:
@@ -1267,7 +1362,14 @@ def _setHTTPUserAgent():
          file choosed as user option
    """

+    debugMsg = "setting the HTTP User-Agent header"
+    logger.debug(debugMsg)
+
    if conf.mobile:
+        if conf.randomAgent:
+            _ = random.sample([_[1] for _ in getPublicTypeMembers(MOBILES, True)], 1)[0]
+            conf.httpHeaders.append((HTTP_HEADER.USER_AGENT, _))
+        else:
            message = "which smartphone do you want sqlmap to imitate "
            message += "through HTTP User-Agent header?\n"
            items = sorted(getPublicTypeMembers(MOBILES, True))
@@ -1286,9 +1388,6 @@ def _setHTTPUserAgent():
            conf.httpHeaders.append((HTTP_HEADER.USER_AGENT, item[1]))

    elif conf.agent:
-        debugMsg = "setting the HTTP User-Agent header"
-        logger.debug(debugMsg)
-
        conf.httpHeaders.append((HTTP_HEADER.USER_AGENT, conf.agent))

    elif not conf.randomAgent:
@@ -1366,8 +1465,8 @@ def _setHostname():

    if conf.url:
        try:
-            conf.hostname = urlparse.urlsplit(conf.url).netloc.split(':')[0]
-        except ValueError, ex:
+            conf.hostname = _urllib.parse.urlsplit(conf.url).netloc.split(':')[0]
+        except ValueError as ex:
            errMsg = "problem occurred while "
            errMsg += "parsing an URL '%s' ('%s')" % (conf.url, getSafeExString(ex))
            raise SqlmapDataException(errMsg)
@@ -1402,6 +1501,41 @@ def _checkDependencies():
    if conf.dependencies:
        checkDependencies()

+def _createHomeDirectories():
+    """
+    Creates directories inside sqlmap's home directory
+    """
+
+    for context in "output", "history":
+        directory = paths["SQLMAP_%s_PATH" % context.upper()]
+        try:
+            if not os.path.isdir(directory):
+                os.makedirs(directory)
+
+            _ = os.path.join(directory, randomStr())
+            open(_, "w+b").close()
+            os.remove(_)
+
+            if conf.outputDir and context == "output":
+                warnMsg = "using '%s' as the %s directory" % (directory, context)
+                logger.warn(warnMsg)
+        except (OSError, IOError) as ex:
+            try:
+                tempDir = tempfile.mkdtemp(prefix="sqlmap%s" % context)
+            except Exception as _:
+                errMsg = "unable to write to the temporary directory ('%s'). " % _
+                errMsg += "Please make sure that your disk is not full and "
+                errMsg += "that you have sufficient write permissions to "
+                errMsg += "create temporary files and/or directories"
+                raise SqlmapSystemException(errMsg)
+
+            warnMsg = "unable to %s %s directory " % ("create" if not os.path.isdir(directory) else "write to the", context)
+            warnMsg += "'%s' (%s). " % (directory, getUnicode(ex))
+            warnMsg += "Using temporary directory '%s' instead" % getUnicode(tempDir)
+            logger.warn(warnMsg)
+
+            paths["SQLMAP_%s_PATH" % context.upper()] = tempDir
+
 def _createTemporaryDirectory():
    """
    Creates temporary directory for this run.
@@ -1421,7 +1555,7 @@ def _createTemporaryDirectory():

            warnMsg = "using '%s' as the temporary directory" % conf.tmpDir
            logger.warn(warnMsg)
-        except (OSError, IOError), ex:
+        except (OSError, IOError) as ex:
            errMsg = "there has been a problem while accessing "
            errMsg += "temporary directory location(s) ('%s')" % getSafeExString(ex)
            raise SqlmapSystemException(errMsg)
@@ -1429,7 +1563,7 @@ def _createTemporaryDirectory():
        try:
            if not os.path.isdir(tempfile.gettempdir()):
                os.makedirs(tempfile.gettempdir())
-        except Exception, ex:
+        except Exception as ex:
            warnMsg = "there has been a problem while accessing "
            warnMsg += "system's temporary directory location(s) ('%s'). Please " % getSafeExString(ex)
            warnMsg += "make sure that there is enough disk space left. If problem persists, "
@@ -1448,7 +1582,7 @@ def _createTemporaryDirectory():
    if not os.path.isdir(tempfile.tempdir):
        try:
            os.makedirs(tempfile.tempdir)
-        except Exception, ex:
+        except Exception as ex:
            errMsg = "there has been a problem while setting "
            errMsg += "temporary directory location ('%s')" % getSafeExString(ex)
            raise SqlmapSystemException(errMsg)
@@ -1491,11 +1625,8 @@ def _cleanupOptions():
    else:
        conf.rParam = []

-    if conf.paramDel and '\\' in conf.paramDel:
-        try:
-            conf.paramDel = conf.paramDel.decode("string_escape")
-        except ValueError:
-            pass
+    if conf.paramDel:
+        conf.paramDel = decodeStringEscape(conf.paramDel)

    if conf.skip:
        conf.skip = conf.skip.replace(" ", "")
@@ -1511,6 +1642,8 @@ def _cleanupOptions():

    if conf.url:
        conf.url = conf.url.strip()
+        if not re.search(r"\A\w+://", conf.url):
+            conf.url = "http://%s" % conf.url

    if conf.fileRead:
        conf.fileRead = ntToPosixSlashes(normalizePath(conf.fileRead))
@@ -1560,6 +1693,23 @@ def _cleanupOptions():
        except re.error:
            conf.testFilter = re.escape(conf.testFilter)

+    if conf.csrfToken:
+        original = conf.csrfToken
+        try:
+            re.compile(conf.csrfToken)
+
+            if re.escape(conf.csrfToken) != conf.csrfToken:
+                message = "provided value for option '--csrf-token' is a regular expression? [y/N] "
+                if not readInput(message, default='N', boolean=True):
+                    conf.csrfToken = re.escape(conf.csrfToken)
+        except re.error:
+            conf.csrfToken = re.escape(conf.csrfToken)
+        finally:
+            class _(unicode):
+                pass
+            conf.csrfToken = _(conf.csrfToken)
+            conf.csrfToken._original = original
+
    if conf.testSkip:
        conf.testSkip = conf.testSkip.strip('*+')
        conf.testSkip = re.sub(r"([^.])([*+])", r"\g<1>.\g<2>", conf.testSkip)
@@ -1588,9 +1738,9 @@ def _cleanupOptions():
        conf.code = int(conf.code)

    if conf.csvDel:
-        conf.csvDel = conf.csvDel.decode("string_escape")  # e.g. '\\t' -> '\t'
+        conf.csvDel = decodeStringEscape(conf.csvDel)

-    if conf.torPort and isinstance(conf.torPort, basestring) and conf.torPort.isdigit():
+    if conf.torPort and hasattr(conf.torPort, "isdigit") and conf.torPort.isdigit():
        conf.torPort = int(conf.torPort)

    if conf.torType:
@@ -1601,18 +1751,13 @@ def _cleanupOptions():
        setPaths(paths.SQLMAP_ROOT_PATH)

    if conf.string:
-        try:
-            conf.string = conf.string.decode("unicode_escape")
-        except:
-            charset = string.whitespace.replace(" ", "")
-            for _ in charset:
-                conf.string = conf.string.replace(_.encode("string_escape"), _)
+        conf.string = decodeStringEscape(conf.string)

    if conf.getAll:
        map(lambda _: conf.__setitem__(_, True), WIZARD.ALL)

    if conf.noCast:
-        for _ in DUMP_REPLACEMENTS.keys():
+        for _ in list(DUMP_REPLACEMENTS.keys()):
            del DUMP_REPLACEMENTS[_]

    if conf.dumpFormat:
@@ -1644,8 +1789,8 @@ def _cleanupEnvironment():
    Cleanup environment (e.g. from leftovers after --sqlmap-shell).
    """

-    if issubclass(urllib2.socket.socket, socks.socksocket):
-        socks.unwrapmodule(urllib2)
+    if issubclass(_http_client.socket.socket, socks.socksocket):
+        socks.unwrapmodule(_http_client)

    if hasattr(socket, "_ready"):
        socket._ready.clear()
@@ -1742,6 +1887,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.connErrorChoice = None
    kb.connErrorCounter = 0
    kb.cookieEncodeChoice = None
+    kb.copyExecTest = None
    kb.counters = {}
    kb.customInjectionMark = CUSTOM_INJECTION_MARK_CHAR
    kb.data = AttribDict()
@@ -1792,7 +1938,6 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.injection = InjectionDict()
    kb.injections = []
    kb.laggingChecked = False
-    kb.lastInputMessage = None
    kb.lastParserStatus = None

    kb.locks = AttribDict()
@@ -1802,7 +1947,6 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.matchRatio = None
    kb.maxConnectionsFlag = False
    kb.mergeCookies = None
-    kb.multiThreadMode = False
    kb.negativeLogic = False
    kb.nullConnection = None
    kb.oldMsf = None
@@ -1855,7 +1999,6 @@ def _setKnowledgeBaseAttributes(flushAll=True):
    kb.reduceTests = None
    kb.tlsSNI = {}
    kb.stickyDBMS = False
-    kb.stickyLevel = None
    kb.storeCrawlingChoice = None
    kb.storeHashesChoice = None
    kb.suppressResumeInfo = False
@@ -1879,9 +2022,10 @@ def _setKnowledgeBaseAttributes(flushAll=True):
        kb.headerPaths = {}
        kb.keywords = set(getFileItems(paths.SQL_KEYWORDS))
        kb.passwordMgr = None
+        kb.preprocessFunctions = []
        kb.skipVulnHost = None
        kb.tamperFunctions = []
-        kb.targets = oset()
+        kb.targets = OrderedSet()
        kb.testedParams = set()
        kb.userAgents = None
        kb.vainRun = True
@@ -2109,9 +2253,9 @@ def _setDNSServer():
        try:
            conf.dnsServer = DNSServer()
            conf.dnsServer.run()
-        except socket.error, msg:
+        except socket.error as ex:
            errMsg = "there was an error while setting up "
-            errMsg += "DNS server instance ('%s')" % msg
+            errMsg += "DNS server instance ('%s')" % getSafeExString(ex)
            raise SqlmapGenericException(errMsg)
    else:
        errMsg = "you need to run sqlmap as an administrator "
@@ -2174,7 +2318,11 @@ def _setTorSocksProxySettings():

    # SOCKS5 to prevent DNS leaks (http://en.wikipedia.org/wiki/Tor_%28anonymity_network%29)
    socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5 if conf.torType == PROXY_TYPE.SOCKS5 else socks.PROXY_TYPE_SOCKS4, LOCALHOST, port)
-    socks.wrapmodule(urllib2)
+    socks.wrapmodule(_http_client)
+
+def _setHttpChunked():
+    if conf.chunked and conf.data:
+        _http_client.HTTPConnection._set_content_length = lambda self, a, b: None

 def _checkWebSocket():
    if conf.url and (conf.url.startswith("ws:/") or conf.url.startswith("wss:/")):
@@ -2247,6 +2395,10 @@ def _basicOptionValidation():
        errMsg = "option '-d' is incompatible with option '-u' ('--url')"
        raise SqlmapSyntaxException(errMsg)

+    if conf.direct and conf.dbms:
+        errMsg = "option '-d' is incompatible with option '--dbms'"
+        raise SqlmapSyntaxException(errMsg)
+
    if conf.identifyWaf and conf.skipWaf:
        errMsg = "switch '--identify-waf' is incompatible with switch '--skip-waf'"
        raise SqlmapSyntaxException(errMsg)
@@ -2259,6 +2411,10 @@ def _basicOptionValidation():
        errMsg = "switch '--dump' is incompatible with switch '--search'"
        raise SqlmapSyntaxException(errMsg)

+    if conf.chunked and not any((conf.data, conf.requestFile, conf.forms)):
+        errMsg = "switch '--chunked' requires usage of (POST) options/switches '--data', '-r' or '--forms'"
+        raise SqlmapSyntaxException(errMsg)
+
    if conf.api and not conf.configFile:
        errMsg = "switch '--api' requires usage of option '-c'"
        raise SqlmapSyntaxException(errMsg)
@@ -2275,7 +2431,7 @@ def _basicOptionValidation():
        errMsg = "option '--not-string' is incompatible with switch '--null-connection'"
        raise SqlmapSyntaxException(errMsg)

-    if conf.notString and conf.nullConnection:
+    if conf.tor and conf.osPwn:
        errMsg = "option '--tor' is incompatible with switch '--os-pwn'"
        raise SqlmapSyntaxException(errMsg)

@@ -2298,14 +2454,14 @@ def _basicOptionValidation():
    if conf.regexp:
        try:
            re.compile(conf.regexp)
-        except Exception, ex:
+        except Exception as ex:
            errMsg = "invalid regular expression '%s' ('%s')" % (conf.regexp, getSafeExString(ex))
            raise SqlmapSyntaxException(errMsg)

    if conf.crawlExclude:
        try:
            re.compile(conf.crawlExclude)
-        except Exception, ex:
+        except Exception as ex:
            errMsg = "invalid regular expression '%s' ('%s')" % (conf.crawlExclude, getSafeExString(ex))
            raise SqlmapSyntaxException(errMsg)

@@ -2394,9 +2550,15 @@ def _basicOptionValidation():
        raise SqlmapSyntaxException(errMsg)

    if conf.skip and conf.testParameter:
+        if intersect(conf.skip, conf.testParameter):
            errMsg = "option '--skip' is incompatible with option '-p'"
            raise SqlmapSyntaxException(errMsg)

+    if conf.rParam and conf.testParameter:
+        if intersect(conf.rParam, conf.testParameter):
+            errMsg = "option '--randomize' is incompatible with option '-p'"
+            raise SqlmapSyntaxException(errMsg)
+
    if conf.mobile and conf.agent:
        errMsg = "switch '--mobile' is incompatible with option '--user-agent'"
        raise SqlmapSyntaxException(errMsg)
@@ -2413,7 +2575,11 @@ def _basicOptionValidation():
        errMsg = "value for option '--union-char' must be an alpha-numeric value (e.g. 1)"
        raise SqlmapSyntaxException(errMsg)

-    if isinstance(conf.uCols, basestring):
+    if conf.hashFile and any((conf.direct, conf.url, conf.logFile, conf.bulkFile, conf.googleDork, conf.configFile, conf.requestFile, conf.updateAll, conf.smokeTest, conf.liveTest, conf.wizard, conf.dependencies, conf.purge, conf.sitemapUrl, conf.listTampers)):
+        errMsg = "option '--crack' should be used as a standalone"
+        raise SqlmapSyntaxException(errMsg)
+
+    if isinstance(conf.uCols, six.string_types):
        if not conf.uCols.isdigit() and ("-" not in conf.uCols or len(conf.uCols.split("-")) != 2):
            errMsg = "value for option '--union-cols' must be a range with hyphon "
            errMsg += "(e.g. 1-10) or integer value (e.g. 5)"
@@ -2467,6 +2633,7 @@ def init():
    _cleanupEnvironment()
    _purge()
    _checkDependencies()
+    _createHomeDirectories()
    _createTemporaryDirectory()
    _basicOptionValidation()
    _setProxyList()
@@ -2476,10 +2643,12 @@ def init():
    _setMultipleTargets()
    _listTamperingFunctions()
    _setTamperingFunctions()
+    _setPreprocessFunctions()
    _setWafFunctions()
    _setTrafficOutputFP()
    _setupHTTPCollector()
    _resolveCrossReferences()
+    _setHttpChunked()
    _checkWebSocket()

    parseTargetDirect()
--- a/lib/core/optiondict.py
+++ b/lib/core/optiondict.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -61,6 +61,7 @@ optDict = {
        "csrfToken": "string",
        "csrfUrl": "string",
        "forceSSL": "boolean",
+        "chunked": "boolean",
        "hpp": "boolean",
        "evalCode": "string",
    },
@@ -192,7 +193,6 @@ optDict = {
    },

    "General": {
-        # "xmlFile": "string",
        "trafficFile": "string",
        "batch": "boolean",
        "binaryFields": "string",
@@ -211,6 +211,8 @@ optDict = {
        "hexConvert": "boolean",
        "outputDir": "string",
        "parseErrors": "boolean",
+        "preprocess": "string",
+        "repair": "boolean",
        "saveConfig": "string",
        "scope": "string",
        "testFilter": "string",
--- a/lib/core/patch.py
+++ b/lib/core/patch.py
@@ -1,14 +1,14 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

 import codecs
-import httplib

 from lib.core.settings import IS_WIN
+from thirdparty.six.moves import http_client as _http_client

 def dirtyPatches():
    """
@@ -16,7 +16,7 @@ def dirtyPatches():
    """

    # accept overly long result lines (e.g. SQLi results in HTTP header responses)
-    httplib._MAXLINE = 1 * 1024 * 1024
+    _http_client._MAXLINE = 1 * 1024 * 1024

    # add support for inet_pton() on Windows OS
    if IS_WIN:
@@ -24,3 +24,11 @@ def dirtyPatches():

    # Reference: https://github.com/nodejs/node/issues/12786#issuecomment-298652440
    codecs.register(lambda name: codecs.lookup("utf-8") if name == "cp65001" else None)
+
+    # Reference: http://bugs.python.org/issue17849
+    if hasattr(_http_client, "LineAndFileWrapper"):
+        def _(self, *args):
+            return self._readline()
+
+        _http_client.LineAndFileWrapper._readline = _http_client.LineAndFileWrapper.readline
+        _http_client.LineAndFileWrapper.readline = _
--- a/lib/core/profiling.py
+++ b/lib/core/profiling.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -9,7 +9,7 @@ import codecs
 import os
 import cProfile

-from lib.core.common import getUnicode
+from lib.core.common import getSafeExString
 from lib.core.data import logger
 from lib.core.data import paths
 from lib.core.settings import UNICODE_ENCODING
@@ -25,8 +25,8 @@ def profile(profileOutputFile=None, dotOutputFile=None, imageOutputFile=None):
        from thirdparty.xdot import xdot
        import gtk
        import pydot
-    except ImportError, e:
-        errMsg = "profiling requires third-party libraries ('%s') " % getUnicode(e, UNICODE_ENCODING)
+    except ImportError as ex:
+        errMsg = "profiling requires third-party libraries ('%s') " % getSafeExString(ex)
        errMsg += "(Hint: 'sudo apt-get install python-pydot python-pyparsing python-profiler graphviz')"
        logger.error(errMsg)

--- a/lib/core/readlineng.py
+++ b/lib/core/readlineng.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -56,9 +56,7 @@ if PLATFORM == 'mac' and _readline:
 # http://mail.python.org/pipermail/python-dev/2003-August/037845.html
 # has the original discussion.
 if _readline:
-    try:
-        _readline.clear_history()
-    except AttributeError:
+    if not hasattr(_readline, "clear_history"):
        def clear_history():
            pass

--- a/lib/core/replication.py
+++ b/lib/core/replication.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -27,7 +27,7 @@ class Replication(object):
            self.connection = sqlite3.connect(dbpath)
            self.connection.isolation_level = None
            self.cursor = self.connection.cursor()
-        except sqlite3.OperationalError, ex:
+        except sqlite3.OperationalError as ex:
            errMsg = "error occurred while opening a replication "
            errMsg += "file '%s' ('%s')" % (self.filepath, getSafeExString(ex))
            raise SqlmapConnectionException(errMsg)
@@ -63,7 +63,7 @@ class Replication(object):
                        self.execute('CREATE TABLE "%s" (%s)' % (self.name, ','.join('"%s" %s' % (unsafeSQLIdentificatorNaming(colname), coltype) for colname, coltype in self.columns)))
                    else:
                        self.execute('CREATE TABLE "%s" (%s)' % (self.name, ','.join('"%s"' % unsafeSQLIdentificatorNaming(colname) for colname in self.columns)))
-                except Exception, ex:
+                except Exception as ex:
                    errMsg = "problem occurred ('%s') while initializing the sqlite database " % getSafeExString(ex, UNICODE_ENCODING)
                    errMsg += "located at '%s'" % self.parent.dbpath
                    raise SqlmapGenericException(errMsg)
@@ -82,7 +82,7 @@ class Replication(object):
        def execute(self, sql, parameters=[]):
            try:
                self.parent.cursor.execute(sql, parameters)
-            except sqlite3.OperationalError, ex:
+            except sqlite3.OperationalError as ex:
                errMsg = "problem occurred ('%s') while accessing sqlite database " % getSafeExString(ex, UNICODE_ENCODING)
                errMsg += "located at '%s'. Please make sure that " % self.parent.dbpath
                errMsg += "it's not used by some other program"
--- a/lib/core/revision.py
+++ b/lib/core/revision.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -44,9 +44,12 @@ def getRevisionNumber():
            break

    if not retVal:
+        try:
            process = subprocess.Popen("git rev-parse --verify HEAD", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            stdout, _ = process.communicate()
            match = re.search(r"(?i)[0-9a-f]{32}", stdout or "")
            retVal = match.group(0) if match else None
+        except:
+            pass

    return retVal[:7] if retVal else None
--- a/lib/core/session.py
+++ b/lib/core/session.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/lib/core/settings.py
+++ b/lib/core/settings.py
@@ -1,25 +1,23 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

+import codecs
 import os
 import random
 import re
-import subprocess
 import string
 import sys
-import types

-from lib.core.datatype import AttribDict
 from lib.core.enums import DBMS
 from lib.core.enums import DBMS_DIRECTORY_NAME
 from lib.core.enums import OS

 # sqlmap version (<major>.<minor>.<month>.<monthly commit>)
-VERSION = "1.2.12.0"
+VERSION = "1.3.4.0"
 TYPE = "dev" if VERSION.count('.') > 2 and VERSION.split('.')[-1] != '0' else "stable"
 TYPE_COLORS = {"dev": 33, "stable": 90, "pip": 34}
 VERSION_STRING = "sqlmap/%s#%s" % ('.'.join(VERSION.split('.')[:-1]) if VERSION.count('.') > 2 and VERSION.split('.')[-1] == '0' else VERSION, TYPE)
@@ -39,7 +37,7 @@ BANNER = """\033[01;33m\
 ___ ___[.]_____ ___ ___  \033[01;37m{\033[01;%dm%s\033[01;37m}\033[01;33m
 |_ -| . [.]     | .'| . |
 |___|_  [.]_|_|_|__,|  _|
-      |_|V          |_|   \033[0m\033[4;37m%s\033[0m\n
+      |_|V...       |_|   \033[0m\033[4;37m%s\033[0m\n
 """ % (TYPE_COLORS.get(TYPE, 31), VERSION_STRING.split('/')[-1], SITE)

 # Minimum distance of ratio from kb.matchRatio to result in True
@@ -100,7 +98,7 @@ MAX_CONSECUTIVE_CONNECTION_ERRORS = 15
 PRECONNECT_CANDIDATE_TIMEOUT = 10

 # Servers known to cause issue with pre-connection mechanism (because of lack of multi-threaded support)
-PRECONNECT_INCOMPATIBLE_SERVERS = ("SimpleHTTP",)
+PRECONNECT_INCOMPATIBLE_SERVERS = ("SimpleHTTP", "BaseHTTP")

 # Maximum sleep time in "Murphy" (testing) mode
 MAX_MURPHY_SLEEP_TIME = 3
@@ -109,7 +107,7 @@ MAX_MURPHY_SLEEP_TIME = 3
 GOOGLE_REGEX = r"webcache\.googleusercontent\.com/search\?q=cache:[^:]+:([^+]+)\+&amp;cd=|url\?\w+=((?![^>]+webcache\.googleusercontent\.com)http[^>]+)&(sa=U|rct=j)"

 # Regular expression used for extracting results from DuckDuckGo search
-DUCKDUCKGO_REGEX = r'"u":"([^"]+)'
+DUCKDUCKGO_REGEX = r'<a class="result__url" href="(htt[^"]+)'

 # Regular expression used for extracting results from Bing search
 BING_REGEX = r'<h2><a href="([^"]+)" h='
@@ -166,6 +164,9 @@ MAX_TECHNIQUES_PER_VALUE = 2
 # In case of missing piece of partial union dump, buffered array must be flushed after certain size
 MAX_BUFFERED_PARTIAL_UNION_LENGTH = 1024

+# Maximum size of cache used in @cachedmethod decorator
+MAX_CACHE_ITEMS = 256
+
 # Suffix used for naming meta databases in DBMS(es) without explicit database name
 METADB_SUFFIX = "_masterdb"

@@ -182,7 +183,7 @@ MAX_TIME_RESPONSES = 200
 MIN_UNION_RESPONSES = 5

 # After these number of blanks at the end inference should stop (just in case)
-INFERENCE_BLANK_BREAK = 10
+INFERENCE_BLANK_BREAK = 5

 # Use this replacement character for cases when inference is not able to retrieve the proper character value
 INFERENCE_UNKNOWN_CHAR = '?'
@@ -215,23 +216,24 @@ DUMMY_USER_PREFIX = "__dummy__"
 DEFAULT_PAGE_ENCODING = "iso-8859-1"

 try:
-    unicode(DEFAULT_PAGE_ENCODING, DEFAULT_PAGE_ENCODING)
+    codecs.lookup(DEFAULT_PAGE_ENCODING)
 except LookupError:
    DEFAULT_PAGE_ENCODING = "utf8"

+# Marker for program piped input
+STDIN_PIPE_DASH = '-'
+
 # URL used in dummy runs
 DUMMY_URL = "http://foo/bar?id=1"

-# System variables
-IS_WIN = subprocess.mswindows
-
 # The name of the operating system dependent module imported. The following names have currently been registered: 'posix', 'nt', 'mac', 'os2', 'ce', 'java', 'riscos'
 PLATFORM = os.name
 PYVERSION = sys.version.split()[0]
+IS_WIN = PLATFORM == "nt"

 # DBMS system databases
 MSSQL_SYSTEM_DBS = ("Northwind", "master", "model", "msdb", "pubs", "tempdb")
-MYSQL_SYSTEM_DBS = ("information_schema", "mysql", "performance_schema")
+MYSQL_SYSTEM_DBS = ("information_schema", "mysql", "performance_schema", "sys")
 PGSQL_SYSTEM_DBS = ("information_schema", "pg_catalog", "pg_toast", "pgagent")
 ORACLE_SYSTEM_DBS = ('ANONYMOUS', 'APEX_030200', 'APEX_PUBLIC_USER', 'APPQOSSYS', 'BI', 'CTXSYS', 'DBSNMP', 'DIP', 'EXFSYS', 'FLOWS_%', 'FLOWS_FILES', 'HR', 'IX', 'LBACSYS', 'MDDATA', 'MDSYS', 'MGMT_VIEW', 'OC', 'OE', 'OLAPSYS', 'ORACLE_OCM', 'ORDDATA', 'ORDPLUGINS', 'ORDSYS', 'OUTLN', 'OWBSYS', 'PM', 'SCOTT', 'SH', 'SI_INFORMTN_SCHEMA', 'SPATIAL_CSW_ADMIN_USR', 'SPATIAL_WFS_ADMIN_USR', 'SYS', 'SYSMAN', 'SYSTEM', 'WKPROXY', 'WKSYS', 'WK_TEST', 'WMSYS', 'XDB', 'XS$NULL')
 SQLITE_SYSTEM_DBS = ("sqlite_master", "sqlite_temp_master")
@@ -327,18 +329,19 @@ CURRENT_DB = "CD"
 SESSION_SQLITE_FILE = "session.sqlite"

 # Regular expressions used for finding file paths in error messages
-FILE_PATH_REGEXES = (r"<b>(?P<result>[^<>]+?)</b> on line \d+", r"in (?P<result>[^<>'\"]+?)['\"]? on line \d+", r"(?:[>(\[\s])(?P<result>[A-Za-z]:[\\/][\w. \\/-]*)", r"(?:[>(\[\s])(?P<result>/\w[/\w.~-]+)", r"href=['\"]file://(?P<result>/[^'\"]+)")
+FILE_PATH_REGEXES = (r"<b>(?P<result>[^<>]+?)</b> on line \d+", r"\bin (?P<result>[^<>'\"]+?)['\"]? on line \d+", r"(?:[>(\[\s])(?P<result>[A-Za-z]:[\\/][\w. \\/-]*)", r"(?:[>(\[\s])(?P<result>/\w[/\w.~-]+)", r"\bhref=['\"]file://(?P<result>/[^'\"]+)", r"\bin <b>(?P<result>[^<]+): line \d+")

 # Regular expressions used for parsing error messages (--parse-errors)
 ERROR_PARSING_REGEXES = (
    r"\[Microsoft\]\[ODBC SQL Server Driver\]\[SQL Server\](?P<result>[^<]+)",
-    r"<b>[^<]*(fatal|error|warning|exception)[^<]*</b>:?\s*(?P<result>.+?)<br\s*/?\s*>",
+    r"<b>[^<]*(fatal|error|warning|exception)[^<]*</b>:?\s*(?P<result>[^<]+)",
    r"(?m)^\s*(fatal|error|warning|exception):?\s*(?P<result>[^\n]+?)$",
    r"(?P<result>[^\n>]*SQL Syntax[^\n<]+)",
-    r"<li>Error Type:<br>(?P<result>.+?)</li>",
+    r"(?s)<li>Error Type:<br>(?P<result>.+?)</li>",
    r"CDbCommand (?P<result>[^<>\n]*SQL[^<>\n]+)",
    r"error '[0-9a-f]{8}'((<[^>]+>)|\s)+(?P<result>[^<>]+)",
-    r"\[[^\n\]]+(ODBC|JDBC)[^\n\]]+\](\[[^\]]+\])?(?P<result>[^\n]+(in query expression|\(SQL| at /[^ ]+pdo)[^\n<]+)"
+    r"\[[^\n\]]+(ODBC|JDBC)[^\n\]]+\](\[[^\]]+\])?(?P<result>[^\n]+(in query expression|\(SQL| at /[^ ]+pdo)[^\n<]+)",
+    r"(?P<result>query error: SELECT[^<>]+)"
 )

 # Regular expression used for parsing charset info from meta html headers
@@ -360,7 +363,7 @@ COMMON_PASSWORD_SUFFIXES += ("!", ".", "*", "!!", "?", ";", "..", "!!!", ", ", "
 WEBSCARAB_SPLITTER = "### Conversation"

 # Splitter used between requests in BURP log files
-BURP_REQUEST_REGEX = r"={10,}\s+[^=]+={10,}\s(.+?)\s={10,}"
+BURP_REQUEST_REGEX = r"={10,}\s+([A-Z]{3,} .+?)\s+={10,}"

 # Regex used for parsing XML Burp saved history items
 BURP_XML_HISTORY_REGEX = r'<port>(\d+)</port>.+?<request base64="true"><!\[CDATA\[([^]]+)'
@@ -375,7 +378,7 @@ URI_HTTP_HEADER = "URI"
 URI_INJECTABLE_REGEX = r"//[^/]*/([^\.*?]+)\Z"

 # Regex used for masking sensitive data
-SENSITIVE_DATA_REGEX = r"(\s|=)(?P<result>[^\s=]*%s[^\s]*)\s"
+SENSITIVE_DATA_REGEX = r"(\s|=)(?P<result>[^\s=]*\b%s\b[^\s]*)\s"

 # Options to explicitly mask in anonymous (unhandled exception) reports (along with anything carrying the <hostname> inside)
 SENSITIVE_OPTIONS = ("hostname", "answers", "data", "dnsDomain", "googleDork", "authCred", "proxyCred", "tbl", "db", "col", "user", "cookie", "proxy", "fileRead", "fileWrite", "fileDest", "testParameter", "authCred")
@@ -441,7 +444,7 @@ HASH_MOD_ITEM_DISPLAY = 11
 HASH_EMPTY_PASSWORD_MARKER = "<empty>"

 # Maximum integer value
-MAX_INT = sys.maxint
+MAX_INT = sys.maxsize

 # Replacement for unsafe characters in dump table filenames
 UNSAFE_DUMP_FILEPATH_REPLACEMENT = '_'
@@ -506,8 +509,6 @@ HTML_TITLE_REGEX = r"<title>(?P<result>[^<]+)</title>"
 # Table used for Base64 conversion in WordPress hash cracking routine
 ITOA64 = "./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"

-PICKLE_REDUCE_WHITELIST = (types.BooleanType, types.DictType, types.FloatType, types.IntType, types.ListType, types.LongType, types.NoneType, types.StringType, types.TupleType, types.UnicodeType, types.XRangeType, type(AttribDict()), type(set()))
-
 # Chars used to quickly distinguish if the user provided tainted parameter values
 DUMMY_SQL_INJECTION_CHARS = ";()'"

@@ -526,9 +527,6 @@ BRUTE_TABLE_EXISTS_TEMPLATE = "EXISTS(SELECT %d FROM %s)"
 # Template used for common column existence check
 BRUTE_COLUMN_EXISTS_TEMPLATE = "EXISTS(SELECT %s FROM %s)"

-# Payload used for checking of existence of IDS/IPS/WAF (dummier the better)
-IDS_WAF_CHECK_PAYLOAD = "AND 1=1 UNION ALL SELECT 1,NULL,'<script>alert(\"XSS\")</script>',table_name FROM information_schema.tables WHERE 2>1--/**/; EXEC xp_cmdshell('cat ../../../etc/passwd')#"
-
 # Data inside shellcodeexec to be filled with random string
 SHELLCODEEXEC_RANDOM_STRING_MARKER = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

@@ -538,13 +536,16 @@ CHECK_INTERNET_ADDRESS = "https://ipinfo.io/"
 # Value to look for in response to CHECK_INTERNET_ADDRESS
 CHECK_INTERNET_VALUE = "IP Address Details"

+# Payload used for checking of existence of WAF/IPS (dummier the better)
+IPS_WAF_CHECK_PAYLOAD = "AND 1=1 UNION ALL SELECT 1,NULL,'<script>alert(\"XSS\")</script>',table_name FROM information_schema.tables WHERE 2>1--/**/; EXEC xp_cmdshell('cat ../../../etc/passwd')#"
+
 # Vectors used for provoking specific WAF/IPS behavior(s)
 WAF_ATTACK_VECTORS = (
    "",  # NIL
    "search=<script>alert(1)</script>",
    "file=../../../../etc/passwd",
    "q=<invalid>foobar",
-    "id=1 %s" % IDS_WAF_CHECK_PAYLOAD
+    "id=1 %s" % IPS_WAF_CHECK_PAYLOAD
 )

 # Used for status representation in dictionary attack phase
@@ -585,7 +586,7 @@ UNION_CHAR_REGEX = r"\A\w+\Z"
 UNENCODED_ORIGINAL_VALUE = "original"

 # Common column names containing usernames (used for hash cracking in some cases)
-COMMON_USER_COLUMNS = ("login", "user", "username", "user_name", "user_login", "benutzername", "benutzer", "utilisateur", "usager", "consommateur", "utente", "utilizzatore", "usufrutuario", "korisnik", "usuario", "consumidor", "client", "cuser")
+COMMON_USER_COLUMNS = ("login", "user", "username", "user_name", "user_login", "benutzername", "benutzer", "utilisateur", "usager", "consommateur", "utente", "utilizzatore", "utilizator", "utilizador", "usufrutuario", "korisnik", "uporabnik", "usuario", "consumidor", "client", "cuser")

 # Default delimiter in GET/POST values
 DEFAULT_GET_POST_DELIMITER = '&'
@@ -675,7 +676,10 @@ VALID_TIME_CHARS_RUN_THRESHOLD = 100
 CHECK_ZERO_COLUMNS_THRESHOLD = 10

 # Boldify all logger messages containing these "patterns"
-BOLD_PATTERNS = ("' injectable", "provided empty", "leftover chars", "might be injectable", "' is vulnerable", "is not injectable", "does not seem to be", "test failed", "test passed", "live test final result", "test shows that", "the back-end DBMS is", "created Github", "blocked by the target server", "protection is involved", "CAPTCHA", "specific response", "NULL connection is supported")
+BOLD_PATTERNS = ("' injectable", "provided empty", "leftover chars", "might be injectable", "' is vulnerable", "is not injectable", "does not seem to be", "test failed", "test passed", "live test final result", "test shows that", "the back-end DBMS is", "created Github", "blocked by the target server", "protection is involved", "CAPTCHA", "specific response", "NULL connection is supported", "PASSED", "FAILED")
+
+# TLDs used in randomization of email-alike parameter values
+RANDOMIZATION_TLDS = ("com", "net", "ru", "org", "de", "jp", "cn", "fr", "it", "pl", "tv", "edu", "in", "ir", "es", "me", "info", "gr", "gov", "ca", "co", "se", "cz", "to", "vn", "nl", "cc", "az", "hu", "ua", "be", "no", "biz", "io", "ch", "ro", "sk", "eu", "us", "tw", "pt", "fi", "at", "lt", "kz", "cl", "hr", "pk", "lv", "la", "pe")

 # Generic www root directory names
 GENERIC_DOC_ROOT_DIRECTORY_NAMES = ("htdocs", "httpdocs", "public", "wwwroot", "www")
@@ -687,7 +691,7 @@ MAX_HELP_OPTION_LENGTH = 18
 MAX_CONNECT_RETRIES = 100

 # Strings for detecting formatting errors
-FORMAT_EXCEPTION_STRINGS = ("Type mismatch", "Error converting", "Conversion failed", "String or binary data would be truncated", "Failed to convert", "unable to interpret text value", "Input string was not in a correct format", "System.FormatException", "java.lang.NumberFormatException", "ValueError: invalid literal", "TypeMismatchException", "CF_SQL_INTEGER", " for CFSQLTYPE ", "cfqueryparam cfsqltype", "InvalidParamTypeException", "Invalid parameter type", "is not of type numeric", "<cfif Not IsNumeric(", "invalid input syntax for integer", "invalid input syntax for type", "invalid number", "character to number conversion error", "unable to interpret text value", "String was not recognized as a valid", "Convert.ToInt", "cannot be converted to a ", "InvalidDataException")
+FORMAT_EXCEPTION_STRINGS = ("Type mismatch", "Error converting", "Please enter a", "Conversion failed", "String or binary data would be truncated", "Failed to convert", "unable to interpret text value", "Input string was not in a correct format", "System.FormatException", "java.lang.NumberFormatException", "ValueError: invalid literal", "TypeMismatchException", "CF_SQL_INTEGER", "CF_SQL_NUMERIC", " for CFSQLTYPE ", "cfqueryparam cfsqltype", "InvalidParamTypeException", "Invalid parameter type", "Attribute validation error for tag", "is not of type numeric", "<cfif Not IsNumeric(", "invalid input syntax for integer", "invalid input syntax for type", "invalid number", "character to number conversion error", "unable to interpret text value", "String was not recognized as a valid", "Convert.ToInt", "cannot be converted to a ", "InvalidDataException")

 # Regular expression used for extracting ASP.NET view state values
 VIEWSTATE_REGEX = r'(?i)(?P<name>__VIEWSTATE[^"]*)[^>]+value="(?P<result>[^"]+)'
@@ -755,8 +759,8 @@ METASPLOIT_SESSION_TIMEOUT = 120
 # Reference: http://www.postgresql.org/docs/9.0/static/catalog-pg-largeobject.html
 LOBLKSIZE = 2048

-# Suffix used to mark variables having keyword names
-EVALCODE_KEYWORD_SUFFIX = "_KEYWORD"
+# Prefix used to mark special variables (e.g. keywords, having special chars, etc.)
+EVALCODE_ENCODED_PREFIX = "EVAL_"

 # Reference: http://www.cookiecentral.com/faq/#3.5
 NETSCAPE_FORMAT_HEADER_COOKIES = "# Netscape HTTP Cookie File."
@@ -770,6 +774,9 @@ BRUTE_DOC_ROOT_PREFIXES = {
    OS.WINDOWS: ("/xampp", "/Program Files/xampp", "/wamp", "/Program Files/wampp", "/apache", "/Program Files/Apache Group/Apache", "/Program Files/Apache Group/Apache2", "/Program Files/Apache Group/Apache2.2", "/Program Files/Apache Group/Apache2.4", "/Inetpub/wwwroot", "/Inetpub/wwwroot/%TARGET%", "/Inetpub/vhosts/%TARGET%")
 }

+# Table prefix to use in "takeover" functionalities (i.e. auxiliary tables used by sqlmap at the vulnerable DBMS)
+TAKEOVER_TABLE_PREFIX = "sqlmap"
+
 # Suffixes used in brute force search for web server document root
 BRUTE_DOC_ROOT_SUFFIXES = ("", "html", "htdocs", "httpdocs", "php", "public", "src", "site", "build", "web", "www", "data", "sites/all", "www/build")

@@ -782,6 +789,9 @@ KB_CHARS_BOUNDARY_CHAR = 'q'
 # Letters of lower frequency used in kb.chars
 KB_CHARS_LOW_FREQUENCY_ALPHABET = "zqxjkvbp"

+# SQL keywords used for splitting in HTTP chunked transfer encoded requests (switch --chunk)
+HTTP_CHUNKED_SPLIT_KEYWORDS = ("SELECT", "UPDATE", "INSERT", "FROM", "LOAD_FILE", "UNION", "information_schema", "sysdatabases", "msysaccessobjects", "msysqueries", "sysmodules")
+
 # CSS style used in HTML dump format
 HTML_DUMP_CSS_STYLE = """<style>
 table{
@@ -806,3 +816,11 @@ th{
    font-size:12px;
 }
 </style>"""
+
+# Leaving (dirty) possibility to change values from here (e.g. `export SQLMAP__MAX_NUMBER_OF_THREADS=20`)
+
+for key, value in os.environ.items():
+    if key.upper().startswith("%s_" % SQLMAP_ENVIRONMENT_PREFIX):
+        _ = key[len(SQLMAP_ENVIRONMENT_PREFIX) + 1:].upper()
+        if _ in globals():
+            globals()[_] = value
--- a/lib/core/shell.py
+++ b/lib/core/shell.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -9,6 +9,7 @@ import atexit
 import os

 from lib.core import readlineng as readline
+from lib.core.common import getSafeExString
 from lib.core.data import logger
 from lib.core.data import paths
 from lib.core.enums import AUTOCOMPLETE_TYPE
@@ -75,8 +76,8 @@ def saveHistory(completion=None):
        readline.set_history_length(MAX_HISTORY_LENGTH)
        try:
            readline.write_history_file(historyPath)
-        except IOError, msg:
-            warnMsg = "there was a problem writing the history file '%s' (%s)" % (historyPath, msg)
+        except IOError as ex:
+            warnMsg = "there was a problem writing the history file '%s' (%s)" % (historyPath, getSafeExString(ex))
            logger.warn(warnMsg)
    except KeyboardInterrupt:
        pass
@@ -99,8 +100,8 @@ def loadHistory(completion=None):
    if os.path.exists(historyPath):
        try:
            readline.read_history_file(historyPath)
-        except IOError, msg:
-            warnMsg = "there was a problem loading the history file '%s' (%s)" % (historyPath, msg)
+        except IOError as ex:
+            warnMsg = "there was a problem loading the history file '%s' (%s)" % (historyPath, getSafeExString(ex))
            logger.warn(warnMsg)

 def autoCompletion(completion=None, os=None, commands=None):
--- a/lib/core/subprocessng.py
+++ b/lib/core/subprocessng.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -31,7 +31,7 @@ def blockingReadFromFD(fd):
    while True:
        try:
            output += os.read(fd, 8192)
-        except (OSError, IOError), ioe:
+        except (OSError, IOError) as ioe:
            if ioe.args[0] in (errno.EAGAIN, errno.EINTR):
                # Uncomment the following line if the process seems to
                # take a huge amount of cpu time
@@ -52,7 +52,7 @@ def blockingWriteToFD(fd, data):
        try:
            data_length = len(data)
            wrote_data = os.write(fd, data)
-        except (OSError, IOError), io:
+        except (OSError, IOError) as io:
            if io.errno in (errno.EAGAIN, errno.EINTR):
                continue
            else:
@@ -85,7 +85,7 @@ class Popen(subprocess.Popen):
        getattr(self, which).close()
        setattr(self, which, None)

-    if subprocess.mswindows:
+    if IS_WIN:
        def send(self, input):
            if not self.stdin:
                return None
@@ -95,8 +95,8 @@ class Popen(subprocess.Popen):
                (errCode, written) = WriteFile(x, input)
            except ValueError:
                return self._close('stdin')
-            except (subprocess.pywintypes.error, Exception), why:
-                if why[0] in (109, errno.ESHUTDOWN):
+            except (subprocess.pywintypes.error, Exception) as ex:
+                if ex[0] in (109, errno.ESHUTDOWN):
                    return self._close('stdin')
                raise

@@ -116,8 +116,8 @@ class Popen(subprocess.Popen):
                    (errCode, read) = ReadFile(x, nAvail, None)
            except (ValueError, NameError):
                return self._close(which)
-            except (subprocess.pywintypes.error, Exception), why:
-                if why[0] in (109, errno.ESHUTDOWN):
+            except (subprocess.pywintypes.error, Exception) as ex:
+                if ex[0] in (109, errno.ESHUTDOWN):
                    return self._close(which)
                raise

@@ -134,8 +134,8 @@ class Popen(subprocess.Popen):

            try:
                written = os.write(self.stdin.fileno(), input)
-            except OSError, why:
-                if why[0] == errno.EPIPE:  # broken pipe
+            except OSError as ex:
+                if ex[0] == errno.EPIPE:  # broken pipe
                    return self._close('stdin')
                raise

--- a/lib/core/target.py
+++ b/lib/core/target.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -12,7 +12,6 @@ import subprocess
 import sys
 import tempfile
 import time
-import urlparse

 from lib.core.common import Backend
 from lib.core.common import getSafeExString
@@ -27,6 +26,7 @@ from lib.core.common import randomStr
 from lib.core.common import readInput
 from lib.core.common import resetCookieJar
 from lib.core.common import urldecode
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
@@ -73,7 +73,8 @@ from lib.core.settings import URI_INJECTABLE_REGEX
 from lib.core.settings import USER_AGENT_ALIASES
 from lib.core.settings import XML_RECOGNITION_REGEX
 from lib.utils.hashdb import HashDB
-from thirdparty.odict.odict import OrderedDict
+from thirdparty.odict import OrderedDict
+from thirdparty.six.moves import urllib as _urllib

 def _setRequestParams():
    """
@@ -212,7 +213,7 @@ def _setRequestParams():
                if not (kb.processUserMarks and kb.customInjectionMark in conf.data):
                    conf.data = getattr(conf.data, UNENCODED_ORIGINAL_VALUE, conf.data)
                    conf.data = conf.data.replace(kb.customInjectionMark, ASTERISK_MARKER)
-                    conf.data = re.sub(r"(?si)((Content-Disposition[^\n]+?name\s*=\s*[\"']?(?P<name>[^\"'\r\n]+)[\"']?).+?)(((\r)?\n)+--)", functools.partial(process, repl=r"\g<1>%s\g<4>" % kb.customInjectionMark), conf.data)
+                    conf.data = re.sub(r"(?si)((Content-Disposition[^\n]+?name\s*=\s*[\"']?(?P<name>[^\"'\r\n]+)[\"']?).+?)((%s)+--)" % ("\r\n" if "\r\n" in conf.data else '\n'), functools.partial(process, repl=r"\g<1>%s\g<4>" % kb.customInjectionMark), conf.data)

                kb.postHint = POST_HINT.MULTIPART

@@ -276,7 +277,7 @@ def _setRequestParams():

            if not kb.processUserMarks:
                if place == PLACE.URI:
-                    query = urlparse.urlsplit(value).query
+                    query = _urllib.parse.urlsplit(value).query
                    if query:
                        parameters = conf.parameters[PLACE.GET] = query
                        paramDict = paramToDict(PLACE.GET, parameters)
@@ -393,8 +394,8 @@ def _setRequestParams():
        raise SqlmapGenericException(errMsg)

    if conf.csrfToken:
-        if not any(conf.csrfToken in _ for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}))) and not re.search(r"\b%s\b" % re.escape(conf.csrfToken), conf.data or "") and conf.csrfToken not in set(_[0].lower() for _ in conf.httpHeaders) and conf.csrfToken not in conf.paramDict.get(PLACE.COOKIE, {}):
-            errMsg = "anti-CSRF token parameter '%s' not " % conf.csrfToken
+        if not any(re.search(conf.csrfToken, ' '.join(_), re.I) for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}))) and not re.search(r"\b%s\b" % conf.csrfToken, conf.data or "") and conf.csrfToken not in set(_[0].lower() for _ in conf.httpHeaders) and conf.csrfToken not in conf.paramDict.get(PLACE.COOKIE, {}):
+            errMsg = "anti-CSRF token parameter '%s' not " % conf.csrfToken._original
            errMsg += "found in provided GET, POST, Cookie or header values"
            raise SqlmapGenericException(errMsg)
    else:
@@ -408,7 +409,10 @@ def _setRequestParams():
                    message += "Do you want sqlmap to automatically update it in further requests? [y/N] "

                    if readInput(message, default='N', boolean=True):
-                        conf.csrfToken = getUnicode(parameter)
+                        class _(unicode):
+                            pass
+                        conf.csrfToken = _(re.escape(getUnicode(parameter)))
+                        conf.csrfToken._original = getUnicode(parameter)
                        break

 def _setHashDB():
@@ -424,8 +428,8 @@ def _setHashDB():
            try:
                os.remove(conf.hashDBFile)
                logger.info("flushing session file")
-            except OSError, msg:
-                errMsg = "unable to flush the session file (%s)" % msg
+            except OSError as ex:
+                errMsg = "unable to flush the session file ('%s')" % getSafeExString(ex)
                raise SqlmapFilePathException(errMsg)

    conf.hashDB = HashDB(conf.hashDBFile)
@@ -555,7 +559,7 @@ def _setResultsFile():
        conf.resultsFilename = os.path.join(paths.SQLMAP_OUTPUT_PATH, time.strftime(RESULTS_FILE_FORMAT).lower())
        try:
            conf.resultsFP = openFile(conf.resultsFilename, "a", UNICODE_ENCODING, buffering=0)
-        except (OSError, IOError), ex:
+        except (OSError, IOError) as ex:
            try:
                warnMsg = "unable to create results file '%s' ('%s'). " % (conf.resultsFilename, getUnicode(ex))
                handle, conf.resultsFilename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.RESULTS, suffix=".csv")
@@ -563,7 +567,7 @@ def _setResultsFile():
                conf.resultsFP = openFile(conf.resultsFilename, "w+", UNICODE_ENCODING, buffering=0)
                warnMsg += "Using temporary file '%s' instead" % conf.resultsFilename
                logger.warn(warnMsg)
-            except IOError, _:
+            except IOError as _:
                errMsg = "unable to write to the temporary directory ('%s'). " % _
                errMsg += "Please make sure that your disk is not full and "
                errMsg += "that you have sufficient write permissions to "
@@ -587,7 +591,7 @@ def _createFilesDir():
    if not os.path.isdir(conf.filePath):
        try:
            os.makedirs(conf.filePath)
-        except OSError, ex:
+        except OSError as ex:
            tempDir = tempfile.mkdtemp(prefix="sqlmapfiles")
            warnMsg = "unable to create files directory "
            warnMsg += "'%s' (%s). " % (conf.filePath, getUnicode(ex))
@@ -609,7 +613,7 @@ def _createDumpDir():
    if not os.path.isdir(conf.dumpPath):
        try:
            os.makedirs(conf.dumpPath)
-        except OSError, ex:
+        except OSError as ex:
            tempDir = tempfile.mkdtemp(prefix="sqlmapdump")
            warnMsg = "unable to create dump directory "
            warnMsg += "'%s' (%s). " % (conf.dumpPath, getUnicode(ex))
@@ -627,45 +631,15 @@ def _createTargetDirs():
    Create the output directory.
    """

-    for context in "output", "history":
-        directory = paths["SQLMAP_%s_PATH" % context.upper()]
-        try:
-            if not os.path.isdir(directory):
-                os.makedirs(directory)
-
-            _ = os.path.join(directory, randomStr())
-            open(_, "w+b").close()
-            os.remove(_)
-
-            if conf.outputDir and context == "output":
-                warnMsg = "using '%s' as the %s directory" % (directory, context)
-                logger.warn(warnMsg)
-        except (OSError, IOError), ex:
-            try:
-                tempDir = tempfile.mkdtemp(prefix="sqlmap%s" % context)
-            except Exception, _:
-                errMsg = "unable to write to the temporary directory ('%s'). " % _
-                errMsg += "Please make sure that your disk is not full and "
-                errMsg += "that you have sufficient write permissions to "
-                errMsg += "create temporary files and/or directories"
-                raise SqlmapSystemException(errMsg)
-
-            warnMsg = "unable to %s %s directory " % ("create" if not os.path.isdir(directory) else "write to the", context)
-            warnMsg += "'%s' (%s). " % (directory, getUnicode(ex))
-            warnMsg += "Using temporary directory '%s' instead" % getUnicode(tempDir)
-            logger.warn(warnMsg)
-
-            paths["SQLMAP_%s_PATH" % context.upper()] = tempDir
-
    conf.outputPath = os.path.join(getUnicode(paths.SQLMAP_OUTPUT_PATH), normalizeUnicode(getUnicode(conf.hostname)))

    try:
        if not os.path.isdir(conf.outputPath):
            os.makedirs(conf.outputPath)
-    except (OSError, IOError, TypeError), ex:
+    except (OSError, IOError, TypeError) as ex:
        try:
            tempDir = tempfile.mkdtemp(prefix="sqlmapoutput")
-        except Exception, _:
+        except Exception as _:
            errMsg = "unable to write to the temporary directory ('%s'). " % _
            errMsg += "Please make sure that your disk is not full and "
            errMsg += "that you have sufficient write permissions to "
@@ -688,7 +662,7 @@ def _createTargetDirs():
            f.write("  # %s" % getUnicode(subprocess.list2cmdline(sys.argv), encoding=sys.stdin.encoding))
            if conf.data:
                f.write("\n\n%s" % getUnicode(conf.data))
-    except IOError, ex:
+    except IOError as ex:
        if "denied" in getUnicode(ex):
            errMsg = "you don't have enough permissions "
        else:
--- a/lib/core/testing.py
+++ b/lib/core/testing.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -17,6 +17,7 @@ import traceback

 from extra.beep.beep import beep
 from lib.controller.controller import start
+from lib.core.common import checkIntegrity
 from lib.core.common import clearConsoleLine
 from lib.core.common import dataToStdout
 from lib.core.common import getUnicode
@@ -51,6 +52,9 @@ def smokeTest():
    retVal = True
    count, length = 0, 0

+    if not checkIntegrity():
+        retVal = False
+
    for root, _, files in os.walk(paths.SQLMAP_ROOT_PATH):
        if any(_ in root for _ in ("thirdparty", "extra")):
            continue
@@ -71,10 +75,10 @@ def smokeTest():
                try:
                    __import__(path)
                    module = sys.modules[path]
-                except Exception, msg:
+                except Exception as ex:
                    retVal = False
                    dataToStdout("\r")
-                    errMsg = "smoke test failed at importing module '%s' (%s):\n%s" % (path, os.path.join(root, filename), msg)
+                    errMsg = "smoke test failed at importing module '%s' (%s):\n%s" % (path, os.path.join(root, filename), ex)
                    logger.error(errMsg)
                else:
                    # Run doc tests
@@ -271,10 +275,10 @@ def runCase(parse):
        result = start()
    except KeyboardInterrupt:
        pass
-    except SqlmapBaseException, e:
-        handled_exception = e
-    except Exception, e:
-        unhandled_exception = e
+    except SqlmapBaseException as ex:
+        handled_exception = ex
+    except Exception as ex:
+        unhandled_exception = ex
    finally:
        sys.stdout.seek(0)
        console = sys.stdout.read()
--- a/lib/core/threads.py
+++ b/lib/core/threads.py
@@ -1,21 +1,25 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

+from __future__ import print_function
+
 import difflib
-import random
 import threading
 import time
 import traceback

+from lib.core.compat import WichmannHill
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.datatype import AttribDict
 from lib.core.enums import PAYLOAD
+from lib.core.exception import SqlmapBaseException
 from lib.core.exception import SqlmapConnectionException
 from lib.core.exception import SqlmapThreadException
 from lib.core.exception import SqlmapUserQuitException
@@ -54,7 +58,7 @@ class _ThreadData(threading.local):
        self.lastRequestMsg = None
        self.lastRequestUID = 0
        self.lastRedirectURL = None
-        self.random = random.WichmannHill()
+        self.random = WichmannHill()
        self.resumed = False
        self.retriesCount = 0
        self.seqMatcher = difflib.SequenceMatcher(None)
@@ -91,9 +95,10 @@ def exceptionHandledFunction(threadFunction, silent=False):
        kb.threadContinue = False
        kb.threadException = True
        raise
-    except Exception, ex:
+    except Exception as ex:
        if not silent and kb.get("threadContinue"):
-            logger.error("thread %s: %s" % (threading.currentThread().getName(), ex.message))
+            errMsg = ex.message if isinstance(ex, SqlmapBaseException) else "%s: %s" % (type(ex).__name__, ex.message)
+            logger.error("thread %s: '%s'" % (threading.currentThread().getName(), errMsg))

            if conf.get("verbose") > 1:
                traceback.print_exc()
@@ -108,7 +113,6 @@ def setDaemon(thread):
 def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardException=True, threadChoice=False, startThreadMsg=True):
    threads = []

-    kb.multiThreadMode = True
    kb.threadContinue = True
    kb.threadException = False

@@ -150,7 +154,7 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio

            try:
                thread.start()
-            except Exception, ex:
+            except Exception as ex:
                errMsg = "error occurred while starting new thread ('%s')" % ex.message
                logger.critical(errMsg)
                break
@@ -166,8 +170,8 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio
                    alive = True
                    time.sleep(0.1)

-    except (KeyboardInterrupt, SqlmapUserQuitException), ex:
-        print
+    except (KeyboardInterrupt, SqlmapUserQuitException) as ex:
+        print()
        kb.prependFlag = False
        kb.threadContinue = False
        kb.threadException = True
@@ -184,8 +188,8 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio
        if forwardException:
            raise

-    except (SqlmapConnectionException, SqlmapValueException), ex:
-        print
+    except (SqlmapConnectionException, SqlmapValueException) as ex:
+        print()
        kb.threadException = True
        logger.error("thread %s: %s" % (threading.currentThread().getName(), ex.message))

@@ -195,14 +199,13 @@ def runThreads(numThreads, threadFunction, cleanupFunction=None, forwardExceptio
    except:
        from lib.core.common import unhandledExceptionMessage

-        print
+        print()
        kb.threadException = True
        errMsg = unhandledExceptionMessage()
        logger.error("thread %s: %s" % (threading.currentThread().getName(), errMsg))
        traceback.print_exc()

    finally:
-        kb.multiThreadMode = False
        kb.bruteMode = False
        kb.threadContinue = True
        kb.threadException = False
--- a/lib/core/unescaper.py
+++ b/lib/core/unescaper.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/lib/core/update.py
+++ b/lib/core/update.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -51,7 +51,7 @@ def update():

            try:
                open(os.path.join(directory, "sqlmap.py"), "w+b")
-            except Exception, ex:
+            except Exception as ex:
                errMsg = "unable to update content of directory '%s' ('%s')" % (directory, getSafeExString(ex))
                logger.error(errMsg)
            else:
@@ -85,7 +85,7 @@ def update():
                                version = re.search(r"(?m)^VERSION\s*=\s*['\"]([^'\"]+)", f.read()).group(1)
                                logger.info("updated to the latest version '%s#dev'" % version)
                                success = True
-                    except Exception, ex:
+                    except Exception as ex:
                        logger.error("update could not be completed ('%s')" % getSafeExString(ex))
                    else:
                        if not success:
@@ -110,7 +110,7 @@ def update():
            pollProcess(process, True)
            stdout, stderr = process.communicate()
            success = not process.returncode
-        except (IOError, OSError), ex:
+        except (IOError, OSError) as ex:
            success = False
            stderr = getSafeExString(ex)

--- a/lib/core/wordlist.py
+++ b/lib/core/wordlist.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -43,7 +43,7 @@ class Wordlist(object):
            if os.path.splitext(self.current)[1].lower() == ".zip":
                try:
                    _ = zipfile.ZipFile(self.current, 'r')
-                except zipfile.error, ex:
+                except zipfile.error as ex:
                    errMsg = "something appears to be wrong with "
                    errMsg += "the file '%s' ('%s'). Please make " % (self.current, getSafeExString(ex))
                    errMsg += "sure that you haven't made any changes to it"
@@ -68,15 +68,15 @@ class Wordlist(object):
        while True:
            self.counter += 1
            try:
-                retVal = self.iter.next().rstrip()
-            except zipfile.error, ex:
+                retVal = next(self.iter).rstrip()
+            except zipfile.error as ex:
                errMsg = "something appears to be wrong with "
                errMsg += "the file '%s' ('%s'). Please make " % (self.current, getSafeExString(ex))
                errMsg += "sure that you haven't made any changes to it"
                raise SqlmapInstallationException(errMsg)
            except StopIteration:
                self.adjust()
-                retVal = self.iter.next().rstrip()
+                retVal = next(self.iter).rstrip()
            if not self.proc_count or self.counter % self.proc_count == self.proc_id:
                break
        return retVal
--- a/lib/parse/init.py
+++ b/lib/parse/init.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/lib/parse/banner.py
+++ b/lib/parse/banner.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/lib/parse/cmdline.py
+++ b/lib/parse/cmdline.py
@@ -1,10 +1,12 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

+from __future__ import print_function
+
 import os
 import re
 import shlex
@@ -20,6 +22,7 @@ from lib.core.common import checkSystemEncoding
 from lib.core.common import dataToStdout
 from lib.core.common import expandMnemonics
 from lib.core.common import getUnicode
+from lib.core.compat import xrange
 from lib.core.data import cmdLineOptions
 from lib.core.data import conf
 from lib.core.data import logger
@@ -29,6 +32,7 @@ from lib.core.exception import SqlmapShellQuitException
 from lib.core.exception import SqlmapSyntaxException
 from lib.core.settings import BASIC_HELP_ITEMS
 from lib.core.settings import DUMMY_URL
+from lib.core.settings import INFERENCE_UNKNOWN_CHAR
 from lib.core.settings import IS_WIN
 from lib.core.settings import MAX_HELP_OPTION_LENGTH
 from lib.core.settings import VERSION_STRING
@@ -212,6 +216,9 @@ def cmdLineParser(argv=None):
        request.add_option("--force-ssl", dest="forceSSL", action="store_true",
                           help="Force usage of SSL/HTTPS")

+        request.add_option("--chunked", dest="chunked", action="store_true",
+                           help="Use HTTP chunked transfer encoded (POST) requests")
+
        request.add_option("--hpp", dest="hpp", action="store_true",
                           help="Use HTTP parameter pollution method")

@@ -592,6 +599,12 @@ def cmdLineParser(argv=None):
        general.add_option("--parse-errors", dest="parseErrors", action="store_true",
                           help="Parse and display DBMS error messages from responses")

+        general.add_option("--preprocess", dest="preprocess",
+                             help="Use given script(s) for preprocessing of response data")
+
+        general.add_option("--repair", dest="repair", action="store_true",
+                           help="Redump entries having unknown character marker (%s)" % INFERENCE_UNKNOWN_CHAR)
+
        general.add_option("--save", dest="saveConfig",
                           help="Save options to a configuration INI file")

@@ -668,6 +681,10 @@ def cmdLineParser(argv=None):
                                 help="Simple wizard interface for beginner users")

        # Hidden and/or experimental options
+        parser.add_option("--crack", dest="hashFile",
+                          help=SUPPRESS_HELP)
+#                          help="Load and crack hashes from a file (standalone)")
+
        parser.add_option("--dummy", dest="dummy", action="store_true",
                          help=SUPPRESS_HELP)

@@ -788,7 +805,7 @@ def cmdLineParser(argv=None):
                    command = raw_input("sqlmap-shell> ").strip()
                    command = getUnicode(command, encoding=sys.stdin.encoding)
                except (KeyboardInterrupt, EOFError):
-                    print
+                    print()
                    raise SqlmapShellQuitException

                if not command:
@@ -810,7 +827,7 @@ def cmdLineParser(argv=None):
            try:
                for arg in shlex.split(command):
                    argv.append(getUnicode(arg, encoding=sys.stdin.encoding))
-            except ValueError, ex:
+            except ValueError as ex:
                raise SqlmapSyntaxException("something went wrong during command line parsing ('%s')" % ex.message)

        for i in xrange(len(argv)):
@@ -838,7 +855,7 @@ def cmdLineParser(argv=None):
                argv[i] = argv[i][:-1]
                conf.skipThreadCheck = True
            elif argv[i] == "--version":
-                print VERSION_STRING.split('/')[-1]
+                print(VERSION_STRING.split('/')[-1])
                raise SystemExit
            elif argv[i] in ("-h", "--help"):
                advancedHelp = False
@@ -862,7 +879,7 @@ def cmdLineParser(argv=None):

        try:
            (args, _) = parser.parse_args(argv)
-        except UnicodeEncodeError, ex:
+        except UnicodeEncodeError as ex:
            dataToStdout("\n[!] %s\n" % ex.object.encode("unicode-escape"))
            raise SystemExit
        except SystemExit:
@@ -884,15 +901,15 @@ def cmdLineParser(argv=None):
        if args.dummy:
            args.url = args.url or DUMMY_URL

-        if not any((args.direct, args.url, args.logFile, args.bulkFile, args.googleDork, args.configFile, args.requestFile, args.updateAll, args.smokeTest, args.liveTest, args.wizard, args.dependencies, args.purge, args.sitemapUrl, args.listTampers)):
+        if not any((args.direct, args.url, args.logFile, args.bulkFile, args.googleDork, args.configFile, args.requestFile, args.updateAll, args.smokeTest, args.liveTest, args.wizard, args.dependencies, args.purge, args.sitemapUrl, args.listTampers, args.hashFile)):
            errMsg = "missing a mandatory option (-d, -u, -l, -m, -r, -g, -c, -x, --list-tampers, --wizard, --update, --purge or --dependencies). "
            errMsg += "Use -h for basic and -hh for advanced help\n"
            parser.error(errMsg)

        return args

-    except (OptionError, TypeError), e:
-        parser.error(e)
+    except (OptionError, TypeError) as ex:
+        parser.error(ex)

    except SystemExit:
        # Protection against Windows dummy double clicking
--- a/lib/parse/configfile.py
+++ b/lib/parse/configfile.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -39,7 +39,7 @@ def configFileProxy(section, option, datatype):
                value = config.getfloat(section, option) if config.get(section, option) else 0.0
            else:
                value = config.get(section, option)
-        except ValueError, ex:
+        except ValueError as ex:
            errMsg = "error occurred while processing the option "
            errMsg += "'%s' in provided configuration file ('%s')" % (option, getUnicode(ex))
            raise SqlmapSyntaxException(errMsg)
@@ -71,7 +71,7 @@ def configFileParser(configFile):
    try:
        config = UnicodeRawConfigParser()
        config.readfp(configFP)
-    except Exception, ex:
+    except Exception as ex:
        errMsg = "you have provided an invalid and/or unreadable configuration file ('%s')" % getSafeExString(ex)
        raise SqlmapSyntaxException(errMsg)

--- a/lib/parse/handler.py
+++ b/lib/parse/handler.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -35,7 +35,7 @@ class FingerprintHandler(ContentHandler):
        if key == "dbmsVersion":
            self._info[key] = value
        else:
-            if key not in self._info.keys():
+            if key not in self._info:
                self._info[key] = set()

            for _ in value.split("|"):
--- a/lib/parse/headers.py
+++ b/lib/parse/headers.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/lib/parse/html.py
+++ b/lib/parse/html.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/lib/parse/payloads.py
+++ b/lib/parse/payloads.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -11,6 +11,7 @@ import re
 from xml.etree import ElementTree as et

 from lib.core.common import getSafeExString
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import paths
 from lib.core.datatype import AttribDict
@@ -24,8 +25,8 @@ def cleanupVals(text, tag):
    if tag in ("clause", "where"):
        text = text.split(',')

-    if isinstance(text, basestring):
-        text = int(text) if text.isdigit() else text
+    if hasattr(text, "isdigit") and text.isdigit():
+        text = int(text)

    elif isinstance(text, list):
        count = 0
@@ -78,7 +79,7 @@ def parseXmlNode(node):
 def loadBoundaries():
    try:
        doc = et.parse(paths.BOUNDARIES_XML)
-    except Exception, ex:
+    except Exception as ex:
        errMsg = "something appears to be wrong with "
        errMsg += "the file '%s' ('%s'). Please make " % (paths.BOUNDARIES_XML, getSafeExString(ex))
        errMsg += "sure that you haven't made any changes to it"
@@ -93,7 +94,7 @@ def loadPayloads():

        try:
            doc = et.parse(payloadFilePath)
-        except Exception, ex:
+        except Exception as ex:
            errMsg = "something appears to be wrong with "
            errMsg += "the file '%s' ('%s'). Please make " % (payloadFilePath, getSafeExString(ex))
            errMsg += "sure that you haven't made any changes to it"
--- a/lib/parse/sitemap.py
+++ b/lib/parse/sitemap.py
@@ -1,19 +1,19 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

-import httplib
 import re

 from lib.core.common import readInput
 from lib.core.data import kb
 from lib.core.data import logger
+from lib.core.datatype import OrderedSet
 from lib.core.exception import SqlmapSyntaxException
 from lib.request.connect import Connect as Request
-from thirdparty.oset.pyoset import oset
+from thirdparty.six.moves import http_client as _http_client

 abortedFlag = None

@@ -26,11 +26,11 @@ def parseSitemap(url, retVal=None):
    try:
        if retVal is None:
            abortedFlag = False
-            retVal = oset()
+            retVal = OrderedSet()

        try:
            content = Request.getPage(url=url, raise404=True)[0] if not abortedFlag else ""
-        except httplib.InvalidURL:
+        except _http_client.InvalidURL:
            errMsg = "invalid URL given for sitemap ('%s')" % url
            raise SqlmapSyntaxException(errMsg)

--- a/lib/request/init.py
+++ b/lib/request/init.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/lib/request/basic.py
+++ b/lib/request/basic.py
@@ -1,22 +1,24 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

 import codecs
 import gzip
+import io
 import logging
 import re
-import StringIO
 import struct
 import zlib

 from lib.core.common import Backend
 from lib.core.common import extractErrorMessage
 from lib.core.common import extractRegexResult
+from lib.core.common import filterNone
 from lib.core.common import getPublicTypeMembers
+from lib.core.common import getSafeExString
 from lib.core.common import getUnicode
 from lib.core.common import isListLike
 from lib.core.common import randomStr
@@ -46,7 +48,7 @@ from lib.parse.headers import headersParser
 from lib.parse.html import htmlParser
 from lib.utils.htmlentities import htmlEntities
 from thirdparty.chardet import detect
-from thirdparty.odict.odict import OrderedDict
+from thirdparty.odict import OrderedDict

 def forgeHeaders(items=None, base=None):
    """
@@ -56,7 +58,7 @@ def forgeHeaders(items=None, base=None):

    items = items or {}

-    for _ in items.keys():
+    for _ in list(items.keys()):
        if items[_] is None:
            del items[_]

@@ -99,7 +101,7 @@ def forgeHeaders(items=None, base=None):

                if ("%s=" % getUnicode(cookie.name)) in getUnicode(headers[HTTP_HEADER.COOKIE]):
                    if conf.loadCookies:
-                        conf.httpHeaders = filter(None, ((item if item[0] != HTTP_HEADER.COOKIE else None) for item in conf.httpHeaders))
+                        conf.httpHeaders = filterNone((item if item[0] != HTTP_HEADER.COOKIE else None) for item in conf.httpHeaders)
                    elif kb.mergeCookies is None:
                        message = "you provided a HTTP %s header value. " % HTTP_HEADER.COOKIE
                        message += "The target URL provided its own cookies within "
@@ -256,12 +258,12 @@ def decodePage(page, contentEncoding, contentType):
    if not page or (conf.nullConnection and len(page) < 2):
        return getUnicode(page)

-    if isinstance(contentEncoding, basestring) and contentEncoding:
+    if hasattr(contentEncoding, "lower"):
        contentEncoding = contentEncoding.lower()
    else:
        contentEncoding = ""

-    if isinstance(contentType, basestring) and contentType:
+    if hasattr(contentType, "lower"):
        contentType = contentType.lower()
    else:
        contentType = ""
@@ -272,18 +274,18 @@ def decodePage(page, contentEncoding, contentType):

        try:
            if contentEncoding == "deflate":
-                data = StringIO.StringIO(zlib.decompress(page, -15))  # Reference: http://stackoverflow.com/questions/1089662/python-inflate-and-deflate-implementations
+                data = io.BytesIO(zlib.decompress(page, -15))  # Reference: http://stackoverflow.com/questions/1089662/python-inflate-and-deflate-implementations
            else:
-                data = gzip.GzipFile("", "rb", 9, StringIO.StringIO(page))
+                data = gzip.GzipFile("", "rb", 9, io.BytesIO(page))
                size = struct.unpack("<l", page[-4:])[0]  # Reference: http://pydoc.org/get.cgi/usr/local/lib/python2.5/gzip.py
                if size > MAX_CONNECTION_TOTAL_SIZE:
                    raise Exception("size too large")

            page = data.read()
-        except Exception, msg:
+        except Exception as ex:
            if "<html" not in page:  # in some cases, invalid "Content-Encoding" appears for plain HTML (should be ignored)
                errMsg = "detected invalid data for declared content "
-                errMsg += "encoding '%s' ('%s')" % (contentEncoding, msg)
+                errMsg += "encoding '%s' ('%s')" % (contentEncoding, getSafeExString(ex))
                singleTimeLogMessage(errMsg, logging.ERROR)

                warnMsg = "turning off page compression"
@@ -312,12 +314,9 @@ def decodePage(page, contentEncoding, contentType):

    # can't do for all responses because we need to support binary files too
    if not isinstance(page, unicode) and "text/" in contentType:
-        if kb.heuristicMode:
-            kb.pageEncoding = kb.pageEncoding or checkCharEncoding(getHeuristicCharEncoding(page))
-            page = getUnicode(page, kb.pageEncoding)
-        else:
-            # e.g. &#195;&#235;&#224;&#226;&#224;
+        # e.g. &#x9;&#195;&#235;&#224;&#226;&#224;
        if "&#" in page:
+            page = re.sub(r"&#x([0-9a-f]{1,2});", lambda _: (_.group(1) if len(_.group(1)) == 2 else "0%s" % _.group(1)).decode("hex"), page)
            page = re.sub(r"&#(\d{1,3});", lambda _: chr(int(_.group(1))) if int(_.group(1)) < 256 else _.group(0), page)

        # e.g. %20%28%29
@@ -342,7 +341,7 @@ def decodePage(page, contentEncoding, contentType):
                retVal = match.group(0)
                try:
                    retVal = unichr(int(match.group(1)))
-                    except ValueError:
+                except (ValueError, OverflowError):
                    pass
                return retVal
            page = re.sub(r"&#(\d+);", _, page)
--- a/lib/request/basicauthhandler.py
+++ b/lib/request/basicauthhandler.py
@@ -1,19 +1,19 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

-import urllib2
+from thirdparty.six.moves import urllib as _urllib

-class SmartHTTPBasicAuthHandler(urllib2.HTTPBasicAuthHandler):
+class SmartHTTPBasicAuthHandler(_urllib.request.HTTPBasicAuthHandler):
    """
    Reference: http://selenic.com/hg/rev/6c51a5056020
    Fix for a: http://bugs.python.org/issue8797
    """
    def __init__(self, *args, **kwargs):
-        urllib2.HTTPBasicAuthHandler.__init__(self, *args, **kwargs)
+        _urllib.request.HTTPBasicAuthHandler.__init__(self, *args, **kwargs)
        self.retried_req = set()
        self.retried_count = 0

@@ -30,8 +30,8 @@ class SmartHTTPBasicAuthHandler(urllib2.HTTPBasicAuthHandler):
            self.retried_count = 0
        else:
            if self.retried_count > 5:
-                raise urllib2.HTTPError(req.get_full_url(), 401, "basic auth failed", headers, None)
+                raise _urllib.error.HTTPError(req.get_full_url(), 401, "basic auth failed", headers, None)
            else:
                self.retried_count += 1

-        return urllib2.HTTPBasicAuthHandler.http_error_auth_reqed(self, auth_header, host, req, headers)
+        return _urllib.request.HTTPBasicAuthHandler.http_error_auth_reqed(self, auth_header, host, req, headers)
--- a/lib/request/chunkedhandler.py
+++ b/lib/request/chunkedhandler.py
@@ -0,0 +1,43 @@
+#!/usr/bin/env python2
+
+"""
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+from lib.core.data import conf
+from thirdparty.six.moves import urllib as _urllib
+
+class ChunkedHandler(_urllib.request.HTTPHandler):
+    """
+    Ensures that HTTPHandler is working properly in case of Chunked Transfer-Encoding
+    """
+
+    def _http_request(self, request):
+        host = request.get_host()
+        if not host:
+            raise _urllib.error.URLError("no host given")
+
+        if request.has_data():  # POST
+            data = request.get_data()
+            if not request.has_header("Content-type"):
+                request.add_unredirected_header(
+                    "Content-type",
+                    "application/x-www-form-urlencoded")
+            if not request.has_header("Content-length") and not conf.chunked:
+                request.add_unredirected_header(
+                    "Content-length", "%d" % len(data))
+
+        sel_host = host
+        if request.has_proxy():
+            sel_host = _urllib.parse.urlsplit(request.get_selector()).netloc
+
+        if not request.has_header("Host"):
+            request.add_unredirected_header("Host", sel_host)
+        for name, value in self.parent.addheaders:
+            name = name.capitalize()
+            if not request.has_header(name):
+                request.add_unredirected_header(name, value)
+        return request
+
+    http_request = _http_request
--- a/lib/request/comparison.py
+++ b/lib/request/comparison.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

--- a/lib/request/connect.py
+++ b/lib/request/connect.py
@@ -1,14 +1,11 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

 import binascii
-import compiler
-import httplib
-import keyword
 import logging
 import re
 import socket
@@ -16,9 +13,6 @@ import string
 import struct
 import time
 import traceback
-import urllib
-import urllib2
-import urlparse

 try:
    import websocket
@@ -32,11 +26,13 @@ from lib.core.agent import agent
 from lib.core.common import asciifyUrl
 from lib.core.common import calculateDeltaSeconds
 from lib.core.common import checkSameHost
+from lib.core.common import chunkSplitPostData
 from lib.core.common import clearConsoleLine
 from lib.core.common import dataToStdout
 from lib.core.common import escapeJsonValue
 from lib.core.common import evaluateCode
 from lib.core.common import extractRegexResult
+from lib.core.common import filterNone
 from lib.core.common import findMultipartPostBoundary
 from lib.core.common import getCurrentThreadData
 from lib.core.common import getHeader
@@ -44,6 +40,7 @@ from lib.core.common import getHostHeader
 from lib.core.common import getRequestHeader
 from lib.core.common import getSafeExString
 from lib.core.common import getUnicode
+from lib.core.common import isMultiThreadMode
 from lib.core.common import logHTTPTraffic
 from lib.core.common import pushValue
 from lib.core.common import popValue
@@ -61,9 +58,11 @@ from lib.core.common import unicodeencode
 from lib.core.common import unsafeVariableNaming
 from lib.core.common import urldecode
 from lib.core.common import urlencode
+from lib.core.compat import xrange
 from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
+from lib.core.datatype import AttribDict
 from lib.core.decorators import stackedmethod
 from lib.core.dicts import POST_HINT_CONTENT_TYPES
 from lib.core.enums import ADJUST_TIME_DELAY
@@ -77,7 +76,7 @@ from lib.core.enums import PAYLOAD
 from lib.core.enums import PLACE
 from lib.core.enums import POST_HINT
 from lib.core.enums import REDIRECTION
-from lib.core.enums import WEB_API
+from lib.core.enums import WEB_PLATFORM
 from lib.core.exception import SqlmapCompressionException
 from lib.core.exception import SqlmapConnectionException
 from lib.core.exception import SqlmapGenericException
@@ -90,7 +89,7 @@ from lib.core.settings import DEFAULT_CONTENT_TYPE
 from lib.core.settings import DEFAULT_COOKIE_DELIMITER
 from lib.core.settings import DEFAULT_GET_POST_DELIMITER
 from lib.core.settings import DEFAULT_USER_AGENT
-from lib.core.settings import EVALCODE_KEYWORD_SUFFIX
+from lib.core.settings import EVALCODE_ENCODED_PREFIX
 from lib.core.settings import HTTP_ACCEPT_HEADER_VALUE
 from lib.core.settings import HTTP_ACCEPT_ENCODING_HEADER_VALUE
 from lib.core.settings import MAX_CONNECTION_CHUNK_SIZE
@@ -101,7 +100,7 @@ from lib.core.settings import MAX_MURPHY_SLEEP_TIME
 from lib.core.settings import META_REFRESH_REGEX
 from lib.core.settings import MIN_TIME_RESPONSES
 from lib.core.settings import MAX_TIME_RESPONSES
-from lib.core.settings import IDS_WAF_CHECK_PAYLOAD
+from lib.core.settings import IPS_WAF_CHECK_PAYLOAD
 from lib.core.settings import IS_WIN
 from lib.core.settings import LARGE_CHUNK_TRIM_MARKER
 from lib.core.settings import PAYLOAD_DELIMITER
@@ -121,7 +120,10 @@ from lib.request.basic import processResponse
 from lib.request.direct import direct
 from lib.request.comparison import comparison
 from lib.request.methodrequest import MethodRequest
-from thirdparty.odict.odict import OrderedDict
+from thirdparty import six
+from thirdparty.odict import OrderedDict
+from thirdparty.six.moves import http_client as _http_client
+from thirdparty.six.moves import urllib as _urllib
 from thirdparty.socks.socks import ProxyError

 class Connect(object):
@@ -173,9 +175,11 @@ class Connect(object):
                    warnMsg += "(e.g. 'https://help.ubuntu.com/community/Tor')"
            else:
                warnMsg = "if the problem persists please check that the provided "
-                warnMsg += "target URL is valid. In case that it is, you can try to rerun "
-                warnMsg += "with the switch '--random-agent' turned on "
-                warnMsg += "and/or proxy switches ('--ignore-proxy', '--proxy',...)"
+                warnMsg += "target URL is reachable. In case that it is, "
+                warnMsg += "you can try to rerun with "
+                if not conf.randomAgent:
+                    warnMsg += "switch '--random-agent' and/or "
+                warnMsg += "proxy switches ('--ignore-proxy', '--proxy',...)"
            singleTimeWarnMessage(warnMsg)

        elif conf.threads > 1:
@@ -267,14 +271,20 @@ class Connect(object):
        crawling = kwargs.get("crawling", False)
        checking = kwargs.get("checking", False)
        skipRead = kwargs.get("skipRead", False)
+        finalCode = kwargs.get("finalCode", False)
+        chunked = kwargs.get("chunked", False) or conf.chunked

        if multipart:
            post = multipart

+        if chunked and post:
+            post = _urllib.parse.unquote(post)
+            post = chunkSplitPostData(post)
+
        websocket_ = url.lower().startswith("ws")

-        if not urlparse.urlsplit(url).netloc:
-            url = urlparse.urljoin(conf.url, url)
+        if not _urllib.parse.urlsplit(url).netloc:
+            url = _urllib.parse.urljoin(conf.url, url)

        # flag to know if we are dealing with the same target host
        target = checkSameHost(url, conf.url)
@@ -295,7 +305,7 @@ class Connect(object):
        code = None
        status = None

-        _ = urlparse.urlsplit(url)
+        _ = _urllib.parse.urlsplit(url)
        requestMsg = u"HTTP request [#%d]:\r\n%s " % (threadData.lastRequestUID, method or (HTTPMETHOD.POST if post is not None else HTTPMETHOD.GET))
        requestMsg += getUnicode(("%s%s" % (_.path or "/", ("?%s" % _.query) if _.query else "")) if not any((refreshing, crawling, checking)) else url)
        responseMsg = u"HTTP response "
@@ -323,7 +333,7 @@ class Connect(object):
                pass

            elif target:
-                if conf.forceSSL and urlparse.urlparse(url).scheme != "https":
+                if conf.forceSSL and _urllib.parse.urlparse(url).scheme != "https":
                    url = re.sub(r"(?i)\Ahttp:", "https:", url)
                    url = re.sub(r"(?i):80/", ":443/", url)

@@ -348,7 +358,7 @@ class Connect(object):
                url = "%s?%s" % (url, get)
                requestMsg += "?%s" % get

-            requestMsg += " %s" % httplib.HTTPConnection._http_vsn_str
+            requestMsg += " %s" % _http_client.HTTPConnection._http_vsn_str

            # Prepare HTTP headers
            headers = forgeHeaders({HTTP_HEADER.COOKIE: cookie, HTTP_HEADER.USER_AGENT: ua, HTTP_HEADER.REFERER: referer, HTTP_HEADER.HOST: host}, base=None if target else {})
@@ -393,6 +403,9 @@ class Connect(object):
            if conf.keepAlive:
                headers[HTTP_HEADER.CONNECTION] = "keep-alive"
            
+            if chunked:
+                headers[HTTP_HEADER.TRANSFER_ENCODING] = "chunked"
+
            if auxHeaders:
                headers = forgeHeaders(auxHeaders, headers)

@@ -414,7 +427,7 @@ class Connect(object):
                page = ws.recv()
                ws.close()
                code = ws.status
-                status = httplib.responses[code]
+                status = _http_client.responses[code]

                class _(dict):
                    pass
@@ -422,7 +435,7 @@ class Connect(object):
                responseHeaders = _(ws.getheaders())
                responseHeaders.headers = ["%s: %s\r\n" % (_[0].capitalize(), _[1]) for _ in responseHeaders.items()]

-                requestHeaders += "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
+                requestHeaders += "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if hasattr(key, "capitalize") else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
                requestMsg += "\r\n%s" % requestHeaders

                if post is not None:
@@ -439,11 +452,11 @@ class Connect(object):
                    req = MethodRequest(url, post, headers)
                    req.set_method(method)
                elif url is not None:
-                    req = urllib2.Request(url, post, headers)
+                    req = _urllib.request.Request(url, post, headers)
                else:
                    return None, None, None

-                requestHeaders += "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in req.header_items()])
+                requestHeaders += "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if hasattr(key, "capitalize") else key), getUnicode(value)) for (key, value) in req.header_items()])

                if not getRequestHeader(req, HTTP_HEADER.COOKIE) and conf.cj:
                    conf.cj._policy._now = conf.cj._now = int(time.time())
@@ -451,7 +464,7 @@ class Connect(object):
                    requestHeaders += "\r\n%s" % ("Cookie: %s" % ";".join("%s=%s" % (getUnicode(cookie.name), getUnicode(cookie.value)) for cookie in cookies))

                if post is not None:
-                    if not getRequestHeader(req, HTTP_HEADER.CONTENT_LENGTH):
+                    if not getRequestHeader(req, HTTP_HEADER.CONTENT_LENGTH) and not chunked:
                        requestHeaders += "\r\n%s: %d" % (string.capwords(HTTP_HEADER.CONTENT_LENGTH), len(post))

                if not getRequestHeader(req, HTTP_HEADER.CONNECTION):
@@ -462,6 +475,7 @@ class Connect(object):
                if post is not None:
                    requestMsg += "\r\n\r\n%s" % getUnicode(post)

+                if not chunked:
                    requestMsg += "\r\n"

                if not multipart:
@@ -477,7 +491,7 @@ class Connect(object):
                            for char in (r"\r", r"\n"):
                                cookie.value = re.sub(r"(%s)([^ \t])" % char, r"\g<1>\t\g<2>", cookie.value)

-                conn = urllib2.urlopen(req)
+                conn = _urllib.request.urlopen(req)

                if not kb.authHeader and getRequestHeader(req, HTTP_HEADER.AUTHORIZATION) and (conf.authType or "").lower() == AUTH_TYPE.BASIC.lower():
                    kb.authHeader = getRequestHeader(req, HTTP_HEADER.AUTHORIZATION)
@@ -493,7 +507,7 @@ class Connect(object):
                if hasattr(conn, "redurl"):
                    page = (threadData.lastRedirectMsg[1] if kb.redirectChoice == REDIRECTION.NO else Connect._connReadProxy(conn)) if not skipRead else None
                    skipLogTraffic = kb.redirectChoice == REDIRECTION.NO
-                    code = conn.redcode
+                    code = conn.redcode if not finalCode else code
                else:
                    page = Connect._connReadProxy(conn) if not skipRead else None

@@ -533,7 +547,7 @@ class Connect(object):
                        if re.search(r"\Ahttps?://", refresh, re.I):
                            url = refresh
                        else:
-                            url = urlparse.urljoin(url, refresh)
+                            url = _urllib.parse.urljoin(url, refresh)

                        threadData.lastRedirectMsg = (threadData.lastRequestUID, page)
                        kwargs["refreshing"] = True
@@ -552,11 +566,11 @@ class Connect(object):
                    if hasattr(conn.fp, '_sock'):
                        conn.fp._sock.close()
                    conn.close()
-                except Exception, ex:
+                except Exception as ex:
                    warnMsg = "problem occurred during connection closing ('%s')" % getSafeExString(ex)
                    logger.warn(warnMsg)

-        except SqlmapConnectionException, ex:
+        except SqlmapConnectionException as ex:
            if conf.proxyList and not kb.threadException:
                warnMsg = "unable to connect to the target URL ('%s')" % ex
                logger.critical(warnMsg)
@@ -565,7 +579,7 @@ class Connect(object):
            else:
                raise

-        except urllib2.HTTPError, ex:
+        except _urllib.error.HTTPError as ex:
            page = None
            responseHeaders = None

@@ -587,7 +601,7 @@ class Connect(object):
            except:
                pass
            finally:
-                page = page if isinstance(page, unicode) else getUnicode(page)
+                page = getUnicode(page)

            code = ex.code
            status = getSafeExString(ex)
@@ -599,7 +613,7 @@ class Connect(object):
            responseMsg += "[#%d] (%s %s):\r\n" % (threadData.lastRequestUID, code, status)

            if responseHeaders:
-                logHeaders = "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
+                logHeaders = getUnicode("".join(responseHeaders.headers).strip())

            logHTTPTraffic(requestMsg, "%s%s\r\n\r\n%s" % (responseMsg, logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE]), start, time.time())

@@ -614,22 +628,22 @@ class Connect(object):
                logger.log(CUSTOM_LOGGING.TRAFFIC_IN, responseMsg)

            if ex.code != conf.ignoreCode:
-                if ex.code == httplib.UNAUTHORIZED:
+                if ex.code == _http_client.UNAUTHORIZED:
                    errMsg = "not authorized, try to provide right HTTP "
                    errMsg += "authentication type and valid credentials (%d)" % code
                    raise SqlmapConnectionException(errMsg)
-                elif ex.code == httplib.NOT_FOUND:
+                elif ex.code == _http_client.NOT_FOUND:
                    if raise404:
                        errMsg = "page not found (%d)" % code
                        raise SqlmapConnectionException(errMsg)
                    else:
                        debugMsg = "page not found (%d)" % code
                        singleTimeLogMessage(debugMsg, logging.DEBUG)
-                elif ex.code == httplib.GATEWAY_TIMEOUT:
+                elif ex.code == _http_client.GATEWAY_TIMEOUT:
                    if ignoreTimeout:
                        return None if not conf.ignoreTimeouts else "", None, None
                    else:
-                        warnMsg = "unable to connect to the target URL (%d - %s)" % (ex.code, httplib.responses[ex.code])
+                        warnMsg = "unable to connect to the target URL (%d - %s)" % (ex.code, _http_client.responses[ex.code])
                        if threadData.retriesCount < conf.retries and not kb.threadException:
                            warnMsg += ". sqlmap is going to retry the request"
                            logger.critical(warnMsg)
@@ -643,7 +657,7 @@ class Connect(object):
                    debugMsg = "got HTTP error code: %d (%s)" % (code, status)
                    logger.debug(debugMsg)

-        except (urllib2.URLError, socket.error, socket.timeout, httplib.HTTPException, struct.error, binascii.Error, ProxyError, SqlmapCompressionException, WebSocketException, TypeError, ValueError):
+        except (_urllib.error.URLError, socket.error, socket.timeout, _http_client.HTTPException, struct.error, binascii.Error, ProxyError, SqlmapCompressionException, WebSocketException, TypeError, ValueError, OverflowError):
            tbMsg = traceback.format_exc()

            if checking:
@@ -727,24 +741,36 @@ class Connect(object):
                else:
                    logger.debug(warnMsg)
                return Connect._retryProxy(**kwargs)
-            elif kb.testMode or kb.multiThreadMode:
+            elif kb.testMode or isMultiThreadMode():
                logger.critical(warnMsg)
                return None, None, None
            else:
                raise SqlmapConnectionException(warnMsg)

        finally:
-            if isinstance(page, basestring) and not isinstance(page, unicode):
+            if isinstance(page, six.binary_type):
                if HTTP_HEADER.CONTENT_TYPE in (responseHeaders or {}) and not re.search(TEXT_CONTENT_TYPE_REGEX, responseHeaders[HTTP_HEADER.CONTENT_TYPE]):
-                    page = unicode(page, errors="ignore")
+                    page = six.text_type(page, errors="ignore")
                else:
                    page = getUnicode(page)
+
+            for function in kb.preprocessFunctions:
+                try:
+                    page, responseHeaders, code = function(page, responseHeaders, code)
+                except Exception as ex:
+                    errMsg = "error occurred while running preprocess "
+                    errMsg += "function '%s' ('%s')" % (function.__name__, getSafeExString(ex))
+                    raise SqlmapGenericException(errMsg)
+
+            threadData.lastPage = page
+            threadData.lastCode = code
+
            socket.setdefaulttimeout(conf.timeout)

        processResponse(page, responseHeaders, status)

        if conn and getattr(conn, "redurl", None):
-            _ = urlparse.urlsplit(conn.redurl)
+            _ = _urllib.parse.urlsplit(conn.redurl)
            _ = ("%s%s" % (_.path or "/", ("?%s" % _.query) if _.query else ""))
            requestMsg = re.sub(r"(\n[A-Z]+ ).+?( HTTP/\d)", r"\g<1>%s\g<2>" % getUnicode(_).replace("\\", "\\\\"), requestMsg, 1)

@@ -753,12 +779,12 @@ class Connect(object):
                requestMsg = re.sub(r"(?i)Content-length: \d+\n", "", requestMsg)
                requestMsg = re.sub(r"(?s)\n\n.+", "\n", requestMsg)

-            responseMsg += "[#%d] (%d %s):\r\n" % (threadData.lastRequestUID, conn.code, status)
+            responseMsg += "[#%d] (%s %s):\r\n" % (threadData.lastRequestUID, conn.code, status)
        else:
            responseMsg += "[#%d] (%s %s):\r\n" % (threadData.lastRequestUID, code, status)

        if responseHeaders:
-            logHeaders = "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if isinstance(key, basestring) else key), getUnicode(value)) for (key, value) in responseHeaders.items()])
+            logHeaders = getUnicode("".join(responseHeaders.headers).strip())

        if not skipLogTraffic:
            logHTTPTraffic(requestMsg, "%s%s\r\n\r\n%s" % (responseMsg, logHeaders, (page or "")[:MAX_CONNECTION_CHUNK_SIZE]), start, time.time())
@@ -813,7 +839,7 @@ class Connect(object):

        if conf.httpHeaders:
            headers = OrderedDict(conf.httpHeaders)
-            contentType = max(headers[_] if _.upper() == HTTP_HEADER.CONTENT_TYPE.upper() else None for _ in headers.keys())
+            contentType = max(headers[_] if _.upper() == HTTP_HEADER.CONTENT_TYPE.upper() else None for _ in headers)

            if (kb.postHint or conf.skipUrlEncode) and postUrlEncode:
                postUrlEncode = False
@@ -830,13 +856,13 @@ class Connect(object):

                    try:
                        payload = function(payload=payload, headers=auxHeaders, delimiter=delimiter, hints=hints)
-                    except Exception, ex:
+                    except Exception as ex:
                        errMsg = "error occurred while running tamper "
-                        errMsg += "function '%s' ('%s')" % (function.func_name, getSafeExString(ex))
+                        errMsg += "function '%s' ('%s')" % (function.__name__, getSafeExString(ex))
                        raise SqlmapGenericException(errMsg)

-                    if not isinstance(payload, basestring):
-                        errMsg = "tamper function '%s' returns " % function.func_name
+                    if not isinstance(payload, six.string_types):
+                        errMsg = "tamper function '%s' returns " % function.__name__
                        errMsg += "invalid payload type ('%s')" % type(payload)
                        raise SqlmapValueException(errMsg)

@@ -888,7 +914,7 @@ class Connect(object):
                        postUrlEncode = False

            if conf.hpp:
-                if not any(conf.url.lower().endswith(_.lower()) for _ in (WEB_API.ASP, WEB_API.ASPX)):
+                if not any(conf.url.lower().endswith(_.lower()) for _ in (WEB_PLATFORM.ASP, WEB_PLATFORM.ASPX)):
                    warnMsg = "HTTP parameter pollution should work only against "
                    warnMsg += "ASP(.NET) targets"
                    singleTimeWarnMessage(warnMsg)
@@ -960,83 +986,92 @@ class Connect(object):
        if conf.csrfToken:
            def _adjustParameter(paramString, parameter, newValue):
                retVal = paramString
-                match = re.search(r"%s=[^&]*" % re.escape(parameter), paramString)
+
+                if urlencode(parameter) in paramString:
+                    parameter = urlencode(parameter)
+
+                match = re.search(r"%s=[^&]*" % re.escape(parameter), paramString, re.I)
                if match:
-                    retVal = re.sub(re.escape(match.group(0)), ("%s=%s" % (parameter, newValue)).replace('\\', r'\\'), paramString)
+                    retVal = re.sub("(?i)%s" % re.escape(match.group(0)), ("%s=%s" % (parameter, newValue)).replace('\\', r'\\'), paramString)
                else:
-                    match = re.search(r"(%s[\"']:[\"'])([^\"']+)" % re.escape(parameter), paramString)
+                    match = re.search(r"(%s[\"']:[\"'])([^\"']+)" % re.escape(parameter), paramString, re.I)
                    if match:
-                        retVal = re.sub(re.escape(match.group(0)), "%s%s" % (match.group(1), newValue), paramString)
+                        retVal = re.sub("(?i)%s" % re.escape(match.group(0)), "%s%s" % (match.group(1), newValue), paramString)
+
                return retVal

+            token = AttribDict()
            page, headers, code = Connect.getPage(url=conf.csrfUrl or conf.url, data=conf.data if conf.csrfUrl == conf.url else None, method=conf.method if conf.csrfUrl == conf.url else None, cookie=conf.parameters.get(PLACE.COOKIE), direct=True, silent=True, ua=conf.parameters.get(PLACE.USER_AGENT), referer=conf.parameters.get(PLACE.REFERER), host=conf.parameters.get(PLACE.HOST))
-            token = extractRegexResult(r"(?i)<input[^>]+\bname=[\"']?%s\b[^>]*\bvalue=[\"']?(?P<result>[^>'\"]*)" % re.escape(conf.csrfToken), page or "")
+            page = urldecode(page)  # for anti-CSRF tokens with special characters in their name (e.g. 'foo:bar=...')

-            if not token:
-                token = extractRegexResult(r"(?i)<input[^>]+\bvalue=[\"']?(?P<result>[^>'\"]*)[\"']?[^>]*\bname=[\"']?%s\b" % re.escape(conf.csrfToken), page or "")
+            match = re.search(r"(?i)<input[^>]+\bname=[\"']?(?P<name>%s)\b[^>]*\bvalue=[\"']?(?P<value>[^>'\"]*)" % conf.csrfToken, page or "", re.I)

-                if not token:
-                    match = re.search(r"%s[\"']:[\"']([^\"']+)" % re.escape(conf.csrfToken), page or "")
-                    token = match.group(1) if match else None
+            if not match:
+                match = re.search(r"(?i)<input[^>]+\bvalue=[\"']?(?P<value>[^>'\"]*)[\"']?[^>]*\bname=[\"']?(?P<name>%s)\b" % conf.csrfToken, page or "", re.I)

-                    if not token:
-                        token = extractRegexResult(r"\b%s\s*[:=]\s*(?P<result>\w+)" % re.escape(conf.csrfToken), str(headers))
+                if not match:
+                    match = re.search(r"(?P<name>%s)[\"']:[\"'](?P<value>[^\"']+)" % conf.csrfToken, page or "", re.I)

-                        if not token:
-                            token = extractRegexResult(r"\b%s\s*=\s*['\"]?(?P<result>[^;'\"]+)" % re.escape(conf.csrfToken), page or "")
+                    if not match:
+                        match = re.search(r"\b(?P<name>%s)\s*[:=]\s*(?P<value>\w+)" % conf.csrfToken, str(headers), re.I)

-                            if token:
-                                match = re.search(r"String\.fromCharCode\(([\d+, ]+)\)", token)
+                        if not match:
+                            match = re.search(r"\b(?P<name>%s)\s*=\s*['\"]?(?P<value>[^;'\"]+)" % conf.csrfToken, page or "", re.I)

            if match:
-                                    token = "".join(chr(int(_)) for _ in match.group(1).replace(' ', "").split(','))
+                token.name, token.value = match.group("name"), match.group("value")
+
+                match = re.search(r"String\.fromCharCode\(([\d+, ]+)\)", token.value)
+                if match:
+                    token.value = "".join(chr(int(_)) for _ in match.group(1).replace(' ', "").split(','))

            if not token:
-                if conf.csrfUrl != conf.url and code == httplib.OK:
+                if conf.csrfUrl and conf.csrfToken and conf.csrfUrl != conf.url and code == _http_client.OK:
                    if headers and "text/plain" in headers.get(HTTP_HEADER.CONTENT_TYPE, ""):
-                        token = page
+                        token.name = conf.csrfToken
+                        token.value = page

-                if not token and conf.cj and any(_.name == conf.csrfToken for _ in conf.cj):
+                if not token and conf.cj and any(re.search(conf.csrfToken, _.name, re.I) for _ in conf.cj):
                    for _ in conf.cj:
-                        if _.name == conf.csrfToken:
-                            token = _.value
-                            if not any(conf.csrfToken in _ for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}))):
+                        if re.search(conf.csrfToken, _.name, re.I):
+                            token.name, token.value = _.name, _.value
+                            if not any(re.search(conf.csrfToken, ' '.join(_), re.I) for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}))):
                                if post:
-                                    post = "%s%s%s=%s" % (post, conf.paramDel or DEFAULT_GET_POST_DELIMITER, conf.csrfToken, token)
+                                    post = "%s%s%s=%s" % (post, conf.paramDel or DEFAULT_GET_POST_DELIMITER, token.name, token.value)
                                elif get:
-                                    get = "%s%s%s=%s" % (get, conf.paramDel or DEFAULT_GET_POST_DELIMITER, conf.csrfToken, token)
+                                    get = "%s%s%s=%s" % (get, conf.paramDel or DEFAULT_GET_POST_DELIMITER, token.name, token.value)
                                else:
-                                    get = "%s=%s" % (conf.csrfToken, token)
+                                    get = "%s=%s" % (token.name, token.value)
                            break

                if not token:
-                    errMsg = "anti-CSRF token '%s' can't be found at '%s'" % (conf.csrfToken, conf.csrfUrl or conf.url)
+                    errMsg = "anti-CSRF token '%s' can't be found at '%s'" % (conf.csrfToken._original, conf.csrfUrl or conf.url)
                    if not conf.csrfUrl:
                        errMsg += ". You can try to rerun by providing "
                        errMsg += "a valid value for option '--csrf-url'"
                    raise SqlmapTokenException(errMsg)

            if token:
-                token = token.strip("'\"")
+                token.value = token.value.strip("'\"")

                for place in (PLACE.GET, PLACE.POST):
                    if place in conf.parameters:
                        if place == PLACE.GET and get:
-                            get = _adjustParameter(get, conf.csrfToken, token)
+                            get = _adjustParameter(get, token.name, token.value)
                        elif place == PLACE.POST and post:
-                            post = _adjustParameter(post, conf.csrfToken, token)
+                            post = _adjustParameter(post, token.name, token.value)

                for i in xrange(len(conf.httpHeaders)):
-                    if conf.httpHeaders[i][0].lower() == conf.csrfToken.lower():
-                        conf.httpHeaders[i] = (conf.httpHeaders[i][0], token)
+                    if conf.httpHeaders[i][0].lower() == token.name.lower():
+                        conf.httpHeaders[i] = (conf.httpHeaders[i][0], token.value)

        if conf.rParam:
            def _randomizeParameter(paramString, randomParameter):
                retVal = paramString
-                match = re.search(r"(\A|\b)%s=(?P<value>[^&;]+)" % re.escape(randomParameter), paramString)
+                match = re.search(r"(\A|\b)%s=(?P<value>[^&;]*)" % re.escape(randomParameter), paramString)
                if match:
                    origValue = match.group("value")
-                    retVal = re.sub(r"(\A|\b)%s=[^&;]+" % re.escape(randomParameter), "%s=%s" % (randomParameter, randomizeParameterValue(origValue)), paramString)
+                    retVal = re.sub(r"(\A|\b)%s=[^&;]*" % re.escape(randomParameter), "%s=%s" % (randomParameter, randomizeParameterValue(origValue)), paramString)
                return retVal

            for randomParameter in conf.rParam:
@@ -1055,14 +1090,13 @@ class Connect(object):
            delimiter = conf.paramDel or DEFAULT_GET_POST_DELIMITER
            variables = {"uri": uri, "lastPage": threadData.lastPage, "_locals": locals()}
            originals = {}
-            keywords = keyword.kwlist

            if not get and PLACE.URI in conf.parameters:
-                query = urlparse.urlsplit(uri).query or ""
+                query = _urllib.parse.urlsplit(uri).query or ""
            else:
                query = None

-            for item in filter(None, (get, post if not kb.postHint else None, query)):
+            for item in filterNone((get, post if not kb.postHint else None, query)):
                for part in item.split(delimiter):
                    if '=' in part:
                        name, value = part.split('=', 1)
@@ -1070,8 +1104,6 @@ class Connect(object):
                        if safeVariableNaming(name) != name:
                            conf.evalCode = re.sub(r"\b%s\b" % re.escape(name), safeVariableNaming(name), conf.evalCode)
                            name = safeVariableNaming(name)
-                        elif name in keywords:
-                            name = "%s%s" % (name, EVALCODE_KEYWORD_SUFFIX)
                        value = urldecode(value, convall=True, spaceplus=(item == post and kb.postSpaceToPlus))
                        variables[name] = value

@@ -1083,31 +1115,29 @@ class Connect(object):
                        if safeVariableNaming(name) != name:
                            conf.evalCode = re.sub(r"\b%s\b" % re.escape(name), safeVariableNaming(name), conf.evalCode)
                            name = safeVariableNaming(name)
-                        elif name in keywords:
-                            name = "%s%s" % (name, EVALCODE_KEYWORD_SUFFIX)
                        value = urldecode(value, convall=True)
                        variables[name] = value

            while True:
                try:
-                    compiler.parse(unicodeencode(conf.evalCode.replace(';', '\n')))
-                except SyntaxError, ex:
+                    compile(unicodeencode(conf.evalCode.replace(';', '\n')), "", "exec")
+                except SyntaxError as ex:
                    if ex.text:
                        original = replacement = ex.text.strip()
+
                        if '=' in original:
                            name, value = original.split('=', 1)
                            name = name.strip()
                            if safeVariableNaming(name) != name:
                                replacement = re.sub(r"\b%s\b" % re.escape(name), safeVariableNaming(name), replacement)
-                            elif name in keywords:
-                                replacement = re.sub(r"\b%s\b" % re.escape(name), "%s%s" % (name, EVALCODE_KEYWORD_SUFFIX), replacement)
                        else:
                            for _ in re.findall(r"[A-Za-z_]+", original)[::-1]:
-                                if _ in keywords:
-                                    replacement = replacement.replace(_, "%s%s" % (_, EVALCODE_KEYWORD_SUFFIX))
+                                if safeVariableNaming(_) != _:
+                                    replacement = replacement.replace(_, safeVariableNaming(_))
                                    break
+
                        if original == replacement:
-                            conf.evalCode = conf.evalCode.replace(EVALCODE_KEYWORD_SUFFIX, "")
+                            conf.evalCode = conf.evalCode.replace(EVALCODE_ENCODED_PREFIX, "")
                            break
                        else:
                            conf.evalCode = conf.evalCode.replace(getUnicode(ex.text.strip(), UNICODE_ENCODING), replacement)
@@ -1119,12 +1149,7 @@ class Connect(object):
            originals.update(variables)
            evaluateCode(conf.evalCode, variables)

-            for variable in variables.keys():
-                if variable.endswith(EVALCODE_KEYWORD_SUFFIX):
-                    value = variables[variable]
-                    del variables[variable]
-                    variables[variable.replace(EVALCODE_KEYWORD_SUFFIX, "")] = value
-
+            for variable in list(variables.keys()):
                if unsafeVariableNaming(variable) != variable:
                    value = variables[variable]
                    del variables[variable]
@@ -1134,7 +1159,7 @@ class Connect(object):

            for name, value in variables.items():
                if name != "__builtins__" and originals.get(name, "") != value:
-                    if isinstance(value, (basestring, int)):
+                    if isinstance(value, (int, six.string_types)):
                        found = False
                        value = getUnicode(value, UNICODE_ENCODING)

@@ -1254,14 +1279,17 @@ class Connect(object):
                _, headers, code = Connect.getPage(url=uri, get=get, post=post, method=method, cookie=cookie, ua=ua, referer=referer, host=host, silent=silent, auxHeaders=auxHeaders, raise404=raise404, skipRead=(kb.nullConnection == NULLCONNECTION.SKIP_READ))

                if headers:
+                    try:
                        if kb.nullConnection in (NULLCONNECTION.HEAD, NULLCONNECTION.SKIP_READ) and headers.get(HTTP_HEADER.CONTENT_LENGTH):
-                        pageLength = int(headers[HTTP_HEADER.CONTENT_LENGTH])
+                            pageLength = int(headers[HTTP_HEADER.CONTENT_LENGTH].split(',')[0])
                        elif kb.nullConnection == NULLCONNECTION.RANGE and headers.get(HTTP_HEADER.CONTENT_RANGE):
                            pageLength = int(headers[HTTP_HEADER.CONTENT_RANGE][headers[HTTP_HEADER.CONTENT_RANGE].find('/') + 1:])
+                    except ValueError:
+                        pass
            finally:
                kb.pageCompress = popValue()

-        if not pageLength:
+        if pageLength is None:
            try:
                page, headers, code = Connect.getPage(url=uri, get=get, post=post, method=method, cookie=cookie, ua=ua, referer=referer, host=host, silent=silent, auxHeaders=auxHeaders, response=response, raise404=raise404, ignoreTimeout=timeBasedCompare)
            except MemoryError:
@@ -1274,7 +1302,7 @@ class Connect(object):

        if conf.secondUrl:
            page, headers, code = Connect.getPage(url=conf.secondUrl, cookie=cookie, ua=ua, silent=silent, auxHeaders=auxHeaders, response=response, raise404=False, ignoreTimeout=timeBasedCompare, refreshing=True)
-        elif kb.secondReq and IDS_WAF_CHECK_PAYLOAD not in urllib.unquote(value or ""):
+        elif kb.secondReq and IPS_WAF_CHECK_PAYLOAD not in _urllib.parse.unquote(value or ""):
            def _(value):
                if kb.customInjectionMark in (value or ""):
                    if payload is None:
@@ -1285,10 +1313,9 @@ class Connect(object):
            page, headers, code = Connect.getPage(url=_(kb.secondReq[0]), post=_(kb.secondReq[2]), method=kb.secondReq[1], cookie=kb.secondReq[3], silent=silent, auxHeaders=dict(auxHeaders, **dict(kb.secondReq[4])), response=response, raise404=False, ignoreTimeout=timeBasedCompare, refreshing=True)

        threadData.lastQueryDuration = calculateDeltaSeconds(start)
-        threadData.lastPage = page
-        threadData.lastCode = code

-        kb.originalCode = kb.originalCode or code
+        kb.originalCode = code if kb.originalCode is None else kb.originalCode
+        kb.originalPage = page if kb.originalPage is None else kb.originalPage

        if kb.testMode:
            kb.testQueryCount += 1
@@ -1298,8 +1325,8 @@ class Connect(object):
        elif noteResponseTime:
            kb.responseTimes.setdefault(kb.responseTimeMode, [])
            kb.responseTimes[kb.responseTimeMode].append(threadData.lastQueryDuration)
-            if len(kb.responseTimes) > MAX_TIME_RESPONSES:
-                kb.responseTimes = kb.responseTimes[-MAX_TIME_RESPONSES:]
+            if len(kb.responseTimes[kb.responseTimeMode]) > MAX_TIME_RESPONSES:
+                kb.responseTimes[kb.responseTimeMode] = kb.responseTimes[kb.responseTimeMode][-MAX_TIME_RESPONSES // 2:]

        if not response and removeReflection:
            page = removeReflectiveValues(page, payload)
--- a/lib/request/direct.py
+++ b/lib/request/direct.py
@@ -1,7 +1,7 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

@@ -25,6 +25,7 @@ from lib.core.enums import CUSTOM_LOGGING
 from lib.core.enums import DBMS
 from lib.core.enums import EXPECTED
 from lib.core.enums import TIMEOUT_STATE
+from lib.core.settings import TAKEOVER_TABLE_PREFIX
 from lib.core.settings import UNICODE_ENCODING
 from lib.utils.timeout import timeout

@@ -53,7 +54,7 @@ def direct(query, content=True):

    if not select and "EXEC " not in query.upper():
        timeout(func=conf.dbmsConnector.execute, args=(query,), duration=conf.timeout, default=None)
-    elif not (output and "sqlmapoutput" not in query and "sqlmapfile" not in query):
+    elif not (output and ("%soutput" % TAKEOVER_TABLE_PREFIX) not in query and ("%sfile" % TAKEOVER_TABLE_PREFIX) not in query):
        output, state = timeout(func=conf.dbmsConnector.select, args=(query,), duration=conf.timeout, default=None)
        if state == TIMEOUT_STATE.NORMAL:
            hashDBWrite(query, output, True)
--- a/lib/request/dns.py
+++ b/lib/request/dns.py
@@ -1,10 +1,12 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2

 """
-Copyright (c) 2006-2018 sqlmap developers (http://sqlmap.org/)
+Copyright (c) 2006-2019 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """

+from __future__ import print_function
+
 import os
 import re
 import socket
@@ -145,13 +147,13 @@ if __name__ == "__main__":
                if _ is None:
                    break
                else:
-                    print "[i] %s" % _
+                    print("[i] %s" % _)

            time.sleep(1)

-    except socket.error, ex:
+    except socket.error as ex:
        if 'Permission' in str(ex):
-            print "[x] Please run with sudo/Administrator privileges"
+            print("[x] Please run with sudo/Administrator privileges")
        else:
            raise
    except KeyboardInterrupt:
--- a/Show More
+++ b/Show More