Blame: lib/request/comparison.py - sqlmapproject/sqlmap

Automatic SQL injection and database takeover tool

36950 0 0 Python

Last preparations for DREI 2019-05-08 12:47:52 +02:00			`#!/usr/bin/env python`
Minor enhancemet to support also --regexp, --excl-str and --excl-reg options rather than only --string when comparing HTTP responses page content 2008-12-05 15:34:13 +00:00
			`"""`
Patch for #5897 2025-05-08 23:54:39 +02:00			`Copyright (c) 2006-2025 sqlmap developers (https://sqlmap.org)`
Replacing doc/COPYING to LICENSE 2017-10-11 14:50:46 +02:00			`See the file 'LICENSE' for copying permission`
Minor enhancemet to support also --regexp, --excl-str and --excl-reg options rather than only --string when comparing HTTP responses page content 2008-12-05 15:34:13 +00:00			`"""`

Further pleasing pylint deity 2019-06-04 12:15:39 +02:00			`from __future__ import division`

Minor enhancemet to support also --regexp, --excl-str and --excl-reg options rather than only --string when comparing HTTP responses page content 2008-12-05 15:34:13 +00:00			`import re`

adding --titles switch (killer switch for pages with lots of dynamicity and/or international ones) 2011-06-10 23:18:43 +00:00			`from lib.core.common import extractRegexResult`
refactoring, code clearing and removal of obsolete switch --longest-common 2011-01-14 14:37:03 +00:00			`from lib.core.common import getFilteredPageContent`
Search for --string and --regexp matches also in HTTP response headers 2011-08-12 15:33:37 +00:00			`from lib.core.common import listToStrValue`
code refactoring 2010-12-04 10:13:18 +00:00			`from lib.core.common import removeDynamicContent`
Fixes #4671 2021-05-25 14:40:15 +02:00			`from lib.core.common import getLastRequestHTTPError`
Better naming 2013-01-29 20:53:11 +01:00			`from lib.core.common import wasLastResponseDBMSError`
			`from lib.core.common import wasLastResponseHTTPError`
Stabilizing DREI 2019-05-03 13:20:15 +02:00			`from lib.core.convert import getBytes`
Minor enhancemet to support also --regexp, --excl-str and --excl-reg options rather than only --string when comparing HTTP responses page content 2008-12-05 15:34:13 +00:00			`from lib.core.data import conf`
fix for Bug #165 2010-09-13 13:31:01 +00:00			`from lib.core.data import kb`
Major bug fix in the comparison algorithm to correctly handle also the case that the url is stable and the False response changes the page content very little. 2009-02-09 10:28:03 +00:00			`from lib.core.data import logger`
Doing some more style updating (capitalization of exception classes; using _ is enough for private members - __ is used in Python specific methods) 2012-12-06 14:14:19 +01:00			`from lib.core.exception import SqlmapNoneDataException`
fix for cases with retrieved binary files (preventing difflib nagging around comparison) 2011-05-25 20:54:30 +00:00			`from lib.core.settings import DEFAULT_PAGE_ENCODING`
another name change 2010-11-09 22:49:31 +00:00			`from lib.core.settings import DIFF_TOLERANCE`
adding --titles switch (killer switch for pages with lots of dynamicity and/or international ones) 2011-06-10 23:18:43 +00:00			`from lib.core.settings import HTML_TITLE_REGEX`
Trivial update 2019-06-04 14:44:06 +02:00			`from lib.core.settings import LOWER_RATIO_BOUND`
Definite patch for MemoryError(s) (fixes #1991) 2016-06-30 14:57:56 +02:00			`from lib.core.settings import MAX_DIFFLIB_SEQUENCE_LENGTH`
refactoring 2011-02-03 23:25:56 +00:00			`from lib.core.settings import MAX_RATIO`
Trivial update 2019-06-04 14:44:06 +02:00			`from lib.core.settings import MIN_RATIO`
Improving comparison engine (removing shared prelude part to further sharpen if pages are identical - especially noticable in small test pages) 2012-11-13 15:22:59 +01:00			`from lib.core.settings import REFLECTED_VALUE_MARKER`
minor refactoring/cosmetics 2010-12-24 11:06:57 +00:00			`from lib.core.settings import UPPER_RATIO_BOUND`
Bug fix in detection engine (abstract URI header sometimes caused problems - e.g. when automatic --string used) 2016-09-23 17:38:14 +02:00			`from lib.core.settings import URI_HTTP_HEADER`
slightly faster and thread safer inference 2011-01-16 10:52:42 +00:00			`from lib.core.threads import getCurrentThreadData`
Some more DREI stuff 2019-04-19 11:24:34 +02:00			`from thirdparty import six`
Minor enhancemet to support also --regexp, --excl-str and --excl-reg options rather than only --string when comparing HTTP responses page content 2008-12-05 15:34:13 +00:00
Added --code switch to match in boolean-based tests against the HTTP response code 2011-08-12 16:48:11 +00:00			`def comparison(page, headers, code=None, getRatioValue=False, pageLength=None):`
Nobody is reporting comparison bug, thus, changing behavior 2025-07-11 12:21:20 +02:00			`if not isinstance(page, (six.text_type, six.binary_type, type(None))):`
			`logger.critical("got page of type %s; repr(page)[:200]=%s" % (type(page), repr(page)[:200]))`

			`try:`
			`page = b"".join(page)`
			`except:`
			`page = six.text_type(page)`

			`_ = _adjust(_comparison(page, headers, code, getRatioValue, pageLength), getRatioValue)`
			`return _`
improvement toward proper implementation of OR-based injection by usage of "negative logic" mechanism 2012-03-29 14:33:27 +00:00
			`def _adjust(condition, getRatioValue):`
El Cosmeticado 2012-08-22 09:58:39 +02:00			`if not any((conf.string, conf.notString, conf.regexp, conf.code)):`
minor comment update 2012-03-29 14:35:47 +00:00			`# Negative logic approach is used in raw page comparison scheme as that what is "different" than original`
			`# PAYLOAD.WHERE.NEGATIVE response is considered as True; in switch based approach negative logic is not`
			`# applied as that what is by user considered as True is that what is returned by the comparison mechanism`
			`# itself`
Minor fix related to Issue #94 2012-07-16 16:06:39 +02:00			`retVal = not condition if kb.negativeLogic and condition is not None and not getRatioValue else condition`
improvement toward proper implementation of OR-based injection by usage of "negative logic" mechanism 2012-03-29 14:33:27 +00:00			`else:`
			`retVal = condition if not getRatioValue else (MAX_RATIO if condition else MIN_RATIO)`

			`return retVal`
disabling "negative logic" as it's not half done (it was "luckily" working for --string/--regex/--code but it was a sheer luck); removing "dirty fix" from checks.py; proof that this was not ready for the release is that there was not check for negative logic anywhere for anything more then --string/--regex/--code 2012-03-29 13:39:12 +00:00
			`def _comparison(page, headers, code, getRatioValue, pageLength):`
improving "boolean detection" by automatic recognition of convenient --string candidate 2012-04-10 21:48:34 +00:00			`threadData = getCurrentThreadData()`

Revert of previous commit (more care has to be done regarding headers dynamicity) 2013-01-18 16:49:35 +01:00			`if kb.testMode:`
Some more refactoring 2017-11-08 15:58:23 +01:00			`threadData.lastComparisonHeaders = listToStrValue(_ for _ in headers.headers if not _.startswith("%s:" % URI_HTTP_HEADER)) if headers else ""`
Revert of previous commit (more care has to be done regarding headers dynamicity) 2013-01-18 16:49:35 +01:00			`threadData.lastComparisonPage = page`
Minor update 2016-06-03 14:29:32 +02:00			`threadData.lastComparisonCode = code`
improving "boolean detection" by automatic recognition of convenient --string candidate 2012-04-10 21:48:34 +00:00
several bug fixes 2010-11-03 21:51:36 +00:00			`if page is None and pageLength is None:`
			`return None`

Revert of previous commit (more care has to be done regarding headers dynamicity) 2013-01-18 16:49:35 +01:00			`if any((conf.string, conf.notString, conf.regexp)):`
Some more refactoring 2017-11-08 15:58:23 +01:00			`rawResponse = "%s%s" % (listToStrValue(_ for _ in headers.headers if not _.startswith("%s:" % URI_HTTP_HEADER)) if headers else "", page)`
minor improvement for special cases with --string/--regexp 2011-06-10 23:05:47 +00:00
Minor patching (--not-string related) 2020-05-05 13:31:44 +02:00			`# String to match in page when the query is True`
Revert of previous commit (more care has to be done regarding headers dynamicity) 2013-01-18 16:49:35 +01:00			`if conf.string:`
			`return conf.string in rawResponse`
Search for --string and --regexp matches also in HTTP response headers 2011-08-12 15:33:37 +00:00
Minor patching (--not-string related) 2020-05-05 13:31:44 +02:00			`# String to match in page when the query is False`
Revert of previous commit (more care has to be done regarding headers dynamicity) 2013-01-18 16:49:35 +01:00			`if conf.notString:`
Minor patching (--not-string related) 2020-05-05 13:31:44 +02:00			`if conf.notString in rawResponse:`
			`return False`
			`else:`
			`if kb.errorIsNone and (wasLastResponseDBMSError() or wasLastResponseHTTPError()):`
			`return None`
			`else:`
			`return True`
Revert of previous commit (more care has to be done regarding headers dynamicity) 2013-01-18 16:49:35 +01:00
			`# Regular expression to match in page when the query is True and/or valid`
			`if conf.regexp:`
			`return re.search(conf.regexp, rawResponse, re.I \| re.M) is not None`
several bug fixes 2010-11-03 21:51:36 +00:00
first step toward negative logic support 2012-03-15 15:52:12 +00:00			`# HTTP code to match when the query is valid`
Patch for an Issue #304 2012-12-18 09:36:26 +01:00			`if conf.code:`
disabling "negative logic" as it's not half done (it was "luckily" working for --string/--regex/--code but it was a sheer luck); removing "dirty fix" from checks.py; proof that this was not ready for the release is that there was not check for negative logic anywhere for anything more then --string/--regex/--code 2012-03-29 13:39:12 +00:00			`return conf.code == code`
Added --code switch to match in boolean-based tests against the HTTP response code 2011-08-12 16:48:11 +00:00
Minor refactoring 2016-08-26 12:28:35 +02:00			`seqMatcher = threadData.seqMatcher`
			`seqMatcher.set_seq1(kb.pageTemplate)`

minor improvement for special cases with --string/--regexp 2011-06-10 23:05:47 +00:00			`if page:`
update (--string and --regex should be done regardless of wasLastRequestError) 2010-11-12 22:44:15 +00:00			`# In case of an DBMS error page return None`
Patch for negative logic (e.g. OR) cases (reported privately) 2015-08-18 03:09:01 +02:00			`if kb.errorIsNone and (wasLastResponseDBMSError() or wasLastResponseHTTPError()) and not kb.negativeLogic:`
Fixes #4671 2021-05-25 14:40:15 +02:00			`if not (wasLastResponseHTTPError() and getLastRequestHTTPError() in (conf.ignoreCode or [])):`
			`return None`
update (--string and --regex should be done regardless of wasLastRequestError) 2010-11-12 22:44:15 +00:00
dynamicity engine update 2010-11-07 00:12:00 +00:00			`# Dynamic content lines to be excluded before comparison`
refactoring, code clearing and removal of obsolete switch --longest-common 2011-01-14 14:37:03 +00:00			`if not kb.nullConnection:`
code refactoring 2010-12-04 10:13:18 +00:00			`page = removeDynamicContent(page)`
slightly faster and thread safer inference 2011-01-16 10:52:42 +00:00			`seqMatcher.set_seq1(removeDynamicContent(kb.pageTemplate))`
several bug fixes 2010-11-03 21:51:36 +00:00
			`if not pageLength:`
			`pageLength = len(page)`
fix for a bug reported by ToR (TypeError: unsupported operand type(s) for *: 'float' and 'NoneType') 2010-11-03 12:40:11 +00:00
removing obsolete switches (--threshold, --excl-reg, --excl-str) 2011-02-03 15:55:19 +00:00			`if kb.nullConnection and pageLength:`
minor JIC (just-in-case) update 2011-02-22 13:18:47 +00:00			`if not seqMatcher.a:`
Minor language fix 2013-07-31 09:24:34 +02:00			`errMsg = "problem occurred while retrieving original page content "`
error message adjustment 2011-04-21 22:31:02 +00:00			`errMsg += "which prevents sqlmap from continuation. Please rerun, "`
			`errMsg += "and if the problem persists turn off any optimization switches"`
Replacing old and deprecated raise Exception style (PEP8) 2013-01-03 23:20:55 +01:00			`raise SqlmapNoneDataException(errMsg)`
minor JIC (just-in-case) update 2011-02-22 13:18:47 +00:00
slightly faster and thread safer inference 2011-01-16 10:52:42 +00:00			`ratio = 1. * pageLength / len(seqMatcher.a)`
Cosmetics 2010-11-07 16:23:03 +00:00
implementation of HEAD/Range methods 2010-09-16 09:32:09 +00:00			`if ratio > 1.:`
			`ratio = 1. / ratio`
			`else:`
fix for cases with retrieved binary files (preventing difflib nagging around comparison) 2011-05-25 20:54:30 +00:00			`# Preventing "Unicode equal comparison failed to convert both arguments to Unicode"`
			`# (e.g. if one page is PDF and the other is HTML)`
Some more DREI stuff 2019-04-19 11:24:34 +02:00			`if isinstance(seqMatcher.a, six.binary_type) and isinstance(page, six.text_type):`
			`page = getBytes(page, kb.pageEncoding or DEFAULT_PAGE_ENCODING, "ignore")`
			`elif isinstance(seqMatcher.a, six.text_type) and isinstance(page, six.binary_type):`
Minor patch 2025-02-26 17:36:02 +01:00			`seqMatcher.set_seq1(getBytes(seqMatcher.a, kb.pageEncoding or DEFAULT_PAGE_ENCODING, "ignore"))`
fix for cases with retrieved binary files (preventing difflib nagging around comparison) 2011-05-25 20:54:30 +00:00
Fixes #2828 2017-12-13 10:46:46 +01:00			`if any(_ is None for _ in (page, seqMatcher.a)):`
			`return None`
			`elif seqMatcher.a and page and seqMatcher.a == page:`
			`ratio = 1.`
Definite patch for MemoryError(s) (fixes #1991) 2016-06-30 14:57:56 +02:00			`elif kb.skipSeqMatcher or seqMatcher.a and page and any(len(_) > MAX_DIFFLIB_SEQUENCE_LENGTH for _ in (seqMatcher.a, page)):`
Fixes #2828 2017-12-13 10:46:46 +01:00			`if not page or not seqMatcher.a:`
			`return float(seqMatcher.a == page)`
			`else:`
			`ratio = 1. * len(seqMatcher.a) / len(page)`
			`if ratio > 1:`
			`ratio = 1. / ratio`
minor fix by request 2011-06-17 16:58:50 +00:00			`else:`
Definite patch for MemoryError(s) (fixes #1991) 2016-06-30 14:57:56 +02:00			`seq1, seq2 = None, None`
minor fix by request 2011-06-17 16:58:50 +00:00
Definite patch for MemoryError(s) (fixes #1991) 2016-06-30 14:57:56 +02:00			`if conf.titles:`
			`seq1 = extractRegexResult(HTML_TITLE_REGEX, seqMatcher.a)`
			`seq2 = extractRegexResult(HTML_TITLE_REGEX, page)`
			`else:`
			`seq1 = getFilteredPageContent(seqMatcher.a, True) if conf.textOnly else seqMatcher.a`
			`seq2 = getFilteredPageContent(page, True) if conf.textOnly else page`
Improving comparison engine (removing shared prelude part to further sharpen if pages are identical - especially noticable in small test pages) 2012-11-13 15:22:59 +01:00
Definite patch for MemoryError(s) (fixes #1991) 2016-06-30 14:57:56 +02:00			`if seq1 is None or seq2 is None:`
			`return None`
Improving comparison engine (removing shared prelude part to further sharpen if pages are identical - especially noticable in small test pages) 2012-11-13 15:22:59 +01:00
Minor patch 2025-02-26 17:36:02 +01:00			`if isinstance(seq1, six.binary_type):`
			`seq1 = seq1.replace(REFLECTED_VALUE_MARKER.encode(), b"")`
			`elif isinstance(seq1, six.text_type):`
			`seq1 = seq1.replace(REFLECTED_VALUE_MARKER, "")`

			`if isinstance(seq2, six.binary_type):`
			`seq2 = seq2.replace(REFLECTED_VALUE_MARKER.encode(), b"")`
			`elif isinstance(seq2, six.text_type):`
			`seq2 = seq2.replace(REFLECTED_VALUE_MARKER, "")`
Patch for an Issue #1066 2014-12-26 22:24:28 +01:00
Performance improvement and reducing number of false-positives in heavily dynamic pages 2018-09-04 22:39:07 +02:00			`if kb.heavilyDynamic:`
Minor patch 2025-02-26 17:36:02 +01:00			`seq1 = seq1.split("\n" if isinstance(seq1, six.text_type) else b"\n")`
			`seq2 = seq2.split("\n" if isinstance(seq2, six.text_type) else b"\n")`
Performance improvement and reducing number of false-positives in heavily dynamic pages 2018-09-04 22:39:07 +02:00
Proper patch for #4591 2021-03-08 15:40:05 +01:00			`key = None`
			`else:`
			`key = (hash(seq1), hash(seq2))`

Definite patch for MemoryError(s) (fixes #1991) 2016-06-30 14:57:56 +02:00			`seqMatcher.set_seq1(seq1)`
			`seqMatcher.set_seq2(seq2)`
Improving comparison engine (removing shared prelude part to further sharpen if pages are identical - especially noticable in small test pages) 2012-11-13 15:22:59 +01:00
Trivial speedup 2021-03-03 23:28:27 +01:00			`if key in kb.cache.comparison:`
			`ratio = kb.cache.comparison[key]`
			`else:`
			`ratio = round(seqMatcher.quick_ratio() if not kb.heavilyDynamic else seqMatcher.ratio(), 3)`

Fixes #4591 2021-03-04 08:26:23 +01:00			`if key:`
			`kb.cache.comparison[key] = ratio`
fixed threading bug (difflib :) 2010-03-10 14:14:27 +00:00
Major bug fix in the comparison algorithm to correctly handle also the case that the url is stable and the False response changes the page content very little. 2009-02-09 10:28:03 +00:00			`# If the url is stable and we did not set yet the match ratio and the`
			`# current injected value changes the url page content`
code refactoring and some fixes 2010-12-18 09:51:34 +00:00			`if kb.matchRatio is None:`
Definite patch for MemoryError(s) (fixes #1991) 2016-06-30 14:57:56 +02:00			`if ratio >= LOWER_RATIO_BOUND and ratio <= UPPER_RATIO_BOUND:`
code refactoring and some fixes 2010-12-18 09:51:34 +00:00			`kb.matchRatio = ratio`
			`logger.debug("setting match ratio for current parameter to %.3f" % kb.matchRatio)`
sqlmap 0.8-rc3: Merge from Miroslav Stampar's branch fixing a bug when verbosity > 2, another major bug with urlencoding/urldecoding of POST data and Cookies, adding --drop-set-cookie option, implementing support to automatically decode gzip and deflate HTTP responses, support for Google dork page result (--gpage) and a minor code cleanup. 2010-01-02 02:02:12 +00:00
Update for an Issue #2384 2017-02-06 13:28:33 +01:00			`if kb.testMode:`
			`threadData.lastComparisonRatio = ratio`

Major bug fix in the comparison algorithm to correctly handle also the case that the url is stable and the False response changes the page content very little. 2009-02-09 10:28:03 +00:00			`# If it has been requested to return the ratio and not a comparison`
			`# response`
refactoring 2011-02-03 23:25:56 +00:00			`if getRatioValue:`
Major bug fix in the comparison algorithm to correctly handle also the case that the url is stable and the False response changes the page content very little. 2009-02-09 10:28:03 +00:00			`return ratio`

minor update 2010-12-27 18:27:42 +00:00			`elif ratio > UPPER_RATIO_BOUND:`
update of dynamicity testing and few misc fixes 2010-11-05 13:14:12 +00:00			`return True`

Patch for negative logic (e.g. OR) cases (reported privately) 2015-08-18 03:09:01 +02:00			`elif ratio < LOWER_RATIO_BOUND:`
			`return False`

code refactoring and some fixes 2010-12-18 09:51:34 +00:00			`elif kb.matchRatio is None:`
just a precaution (now i really need to go for a sleep) 2010-11-09 23:38:29 +00:00			`return None`

Major enhancement to make the comparison algorithm work properly also on url not stables automatically by using the difflib SequenceMatcher object: this changed a lot into the structure of the code, has to be extensively beta-tested! Please, do report bugs on sqlmap-users mailing list if you scout them. Cheers, Bernardo 2008-12-20 01:54:08 +00:00			`else:`
that CONSTANT_RATIO was a pure black magic for dynamic pages. now we have better injection detection workflow than before (False, True, False) and it was just a matter of time for removing this one 2011-04-17 21:58:34 +00:00			`return (ratio - kb.matchRatio) > DIFF_TOLERANCE`