kylefang/mitmproxy
1
0
Fork 0
mirror of https://github.com/zhigang1992/mitmproxy.git synced 2026-04-29 12:45:00 +08:00
Code Issues Packages Projects Releases Wiki Activity

move mitmproxy

Browse Source
This commit is contained in:
Maximilian Hils
2016-02-15 14:58:46 +01:00
parent 36f34f7019
commit 33fa49277a
444 changed files with 0 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
docs/features/anticache.rst
View File

@@ -1,15 +0,0 @@
.. _anticache:
Anticache
=========
When the :option:`--anticache` option is passed to mitmproxy, it removes headers
(``if-none-match`` and ``if-modified-since``) that might elicit a
``304 not modified`` response from the server. This is useful when you want to make
sure you capture an HTTP exchange in its totality. It's also often used during
:ref:`clientreplay`, when you want to make sure the server responds with complete data.
================== ======================
command-line :option:`--anticache`
mitmproxy shortcut :kbd:`o` then :kbd:`a`
================== ======================

18
docs/features/clientreplay.rst
View File

@@ -1,18 +0,0 @@
.. _clientreplay:
Client-side replay
==================
Client-side replay does what it says on the tin: you provide a previously saved
HTTP conversation, and mitmproxy replays the client requests one by one. Note
that mitmproxy serializes the requests, waiting for a response from the server
before starting the next request. This might differ from the recorded
conversation, where requests may have been made concurrently.
You may want to use client-side replay in conjunction with the
:ref:`anticache` option, to make sure the server responds with complete data.
================== =================
command-line :option:`-c path`
mitmproxy shortcut :kbd:`c`
================== =================

39
docs/features/filters.rst
View File

@@ -1,39 +0,0 @@
.. _filters:
Filter expressions
==================
Many commands in :program:`mitmproxy` and :program:`mitmdump` take a filter expression.
Filter expressions consist of the following operators:
.. documentedlist::
:header: "Expression" "Description"
:listobject: libmproxy.filt.help
- Regexes are Python-style
- Regexes can be specified as quoted strings
- Header matching (~h, ~hq, ~hs) is against a string of the form "name: value".
- Strings with no operators are matched against the request URL.
- The default binary operator is &.
Examples
--------
URL containing "google.com":
.. code-block:: none
google\.com
Requests whose body contains the string "test":
.. code-block:: none
~q ~b test
Anything but requests with a text/html content type:
.. code-block:: none
!(~q & ~t "text/html")

97
docs/features/passthrough.rst
View File

@@ -1,97 +0,0 @@
.. _passthrough:
Ignore Domains
==============
There are two main reasons why you may want to exempt some traffic from mitmproxy's interception
mechanism:
- **Certificate pinning:** Some traffic is is protected using `Certificate Pinning`_ and
mitmproxy's interception leads to errors. For example, the Twitter app, Windows Update or
the Apple App Store fail to work if mitmproxy is active.
- **Convenience:** You really don't care about some parts of the traffic and just want them to go
away.
If you want to peek into (SSL-protected) non-HTTP connections, check out the :ref:`tcpproxy`
feature.
If you want to ignore traffic from mitmproxy's processing because of large response bodies,
take a look at the :ref:`responsestreaming` feature.
How it works
------------
================== =============================
command-line :option:`--ignore regex`
mitmproxy shortcut :kbd:`o` then :kbd:`I`
================== =============================
mitmproxy allows you to specify a regex which is matched against a ``host:port`` string
(e.g. "example.com:443") to determine hosts that should be excluded.
There are two important quirks to consider:
- **In transparent mode, the ignore pattern is matched against the IP and ClientHello SNI host.** While we usually infer the
hostname from the Host header if the :option:`--host` argument is passed to mitmproxy, we do not
have access to this information before the SSL handshake. If the client uses SNI however, then we treat the SNI host as an ignore target.
- In regular mode, explicit HTTP requests are never ignored. [#explicithttp]_ The ignore pattern is
applied on CONNECT requests, which initiate HTTPS or clear-text WebSocket connections.
Tutorial
--------
If you just want to ignore one specific domain, there's usually a bulletproof method to do so:
1. Run mitmproxy or mitmdump in verbose mode (:option:`-v`) and observe the ``host:port``
information in the serverconnect messages. mitmproxy will filter on these.
2. Take the ``host:port`` string, surround it with ^ and $, escape all dots (. becomes \\.)
and use this as your ignore pattern:
.. code-block:: none
:emphasize-lines: 6,7,9
>>> mitmdump -v
127.0.0.1:50588: clientconnect
127.0.0.1:50588: request
-> CONNECT example.com:443 HTTP/1.1
127.0.0.1:50588: Set new server address: example.com:443
127.0.0.1:50588: serverconnect
-> example.com:443
^C
>>> mitmproxy --ignore ^example\.com:443$
Here are some other examples for ignore patterns:
.. code-block:: none
# Exempt traffic from the iOS App Store (the regex is lax, but usually just works):
--ignore apple.com:443
# "Correct" version without false-positives:
--ignore '^(.+\.)?apple\.com:443$'
# Ignore example.com, but not its subdomains:
--ignore '^example.com:'
# Ignore everything but example.com and mitmproxy.org:
--ignore '^(?!example\.com)(?!mitmproxy\.org)'
# Transparent mode:
--ignore 17\.178\.96\.59:443
# IP address range:
--ignore 17\.178\.\d+\.\d+:443
.. seealso::
- :ref:`tcpproxy`
- :ref:`responsestreaming`
.. rubric:: Footnotes
.. [#explicithttp] This stems from an limitation of explicit HTTP proxying:
A single connection can be re-used for multiple target domains - a
``GET http://example.com/`` request may be followed by a ``GET http://evil.com/`` request on the
same connection. If we start to ignore the connection after the first request,
we would miss the relevant second one.
.. _Certificate Pinning: https://security.stackexchange.com/questions/29988/what-is-certificate-pinning

17
docs/features/proxyauth.rst
View File

@@ -1,17 +0,0 @@
.. _proxyauth:
Proxy Authentication
====================
Asks the user for authentication before they are permitted to use the proxy.
Authentication headers are stripped from the flows, so they are not passed to
upstream servers. For now, only HTTP Basic authentication is supported. The
proxy auth options are not compatible with the transparent, socks or reverse proxy
mode.
================== =============================
command-line :option:`--nonanonymous`,
:option:`--singleuser USER`,
:option:`--htpasswd PATH`
================== =============================

72
docs/features/replacements.rst
View File

@@ -1,72 +0,0 @@
.. _replacements:
Replacements
============
Mitmproxy lets you specify an arbitrary number of patterns that define text
replacements within flows. Each pattern has 3 components: a filter that defines
which flows a replacement applies to, a regular expression that defines what
gets replaced, and a target value that defines what is substituted in.
Replace hooks fire when either a client request or a server response is
received. Only the matching flow component is affected: so, for example, if a
replace hook is triggered on server response, the replacement is only run on
the Response object leaving the Request intact. You control whether the hook
triggers on the request, response or both using the filter pattern. If you need
finer-grained control than this, it's simple to create a script using the
replacement API on Flow components.
Replacement hooks are extremely handy in interactive testing of applications.
For instance you can use a replace hook to replace the text "XSS" with a
complicated XSS exploit, and then "inject" the exploit simply by interacting
with the application through the browser. When used with tools like Firebug and
mitmproxy's own interception abilities, replacement hooks can be an amazingly
flexible and powerful feature.
On the command-line
-------------------
The replacement hook command-line options use a compact syntax to make it easy
to specify all three components at once. The general form is as follows:
.. code-block:: none
/patt/regex/replacement
Here, **patt** is a mitmproxy filter expression, **regex** is a valid Python
regular expression, and **replacement** is a string literal. The first
character in the expression (``/`` in this case) defines what the separation
character is. Here's an example of a valid expression that replaces "foo" with
"bar" in all requests:
.. code-block:: none
:~q:foo:bar
In practice, it's pretty common for the replacement literal to be long and
complex. For instance, it might be an XSS exploit that weighs in at hundreds or
thousands of characters. To cope with this, there's a variation of the
replacement hook specifier that lets you load the replacement text from a file.
So, you might start **mitmdump** as follows:
>>> mitmdump --replace-from-file :~q:foo:~/xss-exploit
This will load the replacement text from the file ``~/xss-exploit``.
Both the :option:`--replace` and :option:`--replace-from-file` flags can be passed multiple
times.
Interactively
-------------
The :kbd:`R` shortcut key in the mitmproxy options menu (:kbd:`o`) lets you add and edit
replacement hooks using a built-in editor. The context-sensitive help (:kbd:`?`) has
complete usage information.
================== =============================
command-line :option:`--replace`,
:option:`--replace-from-file`
mitmproxy shortcut :kbd:`o` then :kbd:`R`
================== =============================

69
docs/features/responsestreaming.rst
View File

@@ -1,69 +0,0 @@
.. _responsestreaming:
Response Streaming
==================
By using mitmproxy's streaming feature, response contents can be passed to the client incrementally
before they have been fully received by the proxy. This is especially useful for large binary files
such as videos, where buffering the whole file slows down the client's browser.
By default, mitmproxy will read the entire response, perform any indicated
manipulations on it and then send the (possibly modified) response to
the client. In some cases this is undesirable and you may wish to "stream"
the response back to the client. When streaming is enabled, the response is
not buffered on the proxy but directly sent back to the client instead.
On the command-line
-------------------
Streaming can be enabled on the command line for all response bodies exceeding a certain size.
The SIZE argument understands k/m/g suffixes, e.g. 3m for 3 megabytes.
================== =============================
command-line :option:`--stream SIZE`
================== =============================
.. warning::
When response streaming is enabled, **streamed response contents will not be
recorded or preserved in any way.**
.. note::
When response streaming is enabled, the response body cannot be modified by the usual means.
Customizing Response Streaming
------------------------------
You can also use an :ref:`inlinescripts` to customize exactly
which responses are streamed.
Responses that should be tagged for streaming by setting their ``.stream`` attribute to ``True``:
.. literalinclude:: ../../examples/stream.py
:caption: examples/stream.py
:language: python
Implementation Details
----------------------
When response streaming is enabled, portions of the code which would have otherwise performed
changes on the response body will see an empty response body instead
(:py:data:`netlib.http.CONTENT_MISSING`). Any modifications will be ignored.
Streamed responses are usually sent in chunks of 4096 bytes. If the response is sent with a
``Transfer-Encoding: chunked`` header, the response will be streamed one chunk at a time.
Modifying streamed data
-----------------------
If the ``.stream`` attribute is callable, ``.stream`` will wrap the generator that yields all
chunks.
.. literalinclude:: ../../examples/stream_modify.py
:caption: examples/stream_modify.py
:language: python
.. seealso::
- :ref:`passthrough`

57
docs/features/reverseproxy.rst
View File

@@ -1,57 +0,0 @@
.. _reverseproxy:
Reverse Proxy
=============
In reverse proxy mode, mitmproxy accepts standard HTTP(S) requests and forwards
them to the specified upstream server. This is in contrast to :ref:`upstreamproxy`, in which
mitmproxy forwards HTTP(S) proxy requests to an upstream proxy server.
================== =====================================
command-line :option:`-R http[s]://hostname[:port]`
================== =====================================
Here, **http[s]** signifies if the proxy should use TLS to connect to the server.
mitmproxy always accepts both encrypted and unencrypted requests and transforms
them to what the server expects.
.. code-block:: none
>>> mitmdump -R https://httpbin.org -p 80
>>> curl http://localhost/
# requests will be transparently upgraded to TLS by mitmproxy
>>> mitmdump -R https://httpbin.org -p 443
>>> curl https://localhost/
# mitmproxy will use TLS on both ends.
Host Header
-----------
In reverse proxy mode, mitmproxy does not rewrite the host header. While often useful, this
may lead to issues with public web servers. For example, consider the following scenario:
.. code-block:: none
:emphasize-lines: 5
>>> mitmdump -d -R http://example.com/
>>> curl http://localhost:8080/
>> GET https://example.com/
Host: localhost:8080
User-Agent: curl/7.35.0
[...]
<< 404 Not Found 345B
Since the Host header doesn't match "example.com", an error is returned.
There are two ways to solve this:
1. Modify the hosts file of your OS so that "example.com" resolves to your proxy's IP.
Then, access example.com directly. Make sure that your proxy can still resolve the original IP
or specify an IP in mitmproxy.
2. Use mitmproxy's :ref:`setheaders` feature to rewrite the host header:
``--setheader :~q:Host:example.com``.
However, keep in mind that absolute URLs within the returned document or HTTP redirects will
cause the client application to bypass the proxy.

39
docs/features/serverreplay.rst
View File

@@ -1,39 +0,0 @@
.. _serverreplay:
Server-side replay
==================
Server-side replay lets us replay server responses from a saved HTTP
conversation.
Matching requests with responses
--------------------------------
By default, :program:`mitmproxy` excludes request headers when matching incoming
requests with responses from the replay file. This works in most circumstances,
and makes it possible to replay server responses in situations where request
headers would naturally vary, e.g. using a different user agent.
The :option:`--rheader headername` command-line option allows you to override
this behaviour by specifying individual headers that should be included in matching.
Response refreshing
-------------------
Simply replaying server responses without modification will often result in
unexpected behaviour. For example cookie timeouts that were in the future at
the time a conversation was recorded might be in the past at the time it is
replayed. By default, :program:`mitmproxy` refreshes server responses before sending
them to the client. The **date**, **expires** and **last-modified** headers are
all updated to have the same relative time offset as they had at the time of
recording. So, if they were in the past at the time of recording, they will be
in the past at the time of replay, and vice versa. Cookie expiry times are
updated in a similar way.
You can turn off response refreshing using the :option:`--norefresh` argument, or using
the :kbd:`o` options shortcut within :program:`mitmproxy`.
================== =================
command-line :option:`-S path`
mitmproxy shortcut :kbd:`S`
================== =================

19
docs/features/setheaders.rst
View File

@@ -1,19 +0,0 @@
.. _setheaders:
Set Headers
===========
This feature lets you specify a set of headers to be added to requests or
responses, based on a filter pattern. You can specify these either on the
command-line, or through an interactive editor in mitmproxy.
Example: Set the **Host** header to "example.com" for all requests.
.. code-block:: none
mitmdump -R http://example.com --setheader :~q:Host:example.com
================== =============================
command-line :option:`--setheader PATTERN`
mitmproxy shortcut :kbd:`o` then :kbd:`H`
================== =============================

10
docs/features/socksproxy.rst
View File

@@ -1,10 +0,0 @@
.. _socksproxy:
SOCKS Mode
==========
In this mode, mitmproxy acts as a SOCKS5 proxy server.
================== =================
command-line :option:`--socks`
================== =================

41
docs/features/sticky.rst
View File

@@ -1,41 +0,0 @@
.. _sticky:
Sticky cookies and auth
=======================
Sticky cookies
--------------
When the sticky cookie option is set, __mitmproxy__ will add the cookie most
recently set by the server to any cookie-less request. Consider a service that
sets a cookie to track the session after authentication. Using sticky cookies,
you can fire up mitmproxy, and authenticate to a service as you usually would
using a browser. After authentication, you can request authenticated resources
through mitmproxy as if they were unauthenticated, because mitmproxy will
automatically add the session tracking cookie to requests. Among other things,
this lets you script interactions with authenticated resources (using tools
like wget or curl) without having to worry about authentication.
Sticky cookies are especially powerful when used in conjunction with :ref:`clientreplay` - you can
record the authentication process once, and simply replay it on startup every time you need
to interact with the secured resources.
================== ======================
command-line :option:`-t FILTER`
mitmproxy shortcut :kbd:`o` then :kbd:`t`
================== ======================
Sticky auth
-----------
The sticky auth option is analogous to the sticky cookie option, in that HTTP
**Authorization** headers are simply replayed to the server once they have been
seen. This is enough to allow you to access a server resource using HTTP Basic
authentication through the proxy. Note that :program:`mitmproxy` doesn't (yet) support
replay of HTTP Digest authentication.
================== ======================
command-line :option:`-u FILTER`
mitmproxy shortcut :kbd:`o` then :kbd:`A`
================== ======================

31
docs/features/tcpproxy.rst
View File

@@ -1,31 +0,0 @@
.. _tcpproxy:
TCP Proxy
=========
WebSockets or other non-HTTP protocols are not supported by mitmproxy yet. However, you can exempt
hostnames from processing, so that mitmproxy acts as a generic TCP forwarder.
This feature is closely related to the :ref:`passthrough` functionality,
but differs in two important aspects:
- The raw TCP messages are printed to the event log.
- SSL connections will be intercepted.
Please note that message interception or modification are not possible yet.
If you are not interested in the raw TCP messages, you should use the ignore domains feature.
How it works
------------
================== ======================
command-line :option:`--tcp HOST`
mitmproxy shortcut :kbd:`o` then :kbd:`T`
================== ======================
For a detailed description how the hostname pattern works, please look at the :ref:`passthrough`
feature.
.. seealso::
- :ref:`passthrough`
- :ref:`responsestreaming`

23
docs/features/upstreamcerts.rst
View File

@@ -1,23 +0,0 @@
.. _upstreamcerts:
Upstream Certificates
=====================
When mitmproxy receives a connection destined for an SSL-protected service, it
freezes the connection before reading its request data, and makes a connection
to the upstream server to "sniff" the contents of its SSL certificate. The
information gained - the **Common Name** and **Subject Alternative Names** - is
then used to generate the interception certificate, which is sent to the client
so the connection can continue.
This rather intricate little dance lets us seamlessly generate correct
certificates even if the client has specified only an IP address rather than the
hostname. It also means that we don't need to sniff additional data to generate
certs in transparent mode.
Upstream cert sniffing is on by default, and can optionally be turned off.
================== =============================
command-line :option:`--no-upstream-cert`
mitmproxy shortcut :kbd:`o` then :kbd:`U`
================== =============================

12
docs/features/upstreamproxy.rst
View File

@@ -1,12 +0,0 @@
.. _upstreamproxy:
Upstream proxy mode
===================
In this mode, mitmproxy accepts proxy requests and unconditionally forwards all
requests to a specified upstream proxy server. This is in contrast to :ref:`reverseproxy`,
in which mitmproxy forwards ordinary HTTP requests to an upstream server.
================== ===================================
command-line :option:`-U http://hostname[:port]`
================== ===================================