Final prep for 0.7.

This gives a big boost to scroll performance for the flow list.

CHANGELOG

@@ -1,3 +1,33 @@
+
+20 February 2012: mitmproxy 0.7:
+
+    * New built-in key/value editor. This lets you interactively edit URL query
+      strings, headers and URL-encoded form data. 
+
+    * Extend script API to allow duplication and replay of flows.
+    
+    * API for easy manipulation of URL-encoded forms and query strings.
+    
+    * Add "D" shortcut in mitmproxy to duplicate a flow.
+    
+    * Reverse proxy mode. In this mode mitmproxy acts as an HTTP server,
+      forwarding all traffic to a specified upstream server.
+
+    * UI improvements - use unicode characters to make GUI more compact,
+      improve spacing and layout throughout.
+
+    * Add support for filtering by HTTP method.
+   
+    * Add the ability to specify an HTTP body size limit.
+    
+    * Move to typed netstrings for serialization format - this makes 0.7
+      backwards-incompatible with serialized data from 0.6!
+
+    * Significant improvements in speed and responsiveness of UI. 
+
+    * Many minor bugfixes and improvements.
+
+
 7 August 2011: mitmproxy 0.6:
 
     * New scripting API that allows much more flexible and fine-grained
@@ -92,6 +122,3 @@
     * "A" will now accept all intercepted connections
     
     * Lots of bugfixes
-
-    
-

CONTRIBUTORS

@@ -1,8 +1,15 @@
-    278 Aldo Cortesi
-     18 Henrik Nordstrom
-     13 Thomas Roth
-     10 Stephen Altamirano
-      2 alts
-      1 Yuangxuan Wang
-      1 Henrik Nordström
-      1 Felix Wolfsteller
+ 395 Aldo Cortesi
+  18 Henrik Nordstrom
+  13 Thomas Roth
+  11 Stephen Altamirano
+   5 András Veres-Szentkirályi
+   2 alts
+   2 Mark E. Haase
+   2 Heikki Hannikainen
+   1 meeee
+   1 capt8bit
+   1 Yuangxuan Wang
+   1 Rune Halvorsen
+   1 Rory McCann
+   1 Henrik Nordström
+   1 Felix Wolfsteller

MANIFEST.in

@@ -1,7 +1,8 @@
 include LICENSE
 include CHANGELOG
 include CONTRIBUTORS
-include README.mkd
+include README.txt
+exclude README.mkd
 recursive-include examples *
 recursive-include doc *
 recursive-include test *

README.mkd

@@ -1,23 +1,23 @@
-
 __mitmproxy__ is an SSL-capable man-in-the-middle proxy for HTTP. It provides a
 console interface that allows traffic flows to be inspected and edited on the
 fly.
 
 __mitmdump__ is the command-line version of mitmproxy, with the same
-functionality but without the frills. Think tcpdump for HTTP.
+functionality but without the user interface. Think tcpdump for HTTP.
 
-Both tools are fully documentented in the commandline _--help_ flag, and, in
-the case of __mitmproxy__, a built-in help page accessible through the _?_
-keyboard shortcut.
+Complete documentation and a set of practical tutorials is included in the
+distribution package, and is also available at
+[mitmproxy.org](http://mitmproxy.org).
 
 
-Capabilities
-------------
+Features
+--------
 
 - Intercept HTTP requests and responses and modify them on the fly.
 - Save complete HTTP conversations for later replay and analysis.
 - Replay the client-side of an HTTP conversations.
 - Replay HTTP responses of a previously recorded server.
+- Reverse proxy mode to forward traffic to a specified server.
 - Make scripted changes to HTTP traffic using Python. 
 - SSL certificates for interception are generated on the fly.
 

README.txt

@@ -0,0 +1,74 @@
+**mitmproxy** is an SSL-capable man-in-the-middle proxy for HTTP. It provides a
+console interface that allows traffic flows to be inspected and edited on the
+fly.
+
+**mitmdump** is the command-line version of mitmproxy, with the same
+functionality but without the user interface. Think tcpdump for HTTP.
+
+Complete documentation and a set of practical tutorials is included in the
+distribution package, and is also available at mitmproxy.org_.
+
+
+Features
+--------
+
+- Intercept HTTP requests and responses and modify them on the fly.
+- Save complete HTTP conversations for later replay and analysis.
+- Replay the client-side of an HTTP conversations.
+- Replay HTTP responses of a previously recorded server.
+- Reverse proxy mode to forward traffic to a specified server.
+- Make scripted changes to HTTP traffic using Python. 
+- SSL certificates for interception are generated on the fly.
+
+
+Download
+--------
+
+Releases and rendered documentation can be found on the mitmproxy website:
+
+mitmproxy.org_
+
+Source is hosted on github: 
+
+`github.com/cortesi/mitmproxy`_
+
+
+Community
+---------
+
+Come join us in the #mitmproxy channel on the OFTC IRC network
+(irc://irc.oftc.net:6667).
+
+We also have a mailing list, hosted here:
+
+http://groups.google.com/group/mitmproxy
+
+
+Requirements
+------------
+
+* Python_ 2.6.x or 2.7.x.
+* openssl_ - installed by default on most systems.
+* urwid_  version 0.9.8 or newer.
+* The test suite uses the pry_ unit testing
+  library.
+* Rendering the documentation requires countershape_.
+
+**mitmproxy** is tested and developed on OSX, Linux and OpenBSD. 
+
+You should also make sure that your console environment is set up with the
+following: 
+    
+* EDITOR environment variable to determine the external editor.
+* PAGER environment variable to determine the external pager.
+* Appropriate entries in your mailcap files to determine external
+  viewers for request and response contents.
+
+.. _mitmproxy.org: http://mitmproxy.org
+.. _github.com/cortesi/mitmproxy: http://github.com/cortesi/mitmproxy
+.. _python: http://www.python.org
+.. _openssl: http://www.openssl.org/
+.. _urwid: http://excess.org/urwid/
+.. _pry: http://github.com/cortesi/pry
+.. _countershape: http://github.com/cortesi/countershape
+

doc-src/_layout.html

@@ -14,7 +14,7 @@
         </div>
         <!--(end)-->
         $!nav if this.title!="docs" else ""!$
-        <h1><a href="@!urlTo("/index.html")!@">mitmproxy 0.6 docs</a></h1>
+        <h1><a href="@!urlTo("/index.html")!@">mitmproxy 0.7 docs</a></h1>
     </div>
     <div id="bd">
         <div id="yui-main">

doc-src/_websitelayout.html

@@ -1,7 +1,6 @@
 <a href="http://github.com/cortesi/mitmproxy"><img style="position: absolute; top: 0; right: 0; border: 0;" src="https://d3nwyuy0nl342s.cloudfront.net/img/e6bef7a091f5f3138b8cd40bc3e114258dd68ddf/687474703a2f2f73332e616d617a6f6e6177732e636f6d2f6769746875622f726962626f6e732f666f726b6d655f72696768745f7265645f6161303030302e706e67" alt="Fork me on GitHub"></a>
 <div class="yui-t7" id="doc">
     <div style="" id="hd">
-        <h1><a href="@!urlTo("/index.html")!@">mitmproxy</a> </h1>
         <div class="HorizontalNavBar">
             <ul>
                 <li class="inactive"><a href="@!urlTo("/index.html")!@">home</a></li>
@@ -9,6 +8,7 @@
                 <li class="inactive"><a href="@!urlTo("/about.html")!@">about</a></li>
             </ul>
         </div>
+        <h1><a href="@!urlTo("/index.html")!@">mitmproxy</a> </h1>
         <br>
         <p>an SSL-capable man-in-the-middle proxy</p>
     </div>
@@ -29,7 +29,7 @@
                 </div>
                 <!--(end)-->
                 $!nav if this.title!="docs" else ""!$
-                $!title if this.title!="docs" else "<h1>mitmproxy 0.6 docs</h1>"!$
+                $!title if this.title!="docs" else "<h1>mitmproxy 0.7 docs</h1>"!$
                 $!body!$
             </div>
         </div>

doc-src/faq.html

@@ -1,5 +1,5 @@
 
-### Any tips for running mitmproxy on OSX?
+## Any tips for running mitmproxy on OSX?
 
 You can use the OSX <b>open</b> program to create a simple and effective
 <b>~/.mailcap</b> file to view HTTP bodies:
@@ -12,7 +12,7 @@ video/*; /usr/bin/open -Wn %s
 </pre>
 
 
-### I'd like to hack on mitmproxy. What should I work on?
+## I'd like to hack on mitmproxy. What should I work on?
 
 There's a __todo__ file at the top of the source tree that outlines a variety
 of tasks, from simple to complex. If you don't have your own itch, feel free to

doc-src/filters.html

@@ -26,9 +26,9 @@ URL containing "google.com":
 
 Requests whose body contains the string "test":
 
-    ~r ~b test
+    ~q ~b test
 
 Anything but requests with a text/html content type:
 
-    !(~r & ~t \"text/html\")
+    !(~q & ~t \"text/html\")
 

doc-src/index.html

@@ -3,11 +3,12 @@
     <li><a href="@!urlTo("intro.html")!@">Introduction</a></li>
     <li><a href="@!urlTo("mitmproxy.html")!@">mitmproxy</a></li>
     <li><a href="@!urlTo("mitmdump.html")!@">mitmdump</a></li>
-    <li>Concepts</li>
+    <li>Features</li>
     <ul>
         <li><a href="@!urlTo("clientreplay.html")!@">Client-side replay</a></li>
         <li><a href="@!urlTo("serverreplay.html")!@">Server-side replay</a></li>
         <li><a href="@!urlTo("sticky.html")!@">Sticky cookies and auth</a></li>
+        <li><a href="@!urlTo("reverseproxy.html")!@">Reverse proxy mode</a></li>
         <li><a href="@!urlTo("anticache.html")!@">Anticache</a></li>
         <li><a href="@!urlTo("filters.html")!@">Filter expressions</a></li>
     </ul>

doc-src/index.py

@@ -17,7 +17,7 @@ else:
 this.markup = markup.Markdown()
 ns.docMaintainer = "Aldo Cortesi"
 ns.docMaintainerEmail = "aldo@corte.si"
-ns.copyright = u"\u00a9 mitmproxy project, 2011"
+ns.copyright = u"\u00a9 mitmproxy project, 2012"
 
 ns.index = countershape.widgets.SiblingPageIndex('/index.html', divclass="pageindex")
 
@@ -62,7 +62,7 @@ filt_help.extend(
     ]
 )
 ns.filt_help = filt_help
- 
+
 
 
 pages = [
@@ -73,6 +73,7 @@ pages = [
     Page("clientreplay.html", "Client-side replay"),
     Page("serverreplay.html", "Server-side replay"),
     Page("sticky.html", "Sticky cookies and auth"),
+    Page("reverseproxy.html", "Reverse proxy mode"),
     Page("anticache.html", "Anticache"),
     Page("filters.html", "Filter expressions"),
     Page("scripts.html", "Scripts"),

doc-src/mitmproxy.html

@@ -1,28 +1,79 @@
 
 __mitmproxy__ is a console tool that allows interactive examination and
-modification of HTTP traffic. The _?_ shortcut key shows complete documentation
-on __mitmproxy__'s functionality.
+modification of HTTP traffic. Use the _?_ shortcut key to view,
+context-sensitive documentation from any __mitmproxy__ screen.
 
+## Flow list
 
-## The interface: connection list
+The flow list shows an index of captured flows in chronological order. 
 
 <img src="@!urlTo("screenshots/mitmproxy.png")!@"/>
 
-
-The connection list shows an index of captured flows in chronological order.
-So, in this case, we can we can see that we visited __gmail.com__, which then
-returned a 301 redirect to mail.google.com.
-
-The statusbar at the bottom tells us that there are 11 flows in the view, that
-we are using the "pretty" view mode (more on that below), and that the proxy is
-bound to port 8080 of all interfaces.
-
-Also visible is the __Event log__, which can be toggled on and off with the _v_
-keyboard shortcut. This displays events like client connection information,
-errors, and script output.
+- __1__: A GET request, returning a 302 Redirect response.
+- __2__: A GET request, returning 16.75kb of text/html data.
+- __3__: A replayed request. 
+- __4__: Intercepted flows are indicated with orange text. The user may edit
+these flows, and then accept them (using the _a_ key) to continue. In this
+case, the request has been intercepted on the way to the server.
+- __5__: A response intercepted from the server on the way to the client.
+- __6__: The event log can be toggled on and off using the _e_ shorcut key. This
+pane shows events and errors that may not result in a flow that shows up in the
+flow pane.
+- __7__: Flow count.
+- __8__: Various information on mitmproxy's state. In this case, we have an
+interception pattern set to ".*".
+- __9__: Bind address indicator - mitmproxy is listening on port 8080 of all
+interfaces.
 
 
-## Example: Interception
+## Flow view
+
+The __Flow View__ lets you inspect and manipulate a single flow:
+
+<img src="@!urlTo("screenshots/mitmproxy-flowview.png")!@"/>
+
+- __1__: Flow summary.
+- __2__: The Request/Response tabs, showing you which part of the flow you are
+currently viewing. In the example above, we're viewing the Response. Hit _tab_
+to switch between the Response and the Request.
+- __3__: Headers.
+- __4__: Body.
+- __5__: View Mode indicator. In this case, we're viewing the body in __hex__
+mode. The other available modes are __pretty__, which uses a number of
+heuristics to show you a friendly view of various content types, and __raw__,
+which shows you exactly what's there without any changes. You can change modes
+using the _m_ key.
+
+
+## Key/Value Editor
+
+It turns out that ordered key/value data is pervasive in HTTP communications,
+so mitmproxy has a built-in editor to help edit and create this kind of data.
+There are three ways to reach the __K/V Editor__ from the __Flow View__ screen:
+
+- Editing request or response headers (_e_ for edit, then _h_ for headers) 
+- Editing a query string (_e_ for edit, then _q_ for query)
+- Editing a URL-encoded form (_e_ for edit, then _f_ for form)
+
+If there is is no form or query string, an empty __K/V Editor__ will be started
+to let you add one. Here is the __K/V Editor__ showing the headers from a
+request:
+
+<img src="@!urlTo("screenshots/mitmproxy-kveditor.png")!@"/>
+
+To edit, navigate to the key or value you want to modify using the arrow or vi
+navigation keys, and press enter. The background color will change to show that
+you are in edit mode for the specified field:
+
+<img src="@!urlTo("screenshots/mitmproxy-kveditor-editmode.png")!@"/>
+
+Modify the field as desired, and press escape or enter to exit edit mode when
+you're done. You can also add a key/value pair (_a_ key), delete a pair (_d_
+key), spawn an external editor on a field (_e_ key). Be sure to consult the
+context-sensitive help (_?_ key) for more.
+
+
+# Example: Interception
 
 __mitmproxy__'s interception functionality lets you pause an HTTP request or
 response, inspect and modify it, and then accept it to send it on to the server
@@ -31,28 +82,28 @@ or client.
 
 ### 1: Set an interception pattern
 
-<img src="@!urlTo('intercept-filt.png')!@"/>
+<img src="@!urlTo('mitmproxy-intercept-filt.png')!@"/>
 
 We press _i_ to set an interception pattern. In this case, the __~q__ filter
 pattern tells __mitmproxy__ to intercept all requests. For complete filter
 syntax, see the [Filter expressions](@!urlTo("filters.html")!@) section of this
 document, or the built-in help function in __mitmproxy__.
 
-### 2: Intercepted connections are indicated with a red exclamation mark:
+### 2: Intercepted connections are indicated with orange text:
 
-<img src="@!urlTo('intercept-mid.png')!@"/>
+<img src="@!urlTo('mitmproxy-intercept-mid.png')!@"/>
 
 ### 3: You can now view and modify the request:
 
-<img src="@!urlTo('intercept-options.png')!@"/>
+<img src="@!urlTo('mitmproxy-intercept-options.png')!@"/>
 
 In this case, we viewed the request by selecting it, pressed _e_ for "edit"
 and _m_ for "method" to change the HTTP request method.
 
-### 4: Accept the intercept to continue
+### 4: Accept the intercept to continue:
 
-<img src="@!urlTo('intercept-result.png')!@"/>
+<img src="@!urlTo('mitmproxy-intercept-result.png')!@"/>
 
 Finally, we press _a_ to accept the modified request, which is then sent on to
-the server. In this case, we changed the request from an HTTP GET to to
+the server. In this case, we changed the request from an HTTP GET to
 OPTIONS, and Google's server has responded with a 405 "Method not allowed". 

doc-src/reverseproxy.html

@@ -0,0 +1,8 @@
+
+- command-line: _-R_ http[s]://hostname[:port]
+- mitmproxy shortcut: _R_
+
+In reverse proxy mode, mitmproxy acts as a standard HTTP server and forwards
+all requests to the specified upstream server. Note that the displayed URL for
+flows in this mode will use the value of the __Host__ header field from the
+request, not the reverse proxy server.

doc-src/scripts.html

@@ -62,26 +62,6 @@ Called once on script shutdown, after any other events.
 The main classes you will deal with in writing mitmproxy scripts are:
 
 <table class="kvtable">
-    <tr>
-        <th>libmproxy.flow.ScriptContext</th>
-        <td>A handle for interacting with mitmproxy's global state.</td>
-    </tr>
-    <tr>
-        <th>libmproxy.flow.Flow</th>
-        <td>A collection of objects representing a single HTTP transaction.</td>
-    </tr>
-    <tr>
-        <th>libmproxy.flow.Request</th>
-        <td>An HTTP request.</td>
-    </tr>
-    <tr>
-        <th>libmproxy.flow.Response</th>
-        <td>An HTTP response.</td>
-    </tr>
-    <tr>
-        <th>libmproxy.flow.Error</th>
-        <td>A communications error.</td>
-    </tr>
     <tr>
         <th>libmproxy.flow.ClientConnection</th>
         <td>Describes a client connection.</td>
@@ -90,12 +70,41 @@ The main classes you will deal with in writing mitmproxy scripts are:
         <th>libmproxy.flow.ClientDisconnection</th>
         <td>Describes a client disconnection.</td>
     </tr>
+    <tr>
+        <th>libmproxy.flow.Error</th>
+        <td>A communications error.</td>
+    </tr>
+    <tr>
+        <th>libmproxy.flow.Flow</th>
+        <td>A collection of objects representing a single HTTP transaction.</td>
+    </tr>
     <tr>
         <th>libmproxy.flow.Headers</th>
         <td>HTTP headers for a request or response.</td>
     </tr>
-</table>
+    <tr>
+        <th>libmproxy.flow.ODict</th>
 
+        <td>A dictionary-like object for managing sets of key/value data. There
+        is also a variant called CaselessODict that ignores key case for some
+        calls.</td>
+    </tr>
+    <tr>
+        <th>libmproxy.flow.Response</th>
+        <td>An HTTP response.</td>
+    </tr>
+    <tr>
+        <th>libmproxy.flow.Request</th>
+        <td>An HTTP request.</td>
+    </tr>
+    <tr>
+        <th>libmproxy.flow.ScriptContext</th>
+
+
+        <td> A handle for interacting with mitmproxy's from within scripts.  </td>
+
+    </tr>
+</table>
 
 The canonical API documentation is the code. You can view the API documentation
 using pydoc (which is installed with Python by default), like this:

doc-src/tutorials/30second.html

@@ -49,13 +49,13 @@ voila! - totally hands-free wireless network startup.
 
 We might also want to prune requests that download CSS, JS, images and so
 forth. These add only a few moments to the time it takes to replay, but they're
-not really needed and I somehow feel compelled trim them anyway. So, we fire up
+not really needed and I somehow feel compelled to trim them anyway. So, we fire up
 the mitmproxy console tool on our serialized conversation, like so:
 
 <pre class="terminal">
-> mitmproxy wireless-login
+> mitmproxy -r wireless-login
 </pre>
 
 We can now go through and manually delete (using the __d__ keyboard shortcut)
-everything we want to trim. When we're done, we use __S__ to save the
+everything we want to trim. When we're done, we use __w__ to save the
 conversation back to the file.

examples/README

@@ -1,5 +1,8 @@
-
-add_header.py   Simple script that just adds a header to every request.
-stub.py         Script stub with a method definition for every event. 
-stickycookies   An example of writing a custom proxy with libmproxy
-
+add_header.py           Simple script that just adds a header to every request.
+dup_and_replay.py       Duplicates each request, changes it, and then replays the modified request.
+flowbasic               Basic use of mitmproxy as a library.
+modify_form.py          Modify all form submissions to add a parameter. 
+modify_querystring.py   Modify all query strings to add a parameters.
+stub.py                 Script stub with a method definition for every event. 
+stickycookies           An example of writing a custom proxy with libmproxy.
+upsidedownternet.py     Rewrites traffic to turn PNGs upside down.

examples/dup_and_replay.py

@@ -0,0 +1,4 @@
+def request(ctx, flow):
+   f = ctx.duplicate_flow(flow)
+   f.request.path = "/changed"
+   ctx.replay_request(f)

examples/flowbasic

@@ -0,0 +1,39 @@
+#!/usr/bin/env python
+"""
+    This example shows how to build a proxy based on mitmproxy's Flow
+    primitives.
+
+    Note that request and response messages are not automatically acked, so we
+    need to implement handlers to do this.
+"""
+import os
+from libmproxy import proxy, flow
+
+class MyMaster(flow.FlowMaster):
+    def run(self):
+        try:
+            flow.FlowMaster.run(self)
+        except KeyboardInterrupt:
+            self.shutdown()
+
+    def handle_request(self, r):
+        f = flow.FlowMaster.handle_request(self, r)
+        if f:
+            r._ack()
+        return f
+
+    def handle_response(self, r):
+        f = flow.FlowMaster.handle_response(self, r)
+        if f:
+            r._ack()
+        print f
+        return f
+
+
+config = proxy.ProxyConfig(
+    cacert = os.path.expanduser("~/.mitmproxy/mitmproxy-ca.pem")
+)
+state = flow.State()
+server = proxy.ProxyServer(config, 8080)
+m = MyMaster(server, state)
+m.run()

examples/modify_form.py

@@ -0,0 +1,8 @@
+
+def request(context, flow):
+    if "application/x-www-form-urlencoded" in flow.request.headers["content-type"]:
+        frm = flow.request.get_form_urlencoded()
+        frm["mitmproxy"] = ["rocks"]
+        flow.request.set_form_urlencoded(frm)
+
+

examples/modify_querystring.py

@@ -0,0 +1,7 @@
+
+def request(context, flow):
+    q = flow.request.get_query()
+    if q:
+        q["mitmproxy"] = ["rocks"]
+        flow.request.set_query(q)
+

examples/stickycookies

@@ -1,5 +1,12 @@
 #!/usr/bin/env python
+"""
+    This example builds on mitmproxy's base proxying infrastructure to
+    implement functionality similar to the "sticky cookies" option. This is at
+    a lower level than the Flow mechanism, so we're dealing directly with
+    request and response objects.
+"""
 from libmproxy import controller, proxy
+import os
 
 class StickyMaster(controller.Master):
     def __init__(self, server):
@@ -23,13 +30,13 @@ class StickyMaster(controller.Master):
     def handle_response(self, msg):
         hid = (msg.request.host, msg.request.port)
         if msg.headers["set-cookie"]:
-            self.stickyhosts[hid] = f.response.headers["set-cookie"]
+            self.stickyhosts[hid] = msg.headers["set-cookie"]
         msg._ack()
 
 
-ssl_config = proxy.SSLConfig(
-    "~/.mitmproxy/cert.pem"
+config = proxy.ProxyConfig(
+    cacert = os.path.expanduser("~/.mitmproxy/mitmproxy-ca.pem")
 )
-server = proxy.ProxyServer(ssl_config, 8080)
+server = proxy.ProxyServer(config, 8080)
 m = StickyMaster(server)
 m.run()

examples/upsidedownternet.py

@@ -0,0 +1,8 @@
+import Image, cStringIO
+def response(context, flow):
+    if flow.response.headers["content-type"] == ["image/png"]:
+        s = cStringIO.StringIO(flow.response.content)
+        img = Image.open(s).rotate(180)
+        s2 = cStringIO.StringIO()
+        img.save(s2, "png")
+        flow.response.content = s2.getvalue()

libmproxy/cmdline.py

@@ -1,3 +1,18 @@
+# Copyright (C) 2012  Aldo Cortesi
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 import proxy
 import optparse
 
@@ -64,6 +79,11 @@ def common_options(parser):
         action="store", type = "int", dest="port", default=8080,
         help = "Proxy service port."
     )
+    parser.add_option(
+        "-R",
+        action="store", dest="reverse_proxy", default=None,
+        help="Reverse proxy to upstream server: http[s]://host[:port]"
+    )
     parser.add_option(
         "-q",
         action="store_true", dest="quiet",
@@ -114,7 +134,13 @@ def common_options(parser):
         action="store_true", dest="anticomp", default=False,
         help="Try to convince servers to send us un-compressed data."
     )
-
+    parser.add_option(
+        "-Z",
+        action="store", dest="body_size_limit", default=None,
+        metavar="SIZE",
+        help="Byte size limit of HTTP request and response bodies."\
+             " Understands k/m/g suffixes, i.e. 3m for 3 megabytes."
+    )
     group = optparse.OptionGroup(parser, "Client Replay")
     group.add_option(
         "-c",

libmproxy/console/__init__.py

@@ -0,0 +1,913 @@
+# Copyright (C) 2010  Aldo Cortesi
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import mailcap, mimetypes, tempfile, os, subprocess, glob, time, shlex
+import os.path, sys
+import urwid
+from .. import controller, utils, flow, version
+import flowlist, flowview, help, common, kveditor, palettes
+
+EVENTLOG_SIZE = 500
+
+
+class Stop(Exception): pass
+
+
+#begin nocover
+
+
+class _PathCompleter:
+    def __init__(self, _testing=False):
+        """
+            _testing: disables reloading of the lookup table to make testing possible.
+        """
+        self.lookup, self.offset = None, None
+        self.final = None
+        self._testing = _testing
+
+    def reset(self):
+        self.lookup = None
+        self.offset = -1
+
+    def complete(self, txt):
+        """
+            Returns the next completion for txt, or None if there is no completion.
+        """
+        path = os.path.expanduser(txt)
+        if not self.lookup:
+            if not self._testing:
+                # Lookup is a set of (display value, actual value) tuples.
+                self.lookup = []
+                if os.path.isdir(path):
+                    files = glob.glob(os.path.join(path, "*"))
+                    prefix = txt
+                else:
+                    files = glob.glob(path+"*")
+                    prefix = os.path.dirname(txt)
+                prefix = prefix or "./"
+                for f in files:
+                    display = os.path.join(prefix, os.path.basename(f))
+                    if os.path.isdir(f):
+                        display += "/"
+                    self.lookup.append((display, f))
+            if not self.lookup:
+                self.final = path
+                return path
+            self.lookup.sort()
+            self.offset = -1
+            self.lookup.append((txt, txt))
+        self.offset += 1
+        if self.offset >= len(self.lookup):
+            self.offset = 0
+        ret = self.lookup[self.offset]
+        self.final = ret[1]
+        return ret[0]
+
+
+class PathEdit(urwid.Edit, _PathCompleter):
+    def __init__(self, *args, **kwargs):
+        urwid.Edit.__init__(self, *args, **kwargs)
+        _PathCompleter.__init__(self)
+
+    def keypress(self, size, key):
+        if key == "tab":
+            comp = self.complete(self.get_edit_text())
+            self.set_edit_text(comp)
+            self.set_edit_pos(len(comp))
+        else:
+            self.reset()
+        return urwid.Edit.keypress(self, size, key)
+
+
+class ActionBar(common.WWrap):
+    def __init__(self):
+        self.message("")
+
+    def selectable(self):
+        return True
+
+    def path_prompt(self, prompt, text):
+        self.w = PathEdit(prompt, text)
+
+    def prompt(self, prompt, text = ""):
+        self.w = urwid.Edit(prompt, text or "")
+
+    def message(self, message):
+        self.w = urwid.Text(message)
+
+
+class StatusBar(common.WWrap):
+    def __init__(self, master, helptext):
+        self.master, self.helptext = master, helptext
+        self.expire = None
+        self.ab = ActionBar()
+        self.ib = common.WWrap(urwid.Text(""))
+        self.w = urwid.Pile([self.ib, self.ab])
+
+    def get_status(self):
+        r = []
+
+        if self.master.client_playback:
+            r.append("[")
+            r.append(("heading_key", "cplayback"))
+            r.append(":%s to go]"%self.master.client_playback.count())
+        if self.master.server_playback:
+            r.append("[")
+            r.append(("heading_key", "splayback"))
+            r.append(":%s to go]"%self.master.server_playback.count())
+        if self.master.state.intercept_txt:
+            r.append("[")
+            r.append(("heading_key", "i"))
+            r.append(":%s]"%self.master.state.intercept_txt)
+        if self.master.state.limit_txt:
+            r.append("[")
+            r.append(("heading_key", "l"))
+            r.append(":%s]"%self.master.state.limit_txt)
+        if self.master.stickycookie_txt:
+            r.append("[")
+            r.append(("heading_key", "t"))
+            r.append(":%s]"%self.master.stickycookie_txt)
+        if self.master.stickyauth_txt:
+            r.append("[")
+            r.append(("heading_key", "u"))
+            r.append(":%s]"%self.master.stickyauth_txt)
+        if self.master.server and self.master.server.config.reverse_proxy:
+            r.append("[")
+            r.append(("heading_key", "R"))
+            r.append(":%s]"%utils.unparse_url(*self.master.server.config.reverse_proxy))
+
+        opts = []
+        if self.master.anticache:
+            opts.append("anticache")
+        if self.master.anticomp:
+            opts.append("anticomp")
+        if not self.master.refresh_server_playback:
+            opts.append("norefresh")
+        if self.master.killextra:
+            opts.append("killextra")
+
+        if opts:
+            r.append("[%s]"%(":".join(opts)))
+
+        if self.master.script:
+            r.append("[script:%s]"%self.master.script.path)
+
+        if self.master.debug:
+            r.append("[lt:%0.3f]"%self.master.looptime)
+
+        return r
+
+    def redraw(self):
+        if self.expire and time.time() > self.expire:
+            self.message("")
+
+        t = [
+                ('heading', ("[%s]"%self.master.state.flow_count()).ljust(7)),
+            ]
+        if self.master.server:
+            boundaddr = "[%s:%s]"%(self.master.server.address or "*", self.master.server.port)
+        else:
+            boundaddr = ""
+        t.extend(self.get_status())
+        status = urwid.AttrWrap(urwid.Columns([
+            urwid.Text(t),
+            urwid.Text(
+                [
+                    self.helptext,
+                    boundaddr
+                ],
+                align="right"
+            ),
+        ]), "heading")
+        self.ib.set_w(status)
+
+    def update(self, text):
+        self.helptext = text
+        self.redraw()
+        self.master.drawscreen()
+
+    def selectable(self):
+        return True
+
+    def get_edit_text(self):
+        return self.ab.w.get_edit_text()
+
+    def path_prompt(self, prompt, text):
+        return self.ab.path_prompt(prompt, text)
+
+    def prompt(self, prompt, text = ""):
+        self.ab.prompt(prompt, text)
+
+    def message(self, msg, expire=None):
+        if expire:
+            self.expire = time.time() + float(expire)/1000
+        else:
+            self.expire = None
+        self.ab.message(msg)
+
+
+#end nocover
+
+class ConsoleState(flow.State):
+    def __init__(self):
+        flow.State.__init__(self)
+        self.focus = None
+        self.view_body_mode = common.VIEW_BODY_PRETTY
+        self.view_flow_mode = common.VIEW_FLOW_REQUEST
+        self.last_script = ""
+        self.last_saveload = ""
+
+    def add_request(self, req):
+        f = flow.State.add_request(self, req)
+        if self.focus is None:
+            self.set_focus(0)
+        return f
+
+    def add_response(self, resp):
+        f = flow.State.add_response(self, resp)
+        if self.focus is None:
+            self.set_focus(0)
+        return f
+
+    def set_limit(self, limit):
+        ret = flow.State.set_limit(self, limit)
+        self.set_focus(self.focus)
+        return ret
+
+    def get_focus(self):
+        if not self.view or self.focus is None:
+            return None, None
+        return self.view[self.focus], self.focus
+
+    def set_focus(self, idx):
+        if self.view:
+            if idx >= len(self.view):
+                idx = len(self.view) - 1
+            elif idx < 0:
+                idx = 0
+            self.focus = idx
+
+    def get_from_pos(self, pos):
+        if len(self.view) <= pos or pos < 0:
+            return None, None
+        return self.view[pos], pos
+
+    def get_next(self, pos):
+        return self.get_from_pos(pos+1)
+
+    def get_prev(self, pos):
+        return self.get_from_pos(pos-1)
+
+    def delete_flow(self, f):
+        ret = flow.State.delete_flow(self, f)
+        self.set_focus(self.focus)
+        return ret
+
+
+
+class Options(object):
+    __slots__ = [
+        "anticache",
+        "anticomp",
+        "client_replay",
+        "debug",
+        "eventlog",
+        "keepserving",
+        "kill",
+        "intercept",
+        "no_server",
+        "refresh_server_playback",
+        "rfile",
+        "script",
+        "rheaders",
+        "server_replay",
+        "stickycookie",
+        "stickyauth",
+        "verbosity",
+        "wfile",
+    ]
+    def __init__(self, **kwargs):
+        for k, v in kwargs.items():
+            setattr(self, k, v)
+        for i in self.__slots__:
+            if not hasattr(self, i):
+                setattr(self, i, None)
+
+
+#begin nocover
+
+
+class ConsoleMaster(flow.FlowMaster):
+    palette = []
+    footer_text_default = [
+        ('heading_key', "?"), ":help ",
+    ]
+    footer_text_help = [
+        ("heading", 'mitmproxy v%s '%version.VERSION),
+        ('heading_key', "q"), ":back ",
+    ]
+    footer_text_flowview = [
+        ('heading_key', "?"), ":help ",
+        ('heading_key', "q"), ":back ",
+    ]
+    def __init__(self, server, options):
+        flow.FlowMaster.__init__(self, server, ConsoleState())
+        self.looptime = 0
+        self.options = options
+
+        self.flow_list_view = None
+        self.set_palette()
+
+        r = self.set_intercept(options.intercept)
+        if r:
+            print >> sys.stderr, "Intercept error:", r
+            sys.exit(1)
+
+        r = self.set_stickycookie(options.stickycookie)
+        if r:
+            print >> sys.stderr, "Sticky cookies error:", r
+            sys.exit(1)
+
+        r = self.set_stickyauth(options.stickyauth)
+        if r:
+            print >> sys.stderr, "Sticky auth error:", r
+            sys.exit(1)
+
+        self.refresh_server_playback = options.refresh_server_playback
+        self.anticache = options.anticache
+        self.anticomp = options.anticomp
+        self.killextra = options.kill
+        self.rheaders = options.rheaders
+
+        self.eventlog = options.eventlog
+        self.eventlist = urwid.SimpleListWalker([])
+
+        if options.client_replay:
+            self.client_playback_path(options.client_replay)
+
+        if options.server_replay:
+            self.server_playback_path(options.server_replay)
+
+        self.debug = options.debug
+
+        if options.script:
+            err = self.load_script(options.script)
+            if err:
+                print >> sys.stderr, "Script load error:", err
+                sys.exit(1)
+
+    def run_script_once(self, path, f):
+        if not path:
+            return
+        ret = self.get_script(path)
+        if ret[0]:
+            self.statusbar.message(ret[0])
+            return
+        s = ret[1]
+        if f.request:
+            s.run("request", f)
+        if f.response:
+            s.run("response", f)
+        if f.error:
+            s.run("error", f)
+        s.run("done")
+        self.refresh_flow(f)
+        self.state.last_script = path
+
+    def set_script(self, path):
+        if not path:
+            return
+        ret = self.load_script(path)
+        if ret:
+            self.statusbar.message(ret)
+        self.state.last_script = path
+
+    def toggle_eventlog(self):
+        self.eventlog = not self.eventlog
+        self.view_flowlist()
+
+    def _readflow(self, path):
+        path = os.path.expanduser(path)
+        try:
+            f = file(path, "r")
+            flows = list(flow.FlowReader(f).stream())
+        except (IOError, flow.FlowReadError), v:
+            return True, v.strerror
+        return False, flows
+
+    def client_playback_path(self, path):
+        err, ret = self._readflow(path)
+        if err:
+            self.statusbar.message(ret)
+        else:
+            self.start_client_playback(ret, False)
+
+    def server_playback_path(self, path):
+        err, ret = self._readflow(path)
+        if err:
+            self.statusbar.message(ret)
+        else:
+            self.start_server_playback(
+                ret,
+                self.killextra, self.rheaders,
+                False
+            )
+
+    def spawn_editor(self, data):
+        fd, name = tempfile.mkstemp('', "mproxy")
+        os.write(fd, data)
+        os.close(fd)
+        c = os.environ.get("EDITOR")
+        #If no EDITOR is set, assume 'vi'
+        if not c:
+            c = "vi"
+        cmd = shlex.split(c)
+        cmd.append(name)
+        self.ui.stop()
+        try:
+            subprocess.call(cmd)
+        except:
+            self.statusbar.message("Can't start editor: %s" % c)
+            self.ui.start()
+            os.unlink(name)
+            return data
+        self.ui.start()
+        data = open(name).read()
+        os.unlink(name)
+        return data
+
+    def spawn_external_viewer(self, data, contenttype):
+        if contenttype:
+            ext = mimetypes.guess_extension(contenttype) or ""
+        else:
+            ext = ""
+        fd, name = tempfile.mkstemp(ext, "mproxy")
+        os.write(fd, data)
+        os.close(fd)
+
+        cmd = None
+        shell = False
+
+        if contenttype:
+            c = mailcap.getcaps()
+            cmd, _ = mailcap.findmatch(c, contenttype, filename=name)
+            if cmd:
+                shell = True
+        if not cmd:
+            c = os.environ.get("PAGER") or os.environ.get("EDITOR")
+            cmd = [c, name]
+        self.ui.stop()
+        subprocess.call(cmd, shell=shell)
+        self.ui.start()
+        os.unlink(name)
+
+    def set_palette(self):
+        self.palette = palettes.dark
+
+    def run(self):
+        self.currentflow = None
+
+        self.ui = urwid.raw_display.Screen()
+        self.ui.set_terminal_properties(256)
+        self.ui.register_palette(self.palette)
+        self.flow_list_view = flowlist.ConnectionListView(self, self.state)
+
+        self.view = None
+        self.statusbar = None
+        self.header = None
+        self.body = None
+        self.help_context = None
+
+        self.prompting = False
+        self.onekey = False
+
+        self.view_flowlist()
+
+        if self.server:
+            slave = controller.Slave(self.masterq, self.server)
+            slave.start()
+
+        if self.options.rfile:
+            ret = self.load_flows(self.options.rfile)
+            if ret:
+                self.shutdown()
+                print >> sys.stderr, "Could not load file:", ret
+                sys.exit(1)
+
+        self.ui.run_wrapper(self.loop)
+        # If True, quit just pops out to flow list view.
+        print >> sys.stderr, "Shutting down..."
+        sys.stderr.flush()
+        self.shutdown()
+
+    def focus_current(self):
+        if self.currentflow:
+            try:
+                self.flow_list_view.set_focus(self.state.index(self.currentflow))
+            except (IndexError, ValueError):
+                pass
+
+    def make_view(self):
+        self.view = urwid.Frame(
+                        self.body,
+                        header = self.header,
+                        footer = self.statusbar
+                    )
+        self.view.set_focus("body")
+
+    def view_help(self):
+        h = help.HelpView(self, self.help_context, (self.statusbar, self.body, self.header))
+        self.statusbar = StatusBar(self, self.footer_text_help)
+        self.body = h
+        self.header = None
+        self.make_view()
+
+    def view_kveditor(self, title, value, callback, *args, **kwargs):
+        self.body = kveditor.KVEditor(self, title, value, callback, *args, **kwargs)
+        self.header = None
+        self.help_context = kveditor.help_context
+        self.statusbar = StatusBar(self, self.footer_text_help)
+        self.make_view()
+
+    def view_flowlist(self):
+        if self.ui.started:
+            self.ui.clear()
+        self.focus_current()
+        if self.eventlog:
+            self.body = flowlist.BodyPile(self)
+        else:
+            self.body = flowlist.ConnectionListBox(self)
+        self.statusbar = StatusBar(self, self.footer_text_default)
+        self.header = None
+        self.currentflow = None
+
+        self.make_view()
+        self.help_context = flowlist.help_context
+
+    def view_flow(self, flow):
+        self.body = flowview.ConnectionView(self, self.state, flow)
+        self.header = flowview.ConnectionViewHeader(self, flow)
+        self.statusbar = StatusBar(self, self.footer_text_flowview)
+        self.currentflow = flow
+
+        self.make_view()
+        self.help_context = flowview.help_context
+
+    def _write_flows(self, path, flows):
+        self.state.last_saveload = path
+        if not path:
+            return
+        path = os.path.expanduser(path)
+        try:
+            f = file(path, "wb")
+            fw = flow.FlowWriter(f)
+            for i in flows:
+                fw.add(i)
+            f.close()
+        except IOError, v:
+            self.statusbar.message(v.strerror)
+
+    def save_one_flow(self, path, flow):
+        return self._write_flows(path, [flow])
+
+    def save_flows(self, path):
+        return self._write_flows(path, self.state.view)
+
+    def load_flows_callback(self, path):
+        if not path:
+            return
+        ret = self.load_flows(path)
+        return ret or "Flows loaded from %s"%path
+
+    def load_flows(self, path):
+        self.state.last_saveload = path
+        path = os.path.expanduser(path)
+        try:
+            f = file(path, "r")
+            fr = flow.FlowReader(f)
+        except IOError, v:
+            return v.strerror
+        try:
+            flow.FlowMaster.load_flows(self, fr)
+        except flow.FlowReadError, v:
+            return v.strerror
+        f.close()
+        if self.flow_list_view:
+            self.sync_list_view()
+            self.focus_current()
+
+    def path_prompt(self, prompt, text, callback, *args):
+        self.statusbar.path_prompt(prompt, text)
+        self.view.set_focus("footer")
+        self.prompting = (callback, args)
+
+    def prompt(self, prompt, text, callback, *args):
+        self.statusbar.prompt(prompt, text)
+        self.view.set_focus("footer")
+        self.prompting = (callback, args)
+
+    def prompt_edit(self, prompt, text, callback):
+        self.statusbar.prompt(prompt + ": ", text)
+        self.view.set_focus("footer")
+        self.prompting = (callback, [])
+
+    def prompt_onekey(self, prompt, keys, callback, *args):
+        """
+            Keys are a set of (word, key) tuples. The appropriate key in the
+            word is highlighted.
+        """
+        prompt = [prompt, " ("]
+        mkup = []
+        for i, e in enumerate(keys):
+            mkup.extend(common.highlight_key(e[0], e[1]))
+            if i < len(keys)-1:
+                mkup.append(",")
+        prompt.extend(mkup)
+        prompt.append(")? ")
+        self.onekey = "".join(i[1] for i in keys)
+        self.prompt(prompt, "", callback, *args)
+
+    def prompt_done(self):
+        self.prompting = False
+        self.onekey = False
+        self.view.set_focus("body")
+        self.statusbar.message("")
+
+    def prompt_execute(self, txt=None):
+        if not txt:
+            txt = self.statusbar.get_edit_text()
+        p, args = self.prompting
+        self.prompt_done()
+        msg = p(txt, *args)
+        if msg:
+            self.statusbar.message(msg, 1000)
+
+    def prompt_cancel(self):
+        self.prompt_done()
+
+    def accept_all(self):
+        self.state.accept_all()
+
+    def set_limit(self, txt):
+        return self.state.set_limit(txt)
+
+    def set_intercept(self, txt):
+        return self.state.set_intercept(txt)
+
+    def set_reverse_proxy(self, txt):
+        if not txt:
+            self.server.config.reverse_proxy = None
+        else:
+            s = utils.parse_proxy_spec(txt)
+            if not s:
+                return "Invalid reverse proxy specification"
+            self.server.config.reverse_proxy = s
+
+    def changeview(self, v):
+        if v == "r":
+            self.state.view_body_mode = common.VIEW_BODY_RAW
+        elif v == "h":
+            self.state.view_body_mode = common.VIEW_BODY_HEX
+        elif v == "p":
+            self.state.view_body_mode = common.VIEW_BODY_PRETTY
+        self.refresh_flow(self.currentflow)
+
+    def drawscreen(self):
+        size = self.ui.get_cols_rows()
+        canvas = self.view.render(size, focus=1)
+        self.ui.draw_screen(size, canvas)
+        return size
+
+    def pop_view(self):
+        if self.currentflow:
+            self.view_flow(self.currentflow)
+        else:
+            self.view_flowlist()
+
+    def loop(self):
+        changed = True
+        try:
+            while not controller.should_exit:
+                startloop = time.time()
+                if changed:
+                    self.statusbar.redraw()
+                    size = self.drawscreen()
+                changed = self.tick(self.masterq)
+                self.ui.set_input_timeouts(max_wait=0.1)
+                keys = self.ui.get_input()
+                if keys:
+                    changed = True
+                for k in keys:
+                    if self.prompting:
+                        if k == "esc":
+                            self.prompt_cancel()
+                        elif self.onekey:
+                            if k == "enter":
+                                self.prompt_cancel()
+                            elif k in self.onekey:
+                                self.prompt_execute(k)
+                        elif k == "enter":
+                            self.prompt_execute()
+                        else:
+                            self.view.keypress(size, k)
+                    else:
+                        k = self.view.keypress(size, k)
+                        if k:
+                            self.statusbar.message("")
+                            if k == "?":
+                                self.view_help()
+                            elif k == "c":
+                                if not self.client_playback:
+                                    self.path_prompt(
+                                        "Client replay: ",
+                                        self.state.last_saveload,
+                                        self.client_playback_path
+                                    )
+                                else:
+                                    self.prompt_onekey(
+                                        "Stop current client replay?",
+                                        (
+                                            ("yes", "y"),
+                                            ("no", "n"),
+                                        ),
+                                        self.stop_client_playback_prompt,
+                                    )
+                            elif k == "i":
+                                self.prompt(
+                                    "Intercept filter: ",
+                                    self.state.intercept_txt,
+                                    self.set_intercept
+                                )
+                                self.sync_list_view()
+                            elif k == "Q":
+                                raise Stop
+                            elif k == "q":
+                                self.prompt_onekey(
+                                    "Quit",
+                                    (
+                                        ("yes", "y"),
+                                        ("no", "n"),
+                                    ),
+                                    self.quit,
+                                )
+                            elif k == "R":
+                                if self.server.config.reverse_proxy:
+                                    p = utils.unparse_url(*self.server.config.reverse_proxy)
+                                else:
+                                    p = ""
+                                self.prompt(
+                                    "Reverse proxy: ",
+                                    p,
+                                    self.set_reverse_proxy
+                                )
+                                self.sync_list_view()
+                            elif k == "s":
+                                if self.script:
+                                    self.load_script(None)
+                                else:
+                                    self.path_prompt(
+                                        "Set script: ",
+                                        self.state.last_script,
+                                        self.set_script
+                                    )
+                            elif k == "S":
+                                if not self.server_playback:
+                                    self.path_prompt(
+                                        "Server replay: ",
+                                        self.state.last_saveload,
+                                        self.server_playback_path
+                                    )
+                                else:
+                                    self.prompt_onekey(
+                                        "Stop current server replay?",
+                                        (
+                                            ("yes", "y"),
+                                            ("no", "n"),
+                                        ),
+                                        self.stop_server_playback_prompt,
+                                    )
+                            elif k == "o":
+                                self.prompt_onekey(
+                                        "Options",
+                                        (
+                                            ("anticache", "a"),
+                                            ("anticomp", "c"),
+                                            ("killextra", "k"),
+                                            ("norefresh", "n"),
+                                        ),
+                                        self._change_options
+                                )
+                            elif k == "t":
+                                self.prompt(
+                                    "Sticky cookie filter: ",
+                                    self.stickycookie_txt,
+                                    self.set_stickycookie
+                                )
+                            elif k == "u":
+                                self.prompt(
+                                    "Sticky auth filter: ",
+                                    self.stickyauth_txt,
+                                    self.set_stickyauth
+                                )
+                self.looptime = time.time() - startloop
+        except (Stop, KeyboardInterrupt):
+            pass
+
+    def stop_client_playback_prompt(self, a):
+        if a != "n":
+            self.stop_client_playback()
+
+    def stop_server_playback_prompt(self, a):
+        if a != "n":
+            self.stop_server_playback()
+
+    def quit(self, a):
+        if a != "n":
+            raise Stop
+
+    def _change_options(self, a):
+        if a == "a":
+            self.anticache = not self.anticache
+        if a == "c":
+            self.anticomp = not self.anticomp
+        elif a == "k":
+            self.killextra = not self.killextra
+        elif a == "n":
+            self.refresh_server_playback = not self.refresh_server_playback
+
+    def shutdown(self):
+        self.state.killall(self)
+        controller.Master.shutdown(self)
+
+    def sync_list_view(self):
+        self.flow_list_view._modified()
+
+    def clear_flows(self):
+        self.state.clear()
+        self.sync_list_view()
+
+    def delete_flow(self, f):
+        self.state.delete_flow(f)
+        self.sync_list_view()
+
+    def refresh_flow(self, c):
+        if hasattr(self.header, "refresh_flow"):
+            self.header.refresh_flow(c)
+        if hasattr(self.body, "refresh_flow"):
+            self.body.refresh_flow(c)
+        if hasattr(self.statusbar, "refresh_flow"):
+            self.statusbar.refresh_flow(c)
+
+    def process_flow(self, f, r):
+        if self.state.intercept and f.match(self.state.intercept) and not f.request.is_replay():
+            f.intercept()
+        else:
+            r._ack()
+        self.sync_list_view()
+        self.refresh_flow(f)
+
+    def clear_events(self):
+        self.eventlist[:] = []
+
+    def add_event(self, e, level="info"):
+        if level == "info":
+            e = urwid.Text(e)
+        elif level == "error":
+            e = urwid.Text(("error", e))
+
+        self.eventlist.append(e)
+        if len(self.eventlist) > EVENTLOG_SIZE:
+            self.eventlist.pop(0)
+        self.eventlist.set_focus(len(self.eventlist))
+
+    # Handlers
+    def handle_error(self, r):
+        f = flow.FlowMaster.handle_error(self, r)
+        if f:
+            self.process_flow(f, r)
+        return f
+
+    def handle_request(self, r):
+        f = flow.FlowMaster.handle_request(self, r)
+        if f:
+            self.process_flow(f, r)
+        return f
+
+    def handle_response(self, r):
+        f = flow.FlowMaster.handle_response(self, r)
+        if f:
+            self.process_flow(f, r)
+        return f
+

libmproxy/console/common.py

@@ -0,0 +1,237 @@
+# Copyright (C) 2012  Aldo Cortesi
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import urwid
+import urwid.util
+from .. import utils
+
+
+VIEW_BODY_RAW = 0
+VIEW_BODY_HEX = 1
+VIEW_BODY_PRETTY = 2
+
+BODY_VIEWS = {
+    VIEW_BODY_RAW: "raw",
+    VIEW_BODY_HEX: "hex",
+    VIEW_BODY_PRETTY: "pretty"
+}
+
+VIEW_FLOW_REQUEST = 0
+VIEW_FLOW_RESPONSE = 1
+
+
+def highlight_key(s, k):
+    l = []
+    parts = s.split(k, 1)
+    if parts[0]:
+        l.append(("text", parts[0]))
+    l.append(("key", k))
+    if parts[1]:
+        l.append(("text", parts[1]))
+    return l
+
+
+KEY_MAX = 30
+def format_keyvals(lst, key="key", val="text", indent=0):
+    """
+        Format a list of (key, value) tuples.
+
+        If key is None, it's treated specially:
+            - We assume a sub-value, and add an extra indent.
+            - The value is treated as a pre-formatted list of directives.
+    """
+    ret = []
+    if lst:
+        maxk = min(max(len(i[0]) for i in lst if i and i[0]), KEY_MAX)
+        for i, kv in enumerate(lst):
+            if kv is None:
+                ret.append(urwid.Text(""))
+            else:
+                cols = []
+                # This cumbersome construction process is here for a reason:
+                # Urwid < 1.0 barfs if given a fixed size column of size zero.
+                if indent:
+                    cols.append(("fixed", indent, urwid.Text("")))
+                cols.extend([
+                    (
+                        "fixed",
+                        maxk,
+                        urwid.Text([(key, kv[0] or "")])
+                    ),
+                    urwid.Text([(val, kv[1])])
+                ])
+                ret.append(urwid.Columns(cols, dividechars = 2))
+    return ret
+
+
+def shortcuts(k):
+    if k == " ":
+        k = "page down"
+    elif k == "j":
+        k = "down"
+    elif k == "k":
+        k = "up"
+    return k
+
+
+def fcol(s, attr):
+    s = unicode(s)
+    return (
+        "fixed",
+        len(s),
+        urwid.Text(
+            [
+                (attr, s)
+            ]
+        )
+    )
+
+if urwid.util.detected_encoding:
+    SYMBOL_REPLAY = u"\u21ba"
+    SYMBOL_RETURN = u"\u2190"
+else:
+    SYMBOL_REPLAY = u"[r]"
+    SYMBOL_RETURN = u"<-"
+
+
+
+def raw_format_flow(f, focus, extended, padding):
+    f = dict(f)
+
+    pile = []
+    req = []
+    if extended:
+        req.append(
+            fcol(
+                utils.format_timestamp(f["req_timestamp"]),
+                "highlight"
+            )
+        )
+    else:
+        req.append(fcol(">>" if focus else "  ", "focus"))
+    if f["req_is_replay"]:
+        req.append(fcol(SYMBOL_REPLAY, "replay"))
+    req.append(fcol(f["req_method"], "method"))
+
+    preamble = sum(i[1] for i in req) + len(req) -1
+
+    if f["intercepting"] and not f["req_acked"]:
+        uc = "intercept"
+    elif f["resp_code"] or f["err_msg"]:
+        uc = "text"
+    else:
+        uc = "title"
+
+    req.append(
+        urwid.Text([(uc, f["req_url"])])
+    )
+
+    pile.append(urwid.Columns(req, dividechars=1))
+
+    resp = []
+    resp.append(
+        ("fixed", preamble, urwid.Text(""))
+    )
+
+    if f["resp_code"]:
+        if f["resp_code"] in [200, 304]:
+            resp.append(fcol(SYMBOL_RETURN, "goodcode"))
+        else:
+            resp.append(fcol(SYMBOL_RETURN, "error"))
+        if f["resp_is_replay"]:
+            resp.append(fcol(SYMBOL_REPLAY, "replay"))
+        if f["resp_code"] in [200, 304]:
+            resp.append(fcol(f["resp_code"], "goodcode"))
+        else:
+            resp.append(fcol(f["resp_code"], "error"))
+
+        if f["intercepting"] and f["resp_code"] and not f["resp_acked"]:
+            rc = "intercept"
+        else:
+            rc = "text"
+
+        if f["resp_ctype"]:
+            resp.append(fcol(f["resp_ctype"], rc))
+        resp.append(fcol(f["resp_clen"], rc))
+    elif f["err_msg"]:
+        resp.append(fcol(SYMBOL_RETURN, "error"))
+        resp.append(
+            urwid.Text([
+                (
+                    "error",
+                    f["err_msg"]
+                )
+            ])
+        )
+    pile.append(urwid.Columns(resp, dividechars=1))
+    return urwid.Pile(pile)
+
+
+class FlowCache:
+    @utils.LRUCache(200)
+    def format_flow(self, *args):
+        return raw_format_flow(*args)
+flowcache = FlowCache()
+
+
+def format_flow(f, focus, extended=False, padding=2):
+    d = dict(
+        intercepting = f.intercepting,
+
+        req_timestamp = f.request.timestamp,
+        req_is_replay = f.request.is_replay(),
+        req_method = f.request.method,
+        req_acked = f.request.acked,
+        req_url = f.request.get_url(),
+
+        err_msg = f.error.msg if f.error else None,
+        resp_code = f.response.code if f.response else None,
+    )
+    if f.response:
+        d.update(dict(
+            resp_code = f.response.code,
+            resp_is_replay = f.response.is_replay(),
+            resp_acked = f.response.acked,
+            resp_clen = utils.pretty_size(len(f.response.content)) if f.response.content else "[empty content]"
+        ))
+        t = f.response.headers["content-type"]
+        if t:
+            d["resp_ctype"] = t[0].split(";")[0]
+        else:
+            d["resp_ctype"] = ""
+    return flowcache.format_flow(tuple(sorted(d.items())), focus, extended, padding)
+
+
+
+def int_version(v):
+    SIG = 3
+    v = urwid.__version__.split("-")[0].split(".")
+    x = 0
+    for i in range(min(SIG, len(v))):
+        x += int(v[i]) * 10**(SIG-i)
+    return x
+
+
+# We have to do this to be portable over 0.9.8 and 0.9.9 If compatibility
+# becomes a pain to maintain, we'll just mandate 0.9.9 or newer.
+class WWrap(urwid.WidgetWrap):
+    if int_version(urwid.__version__) >= 990:
+        def set_w(self, x):
+            self._w = x
+        def get_w(self):
+            return self._w
+        w = property(get_w, set_w)
+
+

libmproxy/console/flowlist.py

@@ -0,0 +1,211 @@
+# Copyright (C) 2012  Aldo Cortesi
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import urwid
+import common
+
+def _mkhelp():
+    text = []
+    keys = [
+        ("A", "accept all intercepted flows"),
+        ("a", "accept this intercepted flows"),
+        ("C", "clear flow list or eventlog"),
+        ("d", "delete flow"),
+        ("D", "duplicate flow"),
+        ("e", "toggle eventlog"),
+        ("l", "set limit filter pattern"),
+        ("L", "load saved flows"),
+        ("r", "replay request"),
+        ("V", "revert changes to request"),
+        ("w", "save all flows matching current limit"),
+        ("W", "save this flow"),
+        ("X", "kill and delete flow, even if it's mid-intercept"),
+        ("tab", "tab between eventlog and flow list"),
+        ("enter", "view flow"),
+        ("|", "run script on this flow"),
+    ]
+    text.extend(common.format_keyvals(keys, key="key", val="text", indent=4))
+    return text
+help_context = _mkhelp()
+
+
+class EventListBox(urwid.ListBox):
+    def __init__(self, master):
+        self.master = master
+        urwid.ListBox.__init__(self, master.eventlist)
+
+    def keypress(self, size, key):
+        key = common.shortcuts(key)
+        if key == "C":
+            self.master.clear_events()
+            key = None
+        return urwid.ListBox.keypress(self, size, key)
+
+
+class BodyPile(urwid.Pile):
+    def __init__(self, master):
+        h = urwid.Text("Event log")
+        h = urwid.Padding(h, align="left", width=("relative", 100))
+
+        self.inactive_header = urwid.AttrWrap(h, "heading_inactive")
+        self.active_header = urwid.AttrWrap(h, "heading")
+
+        urwid.Pile.__init__(
+            self,
+            [
+                ConnectionListBox(master),
+                urwid.Frame(EventListBox(master), header = self.inactive_header)
+            ]
+        )
+        self.master = master
+        self.focus = 0
+
+    def keypress(self, size, key):
+        if key == "tab":
+            self.focus = (self.focus + 1)%len(self.widget_list)
+            self.set_focus(self.focus)
+            if self.focus == 1:
+                self.widget_list[1].header = self.active_header
+            else:
+                self.widget_list[1].header = self.inactive_header
+            key = None
+        elif key == "v":
+            self.master.toggle_eventlog()
+            key = None
+
+        # This is essentially a copypasta from urwid.Pile's keypress handler.
+        # So much for "closed for modification, but open for extension".
+        item_rows = None
+        if len(size)==2:
+            item_rows = self.get_item_rows( size, focus=True )
+        i = self.widget_list.index(self.focus_item)
+        tsize = self.get_item_size(size,i,True,item_rows)
+        return self.focus_item.keypress( tsize, key )
+
+
+class ConnectionItem(common.WWrap):
+    def __init__(self, master, state, flow, focus):
+        self.master, self.state, self.flow = master, state, flow
+        self.focus = focus
+        w = self.get_text()
+        common.WWrap.__init__(self, w)
+
+    def get_text(self):
+        return common.format_flow(self.flow, self.focus)
+
+    def selectable(self):
+        return True
+
+    def keypress(self, (maxcol,), key):
+        key = common.shortcuts(key)
+        if key == "a":
+            self.flow.accept_intercept()
+            self.master.sync_list_view()
+        elif key == "d":
+            self.flow.kill(self.master)
+            self.state.delete_flow(self.flow)
+            self.master.sync_list_view()
+        elif key == "D":
+            f = self.master.duplicate_flow(self.flow)
+            self.master.currentflow = f
+            self.master.focus_current()
+        elif key == "r":
+            r = self.master.replay_request(self.flow)
+            if r:
+                self.master.statusbar.message(r)
+            self.master.sync_list_view()
+        elif key == "V":
+            self.state.revert(self.flow)
+            self.master.sync_list_view()
+        elif key == "w":
+            self.master.path_prompt(
+                "Save flows: ",
+                self.state.last_saveload,
+                self.master.save_flows
+            )
+        elif key == "W":
+            self.master.path_prompt(
+                "Save this flow: ",
+                self.state.last_saveload,
+                self.master.save_one_flow,
+                self.flow
+            )
+        elif key == "X":
+            self.flow.kill(self.master)
+        elif key == "enter":
+            if self.flow.request:
+                self.master.view_flow(self.flow)
+        elif key == "|":
+            self.master.path_prompt(
+                "Send flow to script: ",
+                self.state.last_script,
+                self.master.run_script_once,
+                self.flow
+            )
+        else:
+            return key
+
+
+class ConnectionListView(urwid.ListWalker):
+    def __init__(self, master, state):
+        self.master, self.state = master, state
+        if self.state.flow_count():
+            self.set_focus(0)
+
+    def get_focus(self):
+        f, i = self.state.get_focus()
+        f = ConnectionItem(self.master, self.state, f, True) if f else None
+        return f, i
+
+    def set_focus(self, focus):
+        ret = self.state.set_focus(focus)
+        return ret
+
+    def get_next(self, pos):
+        f, i = self.state.get_next(pos)
+        f = ConnectionItem(self.master, self.state, f, False) if f else None
+        return f, i
+
+    def get_prev(self, pos):
+        f, i = self.state.get_prev(pos)
+        f = ConnectionItem(self.master, self.state, f, False) if f else None
+        return f, i
+
+
+class ConnectionListBox(urwid.ListBox):
+    def __init__(self, master):
+        self.master = master
+        urwid.ListBox.__init__(self, master.flow_list_view)
+
+    def keypress(self, size, key):
+        key = common.shortcuts(key)
+        if key == "A":
+            self.master.accept_all()
+            self.master.sync_list_view()
+        elif key == "C":
+            self.master.clear_flows()
+        elif key == "e":
+            self.master.toggle_eventlog()
+        elif key == "l":
+            self.master.prompt("Limit: ", self.master.state.limit_txt, self.master.set_limit)
+            self.master.sync_list_view()
+        elif key == "L":
+            self.master.path_prompt(
+                "Load flows: ",
+                self.master.state.last_saveload,
+                self.master.load_flows_callback
+            )
+        else:
+            return urwid.ListBox.keypress(self, size, key)

libmproxy/console/flowview.py

@@ -0,0 +1,608 @@
+# Copyright (C) 2012  Aldo Cortesi
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os, re
+import urwid
+import common
+from .. import utils, encoding, flow
+
+def _mkhelp():
+    text = []
+    keys = [
+        ("A", "accept all intercepted flows"),
+        ("a", "accept this intercepted flow"),
+        ("b", "save request/response body"),
+        ("d", "delete flow"),
+        ("D", "duplicate flow"),
+        ("e", "edit request/response"),
+        ("m", "change body display mode"),
+            (None,
+                common.highlight_key("raw", "r") +
+                [("text", ": raw data")]
+            ),
+            (None,
+                common.highlight_key("pretty", "p") +
+                [("text", ": pretty-print XML, HTML and JSON")]
+            ),
+            (None,
+                common.highlight_key("hex", "h") +
+                [("text", ": hex dump")]
+            ),
+        ("p", "previous flow"),
+        ("r", "replay request"),
+        ("V", "revert changes to request"),
+        ("v", "view body in external viewer"),
+        ("w", "save all flows matching current limit"),
+        ("W", "save this flow"),
+        ("z", "encode/decode a request/response"),
+        ("tab", "toggle request/response view"),
+        ("space", "next flow"),
+        ("|", "run script on this flow"),
+    ]
+    text.extend(common.format_keyvals(keys, key="key", val="text", indent=4))
+    return text
+help_context = _mkhelp()
+
+
+VIEW_CUTOFF = 1024*100
+
+class ConnectionViewHeader(common.WWrap):
+    def __init__(self, master, f):
+        self.master, self.flow = master, f
+        self.w = common.format_flow(f, False, extended=True, padding=0)
+
+    def refresh_flow(self, f):
+        if f == self.flow:
+            self.w = common.format_flow(f, False, extended=True, padding=0)
+
+
+class CallbackCache:
+    @utils.LRUCache(20)
+    def callback(self, obj, method, *args, **kwargs):
+        return getattr(obj, method)(*args, **kwargs)
+cache = CallbackCache()
+
+
+class ConnectionView(common.WWrap):
+    REQ = 0
+    RESP = 1
+    method_options = [
+        ("get", "g"),
+        ("post", "p"),
+        ("put", "u"),
+        ("head", "h"),
+        ("trace", "t"),
+        ("delete", "d"),
+        ("options", "o"),
+        ("edit raw", "e"),
+    ]
+    def __init__(self, master, state, flow):
+        self.master, self.state, self.flow = master, state, flow
+        if self.state.view_flow_mode == common.VIEW_FLOW_RESPONSE and flow.response:
+            self.view_response()
+        else:
+            self.view_request()
+
+    def _trailer(self, clen, txt):
+        rem = clen - VIEW_CUTOFF
+        if rem > 0:
+            txt.append(urwid.Text(""))
+            txt.append(
+                urwid.Text(
+                    [
+                        ("highlight", "... %s of data not shown"%utils.pretty_size(rem))
+                    ]
+                )
+            )
+
+    def _view_flow_raw(self, content):
+        txt = []
+        for i in utils.cleanBin(content[:VIEW_CUTOFF]).splitlines():
+            txt.append(
+                urwid.Text(("text", i))
+            )
+        self._trailer(len(content), txt)
+        return txt
+
+    def _view_flow_binary(self, content):
+        txt = []
+        for offset, hexa, s in utils.hexdump(content[:VIEW_CUTOFF]):
+            txt.append(urwid.Text([
+                ("offset", offset),
+                " ",
+                ("text", hexa),
+                "   ",
+                ("text", s),
+            ]))
+        self._trailer(len(content), txt)
+        return txt
+
+    def _view_flow_xmlish(self, content):
+        txt = []
+        for i in utils.pretty_xmlish(content[:VIEW_CUTOFF]):
+            txt.append(
+                urwid.Text(("text", i)),
+            )
+        self._trailer(len(content), txt)
+        return txt
+
+    def _view_flow_json(self, lines):
+        txt = []
+        sofar = 0
+        for i in lines:
+            sofar += len(i)
+            txt.append(
+                urwid.Text(("text", i)),
+            )
+            if sofar > VIEW_CUTOFF:
+                break
+        self._trailer(sum(len(i) for i in lines), txt)
+        return txt
+
+    def _view_flow_formdata(self, content, boundary):
+        rx = re.compile(r'\bname="([^"]+)"')
+        keys = []
+        vals = []
+
+        for i in content.split("--" + boundary):
+            parts = i.splitlines()
+            if len(parts) > 1 and parts[0][0:2] != "--":
+                match = rx.search(parts[1])
+                if match:
+                    keys.append(match.group(1) + ":")
+                    vals.append(utils.cleanBin(
+                        "\n".join(parts[3+parts[2:].index(""):])
+                    ))
+        r = [
+            urwid.Text(("highlight", "Form data:\n")),
+        ]
+        r.extend(common.format_keyvals(
+            zip(keys, vals),
+            key = "header",
+            val = "text"
+        ))
+        return r
+
+    def _view_flow_urlencoded(self, lines):
+        return common.format_keyvals(
+                    [(k+":", v) for (k, v) in lines],
+                    key = "header",
+                    val = "text"
+               )
+
+
+    def _find_pretty_view(self, content, hdrItems):
+        ctype = None
+        for i in hdrItems:
+            if i[0].lower() == "content-type":
+                ctype = i[1]
+                break
+        if ctype and flow.HDR_FORM_URLENCODED in ctype:
+            data = utils.urldecode(content)
+            if data:
+                return "URLEncoded form", self._view_flow_urlencoded(data)
+        if utils.isXML(content):
+            return "Indented XML-ish", self._view_flow_xmlish(content)
+        elif ctype and "application/json" in ctype:
+            lines = utils.pretty_json(content)
+            if lines:
+                return "JSON", self._view_flow_json(lines)
+        elif ctype and "multipart/form-data" in ctype:
+            boundary = ctype.split('boundary=')
+            if len(boundary) > 1:
+                return "Form data", self._view_flow_formdata(content, boundary[1].split(';')[0])
+        return "", self._view_flow_raw(content)
+
+    def _cached_conn_text(self, e, content, hdrItems, viewmode):
+        txt = common.format_keyvals(
+                [(h+":", v) for (h, v) in hdrItems],
+                key = "header",
+                val = "text"
+            )
+        if content:
+            msg = ""
+            if viewmode == common.VIEW_BODY_HEX:
+                body = self._view_flow_binary(content)
+            elif viewmode == common.VIEW_BODY_PRETTY:
+                emsg = ""
+                if e:
+                    decoded = encoding.decode(e, content)
+                    if decoded:
+                        content = decoded
+                        if e and e != "identity":
+                            emsg = "[decoded %s]"%e
+                msg, body = self._find_pretty_view(content, hdrItems)
+                if emsg:
+                    msg = emsg + " " + msg
+            else:
+                body = self._view_flow_raw(content)
+
+            title = urwid.AttrWrap(urwid.Columns([
+                urwid.Text(
+                    [
+                        ("heading", msg),
+                    ]
+                ),
+                urwid.Text(
+                    [
+                        " ",
+                        ('heading', "["),
+                        ('heading_key', "m"),
+                        ('heading', (":%s]"%common.BODY_VIEWS[self.master.state.view_body_mode])),
+                    ],
+                    align="right"
+                ),
+            ]), "heading")
+            txt.append(title)
+            txt.extend(body)
+        return urwid.ListBox(txt)
+
+    def _tab(self, content, attr):
+        p = urwid.Text(content)
+        p = urwid.Padding(p, align="left", width=("relative", 100))
+        p = urwid.AttrWrap(p, attr)
+        return p
+
+    def wrap_body(self, active, body):
+        parts = []
+
+        if self.flow.intercepting and not self.flow.request.acked:
+            qt = "Request intercepted"
+        else:
+            qt = "Request"
+        if active == common.VIEW_FLOW_REQUEST:
+            parts.append(self._tab(qt, "heading"))
+        else:
+            parts.append(self._tab(qt, "heading_inactive"))
+
+        if self.flow.intercepting and self.flow.response and not self.flow.response.acked:
+            st = "Response intercepted"
+        else:
+            st = "Response"
+        if active == common.VIEW_FLOW_RESPONSE:
+            parts.append(self._tab(st, "heading"))
+        else:
+            parts.append(self._tab(st, "heading_inactive"))
+
+        h = urwid.Columns(parts)
+        f = urwid.Frame(
+                    body,
+                    header=h
+                )
+        return f
+
+    def _conn_text(self, conn, viewmode):
+        e = conn.headers["content-encoding"]
+        e = e[0] if e else None
+        return cache.callback(
+                    self, "_cached_conn_text",
+                    e,
+                    conn.content,
+                    tuple(tuple(i) for i in conn.headers.lst),
+                    viewmode
+                )
+
+    def view_request(self):
+        self.state.view_flow_mode = common.VIEW_FLOW_REQUEST
+        body = self._conn_text(
+            self.flow.request,
+            self.state.view_body_mode
+        )
+        self.w = self.wrap_body(common.VIEW_FLOW_REQUEST, body)
+        self.master.statusbar.redraw()
+
+    def view_response(self):
+        self.state.view_flow_mode = common.VIEW_FLOW_RESPONSE
+        if self.flow.response:
+            body = self._conn_text(
+                self.flow.response,
+                self.state.view_body_mode
+            )
+        else:
+            body = urwid.ListBox(
+                        [
+                            urwid.Text(""),
+                            urwid.Text(
+                                [
+                                    ("highlight", "No response. Press "),
+                                    ("key", "e"),
+                                    ("highlight", " and edit any aspect to add one."),
+                                ]
+                            )
+                        ]
+                   )
+        self.w = self.wrap_body(common.VIEW_FLOW_RESPONSE, body)
+        self.master.statusbar.redraw()
+
+    def refresh_flow(self, c=None):
+        if c == self.flow:
+            if self.state.view_flow_mode == common.VIEW_FLOW_RESPONSE and self.flow.response:
+                self.view_response()
+            else:
+                self.view_request()
+
+    def set_method_raw(self, m):
+        if m:
+            self.flow.request.method = m
+            self.master.refresh_flow(self.flow)
+
+    def edit_method(self, m):
+        if m == "e":
+            self.master.prompt_edit("Method", self.flow.request.method, self.set_method_raw)
+        else:
+            for i in self.method_options:
+                if i[1] == m:
+                    self.flow.request.method = i[0].upper()
+            self.master.refresh_flow(self.flow)
+
+    def save_body(self, path):
+        if not path:
+            return
+        self.state.last_saveload = path
+        if self.state.view_flow_mode == common.VIEW_FLOW_REQUEST:
+            c = self.flow.request
+        else:
+            c = self.flow.response
+        path = os.path.expanduser(path)
+        try:
+            f = file(path, "wb")
+            f.write(str(c.content))
+            f.close()
+        except IOError, v:
+            self.master.statusbar.message(v.strerror)
+
+    def set_url(self, url):
+        request = self.flow.request
+        if not request.set_url(str(url)):
+            return "Invalid URL."
+        self.master.refresh_flow(self.flow)
+
+    def set_resp_code(self, code):
+        response = self.flow.response
+        try:
+            response.code = int(code)
+        except ValueError:
+            return None
+        import BaseHTTPServer
+        if BaseHTTPServer.BaseHTTPRequestHandler.responses.has_key(int(code)):
+            response.msg = BaseHTTPServer.BaseHTTPRequestHandler.responses[int(code)][0]
+        self.master.refresh_flow(self.flow)
+
+    def set_resp_msg(self, msg):
+        response = self.flow.response
+        response.msg = msg
+        self.master.refresh_flow(self.flow)
+
+    def set_headers(self, lst, conn):
+        conn.headers = flow.ODict(lst)
+
+    def set_query(self, lst, conn):
+        conn.set_query(flow.ODict(lst))
+
+    def set_form(self, lst, conn):
+        conn.set_form_urlencoded(flow.ODict(lst))
+
+    def edit_form(self, conn):
+        self.master.view_kveditor("Editing form", conn.get_form_urlencoded().lst, self.set_form, conn)
+
+    def edit_form_confirm(self, key, conn):
+        if key == "y":
+            self.edit_form(conn)
+
+    def edit(self, part):
+        if self.state.view_flow_mode == common.VIEW_FLOW_REQUEST:
+            conn = self.flow.request
+        else:
+            if not self.flow.response:
+                self.flow.response = flow.Response(self.flow.request, 200, "OK", flow.ODict(), "")
+            conn = self.flow.response
+
+        self.flow.backup()
+        if part == "r":
+            c = self.master.spawn_editor(conn.content or "")
+            conn.content = c.rstrip("\n")
+        elif part == "f":
+            if not conn.get_form_urlencoded() and conn.content:
+                self.master.prompt_onekey(
+                    "Existing body is not a URL-encoded form. Clear and edit?",
+                    [
+                        ("yes", "y"),
+                        ("no", "n"),
+                    ],
+                    self.edit_form_confirm,
+                    conn
+                )
+            else:
+                self.edit_form(conn)
+        elif part == "h":
+            self.master.view_kveditor("Editing headers", conn.headers.lst, self.set_headers, conn)
+        elif part == "q":
+            self.master.view_kveditor("Editing query", conn.get_query().lst, self.set_query, conn)
+        elif part == "u" and self.state.view_flow_mode == common.VIEW_FLOW_REQUEST:
+            self.master.prompt_edit("URL", conn.get_url(), self.set_url)
+        elif part == "m" and self.state.view_flow_mode == common.VIEW_FLOW_REQUEST:
+            self.master.prompt_onekey("Method", self.method_options, self.edit_method)
+        elif part == "c" and self.state.view_flow_mode == common.VIEW_FLOW_RESPONSE:
+            self.master.prompt_edit("Code", str(conn.code), self.set_resp_code)
+        elif part == "m" and self.state.view_flow_mode == common.VIEW_FLOW_RESPONSE:
+            self.master.prompt_edit("Message", conn.msg, self.set_resp_msg)
+        self.master.refresh_flow(self.flow)
+
+    def _view_nextprev_flow(self, np, flow):
+        try:
+            idx = self.state.view.index(flow)
+        except IndexError:
+            return
+        if np == "next":
+            new_flow, new_idx = self.state.get_next(idx)
+        else:
+            new_flow, new_idx = self.state.get_prev(idx)
+        if new_idx is None:
+            return
+        self.master.view_flow(new_flow)
+
+    def view_next_flow(self, flow):
+        return self._view_nextprev_flow("next", flow)
+
+    def view_prev_flow(self, flow):
+        return self._view_nextprev_flow("prev", flow)
+
+    def keypress(self, size, key):
+        if key == " ":
+            self.view_next_flow(self.flow)
+            return key
+
+        key = common.shortcuts(key)
+        if self.state.view_flow_mode == common.VIEW_FLOW_REQUEST:
+            conn = self.flow.request
+        else:
+            conn = self.flow.response
+
+        if key == "q":
+            self.master.view_flowlist()
+            key = None
+        elif key == "tab":
+            if self.state.view_flow_mode == common.VIEW_FLOW_REQUEST:
+                self.view_response()
+            else:
+                self.view_request()
+        elif key in ("up", "down", "page up", "page down"):
+            # Why doesn't this just work??
+            self.w.keypress(size, key)
+        elif key == "a":
+            self.flow.accept_intercept()
+            self.master.view_flow(self.flow)
+        elif key == "A":
+            self.master.accept_all()
+            self.master.view_flow(self.flow)
+        elif key == "d":
+            if self.state.flow_count() == 1:
+                self.master.view_flowlist()
+            elif self.state.view.index(self.flow) == len(self.state.view)-1:
+                self.view_prev_flow(self.flow)
+            else:
+                self.view_next_flow(self.flow)
+            f = self.flow
+            f.kill(self.master)
+            self.state.delete_flow(f)
+        elif key == "D":
+            f = self.master.duplicate_flow(self.flow)
+            self.master.view_flow(f)
+            self.master.currentflow = f
+            self.master.statusbar.message("Duplicated.")
+        elif key == "e":
+            if self.state.view_flow_mode == common.VIEW_FLOW_REQUEST:
+                self.master.prompt_onekey(
+                    "Edit request",
+                    (
+                        ("query", "q"),
+                        ("form", "f"),
+                        ("url", "u"),
+                        ("header", "h"),
+                        ("raw body", "r"),
+                        ("method", "m"),
+                    ),
+                    self.edit
+                )
+            else:
+                self.master.prompt_onekey(
+                    "Edit response",
+                    (
+                        ("code", "c"),
+                        ("message", "m"),
+                        ("header", "h"),
+                        ("raw body", "r"),
+                    ),
+                    self.edit
+                )
+            key = None
+        elif key == "m":
+            self.master.prompt_onekey(
+                "View",
+                (
+                    ("raw", "r"),
+                    ("pretty", "p"),
+                    ("hex", "h"),
+                ),
+                self.master.changeview
+            )
+            key = None
+        elif key == "p":
+            self.view_prev_flow(self.flow)
+        elif key == "r":
+            r = self.master.replay_request(self.flow)
+            if r:
+                self.master.statusbar.message(r)
+            self.master.refresh_flow(self.flow)
+        elif key == "V":
+            self.state.revert(self.flow)
+            self.master.refresh_flow(self.flow)
+        elif key == "W":
+            self.master.path_prompt(
+                "Save this flow: ",
+                self.state.last_saveload,
+                self.master.save_one_flow,
+                self.flow
+            )
+        elif key == "v":
+            if conn and conn.content:
+                t = conn.headers["content-type"] or [None]
+                t = t[0]
+                self.master.spawn_external_viewer(conn.content, t)
+        elif key == "b":
+            if conn:
+                if self.state.view_flow_mode == common.VIEW_FLOW_REQUEST:
+                    self.master.path_prompt(
+                        "Save request body: ",
+                        self.state.last_saveload,
+                        self.save_body
+                    )
+                else:
+                    self.master.path_prompt(
+                        "Save response body: ",
+                        self.state.last_saveload,
+                        self.save_body
+                    )
+        elif key == "|":
+            self.master.path_prompt(
+                "Send flow to script: ", self.state.last_script,
+                self.master.run_script_once, self.flow
+            )
+        elif key == "z":
+            if conn:
+                e = conn.headers["content-encoding"] or ["identity"]
+                if e[0] != "identity":
+                    conn.decode()
+                else:
+                    self.master.prompt_onekey(
+                        "Select encoding: ",
+                        (
+                            ("gzip", "z"),
+                            ("deflate", "d"),
+                        ),
+                        self.encode_callback,
+                        conn
+                    )
+                self.master.refresh_flow(self.flow)
+        else:
+            return key
+
+    def encode_callback(self, key, conn):
+        encoding_map = {
+            "z": "gzip",
+            "d": "deflate",
+        }
+        conn.encode(encoding_map[key])
+        self.master.refresh_flow(self.flow)

libmproxy/console/help.py

@@ -0,0 +1,132 @@
+# Copyright (C) 2012  Aldo Cortesi
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import urwid
+import common
+from .. import filt
+
+class HelpView(urwid.ListBox):
+    def __init__(self, master, help_context, state):
+        self.master, self.state = master, state
+        self.help_context = help_context or []
+        urwid.ListBox.__init__(
+            self,
+            self.helptext()
+        )
+
+    def keypress(self, size, key):
+        key = common.shortcuts(key)
+        if key == "q":
+            self.master.statusbar = self.state[0]
+            self.master.body = self.state[1]
+            self.master.header = self.state[2]
+            self.master.make_view()
+            return None
+        return urwid.ListBox.keypress(self, size, key)
+
+    def helptext(self):
+        text = []
+        text.append(urwid.Text([("head", "Keys for this view:\n")]))
+        text.extend(self.help_context)
+
+        text.append(urwid.Text([("head", "\n\nMovement:\n")]))
+        keys = [
+            ("j, k", "up, down"),
+            ("h, l", "left, right (in some contexts)"),
+            ("space", "page down"),
+            ("pg up/down", "page up/down"),
+            ("arrows", "up, down, left, right"),
+        ]
+        text.extend(common.format_keyvals(keys, key="key", val="text", indent=4))
+
+        text.append(urwid.Text([("head", "\n\nGlobal keys:\n")]))
+        keys = [
+            ("c", "client replay"),
+            ("i", "set interception pattern"),
+
+            ("o", "toggle options:"),
+            (None,
+                common.highlight_key("anticache", "a") +
+                [("text", ": prevent cached responses")]
+            ),
+            (None,
+                common.highlight_key("anticomp", "c") +
+                [("text", ": prevent compressed responses")]
+            ),
+            (None,
+                common.highlight_key("killextra", "k") +
+                [("text", ": kill requests not part of server replay")]
+            ),
+            (None,
+                common.highlight_key("norefresh", "n") +
+                [("text", ": disable server replay response refresh")]
+            ),
+
+            ("q", "quit / return to flow list"),
+            ("Q", "quit without confirm prompt"),
+            ("R", "set reverse proxy mode"),
+            ("s", "set/unset script"),
+            ("S", "server replay"),
+            ("t", "set sticky cookie expression"),
+            ("u", "set sticky auth expression"),
+        ]
+        text.extend(common.format_keyvals(keys, key="key", val="text", indent=4))
+
+        text.append(urwid.Text([("head", "\n\nFilter expressions:\n")]))
+        f = []
+        for i in filt.filt_unary:
+            f.append(
+                ("~%s"%i.code, i.help)
+            )
+        for i in filt.filt_rex:
+            f.append(
+                ("~%s regex"%i.code, i.help)
+            )
+        for i in filt.filt_int:
+            f.append(
+                ("~%s int"%i.code, i.help)
+            )
+        f.sort()
+        f.extend(
+            [
+                ("!", "unary not"),
+                ("&", "and"),
+                ("|", "or"),
+                ("(...)", "grouping"),
+            ]
+        )
+        text.extend(common.format_keyvals(f, key="key", val="text", indent=4))
+
+        text.append(
+            urwid.Text(
+               [
+                    "\n",
+                    ("text", "    Regexes are Python-style.\n"),
+                    ("text", "    Regexes can be specified as quoted strings.\n"),
+                    ("text", "    Header matching (~h, ~hq, ~hs) is against a string of the form \"name: value\".\n"),
+                    ("text", "    Expressions with no operators are regex matches against URL.\n"),
+                    ("text", "    Default binary operator is &.\n"),
+                    ("head", "\n    Examples:\n"),
+               ]
+            )
+        )
+        examples = [
+                ("google\.com", "Url containing \"google.com"),
+                ("~q ~b test", "Requests where body contains \"test\""),
+                ("!(~q & ~t \"text/html\")", "Anything but requests with a text/html content type."),
+        ]
+        text.extend(common.format_keyvals(examples, key="key", val="text", indent=4))
+        return text
+

libmproxy/console/kveditor.py

@@ -0,0 +1,270 @@
+# Copyright (C) 2012  Aldo Cortesi
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import copy
+import urwid
+import common
+from .. import utils
+
+
+def _mkhelp():
+    text = []
+    keys = [
+        ("A", "insert row before cursor"),
+        ("a", "add row after cursor"),
+        ("d", "delete row"),
+        ("e", "spawn external editor on current field"),
+        ("q", "return to flow view"),
+        ("esc", "return to flow view/exit field edit mode"),
+        ("tab", "next field"),
+        ("enter", "edit field"),
+    ]
+    text.extend(common.format_keyvals(keys, key="key", val="text", indent=4))
+    return text
+help_context = _mkhelp()
+
+
+class SText(common.WWrap):
+    def __init__(self, txt, focused):
+        w = urwid.Text(txt, wrap="any")
+        if focused:
+            w = urwid.AttrWrap(w, "focusfield")
+        common.WWrap.__init__(self, w)
+
+    def get_text(self):
+        return self.w.get_text()[0]
+
+    def keypress(self, size, key):
+        return key
+
+    def selectable(self):
+        return True
+
+
+class SEdit(common.WWrap):
+    def __init__(self, txt):
+        w = urwid.Edit(edit_text=txt, wrap="any", multiline=True)
+        w = urwid.AttrWrap(w, "editfield")
+        common.WWrap.__init__(self, w)
+
+    def get_text(self):
+        return self.w.get_text()[0]
+
+    def selectable(self):
+        return True
+
+
+class KVItem(common.WWrap):
+    def __init__(self, focused, editing, maxk, k, v):
+        self.focused, self.editing, self.maxk = focused, editing, maxk
+        if focused == 0 and editing:
+            self.editing = self.kf = SEdit(k)
+        else:
+            self.kf = SText(k, True if focused == 0 else False)
+
+        if focused == 1 and editing:
+            self.editing = self.vf = SEdit(v)
+        else:
+            self.vf = SText(v, True if focused == 1 else False)
+
+        w = urwid.Columns(
+            [
+                ("fixed", maxk + 2, self.kf),
+                self.vf
+            ],
+            dividechars = 2
+        )
+        if focused is not None:
+            w.set_focus_column(focused)
+        common.WWrap.__init__(self, w)
+
+    def get_kv(self):
+        return (self.kf.get_text(), self.vf.get_text())
+
+    def keypress(self, s, k):
+        if self.editing:
+            k = self.editing.keypress((s[0]-self.maxk-4,), k)
+        return k
+
+    def selectable(self):
+        return True
+
+
+KEY_MAX = 30
+class KVWalker(urwid.ListWalker):
+    def __init__(self, lst, editor):
+        self.lst, self.editor = lst, editor
+        self.maxk = min(max(len(v[0]) for v in lst), KEY_MAX) if lst else 20
+        if self.maxk < 20:
+            self.maxk = 20
+        self.focus = 0
+        self.focus_col = 0
+        self.editing = False
+
+    def _modified(self):
+        self.editor.show_empty_msg()
+        return urwid.ListWalker._modified(self)
+
+    def get_current_value(self):
+        if self.lst:
+            return self.lst[self.focus][self.focus_col]
+
+    def set_current_value(self, val):
+        row = list(self.lst[self.focus])
+        row[self.focus_col] = val
+        self.lst[self.focus] = tuple(row)
+
+    def delete_focus(self):
+        if self.lst:
+            del self.lst[self.focus]
+            self.focus = min(len(self.lst)-1, self.focus)
+            self._modified()
+
+    def _insert(self, pos):
+        self.focus = pos
+        self.lst.insert(self.focus, ("", ""))
+        self.focus_col = 0
+        self.start_edit()
+
+    def insert(self):
+        return self._insert(self.focus)
+
+    def add(self):
+        return self._insert(min(self.focus + 1, len(self.lst)))
+
+    def start_edit(self):
+        if self.lst:
+            self.editing = KVItem(self.focus_col, True, self.maxk, *self.lst[self.focus])
+            self._modified()
+
+    def stop_edit(self):
+        if self.editing:
+            self.lst[self.focus] = self.editing.get_kv()
+            self.editing = False
+            self._modified()
+
+    def left(self):
+        self.focus_col = 0
+        self._modified()
+
+    def right(self):
+        self.focus_col = 1
+        self._modified()
+
+    def tab_next(self):
+        self.stop_edit()
+        if self.focus_col == 0:
+            self.focus_col = 1
+        elif self.focus != len(self.lst)-1:
+            self.focus_col = 0
+            self.focus += 1
+        self._modified()
+
+    def get_focus(self):
+        if self.editing:
+            return self.editing, self.focus
+        elif self.lst:
+            return KVItem(self.focus_col, False, self.maxk, *self.lst[self.focus]), self.focus
+        else:
+            return None, None
+
+    def set_focus(self, focus):
+        self.stop_edit()
+        self.focus = focus
+
+    def get_next(self, pos):
+        if pos+1 >= len(self.lst):
+            return None, None
+        return KVItem(None, False, self.maxk, *self.lst[pos+1]), pos+1
+
+    def get_prev(self, pos):
+        if pos-1 < 0:
+            return None, None
+        return KVItem(None, False, self.maxk, *self.lst[pos-1]), pos-1
+
+
+class KVListBox(urwid.ListBox):
+    def __init__(self, lw):
+        urwid.ListBox.__init__(self, lw)
+
+
+class KVEditor(common.WWrap):
+    def __init__(self, master, title, value, callback, *cb_args, **cb_kwargs):
+        value = copy.deepcopy(value)
+        self.master, self.title, self.value, self.callback = master, title, value, callback
+        self.cb_args, self.cb_kwargs = cb_args, cb_kwargs
+        p = urwid.Text(title)
+        p = urwid.Padding(p, align="left", width=("relative", 100))
+        p = urwid.AttrWrap(p, "heading")
+        self.walker = KVWalker(self.value, self)
+        self.lb = KVListBox(self.walker)
+        self.w = urwid.Frame(self.lb, header = p)
+        self.master.statusbar.update("")
+        self.show_empty_msg()
+
+    def show_empty_msg(self):
+        if self.walker.lst:
+            self.w.set_footer(None)
+        else:
+            self.w.set_footer(
+                urwid.Text(
+                    [
+                        ("highlight", "No values. Press "),
+                        ("key", "a"),
+                        ("highlight", " to add some."),
+                    ]
+                )
+            )
+
+    def keypress(self, size, key):
+        if self.walker.editing:
+            if key in ["esc", "enter"]:
+                self.walker.stop_edit()
+            elif key == "tab":
+                pf, pfc = self.walker.focus, self.walker.focus_col
+                self.walker.tab_next()
+                if self.walker.focus == pf and self.walker.focus_col != pfc:
+                    self.walker.start_edit()
+            else:
+                self.w.keypress(size, key)
+            return None
+
+        key = common.shortcuts(key)
+        if key in ["q", "esc"]:
+            self.callback(self.walker.lst, *self.cb_args, **self.cb_kwargs)
+            self.master.pop_view()
+        elif key in ["h", "left"]:
+            self.walker.left()
+        elif key in ["l", "right"]:
+            self.walker.right()
+        elif key == "tab":
+            self.walker.tab_next()
+        elif key == "a":
+            self.walker.add()
+        elif key == "A":
+            self.walker.insert()
+        elif key == "d":
+            self.walker.delete_focus()
+        elif key == "e":
+            o = self.walker.get_current_value()
+            if o is not None:
+                n = self.master.spawn_editor(o)
+                n = utils.clean_hanging_newline(n)
+                self.walker.set_current_value(n)
+                self.walker._modified()
+        elif key in ["enter"]:
+            self.walker.start_edit()
+        else:
+            return self.w.keypress(size, key)

libmproxy/console/palettes.py

@@ -0,0 +1,50 @@
+# Copyright (C) 2012  Aldo Cortesi
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+dark = [
+    ('body', 'black', 'dark cyan'),
+    ('foot', 'light gray', 'default'),
+    ('title', 'white,bold', 'default',),
+    ('editline', 'white', 'default',),
+
+    # Status bar & heading
+    ('heading', 'light gray', "dark blue", None, "g85", "dark blue"),
+    ('heading_key', 'light cyan', "dark blue", None, "light cyan", "dark blue"),
+    ('heading_inactive', 'white', 'dark gray', None, "g58", "g11"),
+
+    # Help
+    ('key', 'light cyan', 'default'),
+    ('head', 'white,bold', 'default'),
+    ('text', 'light gray', 'default'),
+
+    # List and Connections
+    ('method', 'dark cyan', 'default'),
+    ('focus', 'yellow', 'default'),
+    ('goodcode', 'light green', 'default'),
+    ('error', 'light red', 'default'),
+    ('header', 'dark cyan', 'default'),
+    ('highlight', 'white,bold', 'default'),
+    ('intercept', 'brown', 'default', None, "#f60", "default"),
+    ('replay', 'light green', 'default', None, "#0f0", "default"),
+    ('ack', 'light red', 'default'),
+
+    # Hex view
+    ('offset', 'dark cyan', 'default'),
+
+    # KV Editor
+    ('focusfield', 'black', 'light gray'),
+    ('editfield', 'black', 'light cyan'),
+]

libmproxy/controller.py

@@ -1,19 +1,18 @@
 # Copyright (C) 2010  Aldo Cortesi
-# 
+#
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
-# 
+#
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
-# 
+#
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import sys
 import Queue, threading
 
 should_exit = False
@@ -26,11 +25,12 @@ class Msg:
         self.acked = False
 
     def _ack(self, data=False):
-        self.acked = True
-        if data is None:
-            self.q.put(data)
-        else:
-            self.q.put(data or self)
+        if not self.acked:
+            self.acked = True
+            if data is None:
+                self.q.put(data)
+            else:
+                self.q.put(data or self)
 
     def _send(self, masterq):
         self.acked = False
@@ -81,6 +81,8 @@ class Master:
         return changed
 
     def run(self):
+        global should_exit
+        should_exit = False
         if self.server:
             slave = Slave(self.masterq, self.server)
             slave.start()

libmproxy/dump.py

@@ -1,5 +1,20 @@
+# Copyright (C) 2012  Aldo Cortesi
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 import sys, os
-import flow, filt, utils, script
+import flow, filt, utils
 
 class DumpError(Exception): pass
 
@@ -103,7 +118,10 @@ class DumpMaster(flow.FlowMaster):
                 freader = flow.FlowReader(f)
             except IOError, v:
                 raise DumpError(v.strerror)
-            self.load_flows(freader)
+            try:
+                self.load_flows(freader)
+            except flow.FlowReadError, v:
+                raise DumpError(v)
 
 
     def _readflow(self, path):
@@ -187,7 +205,6 @@ class DumpMaster(flow.FlowMaster):
     def handle_error(self, msg):
         f = flow.FlowMaster.handle_error(self, msg)
         if f:
-            msg._ack()
             self._process_flow(f)
         return f
 

libmproxy/encoding.py

@@ -1,3 +1,18 @@
+# Copyright (C) 2012  Aldo Cortesi
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 """
     Utility functions for decoding response bodies.
 """

libmproxy/filt.py

@@ -1,16 +1,15 @@
-
 # Copyright (C) 2010  Aldo Cortesi
-# 
+#
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
-# 
+#
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
-# 
+#
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
@@ -19,7 +18,7 @@
 
         ~q          Request
         ~s          Response
-    
+
     Headers:
 
         Patterns are matched against "name: value" strings. Field names are
@@ -34,12 +33,14 @@
         ~bq rex     Expression in the body of response
         ~t rex      Shortcut for content-type header.
 
+        ~m rex      Method
         ~u rex      URL
         ~c CODE     Response code.
-        rex         Equivalent to ~u rex 
+        rex         Equivalent to ~u rex
 """
 import re, sys
 import contrib.pyparsing as pp
+import flow
 
 
 class _Token:
@@ -56,19 +57,27 @@ class _Action(_Token):
         return klass(*toks[1:])
 
 
+class FErr(_Action):
+    code = "e"
+    help = "Match error"
+    def __call__(self, f):
+        return True if f.error else False
+
+
 class FReq(_Action):
     code = "q"
-    help = "Match request"
-    def __call__(self, conn):
-        return not conn._is_response()
+    help = "Match request with no response"
+    def __call__(self, f):
+        if not f.response:
+            return True
 
 
 class FResp(_Action):
     code = "s"
     help = "Match response"
-    def __call__(self, conn):
-        return conn._is_response()
-    
+    def __call__(self, f):
+        return True if f.response else False
+
 
 class _Rex(_Action):
     def __init__(self, expr):
@@ -83,77 +92,69 @@ def _check_content_type(expr, o):
     if val and re.search(expr, val[0]):
         return True
     return False
-    
+
 
 class FContentType(_Rex):
     code = "t"
     help = "Content-type header"
-    def __call__(self, o):
-        if _check_content_type(self.expr, o):
+    def __call__(self, f):
+        if _check_content_type(self.expr, f.request):
             return True
-        elif o._is_response() and _check_content_type(self.expr, o.request):
+        elif f.response and _check_content_type(self.expr, f.response):
             return True
-        else:
-            return False
+        return False
 
 
 class FRequestContentType(_Rex):
     code = "tq"
     help = "Request Content-Type header"
-    def __call__(self, o):
-        if o._is_response():
-            return _check_content_type(self.expr, o.request)
-        else:
-            return _check_content_type(self.expr, o)
+    def __call__(self, f):
+        return _check_content_type(self.expr, f.request)
 
 
 class FResponseContentType(_Rex):
     code = "ts"
     help = "Request Content-Type header"
-    def __call__(self, o):
-        if o._is_response():
-            return _check_content_type(self.expr, o)
-        else:
-            return False
+    def __call__(self, f):
+        if f.response:
+            return _check_content_type(self.expr, f.response)
+        return False
 
 
 class FHead(_Rex):
     code = "h"
     help = "Header"
-    def __call__(self, o):
-        val = o.headers.match_re(self.expr)
-        if not val and o._is_response():
-            val = o.request.headers.match_re(self.expr)
-        return val
-    
+    def __call__(self, f):
+        if f.request.headers.match_re(self.expr):
+            return True
+        elif f.response and f.response.headers.match_re(self.expr):
+            return True
+        return False
+
 
 class FHeadRequest(_Rex):
     code = "hq"
     help = "Request header"
-    def __call__(self, o):
-        if o._is_response():
-            h = o.request.headers
-        else:
-            h = o.headers
-        return h.match_re(self.expr)
+    def __call__(self, f):
+        if f.request.headers.match_re(self.expr):
+            return True
 
 
 class FHeadResponse(_Rex):
     code = "hs"
     help = "Response header"
-    def __call__(self, o):
-        if not o._is_response():
-            return False
-        return o.headers.match_re(self.expr)
+    def __call__(self, f):
+        if f.response and f.response.headers.match_re(self.expr):
+            return True
 
 
 class FBod(_Rex):
     code = "b"
     help = "Body"
-    def __call__(self, o):
-        if o.content and re.search(self.expr, o.content):
+    def __call__(self, f):
+        if f.request.content and re.search(self.expr, f.request.content):
             return True
-        elif o._is_response() and o.request.content and re.search(self.expr, o.request.content):
+        elif f.response and f.response.content and re.search(self.expr, f.response.content):
             return True
         return False
 
@@ -161,24 +162,25 @@ class FBod(_Rex):
 class FBodRequest(_Rex):
     code = "bq"
     help = "Request body"
-    def __call__(self, o):
-        if o._is_response() and o.request.content and re.search(self.expr, o.request.content):
+    def __call__(self, f):
+        if f.request.content and re.search(self.expr, f.request.content):
             return True
-        elif not o._is_response() and o.content and re.search(self.expr, o.content):
-            return True
-        return False
 
 
 class FBodResponse(_Rex):
     code = "bs"
     help = "Response body"
-    def __call__(self, o):
-        if not o._is_response():
-            return False
-        elif o.content and re.search(self.expr, o.content):
+    def __call__(self, f):
+        if f.response and f.response.content and re.search(self.expr, f.response.content):
             return True
-        return False
-        
+
+
+class FMethod(_Rex):
+    code = "m"
+    help = "Method"
+    def __call__(self, f):
+        return bool(re.search(self.expr, f.request.method, re.IGNORECASE))
+
 
 class FUrl(_Rex):
     code = "u"
@@ -190,12 +192,8 @@ class FUrl(_Rex):
             toks = toks[1:]
         return klass(*toks)
 
-    def __call__(self, o):
-        if o._is_response():
-            c = o.request
-        else:
-            c = o
-        return re.search(self.expr, c.get_url())
+    def __call__(self, f):
+        return re.search(self.expr, f.request.get_url())
 
 
 class _Int(_Action):
@@ -206,10 +204,9 @@ class _Int(_Action):
 class FCode(_Int):
     code = "c"
     help = "HTTP response code"
-    def __call__(self, o):
-        if o._is_response():
-            return o.code == self.num
-        return False
+    def __call__(self, f):
+        if f.response and f.response.code == self.num:
+            return True
 
 
 class FAnd(_Token):
@@ -221,8 +218,8 @@ class FAnd(_Token):
         for i in self.lst:
             i.dump(indent+1, fp)
 
-    def __call__(self, o):
-        return all([i(o) for i in self.lst])
+    def __call__(self, f):
+        return all(i(f) for i in self.lst)
 
 
 class FOr(_Token):
@@ -234,8 +231,8 @@ class FOr(_Token):
         for i in self.lst:
             i.dump(indent+1, fp)
 
-    def __call__(self, o):
-        return any([i(o) for i in self.lst])
+    def __call__(self, f):
+        return any(i(f) for i in self.lst)
 
 
 class FNot(_Token):
@@ -246,12 +243,14 @@ class FNot(_Token):
         print >> fp, "\t"*indent, self.__class__.__name__
         self.itm.dump(indent + 1, fp)
 
-    def __call__(self, o):
-        return not self.itm(o)
+    def __call__(self, f):
+        return not self.itm(f)
+
 
 filt_unary = [
     FReq,
-    FResp
+    FResp,
+    FErr
 ]
 filt_rex = [
     FHeadRequest,
@@ -260,6 +259,7 @@ filt_rex = [
     FBodRequest,
     FBodResponse,
     FBod,
+    FMethod,
     FUrl,
     FRequestContentType,
     FResponseContentType,
@@ -277,7 +277,7 @@ def _make():
         f.setParseAction(klass.make)
         parts.append(f)
 
-    simplerex = "".join([c for c in pp.printables if c not in  "()~'\""])
+    simplerex = "".join(c for c in pp.printables if c not in  "()~'\"")
     rex = pp.Word(simplerex) |\
           pp.QuotedString("\"", escChar='\\') |\
           pp.QuotedString("'", escChar='\\')

libmproxy/flow.py

@@ -1,18 +1,29 @@
+# Copyright (C) 2012  Aldo Cortesi
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 """
     This module provides more sophisticated flow tracking. These match requests
     with their responses, and provide filtering and interception facilities.
 """
-import json, hashlib, Cookie, cookielib, base64, copy, re
+import hashlib, Cookie, cookielib, copy, re, urlparse
 import time
-import netstring, filt, script, utils, encoding, proxy
+import tnetstring, filt, script, utils, encoding, proxy
 from email.utils import parsedate_tz, formatdate, mktime_tz
 import controller, version
 
-class RunException(Exception):
-    def __init__(self, msg, returncode, errout):
-        Exception.__init__(self, msg)
-        self.returncode = returncode
-        self.errout = errout
+HDR_FORM_URLENCODED = "application/x-www-form-urlencoded"
 
 
 class ScriptContext:
@@ -21,29 +32,51 @@ class ScriptContext:
 
     def log(self, *args, **kwargs):
         """
-            Logs an event. 
-            
+            Logs an event.
+
             How this is handled depends on the front-end. mitmdump will display
             events if the eventlog flag ("-e") was passed. mitmproxy sends
             output to the eventlog for display ("v" keyboard shortcut).
         """
         self._master.add_event(*args, **kwargs)
 
+    def duplicate_flow(self, f):
+        """
+            Returns a duplicate of the specified flow. The flow is also
+            injected into the current state, and is ready for editing, replay,
+            etc.
+        """
+        self._master.pause_scripts = True
+        f = self._master.duplicate_flow(f)
+        self._master.pause_scripts = False
+        return f
 
-class Headers:
+    def replay_request(self, f):
+        """
+            Replay the request on the current flow. The response will be added
+            to the flow object.
+        """
+        self._master.replay_request(f)
+
+
+class ODict:
+    """
+        A dictionary-like object for managing ordered (key, value) data.
+    """
     def __init__(self, lst=None):
-        if lst:
-            self.lst = lst
-        else:
-            self.lst = []
+        self.lst = lst or []
 
     def _kconv(self, s):
-        return s.lower()
+        return s
 
     def __eq__(self, other):
         return self.lst == other.lst
 
     def __getitem__(self, k):
+        """
+            Returns a list of values matching key.
+
+        """
         ret = []
         k = self._kconv(k)
         for i in self.lst:
@@ -58,14 +91,29 @@ class Headers:
                 new.append(i)
         return new
 
-    def __setitem__(self, k, hdrs):
+    def __len__(self):
+        """
+            Total number of (key, value) pairs.
+        """
+        return len(self.lst)
+
+    def __setitem__(self, k, valuelist):
+        """
+            Sets the values for key k. If there are existing values for this
+            key, they are cleared.
+        """
+        if isinstance(valuelist, basestring):
+            raise ValueError("ODict valuelist should be lists.")
         k = self._kconv(k)
         new = self._filter_lst(k, self.lst)
-        for i in hdrs:
+        for i in valuelist:
             new.append((k, i))
         self.lst = new
 
     def __delitem__(self, k):
+        """
+            Delete all items matching k.
+        """
         self.lst = self._filter_lst(k, self.lst)
 
     def __contains__(self, k):
@@ -89,22 +137,34 @@ class Headers:
             Returns a copy of this object.
         """
         lst = copy.deepcopy(self.lst)
-        return Headers(lst)
+        return self.__class__(lst)
 
     def __repr__(self):
-        """
-            Returns a string containing a formatted header string.
-        """
-        headerElements = []
+        elements = []
         for itm in self.lst:
-            headerElements.append(itm[0] + ": " + itm[1])
-        headerElements.append("")
-        return "\r\n".join(headerElements)
+            elements.append(itm[0] + ": " + itm[1])
+        elements.append("")
+        return "\r\n".join(elements)
+
+    def in_any(self, key, value, caseless=False):
+        """
+            Do any of the values matching key contain value?
+
+            If caseless is true, value comparison is case-insensitive.
+        """
+        if caseless:
+            value = value.lower()
+        for i in self[key]:
+            if caseless:
+                i = i.lower()
+            if value in i:
+                return True
+        return False
 
     def match_re(self, expr):
         """
-            Match the regular expression against each header. For each (key,
-            value) pair a string of the following format is matched against:
+            Match the regular expression against each (key, value) pair. For
+            each pair a string of the following format is matched against:
 
             "key: value"
         """
@@ -114,32 +174,9 @@ class Headers:
                 return True
         return False
 
-    def read(self, fp):
-        """
-            Read a set of headers from a file pointer. Stop once a blank line
-            is reached.
-        """
-        ret = []
-        name = ''
-        while 1:
-            line = fp.readline()
-            if not line or line == '\r\n' or line == '\n':
-                break
-            if line[0] in ' \t':
-                # continued header
-                ret[-1][1] = ret[-1][1] + '\r\n ' + line.strip()
-            else:
-                i = line.find(':')
-                # We're being liberal in what we accept, here.
-                if i > 0:
-                    name = line[:i]
-                    value = line[i+1:].strip()
-                    ret.append([name, value])
-        self.lst = ret
-
     def replace(self, pattern, repl, *args, **kwargs):
         """
-            Replaces a regular expression pattern with repl in both header keys
+            Replaces a regular expression pattern with repl in both keys
             and values. Returns the number of replacements made.
         """
         nlst, count = [], 0
@@ -153,6 +190,15 @@ class Headers:
         return count
 
 
+class ODictCaseless(ODict):
+    """
+        A variant of ODict with "caseless" keys. This version _preserves_ key
+        case, but does not consider case when setting or getting items.
+    """
+    def _kconv(self, s):
+        return s.lower()
+
+
 class HTTPMsg(controller.Msg):
     def decode(self):
         """
@@ -184,20 +230,21 @@ class Request(HTTPMsg):
         An HTTP request.
 
         Exposes the following attributes:
-            
+
             client_conn: ClientConnection object, or None if this is a replay.
-            headers: Headers object
+            headers: ODictCaseless object
             content: Content of the request, or None
-            
+
             scheme: URL scheme (http/https)
             host: Host portion of the URL
             port: Destination port
             path: Path portion of the URL
 
             timestamp: Seconds since the epoch
-            method: HTTP method 
+            method: HTTP method
     """
     def __init__(self, client_conn, host, port, scheme, method, path, headers, content, timestamp=None):
+        assert isinstance(headers, ODictCaseless)
         self.client_conn = client_conn
         self.host, self.port, self.scheme = host, port, scheme
         self.method, self.path, self.headers, self.content = method, path, headers, content
@@ -234,9 +281,9 @@ class Request(HTTPMsg):
             decode appropriately.
         """
         if self.headers["accept-encoding"]:
-            self.headers["accept-encoding"] = [', '.join([
+            self.headers["accept-encoding"] = [', '.join(
                 e for e in encoding.ENCODINGS if e in self.headers["accept-encoding"][0]
-            ])]
+            )]
 
     def _set_replay(self):
         self.client_conn = None
@@ -263,8 +310,8 @@ class Request(HTTPMsg):
         self.scheme = state["scheme"]
         self.method = state["method"]
         self.path = state["path"]
-        self.headers = Headers._from_state(state["headers"])
-        self.content = base64.decodestring(state["content"])
+        self.headers = ODictCaseless._from_state(state["headers"])
+        self.content = state["content"]
         self.timestamp = state["timestamp"]
 
     def _get_state(self):
@@ -276,7 +323,7 @@ class Request(HTTPMsg):
             method = self.method,
             path = self.path,
             headers = self.headers._get_state(),
-            content = base64.encodestring(self.content),
+            content = self.content,
             timestamp = self.timestamp,
         )
 
@@ -289,8 +336,8 @@ class Request(HTTPMsg):
             str(state["scheme"]),
             str(state["method"]),
             str(state["path"]),
-            Headers._from_state(state["headers"]),
-            base64.decodestring(state["content"]),
+            ODictCaseless._from_state(state["headers"]),
+            state["content"],
             state["timestamp"]
         )
 
@@ -305,28 +352,58 @@ class Request(HTTPMsg):
             Returns a copy of this object.
         """
         c = copy.copy(self)
+        c.acked = True
         c.headers = self.headers.copy()
         return c
 
-    def _hostport(self):
-        if (self.port, self.scheme) in [(80, "http"), (443, "https")]:
-            host = self.host
-        else:
-            host = "%s:%s"%(self.host, self.port)
-        return host
+    def get_form_urlencoded(self):
+        """
+            Retrieves the URL-encoded form data, returning an ODict object.
+            Returns an empty ODict if there is no data or the content-type
+            indicates non-form data.
+        """
+        if self.headers.in_any("content-type", HDR_FORM_URLENCODED, True):
+            return ODict(utils.urldecode(self.content))
+        return ODict([])
+
+    def set_form_urlencoded(self, odict):
+        """
+            Sets the body to the URL-encoded form data, and adds the
+            appropriate content-type header. Note that this will destory the
+            existing body if there is one.
+        """
+        self.headers["Content-Type"] = [HDR_FORM_URLENCODED]
+        self.content = utils.urlencode(odict.lst)
+
+    def get_query(self):
+        """
+            Gets the request query string. Returns an ODict object.
+        """
+        _, _, _, _, query, _ = urlparse.urlparse(self.get_url())
+        if query:
+            return ODict(utils.urldecode(query))
+        return ODict([])
+
+    def set_query(self, odict):
+        """
+            Takes an ODict object, and sets the request query string.
+        """
+        scheme, netloc, path, params, _, fragment = urlparse.urlparse(self.get_url())
+        query = utils.urlencode(odict.lst)
+        self.set_url(urlparse.urlunparse([scheme, netloc, path, params, query, fragment]))
 
     def get_url(self):
         """
             Returns a URL string, constructed from the Request's URL compnents.
         """
-        return "%s://%s%s"%(self.scheme, self._hostport(), self.path)
+        return utils.unparse_url(self.scheme, self.host, self.port, self.path)
 
     def set_url(self, url):
         """
             Parses a URL specification, and updates the Request's information
             accordingly.
 
-            Returns False if the URL was invalid, True if the request succeeded. 
+            Returns False if the URL was invalid, True if the request succeeded.
         """
         parts = utils.parse_url(url)
         if not parts:
@@ -334,9 +411,6 @@ class Request(HTTPMsg):
         self.scheme, self.host, self.port, self.path = parts
         return True
 
-    def _is_response(self):
-        return False
-
     def _assemble(self, _proxy = False):
         """
             Assembles the request for transmission to the server. We make some
@@ -357,12 +431,12 @@ class Request(HTTPMsg):
             ]
         )
         if not 'host' in headers:
-            headers["host"] = [self._hostport()]
+            headers["host"] = [utils.hostport(self.scheme, self.host, self.port)]
         content = self.content
-        if content is not None:
-            headers["content-length"] = [str(len(content))]
-        else:
+        if content is None:
             content = ""
+        else:
+            headers["content-length"] = [str(len(content))]
         if self.close:
             headers["connection"] = ["close"]
         if not _proxy:
@@ -392,11 +466,12 @@ class Response(HTTPMsg):
             request: Request object.
             code: HTTP response code
             msg: HTTP response message
-            headers: Headers object
+            headers: ODict object
             content: Response content
             timestamp: Seconds since the epoch
     """
     def __init__(self, request, code, msg, headers, content, timestamp=None):
+        assert isinstance(headers, ODictCaseless)
         self.request = request
         self.code, self.msg = code, msg
         self.headers, self.content = headers, content
@@ -466,8 +541,8 @@ class Response(HTTPMsg):
     def _load_state(self, state):
         self.code = state["code"]
         self.msg = state["msg"]
-        self.headers = Headers._from_state(state["headers"])
-        self.content = base64.decodestring(state["content"])
+        self.headers = ODictCaseless._from_state(state["headers"])
+        self.content = state["content"]
         self.timestamp = state["timestamp"]
 
     def _get_state(self):
@@ -476,7 +551,7 @@ class Response(HTTPMsg):
             msg = self.msg,
             headers = self.headers._get_state(),
             timestamp = self.timestamp,
-            content = base64.encodestring(self.content)
+            content = self.content
         )
 
     @classmethod
@@ -485,8 +560,8 @@ class Response(HTTPMsg):
             request,
             state["code"],
             str(state["msg"]),
-            Headers._from_state(state["headers"]),
-            base64.decodestring(state["content"]),
+            ODictCaseless._from_state(state["headers"]),
+            state["content"],
             state["timestamp"],
         )
 
@@ -498,12 +573,10 @@ class Response(HTTPMsg):
             Returns a copy of this object.
         """
         c = copy.copy(self)
+        c.acked = True
         c.headers = self.headers.copy()
         return c
 
-    def _is_response(self):
-        return True
-
     def _assemble(self):
         """
             Assembles the response for transmission to the client. We make some
@@ -516,10 +589,10 @@ class Response(HTTPMsg):
             ['proxy-connection', 'connection', 'keep-alive', 'transfer-encoding']
         )
         content = self.content
-        if content is not None:
-            headers["content-length"] = [str(len(content))]
-        else:
+        if content is None:
             content = ""
+        else:
+            headers["content-length"] = [str(len(content))]
         if self.request.client_conn.close:
             headers["connection"] = ["close"]
         proto = "HTTP/1.1 %s %s"%(self.code, str(self.msg))
@@ -539,7 +612,7 @@ class Response(HTTPMsg):
 
 class ClientDisconnect(controller.Msg):
     """
-        A client disconnection event. 
+        A client disconnection event.
 
         Exposes the following attributes:
 
@@ -556,7 +629,7 @@ class ClientConnect(controller.Msg):
         Requests.
 
         Exposes the following attributes:
-            
+
             address: (address, port) tuple, or None if the connection is replayed.
             requestcount: Number of requests created by this client connection.
             close: Is the client connection closed?
@@ -593,12 +666,14 @@ class ClientConnect(controller.Msg):
         """
             Returns a copy of this object.
         """
-        return copy.copy(self)
+        c = copy.copy(self)
+        c.acked = True
+        return c
 
 
 class Error(controller.Msg):
     """
-        An Error. 
+        An Error.
 
         This is distinct from an HTTP error response (say, a code 500), which
         is represented by a normal Response object. This class is responsible
@@ -624,7 +699,9 @@ class Error(controller.Msg):
         """
             Returns a copy of this object.
         """
-        return copy.copy(self)
+        c = copy.copy(self)
+        c.acked = True
+        return c
 
     def _get_state(self):
         return dict(
@@ -633,9 +710,9 @@ class Error(controller.Msg):
         )
 
     @classmethod
-    def _from_state(klass, state):
+    def _from_state(klass, request, state):
         return klass(
-            None,
+            request,
             state["msg"],
             state["timestamp"],
         )
@@ -704,7 +781,7 @@ class ServerPlaybackState:
                 l.append(i)
 
     def count(self):
-        return sum([len(i) for i in self.fmap.values()])
+        return sum(len(i) for i in self.fmap.values())
 
     def _hash(self, flow):
         """
@@ -759,6 +836,13 @@ class StickyCookieState:
             m["path"] or "/"
         )
 
+    def domain_match(self, a, b):
+        if cookielib.domain_match(a, b):
+            return True
+        elif cookielib.domain_match(a, b.strip(".")):
+            return True
+        return False
+
     def handle_response(self, f):
         for i in f.response.headers["set-cookie"]:
             # FIXME: We now know that Cookie.py screws up some cookies with
@@ -766,22 +850,23 @@ class StickyCookieState:
             c = Cookie.SimpleCookie(str(i))
             m = c.values()[0]
             k = self.ckey(m, f)
-            if cookielib.domain_match(f.request.host, k[0]):
+            if self.domain_match(f.request.host, k[0]):
                 self.jar[self.ckey(m, f)] = m
 
     def handle_request(self, f):
+        l = []
         if f.match(self.flt):
             for i in self.jar.keys():
                 match = [
-                    cookielib.domain_match(i[0], f.request.host),
+                    self.domain_match(f.request.host, i[0]),
                     f.request.port == i[1],
                     f.request.path.startswith(i[2])
                 ]
                 if all(match):
-                    l = f.request.headers["cookie"]
-                    f.request.stickycookie = True
                     l.append(self.jar[i].output(header="").strip())
-                    f.request.headers["cookie"] = l
+        if l:
+            f.request.stickycookie = True
+            f.request.headers["cookie"] = l
 
 
 class StickyAuthState:
@@ -804,7 +889,7 @@ class Flow:
     """
         A Flow is a collection of objects representing a single HTTP
         transaction. The main attributes are:
-            
+
             request: Request object
             response: Response object
             error: Error object
@@ -823,6 +908,17 @@ class Flow:
         self.intercepting = False
         self._backup = None
 
+    def copy(self):
+        rc = self.request.copy()
+        f = Flow(rc)
+        if self.response:
+            f.response = self.response.copy()
+            f.response.request = rc
+        if self.error:
+            f.error = self.error.copy()
+            f.error.request = rc
+        return f
+
     @classmethod
     def _from_state(klass, state):
         f = klass(None)
@@ -861,7 +957,7 @@ class Flow:
             if self.error:
                 self.error._load_state(state["error"])
             else:
-                self.error = Error._from_state(state["error"])
+                self.error = Error._from_state(self.request, state["error"])
         else:
             self.error = None
 
@@ -897,12 +993,8 @@ class Flow:
             if matched, False if not.
         """
         if f:
-            if self.response:
-                return f(self.response)
-            elif self.request:
-                return f(self.request)
-        else:
-            return True
+            return f(self)
+        return True
 
     def kill(self, master):
         """
@@ -965,6 +1057,9 @@ class State(object):
     def flow_count(self):
         return len(self._flow_map)
 
+    def index(self, f):
+        return self._flow_list.index(f)
+
     def active_flow_count(self):
         c = 0
         for i in self._flow_list:
@@ -979,6 +1074,7 @@ class State(object):
         f = Flow(req)
         self._flow_list.append(f)
         self._flow_map[req] = f
+        assert len(self._flow_list) == len(self._flow_map)
         if f.match(self._limit):
             self.view.append(f)
         return f
@@ -1000,11 +1096,8 @@ class State(object):
             Add an error response to the state. Returns the matching flow, or
             None if there isn't one.
         """
-        if err.request:
-            f = self._flow_map.get(err.request)
-            if not f:
-                return None
-        else:
+        f = self._flow_map.get(err.request)
+        if not f:
             return None
         f.error = err
         if f.match(self._limit) and not f in self.view:
@@ -1078,6 +1171,7 @@ class FlowMaster(controller.Master):
         self.client_playback = None
         self.kill_nonreplay = False
         self.script = None
+        self.pause_scripts = False
 
         self.stickycookie_state = False
         self.stickycookie_txt = None
@@ -1203,17 +1297,27 @@ class FlowMaster(controller.Master):
 
         return controller.Master.tick(self, q)
 
+    def duplicate_flow(self, f):
+        return self.load_flow(f.copy())
+
+    def load_flow(self, f):
+        """
+            Loads a flow, and returns a new flow object.
+        """
+        if f.request:
+            fr = self.handle_request(f.request)
+        if f.response:
+            self.handle_response(f.response)
+        if f.error:
+            self.handle_error(f.error)
+        return fr
+
     def load_flows(self, fr):
         """
             Load flows from a FlowReader object.
         """
         for i in fr.stream():
-            if i.request:
-                self.handle_request(i.request)
-            if i.response:
-                self.handle_response(i.response)
-            if i.error:
-                self.handle_error(i.error)
+            self.load_flow(i)
 
     def process_new_request(self, f):
         if self.stickycookie_state:
@@ -1252,12 +1356,16 @@ class FlowMaster(controller.Master):
             f.response = None
             f.error = None
             self.process_new_request(f)
-            rt = proxy.RequestReplayThread(f, self.masterq)
+            rt = proxy.RequestReplayThread(
+                    self.server.config,
+                    f,
+                    self.masterq,
+                )
             rt.start()
         #end nocover
 
     def run_script_hook(self, name, *args, **kwargs):
-        if self.script:
+        if self.script and not self.pause_scripts:
             ret = self.script.run(name, *args, **kwargs)
             if not ret[0] and ret[1]:
                 e = "Script error:\n" + ret[1][1]
@@ -1304,7 +1412,8 @@ class FlowMaster(controller.Master):
             self.client_playback.clear(f)
         if not f:
             r._ack()
-        self.process_new_response(f)
+        if f:
+            self.process_new_response(f)
         return f
 
     def shutdown(self):
@@ -1316,12 +1425,10 @@ class FlowMaster(controller.Master):
 class FlowWriter:
     def __init__(self, fo):
         self.fo = fo
-        self.ns = netstring.FileEncoder(fo)
 
     def add(self, flow):
         d = flow._get_state()
-        s = json.dumps(d)
-        self.ns.write(s)
+        tnetstring.dump(d, self.fo)
 
 
 class FlowReadError(Exception):
@@ -1333,16 +1440,20 @@ class FlowReadError(Exception):
 class FlowReader:
     def __init__(self, fo):
         self.fo = fo
-        self.ns = netstring.decode_file(fo)
 
     def stream(self):
         """
             Yields Flow objects from the dump.
         """
+        off = 0
         try:
-            for i in self.ns:
-                data = json.loads(i)
+            while 1:
+                data = tnetstring.load(self.fo)
+                off = self.fo.tell()
                 yield Flow._from_state(data)
-        except netstring.DecoderError:
+        except ValueError:
+            # Error is due to EOF
+            if self.fo.tell() == off and self.fo.read() == '':
+                return
             raise FlowReadError("Invalid data format.")
 

libmproxy/netstring.py

@@ -1,151 +0,0 @@
-"""
-    Netstring is a module for encoding and decoding netstring streams.
-    See http://cr.yp.to/proto/netstrings.txt for more information on netstrings.
-    Author: Will McGugan (http://www.willmcgugan.com)
-"""
-from cStringIO import StringIO
-
-    
-def header(data):
-    return str(len(data))+":"
-    
-    
-class FileEncoder(object):
-    def __init__(self, file_out):
-        """"
-            file_out -- A writable file object
-        """
-        self.file_out = file_out
-        
-    def write(self, data):
-        """
-            Encodes a netstring and writes it to the file object.
-            
-            data -- A string to be encoded and written
-        """  
-        write = self.file_out.write
-        write(header(data))
-        write(data)
-        write(',')
-        return self
-        
-    
-class DecoderError(Exception):
-    PRECEDING_ZERO_IN_SIZE = 0
-    MAX_SIZE_REACHED = 1
-    ILLEGAL_DIGIT_IN_SIZE = 2
-    ILLEGAL_DIGIT = 3
-    error_text = {    
-        PRECEDING_ZERO_IN_SIZE: "PRECEDING_ZERO_IN_SIZE",
-        MAX_SIZE_REACHED: "MAX_SIZE_REACHED",
-        ILLEGAL_DIGIT_IN_SIZE: "ILLEGAL_DIGIT_IN_SIZE",
-        ILLEGAL_DIGIT: "ILLEGAL_DIGIT"
-    }
-    def __init__(self, code, text):
-        Exception.__init__(self)
-        self.code = code
-        self.text = text
-        
-    def __str__(self):
-        return "%s (#%i), %s" % (DecoderError.error_text[self.code], self.code, self.text)
-        
-
-class Decoder(object):
-    """
-        A netstring decoder.
-        Turns a netstring stream in to a number of discreet strings.    
-    """
-    def __init__(self, max_size=None):
-        """
-            Create a netstring-stream decoder object.
-            
-            max_size -- The maximum size of a netstring encoded string, after which
-            a DecoderError will be throw. A value of None (the default) indicates
-            that there should be no maximum string size.
-        """        
-        self.max_size = max_size
-        self.data_pos = 0
-        self.string_start = 0        
-        self.expecting_terminator = False
-        self.size_string = ""
-        self.data_size = None
-        self.remaining_bytes = 0
-        self.data_out = StringIO()
-        self.yield_data = ""
-        
-    def feed(self, data):
-        """
-            A generator that yields 0 or more strings from the given data.
-            
-            data -- A string containing complete or partial netstring data
-        """
-        self.data_pos = 0 
-        self.string_start = 0        
-        while self.data_pos < len(data):        
-            if self.expecting_terminator:
-                c = data[self.data_pos]                                
-                self.data_pos += 1
-                if c != ',':
-                    raise DecoderError(DecoderError.ILLEGAL_DIGIT, "Illegal digit (%s) at end of data"%repr(c))
-                yield self.yield_data
-                self.yield_data = ""
-                self.expecting_terminator = False
-            elif self.data_size is None:
-                c = data[self.data_pos]                    
-                self.data_pos += 1
-                
-                if not len(self.size_string):
-                    self.string_start = self.data_pos-1                            
-                    
-                if c in "0123456789":
-                    if self.size_string == '0':
-                        raise DecoderError(DecoderError.PRECEDING_ZERO_IN_SIZE, "Preceding zeros in size field illegal")                    
-                    self.size_string += c                                            
-                    if self.max_size is not None and int(self.size_string) > self.max_size:
-                        raise DecoderError(DecoderError.MAX_SIZE_REACHED, "Maximum size of netstring exceeded")
-                                                
-                elif c == ":":        
-                    if not len(self.size_string):
-                        raise DecoderError(DecoderError.ILLEGAL_DIGIT_IN_SIZE, "Illegal digit (%s) in size field"%repr(c))
-                    self.data_size = int(self.size_string)                    
-                    self.remaining_bytes = self.data_size
-                    
-                else:
-                    raise DecoderError(DecoderError.ILLEGAL_DIGIT_IN_SIZE, "Illegal digit (%s) in size field"%repr(c))
-                        
-            elif self.data_size is not None:                                
-                get_bytes = min(self.remaining_bytes, len(data)-self.data_pos)                
-                chunk = data[self.data_pos:self.data_pos+get_bytes] 
-                whole_string = len(chunk) == self.data_size
-                if not whole_string:
-                    self.data_out.write(chunk)
-                self.data_pos += get_bytes                
-                self.remaining_bytes -= get_bytes
-                if self.remaining_bytes == 0:
-                    if whole_string:
-                        self.yield_data = chunk
-                    else:
-                        self.yield_data = self.data_out.getvalue()                        
-                        self.data_out.reset()
-                        self.data_out.truncate()
-                    self.data_size = None
-                    self.size_string = "" 
-                    self.remaining_bytes = 0                    
-                    self.expecting_terminator = True                            
-        
-        
-def decode_file(file_in, buffer_size=1024):
-    """
-        Generates 0 or more strings from a netstring file.
-        
-        file_in -- A readable file-like object containing netstring data 
-        buffer_size -- The number of bytes to attempt to read in each iteration
-        (default = 1024).
-    """
-    decoder = Decoder()
-    while True:
-        data = file_in.read(buffer_size)        
-        if not len(data):
-            return
-        for s in decoder.feed(data):
-            yield s

libmproxy/proxy.py

@@ -1,11 +1,24 @@
+# Copyright (C) 2012  Aldo Cortesi
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 """
     A simple proxy server implementation, which always reads all of a server
     response into memory, performs some transformation, and then writes it back
     to the client.
-
-    Development started from Neil Schemenauer's munchy.py
 """
-import sys, os, string, socket, select, time
+import sys, os, string, socket, time
 import shutil, tempfile, threading
 import optparse, SocketServer, ssl
 import utils, flow
@@ -21,28 +34,65 @@ class ProxyError(Exception):
         return "ProxyError(%s, %s)"%(self.code, self.msg)
 
 
-class SSLConfig:
-    def __init__(self, certfile = None, ciphers = None, cacert = None, cert_wait_time=None):
+class ProxyConfig:
+    def __init__(self, certfile = None, ciphers = None, cacert = None, cert_wait_time=0, body_size_limit = None, reverse_proxy=None):
         self.certfile = certfile
         self.ciphers = ciphers
         self.cacert = cacert
         self.certdir = None
         self.cert_wait_time = cert_wait_time
+        self.body_size_limit = body_size_limit
+        self.reverse_proxy = reverse_proxy
 
 
-def read_chunked(fp):
-    content = ""
+def read_headers(fp):
+    """
+        Read a set of headers from a file pointer. Stop once a blank line
+        is reached. Return a ODict object.
+    """
+    ret = []
+    name = ''
     while 1:
         line = fp.readline()
+        if not line or line == '\r\n' or line == '\n':
+            break
+        if line[0] in ' \t':
+            # continued header
+            ret[-1][1] = ret[-1][1] + '\r\n ' + line.strip()
+        else:
+            i = line.find(':')
+            # We're being liberal in what we accept, here.
+            if i > 0:
+                name = line[:i]
+                value = line[i+1:].strip()
+                ret.append([name, value])
+    return flow.ODictCaseless(ret)
+
+
+def read_chunked(fp, limit):
+    content = ""
+    total = 0
+    while 1:
+        line = fp.readline(128)
         if line == "":
             raise IOError("Connection closed")
         if line == '\r\n' or line == '\n':
             continue
-        length = int(line,16)
+        try:
+            length = int(line,16)
+        except ValueError:
+            # FIXME: Not strictly correct - this could be from the server, in which
+            # case we should send a 502.
+            raise ProxyError(400, "Invalid chunked encoding length: %s"%line)
         if not length:
             break
+        total += length
+        if limit is not None and total > limit:
+            msg = "HTTP Body too large."\
+                  " Limit is %s, chunked content length was at least %s"%(limit, total)
+            raise ProxyError(509, msg)
         content += fp.read(length)
-        line = fp.readline()
+        line = fp.readline(5)
         if line != '\r\n':
             raise IOError("Malformed chunked body")
     while 1:
@@ -54,15 +104,23 @@ def read_chunked(fp):
     return content
 
 
-def read_http_body(rfile, connection, headers, all):
+def read_http_body(rfile, connection, headers, all, limit):
     if 'transfer-encoding' in headers:
         if not ",".join(headers["transfer-encoding"]) == "chunked":
             raise IOError('Invalid transfer-encoding')
-        content = read_chunked(rfile)
+        content = read_chunked(rfile, limit)
     elif "content-length" in headers:
-        content = rfile.read(int(headers["content-length"][0]))
+        try:
+            l = int(headers["content-length"][0])
+        except ValueError:
+            # FIXME: Not strictly correct - this could be from the server, in which
+            # case we should send a 502.
+            raise ProxyError(400, "Invalid content-length header: %s"%headers["content-length"])
+        if limit is not None and l > limit:
+            raise ProxyError(509, "HTTP Body too large. Limit is %s, content-length was %s"%(limit, l))
+        content = rfile.read(l)
     elif all:
-        content = rfile.read()
+        content = rfile.read(limit if limit else None)
         connection.close = True
     else:
         content = ""
@@ -102,7 +160,6 @@ def parse_request_line(request):
     if major != 1:
         raise ProxyError(400, "Unsupported protocol")
     return method, scheme, host, port, path, minor
-    
 
 
 class FileLike:
@@ -142,14 +199,14 @@ class FileLike:
 
 #begin nocover
 class RequestReplayThread(threading.Thread):
-    def __init__(self, flow, masterq):
-        self.flow, self.masterq = flow, masterq
+    def __init__(self, config, flow, masterq):
+        self.config, self.flow, self.masterq = config, flow, masterq
         threading.Thread.__init__(self)
 
     def run(self):
         try:
-            server = ServerConnection(self.flow.request)
-            server.send_request(self.flow.request)
+            server = ServerConnection(self.config, self.flow.request)
+            server.send()
             response = server.read_response()
             response._send(self.masterq)
         except ProxyError, v:
@@ -158,10 +215,14 @@ class RequestReplayThread(threading.Thread):
 
 
 class ServerConnection:
-    def __init__(self, request):
-        self.host = request.host
-        self.port = request.port
-        self.scheme = request.scheme
+    def __init__(self, config, request):
+        self.config, self.request = config, request
+        if config.reverse_proxy:
+            self.scheme, self.host, self.port = config.reverse_proxy
+        else:
+            self.host = request.host
+            self.port = request.port
+            self.scheme = request.scheme
         self.close = False
         self.server, self.rfile, self.wfile = None, None, None
         self.connect()
@@ -174,18 +235,17 @@ class ServerConnection:
                 server = ssl.wrap_socket(server)
             server.connect((addr, self.port))
         except socket.error, err:
-            raise ProxyError(504, 'Error connecting to "%s": %s' % (self.host, err))
+            raise ProxyError(502, 'Error connecting to "%s": %s' % (self.host, err))
         self.server = server
         self.rfile, self.wfile = server.makefile('rb'), server.makefile('wb')
 
-    def send_request(self, request):
-        self.request = request
-        request.close = self.close
+    def send(self):
+        self.request.close = self.close
         try:
-            self.wfile.write(request._assemble())
+            self.wfile.write(self.request._assemble())
             self.wfile.flush()
         except socket.error, err:
-            raise ProxyError(504, 'Error sending data to "%s": %s' % (request.host, err))
+            raise ProxyError(502, 'Error sending data to "%s": %s' % (self.request.host, err))
 
     def read_response(self):
         line = self.rfile.readline()
@@ -194,18 +254,22 @@ class ServerConnection:
         if not line:
             raise ProxyError(502, "Blank server response.")
         parts = line.strip().split(" ", 2)
+        if len(parts) == 2: # handle missing message gracefully
+            parts.append("")
         if not len(parts) == 3:
             raise ProxyError(502, "Invalid server response: %s."%line)
         proto, code, msg = parts
-        code = int(code)
-        headers = flow.Headers()
-        headers.read(self.rfile)
+        try:
+            code = int(code)
+        except ValueError:
+            raise ProxyError(502, "Invalid server response: %s."%line)
+        headers = read_headers(self.rfile)
         if code >= 100 and code <= 199:
             return self.read_response()
         if self.request.method == "HEAD" or code == 204 or code == 304:
             content = ""
         else:
-            content = read_http_body(self.rfile, self, headers, True)
+            content = read_http_body(self.rfile, self, headers, True, self.config.body_size_limit)
         return flow.Response(self.request, code, msg, headers, content)
 
     def terminate(self):
@@ -248,13 +312,13 @@ class ProxyHandler(SocketServer.StreamRequestHandler):
                 cc.close = True
                 return
 
-            if request._is_response():
+            if isinstance(request, flow.Response):
                 response = request
                 request = False
                 response = response._send(self.mqueue)
             else:
-                server = ServerConnection(request)
-                server.send_request(request)
+                server = ServerConnection(self.config, request)
+                server.send()
                 try:
                     response = server.read_response()
                 except IOError, v:
@@ -286,7 +350,7 @@ class ProxyHandler(SocketServer.StreamRequestHandler):
             ret = utils.dummy_cert(self.config.certdir, self.config.cacert, host)
             time.sleep(self.config.cert_wait_time)
             if not ret:
-                raise ProxyError(400, "mitmproxy: Unable to generate dummy cert.")
+                raise ProxyError(502, "mitmproxy: Unable to generate dummy cert.")
             return ret
 
     def read_request(self, client_conn):
@@ -324,8 +388,7 @@ class ProxyHandler(SocketServer.StreamRequestHandler):
             method, scheme, host, port, path, httpminor = parse_request_line(self.rfile.readline())
         if scheme is None:
             scheme = "https"
-        headers = flow.Headers()
-        headers.read(self.rfile)
+        headers = read_headers(self.rfile)
         if host is None and "host" in headers:
             netloc = headers["host"][0]
             if ':' in netloc:
@@ -339,9 +402,12 @@ class ProxyHandler(SocketServer.StreamRequestHandler):
                     port = 80
             port = int(port)
         if host is None:
-            # FIXME: We only specify the first part of the invalid request in this error.
-            # We should gather up everything read from the socket, and specify it all.
-            raise ProxyError(400, 'Invalid request: %s'%line)
+            if self.config.reverse_proxy:
+                scheme, host, port = self.config.reverse_proxy
+            else:
+                # FIXME: We only specify the first part of the invalid request in this error.
+                # We should gather up everything read from the socket, and specify it all.
+                raise ProxyError(400, 'Invalid request: %s'%line)
         if "expect" in headers:
             expect = ",".join(headers['expect'])
             if expect == "100-continue" and httpminor >= 1:
@@ -360,7 +426,7 @@ class ProxyHandler(SocketServer.StreamRequestHandler):
                     client_conn.close = True
                 if value == "keep-alive":
                     client_conn.close = False
-        content = read_http_body(self.rfile, client_conn, headers, False)
+        content = read_http_body(self.rfile, client_conn, headers, False, self.config.body_size_limit)
         return flow.Request(client_conn, host, port, scheme, method, path, headers, content)
 
     def send_response(self, response):
@@ -422,8 +488,11 @@ class ProxyServer(ServerBase):
         self.RequestHandlerClass(self.config, request, client_address, self, self.masterq)
 
     def shutdown(self):
-        shutil.rmtree(self.certdir)
         ServerBase.shutdown(self)
+        try:
+            shutil.rmtree(self.certdir)
+        except OSError:
+            pass
 
 
 # Command-line utils
@@ -442,7 +511,7 @@ def certificate_option_group(parser):
     parser.add_option_group(group)
 
 
-def process_certificate_option_group(parser, options):
+def process_proxy_options(parser, options):
     if options.cert:
         options.cert = os.path.expanduser(options.cert)
         if not os.path.exists(options.cert):
@@ -454,9 +523,20 @@ def process_certificate_option_group(parser, options):
         utils.dummy_ca(cacert)
     if getattr(options, "cache", None) is not None:
         options.cache = os.path.expanduser(options.cache)
-    return SSLConfig(
+    body_size_limit = utils.parse_size(options.body_size_limit)
+
+    if options.reverse_proxy:
+        rp = utils.parse_proxy_spec(options.reverse_proxy)
+        if not rp:
+            parser.error("Invalid reverse proxy specification: %s"%options.reverse_proxy)
+    else:
+        rp = None
+
+    return ProxyConfig(
         certfile = options.cert,
         cacert = cacert,
         ciphers = options.ciphers,
-        cert_wait_time = options.cert_wait_time
+        cert_wait_time = options.cert_wait_time,
+        body_size_limit = body_size_limit,
+        reverse_proxy = rp
     )

libmproxy/script.py

@@ -1,3 +1,18 @@
+# Copyright (C) 2012  Aldo Cortesi
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 import os, traceback
 
 class ScriptError(Exception):

libmproxy/tnetstring.py

@@ -0,0 +1,398 @@
+# imported from the tnetstring project: https://github.com/rfk/tnetstring
+#
+# Copyright (c) 2011 Ryan Kelly
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+"""
+tnetstring:  data serialization using typed netstrings
+======================================================
+
+
+This is a data serialization library. It's a lot like JSON but it uses a
+new syntax called "typed netstrings" that Zed has proposed for use in the
+Mongrel2 webserver.  It's designed to be simpler and easier to implement
+than JSON, with a happy consequence of also being faster in many cases.
+
+An ordinary netstring is a blob of data prefixed with its length and postfixed
+with a sanity-checking comma.  The string "hello world" encodes like this::
+
+    11:hello world,
+
+Typed netstrings add other datatypes by replacing the comma with a type tag.
+Here's the integer 12345 encoded as a tnetstring::
+
+    5:12345#
+
+And here's the list [12345,True,0] which mixes integers and bools::
+
+    19:5:12345#4:true!1:0#]
+
+Simple enough?  This module gives you the following functions:
+
+    :dump:    dump an object as a tnetstring to a file
+    :dumps:   dump an object as a tnetstring to a string
+    :load:    load a tnetstring-encoded object from a file
+    :loads:   load a tnetstring-encoded object from a string
+    :pop:     pop a tnetstring-encoded object from the front of a string
+
+Note that since parsing a tnetstring requires reading all the data into memory
+at once, there's no efficiency gain from using the file-based versions of these
+functions.  They're only here so you can use load() to read precisely one
+item from a file or socket without consuming any extra data.
+
+By default tnetstrings work only with byte strings, not unicode.  If you want
+unicode strings then pass an optional encoding to the various functions, 
+like so::
+
+    >>> print repr(tnetstring.loads("2:\\xce\\xb1,"))
+    '\\xce\\xb1'
+    >>> 
+    >>> print repr(tnetstring.loads("2:\\xce\\xb1,","utf8"))
+    u'\u03b1'
+
+"""
+
+__ver_major__ = 0
+__ver_minor__ = 2
+__ver_patch__ = 0
+__ver_sub__ = ""
+__version__ = "%d.%d.%d%s" % (__ver_major__,__ver_minor__,__ver_patch__,__ver_sub__)
+
+
+from collections import deque
+
+
+def dumps(value,encoding=None):
+    """dumps(object,encoding=None) -> string
+
+    This function dumps a python object as a tnetstring.
+    """
+    #  This uses a deque to collect output fragments in reverse order,
+    #  then joins them together at the end.  It's measurably faster
+    #  than creating all the intermediate strings.
+    #  If you're reading this to get a handle on the tnetstring format,
+    #  consider the _gdumps() function instead; it's a standard top-down
+    #  generator that's simpler to understand but much less efficient.
+    q = deque()
+    _rdumpq(q,0,value,encoding)
+    return "".join(q)
+
+
+def dump(value,file,encoding=None):
+    """dump(object,file,encoding=None)
+
+    This function dumps a python object as a tnetstring and writes it to
+    the given file.
+    """
+    file.write(dumps(value,encoding))
+
+
+def _rdumpq(q,size,value,encoding=None):
+    """Dump value as a tnetstring, to a deque instance, last chunks first.
+
+    This function generates the tnetstring representation of the given value,
+    pushing chunks of the output onto the given deque instance.  It pushes
+    the last chunk first, then recursively generates more chunks.
+
+    When passed in the current size of the string in the queue, it will return
+    the new size of the string in the queue.
+
+    Operating last-chunk-first makes it easy to calculate the size written
+    for recursive structures without having to build their representation as
+    a string.  This is measurably faster than generating the intermediate
+    strings, especially on deeply nested structures.
+    """
+    write = q.appendleft
+    if value is None:
+        write("0:~")
+        return size + 3
+    if value is True:
+        write("4:true!")
+        return size + 7
+    if value is False:
+        write("5:false!")
+        return size + 8
+    if isinstance(value,(int,long)):
+        data = str(value) 
+        ldata = len(data)
+        span = str(ldata)
+        write("#")
+        write(data)
+        write(":")
+        write(span)
+        return size + 2 + len(span) + ldata
+    if isinstance(value,(float,)):
+        #  Use repr() for float rather than str().
+        #  It round-trips more accurately.
+        #  Probably unnecessary in later python versions that
+        #  use David Gay's ftoa routines.
+        data = repr(value) 
+        ldata = len(data)
+        span = str(ldata)
+        write("^")
+        write(data)
+        write(":")
+        write(span)
+        return size + 2 + len(span) + ldata
+    if isinstance(value,str):
+        lvalue = len(value)
+        span = str(lvalue)
+        write(",")
+        write(value)
+        write(":")
+        write(span)
+        return size + 2 + len(span) + lvalue
+    if isinstance(value,(list,tuple,)):
+        write("]")
+        init_size = size = size + 1
+        for item in reversed(value):
+            size = _rdumpq(q,size,item,encoding)
+        span = str(size - init_size)
+        write(":")
+        write(span)
+        return size + 1 + len(span)
+    if isinstance(value,dict):
+        write("}")
+        init_size = size = size + 1
+        for (k,v) in value.iteritems():
+            size = _rdumpq(q,size,v,encoding)
+            size = _rdumpq(q,size,k,encoding)
+        span = str(size - init_size)
+        write(":")
+        write(span)
+        return size + 1 + len(span)
+    if isinstance(value,unicode):
+        if encoding is None:
+            raise ValueError("must specify encoding to dump unicode strings")
+        value = value.encode(encoding)
+        lvalue = len(value)
+        span = str(lvalue)
+        write(",")
+        write(value)
+        write(":")
+        write(span)
+        return size + 2 + len(span) + lvalue
+    raise ValueError("unserializable object")
+
+
+def _gdumps(value,encoding):
+    """Generate fragments of value dumped as a tnetstring.
+
+    This is the naive dumping algorithm, implemented as a generator so that
+    it's easy to pass to "".join() without building a new list.
+
+    This is mainly here for comparison purposes; the _rdumpq version is
+    measurably faster as it doesn't have to build intermediate strins.
+    """
+    if value is None:
+        yield "0:~"
+    elif value is True:
+        yield "4:true!"
+    elif value is False:
+        yield "5:false!"
+    elif isinstance(value,(int,long)):
+        data = str(value) 
+        yield str(len(data))
+        yield ":"
+        yield data
+        yield "#"
+    elif isinstance(value,(float,)):
+        data = repr(value) 
+        yield str(len(data))
+        yield ":"
+        yield data
+        yield "^"
+    elif isinstance(value,(str,)):
+        yield str(len(value))
+        yield ":"
+        yield value
+        yield ","
+    elif isinstance(value,(list,tuple,)):
+        sub = []
+        for item in value:
+            sub.extend(_gdumps(item))
+        sub = "".join(sub)
+        yield str(len(sub))
+        yield ":"
+        yield sub
+        yield "]"
+    elif isinstance(value,(dict,)):
+        sub = []
+        for (k,v) in value.iteritems():
+            sub.extend(_gdumps(k))
+            sub.extend(_gdumps(v))
+        sub = "".join(sub)
+        yield str(len(sub))
+        yield ":"
+        yield sub
+        yield "}"
+    elif isinstance(value,(unicode,)):
+        if encoding is None:
+            raise ValueError("must specify encoding to dump unicode strings")
+        value = value.encode(encoding)
+        yield str(len(value))
+        yield ":"
+        yield value
+        yield ","
+    else:
+        raise ValueError("unserializable object")
+
+
+def loads(string,encoding=None):
+    """loads(string,encoding=None) -> object
+
+    This function parses a tnetstring into a python object.
+    """
+    #  No point duplicating effort here.  In the C-extension version,
+    #  loads() is measurably faster then pop() since it can avoid
+    #  the overhead of building a second string.
+    return pop(string,encoding)[0]
+
+
+def load(file,encoding=None):
+    """load(file,encoding=None) -> object
+
+    This function reads a tnetstring from a file and parses it into a
+    python object.  The file must support the read() method, and this
+    function promises not to read more data than necessary.
+    """
+    #  Read the length prefix one char at a time.
+    #  Note that the netstring spec explicitly forbids padding zeros.
+    c = file.read(1)
+    if not c.isdigit():
+        raise ValueError("not a tnetstring: missing or invalid length prefix")
+    datalen = ord(c) - ord("0")
+    c = file.read(1)
+    if datalen != 0:
+        while c.isdigit():
+            datalen = (10 * datalen) + (ord(c) - ord("0"))
+            if datalen > 999999999:
+                errmsg = "not a tnetstring: absurdly large length prefix"
+                raise ValueError(errmsg)
+            c = file.read(1)
+    if c != ":":
+        raise ValueError("not a tnetstring: missing or invalid length prefix")
+    #  Now we can read and parse the payload.
+    #  This repeats the dispatch logic of pop() so we can avoid
+    #  re-constructing the outermost tnetstring.
+    data = file.read(datalen)
+    if len(data) != datalen:
+        raise ValueError("not a tnetstring: length prefix too big")
+    type = file.read(1)
+    if type == ",":
+        if encoding is not None:
+            return data.decode(encoding)
+        return data
+    if type == "#":
+        try:
+            return int(data)
+        except ValueError:
+            raise ValueError("not a tnetstring: invalid integer literal")
+    if type == "^":
+        try:
+            return float(data)
+        except ValueError:
+            raise ValueError("not a tnetstring: invalid float literal")
+    if type == "!":
+        if data == "true":
+            return True
+        elif data == "false":
+            return False
+        else:
+            raise ValueError("not a tnetstring: invalid boolean literal")
+    if type == "~":
+        if data:
+            raise ValueError("not a tnetstring: invalid null literal")
+        return None
+    if type == "]":
+        l = []
+        while data:
+            (item,data) = pop(data,encoding)
+            l.append(item)
+        return l
+    if type == "}":
+        d = {}
+        while data:
+            (key,data) = pop(data,encoding)
+            (val,data) = pop(data,encoding)
+            d[key] = val
+        return d
+    raise ValueError("unknown type tag")
+    
+
+
+def pop(string,encoding=None):
+    """pop(string,encoding=None) -> (object, remain)
+
+    This function parses a tnetstring into a python object.
+    It returns a tuple giving the parsed object and a string
+    containing any unparsed data from the end of the string.
+    """
+    #  Parse out data length, type and remaining string.
+    try:
+        (dlen,rest) = string.split(":",1)
+        dlen = int(dlen)
+    except ValueError:
+        raise ValueError("not a tnetstring: missing or invalid length prefix")
+    try:
+        (data,type,remain) = (rest[:dlen],rest[dlen],rest[dlen+1:])
+    except IndexError:
+        #  This fires if len(rest) < dlen, meaning we don't need
+        #  to further validate that data is the right length.
+        raise ValueError("not a tnetstring: invalid length prefix")
+    #  Parse the data based on the type tag.
+    if type == ",":
+        if encoding is not None:
+            return (data.decode(encoding),remain)
+        return (data,remain)
+    if type == "#":
+        try:
+            return (int(data),remain)
+        except ValueError:
+            raise ValueError("not a tnetstring: invalid integer literal")
+    if type == "^":
+        try:
+            return (float(data),remain)
+        except ValueError:
+            raise ValueError("not a tnetstring: invalid float literal")
+    if type == "!":
+        if data == "true":
+            return (True,remain)
+        elif data == "false":
+            return (False,remain)
+        else:
+            raise ValueError("not a tnetstring: invalid boolean literal")
+    if type == "~":
+        if data:
+            raise ValueError("not a tnetstring: invalid null literal")
+        return (None,remain)
+    if type == "]":
+        l = []
+        while data:
+            (item,data) = pop(data,encoding)
+            l.append(item)
+        return (l,remain)
+    if type == "}":
+        d = {}
+        while data:
+            (key,data) = pop(data,encoding)
+            (val,data) = pop(data,encoding)
+            d[key] = val
+        return (d,remain)
+    raise ValueError("unknown type tag")

libmproxy/utils.py

@@ -12,8 +12,8 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
-import re, os, subprocess, datetime, urlparse, string
-import time, functools, cgi, textwrap
+import re, os, subprocess, datetime, urlparse, string, urllib
+import time, functools, cgi, textwrap, hashlib
 import json
 
 CERT_SLEEP_TIME = 1
@@ -123,9 +123,19 @@ def pretty_json(s):
 
 
 def urldecode(s):
+    """
+        Takes a urlencoded string and returns a list of (key, value) tuples.
+    """
     return cgi.parse_qsl(s)
 
 
+def urlencode(s):
+    """
+        Takes a list of (key, value) tuples and returns a urlencoded string.
+    """
+    return urllib.urlencode(s, False)
+
+
 def hexdump(s):
     """
         Returns a set of typles:
@@ -135,10 +145,10 @@ def hexdump(s):
     for i in range(0, len(s), 16):
         o = "%.10x"%i
         part = s[i:i+16]
-        x = " ".join(["%.2x"%ord(i) for i in part])
+        x = " ".join("%.2x"%ord(i) for i in part)
         if len(part) < 16:
             x += " "
-            x += " ".join(["  " for i in range(16-len(part))])
+            x += " ".join("  " for i in range(16 - len(part)))
         parts.append(
             (o, x, cleanBin(part))
         )
@@ -151,7 +161,6 @@ def del_all(dict, keys):
             del dict[key]
 
 
-
 def pretty_size(size):
     suffixes = [
         ("B",   2**10),
@@ -275,12 +284,13 @@ def dummy_cert(certdir, ca, commonname):
 
         Returns cert path if operation succeeded, None if not.
     """
-    certpath = os.path.join(certdir, commonname + ".pem")
+    namehash = hashlib.sha256(commonname).hexdigest()
+    certpath = os.path.join(certdir, namehash + ".pem")
     if os.path.exists(certpath):
         return certpath
 
-    confpath = os.path.join(certdir, commonname + ".cnf")
-    reqpath = os.path.join(certdir, commonname + ".req")
+    confpath = os.path.join(certdir, namehash + ".cnf")
+    reqpath = os.path.join(certdir, namehash + ".req")
 
     template = open(pkg_data.path("resources/cert.cnf")).read()
     f = open(confpath, "w")
@@ -393,8 +403,11 @@ def parse_url(url):
     if not scheme:
         return None
     if ':' in netloc:
-        host, port = string.split(netloc, ':')
-        port = int(port)
+        host, port = string.rsplit(netloc, ':', maxsplit=1)
+        try:
+            port = int(port)
+        except ValueError:
+            return None
     else:
         host = netloc
         if scheme == "https":
@@ -407,3 +420,66 @@ def parse_url(url):
     return scheme, host, port, path
 
 
+def parse_proxy_spec(url):
+    p = parse_url(url)
+    if not p or not p[1]:
+        return None
+    return p[:3]
+
+
+def hostport(scheme, host, port):
+    """
+        Returns the host component, with a port specifcation if needed.
+    """
+    if (port, scheme) in [(80, "http"), (443, "https")]:
+        return host
+    else:
+        return "%s:%s"%(host, port)
+
+
+def unparse_url(scheme, host, port, path=""):
+    """
+        Returns a URL string, constructed from the specified compnents.
+    """
+    return "%s://%s%s"%(scheme, hostport(scheme, host, port), path)
+
+
+def clean_hanging_newline(t):
+    """
+        Many editors will silently add a newline to the final line of a
+        document (I'm looking at you, Vim). This function fixes this common
+        problem at the risk of removing a hanging newline in the rare cases
+        where the user actually intends it.
+    """
+    if t[-1] == "\n":
+        return t[:-1]
+    return t
+
+
+def parse_size(s):
+    """
+        Parses a size specification. Valid specifications are:
+
+            123: bytes
+            123k: kilobytes
+            123m: megabytes
+            123g: gigabytes
+    """
+    if not s:
+        return None
+    mult = None
+    if s[-1].lower() == "k":
+        mult = 1024**1
+    elif s[-1].lower() == "m":
+        mult = 1024**2
+    elif s[-1].lower() == "g":
+        mult = 1024**3
+
+    if mult:
+        s = s[:-1]
+    else:
+        mult = 1
+    try:
+        return int(s) * mult
+    except ValueError:
+        raise ValueError("Invalid size specification: %s"%s)

libmproxy/version.py

@@ -1,2 +1,2 @@
-IVERSION = (0, 6)
-VERSION = ".".join([str(i) for i in IVERSION])
+IVERSION = (0, 7)
+VERSION = ".".join(str(i) for i in IVERSION)

mitmdump

@@ -1,24 +1,24 @@
 #!/usr/bin/env python
 
 # Copyright (C) 2010  Aldo Cortesi
-# 
+#
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
-# 
+#
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
-# 
+#
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import sys, os.path
-from libmproxy import proxy, dump, utils, cmdline
+import sys
+from libmproxy import proxy, dump, cmdline
 from libmproxy.version import VERSION
-from optparse import OptionParser, OptionGroup
+from optparse import OptionParser
 
 
 if __name__ == '__main__':
@@ -38,12 +38,12 @@ if __name__ == '__main__':
     if options.quiet:
         options.verbose = 0
 
-    config = proxy.process_certificate_option_group(parser, options)
+    proxyconfig = proxy.process_proxy_options(parser, options)
     if options.no_server:
         server = None
     else:
         try:
-            server = proxy.ProxyServer(config, options.port, options.addr)
+            server = proxy.ProxyServer(proxyconfig, options.port, options.addr)
         except proxy.ProxyServerError, v:
             print >> sys.stderr, "mitmdump:", v.args[0]
             sys.exit(1)

mitmproxy

@@ -1,22 +1,22 @@
 #!/usr/bin/env python
 
 # Copyright (C) 2010  Aldo Cortesi
-# 
+#
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
-# 
+#
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
-# 
+#
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import sys, os.path
-from libmproxy import proxy, controller, console, utils, flow, cmdline
+import sys
+from libmproxy import proxy, console, cmdline
 from libmproxy.version import VERSION
 from optparse import OptionParser, OptionGroup
 
@@ -35,11 +35,6 @@ if __name__ == '__main__':
                         "Filters",
                         "See help in mitmproxy for filter expression syntax."
                     )
-    group.add_option(
-        "-l", "--limit", action="store",
-        type = "str", dest="limit", default=None,
-        help = "Limit filter expression."
-    )
     group.add_option(
         "-i", "--intercept", action="store",
         type = "str", dest="intercept", default=None,
@@ -48,7 +43,7 @@ if __name__ == '__main__':
     parser.add_option_group(group)
     options, args = parser.parse_args()
 
-    config = proxy.process_certificate_option_group(parser, options)
+    config = proxy.process_proxy_options(parser, options)
 
     if options.no_server:
         server = None
@@ -61,10 +56,8 @@ if __name__ == '__main__':
 
     opts = console.Options(**cmdline.get_common_options(options))
     opts.intercept = options.intercept
-    opts.limit = options.limit
     opts.debug = options.debug
     m = console.ConsoleMaster(server, opts)
-
     m.run()
 
 

setup.py

@@ -27,7 +27,7 @@ def findPackages(path, dataExclude=[]):
         that only data _directories_ and their contents are returned -
         non-Python files at module scope are not, and should be manually
         included.
-        
+
         dataExclude is a list of fnmatch-compatible expressions for files and
         directories that should not be included in pakcage_data.
 
@@ -65,14 +65,12 @@ def findPackages(path, dataExclude=[]):
     return packages, package_data
 
 
-
-
-long_description = file("README.mkd").read()
+long_description = file("README.txt").read()
 packages, package_data = findPackages("libmproxy")
 setup(
         name = "mitmproxy",
         version = version.VERSION,
-        description = "An interactive, SSL-capable man-in-the-middle HTTP proxy for penetration testers and software developers.",
+        description = "An interactive, SSL-capable, man-in-the-middle HTTP proxy for penetration testers and software developers.",
         long_description = long_description,
         author = "Aldo Cortesi",
         author_email = "aldo@corte.si",
@@ -85,10 +83,14 @@ setup(
             "Development Status :: 5 - Production/Stable",
             "Environment :: Console",
             "Environment :: Console :: Curses",
+            "Operating System :: MacOS :: MacOS X",
+            "Operating System :: POSIX",
             "Programming Language :: Python",
             "Topic :: Security",
+            "Topic :: Internet",
             "Topic :: Internet :: WWW/HTTP",
             "Topic :: Internet :: Proxy Servers",
             "Topic :: Software Development :: Testing"
-        ]
+        ],
+        install_requires=['urwid'],
 )

test/.pry

@@ -2,4 +2,5 @@ base = ..
 coverage = ../libmproxy
 exclude = .
           ../libmproxy/contrib
+          ../libmproxy/tnetstring.py
 

test/scripts/duplicate_flow.py

@@ -0,0 +1,5 @@
+
+def request(ctx, f):
+    f = ctx.duplicate_flow(f)
+    ctx.replay_request(f)
+

test/scripts/reqerr.py

@@ -0,0 +1,2 @@
+def request(ctx, r):
+    raise ValueError

test/serv.py

@@ -1,10 +1,6 @@
-import socket
-from SocketServer import BaseServer
 from BaseHTTPServer import HTTPServer
 import handler
 
 def make(port):
-    server_address = ('', port)
+    server_address = ('127.0.0.1', port)
     return HTTPServer(server_address, handler.TestRequestHandler)
-
-

test/sslserv.py

@@ -18,5 +18,5 @@ class SecureHTTPServer(HTTPServer):
 
 
 def make(port):
-    server_address = ('', port)
+    server_address = ('127.0.0.1', port)
     return SecureHTTPServer(server_address, handler.TestRequestHandler)

test/test_console.py

@@ -1,4 +1,5 @@
-from libmproxy import console, filt, flow
+from libmproxy import console
+from libmproxy.console import common
 import tutils
 import libpry
 
@@ -71,14 +72,14 @@ class uState(libpry.AutoTree):
         self._add_response(c)
         self._add_request(c)
         self._add_response(c)
-        assert not c.set_limit("~q")
+        assert not c.set_limit("~s")
         assert len(c.view) == 3
         assert c.focus == 0
 
 
 class uformat_keyvals(libpry.AutoTree):
     def test_simple(self):
-        assert console.format_keyvals(
+        assert common.format_keyvals(
             [
                 ("aa", "bb"),
                 None,
@@ -89,36 +90,6 @@ class uformat_keyvals(libpry.AutoTree):
         )
 
 
-class uformat_flow(libpry.AutoTree):
-    def test_simple(self):
-        f = tutils.tflow()
-        foc = ('focus', '>>')
-        assert foc not in console.format_flow(f, False)
-        assert foc in console.format_flow(f, True)
-
-        assert foc not in console.format_flow(f, False, True)
-        assert foc in console.format_flow(f, True, True)
-
-        f.response = tutils.tresp()
-        f.request = f.response.request
-        f.backup()
-
-        f.request._set_replay()
-        f.response._set_replay()
-        assert ('method', '[replay]') in console.format_flow(f, True)
-        assert ('method', '[replay]') in console.format_flow(f, True, True)
-
-        f.response.code = 404
-        assert ('error', '404') in console.format_flow(f, True, True)
-        f.response.headers["content-type"] = ["text/html"]
-        assert ('text', ' text/html') in console.format_flow(f, True, True)
-
-        f.response =None
-        f.error = flow.Error(f.request, "error")
-        assert ('error', 'error') in console.format_flow(f, True, True)
-
-
-
 class uPathCompleter(libpry.AutoTree):
     def test_lookup_construction(self):
         c = console._PathCompleter()
@@ -168,8 +139,7 @@ class uOptions(libpry.AutoTree):
 
 tests = [
     uformat_keyvals(),
-    uformat_flow(),
-    uState(), 
+    uState(),
     uPathCompleter(),
     uOptions()
 ]

test/test_dump.py

@@ -1,7 +1,7 @@
 import os
 from cStringIO import StringIO
 import libpry
-from libmproxy import dump, flow, proxy
+from libmproxy import dump, flow
 import tutils
 
 class uStrFuncs(libpry.AutoTree):
@@ -58,7 +58,7 @@ class uDumpMaster(libpry.AutoTree):
 
         o = dump.Options(server_replay=p, kill=True)
         m = dump.DumpMaster(None, o, None, outfile=cs)
-        
+
         self._cycle(m, "content")
         self._cycle(m, "content")
 
@@ -80,6 +80,11 @@ class uDumpMaster(libpry.AutoTree):
             0, None, "", verbosity=1, rfile="/nonexistent"
         )
 
+        libpry.raises(
+            dump.DumpError, self._dummy_cycle,
+            0, None, "", verbosity=1, rfile="test_dump.py"
+        )
+
     def test_options(self):
         o = dump.Options(verbosity = 2)
         assert o.verbosity == 2
@@ -106,7 +111,7 @@ class uDumpMaster(libpry.AutoTree):
             self._dummy_cycle,
             1,
             None,
-            "", 
+            "",
             wfile = "nonexistentdir/foo"
         )
 

test/test_encoding.py

@@ -1,12 +1,11 @@
 from libmproxy import encoding
 import libpry
 
-import cStringIO
-
 class uidentity(libpry.AutoTree):
     def test_simple(self):
         assert "string" == encoding.decode("identity", "string")
         assert "string" == encoding.encode("identity", "string")
+        assert not encoding.encode("nonexistent", "string")
 
     def test_fallthrough(self):
         assert None == encoding.decode("nonexistent encoding", "string")

test/test_filt.py

@@ -9,13 +9,14 @@ class uParsing(libpry.AutoTree):
         x.dump(fp=c)
         assert c.getvalue()
 
-    def test_err(self):
+    def test_parse_err(self):
         assert filt.parse("~h [") is None
 
     def test_simple(self):
         assert not filt.parse("~b")
         assert filt.parse("~q")
         assert filt.parse("~c 10")
+        assert filt.parse("~m foobar")
         assert filt.parse("~u foobar")
         assert filt.parse("~q ~c 10")
         p = filt.parse("~q ~c 10")
@@ -73,9 +74,9 @@ class uParsing(libpry.AutoTree):
 class uMatching(libpry.AutoTree):
     def req(self):
         conn = flow.ClientConnect(("one", 2222))
-        headers = flow.Headers()
+        headers = flow.ODictCaseless()
         headers["header"] = ["qvalue"]
-        return flow.Request(
+        req = flow.Request(
                     conn,
                     "host",
                     80,
@@ -85,18 +86,26 @@ class uMatching(libpry.AutoTree):
                     headers,
                     "content_request"
         )
+        return flow.Flow(req)
 
     def resp(self):
-        q = self.req()
-        headers = flow.Headers()
+        f = self.req()
+
+        headers = flow.ODictCaseless()
         headers["header_response"] = ["svalue"]
-        return flow.Response(
-                    q,
+        f.response = flow.Response(
+                    f.request,
                     200,
                     "message",
                     headers,
                     "content_response"
                 )
+        return f
+
+    def err(self):
+        f = self.req()
+        f.error = flow.Error(f.request, "msg")
+        return f
 
     def q(self, q, o):
         return filt.parse(q)(o)
@@ -107,15 +116,15 @@ class uMatching(libpry.AutoTree):
         assert not self.q("~t content", q)
         assert not self.q("~t content", s)
 
-        q.headers["content-type"] = ["text/json"]
+        q.request.headers["content-type"] = ["text/json"]
         assert self.q("~t json", q)
         assert self.q("~tq json", q)
         assert not self.q("~ts json", q)
 
-        s.headers["content-type"] = ["text/json"]
+        s.response.headers["content-type"] = ["text/json"]
         assert self.q("~t json", s)
 
-        del s.headers["content-type"]
+        del s.response.headers["content-type"]
         s.request.headers["content-type"] = ["text/json"]
         assert self.q("~t json", s)
         assert self.q("~tq json", s)
@@ -131,6 +140,10 @@ class uMatching(libpry.AutoTree):
         assert not self.q("~s", q)
         assert self.q("~s", s)
 
+    def test_ferr(self):
+        e = self.err()
+        assert self.q("~e", e)
+
     def test_head(self):
         q = self.req()
         s = self.resp()
@@ -171,6 +184,15 @@ class uMatching(libpry.AutoTree):
         assert not self.q("~bs response", q)
         assert self.q("~bs response", s)
 
+    def test_method(self):
+        q = self.req()
+        s = self.resp()
+        assert self.q("~m get", q)
+        assert not self.q("~m post", q)
+
+        q.request.method = "oink"
+        assert not self.q("~m get", q)
+
     def test_url(self):
         q = self.req()
         s = self.resp()
@@ -192,6 +214,7 @@ class uMatching(libpry.AutoTree):
     def test_and(self):
         s = self.resp()
         assert self.q("~c 200 & ~h head", s)
+        assert self.q("~c 200 & ~h head", s)
         assert not self.q("~c 200 & ~h nohead", s)
         assert self.q("(~c 200 & ~h head) & ~b content", s)
         assert not self.q("(~c 200 & ~h head) & ~b nonexistent", s)
@@ -203,7 +226,6 @@ class uMatching(libpry.AutoTree):
         assert self.q("~c 201 | ~h head", s)
         assert not self.q("~c 201 | ~h nohead", s)
         assert self.q("(~c 201 | ~h nohead) | ~s", s)
-        assert not self.q("(~c 201 | ~h nohead) | ~q", s)
 
     def test_not(self):
         s = self.resp()

test/test_flow.py

@@ -1,7 +1,7 @@
-import Queue, time, textwrap
+import Queue, time
 from cStringIO import StringIO
 import email.utils
-from libmproxy import console, proxy, filt, flow, controller, utils
+from libmproxy import filt, flow, controller, utils
 import tutils
 import libpry
 
@@ -10,11 +10,16 @@ class uStickyCookieState(libpry.AutoTree):
     def _response(self, cookie, host):
         s = flow.StickyCookieState(filt.parse(".*"))
         f = tutils.tflow_full()
-        f.request.host = host 
+        f.request.host = host
         f.response.headers["Set-Cookie"] = [cookie]
         s.handle_response(f)
         return s, f
 
+    def test_domain_match(self):
+        s = flow.StickyCookieState(filt.parse(".*"))
+        assert s.domain_match("www.google.com", ".google.com")
+        assert s.domain_match("google.com", ".google.com")
+
     def test_handle_response(self):
         c = "SSID=mooo, FOO=bar; Domain=.google.com; Path=/; "\
             "Expires=Wed, 13-Jan-2021 22:23:01 GMT; Secure; "
@@ -131,12 +136,35 @@ class uServerPlaybackState(libpry.AutoTree):
 
 
 class uFlow(libpry.AutoTree):
+    def test_copy(self):
+        f = tutils.tflow_full()
+        f2 = f.copy()
+        assert not f is f2
+        assert not f.request is f2.request
+        assert f.request.headers == f2.request.headers
+        assert not f.request.headers is f2.request.headers
+        assert f.response == f2.response
+        assert not f.response is f2.response
+
+        f = tutils.tflow_err()
+        f2 = f.copy()
+        assert not f is f2
+        assert not f.request is f2.request
+        assert f.request.headers == f2.request.headers
+        assert not f.request.headers is f2.request.headers
+        assert f.error == f2.error
+        assert not f.error is f2.error
+
     def test_match(self):
         f = tutils.tflow()
         f.response = tutils.tresp()
         f.request = f.response.request
         assert not f.match(filt.parse("~b test"))
         assert f.match(None)
+        assert not f.match(filt.parse("~b test"))
+
+        f = tutils.tflow_err()
+        assert f.match(filt.parse("~e"))
 
     def test_backup(self):
         f = tutils.tflow()
@@ -153,12 +181,12 @@ class uFlow(libpry.AutoTree):
     def test_getset_state(self):
         f = tutils.tflow()
         f.response = tutils.tresp(f.request)
-        state = f._get_state() 
+        state = f._get_state()
         assert f._get_state() == flow.Flow._from_state(state)._get_state()
 
         f.response = None
         f.error = flow.Error(f.request, "error")
-        state = f._get_state() 
+        state = f._get_state()
         assert f._get_state() == flow.Flow._from_state(state)._get_state()
 
         f2 = tutils.tflow()
@@ -284,7 +312,6 @@ class uState(libpry.AutoTree):
         assert c.add_response(resp)
         assert c.active_flow_count() == 0
 
-
     def test_err(self):
         c = flow.State()
         req = tutils.treq()
@@ -295,6 +322,15 @@ class uState(libpry.AutoTree):
         e = flow.Error(tutils.tflow().request, "message")
         assert not c.add_error(e)
 
+        c = flow.State()
+        req = tutils.treq()
+        f = c.add_request(req)
+        e = flow.Error(f.request, "message")
+        c.set_limit("~e")
+        assert not c.view
+        assert not c.view
+        assert c.add_error(e)
+        assert c.view
 
     def test_set_limit(self):
         c = flow.State()
@@ -368,7 +404,7 @@ class uState(libpry.AutoTree):
 
         flows = c.view[:]
         c.clear()
-        
+
         c.load_flows(flows)
         assert isinstance(c._flow_list[0], flow.Flow)
 
@@ -434,9 +470,19 @@ class uFlowMaster(libpry.AutoTree):
         s = flow.State()
         fm = flow.FlowMaster(None, s)
         assert not fm.load_script("scripts/a.py")
+        assert not fm.load_script("scripts/a.py")
+        assert not fm.load_script(None)
         assert fm.load_script("nonexistent")
         assert "ValueError" in fm.load_script("scripts/starterr.py")
 
+    def test_script_reqerr(self):
+        s = flow.State()
+        fm = flow.FlowMaster(None, s)
+        assert not fm.load_script("scripts/reqerr.py")
+        req = tutils.treq()
+        fm.handle_clientconnect(req.client_conn)
+        assert fm.handle_request(req)
+
     def test_script(self):
         s = flow.State()
         fm = flow.FlowMaster(None, s)
@@ -456,6 +502,17 @@ class uFlowMaster(libpry.AutoTree):
         fm.handle_error(err)
         assert fm.script.ns["log"][-1] == "error"
 
+    def test_duplicate_flow(self):
+        s = flow.State()
+        fm = flow.FlowMaster(None, s)
+        f = tutils.tflow_full()
+        fm.load_flow(f)
+        assert s.flow_count() == 1
+        f2 = fm.duplicate_flow(f)
+        assert f2.response
+        assert s.flow_count() == 2
+        assert s.index(f2)
+
     def test_all(self):
         s = flow.State()
         fm = flow.FlowMaster(None, s)
@@ -473,13 +530,17 @@ class uFlowMaster(libpry.AutoTree):
 
         rx = tutils.tresp()
         assert not fm.handle_response(rx)
-        
+
         dc = flow.ClientDisconnect(req.client_conn)
+        req.client_conn.requestcount = 1
         fm.handle_clientdisconnect(dc)
 
         err = flow.Error(f.request, "msg")
         fm.handle_error(err)
 
+        fm.load_script("scripts/a.py")
+        fm.shutdown()
+
     def test_client_playback(self):
         s = flow.State()
 
@@ -539,6 +600,7 @@ class uFlowMaster(libpry.AutoTree):
         fm.handle_response(tf.response)
         assert fm.stickycookie_state.jar
         assert not "cookie" in tf.request.headers
+        tf = tf.copy()
         fm.handle_request(tf.request)
         assert tf.request.headers["cookie"] == ["foo=bar"]
 
@@ -564,7 +626,7 @@ class uFlowMaster(libpry.AutoTree):
 
 class uRequest(libpry.AutoTree):
     def test_simple(self):
-        h = flow.Headers()
+        h = flow.ODictCaseless()
         h["test"] = ["test"]
         c = flow.ClientConnect(("addr", 2222))
         r = flow.Request(c, "host", 22, "https", "GET", "/", h, "content")
@@ -577,8 +639,53 @@ class uRequest(libpry.AutoTree):
         r2 = r.copy()
         assert r == r2
 
+        r.content = None
+        assert r._assemble()
+
+        r.close = True
+        assert "connection: close" in r._assemble()
+
+        assert r._assemble(True)
+
+    def test_getset_form_urlencoded(self):
+        h = flow.ODictCaseless()
+        h["content-type"] = [flow.HDR_FORM_URLENCODED]
+        d = flow.ODict([("one", "two"), ("three", "four")])
+        r = flow.Request(None, "host", 22, "https", "GET", "/", h, utils.urlencode(d.lst))
+        assert r.get_form_urlencoded() == d
+
+        d = flow.ODict([("x", "y")])
+        r.set_form_urlencoded(d)
+        assert r.get_form_urlencoded() == d
+
+        r.headers["content-type"] = ["foo"]
+        assert not r.get_form_urlencoded()
+
+    def test_getset_query(self):
+        h = flow.ODictCaseless()
+
+        r = flow.Request(None, "host", 22, "https", "GET", "/foo?x=y&a=b", h, "content")
+        q = r.get_query()
+        assert q.lst == [("x", "y"), ("a", "b")]
+
+        r = flow.Request(None, "host", 22, "https", "GET", "/", h, "content")
+        q = r.get_query()
+        assert not q
+
+        r = flow.Request(None, "host", 22, "https", "GET", "/?adsfa", h, "content")
+        q = r.get_query()
+        assert not q
+
+        r = flow.Request(None, "host", 22, "https", "GET", "/foo?x=y&a=b", h, "content")
+        assert r.get_query()
+        r.set_query(flow.ODict([]))
+        assert not r.get_query()
+        qv = flow.ODict([("a", "b"), ("c", "d")])
+        r.set_query(qv)
+        assert r.get_query() == qv
+
     def test_anticache(self):
-        h = flow.Headers()
+        h = flow.ODictCaseless()
         r = flow.Request(None, "host", 22, "https", "GET", "/", h, "content")
         h["if-modified-since"] = ["test"]
         h["if-none-match"] = ["test"]
@@ -587,7 +694,7 @@ class uRequest(libpry.AutoTree):
         assert not "if-none-match" in r.headers
 
     def test_getset_state(self):
-        h = flow.Headers()
+        h = flow.ODictCaseless()
         h["test"] = ["test"]
         c = flow.ClientConnect(("addr", 2222))
         r = flow.Request(c, "host", 22, "https", "GET", "/", h, "content")
@@ -617,6 +724,12 @@ class uRequest(libpry.AutoTree):
         assert not "foo" in r.content
         assert r.headers["boo"] == ["boo"]
 
+    def test_constrain_encoding(self):
+        r = tutils.treq()
+        r.headers["accept-encoding"] = ["gzip", "oink"]
+        r.constrain_encoding()
+        assert "oink" not in r.headers["accept-encoding"]
+
     def test_decodeencode(self):
         r = tutils.treq()
         r.headers["content-encoding"] = ["identity"]
@@ -625,6 +738,10 @@ class uRequest(libpry.AutoTree):
         assert not r.headers["content-encoding"]
         assert r.content == "falafel"
 
+        r = tutils.treq()
+        r.content = "falafel"
+        assert not r.decode()
+
         r = tutils.treq()
         r.headers["content-encoding"] = ["identity"]
         r.content = "falafel"
@@ -645,7 +762,7 @@ class uRequest(libpry.AutoTree):
 
 class uResponse(libpry.AutoTree):
     def test_simple(self):
-        h = flow.Headers()
+        h = flow.ODictCaseless()
         h["test"] = ["test"]
         c = flow.ClientConnect(("addr", 2222))
         req = flow.Request(c, "host", 22, "https", "GET", "/", h, "content")
@@ -655,6 +772,12 @@ class uResponse(libpry.AutoTree):
         resp2 = resp.copy()
         assert resp2 == resp
 
+        resp.content = None
+        assert resp._assemble()
+
+        resp.request.client_conn.close = True
+        assert "connection: close" in resp._assemble()
+
     def test_refresh(self):
         r = tutils.tresp()
         n = time.time()
@@ -684,7 +807,7 @@ class uResponse(libpry.AutoTree):
 
 
     def test_getset_state(self):
-        h = flow.Headers()
+        h = flow.ODictCaseless()
         h["test"] = ["test"]
         c = flow.ClientConnect(("addr", 2222))
         req = flow.Request(c, "host", 22, "https", "GET", "/", h, "content")
@@ -736,7 +859,7 @@ class uError(libpry.AutoTree):
     def test_getset_state(self):
         e = flow.Error(None, "Error")
         state = e._get_state()
-        assert flow.Error._from_state(state) == e
+        assert flow.Error._from_state(None, state) == e
 
         assert e.copy()
 
@@ -770,72 +893,38 @@ class uClientConnect(libpry.AutoTree):
         assert c3 == c
 
 
-class uHeaders(libpry.AutoTree):
+class uODict(libpry.AutoTree):
     def setUp(self):
-        self.hd = flow.Headers()
+        self.od = flow.ODict()
 
-    def test_read_simple(self):
-        data = """
-            Header: one
-            Header2: two
-            \r\n
-        """
-        data = textwrap.dedent(data)
-        data = data.strip()
-        s = StringIO(data)
-        self.hd.read(s)
-        assert self.hd["header"] == ["one"]
-        assert self.hd["header2"] == ["two"]
-
-    def test_read_multi(self):
-        data = """
-            Header: one
-            Header: two
-            \r\n
-        """
-        data = textwrap.dedent(data)
-        data = data.strip()
-        s = StringIO(data)
-        self.hd.read(s)
-        assert self.hd["header"] == ["one", "two"]
-
-    def test_read_continued(self):
-        data = """
-            Header: one
-            \ttwo
-            Header2: three
-            \r\n
-        """
-        data = textwrap.dedent(data)
-        data = data.strip()
-        s = StringIO(data)
-        self.hd.read(s)
-        assert self.hd["header"] == ['one\r\n two']
+    def test_str_err(self):
+        h = flow.ODict()
+        libpry.raises(ValueError, h.__setitem__, "key", "foo")
 
     def test_dictToHeader1(self):
-        self.hd.add("one", "uno")
-        self.hd.add("two", "due")
-        self.hd.add("two", "tre")
+        self.od.add("one", "uno")
+        self.od.add("two", "due")
+        self.od.add("two", "tre")
         expected = [
             "one: uno\r\n",
             "two: due\r\n",
             "two: tre\r\n",
             "\r\n"
         ]
-        out = repr(self.hd)
+        out = repr(self.od)
         for i in expected:
             assert out.find(i) >= 0
 
     def test_dictToHeader2(self):
-        self.hd["one"] = ["uno"]
+        self.od["one"] = ["uno"]
         expected1 = "one: uno\r\n"
         expected2 = "\r\n"
-        out = repr(self.hd)
+        out = repr(self.od)
         assert out.find(expected1) >= 0
         assert out.find(expected2) >= 0
 
     def test_match_re(self):
-        h = flow.Headers()
+        h = flow.ODict()
         h.add("one", "uno")
         h.add("two", "due")
         h.add("two", "tre")
@@ -844,36 +933,58 @@ class uHeaders(libpry.AutoTree):
         assert not h.match_re("nonono")
 
     def test_getset_state(self):
-        self.hd.add("foo", 1)
-        self.hd.add("foo", 2)
-        self.hd.add("bar", 3)
-        state = self.hd._get_state()
-        nd = flow.Headers._from_state(state)
-        assert nd == self.hd
+        self.od.add("foo", 1)
+        self.od.add("foo", 2)
+        self.od.add("bar", 3)
+        state = self.od._get_state()
+        nd = flow.ODict._from_state(state)
+        assert nd == self.od
+
+    def test_in_any(self):
+        self.od["one"] = ["atwoa", "athreea"]
+        assert self.od.in_any("one", "two")
+        assert self.od.in_any("one", "three")
+        assert not self.od.in_any("one", "four")
+        assert not self.od.in_any("nonexistent", "foo")
+        assert not self.od.in_any("one", "TWO")
+        assert self.od.in_any("one", "TWO", True)
 
     def test_copy(self):
-        self.hd.add("foo", 1)
-        self.hd.add("foo", 2)
-        self.hd.add("bar", 3)
-        assert self.hd == self.hd.copy()
+        self.od.add("foo", 1)
+        self.od.add("foo", 2)
+        self.od.add("bar", 3)
+        assert self.od == self.od.copy()
 
     def test_del(self):
-        self.hd.add("foo", 1)
-        self.hd.add("Foo", 2)
-        self.hd.add("bar", 3)
-        del self.hd["foo"]
-        assert len(self.hd.lst) == 1
+        self.od.add("foo", 1)
+        self.od.add("Foo", 2)
+        self.od.add("bar", 3)
+        del self.od["foo"]
+        assert len(self.od.lst) == 2
 
     def test_replace(self):
-        self.hd.add("one", "two")
-        self.hd.add("two", "one")
-        assert self.hd.replace("one", "vun") == 2
-        assert self.hd.lst == [
+        self.od.add("one", "two")
+        self.od.add("two", "one")
+        assert self.od.replace("one", "vun") == 2
+        assert self.od.lst == [
             ["vun", "two"],
             ["two", "vun"],
         ]
 
 
+class uODictCaseless(libpry.AutoTree):
+    def setUp(self):
+        self.od = flow.ODictCaseless()
+
+    def test_del(self):
+        self.od.add("foo", 1)
+        self.od.add("Foo", 2)
+        self.od.add("bar", 3)
+        del self.od["foo"]
+        assert len(self.od) == 1
+
+
+
 tests = [
     uStickyCookieState(),
     uStickyAuthState(),
@@ -887,5 +998,6 @@ tests = [
     uResponse(),
     uError(),
     uClientConnect(),
-    uHeaders(),
+    uODict(),
+    uODictCaseless(),
 ]

test/test_netstring.py

@@ -1,65 +0,0 @@
-from libmproxy import netstring
-from cStringIO import StringIO
-import libpry
-
-
-class uNetstring(libpry.AutoTree):
-    def setUp(self):
-        self.test_data = "Netstring module by Will McGugan"
-        self.encoded_data = "9:Netstring,6:module,2:by,4:Will,7:McGugan,"
-
-    def test_header(self):
-        t = [ ("netstring", "9:"),
-                  ("Will McGugan", "12:"),
-                  ("", "0:") ]
-        for test, result in t:
-            assert netstring.header(test) == result 
-            
-    def test_file_encoder(self):
-        file_out = StringIO()
-        data = self.test_data.split()
-        encoder = netstring.FileEncoder(file_out)
-        for s in data:
-            encoder.write(s)
-        encoded_data = file_out.getvalue()            
-        assert encoded_data == self.encoded_data
-        
-    def test_decode_file(self):
-        data = self.test_data.split()
-        for buffer_size in range(1, len(self.encoded_data)):
-            file_in = StringIO(self.encoded_data[:])            
-            decoded_data = list(netstring.decode_file(file_in, buffer_size = buffer_size))
-            assert decoded_data == data
-        
-    def test_decoder(self):
-        encoded_data = self.encoded_data
-        for step in range(1, len(encoded_data)):
-            i = 0
-            chunks = []
-            while i < len(encoded_data):
-                chunks.append(encoded_data[i:i+step])
-                i += step                
-            decoder = netstring.Decoder()
-            decoded_data = [] 
-            for chunk in chunks:
-                for s in decoder.feed(chunk):
-                    decoded_data.append(s)
-            assert decoded_data == self.test_data.split()
-
-    def test_errors(self):
-        d = netstring.Decoder()
-        libpry.raises("Illegal digit", list, d.feed("1:foo"))
-        d = netstring.Decoder()
-        libpry.raises("Preceding zero", list, d.feed("01:f"))
-        d = netstring.Decoder(5)
-        libpry.raises("Maximum size", list, d.feed("500:f"))
-        d = netstring.Decoder()
-        libpry.raises("Illegal digit", list, d.feed(":f"))
-
-
-
-
-tests = [
-    uNetstring()
-]
-

test/test_proxy.py

@@ -1,22 +1,53 @@
-import cStringIO, time
+import cStringIO, textwrap
+from cStringIO import StringIO
 import libpry
-from libmproxy import proxy, controller, utils, dump
+from libmproxy import proxy, flow
 
 
 class u_read_chunked(libpry.AutoTree):
     def test_all(self):
         s = cStringIO.StringIO("1\r\na\r\n0\r\n")
-        libpry.raises(IOError, proxy.read_chunked, s)
+        libpry.raises(IOError, proxy.read_chunked, s, None)
 
         s = cStringIO.StringIO("1\r\na\r\n0\r\n\r\n")
-        assert proxy.read_chunked(s) == "a"
+        assert proxy.read_chunked(s, None) == "a"
 
         s = cStringIO.StringIO("\r\n")
-        libpry.raises(IOError, proxy.read_chunked, s)
+        libpry.raises(IOError, proxy.read_chunked, s, None)
 
         s = cStringIO.StringIO("1\r\nfoo")
-        libpry.raises(IOError, proxy.read_chunked, s)
+        libpry.raises(IOError, proxy.read_chunked, s, None)
 
+        s = cStringIO.StringIO("foo\r\nfoo")
+        libpry.raises(proxy.ProxyError, proxy.read_chunked, s, None)
+
+
+class Dummy: pass
+
+
+class u_read_http_body(libpry.AutoTree):
+    def test_all(self):
+
+        d = Dummy()
+        h = flow.ODict()
+        s = cStringIO.StringIO("testing")
+        assert proxy.read_http_body(s, d, h, False, None) == ""
+
+        h["content-length"] = ["foo"]
+        s = cStringIO.StringIO("testing")
+        libpry.raises(proxy.ProxyError, proxy.read_http_body, s, d, h, False, None)
+
+        h["content-length"] = [5]
+        s = cStringIO.StringIO("testing")
+        assert len(proxy.read_http_body(s, d, h, False, None)) == 5
+        s = cStringIO.StringIO("testing")
+        libpry.raises(proxy.ProxyError, proxy.read_http_body, s, d, h, False, 4)
+
+        h = flow.ODict()
+        s = cStringIO.StringIO("testing")
+        assert len(proxy.read_http_body(s, d, h, True, 4)) == 4
+        s = cStringIO.StringIO("testing")
+        assert len(proxy.read_http_body(s, d, h, True, 100)) == 7
 
 
 class u_parse_request_line(libpry.AutoTree):
@@ -64,9 +95,51 @@ class uProxyError(libpry.AutoTree):
         assert repr(p)
 
 
+class u_read_headers(libpry.AutoTree):
+    def test_read_simple(self):
+        data = """
+            Header: one
+            Header2: two
+            \r\n
+        """
+        data = textwrap.dedent(data)
+        data = data.strip()
+        s = StringIO(data)
+        headers = proxy.read_headers(s)
+        assert headers["header"] == ["one"]
+        assert headers["header2"] == ["two"]
+
+    def test_read_multi(self):
+        data = """
+            Header: one
+            Header: two
+            \r\n
+        """
+        data = textwrap.dedent(data)
+        data = data.strip()
+        s = StringIO(data)
+        headers = proxy.read_headers(s)
+        assert headers["header"] == ["one", "two"]
+
+    def test_read_continued(self):
+        data = """
+            Header: one
+            \ttwo
+            Header2: three
+            \r\n
+        """
+        data = textwrap.dedent(data)
+        data = data.strip()
+        s = StringIO(data)
+        headers = proxy.read_headers(s)
+        assert headers["header"] == ['one\r\n two']
+
+
 tests = [
     uProxyError(),
     uFileLike(),
     u_parse_request_line(),
     u_read_chunked(),
+    u_read_http_body(),
+    u_read_headers()
 ]

test/test_script.py

@@ -1,6 +1,7 @@
 import os
 from libmproxy import script, flow
 import libpry
+import tutils
 
 class uScript(libpry.AutoTree):
     def test_simple(self):
@@ -13,15 +14,25 @@ class uScript(libpry.AutoTree):
         assert "here" in p.ns
         assert p.run("here") == (True, 1)
         assert p.run("here") == (True, 2)
-        
+
         ret = p.run("errargs")
-        assert not ret[0] 
+        assert not ret[0]
         assert len(ret[1]) == 2
 
         # Check reload
         p.load()
         assert p.run("here") == (True, 1)
 
+    def test_duplicate_flow(self):
+        s = flow.State()
+        fm = flow.FlowMaster(None, s)
+        fm.load_script(os.path.join("scripts", "duplicate_flow.py"))
+        r = tutils.treq()
+        fm.handle_request(r)
+        assert fm.state.flow_count() == 2
+        assert not fm.state.view[0].request.is_replay()
+        assert fm.state.view[1].request.is_replay()
+
     def test_err(self):
         s = flow.State()
         fm = flow.FlowMaster(None, s)

test/test_server.py

@@ -1,5 +1,4 @@
 import urllib, urllib2
-from libmproxy import flow
 import tutils
 
 class uSanity(tutils.ProxTest):
@@ -41,7 +40,7 @@ class uProxy(tutils.ProxTest):
         assert f.code == 200
         assert f.read()
         f.close()
-            
+
         l = self.log()
         assert l[0].address
         assert "host" in l[1].headers

test/test_utils.py

@@ -1,4 +1,4 @@
-import textwrap, cStringIO, os, time, re, json
+import textwrap, os, re, json
 import libpry
 from libmproxy import utils
 
@@ -29,6 +29,7 @@ class uhexdump(libpry.AutoTree):
     def test_simple(self):
         assert utils.hexdump("one\0"*10)
 
+
 class udel_all(libpry.AutoTree):
     def test_simple(self):
         d = dict(a=1, b=2, c=3)
@@ -36,6 +37,13 @@ class udel_all(libpry.AutoTree):
         assert d.keys() == ["c"]
 
 
+class uclean_hanging_newline(libpry.AutoTree):
+    def test_simple(self):
+        s = "foo\n"
+        assert utils.clean_hanging_newline(s) == "foo"
+        assert utils.clean_hanging_newline("foo") == "foo"
+
+
 class upretty_size(libpry.AutoTree):
     def test_simple(self):
         assert utils.pretty_size(100) == "100B"
@@ -138,12 +146,12 @@ class udummy_cert(libpry.AutoTree):
         d = self.tmpdir()
         cacert = os.path.join(d, "foo/cert.cnf")
         assert utils.dummy_ca(cacert)
-        assert utils.dummy_cert(
+        p = utils.dummy_cert(
             os.path.join(d, "foo"),
             cacert,
             "foo.com"
         )
-        assert os.path.exists(os.path.join(d, "foo", "foo.com.pem"))
+        assert os.path.exists(p)
         # Short-circuit
         assert utils.dummy_cert(
             os.path.join(d, "foo"),
@@ -153,12 +161,12 @@ class udummy_cert(libpry.AutoTree):
 
     def test_no_ca(self):
         d = self.tmpdir()
-        assert utils.dummy_cert(
+        p = utils.dummy_cert(
             d,
             None,
             "foo.com"
         )
-        assert os.path.exists(os.path.join(d, "foo.com.pem"))
+        assert os.path.exists(p)
 
 
 class uLRUCache(libpry.AutoTree):
@@ -192,6 +200,22 @@ class uLRUCache(libpry.AutoTree):
         assert len(f._cachelist_one) == 2
 
 
+class u_parse_proxy_spec(libpry.AutoTree):
+    def test_simple(self):
+        assert not utils.parse_proxy_spec("")
+        assert utils.parse_proxy_spec("http://foo.com:88") == ("http", "foo.com", 88)
+        assert utils.parse_proxy_spec("http://foo.com") == ("http", "foo.com", 80)
+        assert not utils.parse_proxy_spec("foo.com")
+        assert not utils.parse_proxy_spec("http://")
+
+
+class u_unparse_url(libpry.AutoTree):
+    def test_simple(self):
+        assert utils.unparse_url("http", "foo.com", 99, "") == "http://foo.com:99"
+        assert utils.unparse_url("http", "foo.com", 80, "") == "http://foo.com"
+        assert utils.unparse_url("https", "foo.com", 80, "") == "https://foo.com:80"
+        assert utils.unparse_url("https", "foo.com", 443, "") == "https://foo.com"
+
 
 class u_parse_url(libpry.AutoTree):
     def test_simple(self):
@@ -216,6 +240,21 @@ class u_parse_url(libpry.AutoTree):
         s, h, po, pa = utils.parse_url("https://foo")
         assert po == 443
 
+        assert not utils.parse_url("https://foo:bar")
+        assert not utils.parse_url("https://foo:")
+
+
+class u_parse_size(libpry.AutoTree):
+    def test_simple(self):
+        assert not utils.parse_size("")
+        assert utils.parse_size("1") == 1
+        assert utils.parse_size("1k") == 1024
+        assert utils.parse_size("1m") == 1024**2
+        assert utils.parse_size("1g") == 1024**3
+        libpry.raises(ValueError, utils.parse_size, "1f")
+        libpry.raises(ValueError, utils.parse_size, "ak")
+
+
 tests = [
     uformat_timestamp(),
     uisBin(),
@@ -230,5 +269,9 @@ tests = [
     udummy_ca(),
     udummy_cert(),
     uLRUCache(),
-    u_parse_url()
+    u_parse_url(),
+    u_parse_proxy_spec(),
+    u_unparse_url(),
+    u_parse_size(),
+    uclean_hanging_newline()
 ]

test/tutils.py

@@ -1,13 +1,13 @@
-import os.path, threading, Queue
+import threading, Queue
 import libpry
-from libmproxy import proxy, filt, flow, controller
+from libmproxy import proxy, flow, controller
 import serv, sslserv
 import random
 
 def treq(conn=None):
     if not conn:
         conn = flow.ClientConnect(("address", 22))
-    headers = flow.Headers()
+    headers = flow.ODictCaseless()
     headers["header"] = ["qvalue"]
     return flow.Request(conn, "host", 80, "http", "GET", "/path", headers, "content")
 
@@ -15,7 +15,7 @@ def treq(conn=None):
 def tresp(req=None):
     if not req:
         req = treq()
-    headers = flow.Headers()
+    headers = flow.ODictCaseless()
     headers["header_response"] = ["svalue"]
     return flow.Response(req, 200, "message", headers, "content_response")
 
@@ -50,8 +50,8 @@ HTTPS_PORT = random.randint(30000, 40000)
 
 class TestMaster(controller.Master):
     def __init__(self, port, testq):
-        serv = proxy.ProxyServer(proxy.SSLConfig("data/testkey.pem"), port)
-        controller.Master.__init__(self, serv)
+        s = proxy.ProxyServer(proxy.ProxyConfig("data/testkey.pem"), port)
+        controller.Master.__init__(self, s)
         self.testq = testq
         self.log = []
 

todo

@@ -2,17 +2,25 @@ This is a loose collection of todo items, in case someone else wants to start
 hacking on mitmproxy. Drop me a line (aldo@corte.si) if you want to tackle any
 of these and need some pointers.
 
-Features:
+Targeted for 0.8:
+    - Upstream proxy support.
+    - Improve worst-case performance problem with XML-ish indenter
+    - Follow mode to keep most recent flow in view
+    - Rewrite the core to be asynchronous. I've done some research, and
+    although it's a bit of a bloated monster, it looks like Twisted is the way
+    to go.
+    - Verbose view to show timestamps
+    - Search within requests/responses
+    - Transparent proxy support
+    - Ordering a-la mutt's "o" shortcut
 
+
+Further ideas:
     - Add some "workspace" features to mitmproxy:
         - Flow comments
         - Copying/duplicating flows
         - Ordering by time, size, etc. a-la-mutt (o keyboard shorcut is reserved for this)
-    - Post and URL field parsing and editing.
-        - We need a built-in graphical editor for these, that knows how to
-        break fields up and present them for modificatio individually.
-    - Upstream proxy support.
     - Support HTTP Digest authentication through the stickyauth option. We'll
-    have to save the server nonce, and recalculate the hashes for each request.
+      have to save the server nonce, and recalculate the hashes for each request.
     - Chunked encoding support for requests (we already support it for responses).
     - A progress indicator for large files