Mobile Development with Mitmproxy: mocking stuff is sooo cool!!!

Hey everyone, I’m Rafa and I’ll be sharing with you a little bit about a tool that is helping me a lot these days: Mitmproxy. Although, lets start from the beginning…

If you don’t know or never heard about what a Http Proxy is, I really encourage you to spend a little bit of you precious programming time to give it a try. Basically, a Http Proxy is a machine that stands between the router and the client (mobile, browser, etc.). It gives you a broad view on which kind of information you are exchanging while surfing the internet. It gives you the opportunity to monitor the network traffic completely (if you have the correct certificates installed).

But some of you might ask: “what do proxies have to do with software development?”. Nowadays, almost every app needs to be connected with the internet to give more value to the user. Up to date and synchronized data are key. That means the products needs to be connected to extend the user experience to the next level. During an app development, for instance, we always face the disparity with the backend development team in velocity to deliver new features. Since the server is a blocker to deliver the majority of the big features, it can be a problem if not well planned. That’s where the Http Proxies come in.

Http Proxies also lets the developer to mock data. Since the Http Proxy serves as a transparent server, it is invisible for the app to where its data is being sent. That opens a room of possibilities for us developers.

There are a lot of options out there but the majority of them are paid. I was looking for an open source Http Proxy that could be easily configured, easy to use and, obviously, free ☺️. Then I found Mitmproxy.

Mitmproxy

This powerful proxy is an open source project that has been maintained for over 10 years. It is very easy to configure and provides some cool features like traffic sniffing and data interception / modification.

It has 3 different user interfaces:
1. mitmproxy: this is a terminal interface where you can interact with it and see the requests going through it.
2. mitmdump: this is also a terminal interface but just for visualization. No interaction available.
3. mitmweb: this is a web app where you can interact and see all the requests in a browser.

All of them use the same code behind the scenes, only the interface changes.
I’ll be focusing on the mitmweb in this tutorial since I don’t really use the other ones.

Installation

The installation is very straightforward. In less then 5 minutes you have everything setup and running to start using it.

Note: I’ll be focusing on Mac installation for this entire tutorial.

First of all you’ll need to download the Mitmproxy tool. You can do this by typing the following command on your terminal app:

brew install mitmproxy

This is going to install the proxy command line tool and place it on your environment path so you can access it from anywhere in your computer.

To run the proxy, type:

mitmweb

You’ll see that nothing happens which means the network traffic is not going through the proxy. There are two problems there: the first one is that you need to install its certificates (for security purposes) before start using it. The second one is that you need to configure your machine to enable the proxy.

To solve the first problem, open the Mimtproxy installation directory by typing:

open ~/.mitmproxy

You’ll see some certificates like the image below:

You want to choose the mitmproxy-ca.pem file and double click it. It will open the Keychain Access app and install the certificate there. This should solve the first problem.

To solve the second problem you’ll need to go to:

System Preferences > Network > Advanced > Proxies

Then you should enable Web Proxy (HTTP) and Secure Web Proxy (HTTPS). For both of them you’ll need to fill out the proxy server and the corresponding port like the image below:

Hit OK and Apply. And that’s it 🚀!! Now if you run the proxy again, open another tab on your browser and access google.com, for instance, you will see something like this:

Woo woo!!! You are up and running 🎉.

I’m not going through all the features the web app provides in this tutorial. But you can check this out in the oficial Mitmproxy site.

The mobile app

To exemplify the use of the proxy in a real scenario, imagine that you have a mobile development that is walking side by side with the cloud development. You already discussed a cloud contract and know how the API looks like. The app I’m going to showcase here is a simple To Do app that fetches a To Do list from a server that doesn’t exist yet (that means without the proxy the app doesn’t work). You can checkout the mobile app code in here https://github.com/rafellk/Mitmproxy-example (if you like it please leave a star there 🎉).

As I said, the contract is already defined and looks like the following:

{
    "resolved": [
        {
            "title": "Create iOS app",
            "resolved": true
        },
    ],
    "unresolved": [
        {
            "title": "Write a new article and share with the community",
            "resolved": false
        }
    ]
}

Basically this consists into two models that rules the entire app:

private struct TodosResponse: Codable {
    let resolved: [TodoModel]
    let unresolved: [TodoModel]
}
private struct TodoModel: Codable {
    let title: String
    let resolved: Bool
}

The app running looks like this:

Mitmproxy code

The Mitmproxy tool is totally scriptable. That means you write code to intercept the requests or do whatever you want with them.

Basically, for this simple app I just need to intercept a request and return the response I want instead of reaching the actual server for it.
Mitmproxy also supports response interception where you can intercept the server response and alter any data you want. For more information about the response interception and other events hooks see the Addons Example in the oficial Mitmproxy web site.

The Mitmproxy scripts are written in Python but you don’t actually need to know Python to understand it since it is very simple and straightforward.

The script we are going to write is going to intercept the request we want and print at first so we can see it is actually intercepting the request or not. First of all, create a new folder in the directory you find more suitable called MockedResponses (or any name you like the most). Then you access that repository and create a file that will be the main script you are going to run. I’m going to call it mock.py. To intercept a request you basically need to implement a function that is going to be called every time a new request is performed. This method is signature is:

request(flow: http.HTTPFlow) -> None

To make this method available you will need to import the Http library:

from mitmproxy import http

Now implement the request method like the following:

def request(flow: http.HTTPFlow) -> None:
    print("testing")

Then run the proxy pointing to the script you just created and activate the proxy settings in System Preferences > Networks > Advanced > Proxies.

mitmweb -s mock.py

Now, open a new browser window and search anything in the search bar. You should see a lot of “testing” logs in the terminal like the following:

If you can see the logs it means you are successfully capturing the request before is being performed. That’s where we will start to work on right now!!

Given we have the request method capturing the request that means we can manipulate the request before it actually reaches the server. In this tutorial, I’ll override the request with an entire json that conforms to the cloud contract previously defined.

But before that I you may have noticed that every time you need to run the proxy you need to go activate it in System Preferences. I wrote a script to automatically activate it (when launch) and deactivate it (when terminated) so you only care about running and killing the process. You are welcome ❤️!!! You can find both scripts below:

proxy_on.sh

#!/bin/sh
activeNetwork=$(route get default | grep interface | awk ‘{print $2}’)
activeNetworkName=$(networksetup -listallhardwareports | grep -B 1 “$activeNetwork” | awk ‘/Hardware Port/{ print }’|cut -d “ “ -f3-)
networksetup -setwebproxy “$activeNetworkName” localhost 8080 off
networksetup -setsecurewebproxy “$activeNetworkName” localhost 8080 off

proxy_off.sh

#!/bin/sh
activeNetwork=$(route get default | grep interface | awk '{print $2}')
activeNetworkName=$(networksetup -listallhardwareports | grep -B 1 "$activeNetwork" | awk '/Hardware Port/{ print }'|cut -d " " -f3-)
networksetup -setwebproxystate "$activeNetworkName" off
networksetup -setsecurewebproxystate "$activeNetworkName" off
networksetup -setwebproxystate "Wi-Fi" off
networksetup -setsecurewebproxystate "Wi-Fi" off

Place both the scripts in the same directory as the mock.py file and lets call the scripts on launch and termination process events:

import subprocess
import signal
# Setup onStart handler
def load(l):
    print("setting up network proxy")
    subprocess.Popen("sh proxy_on.sh".split(), stdout=subprocess.PIPE)
# Setup onExit handler
def handler(signum, frame):
    print("shutting down")
    subprocess.Popen("sh proxy_off.sh".split(), stdout=subprocess.PIPE)
signal.signal(signal.SIGINT, handler)
signal.signal(signal.SIGTERM, handler)

Now if you run your proxy again pointing to the mock.py file you will see the traffic passing through automatically. Woo woo!!! It beginning to get in a beautiful shape. Let’s move forward to the actual request interception code.

Again using the flow parameter you will be able to access the request url so you can identify which request you want to intercept. Remove the print statement inside the request(flow: http.HTTPFlow) method and replace with the following:

url = "http://com.rlmg.mitmproxy.server.com/todos"
if url == flow.request.pretty_url:
    print("intercepting request")
    return

Then run your proxy again and open a new browser tab and paste http://com.rlmg.mitmproxy.server.com/todos. You should see a 502 Bad Gateway error in the browser but that’s expected since we are not mocking anything yet and the actual server doesn’t exist 😬. Although, if you see the terminal logs, you will be able to see some intercepting request for every time you tried to perform the request. That means we are intercepting the correct url.

Now that we are all set, we just need to override the request and send a json as its response. Note: overriding the request will prevent the request to reach the actual server. I’ll use the same json I presented in the beginning as my response. To be more organized, create a folder called jsons in the same directory of your mock.py file then create a file to store the json content in the jsons directory. I’ll call it todos.json and the content will be:

{
    "resolved": [
        {
            "title": "Create iOS app",
            "resolved": true
        },
        {
            "title": "Create Mitmproxy template",
            "resolved": true
        },
        {
            "title": "Connect both iOS and Mitmproxy",
            "resolved": true
        }
    ],
    "unresolved": [
        {
            "title": "Write a new article and share with the community",
            "resolved": false
        }
    ]
}

Now that I have the json response I want to use as the request response then I just need to load it in the script. One more time I’ll use the flow parameter to override the response:

import os
directory_path = os.path.dirname(os.path.abspath(__file__))
flow.response = http.HTTPResponse.make(
                200,
                open(directory_path + "/jsons/todos.json", "r").read(),
                {"Content-Type": "application/json"}
            )
            return

The code above finds the relative path to the todos.json file and use the HTTPResponse.make method to create a mocked response with json content-type and a 200 status code. Tip: you can change the status code to 400 or 500 if you want to test error state for instance.

Now that we have the json setup, we should see the json response in the browser. Run the proxy and access http://com.rlmg.mitmproxy.server.com/todos in the browser and you should see something like:

If you can successfully see it then it means we are good to go 🎉🎉🎉. We successfully mocked the request response and if you run the app you should see it fully functional. You can also play around with the json to test different stuff like error handling, big list, empty list, etc.

This is what you should see after you run the app 🎉:

Conclusion

Mitmproxy is a very nice, open source, easy to configure, easy to use and incredibly good to scale alternative to CharlesProxy for example. Since it is scriptable you can make it a more complex program more than just mock a request response. You can change the mocked response json file dynamically after some tries to test a client retry action for instance.

You can find the repository with the iOS and Mitmproxy code in here: https://github.com/rafellk/Mitmproxy-example.

Well that’s all folks.

Thanks for staying until this very end. Leave a star on Github and a clap here if it was helpful. See you in the next one!!!