At Ginkgo we use the Python standard library module unittest for automating the testing of our Python code which is a powerful class-based approach to catching bugs and regressions. Philosophically, a test should only target a small piece of code and every HTTP request should be mocked so it can be relied on to always provide the same result. In practice, making sure tests don’t make unintended network requests make the tests more resilient so they don’t fail if an API is down or the environment in which the test is running has connection issues. A test that includes an inadvertent request could accidentally cause unintentional mutation of data, for example, by calling an API endpoint that updates a database with a payload of test data. The database could be updated every time the test suite is run and could could conceivably fill the database with unexpected junk data and cause problems for the rest of the testing environment.
Code often has to call HTTP APIs, so when testing, to mock any external requests we utilize VCR.py to record the requests and responses which increases the deterministic nature of the tests and removes the time it takes for a live request to be completed. The first time a test with a HTTP request is run within a VCR.py context manager VCR.py will record the HTTP traffic and create a flat file, called a “cassette”, that serializes the interaction. When the test is run again VCR.py will recognize the request and return the cassette file instead, preventing any live HTTP traffic. Writing tests can be complicated and sometimes a test will use code that can unknowingly include an external request to an API which is why it would be beneficial for the tests to warn contributors about unmocked external requests so they can be wrapped in a VCR.py context manager and have a cassette generated.
To implement this behavior we will extend TestCase from django.test and utilize the setUpClass to preform that patching during initialization. (As an aside don’t forget super, the absence of which can lead to some nasty bugs.) When tackling this, my first approach involved monkey patching urllib3 which is called on every external request. VCR.py already monkey patches HTTPConnection from urllib3 so it can return cassettes instead of live requests. If we want to avoid incorrectly flagging VCR cassettes as outgoing requests, we have to go lower and monkey patch urllib3.util.connection.create_connection instead.
class RequireMockingForHttpRequestsTestCase(TestCase):
@classmethod
def setUpClass(cls):
super(RequireMockingForHttpRequestsTestCase, cls).setUpClass()
def mock_create_conn(address, timeout, *args, **kw):
raise Exception(f'Unmocked request for URL {address}')
# monkey patching urllib3 to throw an exception on unmocked requests
# must be commented out to generate cassettes for vcrpy
urllib3_patch = mock.patch(
target="urllib3.util.connection.create_connection", new=mock_create_conn
)
urllib3_patch.start()
While this approach is successful in catching requests it has one glaring issue when combined with VCR.py, that VCR.py actually needs external requests to complete when it is first run in order to generate the cassette that is then used as the mock response for any further calls. With a simple patching of urllib3 this would block VCR.py from being able to generate the cassette and require the developer to comment out this line when first creating a new test — unideal.
Here is an approach that patches urllib3 more intelligently. Based on parsing the stack trace this will allow requests originating from VCR cassette creation to make a network request as expected but everything else will raise an exception. We decided not to go with this approach, deeming it too hacky because of its reliance on string manipulation and the install location of the Python package.
class RequireMockingForHttpRequestsTestCase(TestCase):
@classmethod
def setUpClass(cls):
super(RequireMockingForHttpRequestsTestCase, cls).setUpClass()
def mock_create_conn(adapter, request, **kw):
if any('dist-packages/vcr' in line.filename for line in traceback.extract_stack()):
return real_create_conn(adapter, request, **kw)
else:
raise Exception(f'Unmocked request for URL {adapter[0]}')
urllib3_patch = mock.patch(
target="urllib3.util.connection.create_connection", new=mock_create_conn
)
urllib3_patch.start()
Our solution was to allow external requests based on an environment variable which would preserve the ability to locally generate VCR cassettes but fail any outgoing connections when the tests are run in our GitLab pipelines.
class RequireMockingForHttpRequestsTestCase(TestCase):
@classmethod
def setUpClass(cls):
super(RequireMockingForHttpRequestsTestCase, cls).setUpClass()
def mock_create_conn(adapter, request, **kw):
raise Exception(f'Unmocked request for URL {adapter[0]}')
if os.environ.get("GITLAB_CI"):
urllib3_patch = mock.patch(
target="urllib3.util.connection.create_connection", new=mock_create_conn
)
urllib3_patch.start()
This comes with a tradeoff as now an unmocked request will pass all tests within a local environment but fail in the Gitlab pipeline but we decided that was the most robust failure mode because it prevents bad test from entering into production code.
Even thought this change is very recent it has already caught unmocked requests for our developers making our Python tests stronger. We hope you find this helpful as well!
(Feature photo by Daniel Schludi on Unsplash)
