Intercepting Network Requests with Python and Playwright

Learn how to mock, wait, and abort web requests for test automation.

Photo by James Harrison on Unsplash

Mocking is a testing convention normally found in unit testing. The purpose of mocking is to isolate behavior of an element under test by replacing other elements with simulations. In test automation, mocking is useful for creating a curated and sterile testing environment.

This tutorial will focus on how to intercept web requests using Python and Playwright for the purpose of test automation. We will use the DemoQA Bookstore application as a base in order to create a user journey where a user may select a single book.

Getting Started

You will need to install the following packages using Pip or the package manager of your choosing:

• Playwright
• Pytest
• Pytest-Playwright

The pytest-playwright library is maintained by the creators of Playwright. It comes with a bunch of useful fixtures and methods for engineering convenience.

Listening to the Network

Playwright makes it easy to intercept network traffic using the page.on method. From here, all requests or responses may be monitored for a specific browser page.

page.on(
    "request", lambda request: print(request.method, request.url)
)

The above captures all network requests, then prints the request method and URL to the console when a request is intercepted. Opening the DemoQA Bookstore application with Playwright and the above code will output the following to your terminal:

A printout of /books requests.

For the sake of this tutorial, we will only take action against the /Books and /images requests.

Mocking Network Traffic

The /Books endpoint returns a JSON object containing a “books” array. This array is further populated by objects containing book data. The front end iterates through the array of books and shows each entry in the bookstore application. Knowing this, we can support our user journey by curating the list down to a single entity.

// books.json
{
    "books": [
        {
            "isbn": "9781449337711",
            "title": "Designing Evolvable Web APIs with ASP.NET",
            "subTitle": "Harnessing the Power of the Web",
            "author": "Glenn Block et al.",
            "publish_date": "2020-06-04T09:12:43.000Z",
            "publisher": "O'Reilly Media",
            "pages": 238,
            "description": "Design and build Web APIs for a...",
            "website": "https://chimera.labs.oreilly.com/books/..."
        }
    ]
}

Using Playwright’s page.route method, we can create a lambda function which uses the route.fulfill method to intercept requests made to the provided URL, then mock a response.

def test_select_single_book(page):
    """Using a mock, select a single book in the application."""
    book_title = "Designing Evolvable Web APIs with ASP.NET"
    page.route(
        "**/BookStore/v1/Books",
        lambda route: route.fulfill(path="./data/books.json")
    )

The fulfill method may be used to mock response path, body, status, headers, or content type. In this case, we mock the response path using our books.json file from above.

We can now finish writing our test.

#test_books.py
def test_select_single_book(page):
    """Using a mock, select a single book in the application."""
    book_title = "Designing Evolvable Web APIs with ASP.NET"
    page.route(
        "**/BookStore/v1/Books",
        lambda route: route.fulfill(path="./data/books.json")
    )
    page.goto("https://www.demoqa.com/books")
    book = page.wait_for_selector(
        f"a >> text={book_title}"
    )
    book.click()
    visible = page.wait_for_selector(
        f"label >> text={book_title}"
    ).is_visible()
    assert visible

When run, our “user” will see a single entry within the bookstore application, click the entry, then verify that the book’s title appears on the page.

Waiting…

What if we need to wait for the response to finish before taking action against the page?

Waiting for requests and responses has become more common in test automation, especially for applications with long load times. Playwright makes it simple with the expect_response and expect_request methods.

with page.expect_response("**/BookStore/v1/Books") as response:
    page.goto("https://www.demoqa.com/books")
assert response.value.ok

The expect_response method returns an EventContextManager which is invoked using the with statement. Calling response.value returns a Response class which features an ok property. The ok property returns a boolean value for whether a 200-level response was received or not.

The above code will wait for the default timeout period until a response is received. If not, a timeout error will be raised. The same implementation may be used with expect_request.

Abort!

Playwright affords engineers with the opportunity to abort requests. This is especially useful for pages or applications which are image heavy, thereby increasing load time. Similar to a mock, aborting requests intercept a provided URL and then pass an abort method using a lambda function.

page.route("**/*.jpg", lambda route: route.abort())

In this example, all requests ending in “.jpg” will be aborted.

The DemoQA Bookstore application with aborted images.

Verification

To run this test, input one of the following commands to your terminal:

• pytest to run headlessly
• pytest --headful to run in a headful state

Project Directory

Your directory should resemble the following upon completion of this tutorial:

data
    |__ books.json
tests
    |__ test_books.py
.gitignore
README.md
requirements.txt

Summary

Playwright is a powerful tool for intercepting web requests. Using mocks, engineers can create sterile test environments while waits and aborts are great for handling slow-loading applications.

Jonathan Thompson is a Senior Quality Engineer specializing in test automation. He currently resides in Raleigh, NC with his wife and a Goldendoodle named Winston. You can connect with him on LinkedIn, or follow him on either Twitter (@jacks_elsewhere) or Github (ThompsonJonM).