Unit Testing with Mocks

Introduction

About Me

Professional Work

Works for Zilogic Systems
Technologist and Trainer
Specializes in
- Linux
- Embedded Systems
- Python

Community Work

Co-organizer of Chennaipy
Previously
- Co-organizer of PyCon India 2019
- Co-organizer of PySangamam 2018
Twitter: @bravegnu
Follow on LinkedIn

E2E vs Unit Testing

People generally tend to test their software as a whole
- Run the program
- Input data through the UI
- Check program’s response
Tests done this way automated or manual is called End-to-End Testing
E2E Testing is required but not sufficient

Drawbacks of E2E Testing

It is terribly slow
- Accessing database, networks, files, etc. is slow
- Implication: Will not be run often
Running test cases has dependencies
- Depends on the availability of test server, input emulators, etc.
- Implication: Only people with access to resources can execute it
Hard to determine root cause of failing test
- Implication: For a large code base, takes time and effort to zero in on the faulty code

Drawback of E2E Testing (Contd.)

Cannot drive program through all possible code path, low code coverage
- Some error scenarios is almost impossible to create
- Implication: Low code coverage, and typos lurking in untested code paths
E2E tests can be flaky
- Intermittent network failures, power outages, device failures, can cause tests to fail
- Implication: Reduces trust on the tests

Unit Testing

Unit Tests, verify the public interface of a module / class, in isolation
Dependencies are mocked or faked
Written and executed by developers

Advantages of Unit Testing

Unit tests are extremely fast
- They do not use databases, network, file system, other applications, console, etc.
Unit tests provide early feedback
- No test setup or resources are required to run the tests
- Developers can run them every time they make a change and before committing the code

Advantages of Unit Testing (Contd.)

Unit tests makes it easier to refactor
- Any regression due to refactoring, will be caught by unit tests
- Code can be refactored / improved without the fear of regression

Testing Pyramid

Unit tests verify the functionality of each unit
Integration tests verify that the units work together at the sub-system level
E2E tests verify that the product works as a whole
A good test strategy will have, a mix of each approach, with most of the testing done at the unit level

Writing Unit Tests

Sokoban

A puzzle game, where the player pushes boxes to docks
Restrictions
- Boxes cannot be pulled
- Boxes cannot be pushed into walls or other boxes
Under 400 lines of code, written using Pygame
Will serve as base for practicing writing unit test cases

Sokoban

First, let’s try playing around with it
You need to have pygame installed
Download game.zip

Extract and run the program

$ unzip game.zip
$ cd game
$ python3 sokoban.py

Arrow keys to move, S to skip to next level

Unit Testing Frameworks

Unit testing frameworks provide the necessary infrastructure to
- Write test cases
- Run the test cases
- Generate test reports
Available unit testing frameworks
- unittest, part of standard library
- py.test, popular third party framework
- nose2, extends unittest, making nicer and easier

Sokoban Levels

from sokoban import World

level = [
    "#########",
    "#.$ @   #",
    "#########",
]

world = World(level)
print(world.nrows)      # 3
print(world.ncols)      # 9
print(world.worker_pos) # [(4, 1)]
print(world.box_pos)    # [(2, 1)]
print(world.dock_pos)   # [(1, 1)]

A level is represented as a list of strings.
Corresponding graphical representation
The World class parses the given level, and sets up its attributes

Writing a Test Case

from sokoban import World

SIMPLE_LEVEL = [
    "#########",
    "#.$ @   #",
    "#########",
]

def test_rect_world_dimensions():
    world = World(SIMPLE_LEVEL)

    assert world.nrows == 3
    assert world.ncols == 9

Try Out

Create a file test_sokoban.py in the same directory as sokoban.py
Add the test case from the previous slide
Try running the test case using
```
$ py.test
```

Assert Raises Exception

A ValueError is thrown
- When an invalid character is present
- Or when the worker is missing
- Or when the number of boxes and docks do not match
- Or when the no. of boxes is zero

Assert Raises Exception (Contd.)

pytest.raises can be used to verify that an exception was raised
```
pytest.raises(exception, func, *args)
```

Verify that ValueError is raised when an invalid character is present

def test_invalid_character_error():
    level = [
        "########!",
        "#.$     #",
        "#########",
    ]

    pytest.raises(ValueError, World, level)

Testing `get()` method

get() method returns a Tile, at the specified location.

The Tile object indicates the properties of the tile in world.

def test_world_get_wall():
    world = World([
        "#########",
        "#.$  @  #",
        "#########",
    ])
    expected = Tile(wall=True, worker=False, dock=False, box=False)

    got = world.get((0, 0))

    assert got == expected

Try Out

Add a test case to check if there is a dock in position (1, 1)

Testing `move_worker()` method

move_worker() method moves the worker to a specified location within the world.

def test_world_move_worker():
    world = World([
        "#########",
        "#.$     #",
        "#  @    #",
        "#########",
    ])

    world.move_worker((2, 2))

    assert world.worker_pos == [(2, 2)]

Testing `push_box()` method

push_box() method moves a box from a one location to another within the world.

def test_world_push_box():
    world = World([
        "#########",
        "#.$     #",
        "# @     #",
        "#########",
    ])

    world.push_box((2, 1), (2, 2))

    assert world.box_pos == [(2, 2)]

Short Detour: Game Objects

Overview

Model: World
Controller: GameEngine
View: GameView

Review of World Object

get(pos)
- Query tile at the specified (x, y) position
- Returns a Tile, with boolean attributes: wall, worker, dock, box
move_worker(to_pos)
- Move worker to the specified position
push_box(from_pos, to_pos)
- Move a box from one position to another

GameEngine Object

Rules engines, decides what is possible and what is not within a world
move(dir, world)
- Validates and moves the worker in the specified direction, by updating the World object
- Validates and pushes the box, if present
is_game_over(world)
- Checks if the game is over
- All boxes are in docks

Mocking

What is Mocking?

Unit testing requirements:
- A unit needs to be tested, in isolation from the rest of the units
- Units being tested should not access external resources like network, filesystem, databases.
Mocking provides one way of satisfying these requirements
A Mock objects can be used to simulate real objects
In a unit test, Mock objects are deliberately used in place of real objects

Testing GameEngine

GameEngine manipulates the World object
Testing the GameEngine requires that we isolate it from World
World is simulated using a Mock object
Assertions on the Mock object will be used to verify the functionality of the GameEngine

Simulating Objects

Simulating an object with attributes

>>> from unittest.mock import Mock
>>> m = Mock(a=1, b=2)
>>> m.a
1
>>> m.b
2
>>> m.b = 3
>>> m.b
3

Simulating Functions

Simulating a function like object, returing a fixed value

>>> mfunc = Mock(return_value=10)
>>> mfunc()
10
>>> mfunc()
10

Simulating Functions (Contd.)

Simulating a function like object, returning a series of values

>>> mfunc = Mock(side_effect=["a", "b", "c"])
>>> mfunc()
'a'
>>> mfunc()
'b'
>>> mfunc()
'c'
>>> mfunc()
...
StopIteration

Simulating Functions (Contd.)

Simulating a function like object, raising an exception

>>> mfunc = Mock(side_effect=ValueError("test error"))
>>> mfunc()
...
ValueError: test error

On the Fly, Attributes

Object attribute assignment, creates attributes on the fly
```
>>> m = Mock()
>>> m.a = 10
>>> print(m.a)
10
```

On the Fly, Object Hierarchy

Object attribute access, auto-creates Mock objects

>>> m = Mock()
>>> m.b
<Mock ...>
>>> m = Mock()
>>> m.b.c = 30
>>> m.b
<Mock ...>
>>> m.b.c
30

Hierarchy of objects and attributes can be easily created

On the Fly, Functions

Turn any Mock object into a function like object by setting return_value attribute
```
>>> m = Mock()
>>> m.func.return_value = 20
>>> m.func()
20
```

Call Tracking

All calls made on Mock objects are captured and stored

>>> mfunc = Mock(return_value=None)
>>> mfunc(1, 2)
>>> mfunc(2, 3)
>>> mfunc.call_args_list
[calls(1, 2), calls(2, 3)]

Asserting on Calls

Helper methods to assert calls were made on the Mock object

assert_called_with(), checks if the last call was made with given arguments

>>> mfunc = Mock(return_value=None)
>>> mfunc(1, 2)
>>> mfunc(2, 3)
>>> mfunc.assert_called_with(2, 3)
>>> mfunc.assert_called_with(1, 2)
...
AssertionError

Asserting on Calls (Contd.)

assert_any_call(), checks if one of the calls was made with given arguments

>>> mfunc = Mock(return_value=None)
>>> mfunc(1, 2)
>>> mfunc(2, 3)
>>> mfunc.assert_any_call(1, 2)
>>> mfunc.assert_any_call(2, 3)

Review of Object Methods

class GameEngine:
    def move():
        ...

    def is_game_over():
        ...

class World:
    def get():
        ...

    def move_worker():
        ...

    def push_box():
        ...

Test Case for `is_game_over()`

Implementation of is_game_over()
Check if each box is in a dock position.

    def is_game_over(world):
        for box in world.box_pos:
            if box not in world.dock_pos:
                return False
        return True

Test Case for `is_game_over()`

is_game_over() checks if each box is in a dock position
Create a Mock() for the world, with attributes box_pos and dock_pos
Invoke is_game_over() with the mocked world

Test Case Implementation

from unittest.mock import Mock
from sokoban import GameEngine

def test_game_is_over_true():
    # Arrange
    engine = GameEngine()
    world = Mock()
    world.box_pos = [(1, 1), (1, 2)]
    world.dock_pos = [(1, 1), (1, 2)]
    # Act
    result = engine.is_game_over(world)
    # Assert
    assert result == True

Try Out

Update the test cases with the code from the previous slide.
Execute the test cases and verify that the tests pass.
Add a test case to verify that is_game_over() returns False
- When the boxes or not in the docks

Simplified `move()`

    def move(direction, world):
        x, y = world.worker_pos[0]

        if direction == Dir.UP:
            next_pos = (x, y - 1)
        elif direction == Dir.DN:
            next_pos = (x, y + 1)
        elif direction == Dir.RT:
            next_pos = (x + 1, y)
        else:  # if direction == Dir.LT:
            next_pos = (x - 1, y)

        next_tile = world.get(next_pos)
        if next_tile.wall:
            return

        world.move_worker(next_pos)

Testing `move()`

mock_world = Mock()
mock_world.worker_pos = [(3, 1)]
mock_world.get.return_value = Tile(wall=False,
                                   dock=False,
                                   worker=False,
                                   box=False)
engine = GameEngine()
engine.move(Dir.RT, mock_world)

mock_world.move_worker.assert_called_with((4, 1))

Testing `move()` into Wall

mock_world = Mock()
mock_world.worker_pos = [(3, 1)]
mock_world.get.return_value = Tile(wall=True,
                                   dock=False,
                                   worker=False,
                                   box=False)
engine = GameEngine()
engine.move(Dir.RT, mock_world)

assert mock_world.move_worker.called == False

Real `move()` Implementation

Simplified move, only moves the worker
The real move() needs to move the boxes as well
- If we move, into a box
- The box needs to be moved

Real `move()` Implementation

But only if the box can be moved
Box cannot be pushed into a wall

Real `move()` Implementation

But only if the box can be moved
Box cannot be pushed into another box

Testing `move()` Box

mock_world = Mock()
mock_world.worker_pos = [(2, 1)]
mock_world.get.side_effect = [
        Tile(wall=False, dock=False, worker=False, box=True),
        Tile(wall=False, dock=False, worker=False, box=False)
]
engine = GameEngine()
engine.move(Dir.RT, mock_world)

mock_world.push_box.assert_called_with((3, 1), (4, 1))
mock_world.move_worker.assert_called_with((3, 1))

Patching

Testing the View

View layer of Sokoban is written in Pygame
GameView object is where all access to Pygame is done

GameView initializes Pygame and sets the window caption

    def __init__(self):
        self._screen = None
        self._images = {}
        self._done = False
        pygame.init()
        pygame.display.set_caption("Sokoban!")

Testing the View (Contd.)

Unit testing best practices suggest that the pygame should be isolated, while testing GameView()
pygame is not passed into the function as argument
If it were passed in as argument, we could pass in a Mock object for purpose of testing
How do we replace pygame with a Mock Object?

Enter Monkey Patching

Monkey patch is a piece of code, which extends or modifies other code at runtime

>>> import sokoban
>>> sokoban.pygame
<module ...>
>>> mpygame = Mock()
>>> sokoban.pygame = mpygame

With this patch in place, all the accesses to pygame, within the view module will receive our Mock object

Verifying the Patch

Let’s verify that our Mock object is indeed, being used, within view

>>> view = sokoban.GameView()
>>> mpygame.init.called
True
>>> mpygame.display.set_caption.call_args_list
[call("Sokoban!")]

This proves that the set_caption() was infact called on the Mock object we patched into pygame

Managing the Patch

A patch should be reverted when no longer required
Forgetting to revert the patch will affect the functionality of the patched code
Patching for unit testing, should be restricted to the duration of execution of the test
- Install the patch
- Run the test
- Revert the patch
mock module provides convient helpers for this purpose

Using `patch()`

patch() should be provided the target to be patched
Target is specified as a string, of the form package.module.name
.start() patches the target with a mock object, and returns it
.stop() restores the original object

>>> patcher = patch("sokoban.pygame")
>>> mpygame = patcher.start()
>>> import sokoban
>>> sokoban.pygame
<MagicMock ...>
>>> patcher.stop()
>>> sokoban.pygame
<module ...>

Patching Custom Object

A custom object to be patched, can be provided as argument.

>>> mpygame = Mock()
>>> patcher = patch("sokoban.pygame", mpygame)

Test Case Implementation

from unittest.mock import patch
from sokoban import GameView

def test_set_window_title():
    patcher = patch("sokoban.pygame")
    mock_pygame = patcher.start()

    view = GameView()

    mock_pygame.display.set_caption.assert_called_with("Sokoban!")

    patcher.stop()

Testing Load Levels

def load_levels():
    """Returns levels loaded from a JSON file."""
    try:
        return json.load(xopen("levels.json"))
    except (OSError, IOError, ValueError):
        print("sokoban: loading levels failed!", file=sys.stderr)
        exit(1)

Testing Load Levels (Contd.)

from unittest.mock import Mock, patch
from io import StringIO

from sokoban import load_levels

def test_load_levels_success():
    mock_open = Mock()
    mock_open.return_value = StringIO("[]")
    patcher = patch("sokoban.xopen", mock_open)
    patcher.start()

    levels = load_levels()

    assert levels == []

    patcher.stop()

Try Out

Update the test cases with the code from the previous slide
Execute the test cases and verify that the tests pass
Add a test case to verify that SystemExit exception is raised when, a invalid JSON is passed

Try Out (Contd.)

def test_load_levels_invalid_json():
    mock_open = Mock()
    mock_open.return_value = StringIO("!!!")
    patcher = patch("sokoban.xopen", mock_open)
    patcher.start()

    pytest.raises(SystemExit, load_levels)

    patcher.stop()

Injecting Exceptions

def test_load_levels_os_error():
    mock_open = Mock()
    mock_open.side_effect = OSError
    patcher = patch("sokoban.xopen", mock_open)
    patcher.start()

    pytest.raises(SystemExit, load_levels)

    patcher.stop()

Resources

Documentation

pytest Documentation
- https://docs.pytest.org/en/latest/
mock / patch Documentation
- https://docs.python.org/3/library/unittest.mock.html
Pygame Documentation
- https://www.pygame.org/docs/

Unit Testing with Mocks

Introduction

About Me

Professional Work

Community Work

E2E vs Unit Testing

Drawbacks of E2E Testing

Drawback of E2E Testing (Contd.)

Unit Testing

Advantages of Unit Testing

Advantages of Unit Testing (Contd.)

Testing Pyramid

Writing Unit Tests

Sokoban

Sokoban

Unit Testing Frameworks

Sokoban Levels

Writing a Test Case

Try Out

Assert Raises Exception

Assert Raises Exception (Contd.)

Testing get() method

Try Out

Testing move_worker() method

Testing push_box() method

Short Detour: Game Objects

Overview

Review of World Object

GameEngine Object

Mocking

What is Mocking?

Testing GameEngine

Simulating Objects

Simulating Functions

Simulating Functions (Contd.)

Simulating Functions (Contd.)

On the Fly, Attributes

On the Fly, Object Hierarchy

On the Fly, Functions

Call Tracking

Asserting on Calls

Asserting on Calls (Contd.)

Review of Object Methods

Test Case for is_game_over()

Test Case for is_game_over()

Test Case Implementation

Try Out

Simplified move()

Testing move()

Testing move() into Wall

Real move() Implementation

Real move() Implementation

Real move() Implementation

Testing move() Box

Patching

Testing the View

Testing the View (Contd.)

Enter Monkey Patching

Verifying the Patch

Managing the Patch

Using patch()

Patching Custom Object

Test Case Implementation

Testing Load Levels

Testing Load Levels (Contd.)

Try Out

Try Out (Contd.)

Injecting Exceptions

Resources

Documentation

Questions

Testing `get()` method

Testing `move_worker()` method

Testing `push_box()` method

Test Case for `is_game_over()`

Test Case for `is_game_over()`

Simplified `move()`

Testing `move()`

Testing `move()` into Wall

Real `move()` Implementation

Real `move()` Implementation

Real `move()` Implementation

Testing `move()` Box

Using `patch()`