About Me

Professional Work

  • Works for Zilogic Systems
  • Technologist and Trainer
  • Specializes in
    • Linux
    • Embedded Systems
    • Python

Community Work

  • Co-organizer of Chennaipy
  • Previously
    • Co-organizer of PyCon India 2019
    • Co-organizer of PySangamam 2018
  • Twitter: @bravegnu
  • Follow on LinkedIn

E2E vs Unit Testing

  • People generally tend to test their software as a whole
    • Run the program
    • Input data through the UI
    • Check program’s response
  • Tests done this way automated or manual is called End-to-End Testing
  • E2E Testing is required but not sufficient

Drawbacks of E2E Testing

  • It is terribly slow
    • Accessing database, networks, files, etc. is slow
    • Implication: Will not be run often
  • Running test cases has dependencies
    • Depends on the availability of test server, input emulators, etc.
    • Implication: Only people with access to resources can execute it
  • Hard to determine root cause of failing test
    • Implication: For a large code base, takes time and effort to zero in on the faulty code

Drawback of E2E Testing (Contd.)

  • Cannot drive program through all possible code path, low code coverage
    • Some error scenarios is almost impossible to create
    • Implication: Low code coverage, and typos lurking in untested code paths
  • E2E tests can be flaky
    • Intermittent network failures, power outages, device failures, can cause tests to fail
    • Implication: Reduces trust on the tests

Unit Testing

  • Unit Tests, verify the public interface of a module / class, in isolation
  • Dependencies are mocked or faked
  • Written and executed by developers

Advantages of Unit Testing

  • Unit tests are extremely fast
    • They do not use databases, network, file system, other applications, console, etc.
  • Unit tests provide early feedback
    • No test setup or resources are required to run the tests
    • Developers can run them every time they make a change and before committing the code

Advantages of Unit Testing (Contd.)

  • Unit tests makes it easier to refactor
    • Any regression due to refactoring, will be caught by unit tests
    • Code can be refactored / improved without the fear of regression

Testing Pyramid

  • Unit tests verify the functionality of each unit
  • Integration tests verify that the units work together at the sub-system level
  • E2E tests verify that the product works as a whole
  • A good test strategy will have, a mix of each approach, with most of the testing done at the unit level

Writing Unit Tests


  • A puzzle game, where the player pushes boxes to docks
  • Restrictions
    • Boxes cannot be pulled
    • Boxes cannot be pushed into walls or other boxes
  • Under 400 lines of code, written using Pygame
  • Will serve as base for practicing writing unit test cases


  • First, let’s try playing around with it
  • You need to have pygame installed
  • Download
  • Extract and run the program
    $ unzip
    $ cd game
    $ python3
  • Arrow keys to move, S to skip to next level

Unit Testing Frameworks

  • Unit testing frameworks provide the necessary infrastructure to
    • Write test cases
    • Run the test cases
    • Generate test reports
  • Available unit testing frameworks
    • unittest, part of standard library
    • py.test, popular third party framework
    • nose2, extends unittest, making nicer and easier

Sokoban Levels

from sokoban import World

level = [
    "#.$ @   #",

world = World(level)
print(world.nrows)      # 3
print(world.ncols)      # 9
print(world.worker_pos) # [(4, 1)]
print(world.box_pos)    # [(2, 1)]
print(world.dock_pos)   # [(1, 1)]
  • A level is represented as a list of strings.
  • Corresponding graphical representation
  • The World class parses the given level, and sets up its attributes

Writing a Test Case

from sokoban import World

    "#.$ @   #",

def test_rect_world_dimensions():
    world = World(SIMPLE_LEVEL)

    assert world.nrows == 3
    assert world.ncols == 9

Try Out

  • Create a file in the same directory as
  • Add the test case from the previous slide
  • Try running the test case using
    $ py.test

Assert Raises Exception

  • A ValueError is thrown
    • When an invalid character is present
    • Or when the worker is missing
    • Or when the number of boxes and docks do not match
    • Or when the no. of boxes is zero

Assert Raises Exception (Contd.)

  • pytest.raises can be used to verify that an exception was raised
    pytest.raises(exception, func, *args)
  • Verify that ValueError is raised when an invalid character is present
    def test_invalid_character_error():
        level = [
            "#.$     #",
        pytest.raises(ValueError, World, level)

Testing get() method

  • get() method returns a Tile, at the specified location.
  • The Tile object indicates the properties of the tile in world.
    def test_world_get_wall():
        world = World([
            "#.$  @  #",
        expected = Tile(wall=True, worker=False, dock=False, box=False)
        got = world.get((0, 0))
        assert got == expected

Try Out

  • Add a test case to check if there is a dock in position (1, 1)

Testing move_worker() method

  • move_worker() method moves the worker to a specified location within the world.
    def test_world_move_worker():
        world = World([
            "#.$     #",
            "#  @    #",
        world.move_worker((2, 2))
        assert world.worker_pos == [(2, 2)]

Testing push_box() method

  • push_box() method moves a box from a one location to another within the world.
    def test_world_push_box():
        world = World([
            "#.$     #",
            "# @     #",
        world.push_box((2, 1), (2, 2))
        assert world.box_pos == [(2, 2)]

Short Detour: Game Objects


  • Model: World
  • Controller: GameEngine
  • View: GameView

Review of World Object

  • get(pos)
    • Query tile at the specified (x, y) position
    • Returns a Tile, with boolean attributes: wall, worker, dock, box
  • move_worker(to_pos)
    • Move worker to the specified position
  • push_box(from_pos, to_pos)
    • Move a box from one position to another

GameEngine Object

  • Rules engines, decides what is possible and what is not within a world
  • move(dir, world)
    • Validates and moves the worker in the specified direction, by updating the World object
    • Validates and pushes the box, if present
  • is_game_over(world)
    • Checks if the game is over
    • All boxes are in docks


What is Mocking?

  • Unit testing requirements:
    • A unit needs to be tested, in isolation from the rest of the units
    • Units being tested should not access external resources like network, filesystem, databases.
  • Mocking provides one way of satisfying these requirements
  • A Mock objects can be used to simulate real objects
  • In a unit test, Mock objects are deliberately used in place of real objects

Testing GameEngine

  • GameEngine manipulates the World object
  • Testing the GameEngine requires that we isolate it from World
  • World is simulated using a Mock object
  • Assertions on the Mock object will be used to verify the functionality of the GameEngine

Simulating Objects

  • Simulating an object with attributes
    >>> from unittest.mock import Mock
    >>> m = Mock(a=1, b=2)
    >>> m.a
    >>> m.b
    >>> m.b = 3
    >>> m.b

Simulating Functions

  • Simulating a function like object, returing a fixed value
    >>> mfunc = Mock(return_value=10)
    >>> mfunc()
    >>> mfunc()

Simulating Functions (Contd.)

  • Simulating a function like object, returning a series of values
    >>> mfunc = Mock(side_effect=["a", "b", "c"])
    >>> mfunc()
    >>> mfunc()
    >>> mfunc()
    >>> mfunc()

Simulating Functions (Contd.)

  • Simulating a function like object, raising an exception
    >>> mfunc = Mock(side_effect=ValueError("test error"))
    >>> mfunc()
    ValueError: test error

On the Fly, Attributes

  • Object attribute assignment, creates attributes on the fly
    >>> m = Mock()
    >>> m.a = 10
    >>> print(m.a)

On the Fly, Object Hierarchy

  • Object attribute access, auto-creates Mock objects
    >>> m = Mock()
    >>> m.b
    <Mock ...>
    >>> m = Mock()
    >>> m.b.c = 30
    >>> m.b
    <Mock ...>
    >>> m.b.c
  • Hierarchy of objects and attributes can be easily created

On the Fly, Functions

  • Turn any Mock object into a function like object by setting return_value attribute
    >>> m = Mock()
    >>> m.func.return_value = 20
    >>> m.func()

Call Tracking

  • All calls made on Mock objects are captured and stored
    >>> mfunc = Mock(return_value=None)
    >>> mfunc(1, 2)
    >>> mfunc(2, 3)
    >>> mfunc.call_args_list
    [calls(1, 2), calls(2, 3)]

Asserting on Calls

  • Helper methods to assert calls were made on the Mock object
  • assert_called_with(), checks if the last call was made with given arguments
    >>> mfunc = Mock(return_value=None)
    >>> mfunc(1, 2)
    >>> mfunc(2, 3)
    >>> mfunc.assert_called_with(2, 3)
    >>> mfunc.assert_called_with(1, 2)

Asserting on Calls (Contd.)

  • assert_any_call(), checks if one of the calls was made with given arguments
    >>> mfunc = Mock(return_value=None)
    >>> mfunc(1, 2)
    >>> mfunc(2, 3)
    >>> mfunc.assert_any_call(1, 2)
    >>> mfunc.assert_any_call(2, 3)

Review of Object Methods

class GameEngine:
    def move():

    def is_game_over():
class World:
    def get():

    def move_worker():

    def push_box():

Test Case for is_game_over()

  • Implementation of is_game_over()
  • Check if each box is in a dock position.
    def is_game_over(world):
        for box in world.box_pos:
            if box not in world.dock_pos:
                return False
        return True

Test Case for is_game_over()

  • is_game_over() checks if each box is in a dock position
  • Create a Mock() for the world, with attributes box_pos and dock_pos
  • Invoke is_game_over() with the mocked world

Test Case Implementation

from unittest.mock import Mock
from sokoban import GameEngine

def test_game_is_over_true():
    # Arrange
    engine = GameEngine()
    world = Mock()
    world.box_pos = [(1, 1), (1, 2)]
    world.dock_pos = [(1, 1), (1, 2)]
    # Act
    result = engine.is_game_over(world)
    # Assert
    assert result == True

Try Out

  • Update the test cases with the code from the previous slide.
  • Execute the test cases and verify that the tests pass.
  • Add a test case to verify that is_game_over() returns False
    • When the boxes or not in the docks

Simplified move()

    def move(direction, world):
        x, y = world.worker_pos[0]

        if direction == Dir.UP:
            next_pos = (x, y - 1)
        elif direction == Dir.DN:
            next_pos = (x, y + 1)
        elif direction == Dir.RT:
            next_pos = (x + 1, y)
        else:  # if direction == Dir.LT:
            next_pos = (x - 1, y)

        next_tile = world.get(next_pos)
        if next_tile.wall:


Testing move()

mock_world = Mock()
mock_world.worker_pos = [(3, 1)]
mock_world.get.return_value = Tile(wall=False,
engine = GameEngine()
engine.move(Dir.RT, mock_world)

mock_world.move_worker.assert_called_with((4, 1))

Testing move() into Wall

mock_world = Mock()
mock_world.worker_pos = [(3, 1)]
mock_world.get.return_value = Tile(wall=True,
engine = GameEngine()
engine.move(Dir.RT, mock_world)

assert mock_world.move_worker.called == False

Real move() Implementation

  • Simplified move, only moves the worker
  • The real move() needs to move the boxes as well
    • If we move, into a box
    • The box needs to be moved

Real move() Implementation

  • But only if the box can be moved
  • Box cannot be pushed into a wall

Real move() Implementation

  • But only if the box can be moved
  • Box cannot be pushed into another box

Testing move() Box

mock_world = Mock()
mock_world.worker_pos = [(2, 1)]
mock_world.get.side_effect = [
        Tile(wall=False, dock=False, worker=False, box=True),
        Tile(wall=False, dock=False, worker=False, box=False)
engine = GameEngine()
engine.move(Dir.RT, mock_world)

mock_world.push_box.assert_called_with((3, 1), (4, 1))
mock_world.move_worker.assert_called_with((3, 1))


Testing the View

  • View layer of Sokoban is written in Pygame
  • GameView object is where all access to Pygame is done
  • GameView initializes Pygame and sets the window caption
        def __init__(self):
            self._screen = None
            self._images = {}
            self._done = False

Testing the View (Contd.)

  • Unit testing best practices suggest that the pygame should be isolated, while testing GameView()
  • pygame is not passed into the function as argument
  • If it were passed in as argument, we could pass in a Mock object for purpose of testing
  • How do we replace pygame with a Mock Object?

Enter Monkey Patching

  • Monkey patch is a piece of code, which extends or modifies other code at runtime
    >>> import sokoban
    >>> sokoban.pygame
    <module ...>
    >>> mpygame = Mock()
    >>> sokoban.pygame = mpygame
  • With this patch in place, all the accesses to pygame, within the view module will receive our Mock object

Verifying the Patch

  • Let’s verify that our Mock object is indeed, being used, within view
    >>> view = sokoban.GameView()
    >>> mpygame.init.called
    >>> mpygame.display.set_caption.call_args_list
  • This proves that the set_caption() was infact called on the Mock object we patched into pygame

Managing the Patch

  • A patch should be reverted when no longer required
  • Forgetting to revert the patch will affect the functionality of the patched code
  • Patching for unit testing, should be restricted to the duration of execution of the test
    • Install the patch
    • Run the test
    • Revert the patch
  • mock module provides convient helpers for this purpose

Using patch()

  • patch() should be provided the target to be patched
  • Target is specified as a string, of the form
  • .start() patches the target with a mock object, and returns it
  • .stop() restores the original object
>>> patcher = patch("sokoban.pygame")
>>> mpygame = patcher.start()
>>> import sokoban
>>> sokoban.pygame
<MagicMock ...>
>>> patcher.stop()
>>> sokoban.pygame
<module ...>

Patching Custom Object

  • A custom object to be patched, can be provided as argument.
    >>> mpygame = Mock()
    >>> patcher = patch("sokoban.pygame", mpygame)

Test Case Implementation

from unittest.mock import patch
from sokoban import GameView

def test_set_window_title():
    patcher = patch("sokoban.pygame")
    mock_pygame = patcher.start()

    view = GameView()



Testing Load Levels

def load_levels():
    """Returns levels loaded from a JSON file."""
        return json.load(xopen("levels.json"))
    except (OSError, IOError, ValueError):
        print("sokoban: loading levels failed!", file=sys.stderr)

Testing Load Levels (Contd.)

from unittest.mock import Mock, patch
from io import StringIO

from sokoban import load_levels

def test_load_levels_success():
    mock_open = Mock()
    mock_open.return_value = StringIO("[]")
    patcher = patch("sokoban.xopen", mock_open)

    levels = load_levels()

    assert levels == []


Try Out

  • Update the test cases with the code from the previous slide
  • Execute the test cases and verify that the tests pass
  • Add a test case to verify that SystemExit exception is raised when, a invalid JSON is passed

Try Out (Contd.)

def test_load_levels_invalid_json():
    mock_open = Mock()
    mock_open.return_value = StringIO("!!!")
    patcher = patch("sokoban.xopen", mock_open)

    pytest.raises(SystemExit, load_levels)


Injecting Exceptions

def test_load_levels_os_error():
    mock_open = Mock()
    mock_open.side_effect = OSError
    patcher = patch("sokoban.xopen", mock_open)

    pytest.raises(SystemExit, load_levels)