Python Testing like a Pro

Coding test-driven Python package
with CI and Cloud

Keith Yang

2017 August

Python Testing like a Pro 🚀

Keith Yang

Formosat-5 launching

Coding test-driven Python package with CI and cloud

My talk in PyCon APAC 2017


alog v.s. OpenStack

keithis/alog <master>
% find . -name '*.py' | xargs cat | wc -l

Total 418 Python lines of code in alog

Versus Over 20,000,000 Lines
of Code of


The same part: BUG FREE!?

Another Launching Image

Keith Yang

Software Engineering Artisan

Caring from code, tesitng, to delivery on Web & Cloud

Keith's avatar

Founder of @Taipeipy since 2012
Chairpersion of PyCon APAC 2015
Cloud Senior Engineer in iCHEF


Expected Experience

For participating a serious project
Testing in OpenStack

Testing in an OpenStack project

& Categories


Some terms marked in the following slides for easier remember while going through slides.

Software Application Categories

CompilerProtocolsCryptographyEmbedded system
ScienceTrainingVotingWeb browser
RouterCloud architectureOSRobotic
Image, speech, or handwriting processing

by Human?

Titanic 2003

Software Testing Topic

Testing Methods

Box approach

Testing levels

Non-functional testing

Testing Process

No enough experience?
Not really a blocking problem!

Learn from
Open Source!

keitheis/alog and pypi version badges of alog

build | passing

Builds on Travis CI

Build results of alog on Travis CI

Concept Confirm Action Success

Use Email, IM notification

Build results of alog on Travis CI

Concept Alert on Broken

Use Email, SMS notification

Build results of alog on Travis CI

NO, it's me! Ask me to fix it!

codecov | 100%

Coverages Report on Codecov

Coverage report of alog on Codecov

Files coverage of alog on Codecov

Coverage of alog files on Codecov

It looks 100% good.

Lines coverage of alog module

Coverage lines of an alog module

Coverage 💯 %?

pypi | v0.9.13

Package ready on (new) PyPI

Alog package page on PyPI

How? Tox!


virtualenv management and test command line tool
# content of: tox.ini , put in same dir as
envlist = py26,py27
deps=pytest       # install pytest in the venvs
commands=pytest  # or 'nosetests' or ...


virtualenv management and test command line tool

tox.ini of alog

envlist =

basepython =
    py27: python2.7
    py36: python3.6
    pypy: pypy
    py2: python2.7
    py3: python3.4

Services Automation by GitHub

.travis.yml of alog

    - travis_retry pip install tox
    - travis_retry tox
    - bash <(curl -s

.travis.yml of alog

Tips of using Travis CI

Focus on Coding!

how about Jenkins?

Fun Facts of Jenkins

Swift builds on Jenkins

Customzied Jenkins page of Swift

Code Coverage and Test Trend
on Jenkins

Tips of Coverage on Jenkins

Avoid keeping failing or

Fire Jenkins

What it is all for

Keep failures before release

It's really hard.
After release:

Even AWS Serveless Can Fail

AWS Serveless Errors

My HeardBleed with OpenSSL

Heardbleed Page

A Little Less Surprise?

Yahoo Error Page

Forget it,
lets go back to CODE!

Don't forget it
please 🙇

Back to Python World Finally!

2 strategieswill be focus here

Strategy 1. Show Me Errors Early

Some tools

IPython (& Jupyter Notebook)

Step by Step Extension

Tip: Install IPython first

Code Linter on IDE/Editors/CI

Super helpful for human typo

Concept: WYSWYG

What you see is what you get

See it right right away!React Storybook


... based off of 'clang-format' ... In essence, the algorithm takes the code and reformats it to the best formatting that conforms to the style guide, ...
even if the original code didn't violate the style guide.


Tips of using yapf



makes it easy to write small tests, yet scales to support complex functional testing for applications and libraries.

Tips of using py.test


... lets you write tests which are parametrized by a source of examples, and then generates simple and comprehensible examples that make your tests fail.


        allow_infinity=False), min_size=1))
def test_mean(xs):
        assert min(xs) <= mean(xs) <= max(xs)

Falsifying example: test_mean(
    xs=[1.7976321856e+308, 6.102322755e+303]

Many to discover


Tips of mock

Assert it works before and mock it with success result

Strategy 2. Let Me Debug Easily

Tips of Sentry

Things Happen

Prepare most error handles during development

You can invest the minimal time cost for maximum gain by paying attention during development

Pay something before release
Just Read the Instructions

Some References


No tested backup plan?
equals no backup


Thank you!
Things Happen