Modern Programming for Data Analytics - Testing and Continuous Integration
Why do we do testing?
- Scientific software development can become quite complex, with multiple moving parts where one error somewhere can have global effect on the outcomes.
- Multiple people working on the same code can cause errors is a team member introduces a bug
- Bugs will happen, they are enivitable, the key is being able to find them quickly and how they effect the outcomnes
Why do we do testing?
- Building testing into your software develop process can make it easier to find bugs as they develop and prevent them from making it to users
- Testing can give you the confidence that the answers your code is getting are correct
- Testing can save far more time in the long-run than having to chase down difficult bugs
Types of Testing
- Unit Testing - Testing individual components of the software
- Functional Testing - Testing individual features of the software
- End-to-End Testing - Testing the use through all the functionality of the software (including users interacting with the software)
- Integration Testing - Testing how the software interacts with other external systems and services.
- Acceptance Testing - Formal requirements for declaring a process completed
- Performance Testing - Testing the time to solution for the software
Types of Testing
- Manual Testing - Tests that requires human interaction to complete
- Automatic Testing - Using some form of infrastructure to run testing / verification
- Interactive Testing - Bug fixing and developing
Unit Testing
- As codes get highly complex, it becomes important to know that individual components give expected results.
- Unit testing involves creating a test for each of these individual components
- Unit testing is not a garuntee that the components work well together, but it does ensure that if you expect something to give a result, you get that.
- Basically, it involves giving a component a set of known inputs that have expected outputs that test all of the expected functionality of the component.
- In Python, unit testing can be done over an entire module, an individual function or a class.
Unit Testing
- Mocking - creating a "fake" version of an external or internal service that can stand in for the real one
- For example - if function needs a class or object, a "fake" complete object can be created as a stand-in for testing
- Stubbing - stubbing only replicates the behavior of parts of an object, not using the actual object
- For example - if I have a member of a class that requires a string, but the string is intended to come from another object in the program, I could test the base functionality by just creating a string
Basic Steps of a Unit Test
- Setup - create the data needed for the test
- Exercise - execute the functionality on the data
- Verify - perform a test (usually an assertion) to verify the answer is expected
- Teardown - Clean up all data so that the test remains independent
High-level example
Let's say I have a class foo that relies on processing a collection units of class bar
To create a unit test for function foo, the steps would be conceptually:
- Generate a mock set of bar instances
- Run the members of foo using the mock instances of bar
- Pass through a series of assertions that the answer is the expected answer
- Destroy the mock set of bar instances
def add_x_and_k(x, k):
if isinstance(x, str):
return x + str(k)
elif isinstance(x, int):
return x + int(k)
elif isinstance(x, float):
return x + float(k)
else:
return "this would be an error"
if __name__ == "__main__":
### Unit testing for module
pass_flag = True
if add_x_and_k("x", "0.001") != "x0.001":
print("string failed")
pass_flag = False
if add_x_and_k(1, 1.022) != 2:
print("int failed")
pass_flag = False
if add_x_and_k(1.01, 2.01) != 1.01 + 2.01:
print("float failed")
pass_flag = False
if add_x_and_k(True, 1.020) != "this would be an error":
print("error failed")
pass_flag = False
if pass_flag:
print("All Tests Passed")
else:
print("Some Tests Failed")
unittest - a simple unittesting library for Python
import unittest
class T:
def __init__(self):
self.x = 1.000
def add_x_and_k(x, k):
if isinstance(x, str):
return x + str(k)
elif isinstance(x, int):
return x + int(k)
elif isinstance(x, float):
return x + float(k)
else:
return "this would be an error"
class TestAddXAndK(unittest.TestCase):
def test_add(self):
self.assertEqual(add_x_and_k("x", "0.001"), "x0.001")
self.assertEqual(add_x_and_k(1, 1.022), 2)
self.assertAlmostEqual(add_x_and_k(1.01, 2.01), 3.02)
self.assertEqual(add_x_and_k(True, 0.01), "this would be an error")
y = T()
self.assertEqual(add_x_and_k(y, 0.001), "this would be an error")
if __name__ == "__main__":
unittest.main()
$ python -m unittest unit_test.py
======================================================================
FAIL: test_add (unit_test.TestAddXAndK)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/mnt/c/Users/shawn/Programs/ms-das/TestingAndContinuousIntegration/static/scripts/unit_test.py", line 25, in test_add
self.assertEqual(add_x_and_k(True, 0.01), "this would be an error")
AssertionError: 1 != 'this would be an error'
----------------------------------------------------------------------
Ran 1 test in 0.004s
FAILED (failures=1)
For more information and a tutorial, see https://www.datacamp.com/community/tutorials/unit-testing-python
Pytest - a framework for unit testing in Python
- provides a complete unit testing framework for complex programs and resources
- allows the definition of complex inputs for unit testing
- can be easily integrated with most software frameworks
Simple Pytest example
from unit_test import *
def test_add_x_and_k_string():
"""
Verifies string output
"""
assert add_x_and_k("x", "0.001") == "x0.001"
def test_add_x_and_k_int():
"""
Verifies integer output
"""
assert add_x_and_k(1, 1.022) == 2
def test_add_x_and_k_float():
"""
Verifies float output
"""
assert round(add_x_and_k(1.01, 2.01), 2) == 3.02
def test_add_x_and_k_bool():
"""
Verifies that this should fail with a boolean
"""
assert add_x_and_k(True, 0.001) == "this would be an error"
def test_add_x_and_k_error():
"""
Verifies that this should fail with an error
"""
t = T()
assert add_x_and_k(t, 0.001) == "this would be an error"
Simple Pytest example
=================================================== test session starts ===================================================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /mnt/c/Users/shawn/Programs/ms-das/unit_testing_examples
collected 5 items
test_unit_test.py ...F. [100%]
======================================================== FAILURES =========================================================
__________________________________________________ test_add_x_and_k_bool __________________________________________________
def test_add_x_and_k_bool():
"""
Verifies that this should fail with a boolean
"""
> assert add_x_and_k(True, 0.001) == "this would be an error"
E AssertionError: assert 1 == 'this would be an error'
E + where 1 = add_x_and_k(True, 0.001)
test_unit_test.py:33: AssertionError
================================================= short test summary info =================================================
FAILED test_unit_test.py::test_add_x_and_k_bool - AssertionError: assert 1 == 'this would be an error'
=============================================== 1 failed, 4 passed in 1.50s ===============================================
pytest - Fixtures
- Fixtures allow you to define reusable mock and stub objects for use in testing
- Can define objects, services, inputs, etc..
- Remove to the need to create mock objects for every test
- Allow one to define the scope of the object, and the setup and teardown procedures
pytest - Fixtures
A word about directory structure
.
├── LICENSE
├── README.md
├── SCF.py
├── main.py
├── mol.py
├── requirements.txt
└── tests
├── SCF
│ ├── __init__.py
│ └── test_SCF.py
├── __init__.py
└── conftest.py
pytest - fixtures
conftest.py is where we define fixtures
"""
The PyTest Configuration code
"""
import pytest
from mol import mol
@pytest.fixture
def mol_h2o():
"""
Fixture that creates a specific water molecule
"""
atom = "8 0.000000000000 -0.143225816552 0.000000000000;" \
+ "1 1.638036840407 1.136548822547 -0.000000000000;" \
+ "1 -1.638036840407 1.136548822547 -0.000000000000"
return mol(atom, 7, 5, 7)
pytest - Fixtures
A more complext example https://github.com/shots47s/conp-portal/blob/master/tests/conftest.py
pyttest - testing a function
import pytest
import SCF
def test_calc_nuclear_repulsion_energy(mol_h2o):
"""
Tests that the nuclear repulsion energy is correct
"""
assert SCF.calc_nuclear_repulsion_energy(mol_h2o) == 8.00236706181077,\
"Nuclear Repulsion Energy Test (H2O) Failed"
def test_calc_initial_density(mol_h2o):
"""
Tests that the initial density returns a zero matrix
and tests dimensions
"""
Duv = SCF.calc_initial_density(mol_h2o)
assert Duv.sum() == 0.0
assert Duv.shape == (mol_h2o.nao, mol_h2o.nao)
pytest - Running the tests
$pytest
================================== test session starts ==================================
platform linux -- Python 3.7.6, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /mnt/c/Users/shawn/Programs/ms-das/HF_SCF_Assignment
plugins: cov-2.12.1
collected 2 items
tests/SCF/test_SCF.py F. [100%]
======================================= FAILURES ========================================
________________________________ test_calc_nuclear_repulsion_energy _____________________
mol_h2o = <mol.mol object at 0x7eff07b6d250>
def test_calc_nuclear_repulsion_energy(mol_h2o):
assert True
> assert SCF.calc_nuclear_repulsion_energy(mol_h2o) == 8.00236706181077,\
"Nuclear Repulsion Energy Test (H2O) Failed"
E AssertionError: Nuclear Repulsion Energy Test (H2O) Failed
E assert 0 == 8.00236706181077
E + where 0 = <function calc_nuclear_repulsion_energy at 0x7eff0a32ab00> (<mol.mol object at 0x7eff07b6d250>)
E + where <function calc_nuclear_repulsion_energy at 0x7eff0a32ab00>
= SCF.calc_nuclear_repulsion_energy
tests/SCF/test_SCF.py:7: AssertionError
--------------------------------- Captured stdout setup --------------------------------
================================ short test summary info ===============================
FAILED tests/SCF/test_SCF.py::test_calc_nuclear_repulsion_energy - AssertionError:
Nuclear Repulsion Energy Test...
============================== 1 failed, 1 passed in 2.00s =============================
Continuous Integration, Deployment and Testing
- CI/CD is essentially the process of automatically deploying or integrating software based on an event (e.g. pull requests, version release, manual triggering,...)
- This process can also be used for testing to ensure that changes to a platform do not break the whole system and do not pollute the source code.
- There are many systems for doing this:
- Jenkins - the leading open source CI platform, it is a completely self-contained Java-based solution
- CircleCI - cloud based CI platform, strong integration with GitHub and Bitbucket
- TravisCI virtuall the same as CicleCI in features, cloud based solution with strong intergration to GitHub and Bitbucket
Using TravisCI with Pytest
adding a .travic.yml to your repository
language: python
python:
- 3.7
install:
- pip install -r requirements.txt
- pip install coveralls
script:
- pycodestyle --max-line-length=99 -r *.py
- pytest --cov ../HF_SCF_Assignment
after_success:
- coveralls