<date/time>
<number>
Testing strategy woes
Addressing shortcomings in testing strategyStefano Borini
Problems: developer side
Developers don't run tests before committing. Why?
The full testsuite is slow.The GUI testsuite is annoying.During the full run, you can't keep working, or you will potentially ruin the run.Code layout and interdependencies make it hard, if not impossible,to limit the test run only to a subset of the suite, without worrying aboutpossible consequences in other modules, even when there should not be.Forget to run it. Forget to check if that tiny change we did to a common module had an overall effect. Trusting developers too much. Too little focus.With two development teams, they disrupt each other much more.
Problems: unittest organization
Intermixing of functional tests in the unit tests.Functional tests happen at unspecified times. Different machines may run functests on different versions.Functional tests run only if unit tests pass, leaving holes of knowledgeabout the functionality of our code.Some GUI tests are overly sensitive to small changes in positioning.The lower the layer, the more it is fundamental. A failure there prevents to test the full application.
Why our test run fails?
We forget a file
We forget an import/make a typo
Weird changes in external-libs
Actual broken test
Platform dependencies
Random segfaults
Problem
Mitigation
Use git ignores
As-you-type linting with pylint andAn editor that supports it
Generally self-solved within 24 hours
Local testing
Hard to predict and mitigate
Simplify runtime, simpler C++,C++ debug checks, Valgrind runs
Fixed bug
Need ability for a machine to switchBranch. Only developer in charge getsReports until back on master.
Proposed changes: development
Toyota “Stop the production line” approach (eg. At Friday we always focus on bringing back to green if we aren't)Have tests for the code we are working on constantly running in a separate window (can be a problem with GUI stuff), e.g. by “while true”Should mock more (_MUCH_ more), especially calculation. Our design does not help, but we should not trigger evaluations. That's what functional testing is for. Unit testing checks the interfaces and overall correctness. Can we mock the GUI as well?GUI should be tested less, or in a separate run.Parallelize tests via nose or testoobRequire strict test independence (e.g. no shared files)Have a local test run during lunch break, push when green.Start the sprint from a green. Branch, feature merge back into a green state. Repeat.Rebase only to another green.Force git precommit-hooks ?Generational Testing? Old, generally passing tests are run less often.
Proposed changes: Test machines
Run tests only when actual changes appear in master.Run functional tests only once, during the night. Not constantly.Run tests regardless if the previous tests failed. (e.g. do run functestseven if some unittests failed. Also for C++/python)Improve the email report. Too much inbox noise.Have a public screen hanging on the wall with test dashboard.Improve the web report. The web report should show a git tree with passed/failed.Easier to pinpoint changes if a red occurs.