Nate Meyvis

15 notes on testing software

  1. If there is a One True Way to test software, we can’t know what it is until we know a One True Way to design software.
  2. But we don’t know such a One True Way.
  3. So, for now, it’s more productive to think about empirical, surface-level claims about testing. (That’s why this list is not an essay.)
  4. Most of the world’s production software is not tested.
  5. The difference between “tested at all” and “not tested at all” is much bigger than the difference between “tested really well” and “tested at all.”
  6. Whereas the main practical difficulty in software is getting people to write software well, the main practical difficulty in software testing is getting people to write tests at all.
  7. So beware of instituting any policy that could be willfully misinterpreted in a way that will cause fewer tests to be written.
  8. And beware of responding to having poor test coverage by thinking about how ideal tests look instead of the motivational causes of proper tests’ not getting written.
  9. Bug-report templates with too many required fields discourage people from reporting bugs and distract them from communicating what’s most relevant.
  10. So optimize your bug reporting to make capture easy.
  11. If you’re picking an aspect of discipline to develop, choose “actually goes and writes the test when something fails, even in development” over “writes beautiful and comprehensive tests for whatever thing I’m actually writing tests for.”
  12. Pytest is great.
  13. When there’s a performance problem with tests, most of the problem usually comes from just a few tests (or a single technique for setting up a fixture).
  14. So a little profiling can go a long way.
  15. And “I don’t want to slow down the test suite” is not a good reason not to write that extra test.

natemeyvis