Nate Meyvis


A note on unit testing (or, writing to your tools)

Let’s say you like (not just testing but) unit testing. I do, and I think you should, but that’s a separate argument–for now. And let’s say you see a test like this:

def test_only_foo_is_green(self):
self.assertEqual(, self.filter_by_color(Color.GREEN))

This test is really two tests: you’re testing both that all the foos are green and that all the green things are foos. So, if you’re really committed to unit testing, this ought to be:

def test_foo_is_green(self):
# The best way to test this will vary a lot by context
self.assertTrue(all([foo_thing.color == Color.GREEN for foo_thing in])

def test_nonfoos_are_not_green(self):
# The best way to test this will vary a lot by context
self.assertTrue(all([green_thing.is_foo() for green_thing in self.filter_by_color(Color.GREEN)]))

More justifications for this refactoring, from roughly most to least trivial:

  1. The initial test name (ex hypothesi) used “only” to mean “exactly,” and that’s a perfectly idiomatic English sense of “only,” but it’s imprecise when labelling a test, and the rewrite avoids this temptation.
  2. There’s a temptation to think of “the foos are exactly the green things” as the right scope for a unit test because you can put it in one tidy simple English sentence, but the right level of simplicity for the purposes of individuating unit tests can be finer-grained than that.
  3. Lots of testing frameworks offer something like assertItemsEqual or assertContainsExactly; these have their place, but the fact that you can test for something in one assertion is no more evidence that it individuates a proper unit test than the fact that you can express something in a single simple English sentence.
  4. In practice there might be a bit of overhead in splitting out these tests, usually because some setup work (that doesn’t naturally fit in a suite-level setup function) must be copied and pasted. But as IDEs get better and better, this copy/pasting work, and all the future navigation and reading work this imposes on maintainers, gets cheaper and cheaper.

This last is crucial. When we learn tools, we mostly hear and think about how to do our current tasks better. But as our tools improve, what we do doesn’t just get more efficient, it changes. This is very obvious in other parts of life (of course you buy different groceries and cook different meals if your kitchen equipment changes), and obvious in computer systems design, but often overlooked in (the micro level of) programming. For all the talk about making one’s architectural choices robust in light of unspecifiable future improvements in technology, we don’t talk enough about making micro-level coding choices robust in light of such improvements. A more granular, repetitive approach to unit testing can’t be the only way in which a programmer can put oneself on the right side of history.