What is up with these tests?

there are plenty I'm just confused

By Faith Okamoto

I’ve done critiques of tests before, but here I’ll look at a much more comprehensive suite. Or, at least a bigger suite. And I’ll still complain some, because that’s the kind of person I am. Persnickety.

This month’s paper: Dayan D and Gersein M. pyGENIE: A Python library for programmatic access and mutational analysis of AACR Project GENIE. Bioinformatics 2026. doi: 10.1093/bioinformatics/btag844

Original code

The tests for this paper are on GitHub.

Critique

Standard disclaimer: issues with published code are not necessarily anyone’s fault, and often are due to nothing more nefarious than time constraints.

Test names should be accurate

def test_cache_starts_empty(genie):
    # note: cache may already have data from previous tests in session
    # just check it's a dict
    assert isinstance(genie._cache, dict)

If you read the name of this test, you might think it’s about testing whether, y’know, the cache starts empty. But instead, it’s testing… whether the cache is a dictionary type? And the comment says that the cache might not actually be empty. So the test ain’t doing the thing it says. I’m not sure if what it’s doing is even useful. (Maybe a better test name would tell me the point?)

My best guess is that the test originally was about whether the cache was empty. It then failed weirdly, due to race conditions (tricky goblins, those) where “data from previous tests in session” caused issues. To make the test pass consistently, it was softened into what is here today.

But now if this version fails, that test case name is of negative utility.

Tests should be written such that if they fail, it is obvious what failed and why that’s a problem. If you name the test in a confusing manner, that will be, you guessed it, needlessly confusing. What should’ve been done is either finding a way to make a test matching its original name pass, or renaming the test for its new purpose. Not this chimera of name and function.

What is this one doing?

def test_summary_no_unknown_cancer_types(genie):
    samples = genie._load("data_clinical_sample.txt")
    known = samples[samples["CANCER_TYPE"] != "UNKNOWN"]
    s = genie.summary()
    assert s["n_cancer_types"] == known["CANCER_TYPE"].nunique()

It took me a while to figure out that this is testing for the number of cancer types in .summary() to be equal to the number of cancer types in the raw data but sans the "UNKNOWN" types. That could’ve been explained with a comment. For example, # n_cancer_types should not include UNKNOWN.

Documenting what the expected behavior is, and crucially why it’s expected, does legions to help debugging down the line when things start acting up.

Organization is nice

Hey, ending with praise again. I do note that the sections of tests have comments such as # query fusions to head each few test cases, which is nice for understanding which ones should be thought of together, and for finding the ones which you might need.


Tests are, more than anything else, for the future. Specifically, the future in which things break and you want to narrow down why. So it behooves you to make them easy to navigate.

If there’s a recent paper you’d like me to look through, shoot me an email. Address in my CV.

Share: X (Twitter) Facebook LinkedIn