Philip Guo (Phil Guo, Philip J. Guo, Philip Jia Guo, pgbovine)

A Spectrum of Research Software Quality

Research software varies widely in its quality or “realness.” Here's a spectrum from least to most “real.”

I've been building research software since my sophomore year of college when I joined my first lab; nearly all my publications are about research software or involve writing research software to do the underlying work. And in recent years, I've mostly been supervising students on building their own research software.

An age-old question that always comes up is, “How high-quality does my research software need to be?” Like many things in life, the answer is: it depends! Sometimes a hacked-together demo is all you need to demonstrate a proof-of-concept for a grant proposal or academic paper; other times your software must be good enough to deploy to users to measure real-world usage.

Anyhow, here's my quick attempt at a spectrum of research software quality, in ascending order:

  • barely works well enough for you to take some screenshots (the least real ... maybe it's even a mockup!)
  • works well enough for you to make a screencast video demo about it (videos are harder to fake than screenshots)
  • you can live demo it in front of a small crowd as long as you rehearse a ton beforehand, maintain full control over your computer, and exercise only a few “happy paths” in the code
  • you can live demo and also react to some audience questions to try out different code paths beside the expected happy path
  • your software can actually be installed and run on a computer that's not your own (anyone who's tried to do this can tell you it's easier said than done!)
  • you can allow someone else to use your software under your close supervision, like in a live demo session or in a highly-constrained limited user study
  • you can allow someone else to use it mostly on their own in a regular user study lasting ~1 hour, under your light supervision and with the understanding that it's a prototype (most research projects, including my own, never go beyond this point)
  • someone needs your help to install and configure the software, but it's reasonably usable by others without your supervision (e.g., for a week- or month-long longitudinal deployment study), as long as they understand that it's a prototype
  • people you personally know can install, configure, and use it decently well on their own without your direct supervision
  • total strangers can install, configure, and use it on their own; a subtly-different related point: strangers actually want to install and use it without you advertising to them (at this point, there's demand beyond people you personally know)
  • large numbers of strangers find and use it on their own; at this scale, you'll need to deal with inevitable edge cases and scalability concerns from having a diverse userbase (e.g., CDE)
  • same as above, except it keeps working and being maintained N years into the future, where N is non-trivial like 3 years, 5 years, or even 10+ years (e.g., Python Tutor)
  • ... also grows a sizable community of users who produce original creative content with it (e.g., Scratch, d3 and Vega)
  • ... also grows a sizable community of open-source developers who build tool extensions upon it (e.g., Jupyter)
  • becomes a production-grade platform that others use to create major research and product innovations (e.g., LLVM)
  • ... same as above, but opens up an entirely new industry with worldwide impact (e.g., Unix at Bell Labs, WWW at CERN)

Appendix: related links

Thanks to Jonathan Edwards and John Regehr for feedback.

Subscribe to email newsletter
Donate to help pay for web hosting costs

Created: 2020-02-17
Last modified: 2020-02-17
Related pages tagged as software:
Related pages tagged as research: