What part of your API should you test for usability?
Written by Gopal V.
Published on Wed Aug 11 2021
So you’ve decided you want to run an usability test to discover ways in which you could improve the developer experience with your APIs. Congratulations! The next step is to decide what exactly to test. You might be thinking of simply taking one of your public samples, removing some of the code, and asking the tester to fill in the parts that are missing. While this is certainly one option, it’s probably not best for a few reasons:
- Your tester has access to Google and could easily find the public sample while looking for other documentation
- Most public samples are actually too long for an initial task (see below on choosing a task with the right size).
Now that I’ve hopefully discouraged you from the easy (but uninformative) path, let’s talk about what you should do.
Enumerate your product hypotheses
Iterating on a software product is about testing a series of hypotheses of the form: “I hypothesize that customers will be able to use solution X to solve problem Y.” So to decide what to test, start by enumerating your product’s hypotheses.
For example, if you’re working on SignalWire’s new Video API, your product hypotheses might be:
- I hypothesize that customers will be able to manage permissions for users in a video call by assigning each user a JSON Web Token (JWT)
- I hypothesize that customers will be able to get JWTs for each user by submitting authenticated POST requests from a backend server to a token endpoint
Choose your riskiest hypothesis
Now that you’ve enumerated your hypotheses, decide which of these hypotheses you have the least confidence in. Continuing the SignalWire example, there are a few ways to think about which hypothesis is riskiest:
- Which hypothesis involves a step that is least similar to the existing API offerings?
- Which hypothesis is associated with a step that results in the most customer support tickets?
- Which hypothesis is associated with a step that was not tested internally or with beta testers prior to being released?
Depending on your answers to the above questions, you should have a sense of which hypothesis you want to learn more about.
Let’s imagine that in the SignalWire case, the client SDK has resulted in a few customer support calls and that it’s the least similar to SignalWire’s existing offerings. So they decide to move forward with testing hypothesis number 3.
Choose a right-sized task
Now that we’ve decided to test hypothesis 3, we just need to come up with a task for a tester to complete. The rule of thumb I like to apply here is that any task which takes an experienced user of the product (like you) ~30 seconds to complete will take ~10 minutes for a new developer. So for the client SDK example, if your goal is to run a 20-minute usability test, think about what you can accomplish in 1 minute. That might be:
- Creating a simple HTML page
- Opening the HTML page in the browser
So there you have it! A 20-minute usability test of your riskiest hypothesis. To learn more and schedule a demo of Usabl’s offering for API usability tests, sign up here.