So you’ve decided you want to run an usability test to discover ways in which you could improve the developer experience with your APIs. Congratulations! The next step is to decide what exactly to test. You might be thinking of simply taking one of your public samples, removing some of the code, and asking the tester to fill in the parts that are missing. While this is certainly one option, it’s probably not best for a few reasons:
Now that I’ve hopefully discouraged you from the easy (but uninformative) path, let’s talk about what you should do.
Iterating on a software product is about testing a series of hypotheses of the form: “I hypothesize that customers will be able to use solution X to solve problem Y.” So to decide what to test, start by enumerating your product’s hypotheses.
For example, if you’re working on SignalWire’s new Video API, your product hypotheses might be:
Now that you’ve enumerated your hypotheses, decide which of these hypotheses you have the least confidence in. Continuing the SignalWire example, there are a few ways to think about which hypothesis is riskiest:
Depending on your answers to the above questions, you should have a sense of which hypothesis you want to learn more about.
Let’s imagine that in the SignalWire case, the client SDK has resulted in a few customer support calls and that it’s the least similar to SignalWire’s existing offerings. So they decide to move forward with testing hypothesis number 3.
Now that we’ve decided to test hypothesis 3, we just need to come up with a task for a tester to complete. The rule of thumb I like to apply here is that any task which takes an experienced user of the product (like you) ~30 seconds to complete will take ~10 minutes for a new developer. So for the client SDK example, if your goal is to run a 20-minute usability test, think about what you can accomplish in 1 minute. That might be:
So there you have it! A 20-minute usability test of your riskiest hypothesis. To learn more and schedule a demo of Usabl’s offering for API usability tests, sign up here.
Learn more about how API usability impacts your business
Make your quickstart harder
When I was at Microsoft, one of the Azure Machine Learning team’s strategic initiatives was “five minutes to wow.” Our aim was to ensure that a developer new to Azure ML should, within 5 minutes, achieve such a surprising/useful/novel outcome with Azure ML that they say “wow!” Inspired by the goal, I worked on a […]
How Courier is using Usabl to unlock developer virality for its multi-channel notification API
“Videos from Usabl are timely, informative, and actionable. They help us uncover previously unknown issues in our developer experience and build internal consensus around real solutions” – Donnie Wang, Growth PM @ Courier Courier’s API makes it easy to send multi-channel product notifications reliably and in a way that is respectful to the end user. […]
We evaluated the usability of 4 leading eSignature APIs
Most were surprisingly bad eSignature as a space has been heating up for a while now. Look at Dropbox’s $230 million acquisition of HelloSign in 2019 or PandaDoc’s recent unicorn status for evidence of the growth in interest in the sector. The high valuations reflect the fact that signatures are involved in some of the […]
Why you, a UI-first business, should care about developer experience
What do the video game Doom, shopping malls, and Salesforce have in common? (No, it’s not that they all are fun ways to spend an afternoon). Doom was remarkable as one of the first PC games to support “modding.” Shopping malls are only as successful as the tenants that rent space within them. And Salesforce […]
Tag yourself: a framework for developer experience maturity
As we’ve built out Usabl’s platform for getting feedback from real developers, we’ve spoken to dozens of companies building API- and developer-first products for everything from payments processing to security scanning. In each conversation, we’ve consistently seen patterns in the way these businesses engage with developers. In this piece I propose that developer-first businesses can […]
Understanding the developer activation funnel
Your developer experience should be your best marketing tool As we’ve interviewed companies building developer-facing SaaS products, I’ve seen some common patterns emerging in the way companies build for developers. In this blog post, I’ll explain at a high level the journey that a developer will typically take with a SaaS product. To the best […]
API usability for developer-first products
It’s 2021, and developer experience is coming more and more into focus. In fact, companies are hiring product managers specifically with the goal of improving their developer experience. Staying close to your customer matters in every industry. Though it feels harder to get feedback as developers are expensive and specialized, Usabl is focusing on helping […]
Writing your first task
Writing your a task for an API usability test Let’s say you’ve gone through the steps here and chosen a well-sized task that tests your riskiest hypothesis about the usability of your API. You’re now ready to actually write your first task. How should you go about doing this? Minimize configuration time Remember, your goal […]
What part of your API should you test for usability?
So you’ve decided you want to run an usability test to discover ways in which you could improve the developer experience with your APIs. Congratulations! The next step is to decide what exactly to test. You might be thinking of simply taking one of your public samples, removing some of the code, and asking the […]