Make your quickstart harder

Written by Gopal Vashishtha

Published on Mon Mar 28 2022

When I was at Microsoft, one of the Azure Machine Learning team’s strategic initiatives was “five minutes to wow.” Our aim was to ensure that a developer new to Azure ML should, within 5 minutes, achieve such a surprising/useful/novel outcome with Azure ML that they say “wow!” Inspired by the goal, I worked on a feature in my product for real-time inference called “no-code deploy (NCD).” With NCD, a developer new to Azure ML could click on a button in a GitHub repo containing a machine learning model, fill out some fields in a form, and deploy that model as a REST endpoint in Azure. While NCD really impressed my leadership, it failed to make a significant impact on the top-line metrics we cared about: number of active users and retention.

Help developers learn not by hiding hard concepts, but by introducing those concepts gradually

It was literally impossible to make my product any easier to use, and yet users still weren’t sticking around. How could I explain this conundrum?  As I talked with churning users I realized that I had simply replaced their “day 1” questions with “day 2” questions. Users went from “how do I deploy a model with Azure ML?” to “how do I deploy my model with Azure ML?” Because we had obscured all the details about what happens when deploying a model, like defining a Docker container, uploading the model to storage, or running the model in an entry script, as soon as users wanted to actually use the product, they were actually worse off than they had been before, because they had the same knowledge of my product but higher expectations of what it should do.

Armed with this new insight, I focused on refactoring our tutorial to introduce concepts rather than to hide them. Start by creating an entry script with no model, then add in your model, then modify the Docker container. The quickstart actually got harder, but more users were successful when using it. Our support team saw an immediate reduction in basic conceptual questions from users.

My personal experience was only one example of a frequent misconception I see among developer-first companies that their quickstarts need to be as easy to use as possible. Taking this idea to an extreme, some developer-first companies will actually build quickstarts similar to mine where all the code has been written in a code sandbox, and all that remains for developers to do is to click buttons in the correct order. Unfortunately, because there is no meaningful cognitive engagement on the part of the user, the developer can very easily zone out and learn nothing during the quickstart.

Csikszentmihalyi’s concept of “flow” captures roughly what I’m trying to say about developer quickstarts:

Csikszentmihalyi tells us that when a task is challenging but also well-matched to an individual’s skill level, they enter a state of “Flow”

You don’t want your quickstart to be like vim’s, where the learning curve is so steep that developers enter the “anxiety” section and avoid learning the tool at all. But you also don’t want a highly guardrailed experience like AWS Amplify’s that removes all agency from the user and relegates them to “boredom.” You want the developer to find a “flow,” the happy medium that gives them opportunities to think about real problems and how your product can solve those problems.

The Python Django quickstart is a great example that gets “developer flow” right. This tutorial goes so far as to intentionally introduce a bug for a developer to resolve. Because this quickstart requires developers to take non-trivial actions to succeed, they leave feeling not just motivated to do more but equipped to do so.

When you’re building onboarding experiences for developers and make a decision to hide something, ask yourself: is this step I just hid something that most developers will probably need to learn at some point? If the answer is yes, find a way to reintroduce that concept into the quickstart in a way that makes sense. Change your goal from “5 minutes to wow” to “10 minutes to understanding.”