Developer extensions

 

Strategizing and shipping a major release of Sourcegraph’s editor extensions

 
 

I worked as the product designer on a major new release of our VS Code extension that saw a 342% increase in net new installs and a 37.2% conversion rate from CTAs to net new user accounts on the days following release

 
 

My Role

Design and test experience end to end, collaborate closely with PM on market strategy

Team

Lead Product Designer, Product Manager, 1 Engineering Manager, 3 Engineers

Timeline

Nov 2021-Feb 2022. MVP released, and team is planning future iterative improvements

 

01/

Discovery

 

What problems were we solving?

We frequently heard from our users that they would like to access more Sourcegraph features in their IDEs (integrated development environments). While Sourcegraph already had a VS Code and JetBrains extension, these legacy extensions only offered one feature, the ability to generate a Sourcegraph browser link.

The pain point underlying this feature request was that users disliked context switching back and forth from the tool where they write code to the browser to use our product. At its simplest, Sourcegraph helps devs answer questions about their code. These questions arise most often when devs are reading or writing code, which happen most often in their editor tools. There was an opportunity here to drive usage by delivering value in the tools and context our users preferred.

“There was an opportunity here to drive usage by delivering value in the tools and context our users preferred.”
 
 
 

Why prioritize this now?

Our continuous touchpoints with our users had uncovered this same pain point from multiple personas. We heard from multiple dev tool and productivity managers that it was hard to get devs to try out, much less learn, new tools. We heard from individual developers that their companies had too many tools that was required of them to use. We knew from our prior user research that the IC developer persona is typically very specific about their workflow and setup. These factors combined meant that only hosting our features on the browser was hurting our adoption and growth. 

Solving this problem also presented a business opportunity to increase growth and retention by allowing our users to use our product in the tools and context that they most preferred. We also knew from talking to our customer facing teams that a robust editor extension and extending into the developer toolset would make for a compelling sales and demo story. 

With this understanding, our team decided to prioritize improving our IDE extensions, starting first with VS Code.

 
 

02/

Definition

 

We knew that our users wanted more of Sourcegraph in their editors, but we didn’t have clarity on what features or for what workflows. A product goal of “extend our product into the editor” was an output , and it wasn’t structured or specific enough to drive our planning. We needed to define and structure the opportunity further.

 
 
“We needed to define and structure the opportunity further”

User research

I started by combing through existing user data from customer requests in Salesforce and NPS forms. I was also able to conduct quick user interviews with customers who had expressed already interested in better IDE extensions to our customer teams.

The top level goal of this research was to identify the common JTBD (Jobs to be Done) and workflows when developers use an editor and surface the corresponding opportunities for Sourcegraph.

This research helped us to understand that many users were content with the native search built into their editors — a “better search” would not be a compelling product opportunity unless users utilized the power features that we offered. However, these were features Sourcegraph traditionally had low adoption on even in the browser. 

However, we identified one clear opportunity that presented a competitive advantage over an editor’s native search: users wanted to search and read code that they did not have locally. While users were fairly content with their editor’s native search, this search only functioned for files that a user had downloaded. We offered the competitive advantage that you could search and read millions of public code without having to download the file. 

The feature also had a usability advantage over a user’s current existing alternatives: Users would leave their editors and search either in our product, a code host like GitHub which had a poor search experience, or sparse alternative search engines. This tied back to the larger parent pain point that devs don’t like context switching when in their editors. 

 

03/

Testing

 

One month in, we launched a preview version of the extension internally for dogfooding as well as external testing. At this point the preview release only had a barebones UI, but testing early on helped us to gauge whether our initial assumptions were sound and helped shape what improvements we should spend our remaining time on.

“Testing early on helped us to gauge whether our initial assumptions were sound and helped shape what improvements we should spend our remaining time on.”

We conducted two types of external testing: 

  • Cheap, quick tests on usertesting.com (less than one day to set up and execute)

  • Moderated tests with customers that had expressed interest in an extension (around two weeks to set up interview tasks, schedule, and synthesize) 

Testing helped validate that users were indeed keen on an alternative solution that would help them search through code when they had questions or wanted to see how others had solved a problem without having to download the file or leave their editor. More importantly, testing helped us zone in on the improvements that we needed to make. 

 
 

04/

Design

 

We need to respect a user’s mental model in their third party tool

Our preview release uncovered that the usability of our extension was negatively affected by the incongruous behavior of our extension. We heard a consistent sentiment from our users, “it looks unnatural.” This went beyond the UI to the expected UX - “I didn’t expect it to open up here.” We were focused on getting features into the extension and overlooked that we needed to respect a user’s mental model in a third party tool when building an extension.

One design change we made as a result was how the extension could be accessed. I looked at popular extensions and the default search UX in each IDE to recognize common UX patterns that we should be utilizing in our own search.

With VS Code, we learned that extensions typically utilized the sidebar and/or incorporated features directly inline to code. We changed the interaction of our extension such that file navigation, authentication, and even onboarding was done in the sidebar. Due to time and technical constraints, our v1 search input and search results were housed in a separate “tab”, but we flagged it as a potential future improvement to explore a UX where a search could be kicked off while directly in a file.

With JetBrains, their native search and “find in files” was very popular with their userbase and utilized modals. Our first release of the JetBrains extension mirrored their native search by housing the search input and results in a modal. We also modified the syntax of how our search results were displayed to be more closely aligned to JetBrains, and introduced a scrollable file preview, which was a notable differentiator from our browser, which only shows a preview of 3 lines.

 
 
 

We need to promote product growth

Testing uncovered that users expected to be able to search private code through our extension by default, when they needed to create an account and authenticate. We needed to better communicate this requirement and also display the benefits of a Sourcegraph account to promote growth. 

At this point, Sourcegraph was focused on individual developers creating Cloud accounts as a business outcome. This meant that we needed to focus on messaging for the individual developer persona to drive account creation.

To achieve this, we implemented two types of CTAs: a one-time dismissible banner as well as embedded CTAs for when users tried to use advanced features such as saved searches or code monitoring without authenticating. We also implemented a landing page specific to each editor when a user clicked on a CTA.

Overall, these CTAs were very successful upon launch - seeing a 37.2% CTA click to account creation conversion rate the week of launch.

 
 

05/

Launch & Beyond

 

On February 8th 2022, we launched the VS Code extension, working with Product Marketing for the launch. We tracked net new installs, unique searches performed, and conversion rates to Sourcegraph accounts to measure our success. Notably, the extension was an explosive hit on Twitter with 114k impressions and a retweet from the official VS Code account, which helped to drive traffic near launch days.

We also set up metrics to track correlation between the extension use and retention of our enterprise customers. However this was a lagging indicator, and we were able to gauge interest from customer facing teams who shared that the extension was immediately a hit in demos as well as prospect calls. 

While we the celebrated the launch, it was important to keep in mind that the launch was an MVP. As we continued to monitor metrics post launch, we uncovered a number of concerning trends that we want to improve:

  • We weren’t successfully activating users. A relatively small percentage of users who installed the extension were performing searches.

  • Our unique active usage was low; a relatively small proportion of users were conducting many searches, which drove overall usage.

While iterations of the editor extensions were put on hold due to shifting business priorities, we are continuing to monitor metrics and outreaching to our users.