Developer extensions

 

Strategizing and shipping a major relaunch of Sourcegraph’s editor extensions

 
 
 
 
 

I worked as the product designer on a new release of our editor extension that saw a 342% increase in net new installs and a 37.2% conversion rate from CTAs to net new user accounts on the days following release

 
 

My Role

Design and test experience end to end, collaborate closely with PM on market strategy

Team

Lead Product Designer, Product Manager, 1 Engineering Manager, 3 Engineers

Timeline

Nov 2021-Feb 2022. MVP released, and team is planning future iterative improvements

 

01/

Discovery

 

What problems were we solving?

We frequently heard from our users that they would like to access more Sourcegraph features in their IDEs (integrated development environments). While Sourcegraph already had a VS Code and JetBrains extension, these legacy extensions only offered one feature, the ability to generate a Sourcegraph browser link.

The pain point underlying this feature request was that users disliked context switching back and forth from the tool where they write code to the browser to use our product. At its simplest, Sourcegraph helps devs answer questions about their code. These questions arise most often when devs are reading or writing code, which happen most often in their editor tools. There was an opportunity here to drive usage by delivering value in the tools and context our users preferred.

“There was an opportunity here to drive usage by delivering value in the tools and context our users preferred.”
 
 
 

Why prioritize this now?

Our continuous touchpoints with our users had uncovered this same pain point from multiple personas. We heard from multiple dev tool and productivity managers that it was hard to get devs to try out, much less learn, new tools. We heard from individual developers that their companies had too many tools that was required of them to use. We knew from our prior user research that the IC developer persona is typically very specific about their workflow and setup. These factors combined meant that only hosting our features on the browser was hurting our adoption and growth. 

Solving this problem also presented a business opportunity to increase growth and retention by allowing our users to use our product in the tools and context that they most preferred. We also knew from talking to our customer facing teams that a robust editor extension and extending into the developer toolset would make for a compelling sales and demo story. 

With this understanding, our team decided to prioritize improving our IDE extensions. We had key customers asking for both an improved JetBrains and VS Code extension, so we needed to work quickly to release both extensions sequentially.

 
 

02/

Definition

 

We knew that our users wanted more of Sourcegraph in their editors, but we didn’t have clarity on what features or for what workflows. A product goal of “extend our product into the editor” was an output , and it wasn’t structured or specific enough to drive our planning. We needed to define and structure the opportunity further.

 
 
 

User research

I started by combing through existing user data from customer requests in Salesforce and NPS forms. I was also able to quickly schedule user interviews with customers who had expressed already interested in better IDE extensions to our customer teams.

The top level goal of this research was to identify the common JTBD (Jobs to be Done) and workflows when developers use an editor and surface the corresponding opportunities for Sourcegraph.

This research helped us to understand that many users were content with the native search built into their editors — a “better search” would not be a compelling product opportunity unless users utilized the power features that we offered. However, Sourcegraph had a relatively low adoption of our power features even in browser.

 
 
 

However, we identified one clear opportunity that presented a competitive advantage over an editor’s native search: users wanted to search and read code that they did not have locally. While users were fairly content with their editor’s native search, this search only functioned for files that a user had downloaded. We offered the competitive advantage that you could search and read millions of public code without having to download the file. 

The feature also had a usability advantage over a user’s current existing alternatives: Users would leave their editors and search either in our product, a code host like GitHub which had a poor search experience, or sparse alternative search engines. This tied back to the larger parent pain point that devs don’t like context switching when in their editors. 

As a result of this user research, we decided that we would focus just on getting the core search workflow into the extension for MVP, and ruthlessly leave out any additional Sourcegraph features such as code intelligence and features such as code monitoring and batch changes. This was because the core search workflow was already a large endeavor, and sticking to just one core workflow for the MVP would allow us to more effectively measure and gauge the success of this extension for future improvements.

 

03/

Testing

 

Our devs were able to quickly hack together a prototype of the extension within a week by leveraging webviews. This allowed us to quickly plug and play browser components into the extension. We decided to test with this barebones prototype early on with internal dogfooding as well as internal testing.

The goal of this early testing was to gauge whether our initial assumptions to focus on the core search workflow were sound, as well as drive what improvements we should focus on prior to MVP release.

“Testing early on helped us to gauge whether our initial assumptions were sound and helped shape what improvements we should spend our remaining time on.”
 

We conducted two types of testing

  • Fast, quick tests on usertesting.com. We defined a user group to be specific to developers that used VS Code and JetBrains in two different tests. These tests took less than one day to set up and execute.

  • Internal dogfooding. We shared our prototype in our #dev-chat Slack group and asked our devs to use Sourcegraph in their editors for a few days and send over any feedback to our team Slack channel.

 

Testing helped validate that users were indeed keen on an alternative solution that would help them search through code when they had questions or wanted to see how others had solved a problem without having to download the file or leave their editor. More importantly, testing helped us zone in on the improvements that we needed to make. 

 
 

04/

Design

 

We need to respect a user’s mental model in their third party tool

Our preview release uncovered that the usability of our extension was negatively affected by the incongruous behavior of our extension. We heard a consistent sentiment from our users, “it looks unnatural.” This went beyond the UI to the expected UX - “I didn’t expect it to open up here.” Our webview preview allowed us to very quickly reuse web components, but we weren’t respecting the mental model of a user insider their editor.

We ended up going different directions with VS Code and JetBrains, due to the different UX patterns in each editor as well as the capabilities of the editor’s extension framework. The process to refine the user experience required heavy collaboration between me and our engineers; As I explored visual designs and workflows that were better tailored to each editor, engineers were deep diving into the extension frameworks. It was an ongoing process to research, refine, and scope until we landed at a solution that met our MVP requirements with a reasonable engineering time.

With VS Code, it was a common UX pattern with popular extensions to utilize the sidebar. We changed the interaction of our extension such that file navigation and authentication was done in the sidebar. Due to time and technical constraints, our v1 search input and search results were housed in a file or “tab” and heavily leveraged webview components, but we flagged it as a potential future improvement to explore a UX where a search could be launched inline while coding.

 
 

With JetBrains, their native search and “find in files” was very popular with their userbase and utilized modals. The JetBrains extension framework allowed for a better mixture of using webview components and JetBrains native components, and gave us more flexbility with our MVP solution.

Our first release of the JetBrains extension mirrored their native search by housing the search input and results in a modal. We modified the syntax and visual design of how our search results were displayed to be more closely aligned to JetBrains. I leveraged key iconography from Sourcegraph to denote the different result types (commits, diffs, repositories, file paths), while modifying the syntax to be more familiar to a JetBrains user.

Another key difference was that we leveraged JetBrain’s native file preview. This was a notable differentiator from our browser experience, which only shows a preview of 3 lines on the search results page. This was a great combination of leveraging a UX pattern that JetBrains user's would be familiar with while saving us development time by using a native component.

 
 
 
 

We need better onboarding

Our external testing showed that users who were unfamiliar with Sourcegraph had a hard time understanding the concept of a code search. While Sourcegraph in the browser had various features in place to promote onboarding, I needed to think about how I could adapt or create new ways of onboarding to be specific to an editor’s UX.

This was especially a challenge with JetBrains as the modal we housed our extension in was a very different context than the browser. My end solution ultimately leveraged multiple aspects of the JetBrain’s interface.

Upon installing the extension, we leveraged the JetBrains native notification balloon to onboard users onto how they could access Sourcegraph through hotkeys, as well as learn more about the extension. After opening the extension and activating the searchbar, we showed various query builders by default to help users onboard onto our search syntax. Finally, I leaned into JetBrain’s pattern for providing hotkey and daily tips in the footer.

While these were subtle features to aid a user’s onboarding, we decided that we would measure the effectiveness of especially the query builders in the MVP release to understand how we might iterate on the onboarding workflow.

 
 
 

We need to promote product growth

While we were solving core user needs through this extension, we needed to ultimately drive revenue. At this point, Sourcegraph was focused on individual developers creating Cloud accounts as a business outcome. This meant that we needed to focus on messaging for the individual developer persona to drive account creation.

To achieve this, we implemented two types of CTAs: a one-time dismissible banner as well as embedded CTAs for when users tried to use advanced features such as saved searches or code monitoring without authenticating. We also implemented a landing page specific to each editor when a user clicked on a CTA.

 

These CTAs saw a 37.2% CTA click to account creation conversion rate the week of launch.

 
 

05/

Launch & Beyond

 

We worked with Product Marketing for the VS Code and JetBrains launch. We tracked net new installs, unique searches performed, and conversion rates to Sourcegraph accounts to measure our success. Notably, the extension was an explosive hit on Twitter with 114k impressions and a retweet from the official VS Code account, which helped to drive traffic near launch days.

 

We also set up metrics to track correlation between the extension use and retention of our enterprise customers. However this was a lagging indicator, and we were able to gauge interest from customer facing teams who shared that the extension was immediately a hit in demos as well as prospect calls. 

While we the celebrated the launch, it was important to keep in mind that the launch was an MVP. As we continued to monitor metrics post launch, we uncovered a number of concerning trends that we want to improve:

  • We weren’t successfully activating users. A relatively small percentage of users who installed the extension were performing searches.

  • Our unique active usage was low; a relatively small proportion of users were conducting many searches, which drove overall usage.

Our hypothesis to improve both of these trends is that we need to improve how native our extensions felt — we believed that our extension helped solve user problems in key use cases, but we needed to expand how users could launch and explore searches to be more natural to an editor’s workflow.

While iterations of the editor extensions were put on hold due to shifting business priorities, we are continuing to monitor metrics and outreach to our users.