Coinbase, a global leader in the cryptocurrency industry with a mission to "increase economic freedom in the world," has a reputation for innovation not only with financial systems but also with technological innovation. "We want to be a leader and show we're not just thinking about crypto, we're also thinking about other innovative technologies and how they can remove friction, add speed and increase efficiency as a whole," says Roderick Randolph, a Principal Engineer with the Developer Experience team at Coinbase.

As the interest in AI hit a fever pitch in late 2022, it was natural that Coinbase was interested in ways they could leverage AI. "Our CEO Brian Armstrong is a big proponent of identifying ways we could adopt AI across the entire company," says Roderick, "and Coinbase has a very innovation-driven engineering culture. My job is to elevate developer productivity and enable developers to move fast with velocity, so it made sense to see how we could leverage AI code assistants to improve developer productivity."

With this desire for innovation came security and legal challenges for Coinbase, which not only deals with customer data and digital assets but also does so in an industry where the stakes are exceptionally high. Brady Thornton, a Staff Security Engineer and Technical Lead of the Security Advisory Services team at Coinbase, says: "Our mission is to be the most trusted crypto platform. Maintaining a high security bar while innovating is essential to earning and keeping that trust."

These challenges were compounded by the newness of AI and the lack of information about companies adopting the technology. "We were evaluating AI code assistants at a time when there was very limited public data on their use at large organizations,” says Brady. “There were a few academic studies related to AI code assistant adoption, and even fewer related to security, so we had to be creative with our approach.”

To help Coinbase adopt an AI code assistant, Roderick and Brady joined forces to evaluate options that met both requirements to improve developer productivity while ensuring Coinbase's strict security standards weren't compromised.

Cody set itself apart from the competition around security. "Some of our code directly handles digital assets and transactions," says Brady, "and our willingness to accept risk from AI code assistants with that code is especially minimal.”

As a result, Brady says the team came up with a threat model to identify risks and their severities across AI code assistants in a few key areas:

• Code exposure: "One potential show-stopping issue was sensitive code leaving our environment," says Brady, "Without using self-hosted models, there aren’t any guarantees of what happens with that data in the backend.”
• Model training and data use: The risk of inadvertently using proprietary code to train external models could expose confidential information. "There's a potential impact if proprietary Coinbase code or non-public information made its way into AI models."
• Attack vectors: Brady also expressed concern about the potential risk that attackers could influence AI training models by inserting malicious code patterns into open source projects, potentially causing those patterns to be suggested to Coinbase developers.

With these considerations in mind, Brady developed a process to evaluate which tool best met their security requirements. This included a unique statistical risk analysis where code generated with AI by engineers involved in the evaluation was compared not only to those not involved, but also to the code that participants wrote before using AI. “We were interested in whether AI coding assistants produced code of equal security quality compared to code written by engineers on their own. This was our null hypothesis. If we observed no significant increase in the rate of insecure coding patterns introduced between experimental groups, we could conclude that AI assistants weren't making things worse.” says Brady. “So, we designed and conducted an experiment to look for security issues across a number of groups, per PR, per business unit and after comparing our groups we determined using AI coding assistants made no statistically significant difference in the rate of observed security issues.”

In addition, Brady randomly sampled PRs from these groups, again ensuring statistical significance, and had people review them for potential security issues. The reviews were blind, meaning reviewers didn’t know whether the PR was written with or without AI assistance. The team found no statistical difference in the quality of the PRs.

Across all of the security testing and threat modeling conducted, Cody was determined as the best fit for Coinbase’s security requirements. These requirements haven’t hampered the impact it has had on developer productivity either, with Roderick highlighting Cody’s context awareness as a critical reason why Coinbase developers are now more productive. “Cody really stood out because of its context awareness. It wasn’t just suggesting random pieces of code; it generated boilerplate code based on our internal SDKs and frameworks. It was a good signal we were on to something powerful.”

One final decisive factor in Coinbase's selection of Cody was its support for Amazon Bedrock. Amazon Bedrock is a fully managed service that allows teams to set up Cody within their Virtual Private Cloud (VPC) and create an isolated environment where data doesn’t have to be sent over the public internet to an LLM provider.

Developers at Coinbase have access to a number of AI code assistants for their daily use, but Cody’s the only one among them that prevents code from leaving Coinbase’s environment. In addition to meeting its stringent security requirements, Cody's context awareness of Coinbase’s codebase has seen tangible improvements in productivity and time saved. “We’ve found engineers are saving roughly 5-6 hours per week using AI code assistant tools like Cody,” Roderick says, “and writing code 2x faster than without it.” Coinbase developers are also feeling these benefits, with 75% noting they were more productive in a recent survey.

Roderick believes that by using Cody, Coinbase can innovate while maintaining the security and trust of its customers. “We take security very seriously, and building and maintaining trust with our customers is incredibly important. Cody enables us to not only leverage technology that enables productivity but also keeps us secure.”