Etsy: Doing more with less cost and infrastructure

Google Cloud results

• More than 50 percent savings in compute energy
• 42 percent reduction in compute costs using committed-use discounts and other optimizations
• 5.5 petabytes of data migrated

Keeping commerce human

With over 2.8 million sellers and more than 66 million unique items, Etsy enables commerce in an enormous, global community. The company’s platform makes it easier for people to supplement their income, or even make a living, from the comfort of their own home, creating opportunities for people who might face obstacles in traditional marketplaces. This is reflected in the demographics of its sellers, 83 percent of whom are women, and 95 percent of whom work from home, making Etsy even more important during times when people may be unable to leave their homes, as has been the case during the COVID-19 pandemic. As an organization, Etsy has a strong commitment to the people who use the company’s platform, and the families those individuals support, as evidenced by Etsy’s mission, Keep Commerce Human.®

From a platform standpoint, the best way to do this is by staying ahead of the technology curve. Etsy wants to make sure that its platform functions smoothly during peak periods of sales, is cost-efficient, and is environmentally sustainable. The company also wanted to ensure that it had the best and most sustainable collaboration tools for communication both within the company and with the sellers using Etsy’s platform. With those goals in mind, Etsy began the process of moving to the cloud.

Choose your cloud

“We wanted to make sure we weren’t biased,” says Mike Fisher, Etsy’s Chief Technology Officer, when discussing the process by which Etsy evaluated potential cloud providers. With that in mind, Etsy embarked on a detailed process during which senior engineers and key decision-makers collaborated to map out Etsy’s existing on-premises architecture. This map was used to identify and rank Etsy’s requirements for the capabilities of its cloud provider. Etsy then evaluated potential cloud providers using the ranked requirements. “The total scores for each vendor reached over 50,000 points, with Google Cloud exceeding the others by more than 10 percent,” Fisher noted in a post describing the evaluation process on Code as Craft, Etsy’s engineering blog.

When looking at the reasons why migrating to the cloud would be a clear win for Etsy, as well as the ways in which Google Cloud excelled, three areas of alignment stood out. “It came down to three major criteria,” shares Dany Daya, Senior Program Manager for cloud migration at Etsy. “Number one is the ability to scale on demand. With data centers, you have to plan ahead of time, probably six or nine months ahead, to do provisioning, to purchase the hardware, to be able to install, test, and run it. On the cloud, we can do that in a significantly shorter time frame, and we can scale down after we finish.”

Sustainability was another area in which Etsy and Google Cloud were closely aligned. “Sustainability is one of Etsy’s major goals, and we track sustainability just like we track revenue and profit,” shares Daya. Google Cloud’s commitment to carbon neutrality—and especially its emphasis on energy efficiency in artificial intelligence (AI) and machine learning (ML)—resonated with Etsy’s emphasis on serving its community by maintaining sustainability. “We wanted to take advantage of Google’s machine learning technology to minimize Etsy’s carbon footprint and energy usage.”

The third area of alignment was an emphasis on an open culture of sharing between the two organizations. As Fisher explains in a Code as Craft post, “Although we use the term ‘vendor’ when referring to a cloud provider, we viewed this as much more than a simple vendor selection process. We are entering into a partnership and a long-term relationship.”

Daya explains how the relationship with Google Cloud facilitates these efforts: “We call it enabling—enabling our engineers and developers to leverage the latest Google technologies, such as machine learning and AI.” This shared vision for the partnership between Etsy and Google Cloud involved making it possible for engineers and developers to do more than just use Google Cloud’s latest features. By eliminating the effort and costs associated with maintaining on-premises data centers, Etsy’s team would be free to focus on optimizing the features that mattered most to its buyers and sellers, like search and recommendations. Partnering with Google Cloud also meant that Etsy could maintain its commitment to the open source development community.

It’s not (just) about the technology

The bidirectional relationship between Etsy and Google Cloud has proven to be a crucial element to the success of its migration. Even before a contract was signed, Fisher relates, “We found that Google would come into meetings, pull their chairs up, meet us halfway and say, ‘we don’t do that, but let’s figure out a way that we can do that for you.’” The Google Cloud team knew they were being entrusted with a very important task. Before the migration, Etsy had always run its own data centers and been exclusively responsible for everything related to those data centers. Now, Etsy would share that responsibility with Google Cloud engineers.

To do that job well, the Cloud team needed to have a deep working knowledge of Etsy’s system. “What we did to support them is we went in and trained alongside Etsy engineers,” explained Danny Rosen, Technical Account Manager with Google Cloud, at Google Cloud’s Next conference in 2019. “We went through a two-week immersed training in Etsy.” This core support team then created tools that could be used by other members of the Google Cloud team whenever Etsy required infrastructure support, both during key moments in the migration and during Google Cloud’s ongoing partnership with Etsy once the migration was complete. The core team also was available during the scheduled cut over from the on-premises data centers to Google Cloud—at 3 a.m. at Etsy’s headquarters.

After overcoming some initial complexities in the migration, Etsy was very quickly able to shift some workloads onto Google Cloud. Other elements of the company’s infrastructure were moved using a lift-and-shift approach, followed by post-migration optimization for the cloud. And when it came to improving code, Etsy found an unexpected resource that proved valuable for identifying opportunities to increase efficiency: cost data.

Doing more with cost data

“Being in the data center, the cost equation was relatively straightforward,” explained Keyur Govande, chief architect at Etsy. “You buy some hardware and then you use it for three years. Or you don’t use it. Whereas in Google Cloud, it really matters if the box is up and running and you’re not using it.” Having compute resources that scale on demand means that cloud users also have compute costs that scale with demand. The problem is that sometimes you find spikes in demand—and cost—where you don’t expect them.

“The first priority was to clear away the wasted resources created by our use of a lift-and-shift approach,” explains Daya. “Some of this was low-hanging fruit, such as minimizing or resizing solid state drives (SSDs) to match their load, and resizing virtual machines for utilization and performance.” Storage was also optimized to minimize compute costs. “We compressed the data and used the different storage categories provided by Google Cloud,” Daya shares. In addition to these optimizations, Etsy took advantage of Google Cloud’s committed-use discounts (CUDs), which helped lower its compute cost by 42 percent.

Nevertheless, Etsy still saw some unexpected variability in costs. “You will have data scientists, developers, and engineers testing new ideas that potentially come up with great results for our sellers and buyers. However, when they are testing it, we are seeing spikes in costs,” says Daya. Etsy saw these unexpected costs as an opportunity to increase cost visibility among the company’s teams. “Dany (Daya) was an early adopter of several Google Cloud Billing and Cost Management features, and we wanted to give that same visibility to everyone, to introduce accountability across the company about each person’s impact on cost,” explains Jack Perkins, Data Analyst at Etsy. That meant making cost data accessible with reporting views that made sense to everyone.

“This is all run through Looker,” shares Perkins. “The financial team comes to an agreement on the budget, and we integrate that and replicate it on the BigQuery side. That is integrated with Google Cloud’s billing export. People can view the data through a dashboard and receive emails highlighting variance to the budget.” Making sure that the Etsy team can easily find the cost data it needs has been especially valuable. “The ability to add any kind of arbitrary metadata into resource labeling has really helped us map costs to different types of activity, cost centers, things like that,” he explains.

Making cost data available to the engineers has also had an unexpected upside: providing visibility into the interactions between elements of Etsy’s system. “Because of the complexity of some of the systems, there are times when we don’t have enough understanding of the small steps that are happening in the background,” says Perkins. “We have noticed that this cost data, these tools, have served us well and helped us identify and fix underlying technical issues.”

Collaboration that ensures everyone is heard

The spirit of internal collaboration reflected in the way Etsy democratized its cost data can also be seen in the way the company uses collaboration tools. Etsy was an early adopter of Google Workspace, and it has put significant effort into being innovative and effective in its use of Calendar, Docs, Sheets, Slides, Meet, and Forms. Shawn Carney, director of engineering, IT, and compliance, explains, “Our working culture matches our guiding principles: digging deeper, minimizing waste, embracing differences, committing to our craft, and leading with optimism. Across Etsy, we use comments to dig deeper and embrace different ideas. Our Internal Comms team uses Docs and Slides to asynchronously pull for content, cast a wide net for feedback, and collaborate cross-functionally. This also helps us minimize waste from unnecessary meetings.”

Thoughtful use of Google Workspace isn’t limited to the Comms team. The Engineering team uses Docs to gather input for, and feedback on, architectural designs. Customer Support Agents use Google Translate when they need to help someone whose language isn’t supported, and they are sensitive to the customer’s experience by being transparent about their use of an automatic translation tool. Etsy’s IT department even used Google Workspace to collaborate while writing a post for Code As Craft regarding the company’s recent transition to increased use of Meet during the COVID-19 pandemic. Carney shared the methodology: “We used a Doc to asynchronously gather initial ideas, and then we held an optional Meet where people could help build the narrative up into a story, while taking notes in the Doc. Afterward, we shared it broadly for internal feedback before posting on our blog.”

Etsy’s cross-organizational culture of collaboration, in the context of the pandemic-driven imperative to work from home, has led the company to identify opportunities for further growth. Carney says, “We’re currently thinking about collaboration as it intersects with the future of work. We’ve recently seen enthusiastic adoption of new tools (Google Meet) and ways of working (more asynchronous collaboration) as part of our pivot to a fully distributed model. We are paying attention to the changing needs of our employees at Etsy and actively investigating solutions to enable teams to collaborate together outside of offices, inside of offices, and across time zones.”

Enabling a focus on community with Cloud and Google Workspace

Etsy completed its migration to Google Cloud in February 2020, moving 5.5 petabytes of data in record time. Over the course of the migration, Etsy has been able to shift 15 percent of its engineering headcount from managing system infrastructure to focusing on customer experience. This has, in turn, led to significant improvements in the time to ship new features, which keeps Etsy as a whole more focused on its mission of keeping commerce human. “Our customers don’t care if we’re the best in the world at supporting hardware,” relates Fisher. “They care about us having the best marketplace, with the functionality and features they want.”

Etsy’s engineers and data scientists are also carrying out twice as many experiments, including those in the areas of AI and data analytics. In addition, Etsy’s migration to Google Cloud significantly reduced Etsy’s carbon footprint, with more than 50 percent savings in compute energy. Finally, through careful application of CUDs and the use of cost data and reporting as an optimization tool, Etsy was able to reduce compute costs by 42 percent and improve cost predictability. Etsy’s Google Workspace metrics are equally impressive; in March 2020, the company’s employees averaged 1,050 Meets per day, sent 24.5 million emails, and shared nearly 600,000 files. Etsy is leading the way through its commitment to the community and by doing more with less in partnership with Google Cloud.