How Duolingo built their Viral Superbowl Ad w/ Paul Rothenberg, Sr. TPM

Every month I teach a Tech Term You Should Know (TTYSK) and a tech essay to level up your technical literacy and communicate well with dev teams. Ask me anything and I'll cover it in an upcoming issue.

This issue's TTYSK is "Boolean". Boolean is a common term in data and understanding how they're used in programming will help you connect user flow logic to algorithms. Scroll to end of issue for full definition 👇

Paul Rothenberg is a technical program manager at Duolingo, where he looks after marketing technology and growth. Before the big green owl, he worked in marketing, analytics, and software development at companies like Verizon and JetBlue Airways. He lives in Brooklyn with his wife and highly-opinionated infant son.

Some takeaways from our conversation:

1. Duolingo's product teams are cross-functional and focused on specific metrics. For instance, the retention team collaborates to enhance user retention rates through experiments. Platform teams support product teams, emphasizing speed over a single metric.

2. Duolingo prioritizes a strong product focus, ensuring close collaboration between product and engineering, as well as product and design teams. Cross-functional teams guarantee continuous input exchange from ideation to project completion.

3. Product managers craft specs with early engineering input, fostering a close-knit collaboration among design, engineering, and product teams.

4. To address the challenge of live-event push notifications at a large scale, the team explored various solutions but opted for simplicity and rapid prototyping. Through multiple tests and iterations, they refined a system that achieved both speed and predictability.

5. Early alignment among marketing, product, and engineering on a shared goal. is crucial. This collaboration provided time for prototyping and iterative testing cycles. Emphasizing and addressing risks early on, openly and vocally, ensured the necessary resources to validate assumptions, crucial for a high-stakes, low-margin-for-error project.

Before we get into the nitty gritty, can you tell us a bit more about how teams at Duolingo are structured and how everyone works together?

PAUL: Product teams at Duolingo are typically structured cross-functionally and align to a specific metric. For example, our retention team has product managers, designers, and engineers, and they're solely responsible for running experiments to move our current user retention rate in the right direction. These teams are supported by platform teams, which don't have a single metric like that, but exist to speed up the product teams. These platform teams, including my own, are often composed primarily of backend engineers, and have a technical program manager instead of a product manager – although in this context, they perform similar functions. 

Process-wise, teams here are empowered to adopt the amount of process that fits their team best. We have best-practices that we start from (classic scrum), but generally we're not prescriptive when it comes to how teams should run. The last thing we want is for process to become a hindrance rather than something that adds value. 

Product managers write requirements in the form of "specs", which include engineering input very early on. As I said, our product teams are very close-knit between design, eng, and product.

Duolingo's 5-second Superbowl ad and the subsequent push notifications to 4 million learners in 5 seconds garnered a lot of attention. Can you share the inspiration behind the decision to create such a short yet impactful ad, and what goals did the team aim to achieve with this campaign?

PAUL: We know there was a precedent for 5-second SB ads breaking through, we saw it in recent years with Reddit and others. We’ve long been a scrappy, social-first brand. The thinking was – if we bought a 5-second spot, and really did it right, could we “hack” our way into getting similar buzz and similar media coverage as we would have if we were to buy a regular Super Bowl commercial – except without spending $10M? The push notification was part of that strategy. So far the results have been excellent.

Duolingo wrote a blog about how your company created and executed the Superbowl ad. It talked a bit about the collaboration between the product and engineering teams. Before diving into the technical details, could you elaborate on how these two teams typically work together at Duolingo and what makes this collaboration successful?

PAUL: We’re very much a product-focused company. So product and engineering, product and design, they’re pretty much in lock-step with each other. In fact, we organize teams cross-functionally to include members from each group, so that they’re giving each other input from ideation all the way through the end of a project.

Can you walk us through the initial stages of the project, highlighting how the product and engineering teams collaborated in the ideation and conceptualization phase?

PAUL: This project was actually a bit different, as the requirements didn’t come from product, but rather marketing. That happens a lot more rarely, but when it does, it’s always fun. The earliest concepts of the ad creative had an even stronger tie-in with the push notification, so we treated the timelines and scale as a non-negotiable from the start. I was a little bit nervous that the engineers on the project would have been skeptical or resistant to the idea – but that wasn’t the case at all. We knew it would be a major challenge, but we also knew it would result in a breakthrough brand moment for us. The idea was just too fun.

Could you explain the key technological components and strategies employed to achieve the remarkable speed in delivering notifications to 4 million learners immediately after the ad aired?

PAUL: We talked to a lot of folks in the industry who work on similar problems, however most of the potential solutions we came up with through that either relied on knowing exactly when the push notification was to be delivered in advance (didn't have that luxury since it was a live event), or had a slightly more forgiving time window (minutes, not seconds). We had a few different potential approaches in mind, but we ended up trying out the one that we felt was both the simplest implementation and the one that gave us the best chance to get a prototype in place the soonest.

Since we had medium confidence (at best) that any particular option would have worked, we wanted to leave as much time as possible for trial and error. So, the strategy early on was to get to testing as soon as possible. In total we did seven or eight full-scale tests to go with countless smaller ones. When the first test went out, we liked the send speed we were seeing overall, but saw odd distributions in the length of time it was taking for the messages to be received. But experimenting with the number of threads and processes the system was using over several weeks, we were able to increase both the send speed and the predictability over time. 

At first we looked a lot into storing the notification locally and simply “revealing it” at the right time. This wouldn’t work because we didn’t know exactly when the ad would air. Then we talked about having the client constantly poll in the background for a change in an S3 bucket – then when the ad airs, we’d make that change in S3, which would reveal the local notification. That didn’t work because of how iOS restricts background app activity. The system that we ended up with is pretty elegant in its simplicity: it’s essentially tiers of workers that keep user information read from an S3 bucket in memory, working asynchronously off an Amazon SQS queue. It’s then able to hit the Apple and Google APIs to send the message with massive concurrency. Thankfully, those APIs were up to the task.
 
In terms of project management, how did you ensure effective communication and coordination between the product and engineering teams throughout the development and execution of the Superbowl ad campaign?

PAUL: Our product managers have excellent working relationships with our engineers and don’t need very much of this at all. Our marketers, however, are far less accustomed to working with engineering folks – and vice versa. Having a background in both makes this easy for me when I’m in the room, or on the Slack thread – but since I can’t always be there, I encouraged my team to use simple, straightforward language, and to use empathy when delivering information across functions. For example, always think first – what does the other person want to get from this interaction? It’s probably not a low-level accounting of technical considerations we’re working through at this point in the project, it’s more likely a sense of confidence that we’re on the same page, and we’re on track – or if not, what we’re doing about it, and how important this is to us.  

Were there any specific challenges or obstacles that the teams encountered during the process, and how did you address them to keep the project on track?

PAUL: I know that creating a five-second ad that breaks through is a massive challenge by itself, but for our part it was about how the app handled notifications when we started on this. Long story short, the app was doing "too many things" when it received a push notification. That's fine over a more normal period of time, however with a spike of the magnitude we were looking at, it was dangerous since notifications had caused a disruption in service for our learners in the past, and this was something we were keen to avoid.

The key was to rework the app to be more efficient about what it does when it gets a notification so as to not cause any kind of interruption for users. We did this by building a new system that included AWS SQS for our queue service, AWS S3 for storage, and a single microservice, which we dubbed, of course, “Superb owl”. This was completely separate from the new system we had to build, but it was necessary for the campaign to be a success – so we had to loop in additional teams.

The article mentions the success of the ad, but could you share specific metrics or key performance indicators (KPIs) that the team used to measure the impact and effectiveness of the campaign from both a product and technical perspective?

PAUL: From the technical perspective, our success criteria was pretty simple: we had a speed goal – to send out the 4M notifications in under 5 seconds – and a goal to ensure there were no issues with our infrastructure that would cause a poor experience for our learners when it happened. Thankfully, we were successful on both. On the marketing side, we focused on social impressions (which is relatively straightforward to measure) and PR pickup (for which we’ve devised an internal scoring system). 

For testing the speed goal – we knew we didn't have time to send tracking events as part of this, so we ended up measuring this in two ways: simply measuring how quickly the workers got through the queue (which told us how quickly we were hitting the Apple/Google APIs), and by putting a handful of test accounts at the very beginning and the very end of the queue, so that we could measure on those test devices how long the whole process took.

Were there any lessons learned or key takeaways from the Superbowl ad project that might inform future endeavors and collaborations at Duolingo?

PAUL: I think a lot of the success of this project can be owed to how marketing, product, and engineering were aligned on a mutual goal very early on in the process. This allowed us the time and space needed to prototype and to iterate on our system over several testing cycles. Another thing that made it successful is that we put a lot of emphasis on our risks early on – very vocally, very publicly. This gave us the capital we needed to validate every assumption we were making about how the system would ultimately work. When you have such a slim margin for error, it’s essential not to leave these types of things unresolved. 

Missed the mid-month PM & Tech Jobs Newsletter?

Looking for a new job? Our PM & Tech Jobs newsletter is issued monthly with product role job listings from senior to entry-level roles.