This is a post about my main project this summer at Khan Academy, both the technical and non-technical aspects.
The project revolves around enabling Khan Academy to send more emails to its users. These emails come in two varieties:
Campaign emails. Monthly newsletters, emails to our teachers, big news announcements, or alerting folks about big changes to the site.
Transactional emails. One-off emails sent on an individual-user basis, like a “Welcome” email you get when you first sign up for Khan Academy, or a notification email for getting a new badge, or an email with customized video recommendations.
But don’t people hate emails?
Not unless they provide important value, like the examples above. And like most non-Facebook websites, Khan Academy struggles with user retention…lots of people sign up and stop visiting the site after a while. Email is a tool to help keep these users engaged and coming back to the site. More visits to the site means more net learning for world. That’s what motivates the Growth team at Khan, and setting up infrastructure to send better emails was high on the to-do list.
Step 1: The hunt an email service
First step in sending good, professional emails—get a service that handles the hard part for you. Previously, Khan Academy used a combination of Sendgrid and Google App Engine’s built-in email server. In addition to meaning that only engineers could write emails (which might be a very bad idea in itself), it also meant a lot of basic features required a lot more technical labor.
Enlisting the help of an email marketing software solution would provide certain key benefits:
- An interface for non-coders to write and design emails, freeing up developer time
- Tools for segmenting users (i.e. send this email to all teachers who joined 6 months ago)
- An A/B testing framework to test subject lines, “From” emails, and email content
- Tracking open rates and click-through rates on emails
- Deliverability assurance: making sure emails don’t get flagged as spam or bounce while sending to 6 million users at once
Picking the right solution was a lot harder than expected. We looked at services of various shapes and sizes, including Mailchimp, ExactTarget, Sailthru, customer.io, Marketo, and others.
Most services fell into two camps:
- start-up email companies providing email-as-a-service to other technical start-ups. These services usually had strong and well-documented APIs and were more flexible for small, agile teams of developers.
- enterprise email solutions, meant for large corporations with sales teams to handle email marketing. Thus, these services were often very good at handling large volumes of email (i.e. to millions of users), but were less developer-friendly.
The challenge? Khan Academy is a start-up, with an agile team of developers. A solid API is a must. At the same time, our user base is on the scale of millions. Thus, none of the small solutions were optimized (both from a technical and cost perspective) for our needs.
We had lots of phone calls and meetings, and we eventually decided to pick one that best balanced between those two needs. We settled on Sailthru, and we are giving it a run now.
Step 2: The integration process
Now to integrate the Khan Academy app with our new email service. This involved a few things:
- Adding every user that signs up for Khan Academy to the appropriate email list on Sailthru, by making API calls to Sailthru on user signup
- Inserting tracking code to see how much a user engaged with Khan Academy after receiving an email
Sailthru’s Python client library made this a pretty straightforward task. API calls to add users to Sailthru also return a cookie which we can drop on the client’s browser, so Sailthru knows which user is engaging with the site (connecting the two bullet points above).
Once this integration was complete, all new Khan Academy users were on Sailthru, and we could start sending them emails. As for the old and existing user, who needs ‘em? At Khan Academy, we only look forward. (Just kidding, that’s the next step.)
Step 2.5 A mini on-boarding email experiment
Before bringing in all the old and existing users onto Sailthru, we wanted to see what email experiments we could do with what we had, namely all the new users. Naturally, we decided on trying out a very basic form of new-user onboarding email campaign.
This campaign consisted of a two-part series of emails:
When the user first joins, a “Welcome email” directly from Sal 7 days after joining, they receive a follow-up email, also from Sal, highlighting cool content on the site
We did a simple A/B test, where half the users did not receive the 2nd email. We were testing if receiving that follow-up email made a user more likely to return to the website within their first two weeks of joining.
As expected, the follow-up email led to more return visits. However, it was not statistically significant increase, most likely due to small sample size and the short duration of the experiment. But we took our learnings, and the team is working on a more thorough version of this campaign.
Also, working with Sal to come up these emails was a cool experience. He stressed the importance of keeping a fun, light-hearted tone in the emails, to make sure the emails matched the culture of the organization of the email. Thus came things like referring to the user as a “knowledge sponge” and including a cute dog photo in the email.
Step 3: Importing 6 million
Time to take care of all 6 million students that joined Khan Academy in the past several years. Most email services have a way to upload a large list of existing users onto their platform, in the form of a CSV file.
We had to build that file. (Or in our case, many files, since a list of 6 million users requires much more than Sailthru’s 100MB file size limit.) Our default method of doing that was a mapreduce job over all of our user data. This would involve a hairy Hive set-up process, and a whole lot of waiting. But just as I was about embark that adventure, Kamens intervened with a shiny, new tool that might allow me to do this a lot more easily.
In came BigQuery.
Because one of our developers had recently backed up our entire Google App Engine datastore into Google Cloud Storage a few weeks prior, we know had the ability to use BigQuery (Google’s consumer offering of its super fast internal mapreduce infrastructure) to query that data.
Let’s just say that in one day, not having written any SQL before, I was downloading CSVs of our user data for import into Sailthru. BigQuery was fast. Because of that, it was also highly interactive, making it really easy (and fun) to try and debug all kinds of queries. In the process, I learned only Sal has more than 100 million energy points. (He has infinity. I guess he deserves it…)
Step 4: Send some emails!
With our users all into Sailthru, we were off to the races. Tom, who was coordinating the email project from the strategy side, had all kinds of emails ready to go and A/B tests ready to run. So far, we’ve sent a batch of useful emails to our teacher community, a huge announcement to all of our users about the new learning dashboard, and others.
In the pipeline is a more thorough onboarding campaign, a new content recommendation email, and some top-secret things you’ll see soon enough. :)