I’m currently sitting on a super comfy gray chair, listening to folks argue about who is or isn’t a spy… That’s because it’s Thursday night–which means it’s weekly game night here at the Khan Academy’s Mountain View HQ. I’m sitting out for this round of Resistance, a fancier version of Mafia. A lot of other games are happening, too, along with some great conversations, amongst the 20 or so Khan Academy employees and interns and a few of their guests.

I’m a Software Engineering Intern at Khan Academy for the summer, a job I had wanted for over two years. It still hasn’t quite sunk in that I’m here. When Salman Khan gave a talk at Berkeley two and half years ago during my freshman year of college, I fell in love with Khan Academy. I wasn’t the only one, of course. Khan Academy has gotten tons of media attention in the past few years, much to the delight of the thousands of people eager to read and hear more about the incredible story of the organization.

I, myself, was most mesmerized right after hearing Sal share his story that night at Cal, about how a simple tutoring gig with his younger cousin Nadia turned into a “free, world-class education for everyone”. The genuineness of Sal’s narrative, the purity (and audacity) of the organization’s mission, and the demonstrable impact it was having on students all over world, made for an entrepreneurial tale that was beautifully refreshing and totally inspiring.

That’s why, when they said they were hiring Software Engineers for the summer…I was almost in shock. You mean, they were hiring people like me? At that point, I had no clue about the interactive exercise framework being built and how core engineering was to the organization’s efforts. Yes, folks, Khan Academy does more than make videos. Much more. But anyway, I was ecstatic.

Too bad I was a freshman who had only been programming for less than 8 months…and that the only programming languages I knew were BYOB (Berkeley’s visual programming platform) and Scheme (an awesome language, but unfortunately rarely used in practice). But, I decided to apply anyway. But at that point, it was May…way after companies usually finish their hiring. Combining that with my lack of experience, let’s just say the odds weren’t totally on my side.

To make up for it, I put the most amount of time I’ve ever put into a job application. I made my own Khan-style video with “three reasons why Khan Academy should hire Aatash Parikh”, and put it on a page that resembled an official Khan Academy video page, replacing the comments section with ideas I had for the organization and some of my qualifications. I spent 20-30 hours on that app, when I should have been studying for finals. But it worked: Ben Kamens emailed me the next day, requesting a phone interview. A couple of interviews later, I would find out I didn’t get the job. Looking back, I’m not that surprised, seeing the caliber of the interns that did end up working there that summer.

But that was just the first time I applied. It would take me two more to finally get the job, and here I am, summer after my junior year. Now, I am not only a much more experienced programmer, but I have also spent two years immersed in the online education space. I have worked on projects, gone to events, organized events, and am involved in creating a MOOC-version of a popular Berkeley course. When I come to Khan Academy now, it’s not as a fanboy. This summer, I plan to use my skills and my experience in this space to really help take Khan Academy to the next level and further fulfill its potential to change what education means in today’s society.

Of course, there’s still a bit of fanboy in me. I get to work with a modern day celebrity in Sal, which means things like visits from Elon Musk and a collab with Lebron James. Yeah. And the rest of the team members are some of the smartest people in the world, celebrities in their own right. In my first two weeks, my mentors Stephanie Chang and Desmond Brand have been nothing short of awesome–the right combination of instructing and empowering, along with being great role models.

Plus, I’m working at the organization that sparked almost all of the innovation happening in education technology in the last couple years. And don’t worry, there’s a lot more to come.

It is currently a common misconception that Khan Academy revolves around videos. While the videos are core to the organization’s product and mission, Khan Academy–from its inception–has had a whole, separate interactive learning component to accompany the videos. After making the first few dozen videos for his cousins, Sal wanted a way to evaluate how well they were actually absorbing the concepts he was teaching–so he coded up a math problem generator. That problem generator has since evolved into 400+ exercises (as of May 2013), and Khan Academy has a large, full-time team working on this interactive learning platform itself.

Okay, so you believe me. Khan Academy does more than videos. What’s the big deal? The big deal is that Khan Academy is gearing up to be more than just a good-to-have supplementary educational resource for students needing a little extra math tutoring. It’s aiming to be a truly comprehensive learning platform…and that’s going to take a lot more than just videos. Yes, the videos help create a strong conceptual understanding of a topic. You can know absolutely nothing about a subject, and then go watch Khan’s introductory video on it, and immediately start grasping the core concepts. The more videos you watch on a particular topic, the more comprehensive your grasp becomes. But without attempting to apply that learning to an actual problem or exercise (and gauge your success at doing so), the learning is incomplete.

That’s where the interactive components come in. If you’ve seen Sal’s TED talk, it’s actually the stuff he starts talking about at 7:42.

And that was 2 years ago. Since then, the Khan Academy team has grown to 40 full-timers--not including the interns that are on board every summer (there will be 18 this summer). And the vast majority of these team members are engineers, building everything BUT the videos on the website. Things like: interactive problems for students to answer, dashboards for students to see their progress, tools for coaches and teachers to visualize strong and weak points in their students’ learning. There’s a LOT of work being put into this side of things by the Khan Academy team. And even though it may not be what people immediately associate Khan with, it will be very soon.

One of the most common questions I get about how Khan Academy operates is about Sal Khan and what he does these days. Surely, he doesn’t make videos anymore… After all, Khan Academy has plently of traction now–Sal can get other folks to do videos. He’s probably got more important stuff to do.

Well, that’s what I thought, too. Until last week’s team meeting, when it was Sal’s turn to give his weekly update. He mentioned a few big meetings he attended, some upcoming partnerships, and then talked a bit about the Common Core. And speaking of the Common Core, he said, he had just made NINE videos THAT MORNING to add to Khan Academy’s Common Core math collection. 9 videos. In one morning. And here I thought Sal didn’t make videos anymore. 

But the fact that Sal still makes so many videos (he made 30 that week) is interesting not because he’s the only one still making videos. In fact, he’s not: Khan Academy has hired several talented people to also make videos (including Brit Cruise, Vi Hart, Rishi Desai, Beth Harris, Steven Zucker and others) and is working on all sorts of cool partnerships for more content. 

The reason it’s interesting is that over the several years Khan Academy has existed, Sal has received countless opportunities to stop making videos and spend more time doing other things. Instead, he has chosen to continue making videos. His routine involves coming into work early in the morning and using the first half of his day to make anywhere from 3-10 videos. Once he’s done with that, he’ll attend any meetings, respond to emails, and do other management-type tasks. But the first thing he does in the day is make videos. It’s this deep prioritization of working on the core content of the site (and on his core competency–absorbing and distilling information and teaching it engagingly) that led to Khan Academy’s huge growth.

Think about it. What made Khan Academy so cool that people started talking about it and spreading the word? Not just that it just had a couple cool math videos. When people went to khanacademy.org, they’re reaction was “Damn, they have so many videos!” When Bill Gates found Khan Academy, he was awestruck by breadth of content on the site–which led to a chain reaction of attention that skyrocketed Khan Academy’s growth.

That was in 2009. Sal started Khan Academy in 2006. What did he do in those three years? Make videos, and more videos, and more videos. And an interactive math problem tool. And more videos. This is a fact lost in the glamour of the social entrepreneurial fairy tale portayal of Khan Academy’s success. It would not have gotten there without the consistent hours that Sal put in, in his closet studio, building up the library of content that would become the foundation of Khan Academy.

Basically, he put in the work. And he continues to do so. At this point, if Khan Academy loses all its funding, and we all have to find new jobs, it would not be surprising to me if Sal continued to make new videos. And when our grandkids stumble upon his elementary calculus video and have that “aha” moment…we’ll be glad he did.

This past week I deployed my first change to the Khan Academy website (other than my About listing).

Every video on the Khan Academy website now has a nifty little “Info” button underneath it. It pops up a dialog showing the name of the video author, the date the video was uploaded, a link to download the video, and the copyright information of the video.

Okay, so it’s cute. But it actually serves to do a few important things:

1) Higlight our “non-Sal” authors. Sal made the majority of videos on Khan Academy (and continues to do so). However, he is gradually receiving more and more help. In fact, Khan Academy has videos from several other team members: Vi Hart, Elizabeth Slavitt, Rishi Desai, Brit Cruise, Karl Wendt, Steven Zucker, Beth Harris. Now, users will be able to easily see who made a particular video, especially if they recognize that it doesn’t have Sal’s voice.

Important to note: the author’s name still takes an extra click to reach. You have to actually click on “Info” and wait for the tooltip to come up. Why not always display the author’s name right under the video? This was a deliberate decision: the “who” behind a video is still secondary to the “what” of the video. The content is what’s important, and we won’t distract you with auxiliary information about the video unless you’re looking for it.

2) Clear communication about licensing restrictions. We never displayed the copyright license of a video before. Except for a few special cases (where we made a note in the video itself), the general case was covered in the Terms of Service. That worked pretty well because most videos fell under the same license (Creative Commons Attribution/Non-Commercial/Share-Alike). That’s slowly starting to change as we get more 3rd party video content, making it more important to display license information on a per-partner basis. This information is planned to be available via the Khan Academy API as well.

It’s pretty awesome to deploy code used by millions of people. Let me know if you catch any bugs or have any suggestions.

I realized I’ve been at Khan for 7 weeks now. 

It’s awesome, but also sad–because I’m already halfway through. :(

In these 7 weeks, in the midst of all the coding and fun, I have tried to keep my head up a little bit and see how things work around here. And one of the most striking things is how much this company cares about its interns. It’s not an accident, that’s for sure. Khan Academy values its interns heavily, not only as future full-time hires, but as important contributors to the mission and to the culture of the organization.

It shows this value in several ways. Some of these are common in today’s best tech companies:

No differentiation between interns and full-time engineer

At Khan, just like at some of the other great engineering companies in the Valley, very few things distinguish an intern from a full-time engineer. For example:

• Shipping real features. Interns don’t work on temporary, bogus projects just for learning, that will be thrown out once they go back to school. They also don’t make coffee (unless it’s for themselves). They work on the same stuff the full-timers do…real features, seen by millions of people. Need proof? See Dylan Vassalo’s summary of last year’s intern projects.
• No social discrimination. Okay, discrimination is a harsh word, but what I mean is: interns don’t wear a big red stamp on the forehead screaming ”I’m an intern!”. They are just part of the team, like everyone else… Whether in a conversation at the lunch table or at a team event (like a recent BBQ), everyone is respected as a highly valuable member of the team.

A great development experience

This means things like great (free) food, unlimited drinks, a large monitor, and a healthy development abstraction that allows for digging deep into the code.

The part about the development abstraction is one of those interesting things that makes software engineering jobs awesome when it’s present. It’s why people love working at Google (other than the perks), and it’s why Ben Kamens loved working at Fog Creek Software (and why he made sure Khan Academy kept it up.) Joel Spolsky describes it best, and I highly recommend reading it.

A fun and quirky culture

We have weekly board game nights. We did an intern scavenger hunt. We just celebrated Tau day. It’s a fun place to be.

Khan Academy also does a few things that are less common in other companies, including:

Explicit (and implicit) mentorship

As Ben wrote about here, all interns are explicitly given a specific mentor on their first day of work. Thus, I have someone who has willingly made himself available to answer my questions at all times. I don’t know how I would have survived without this in the beginning of my internship, navigating a foreign codebase and learning how everything worked. But even now, it’s awesome to have someone I can go to for help when I’m stuck for sufficiently long on something. My mentor and I also have weekly one-on-one walks that are tremendously useful for hashing out any issues stuck in the back of our minds.

And that’s just the explicit part. I’ve had so many conversations with various full-timers who are some of the smartest people that I’ve met. I’m trying to soak in as much wisdom as I can before I’m kicked out in August. Sad as it is, I don’t learn nearly this much in school.

Involving interns in the conversation

At Khan, we have weekly team meetings led by Sal. We have regular “Big Ideas” meetings, where we discuss various philosophical questions, related and unrelated to the company. We have amazing guest speakers, advising us on how to focus the organization’s efforts.

And along with such events come conversations–chances for team members to share their thoughts on where Khan Academy is headed, where it needs to go, what our users care about, and what, in general, we are doing right or wrong. From what I’ve seen, interns don’t hesitate to chime in, and their thoughts are heard as loudly as anyone else’s.

A name we’re proud to wear

The fact that I’m writing all these blog posts about Khan Academy says something. I’m not the only one…several team members blog, and interns have done awesome reflections on their work at Khan.

I think a huge part of it is that we are proud to be at Khan Academy. It’s a noble mission that’s truly changing the world. And it’s a brand name that people recognize, which is undoubtedly a plus when you consider that an internship is an investment in your future.

And the changing the world part…it’s no joke. Even if you ignore everything else, and you put in the same room a bunch of people who care deeply about a single mission, you’ll still have something pretty darn good.

An expectation of excellence

The points above are undoubtedly great perks. But why would a company invest so much into these details? 

Well, it helps with the recruiting, clearly. But what it further means is that joining this organization comes with a high expectation of excellence. Khan Academy’s goal is to hire the best employees and interns…and then give them the freedom and support to perform. These things won’t turn the average performer into a great one, but it will make sure the great performer does his or her best work.

Also, these things don’t just magically show up in organizations. At the end of the day, someone has to care enough to make them happpen. That’s where people like Ben come in; he really believes in using his role as our lead developer to cultivate an environment where others can thrive. And Jessica Yuen was instrumental in establishing Khan’s culture and running the intern program for the past couple years. Kayla Bills, who’s since taken over, is doing an awesome job so far this year.

As I’m seeing now, building such a culture is craft in its own and it takes people like these who really care enough to get the details right.

When’s the last time you ended your team meeting with tears welling up in your eyes? Now, when’s the last time you worked at a company where this happened EVERY WEEK at the end of your team’s weekly check-in meeting?

Every Monday’s “Company Update” at Khan Academy ends with team members reading aloud 3 letters that we’ve recently received in the mail. One might be from a college student in Canada telling us how Khan Academy helped him get through calculus. Another might be from a 40-year-old mom in Minnesota discovering that she doesn’t actually hate math. The crazy thing is that all of them will have been from the past two or three days.

So where do the tears come in? Well, the impact Khan Academy has had on so many people’s lives is nothing short of astounding, and hearing them convey it in words is immensely touching. We recently heard from a girl who had dropped out of high school because of poor performance in math and also lost the faith of her parents. After discovering Khan Academy years later and building up her math skills, she came back to high school, graduated, and got accepted by Juilliard, one of the most competitive and famous performing arts colleges in the world. Hearing this story had everyone smiling. And for me, tears were close.

I’ve thought a lot about the power of stories and testimonials–in the two years I’ve worked with Ramit Sethi at I Will Teach You to be Rich, my entire job has been to organize the influx of testimonials that come into Ramit’s email inbox. He gets incredibly amazing stories as well. Both IWT and Khan Academy have helped me realize two big reasons why such stories can be so valuable to an organization (in addition to just being awesome to hear):

1) Social proof for your product

Especially in Ramit’s case, testimonials help drive sales for his paid products. For a customer, hearing the results that your peers have gotten with a certain product can be the reason you make the leap to invest for yourself. It’s been proven to make a huge difference.

Similarly, Khan Academy has a whole page highlighting some of its users’ stories. Khan Academy has no paid products, but even spending the time to try out a free tool is an investment in time and energy that becomes more doable if you hear how it has been worth it for others. This will be especially important as we continue to encourage more teachers to use Khan Academy in their classrooms.

2) They remind us why we do what we do

I wrote about the years that Sal spent making videos with very little public recognition. For a certain portion of that time, he had quit his job and had no external funding for his work. What kept him going during that time? He says it himself: the letters and emails of people all over the world saying how instrumental Khan Academy was for them.

It’s the same reason we end every team meeting with the letters from our users. I literally walk back to my desk with a tingling sensation and a crystal clear vision of how valuable my contributions are to real people across the globe.

Ramit Sethi has had all kinds of nay-sayers in the years he’s spent building his products and and his brand. Such critics can make you question your work and its value. But when you get an email saying, “Thanks to your recent course, I’ve gotten my finances under control, found my dream job, and recovered my marriage and my life.” – that’s when you close your eyes and say to yourself:

“OH YEAH. This is why I do what I do. Now back to work.”

I’ll end with this recent note from our stories page:

I really can’t begin to explain what seeing your site means to me. I’m a 36 year old man who 18 years ago wanted to be an engineer but as I progressed through school I got further and further behind and I dropped what I wanted to do in favour of what I thought I could do. As I watched your video explaining your vision I literally started crying because I saw that this is exactly what happened to me, the swiss cheese of the holes in my education added up to an insurmountable wall. I’m now on the way to becoming an engineer and I wanted to let you know that your videos have made a world of difference to me and mean that its possible for me to live a life I want rather than exist in a boring drudgery… wish you the very best, you’ve blessed me more than I can express…

Best Regards -Dave

See what I mean? ;)

P.S. It’s a double bonus when you meet someone in real life who uses and loves your product. That happened to me twice today. Knowing that REAL people thatyou know use your product and love it is excellent motivation to show up to work in the morning.

A couple weeks ago, I donated a little poster to the Khan Academy office:

I got it from here, and it represents YCombinator’s unofficial motto.

But the reason I put it up (other than the fact that it had been sitting unopened in my apartment for months…) is that it already seemed to be a philosophy practiced strongly at khan academy, namely on the engineering team. So I thought it was very fitting to be on the wall.

So what does it mean?

Well, like it sounds, it means building the features that your customers want. But more importantly, it means NOT building the features your customers DON’T really care all that much about.

It’s harder than it sounds. Too often, we as engineers want to build out a bunch of features we think are useful. Or even if we don’t, people ask for them or suggest them and they slowly creep onto the to-do list.

But often these features have been added or requested without the due diligence (i.e. talking to your users) required to confirm that it is something our users actually want. That it’s not just a nice-to-have for the one-off user, but something that solves a critical pain point or greatly improves the product for a substantial amount of users.

When you stop and ask the question, “Is this something people really want?” you might not only end up with a product vastly different from the one you imagined in your head, but you also might end up saying “No” to a lot of requests. And saying “No” ain’t easy. Especially when someone else on the team really wants a feature to be built.

At Khan Academy, I see every to-do item and feature request being questioned. Is this something the user really needs? Can we ship the product with this minimal thing that we know they need, and then think about this other (optional?) thing later? In other words, can we punt?

In a world where there’s too much to do and not enough time to do it, this focus on the essential goes a long way in making sure we get stuff out the door and into our users’ hands. And if we’re doing our job well, then we know it’ll actually be stuff that they want.

Last weekend was Khan Academy’s 2nd Annual Healthy Hackathon! It was a fun weekend, and the resulting projects were amazing. I was blown away by what built–from a new iPad app with support for interactive exercise to a Python version of our API to a full Blackberry app for Khan Academy to a full computer made of nothing but logic gates. The biggest difference I saw between this hackathon and every other hackathon I’ve been to (other than the healthy part), was that almost everything was built to be shipped. The fact that it was a company/internal hackathon explains some of that, but even so, it was clear that building something useful and/or passion-driven mattered more than winning the competition. I have seen the competion aspect of hackathons lead to a lot of flashy and hollow projects that are dropped after the event. Not so around here.

As for myself, I worked on a fun project with James Tynan, Matt Dunn-Rankin, and Julia Lee. Our idea was to use celebrity personalities to encourage a growth mindset in Khan Academy learners. How? We made it so that in the same place where you add a real-life coach on Khan Academy, you could now add an “Inspirational Coach”–anyone from LeBron James to Abraham Lincoln. This inspirational coach would pop into your learning dashboard every once in a while to either congratulate a streak of consistent effort or give encouraging message when you hit a dry spell. (Note: the learning dashboard is a new feature and you may not have it till Fall 2013)

For example, let’s say you add Abraham Lincoln as a coach. You might get a video like this (thanks to Sal for the Lincoln get-up and James for the excellent delivery):

Now, let’s say you’ve also added Lady Gaga as a coach. When you successfully complete three tasks in a row, Gaga (portrayed here by the awesome Eliana Feasley) might pop into your task queue with a congratulatory message.

Let’s say you’ve also added a famous athlete as a coach. (Yes, clearly you can add a lot of coaches.) And while doing a particular task, you struggle to get it right on several attempts, but then you persist and eventually complete the task. Our athlete might jump in with an encouraging story from his career when he too persisted through difficulties to reach success.

Finally, after some intitial success on Khan Academy, life gets in the way and you altogether stop visiting Khan Academy. Three months pass by, and one day, you open your email and are surprised to see an email in your inbox from none other than… Abraham Lincoln.

You click the link in the email, and then find this message from Coach Lincoln:

Man, James does a good Lincoln, no?

On the technical side, one of the key aspects of the project was adding video tasks to the learning dashboard, which previously only allowed for exercise tasks. With some help from roaming hacker Marcia, we were able to use some already-written code to push this relatively quickly. Then it was a matter of 1) registering when a user actually completed these videos, 2) adding these tasks intelligently based on the user’s activity.

The result was a pretty seamless experience of inspirational video tasks popping in and out of the user’s learning dashboard:

The next “task”: get some real celebrity footage, improve + clean up the code, and ship it!

The next next task: find other ways to incorporate growth mindset and other important soft skills into Khan Academy. Let me know if you have any ideas!

I’ve recently come to the conclusion that there’s no place in the world to work like Khan Academy.

Here are a few facts about Khan Academy:

• It operates like a lean tech start-up
• It’s working to solve a major world problem
• Salaries are competitive with Google and Facebook
• It’s prestigious on your resume
• It’s completely non-profit

None of the individual points above are too unique on their own. But how often do you find a place that has all of them? Yeah, I don’t think so.

Yesterday, we had the folks from Participant Media come and screen CBS’ upcoming television special, called TEACH. Sal and Khan Academy make an appearance in the film, and it was a testament to the impact and reach Khan Academy has been having in the past couple years.

Last week, Sal was on the cover of Costco Connection, reaching tiger moms everywhere.

Last month, Bill Gates visited our office, causing the interns to jump up and down, and reminding everyone that we have the most influential support system out there.

When we aren’t graced by the visit of an eminent supporter or by seeing Khan Academy in the news, we are hard at work building out the best learning experience in the world. It can be easy to get caught up in the glamour and press, and while the team here is totally inspired by the external goodwill and a mission that’s pie in the sky, it is also intensely focused on working on the next concrete steps to making that vision a reality. We know we’re only a fraction of the way there, so continually moving forward is the only option.

Various factors have come together to make Khan Academy so special. Some of it is circumstance and timing, and a lot of it is the genius and persistence of some amazing people. I’m so fortunate to have been able to spend a summer here to see it firsthand.

This is a post about my main project this summer at Khan Academy, both the technical and non-technical aspects.

The project revolves around enabling Khan Academy to send more emails to its users. These emails come in two varieties:

Campaign emails. Monthly newsletters, emails to our teachers, big news announcements, or alerting folks about big changes to the site.
Transactional emails. One-off emails sent on an individual-user basis, like a “Welcome” email you get when you first sign up for Khan Academy, or a notification email for getting a new badge, or an email with customized video recommendations.

But don’t people hate emails?

Not unless they provide important value, like the examples above. And like most non-Facebook websites, Khan Academy struggles with user retention…lots of people sign up and stop visiting the site after a while. Email is a tool to help keep these users engaged and coming back to the site. More visits to the site means more net learning for world. That’s what motivates the Growth team at Khan, and setting up infrastructure to send better emails was high on the to-do list.

Step 1: The hunt an email service

First step in sending good, professional emails—get a service that handles the hard part for you. Previously, Khan Academy used a combination of Sendgrid and Google App Engine’s built-in email server. In addition to meaning that only engineers could write emails (which might be a very bad idea in itself), it also meant a lot of basic features required a lot more technical labor.

Enlisting the help of an email marketing software solution would provide certain key benefits:

• An interface for non-coders to write and design emails, freeing up developer time
• Tools for segmenting users (i.e. send this email to all teachers who joined 6 months ago)
• An A/B testing framework to test subject lines, “From” emails, and email content
• Tracking open rates and click-through rates on emails
• Deliverability assurance: making sure emails don’t get flagged as spam or bounce while sending to 6 million users at once

Picking the right solution was a lot harder than expected. We looked at services of various shapes and sizes, including Mailchimp, ExactTarget, Sailthru, customer.io, Marketo, and others.

Most services fell into two camps:

• start-up email companies providing email-as-a-service to other technical start-ups. These services usually had strong and well-documented APIs and were more flexible for small, agile teams of developers.
• enterprise email solutions, meant for large corporations with sales teams to handle email marketing. Thus, these services were often very good at handling large volumes of email (i.e. to millions of users), but were less developer-friendly.

The challenge? Khan Academy is a start-up, with an agile team of developers. A solid API is a must. At the same time, our user base is on the scale of millions. Thus, none of the small solutions were optimized (both from a technical and cost perspective) for our needs.

We had lots of phone calls and meetings, and we eventually decided to pick one that best balanced between those two needs. We settled on Sailthru, and we are giving it a run now.

Step 2: The integration process

Now to integrate the Khan Academy app with our new email service. This involved a few things:

• Adding every user that signs up for Khan Academy to the appropriate email list on Sailthru, by making API calls to Sailthru on user signup
• Inserting tracking code to see how much a user engaged with Khan Academy after receiving an email

Sailthru’s Python client library made this a pretty straightforward task. API calls to add users to Sailthru also return a cookie which we can drop on the client’s browser, so Sailthru knows which user is engaging with the site (connecting the two bullet points above).

Once this integration was complete, all new Khan Academy users were on Sailthru, and we could start sending them emails. As for the old and existing user, who needs ‘em? At Khan Academy, we only look forward. (Just kidding, that’s the next step.)

Step 2.5 A mini on-boarding email experiment

Before bringing in all the old and existing users onto Sailthru, we wanted to see what email experiments we could do with what we had, namely all the new users. Naturally, we decided on trying out a very basic form of new-user onboarding email campaign.

This campaign consisted of a two-part series of emails:

When the user first joins, a “Welcome email” directly from Sal 7 days after joining, they receive a follow-up email, also from Sal, highlighting cool content on the site

We did a simple A/B test, where half the users did not receive the 2nd email. We were testing if receiving that follow-up email made a user more likely to return to the website within their first two weeks of joining.

As expected, the follow-up email led to more return visits. However, it was not statistically significant increase, most likely due to small sample size and the short duration of the experiment. But we took our learnings, and the team is working on a more thorough version of this campaign.

Also, working with Sal to come up these emails was a cool experience. He stressed the importance of keeping a fun, light-hearted tone in the emails, to make sure the emails matched the culture of the organization of the email. Thus came things like referring to the user as a “knowledge sponge” and including a cute dog photo in the email.

Step 3: Importing 6 million

Time to take care of all 6 million students that joined Khan Academy in the past several years. Most email services have a way to upload a large list of existing users onto their platform, in the form of a CSV file.

We had to build that file. (Or in our case, many files, since a list of 6 million users requires much more than Sailthru’s 100MB file size limit.) Our default method of doing that was a mapreduce job over all of our user data. This would involve a hairy Hive set-up process, and a whole lot of waiting. But just as I was about embark that adventure, Kamens intervened with a shiny, new tool that might allow me to do this a lot more easily.

In came BigQuery.

Because one of our developers had recently backed up our entire Google App Engine datastore into Google Cloud Storage a few weeks prior, we know had the ability to use BigQuery (Google’s consumer offering of its super fast internal mapreduce infrastructure) to query that data.

Let’s just say that in one day, not having written any SQL before, I was downloading CSVs of our user data for import into Sailthru. BigQuery was fast. Because of that, it was also highly interactive, making it really easy (and fun) to try and debug all kinds of queries. In the process, I learned only Sal has more than 100 million energy points. (He has infinity. I guess he deserves it…)

Step 4: Send some emails!

With our users all into Sailthru, we were off to the races. Tom, who was coordinating the email project from the strategy side, had all kinds of emails ready to go and A/B tests ready to run. So far, we’ve sent a batch of useful emails to our teacher community, a huge announcement to all of our users about the new learning dashboard, and others.

In the pipeline is a more thorough onboarding campaign, a new content recommendation email, and some top-secret things you’ll see soon enough. :)