Personalized AI Language Education — with Andrew Hsu, Speak

First class. First class? Yeah.

Is that the one with SBF? No. He was, I think several years later, actually.

Okay. Yeah. Yeah.

What was it like? Just talk about the That's a good question. Haven't been asked that one in a while.

It was a really crazy idea at the time and very controversial. And I think the first few years of the fellowship were definitely let's just find 20 people in their 20 and give them a $100,000 to drop out of college. And it could be it was no holds barred.

You could do anything. You could be doing some crazy research idea, a startup, anything. And I actually met my current co founder at Speak.

He was in the second year of the fellowship and made many very close friends from the first few years. But I mean, for me, was life changing. I had a very unusual path where I was actually I did finish college unfortunately.

I was in grad school at the time because I went to school really early. Yeah. Was like, aren't you too old?

19 at the time and in grad school. It was a very accelerated path, but I think like I knew at the time that I was going to leave grad school and do startups anyway. And the timing lined up really well.

Yeah, yeah. Vitalik, I think. He was also in a later year.

Ah, yeah. Damn. Okay, anyway.

But the first two years, I mean, there are some crazy successes. Dylan from Figma.

despite our current revenue scale and objectively, I think how successful we are, we've always operated in a market, at least initially, on the other side of the world and been much, much more popular in the Eastern world and a bunch of Asian markets and relatively unknown in the West. So it hasn't really felt like we've had that sort of awareness until the past few years really. But brief story is that my co founder and I back in 2016 were fascinated by the promise of AI.

And we spent a year sabbatical basically learning everything we could. We talked to Karpathy back then actually when he was like just finishing grad school and did a lot of sort of self study research. And we were just so convinced, I think fundamentally, that speech models were going like this, language models were going like this.

And in the five to ten year span, they would become superhuman. And we were utterly convinced of this future and we saw that the way people learn things and specifically learn languages which was a very sort of human based thing if you really care about fluency, that would completely change and we'd be able to build language shooters that were pure software, pure AI. So that was kind of the Genesis story of Speak.

It took much much longer than we expected to build a great product and find good PMF. The first few years were very painful. And I think without this really compelling vision of the future, we would have quit.

We actually like never pivoted. Last year, we brought the entire company to Taipei. We do this company trip every year.

And we played our original YC application video on screen. And it was really funny because the things we were saying in that video were the exact same things that I still say today about the long term vision and what we're building towards. So that was really cool to see.

Can you summarize the long term vision again?

act of the company, if you will, was before LMs, before 2022 when Whisper came out, when ChatGPT came out. In the years before that, you know, like roughly two to three years is when we feel like we found PMF in South Korea and then started growing still only in that market, still only teaching English. And we developed custom speech recognition models and users were speaking into the app all day.

So we had a ton of this non native English speaker data. And we would use that to fine tune models, understand our users better. We still do that today.

And it's important for us for the core recording loop in many of our lessons that it's extremely fast. So we're very latency sensitive. There's many other sort of product surfaces within the app today that are more LM powered where it's more open ended real tutoring where we actually give you feedback on what you said in the semantics and so on.

So that stuff is more like whisper powered, more LM powered.

more conversational. We wanted it to feel more like you were talking with a tutor and they were learning things about you and we would use that later to personalize the experience. Before that, we had a much more traditional app onboarding.

There's still a lot of open questions, interesting questions around what is the proper onboarding UX because a lot of people start using Speak and they're not in a situation where they can actually speak aloud. So we have fallback outlets and so on. But it's something we're super actively experimenting with.

Is there a structured output behind that?

Or was there one? We still don't know yet. The interesting thing is that in general, because it's speaking based, which is a much higher barrier than just tapping a multiple choice button, what we see is that install to sign up rate is a decent amount lower, but trial start rate is higher.

It's still something that we have an active experiment that that's running and we're trying to be super agile about testing many different sort of like formats of this. I don't think I have like the final answer yet. Yeah.

But I think the intent that really like vision that we're going for here is that as soon as you download the app from the App Store, maybe you see it in an ad, the first thing that the first interaction when you have like a fresh open of the app should feel pretty futuristic. It should feel like, okay, this is like the new AI native next gen way of learning a language to fluency. And that's kind of been always our ambition.

Like, we want to build something that wasn't possible before without LM and, like, AI technology.

And I'm just wondering if you had any insights there or is it literally just a state machine? We tried both actually. Yeah.

Right now, I think probably what you saw is a state machine, but I think that- trust the AGI.

I think that things should move in a direction where it's much more of a natural conversation. Yeah. There is a general sense of a goal in the prompt that you can specify.

And part of the hard thing here is all the guardrails. Right? Like, once you start talking about what you had for breakfast yesterday, right, and trying to be, like, antagonistic to the system, then things start, like, really going off the rails.

So for a bunch of these experiences, we're pretty careful about the fallbacks and we have a lot of evals around that. But I think where it should end up is just feeling like you have a quick three to five minute conversation with your tutor and then it knows a lot about you.

Like Yeah. So we we store what you're saying. We summarize.

In the experience, the way it works is the tutor will ask you some sort of question like what are your goals around learning English or the language? And then we will basically use a separate LM prompt to summarize. So it's not the full transcript for what you said that you see.

It's more of an abstracted, Okay, here's what you care about. And we think that's a better product experience.

level, this is actually a pretty new product category, AI powered language learning. And all of these apps that you mentioned, Duolingo, Babbel, etcetera, they're more of like the gen two of language learning. So like if you think about gen one was Rosetta Stone if you remember, right?

CD ROMs in airports. And then gen two was basically mobile. So you have these very casual massively popular mobile apps like Duolingo that I think the comp there is probably closer to a mobile game, something that feels productive, something that's very engaging, very gamified.

And Duolingo is very leaning into it, the gamification. And they've done an amazing job of that to be clear. Yeah.

They might be the world's best people at it. Yeah. And our view is that LLMs and AI now enable gen three of language learning, which is something that is very AI native, very focused on functional fluency, which is why we do all these role plays and let you practice Spanish by talking to your Uber driver.

We don't teach vocabulary and grammar, we teach sentence patterns and we try to get you to just repeat and drill and drill and drill almost like you're in a gym until it's automatic because that's what speaking is. Right? Like it has to be spontaneous and automatic.

In terms of the other aspects of the design though, we went through many, many iterations over the first few years of starting the company. This is kind of what I was mentioning about. It was like really painful in the first four or five years.

And in fact, the current version of the Speak app is not the first thing that we launched. We had something that we call internally the Red app, which was like a red app icon, still a similar logo. And it was more around packs of content instead of courses where you could sort of choose any topic that you wanted to learn.

It was from many different languages for learning. It was essentially like not a very directed experience and it didn't really work. It was free.

It was a very basic thing. But we, in 2018, tore everything down and realized that we had to really fully change what we were doing. And that's when we decided to focus on South Korea specifically on teaching English.

We built a bunch of new lesson types and we created our courses so that the experience is much more on rails. We realized people don't want to choose. They're already using some of their motivation on a daily basis just to open the app.

They don't want to make another choice after that. Right? Just tell me what to do.

Right? Like, you know, give me a big button and then I can tap it and just start a video lesson or whatever. We also pretty critically, I think, abandoned the free version and just went straight premium.

And we kind of sidestepped the motivation question that way because we knew that there were a ton of users that really wanted to learn English and were already really motivated. So we wanted to basically filter for these users. So they you know, like it was I wouldn't say there was one silver bullet.

It was kind of the combination of many learnings over three or four years. And then that started really growing in South Korea. And from there, I guess like phase two was really 2022 when LLMs came out and Whisper came out.

And that allowed us to go from this more supplemental speaking practice tool to more full featured language tutoring where we could use LM's like 3.

So we're now the biggest English app in South Korea. Yeah. We do billboards, big celebrity campaigns, that sort of scale.

Like, we're, you know, very popular there. I think, like, 6% of the Korean population has tried us. We're, you know, like, well on the way in a bunch of other Asian markets like Japan, Taiwan.

So the Asian markets are currently our mainstay. We also teach English in 40 more countries. We're coming to The US as well launching I mean, we we have Spanish, French life, and several more languages are coming this year.

That's a huge focus of the company right now. In terms of revenue scale, well over 50,000,000 ARR. It's a pretty simple business model.

It's like mostly consumer. Subscription. Yeah.

The b to b stuff is super, super exciting, and that's also growing really fast. And I think it'll be a really meaningful part of the business. When did you start B2B?

About a year ago. It would Okay. It kind of it was like very much a side bet slash experiment at first.

And then it just started working. And Of course, it's going to work. Yeah.

And now it's like, Okay, this is part of the future. Right? This is a real thing.

So that's exciting. What's the B2B race between learning language and real time AI translation?

Yeah. Yeah. Yeah.

So people always ask this. They're always like, what happens when the Babelfish comes, when the real time translation comes? And Babelfish is the hitchhiker's guide.

Right. Yes, exactly. The counterexample that I always have that I think is quite illustrative is, in German, the verb is at the end of the sentence.

Right? So if you're trying to do real time translation from German to English as an example, you can't actually make any progress on the English until you hear the whole German sentence and you know what the verb is at the end. So the minimum latency there is the full sentence.

And that's an example of the technical blocker for why it'll never be truly perfect. But also, I think besides that, if you talk to all of our users in Asia, they don't want a translator. The reason that they are trying to learn English is to make themselves a better person to connect with other people.

Like, they wanna be able to look you in the eye and speak English, speak the same language as you. Right? So it's actually like a very different thing.

I think what will end up happening is that we will build a real time translation feature into Speak and have it integrated into the learning experience. And also, like, there's always that human side. Right?

Like, I'm dating a Romanian woman. Yeah. His wife is trying to learn Italian.

Like, there's always that. Yeah. Absolutely.

Gonna keep happening. I wanna double click on Korea. Yeah.

I think it's, like, a very insightful, smart decision.

Was it obvious? Were you sure when you went into Korea? Was it just a test?

We visited a bunch of Asian countries when we were thinking about how do we relaunch things, how do we focus in. And we almost chose Taiwan actually. But I think it was a little bit serendipitous.

So our first employee is Korean and was my co founder's college roommate actually. When my co founder visited Seoul to check out the market, he asked SJ to come along as essentially a translator and to facilitate. And I think that just went really well.

And it was just very obvious from being on the ground in the market that Korea is pretty obsessed with learning English. And there is every human based solution possible. Right?

You know, like English academies, classes, skyscrapers full of classrooms, stuff like that. And our logic was basically, if we can really make headway and win this market that is chock full of these human competitor products and all these people that fundamentally care about fluency, then we probably have something pretty real and strong PMF that we could win other markets with. So that was the original logic.

And so far it's been working.

sitting in an office here in San Francisco operating with users in a market all the way on the other side of the world, it would not have worked without Sunjay. Have to give him a lot of credit here because we paid a lot of attention to the specific wording of button text in the app and local you know, like like localized strings. We had a lot of reports from users pretty early on that they were shocked that it was an American company.

Like, they thought, dude, right? Because you can always tell. There's always some weird wording or whatever, but there wasn't in speak.

And I think that probably had like a large sort of non tangible effect. Yeah. Focus attention to detail.

Tech Stack.

What were you rolling? You you just did ASR and there's no LLM. So BERT maybe?

I don't know. We actually had really no LLM component of it. So all of the content oh, yeah.

Another thing we did that I forgot to mention was we decided we needed to fully own all the content. So the way that we teach all in house, all sort of thought from first principles, we build this thing called the speak method, which is basically like a pedagogical philosophy around teaching sentence patterns that you drill and then sort of combine it to higher order patterns. And all of that was in house with our content team and our teachers.

And we built a lot of internal tooling to make this possible. There's just a lot of operational overhead, I would say. This is something we've struggled with to scale to many more languages, and that's like a big research effort within the company right now.

We're building a mobile product. Right? My co founder and I have always just loved apps and been big iPhone users.

So we cared a lot about the app being native, feeling great, being high performance. The DNA of the company was always consumer. Frankly, my cofounder and I had never worked in a real company.

I dropped out of grad school, had a few failed startups, and then eventually started Speak. And he had never worked in real company either. He, just startups in the past.

So we didn't know anything about enterprise, enterprise workflows or what sort of software real companies used. So I think, frankly, consumer was the only path. I don't think we could have done anything else.

We just didn't know enough.

addition to the market on the other side of the world, our first iOS engineer that we hired through a YC referral was in Slovenia. If you don't know where Slovenia is, look it up on Google Maps. But it's, you know, it's like a pretty obscure little country.

Middle of nowhere. Yeah. And then we needed to hire a back end engineer.

And one of his best friends was a great back end engineer, we hired him. And then this happened four more times, all in the same city. And then we were like, Okay, we should probably just open a physical office.

So In for Yes. So for several years, we had an engineering office in Slovenia. What?

And then a few people here in San Francisco. And we still do. Now we have 90% of our core product development team in San Francisco here.

Office in FiDi, we're really only hiring here. But for the first, like, several years, that, you know, that was like another very interesting sort of cultural aspect of the company, I guess. I think a lot of early stage founders have to do that.

This that's the only people they can afford or whatever.

price thing.

we were good at remote work. I don't think that my personality or my co founder's personality is inherently very good at async just to be perfectly frank. I actually think that almost like in spite of it, we made it work.

It was a little bit brute force. Like I would just sync with them every single day, And there was pain because the time zone overlaps, like it was like exactly the most inconvenient. Yep.

But I think for several years we did that. We got really good at the cadence of it. I think they were excellent engineers as well.

So it worked out. But if I had to do it over again, I probably wouldn't do it. It's hard to say, yeah.

Should we move to phase two on the LLM side?

So that was also when Whisper dropped and Whisper was a really exciting moment for us. It was actually since we started the company and made that prediction of, okay, in five or ten years, speech models, language models will become superhuman level. Whisper was really that magic moment for us where we were like, oh shit.

I think what we predicted is here. And I pretty distinctly remember this moment in the office when we got access to the model and we were testing it on an audio clip of like a very beginner English learner in Korea saying something. And it was, if you close your eyes as a human, you have no idea what they were saying.

There were four of us in the room, we all closed our eyes and none of us had any idea and the model got it right. So I mean, superhuman. I think that was the moment that we had been waiting on.

And at the same time, LMs were on the Ascendancy. ChatGPT would come out, I think on 2022. And 3.

5 turbo came out. And I think like we kind of realized very quickly that all the pieces were clicking now. We have what we need at our fingertips now to go from something that was listen and repeat, where the user would see something on screen, hear a reference of the teacher saying the thing, and then they would just repeat the thing.

Right? It was like very simple. Still a great product by way.

You know, still grew to like several million ARR in South Korea. So clearly, is like a big marketing for that. Pre Whisper.

Yes. Wow. This is from 2019 through 2022.

And again, I think that if we there were many moments when things weren't working from 2017 through 2019. We were looking in the mirror and we were like, why are we doing this? This is this is crazy.

But I think we were so convinced about the vision. We just like couldn't believe that the vision would not Yep. Come true.

So we stuck with it. So fast forward to 2022, the pieces started coming together. We realized that we could start building something that felt more like a language tutor that could give you feedback, that could start explaining to you why you did something wrong.

positive for us. We still kept using our custom ASR system because it was streaming real time, really fast, really well fine tuned. Whisper wasn't streaming, so it was a different use case.

We used it for the more spontaneous stuff. And I think in almost every way, we were just really excited because pretty directly as the frontier of model intelligence improved, it would just unlock things on a roadmap that were locked before, if that makes sense. And we still really operate in that mode today where we take a model and then we try to think about, Okay, how do we saturate model capability by building product on top of it?

And then it happens again. Right? And then we build and saturate the model capability again.

I think that's a really cool paradigm to, like, you know, think about. But all the LM stuff basically allowed us to build a tutor for English, and we still didn't have, like, real time voice, for example. Right?

But the barriers are coming down now. Obviously, it's a really hot topic. We're actively building out a real time voice platform that we can build a lot of more verticalized specific lesson experiences on top of that I'm super super excited about.

I don't think they're going to replace our current lessons. They're going to be more immersive, just a different thing probably for more advanced learners. Still language learning though, not broadening out for language.

Yeah. So I think that language learning is interesting because it is so universal. 99% of people you know have certainly tried to learn a language.

And it's so hard. Right? Becoming fluent just has a huge failure rate.

And it's something people are willing to pay for. So I think that has been just like a pretty amazing beachhead for us. And I think we'll be doing language learning for a long time.

There's a huge, huge, huge company to be built here. But our even longer term ambition is really this idea that even beyond language, we think AI will reinvent how people learn anything. It already has for me.

Right? I use ChatGPT to learn things every ten minutes. And I think I'm just naturally like a very curious person.

So whenever I'm thinking about something, I want to know more about it. And then I'll naturally go to TatchGPT and then I'll learn about it. It's unlocked this like entirely new dimension of learning.

And I'm spending way more time learning as well as an adult, which is really cool. And I wanna bring that in a more structured systematic way to everyone.

any other major pieces of the puzzle that really unlocked it for you? So there's a few things I can talk about. I think one thing is in order to go from teaching English to teaching a bunch more languages, we needed to really figure out more direct AI content generation.

That was a pretty right? Because it's hard to scale like our little studio in LA where we shoot a lot of the video lessons. All of the scripts were written manually before by our content team.

But we want like a 100 x more content. Right? And 10 x more languages.

Eventually, a 100 x more language pairs, which is how we think about it. It's like, what's your native language and then what language are you learning? And really, the only way to do that is to make it more AI generated.

And very much like a AI native company, we want to be on a frontier here. We want to keep a small team and to have as much leverage as possible through these types of tools. So that's a big active area where we're building out.

I think using people overuse the word agent. But we have a tutor agent. We have a curriculum writing agent.

We have a giant LM based pipeline that creates curriculum, scaffolds it in the right way, writes the lessons themselves. That's a big active area that will basically help us to scale to a lot more markets and a lot more languages. So that's like one big thing.

Another big thing is we care a lot about fluency obviously. Specifically, we want to be able to quantify how fluent you are. So if you're learning Spanish, it's like, okay, what what does it mean to be fluent?

Right? There's a real world test for that. We care about real world fluency.

Your ability to go to Mexico City and go to a street taco stand and actually order, right? That's very functional fluency in one aspect. You might be really good at that but be completely unable to like talk about your family, right?

So the frontier of fluency is very jagged but we're very pragmatic and we care a lot about meeting user goals and helping them become fluent at what they care about. And we're thinking a lot about, okay, how do you quantify that? How do you actually store a knowledge graph of everything you know about Spanish in terms of the vocabulary you know or you don't know, the patterns that you know or you don't know, the mistakes you made using Speak over the last month that are clustered.

You said the magic word of knowledge graphs.

Is that experimental? There are aspects of it that are live. And it's a very multidimensional system where we think of it as there are many aspects of fluency.

There's many subscores. And we have a few of them that are currently live and we're actively developing other aspects of it. And then all those will fold up into a more holistic fluency score.

The idea is that eventually once we have a complete enough picture, everything will fold up into a number that we call the speak score that is a very sort of holistic measure of just like how good are you at Spanish. Right? And obviously 54 is kind of meaningless by itself, but it does give you a general sense.

Right? Like being at 54 versus being at five is very different, right? And I think everyone can kind of like intuitively understand that.

And it's surprising like I would have grounded it more in real world. Like we will get you to pass this exam that is a standard that is like the ESL standard or whatever. So the way that we think about that is we don't really teach for the test.

I think it's possible in the future that we'll do a test prep product. But in general, we care about real world proficiency in various functional situations. So the way that we think about it is if you're at this level, then these are the things you can do.

which doesn't really help you if you're traveling somewhere. That's new for Chinese because my parents spoke Mandarin to me growing up. So I can understand a nontrivial amount, but I'm very bad at speaking.

how do you figure out the courses are going to be good? Rely very heavily on our content team. And we are trying to build out an eval suite.

It's really hard. The illustrative example here is that as we try to hire and train new content writers on our content team, It's so nuanced. There's many different aspects of training them in the speak method and how to write the right types of lessons and articulating why this form of lesson, which is subtly different from this other form of lesson is better.

Right? So we try as hard as we can to articulate that. So I think forming a sense of evals using model graded evals like that, that's one piece of it.

And I also think in the future, a really good curriculum or lesson writer agent will probably be reinforcement fine tuned on a lot of our internal data as well. That's something we're experimenting with, but it's still pretty early.

our content team members, but also I think it's perfectly applicable to engineering is that it's it's leverage. It just allows you to do 100x in the same amount of time. We still need human review of the syllabus, the curriculum, the specific lines, etcetera.

But the hope is that this will allow us to launch 100x more courses.

Entirely. That's one of our fundamental tenets, which is that we don't teach textbook English or textbook language. Like we try very hard to Teach Gen Z slang.

We don't go quite that far, but Slaking. We try to teach very casual conversational language that is actually what real people use. And like you said, that's usually very, different.

Like if you pick up like a typical English textbook in Korea, it's all really traditional and weird formulations.

for example, currently, we teach American English, standard American English. We don't really teach other accents or other dialects. For now, given how small we are, we just have to be pragmatic and teach in the direction that most people want and most of our users know.

So we've made those decisions like on the content team side for American Spanish, every language that we're teaching. But I do expect that in the future, we're gonna get a lot more sharply differentiated. Like if you wanna learn British English, then we'll teach you British English.

We'll teach you how to pronounce it, etcetera. I think all of that feels like something that superhuman language tutor should be able to do. I just think it'd be very funny if all the Koreans had like a very distinct Southern accent.

I have two main thoughts on this. I think the first one is that communication and your ability to speak spontaneously and get us on a concept across, an idea across is almost fully orthogonal to pronunciation. You can be really bad at pronunciation but still communicate effectively.

So a lot of the current core project experience is about just speak as much as possible, make mistakes, don't worry about screwing something up on the accent or the pronunciation side. Yeah. The important thing is that you literally move your mouth and you make the sounds.

Right? And it turns out there's like a really key psychological barrier there where people are just not willing to do this in front of a human, even if it's a human that is a teacher that you're paying. Right?

So a lot of the core message of our marketing campaigns in many of our biggest markets is along the lines of like you can make mistakes in this private space with And I think psychologically, that's extremely powerful. And then you can go and get it right more confidently in the real world after you practice with Speak. Now having said that, people do care about their pronunciation and their accent.

Right? So we we have for English only right now, a pronunciation coach that is basically like a fine tuned version of Wave two VAC, which is a meta model. But we basically fine tune it on a bunch of our own phonetic transcripts like fine tuned data.

It works pretty well. It's currently for single words. We're going to expand it to full sentences, to more languages, etcetera.

is that the right product?

So for Speak for Business, the B2B product, another part of the vision is really like, what should a superhuman language tutor be able to do? It should probably be able to handle kids as well as a Samsung employee that wants to transfer to The US office and wants to use Speak for work. Right?

So our view there is that it's the same product. It's a different distribution mechanism, consumer versus B2B. And I think that we will eventually build something like a Mac app.

Maybe it'll be integrated with the browser in some way. We're not really sure yet. But obviously in order to apply it to your day to day, there needs to be some way to hook into your actual sort of work documents, whatever.

That's a whole can of worms. We are actively thinking about it. But I think my sense is that it's not clear to me that any of these products have really taken off.

And I think that there's many other approaches that are possible. I don't have the answer. But like another example, very hypothetical future world is maybe OpenAI, the new Johnny I thing, will come out with some hardware that will be listening to you all day.

Probably that's not true. What we've done so far is part of the course experience is a teacher video. We've tested other more audio forward types as well.

We found that, of course, like you said, video is very engaging. But at the same time, we have a lot of users that do want to be able to walk around with the phone locked in their pocket. So doing something that is more like voice mode with optional visuals, I think, is really good.

I think there's huge opportunity for a better way to learn things like listening comprehension. So I took German in grad school for two years and I thought I was getting somewhere. But anytime I listen to a native German speaker, it's so fast.

It's completely on a different level. And I think you can imagine a plethora of really cool experiences that feel kind of like you're listening to a podcast. But it's all AI generated.

It's fully controllable. It's integrated with the app. There's something there for sure.

Yeah. Don't want to do AI podcast, man. We're cooked.

Zoom out a little bit, in the pretty near future, multimodal models will cross the threshold where they will be able to generate images a lot faster than they currently are, maybe somewhat close to real time even, right? And audio at the same time, text at the same time. And you can imagine just like a very powerful multimodal tutor that can kind of do it all at once where there's an audio track.

And then if the teacher is teaching you something with the right timing, it chooses, okay, at this point, I'm about to introduce a new concept. So I'm going to show the word on screen so that the user can see how it's spelled. Right?

There's a lot there. You can do generative UI. A lot of nuance there where it's easy to do it badly, but to do it well requires a fair amount of reasoning and mental modeling of what the user knows.

Yeah.

Right? So we're just scaling and trying to hire as fast as possible like everyone else is. And as a result, we're really focusing on just like the most in reach, highest impact things.

I do think that the barriers are coming down very fast for all of this sort of stuff. I'm just so excited about multi modality and where things are going here because imagine if you're learning Spanish, being able to look at an image that the model generates for you and then doing Q and A on it, right? Like a beach scene.

And then the model will ask you like, oh, how many people are running on the beach? And then you have to sort of respond in the target language that you're learning. Very traditional language learning exercise, but you can imagine it being fully generative, which is really cool.

Awesome. Lots of stuff like that. The engineering in me worries about inference costs, but I think you can just sweep that under the rug.

And you build a bunch of tooling around it? Yeah. So we, last year, had early access to the Realtime API.

And there's a very obvious use case for language learning. I think one common theme that has just been pretty awesome since LMs came out is that language learning as an application is just a really good fit for LMs, all these model types in almost every way, which has been just really great for Speak specifically. For real time, I think the audio piece promises to really infuse almost every surface in the app.

You can imagine this is the primary way that you talk to your tutor. And an additional complication is that it needs to be multilingual and there needs to be code switching. So that's a pretty frontier problem right now.

Right? So like I should be able, if I'm learning Spanish, to speak both English and Spanish and vice versa from the model. That's a pretty hard TTS problem today.

a subword in a single sentence in a different language. So you can't just concatenate either because it won't sound right. It won't sound natural.

That's not how humans do it. So this seems to be like a very native controllable audio, you know, function. Yeah.

But we are in the process of building a variety of experiences on top of the real time API. I wanna clarify that actually nothing is in production yet, mostly for price reasons, frankly. The pricing model of the real time API makes more sense for something like a customer support agent where you're very directly replacing somebody that you would pay hourly otherwise.

And that's how you're seeing the price model for a lot of these initial agents work out. For us, we want our users to be able to do these real time role plays and have these conversations for many hours a day, right, if they want. Getting cost under control is definitely a pretty key consideration right now, but we are pretty close.

Maybe actually even by the time that this episode is released, we'll have something live. But we have what I think is a really cool application of the real time API, which is basically a new instructional lesson where it's the model actually teaching you something, like a new language concept. And it's intended to sort of augment slash play the same role as our current video lessons, which are the instructional lesson type.

And it's interactive obviously at certain points in the three to five minute lesson. You're interacting with the real time API. It's semi on guardrails.

There was a lot of scaffolding we needed to build to basically, number one, switch between the interactive and non interactive portions of this lesson properly, if that makes sense. Right? There there's there's some portions where you're just listening or looking.

And then some portions where you're actively in a short conversation. And we kind of swap back and forth. And we have like a bunch of sort of custom architecture and infra around that.

And then there's also making the cost make sense or at least like semi makes sense. And then there's a bunch of WebRTC infrastructure. We're at sort of not huge but nontrivial scale either.

So we definitely just it'll cost us millions of dollars if we do something wrong.

have been increasingly paying attention to for all the real time pads. Like I would say two or three years ago when real time stuff was still quite nascent, users didn't really care as much. But I think now the standards have risen.

Right? Like, latency has to be low.

from the real time engineering side, everyone talks about, Okay, submit user requests to get agent audio response, like first bytes, first audio bytes. What's that latency? And then we try to get that as low as possible.

I would argue that's actually like a vanity metric because what you don't take into account is how the VAD works. How do you do turn detection to detect when the user has finished speaking? Right?

Because that can easily add like another second if you do it badly. Nobody talks about that for some reason. Right?

Like, what you need to measure is actually when does the user stop talking to when does the model first audio come. And usually that number is much larger. That is a very domain specific problem.

You can use, like, the semantic VAD on real time API for regular English conversation, and that will basically classify at every token how likely it is that you're done speaking as a sort of normal conversational English speaker, like in this conversation, that's fine. But it doesn't work at all for language learners. If I am trying to respond in a language that I'm learning, I'm going to be hesitating halfway through for ten seconds for more.

Right? So it needs to be fully custom probably. This is something that we're also actively working on.

But that is actually the dominating factor in perceived latency.

the CTO, I view it as part of my responsibility to really set expectations, push everyone on the team, show them what's possible. We've been trying everything. Yep.

And I think we tried to basically set the expectation that the frontier is moving so fast, it's deeply non intuitive. Mhmm.

try to set a culture in the engineering team where usage of these tools as much as possible and as a default path is the expectation. And in hiring, we are now explicitly asking about this a lot, thinking about what are the types of people that are going to be better higher agency at trying these types of tools.

foundational pieces that we're building around knowledge graph, for example, a lot of these concepts should be applicable to not just learning language, but also other things in the future. We're already starting to see the very beginnings of this on the b to b side where a lot of it is more like management skills and hospitality skills, communication skills, more like true L and D for enterprise, less like core pure English proficiency. So I think you're you know, that's like obviously immediate neighborhood.

But you can imagine many academic subjects, math, biology, etcetera, know, schools work for.

We are approaching that problem from a few different angles. I think part of it is knowledge graph. Part of it is being very careful in how we structure the curriculum so that you're placed at the right level so that the learning path itself, which has a foundational backbone.

Because beginner to intermediate English learners actually like all need to know a bunch of similar concepts. It isn't really until you get intermediate and more advanced where that starts to like more sharply diverge and a zero through b one, I would say, there's a pretty well defined like sort of linear path actually. A lot of the deep thinking that we've done around how do we structure the pedagogy is also super useful in terms of just matching people to the right level.

Chat.com. Okay.

Yeah. We bought it several years ago. It felt very expensive for us at the time.

It was a little bit of a crazy move, but I think we were very convinced that we needed a super strong consumer brand that was scalable globally. And that was just always our ambition. We want to be the way the next billion people learn languages and we need speak.

com.

We have a bunch baking right now, but I think it was, you know, some something more generally that has just been so fun on the journey. So we we would visit Seoul every year. Yeah.

And seeing Speak go from nothing to the first time we saw somebody on the street using Speak to now our main teacher in the app is like a mini celebrity. People come up to her on the street as she's just walking around Seoul and recognize her from the app, which is really cool. Now we do a lot of advertising.

We do billboards, TV commercials. We work with big influencers and so on. So just like seeing the scale of that has me kind of like in awe.

I've always been interested in learning and education. One of the other field startups that I did in that time was it feels silly to even talk about this because amounted to nothing. But it was called Bloom, the Bloom two Sigma problem.

It was actually named after that. And we were trying to basically build like a better adult learning platform and have really cool interactive JavaScript widgets for various concepts that you could learn. Didn't find PMF.

I was young and didn't really know anything about business at the time either. But I think that the common thread through actually like everything that I've been interested in since leaving grad school has been how do we build software, build tools that help people learn things more effectively and better and faster. And now I feel very lucky to be in this position because obviously AI is the ultimate version of that.

Right? And it's been completely transformative for me personally because I just get a lot of just inherent fun and pleasure out of being able to like think of a concept. And then, oh, now I can talk to this omniscient LLM that can tell me more about it.

And I'm really good at asking the right follow-up questions that I want to know. So that's been completely transformative for me. Do you get a lot of people using Speak for therapy?

In 2023, when we first launched our AI role plays using GPT four, back then, people were way more concerned about safety. Right? And obviously, the models now are much better at refusals and the line is sharper between what's appropriate and not.

But we did see a lot of our first users start to put in pretty questionable custom scenarios. You probably guessed. And this was something we expected, but I think seeing the logs in person is like very different.

Got it. Some shocking stuff in there.

Thing that I think we didn't really realize early on or fully internalized at least was just how deep the market is. Say more? It was so universal where we really struggled to do some of the basic startup stuff around define your ideal customer profile and segment your users because our users were everyone.

We had parents using it with their kids. We had really old people using it. We had people using it for work.

So that was kind of like mind boggling. You still did customer segmentation or are you saying it doesn't matter? I'm saying it was hard to do.

Like, we tried and we have a sweet spot in Korea. It's like 25 to 45 more professional, more white collar. But it's very it's like like a very long tail on either side.

Yeah. Think it's, you know, it's a huge market. And I think it's a very special moment in time right now where it's obvious that a lot of the tech is here.

I think it's really good for humanity if we make a lot of progress here. So I'm really excited for his company too. We started asking about the Thiel Fellowship.

So maybe we can wrap with one of Thiel's favorite questions, which is, was something you believe in today that most people would not agree with you on? I think that people, if you recall, expected the world to kind of explode when GPT-four came out. And everything would change.

And I think if you like go to another state outside of the Bay Area, probably even in California outside of the Bay Area, and then you ask somebody how much their life has materially changed, it's like pretty close to zero. Real world inertia is enormous. Obviously, AI is probably the most transformative technology we've ever built.

But I think in a very real sense, the world hasn't changed that much either. And that's a really weird thing. Right?

So I think we need more builders. We need more people building applications. It's weird to me that Speak is actually not that many net new consumer AI native applications at scale.

There should be way more. I would love for there to be way more. Consumer is hard.

Yeah. I'm intimidated.

so hard to predict. Yeah. Sometimes I get AI anxiety.

And then I just You get anxiety? Yeah. Okay.

And I just focus on our users. That's a perfect place to wrap. Thank you so much for taking the time.

Yeah. Thank you both so much. This is great.