#94 – Lifecycle, e-mail and some of the strongest growth levers today (Will Wong, Ben Grynol & Josh Mohrer)

Email as a marketing channel is here to stay. It can be a strong awareness tool and a way to keep users engaged and coming back to your product. In this episode, Levels Head of Growth Ben Grynol and Head of Global Operations Josh Mohrer talked to Will Wong, a partner at Andreessen Horowitz, the lead investor in Levels. Will shared his deep experience with email marketing from launching Disney+ to bringing in new Dropbox users. He shares his biggest learnings from his career and how companies should use email to reach their goals.

05:45 – Applying baseball vendor logic to marketing

Will used to work in professional baseball stadiums as a teenager selling cotton candy, peanuts, and ice cream. A lot of what he learned can be applied to performance marketing.

What happens is, there’s an unwritten code where if you see a vendor in the same aisle, you’re not allowed to go down that same path. And what happens is, around the fourth inning, everyone wants ice cream, especially in the nice seats, and so all the ice cream guys rush there, and it becomes a little game of which aisles do you go first, you’re eyeing all the other ice cream guys so that you get there before they do. And when you get there, you have to go figure out what your pitch is, what your call to action is, and over time, you have to figure out what actually sells the most. You figure out how long you linger, how long you go down aisles, how good you are at spotting people who are looking to buy what you’ve got to sell. And you’d be surprised that that’s almost exactly how early days performance marketing was. It’s just how do you optimize your CTA, what message do you use, what time do you target a certain ad, or target a certain audience, all that type of stuff. And I was actually surprised like, “Oh, hey, these peanut days really apply to my adult life career.”

12:14 – Minimize the risk of launching Disney+

Will worked at Disney during the launch of Disney+. Because it was going to be a huge launch, they stuck to the basics to keep it low risk.

And then you kind of jump into, okay, what is the actual email, SMS push, in-product alerts that you actually need to build out? What are the things that are mandatory? What are things that we need to do to really kind of tune our activation rates, engagement rates, upsell rates, all that type? And how do we actually just start from the ground up? At the same time, knowing that this is a huge brand and we’re going to have a huge demand early on. And I think what was really interesting about that is, we actually made a very distinct decision to not run any A/B testing in those first two, three months. We just knew that so many people were going to sign up. The most important thing that was in my head was, we have to de-risk the launch, because, yes, a lot of people want to test, “Hey, what’s the best way to optimize your…” At the time there was a free trial, like, “How do you optimize trial to paid? How do you optimize month one to month two?” But our systems also ran a lot of things that could actually blow up the entire sign up and launch day, and things like one-time passcode. This is a brand new email domain, it’s an IP that we had to warm-up.

16:15 – Be prepared for the end of free trial period

Will knew that a majority of users would unsubscribe after the 90-day trial period ended, so they got all the email messages figured out ahead of time.

So we wanted to make sure that we had all the messaging setup automated, so that… The situation we didn’t want to be in is, wait all the way until month four until we had now four monthly cohorts of data, and then realize, “Oh, hey, this is the month we need to focus on. Great, what do we need?” And then that was the time we started building, what we wanted to do is build all of that ahead of time, so that when we got to that point, the only thing we needed to do is really test messaging, test what the right recommendations were, test what content things, or offers that we needed to make at that point, so that all the systems were in place, as opposed to just waiting for…Maybe a better way of saying is, to be proactive with what kinds of programs we would need, versus being entirely reactive. And so the hard part is of course figuring out what the content and what the messages need to be, but we were pretty sure that we needed a user onboarding flow, a re-engagement flow, a win-back flow, and an upsell flow.

18:27 – Segments need to be distinct

Segmentation works best when the segments have distinct behavior patterns.

This is my biggest learning is, how do you implement segmentation? And my approach has always been, generally, I don’t care what the segments are, I don’t have a bias, but for them to be good segments, one, each segment needs to be distinct from another, and they’re distinct because their behaviors, and patterns, and the KPIs between sign up rates, and engagement rates, retention rates, are pretty consistent within that group, and don’t very much over time. So if group A performs at 50% sign up rate, and has 80% monthly retention, if you wait a year and inquire that same segment, they should perform something close to that band.

22:03 – Get the messaging right for each group

They discovered there were four distinct segments of Disney+ users and started to customize the messages and frequency to each.

We just spent a lot of time into spending on how the logistics were done, but where this kind of really materializes is from a messaging standpoint, it’s, “Who do you message what? How often do you need to message them? And when do you need to message them?” Because it’s Disney, no surprise, both of the segments that involved basically young adults and kids, their engagement rates were through the roof, we were never worried about churn whatsoever. And so that actually really just turned out to be one of our least communicated towards groups, and where our focus area basically focused in on was basically all our action content that was focused on people without kids, and then a lot more of the female-leaning content.

23:26 – Keep people engaged in the short term

The short-term goal was that the messages needed to keep people watching at least once a month so that they wouldn’t leave the service.

So this was a regular debate, less so within our team, but in terms of how we service all the different teams, and all the different goals for the service, it was an interesting conversation to say the least. But the philosophy that, at least our team took, was that there’s a near-term goal, and then there’s a long-term goal. The near-term goal is that the purpose of this messaging is to keep subscribers engaged. And for us, through a lot of our data analyses, things like that is, for the most part, if folks were at least engaged within a two-week period, we’re in a good spot. This is a monthly service, as long as you watch something once a month, you derive some kind of value from it. Once you get into that three plus weeks inactive state, and you can imagine, if you go from three weeks, to two months, to three months, and four months- At some point you’re going to get to a 100% probability to churn, and the only people who don’t are probably the people who don’t realize you’re paying for the service, to be honest. And so the near-term was that, how do we keep people from falling beyond that three weeks of inactivity?

25:06 – Create long term fans

The long-term goal is to build a fan base that went beyond just the streaming service and into other Disney properties. Too many messages could push people away and burn that relationship.

The other part of that around the long-term consideration, because the flip side is, well, email’s free, so to speak, why don’t you just send it one a day, and dial it down? Is that, where the business hasn’t evolved yet, but this is my now an outsider hunch, is that the Disney machine is that most people become Disney fans become Disney fans for life, and there’s all sorts of ways you engage with the business, whether it’s going to parks, going to theaters, buying merchandise, buying trading cards, and now subscribing to Disney+. The business really hasn’t really, from a Disney+ perspective, the sole goal right now is to get people into the service, and fully engaged, and there isn’t a huge goal on like, “Hey, how do you get these people into parks? And how do you sell people more hoodies?” Or things like that. The one thing we wanted to make sure we preserved in our short-term optimization is, by the time that becomes a priority, I don’t want people rolling their eyes at Disney+ emails, because this is a machine that spams you two times a day. We would really destroy the channel, and really destroy the long term strategic value that a life-cycle channel can provide the business. And so that was a little bit of a mix of the short and long-term, things we were looking to balance.

30:18 – Make content for all the segments

Marketing can see data before people up the ladder can, so sometimes they’d have to sound the alarm that there wasn’t enough content for certain segments.

You brought up mass appeal content, on the life-cycle front, you’re really kind of on the front lines of things, so you will see data, and you’ll be able to interpret the data quicker than you can kind of pass it up the line to the people, the rear general, so to speak. And so from our segmentation very, very early on, we could take a look at our upcoming calendar release, we know which audience release is really kind of meant to be. Some things are massive appeal, but a lot of things kind of tend to fall between, at most one of those four segments. And there will be times where we’ll see, hey, we have to ring the bell, there’s not a lot of content for this, people who like female identified leaning content who don’t have kids. And there’s nothing on the calendar for four or five months. And we could flag that with a lot of folks who handle content, but the tricky thing about this business is that everything you’re watching today was greenlit a year plus ago, so there isn’t this immediate, “Hey, how do you spin things up?”

40:28 – Lean into product-led growth

When Will was working at Dropbox, the winning approach to getting more paying users ended up being a product-led approach.

What’s really interesting is, as I was testing that self-serve side that was run by a gentleman named Giancarlo, who’s awesome, early days, product-led growth person. He came over from my last hand, who kind of knew a lot of how that stuff was done. His group kind of ended up being kind of the winning formula, which is, they were basically doing product-led growth before that term became as popular as it is today. And a lot of it was, how do you basically fish from an existing user base? How do you identify free and individual users within a company? And how do you get them into a business plan?

50:31 – Sometimes a left-field idea is the best one

When Will was working at Oscar, their leading conversion channel was subway ads in New York. It taught him that ideas without a lot of proof can go far sometimes.

And the longest subway car is typically 10 cars, so one car will be all Oscar stuff. You see any other car, at the time it was basically personal injury lawyer ads, and, “Do you want to try this clinical trial” kind of thing. And basically it was dirt cheap, that’s how cheap it was. And the first two years of Oscar before I joined, so I take no credit in any of this stuff, the subway ads was their number one sign up driver, it was very low CATs, you couldn’t measure it from an itemized perspective, but total invested, and total people in a sign up survey that said, “I’m here because of the subway ad.” Was phenomenal. And my biggest takeaway was basically there was, something like that every year, so don’t write something off just because it doesn’t make sense in your head, or there’s no case studies to it.

Will Wong (00:06):

You would be surprised at how much I’ve actually learned from selling cotton candy and peanuts, and how that’s actually applied to performance marketing. I kind of think of this from a cohort perspective, there’s so much demand for every section when they want beer, when they want ice cream, when they want hot dogs and things like that. And so much of it is really more being at the right place at the right time, a lot less of it is getting someone to buy a bag of peanuts who has no interest in buying a bag of peanuts. So I would say this is definitely much more performance marketing, how do you harvest demand, than it is, [crosstalk 00:00:37] how do I go create desires and wants?

Ben Grynol (00:45):

I’m Ben Grynol, part of the early startup team here at Levels. We’re building tech that helps people to understand their metabolic health, and this is your front row seat to everything we do. This is a whole new level.

Ben Grynol (00:59):

When it comes to the distribution of information, one of the oldest channels on the internet, that’s email, well, some people say email is a channel that is dead, no longer relevant. Other people say it’s underutilized in every respect. And when it comes to email, well, it’s been a very strong tool, a very strong distribution channel for us in being able to spread our information, everything we do from an education and editorial standpoint.

Ben Grynol (01:40):

And so Will Wong, one of the partners at Andreessen Horowitz, the lead investor in Levels. Will has deep experience when it comes to things like email marketing. Will spent a lot of time throughout his career learning about the strength of email, the ways in which you can use it as a channel to engage people. Sometimes email’s thought of as being a distraction, something that’s low value, but when done right, email can be a very strong lever in building engagement, and an audience cares about what you’re doing. And so JM, Josh Mohrer, head of global operations, Will Wong, and myself, the three of us sat down and discussed this idea of email marketing, why is email such a strong channel to consider, and how can companies think about it moving forward? It was a great conversation, and here’s where we kicked things off.

Ben Grynol (02:29):

Well, this is awesome. I appreciate you doing this, and taking the time to walk through. I thought it would be fun for the three of us to jam on all things, email, life-cycle, growth, and marketing in general. Riff on both your path, you’ve had such an interesting path, and seen so many different aspects of growth and digital through different companies, and then now, being a partner in VC on the other side of the table, and walk through some of those things. I mean, if we take it all the way back, as a digression, there must have been some contention between being the cotton candy guy for the A’s, at the same time that you’re the peanut guy for the Giants, which is an interesting juxtaposition between the two teams, but-

Josh Mohrer (03:23):

It’s the American League and National League, so I think you’re typically safe, right?

Will Wong (03:27):

Yeah, we have a Bay Area rivalry, it’s not so big these days, but every year, at least in those early 2000s, they would have a battle [inaudible 00:03:35] Bay, even bragging right between the East Bay and San Francisco. And I guess folks would remember this, but the last time the major earthquake happened here in ’89, that was the A’s World Series, A’s and Giants, A’s won, and because the World Series was going on during the earthquake here, it actually ended up saving a ton of lives, because everyone was home watching the game.

Josh Mohrer (03:55):

Wow.

Ben Grynol (03:56):

I remember that.

Josh Mohrer (03:58):

That’s a good step.

Ben Grynol (03:59):

That was a stacked team to go down-

Will Wong (04:01):

Yep.

Ben Grynol (04:01):

That path. Jose Canseco.

Will Wong (04:03):

I think McGuire was on that team.

Ben Grynol (04:05):

McGwire, Canseco. What was the-

Will Wong (04:07):

Rickey Henderson was also on the team.

Ben Grynol (04:08):

Rickey Henderson.

Will Wong (04:09):

Yep, yep. All-time steals leader. But those were some stacked teams. And I think for me, I’m Oakland born and raised, I’m a die-hard A’s fan, and so on one hand, I loved being at A’s games, but I would never focus on selling cotton candy. And when I took the Giants job in high school, it was called [inaudible 00:04:27], and now they went through its fifth name change, I think it’s now Oracle. It was a lot nicer. So I got to do best of both. And I think what most folks don’t know was, there’s all these friendly local team deals, so generally what happens is when the A’s are in town, the Giants are on the road, and then when the Giants are in town, the A’s are on the road. So you very rarely get any type of overlap.

Ben Grynol (04:48):

You’d balance both. I was a die-hard A’s fan in the late ’80s. And then when Barry Bonds joined the Giants, I was a die-hard Giants fan, that was all-

Will Wong (04:56):

Yeah.

Ben Grynol (04:57):

I could do, but you followed the path of the top performers.

Will Wong (05:02):

Yeah, and you would be surprised at how much I’ve actually learned from selling cotton candy and peanuts, and how that’s actually applied to performance marketing. And so how it actually works in the vending game is that you make about 20% of all commissions, so it’s entirely commissioned. And before you go out, there’s a pecking order between most tenured, and who’s newest, we were in high school, so we’d get the last pick. And so you always have to strategically pick what you’re going to sell. So if it’s a hot day, go for ice cream, you could go for soda and water, but you’re carrying 50 pounds up and down the aisles, and you really have to sell those quick. If you’re in Oakland, there’s a lot more kids, cotton candy’s the way to go. The Giants have the older fan base, and peanuts are the way to go.

Will Wong (05:45):

And what happens is, there’s an unwritten code where if you see a vendor in the same aisle, you’re not allowed to go down that same path. And what happens is, around the fourth inning, everyone wants ice cream, especially in the nice seats, and so all the ice cream guys rush there, and it becomes a little game of which aisles do you go first, you’re eyeing all the other ice cream guys so that you get there before they do.

Will Wong (06:08):

And when you get there, you have to go figure out what your pitch is, what your call to action is, and over time, you have to figure out what actually sells the most. You figure out how long you linger, how long you go down aisles, how good you are at spotting people who are looking to buy what you’ve got to sell. And you’d be surprised that that’s almost exactly how early days performance marketing was. It’s just how do you optimize your CTA, what message do you use, what time do you target a certain ad, or target a certain audience, all that type of stuff. And I was actually surprised like, “Oh, hey, these peanut days really apply to my adult life career.”

Ben Grynol (06:40):

Yeah, that’s so funny. I mean it’s true, it’s one of those things that… This American Life actually did an episode, and I want to say it might have been on one of the teams in the Bay Area, this is confirmation, but I said it probably is not that, but they definitely did an episode, and it was all about exactly what you’re saying. And it’s funny that the way it works is very analogous too, if you think about marketing, you’re essentially running mini campaigns every time you go to a new area.

Will Wong (07:11):

Yep. You had to know your audience, and in this case, you need to know your section, you know what that section buys at what inning they buy it, and you need to also get there before your competitors does. Friendly competition, but at the end of it, you need to get there, or you need to know that if the ice cream guy just hit that aisle… It’s actually okay to be right after, because some people need a little bit more time to decide that they want ice cream, but if you wait a little too long, there’s an optimal time and a poor time as well.

Josh Mohrer (07:36):

I love the analogy, because really, if you all head to the best section, it kind of becomes not the best section anymore-

Will Wong (07:43):

Yep, yep.

Josh Mohrer (07:44):

Because all the supply is there, and you could be better off going somewhere else. And I think that metaphor really extends, there are places where you can go, and selling through ads is just going to be so expensive because everyone else is there.

Will Wong (07:58):

And just so you all know too, these are real differences, especially if this is a side income for you, the minimum you can make by making the wrong choice, the wrong product, and hitting all the wrong aisles, you make maybe $40 a game. If you get the best product, and you hustle, you move fast, you get every aisle, you’re making $300 a game. So it’s [crosstalk 00:08:18] a real difference, and you can pound that over the 80 home games you get per team, it starts to add up.

Ben Grynol (08:23):

It sounds like you almost have to manufacture your own demand in some cases.

Will Wong (08:27):

Yeah.

Ben Grynol (08:27):

Like you’re going, and if it’s not a hot day, and you’re on ice cream, you really have to manufacture the demand for that.

Will Wong (08:34):

Yep.

Ben Grynol (08:35):

You have to run a pretty tight process to make sure that you are getting people the product that they think they want.

Will Wong (08:42):

100%.

Josh Mohrer (08:43):

Were people looking for the thing you were selling, or did you feel like it was more a split second decision, they see you, “Oh, I’d like that. I could use some ice cream or cotton candy”? Or is it someone who’s saying, “When I sit down, I’m going to look for someone who’s selling this”? Because I bet it’s more the former, and so there’s so much to the creative, and sort of what you’re shouting, and when you do it, is it between innings, or during an inning and kind of how you carry yourself. There’s a lot of subtle variable there that I would imagine really impact results.

Will Wong (09:18):

I kind of think of this from a cohort perspective, there’s so much demand for every section, when they want beer, when they want ice cream, when they want hot dogs, and things like that. And so much of it is really more being at the right place at the right time. I do think to your point, a lot less of it is getting someone to buy bag of peanuts who has no interest in buying a bag of peanut, so I would say this is definitely much more performance marketing, how do you harvest demand, than it is, [crosstalk 00:09:43] how do I go create desires and wants?

Josh Mohrer (09:45):

You’re not generating demand, you’re harvesting demand.

Will Wong (09:47):

Yep.

Josh Mohrer (09:48):

Got it.

Will Wong (09:48):

And it’s really how do you optimize that bottom of the funnel, is the way I think about it.

Josh Mohrer (09:52):

Right on.

Ben Grynol (09:53):

Very interesting. We should jump into this idea of acquisition. So at Disney, we’ve had conversations before about the way you built life-cycle at Disney from the ground up, Disney Digital, and also did it with Etsy. So when you were thinking through acquisition at each company, what were some of the things that you considered? Maybe start with Disney, because that was such an interesting process, where it was like, “Hey, Will, there’s nothing, go.” And then you had to basically grab all the tools and start building.

Will Wong (10:27):

Yeah. I guess for context, so I joined the Disney streaming team about a year before Disney+ launched, and about two or three months before ESPN+ launched. And all the digital products were all launched out of the New York office over in Chelsea Market, and I think at the time, at least for the better part of the year, everyone knew Disney was launching a streaming product, there was no name for it, folks didn’t know how big it was going to be, but for Bob Iger, who was CEO at the time, and Kevin Mayer, who was the chairman of the group and kind of the leading exec spearheading this, this was the most important thing for the company to focus on. And they made sure that there was essentially infinite resource staff to it.

Will Wong (11:08):

So Disney’s a long storied company with a lot of enterprise contracts, and deals, and a lot of tools that were already in place. The guidance given to the streaming team was, you don’t have to go by any of those, you choose whatever it is you want from a software perspective, you can hire whomever you want. And so when I came into the role, it was basically, you can hire up to 25 to 30 people, you can choose the user existing tool stack where you can choose not to use it, that’s entirely up to you. I think the way we thought about it was just very your first principles, this is, on the shorthand, we have a very fixed timeline on when we’re going to launch Disney+, I think, unlike a traditional startup where there’s a lot of product market fit testing, you have an undefined beta period.

Will Wong (11:54):

Once we announce a date, we have to hit that date, and a lot of what we needed to do is kind of back around, “Okay, what’s the CRM that we need for this? Do we need a CDP? Do we need a [inaudible 00:12:03]? Is it all in-house? What do we do for that? Okay, great. Which countries do we need to be in? What are all the relevant, legal constraints within each country? And is the toolset we’re building out that fits that?”

Will Wong (12:14):

And then you kind of jump into, “Okay, what is the actual email, SMS, push, in product alerts that you actually need to build out? What are the things that are mandatory? What are things that we need to do to really kind of tune our activation rates, engagement rates, upsell rates, all that type? And how do we actually just start from the ground up? At the same time, knowing that this is a huge brand and we’re going to have a huge demand early on.” And I think what was really interesting about that is, we actually made a very distinct decision to not run any A/B testing in those first two, three months. We just knew that so many people were going to sign up.

Will Wong (12:50):

The most important thing that was in my head was, we have to de-risk the launch, because, yes, a lot of people want to test, “Hey, what’s the best way to optimize your…” At the time there was a free trial, like, “How do you optimize trial to paid? How do you optimize month one to month two?” But our systems also ran a lot of things that could actually blow up the entire sign up and launch day, and things like one-time passcode. This is a brand new email domain, it’s an IP that we had to warm-up.

Will Wong (13:17):

I think we signed up something like 10 million trials on week one, I can just say that was vastly greater than what anyone anticipated, and nothing blew up from that front. And so on that end, I think that was a huge success, and while, I’m sure folks wanted to hear like, “Hey, what kind of great testing that you were running in those early weeks and months?” I think we all knew how big the brand was, and month one or two is make sure, don’t jeopardize the launch, and we’ll start rolling in testing following those first few months.

Ben Grynol (13:45):

Yeah, so interesting.

Josh Mohrer (13:46):

Wow.

Ben Grynol (13:47):

When you were thinking about some of the opportunity costs, as far as trade-offs go, the constraints were that there weren’t any constraints, and so you had to be pretty diligent about decision-making for things like localization. If you’re thinking about countries, you’re not just thinking first principles as far as, “What’s the stack that we’re going to use?” It was so wide, and part of that I think is the byproduct of Disney being such an established and strong brand. What were some of the things that you thought about as a team when you had to make calculated decisions, and strong decisions, but you’re also going on limited data, and you’re making them and moving them forward, and moving quickly, because you’re working towards a date, as opposed to arbitrary, like, “We’ll launch eventually.”

Will Wong (14:31):

Yeah.

Ben Grynol (14:31):

How did you think about that?

Will Wong (14:33):

I think the way that we approached that was that we wanted to prioritize the shell, and not exactly what goes in it. So before we launched, because we had a relationship with Hulu, Hulu was majority owned by Disney at that point, now I think it’s almost wholly-owned. We already had a number of streaming comps, and the fact that the company that Disney acquired to do much of the engineering horsepower for Disney+ is Pantech, which basically developed 20 streaming services for a number of household names, WWE, MLB, F1, all that type of stuff. We already had a general sense of where we thought trial to paid conversion would be, and all the M1’s, M2’s engagement and retention numbers would be, but because Disney+ was a brand new product, we would just have no idea where we would need to focus, like which part of that customer life-cycle.

Will Wong (15:26):

And so what we knew for sure though, was we needed to build a new user onboarding drip campaign, that is both email and push. We knew that if people canceled the trial, or they canceled the paid subscription, we would need a service, or targeted messages to bring people back. And so what we instead launched was these basic campaigns on a very basic, heuristics based on what we kind of knew typical streamers’ behavior would be. It was pretty straight-forward, we had a pretty good guess as to how many people abandoned a trial. Cool, how many subs would that lead to? What is the right messaging that we need to set for that? And we already knew, this is pretty normal for all subscription services, you’re going to lose a majority of users in the first 90 days, the curve will flatten generally anywhere between month three and month nine, which is pretty typical.

Will Wong (16:15):

So we wanted to make sure that we had all the messaging setup automated, so that… The situation we didn’t want to be in is, wait all the way until month four until we had now four monthly cohorts of data, and then realize, “Oh, hey, this is the month we need to focus on. Great, what do we need?” And then that was the time we started building, what we wanted to do is build all of that ahead of time, so that when we got to that point, the only thing we needed to do is really test messaging, test what the right recommendations were, test what content things, or offers that we needed to make at that point, so that all the systems were in place, as opposed to just waiting for…

Will Wong (16:48):

Maybe a better way of saying is, to be proactive with what kinds of programs we would need, versus being entirely React. And so the hard part is of course figuring out what the content and what the messages need to be, but we were pretty sure that we needed a user onboarding flow, a re-engagement flow, a win-back flow, and an upsell flow.

Ben Grynol (17:06):

Very, very cool. Yeah, JM’s been doing a lot of deep diving on cohort analysis, just did one, I guess it was yesterday when drifted it out to the team, but looking at the data that we have now, and then thinking about how can we glean insights from that moving forward based on what we have, and so a lot of learnings around there, but there’s also so much work to do to get to a point where we have, as we ramp up to lift off, which is something JM’s been spearhead, how do we think about all of the flows that we need to build out from an engagement standpoint, from an awareness standpoint, from a convergence standpoint, there’s so much work to do.

Josh Mohrer (17:48):

Were there any lessons that you learned through those experiments that you ran early on that yielded something that was surprising to you, and maybe unintuitive?

Will Wong (18:00):

So my biggest learning, and because the organization that I was in is just a giant machine, and we weren’t the only life-cycle team, Disney has maybe hundreds of products, and each one has their own life-cycle team. So you can imagine the noise of ideas and best practices that were true for business A, that weren’t true for business B, but then you have so much different feedback. And I think the interesting part where this kind of really materialized is, and this is my biggest learning is, how do you implement segmentation? And my approach has always been, generally, I don’t care what the segments are, I don’t have a bias, but for them to be good segments, one, each segment needs to be distinct from another, and they’re distinct because their behaviors, and patterns, and the KPIs between sign up rates, and engagement rates, retention rates, are pretty consistent within that group, and don’t very much over time.

Will Wong (18:57):

So if group A performs at 50% sign up rate, and has 80% monthly retention, if you wait a year and inquire that same segment, they should perform something close to that band. And I think we first kicked off this exercise, and of course, every group that was involved, not just our group. You can imagine, everyone had an opinion around what Disney+’s segmentation should be. You have one school of thought, which is, “Oh, we should bucket these by fans. We have Marvel fans, and StarWars fans, and these type of things.” You have other buckets who think it should be regional, or by country, or something like that, and on, and on, and on, you can assume all those things.

Will Wong (19:33):

And one of the great things that our data team there did was, “Okay, we’re just going to go look at all the data, and figure out what these distinct groups are. And then in partnership with our product marketing team, we are going to go do some user research and figure out who these folks are, draw the commonalities, survey them, and actually then come up with a basic heuristic name that people can then understand, and start thinking about their marketing and messaging toward. Again, I would say the number one thing everyone thought our segmentation was going to be was based on fandom. What it ended up turning out to be was actually just… We had four groups, we basically had people who liked, I wish we had a better word leaning into this, but it was basically very male leaning content, so a lot of StarWars-

Josh Mohrer (20:15):

StarWars.

Will Wong (20:16):

[crosstalk 00:20:16] type stuff.

Josh Mohrer (20:17):

Marvel, yeah.

Will Wong (20:18):

Yep. You had a lot more content that tends to appeal to more of a female audience, the Billie Eilish concert, the Beyonce concert, things like that. And then you had kids content for young kids, so think grade school kids, and then teenagers. Those are the four groups, all their behaviors are completely different, they were also consistently different over a one plus year cohort, like monthly cohorts over a year plus. And that ended up being the true segmentation, and all of our content marketing, engagement marketing, retention marketing, was kind of built upon. I don’t know if this answers your question, but I think what was most interesting to me was, I think being from smaller companies prior to Disney, you’re like, “Oh, we’ll look at the data, it’ll take you about a week, you’ll you’ll jam on it by-”

Josh Mohrer (21:01):

Right.

Will Wong (21:02):

“[crosstalk 00:21:02] groups.” This was maybe a six month process-

Josh Mohrer (21:04):

Totally. So I’d like to double-click on that a bit. That was super interesting. Thank you for sharing that. And I love the service, and we are a grade school/female home-

Will Wong (21:15):

Yep.

Josh Mohrer (21:15):

Here, Marvel and StarWars are not my jam, but whatever dollars I spend, it’s the best that I spend every month. So it sounds like the instinct was, “Oh, we’ll have our Marvel people, and our StarWars people, and maybe our Simpsons people, and our Bluey people.” Got to give a shout out to Bluey. But it ended up being something that you derived instead from data, that maybe was a little more blurry than that, is that a fair kind of-

Will Wong (21:41):

Yep, absolutely.

Josh Mohrer (21:41):

Recap of what you said?

Will Wong (21:43):

Yep.

Josh Mohrer (21:43):

Interesting. Because it probably gets hard in families where they’re sharing, where the usage will just look totally bizarre-

Will Wong (21:53):

Yeah.

Josh Mohrer (21:53):

Like Little Mermaid and Marvel. I don’t know, maybe that’s not so bizarre actually, but things like that.

Will Wong (22:03):

We just spent a lot of time into spending on how the logistics were done, but where this kind of really materializes is from a messaging standpoint, it’s, “Who do you message what? How often do you need to message them? And when do you need to message them?” Because it’s Disney, no surprise, both of the segments that involved basically young adults and kids, their engagement rates were through the roof, we were never worried about churn whatsoever. And so that actually really just turned out to be one of our least communicated towards groups, and where our focus area basically focused in on was basically all our action content that was focused on people without kids, and then a lot more of the female-leaning content.

Josh Mohrer (22:46):

So I just opened up my email, and I wanted to count how many emails I got from the service, and it seems like it’s roughly three or four per week-

Will Wong (22:56):

Yep.

Josh Mohrer (22:56):

Which is a lot, but not too much that I’ve noticed that it’s too much. So I was wondering how you think about frequency, and repeating, if someone is opening the emails or not, I tend to not open them, I just sort of archive them. I understand that you’re not there anymore, but would it be their practice to read into those signals? Are there ideas about what is too much? Is it the amount that I use the service that is driving that? I’d love to hear something on that.

Will Wong (23:26):

Yeah, so this was a regular debate, less so within our team, but in terms of how we service all the different teams, and all the different goals for the service, it was an interesting conversation to say the least. But the philosophy that, at least our team took, was that there’s a near-term goal, and then there’s a long-term goal. The near-term goal is that the purpose of this messaging is to keep subscribers engaged. And for us, through a lot of our data analyses, things like that is, for the most part, if folks were at least engaged within a two-week period, we’re in a good spot. This is a monthly service, as long as you watch something once a month, you derive some kind of value from it. Once you get into that three plus weeks inactive state, and you can imagine, if you go from three weeks, to two months, to three months, and four months-

Josh Mohrer (24:18):

Yeah.

Will Wong (24:18):

At some point you’re going to get to a 100% probability to churn, and the only people who don’t are probably the people who don’t realize you’re paying for the service, to be honest. And so the near-term was that, how do we keep people from falling beyond that three weeks of inactivity? So there’s a baseline level of content, but a lot of what we ended up doing was that, when you get to that point, there becomes a much bigger ratchet of how much message is you get. And internally, our whole goal is basically, we need to introduce a new message that can only help, or at least is neutral to the engagement behavior, and so we tested a ton of different messages, and a ton of different content types, but we don’t keep it, unless there was a neutral or positive impact to likelihood, or incremental minutes streamed, is a metric that we use.

Will Wong (25:06):

The other part of that around the long-terms consideration, because the flip side is, well, email’s free, so to speak, why don’t you just send it one a day, and dial it down? Is that, where the business hasn’t evolved yet, but this is my now an outsider, or a hunch, is that the Disney machine is that most people become Disney fans become Disney fans for life, and there’s all sorts of ways you engage with the business, whether it’s going to parks, going to theaters, buying merchandise, buying trading cards, and now subscribing to Disney+.

Will Wong (25:42):

The business really hasn’t really, from a Disney+ perspective, the sole goal right now is to get people into the service, and fully engaged, and there isn’t a huge goal on like, “Hey, how do you get these people into parks? And how do you sell people more hoodies?” Or things like that. The one thing we wanted to make sure we preserved in our short-term optimization is, by the time that becomes a priority, I don’t want people rolling their eyes at Disney+ emails, because this is a machine that spams you two times a day. We would really destroy the channel, and really destroy the long term strategic value that a life-cycle channel can provide the business. And so that was a little bit of a mix of the short and long-term, things we were looking to balance.

Ben Grynol (26:21):

When you first went down the path of conversion, I remember that was the focus, and then it was like a light switch where as soon as people converted, it was all about retention and creating value through the content that you’re putting out, so it wasn’t just a matter of consideration to try to probe people like, “Hey, go use the service.” That’s always sort of an underlying goal, but I remember you framed it as something along the lines where, “We only want to give people information that they find valuable.”

Ben Grynol (26:49):

So the value might actually be consideration for a new piece of content, it also might be, I’m making this up, but it was like, “Read this article about something.” Because you were always thinking through, what’s the Marvel clan? What’s the young family going to be interested in? And it was a really interesting lens, because that’s not a typical approach for life-cycle. Most life-cycles are going to be focused on this idea of just always trying to get people to click on something, to do something else. And it was a different approach, especially given the size of the audience that you were working with.

Will Wong (27:26):

Yeah, yeah. So I guess it’s funny, because one of the big influences on my philosophy going into it actually came from Ben Horowitz and reading his book, I think The Hard Thing About Hard Things, and now that I’m actually working at Andreessen Horowitz, it’s kind of a full-circle moment. But I remember a long time ago reading that book, there’s a passage in there where I think Ben was sitting on a board meeting, and one of the executives at a startup basically was saying, “Oh, hey, retention was really bad, but it’s because we couldn’t get this email campaign app.” And I think the feedback was, “Your retention isn’t bad because you didn’t send an email, retention’s bad because your customers didn’t like the product.”

Will Wong (28:06):

And that really resonated with me. And it’s really easy when your team’s responsibility is all communications, and to really get lost in just saying, “Well, we own emails, push, in products, so let’s just do more of that.” When there is a whole point of people don’t retain, because of this message, but what this message can do is help someone who’s on the fence, who’s otherwise going to drift off, and maybe use it to bring people back. And that was a large part of the philosophy, where it’s, for a streamer that basically releases weekly content, a lot of what’s underpinned, what we were trying to do is, one, is there a piece of content that got released that if someone who’s not using the service knew about, they would come back, stream, and they would stay in that healthy, “I’m streaming at least once every three weeks”? And that was really the philosophy that we took, as opposed to this, we’re blasting people all the time, with no discrimination whatsoever sort of thing.

Josh Mohrer (29:07):

Now it’s what you do now. The Beatles, I mean that must have driven a gajillion subscriptions, it was such an amazing eight hours. And in Ecanto, and Hamilton before that. There must be these big moments where this is something that a very wide audience will enjoy, and could be a reason to sign up. I would imagine it’s also just like a prompt, “Hey, reminder, this movie is on the service.” So when you’re making the decision of what to show your kids, or what to put on at night, you’re like, “Oh yeah, I saw that email.” It makes a ton of sense.

Josh Mohrer (29:37):

But the underlying product is really, really great, and people want it anyway. And so I think what you’re saying is, you can tap it in the basket. It’s like an alley-oop, and you’re giving it a little bit of a push, but the underlying want is sort of already there. And I think that’s very similar for us too, folks are into what we’re doing, because they’ve run at our free content, or they’re just kind of aware, they listen to the podcast, they care about it because they’re human, then we’re just sort of showing them the way in, instead of-

Will Wong (30:10):

Yeah.

Josh Mohrer (30:10):

Making a hard-pitch.

Will Wong (30:12):

Yeah, I think that’s true. You brought up mass appeal content, on the life-cycle front, you’re really kind of on the front lines of things, so you will see data, and you’ll be able to interpret the data quicker than you can kind of pass it up the line to the people, the rear general, so to speak. And so from our segmentation very, very early on, we could take a look at our upcoming calendar release, we know which audience release is really kind of meant to be. Some things are massive appeal, but a lot of things kind of tend to fall between, at most one of those four segments.

Will Wong (30:52):

And there will be times where we’ll see, hey, we have to ring the bell, there’s not a lot of content for this, people who like female identified leaning content who don’t have kids. And there’s nothing on the calendar for four or five months. And we could flag that with a lot of folks who handle content, but the tricky thing about this business is that everything you’re watching today was greenlit a year plus ago, so there isn’t this immediate, “Hey, how do you spin things up?” I think maybe after a few months, that message of how to treat these two, what I would say, greater to be at-risk of churn audiences ended up trickling into across the content orgs, and now there’s a big focus on spending more on content, especially for the non-kids content side of things.

Will Wong (31:38):

And so tying that back to what you brought up about mass appeal, I would say it’s not as mass appeal as you’d think, and there’s certainly key subscriber groups that the content [inaudible 00:31:50] always appeal to. I think that’s going to change very, very soon, but also, the one big thing is when we launched in the first year, because we signed up so many people, we basically hit our five-year target number in under a year, because I think the five-year target was 60 to 90 million subs over five years, and I think we crossed over 100 by month 11, something like that.

Josh Mohrer (32:12):

Amazing, amazing.

Will Wong (32:13):

But the life-cycle challenge was that the content schedule was built for a subscriber base of 30 or 40 million, it’s like, “Oh crap, we got this pull forward. What do we do now?” And so we tried our best to repackage existing library content, and lot of our partners, but then the studio has tried to pull content from elsewhere to get them onto the service. These are things that I think all products, not just Disney to solve over time, where core of Disney+, what the product is, is less the tech end, it’s really the content and the IP, and the folks who are basically optimizing the surface area, be it the actual streaming app, or the extension things like life-cycle, push notifications. Because over time that stuff [inaudible 00:32:53], so everyone is looking at the same data, they’re all completely aligned, it just takes some time in kind of pulling those things together.

Ben Grynol (33:00):

So cool to hear it first-hand from behind the scenes, because from the outside, it was such a killer launch, it was such a cool thing to watch, and to see all the media around it, just following nerdy things like TechCrunch, and whatever media sources, to watch this thing come to life. And everyone’s like, “Oh my gosh, N number of subscribers in a day, a week.” Whatever it was. And it was very visible, it was very cool to see that come together, and then to hear how you hit these targets so quickly, and then some of the implications of, “Uh-oh, we don’t have enough content based on the number of… We do, but we don’t have enough content to really sustain the subscriber base.”

Will Wong (33:40):

Yeah.

Ben Grynol (33:40):

Everything always flexes when you get that type of oscillation.

Will Wong (33:43):

Yeah, or maybe put it in a different way. I wouldn’t say we can’t sustain the audience, but a larger percent of this audience is in this at-risk three plus weeks of inactivity bucket. If you’re not using a monthly service at least once a month, I think that’s a pretty basic understanding, you’re going to be at risk of churn, you’re not any value from it. And so there were these lulls, and I don’t believe they’re there anymore, this is just year one challenges, where it’s, “What do we do about these lulls? The at-risk bucket continues.” I want to say it continues to grow, but it’s large enough where it could be a concern, what’s everything we do from a life-cycle perspective to keep these cohorts and these groups engaged with this service.

Ben Grynol (34:24):

One of the thoughts around what you did, jumping into performance at Dropbox, so Dropbox launches in ’07, there’s this flywheel that is a case study about user referrals, and how that led to all of this initial growth, and I think it was ’15 when you came on. So what did that look like as far as the way you used performance as a lever to drive growth? And how did some of those things change as far as user referrals, and then shifting into maybe a different approach for acquisition?

Will Wong (34:57):

Yep, yep. I think a lot of folks kind of know the early days of the Dropbox story, Drew and the early team launched this amazing, I think it’s 60 and 90 seconds stick figure explanation, where anyone and everyone can explain what Dropbox does. And it was a really groundbreaking product in terms of how things just… It was a model people said internally where it just worked, it just did. And that, plus the viral of mechanism of give space, get space really, really kind of got that entire service on the free side, really, really ramped up.

Will Wong (35:33):

And when I joined around 2015, that was at the same time the launch of Dropbox Business, it was called Dropbox Teams at that time. I think the story internally was that, we’re noticing a lot of business using it, the current SaaS movement wasn’t really a movement back then, people just saw so much value, a lot of employees were just like, “I’m going to sign up for this, I don’t care what IT says. They can slap me on the wrist later.” Sort of thing. And so folks internally noticed this, they’re like, “Oh wow, we should just bundle this service, and sell it to businesses.” And over time, more and more features gone, it became a whole kind of business class product. And at the time, the only go-to-market motion the firm had was the viral loop, and they were only starting to build the sales org. And the viral referral loop, that’s just not how businesses share. That is how businesses share internally, when you get three or four people within a company, the internal loop works, but that doesn’t really jump to another company.

Will Wong (36:27):

And so what’s interesting when I joined was that a lot of the folks there, there’s 300 people there, 20 were business folks, the other 280 all engineering product design, the folks were thinking like, “How do we get business customers?” And I remember coming in, initially as a consultant, just like, “Hey, you can totally run performance marketing.” Like, “Oh, that doesn’t work.” I’m like, “Why do you think it doesn’t work?” I go on YouTube, I find this video of Drew doing a YC presentation, something like that, around tax being $1,000, “We don’t want to do this.” And that’s why they leaned so heavy into it.

Will Wong (36:59):

I’m like, “Okay, that makes sense, but a free user doesn’t generate anything, and let’s say you only convert 1% of them to a $100 product, your user’s only worth a buck. This is a $1,000 minimum business product, your whole CAC to LTV ratio is completely different now, we should drop that, I would say, institutional knowledge, which is probably true for the free user, and is true for the free user aspect, but all the economics has changed. Let’s relaunch performance, let’s test it, let’s see if this works.” And that was really just kind of the story of how performance marketing started, and in my first three months just consulting there. We proved out to be a channel, I think in just six months alone, performance was somewhere in 10 to 20% of all sign ups, and a significant enough number where they’re like, “Okay, here’s more budget, go hire a team, go launch a performance acquisition side of things.”

Ben Grynol (37:50):

And did that lead to-

Josh Mohrer (37:51):

Amazing.

Ben Grynol (37:53):

An enterprise or a B2B approach, where you went deeper from the sales perspective because you had the insight, as opposed to just saying… Because you could run performance, and it can be focused on consumer, but the unlock is, “Oh my goodness, look, we have businesses that are using it.” And then it becomes that play. Was that sort of something that you considered?

Will Wong (38:12):

So my biggest learning there is that performance marketing is just a tactic, but you can’t really change what CPM are, and CPMs, or CPCs, whatever metric you want to go with really affect your CAC. And at the end of the day, everything’s guided by your CAC, LTV ratio, or a more simpler explanation, you can’t overpay for what total revenue your customer results are going to generate for you, you can’t sell knowledge for dimes, that is not a recipe for success. What was interesting, we were resourced at the time, it was unclear what our business go-to-market model was. We were testing everything.

Will Wong (38:51):

So there were literally three different teams. You had an enterprise team, I think it was called Lighthouse at the time, who was focused on selling to the Fortune 500. We want logos, we want case studies, we want to get into the world’s biggest companies. You had a mid-market team, which is more your kind of traditional, go get leads, farm these leads from our existing users, we will run some SDR calls through them, an AE will then talk to them, and then once they sign, we’ll move them into a customer success team sort of thing, or account management team. And then you had the third group, which was the self-serve group. And what Dropbox had at the time was something like 200 million free user accounts, and free individual paid user accounts, I think it’s well over that count. And we were trying all three things, and at least from I recalled, the Lighthouse enterprise stuff always makes sense, because the logos are important, it helps give you social proof from a B2B perspective in terms of winning new customers.

Will Wong (39:49):

The mid-market front, that model always works, but if you really kind of take a look at what our average selling price is, at the time it was sub 10K in that space, in my head, it didn’t really make a lot of sense to spend 25 to 50 bucks a lead, spend all the energy to qualify them, have SDR spend multiple calls to make sure their sales accept the leads, and then handing them off to an AE without any upsell product down the road. A service like Salesforce does that, but I think their ISPs are in the hundreds of thousands, and then once you’re in, you have 10 other products in the hundreds of thousands that you can cross sell to.

Will Wong (40:24):

And that model, I don’t think really kind of worked for that type of business, and what’s really interesting is, as I was testing that self-serve side that was run by a gentleman named Giancarlo, who’s awesome, early days, product-led growth person. He came over from my last hand, who kind of knew a lot of how that stuff was done. His group kind of ended up being kind of the winning formula, which is, they were basically doing product-led growth before that term became as popular as it is today. And a lot of it was, how do you basically fish from an existing user base? How do you identify free and individual users within a company? And how do you get them into a business plan?

Ben Grynol (41:05):

Very, very cool.

Josh Mohrer (41:07):

I could listen to these stories all day. How did you think about LTV in those early days, Dropbox, when you don’t really have a full life-time to measure value off?

Will Wong (41:18):

You know what’s interesting? Well, because I’ve been out of performance marketing for five years now-

Josh Mohrer (41:25):

Yeah.

Will Wong (41:25):

That was kind of one of my last performance marketing roles before I wanted to branch out into life-cycle and the other parts of the funnel, so to speak, that was still very early days, kind of like SaaS metrics and KPIs weren’t really solidified yet.

Josh Mohrer (41:39):

Right.

Will Wong (41:39):

And we actually used to use this metric that no one ever uses anymore, but I thought it was very helpful, we used payback period, which is basically a different way of looking at LTV. So it’s basically, if ASP is $100 a month, and you acquired a customer for $1,000, your payback period is 10 months. And a lot of this was mostly more assumption driven. Because this was month two of the business product ever being in market, no one’s model LTV ever actually materializes to be true.

Josh Mohrer (42:11):

Right.

Will Wong (42:12):

And so it was more-

Josh Mohrer (42:12):

Yeah.

Will Wong (42:13):

Helpful for us to know, at the current rate that we’re acquiring customers, how long will it take for them to kind of pay back, if they 100% stay subscribed at the exact-

Josh Mohrer (42:22):

Right.

Will Wong (42:22):

Same rate. And that was a much more helpful guiding point, I think, in those early days, before we had a real LTV model to smell check your payback targets.

Josh Mohrer (42:30):

Got it. That reminds you a little bit of return on ad spend, it’s sort of the inverse of that, and that’s what we use a little bit now for the little that we do right now. Most of it is organic at this point for us.

Ben Grynol (42:47):

Yeah, I’ve got to dig into this other growth lever, which you had exposure, and you’ve got a lens on, is everything around Twitch, when you worked with Twitch, and driving growth through SEO. What did that lens look like? And that’s why it’s so cool to hear, because you’ve had so many different parts of the stack, and so many different levers that you’ve had exposure to, that everything from cotton candy, and selling peanuts, all the way up to life-cycle with Disney, there are so many stories to share. So I would love to dig into that idea of, when you worked with Twitch, what were some of the things that you were thinking through with SEO? And how were you approaching it, knowing that there’s product-driven SEO, there’s content-driven SEO, and there are all these different aspects to it? Yeah, we’d love to dig into that.

Will Wong (43:35):

Yeah, so I worked with Twitch very briefly as an outside consult as well. I think it was a little bit under a year, working directly with their engineering team, the search and discovery team. And I think at the time Twitch had just been acquired by Amazon for under a year, and there’s just a lot of opportunity. Twitch is this amazing platform, they dominate the platform for streamers, and all types of gamers, and they’ve kind of withstood every single competitor from YouTube Gaming to, I think Microsoft’s Mixer for a little bit, and they’re still the number one streaming platform. And I think at the time, there was questions around, “Hey, is there stuff we can do in SEO?” I went in to basically kind of just do a general audit, to see where things are.

Will Wong (44:22):

And I think the biggest learning here, and there’s a lot more folks who’ve kind of done this now, or are aware of this now, is the opportunity for them was really more of that, it was an engineering-led org, the streaming service wasn’t highly built on React, the way React works is not good for SEO. And most of what we really identified is that Twitch, if you search any game on Google, whether it’s Call of Duty, Warzone, Fortnite, Destiny, whatever it is, there’s no reason why Twitch should not a top five result, much like when you searched a company’s name that their Crunchbase appears, their Wikipedia appears, all below the actual dot com of the home. And at the time, I think all of the Twitch pages were sitting on page two, page three. And if anyone is not familiar with SEO, if you’re not on page one, you’re not getting any traffic, and then all of the traffic generally goes to top three.

Ben Grynol (45:12):

Of the folder.

Will Wong (45:13):

Yes, exactly. And in every year, there’s a new ad that pops up that pushes that fold a little further down. But it really is kind of top half at minimum page one. And that was an interesting experience, and I would like roll that up into a… What I would say is, the general balance between building great product, which Twitch is, but then also not leaving too much growth debt on the table, Where React is absolutely the right technology choice for them, but to make React [inaudible 00:45:44]. And the way React kind of works is, I’m not an engineer, so I’m not to explain this right, if you used a crawler to go to a Reactor in the website, a crawler, which only read text, sees nothing. And so there’s a number of things you need to do to serve up the text so that when the Google crawler, or another search engine crawler goes to a Twitch page for Warzone, they know that this page is about Warzone.

Will Wong (46:03):

At the time, all they could tell was that, “Oh, the URL is twitch.tv/warzone. Maybe it’s about Warzone, I don’t really know.” And by having more content, and actual clips show up, a streamer name up, all the texts around could only have helped send that signal. And because Twitch is just so popular, these are just one of those opportunities where, this is a pure example of, “Let’s make this minor change, and top five is literally in your line of sight. There is no reason why you should not be top five.” Now, my project ended at the recommendation hand-off, so I’m not too sure what ended up getting done, and where that netted, but maybe I’m exaggerating the possibilities of this, but I think a lot of us as just some casual fans, there’s really is no reason why Twitch can’t be the Wikipedia version of [inaudible 00:46:51] result for games.

Ben Grynol (46:53):

Yeah, there’s so many opportunities-

Josh Mohrer (46:55):

Wow.

Ben Grynol (46:55):

With it, because riffing on it, essentially you could have the content that is about the different streamers, and that’s its own play, but there’s also this other play that seems a little bit growth hacky, but it also would help with things like SEO, and that’s, if every single stream has an associated transcript with it, and you’ve got this massive body of text, it’s not going to be perfect, let’s assume that it’s not perfectly corrected, just given the scale of content that’s getting put out, but having that hosted in some way, where that allows the results to come up to the top of fold so much easier, or so much faster, because you’re not just going, “Cool, what’s the slug on the end of that thing?” You actually have meaningful content. Where it gets a little bit hacky and scrappy is, especially we with streamers, there’s going to be a lot of just sort of filler content that might not actually talk about the thing that they’re doing. But yeah, there are so many different avenues that you can go with something like that.

Will Wong (48:02):

Yeah, absolutely.

Josh Mohrer (48:05):

Could we circle back to your time at Oscar?

Will Wong (48:09):

Yeah, so Oscar, for folks who don’t know, so the Affordable Care Act, kind of passed kind of in the tail days of Obama, it did a number of things to basically give affordable access to health insurance plans, to something like the 25% or so, something like that, of basically uninsured Americans at the time. And the biggest challenge with being uninsured was basically, healthcare costs are the leading reason for bankruptcy, and while most people might not need it, it’s one of those low risk, but if you are the unlucky number, the RNG of life chooses, it’s basically financial catastrophe in many types of ways. So Oscar was one of the companies basically emerged as one of the leading, basically Obamacare based plans at the time, and my background’s mostly been 15 years of growth marketing, but I like to change industries every two years.

Will Wong (49:07):

So at that point I did SaaS, I did e-commerce at eBay, I did mobile gaming at this place called Rumble Games, and I’m like, “I want to go do healthcare.” Right now VC, but I wanted to do healthcare at the time, and I think a lot of it was really just the mission-driven part of it as a growth marketer. At this time where I asked myself, “Okay, I love the people at Dropbox, I love what the product does, but let’s say you do a phenomenal job at growth, what does that actually do? You get more people involved.” If you kind of take that to what Oscar’s mission was, is you’re giving more access to healthcare, and you’re mostly just limiting the negative effects of not being insured in case something happened to you.

Will Wong (49:46):

I’m like, “Oh, that seems like a very worthy mission. Let me just kind of jump right in.” I would say the biggest thing I learned about Oscar is that from a performance marketing and growth marketing standpoint, there’s always a dumb idea at any given moment that people would be like, “You’re stupid, you’re crazy. Why would you do that?” That is a phenomenally good idea, because everyone thinks it’s dumb, and it becomes remarkably cheap. I wasn’t responsible for this, the company was already doing this when I joined, but when I first moved to New York, they were the only person are some plastering subway cars, all Oscar ads, very creative, very funny.

Josh Mohrer (50:30):

Oh yeah, oh yeah.

Will Wong (50:31):

Yep. And the longest subway car is typically 10 cars, so one car will be all Oscar stuff, you send any other car, at the time it was basically personal injury lawyer ads, and, “Do you want to try this clinical trial.” Kind of thing. And basically it was dirt cheap, that’s how cheap it was. And the first two years of Oscar before I joined, so I take no credit in any of this stuff, the subway ads was their number one sign up driver, it was very low CATs, you couldn’t measure it from an itemized perspective, but total invested, and total people in a sign up survey that said, “I’m here because of the subway ad.” Was phenomenal. And my biggest takeaway was basically there was, something like that every year, so don’t write something off just because it doesn’t make sense in your head, or there’s no case studies to it.

Josh Mohrer (51:14):

Those are really good ads. I have a few open now on my desktop, and I remember them. I was actually wondering how regulation would impact what you would do for those ads, and really, if there was any impact. I believe selling insurance is a regulated activity, so maybe you can-

Will Wong (51:32):

Yeah, so-

Josh Mohrer (51:33):

Chime on that.

Will Wong (51:34):

This was the most interesting thing, because, to that point, I had not thought about it when I joined. It’s performance marketing in virtually every other sector, it’s this CAC/LTV ratio. And so when, as soon as I went in, I was like, “Oh, CAC/LTV.” All that stuff. And then someone kindly told me, “You can’t look at LTV, it’s illegal.”

Will Wong (51:54):

“Why not?” It’s like, “Oh, I get it, because what adds to cost in health insurance is utilization, how often using your plans, and what you don’t want to do is basically build a program that kind of purposely excludes people who have high needs.” So these could be people of terminal illnesses, things like that. And immediately I thought, “Okay, yeah, that is a really bad idea. We won’t look at LTV. We will like take a look at-”

Josh Mohrer (52:17):

That was sort of the point of the ACA, right?

Will Wong (52:18):

Yep.

Josh Mohrer (52:19):

I mean, in a nutshell.

Will Wong (52:22):

Yep, exactly. And so it became much more of a straight-forward, how do you acquire customers? And what was interesting is this is the ultimate seasonal product, the only close analogy would be something like TurboTax during April 15th, open enrollment is a three month period, but 90% of people sign up right before the deadline to pick a plan. And so a lot of it ended up becoming like, how do you load up for that period? Because if you better or worse try to get too cute to stick a landing, if you miss it, there’s no way to recoup those-

Josh Mohrer (52:56):

Right.

Will Wong (52:56):

Potential subs you could have gotten, until the next open enrollment period.

Ben Grynol (53:03):

It’s very cool to see the way that out-of-home was used, because a lot of companies shy away from it because of the lack of attribution, but then you see some companies go really deep on it. So Spotify is notorious for having these massive out-of-home campaigns where they’re plastered, not just in one city, but they’re plastered across the entire country. And there’s still a ton of opportunity with it, but you have to find what that unlock is. When it was done with Oscar, was there a reason why it was so deep in New York, or was it geographically distributed? What did that look around those decisions?

Will Wong (53:46):

So, I mean, again, I take no credit for any of this, this was kind done by some smarter people there before my time. But what I would say the learning is, this is hindsight learning, is, that was really just unique to the New York market. There were other cities that out-of-home did decently well, but there was a couple things that made it particularly strong in New York. Number one, New York City specifically is the only major metro area where it’s not dominated by one or two health insurer providers. If you look at every other major city, two companies own 60%, 70%, something like that, of all the health insurer market share, like in California, it’s The Blues, it’s Kaiser in certain parts of California, it’s Anthem in certain areas. New York’s kind of the only area where there is no single dominant player.

Will Wong (54:36):

And then the other thing is that it’s also city where everyone just takes the subway. Everyone sits in the subway, they have nothing to do for 20 minutes. And one of my takeaways from it was that the subway ads actually always worked, and I think it took the fact that no one believed in it, where the CPMs actually made sense, the cost of actually getting someone to see a subway ad.

Will Wong (55:00):

Essentially, that’s kind of what we saw on Facebook over the past few years, where there was a period of time, in the early days of say gaming advertising, no one was advertising on Facebook, so everything you saw on your feed maybe about eight years ago was all games. The game LTV for a casual game is under 10 bucks. You can’t really advertising anymore, because basically like people with LTVs over $100 are advertising, and you kind of price out the people who can only afford to pay 10 bucks a user type of thing. And I would say that was basically what was going on with subways in New York, where a huge part of the population was on it every day, the eyeballs were there, but there was no competition for their eyeballs from a cost perspective, because there was no other advertiser in the marketplace.

Josh Mohrer (55:42):

No one was in the aisle selling cotton candy.

Will Wong (55:44):

Yep, that’s exactly what it was.

Ben Grynol (55:47):

Love it. I love that when everyone’s in the office, you’ve got the a16Z background, [inaudible 00:56:00] different ones.

Will Wong (56:01):

Yeah, it’s a nice touch, I think. We have a, I don’t know, it’s almost like a fashion brand. We get quarterly drops, so every time there’s a new season, we get 10 new ones.

Ben Grynol (56:12):

Do you really?

Will Wong (56:13):

Yeah, we do, we do.

Josh Mohrer (56:13):

I like it.

Will Wong (56:15):

Let’s see what we have here.

Josh Mohrer (56:16):

Unlimited fashion budget.

Will Wong (56:18):

Exactly.

Ben Grynol (56:19):

Mark came on our forum last weekend, and he was in like a Dolce & Gabbana. It’s pretty dark, and it was definitely very fashion-driven. It’s funny that’s sort of the brand.

Will Wong (56:33):

That’s amazing.