Josh Clemente (00:00):
Welcome to Friday Forum, August 12th, 2022. Straight into the recent achievements. On the UK expansion side, our infra, the backend, is about 85% complete. A little bit tricky to get exact number there, but actually tracking ahead of schedule on a number of things, which is really exciting. Thanks to the ENG team for crushing on that.
(00:23):
Outreach has started for UK podcast and creator targets. Similar to the expansion of the original levels program here in the US, we really relied on partners who have direct communication with very intent audiences to help us with distribution network very effectively. We’re going to be looking to replicate that in the UK. So, that started. We’ve got some great recommendations and connections already underway.
(00:48):
On the alpha signups, we have something like 18 up and running. You can see the kits down here getting ready to go out. That kicks off on Monday, so things are moving. Research team updated the adverse event reporting approach for our large general glucose trends in the general population study. And then we’d begun analysis on a number of things. So the vinegar experiment, a few of our external partnership studies like the Hull Wellness Study at USF, and then metabolic panel trends.
(01:21):
So as we look to develop our labs 2.0, Metabolic Health Panel 2.0 experience, we also are looking at the numbers from that data that we’ve already collected on metabolic panel trends, and what our member baselines look like. So that will be… Obviously, this is all anonymized analysis, but the objective here is to better understand where the education targets can be, where the platform can expand information about how Metabolic Health Panel trends, these North Star metrics, connect with these sort of continuous feedback loops of CGM and the other education we have on the platform.
(01:57):
We are still in the woods on support, so I just want to continuously hold up the support team as really doing everything that we can hope for in a support team. Maintaining happiness at 90% for two weeks, despite really getting an overwhelming amount of volume and diverse change to the nature of the support tasks that they’ve been having to handle through liftoff. Definitely still experiencing a major backlog. SLAs are behind, but Lynette and Rebecca have made it through onboarding and we’re taking some actions, such as setting up an ask/actions tracker, which Chris is keeping up to speed on, just to handle more of the ad hoc stuff that’s happening. We have a lot of people that are putting in requests with unique projects that require support, and Chris is taking lead on just making sure that we are handling those and prioritizing them against the primary function, which is to keep our members on track satisfied and getting set up in the CGM program. So yeah, big shout out to that whole team.
(03:01):
On the content side. So we’ve been considering doing some SEO experimentation with an external party, but Paul stepped in and made some really good recommendations on SEO updates, a few technical fixes we can make. And then adding attention. I’m excited about this one on information graphics. So basically taking a more visual approach to adding additional SEO juice, but in a way that really adds value, in my opinion. So Google Images ranking and adding additional high value visuals to the posts we’re putting up. So excited for that effort. The stability ring is in trusted user testing, so we have kind of a small subset of our users trying it out along with our team. I hope everybody’s given it a shot and providing consistent feedback on that experience. It’s an entirely new way of looking at in-app scoring.
(03:52):
And beyond that, we’ve got logging, tagging and event insights in engineering right now. And we had a really great week for ENG metrics. So cycle times, review times, deploy times, all improving after a big push and after the introduction of some deeper tracking there. So shout out to the ENG team on continuing to improve process there. On the first third of the month. So we’re in August, we’re what, 11 days in, 12 days in, we’re tracking behind our growth goal. So we’re now in growth mode. This is our second month really, really kind of the first complete month that we’ve been in growth mode formally, and we’re still learning the ropes, still figuring out where our levers are, but we’re behind right now, no changes to burn and runway relatively flat here. And this is just something that we’re going to keep eyes on. We’re going to have a lot of deeper analytics coming up later in this meeting. And then just generally on our growth goals and how we’re focusing on them. We’re also shifting to some more conservative revenue reporting. So Riley’s going to have a big update on that once the July numbers are fully in. So just making sure that we’re doing all the right things, such that we don’t over rotate or overspend.
(05:03):
And then we’ve got some big culture updates, so feedback, leveling, performance processes. All of these were major sort of themes in the company survey, and we have made some quick progress. Lots of those projects were already in flight, and Miz is going to be rolling this stuff out fairly soon, so keep an eye out for updates to all of that, very excited for it and what it’s going to bring for the company. And then Functional OKRs project is still proceeding. I think by now every function has gotten the prompt to start setting up their OKRs. There’s going to be a lot of effort involved in doing this, but it’s going to pay dividends long-term, and keeping everyone aligned and rowing in the same direction.
(05:42):
And then lastly, website redesign memo. We are kicking off the next phase for the site and levels community newsletters in development. We’re going to be pushing that out in September, Cece’s taking lead there. The Getting Started Cohort B finished on Tuesday, the wearable challenge.
(05:56):
On the whole new level, we had Charlotte and we had Drew Manning from Fit to Fat to fit on YouTube as well as audio episodes. And then we had David doing a deep dive into his perspective, his experience as a Levels co-founder and leading design in the very earliest stages, design and product. We’ve got a new modal on the website capturing emails. It’s one of the first changes we’ve made post liftoff, where email capture is not necessarily early in the flow.
(06:25):
A bunch of great content went out, including a deep dive into the Dawn effect, which is very relevant to me, selfishly. Levels hit 25,000 subscribers on YouTube. We had some really nice boosts from David Perlmutter and Ben Greenfield this week on our content. Dr. Perlmutter reshared our year-end review and Ben Greenfield reshared our Healthy Condiment rundown, which both on Twitter. We are doing a new project on levels rendering, so seeing what the product actually is in sort of a real lifestyle context. So we’ll be developing some new beautiful renderings with Mr. Craker.
(07:05):
And then this past week, I was in California in SF for the Abundance 360 Platinum Longevity event. Got a chance to share about our concept of biological observability, our thesis of biological observability, as well as get a number of VIPs set up with the CGM program. Had a ton of amazing support from Jesse and Mike on site. It was a very successful event. We’re going to learn a lot, kind of the first of its kind that we’ve done and definitely some opportunity to improve. But Peter Diamandis and his team at Fountain Life and Abundance really loved what we’re working on.
(07:37):
We got some projects going with Athena to rapidly iterate on our approach to developing assets for YouTube, for example. So thumbnails, kind of insourcing what is otherwise a very tricky thing and requiring many iterations. So I really like the scrappiness here. Excited to see how this goes. We pushed out an effort to kind of boost our focus on ratings for the app that’s in the app store. Had a nice bump from, I don’t know, the three somewhere, up to four, continuing to make progress there.
(08:08):
I want to highlight this image here. I’m not going to say the name of the brand, but there’s a very large food brand that has been paying close attention to what Levels is doing. So some of their team repping Levels, wearing it, and they’re very closely watching and trying to build, it seems, product lines that actually do maintain glycemic control, which I have to say, I’m very excited to see. I will hold out hope that this is going to be a trend that will accelerate. And then some really nice just boosts on Twitter from people who are using Levels, and focusing on the rapid improvement and continuing that trend. I think that’s it.
(08:50):
That I want to welcome Cole Sager. Cole is a partner to Levels, a member of Levels. He’s a D1 athlete. He’s also been ranked in the top 10 in the CrossFit Games. Pretty much the ultimate test of a beastly physical fitness specimen, and a husband and father lives in Washington. Cole, I really appreciate you taking the time to hang out with us, really respect what you’ve accomplished in your career and appreciate any insights you can share with us on your journey, what you’re excited about in metabolic health.
Cole Sager (09:21):
Yeah, thank you. Thanks for having me. That’s a very kind comment. I appreciate that. It’s been a really fun career getting to compete in CrossFit, been doing that for the last 10 years now, which I mean you guys probably all know, goes really fast, but that’s been amazing. And thanks for having me on here. This has been really cool. One, just even just to sit in here and listen to you guys talk about it. When I first found out about Levels, both my wife and I were very intrigued, very excited. We both take Metabolic Health pretty seriously, and it seems to me, I’ve gained so much knowledge from Levels and especially from the blog posts, and things like that, it seems like on my Google Chrome, I’m constantly cycling through a new tab of a blog post of just reading and absorbing, and seeing, hey, what can I take in to better my pursuit as an athlete? So thank you. To whoever writes those blog posts, those are great, really enjoying those.
(10:18):
First off, just to talk about what Levels has been helping me from a training perspective, I first recognize that I’m kind of in a different category from the general population, as an athlete. So there’s a lot of information out to the general population of how they should live and how metabolic health should look, but that doesn’t always coincide with me as an athlete. And I recognize that I stress the body to an extremely high degree on a regular basis, whether that be from a physical perspective, but then also from a nutritionalist perspective. I know that I’m putting my body through a lot of refueling stress, if you will.
(11:01):
So one of the things that I’ve really got a lot of insight from was actually the Metabolic Health Panel, that’s been really cool to see integrated into Levels. I first did the CGM and that was very insightful, and I’ll get to that in a second. But actually the Metabolic Health Panel is something that I’m really excited about and to see that further, because I’m constantly putting my body through stress and damaging it, and I want to make sure that I’m not causing any long-term health or how can I combat that from a longevity standpoint, long-term health and longevity down the road. So that has been really insightful and really cool for me.
(11:40):
And then actually, this is kind of an interesting thing that I did, but I used the CGM to kind of experiment with foods to see what might give me a better metabolic response for performance, like during a competition weekend. And essentially looking at what foods do I respond to and how do I respond to them in a certain way to get the highest spike in blood sugar, so that I can get an extremely high insulin response. For anything to aid in refueling and nutrient replenishment or even into getting into better recovery foods that maybe have a good spike, but then taper off, so they can help carry me through an event. And that’s been kind of an interesting thing to look at. And again, just coupling that with a lot of the blog posts and information that Level has been putting out.
(12:35):
And I just, again, really excited about what you guys are doing and I think that there can be good education for athletes, and I think that I’m doing a lot of the work to try to put that together, but I think it’s there. I think there’s something there and being able to look, especially the Metabolic Health Panel, and it sounds like maybe you guys are coming out with a 2.0, which is kind of cool, but being able to look at that from a long-term perspective, I think, is really good for athletes to know that, hey, we do stress ourselves to a high degree and we may be what might look like causing some damage in ways, but hey, let’s just take a look on the inside and see what’s truly happening, and that can help guide our decisions as athletes. And I think that that’s something that’s really helpful and I’m excited about.
Josh Clemente (13:24):
Really awesome perspective, because I think there are two challenges. First, you have truly elite athletes who are deliberately pushing the boundaries. And you can think of racecars are in the shop after every single race getting oftentimes complete engine rebuilds, and that’s because they’re running at the functional performance limit, and that’s a known condition. This is not the Prius using user condition.
(13:49):
So there’s a lot to learn there. And admittedly, there are only so many in that sample size and the vast majority of people are not that elite athlete group. So it’s really fascinating to see the user experience and how you’re experimenting to potentially even push well outside of or in the opposite direction of our scoring framework. There’s also though a lot of people who look to elite athletes as that’s the peak health, and so I should be doing what they’re doing, even though I’m spending the vast majority of my time at my desk not cranking out reps, and so fueling like elite athletes. And so helping people to understand contextually what is the difference, peak performance versus peak health. And I think Metabolic Health Panel’s a really great way to look at that.
(14:37):
I’d be really curious from your perspective… I think if I could try to characterize what you were describing there, the objective is to function at peak performance, but to maybe minimize unnecessary or excessive damage from that lifestyle. So I’d be curious, what tools could you use? What features would you like to see to aid you in that goal?
Cole Sager (15:00):
I think when I was going through the Metabolic Health Panel, I think one of the things that… It gave me a good picture, a good snapshot, of where things are at, and I could gain enough information from that, and go into each individual biomarker and look, okay, this is where certain things are at. But one of the things that I felt like was missing with the Metabolic Health Panel was deliberate and direct recommendations on, hey, this would be a strong way to combat that on a lifestyle basis. I mean something as simple as like, hey, eat a bowl of oats every morning and this helps with whatever. Something as simple as that, but on a daily basis, what are some of those lifestyle perspectives or things that I could be doing to help combat that?
(15:53):
Because you’re absolutely right, at the end of the day, I want to operate at peak performance, but my end goal is also health and longevity. I’m a fitness fanatic, I’m a health fanatic, and so I want to be operating well. I have this for a short season of life as an elite athlete. When I step away from that, I want to know what are the things that I should be doing on a regular basis. And sure, I’ve been doing this for over a decade so I have a good idea of what that will look like, but maybe not all individuals know that of like, hey, my A1C was really high for where it should be, what are the things on a daily basis that I could implement in order to help lower that and have better metabolic health?
Josh Clemente (16:36):
Yeah, I love it. I’m just having being able to focus on maybe the priority as it relates to some real impactful health metrics, and knowing that there might be some point where you pull back on a certain lifestyle action that you’re taking and then you hit a performance wall or you see some deterioration performance, and then you kind of ratchet it back up a little bit. But you can find the balance potentially. And that’s what we’re all very excited about here at Levels, is being able to meet people where they are, no matter what. Your approach, your goal, is very different than a vast majority of the Levels users’ goals, but that doesn’t mean that the tool set can’t be optimized for your journey or at least balanced towards your personal objective.
(17:18):
Cole, I really appreciate you sharing your experience with the team. And I know on behalf of all of us, this is the most powerful stuff for us, is to hear from real users. If there’s one thing that you would hope to see, and it might be that what you just described in the Metabolic Health Panel, if there’s one thing that you hope for Levels to introduce in the next few months, could you just describe what that is and feel free to say what I just described as the Metabolic Health Panel 2.0?
Cole Sager (17:42):
Yeah, gosh. I mean, that’s a great question. I probably would just, I mean being put on the spot, I’d probably just lean towards that, of just a better combination of, hey, these three things are a little on the high end, here are some lifestyle implementations or tangible things that you could do and implement to help lower those and optimize those. And I think that I would, like you said, that wouldn’t just be good for me as an athlete, that’d be good for anybody. If your glucose is high and your triglycerides are high, here are some things and how they all play in. I think that was one of the things that I thought was so interesting about the health panel was, in looking at the information, they all integrated and they all played, and it showed that they all worked together a lot more than I realized. They’re not so independent of each other. And so what are the few things that could help optimize all of them to the best of the ability?
Josh Clemente (18:40):
I love that. Yeah, expressing the framework better is definitely a key focus for us, describing how this is a symphony.
Cole Sager (18:48):
Absolutely.
Josh Clemente (18:49):
Well, thanks a ton, Cole. Please feel free to hang out for the rest of the meeting. We’ve got kind of a packed show here, but I know you’re busy hanging out. Otherwise, I really appreciate you taking the time.
Cole Sager (19:01):
Yeah, thank you guys for the time and thank you for what you guys are doing. I think it’s very cool and I’m excited to see where Levels goes.
Josh Clemente (19:07):
Awesome, thank you.
Cole Sager (19:08):
Thank you.
Josh Clemente (19:10):
All right, onto the culture and kudos side. So I want to start off by showing some themes here, not to name any specifics, but essentially, we’ve had a couple great opportunities to highlight people like Riley, Paul, Karen, going out of their way to maintain a super high bar, especially when dealing with third parties, outside vendors. This is a really difficult thing. It doesn’t come naturally for many of us. It’s tough to… Especially when knowing that there’s a particularly challenging process in play, a migration from different platforms. It can be easy to let the standard fall temporarily, but the problem is that that can set the standard going forward and it’s very hard to recover from at that point once an outside team, an agency, has gotten used to a certain necessary output to keep the team happy. So I just want to say, I really appreciate seeing these examples and for those that highlighted this, thank you for doing so.
(20:02):
But holding our vendors accountable to the SLAs we have requested to the statements of work that we’ve requested is critically important. And as we get larger and as we have more of these moving parts, it’s going to get much harder. And so each of us practicing this and doing so, and asking folks like Riley, what are the tips and tricks to be able to do this without feeling a lot of tension and personal stress? I think we could probably provide some education inside the team for how to do this well, and these are exceptional examples. So thanks to both of you.
(20:36):
Also had some appreciation for Lauren, who is taking on a ton with Functional OKRs project, just getting involved in a huge number of the processes on company objectives, and simultaneously rapidly ramping up and killing it on podcasts and IG live circuit, and doing an amazing job of developing a perspective of expertise on women’s health to share with these platforms. So really appreciate the kind of context switching that is necessary in handling both of those types of work.
(21:05):
And lastly, shout out to Mike and Jesse. We had our first onsite with the Platinum Longevity Team and event. This is a shot of Mike with the whole table with all of our kits there. The True Pill team came through really strongly. They hand delivered kits to us. They’re really an amazing example of partnership. And we learned a lot here, we’re going to continue to improve. We may or may not do many more of these events, but I appreciate Jesse and Mike for being… There were some situations where, for example, we weren’t able to get international VIPs set up the way we thought we could, and that could be a very difficult situation, there could be a lot of stress and fallout there, but it was exceptionally well managed. I think this is a core competency of Levels at this point, is helping our members see the positive side and making sure that we issue Mia Culpas, and make sure we do the best possible situational handling. And there were a few examples of this. We’re going to do a full retro on the whole event, but just wanted to shout them out as well.
(22:02):
Okay, so we got some meetups here. Jackie and Steph hung out in Colorado. They’ve been good buddies since starting essentially together and I guess got to meet for the first time, which is always exciting. Looks like an awesome hike. Charu and Paul in Montreal hanging out. And then in the other CA, in California, I got a chance to hang out with Miz and Moz, and they took me gravel biking for my first time. And then Jesse, Mike, Miz and I kind of cruising through SF. So always fun to see these pictures and to get to meet more of the team.
(22:34):
All right. The main thing, Levels shows you how food affects your health. We’re still on task here, should be the main priority for everyone. If it isn’t, please raise the question. And trickling out of that main thing are the top level objectives, still member retention, consistent growth, member health improvement and new member acquisition, to be consistent growth there. So both growing the new membership and also growing the rate of retention. With that, we’ve got Moz with the product update for the week.
Moz (23:09):
Welcome to the Friday Forum Product Update.
(23:09):
Lots of great work going on by the team in both shaping and creating new concepts. The two projects that are being shaped are healthier food choices and labs 2.0. And there are number of concepts that we’re focusing on to really make the product experience much deeper in a way that would drive effectiveness and engagement to help with retaining members over a long period.
(23:29):
Let’s jump into it. On healthier food choices, Allan and David are going through a number of iterations on figuring out how to present people with proactive choices to when each meal. For example, in the morning, when lunch and when the evening, and really being thoughtful about how do we actually present the right content, give people enough choice, but not too much, and also report them by making the right choice. And we’re doing this in an iterative passion. And we’ll also, once we have the initial concept locked, we’ll run it through UXR to make sure before we build anything, we are actually creating something that people value and will help them make the right decision. So a lot of great work, a lot of testing and really doing in skip fashion, and adding on top.
(24:09):
On Labs 2.0, we’re also taking an innovative approach here. The intuitive thing is to expand the labs and provide more, however unintuitive, but something that what we think will drive a lot more value, is to actually provide a set of labs that are actionable, they’re not confounded and people can easily understand, and present it in a way that people understand. So this is really around breaking them down and explaining to people in video form, what does it really mean for them and what can they do about it? And the approach we’re taking is we’re breaking down the labs into two energy systems. So your glucose energy system and your fat energy system, and explaining to people how those systems are optimal or not, and if not, what can they do about it? And really talking about metabolism as those [inaudible 00:24:49] that power your body.
(24:51):
So really excited for this work. There’s a lot of deep thinking going on and a lot of cross collaboration with the research team, really done in the message and creating masses. Again, in this one, what we’re going to do is, we are going to, before we make anything, we’re going to actually run a UXR exercise to see whether this approach is understandable and really adds value for people, and people feel like they know what’s next. Really want to create something here that’s repeatable, that people get this as often as they need. It’s almost like a check on your metabolic health that could happen quarterly for people that need it, or longer if you are optimal. So really this is designed to be something that is really a core part of your Level experience and it is easy to implement.
(25:30):
I actually just relapsed this evening. It took me maybe a minute to order on the app, and so it just feels so magical and frictionless, and really getting also the information in the way that people understand it’s actionable and want to repeat it is really the goal here. It is really the measurement of the efficacy of our program. So really important part of the program, we’re really investing to develop this further. Great work done by Guzma, Victor, Allen and Taylor.
(25:56):
Talking about product concept exploration. These are on the earlier stages, but what we really wanted to do is really invest in these core pillars that will make our app and experience a lot richer. So jumping in, first of all, content is really the lifeblood of our app. We do this very well outside the app. And what we’re really thinking about is going back to the drawing boards and saying, what are the types of contents that we really want on the app that really would drive engagement and efficacy.
(26:20):
And really we are having that project, derived by David, and he is all working with a lot of people to bring this together. Stacy has done a really fantastic job reaching out to our content producers and kicking off their personality driven insight work that really had great internal feedback from the alpha that we did. I’m really excited to see this come together and Stacy’s really done a good job at getting quickly this project off the ground, and hopefully we’ll have something to look at in a week or two. So really excited about this. There’s other content work that also is happening that might be as helping lead and we’ll lean on heavily on other parts of the organization, whether it’s Mike Kaney and Ben’s team to really drive a high volume of high quality content into the app in a way that is actionable and meaningful for people. So really investing in content in a big way.
(27:15):
Next one is personalization. In order for the app to be effective and engaging really needs to be personalized, and we’re going to iteratively go that way. We’re going to work with data science, Jason’s team, to really drive this forward. Things like having context of your day, for example, if somebody has slept a certain way or a certain amount, or having a certain amount of exercise, obviously their food response is different. So understanding that concept for each individual and also understanding that individual based on demographics, what age, what gender are they, what are their motivations? And really tailoring the content, the rewards, the motivation around who this person is, who our member is, what resonates with them, and what is the context that they’re living in. So really thinking about personalization in a deep way, but also building in an iterative of MVP stable fashion. So really excited for this work and investing in this area, and creating concepts that will get us there. And ultimately what this hopefully will help us do is create the opportunity for us to be a trusted partner of people’s health, because we are providing things in a personal way instead, that will work and are engaging, versus something that just feels and falls flat.
(28:22):
The next concept is Levels levels. As you can see, this member is climbing levels. This is the idea of really showing a sense of progress or creating that extrinsic motivation that will compliment intrinsic motivation that people have to make their health better. We know that our motivations ebb and flow and really the extrinsic motivation is part and parcel to that, and really help us to visualize how we’re making progress in our health, and based on the actions in the app that we believe will improve people’s health and wellness by completing those actions, showing that back to the member by helping them increase their levels in Levels. So we’re really excited about this. It is a part of investment in rewards and extrinsic motivation, and so there’s a lot of thinking going through here of how do we incorporate that in a way that it feels contiguous and integrated. So a lot of thinking going on there.
(29:16):
The last one is community. Really using community and the best thinkers at the space, like Seth Godin is thinking here, and also the great work that Cece and team are doing, is how can we use community and our tight-knit community that Levels has to help improve the ability of members to do the things that would be better for their metabolic health, or create or enhance the motivation or the trigger. Really how do we use community in a way that we can drive these behaviors that will help all members improve their metabolic health. Community is a very complicated and important topic. So this is something that we’ll tackle after we’ve tackled the three prior priorities. We’re constantly thinking about this and really looking forward to working with the community team to bring some of these ideas into the app.
(30:06):
So anyways, hopefully this was helpful and you all have a great weekend. Thanks, bye.
Josh Clemente (30:12):
Lot of good themes there. Excited for most of those. Thank you Moz and team.
Speaker 4 (30:18):
You might’ve seen part one of my demand memo that I rolled out on Monday morning. This is a backwards looking kind of retrospective of things that have happened over the last couple of years. Feel free to take a peek at this later if you haven’t already. Basically, I cover five topics, starting with a wait list retrospective and talk about the realities of new member flows during the era of wait list and beta, hopefully some interesting things in there. The next part of this will come soon, but basically outlines things I think we should do going forward. But take a look, it was a good one.
(30:48):
Circling back on some experiments, since I was out last week, I didn’t get to share this. I did an experiment replacing the illustration in checkout with the product box. The product box lost, in fact, the illustration won. I even let it go a little bit longer, because I was surprised by the results. But the numbers don’t lie and the illustration did better, so that was rolled back. The next iteration of this experiment, number three, is taking a different approach on slide one, giving a full accounting of what you get when you join, what will require an additional purchase and what’s included. Thanks to Jen for the language help on this. That experiment is now live on the homepage. It’s only been a couple of days, so far we are seeing the longer one win. The thinking here is that it would add friction on step one, but when it comes to subscription selection, which is a place we’re losing a ton of folks now, we would lose fewer, and that’s what’s happening. So it’s netting out higher so far. It’s too soon to be significant, but I’ll circle back on that next week.
(31:38):
Another experiment, purchase reminder email. This has now been live for one month, so I wanted to circle back on this. What this does is if you are in checkout and you get to the almost last steps, actually have your email address, we send you a reminder a couple hours later, since we’ve all been there, you open it in a tab, you think you’ll circle back on that, and then you forget and don’t do the thing. So this reminder experiment, basically it’s an AB test, half the people who abandoned get it, half don’t. Those that get it are doing better. I have not taken a very deep look at this yet, because we just hit one month now. But we did see a meaningful lift in people both going back to the website to finish off, and among those that do in both groups, those that get the email are finishing at a higher rate. So that’s good. Not just how it works. Let me check. We talked about that.
(32:18):
Another experiment, we took Charlotte’s story and packaged it up in an email. Thank you Ben and Mike Kaney for that. We all really loved hearing from Charlotte when she was on the call a few weeks ago. We sent it to about a thousand waitlist members who had indicated weight loss as an interest. The open rate on the email was as good as it always is. Click through rate, very, very low. All the links worked. I think this particular positioning did not hit in the way we thought. It’s not the end all, this is just one experiment, but that was the result, not even… I wouldn’t do something like this again without a different approach.
(32:47):
And then finally, in my memo that I just referred to, one of the things I talked about is how we collect far fewer email addresses now, that’s because for two reasons. One, the wait list is over, obviously we got a ton of email addresses from that. And the second reason is in checkout, we used to collect email addresses the first step and we recorded it, and now we collect it towards the end. So we’re just getting fewer email addresses. A great place to collect email addresses is the blog. We’re getting thousands of new visitors a day through search. And so Jen, Ben and I did a little experiment yesterday of putting a pop-up, but I hate that word, so maybe a registration modal that pops up on our properties after 30 seconds that asks for a subscription. We collected 100 email addresses in the first day, which would be about 3000 a month. That moves a needle for us. So this is just the first cut of it. There are different ways we can traffic this.
(33:31):
It could wait longer, it could wait until your second visit. We could use different imagery or language, but it’s the first step at marketing a little more explicitly on the blog. The emails we send are amazing and get opened really significantly every week. So I don’t feel too bad about luring folks into these, but this is what we’re doing. Let me know what you think. So far so good on this one. If you’re curious about what I’m working on on an experiment, you can check out the demand capture experiment tracker, find that anywhere notion pages are sold. That’s all for me. Thanks for the long one.
Josh Clemente (34:01):
Awesome. All the experimentation there is amazing. The data in the first half of Jam’s writeup is very counterintuitive for me, very insightful, highly recommend everyone reads through that. It’s approachable, it’s a quick read and super interesting and important for us. Let’s see if I can get through to the next slide. Okay. Onto Tom, Partnerships.
Tom (34:29):
Hey team. Okay, experiments and learning for partnerships. Let’s dig into it. We’re going to start with just a high level look at partnerships and what’s been going on. I know there are a lot of new team members here, so I feel like this is helpful, and then we’ll dive into some recent experiments and results. Going to probably breeze through this pretty quickly, because there’s a lot in here.
(34:46):
All right, so as a reminder of what partnerships is focused on, you can dig into this doc more, but briefly, we are laser focused on content creator relationships. So think thought leaders in the space, paid sponsorships, like podcast advertising on Tim Paris, Andrew Huberman, David Sinclair, as well as content creators broadly speaking through our affiliate program and some organic relationships. This list here includes a bunch of other things, some that are technically partnerships at Levels, others that are not, but none of them are focused right now.
(35:12):
Okay, reminder of the why behind partnerships. Right now, the number one focus is on new member acquisition. I think historically, maybe equally focused on just general education and awareness in the early days, certainly building a premium brand by securing a lot of the top thought leaders community, and then just getting to a lot of these individuals before competitors did. This is what the marketing funnel generally looks like, awareness, consideration, conversion. We think about it in terms of demand gen, which is the top of the funnel, and demand capture, which is the bottom of the funnel. Ben is responsible for demand gen. So think getting people to the blog or YouTube channel and initial hearing about Levels. Jam’s responsible for demand captures, so think about optimizing the checkout flow or landing pages to get as many people as possible converting.
(35:50):
Partnerships spans demand gen and demand capture. So think you hear podcast ad, some people might then sign up our newsletters and purchase three months later. Or you might hear a podcast ad and be so compelled by David Sinclair that you purchased immediately. And why do we care about any of this? Well, we’re in growth mode now and we’ve got targets to hit, 10% month over month.
(36:07):
All right, so the big picture, we’ve been in beta for a couple of years and we’ve done a good job. We’ve secured majority of the top health thought leaders in the space. A lot of our revenue to date has come through partner codes, in part because there haven’t been any other way to get the product, aside from some other different types of tests. And we’ve learned that partnerships can drive a ton of sales, but at the same time, it’s not highly predictable and highly controllable. And the reason is, among others, that there are some power laws at play, like there are certain partners that drive orders of magnitude more than all other partners. And then similarly, there are outlier events that will pop up and basically determine whether or not a month is high performing or low performing.
(36:42):
Now we are in growth mode, and high level that the picture so far has been that attributable partner code conversions have dipped and most of this is expected, some of it maybe not as much. But some of the dynamics here that general availability allows people to purchase through the websites, we’re going to see fewer people coming through the partner codes, especially because the partner codes are not offering anything. In the past, they were offering skip the wait list, but they currently don’t have any other incentive associated with them. Jam’s working on a demand capture series. More info on all that there.
(37:09):
Sinclair performance has been by far a top performing partner and that’s finally dropped off after six months or so post podcast advertising. Conversion rates, this is something we know we need to improve, the checkout flow. Conversion rates are just lower across the board than they were in the past, so we need to improve that. And then Tim Ferris and Andrew Huberman have not been performing as well as expected. And I would say this is certainly related to these dynamics, but I would say also independently, the just amount of engagement on the links is a bit lower than we anticipated. And our general goal is that we are going to experiment and scale partnerships. And so that’s the focus.
(37:42):
And this is looking at all new members by category. Red is paid conversion. So you’ll see that this has sort of steadily declined. A lot of this in here was intentional, based on scraper capacity. And then this is a result of some of the dynamics I just mentioned.
(37:55):
All right, so high level, how we think about experimentation when say a campaign is not performing the way we would like, there are a few areas where we can make changes along the funnel. It starts with just selecting shows and audience, and then the actual ad read itself, as well as the offer, or lack thereof in our case, that we’re extending to an audience. Then landing page optimization and checkout flow optimization. So these are all areas that we’re currently investing in experimenting with. As mentioned, current experimentation, everything from partner selection to we’re going to be expanding with a new special offer at some point soon. And then there are also some new funnels. We’ll go into this right now, but these are sort of categorically different. So an example would be testing whitelisted Facebook ads with Mark Hyman. More on that in the future.
(38:36):
Okay, diving into some performance update. As most of you know, Tim Ferris, Andrew Huberman have not been performing anywhere close to what we saw with David Sinclair. Some of this is expected, some of this is disappointing. This is what the daily new member conversions look like with Ferris. And for the most part, all the data, besides Tim that I’m going to be showing you now, is just podcast ads. But Tim does have a couple of newsletters in here. So we’re seeing, so far we just crossed a hundred members. CAC is still super high at 1100. We’re trying to have CAC, which is customer acquisition costs, to be below 200. The goal of course is over time, the more we advertise, the more these numbers either go up or come down, depending on the numbers. New members go up, CAC goes down.
(39:14):
So we’ll see. That’s what we’ve been told by other people in the space, other brands, is you can’t expect any of these numbers to look good after a couple of ads. So we’ll find out. Just by comparison, David Sinclair, this is what that looked like. Obviously we had more ads running, but across the board this just performed better in terms of clicks, in terms of conversion rate. And obviously, CAC was very low for Sinclair. And Huberman is new, but it looks similar to Ferris and role, in that people are kind of trickling in and CAC right now is north of a thousand, and we’re going to find out how this plays out frankly over the next three to six months as we advertise more.
(39:47):
This is looking at CAC by partner. Again, just providing some benchmarks. We really want it to be closer to Sinclair or at least under a couple hundred bucks, but most partners, in terms of pure podcast advertising tests, are quite a bit higher than that. At the end of the day, what we really want is our holistic marketing spend divided by number of conversions per month. We need that CAC to be around, I think it’s 110. And so far, we’ve been hitting that, so that’s great. But even within these experimental podcast ad tests, we want to get these numbers down as low as possible. And again, we expect them to continue to drop with time, so that will be the real test.
(40:24):
And then I just wanted to flag another metric that we’re sort of using internally and going to be using more, is adjusted CAC. So what this refers to is that, because people can hear about you on a podcast, but then buy through the website, especially in our case, because there’s no incentive to purchase through the podcast link, you no doubt are losing out on some attribution. So for example, 100 people might purchase over the next couple of months from hearing an Andrew Huberman ad, but we’re only going to see say 30 of those actually use the code.
(40:53):
And so what most other brands do, and have advised us to do, is use an adjusted CAC in a particular multiple, so like a three x multiple. So if you see 10 new members come through Huberman, you should assume over a given period of time, actually maybe 30 people have purchased because of Huberman. And these are not the numbers that we report to our investors to signal the health of the business, but they are numbers that are useful in determining whether or not we should continue to run ads with a particular partner. So we would look at adjusted CAC if we get Ferris down, for example, to 150, even if the unadjusted CAC is 450, we, in that case, would continue to advertise with Ferris. So a lot of considerations here and the bottom line is, it’s still very early, so excited to get a lot more data and continue to keep everyone updated. All right, happy Friday, thanks for listening and watching.
Josh Clemente (41:38):
Really interesting. Can’t manage what we don’t measure. So thank you Tom and Jackie, and others, working on the partnership side and I’m looking forward to seeing how these evolve.
(41:49):
All right, into the hiring updates, no update here. Taylor is joining us in just a few weeks now, September. And then on the open role side, still have software engineering with both backend and mobile roles opened, and then the R&D engineering role. Something else, if you or someone you know is interested in Levels, in the culture and what we’re building here, and what our mission is, check out levels.link/careers, shoot us a general application and we will keep you in mind when new roles open.
(42:17):
All right, we are individual contributions with time. I’m going to jump out of the screen share and we’re going to do this by hand raise, as usual. So I will kick things off, but if you’d like to share, the prompt is, love to hear from you on something you’re excited about this week. Personal is encouraged, professional is always accepted. And for me this week, personally, it was nice to hang out with Miz, and Moz, and Mike, and Jesse in SF. SF’s a beautiful place. I had a lot of fun getting out on the gravel bikes.
(42:56):
Professionally, I went back and I was catching up on comms and threads, and there’s a ton of awesome stuff happening and I found that the rigor, the quality of the projects, something seems to have improved in a pretty dramatic way. There’s data driving the vast majority of our programs right now, in a way that maybe we were just at too small of a scale or just didn’t have the resources to dive into it. But I think there’s a real dramatic improvement in rigor and sort of objectivity across all the projects that are happening inside Levels. It feels really great.
(43:27):
And then lastly, I want to just shout out again, the Personas project. We’re going to do a lot to try and surface these personas more aggressively, but they’re very good. They really help you apply someone that it feels like you know, to the problems we’re trying to solve, and it’s way more powerful for me than I had thought it might be. Just knowing the types of brands that this person might use, it’s all very important. So highly recommend that project. Read into that. Rebecca?
Rebecca (43:56):
All right. Well onboarding is complete after today, so that’s very exciting. And I’m just feeling a lot of gratitude, because before I came to Levels, I had my own health coaching business and it was very stressful having my own business. I was losing sleep, all the things that could be going wrong physically were just kind of starting to happen, and I’ve been getting the best sleep of my life over the last two weeks. So I don’t know, there’s a lot of things at play. I’m also now wearing Levels, so that’s been helpful too. But just being here has been really amazing for me personally. Thank you.
Josh Clemente (44:29):
Love to hear that. That’s awesome. Congrats on finishing onboarding. Chris?
Chris (44:35):
On the Level standpoint, I just want a huge call-out and thanks to the support team. I’ve been throwing them some wrenches and some big asks from the team collectively, and the way that they’re kind of stepping up to help out to pitch in, even with a queue that we just can’t seem to get in front of. So it’s already been busy with support volume and I’m asking them for more help on a lot of other fronts, and it’s just really encouraging. I’m just proud to be part of the team.
(45:06):
On the personal side, my friends from Chicago and their 14-year-old son officially left, which means I can go back to having salads and athletic greens, and Element, versus Hot Pockets and Pop-Tarts. So my metabolic health is about ready to make a huge improvement. Unfortunately, I didn’t have sensors in time to actually do the full AV experiments. And I’m excited to get back to farm life. My mower has been falling apart. One wheel came off, I fixed it, the other wheel came off. When you’re mowing and you do a nose dive into the ground, you’re like, all right, I’m beginning to think it’s me that’s the problem. It’s not my mower, it’s the mechanic. So should be back, fun time on the farm.
Josh Clemente (45:54):
Sounds awesome. Love it. Justin?
Justin (45:58):
I feel like it’s been a month and a half or so since I’ve been on a live forum, so it’s great to be live again. Had two weeks off and it’s just crazy how fast you can get back in the swing of things after being off for two weeks, just with how everything is documented and Threads even just helps, I think, versus Slack. Looking forward to other members to finally trying out the stability ring, even though it’s just a small handful, but really excited to find out what they think. And personally, I don’t have anything planned this weekend, but it’s supposed to be nice, sunny, and not too hot. So yeah, I’m excited.
Josh Clemente (46:38):
Welcome back, Justin. Great to have you. Miz?
Speaker 9 (46:42):
Yeah, great seeing you in person, Josh, and also Mike D and Jesse, and MAs, of course, so enjoyed that. On the work side, excited about this kind of suite of memos and processes coming out. So a lot of the questions that people have been asking about require a lot of pieces that need to fit together as a puzzle. So there’s the job titles memo, there’s our promotions process, there’s performance and feedback, and ratings. There’s manager training on 360. And these all kind of had to come together at the same moment and I’m happy they finally did. So we’re pressure testing that with some people managers later today and early next week, and then rolling this out a little bit wider. So it’s been great working with [inaudible 00:47:23] on that. It’s nice to have her people management expertise lending a hand. So that’s been great. And otherwise, nothing too special this weekend. Have parents visiting late next week, so getting ready for that.
Josh Clemente (47:37):
Super nice. Ben?
Ben (47:41):
Yeah. Work front, super stoked on the memo Paul did about website. It reminded me of Corinne’s memo on international. It was just so deep and there was so much diligence that went into it. So love seeing all the thought that goes into it and really dissecting all the infrastructure behind it. So that was cool. And then also, week over week, I know I feel like a broken record saying this, but it’s so fun being in this growth mode now, where there’s a lot of demand gen and demand capture overlap. And so running all these little lever experiments, we’re like, let’s try this thing. We still have the ability to be sort of, I don’t want to say scrappy, but somewhat calculated. We measure it, we look at what’s happening and we’re like constantly tweaking these things, based on we’ve got something online now, which is we have these targets to hit. So it’s very energizing to do that. And being able to work with [inaudible 00:48:32] is super fun. So love that. And then yeah, back from the lake, that was completely oh oh oh, didn’t touch a device really the whole time I was away and it was super soothing, is probably the word. But yeah, fun to be back and just love being out there. So that is it.
Josh Clemente (48:51):
Glad to hear you stuck with it, Ben. Looking forward to hearing how the lake was. Tom?
Tom (48:56):
Yeah, plus one to everything that Ben said about being in growth mode. I think it’s going to be a fun few months, probably a little bit stressful, but I’m excited for it and I think it’s fun having the pressure on us to figure things out as quickly as possible. And I think going into launch, I was thinking to myself, we’re either going to find out that this is going to be easy for the next six months or difficult, and I think we know enough to know that it might be a little bit more on the difficult side. But as a result of that, I think we’re going to learn a hell of a lot more about all of our sales and marketing efforts across the board, demand gen and demand capture. So I’m excited for all of that learning to happen at maybe more of an accelerated pace than we initially anticipated, but I think we’re going to figure it out.
(49:41):
And then, yeah, personally, I had a family reunion last weekend in Iowa. My mom is from a small town in Iowa, that Chris knows, and that was awesome. It was just really cool to get a deep dive into, I’ve been there before, but how she grew up. And one of the highlights was my uncle Steve took us on a dive bar crawl that really took dive bars to an entirely new level for me and all of my cousins. So that was a highlight.
Josh Clemente (50:12):
I hope you have pictures.
Tom (50:15):
Karaoke pictures.
Josh Clemente (50:18):
Nice.
David (50:19):
I was going to plus one all the growth and conversion optimization things, that’s really exciting to see. And complimentary to that, I think at the same time that we’re starting to do that, the app is really tightening up and the experience is coming together very cohesive, very simple, very beautiful. And I think it’s going to set the stage really well for us as we enter growth mode, to have that premium experience and that simple experience come together. So like Justin mentioned, we’re going to get some good feedback on the experience with members, but overall, I think from those two paths, tightening up on the business side and growth metrics and the products really coming together, I think it’s going to put us out there for a good chance at success, for a definite success.
Josh Clemente (50:55):
Love it. All right. Anybody else on the shares this week? Okay, well, get seven minutes back. Yeah, overall, really appreciate all the rigor, like I said, across the board. Support team, thanks for crushing it still, despite the difficulty of the circumstances. And partnerships team, thanks for doing the deep dives and staying optimistic about things, even though our experiments don’t always pan out as flawlessly as they used to. I think there’s something to potentially… We’re going to learn more about demographics as well. It wouldn’t surprise me if Huberman and Ferris have bias towards a demographic subset that actually isn’t representative of what resonates most the way Hyman’s audience might skew maybe older and more female. And so those sorts of things are also very interesting and don’t necessarily speak to the quality of our partnership, more so to where our product is resonating right now. So anyway, across the board. Thanks for all the awesome work and enjoy your extra five minutes of Friday, and have a great weekend. See you all next week.