Ben Grynol (00:00:00):
Welcome to Friday Forum, March 17th. Happy St. Patty’s Day for those who are interested and celebrating. Here we go. Reminder, so Forum is our weekly sync time to celebrate all of our weekly wins, celebrate each other and very much do things that we should do sync as opposed to async. So moving into the weekly recap, lots of things have gone on this week. Super important to highlight. Team cohesion. I think that’s the theme of the week. You can really feel it, you can see it, and there’s lots of ways that qualitatively it can be measured and sometimes quantitatively it’s a little bit harder to measure, but very much firing on all cylinders. And a few examples, despite Alan and David might have a little pushback against the green, had to go for the highlight the accent, because it is St. Patty’s day.
(00:00:48):
So here we go. Eng metrics. Cycle time for getting code into production is down to one day from the peak of four. So these are metrics from the beginning of this year. Pick up time for PRs is down to 12 hours from a peak of 37. Review time is down eight hours from peak of 28 and deploy time is down to three hours from peak of four. So again, it’s just a matter of asynchronously we’re not tapping each other on the shoulder, we’re doing it virtually and making sure that we’ve got good process and good documentation in place. So big hat tip to the Eng team for bringing it all together.
(00:01:21):
Tightening up process. So this is something that Taylor has been spearheading, and it’s very much a process to make sure we get support tickets for bug reporting and Eng fixes actually taking care of and documented well and that all teams are in the loop, know who’s holding the ball and where things stand so that we can have that accountability and get things over the line. So hat tip two, Ops Eng and everyone working together to make sure that is a cohesive process.
(00:01:49):
And then Stacie’s Alpha 2 cohort is underway, and there’s so many aspects of bringing this all together. So multiple team members have been contributing to support the initiative across content creation, product insights, aggregating feedback, and just giving product feedback as it is. So you can see how important it is that we’re all working together and everybody is very much pushing that forward. So thank you for everyone for being a part of that.
(00:02:12):
Happiness. So we are above 90% for six weeks in a row. The streak continues. So this is something, as a refresh, Chris had explored what would it mean if we moved to bubble mailers as opposed to boxes for subscription orders only. And we get roughly $5 savings per order through USPS, which is 50K a month through True Pill. So a few more things to work out, but these are all the initiatives that help push us forward and help iterate on everything so everyone is very much taking ownership over all these different aspects that get us attraction we need and make us improve, improve, improve.
(00:02:47):
Retention rate for discounted membership. So this is something that Chris had looked into. There was an initiative for membership renewal in January and February that kicked off with some pricing tests, and he and Ryley had coordinated on it, very small number of people. It’s only over a thousand people, but signups at 199 had a retention rate of roughly 25%, and the 100 and $150 orders were roughly 33%, so pretty big delta of 8% there. The takeaway is that there’s so much room for optimization and for improvements to maximize revenue or subscription. And as we move into CGM optional product, this will be increasingly important to make sure that we’re pricing things properly, we’re pricing things in a way that people engage with and that we get great retention out of. So more work to come there. Lots of exploration underway.
(00:03:40):
The Alpha app, so this is more a call out to the team. The Alpha app is enabled for the team. If you’re not on it right now, make sure you message Cissy, there is a comms note about this, but test the app, give feedback about the experience and let’s continue to surface that and improve the product as we move into general availability, because all of our insight into what we’ve built and we are building is equally as important as getting member feedback.
(00:04:04):
Last. So A/B tests across the board. Beta 1 is now filled with 338 people. That’s the Beta cohort starting on April 3rd. There’s a How it Works page shipped in the checkout flow that Karen’s been spearheading. So against the control group, we’ve seen an increase of 10% on the conversion rate, which is great. And then there’s some other A/B testing things happening in the checkout flow for sensor education and comparison, so giving people more insight into the difference between Libre and Dexcom. And then social proof with review. So pretty standard e-comm stuff, but showing people that other people trust the product is so important and sometimes getting that conversion over the line. So lots of A/B testing going on there and very cool things in growth product.
(00:04:50):
So we will hammer the last little bit here, but podcasts Zach and I did on working with EAs, wasn’t planned to be a podcast, ended up turning out to be one and it was shipped on inside the company. We’ve got the app images, here are some Instagram templates. So V1 iteration, there’s still some development to be done, but to create more cohesion across our feed. Gardening email went out, so this is a touchpoint that we’re doing for anybody who is in the Beta cohorts to keep them warm. So the one thing we don’t want to do is have people sign up and then they go, “Wait, did you forget about me?”
(00:05:27):
This is the interesting metric though. 90% open rates, so best email open rate to date of any email, and 41% click-through rate. So again, these can be looked at as vanity metrics, it can be a little bit arbitrary, the group is only 170 people, but what it shows is that people are very interested in learning what we’re building and actually getting into this Beta and testing things out. So we’ve got another email going out this week, another one the week after, and then people will be onboarded. So that’s something that we’re working with product to do a handoff there.
(00:05:59):
50K subscribers on YouTube. So this continues to grow our audience there. Month or month is roughly 10%, so that is cool to see lots of perennial content that compounds over time, and fingers crossed that we’ll get that YouTube button for 100K pretty soon. We’ll see how it goes. We’ve also got… Oh, this is pretty important. So Paul has been digging into our email service provider. So Drip, we’re talking about moving from Drip to Braze, because we have very much outgrown the functionality that we need in Drip and Braze has deeper functionality across product and things like lifecycle marketing. So to be determined as far as a timeline for that, but it’s something that we’ve decided that we will move towards once we can fit it in the pipeline of priorities.
(00:06:48):
Last thing is abandoned cart. So we’ve got an abandoned cart email, the previous one focused very much on education and the new one is focused more on that social proof. So we’ve got the Apps Store rating the average 4.7 stars and we’ve got some reviews where it just shows people, “Hey, other people trust this product as opposed to the initial email.” The trigger was at four hours and it said, “Do you want more info about the cost of Levels or how it works? “And so we removed those education pieces and went straight for the other people who bought it, so you should too without overselling it. Just showing a lot of people, 40,000 plus people, have used this product. So we will test that, we’ll see the results and we’ll measure them and we’ll keep going.
(00:07:29):
So anyway, very long recap, but excited to move into our guest. So Paola Veglio, she’s here, she’s a Levels member, she is an investor and a tech marketing executive. So, very stoked to have you here and would love to hear about your experience with Levels.
Paola Veglio (00:07:48):
Okay, awesome. Cool. So hi, everyone. My name is Paola and yeah, I’m super excited to meet you all, because I think I can claim to be one of the very official OGs on Levels. I’ve been on it since 2020, I believe, which I think is a good record. But long story short, I met Josh and Sam pretty randomly through some mutual connections and they were kind enough to get me into the Beta program and I’ve been on it pretty much ever since, on and off, but really more on than off. So it’s been over two years at this point.
(00:08:30):
And I was thinking about it as I was walking here this morning in terms of what has happened in my health in the past two years. And I realized that the question is more like what has not happened in my health in the last two years. All sorts of things, of course, pandemic for everyone, but on a personal level, from the simple things like I turned 40 to the more unsettling things like discovering that I have some sort of genetic mutation with my liver. A number of things that have just made me much, much more health focused. And throughout this personal shift, my little CGM was always there with me.
(00:09:08):
So it’s been such a trip, it’s been such a journey. I think it’s really hard to condense it in a few minutes, but there’s two things that stand out to me. The first thing is that in hindsight, putting that CGM in my arm was literally the kickoff of an incredible journey in terms of personal health. I went down a massive rabbit hole of obviously everything that has to do with glucose and metabolism, but also intermittent fasting, Huberman Lab, podcasts, books, you name it, I’ve done it. And it’s all stuff that I had not even remotely considered before. So I’m super grateful to you guys for getting me onto that train.
(00:09:59):
The second thing that stands out to me is how the role of the CGM in my health has shifted over time. And I feel like there were three phases. I think in the very first phase, in the first couple of months or so, it felt like a really cool science experiment. I went a little Einstein with it and I was like, “Oh, let’s try to put these two different things in my body and see what happens.” Or, “Let’s try to do this workout and see what happens.” And it was so fun and so fascinating. I actually still remember going to SUGARFISH, which is, if some of you guys haven’t encountered it before, it’s an amazing sushi spot in LA. And the spike that I got out of it, it just literally blew my mind and I spent the following three days sending screenshots of the spike to a bunch of people and just like, “Look at this, this is insane.”
(00:10:55):
So I got super engrossed with the whole experimenting process. I got a ton of friends onto Levels, especially pre-diabetic people, but not just that. Actually, my main goal for today is to give a big shout out to your customer support, because in those days I emailed them all the time and they are hands down the fastest, nicest, most effective team I’ve ever encountered, so thank you for being so patient and so nice with me. But that was the beginning, those first couple of months. Then I think I started to see the benefits of actually paying attention to what was happening to my glucose.
(00:11:36):
And without quite realizing, I think my type A personality kicked in. And so I started to become a little bit obsessive about it, and for no good reason, but I guess in my mind… I don’t remember, I think at the time it was maybe being in green zone or whatever, aiming for that 10 out of 10 or a hundred percent stability out of a hundred percent. And part of me is happy I did that, because I really flexed in different ways, everything I do and don’t do. In the short run, it really worked out well for me. I was like, “I’m feeling fit.” I started lifting weights, I shed the pandemic weight, which was major plus, I was like, “This is great.”
(00:12:23):
But then I hit a wall and I started not sleeping well, my energy was off, my mood was off and I was like, “Okay, let’s hold the horses. What am I doing here?” And then the light bulb went on and I realized that I blindly ended up removing all sorts of whole grains. And I’m Italian so there’s no such thing as keto in my life. So I went keto without realizing, but probably a really bad version of keto, where it was just no whole grains whatsoever. And so I think from becoming that bullseye target it, the CGM has then shifted to become much more of a compass for me. I got my brain back on, I started re-feeding my body with what I intuitively feel it needs, when it needs it. And honestly, the CGM confirms. So it’s I’ve stopped obsessing about, “Let’s get a hundred percent.” Or, “Let’s keep the curve as flat as possible.” And try to optimize for other things instead, like, “I’m trying to optimize for no spikes.” Versus no curves, or, “Let’s stay above the 80% stability.” Versus aiming for the 100% all the time.
(00:13:47):
So I feel like it’s been a little bit full circle in a very nice wholesome place for me right now, where I feel like the CGM is a little bit like a live feed from my body. It’s a tool, technology is a tool, but context is everything. And ultimately, what matters the most is how you use it. So I went on a bit of a full circle there, but these days I actually use it… I love it even more now than two years ago. And it has been so transformational, but in a completely different way from how I thought it would be at the beginning. And these days I really use it as, “Oh, let me turn the TV on and see what’s happening in my body now.” But it’s so much more than food, it’s like the movement, stress, anxiety, I feel like it’s the best piece of biofeedback that we have today on our health from a holistic standpoint.
(00:14:55):
And so yeah, I’m pretty hooked. And I am very grateful for all the work that you guys do, and I’m really, really hoping that you are going to be able to make it more affordable and accessible for a wider crowd, because I think there’s so much power in it, and it goes way beyond… It’s lifesaving for sure for diabetic, pre-diabetic, all of that. But even for people who don’t have those conditions, I think it can be really life changing in many ways. So I’m excited to see what you guys do next, but I’m really excited to see that you seem to be taking that holistic direction within the product as well. And again, as a Mediterranean person, that’s way to go.
Ben Grynol (00:15:48):
Amazing, appreciate all the insight and the lens that you’ve been using the product for so long and how it continues to create ongoing value, because it really is our mission of there’s no such thing as a point in time that’s going to give you any indication of how you’re doing from a health perspective.
Paola Veglio (00:16:04):
Yeah, totally.
Ben Grynol (00:16:05):
So I appreciate you sharing with the team and these are the sentiments that keep us doing what we’re doing and pushing things forward, so appreciate you taking the time and sharing everything.
Paola Veglio (00:16:15):
My pleasure.
Ben Grynol (00:16:16):
Very welcome to stay around. We’ve got a packed meeting today, but if you do have to hop off, feel free to hop off whenever-
Paola Veglio (00:16:22):
I need to hop up in five, but thanks again, guys. And really nice to meet you all. And thank you, customer service.
Ben Grynol (00:16:29):
Thank you. And hat tip support as well. Onto Culture Kudos, so Miz is going to take this one. And we’ve been here a couple of times with my sticky fingers today, slippery fingers.
Michael Mizrahi (00:16:41):
Go for it. So a few shout outs this week, starting out with the collaboration happening to make the Guides and Swap Guides, the Food Guides, all the content pieces that are part of the Guides experience, really, really impressive cross-functional alignment here, collaboration. And so a lot of people have been touching this, a lot of really, really good delegation with the Athena team, really mind blowing to see this whole team come together on these. So wanted to call that out, and really appreciate everyone for working so well together and really, really finding that sync. Like Ben called out in the highlight slide, the achievement slide, it really feels like this is a period of really nice alignment and really good cadence. And so that work is going well, and thanks everyone for the effort there.
(00:17:24):
Second one here to call out and recognize is Stoddy, back one slide, Ben, sorry, is Stoddy. He dug in and reviewed a bunch of Advarra, our IRB invoices, found some overspend, some incorrect billing and we’ve been able to recover that. So thanks Stoddy for being an owner, keeping an eye out for the company and being proactive on this. Appreciate that mindset.
(00:17:44):
And then finally, it’s been quite a roller coaster of a week on the finance side from last week to this week. Ryley totally owned the situation, calm, rational, definitely serious, very proactive on decision making, taking action, watching the situation, and keeping us all comfortable and absorbing that for the team. So Ryley, we appreciate you for all that work, and happy to have you here. So thanks, everyone. Really awesome week across the board. There’s a ton of other shout outs in the WOO channel across comms. Alan did a nice shout out for the product design Eng team and the collaboration we’re seeing there. But all across the board, great stuff.
(00:18:18):
And one more slide. It is AMA season, you might have noticed. A handful of these are required viewing, a handful of these are strongly recommended, but across a number of topics, as we’ve grown, we’ve noticed it’s really, really important to maintain team alignment and it’s really hard to do that with a remote and distributed team. And so where we think we might all be aligned, there’s actually gaps and room for comment, for discussion, for understanding different perspectives and aligning on where we’re landing on all these topics. So some of these are applied focused on some of the product strategy, some of our roadmap, how we’re actually implementing certain features. Others are around company culture and values, like treat people like adults, team not a family, and the implications of that in our actual process.
(00:19:02):
So a bunch of those have happened. Recordings are available, a few more are still upcoming, I think we’ve got UK next week and some RD strategy. And so all of this has been great. And if you can find those recordings and watch them, I think you’ll find them very, very beneficial and helpful. And we’ll keep going on this cadence, maybe not as many, but it’s been a good tool, so we’ll lean into it.
Ben Grynol (00:19:21):
Let’s see if we can do this. Levels shows you how food affects your health. So to reiterate, if you’re not working towards this, which we all are, you’re not contributing to the new product build, everything that’s in the pipeline, flag it. People and culture, onto Miz and Nicole.
Nicole Miller (00:19:40):
Thank you so much. I will kick things off. So we’re talking about performance again, and this has come up a lot. And why? And why is it so important? But the truth is performance, documentation and communication around performance is always very vital. And we’ve reiterated at Levels that everyone should always know where they stand.
(00:20:01):
And so in order for this, we have been working really hard to make sure that we are iterating on these and ensuring that there’s a few key foundations when it comes to performance at Levels. And the first thing is thorough documentation. This benefits everyone in every stage of the company. It’s not about seeking out flaws or building up a case to fire someone. It’s about having that complete picture for both promotions and raises and improvement.
(00:20:31):
The other thing is that having continuity and consistent experiences from one manager, from one department, it’s very important. And some of that comes back to knowing performance and making sure that managers are evaluating everyone as fairly as possible. And then finally, structure and documentation really help us create this whole picture of a teammate experience. And so by having this and really anchoring to documentation here, you’re able to see the whole canvas and not just a small section of a teammate’s experience. Next slide.
Michael Mizrahi (00:21:09):
All right, so how’s we get here? And why now? For those who have been at Levels for a while, you know we’ve gone through several iterations of performance management starting around late 2020, early 2021. And as a team grew, we’ve started to find more edge cases and realized that we need better systems in place. As of last year, towards the end of the year in December, Casey, Sonja and Nicole did an audit of all their existing reviews, the check-in tracker, some of the documentation, and found opportunities for both clarification, better structure and more consistency across the company.
(00:21:42):
And so some examples of things that were getting in our way, we found that Levels, we started with five, eventually had six, felt a little bit too restrictive and there was too much time between each one of those levels in order to have attainable career trajectory that kept up with the company’s needs. Performance reviews got a little bit heavy, the process got a little bit overwhelming, and that led to difficulty, both completing them on time and then also calibrating ratings across managers across different teams. There was a lot of variance there. And then finally, people were just using all of our systems differently. And so we wanted to improve that, add more definition and refine it and just make it a lot easier.
(00:22:19):
This has been a little bit tough, this is taking out the foundation of the building. All of these systems are interdependent. So it’s not just about performance reviews, it’s not just about ratings, it’s the role definition leads to… So you get feedback on how you’re doing, that response to how you’re leveled and what the expectations are for that role, which then leads to the reviews of how you’re doing there and the rating system, which then we can use as a foundation for promotions or need for improvements. So all these are pretty interdependent and linked. There’s compensation components, there’s financial plan components, and so we need to take a step back, address all of this at the same time, and that’s what led to this point. So we’re finally at the point where we’re ready to roll a bunch of this out. So, what’s changed? Over to Nicole.
Nicole Miller (00:23:03):
Well said. So yes, so we’ve updated the whole picture here. So we are rolling everything out mainly today and then a little bit more will come in the next few weeks. So you can check out that link at the bottom there, perf-overview. First off, everything has been wrangled into one document for a bit cleaner organization. There were a lot of memos and all of those memos are still very important and valid, but we tried to put everything into a lot more cohesive picture, and that includes role expectations and leveling, performance reviews, performance review ratings, ongoing feedback, promotions and performance improvement plans. And Miz and I’ll go through each of those right now.
Michael Mizrahi (00:23:44):
We’ll keep it quick. So ratings, we’re going from a five star system, one through five, to really descriptive terms at four levels, meeting expectations, exceeding room for improvement and not meeting. This illustration is really important. Hot tip to Sam for the inspiration on this one. Expectations increase over time. So keeping up with the role really means growing and maturing and developing your skills and the application of those skills as the company grows. And so the needs are going to change, the projects are going to change, and keeping up with that is a big task and we have high expectations from the start. So that is the baseline setting here. We’ll save the details for when we actually get to the rating system, but that’s the foundation of it. I’ll leave it there.
Nicole Miller (00:24:30):
And then as mentioned, our five or six levels felt a bit restrictive, and so we’ve essentially branched this out to 11 total. And the reason for this is that it allows for more movement within the framework and then also is much more accurate when it comes to benchmarking salary and equity based on our data source, which is Carta. And so everyone has been re-leveled under this system and we’ve confirmed this with managers and we’re going to be rolling this out today/Monday and ensure that everyone has some clear expectation and is aware of what the current levels look like and new levels. And the other piece of this is that we have now clear attracts for individual contributors, people managers, and functional leads. And so there’s a much more complete picture of the pathways within Levels.
Michael Mizrahi (00:25:22):
And one more slide, Ben. Back one. Okay, there we go. So the place where most of us are going to see this is the performance review process. The big changes here, twice a year. So every April, every October. If you recall the previous rationale, we wanted to avoid perf season. We’re going to do everything we can to keep it extremely lightweight but still thorough and still cover the bases. But the reason why it’s important to do these all at the same time is really to calibrate within teams, across managers and then across the company. And so a whole set of new process coming for this, some Typeform, some really well integrated Athena process, ongoing peer reviews along the way so that it’s not all at once and you have to block off a week to do this. So all that’s coming. And in terms of action items, I know we’re throwing a lot out here, nothing required for now. Our next cycle comes up in April, and that’s when we’re going to handle the next round of calibration of adjustments of performance reviews and feedback.
Nicole Miller (00:26:30):
And final point here. We did add in a little more clarity around promotions and under what conditions promotions apply, and then also, a little bit more around performance improvement plans. And again, all of this really builds toward this more complete picture of an individual’s journey throughout the company. And so we want this to be supportive of everyone, help them grow and develop. And so a lot more information to come, a lot of documents, a lot of comms and the exact templates and forms will be taking shape as well. So thank you for everyone who’s contributed. We’ve gotten a lot of advice over the last few months and it’s been a very big effort. So thank you all so much for tuning into this.
Ben Grynol (00:27:17):
Amazing. Thanks for all the work on this, Nicole and Miz. And I know Sonja has contributed and Casey as well. So lots of work there. I think I need a performance improvement plan on how to work slides properly. So I’m looking forward to that. One note, so a plug here. There’s a podcast that recently shipped or re-shipped on inside the company and it’s a conversation between Tim Kendall. So Tim was early at Facebook, he’s one of our investors if anyone’s not familiar with him, also part of the documentary, the Social Dilemma. But Tim talks about what it was like to be early at Facebook and there was performance reviews that would come out. And it felt terrible to everybody there, because you’d get these reviews and it said meeting expectations, middle of the bell curve. And you thought, “What? I’m a high performer, I always exceed expectations.”
(00:28:03):
But when you’re in a group of very high performance, you should be meeting expectations, that is the middle of the bell curve. And so it can be one of those things where when you think of the way that you should perform, it should always be that you are meeting expectations. That’s where we want to be. That’s in the pocket, that is with a bunch of high performance. So it’s very much a moment of reflection listening to that, but I highly encourage everybody to listen back and to hear his take of what it was like to be around all these really high performing people that have gone on to build amazing companies in the world since then. So we’ll leave it there. And on to growth. We’re pretty tight on time. We’ve got three more sections, we’ve got growth and two product updates. So as much expedition as possible would be amazing.
Jackie Tsontakis (00:28:52):
Okay, cool. I’ll try my best. So I’m going to share a quick update on partnerships, what we’re focused on now, where we’re going to be focused on for the next few months. But just to set the stage really quickly, a quick reminder on our demand capture priorities currently. One, preparing to sell the new product, and then two, selling the current product.
(00:29:14):
So another quick reminder on goals for partnerships, which are to, one, drive awareness for the Levels brand, two, build the brand and trust with our consumers by partnering with key thought leaders in the health space, and then three, drive revenue through new member growth. And then just a quick note on that, over the past few months, partnerships has driven an average of 24% of total new members per month, which is pretty significant. And also just to note that this is a measurement based on last touch attribution. So that means that people who used a partner code is what’s measured here. So it doesn’t count anyone who, for example, heard about Levels on a podcast and then later purchased through our homepage.
(00:29:56):
Cool. And here’s a bit more detail on what our demand od our short priorities look like for partnerships. So our primary focus right now is building a strategy to help us sell the new product. So what that means is we’re working on learning more about Maureen, where she gets her health information. From there, we’re going to determine key partners that we should be experimenting with across those platforms to best reach her. So partnerships will ultimately play a key role in validating the new products among this core target audience. And then our secondary focus is selling the current product right now through our paid partners. So the goal here is to keep conversions through paid partner code steady month over month without increasing our spend.
(00:30:33):
There’s some overlap between these goals. So for example, any new experiments that target Maureen that we’re running right now, like our recent YouTube experiment with Dr. Becky. This experiment is helping us sell the current product. It’s already driven over a hundred conversions in the past month or so, but building on this relationship now is also going to help us sell the new product since Dr. Becky’s audience is so aligned with our Maureen target. We’ll be working with her once we launch the new product. She probably also has a lot of insight to share with us about this core audience and knows other creators who have a similar audience who we should be connected with. So we’ll continue to partner with her and continue working with her as we begin promoting the new product and focusing more on Maureen.
(00:31:18):
So focusing on objective one, our primary focus over the coming months will be building a partnership strategy to sell our new product. So I shared a memo on this in comms yesterday if you’re interested in taking a look at everything in more detail. But at a high level, we’re going to be getting more information on where Maureen gets her health information from a channel perspective. So what that means is pretty much determining is she going to Instagram, blogs, YouTube, searching on Google? We’re going to do this by taking a closer look at member survey and call data. And also just to note that this is a distinct question from where did you hear about Levels, because the goal is to determine if there are any platforms that we’re not focusing on enough or should be focusing on more.
(00:31:59):
And then from there we’ll want to determine what key partners we should be working with across those channels. Sorry, Ben, I’m still on the previous slide, but just the last note. From there, we’re going to determine what key partners we should be working with across those channels just to best reach Maureen. So we’ll do that in a few different ways, talking to other brands, researching brands that have a similar core target audience to see who they’re partnering with, and then also asking our current partners if they have recommendations for creators who we should be in contact with, whose target audience is aligned with ours. And then also just to note, in the future we should consider working with a program to help us narrow in on targeting. There are a number of different influencer and podcast tools that allow you to search for creators and shows based on audience demographics, but we won’t do this right away, but it’s always good to keep options in mind like that for down the line.
(00:32:48):
And then a quick caveat, this doesn’t mean that we would pause partnerships that are successful just because they’re not directly targeting Maureen. So some of our current partners, like Kelly Leveque and Dr. Becky, are targeting Maureen already. Others, like Dr. Sinclair and Huberman, are not really, these audiences are more health optimizers. Huberman’s audience is actually almost 70% male. But these partnerships perform really well for us. They bring in a ton of awareness, brand value and drive significant conversions. So focusing more on targeting Maureen with our new product is not to say we’ll stop these partnerships, but rather we’ll be focusing resources and new experimentation over the next few months mostly on our Maureen target to align with the new product and to help us validate the new product direction among that target audience. And then we’ll definitely be keeping an eye on performance across all partners and just paying closer attention to audience demographic when we’re assessing new ones.
(00:33:51):
And as a quick reminder of… Was there a slide before that, Ben or no? Okay, cool. As a quick reminder of why it’s important to continue working with big partners like Huberman and Sinclair, even if their audiences are not primarily Maureen. So partnerships are a key driver of general awareness. So many of our partners are reaching millions of people daily and there’s huge value to this. For example, there’s a tweet here that Huberman’s episode on alcohol, which we sponsored, was the second most shared podcast episode of 2022. We originally saw that on Twitter, I validated it on Apple podcasts. I think it’s legit. But regardless, there is so much value in driving awareness through partners with massive reach like this.
(00:34:34):
And then secondly, there’s also invaluable benefit to the Levels brand in building relationships with trusted thought leaders in the health space that we’re working with. So these partners build trust among consumers, which is important beyond conversions and revenue. But of course revenue is also a key component of why partnerships are important. As we mentioned before, partnerships drives about 24% of all new members from a directly attributable standpoint disease. But we know this number is much larger if we were to measure on a first touch attribution basis, for example, someone listening to a podcast ad and later searching us and purchasing through the homepage. So these are the main factors that we focus on when we’re assessing a partnership. We’ll be focused mostly on targeting Maureen through partnerships, but we’ll have other partners in the mix as well who are bringing in value to the funnel by driving awareness, brand value and revenue.
(00:35:23):
And then quickly touching on our secondary objective, which is selling the current product. We’re aiming to keep conversions through paid partner code steady month over month without increasing spend. We have a goal right now of 400 conversions through paid partner codes per month, and we’ve been able to do this so far this year by doubling down on our tier one partners and decreasing resources that we were spending on pretty much all new partner experimentation with the exception of a few. And while we’re spending less time on new partnership experiments right now, we’re increasing spend in a few other places across growth like performance marketing. And I’ll also note that we’ll likely be looking to increase partnership spend a bit once we’re ready to run experiments with new partners to sell the new product.
(00:36:10):
And then next slide is the last one, just wrapping up with a quick overview of where partnerships will be focused over the next few months. So there’s more on this in that memo that I shared yesterday, but we’ll at a high level be focusing our resources and almost all new experimentation on Maureen and validating our new product among that core target audience. We’ll be preparing our current and new partners to talk about the new product and familiarizing them with Maureen and her pain points and goals.
(00:36:35):
And then more tactically, which I won’t get into detail on now, we’ll be considering changes to the infrastructure of our partnerships program. So for example, what does our new offer look like? What do our lending pages have to look like to best communicate our new product offering? How does our affiliate commission structure need to change as we introduce new pricing? And then lastly, in the meantime, as we’re thinking about all these things and primarily focused on the new product, we’ll be working on selling the current product by keeping conversions through paid partner codes steady until we’re focused on month over month growth. And that’s it.
Ben Grynol (00:37:09):
Amazing. Thank you, Jackie. And to give Jackie a hat tip, for those of you who aren’t aware, she’s very much been spearheading partnerships basically all by herself independently. Tom is very focused on growth, and so there’s a huge amount of work that goes into this. And you can see she is a top delegator with Athena for a reason. She gets a lot done and a lot of output. So I appreciate all the work on that and the update.
(00:37:35):
A note on time. So we are inevitably going to go over time. If everyone has appetite to keep going or if you have to hop off, suggest that we keep hammering through. I think we’ll end around 12:15. But for those of you who can’t stay, then you can catch the recording async. Don’t want to punt a lot of the great product updates into the next forum. So if that works for everyone, stay on. We’ll keep going. And if you have to hop off, then we can do that. If everyone’s good, I’ll see a thumbs up or two and we’ll keep rocking and rolling. All right, product. This is, I believe David’s going first. No, Sonja and Cissy. Sonja, do you want me to copy the share?
Sonja Manning (00:38:14):
Can you see the screen?
Ben Grynol (00:38:18):
Oops. You should be able to share.
Sonja Manning (00:38:24):
Looks like I’m sharing. Can you guys see?
Ben Grynol (00:38:27):
Here, I’ll stop my share. Can you share now?
Sonja Manning (00:38:32):
Yep. Hopefully people can see it. Give me a thumbs up if you see the…
Cissy Hu (00:38:35):
Looks good from my end.
Sonja Manning (00:38:36):
Yay. Okay, I know that we’re at the end. Feel free to stand up, sit back down again, do five air squats. Buckle up, get ready. Cissy and I are going to tag team this presentation and we’re super excited to tell you what we did and what we learned for Guides v5. So this is our fifth Guides experiment where I led 300 people through a month long metabolic health journey via my close friends group on Instagram.
(00:39:00):
But before I jump in, I want to take a second to talk about how I approached being a Guide. To be honest, I was a little nervous to do this. Not a content creator before this, I did not have a large following or a public persona or a brand, which is one of the things that we wanted to test. More on that in a minute. But conversations with Cissy and Casey and a few articles led me to this mindset that I wanted to share. So I am, and Guides are, really a vehicle of information for people, a vehicle of metabolic health, of our blog information, of our incredible advisors. And so I saw my job for the month of February as being the most empowering, supportive, clear, authentic and engaging vehicle that I could be. This is the awkward part where we watch a little bit of content with me in it, but in case you missed it, here is a short recap of a little bit of what the month’s content was like.
(00:39:55):
Good morning. Oh my god, I’m so excited. Welcome to this month long experience with me where I’m going to be giving you an inside look into my daily life. And I’m here as your Guide and your champion as you learn more about metabolic health and work towards your health goals. So today’s pillar is all about exercise. Today’s principle is going to be all about food. Here are today’s a version of the power bowl. Turkey burgers on the bench night. This is one of my favorite sheet pan meals. Hello and happy Saturday. Today we’re going to talk about environmental toxins. Okay, today we are going to talk about cold exposure. Today’s metabolic health pillar is stress, but just realize I could pinch myself and make my face smaller on this screen. Getting my post meal walk. All right, today is a zone. Today this is your prompt to pause and take a breath break.
(00:41:07):
Okay, I’m not going to play all of it, because we’re a little short on time, but there is a five minute version of this if you want to watch the whole month in five minutes in our retro. But that’s a little bit… I’ll pass it to Cissy.
Cissy Hu (00:41:22):
All right, so you may be thinking, “Guides v5, I don’t remember what happened in one through four.” So we’re going to take a quick walk down memory lane from left to right for Guides v1 through 3. Our key question was, is their appetite to follow a Guide on Instagram for one week if they’re focused on metabolic health content? The answer was yes. We were able to recruit around 1300 people across three cohorts with Casey in cohort one and two and with Stacie in cohort three. These experiments were all free experiments.
(00:41:59):
Once we validated that question, we moved on to Guides v4 with Stacie, where the key question is, will people follow for a month? We see that they’re interested for a week. What does a month long experience look like and is there willingness to pay? So Stacie put out a call, “If you’re interested, you can pay what you want.” And of her 1200 people, 140 of them converted and they averaged a payment of around $11.87.
(00:42:29):
So with that confidence that people are actually generally interested, we were curious, is their willingness to follow someone who has a metabolic health background, but a small following? And with that, Sonja lodged her Guides v5 experiment and validated that thankfully there is in fact interest. So we had precisely 285 people sign up to follow Sonja for one month after receiving either an email via Drip or seeing our real invitation on Instagram.
(00:43:00):
Next slide. For the key hypotheses, the what, we were really focused on these four primary questions and pressure testing them throughout the four weeks. The first one was, is there a willingness to pay for a four week experience for a Guide who has a metabolic health background, but a small following? Number two, can we enable a Guide who hasn’t previously been a creator to spend less than 90 minutes per day on creation? Number three, can Levels impact a Guide’s following materially on Instagram? And number four, is there a willingness to convert after one month for a Guide who has metabolic health background, but a small following?
(00:43:42):
So before we get into the results and our hypothesis testing, we’ll give you a quick preview into the cohort. Most of you likely saw this in the survey recap that came out last week… Actually was it this week? Last week. And I’ll give you the quick highlights in case you aren’t familiar. So the cohort had a baseline knowledge of metabolic health, because they were largely familiar with Levels. They were either existing Levels members, which is the Drip campaign, or they were following Levels already. They had the idea of they knew what metabolic health was and they were largely interested. The key things were 52% of this cohort were Maureen’s and 94% of them were females. When we looked at the top things that were holding them back from achieving their health goals, it was a lack of time, a lack of knowledge and bad eating habits. And so over the four weeks, Sonja effectively tried to help them tackle these challenges. And Sonja will talk a bit about the how.
Sonja Manning (00:44:40):
So the overall original guidance was just to live my life for a month, but given the direction of the product vision and given this survey results, the initial survey, I leaned into a little bit more of a program feel. While it felt like a program to users, I was generally building it out a few days in advance. So it was very agile. And essentially, I reduced all nine of the metabolic health pillars in the first nine days. Then I wove those into my everyday life through recipes, exercise habits, and covered a ton of other adjacent topics based on requests or DMs or topics that I care about. Every day that you see here was anywhere between the five to 10 minutes worth of content, usually 15 to 25 artifacts. So it was more than these key topics, but I tried to have at least one topic or lesson in our teachable moment, planned per day in addition to showing how I lived the metabolic principles.
(00:45:25):
So let’s get into the results. We’ll start with some feedback from comments in the survey. I’ll let you just read a few. I also received a ton of comments and engagement throughout. I provided 57, I believe, engagement opportunities via polls and question boxes. And the feedback enabled me to adjust the content, meet people where they are. This is something we’re actively working on, building into the app experience with quiz and polls. All these feelings that you see here are mutual. I loved engaging with this cohort, our members, and the feedback was incredibly useful, but also really rewarding as a Guide, showing that I was being an effective vehicle of information.
Cissy Hu (00:46:07):
At the end of the four weeks, we sent out a survey, and 90 folks, 30% of the cohort, responded. Of those 90 people, 89% of those people made changes to their diet or lifestyle from following Sonja. And the experiences that people most appreciated were learning new habits and tips following Sonja’s daily life and accessing the Metabolic Health Handbook. One of the really interesting takeaways we had were only 17% of people valued DMing with Sonja, which was interesting considering how many DMs Sonja actually got. And so this led to some interesting thoughts around product, which we’ll share in the upcoming slides.
(00:46:51):
So getting into the hypotheses, we touched on this one. Hypothesis one is, is there a willingness to pay for a four week experience for a Guide who has a metabolic health background, but a small following? So we validated this, and I’ll touch on an interesting operational learning we had, which was as Ben was starting to do the growth experiments for recruiting for Beta 1, he took a look at the, Typeform and what we found was we saw a huge drop off from the start to the end of our registration form. We had around 11 questions, and our effort was to try and capture all the relevant information upfront as much as possible. And by doing that, we created a friction in the signup flow. Ultimately, all we wanted was people to commit to following Sonja for four weeks.
(00:47:37):
So all of the additional demographic information we could have captured after they converted. And so had tip to Ben for doing the initial analysis. And we’ve implemented this learning since in both his Typeform surveys as well as a couple we’ve sent out to Sonja’s cohort to convert for upcoming program registration. For anyone who’s making a Typeform survey, the recommendation is keep it simple when it’s a registration form. So name, email, payment, have them convert, and then we’ll collect the demographic information separately. And on this hypothesis generally, this ultimately gave us the confidence that we should continue building the Guides playbook and train up internal Guides and leverage our brand to validate new Guides as credible coaches, which is exactly what we did with Sonja. Okay, over to you.
Sonja Manning (00:48:29):
Those [inaudible 00:48:29] is mine. As someone with a lot of other hats they wear at Levels, we thought, “Can we try and time box this to 90 minutes a day for content creation?” So here’s my schedule, here’s what we expected, day one, and then this is a little bit more what reality looked like AM to PM, eight hours a day. Those hours did come down and we figured out how to systematize the superpowers of our team with editorial and digital and Athena, but we learned it as a full-time job. And this is live footage of me every day for the month of February.
(00:49:01):
So some of the challenges of being a Guide. And again, we anticipated all these, we knew this was going to be the unsustainable, unscalable version, and part of it was putting me in the loop so we could figure out as a team how to make this sustainable and scalable. Challenges included, drafting and producing content takes a lot of time. I was solely designing the Guides v5 program and content calendar at the time. Keeping up with the DM volume was a doozy. Keeping up with the pace of filming a 24/7 inside look of my life was ultimately unsustainable and led me to feeling a little burned out by the end. Don’t worry, took a few days off. I’m feeling good now.
(00:49:35):
But some learnings is that to really loop in and systematize the support and superpowers of our incredible team from editorial design, digital product, Athena especially, and have them involved throughout the whole process from the creation to the analytics scope back the content volume to be more than one video a day, but less than 15 to 20 content artifacts a day and plan as much content in advance to enable Guides to not be on this treadmill of content creation, but to take a few days off.
(00:50:07):
Overall Guide learnings from my perspective. There’s a ton more in the retro, but it’s a full-time job. We’re going to be treating it that way, at least for our next few Guides. Producing this programmatic or program light content is harder than the femoral content, requires of course more planning and infrastructure and cross-functional support. There’s no such thing as a just film your regular life content. All content must have a story arc, a form of engagement and empowerment to be high quality and really Levels worthy. And then keep Guides close to our members. I learned a ton about our members and potential members and loves being closer to who we’re building for. And Guides have this potential to be mini UX researchers. And so we’re going to continue to work on formalizing feedback process within the app to go from the community to the Guide to Levels team. And last was that it was really fun. I loved doing it.
Cissy Hu (00:50:58):
Hypothesis three, can Levels impact a Guides following on Instagram materially? So we actually saw a 30% increase in Sonja’s following follower base, around 880 people followed her over the course of the four weeks. But we determined that it’s ultimately not enough of a increase to say that the affiliation had a significant impact. And we define significant impact as effectively allowing the Guide to rely on their affiliation with Levels to drive meaningful value from us to them. And so over the four weeks, we drove traffic to Sonja’s Instagram through Drip campaigns, tagging her in three collaboration reels and having Sonja post to the grid more frequently. The confounding factor here was that the experiment was ultimately in close friends, which locked out any public access by design. And so it’s hard to say if it would’ve been much different if this was all public. But again, by design we had this be a close friends experience to simulate what it would be like in the Levels app.
(00:52:01):
So this learning ultimately led us to evolve our thinking around how we think about Guides. From the start, we initially thought Guides would be metabolic health entrepreneurs that we’re building in the space that we were essentially trying to elevate, but their top priority was actually building their own business. We realized that we actually need to think about our Guides as an integral part of our Levels team who is as invested and accountable to the success of Levels as we all are. And so this is ultimately leading us to spend more time building out the playbook for Alpha 2 and beyond, as well as identify both internal and external future Guides who can be champions of Levels versus having Levels be maybe a side hustle for them.
(00:52:48):
Okay, the final and most important learning, is there a willingness to convert after one month for a Guide who has metabolic health background, but a small following? So this is important, because this ultimately dictates whether or not we think that this business is viable. And we found that after four weeks we were able to offer members two paths to conversion. The first path was sign up for an upcoming program and we essentially put them on a reservation list, they were the first on the wait list. 68 participants of the entire cohort converted, so about a quarter of the cohort converted and paid the $10. So those folks will all be going into Beta 1. And then 7% of people, 20 participants in total purchased the 99 Try It kit, which we brought back to life to test with this cohort.
(00:53:38):
And so ultimately what we found that is despite having a small following, Sonja was able to convert people through a combination of her deep metabolic health knowledge. She had a really engaging accessible approach which came out loud and clear in the survey results and then her enthusiasm. So ultimately here we’re going to continue to really lean on partnering with internal talent and test potential internal Guides with guest collaborations with Sonja and Stacie. And we’re going to build out a training plan for new Guides, because clearly we can make it happen with folks who haven’t been creators in the past.
(00:54:15):
And we will wrap with three final learnings. So I’ll touch on these quickly. For engagement, the interesting thing was polls continue to reign supreme here. So there was a ton of passive engagement, about 50% of engagement in the first week, which ended up dropping to around 30%. But people mostly stayed passively engaged through answering polls. And the really interesting piece on the DM piece was that 10 people drove around 30% of DM traffic to Sonja.
(00:54:48):
So despite the fact that she saw a ton of DM volume, there was a few super fans who were constantly hitting Sonja up. And so that led us to realize that we don’t need to prioritize any direct one-to-one contact with Guides that can eventually become a premium offering, but for now it’s not essential to the experience. Over the four weeks we ran a bunch of content experiments, which led us to realize that there’s continued demand for structured accountability, which is great with a nod to our metabolic checklist. But the interesting takeaway here is people really wanted printable PDFs in this cohort. And so keeping that in mind for our non-tech native-
David Flinner (00:55:28):
We’re good? Yeah, fast forwarding to Alpha 2, so picking up where Sonja left off, now we’re bringing everything to the app. So this is, Will Stacie’s Instagram cohort of 1,200 people purchase the program and engage in the software only version of Levels as we bring that Guides experience in? So quick demo. We won’t go through the whole minute, but play a few seconds of the video if you could then. Is there sound? Can you put your sound on?
Stacie Flinner (00:55:54):
Good morning. Our first week together is all about healthy morning habits. We’re starting with a focus on mornings, because it’s typically the time of day we have the most control over. This week-
David Flinner (00:56:05):
Okay, you can pause it for now.
Stacie Flinner (00:56:06):
… you’ll learn how to improve your morning routine.
David Flinner (00:56:09):
Cool. So we kicked off Alpha 2. It’s been about two thirds of a week now that we’ve been going through this. We are using this with Stacie’s cohort and we are implementing in a scrappy way, the programs that Alan laid out in last week’s forum. Really quick, flying through this, wanted to call out that we had about almost a 10% conversion rate from the people who saw Stacie’s pitch to sign up for Alpha 2 in her Instagram call to action. And we will be sharing more on the engagement that we’re seeing in next week, but for this week I just wanted to give you a taste about all of the velocity and the things that we’re learning so quickly from both between the product team, the whole development team, and then operationalizing the content structure. Next slide.
(00:56:49):
So we are moving very fast, extremely fast, and this is to give us more chances at bat to hit a home run. We are learning pretty much across the board. And this is spinning up the entire team to figure out what should we be doing on a day-to-day basis, focusing on morning routines. We have people, Azure, Caitlin, Cissy, Tony, everyone working on spinning up this program and then all of the people on the development team rapidly pushing out more features based on what we knew worked in Instagram to validate that they also work in the app and also fostering a sense of connection. So if you haven’t tried it yet, it’s really special and we’ll dive in a few slides. Next slide.
(00:57:30):
I wanted to share just how much the app is changing between Alpha 1 and Alpha 2 if you haven’t had a chance to use it since then. On the left you see this is the zone review where people had very, very little value in terms of logging their food in the first in-app Alpha a couple of weeks back. The software only pod launched some really beautiful new updated experience that helps people unpack how food is affecting their health at the tag level and we’re getting feedback from people that they love this. So that’s really helpful for them to see that these ingredients that they didn’t realize were bad for them. What we’re seeing is that people are viewing the Levels experience in a system way and that they’re finding that the guidance video is really helpful to orient them on what to do and then they’re logging their food and it’s helping them see what the healthier option is. Next slide.
(00:58:13):
When we launched Alpha 1, we only had the ability to have one video per day and it had nothing on it, no ability to link it to an associated experience to take action or to go deeper, no community engagement on it, like the ability to like things. So over the last week, the development team has been pushing out a lot of excellent features that will reduce friction to content consumption as well as foster an experience of community, things like multiple videos in a feed, adding links, the ability to like the video and the ability to go deeper into who your Guide is. It’s early, we don’t have a lot of results yet, but people are reporting that since we started sharing an app, they’re finding that they don’t crash as much during breakfast, that they have more energy throughout the day. Next slide.
(00:58:56):
In the first version in Alpha 1, we had no Kitchen. Now we are completing the system so that not only do you have guidance from your Guide on what to do and why, but an actionable tip on how to close the loop with that through a delicious and beautiful recipe that you can find at a single tap from the Guide video redirecting you to the kitchen. And we’re hearing that people are finding those recipes very valuable. We also, in a scrappy way, Maurice is on the team to figure out how we could pilot a Levels Food Guide, which is pretty much the first version of a pantry. And this is going to help flush people’s understanding out of what ingredients should you be using in your house. We paired this with a Guide video on how to think about stocking your pantry, gave them an actionable list of really beautiful recommendations on what they could be stocking it up with. People are doing that and they’re reducing their sugar in their pantries. Next slide.
(00:59:48):
Okay, so Alpha 2 is in process. We’re only one week in out of a three week program that we’re piloting with Stacie. I wanted to just give you a bit of a bigger picture here too. We are measuring retention as one of our main metrics from Beta 1 onwards through general availability. Starting not in Alpha 2, but in Beta 1, we will be looking at having a 30% retention of those Beta 1 participants going on to their second month. However, in Alpha 2 we’re looking at a proxy for retention, also engagement. I wanted to surface one of these quotes from our users who sent this in yesterday that I think is emblematic of where we’re going.
(01:00:22):
Once we hit Beta 2, in Beta 2, we’re going to be weaving in the CGM experience with the software only experience and we’re creating this larger user journey arc across Levels. Whether you’re on sensors, off sensors, no matter where you are and what’s right for you, there’s a Levels experience that should resonate with you and help you. This person said that they were an early adopter of Levels and they’ve been enjoying the subscription service, but they let it lapse in December. When they heard about this software only experience, they got excited and they renewed and have been really enjoying the app experience.
(01:00:49):
So this is a preview of what’s to come. This is our vision, this is where we’re going, and I wanted to just encourage you all to keep up at this. It’s a multi-month experience. We’re going to be incrementally layering in experiences that continue to foster community, continue to inspire people, continue to make it easier to see if food affects their health and take action, refine the program to make it more outcome driven and more effective. And each month we’re going to hope to see that retention rate go higher and higher, starting at a 30% rate at the end of April and a 40% rate thereafter, all the way through retaining 60% of our users by the time we hit general availability. Next slide.
(01:01:27):
And just a little visual preview of what’s to come where the teams are working very hard across the board. The software only pod is working on things like seamless onboarding and getting up to speed for the software only users. We don’t really have any in-app onboarding for them right now, as well as increasing our insights that we deliver for meal reviews. The Guides pod is going to be focused on experiences that make it really easy to deliver feedback. Like you heard from Cissy, the polls are dead simple to provide feedback to the Guides. We want to foster an emotional experience here, a sense of connection that is really easy on Instagram but is not yet easy in the Levels app, but we have a roadmap to get there.
(01:02:06):
And then also de-risking some of the big things that we’ve heard work really well, but we have debated internally whether to do, namely comments. We’re queuing up a project to test out comments on Guide videos, recipes, things like that, and see how that works. And I’ll leave it here for now and next week we’ll have more on a deep dive on the quantitative engagement.
Ben Grynol (01:02:26):
Amazing. Well, I think we are at time, we’re past time, but a huge hat tip to everyone. You can see how much work is going into this. It is clear that everyone is pushing hard, so let’s keep pushing hard. Let’s push hard ahead. We’re all here to support each other. Make sure that you are driving as fast as you can without hitting that point of burnout. So if you are feeling that at all, know that we’re here to lean on each other and there’s lots of resourcing to go around, especially with things like utilizing Athena to take work off our plate. So keep going, let’s get into Beta 1. Let’s get all the way through all the Betas into general availability and learn, learn, learn because we are moving fast and you can see how much is going into it. So appreciate everyone’s work. Have a great weekend ahead and happy St. Patty’s Day.