Josh Clemente (00:00):
Let’s jump in. Last Friday Forum September 2022. Also, my sister’s birthday. Happy birthday. All right. This week we hit a pretty huge milestone for the support team, 100,000 cases handled in Help Scout. Through the history of the company, we’ve had many support tools that we’ve used, so this doesn’t scratch the surface of how many happy members we’ve worked with, but just want to shout out that big round number.
(00:29):
We had a major logistics shakeup that is going to affect a number of programs that we have at our company. This is behind the scenes, going to allow some things like we are migrating our study to finally take advantage of the IRB approval and be able to deliver directly to members who opt into our studies without a prescription requirement. That can improve some of the access stuff and then we’re going to likely migrate to a direct partnership for labs and some future products that we’re working on that may require physician orders, et cetera. More updates to come on that, but always stays exciting around here.
(01:05):
We have a UK Closed Beta wait list rollout starting next week. Karen posted a really cool video walkthrough of what the e-commerce like experience is for the Beta members in the UK. You can order in less than three minutes since there’s such a simplified flow. It’s looking really beautiful, getting ready to launch that.
(01:21):
We’re also going to start outreach to tier one press outlets in the UK next week. It’s heating up. On the product Scoring and Logging V2 rolled out to 100% of members. We’re now collecting feedback on that. We’ve already got a few V 2.2 type upgrades planned for those including scoring updates, which I think we have some more info on later in this meeting. This week, upgraded the database to support non-time series data like heightened weight, so that historically we’ve all been time series data or all the data that we’ve been really working with has been time series. This update will allow us to store other physiologic metrics that we care about.
(02:01):
Sean is onboarding this week onto the team. We’ll have a bit more on that in just a minute and then we had some great content this week on some of my favorites, the how to order posts are always really popular. They just give people a very useful playbook for what to do when you walk into one of these favorite places that we’re all going to use because they’re convenient, how can we avoid the landmines?
(02:21):
We had a great Q and A with Dr. Merrill Matschke. We had a really good podcast with him previously, specifically on Men’s Health and this Q and A goes nicely with it. A five question story with our member Susan Bray and then the roundup this week in Metabolic Health.
(02:36):
This week Rich Roll and Ben Greenfield ads are live. We’re also starting, we’re prepared to start the special offer experimentation. Across Mark Hyman’s podcast, where we have a special offer with we’re an exclusive offer with Casey that will be delivered via email. Peter Dimandis Podcast is getting ready. We’re going to have a first ad offer that includes two months additional for people who convert through that link.
(02:57):
Then Tim Ferriss, same deal through 5-Bullet Friday newsletter offer. Then lastly, very exciting. We’re talking to the honestly team after the blockbuster success of that episode, which continues to ripple, you can see this is a little tweet, there are dozens of these rolling in. People are still really resonating and absorbing that episode. But anyway, the honesty team is interested in talking to us about potential sponsorship there.
(03:24):
We’re working on a B2B2C experimentation, so potentially going to partner, we’ll figure out exactly what the details look like, but with the Institute of Integrative Nutrition, so going to see, essentially we’re starting to test the waters for additional opportunities where we can expand the funnel and support experts and practitioners who are working with clients already and could leverage levels.
(03:49):
We got two new emails in work. One is called The Pitch, and this is essentially a conversion. It’s the final email after a conversion campaign when somebody’s in our email flow and then the special offer email, which these are all very nicely designed emails. Special offer is going to go to those folks who convert via partners, and so it offers that extra two months.
(04:10):
The team is building, well thinking through collaboration still with between research content and products. We have a ton of firepower on those three sections of our business and combining them in order to both create internal content that we can all learn from, as well as introducing that through our content to our blog and also into the product. There are opportunities to improve our how we cross-collaborate and there’s been a lot of thinking done recently on this.
(04:35):
Lastly, the fat burning experiment, which many of you, many of us took advantage of or got participated in is now in data analysis as of today, I believe, that wound up and then that we’ll do an iteration and this should lead into a community experiment. Finally, on the backend purchase flow for the US e-commerce experiences in work, we launched a tool for support and retool to be able to build out e-commerce orders themselves. Had a big backfill for stripe orders, which will let us do country code specific tax, which is necessary ahead of our e-commerce migration here in the US. We instance the database which improved performance, so at our current data scale we’re obviously much larger than we used to be and improving database performance through a new instance happened this week.
(05:20):
Then lastly, we’re working on a data science services tool, which will be built in Python and this will allow data science and engineering to collaborate more easily without having to do double work. Nice upgrade there. Let’s see, other items that we want to touch on? Always some great action on Twitter for us to tune into. We had a couple episodes, Andrew Herr from Fount Bio, talked to Azure about biomarkers. Azure also did an episode with Ben on biomarkers and blood work and then we had a great episode with Edie Horstman on putting PCOS into remission. Then we’ve got just a ton of great UGC, a couple videos, both those podcast episodes were published on YouTube and then also some awesome McGuffey content.
(06:06):
Jump ahead and welcome our new team member, Shawn Jones to the software team. Shawn, great slide here. A lot of awesome stuff pulling the pancakes towards you. Those look like some dense flapjacks there, so would love to hear from you, what you’re excited about joining the team. I know you and I have had some great chats so far and I’m really stoked to have another CrossFitter on the group.
Shawn Jones (06:30):
Good. Yeah, I’m super pumped. I’ve been getting a lot more interest. I mean, I haven’t been a longtime fitness guy or anything like that, but was kind of getting into it about, I don’t know, five years ago or so now and just a couple of years into that started. I remember picking up my L1 handbook when I was thinking about getting deeper into the CrossFit stuff and the first chapter had a lot of stuff about nutrition and then my world was just kind of rocked by that. I stopped reading right there and just kept rereading, trying to learn about Zone Diet and counting macros and dietary recommendations. That was my first intro, because I had just kind of ate whatever before, wasn’t that big of a deal.
(07:15):
Getting really into health and then tracking fitness and being data oriented and getting in with WHOOP and using WHOOP for a long time. Just going all in on the fitness and wellness aspect of things in life with career and my local community and the global community, just has me really pumped and really excited because it’s stuff that I feel like has been changing my life. I’ve seen just dramatic changes in four years time, right, just my health and my wellness and being able to make improvements by measuring. I’m just a huge fan of what we’re doing here and I’m excited to be part of it.
Josh Clemente (07:55):
Amazing. Well, let’s get that workout chatter going. I am looking forward to much more of it. Yeah, stoked to have you aboard for the whole team as Shawn gets onboarded. Please, please help him along the way. Obviously there’s a bit of a reintegration process for many of us into new ways of communicating when we get started here. Yeah, make yourselves available, connect with him. I think we have a cafe scheduled or maybe you already did that. My brain is not able to organize chronological time anymore, but excited to have you aboard Shawn, looking forward to working together.
Shawn Jones (08:28):
Yeah. Thanks. Thanks.
Josh Clemente (08:30):
All right, quick culture slide. It’s been a year since Taylor decided to come join the team here on research and I don’t know if he’s on the call right now, but shout out to Taylor. Happy one year. I wanted to highlight Jackie. This was a really good WOO post that Tom put up, but really Jackie’s doing an amazing job of coordinating a ton of behind the scenes work, both with our advisors, with our members, and with our partners.
(08:56):
She is really finding an amazing set of opportunities to just close loops and surface additional. I think kudos for our members personalizing the communications and making them feel just very thoughtful and just intentionally built for them. That goes a super long way. It’s the reason that people… Just this week I was told by one of our advisors that we are by far their most important partnership. They would put everything aside to provide more support for us and it’s in no small part due to this sort of work. We can all learn from this.
(09:29):
Then on the company side, we’ve got Q3 assemblage underway. I think we all know and have been able to join some of those events. Farm assemblage is going on, so Chris Jones launched this Montana Farm assemblage on this ranch. I haven’t gotten the update yet, haven’t yet seen some of the pictures, but I’m sure there will be many. Then next week, just to put this on everyone’s radar, the Q3 culture survey is going to launch next week. We really want to get to 100% participation on this survey. It’s super critical to us as well.
(09:57):
If you have any recommendations for feedback that you think we should add themes, specific questions, whether it’s related to programs that we’re working on or should work on, there’s a thread post that fitness put up and please add that information. With that, I’m going to jump over to Hui, who is going to give us a quick culture aside.
Hui (10:19):
I would like to talk about this Mobile Dev On-call, which is a process the Mobile Engine team recently started to experiment. What it means, is engineers on the Mobile Eng team take turns to serve this Mobile Dev On-call role for a week. During that week, the On-call engineer will very intentionally set aside up to half of their working time and dedicate that time to handle interruptions, whether it’s a production issue, a bug, or just some unknown incident that needs immediate care or just informative questions on threads.
(10:57):
I’ve linked the memo here so we have a memo that documents what this process is and how it works. If you’re curious, you can go there to view more details. We have a thread handle Mobile Eng Dev On-call. You can use that if you are seeking immediate help from the Mobile Eng team.
(11:15):
We also have the linear dashboard, which tracks all the non-trivial requests in one place, so we don’t let them fall through cracks. Within the Dev on-call memo, you can also find a pointer to a On-call run book where we document all their specific domain knowledge and troubleshooting steps for issues that engineers have encountered.
(11:40):
With this process, it not only helps the rest of the team to stay focused throughout the week, it also helps our cross-functional partners such as product and support team to know who exactly to tag for without needing to guess or worry about, “Oh, I’m bothering or I’m interrupting my engineer peers.” Over time we’ll also anticipate the On-call run book to help with knowledge distribution across the team and help reduce single point of failures.
(12:12):
In the bottom of the slide, you will find screenshots of a few examples where this process helped. The first two were, “The On-call engineer helped fix production issues.” The first one was in-app purchase flow being broken. The second one was hyperlinks not working in event-based insights. These cases, the support team raised these own threads and the Mobile Eng Dev On-call handle was tagged.
(12:39):
Their third example was, “Product team tried to push a release but find out their build was taking longer than usual.” In this case, a linear ticket was created labeled with Mobile Dev On-call and assigned to the On-call engine directly. For all these cases, the issues were picked up and addressed in a fairly timely manner.
(13:01):
Yeah, just as any other process here at Levels will continue to improve and iterate over this process. If you have any feedback or have any questions, very welcome. You can talk to me or ask their On-call engineer and shout out to their engineering team and also cross-functional partners who gave the input to make this process happen and try it out. Yeah, that’s all I have for today. Back to you, Josh.
Josh Clemente (13:30):
Awesome, thank you, Hui. Just a really cool upgrade to our systems and processes and improving, I think removing that cognitive burden for where to look and when is just super huge. Anywhere we can systematize this sort of thing, I think this is a great example of how we can improve cross-functional work without creating unnecessary process. Thanks everybody. All right, Tom’s going to do a quick aside on memo versus meeting culture.
Tom (13:57):
Hey team, happy Friday. All right, so the axiom today is memos over meetings. I think first high level, it’s important maybe to call out the context around which this value for levels first came to be, which is that most work environments, at least that we’re all familiar with, I would say default to meetings when getting work done. In short, what this value is saying for levels is that we want to default to deep asynchronous work and then just be very intentional about when and why we push to hop on calls in order to push work forward.
(14:37):
It’s not to say we don’t believe in meetings. There are definitely tons of times that meetings are going to be better than a memo. We’re not going to cover all those examples here, but think when you’re having a more nuanced or emotionally nuanced conversation with the teammate. But as a team and a culture, I think it’s important that we sort of fight the entropy of defaulting to meetings and instead make the effort for our first instinct to be doing the harder work of thinking and writing.
(15:10):
I would say high level, I think there are, at least the way I think about it, two main reasons that we want to default to memos and probably some smaller ones that roll up into these, but number one, memos require deep focused thinking. I would say simply put our position is that we’re going to be better at solving problems and achieving our goals if we orient our culture around deep focused thinking, rather than days full of 30 minute project check-ins and standups. I’m sure this is pretty uncontroversial for this group, but it’s worth calling out that fundamentally this is what we believe.
(15:49):
Then second, I would say the other category is that memos create a record for work that can be referenced later and built upon over time. Whether that’s 24 hours later, a teammate reviewing a proposal on their own time one evening, or that’s a year later when a new teammate is being onboarded and they’re looking for context on a particular project or strategy or experiment that we ran. I would just end by saying, on a more honest and personal note, I’ve definitely found it way harder these days than maybe in the very early days of levels to sort of live this value. I find myself getting on more calls and maybe being a little bit more reluctant to write a memo. I guess what I would encourage, and something that I’m thinking a lot about is just being mindful throughout my days of what I would call triggers, for when I think I should be jumping on a call and then when I think I maybe should be pushing myself to write a memo.
(16:51):
For example, if I find myself saying things like, “Yeah, we should chat about X, Y, or Z strategy or idea sometime,” if I find myself saying that about a particular strategy two or three times, that’s usually a sign to me that I’m being a little bit lazy and that I’m avoiding doing the deep work of thinking and writing about a particular topic.
(17:13):
The flip side of that is one example of a trigger for jumping on a call would be if you find yourself going back and forth in real time on threads with someone for 30 minutes, that’s definitely an example where you probably would’ve been better suited to get on a call. Takeaway, try to be intentional about how you’re doing the work and then always default to deep focused work over the meeting. That’s it.
Josh Clemente (17:38):
Nothing to add on top of that, really appreciate the personal experience and the tips and tricks there. Yeah, keeping track of when we’ve mentioned that we are holding an idea in our heads for more than one or two times, I think that’s a great trigger to start a memo, even if it’s just basic. I think we also need to lower the burden for what a memo actually means and how we can get started on one. Thanks, Tom.
(18:04):
All right. Company objectives. Levels shows you how food affects your health. No changes here. Top priority. The objectives that stack up into this are member retention, new member acquisition and member health improvement. As we all know, and based on last week, if you didn’t see last week’s Friday Forum, definitely watch it. We did a piece on the OKRs project for functional and group levels which stack up into these top level objectives. All right, Demand Gen, this is going to be Ben.
Ben (18:30):
Here we go. The migration begins right now, this time of year, all the wheat fields that surround the city are filled with these Canadian geese and it’s pretty beautiful to see and it’s something that we are undergoing simultaneously. Next slide please. One of our OKRs with Demand Gen is to grow audiences across platforms. The main goal is to focus on the platforms where we have a lot of opportunity for growth.
(18:59):
That being platforms like TikTok, that being platforms like YouTube versus platforms like Instagram where we’ve got a more mature audience and we can think more about curation as opposed to focusing on tactics to actually grow the audience. It happens over time, but this is very much aligned with when we think about the eyeballs, emails, orders, framework, this is aligned with the eyeballs part of it. Next slide please.
(19:23):
To date, we have on YouTube, we’ve been uploading all these different video assets to one channel. We’ve got culture videos, we’ve got productivity videos, Austin McGuffie, Friday Forum Special Guests, you name it, even audio versions of the podcast episodes. What you get is you get a ton of overall impressions and you get a lot of views and you get a lot of subscribers coming to the channel.
(19:47):
But over time, the early adopters start to say, “Hey, the reason that I came,” and maybe it was for productivity videos or startup videos, or maybe it was metabolic health videos, but the more the audience grows, the less that you’re serving up content that is targeted towards that audience. Next slide please.
(20:06):
What we did is we took the data from the assets that we’ve got. We’ve got over 700 different assets on the single channel right now. We reviewed these from both a qualitative and a quantitative perspective to see what resonates with the target audience. Overwhelmingly, not surprisingly, the health related content is the highest performing because that is what we do. It’s a key driver for our audience growth. When you look at the individual videos that actually drive subscribers versus just views and engagement, the health related content is our pinnacle content. Next slide please.
(20:41):
AVD, this is average view duration. This goes down with lower engagement. When you start to upload videos that aren’t related to what somebody might be interested in, if they came initially for, let’s say David Perlmutter, they’re interested in metabolic health or uric acid or whatever it is, and they see Special Guests or Friday Forum or any of these uploads go to the channel and they say, “Wait, this isn’t what I came for.” Over time the algorithm’s going to say, well, we’re serving up videos to this audience, but they’re not really engaged. This lower average view duration starts to drive down all the engagement on the platform.
(21:15):
You can see this is an example of, this was a recent podcast I think with Dorothy and one of our subscribers like, “This is cool, but it’s have to do with levels in metabolic health?” Another person said, “I wish you’d put this type of content on a different channel. I’d like to see health related content on here.” We’ve heard this anecdotally a few times, and so it came time to start to consider it. Next slide please.
(21:40):
What do other creators do? A lot of creators, Mr. Beast being one of them, avoid the Variety channel. This doesn’t mean that you can’t do it. Other creators like Gary Vaynerchuk is notorious for this. He kind of has, that’s his game, is the Variety channel where he puts everything from garage sale content to keynotes that he’ll do at different events.
(21:59):
But some creators like Mr. Beast will create all these different channels. You can see Mr. Beast shorts versus philanthropy, versus gaming, versus reacts, versus main channel. In doing so, this helps to serve up content that aligns with the people that have different interests across all these different pillars that he creates content for. Next slide.
(22:23):
The Initiative Underway is that we’re splitting our channels to generate more audience engagement. Next slide. Right now we’ve got Athena helping with this process, and as of yesterday we’ve got Levels Culture, and then Levels is our main channel. We’re starting with zero subscribers on one of them, but we’ve started to migrate anything that is just culture related content to one channel and then health related content to our main channel. As we do this, we’re ensuring that we’re continuing to get better, but only serving up to the people that want the health related content. Then the other culture is the melting pot of everything related to business startup productivity, you name it. Next slide please.
(23:02):
We are in the process deprecating some content. So long, farewell, it’s been swell, we have all these are the podcasts, the audio only version of the podcast, which sounds weird, but YouTube is an incredible discovery mechanism and some of the episodes have more views and engagement on YouTube versus through Apple or through Spotify. There is the opportunity cost, but you have to say, “Is the discovery worthwhile as compared to saying, we’ll put up the video version of the podcast and just leave these ones out.”
(23:34):
When we looked at the data overall in aggregate, there was less benefit in putting these ones on. That was an experiment when we initially did it and less benefit in continuing to do it versus just putting the video versions up. We’ll get more engagement from those over time versus the audio only ones. Next slide.
(23:53):
100K versus 1K subs. This is one of our OKRs. The goal is to get to 100K subs for Levels in the next 12 months. Consciously, we’re not going to look at data. If we only ever get to 1K subs on Levels Culture, that’s okay because that’s not our goal to grow that channel. It’s that we have an outlet and a distribution mechanism for all the content that we create. There’s a lot of benefit that is harder from a qualitative perspective to actually measure, because some of that business related content helps us with things like hiring and helps us with prospective candidates who are interested in gaining insight to the culture. Last slide.
(24:31):
Why do this> Why does this even matter? Well, splitting the channels will drive us to our OKR of audience growth and it’s going to help to generate more awareness for our content. That being let’s grow audiences, let’s bring the Demand Gen tagline is to build a media company and bring awareness to levels, while educating people about metabolic health. Splitting the channels is exactly doing this, and that is aligned with our OKRs. That is it for Demand Gen.
Josh Clemente (24:59):
Great. Love the change and seeing this evolve. Thanks a lot, Ben. All right, product team. I’m going to give a quick refresh here. Here we go. All right.
Alan (25:12):
Awesome, boss.
Josh Clemente (25:13):
Alan.
Alan (25:14):
Okay, welcome to the PMs Away Edition of product updates this week. Let’s go to the next slide. A lot of that active projects right now, we’ve got Levels Levels underway, me and David working on that. Labs is getting queued up for ENG really soon. We’ve got 1.1 and 2.0 pretty soon on deck, tons of in-app content work, Stacy, Mike. I’m not actually doing that much on there. Why did I put my face in there? That’s mostly David, Stacy and Mike.
(25:44):
In concept stage right now we have Personalization, which should be a pretty exciting integration in the product. We’ve wanted to do that for quite a while and Scoring 2.2. Let’s get into some of the details here.
(25:59):
I think one that we’re really excited about is figuring out a way to increase algorithm development velocity. In the past we’ve had this challenge of translating what the tooling that you typically use in data science into our production environment. Usually most of our data scientists, they’re doing their work in Python for example. What we’re hoping to do is spin up this Python service that will allow us to develop algorithms in production a lot faster.
(26:27):
This is actually subtly sort of a behind-the-scenes effort, but it’s going to make a big difference in things like spike detection or scoring for example, and down the road things like insight identification. That’s some great work by Jason Gleet, Shin Lu, Andrew. Looking forward to seeing that going out. Next slide.
(26:50):
Scoring 2.2 is ready for internal testing. You might recall we’ve launched the spike detection in integrated and we integrated that into My Data and we found through some user feedback and just using it ourselves, that it’s lacking some of the nuance that we hope to have to show progress over time and help you evaluate the good days and bad days.
(27:12):
We’ve evolved the score and this is a… Oh, I missed John, sorry about that. We’ve evolved the score and this is based on a simpler version of the past where we had area under the curve and average glucose and so on. This hopes to mitigate the issues that we had around sensor variation and looking at average glucose. It’s focusing just on the wobble up and down. How stable are you, which should align it pretty nicely back to our spike model that’s on the homepage.
(27:41):
You can go check this out now if you go to My Data, a little dropdown, this is just an internal quick and scrappy test. It’s reusing all the old metabolic score visuals and you can take a look at that. What we’d love to hear from you is how does this work? How does it feel to you? Is it too easy? Is it too hard? Does it largely map to what you’re seeing in your glucose line? This was some great, again, algorithm development, Jason Gleet, probably some others that I’ve forgotten. I’m sorry. Yeah, let us know what you think. Next slide.
(28:16):
Healthy food choices is underway. Justin’s cranking on this, some of you might notice some of the initial modal has been integrated and there’s a lot of content integration underway. Some of the things that you can look forward to here are tons of this rich content being integrated into the app. We’ve got recipe videos, guidance around food. Really looking forward to this. Next slide.
(28:41):
One of the reasons it’s such a wonderful development is that it’s going to set the stage for us to do some of these explorations around food. One of them is this Instacart exploration, where we’re looking at how we can sort of increase ability for our users and surface this little point of action where say on a recipe you can quickly add those ingredients to a shopping list on Instacart or on event-based insights. Perhaps we can surface again another little integration point there to add it to your shopping list.
(29:11):
Next week we’re going to begin exploring the API, looking at how we can integrate it. I think it’s going to be potentially quite a exciting development, which are one of our first partners that we’ll be working with and we’ll see how user interest lines up with that. Next slide. Thank you.
(29:29):
Labs 1.1 is on deck for sprint planning next week. You’ve probably seen a bunch of screens about this in the past, but I think we’re really excited about it. The first evolution of this is going to probably get rid of or smooth out some of the initial signup flow issues. It’s going to be a lot easier to get signed up, it’s going to look a lot more polished, build some trust around it, and we’re going to start integrating some of these conversion points into the app, such as in the right here with these event-based insight.
(30:00):
You can see 2.2 or 2.0 will be on deck as well pretty soon. That’ll have much more refined set of Labs or a simplified set of Labs. The cost will be reduced and the presentation will be cleaned up a ton. Levels Levels. We’re going to begin doing some user testing next week on this. Again, just to recap on what this is about, this is intended to teach users the concepts around metabolic health through gamification. We think this could be a fun way to build some comprehension around what to eat and not to eat.
(30:34):
We are going to begin testing next week just to evaluate how people respond to this. It’s important because it’s kind of an existential change for the product you’re getting things like points and leveling up and so on. The things that we want to learn is, is this a fun game? It’s not going to be effectively gamified if people don’t want to play it. How do people respond to this notion of points. That can be quite polarizing on that.
(30:59):
We expect to hear a variety of different sets of feedback and we want to tailor it so we don’t over index on it, but we incorporate some of their feedback. Essentially, when you’re getting points, does it essentially line up to what you’re expecting to get? That’s it for product this week.
Josh Clemente (31:16):
A lot of beautiful work there. Yeah, really excited for all this, especially I think the new Labs relaunch and Levels levels. These are both going to be really interesting to learn from. I have my assumptions on how this is going to work and I’m very excited to try it out myself. Thanks a lot Alan and product team. All right, on the Experimentation and Learning side, Azure is going to take over the share from here briefly. I’ll stop here.
Azure (31:43):
Even the format of this is going to be an experiment, so we’ll see. I’m going to try a little bit of white boarding. Alrighty. You guys let me know if you can see this screen and me writing. All right, great. Let’s talk a little bit about clinical glucose metrics and the relation to the shape of the glucose curve. If you’re curious or saw this participatory research talk over Assemblage, this pertains to levels wide citizen science, looking at the data sets we’ve already collected in relation to the CGM curve and to blood work.
(32:17):
Our big question is how to relate CGM data to health in people without diabetes, since historically most of the research on CGM data was done in a diabetic context. We have some different pieces of kit in order to do this. One that we’ll talk about is our blood work data.
(32:35):
Another one, are clinical metrics that are derived from the CGM curve itself. These are things like average glucose or glucose variability, think standard deviation. Then at the bottom, I’m calling these Levels Metrics here, I’m sure we have lots of good names for them, but these so far have been things like different kinds of excursions from a glucose average or spikes. We can look at things like their duration, their size, and their stability.
(33:01):
I think the advantage of these ones is that they’re designed for nuanced relation to short-term behavior and for encouraging people along the right track, whereas these more clinical abstract long-term metrics might not do that as effectively. I like lots of analogies. I’ve put three, but I’ll just call out my favorite one. I think of blood work as this single note reflecting the song of your metabolism.
(33:25):
I don’t know how many of us can identify a song by a single note, probably on our favorite ones, but a little more detail clinical metrics that are derived from continuous glucose data. I think of those as a few stanzas of our metabolism song. What we want to move to are these Levels Metrics that help us follow along through time and listen to the whole song.
(33:46):
Let’s start out for a minute and talk about that 10,000 foot single note, eight-bit picture of health that is nonetheless quite important. Blood work. These are some histograms of blood work results from 361 of our members pulled earlier this year. Men are in red, women are in this bluish color and this shading in the background indicates the in range values use.
(34:12):
One thing that I thought was really interesting and is both good and provides challenges, is that if we look at A1C fasting glucose and fasting insulin here on the top, those are mostly in range for these members. If we look at the lipid panel results, so HDL LDL and total cholesterol in the middle, we see we have some more out of range values, particularly in LDL and in total cholesterol. But as you’ve probably heard and in discussions across the team, these metrics are kind of evolving and being replaced with things that might be a little bit more health relevant overall. we may even change our panel to reflect that.
(34:51):
Trigs and CRP. This metric of inflammation, those are largely in range for our members. Then our age distribution here is average at about 40. The men’s skew just a little bit younger and we have a wide overall range from 20 to 80.
(35:08):
Let’s talk about this in the context of how blood work relates to blood work first. One thing you might assume, I had certainly assumed a while back in the past, is that things like fasting glucose and A1C, these labels are a little bit small, would correlate super well to each other, but actually they only relate to each other about 19%. It’s a statistical correlation but not a perfect one.
(35:32):
Relation of A1C to things like fasting insulin and CRP is a little weaker still, so that’s around 2-4% and then not super related to lipid results. Blood work is not definitely a match to all other blood work. Another point I wanted to call out from looking at blood work in relation to CGM data is that timing matters. One thing I saw is that only 14% of people wore a CGM and at least one out of the last two weeks prior to getting their blood sample. I think as we go along we can get a lot more from helping people learn how to relate their blood work results to their continuous glucose, if we can get them to do both at the same time. Josh, yes, you’ll be curious about the rest of this little talk.
(36:20):
Correlation only around 5% with CGM derived metrics to A1C. Super interesting there. It definitely surprised me. These correlations which seem to come out more strongly in diabetes data, don’t necessarily come out as strongly in data from people without diabetes.
(36:40):
For the last couple of minutes, I wanted to talk with you about the shape of the curve and the kinds of shape changes that I’ve seen on the progression from people without diabetes, through pre-diabetes, or very well-managed Type 1 through Type 2 Diabetes or 4T1 that is managed using older therapy strategies. Basically if we look up in the top, these green curves, people without diabetes tend to have more stable, smooth, low amplitude daily and hourly excursions. These little black curves, if this is day 1, 2, 3, these tend to be smooth and low and I think we’re used to seeing that.
(37:19):
In cases of individuals, and this is some data from an old paper my lab worked on in school, this is reflective of what a lot of pre-diabetic logs look like or sometimes some artificial pancreas use. These tend to start by having these taller spicier excursions, which I think matches a lot of what we’ve seen in our data and how spike detection is working. Again, this would be a picture of over about three days.
(37:46):
Then what seems to happen, though I think this is a little more out there, is that when a person goes all the way into a Type 2 Diabetes state or when T1D is less well-managed, it’s almost as though some of these spikes bleed together in these long 5, 6, 7 hour excursions and the curve itself becomes less stable.
(38:06):
With those thoughts really quick, how does this relate to what we’ve seen in our data? A couple takeaways that I’ll walk through one by one are keeping excursions stable on short timescales, we’ll talk about that on the lift, is associated with lower glucose variability. If we take our metric of comparison as standard deviation of the CGM curve, something that’s been used clinically and studied, basically the more stable that curve is, the more you’re bouncing along every one to two hours regularly, the better the outcome and the more unstable. This would be a picture of having a spike, maybe one hour going a long time without one having very irregularly timed glucose excursions. That seems less good.
(38:50):
Again, more stability, less standard deviation, greater, or sorry, less stability, greater standard deviation. Then conversely, long tall excursions are most responsible for high glucose variability I’ve seen. These are those spikes that last five to six hours and they’re very tall. Again, the taller those are, the greater STDEV, the shorter or more absent they are the lower.
(39:16):
Moreover, those long tall excursions seem most responsible of any excursion type I’ve looked at for high average glucose generated from the CGM. In total, these are my big takeaways just at a high level. One is that timing matters. I think if we can encourage people to collect their CGM data in close proximity to their blood work, our lives will be easier in trying to relate that CGM data to the blood work.
(39:42):
Number two is with published A1C to CGM metric correlations, thinking about the correlation of say A1C to average glucose or A1C to STDEV, those improve when A1C is very high or an average glucose is very high. I think we’re already doing this, but just an extra note of being cautious when we try to transfer expectations from the diabetes literature to people without diabetes, we have to recheck and re derive these rules for ourselves, which I think can make our jobs harder but more exciting too.
(40:16):
Then, the third one, which I think is fun from an insight perspective and you can head over to research pearls to learn more details. This is that stable hour to hour glucose oscillations and reducing five to six hour spikes are so far what I’ve seen to be the strongest things that you can do to improve average continuous blood glucose and improve or lower glucose standard deviation.
(40:42):
That’s it for today. I know that was a lot, so I would love to hear your questions and critiques. Hit me up anytime and I can link in threads to the more detailed version of all this.
Josh Clemente (40:56):
Well I think that was about as good as it gets here for the Experimentation and Learning segment. Tons of really awesome stuff there. Thanks a lot Azure, and just stoked to see the analysis and insights that we’re getting from the data we’re collecting. It’s amazing. Lots more that we can do here.
(41:17):
All right, thank you for that quick hiring update. Priya is joining us October 17th. Matt is the only current team member who has not yet onboarded. Open roles haven’t changed week over week, still looking for our engineers, people ops, generalists and R&D engineering.
(41:34):
Then of course if you want to make your own role that you see levels needing when you watch these Friday Forums from the outside, let us know. Head to levels.link/careers. All right, we’ve got a couple of minutes here for some individual contributions. We’ll stop the share here, we’ll do these by hand raise. If you press the reactions button on your Zoom app, you can raise your hand to do a share. Yeah, like this.
(42:02):
I’ll start. Professionally, I’m very excited about some of the developments I think on, probably the most excited about UK expansion right now. It’s really cool to see a bunch of things like e-commerce come to life alongside the expansion opportunities. It’s just going to be a huge part of our business going forward and so much has happened behind the scenes.
(42:27):
Then I’m also very excited about some of the opportunities that are coming to life as a function of the logistics challenges we’re facing. We’re going to be forced into some big upgrades, updates, TBD, exactly how they play out, but I think what we will end up with on the other side is going to be a much more smooth process for our members. On the personal side, the weather has turned here in Austin and it was 58 degrees when we played pickleball this morning and I’m all about it. Hui.
Hui (43:00):
Yeah, I’m I guess professional, I’m excited about the OKR process and just reading all their functional OKRs. I definitely get a better understanding of what the whole company is doing and I really like their reporting dashboard. I think that’s a really great way to make sure we are on track and if you’re not on track, you have a place to read the awareness on.
(43:27):
I think that’s definitely greatly increase my confidence in we’ll be successfully making more impact for our users. I’m also very excited that Shawn joins. Yeah, it has been great talking to him during the hiring process and it was really, really very exciting news that he decides to join and is finally here. Yeah, that’s it.
Josh Clemente (43:52):
The only downside of Shawn joining is that we now have two Shawn both spelled A-W and I don’t don’t know how we’re going to tie-break, who gets to keep the name. Azure.
Azure (44:03):
I don’t know. Maybe we’ll have to go by initials. I’m weirdly excited about the Mobile Dev On-call thing. I think I’m going to use that a lot. Good job making that. I’m also really excited about the process for product research content generation. I think we can make it super efficient. I’m really looking forward to that and everybody’s given a bunch of really good feedback about it.
(44:26):
Personally, I’m kind of bummed. I’m trying to avoid shin splints, but I’m learning all about icing. If anybody has any running recommendations for shin-splint healing, let me know.
Josh Clemente (44:39):
My only recommendation is don’t run in basketball shoes, because that’s how I got shin splints in college. That’s all I got. Ian.
Ian (44:49):
Agreeing with you on the weather. This is about to become the best time to be outside, best time of year to be outside in Austin. On the professional front, we’re sort of entering, I wouldn’t call it the home stretch by any means yet with the e-commerce or now commerce project, but it’s entering a phase where we’re starting to see long investments that have been made over multiple months starting to actually manifest in the world, and that’s been really exciting for me in the last couple of weeks.
Josh Clemente (45:21):
Awesome. Yeah, we’ll have to get the pickleball team together though, while the weather is great. Miz.
Michael (45:29):
Welcome, Shawn. Excited to meet you. Enjoyed the week this week. A few of the Assemblage events, the photos coming out of this Farm Assemblage are very exciting, so I think there’s a lot to learn there, which will take to heart and figure out how we approach that moving forward as this team grows.
(45:43):
But excited for all those [inaudible 00:45:45] meeting and yeah, looking forward to learning. On my end, I was in New York this last week for Rosh Hashanah and saw a handful of people. Nice to meet [inaudible 00:45:54] and Tony and Sam, and a few others in person. Then in Austin next week, so excited to see people there as well. Busy season. I guess on the work side, a lot of really exciting Truepill advancement, so pretty pivotal episode coming up, which I’m excited to watch.
Josh Clemente (46:12):
Likewise, looking forward to seeing you. Scott.
Scott (46:17):
We’ll echo, I think a lot of what Ian said. Consequently, they’re building a house down the street from me right now, and it’s just been months of the framing going up and I think in very short order it’s going to make a lot of sense why they’ve been spending a lot of time on the things. Hopefully commerce will parallel a lot of that. I’m actually pretty positive it will.
(46:34):
I feel like product is cooking right now. It’s really fun. As I’ve transitioned back to engineering, watching it from the outside, it just feels like velocity’s really picking up, which is really, really fun to see. I got to say though, when I saw the general thread last night about the Truepill stuff, I had a moment of panic, but then I was actually really excited, because my first thought it was like, “This is going to be crazy and kind of wild, but this is also what we signed up for and everybody that I’m going to have to work with to do whatever changes we need, I’m excited to work with.”
(46:59):
There wasn’t any moment of dread where I thought, “Man, this is going to suck for whatever reason, because I’ve got to go interface with X or we’ve got to change this thing.” I just felt like universally I was excited to just jump on. I literally was looking at flights. Do I need to go to Montana for a couple of days to battle plan? It’s not that urgent. It was fine, but it was just a good reminder of I feel like I’m happy to be at this company. I feel like in times these, I’m wanting to lean into it as opposed to be scared of it.
Josh Clemente (47:25):
Well said. Montana never requires urgency. I recommend just going just in case.
Scott (47:31):
That’s true.
Josh Clemente (47:33):
All right, anybody else for the weekly share? Okay, we got 60 seconds back. Go enjoy the awesome weather.