[00:00:00] Dr Genevieve Hayes: Hello, and welcome to Value Driven Data Science, the podcast that helps data scientists transform their technical expertise into tangible business value, career autonomy, and financial reward. I’m Dr. Genevieve Hayes, and today I’m joined by Lauren Lang and Dr. Matt Hoffman. Lauren is the Director of Content for Uplevel and is also a Content Strategy Coach for B2B marketers.

[00:00:26] Matt is a Data Analyst and Product Manager at Uplevel and holds a PhD in Physics from the University of Washington. In this episode, we’ll uncover proven strategies for transforming complex technical findings into compelling business narratives that drive real organizational change. So get ready to boost your impact, earn what you’re worth, and rewrite your career algorithm. Lauren, Matt, welcome to the show.

[00:00:55] Lauren Lang: Hi Genevieve, thank you so much.

[00:00:57] Dr Matt Hoffman: Thanks for having us. Excited to be here.

[00:01:00] Dr Genevieve Hayes: In logistics, there’s a concept known as the last mile problem. Which refers to the fact that the last stage of the delivery process of people or goods is typically the most complex and expensive while also being the most essential. For example, it’s typically easier and cheaper to fly a plane full of packages from Australia to the U.

[00:01:22] S. than it is to transport those packages by road to their final destinations within the U. S. Yet if you can’t distribute those packages once they arrive in the U. S., they may as well have never left Australia. It’s for this reason that supply chain managers typically focus a disproportionate amount of effort on planning those final miles.

[00:01:43] Data scientists also face their own last mile problem. Despite many data science projects requiring sophisticated modelling and analysis techniques, the most difficult part of data science is often communicating the results of those projects to senior management and gaining adoption of the project from the business.

[00:02:04] That is the final stage. Yet, unlike in logistics, This is also the stage where data scientists typically focus the least amount of effort, much to the detriment of their work and their careers. Lauren and Matt, the reason why we’ve got both of you as guests in today’s episode is because you’ve recently backed this trend and pooled your combined experience in communications and data science with outstanding results.

[00:02:33] And this is actually the first time I’ve come across a data scientist working directly with the communications expert to address the data science last mile problem. Although, it probably should be far more common. So to begin with, Matt, can you give us an overview of the data science project you were working on and how you came to team up with Lauren when delivering the results?

[00:02:57] Dr Matt Hoffman: So we work at Uplevel and Uplevel is a company that pulls in data about software engineers and we help tell those data stories to our customers. Senior leaders of engineering, like software engineering firms so that they can make data driven decisions and drive change within their organizations.

[00:03:17] One of the things that’s really come up in the past year is this full topic of gen. AI software engineers being able to talk to an AI assistant to help them write code and the thinking was, oh, this is a silver bullet. We’re just going to be able to. Turn on this system. Our developers are going to be more productive.

[00:03:36] Instantly. The code is going to get better. There’s going to be nothing but greenfield. If we just turn this on, it’s a no brainer, we heard those questions and we don’t develop our own gen AI tool. But what we do have is data about software engineers and how they spend their time, the effectiveness of their work.

[00:03:54] Are they able to deliver more? Are they getting more things done? How’s the bug rate of their code? So it was natural for us to go explore that problem and really try to understand what is the impact of Gen AI on software engineers. That’s the problem that we were facing. So I work with our data science team.

[00:04:13] I’m not actually on our data science team, but worked with them to go do this analysis to really try to understand how do people compare to themselves and what changes do we see within this. And then we pulled in Lauren to go start showing off what we found. And that’s where that story kicked off.

[00:04:32] Dr Genevieve Hayes: Prior to working with Lauren, what are some of the challenges you encountered in communicating the results of your analysis?

[00:04:38] Dr Matt Hoffman: Well, it’s always a tricky one when the answer is complicated. The real fundamental place that we at Uplevel are at is that this is human data. While we may be able to measure timestamps to a millisecond, This is all still predicated that this is people data and people do weird things. And the data is messy and the data is muddy.

[00:05:03] So there’s the constant battle of, well, what can we trust? We’re looking for correlations and, you know, you squint to see if like, there’s something there you peel back a layer and then there’s something more, but people data is hard to work with. So that’s really a skill of our data science team to help pull that back.

[00:05:20] But we were. Kind of struggling to make heads and tails of what were the real conclusions. And Lauren really helped clarify that story for us and get that communication there.

[00:05:30] Dr Genevieve Hayes: People are irrational. I mean that’s the big problem with us. Before you did this, had you ever made some massive mistake because you just assumed people were rational when they worked?

[00:05:44] Dr Matt Hoffman: It’s funny stuff so sometime when some work’s becoming delayed and you go ask for the root cause and it’s like, oh, someone’s saying, I thought I did that and I forgot. Like, I never hit the button. That’s the kind of, people data that we see is that, like, yeah, that happened.

[00:05:59] It was late, but that was just because you forgot to hit the button. People’s behavior is really funny. So yeah, we just have to kind of take that into account that everybody’s different. That’s okay. And we need to bake that into our analysis, that people work differently and not try to over fit one model that applies to everybody .

[00:06:18] Dr Genevieve Hayes: Yeah, I actually wrote a LinkedIn post a while ago saying, people are a problem with data and wouldn’t it be nice to just be dealing with mechanical processes? And I had someone reply to that post who works at a water agency where they don’t deal with people, it’s, water going through pipes, and they said, well actually mechanical processes are just as annoying, they just are annoying in different ways because you have the sensors malfunctioning and all this.

[00:06:44] You can dream about not dealing with people but Machines cause problems too .

[00:06:48] Dr Matt Hoffman: Yeah, that’s exactly right. So you just have to know that going in and know that it’s going to be messy. And plan for that.

[00:06:56] Dr Genevieve Hayes: So Lauren, in your content strategy coaching work you’ve done a lot of work with software as a service companies. And as Matt said, Up Level itself is a company that Works with engineers and probably has a lot of engineers as its employees. So, I’d imagine you’ve worked with a lot of very technical people throughout your career.

[00:07:20] Lauren Lang: I have. Yes.

[00:07:21] Dr Genevieve Hayes: What are some of the biggest issues you’ve noticed in how technically minded people, especially data scientists and data analysts, present their findings to business stakeholders?

[00:07:33] Lauren Lang: It’s very funny because I think that there is a lot of similarities actually between how data scientists might present their findings and how a lot of marketers present their findings. And you would think like, Oh, marketing is so much more. We have our thumb on the pulse of the business.

[00:07:48] And, marketers are so much more business driven, but I think, anyone who is looking at data as marketers, we look at data too. We are. Not data scientists, but there’s a fair amount of data science, sometimes in marketing. And there’s a lot of data analysis that happens. And I think there is just this tendency sometimes to.

[00:08:07] Get very myopic and get very focused on your own specific context in looking at the data and forgetting that there is probably a larger story that the data existed to tell. I see this a lot. 1 of the. Challenges that I see a lot is, marketers will go into a meeting with a CEO and they will have dashboard after dashboard and chart after chart.

[00:08:31] And there is a very sort of distinct look on an executive space when. You’ve shown them three charts in a row or three dashboards and it’s like a completely blank look and you know that they are literally anywhere else. but in the conversation and it’s a little bit of like a death now.

[00:08:51] And so I think for anyone who likes to geek out on data, whatever part of the business you’re in, you have to remember that there is this larger value story that you need to be telling, and you need to be showing that data and be mindful of the context in which you’re showing that data.

[00:09:08] To what end? Rather than just taking people down the rabbit hole with you. I think sometimes there’s an assumption that everyone should be as interested about all of the nuances and slight, variances in the data as you are, and that’s not always the case.

[00:09:24] Dr Genevieve Hayes: Yeah the way you’re describing that death knell face, yeah, I’ve seen that before. And worse than that is when the people you’re presenting to start playing with their phones. Then you definitely know that you’ve failed.

[00:09:35] Lauren Lang: Might as well call it right there.

[00:09:37] Dr Genevieve Hayes: Yeah, , just pack up and walk out of the room at that point.

[00:09:39] Lauren Lang: That’s right. That’s right.

[00:09:42] Dr Genevieve Hayes: So, I assume you’ve pointed out these issues to technical people who you’ve worked with. How do they typically respond when you say, hey, not everyone’s as geeky as you?

[00:09:53] Lauren Lang: I think there’s a way to couch that in a way, because I have a lot of empathy for it. Geeky people are excited about what we do. I mean, there’s a passion there. And so you don’t want to not communicate that passion.

[00:10:05] I think that’s really important. And, there’s some exciting results or, even. Not exciting results that you didn’t think were going to pan out, but there’s always a story to tell, but it’s just, can you tell it maybe at a slightly more abstract level of specificity, maybe? Or can you tell it with an understanding of the context in which your audience exists

[00:10:28] I think there’s just a lot of tendency to Just forget that not everyone brings the same experiences and the same understanding and the same depth of knowledge to the table. And so the best way that the stories we tell with data can be impactful is to tell them in context and to be able to pull out the important parts that really can bring the message home.

[00:10:50] Dr Genevieve Hayes: So, put yourself in the shoes of your audience,

[00:10:53] Lauren Lang: absolutely. You should always have empathy with the person you’re trying to communicate to. I think it was Kim Scott said that communication happens at the listener’s ear and not the speaker’s mouth. That’s where meaning is made. It’s really important to keep that in mind as you are stepping into the shoes.

[00:11:09] Of the communicator,

[00:11:11] Dr Genevieve Hayes: so, I’d like to now take a deep dive into the project that the two of you collaborated on so Matt, how did you determine which insights from your analysis were most relevant for communicating with management? Are

[00:11:24] Dr Matt Hoffman: So we have a set of measures at up level that are kind of part of our standard suite of analysis. So 1st, because if you can’t go explore the data for yourself and understand where your team’s at, then that’s a really unsatisfying experiment. So we knew that we wanted to look at some of these measures.

[00:11:43] We’ve also been doing this for a few years now, so we do have a pretty good grasp on. You know, what are appropriate measures to look at for software engineers? And then what is completely inappropriate? That’s like, this is just not a good measure. You shouldn’t use it. It’s problematic for 1 reason or another.

[00:12:01] So choosing those measures that we think. Are kind of universally applicable, are good proxies of how this experience may look, and then really trying to see what’s going to move and shift when we look at these. Those were kind of the criteria. We had a few hypotheses that we went in for how we thought things were going to move once you introduced Gen AI to the mix.

[00:12:22] And we were surprised by our hypotheses, and we had to reject some of them, which was really fun. And it makes you really challenged that you’re doing it right. And then finding that this actually does go against what we thought would happen.

[00:12:36] Dr Genevieve Hayes: you able to share any examples of these?

[00:12:39] Dr Matt Hoffman: One of the things that we wrote about and we can share the link to our study was the general thinking was, hey, if you’re going to use Gen AI, you’re going to be able to ask questions and Jenny is going to help you write better code. So one of the things we looked at was. What’s the defect rate of code that gets merged and then it needs to get fixed later?

[00:13:02] So how often does that happen? You would think that that would go down if the code is going to be of higher quality because Gen AI is helping you. Now what we found was that actually the defect rate went up. Another organization seemed to find the same thing, saying that the result of Gen AI was that there’s larger changes to code.

[00:13:23] And then more things are going to get missed because the batch size is getting larger. So you might find things. four bugs, but there’s five because you’re writing bigger and bigger code changes. So we saw that the defect rate for the cohort that was using Gen AI went up by 40 percent compared to themselves, which is a pretty market change.

[00:13:43] So that was one that , we were very surprised to see and are really interested to see what happens next with that as all these tools get better and better and better.

[00:13:53] Dr Genevieve Hayes: insight you just described, that doesn’t surprise me because my own personal experience I’ve found with writing code using Gen AI, you can produce the code really, really fast. You’re spending. twice as long or three or four times as long debugging it, because there are all these bugs in it that would not be in there if you’d written it yourself.

[00:14:14] And you’re just not used to having that many bugs to fix.

[00:14:19] Dr Matt Hoffman: Yeah, and it might be not stylistic, like, the way that you think that you should write your code it might pull some solution that looks reasonable at first pass, but it’s pretty hard to debug if it’s the right thing when it, looks right, smells right, but then under the hood, there’s something wrong with it.

[00:14:36] Also, Jenna, I doesn’t understand the context of the problem that you’re trying to go write code for. You have that in your head, you know where you’re at and where the destination is, and it’s going to help you write some code. But you have that.

[00:14:49] Dr Genevieve Hayes: Yeah. And I’ve found it creates. Non existent Python packages and non existent Python functions, which is fun, because then you spend half an hour trying to find this package that doesn’t even exist.

[00:15:02] Dr Matt Hoffman: It’s tricky. It really is. The other one that I would just briefly say that we looked at is we thought people would write code faster. That’s the statement that you just said. How quickly does it take to get from commit to merge? Does that really pick up? Because you’re using Gen AI.

[00:15:16] And we found that it didn’t make much of a tangible impact. That there’s still a lot of time that’s spent when you’re trying to understand the problem of what you’re trying to solve, how you might approach it, the architecture of it. None of those things are going to go away.

[00:15:31] Bottlenecks of having another human review your code, that doesn’t change whether they both have Gen AI or not. You’re still working with other people. So those structural factors do tend to be very important in this problem. And those are ones that you need to pursue and kind of conventional means of understanding how your teams work and doing better.

[00:15:51] So that one didn’t move at all. And we thought that that would speed up. That was our hypothesis.

[00:15:56] Dr Genevieve Hayes: Yeah, doesn’t surprise me. So, Lauren, how did you take these insights and structure them into a narrative that maximized their impact?

[00:16:04] Lauren Lang: well, it was funny because even before we had done the research, we knew we wanted to do this research and we wanted to publish it. And looking from a content marketing perspective, I think original research right now is one of the most, potentially impactful formats for creating content.

[00:16:23] And some of that is that, there is so much out there. That is just really bland. And I is not helping. Jenna is not helping with that. There’s a lot of content. That is just not special. It’s not differentiated. It’s not helping to educate or inform anybody or share anything new. And so when you have the opportunity to sort of lend something new to the conversation, that’s an important opportunity.

[00:16:46] So we knew going in that we were going to do it. What we were not expecting were the results that we got. And I laughed a little bit when we got these results. I had a meeting with our data science team and with Matt, and., we all are sitting down and I’m like, lay it on me tell me what the results were and they were a little bit disappointed and they said, it’s kind of we’re not seeing, a big thing from Impact perspective or a data perspective, like, it’s just not that exciting.

[00:17:15] And I said, oh, no, actually, this is very exciting because there were a number of factors. I think that really made this a really impactful report. 1st was just having some new original research on this topic. That is maybe the hot topic of the decade.

[00:17:31] I think was really exciting. So it was like, listen, we know that people are very interested in this. We know that this is the question that they are asking, especially engineers and engineering leaders, the people who we serve from a business standpoint. They want to know is gen AI actually helping my developers be more productive.

[00:17:48] And we have like some. Things that we can show around that. And then also the fact that we were able to then bring a little bit of a spiky and contrarian point of view about this because a lot of the research that’s been published already is either survey based. So, a lot of developers reporting whether or not they feel more productive.

[00:18:11] Which is data as well, but, this is we’re bringing some quantitative data to bear or some of the other data was published by the. AI tools themselves, so you have to take that with a grain of salt. So, we came in

[00:18:27] with this sort of interesting and different point of view. And that really, really took off for folks. And we found that some people were surprised. We found a lot of developers and engineers like you, Genevieve, who are not who said, I have been saying this all along. And this feels very validating because I think there is some anxiety among engineers that, Hey, like leadership just thinks that can be replaced.

[00:18:50] But it really kicked off a really big conversation in the industry where we just said, Hey, you know, there’s a little bit of a hype cycle right now. We don’t know for sure. , we have results from one sample. There’s no big claims that we can make about the efficacy in the long run.

[00:19:06] And things change very quickly. Gen AI is improving all the time, but. We do have some data points that we think are interesting to share and it really took off and it was great for us from a business perspective. It really helped take the work that we do into that last mile. And it helped make the work that we do feel very tangible and accessible for folks.

[00:19:29] Dr Genevieve Hayes: So it sounds like, rather than taking a whole bunch of statistics and graphs, which would have been the output of Matt’s work. You translated those statistics and graphs into a narrative that could be understood by a person who wasn’t a data scientist or wasn’t a data analyst. Is that right?

[00:19:49] Lauren Lang: Yes, we did. And our audience is primarily engineering leaders, engineering leaders are not data scientists, but they’re technical. So we identified three main takeaways. And we presented that we shared a little bit about our methodology.

[00:20:03] And we shared essentially Some thoughts about what does this mean, what is the larger significance of what we found? What does this mean for you as an engineering leader does this mean that we think that you should stop adopting AI?

[00:20:17] Does it mean that, right?, you should be more controlling of how your engineers are experimenting with AI. And, we don’t believe that’s the case at all. But it allowed us to sort of share some of our perspective about, how you build effective engineering organizations and what role we think I may have to play in that.

[00:20:35] And, that is the larger story where data becomes very interesting because there’s sharing the data and then they’re sharing the so what around the data. So, what does this mean for me as an engineering leader? And so we really tried to bring those 2 elements together in the report.

[00:20:51] Dr Genevieve Hayes: How was this report ultimately received by the audience?

[00:20:55] Lauren Lang: Very well. We issued a press release around it. And I think we were picked up globally by somewhere between 50 and 75 media outlets, which. For a small engineering analytics platform, I’m pretty happy about that. It was in some engineering forums, it really became a big topic of discussion. We went sort of medium level viral. And it felt really good. It’s like, this is a really interesting topic. We accept that it’s an interesting topic.

[00:21:22] We think that we have something that is very interesting to add to the conversation. So, yeah, it was good and some folks to it was great, you know, because engineering leaders are naturally skeptical. This is 1 of the most fun parts about marketing to engineering leaders that engineering leaders hate marketing.

[00:21:38] So we got a few emails of folks who are like, tell us more about your methodology. And they really sort of wanted to, see behind the scenes and really, really dig in. And, that is par for the course. And we would expect nothing less

[00:21:51] It was a really positive impact. I’m really glad we did it.

[00:21:53] Dr Genevieve Hayes: So with all that in mind, I’d like to ask this of each of you. What is the single most important change our listeners could make tomorrow to accelerate their data science impact and results?

[00:22:05] Dr Matt Hoffman: I. am very fortunate to have Lauren as an editor even when we collaborate on writing, an article I think having someone who can help you clarify and simplify your story is so important. You really do want to edit and bounce back and forth and try to distill down the most important bits of what you’re doing.

[00:22:28] I tend to want to share, like, Everything, all of the details, all the gritty stuff, the exact perfect chart and it’s like, let’s simplify, simplify, simplify. And part of that conversation is also, who’s going to be receiving this? And what’s their persona? At what level are we going to explain this work?

[00:22:47] Are they going to be familiar with, the methodology that we’re using? Or do we need to explain that too? So, how do we write everything at the most appropriate level and understand the life cycle of? This report that we’re doing. So having an editor would be my big one and understanding your audience would be the other.

[00:23:06] Lauren Lang: I absolutely agree with everything Matt said. I think that the more that you make Sharing the results of your research, a team effort and a team sport, the more you’re likely going to succeed at it. But I think probably, and I’ll just come at it from, more of a technical perspective.

[00:23:23] When you are presenting information, 1 of the things that could be very helpful is to present it at various levels of detail. So, making sure that you are presenting key takeaways or abstracts at 1 level and then. People can always double click into things and dive deeper and, you can include appendices or include links to , more of the detailed research.

[00:23:47] But I think sort of having these executive summaries and really sort of being able to come at things from a very high level Can help sort of get that initial interest so that people understand quickly. what did the research find? What is the impact? And what is the context that this research was performed in?

[00:24:06] Where is the business value, so, being able to connect the dots for your audience in terms of not only did we find this, but here’s what it means. And that thing that it means is actually very impactful to you and the job that you are trying to accomplish .

[00:24:19] Dr Genevieve Hayes: So for listeners who want to get in contact with each of you, what can they do?

[00:24:23] Lauren Lang: I live on LinkedIn. So they can look me up on LinkedIn. I think my little handle there is ask Lauren Lang.

[00:24:31] Dr Matt Hoffman: Likewise, I don’t know what my LinkedIn handle is, but I’m on there. That would be the easiest way to get a hold of me on that.

[00:24:39] Lauren Lang: You obviously need to spend more time on LinkedIn than Matt.

[00:24:42] Dr Genevieve Hayes: Yes. And there you have it. Another value packed episode to help turn your data skills into serious clout, cash, and career freedom. And if you enjoyed this episode, why not make it a double? Next week, catch Lauren and Matt’s Value Boost, a five minute episode where they share one powerful tip for getting real results real fast.

[00:25:08] Make sure you’re subscribed so you don’t miss it. Thanks for joining me today, Lauren and Matt.

[00:25:12] Lauren Lang: Thank you so much for having us.

[00:25:14] Dr Matt Hoffman: Thank you. It was really lovely.

[00:25:16] Dr Genevieve Hayes: And for those in the audience, thanks for listening. I’m Dr. Genevieve Hayes, and this has been value driven data science.