²ÊÃñ±¦µä

Jeff, welcome to another episode of creative distillation. Your hosts, Jeff and Brad from the University of Colorado, Boulder's Leeds School of Business, discuss entrepreneurship research while enjoying fine craft beverages. For this episode, we continue with our guests from the previous episode, Jennifer and Dev Jennings. They're both professors in the department of strategy, entrepreneurship and Management at the University of Alberta's School of Business. They're also the first married couple to appear on creative distillation, as well as the first to enjoy wine. On the podcast, Dev discusses his latest research currently in progress that examines how science based products and processes get passed along to corporations or to startup entrepreneurs, from corporations to be developed for real world application. His paper in progress looks at this through the lens of the Alberta tar sands industry devs findings thus far, which involve not just corporate scientific community interactions, but also observations about gender roles and AI are fascinating, as is the fact that an effort to commercialize a product is at the center of devs project. This action research is another creative distillation. First enjoy and cheers. Welcome to Jeff York 1:41 Creative distillation, where we distill entrepreneurship research into actionable insights. I am your host, Jeff York, Faculty Director at the Deming Center for Entrepreneurship at the Leeds School of Business, joined by Brad Werner and Jeff, it Brad 1:52 is great to be with you today. Jeff York 1:54 It is great to be here with you, Brad. We're still on the lovely front porch of the what is it? Where are we? Joe Mission House. Thank you, Joel. Joel is our official memory of the cast. He's actually not drinking anything right now, and we just wrapped up an awesome discussion with Jennifer Jennings, professor at University of Alberta School of Business. And we're now going to talk to her husband, Dev Jennings, about one of his projects, also, professor at the University, Alberta School of Business, and both of them are here for the reversing the arrow conference. And if you want to know what the heck that is, you got to rewind two episodes and find one that says. Robert Eberhart, you're all about it, how it came to be and why it's such a fascinating thing. Dev and Jen, welcome again. Thanks. Time for us to select the new beverage we enjoyed. Oh yeah. Now, Deb, you said you were a beer guy, so, yeah, easier for me, yeah. So what we got, these are all from, probably from Boulder, or at least with that 15 minute dry bulb, we've got new terrain brewing. Pillow drops, Colorado Pilsner. Dev Jennings 2:55 Okay, it's a pilsner. Now, all right, let's Jeff York 2:56 try the pillow drops, okay, we all get a taste of that. And whippery, Jeff, this is new terrain. Okay, where are they? I can't remember where they are, Colorado. Yes, they're in Colorado. Dev Jennings 3:07 It's in a lavendery Purple, yes. So I guess that's the whole Oh, it is featuring a skier. Jeff York 3:13 There's that. Jennifer Jennings 3:14 There you go, Jeff York 3:16 Jen, You stick with the Prosecco. Are you? Oh, we might be out of Prosecco. I don't know. So the new terrain. Yeah, you're welcome. All right. Cheers, everybody. Hey, cheers. I forgot producer, Joe Oh no, yeah, Joel, we just gave him the whole bottle of deviate. You don't know what that is. You got rewind to the last episode, sure. So what do you guys think pillow drops? Dev Jennings 3:39 Yeah. Brad 3:40 Dov, you liking us, Dev Jennings 3:41 yeah, but I don't think I'd be using it just before hitting my pillow. I'd probably have a few of these. Yeah, Brad 3:46 oh yeah, it's Dev Jennings 3:47 pretty nice. Jeff York 3:48 It says pillows of paradise. Okay, now we're talking much like the voluminous foamy cup on a freshly poured Pilsner pillow. Drops are the highly technical ski lines that take you flying through boulder fields capped with mounds of fresh, fluffy powder. This pills has a dry biscuitty malt backbone, delicate floral aromas balanced by some bitterness and a crisp fit. They Dev Jennings 4:09 actually think you should take those pillows. Jeff York 4:18 Hops, traditional Pilsner hop, one of our favorite German pills, as a lot of those hops in it, I could drink this all day. Jeff, yeah, we're gonna have to seek this out. Sorry, new terrain. We're gonna, we're gonna look up where you guys are, find you and come to your name too, right? Yeah, this is delicious. Goes Jennifer Jennings 4:33 with the whole skiing vibe. Yes, yeah. We're Jeff York 4:35 here on the ski line. Dev Jennings 4:36 When I least live in Palo Alto, we would go and go to this little German brewery place and drink a version of like a stop and type beer. Okay, wait, Brad 4:45 so Palo Alto, tell us about your experience there. Oh, I Dev Jennings 4:48 went to grad school, got a master's in a PhD in the Bay Area, right? Yeah, yeah. So I was just living with a bunch of engineers. And how'd you like they were, that's as Brad 4:58 a as a scholar. So that type of background actually could be really Dev Jennings 5:02 cool. It was great. It was a wonderful era, the late, you know, the 80s was wonderful, very interdisciplinary. You could go across departments and hang and do I think my, my extra love of working with science really was from then, right? And that, you know, one of the reasons I chose the paper today is because I've always had a big interest in sustainability. I used to work for the Forest Service many years back, and just before I went to grad school, I was working for the Forest Service. And my first couple summers during grad school, I worked for the Forest Service, and yeah, and so I was in Palo Alto, and there were engineers Forest Service. I was working in the biz school. I was working in sociology. The Stanford biz school paid for my education, right? Yeah, so it was a wonderful place, wonderful place, and still is a wonderful place. I don't even think I can get in anymore, but I was lucky to be in that and I probably couldn't get in. Now, that's Jeff York 5:57 true every school I went to. I was lucky. I went there. I did wouldn't work out today, yeah, so I just remembered, actually, new terrain. I've actually been there, really, it's in golden and it's at the base of a mountain biking trail. So if you're ever in Golden Colorado, there's some really good mountain biking trails right by where this brewery is up. And you it's a massive, hellish climb, like every, every mountain bike ride in Colorado starts like this. It's gonna be a massive, hellish climb, and that's the star of the ride. But then you end up back at the brewery, and it's an awesome, awesome Brad 6:31 I think we need to do a podcast there. Just, yeah, we Jeff York 6:33 definitely need to get down there. It's beautiful. It's right in the shade of a mountain there. So I feel bad I wasn't properly selling new terrain. Great place. Yeah, the beer is great. So, so Dev, we're gonna talk about paper with a sustainability bent to it. Today. We Dev Jennings 6:45 are, we are sustainability and entrepreneurship. So, you know, Jennifer and I, as she mentioned, and in the prior podcast, we do a lot of work together. We're still doing work, including a work on women's entrepreneurs with Jessica Santana, right, right? Yeah, that looks at sort of their discourse in failure. So that's kind of over there. But I've always been super serious about how science based products and processes get passed over to corporations or to startup entrepreneurs, from corporations to be developed for the street or, you know, for use for us, especially for sustainability, right? So in Alberta, we have the oil sands, yeah, and the oil sands is a massive operation, and it has all the majors there, right, working hard on what's called the oil sand, which is bitumen. It's really heavy oil that you know you have to process. It's like, full of sand, and so you have to put it through thing called sag D, essentially, really high heat, with lots of water and chem processing to separate it, right? And then you can distill it off, but then you've got a lot of toxins. And, you know, you can imagine. So there's been Brad 7:49 a big political discussion in the United States transferring the oil sands, or the oil from the oil sands, down through the US to Houston, right, Dev Jennings 7:56 right? And so there's been a lot of backlash, but then there's been a lot of tech development on trying to improve it, right? And so I've been working at when I was at my former job, I used to work on water management. I was between solder and water management and Institute of resource environment as an associate. And then I at Alberta, I became involved in doing future energy systems. And my part in future energy systems was really around water, because I knew about that. And so water and wetland, especially remediation, after you've got these heavy toxins, you know, that have been used in processing. Well, what? What do you do? And we so we have a lot of science guys on campus and in Alberta and around the world, and also down in Texas especially, and some in Colorado, frankly. Who are, you know, piled in with lots of processes, lots of products to try to do cleanup. But it's a very interesting thing. You know, when you're in the biz school, you tend to, oh, you're a startup entrepreneur, you know, you'll go grab a product process, you might do an app. But when you're over, over in the science or engineering departments, as you know, Jeff, you know, you've kind of got this lineage of science or development of that product or that process, and you have to really leverage off that right? And then you go, Okay, well, we're going to do oil sands, but we got to do this chemistry, that chemistry, that chemistry, that product, that product, and each of these products is based off that science. Then it gets developed, and then it gets chucked over the fence magically to a corporation or to an entrepreneur, and then you just hope, you know, when you chuck it over the fans to shell, or you chuck it over to Exxon Mobil, or you chuck it over to sin crude, they pick it up, yeah, Brad 9:29 through tech transfer, yeah, yeah, but also literally Dev Jennings 9:33 through patent license, right? Tech trend, yeah, all versions, but not Brad 9:36 a lab. It gets there's some sort of IP and now it's on the shelf, yeah, but Dev Jennings 9:41 it's, you sit in the same room. No, it comes out of lab, but you sat in the same room with the development process, with these guys. I mean, you know, you're from Exxon, and you're looking at it, and they're there, and they're literally, you're, you're at the end process, and they're in the same room, some of the same guys, and there's now, like, Okay, now you take it and. Literally chucking. It's literally a handoff. And the thing is, I'm not gonna, I don't I want to be a scientist. I want to develop the next one, Brad 10:09 right? Actually, to continue the development of that technology, to commercialize, Dev Jennings 10:14 you do and you don't. You need the grad students who were on that particular product to maybe consult or be licensed, but then you'll hand it over to somebody else who has been trained in chemistry or trained in this to take care of that. Often they're on the corporate side, not on that side, right? And so there's this handoff, chuck it over. And you hope the handoff is smooth, like in a relay race, but it's chucked over the fence or, you know, they receive it. Well, yeah. And so there's, there's quite a there's a big way, back when I was a grad student, there used to be this thing was developed called the Integrated circuits lab on campus, which was trying to, you know, bridge between business community like the biz school, and I was between social and business. And so I was design and that, and tried to make an integrated statement. And it failed, right? Right? Because there weren't incentives. It almost does, right? So the question then is, how do you actually put incentives in there to keep the science guys engaged, to work with the corporation, and at the same time make sure that when you get down the road, the community isn't the one that's like holding the bag interesting, right? I mean, that's really the question. How do you do it? So you get all three players so that you can improve that the handoff process? Yeah, right. And it's not a simple one, and there are lots of answers that people work on. But I thought, well, you know, this is something that seems super important to me, then people want to know Jeff York 11:34 the answer. Was a critical question. So important everyone. Essentially, you actually seem like all the technologies so, so not even technologies where there might be negative externalities the community and things like that, but things that like, where there's a positive externality that we desperately need to be commercialized as quickly as possible. So I'm thinking a lot about like down the road, near new terrain, actually, in Golden Colorado, is the national Noble Energy Lab, right? And I had a thing where I would work with them to have people bring technologies to my class in renewable energy and just introduce the technology, and then the students would try to figure out how to commercialize it over a period of semester Exactly. And of course, nobody ever did at all. But, gosh, I'm not trying to throw them under the fence. They do a lot of things besides this. I mean, they do amazing job of a lot of commercialization efforts, and then the lab is very complex. There's a lot, but I was kind of thinking to myself, like, wow. Like they would even spend time on this, like, having these students, like, just kind of screw around, shows how hard this process is, where they're willing just to try a lot of different things. Yeah, Dev Jennings 12:35 and this is the National Laboratory. That's a national laboratory, so, yeah, Jeff York 12:40 they're clueless. I mean, yeah, sure, we'll do this one actually. That's not physically possible, but sure try. So Dev Jennings 12:47 anyway, yeah. So my engineer buddies, you know, in grad school, I would talk to them about that, and also UBC, I observed things. So we did this one project just on the side, but it is relevant with the City of Richmond and British Columbia in Vancouver itself. And it was called the 50 year out planning. And we had, like, a simulation that was done 50 years out. And I put in the organization data other people, and you ran with city council, so you had the mayor there, yeah, and, you know? And they sat and they looked and they and then they made a policy for 10 years out, 20 years out, 30 years out, right? And then they said, well, now we're gonna commit to doing what was called the greening of Richmond. We're gonna put riparian edges. We're gonna do a bit on this, on the subway, of this, this change, this change in LED lights. And then three years later, one thing out of 20 was done. Yep, right. So, pattern, right? And so I always worried about that. I'm like, Oh, how do we, how do we actually get engagement, like you said, with the students, in a way that improves it? So the idea was in this paper, and with the science guys, is, okay, how can you know you've made a difference, and how do you actually get that engagement? So we said, Okay, let's take these products where that you want to sell. In this case, biochars are, like, super common. They're the standard product that everybody uses, like your bread of water filter. Yeah. Water Filter, you know, or any pool filter, you know, has charcoal in it, right? And so you activate that in a particular way, and that's like the standard private then you have a novel product. And we looked at a couple, like, one is this thing called chicken feathers. But you when you, interestingly, when you take chicken feathers off the farm and you put it through heavy acids, and you can cart it, just like your fingernail, into little serrated things, just like Chautauqua up there, where you've got the flat iron. Just run it right across the flat irons, and it pulls the colloidal mixture off. So it's exactly like that. So you can use that, or you can use nano solar and the little nano solar chips that take the light and then they refract it and do the chlorination type process like that. So there are three types, and we were, we looked at three types, and we ended up in what the paper in two types. So, you know, the chicken feathers was like, the more novel and versus more standard. And you're like, Okay, how can we induce people to get the more novel product? And in back it right? Of these three players, the core. Operations, these science guys are pushing it and then the community. Okay, so the real problem is the science says that the chicken feathers just doesn't process oil sands process water toxins as well as chicken feathers, like most novel products, right? So science is saying we want to develop this is new, but it's not quite as efficient yet, you know? Because that's how lots of early stage products are, you know, right? And so it says that. And it says also, the wetland design needs to be tweaked so there's more circulation of water. So, but still, let's back that. And so the standard science guys are saying that, right? And so it's cool, the business guys are saying, well, no, you know, no, right? It's too expensive for all that time is and also, you know, do we need that in the community? Folks are going, Hey, are you gonna ask us anything? Right? Because normally you ask those two and then you go to a town hall or a community consultation, and you present, this is the best science. And, yeah, right. And then people put stickies on this, and they go, maybe we'll do that. And then you walk out and you do what you were going to do, right? So we, Jennifer Jennings 16:08 at one point, we came up with the cutesy, academic part of the title that goes before the colon. We thought a really cute one would be, would you like your waist charred or feathered? Sorry, I yeah, Dev Jennings 16:23 that title has not been lost yet. I'm Jeff York 16:26 sure the reviewers Dev Jennings 16:29 will sort of around that title. Jeff York 16:32 This always happens when I have a cool title, like, you get to the like, the last round, I just don't like the title. I'm like, Dude, this was Jennifer Jennings 16:39 so fun. We love doing that. Yes. Isn't that fun? Dev Jennings 16:41 I'm good with that. We haven't lost it, which is good. I'm glad you think it's great, Brad, because that matters. Brad 16:50 Here's the other thing is, I've been helping scientists for a lot of my life, commercialized technology, yeah? Well, then Dev Jennings 16:56 you recognize, yeah. I mean, I'm I love this, right? Yeah, so, so. But then, based on my priors, how do I actually know that you can make a difference? So we chose this version of CO design. My buddy at IDEO had been working on CO design, co design, but there are different ways of doing co design and deep co design. So, you know, we literally went to stakeholders now burrow oil sands and solicit them data. Wintered it down to people who lived there. And then we got 12 teams of one type using Town Hall, and then 12 teams doing co design. And in the CO design, they were working with people who knew the science, with a corporation, person and the group, and they were, you know, given the same spiel and the town hall versus the other. But then the CO design teams were allowed to work with the oil sand product and try to put it in a wetland and choose the chicken feathers versus the biochar. They had the choice he was, like a lot of cooks in the kitchen, though, no, just Yeah, but they're only six people, because we put each of these six person teams so that you have the intermediary was trained, and you had a corporate person. Then you had these other folks who were from oil sands, you know, community, and then they, like, said, Okay, what do we want at the end? Now we took that and then we married it with what's called lifecycle analysis. And, you know, lifecycle analysis, so you've got the immediate impact of the product, but then we're in sustainability, so you want to know the longer range impact, right course. So we took lifecycle analysis, we ran the numbers by inputting their preferences, their new product, what they cared about, and then we tweaked the lifecycle analysis to actually then look at, okay, after feedback, which of the the CO design with this product, is it actually any better? Okay, when you include these guys, and it was, it cut down on water consumption, and it cut down your trophication, it cut down on Carson. Brad 18:36 When you include these guys, what exactly does that mean? That means Dev Jennings 18:39 like, Okay, I'm going to talk to you for a half hour about what the talk today. I'm going to give you this paper. I'm going to put all the pieces on the table, just like a Lego thing. You're going to put this wetland together. Which one do you like? Then you're going to score it after based on the same scoring system as the town hall guys, and you're going to see and then we're going to rate it based on an expert inter rater over here to check it, and then we're going to go and take it, we're going to run it in life cycle analysis. And we actually went through the labs. We went to the oils, and we we measured all all the stages of the life cycle, from farm all the way through the wetland, all the way back to run the LCA with these numbers, you Jeff York 19:16 guys are doing this as your research project. Yeah, yeah. And you just partnered with this organization or this group that was trying to actually do this commercial. Dev Jennings 19:23 No, I've been working you're at the science department. So this is, actually, this is, this is you say my team of, your team is working on, this is my team of so you're Jeff York 19:32 like, kind of doing this. Like, I mean, I don't want to call it action research, but Dev Jennings 19:37 we work with three scientists, like, you know, Muhammad and his teams and stuff, yeah, yeah, yeah, they're great. So Jeff York 19:43 okay, yeah, to clarify that for listeners, because you know now we talk about, we're not usually talking actually doing something. No, no, it's Brad 19:51 like an embedded reporter. Yeah, right, yeah. No, we're embedded with, with the with a developmental team, or the scientist scientific team. Yeah, we Dev Jennings 19:58 are cool. Do. Yeah, part of the science development, yeah, that's awesome team, right? Brad 20:04 That's a first for us. Jeff, actually, oh, two. First Jeff York 20:09 number one, we drank wine. Yeah, Brad 20:12 married couple, but actually embedded with a scientist or embedded, right? That are developing a well. So Jeff York 20:19 I've talked about some work where I was, like, embedded and part of the steering committee for something I was studying. This is different. Brad 20:26 This is totally different. Not Dev Jennings 20:30 about you, Jeff. This is wind studies. No, your studies on wind and things are like, you know, various, various renewables are like, brilliance. No, that AMJ paper is low. There we go. But Jeff York 20:48 I had nothing to do with wind energy. Like, I like, literally going afterwards and like talking to people, like, what do you think about wind energy? You were there? Like, I remember when they protested because it was going to give us all brain cancer. Oh, wow. Okay, that's interesting. But you're actually, like, trying to help commercialize a project, right? Yeah? And you're a part of the project. That's Dev Jennings 21:10 fascinating, yeah? And so the improvement part now, the second stage of the project, which we just ran the experiment on and are about to run in the field, is to take those products, and then we've got in the chucking over the fence. One of the main things that you do with like the novel product at this stage, at least in that part of the world, even with oil sands, is you can give it to corporate and they can run it, but sometimes they'll want to use seed funding and panel funding and let somebody else do the in between. Stage, right? Yeah, sure. Of like, pre beta, beta, like, you know, developments, are Brad 21:42 they funding the research? Corporates Dev Jennings 21:43 are funding there? Are they funding the research the future energy systems is based on the carbon tax and others that gave the big, grand funding. So the big, the overall grant funding is 75 mil for a bunch of oil sands related projects like this is one of 11, okay, of that, and so that came from a big fund based on carbon taxes and a bunch of things. So yes and no, right? Yeah, but I wanted to be clear, no, no, because that's important, as you say, it is important, right as to what your affiliation is, right stuff. So, but again, you guys care a lot about the positive and negative side of, you know, helping corporations. So, you know, I work in a business school because, and not a social department, environmental department, because people need jobs, and people need work, and you got to make changes that work for, you know, everybody can embrace the environment. Jeff York 22:36 I mean, things are actually gonna I mean, yeah, so I'm often trying to convince, like, Doc students over in environmental studies to come over and get this. They're great students too. They're brilliant, like, we also masters of environment programs and and they always want to stay in the Environmental Studies. I'm just like, they feel like they're selling out. And I'm just like, but guys like, if you're just talking to people who already believe what you're saying. There's no change. Aren't going to make any ability to actually implement that. Yeah, I don't know. I just, I mean, obviously I'm completely biased, but I think you can make a real difference in a business school. And I think, you know, unfortunately, we're at the point now where only solutions are going to possibly emerge, any kind of relevant time for environmental problems, are gonna have to come through business, yeah, Brad 23:23 and you have to make an economic case for it. You can't preach to someone to say, just change this because it's better for the environment. I wish this wasn't true, right? But just but this is the reality, right? But if you can show an economic reason why to change, that makes the discussion much easier. Dev Jennings 23:38 Yeah. I mean, if there are a lot of things you need to do, all of us need to do a bunch of things. But for me, I you know you can't, you can't leave organizations out of the equation. No, it's like there's just, especially in North America and the states in the world, the organizations make a huge difference. And so if you can't get those folks on side and science on side, the ability to change things is just Brad 24:01 drama, outsourcing research, though, yeah, no, they Dev Jennings 24:05 do core research and they outsource it. You know, corporations are, it depends on the company. Honestly, shells, shells unit is different than Exxon Mobil's unit is different than syncrudes. It's yeah, and where they actually locate their expertise in Europe versus North America? Why not? All vary quite a lot, but there's a lot of expertise around so trying to figure out how to drop it into any project is one of the things in this one like, what level of expertise is getting dropped in from these guys? And how do you do it so that it doesn't derail it, but it's incorporated, right? So Jeff York 24:37 you guys are doing these, these two dual process. You basically have these two product approaches to Dev Jennings 24:41 this problem, yeah, on two types of wetlands. The two wetlands are one, like, has all the bells and whistles, and one is much more naturally constructed. And there are two other intermediate types that are both in the actual experiment, right? But we just think of these things, yeah, yeah. We're horse racing, yeah, yeah. And Jeff York 24:57 so what'd you find, like, what's the, what's the, what Dev Jennings 24:59 do you think. Found that the nationally constructed one with chicken feathers over the course of one month through multiple rounds, will still be good enough, okay, to actually deal with it, but over the life cycle analysis, it actually is better. Wow. So yeah, long term, so short term, that's long term versus short, but also cheap versus expensive, but also big science versus small size, right? So Jeff York 25:21 that's a lot of challenges to stand up. Yeah, corporations is like, we're worried about the quarterly earnings. And so you make the cheapest possible decision, that actually is not the cheapest possible decision, not even just from a from an environmental perspective or a social perspective, but from the economic perspective. But you still make that choice, because all we care about is quarterly earnings. And Brad 25:39 actually, quarterly earnings disclosures are part of the problem, right? Exactly. I Jeff York 25:43 mean absolutely. So yeah. Dev Jennings 25:44 And so the next stage is, is you know how to pitch these things? Right? Yes. And so we're taking we are in our pitch experiment. We ran a male versus a female scientist running the novel versus standard product in front of government panels in the experiment. And we had first experiment was just vignette experiment with, you know, folks judging it. And the second experiment is actually with this, the seed funders both a VC type and a government type, in order to see which, you know, which of that two by two pairing. Okay, so does a woman. Normally, in research, we know that women, on average, on novel products, will be penalized more than men. Usually put a lead male, you know, senior scientist. And we also did better known versus less well known person. And that came up that way so far in the results. We also did a tweak on it. We ran one condition where all of it was done by the human, the other condition, the script and the slides and everything were all done by AI. And we want to see whether, you know, AI and Jen, you know, would be able to help us in some way to on sustainability, to get these things across more clearly, and whether. And then we had to reveal, like, Okay, guess what? All this was done by AI, by this person, not by that person, right? And so then we tried to reveal and then we wanted to see whether the female scientists would be punished more. And we can't, I can't tell you the results, because they're only, like three weeks old. But when you look at the simple results without controls, it looks like there's it's not what you expect. People were thinking, you see a lot more spread of the results. Certainly some people did punish the female scientist more, but not routinely, no, no. And people were like, okay, there either there's a big deal like NIH in the States is said, no, no, no use of no use of AI and grant writing or this or that. But then plus, and science reported that 25% ish of folks were using it, are using and constructing their grants and stuff. So I'm like, Well, you got to be able to world Right, right. What's the punishment factor after owning up, right? Or if somebody reveals it on panel, or somebody finds out, right? So we wanted to look at the punishment factor too, for men versus women. Interested scientists to see whether we could get over that, to get to better, you know, rollouts So, but Jeff York 28:06 there wasn't a consistent effect of punishing women. No, no, no, AI, but there was for the novel technology. Dev Jennings 28:14 But the novel technology seems like it was getting it was like, Well, do we really? It's almost people, like, there are too many credibility factors, right? With the AI, plus the novel, plus that, are you? Are you using it to, like, hype me sell. That's why, when you mentioned Matt and stuff on the hype thing, I've been, you know, we've been thinking about, you know, well, first Brad 28:36 of all, I know breakfast and Okay, so the first finding is going to get you Brexit. The second finding, though, I think is really interesting because, like, groups like the NIH, I don't understand why they don't kind of recognize what reality is and just say, okay, findings are findings. Why is there this bias against AI, yeah, we don't even that's Jennifer Jennings 29:01 a whole other podcast. Dev Jennings 29:04 No, you know, those are just tools, though, that they are tools, but they're very they're very rapidly evolving, complex tools that people are trying to figure out how to use. So, and I just finished teaching a data analytics class in the winter to the MBAs undergrad combined and and in it we, for instance, I ran a quant and a qual problem. So the quant problem was, you know, creating four data sets and a quadrant sort of scenario based on what was a regression equation of a startup's earnings. Okay? And I messed with the I messed with the intercept, so I knew the equation was wrong. And then I got him to run it in, you know, chat, GPT, four plus. And I got him to run it, I guess I shouldn't mention all these things, but, yeah, somewhere else, Gemini, but, but anyway, I ran Gemini. And the thing is, I mean, a wants to please, so it generated the four, and the approximations were pretty close. So the upside it would generate, but it wouldn't generate like, oh. This is incorrect. It wants to say this is correct within so the real answer was blah. So it became close, but it wouldn't say this is actually Brad 30:09 so if you understand the bias, though, if Dev Jennings 30:12 you understand but then, when I ran on the sustainability thing, like, Okay, on this particular sustainability problem with water, with this toxin, here's what it is, okay, yeah, what would you do in terms of policy with this? It generated nice lists of policies, and then you could drill down and drill down, and it was nowhere close to what the policy was in the region by the end of Brad 30:39 it. So was and so, I mean, how does this happen? Well, Dev Jennings 30:42 some people call it hallucinating, but I would just call it, you know, that's just reached the logical conclusion. But it was no, but it was like, you know, a gap between what was there. So, so you got to be super careful. And so prompting, we were very careful with our prompts, sure, and we didn't use an avatar, you know, we didn't use that we we'd already pre tested so we could equalize across the four cells. So people couldn't actually tell very well between the senior female this that, you know, there was enough range in the values that it wasn't, it was only when AI was put in where you saw differences. So So you have to be very careful, and that's just the stage we're at where people need to experiment a lot and be super cautious. But as you say, you know that's just where it is, and it's better to be transparent and own up. Yeah, totally agree. And Brad 31:35 if we're sitting here 100 years from now, who knows? Yeah, right. I mean it's gonna be really yeah, Dev Jennings 31:43 oh, I agree with that, right? It's Jeff York 31:45 yeah. It reminds me, yeah. I think it's like a more extreme version of the.com boom that is actually going to have far more reaching impacts than and now I'm saying this on the podcast. I guess I'll have to stick with it. But like, I just remember, like, the whole creation No, not the creation there. And, I mean, when Al Gore created the internet for us, but I just remember the whole, like, I remember Amazon being founded, you know, Google, hey, Dev Jennings 32:09 I used to use ARPANET files across campus, Jeff York 32:16 Brent, like, pitch for Google. Like, I mean, for, like, Angel rounds, like, and everybody's just like, ah, we got Alta Vista. What's that stupid? It's a no search engine. And now look where. I mean, so it's like, the implications of AI are much greater, though, and I think the unexpected, both positive and negative aspects of it, are going to be revealed in time. Dev Jennings 32:38 You know? What really helps is living with a technophile. She keeps me super honest. No no, because honestly, like, we've had this discussion with our entrepreneurship grad students and whatnot, like, how do we run exams now be given that AI and chatgpt are there and whatnot. And so she was one of the first ones to bring it up, being worried about technology. So you need that sort of positive right view. Let's embrace it view, along with a really skeptical like, you know, where are we view? And you need to embrace that as a person or have somebody else, but Brad 33:13 if you're sophisticated user, though you understand the downsides. What could come out of your search results or your your query? Yeah, Dev Jennings 33:21 but I still think, like on, you know, CEO or, man, you know how, how VCs will always set up teams. I mean, if you're setting up a team now in Gen AI or whatnot, you'd want somebody super critical, super skeptical, on your top management team of, honestly, you know, in the same way, at one point, you'd have a lawyer in the room on blah, blah, blah. Now you want somebody who's like an ultra skeptic, who knows things in order to keep you honest. I think I Jeff York 33:47 love that. Actually, that's cool. Yeah. Oh, back the paper, though. Actually, what do you think? Like, what is your main takeaway from this paper? Like, for people that are trying to address these problems, there's so many people that have great intentions, they're struggling so hard. Yes, Dev Jennings 34:00 I know. I just think that, you know, we all one of the takeaways is it's, it's a long journey, and just keep going at it and work hard with the folks as best you can, like scientists, care a lot, you know, the community folks, the corporations. They care a lot, if you can get them together, engaged in a way to co design, where people can figure out an incentive. Yeah, they'll do it. They will do it. They will do it. You're figuring out Jeff York 34:26 a collective, or at least different incentive. This is much like entrepreneurship, where you talk about letting people see what they want to see in the project, so they can create their own incentive in some ways, right? At least, that's why I always think about it, letting people see what they want to see and not be like, No, it's not about this. Oh, I just made you not want to do and Dev Jennings 34:43 it's not all Kumbaya, because you have a bunch of folks who in the in among the stakeholders. When we measure the, you know, we had extremes here in polls. We had wedge politics, right? And when we ran the measures to try to pull things together, we deliberately truncated the tails. Yeah, we. Literally took only the 75% Oh, centrist on the weights and tails, not the extremes. Yeah. And, you know, somebody's got to make decisions that say the tails have to be out of this part of the solution if we're going to get any modal stuff that works for the majority of folks. No, I'm just saying that as a takeaway, and it goes down to into the field, right down to the science itself, because that those things become wedged up later on, in a much worse fashion. Okay, right? Honestly. And so if you can avoid the wedging early, you can avoid more wedging later, okay, is one of the other takeaways, very Jeff York 35:41 cool. So a paper, we can't point you to it. It's not published yet. This Dev Jennings 35:45 project has been going five and a half, six years. That's quick in this world. It's gonna go a couple more years. Yeah, Jeff York 35:55 I think we're gonna So, yeah, well, I'll talk about that interview someone who presented their paper at this conference. Not this conference, but the next one we're doing, the Global Entrepreneurship innovation conference. Kirk, thank you. But anyway, they presented it, the last one that happened, which was well before the pandemic, and they just got published it. That's, that's pretty quick. Yeah. They were like, Oh, it took forever. Like, Dev Jennings 36:19 no, yeah. And there are many other things that could come out of this, but, yeah, you only have so much capacity. That's true. Jeff York 36:25 Yeah. Well, Dev and Jen, thank you again for joining us. You're enjoying your visit to Colorado. Absolutely. We're thrilled to have you here. Yeah, I think we better go on over to the barbecue and grab some food before it's all gone. Thanks everybody. Jeff York, Faculty Director at the Leeds School of Business dipping Center for Entrepreneurship. I Brad 36:43 am Brad Werner, and it was great to have both of you on thanks. Jeff York 36:46 A pleasure. Yeah. Dona L 36:54 This episode of creative distillation was recorded on the front porch of the Chautauqua mission house at the reversing the arrow conference held in Boulder in June 2024 devs paper is yet to be titled and is still in progress. Learn more about dev Jennings on his faculty page at the University of Alberta website. We'd love to hear your feedback and ideas, email us at cdpodcast@colorado.edu and please be sure to Subscribe to Creative distillation wherever you get your podcasts. The creative distillation podcast is made possible by the Deming Center for Entrepreneurship at the University of Colorado, Boulder's Leeds School of Business. For more information, please visit deming.colorado.edu that's D, E, M, I, N, G, and click the creative distillation link. Creative distillation is produced by Joel Davis at analog digital arts. Our theme music is whiskey before breakfast, performed by your humble host, Brad and Jeff. Thanks for listening. We'll see you back here next week for the next round of creative distillation.