Sruthi Pisipati - Why Engineering Foundations Still Matter in the Age of Generative AI

Welcome everyone to the Conversations on Applied AI podcast. Today we're speaking with Sruthi Pisipati. Shruthi is a software engineer turned data scientist with more than eleven years of experience across the consulting, IT, insurance, manufacturing, and medical device industries. She holds a Master's in Analytics from Georgia State University and a Bachelor's in Computer Science.

In her current role as a principal data scientist at Boston Scientific, Shruthi builds AI-powered solutions that drive innovation in healthcare and manufacturing. Her expertise spans computer vision, generative AI, time series forecasting, and uncertainty quantification, with a focus on deploying trustworthy, scalable models in high-impact environments.

Shruthi is a recognized speaker and thought leader in the AI space, with talks focused on optimization techniques, responsible AI practices, and real-world applications. Thank you, Shruthi, for taking the time to be on our podcast today. 

If you are interested in learning how AI is being applied across multiple industries, join us at a future AppliedAI Monthly meetup and help support us as we plan future Emerging Technologies North non-profit events!

• Applied AI
• AppliedAI Meetup
• Emerging Technologies North

Resources and Topics Mentioned in this Episode

1. "Attention Is All You Need" — The transformer architecture paper by Vaswani et al. (2017)
   https://arxiv.org/abs/1706.03762
2. ChatGPT — OpenAI's conversational AI
   https://chatgpt.com
3. Claude Code — Anthropic's AI coding assistant
   https://www.anthropic.com/claude-code
4. Chris Gregory's LinkedIn article: "Code is cheap now, but software isn't". -https://www.chrisgregori.dev/opinion/code-is-cheap-now-software-isnt
5. Hugging Face — Open-source model hub (visual-language models, etc.)
   https://huggingface.co
6. Conformal prediction / Uncertainty quantification — A statistical framework for calibrated prediction intervals. A good introductory resource:
   https://en.wikipedia.org/wiki/Conformal_prediction
7. Time series foundation models — This is an emerging topic, not a single resource. Shruthi's upcoming talk compares these transformer-based models against classical approaches like ARIMA. For an overview:
   https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average

Bonus links from the episode:

• Applied AI (podcast community): https://appliedai.mn
• Tesseract (OCR): https://github.com/tesseract-ocr/tesseract
• Google Vision: https://cloud.google.com/vision
• Amazon Mechanical Turk: https://www.mturk.com
• Boston Scientific: https://www.bostonscientific.com
• Novelis: https://www.novelis.com
• Accenture: https://www.accenture.com
• Georgia State University: https://www.gsu.edu
• LexisNexis Risk Solutions: https://risk.lexisnexis.com

Transcript

Copy Transcript Copied

[00:00:00] Shruthi: You can generate one-off tools really quick these days. But if you are thinking of developing full-fledged software or developing tools on which your most important decisions rely on, you should think of the engineering principles on which these tools sit. You can definitely get faster to the deadline with those one-off tools which are helping you analyze faster.

[00:00:23] But when you think about productionizing something, you should have that engineering foundations in place. 

[00:00:30] AI Speaker: Welcome to the Conversations on Applied AI podcast, where Justin Grammens and the team at Emerging Technologies North talk with experts in the fields of artificial intelligence and deep learning.

[00:00:41] In each episode, we cut through the hype and dive into how these technologies are being applied to real world problems today. We hope that you find this episode educational and applicable to your industry, and connect with us to learn more about our organization at appliedai.mn. Enjoy.[00:01:00] 

[00:01:01] Justin: Welcome everyone to the Conversations on Applied AI podcast. Today we're speaking with Shruthi Pisapady. Shruthi is a software engineer turned data scientist with more than eleven years of experience across the consulting, IT, insurance, manufacturing and medical device industries. She holds a Master's in Analytics from Georgia State University and a Bachelor's in Computer Science.

[00:01:20] Her current role as a principal data scientist at Boston Scientific, Shruthi builds AI-powered solutions that drive innovation in healthcare and manufacturing. Her expertise spans computer vision, generative AI, time series forecasting and uncertainty quantification, with a focus on deploying trustworthy, scalable models in high impact environments.

[00:01:37] Shruthi is a recognized speaker and thought leader in the AI space, where her talks focus on optimization techniques, responsible AI practices and real world applications. Thank you, Shruthi, for taking the time to be on our podcast today. 

[00:01:49] Shruthi: Hey, thank you so much, uh, Justin. I'm really honored to be speaking with you today on Conversations on Applied AI and share my perspective on what my journey has been so [00:02:00] far in the AI space.

[00:02:01] Thank you so much. 

[00:02:01] Justin: Yeah, for sure. This is gonna be super fun. I think it'd be great for you just to give a sort of a quick summary with regards to, you know, what was your path? What was your trajectory of your career to, to get you to where you are? 

[00:02:11] Shruthi: Sure, yeah. I'm a software engineer turned data scientist with thirteen years of experience, eleven to thirteen years of experience in this space, which includes software engineering, AI and, and then across various industries like consulting, insurance, manufacturing and now medical devices.

[00:02:27] And when I look back and reflect on my journey, it was never a straight path or it was never a straight plan to be in AI. Rather, it was shaped by my curiosity and course corrections than a straight plan. I started my career in India after earning my bachelor's degree in computer science engineering from a reputed university.

[00:02:45] Interestingly, machine learning was offered as a course in my final year, which was twenty twelve, but at that time, very few people were interested. The industry buzz back then was all about Java, SAP and Oracle, all the enterprise [00:03:00] systems. I knew I loved programming, but I didn't yet have the maturity or exposure to understand where the field was heading.

[00:03:08] And access to the right learning resources wasn't as easy as it is today, and exploring emerging areas like machine learning at the time required very intentional input. So it was a no-brainer for me to just go ahead and pursue my interest in programming while in college in my final year, and I was fortunate enough to land two different opportunities with respect to programming.

[00:03:28] I chose one I started my career as a junior Java developer, which was the buzz around that time. I did that for three months, but that was away from my hometown, and I was looking for opportunities to be back to my hometown, and I did get an opportunity at Accenture as a SAP ABAP developer. So from the Java world, I had to, again, switch back and start from ground in the SAP ABAP world.

[00:03:53] With Accenture, I, I stayed there for four and a half years, and when I look back, that was the [00:04:00] decision which turned out to be foundational with respect to the software engineering thinking mindset which I've got and the communication skills I have picked up. So over those four and a half years at Accenture, I experienced rapid growth.

[00:04:13] I received two promotions, worked on complex large-scale enterprise projects, and gained international exposure through a client assignment in Paris. That phase really taught me how big software systems are designed and executed, the importance of work ethics, effective communication, and the discipline required to deliver reliable solutions at scale.

[00:04:34] I also completed an executive management certification during that time, which gave me strong business and leadership perspective as well. However, despite doing well professionally, I slowly began to feel that my learning curve was flattening I was repeating similar patterns, and I found myself craving deeper technical and intellectual challenges.

[00:04:56] That curiosity led me back to something that I'd [00:05:00] briefly encountered during college but never fully explored, which was machine learning and data science. And it was 2016, 2017, chronologically, if, if I have to go back and look at the years, and that was around the same time when data science was the job which was most sought out.

[00:05:16] Justin: Yeah. They said it was, like, the sexiest job you could have or something like that. I forget what all the headlines were saying. Yeah, it was funny. 

[00:05:21] Shruthi: Yeah, yeah. And around the same time, my personal life was also going through major changes. I got married, moved to United States from India. I decided to take a pause from my successful career at Accenture.

[00:05:31] I took a career break, and I decided to go back to school. Thanks to my partner, he supported me all through, through this journey. It wasn't an easy choice. New country, new into marriage, and new field, l- like, leaving my successful SAP career and then starting from ground. But with the support of my partner, I decided to take that leap.

[00:05:51] I prepared for all exams, like GRE, TOEFL, and applied to several universities who offer fall 2018 admission, and I got an admit from Georgia [00:06:00] State University. And what drew me about that course was its strong focus on real-world industry use cases, affordability, and of course, exposure to Atlanta's growing tech ecosystem.

[00:06:10] But it meant for me to relocate from Minnesota to Georgia and, uh, do a long-distance thing from my life partner, like, new into a relationship. He supported me. I trusted that decision. He trusted it, and I took that leap. I relocated from Minnesota to Georgia and pursued my master's in analytics, but that again meant that I had to start from ground zero, like switching from, from Java to SAP and now to entirely a new different world, which is machine learning data science, which needed strong, rigorous statistics.

[00:06:42] I still remember the first day when I walked into the, the class in Georgia State. All my classmates, they were already mature enough in the data science space. They had some experience in data science. They knew what Jupyter Notebook was. They knew what, what these models are, and I come from a SAP package software space.

[00:06:59] I don't know [00:07:00] what GitHub is, what Jupyter Notebook is. It was very scary, and I really dreaded learning statistics. I took some time to understand this whole new world of machine learning and data science, and quick enough, I found the rhythm, and Statistics, machine learning turned out to be my favorite sub-subject by the end of the first semester.

[00:07:20] Simultaneously, I had an opportunity to do multiple assistantships within Georgia State. Like, I worked with College of Law on few interesting NLP use cases. I worked with professor in statistics department to gain more understanding into that world. All was going great, but then I needed that industry exposure, which was my first hurdle.

[00:07:40] How do I compete with my peers who are experienced in data science, whereas I'm new to this world? I only had SAP knowledge, but not the open source world knowledge. So how do I do that? Thanks to the assistantships which I have got at, uh, Georgia State, I was exposed to cutting-edge NLP [00:08:00] problem statements, which I worked at the law college, and that gained me an internship with Munich Reinsurance America.

[00:08:07] It's a company in Princeton, New Jersey. I got into my internship there with whatever machine learning and NLP knowledge I gained at Georgia State, thanks to the coursework and the industry exposure. I landed an opportunity, so I relocate from Atlanta to Princeton, New Jersey. So- 

[00:08:24] Justin: Sure. 

[00:08:24] Shruthi: Uh Moving all 

[00:08:25] Justin: over.

[00:08:26] Shruthi: Yeah, yeah. So I, I moved there. I worked on this NLP project for, for three months. It was classification of sexual harassment cases. It's a pretty interesting problem statement. I did that, and then back to Atlanta after three months. And there, once I'm back in Atlanta, I had an opportunity to land an internship with Novelis.

[00:08:46] It's a leading aluminum manufacturing company. So that was my first ever exposure to the manufacturing industry. It was pretty interesting. And this internship, which I landed with Novelis, I still would like to see as, as the most defining [00:09:00] internship in my entire data science career so far. That's because of the blend of software engineering principles and the data science capability, which I was exposed to.

[00:09:11] The task for me at that internship was to build end-to-end data science pipelines, so I had to apply software engineering, like solid software engineering principles, and also know much more about the various kinds of machine learning algorithms and data science principles. So my task was to come up with a custom library, so design and develop a custom library.

[00:09:33] So I was the first intern onboarded into that program there, and slowly the other interns joined, and together we built this library. So once the library was up and running, the next task was to take each and every project and make sure we are adopting this library into those data science projects which were already existing.

[00:09:50] So it was productionalizing using this library. So that was a pretty eye-opening or exciting experience for me to think of, like, prototype to deployment. [00:10:00] So it taught me quite a lot in that internship. Things were fine until 2020 May, but 2020 May is when pandemic, middle of pandemic, and my role was impacted at Novelis, so I was let go.

[00:10:11] After that, I was fortunate enough to land immediately other opportunity at LexisNexis Risk Solutions They deal with a lot of insurance claims and providing big data intelligence to insurance companies. So I was there, and that was my first formal exposure to work in a highly regulated environment dealing with large-scale data and dashboards and interpretable machine learning.

[00:10:32] So within in- insurance, they don't accept any model, any shiny model. You need to have interpretable machine learning explanation there. So that role deepened my appreciation for explainability, governance, and trust, and especially in industries where AI decisions carry real consequences. So that was happening until twenty twenty December, and it was time for me to think of moving back to Minnesota to be with my partner.

[00:10:55] Uh, till that point of time, we were doing long distance. And I was looking for [00:11:00] opportunity, and I, I landed this opportunity with Boston Scientific. I joined Boston Scientific in, uh, January twenty twenty-one, where I have now completed five years. At Boston Scientific, I have had the opportunity to work on a range of applied AI initiatives, including computer vision for automating vision, sight inspection, classical machine learning solutions, time series forecasting, and more recently, generative AI.

[00:11:22] Working in this space has reinforced for me that applied AI is not just about accuracy or innovation. It's all about reliability. Interpretability is at, at its crux. So when I reflect back on my journey, what ties together is a willingness to start over again and again, to stay curious, and to build AI systems that work in the real world.

[00:11:43] So that's the reflection of what I have done so far and how I came into AI. 

[00:11:48] Justin: That's amazing. That paints a great picture of sort of your trajectory through, and I think a lot of us have taken a similar approach. I think we're all learning AI as fast as we possibly can, and the [00:12:00] curve is going more and more of a hockey stick, you know?

[00:12:02] It seems like every month things are moving faster. So to have someone say, "Well, I've been studying AI for the past twenty years," it's like that can be useful for a foundation, but things are changing so fast, especially in the world of generative AI. And so, you know, I want you to maybe talk us through a little bit because AI comes out of machine learning, deep learning, and I would say generative AI comes out of that.

[00:12:22] I, I consider it all artificial intelligence in a lot of ways. But generative AI sort of walks this line of it's not deterministic. It's a whole different paradigm. People don't realize that versus some of the statistical analysis and stuff that you're doing around forecasting. How do you distinguish both of them?

[00:12:37] How do you try and explain it to people? Maybe we can start there because I have about a hundred other questions I wanna talk to you about how AI has changed as you've seen it in the past ten years. Maybe explain and how gen AI maybe is different. 

[00:12:47] Shruthi: When I tend to talk to my colleagues or friends when they ask me about, "How do you see Everything is AI.

[00:12:53] Everything is labeled under AI right now. So what is generative AI? Is generative AI something which [00:13:00] generates, which has the capability to, to, to generate? So what people see these days is ChatGPT. Uh, it, it generates text for you, so it does, so it is generative AI. But when you just take a step back and look at the underlying technology which is running ChatGPT or the cutting-edge time series foundational models, the architecture is transformer.

[00:13:19] Attention is all you need. It's the basis of all these cutting-edge generative AI stuff, right? I know that the architecture is the same for when we talk about the generative AI capability, it could be time series or NLP or text-based. When I explain to someone, I just tell them that the architecture is same, but the way it manifests or the way it portrays to you, it's different.

[00:13:42] Justin: So rewinding back to, like, you were working on this NLP use case here with this, with these lawyers and stuff, like, how would that project be different today versus in 2018, 2019 or so when... Or how you would approach it differently? 

[00:13:54] Shruthi: I guess the, the accuracy with which we can get to the solution [00:14:00] today, it's very different than what I did in 2018.

[00:14:03] I'll give an example here. So I was working on a auto insurance claim. The problem statement is to separate a sexual harassment claim and then from a auto insurance claim, okay? I still remember this particular claim where it was mentioned that Car A kissed Car B, but that, that kissed is a word which would come under sexual harassment.

[00:14:24] So immediately, that claim was classified as sexual harassment claim. But when I read it, it was not sexual harassment claim, it was a auto insurance claim. Back then it was difficult to do that. But today, with the advancement we have in, uh, in NLP space, in generative AI, that claim would not be classified as auto insurance claim.

[00:14:43] So This is the clear advantage or clear distinction I see from what we are seeing today than what it was in twenty seventeen, twenty eighteen. 

[00:14:52] Justin: Absolutely. You reminded me of another, of an... I was working with a company, and we were basically taking invoices that were coming through, and we [00:15:00] were essentially doing OCR on them, right?

[00:15:02] Yeah. So character recognition. Yeah. And back in those early days, Google had this thing called Google Vision that you could try and, like, load in documents to try and figure out and read it. We tried that, it was okay. Then we built our own model using this thing called Tesseract, open source project, and we gave it tons and tons of invoices, and it still kept messing up on finding stuff.

[00:15:20] Mm-hmm. And at the end of the day, we actually used Mechanical Turk, actually had humans- ... that actually ended up doing the work. And now with these visual- Yeah ... language models, even just the free ones that are out on Hugging Face, like you can give it an invoice, and it is just so good. So yeah, it's just gotten so good.

[00:15:36] Like, I mean, it's coming closer and closer and even some cases surpassing what humans- Mm-hmm ... would basically figure out. 

[00:15:41] Shruthi: Yeah. If I have to divide this era, like last decade, probably I would divide it pre-attention is all you need and post-attention is all you need, because that's the groundbreaking paper which would change the landscape.

[00:15:55] Justin: Absolutely. By the time this podcast gets out, it will be over, but we're actually doing an event at St. [00:16:00] Thomas this Saturday, and it's building your own transformer. And so we're gonna use a transformer architecture and people can show up. I'm sure we'll do this again, 'cause it's-- people are very interested in sort of diving into what does this mean?

[00:16:10] 'Cause that paper was groundbreaking, but people actually, a lot of people have not actually pulled it back and tried to actually implement some of this stuff. So very cool. The other thing is, is I love your background because it, it's actually very similar to mine in a lot of ways. I came up traditional software developer, right?

[00:16:24] I got a master's in software engineering, as I think a lot of people did, and then got into data or through the Internet of Things, through sensors and real-time streaming of data, large time series stuff, and then got to a point where I tell people, "I realize there's just so much data that humans are not gonna be able to manage this."

[00:16:41] It's not, uh, even gonna work in a dashboard per se, because you're actually gonna need to have very intelligent dashboards, and we're gonna need AI techniques, what have you, to allow us to sort of do this. And as we've seen over the past year, software engineering has totally changed with things like Cloud Code, for example, this vibe coding trend that's going on, but pretty much [00:17:00] anyone can kinda like bang out actually a, a pretty reasonable prototype, right?

[00:17:04] Yeah. In fact, in some cases, I'm some completely floored at what people can sort of do. Still not adhering to software development, like best practices per se, but I would say arguably the software engineering industry is going through a huge transformation. How do you see data science? How do you see a data scientist's role changing over the next year or so, or has it changed in the past year?

[00:17:25] Do you see that being impacted as much, or how do you think AI is impacting that? 

[00:17:30] Shruthi: What I've seen, like, in the past five years on how my work has evolved, things which used to take a week or 10 days, now it just takes an hour thanks to the cloud code and other, other tools, AI tools which we have. I can really get fast to the deadline than sitting there debugging and checking stack overflow to, you know, debug the, the code.

[00:17:52] So that life cycle is, is cut short. So now I can think of more experimentation and [00:18:00] more avenues to search and make sure that I leave no stone unturned when I explore the problem statement. So right now, I could just build code by prompt and then execute it when I... w- when my thinking is active on the problem statement which I want to solve So this has really helped me to explore more problem statements than what I had done three years back.

[00:18:26] So now I'm actively working on three different projects, which probably I would have never imagined three years back if somebody would have asked me. Pursuing time series problem statements would have never been done three years back or two years back. Now I know to do it. So things are moving fast. I'm able to show value out of the work which I'm doing.

[00:18:45] But there is one caveat, though. There are these one-off tools which we generate really quick to reach to the deadline, but maybe we all should pause a bit and try to understand if the foundations on we-- on which these tools are being built, are they wobbly? [00:19:00] So the engi-- the strong engineering foundations, that is one thing which I guess the data scientists who are coming into the industry right now should never miss.

[00:19:10] I came across an article on LinkedIn. It was by Chris Gregory. I don't know him personally, but it just appeared on my LinkedIn feed, and he has written an article saying that the code is cheap now, but software isn't. And I totally resonate with what he has mentioned in this article and what he's written there.

[00:19:25] You can generate one-off tools really quick these days, but if you are thinking of developing full-fledged software or developing tools on which your most important decisions rely on, you should think of the engineering principles on which these tools sit. You can definitely get faster to the deadline with those one-off tools which are helping you analyze faster.

[00:19:49] But when you think about productionizing something, you should have that engineering foundations in place. 

[00:19:54] Justin: Yeah. So they need to understand the software or how software engineering sort of plays out. And that's one of these [00:20:00] things that I saw years ago was as I moved into Jupyter Notebooks and stuff, I don't mean that in a bad way, but like data scientists, they just work in these Jupyter Notebooks and stuff, but there's no concept on how is this actually gonna be deployed.

[00:20:10] Like, how are you gonna set up an API for this? How is this gonna actually integrate with this larger scale system? It was always like, "Hey, I'm gonna do the Jupyter Notebooks, and I'll throw it over the wall, and somebody else will deal with it." And so now I think what I'm hearing is, is... And I come back to my question to you is, what would you recommend people focus on as they're coming out of school, right?

[00:20:27] One is probably definitely good software engineering practices. The other term that I've heard come and go is really around data storytelling, is like sort of like that word. Do you, you s- you see that like a lot? People need to understand the story of what the data is trying to tell and be able to, i-in some ways do what AI can't do, which is the personal side of it.

[00:20:47] Like how would you actually present this? Yeah. Do you see that being done a lot as well? 

[00:20:51] Shruthi: You, you bring up a very good point, Justin, of, of software engineering principles and engineering principles. Engineering foundations is one thing which still s- sticks today [00:21:00] irrespective of how AI is emerging. The other thing is data storytelling or presentation or the ability to convince the stakeholders.

[00:21:08] It could be Novellus or in Munich Reinsurance America or right now in Boston Scientific. Any company I've worked for, one thing I have struggled with machine learning or data science or AI space is I build this model, it produces accurate results, but when I sit in a room with the stakeholders When I show them these accuracy numbers, it doesn't really mean anything to them.

[00:21:28] They say, "Okay, why do we trust it? What does this mean to us?" That stakeholder communication, convincing your stakeholders to trust in the process, make them part of the process, and make sure that they realize value out of this, they see what you're seeing. So the data storytelling, the communication is something which, which goes without saying that should be one of the skills which any aspiring data scientist or any AI engineer should have.

[00:21:52] If you really want to make sure that the value which you're seeing, the others who are invested in the process should also see it. And AI is, is not [00:22:00] cheap. In any company, if you want an AI solution to be deployed, you need to have stakeholders who invest in that solution, who think that is useful, who think that the ROI is there on that solution.

[00:22:11] So you need to convince the stakeholders on that ROI. What is the return on, return on investment? And if you're able to build a quick prototype and show them, "This is the result. This is how accurate it is, and this is the ROI which you can achieve within a few months." And then walk them through on what a full robust end-to-end solution would look like in phases.

[00:22:30] Justin: Yes. Yes. Exactly. Yeah, people need to see the path, and I think far too often, both in the software world and in the data science world, we get so into the solution and get into the technology that we're like, "Look at this. This is amazing." But if people don't have the context and probably, like you say, maybe that path forward of, "Okay, this is what it means to the bottom line," that's, that is definitely helpful.

[00:22:53] And then also like, yeah, here's how just our customers are going to benefit from this is certainly useful. We [00:23:00] use the terminology around here called product-minded engineering. Have the engineer understand the product they're building all the way through, and it's the same thing with data scientists too as well.

[00:23:08] Kinda like, why are you doing all of this? Yeah. Yeah. Sometimes people get so busy in the details, right? 

[00:23:13] Shruthi: And the problem with data science space is it's not like classic software engineering where you have set of rules, and you just build logic for that set of rules, and then boom, you have the output.

[00:23:23] It's not like that. With data science, machine learning, there is a lot of experimentation, iterative development involved until you reach that stage where you can say, "Okay, now this confident about this model." The stakeholders who are working with you, they should be aware of this process in AI and ML. If it is computer vision, if you're deploying a image classification model, it involves a lot of annotation.

[00:23:43] You need to work with annotators who are, like, annotating until your model reaches a stage where it understands what you are trying to tell it. So this iterative back and forth development is something which the stakeholders should understand that the time it needs, this whole [00:24:00] development process needs.

[00:24:01] Justin: Yeah. So the other thing I heard you talk a little bit about with this, as you sort of talk to stakeholders and stuff like that Is around explainability. And I think it's some of the things that I think you have spoken about in both probably at various roles and at various conferences and stuff. How does that change across industry?

[00:24:17] I'm just thinking about right now, you're in med device, right? Probably has to be highly explainable in some capacity, whereas maybe a different industry, it's not as big of a deal. Like, how, how do you see that going forward, and how have you seen it across various industries around that? 

[00:24:30] Shruthi: So it depends on what kind of algorithms you're choosing for your deployment as well.

[00:24:34] It goes without saying that within medical device and even with, within insurance, like the highly regulated spaces, you have more than one barrier, more than a couple of barriers I should say, with respect to the model which you're building. All of these are like holdout sets, which you don't get to see until that particular point of time So we have those guardrails in place from the stakeholder's perspective for them to believe that, okay, this model is good enough to pass.

[00:24:59] [00:25:00] And when we talk about explainability, reliability of a model, the stakeholders are involved in the process right from the very beginning, where they get an opportunity to come up with metrics. So metric definition happens at very early in the process. If you're talking about a gen AI solution, the stakeholders and the AI engineers, data scientists, we just sit together and talk about, okay, when do we say that this particular model is good enough to cross barrier one?

[00:25:26] What do those metrics look like? And how do we design those metrics? So that discussion starts at the very beginning, and I guess right now, even if it's a highly regulated space or not, more often than not, I see that people within various industries doing it just because the nature of generative AI is, it's all over the place.

[00:25:45] You, you need to understand it thoroughly from day one. So I'm seeing that this particular trend of metric definition happening right from the very beginning. So initially people used to say that, "Okay, generative AI application, let's just ask the [00:26:00] generative AI to come up with a justification on a scale of one to five on how confident it feels about a particular answer."

[00:26:07] So that worked for a few days, but then people started to realize that AI is, is trying to be a grader who, who is, like, always around three for everything. So then people came up with, "Okay, this is a bad metric. Probably we should have a binary metric." Binary metric in the sense I would say how robust or how fair it is.

[00:26:24] So fair or not, robust or not. So have a binary definition when you're asking a generative AI about justification in terms of fairness, in terms of relevance, relevant or not, fair or not. So it doesn't have much choice but to choose one exactly, zero or one. These are few metrics which we need to take early on in, in the journey, and this is only going to refine going forward.

[00:26:48] Justin: Yeah. I think the big thing that I heard a lot last year was evaluations, right? Evals. And especially around large language models, I think we've had evaluations in regression analysis and stuff like that. Like, like you said, we pull data out, [00:27:00] we've run it through, how accurate is it? We can measure that.

[00:27:03] Generative AI, especially around large language models, like you say, you ask ChatGPT the same question twice and it will give you a different answer. And so that is like this sort of like very interesting sort of line we're trying to walk, I think, where people throw things into production without actually even setting up any sort of evaluation techniques or even being able to sort of trace and watch stuff.

[00:27:24] So there's a lot of open source projects, I think, that have, that have taken off this past year for that particular reason, because before you deploy it You need a set of this is the questions, these are the answers, and you need to feel confident that it's not gonna go off the rails, especially around different prompts, right?

[00:27:43] So you don't know who, what people are gonna be asking. So the idea is, is, you know, especially if it's open-ended like that, where, you know, you don't know how they're gonna ask a different question or put something bad in there, and all of a sudden the language model opens up and does what you don't want it to.

[00:27:56] Interesting times for sure, but yeah, the evals is what I've been seeing and [00:28:00] hearing a lot about this year. The other thing is, I know you're speaking, I think, coming up at one of the mini analytics conferences. Is that right? 

[00:28:05] Shruthi: Yes. I, I believe in actively engaging in the world outside my professional life when it comes to AI.

[00:28:11] I was fortunate enough to present like thrice at many mini analytics conferences, two times at DataTech, and the recent one was at University of Minnesota Women in Data Science and AI space. And the next one is coming up is the supply chain conference, Fast One. I'm going to speak there on, uh, time series foundation models.

[00:28:28] Are they magic wand or not? So do you need to trust time series foundation models or do you need to go back to your classic algorithms like ARIMA or linear regression? So that's the deep dive I'm going to do and present on that topic. 

[00:28:42] Justin: That's fascinating. We're gonna be there. Applied AI is gonna be there that day, and we're actually working with Dan and the team there to have an Applied AI track.

[00:28:50] I think the mini analytics community, some of the feedback I think that Dan's been getting, and I don't think I'm speaking out of line here, but a lot of people wanna get back to some of the roots of classical data science, which I think is fabulous, [00:29:00] and especially around supply chain, right? That's... It's just such a complex area.

[00:29:04] So many different variables, I'm, I'm guessing, go into this. 

[00:29:07] Shruthi: Yeah. I've definitely heard of the, the desire to get back to the roots, the, the classical data science as well, especially with too much of AI, generative AI around, and which we shouldn't forget about the roots from where this all started, and that's very much essential as well.

[00:29:22] It is fascinating to speak at a conference like that on, on a topic like this because it does right at the crux of AI, which is time series foundational models or transformers. Think of it as ChatGPT for time. So it's, it's fascinating to talk about that in a classical data science, uh, sense. 

[00:29:39] Justin: Sure. Sure.

[00:29:40] The other thing is this uncertainty quantification, that term that I read off in your bio, and it seems to me like with all of these new tools and stuff like that, this idea that you can start getting to closer and closer to what I think we've all been hoping for is much better sort of predictive maintenance.

[00:29:55] Mm-hmm. So I've been in the IoT space for more than a decade here, and I've been working with companies, and the whole-- the gold [00:30:00] standard is we're gonna get enough data to finally have these, to be able to predict ahead of time before this machinery sort of dies out. And so that's one piece of the uncertainty sort of quantification.

[00:30:09] I know... I, I'm not sure how you have been working with it, if you're doing much at Boston Scientific, how you sort of approach that field and worked in that field. 

[00:30:17] Shruthi: Conformal prediction is one of the uncertainty frameworks which I have been really interested in the recent years, like in the past year. So I'm actively working on, like, demand forecasting right now, and one thing which demand planners love is to know what is the range, what is the absolute range the demand for a particular product falls into, and that range should be very strict when I say range.

[00:30:38] It is not like hundred to five hundred. It's not like that. They like to look at hundred to one twenty. I should put up that range with much more confidence. That is what the quantification or the conformal prediction is all about, to be able to give that range to demand planners so that they know that- Demand would fall into that space with ninety-five percent confidence or [00:31:00] ninety-eight percent confidence.

[00:31:00] And this is not just applicable to demand forecasting as such. You can extrapolate this to generative AI classification outputs. So to any outputs, conformal prediction or the uncertainty quantification is just a wrapper around your, your outputs, your ultimate outputs, which come from an algorithm. I like that concept of uncertainty quantification because to be able to have a range and make decisions based on that range, it's pretty lucrative.

[00:31:27] Justin: Yeah. Well, it's almost like magic. When I go to a store to buy something, like you might go to a florist, and they have all these flowers that are there, and you're like, they had to somehow figure out how many fl- because these things don't last for very long, so you gotta be very particular on what you're gonna buy and when and how many, and o-obviously, don't throw away the least amount possible.

[00:31:43] So it's a very interesting problem. And my undergrad was in applied math and physics, so I really love the applications of math. I tell people I don't wanna solve for X. I actually wanna figure out why I'm solving for X and answering a real-world problem. So it's a fascinating area for sure, and I think as these algorithms are [00:32:00] gonna be asked more of them.

[00:32:01] I love how you tied it back to a large language model too because, yes, we can use these underlying principles of, like, uncertainty quantification to actually make us trust now better the output- Yes ... of an LLM. 

[00:32:12] Shruthi: One of the areas I'm working on right now, like how do you tie this conformal prediction or uncertainty quantification framework to LLM outputs, and how do you make your LLM outputs much more reliable?

[00:32:23] So I'm really fascinated by the subject of evaluation of LLMs, and how do you reach that trust point with your LLMs? 

[00:32:32] Justin: Fascinating. Really, really cool stuff. How do people get ahold of you? 

[00:32:36] Shruthi: LinkedIn. I'm, I'm pretty active on LinkedIn. Expect a response within a few hours for sure. 

[00:32:41] Justin: Sure. I'll put a link to your LinkedIn profile and then links to some of these other things that we talked about here.

[00:32:46] Even that article I think you had mentioned, um, I'll make sure to find that and have that in the liner notes here that people can sort of click and follow through. 

[00:32:53] Shruthi: I really wanted to thank you, Justin, for giving me this opportunity. Like I mentioned, I love being engaged in the community outside [00:33:00] of my professional life, just to share my knowledge about applied AI and also get to know more about applied AI and what other people are working on.

[00:33:09] So you have given me a platform today to share my passion and my interests, and I hope to contribute to Applied AI community the whatever knowledge I'm, I'm gaining in this space. So thank you so much for having me on the podcast today. I really loved conversing with you, and I hope for many more great conversations in the future.

[00:33:26] Justin: Oh, for sure. Yeah, it was great. No, no, thank you. Again, this community is built by people like you and everyone else who wants to contribute and share. So just thrilled that you were able to take some time, and I know that people will gain a lot of wisdom here from what you have shared on our program today.

[00:33:40] So look forward to keeping in touch. 

[00:33:44] AI Speaker: You've listened to another episode of the Conversations on Applied AI podcast. We hope you are eager to learn more about applying artificial intelligence and deep learning within your organization. You can visit us at appliedai.mn to keep up to date on our [00:34:00] events and connect with our amazing community.

[00:34:02] Please don't hesitate to reach out to justin@appliedai.mn if you are interested in participating in a future episode. Thank you for listening