Sebastien Briens

Good morning. It's my pleasure to introduce you to Brent, David, and Nagraj who have a lot to talk about based on what we heard. A number of you had the pleasure to meet with David last year.

David is CTO and VP of Strategic Incubation for Microsoft where he is in charge of topics such as AI, quantum computing, and space.

A lot of what he talked about last year actually happened. So I suggest you listen carefully to what he has to say today.

Brent is a senior exec at Salesforce and most recently, CEO of Mulesoft. Probably the most successful acquisition in Salesforce history between 2 billion dollars run rate and 40 % revenue CAGR in the past 5 years.

A number of you in the room actually use Salesforce. I know a vast majority actually do as part of their core infrastructure to run the customer information, customer business. So we're very keen to hear your thoughts.

And then finally, Nagraj is a General Partner at Touring Capital, he has 20 years’ experience in venture capital. It was the out of Microsoft Venture Business, M12. Previously worked for Qualcomm Ventures on hardware side. and then was also a managing partner at the Soft Bank Vision Fund, the largest venture capital fund in the world.

I suspect a number of Nagraj’s previous investments will be part of your daily life, depending on your level of activity: Zoom, Waze, and for some of us - or some of you, I shall say - Fitbit.
So welcome.

I think we're going to talk about AI today. AI tomorrow. We're also going to talk about the scaffolding a fair bit, and how we see those layers moving.

Maybe to ground the conversation first. You know, what does the current moment in AI mean for you personally and your company? So I'll start with you, David.

David Carmona

I love the comparison, by the way, from the presenters on kids versus adults. I loved it. I think it’s something that that we're learning how to manage, right. So under that I immediately consider myself a teenager, right? Because my role in Microsoft is really to work closely with those incubations and turn them into businesses, right? That's my role.

And I was starting to drive with the cloud. Then I moved to AI 8 years ago, where most of my work was the kids, right, with Microsoft research and trying to commercialize that innovation. And most recently, with Gen AI a few years ago, we called that product internally in Microsoft, Aristotle, and that was, I mean, that is what you see today on commercializing that innovation. Which means that it is very troubling years - this teenage era, right - because you are neither an adult nor a kid, and I feel exactly like that. So that's my point in my career personally, with AI.

Brent Hayward

Maybe build on the teenager theme. I mean, personally, I think it's been amazing. I use generative AI every day, I use it primarily to summarize tasks. I feel like the business workplace today is just so noisy. I get literally thousands of Slack messages a day, and to be able to sort of take the team's output and be able to summarize that quickly and act on it is a simple task that probably saves hours and hours a day.

My youngest daughter uses it to write her papers so that she can go perfect her career as a Fortnite player. So there's wonderful other human things going on with the technology. I think from a business perspective, though, and I think, as CEOs, we all need to appreciate that it's an existential threat every single day. I think the difference, though, between this one and other existential threats is very much the hype cycle. I mean I see this somewhat analogous to moving to Cloud, I mean, that was just as existential.

We built a company, MuleSoft, that took advantage of the fact that the juggernauts and IBM and Tipco and others who had cornered the market in Integration software had not figured out a world where all software was gonna run in the cloud.

And so here we go, 2 billion dollars later and growing, you know, we built an entire business out of that. And it's generated an entire ecosystem. So we have to look at these things as a threat. But I do think it's a marathon, not a sprint.

There's just something about AI. I don't know if it appeals to our romantic nature as human beings, but we love that there might be some catch-all technology that solves the hardest problems that we don't know how to solve.

But if you believe that, you know I've got some ocean-front real estate to sell you in Switzerland and found some magic beans today, when I was on a walk, I'll sell those to you for a good deal, too. I think they're like the reality is this is going to take time. The consumer model doesn't necessarily apply in the same way to the business model around UI and data and governance.

And it's gonna take time to emerge. So I'm not particularly surprised that we're 18 months in, and there's a lot of noise around why it hasn't applied the same in the consumer world of the business world.

But there's really exciting innovation. And we're starting to see some of it monetized as well.

Nagraj Kashyap

Yeah, thanks. I really appreciate. Microsoft, of course, was my home for 5 years. And I've had great respect for Salesforce as well.

My career has actually spanned what I call the last 2 platform shifts. So when I was at Qualcomm we went through the mobile platform shift, at Microsoft right at the beginning of the cloud computing platform shift. And I think the sole reason, at the tail end of my career, I decided to start my own VC Firm was Chat GPT in November 2022. This is gonna be another 10 plus years of wealth creation. I couldn't be sitting on the sidelines of.

So I'm personally very excited by Gen AI. What it brings to it. I think what many people have said, consumer is a good example of how to frame it, is AI in sort of some form, has been around for multiple years.

We remember back in Qualcomm Imagenet, which was the advent of very early machine learning. We didn't call it AI at that time. We can call it AI 1.0.

Then we had basically a big leap forward with Ganz in the 2014/15 time frame and then finally transformer models came in 2017 but we didn't have the Chat GPT breakthrough, powered by transformers, until 2022.

And actually, why everybody's so fascinated, and the hype is so much with Gen AI is because consumers can touch and feel it. Before Gen AI, no consumer had any understanding of what AI can do for them every day. Now OpenAI has, and Chat GPT specifically, I think, has made that part of everyday life, and that, I think, has led to this hype cycle. So I'm a very firm believer in the same thing, the long term value of what Gen AI will bring. The short term, we don't see it ready for enterprises. I mean, there's very few use cases. We'll go through that. But clearly the hype cycle is driven by the fact that everybody, not only in this room, but our kids, everyday consumers not in tech, can touch and feel what AI can do for them, and they never knew about that. In the last 2 times you have the AI revolution. So I think this one is real but the hype cycle’s really taken over.

Sebastien Briens

So I guess this begs the question, what do you guys see as actually working today? So if we go in there, almost like in the first bucket that Chris describes, what's really today in production, and I'm going to follow on by saying: I suspect, both in your internal organizations - because you obviously wear 2 hats, you’re software vendors and platform providers, and then within your customers, and maybe within the start-ups, you're seeing. What's working?

Brent Hayward

Within Salesforce today we have over a hundred projects running just to implement AI within our own capability across sales, across service. I mean, what's nice about the software we provide is that we provide the same functions to our customers that customers provide to their customers. So we've really feel like we can be customer zero of the technology.

But panning back a little bit, I think I love the concept if it's a broader wave we're in. It started with sort of predictive AI. I think we bought a company, Relative IQ, for about 400 million enterprise value 10 years ago. It now generates over 2.5 billion predictions a day. Stuff that we all see when we go into like Amazon and shopping carts to say, what's the pair to the product that you would have with this. So predictive AI has been out for a really long time. We're now getting into this generative AI world, and very quickly - I love the agent talk earlier - it's not just human in the middle. It's going to be semi autonomous and then fully autonomous. And then eventually we'll get to, you know, AI around basic human intelligence which I think, those are the doomsday, fun scenarios to talk about.

But in this world of generative moving to semi autonomous, that's where the majority of the experimentation, at least, is happening within Salesforce. We've taken all of our human resources content, all of our medical benefits, all the questions you might ask an HR rep or your manager across 75,000 people in a globally distributed company and Slack is now answering 375,000 questions that it would have taken a human to answer every single quarter.

You know our engineers are using our code development tools and not only producing better code, but saving 20,000 engineering hours a month.

If that is real enterprise value on the human productivity side, I think what's going to be interesting is, how do we deploy that, more standard, with the customer having to do less of the work themselves? I think it's unreasonable that every customer needs to build their own LLM.

The UIs are fundamentally gonna change, and the underlying data needs to be far more secure and accessible within the enterprise, especially in things like regulated industries.

So I might bunk a little bit, we are actually seeing tremendous return on that investment. Would I say it's at a place where the cost equation, the margin equation, is there yet in where it's generally releasable? No, I think there's very rapid innovation. But I do think those things are coming. We have lots of customers that have deployed the technology: Rocket Mortgage, for example, Wonolo and simple metrics like 20% less average handle time. Like, if you're in a service industry, who wouldn't want 20% less average handle time on a particular call.

You know, Rocket Mortgage is actually experimenting with using it to auto loan. So this isn't just work productivity. This is fundamentally changing the way they are going to generate offers and acquire customers around their mortgage products.

So we're seeing that today. I think it is very real.

David Carmona

Yeah. So in in our case, I think it's a little bit different. The difference between 1st party and 3rd party. We have a mantra in Microsoft. You probably remember it was and it is: “first party equals third party”. Right? So if we see anybody doing something that is only for internal usage, that's a red flag, and we have even the 1P = 3P that we're bringing in meetings, right? And that apply also to AI, right?

So the way that we were able to scale AI Microsoft is actually decorating the platform today is exactly the same platform that we offer externally. Now, on what use cases we see more real today. I love, by the way, the framework right? And I think it's something super valuable, and you will see it in different ways in any literature out there but that concept of knowing where you are in your journey of AI to then target the things that will be successful, or don't take too much for where you are. Still the way of looking at that. Super important, right. The good news that now the industry is getting more mature. You have a ton of research. You have a ton of even evaluation frameworks. We just ran one in Microsoft. Maybe 3 weeks ago we released it with 1500 companies, and we were first identifying in what stage those companies were in their maturity of AI.

And then seeing what are the success factors in every stage. And it’s something very similar to what you saw before. So we see that when you're early in that journey, where we see the majority of the use cases being successful, is like that first level: productivity, efficiencies. More than 80% of the use cases are in that stage.

Then the more that you move into a more mature company on AI, then the more you start doing things that are like revenue growing, growth growing, AI first kind of activities. And we seen those companies more than 50% of their use cases more on that side. Now know where you are on what to do, right? Probably I mentioned this last year the pilot purgatory, right? So that notion that we see many companies that are in this pilot purgatory forever, and you don't get out of that. Maybe you don't even need to be in that pilot purgatory from the beginning. So if you are calling something a pilot, maybe that's that's already a wrong first step, right? So you should be doing more projects that have a continuity right where the business is heavily involved.

So you are starting from the beginning. In those cases, you just skip directly, right? So you just do product development, quick iterations on product development instead of doing pilots, right? So if you feel you are in that situation, rethink where you are.

Nagraj Kashyap

Yeah, I wanna give some examples if that's good. I really liked the first session. I think Joe talked about services, we are seeing this actually work really well, when we talk about displacement of services businesses.

So a good example from one of the companies we did. We went after, essentially, digital marketing teams and digital marketing teams are everywhere when the end customer basically is acquiring their customers through Facebook, Google, whatever other channel there is.

Historically, it's been very, very siloed. You had a team that would go at one channel for Google, the other channel for Instagram, one for Facebook. And now using. not Gen AI right now, just using a predictive discriminative AI. They've actually built models where you can do cross channel optimization.

And what are they taking away from? They're not taking away from software. They've been taking away business from agencies.

And if you add that workflow, a digital marketer gets up in the morning, is able to optimize the ad campaign across multiple things. And what else do they want after that? Then they want to eventually be able to build their ads without going to the agencies. That's where Gen AI comes in. So we're seeing actual production of Gen AI in, not necessarily video ads which are much hyped up, you have much bigger basic TAM of static ads that you can generate. You know everybody talks about Sora, and what OpenAI is doing with Sora. You don't even have to go that far, GenAI, using pure static ads, and how you can experiment with them and essentially test across different cohorts is a real use case. But that is taking away from services businesses. It is not taking from software. It's fully disruptive. That's a really large TAM.

Another one of our companies is going after a vertical, which is, I think a lot of the software there is, PE owned.

Truly, the difference between, I think what was talked about is, don't disrupt Salesforce. You can't do it. They're like a new incumbent versus a player that is a Salesforce for the auto industry, call it the DMS software providers, and those are very, very legacy for the most part. I'm sorry if there's one here, I'm just saying, at least in the US.

If you go talk to anybody in the auto dealership, it is the worst customer experience ever to try to buy a car. Tesla's changed that. But if traditional customer service… and it also relates back to one of the things here on the TAM side.

They are not going under there and saying, we have a TAM of support. They're saying, there's 65 billion employee TAM in the front office that doesn't actually need to be there.

What are they doing?

They're coming up taking customer calls most of the times, not answering those back. Then trying to route to a service advisor where their car is, all of that, whether some agent architecture, even if there’s a wrapper, can be fully automated. We are seeing this actually working. And it's interesting. When some of these companies see these solutions, their jaw drops because their frame of reference is literally software they've been using that has not innovated in the last 15 years. It is not Salesforce, it is not something that comes from Microsoft.

It came from providers that would have to re-architect their tech stack so much, it'd be very, very hard to do so. These are real examples of things that are working. You just sort of pick your spots.

Brent Hayward

I love that. I think that point was made like 2 sessions ago as well, that when you really look at enterprise spend in technology and software it’s about 15 %, the average enterprise, the other 80, give or take, is just human capital.

It is a mistake if you're starting a company today, and your only viewfinder is on existing TAM that you wanna replace or take out. I think that needs to be an angle. What software companies are your competitors? What's unique about the value proposition, the workflows you're providing, what's unique about the data you're providing. But I think if you challenge your teams to cast their eye on the human element, that is where we have a massive expansion in the TAM.

Who in this room isn't worried about hiring great people, training them to be effective in our organizations, retention and turnover. Software could be solving a tremendous part of that 80%. If we allow this opportunity to take that angle if we just sort of are fighting over the same 15% it's a much less interesting problem that we're solving.

Sebastien Briens

Can we spend a bit of time on this concept of agents.

I think it was described. It would be helpful if perhaps you know, Brent, David, you could describe how you're seeing those happening. By basic understanding, orchestrate a bunch of models to deliver more complex use cases. So to some it looks magical, to some others it looks suspiciously like RPA, which has not worked as well. So, if you could expand on that, what are those? What needs to happen for these to truly be working in the enterprise space?

David Carmona

So it's funny, when I was seeing the frame that you were showing before, God, it was so similar to our thinking, right, and we refer to it as a slightly different. And let me just go through it because I think the evolution of those give you the answer on what is an agent, right.

For us the first stage, where we are right now, from the Gen AI point of view, is productivity. No need for an explanation there, those are applications that are infused with AI: typical chat bot-centric applications where they can improve your productivity. Let's keep that. That's where the majority of us are therefore now, by the way.

The second one that we see is core reasoning. So that is where you can not only do things that you could do before quicker, but things that you couldn't do before. Because now you are building on top of this AI that can go much further than you, that you can address with that push, cognitive capabilities that you couldn't do before. That's simple, right? And we see a lot there already going on. That's for us is that second wave.

The third wave, the wave that we're referring to is discovery. That's the one that is more aligned with agent. Why do we call it discovery? Because in that case things shift. So in that case it’s not only about that reasoning paradigm, it's really the human in the loop of guiding that AI to perform that reasoning. That's a huge shift, if you think about it, right?

That's for particular fields, for highly cognitive fields. For example, research and development, which is our focus for this technology initially. It’s really, truly transformative, right? Because now in that moment you see a new concept of a autonomous reasoning loop where the human is always in there as well, but their role is different, right. In that autonomous reasoning loop you have from learning right to a hypothesis, to testing, to experimentation, and then all over again for enabling, not only things that you can make better, but really to uncover new insights, to be proactive instead of reactive, just waiting for the prompt to be written.

In that case you have something that is much different. Yes, the starting point of that is an agent. But don't look at an agent like a technology, or an architecture, or something like that. It's much more powerful and deep. What we want to see there.

Brent Hayward

I mean, I agree with the framework, and I agree with the promise of fully autonomous agents. I don't know that there's alignment in the industry about how we're going to get there.

I think we should overlook today, I think what's happening today is a technology like an RPA which is typically a quite rigid architecture to solve a problem. It's extremely brittle to record an existing process across different systems and data. When that process needs to be changed or is dynamic, the bots tend to fall over.

But we have these 2 new things that are really interesting: with generative, we have this ability to have a conversation. So we're making access to these workflows far easier. We can extend a workflow that existed in a contact center to support someone or existed with a field service agent, and we can make that more accessible and more conversational with the end customer. We've just raised a bar for self-service, or by the time you take first call we're solving a real problem for you, and not, you know, press one for change your reservation, press 2 and so on.

So I think this intersection between the conversational side and activating more dynamic workflows is where there's the largest opportunity, and that should replace more traditional brittle RPA architectures over time. I think the question then becomes, how do we get to this fully autonomous idea? And we're ignoring some huge elements when we try to make that leap.

We're ignoring trust. We're ignoring toxicity. We're ignoring data grounding. What's data grounding? It's actually being able to argue articulate where this answer came from. Where did this information come from?

And we're also ignoring data privacy. This is where the consumer model, where basically a whole bunch of companies went out, crawled the internet, effectively, may not like the word, but stole a lot of proprietary data from Time Magazine, from Reddit and from others. And now they're sort of apologizing for doing that, creating agreements with these companies. But that's how those models were built, and they're fascinating.

You cannot do that in the enterprise. Bank of America is not going to just offer their data up for the world to crawl, to create these autonomous agents, to do headless banking. Not only that. But when you crawl sources of data that are not trusted, it ups the bias. It ups the toxicity.

The other big thing that's going on. It's one of the reasons actually Salesforce, I would argue, is a bit late to the market rolling out some of our AI tooling, because until we had an agreement with OpenAI, that every time we feed that model a prompt it guarantees it will not retain the data for its own training purposes. That was essential. I mean, trusted customer data is what our customers own, not Salesforce.

And so I think. as we start to add conversational and workflows, and also pay attention to the fact that in the enterprise this data is yours. It has to be trusted, secured, and there's real liabilities to getting the answers wrong, not just reputationally, but legally.

I think, that creates this interesting sort of human in the middle loop. And I think the best companies will start to evaluate, not just the ROI that they're delivering customers, but they'll also look at things like the accuracy. And when we start seeing 95%, 96%, 97% accuracy.

Of course we will set that free, and that will be semi autonomous and ultimately autonomous. I think the interesting world is when we start having these capabilities, and it's not going to be one company that owns them all. When we have them everywhere. If you go on Hugging Face today, I think this morning I logged in. There's 701,000 LLMs that have been uploaded to Hugging Face.

So massive proliferation, and I would argue commoditization of the LLM side of the equation. But when you start putting all of these things together, I think there's a huge opportunity for orchestration. And it was you were talking about sort of the legacy automotive vendors. A lot of what's coming is just someone that's attaching into these legacy capabilities and providing orchestration of these conversational AI capabilities and workflows on top of that.

So there's gonna be the great unifier of infrastructure, Microsoft, Amazon, Google. But there's also really a lot of room for the great orchestrator of these b2b processes across technologies as well.

Nagraj Kashyap

Yeah, I mean, I couldn't agree more. I think, one of the things coming from Silicon Valley, we have a very different view viewpoint than what you probably heard earlier. We don't think in this wave, you have to be very careful on where to invest, at least for us, and where not to on the AI front.

Coming from the last 2 platform shifts. The lesson we learned was, in mobile, if you look at the incumbents that started out, some of them from here, Nokia, Eriksson, they don't exist. They didn't actually survive the platform shift. Google, Samsung, Apple basically took over the mobile smartphone market.

You look at cloud computing, when I went to Microsoft, Microsoft was not the incumbent. AWS was incumbent. Microsoft was caught flat footed. Google was nowhere there.

So what happened was a lot of the efforts within Microsoft or Google were just catching up to make sure you could have the cloud computing efforts going.

What that allowed was a huge amount of innovation on the software side. Whether it's a data log, it's a snowflake, a lot of companies on the infrastructure and tooling side came up because everybody's ball was somewhere else.

What is fundamentally different about this wave is who started it. It was Microsoft as a big incumbent teaming up with OpenAI, which is a startup. So they're innovating at the speed of a startup with the resources of a Microsoft.

So for us. When we look at companies that are trying to say we are going to build this horizontal thing, we're going to build an agent, we look at who is actually best place to do it today.

I don't think it will come from the startup space. A: The idea on the Enterprise side, how complex it is already was, was mentioned. Addition to that, the level of integrations it is to do a full agent is off the charts, and to say that somebody who's gonna come up with a one startup in the valley with a hundred million dollars from Sequoia. Anybody else you can sort of disrupt that actually is oversimplifying the problem too much.

So I think the only advice I would give is, don't start pilots with Silicon Valley startups on this, because I do think you're not going to see the results here. You're better off waiting for some of the incumbents in this, it's heresy for me to say this, but better off waiting for the incumbents here to come up with products, because the innovation speed in this platform shift by the incumbents is off the charts.

You wait 3 months. The next Open AI developer day. Open AI, Microsoft plugged the gap. If you remember Rag, I mean, I think 12 months back. Look, Drag is this like the new thing. There's 50 startups on Rag, if you think where Rag will come from will come from traditional database vendors which have incorporated vector databases into the workflow. Because you have data prep, you have so many other parts of the process that buying from a single vendor for a single purpose. You can experiment with it. It will not go into production, in my mind.

So I think you'd be just careful of when you spend your time and money, which is very, very important on things like pilots, who you're trying out with. Well, even if you try it out, I don't think will be the eventual provider here.

Sebastien Briens

So I guess this begs the question. I'm talking as an investor. As to where application software starts and ends in this world. So where does the stack, how high do they go? The platform vendors, etc. And where do you invest?

Nagraj Kashyap

I mean, it's a very small portion of the world we see but, for us, the easiest place to invest right now is what we are calling vertical. AI, I mean the auto example I gave.

That is not where the incumbents are focused, and for good reason. Those are very small markets for incumbents. They're going after the horizontal layer. If you talk about LLMs, we absolutely believe LLMs will be commoditized. That is not a venture or investible area from a VC perspective.

The amount of compute and cost required. You raise a hundred million, 80 million goes to Nvidia. That's why they’re 3 trillion today. That is not a sustainable venture model. It should not be backed by venture dollars. So that, I think, will get commoditized. And I think the way to think about LLMs is basically what happened in mobile. There was 50 different mobile operating systems, everybody in the world had an operating system, and who survived? There's like 2, really.

So there will be OpenAI. Clearly, I don't think you can displace that, and there'll be one or two more that'll be at scale, but it'll be a commodity layer at the end of the process. So then it becomes there's some tooling, which was one of the layers, and then there is application software. And there's going to be a little bit of where application software starts and where tooling starts.

Our view is when we look at the vertical AI space, the one advantage we have, we think there is, and a lot of people in the room here have, we do believe in the fundamental advantage of proprietary data. And I think a good example I'll give is - I can't disclose the names and all that - but a European company that is in the space that you'd expect to be able to mine the data from, legal. You can, if you have it you can mine all this data. Is trying to be bought, or it's like it's got an unsolicited offer from a quote unquote Silicon Valley Company. That is a GenAI-based legal company because they cannot get it started. They cannot jump start their business without the hundreds of millions of data points that have been collected over many years by some of the companies in this room and they have the scaffolding. They have a wrapper, but they don't know what to do with it. They cannot jump start their business.

And that data that you're collecting that is proprietary customer data, that has privacy associated with it, that is not available on the wider web is the moat where an Open AI cannot train on that, where a general purpose LLM cannot train on that. So we do believe if you can build on that, ultimately you can argue that some of the companies in the room can take that data and start building domain specific foundation models that are much easier to train, much less cost, much less compute.

And that becomes a proprietary way for you to basically get into Gen AI. I'm not saying that general purpose LLMs are less better at doing domain specific answers. It's just they don't have the data for it.

Brent Hayward

We do a simple 2-by-2, which is value and data, and the upper right is: if you have access to or have the privilege of managing your customers data - like we're given the opportunity to do at Salesforce with our customers most precious customer data - and we can develop specific functionality for a task, whether it's to market sell, to service, to do those things at the intersection of those things. I was actually at the bar last night talking to Soeren from Trackunit. Amazing, like telematics company. He's located in Denmark. But they track heavy equipment at construction sites with this amazing IoT device that's generating just millions and millions and millions and millions of records of data. So they have data, secure proprietary data on behalf of their customers and their customers equipment.

Do they have processes and workflows like when they should replace a part, or where something is located, or even geocoding. We've had horrible weather in an area that's a construction site. Should that change the way we think about servicing and providing for these capabilities? So, in the intersection in your enterprise, whether it's internally or externally, where you have high value and high data. That is, I believe, where you're going to get the most ROI.

Now, we all don't have that privilege, but I think if you don't own the data, you are going to compete with everyone else that is building products to commoditize data feeds.

If you also don't own the data, or have good access to the context behind the data. The metadata. Today, you know, General AI can tell you that you should go return a product.

But the best thing to do because of this error code and issue with the product you have is you should go return it. Great, you saved me a bunch of time and enough to call a call center agent.

What it can't do is do the return for me. I want to say, “return it”. That requires the metadata, the data, the context, the business process. So, I think it's, again, romantic but folly to think that ERP systems, CRM systems, marketing systems, systemes that have all of this data and the context and the functionality of business process behind that data, that simply if you take the data out and put it into a general AI, that somehow magic happens.

It's just not true. I think we're seeing a ton of spend in the LLM. I think surprisingly, for a lot of us that market is… the innovation cycles are rapid, but it's incredible how many very specific LLMs and General LLMs. Are out there. We've taken a stance of open LLM.

Some of that also is for frankly, consumption. If you're a service agent, you don't need GPT4 to run that query. It's bad for the environment. Honestly, it takes too much compute. You need a much less sophisticated LLM.

So, being able to orchestrate: calling the right LLM for the right workload for the right function means that the cost of providing that service is economical as well as the value provided.

But we tend to always start with value first in answering the question of what products and services that we can provide. I think one of the problems with all these pilots is we've lost our mind and forgot that the reason we funded pilots was we used to have this thing called a business case, and ROI.

Most of these pilots have no executive sponsor. They have no ROI. They're put in an IT lab where we ask people to build things they shouldn't be building. Your team should not be building LLMs. And we're not focusing on the real issues, which is, what is the return on the pilot. And we should be willing to experiment a lot to find out that the return on the pilot is negative, and let's throw it away.

I think what most CEOs are frustrated by is that 80% of these pilots that are not working had no pre predetermined outcome. So we are randomly experimenting, which is very different than actually piloting on what we think might be a great idea to take to our customers.

David Carmona

One thing that I would that would add, because I love that concept of data being your differentiation, right? So that's something that you need to keep in mind always. I think what we're seeing now with this new model is not only the data, it’s the knowledge.

So it’s the knowledge that you as a company, that your people in your company, have created over all these years is what makes you very special. And now for the first time with this new generation of AI. You can exploit that knowledge. That's very powerful, right?

Because that adds a new layer. So we are all visualizing this stack, right? It was perfect, the slide that was shown before. Where did you want to focus the value? Well, just remember something, by the way, right? So even in Microsoft, which usually we are close to the bottom of that platform, we are not following that approach. We decided as a company the 1st thing that we say to every business unit in Microsoft, to any developer, to any product manager is, “Hey, if you can do that problem without creating a model, with just an out of the box model, do it.”

That's the first option. That's what you're seeing with M365, it’s actually out of the box models with maybe some rack on top of that. So if we don't do it, should we do it? Should you do it? So think about that, right?

So then, with the application. Now, with this knowledge, you have a very interesting additional layer, which is services. It was mentioned before today. How is services going to transform? I had a conversation a few weeks ago with an audit company, one of the big four, that they are seeing a huge opportunity to this.

So the way that they're visualizing this is, hey, I've been forced to do a PxQ business model for all this year. So how many people, how many hours? That‘s your price. Now we have a new element: now I can do PxQ xT. Technology. Now I can take all this knowledge that is being imagined in all the industry, right, the knowledge that is there. I can put this knowledge to the work of this platform that can then increase the value automatically to all my customers, right? That's something that I think will see more and more. Don't think, hey, this is gonna replace my services. So why should I do it? No, this is gonna amplify your services. So take that into account.

Brent Hayward

Actually, that one point of inspiration. I think organizations that are actually getting these pilots out of the technology lab and into the hands of the actual users they're meant to enrich are seeing more success.

So I know we're all a little bit terrified of “are all our call centers going to go away”. It's sort of analogous to when Cloud came about. It's like, are there no more people in the data center putting racks on, racking up equipment.

And the reality is, that was wildly over hyped. It's just as companies, we don't have data centers anymore. But the job market for the AWS's of the world and other providers, and Azure is probably better than it's ever been, and more sophisticated than it's ever been. But I think you've gotta overcome this fear of: “we don't wanna disrupt our workforce”. Actually. most of the time they're struggling with the tasks that AI would help them be far more proficient at.

If you actually give them these technologies at the tool, at the edge, they will automate their own work. And if they automate their own work in the products that you're building or services that you're building for the market, you can turn around and package that up and offer that as innovation to your customers as well.

So I think these pilots need to get out to the edge where the human productivity opportunity is first. That's where you get to the V in the equation. And then it does become this life cycle of, wow, we really unlock something for ourselves. Let's go unlock it for our customers.

Nagraj Kashyap

Just one comment when David said about the accounting firm, I think this is also a little controversial, but I'll say it because why not?

One of the things we are seeing in our early companies, and where there is some risk for, I would say, not large incumbents, maybe some not, but even for some of the big incumbents here, is the business model for SaaS. Again, it's heresy but seat-based models have survived for many years and I think I go back to the example of one of my companies which sells into traditional marketing teams.

So what they found was after year one, the digital marketing team they were selling into went from 10 to 2. Think of what happens to a seat-based model there, you basically have no NDR, you're going below 100% NDR, so what can you do there.

So that's the thing, food for thought on how you can think of aligning pricing based on value, creation and consumption. It's actually very hard. But if you don't start thinking now, at some point you will have less people in seat that you're selling to and then you have to figure out what to do about that.

That, I think, is the good news in the startup side is actually because they're smaller. They're able to adapt faster to that model.

One of the companies that previously funded when I was at M12 had a seat based model for sales, and they've gone to an account based model which does not mean you could have multiple salespeople there. But it's an account-based model. So I think that is one, I think, from a business model perspective it’s good to start at least thinking about in your own business, because we're gonna be very bespoke.

There's no one answer to change it, because my company on the digital marketing side was easily able to align with ad spend. Well, if you give me this much ad spend, I save you 30%, that basically becomes my business model. That is not like a one size fits all.

Sebastien Briens

Gentlemen, thank you very much.