S4 | E21 | Building the AI Agents That Could Change the Internet | Abhishek Das, Co-founder @ Yutori

Alex Hutchings: 00:00:03

Welcome to the Think Data podcast brought to you in partnership with Mydataworks. If you want to stay up to date with the latest breakthroughs and trends in the world of data and artificial intelligence, and if you're curious about some of the strategies that companies and founders use to launch data and AI products, then you're in the right place. Our aim is to bring together a diverse lineup of fantastic guests from the founders, through to accomplished leaders and product owners at some of the most fascinating data and AI companies worldwide. They will each offer you their own unique insight into what it takes to launch and scale a great data business. Thanks for tuning in and I hope you enjoy the episode. Welcome to the Think Data podcast and today's guest is Abhishek. Das. He is the co-founder and co-CEO of Utori. After spending years researching AI at Meta, Abhishek is now building one of the companies at the forefront of AI agents. His company is creating technology that can browse, understand and interact with the web autonomously. We're going to really discuss his journey from research to startup, finding product market fit, scaling the AI business. and why he genuinely believes web agents could fundamentally change how businesses operate online. Really looking forward to this one. Let's jump right in. Das, it's really good to have you on the show today. And obviously, as I mentioned in the intro, you've obviously been and spent a bit of time at Meta doing some heavy research within the AI space and obviously launched Utori March of 2024. I was really keen to... I get a bit about your backstory, but first and foremost, for the people that haven't come across you or Yotori, firstly, who are you? And secondly, when did the idea for Yotori come about?

Das: 00:02:02

Yeah, thanks for having me. My name is Abhishek. I prefer to go by Das, which is my last name.

Alex Hutchings: 00:02:07

Perfect.

Das: 00:02:08

I grew up in India, moved to the US in 2016. That was for my PhD at Georgia Tech. I'm an AI researcher by background. So just to give you a sense to your listeners. As part of my PhD, I was working on problems at the intersection of computer vision, language, and reinforcement learning. So if you think of, for example, a home robot and asking it the question, where did I leave my keys? In order to answer that question, the robot would need to understand language, it would need to know how to navigate an indoor environment, and then come back to you with the object or with the answer to that question. And in my PhD, I was looking at how do we train deep learning models for these kinds of tasks. This is back in like 2016-17. So AI systems weren't anywhere close to as capable. as they are today. So it was a fairly early explanatory space to be working in. Yeah, I had a lot of fun in my PhD. I finished in 2020. My thesis was called Building Agents That Can See, Talk, and Act. So this idea of an entity, an AI system that can take a sequence of actions towards an objective, and that's basically what an agent or an agentic system means in the machine learning sense of the word. Over the last few years, of course, that that word an agent has picked up a lot and it's overloaded in all sorts of ways. But yeah, my co-founders and I, we've basically been working on agents in some capacity for almost a decade or more at this point. And so, yeah, like after my PhD, I was at Pair at Meta as a research scientist. And then my co-founders and I, we decided to leave Meta in 2024 to start Tutori. At the time, it felt like... chat gpt and a bunch of ai capabilities were starting to to pick up but if we thought about how we interacted with the web that still hadn't quite changed roughly speaking in in the last two three decades right like we still open a web browser click buttons manually navigate pages fight pop-ups and ads scroll etc and it felt like if you look ahead two five ten years into the future that that's ripe for a shift we're going to be operating at a slightly higher level of abstraction where we're where we're talking to an AI assistant or an agent that's operating the web on our behalf. And at the machine learning fundamentals level, it's the same basic principles. You need an AI system that is capable of making a sequence of actions towards an end objective, just that in this case, the substrate is a web browser and a task on the web. And it needs to be able to interact with web pages and APIs towards that task. So yeah, that's the space. That's the vision. and that's what we're doing now.

Alex Hutchings: 00:04:47

Interesting, because I took a look at obviously the founding engineering team and some of the people you brought on board. Obviously really impressive backgrounds, but one thing that obviously leapt out is there's obviously a lot of people coming from that kind of big tech, you obviously deep mind, you've got meta, you've got some fantastic, you know. resource that you brought on board. But what's been the biggest challenge you think getting people to move away from the big tech mindset and into scrappy startup think for yourself? Because obviously we see a lot of fantastic talent coming out of these big AI labs and then spinning up their own startups. But bringing on that first layer of hires, that's typically where the challenge lies. So how did you approach that?

Das: 00:05:27

Yeah, so a lot of the founding team that joined us We had worked with them in some capacity during our time at Meta and other places. We're, of course, like fresh contacts who we didn't know earlier. But I think, and there is quite a bit of selection bias because like this is a startup environment. We're a tiny team. We're like 14 people right now. But it's refreshing, to be honest. Like all three of the founders, we've spent a significant amount of time at Meta. And relative to a sort of large tech company like that. in a 14 person stardom, you can sort of make decisions on a whim. And you sort of each person is trusted with a good amount of ownership and autonomy. And so we select for people who we think have good judgment and will be and have high agency to be able to execute well in this sort of setting. If someone's looking for like constant guidance or direction, then this is not obviously the right size and shape of the company to be working at right but yeah like the short answer is that for many of the for a big part of the founding team we had worked with them in some capacity in our in our previous roles like some of our early hires had led multimodal post-training for llama at at meta others on the team like you said had been involved in post-training of gemini there's a few others who who spent time at startups previously so it's a yeah startups like versef minion etc so it's a mixed up like

Alex Hutchings: 00:06:58

talent from big tech but also startups that that we brought together because it's really interesting because if i go back to your your your thesis and your education obviously you had the well that's 10 years ago now if you think of kind of the problem statement you were trying to address and the reason you know the vision that you had what's been the biggest surprise you've seen in terms of this ai agent space because obviously there is this assumption that kind of Bigger, larger frontier models can solve every single problem out there. But in your experience, in your knowledge, being in the Wiis day to day, what's been the biggest surprise?

Das: 00:07:36

Off late, it seems to be the realization that cost matters. Yeah, a lot of. And historically, I think that is kind of been the arc of technological progress. Like there is this push towards invention and making something possible first. And once it's possible. there's a push towards efficiency and cost. They're like, okay, what is the sort of cheapest unit economics we can do this at, right? And so that's actually helped us to quite an extent in the last six months or so, I would say. So the core of what we build and how we are going after this vision is a computer-used model that we train in-house that can take actions and complete tasks on the web. So that computer-used model, it can... basically look at web pages and predict UI interactions that, okay, where to click, where to type, where to scroll and UI interactions of that sort. Or it can alternatively write short snippets of JavaScript code to interact with web pages. And you can prompt it in all sorts of ways. It is quite general purpose and it will be able to complete tasks on websites the way you and I would interact with a web page in a web browser. But yeah, so this is sort of the core task that we train our in-house model for. And we are not training a single model that can do math, coding, reasoning, and everything under the sun like the Frontier Labs are. So this is a model that's specialized for web use and browser use. But in that space, we've been able to sort of push accuracies to be state of the art. We have a model that's like way better than Opus and GPD 5.5, while still being a step change in latency and cost. Yeah. Two to five X cheaper and faster than the Frontier models. And that, like, just that last bit seems to, yeah, like, in all our enterprise engagements, that seems to matter quite a bit. Like, the frontier models are good for discovering what's possible to do now that wasn't possible six months ago or 12 months ago. But once you realize that, like, okay, like, here's a space of tasks that is now possible to do with AI models, you're starting to look for, okay, what is the right model for this task? Because I don't need Fable 5 to fill a form on a website. Because if that's going to cost me $5 and I can do it in 10 cents with this other model that's a better fit, then I'd much rather use that.

Alex Hutchings: 00:10:00

That makes sense. And so looking at some of your case studies, some of the companies that you work with, there's some really good brands on there. But for the people who are listening here, what is your typical customer profile and kind of why would they engage with Yotori and use your solution?

Das: 00:10:17

Yeah. So we productize our sort of web-agentic tech. So this computer use model and the web-agentic tech in a couple of different ways. One is a prosumer-facing product, which is called Delegate. So delegate is more like an AI chief of staff. It connects to all your internal apps or any websites that you want to log it into. And then you can start throwing tasks at it. And so then what, like the way this technology matters is that let's say you have a task like, okay, I got this support ticket. Can you read the email, what the user is saying? And then like go investigate on the Stripe dashboard or in the logs or whatever to figure out what. what happened here and what needs to be what needs to be done now now the first mile of this which is reading the email or like maybe doing a search api call that stuff is already kind of solved like there are good solutions for for that stuff out there but the last mile off of this which is which if i had to manually do it i would probably open a bunch of websites on my browser and like look at okay like what happened in this user's case what did they try what and look at logs or analytics across a few different dashboards, that's where our computer use model and our web-agent technology really shine. So it can do that in a way that's state of the art, that's frontier level. So that's one category of use cases. The prosumers who can just delegate tasks to a product that connects to their internal apps as well as any websites that they want to log into. We also offer an API which is more catered to enterprises. And that's where a lot of the larger, like any company that has a product use case that interacts with the long, heavy tail of the dynamic web. They would use our API to build automations on top of that. So a bunch of companies trying to automate filling forms of various kinds. Some companies who are using us to do end-to-end checkouts for various use cases. Yeah. some companies are using us to extract data or monitor data on let's say government websites whenever there's an update or something so use cases use cases of that sort and i know you're operating in a space it's like you know let's be honest the web is at

Alex Hutchings: 00:12:39

times a complete mess right it's kind of you know it is you know it's really unpredictable it's you know constantly changing all the time when you were developing more probably that the version two of the product What were some of the biggest technical problems you came up against in terms of training your agent against that kind of really unreliable data source?

Das: 00:13:01

Yeah, the web is a bit of a mess. The long tail of the web is set up in all sorts of quirky ways that's hard to predict and no sort of centralized entity controls it. There's going to be a little bit in the technical weeds, but when we started, one of the sort of hypotheses that we had was that, okay, like... The source code for how websites are laid out is in the underlying HTML and CSS and JavaScript. So that's the sort of input modality that our AI system should be looking at to predict where to click or where to type and what action to take on a particular web page. And so we kind of bet on that hypothesis and we doubled down on that and we spent about four to six months building our AI system on top of that. But what that led to was a whole sort of... rabbit hole of edge cases because the web is a bit of a mess like different like different websites and merchants and web website builders set up their websites in all sorts of weird ways where they may not be using the right let's say html element for the right use case they may have different javascript behaviors like the page loads in in different times maybe they're injecting some contact content dynamically into a certain element so initially what you read in the underlying html is different from what's finally rendered. And like, we're not like we had sort of gotten stuck in this whole of, okay, like, let's go through each one of these and try and sort of hack our way and like, implement a fix for each one of these. But eventually, it just got like untenable. It was it was like a whole slew of edge cases. And so then we took a step back. And then we were like, okay, what is the simplest solution here that we think would would scale much better. And we sort of did a bit of a pivot in the technical direction. And we started and went down this other approach, which is basically, okay, let's just take a screenshot. We don't care about the underlying HTML. So it is, let's rely on vision alone and not the underlying text or DOM or HTML. And yeah, if we take a screenshot and the model realizes the page hasn't loaded it, just wait five more seconds, the page will eventually load. And so with some simple heuristics and just a screenshot-based approach, that just scaled way better. We were no longer sort of fighting all the different ways in which websites are laid out and the model would predict. in coordinate space of where to click, where to type, and so on. And that helped us sort of scale way, way, way better. So it was kind of like the bitter lesson of training these web agents. And we actually have a detailed blog post on it on what the different edge cases and issues were of the former approach that the latter approach helped with. So that was a big unlock early on. Even now, I see other companies in this space who, yeah, like I think the debate isn't quite settled of which one is the correct approach. But for us... being able to train a computer use model that's looking at vision alone and predicting actions and what to do based on that um turned out to scale much better it goes back to your computer vision days at meta right it kind of goes back to you you know that's kind of playing back to your strengths and you know you

Alex Hutchings: 00:16:16

said something really interesting there about this simplifying you know actually there's so so many of these companies and engineers will get really into the technical weeds and over engineer a solution but actually stripping that back means you ultimately have a better solution now. And if you looked at your thesis you did all those years back versus now, in terms of kind of people's interactions with agents and people, the way that us as consumers or enterprises of that kind of instance are, you relying on agents, where does this go? Where are you kind of seeing this market and reliance and potential? potential risk you know whether there's such a reliance on a genetic ai where we stop to think for us we stop thinking for ourselves and it becomes this automation mode where do you see this going yeah i think that's a good question i

Das: 00:17:10

think what's happening both in the software development life cycle but also in knowledge work more broadly when interacting with agents is that like obviously these agents are becoming more and more capable and can like automate more and more workflows but there is still that last 10 percent that requires human judgment and review to get to a get to the right outcome and there is like just so much elastic demand in this in this space that like if you can automate 90 percent of a task people end up using your product or your tool or your model way more than than they used to six months back right And then that 10% becomes the bottleneck that requires human review or human judgment or taste, right? And yeah, so if 90% is automatable and that 10% becomes the bottleneck and you're doing 10x more of the first 90%, then demand for that last 10% also kind of goes up because you're now bottlenecked way more on that last 10%. So to be honest, across the board, I'm not, yeah, like I don't think the way this plays out is that it... replaces humans in a like in in these workflows i think there is still like we've seen massive advances across models from like these frontier models from opening and anthropic but i think there is still a gap in that last 10 percent that requires human judgment and review like what yeah like depending on the domain you might call it pace depending on the domain you might call it like judgment or yeah something else but i yeah i think there is still that piece that isn't quite easily automatable. Maybe there are some domains where that's less important. Like if you're, let's say, doing something that you would classify as busy work or somewhat mechanical, like writing, let's say, a small piece of code with extremely well-specified input-output specifications, then maybe that 10% is not there. But for a lot of other stuff... including like creative writing or software development more broadly not just code generation i think there is that human judgment piece of it which is which is hard to hard to automate i think it's a really fair point we've also from the conversations i've had on here and from the work i do in recruitment is that we've seen the kind of levels of what

Alex Hutchings: 00:19:33

used to be deemed as entry level and those kind of levels have been raised because a lot of the auto automation repetitive tasks that were deemed to be kind of your entry level role So... they're being replaced actually is enabling people and freeing their time up to deliver better value by doing parts of the role they probably enjoy more right that's kind of they're probably gravitating more to that because all the administrative side the heavy burden of coding getting into the weeds actually being that they've got their co-pilot

Das: 00:20:02

Yeah, yeah, yeah, exactly. And yeah, like the web and web agents, I think are still a click or two behind where, for example, coding agents are today. I think in the future that we are headed towards is where agents will become the primary drivers of action on the web. Like right now, yeah, the number of interactions, tasks, workflows that are happening on the web are heavily bottlenecked by the number of people and what our bandwidth looks like, right? Like when agents start... doing a bulk of this i think that is going to be a massive productivity unlock and they will become agents will become the primary drivers of action on the web with humans in the loop in some capacity for that for that last 10 percent let's say i suppose that's what i'm keen to understand obviously you're going back to your point earlier about web data and you know agents

Alex Hutchings: 00:20:52

on on the web being what your prediction is there's going to be they're going to ultimately be heavily influencing what's on the web. But from your Tori standpoint, in terms of validating the information that's on the web, you know, what steps does your Tori take from a governance, from a source of truth standpoint? How do you monitor the validity of what you're pulling out of the web?

Das: 00:21:14

Yeah. So we are, maybe one distinction I'll make is that we are not a search company. We are not building our own index. We're not sort of, yeah, like, like chunking, embedding, indexing the web. to make information retrieval easier. And the reason I bring that up is that it isn't an issue for us that there's, let's say, some offline copy of the web that needs to be constantly kept fresh and so on. And so anytime someone issues a prompt to our APIs and our agents are going out to the website, to a set of web pages, let's say, to execute on that task, we would sort of, they can inspect. the entire trajectory of actions that the agent took on those websites. So there's no notion of that, okay, like maybe the agent hallucinated that like this webpage said something that it actually didn't.

Alex Hutchings: 00:22:09

Okay.

Das: 00:22:10

Even then, when the agent is looking at a webpage, there's a chance that it sort of misreads something that isn't there. And that's on us. Like we work very hard to make sure to reduce the likelihood of mispredictions of those kind. But the entire trajectory, everything is inspectable. And of course, we play by the rules of the web of like, okay, like whenever a certain website doesn't want bots or agents on their website, like automating things. Yeah, like we would play by that. And same for like subtasks, like for example, authentication and payments. Like right now, there's a lot of machinery built to make sure that whenever someone is doing a payment on the web. like there's machinery built to make sure the entity on the other side is a human and not an automated agent i do think that that will change over time and there are already some protocols in development that are enabling agent uh payments and instead of human uh human payments but yeah we're not trying to sort of sidestep that in in any capacity no

Alex Hutchings: 00:23:11

i appreciate clarifying because I think it's... Because there's also a lot of talk about kind of AI agents, and there's a lot of companies in that search space, right? They're extracting web data and obviously packaging it up for other companies. But actually, that's where the source of truth issue and governance comes from. But actually, as you said, there's websites that are restricted from doing it anyway. But B, it goes back to your earlier point, the models you've built really does compete with the top frontier labs anyway in terms of validity and hallucinations are at a minimum anyway. So I guess that's your pitch.

Das: 00:23:44

Yeah, exactly. And one other thing I'd add is we will probably see more and more agentic microtransactions starting to pick up. And what I mean by that is that right now, the web is basically, a large part of the web is built to monetize human attention. Like when you visit a certain website, you might see a certain number of ads and you're spending time there. And so that's how they're monetizing it. But if it's not human eyeballs and if it's agents visiting websites. then that economy needs to change to some extent. And so a version of what that may end up looking like is that, okay, anytime an agent visits a certain website, there's a programmatic automated microtransaction of like, okay, it pays a few cents for access to that webpage. Now, if it was a human visiting websites and if we had to pay like a few cents for every website we visited, that's like untenable, that's overwhelming, that's too much. foot. agents to programmatically do it i think it's it's trivial so we will see that we will see that pick up more and more and it'll be almost like a parallel economy of like agents operating the web for people looking into your tory going to the website you know what

Alex Hutchings: 00:24:57

can they expect to see from you guys over the next 6-12 months i suppose it's the rate at which the space is kind of exploding is, you know, it's probably hard to think that 10 years back, you think we'd actually be at this point now, but what does the next six, 12 months look like for you guys?

Das: 00:25:14

Yeah. So actually we just shipped a new playground on our website. So people should go check that out. It provides sort of an easy way for anybody to play with Navigator, which is our computer.

Alex Hutchings: 00:25:25

Is that the first page where you can actually, it shows it scraping the... where it basically shows you how it's working.

Das: 00:25:33

Yeah, exactly. Yeah, really interesting. It's on our website landing page. We have a pretty big new model release coming up. Maybe if this episode goes out two weeks from now, so maybe like a few weeks after that, that we are super excited about. It is going to be a lot more capable than our current model while still sort of hitting that plateau of latency and cost. And so we are super psyched about it.

Alex Hutchings: 00:26:04

Awesome. Yeah, I think Yotori is so interesting. I think the way it's positioned, let's be honest, you're just over two years old. And actually to have that trajectory, the people you hired, the problems you're solving, actually the customers, you've already got signed using the product. It's a huge testament to you and the team. For everyone listening here, check Yotori's website out. They have got a really interesting, very easy to navigate front page that shows you the product in. action but that's yeah thanks for coming on this morning it's been really interesting i'm looking forward to this one going live awesome thanks alex thanks for having me thanks again