EB-7: Greywing - Transcript

[00:01.945] 👩‍🎤 Ate-A-Pi: So Rishi, what is Greywing?

[00:05.906] Hrishi Olickel: And Greywing, I think, started out as just two people because I met this guy who had about a decade and a half in commercial shipping. I knew nothing about commercial shipping, and except for the fact that I lived in Singapore and there's all these massive ships on the harbor. He was one of the best salesmen I'd ever met, could sell ice to an Eskimo. And so he said, hey, there's a lot of opportunities in shipping. Let's go after it. Let's build something together. And I said, yes. And so for the first year of Greywing, it's like five years old now.
It was just the two of us, right? And we built the first product, got it to six, mid-six of revenue, just building and him selling by day. And it's just grown and changed quite a bit over the last four to five years. But if there's a through line, through all of that, I would say it's really just around trying to solve commercial shipping problems. Most of those problems are related to the data that they have that they can't process. And there's a lot of different ways that you can...
arrange people, improve the lives of people on the ships, optimize the kind of arteries of the world that move things around. And so that's kind of been the core philosophy that we work with.

[01:12.641] 👩‍🎤 Ate-A-Pi: So your co-founder, the salesman being Nick Clark, right? And so commercial shipping is basically our tankers. I mean, our container shipping, right? Container shipping, is it also oil tankers and the rest or?

[01:30.366] Hrishi Olickel: It's everything really, right? So containers are a small part of it. Even crews can be considered commercial shipping in a certain way. But it's also, you know, chemical tankers, product tankers, right? Oil tankers as well. And there's all sorts of research vessels, all sorts of things just fall inside of commercial shipping.

[01:47.213] 👩‍🎤 Ate-A-Pi: So anything non-military basically, is that a good way to look at it?

[01:52.934] Hrishi Olickel: I think anything non-military, but then there's also certain size, usually. I mean, it's kind of a weird comparison, but it's kind of like rag, right? It's one of those terms that came about as a way to describe a phenomenon. So it tends to be loose around the edges, right? People in the industry sort of know that it's above a kind of size where you'd look at it. And then there's always a gray area where people look at a ship and go, ah, that's probably not included, or it is.

[02:17.725] 👩‍🎤 Ate-A-Pi: I see. So when you say size, is that like tonnage? So it's like 10,000, 10,000 deadweight tons, 50,000, something like that. Right. And so, Greywing's first product was to help people charter these ships.

[02:23.178] Hrishi Olickel: Yeah, exactly. Yeah.

[02:36.834] Hrishi Olickel: Ah, so Greywing's very, very first product, and this was sort of when the physical security market, which very simply put, comes out of the need of, that's created by piracy, right? So there's all these dangerous parts of the world, you know, America patrols some of our oceans, but there are just still these dangerous parts of the world that ships need to go through. There's piracy in those waters. And so vessels have gotten to either hiring escort vessels that can escort them through those waters or hiring people to get on board the ship and sort of protect them, right?
That was a very fragmented hyper local market at the time. It just started commoditizing, because the problem is wherever you go, the suppliers are by nature local, right? If you're passing by Nigeria, you can't bring your own security. That security has to come from Nigeria or Gal or somewhere else, right? So we built a digital marketplace to basically pull that market together to add some performance tracking history, you know, compliance and a bunch of those things. And that was the first product.

[03:36.009] 👩‍🎤 Ate-A-Pi: Wow. So you would as so who would be who'd be the user of the product, the captain of the ship or

[03:43.942] Hrishi Olickel: It's usually people on the shore that sometimes make these calls and sort of coordinate. The captain has final say in most of these decisions, but they usually have enough decisions to make anyway. But usually people on shore that go, hey, this vessel is going to be transiting through this area. Let me go over there and look for suppliers. I'm going to coordinate with those suppliers as to when those ships are going to meet my ship and escort it in and out of the area.

[04:08.425] 👩‍🎤 Ate-A-Pi: And then, and then is that, is that, can they like leave like reviews or whatever? Like, you know, this, this particular supplier was good. This particular supplier was late all the time, like that kind of thing.

[04:21.27] Hrishi Olickel: That is exactly it, right? But in terms of commercial shipping, everything's they like a lot more forms and a lot more information. So you had engagement reports, and massive amounts of information on how each specific part of that escort was performed. And that's your review, right? We even had like a star system.

[04:40.345] 👩‍🎤 Ate-A-Pi: Right. Oh, wow, you had a star system. So one to five star system. And do you still have that product right now?

[04:43.156] Hrishi Olickel: Yeah.

[04:52.666] Hrishi Olickel: We have that product effectively somewhere. But what effectively happened is we went through, you know, you might have heard this pandemic, right? And security demand basically went practically to zero, right? Like the bottom fell out of that market. And so we'd actually plan to sort of move out of security anyway, because we saw that commoditizing, we thought we had an extra year, we didn't, right? And sort of what we were moving in the right direction at the time.
So we did was we took all of the different pieces because the biggest problem with shipping, honestly, not that different from AI like a year ago, is a lot of early building blocks needed to get built before you could build anything on top of it. Right? So let's say you wanted to build a marketplace or an engagement system, you still needed a really good routing engine that could predict what the router ship was going to take. You still needed to be able to connect to positioning systems for ships to be able to pull that information in and you needed a good ports database that was constantly being enriched so you knew.

[05:30.646] 👩‍🎤 Ate-A-Pi: Mm-hmm.

[05:46.57] Hrishi Olickel: where the ports and where the regions were. None of those things are directly related to the service you're offering, right? They're almost like the building blocks you need to have, the same way that you build an AI now, and you also need to figure out how to do prompt templating, testing, and all of these other things. And so we built all of those things from the ground up. And we decided we were just going to take those things and repurpose them for crewing, which became sort of the biggest problem during COVID.

[06:11.497] 👩‍🎤 Ate-A-Pi: Right. And crewing meaning, um, you know, let's say you have a vessel, it needs a crew of 10, 15 people. They each need to have certain skillsets and without certain, certain skillsets, it cannot sail or something like that.

[06:27.958] Hrishi Olickel: That's exactly it, right? Vessels are usually, and they're not by definition, there's not a lot of room on the vessel. So they're usually understaffed in a certain way. And the way that this works is a bit more comprehensive than the airline industry. The shipping company is responsible for you, in a lot of cases, from the minute you leave your door at home to when they return you, right? And you're usually doing three months on, three months off, six months on, six months off, and you need to get rotated out.
And that rotation happens, has to happen like clockwork, but it can't really disrupt the vessel schedule or cargo or any of those other things. And what happens if you don't do that thing, which is what happened during COVID, is travel lockdown, or countries weren't accepting anyone. So you really couldn't get people off vessels, is people get tired, above a certain amount of time at sea, all sorts of psychological issues happen, suicide rates skyrocketed, and all of these problems happen.

[07:25.437] 👩‍🎤 Ate-A-Pi: Is the suicide rate for a crew high or low? I know in the airline industry, pilot suicide rates are actually quite high. It's actually quite a significant number.

[07:37.866] Hrishi Olickel: Yeah, that's a, I read something at some point that this was more to do with how, you know, you couldn't really seek mental health or mental health as a pilot, right? But in the, among seafarers, I think it varies really from region to region to class to class to class. But every indicator shot up during COVID, right? Because people on the shore couldn't get to their jobs and they really rely on that income.

[08:00.567] 👩‍🎤 Ate-A-Pi: Oah.

[08:05.886] Hrishi Olickel: and people on vessels were at some point, you know, like overstaying their contracts by a year and a year and a half, right? And you had all sorts of problems come up.

[08:12.621] 👩‍🎤 Ate-A-Pi: Oh wow.

[08:16.506] 👩‍🎤 Ate-A-Pi: So how did you start using AI in your products? Like what was the first AI product that you guys started to do?

[08:26.126] Hrishi Olickel: Ah, so it take you back just a little bit more, right? My original background is in a lot of ways, AI and CV and quite a bunch of stuff, right? So I'd always been the person inside the company. When we went through a deep tech wave here in Singapore, we've been through different AI waves that always resisted having AI inside of your company, your product, unless it serves a very clear use case or improves something directly, right? When LLM sort of launched,
Like, you know, I'd been following GPT-2 and GPT-J. There was an old subreddits where, you know, GPTs talk to each other even way before each other. And what became very quickly obvious was that this was a revolution in LP. Like I was kind of short sighted at the time. But because for me, I was laser focused on the problem of NLP, because most of shipping's data is very, very human, right? It's all sitting in PDFs and emails and all of those things. And that's where your engagements are. That's how you know how your people are doing.
how much profit you're making, what's happening to your company. That's all human information in 8,000 formats, because shipping is the original decentralized industry. So when the GPT transformer model started coming up, what I saw them as was a way to do better NLP. So that was the first product that we released. The first product that we released was a way to pull structured data out of your emails.

[09:30.041] 👩‍🎤 Ate-A-Pi: Mm-hmm.

[09:48.654] Hrishi Olickel: and related to suppliers, right? Because a shipping company, for example, day to day per person might send and receive 100, 300 emails, right, that are relevant to tracking the price of something, purchasing goods or service, getting like a disbursement accounting form, all of these different things. And they're just sat in PDFs, and they're sat with PDFs inside of emails, inside of something else. So that was the first part of the release, is just being able to pull structured information out.

[10:13.565] 👩‍🎤 Ate-A-Pi: Mm.

[10:18.558] Hrishi Olickel: We actually went one step further, which is because these things started to become chat models almost as early. We also built effectively an interface that would automate communications. So you could send in the information to a shipping company about something. And 99% of the time, the first response from the shipping company is you forgot something, right? You didn't include something that we really needed. You're ambiguous about something. And that's a small place where a language model can...

[10:33.133] 👩‍🎤 Ate-A-Pi: Mm hmm.

[10:48.182] Hrishi Olickel: can easily make that decision even way back with the first version of ChatGPT. And so that was the first product.

[10:55.629] 👩‍🎤 Ate-A-Pi: So typically something like, you know, you're always supposed to tell us like, what time the crew is supposed to unload, you know, whenever you send us a message, because, you know, every single time you have to include that piece of information. And you just randomly like, hey, you know, can you just do this for me? And you didn't include that. And they're like immediately like, no, we can't accept anything unless you say this, say like this magic word, right? So, and then you have to, and so the LLM could just do that for you basically.

[11:24.606] Hrishi Olickel: Yeah, and even today I'm surprised that there's not a ton of people doing that because I think that's an easy win in a ton of industries, right? Because I spent three, three and a half years in insurance and reinsurance. And if I was there, I would be doing exactly that because they had the exact same problem, right? Either on the human side, when claims come in, when patients send you information, usually your first response is, hey, you forgot to include the bill or the document that we need to be able to process it. Or on the other side, for the insurer side, they send you information and they're always missing information, right?

[11:56.049] 👩‍🎤 Ate-A-Pi: It's funny because it's a natural language API, right? It's an API between, you know, a human being and an organization. And that API, you need to have certain requirements, right? And if those requirements are not met, you have to send it back and you have to say like, you know, look, you didn't meet the requirements of the API, so you need to send me in this input data or else I can't, I can't, I can't do the work, right?

[12:04.82] Hrishi Olickel: Exactly.

[12:17.486] Hrishi Olickel: I mean, that's a really good way to put it. The way that we kind of seed and isolated a few places were, look, for some reason, we've taken to putting a lot of processes in place in really large industries to get humans to work like computers, right? So insurance shipping, quite a bunch of places, there's humans that are just working through checklists and being kind of being forced to work in a lot of ways like computers do, right? Practically human APIs.
Those are areas where a good chunk of that load could just be taken off their hands. And those are also the same people still making the decisions, but they also have to do all this other work.

[12:54.473] 👩‍🎤 Ate-A-Pi: Yeah, because basically there is in a large company, you need to reduce variance, right? And so you need to make sure that's why you have all these checklists and APIs and et cetera, et cetera. You wanna reduce all of that variance to something which is manageable, but you still want like the human touch, which is, hey, something is going wrong. I need to call it in, you know.
So when you deal with, okay, so now I've noticed that you have a couple of products. You have a CGPT. So what is CGPT?

[13:30.758] Hrishi Olickel: CGPT ended up becoming more of a catch-all term for the different kinds of AI services that we ended up raising, because once that got off the ground, I think it was the most successful launch we'd done up until that point. We just started getting hit with everything that people thought AI could do now or couldn't. And some of them were low-hanging fruit, so we were very happy to pluck them off. Some of them were bigger things that were only now just getting around to solving. And CGPT ended up becoming a catch-all.
We've since then rebranded it to Proteus.

[14:05.633] 👩‍🎤 Ate-A-Pi: Which is a copilot, which is kind of a copilot assistant, right? Is that right?

[14:08.83] Hrishi Olickel: Which is kind of a co-pilot system. We find that interface or that branding makes it a lot easier for people to get an idea of what something does. So almost like the iceberg thing, there's quite a bit under the surface that we explicitly do where it's just AI powered processing. No human's ever going to interact with that system, but it's just another block in a data processing pipeline. And then there's human facing stuff. And that usually falls into the branding of the co-pilot.

[14:38.741] 👩‍🎤 Ate-A-Pi: So let me try to explain Proteus. So Proteus is basically, if you are a crew member on a ship, you get access to it. And you can ask it questions like, hey, what is my schedule? Where am I going to? What's my next stop, et cetera? You also have access to manuals for the ship or manuals for where is the deck? Where do I find this particular object or something like that? Is that correct?

[15:07.922] Hrishi Olickel: Ah, no, it's the other way around, right? And we need to get better at our messaging. What Proteus does is effectively, it's a co-pilot for the vast amount of data that a shipping company has. It's designed for people who sit on the shore, whose job it is to manage these ships, right? And that means that they can quickly pull information out, like, you know, hey, I need a crew replacement that has, you know, these certifications and these things and these things and these things, right? Or I have a vessel that is calling into this particular port, I need to get all the documentation.

[15:10.541] 👩‍🎤 Ate-A-Pi: OK. Yeah.

[15:23.082] 👩‍🎤 Ate-A-Pi: Okay.

[15:37.97] Hrishi Olickel: all of our history of working with suppliers there, because I'm trying to do something specific. Right? Or somewhere else and you go, hey, I need to find...

[15:42.667] 👩‍🎤 Ate-A-Pi: I see.

[15:46.075] 👩‍🎤 Ate-A-Pi: So it's the charter, like the person who's chartered the vessel basically, who used it.

[15:50.426] Hrishi Olickel: That is charters, ship managers, and we honestly will be here for like three hours if I start going into the layers of the shipping industry because there's so many. So it is sometimes charters, it is sometimes ship managers and ship owners, effectively people with data about a vessel that need to manage that vessel, right? It makes accessing that a lot easier. And of course, once you can access it, you can also use it to do all sorts of planning.

[15:56.586] 👩‍🎤 Ate-A-Pi: Right.

[16:07.349] 👩‍🎤 Ate-A-Pi: Right.

[16:15.245] 👩‍🎤 Ate-A-Pi: So do you plug in directly into, let's say your client's system, like let's say it's a shipping company and they have certain databases, et cetera, do you produce plug-in directly into those?

[16:27.178] Hrishi Olickel: That's exactly it. We plug in either directly to their systems, depending on the size of the company, because shipping's one of those industries, same as insurance, that went through one wave of digitization. So there's usually either a homegrown or a bot software system that sits in front of that data, like an ERP. And so we usually plug into those ERPs when they exist, when they don't, we plug into Excel sheets.

[16:50.438] 👩‍🎤 Ate-A-Pi: Right. Right, so basically, the user can make a query. And then you have an LLM that processes that query, breaks it down into, hey, which tools do I need to use, or which databases, et cetera, do I need to call? And then goes off, calls those databases, pulls back the data, formats it into a response for the user, and then returns it to the user.

[17:13.771] Hrishi Olickel: That's exactly it.

[17:15.405] 👩‍🎤 Ate-A-Pi: Right. And how has it been, you know, when did you introduce the product? How has the take up been? How has the response been?

[17:22.766] Hrishi Olickel: So we introduced the product, I think, you know, six to seven months ago. And honestly, the response has just been very positive and very overwhelming, right? I know shipping companies still move pretty slow for practically any other industry, but this is, you know, we saw sales cycles or just generally update cycles go down by massive, massive mods, because I think it resonated the same way that chat GPT or other AI products is it captured people's imagination. So usually we'd go in and work with one particular team inside of a shipping company.
But in this particular case, everyone wanted to be in that meeting. Or we'd go into have a meeting with a technical team and some other teams would hear about it and they'd come to us and go, hey, can we be part of the trial or can we be part of what you guys are doing? Because I think just that question answer aspect, or I guess there's smarter larger companies doing a lot of marketing for us that is also resonating and connecting in this way. But the response has really just been massive.

[17:54.615] 👩‍🎤 Ate-A-Pi: Mm-hmm.

[18:17.117] 👩‍🎤 Ate-A-Pi: I see, I see. And is your client base mostly in Singapore, in Asia, worldwide? Is it, you know, are there particular geographic regions that, you know, your clients are? And I know the shipping industry is vast and everyone sits in different places, but where do the people use the software?

[18:31.756] Hrishi Olickel: Yeah.

[18:36.342] Hrishi Olickel: So the people that use the software, it is worldwide by definition, but there are hubs, right? So the people who use the software usually sit either in Singapore, you know, parts of Europe like Hamburg, or Dubai, and the US, right? Those tend to be your general sort of clustering.

[18:54.357] 👩‍🎤 Ate-A-Pi: Right. And have you noticed take up, which is particularly aggressive in one region versus another or kind of generally everyone's kind of using it, you know, everyone's kind of picking it up at the same time.

[19:06.626] Hrishi Olickel: So everyone is picking it up at the same time, but I will say that the different countries and different jurisdictions just have different patterns of how things get taken up. Europe, for example, understandably very concerned about regulation. So those tend to be the teams that we interface with first. One of the companies that we work with, for example, the team we speak to sits in the exact same floor as the compliance officials for the country. So that changes the pattern of how you interact.
There are other places that are a little bit more, you know, move fast and break things. Let's put this in the hands of people and then we'll figure it out. But it's more of a cultural thing.

[19:44.637] 👩‍🎤 Ate-A-Pi: Interesting. So one European strategy has been we are going to be the hub for regulations. So everyone has to come to us first. So they actually pulled it off, right? So, you know. You have another product called Ocean Oracle. And I think I've seen you kind of like demo some of this. Like you guys basically took kind of schematics and engineering diagrams, and you did like some sophisticated thing and you made it like...

[19:53.43] Hrishi Olickel: Hehehehe

[20:13.209] 👩‍🎤 Ate-A-Pi: queryable using a GPT or something like that. Like, what is it, what did you guys do?

[20:18.862] Hrishi Olickel: Ah, so that honestly came about as a result of so we are, you know, in our DNA, at least originally, we're a vertical SaaS company, right? Meaning, you know, we take existing tech in a lot of ways, and we figure out how to verticalize it to a specific industry using industry knowledge. That was our primary area. And I guess we're not anymore. But Ocean Oracle was one of those things that started it, right? Effectively, what happened was, customers came to us and said, you know, we have PDFs, we have information, we'd like to, you know, upload to your AI. They're really valuable, really important.
And we said yes, right? Because in some ways, we've been responding to the standard marketing and we've been responding to the general perception that AI can do that now, right? Chat with PDF and all that. So we expected there to be either something off the shelf or something that we could work with someone else with the IP, then we could verticalize it, make the adjustments needed for shipping. So we said yes, which was our mistake, but it's also the thing that ended up ended us up in this position. And then they send over the PDFs. And these are, you know,
per document, 5 to 6,000 pages of incredibly dense schematics and diagrams, right? Some pages you OCR and you get basically blank space, right? Nothing comes out, right? And again, these are really important. And the questions they sent over as, you know, can you help figure this out are not simple questions that you find in the document. These are things that would require you to go across the document, right? Like, let's say there's a fire in a specific area.
you would first need to figure out that fire would belong to this particular machine, figure out what the evacuation procedures are for that and something else, and also things you don't want to get right. Sorry, things you don't want to get wrong, right? So we were in that dilemma and we decided, hey, we're just going to go ahead and try to solve that. So what we ended up figuring out is there's a bunch of key pieces missing from the AI infrastructure at the time. I think now there's very smart people building it, is we have no way to visually index information.
We can index text information with embeddings, sort of, where you can sort of search through them. But if there's visual relationships and information, there's no way to index it so you can find it later. And B, there was no sort of good way to do cyclic retrieval, which is thinking about the way the humans do it. If I asked you to find information in new text, you almost always will find some things, use those as reference and look more, until you get to a point where you say, hey, I've looked enough.

[22:22.241] 👩‍🎤 Ate-A-Pi: Mm-hmm.

[22:41.93] Hrishi Olickel: So there was no way to do that. And we thought that was the big key to getting more comprehensive retrieval. So that's what we ended up building with Ocean Oracle, right? It's a way to visually index information. And also for an LLM, whatever it is, that is part of the chain to go, hey, I'd like to look more. And here's where I look more.

[23:03.021] 👩‍🎤 Ate-A-Pi: Interesting. So do you guys use any open, like what are the underlying LMS that you guys use, like GPT-3, 3.5, 4, Opus, Mistral, like what do you guys use under a mix or like, you know, what are the ones that you've worked with in production?

[23:22.286] Hrishi Olickel: So it's definitely just a big mix, right? Because we definitely have, you know, a GPD 3.5, we have four for turbo. We also use Claude, you know, Claude and Shen way back when that was a thing. Now we're starting to switch to Haiku, sort of modern models. We did try out Gemini for a while. And we consistently keep looking at open source models. And at one point, we had moved some parts of our pipeline to open source models. I think just the pace of everything moving.

[23:46.978] 👩‍🎤 Ate-A-Pi: Mm-hmm.

[23:49.026] Hrishi Olickel: Today, I think it's about 90 to 95 percent centralized sort of large models, but we expect that's going to change.

[23:55.933] 👩‍🎤 Ate-A-Pi: I see. What is the process? So I have a bunch of people who are building, I met a bunch of people who are building open source models, right? And so they would all come and end up pitching to you as the CTO of a company that uses models. Like what is a thought process that you would have when you evaluate, let's say a Mistral or someone else who comes into you and says like, hey, we have this Mistral large.
like, you know, what would you look at that model for it to do for you? What is the process that you would evaluate this model?

[24:35.038] Hrishi Olickel: That's a really good question. The way that I would put that is, you know, there's your standard sort of graph and your standard sort of spectrum of task complexity of the different things you need to get it done. And then you've also got the general distribution of is this human facing or not, right? So I think almost any model at any size can be fit for purpose as long as it falls in the right part of that spectrum, right? Meaning it can do really simple tasks, but it's either really cheap to run, really small, easily embedded somewhere else. And that works, right?
or it's human facing and it's expensive, but it's fast. Because when it comes to things that face users, you're almost always looking for latency, right? And Mistrules done a wonderful job there. I don't know what kind of black magic they've got going on. But I don't know if anyone's noticed it, but their time to first token is insane. You can throw in practically, you know, like 20,000 tokens into the context window, and you get the first token out almost instantly, compared to everything else. So there's a bunch of different things that we use to sort of evaluate that.
But I will say for us, the goal with open source models is eventually that we can embed them client side. Like that's the reason we push forward with open source models, especially in enterprise that has huge implications for privacy and data security, right? When you're doing and the number of tasks that need plain text sensitive information are still simple enough that we're right at the cusp of open source models being able to do that, right?
you don't need very high levels of intelligence to turn something into structured data, right? Or make a transformation or sort of build a query from sensitive data. And so I think the next big jump is going to happen for open source when these models get embedded client side.

[26:16.353] 👩‍🎤 Ate-A-Pi: Right, so as you said in your case where a lot of emails, you just need to check that certain things were filled in and if they're not filled in, just respond. And that's not a particularly very intelligent task that you need a GPT-4 turbo or whatever to do. And it's very sensitive because it's reading your email. So you wanna be aware of that. So another question.

[26:30.037] Hrishi Olickel: Exactly.

[26:35.849] Hrishi Olickel: Yeah.

[26:42.509] 👩‍🎤 Ate-A-Pi: When you have, again, like an open source company, like a Mistral or whatever, approach you and they have this open source model which they've already published, why would you as an enterprise choose to purchase their services? Like what kind of services do they offer you that you can purchase versus just downloading the model from Hugging Face? Like what are the typical services that you would expect from an open source provider for you to want to purchase some kind of deal with them rather than just downloading the model?

[27:12.462] Hrishi Olickel: I see what you mean. That changes a fair bit, especially because I don't think a lot of providers have been kind of laggard in providing a suite of services on top. And so then companies like mine end up building those things, right, because you need them because you've hit production and you absolutely need those things. So then implicitly, the providers are competing with your suite of stuff that you use internally, right? And that might be things like weight limiting, token tracking, usage metrics, just all the quality of life things that exist on top, right?
playgrounds and everything else, right? When they don't exist fast enough, companies end up building it internally. And once you do and providers come to you, they're effectively competing in some ways for that quality of life layer. But the biggest one, I would say the biggest reason to use their services or to have them run the model is if they can make use of economies of scale. Right? Like, you know, companies like mine, and even if it's not companies like mine, just much bigger companies included, right?
I still think that providers are always still going to be able to optimize really, really well for stuffing GPUs with as many tokens as possible and pinning them to zero and sort of optimizing for bandwidth, to eventually be able to deliver things. Because most workloads, no matter how many users you have, are usually going to be bursty. And I think the big difference there in some ways is kind of choosing to use a provider like Varsel or running your own servers, and we actually do both.
So we know. And so sometimes you're just saying you don't want to do that DevOps work, especially in the early days.

[28:44.459] 👩‍🎤 Ate-A-Pi: Right, right, right. So.

[28:50.613] 👩‍🎤 Ate-A-Pi: Let's talk a little bit about Lumentis. So Lumentis was a project, I think that you published on GitHub. So it was kind of like more of a personal project. It generates documentation, right? It generates kind of you dump in documents in there and it generates like nice, easy to read online documents.
with kind of like, et cetera, et cetera, and sections, and you can click through. And it went viral on Twitter and on GitHub. And I saw a comment, you just eliminated the entire technical writers job in a day. So what happened with Lamenters? How did you come about to think of the project, and why did you go forward?

[29:32.59] Hrishi Olickel: Hehehehe

[29:42.154] Hrishi Olickel: Oh, man. The Lamentus project effectively came from, and I did not expect that to resonate as much as it did, because it was really trying to solve my specific problem that I had using my specific situation that I had, which is effectively this, we build and launch quite a bunch of things internally, and a bunch of things happen internally, there's a bunch of projects going around, there's a lot happening. And what all of those things are missing is documentation.
And the biggest reason they're missing documentation is the initial step of creating documentation is just a massive piece of work. Someone has to understand everything, go through all of it. But on top of that, they've also been able to build the diagrams and the structure and all of it to make good documentation. Once you have it, keeping it up to date is less work. I know people might disagree with me on that, but it is less work. And creating that is so much work that it usually goes without it. When it goes without it, what ends up happening when you have people that you need to onboard to that project
you need to distribute that knowledge internally as a company or to people is you have, you know, very similar to what we're doing now is you have meetings about it. Right. And so after the sort of fourth meeting where I had to re-explain a project, I went, there's enough information in these two meetings to build good documentation. Right. And I also know that having someone on the other side who's constantly asking questions about what doesn't make sense to them, you know, what actually doesn't fit.
all of that could be turned into really good documentation. Right? Or at least it felt like the content was there. So I threw this thing together like overnight. I honestly didn't expect it to work. I expected it to be kind of a demonstration of the fact that language models could do this some point in the future, but are not at that point yet, because it's a massive amount of information you're dumping into context, right? But Opus blew me away with how well it actually, you know, Cloud 3 Opus blew me away with.
how well it did on the documentation. I was reading through these and I kept going, well, this is right, I haven't seen a mistake just yet. It's not wrong yet. And then Sonnet blew me away because it could do the same thing and Haiku at the end really blew me away because it could do the same thing but for less than 10 cents.

[31:54.397] 👩‍🎤 Ate-A-Pi: Yeah, the recall, right? The faithful recall is quite something to see in those models. Yeah. So yeah, and what do you think, like, when what has been like the one or two anecdotes that users have come back with?

[31:55.963] Hrishi Olickel: So that's the story.

[32:04.206] Hrishi Olickel: Exactly.

[32:20.749] 👩‍🎤 Ate-A-Pi: from using some of the tools that you have put at their disposal using the AI tools. Sometimes anecdotes, rather than data, anecdotes can be interesting or illuminating on how they actually interact. So have there been any interesting, like, oh, you know, I was in a fire and I looked at it and it helped me out, like those kind of moments or stories?

[32:42.346] Hrishi Olickel: Oh, man, I'm trying to remember stuff that is, you know, and just making sure I sanitize some of the proprietary things in my head. I think one of them effectively that was really interesting is we had a user who came and sort of test the Ocean Oracle product. And I was looking at technical manuals for the kind of technical engineers. And he said, it did a great job. Right? He said, I asked about seven questions and it did a great job. And we were like, that's awesome. That's really good. But you seem surprised. And he goes,
Well, that's because I asked the questions that I usually asked to sort of as part of stumping new engineers, because they wouldn't necessarily know it, and it's really hard to find. Right. And so we've also found this kind of divide between users effectively, where quite a bunch of users, and these are surprisingly in the minority, will be very, very nice to the system. Right. They don't really push it. They sort of ask very simple questions, you know, let's say things you can expect your computer to do already,
take it up half an order of magnitude. And then you've got users that start with the most complicated thing they can imagine and then work their way down. But yeah, we've just seen that general spectrum. I was also talking to a friend of mine the other day about the fact that I think a lot of AI products go into this kind of loop where, and Lumentis is also a good example of where that happened, because I showed it to a bunch of people and a bunch of people used it.
and they said, this is amazing, but it really doesn't let you update the documentation. And I went, yes, but that is another massive open problem. And I've been talking to a lot of AI companies, it seems like they all end up in similar loops where you do something really well, but because it triggers that right part of the uncanny valley or the rest of it in your head, you almost immediately jump to more and more complicated questions and that starts that loop of sometimes disappointment.

[34:20.568] 👩‍🎤 Ate-A-Pi: Right.

[34:40.157] 👩‍🎤 Ate-A-Pi: Yeah, I mean, it's the autopilot versus full self-driving, right? Like, okay, you know, you can do cruise control. All right, I wanted to park the car for a week. And like, no, you know, that's not going to happen for another 10 years.

[34:47.4] Hrishi Olickel: Exactly. Yeah.

[34:56.642] Hrishi Olickel: That's exactly it. I mean, granted, we've tried to stay with our messaging on the cruise control side of things. We're like, look, we want to be delightful cruise control instead of work up from there.

[35:10.142] 👩‍🎤 Ate-A-Pi: All right. So I think how important is latency versus accuracy? Like, I mean, do people want like, you know, is that, because GPD4 can be pretty slow, because especially if you're using the API and you don't have a reserve instance, you know, it's fairly, there's a lot of latency there.

[35:28.898] Hrishi Olickel: So latency and accuracy are very strongly connected through more ways than I'd initially realized. One big one is just UX, right? And the way that people see these systems is intelligent systems. And the reason I say that is because we've now come to realize that for users, if you take longer, they expect you to be better.

[35:48.781] 👩‍🎤 Ate-A-Pi: Oh, oh wow. Okay.

[35:49.886] Hrishi Olickel: Right? Because to a user, the system is thinking.

[35:55.454] 👩‍🎤 Ate-A-Pi: Ah.

[35:55.678] Hrishi Olickel: Right? If you take eight seconds to come back to a user, they expect a significantly higher quality of answer or service than if you got back to them in 10 milliseconds.

[36:06.373] 👩‍🎤 Ate-A-Pi: and human-computer interaction. Wow.

[36:09.298] Hrishi Olickel: Exactly, right? Because we've gotten so close to general intelligence and how humans are. And that intuitively kind of makes sense, right? When you think about it as a user, because for us, if we can start putting out a token almost immediately, users tend to react very differently. If we make them wait for 20 seconds, they expect the entire thing to be done and the questions get more and more complex and you get less buy-in, right? Almost everything we build,
it's also just about, I think there's a lot of ways you can also balance latency and accuracy, right? Because there's a lot of things you can do to give users more of an understanding of what's actually happening. And I think that matters more than the answer, because the more you involve people in, and you might have seen this in some of the demos for Ocean Oracle or things, is that it takes you through a sort of chain of thought, it sort of flips you through a bunch of pages that are currently being looked at. And all of those things help with...
user buying because they feel like they know how the system is thinking and they do by looking at it, right? So quite a bunch of cases, they're out before the answers even showed up, because they got what they were looking for. Or the answer shows up and very, very rarely it's wrong. And when it's wrong, instead of users coming to you being like, hey, you had a system for me and it gave me the wrong answer, they come to us with, hey, it gave me the wrong answer, but I know why because it was looking at the wrong pages. And that's an understandable mistake because I know that shows up in those pages.
So there are a lot more bought in, they're a lot more willing to work with the system.

[37:39.261] 👩‍🎤 Ate-A-Pi: You know, there is a paper from Microsoft on the first implementations with clients for GPT-based systems. And it has a lot of like knowledge there about like all of the weird things at the end. And it has this thing where the, our experienced coders are finding it very difficult because they're like, it's non-deterministic. And so it's how do we do like testing? How do we do...
How do we do continuous integration, right? Because it's non-deterministic. So how did you guys deal with those problems? Like where, I mean, you must have some test cases on things that it should say or shouldn't say, et cetera. But then once in a while you have those things failing. How do you deal with those things?

[38:24.234] Hrishi Olickel: Ah, so, you know, I think we have basically two problems, effectively, right? One is iteration. You've got a system and you want to make it better or you want to find better ways to do it. And in the world of, you know, deterministic coding and other things, basically look at the problems you add on to code, right, whereas in the case of prompting and the way you work with models, sometimes you might have to throw the entire thing out and start over and you get a better response, right? Or a new model might require that. So that kind of incremental optimization to the same piece of text, which is, you know, encoding.
perfectly the right thing to do. May not be the right thing here, right? So we had to put in place just ways of thinking about it that made that difference. Now, the other one is in terms of correcting and sort of fixing things or improving general performance. So we do put in place tests and sort of benchmarks, but we might still be at the stage or AI might still be at the stage where I think you really can't beat direct human evals. So.

[39:21.869] 👩‍🎤 Ate-A-Pi: So someone just sitting there and asking questions.

[39:25.154] Hrishi Olickel: someone just sitting there asking questions and someone just actively looking at success and failure cases, right? I think we're too early. I might be wrong on that. But I think we're too early for LLMs to be evaluating themselves and for that to be a viable strategy to take you into production. There might be good classifiers to tell you what stuff you definitely don't have to look at and what stuff you definitely need to look at. But actually reading the responses, actually going through the system, and we've just found so many ways that systems can be better just by doing that.

[39:55.169] 👩‍🎤 Ate-A-Pi: Mm-hmm. So actually just typing out a prompt and putting it in there and seeing what happens and evaluating it, basically.

[40:03.466] Hrishi Olickel: That's, that's, that's exact. I also say that because when this happened among sort of the engineer friends that I knew in my network and sort of inside the company, I also saw this kind of polarizing reaction, right? People split into camps, where some people wanted to get away from prompting as quickly as they could, right? It's non deterministic. It reminded them in some ways. One person actually told me is the only reason I'm saying that, of why they got into development in the first place, because it felt like we're, you know, like managing humans again.
of saying please and all of these different things. You know, because you don't get deterministic repeatable output, right, you could tip it 20 bucks to get a better answer. And so they were all looking for abstractions and ways to get away from prompting. And then another crowd was immediately in love with it, you know, because of what you could do. So to homogenize things a little bit more, I think forcing people to get really, really comfortable with the prompts and the responses.

[40:35.241] 👩‍🎤 Ate-A-Pi: It felt like managing humans again. Wow.

[41:02.827] Hrishi Olickel: especially while these models now are still so young and we don't actually even know the proper distribution of how they respond to different things, I think it's really valuable, right? So it has that ancillary benefit.

[41:15.481] 👩‍🎤 Ate-A-Pi: Indeed.

[41:19.849] 👩‍🎤 Ate-A-Pi: What do you think comes next? Because we are in this process where we think we have agentic, a lot of people are working on AI agents this year. We haven't seen, I mean, we saw like prototypes last year with like maybe, I think maybe AGI and some stuff like that, but stuff that didn't work, right? So, and there's some expectation that we'll have like some kind of working agents sometime this year.
And you know, some people are like, what is an agent? It's just a for loop with a GPT, right? So, yeah, so again, like, but what do you think comes next for you guys as, you know, if you have like a working, somewhat of a working two, three, five step following agent, what do you think happens next? Or do you think that's actually hard to use in an enterprise environment because there's still these kind of unknowns on like, you know, giving it like some access and you don't know what it's gonna do, right? Right, access to something and you don't know what it's gonna do.

[41:50.943] Hrishi Olickel: Yeah, pretty much.

[42:16.93] Hrishi Olickel: I'll give you a few different answers to that one. Right? The first one effectively is I don't think any sort of arbitrary or model written code execution is going to be good for enterprise in the short term. Right? We still haven't begun to think about how we can draw good lines and good controls over LLM written code to make it safe. And arbitrary code execution before LLMs were writing them in an enterprise space already a no go. Right? We haven't ever built a sandbox that can't be broken out of.
when it comes to these things. So that immediately makes it a no go for quite a bunch of different industries. That said, most of our work has focused on and we were quickly coming to the thesis that the tools are what matters, right? And we've kind of moved down the stack a fair bit, is the agents can be dumber if your tools are better, right? If your designation of the tools, if your description of the tools and the way that you've structured and built them, those are what we found accumulate the most improvements, right?
That can be as simple as something that learns how to call an API, or something that learns how to read a SQL database, or something that generates a query, or any number of these different things. So the smarter and better those are, the lower we found the intelligence requirement for agents. And we found that kind of accidentally because we were hoping eventually that we'd be able to get to an open source model that could run a proper agent. So we were trying to lower the intelligence requirement ourselves. But we also then realized that if the tools got thicker and better, then the agent...
even the agent needed to be less smart. But if you gave it a smarter agent, now it could do a lot more, or a smarter model on top, now it could do a lot more. And so that has kind of underpinned a lot of the work that we do. And a big reason for that is also just having been in production this long and just having had users bring us problems and things they want to do. I think a finite set of tools that keep getting improved, it will still cover about 80 to 90% of use cases.
We like to think we're diverse, we're not that diverse. Right? And then you can have code execution or meta agent prompting or meta prompting for the rest. But these ones, I think having a wonderfully curated, highly optimized set of tools is kind of the solution.

[44:17.697] 👩‍🎤 Ate-A-Pi: Right.

[44:32.145] 👩‍🎤 Ate-A-Pi: Indeed. It's funny because, yeah, go ahead.

[44:36.116] Hrishi Olickel: Right? The analogy that I most recently used is instead of having, you know, a big brain 100X CEO and a number of workers, right? I think you can get far better results with a mediocre CEO and 100X people under him, right? Who all really know how to do their jobs, right? And the guy at the top, which is your agent.
just needs to largely give out some instructions and check the outputs and sort of manage that.

[45:05.269] 👩‍🎤 Ate-A-Pi: So there's a Palantir demo from last year where they have this battlefield agent and the agent is a GPT Neo J. It's one of those early open source GPTs. And it basically just does conversation. All of the tools that go into it though, come in through the Palantir system where every single input and output is totally known.

[45:18.508] Hrishi Olickel: Yeah.

[45:34.361] 👩‍🎤 Ate-A-Pi: who has access to it, who has the authority to use it. So it's like so much metadata coming in there that Palantir controls. And then the agent just sits around and just directs traffic. Like you said this, you have access to these tools. Which one, like, you know, and that's basically, the agent really has no way to do anything else. And then they demoed it calling in a strike, but then it just shows a menu and you click and then you're calling in the strike.

[46:02.322] Hrishi Olickel: It goes to the strike. Yeah. That's that's exactly it. Right. And this is something that I now become more confident in, simply because we've had that history of being able to show that. Right. I think, and Tropic was recently talking about parallel to calling instead of larger numbers of tools. Our earliest iterations, and we didn't realize at the time, could very easily do, you know, like 100 tools, and 100 parallel tool calls. And that's because none of the tools were very thick.

[46:04.489] 👩‍🎤 Ate-A-Pi: Yeah, yeah, yeah. So exactly.

[46:31.07] Hrishi Olickel: sort of API calls, most of them were calling intelligent tools and just saying what you wanted.

[46:37.577] 👩‍🎤 Ate-A-Pi: Right, right, right. So, and this is in reference to Anthropic, like in their most recent release of the documentation, they showed a Opus model calling in 100 haiku models, each one doing a call to a webpage, I guess scraping it or something, bringing it back, parsing it, and then like returning. And then so you could get a kind of table or chart of like a summary of all the data that was on like those hundred pages, something like that.

[47:07.891] Hrishi Olickel: Exactly, yeah.

[47:09.581] 👩‍🎤 Ate-A-Pi: Right, right. That's interesting. So it's really like reducing the range of motion, kind of like the range of motion, that the degree of freedom of motion that the agent has to something which is acceptable and within that kind of like controlled, kind of cordoned off like, you know, safety area that you can still kind of manage all of the variances.

[47:36.178] Hrishi Olickel: That's a really good way to put it. But from my perspective, it also sort of increases the freedom for how much the model can sort of ideate and do things, right? If I give you sort of the three examples, right, of the three major architectures that I see run, the standard way to do tool calling would be you give it sort of a bunch of different tools and say, hey, here's four APIs, each one's got 50 parameters, right? I want you to do this thing, but I want you to write detailed API calls for the 50 parameters. So about 400, you know, 200 parameters.
and write that out, I'll execute it for you and I'll give you the response, right? That's one. Ours is somewhere in the middle, it's more of, hey, here's four tools for you, right? This particular tool deals with, let's say, crewing data, here's one that deals with chartering data, here's one that deals with flights. All I want you to do is basically call this tool and then describe what you want to happen with flights. So you could literally just say, there's no parameters, you literally could just say, hey, I would like to know what non-red-eye flights exist.
for the next week to go to Los Angeles. And then that tool will figure it out and go off and figure it out itself. Then you've got the new meta prompting one, which is, hey, I have this thing for you. You can do about four things, but I want you to write the prompt and then execute that for each of these individual tools from scratch. And then if you do a good job, they'll come back and give you a response.
So really for us, it's about the intelligence requirement. And the more we can lower that, the more the model has room to really surprise you and sort of build more things.

[49:08.237] 👩‍🎤 Ate-A-Pi: How many ideas do you prototype and of which, what is your conversion rate to production?

[49:15.926] Hrishi Olickel: Oh, that's a really good question. I think the conversion rate would probably be around, say 10 to 20% of the ideas that we prototype. That's simply because of a lack of resources and time. And also as a company, it's very different. As an individual, I can work on a number of different things and they're all really fun and they all really help me. As a company, you want to make sure that you're pointed in one direction.

[49:41.885] 👩‍🎤 Ate-A-Pi: Right, right on. So, and are there other, so there's definitely a lot of ideas now, right? Like because post-GPDs, there's a, and what do you think is like the gating factor? I mean, like, how would you make a decision between like, hey, this is important enough that we're gonna do it and it's not worth our time. Like, you know.

[50:05.166] Hrishi Olickel: Ah, I think there's a number of different factors there. One of them is, is this solving an internal problem? Is this something we can dog food in any sort of way? Because that makes a huge difference, right? You ideally want to be, especially people working in enterprise, working in industries like shipping or churns or a number of different ones. The Holy Grail is really a product that is also useful to you, that your team can use, that you can then dog food.
because that's a product that can get continuous improvements. Every time we hit on something like that, I'm very, very excited. But on top of that, once you've got enough history as a company, you usually will find something, do something, and then you can just look back to see a trail of requests or problems or pain points that trail behind you from the last multiple years of conversations that you've had.

[50:53.993] 👩‍🎤 Ate-A-Pi: Right on. What kind of, what three, two or three numbers do you carry in your head that you always kind of like either quote or, you know, it's numbers that you define kind of like your business or what do you think about? Because, you know, I carry a bunch of like different stats in my head for early, you know, I just spit them out. So what are those two or three numbers that you carry about, you know, in your industry?

[51:21.382] Hrishi Olickel: Oh man, that's a really good question. Nothing comes to mind right now. And that's because we've just been through going through a bunch of different customers and a bunch of different races. And I guess I just got flushed out. Right? Let me think about that. If I come up with something, I'll come back to you.

[51:34.45] 👩‍🎤 Ate-A-Pi: right on right.

[51:39.818] 👩‍🎤 Ate-A-Pi: What does your typical day look like? Like what did yesterday look like? Like you woke up and you know, are you working on Singapore hours, US hours, you know, because you have clients all over the world, how does that work? You know, what does your typical day look like?

[51:55.426] Hrishi Olickel: So we're based around the Singapore time as a company. So nine to five, I am either available or in meetings or working with the team as much as I can. That said, usually I'll reserve about nine to about 1am for US calls and for working with either investors or customers in that time zone. In between is time that I either spend with my girlfriend, go for a bike ride or do different things. And then there's...
sleep needs to fit in there somewhere. But you figured it out.

[52:30.927] 👩‍🎤 Ate-A-Pi: Is your dev team in Singapore or remote worldwide? Like how does that work?

[52:35.882] Hrishi Olickel: Our dev team is, so we give people a lot of freedom to be able to travel and sort of move around. Almost no questions asked. So they tend to be in Singapore quite a bit at the time, but people also travel, I think. One of our key devs is right now in Brunei. And so people move around a bunch. But usually everyone call us around Singapore time.

[52:57.832] 👩‍🎤 Ate-A-Pi: How big is the firm?

[52:59.682] Hrishi Olickel: The firm, I think at the moment, is about eight people. So it's not, you know, and that's split down the middle on business and death.

[53:08.309] 👩‍🎤 Ate-A-Pi: Right on, right on. And you guys are invested in by Flexport, I think. Flexport and I think Instacart as well. So are they also your customers? Or do you work with them? Or are they basically just like, you're in shipping, they're in shipping related kind of? How did that come about?

[53:33.034] Hrishi Olickel: I think there were quite a bunch of different things that we were thinking of at the time because Flexport is effectively in more and of course Flexport's gotten so much bigger. So I might be wrong about these things. But at the time, my understanding was that Flexport's far more in the logistic side of things than where we are. And those are sort of the yin yang different parts of shipping. It's what goes inside the ship and there's a lot to do with the cargo. And there's a lot about how do you keep that ship afloat and the right people on it and sort of get it to the right places.
So we were sort of different parts of that. I think we did have early conversations about how we could possibly use the information that the other has to better do what we were doing. I don't think anything came out of that because these ended up just being big enough problems to solve on their own.

[54:17.789] 👩‍🎤 Ate-A-Pi: Indeed. What do you think are the next big things, you know, that you guys are looking towards? Like, is it a product? Is it, you know, expansion? Like, what are the next big things?

[54:30.442] Hrishi Olickel: I think the biggest thing for us would be able to take what we've done in a lot of ways with walking rag or sort of vision rag, and then expand that out to larger datasets, right? Because we're starting to see that place where very high value technical expertise inside of a company can now be baked into systems because they already have the documentation. And that can be very, very helpful, right?
especially in shipping, where the biggest problem that the guys were trying to solve when they came to us with the thing that became, you know, walking rag, was they had a lack of technical engineers on board the ship who knew enough about a vessel systems to be able to solve problems. So they usually had to make ship to ship calls, or, you know, talk around and find the right person and you may not always have the time or the money or the ability to do that. And these are very highly critical problems because, you know, a gas tanker is $220 million.
and has a ton of very, very intricate systems on board, the Hobbit function. Long term, I think we can expand that out to the company. We're really hoping things happen with open source and model execution that make it possible for us to do that client side, because I think that is going to be sort of the next massive jump.

[55:41.705] 👩‍🎤 Ate-A-Pi: So is that because there is internet access issues on the ships? Because that's one of Starlink's reasons for existing, is because internet access on ships is... And you know, I had a friend who was a nuclear submariner, and he used to tell me that his boss, the commander of the ship, would reserve all of the internet time for his own emails.

[56:07.06] Hrishi Olickel: Okay.

[56:07.256] 👩‍🎤 Ate-A-Pi: So no one else on the ship had internet access except him, because he wanted to do his emails. So is that still a thing? Yeah.

[56:13.515] Hrishi Olickel: Ben, that is okay.
That's really cool. Okay, remind me to either look that up later because I have no idea how submarines get internet. Do they have to surface?

[56:23.497] 👩‍🎤 Ate-A-Pi: Yeah, they do. They surface and they have like something, they put something that's, you know, trails behind them. Yeah, yeah, yeah.

[56:27.826] Hrishi Olickel: Oh, God, they put up a radio tower. So it is kind of related to the internet access on a vessel, but that's where the chat interface has been really great because we've been trying to do that for the last sort of three years is transform some of our services into chat, so that they're more easily accessed by people on ships. This was before Starlink. Now I think it's becoming less of an issue because Starlink is making a big difference to these ships. But it's more just around.
data protection and sort of privacy, right? It's also a compliance question. It's a win-win if you can do it, right? Because it means that far more can be exposed to models because they're running client-side. It means that companies like us can sell them far easier to companies because there's a lot fewer hoops to go through, right? Understandably, fewer hoops to go through. And there's a lot more optimization that you can do when you've got client-side models running.

[57:21.065] 👩‍🎤 Ate-A-Pi: Right. And, and when you say client side, it will be something that runs on a phone, right? Like ideally, right?

[57:27.426] Hrishi Olickel: Phone would be wonderful, right? I'm really hoping and I've got a lot of faith in the web because and also because a lot of pioneering work has already been done now you need the application sort of stability work to happen. Everyone's got practically the same runtime, you know, through V8 and the browser. Wasm is now you know, web assembly is now getting to the right point where you can run these models client side. So the ideal would be a website. In some ways, yes.

[57:55.069] 👩‍🎤 Ate-A-Pi: Okay, so on browser basically. So I interviewed the Dingboard founder Yaseen and he split his models into a client side and a server side. And then he does a lot of the computing on the client side so that he doesn't have to pay a lot for GPUs on his side. But he did it for cost and latency reasons, right?

[57:58.101] Hrishi Olickel: Exactly.

[58:15.484] Hrishi Olickel: Yeah.

[58:19.85] Hrishi Olickel: That's the third win. That's the third win. You get that for free. At least in our neck of the woods, where people are usually willing to pay for these things more. It's still a win, right? And you get it for free.

[58:31.521] 👩‍🎤 Ate-A-Pi: right on. So having it run in browser and some of these models could be very simple, just kind of like, you know, hide by hide by kind of like, you know, the initial start and then the more complex queries basically being fed into something in the backend.

[58:46.314] Hrishi Olickel: That's exactly it. And it makes a huge difference. Even the enterprise space makes a huge difference. One of the earliest things that I did sort of middle partway to this company was take our routing engine, which needed to run on a cluster in the cloud and sort of move that client side, so that can run in the browser. So once we switched all those routing calculations to client side, that was a big cost saving for what was an even smaller, younger company back then, so made a big difference.

[59:10.485] 👩‍🎤 Ate-A-Pi: Right on, right on. So what detail about your product or service do you put a lot of time into, which you think that users don't really see? Which is sometimes you work on something, like Steve Jobs worked on the insides of the computer and hardly anyone saw that. So what do you put a lot of time into that you think users outside don't actually see?

[59:33.538] Hrishi Olickel: Hehehehe

[59:38.326] Hrishi Olickel: I mean, that's a really good analogy, because we also try to do the equivalent of Steve Jobs and the semi-transparent plastic, which is to try and expose the things we do if we can, even if users may not care as much, because we're proud of them. I think quite a bunch of work internally goes on our side into serializing and deserializing things, so users can sort of jump in exactly when they want to. Right? That is oftentimes unsung work.
And when I say serialization, deserialization, I mean things like being able to save conversation history at the right point, or if someone is in the middle of sending an automated email or receiving one, right? Being able to deal with all of those interstitial states and sort of saving state. Like, I developed a massive amount of respect for all the people that work on those things because you don't think about it, right? So, same way that if you play a modern video game, you don't really think about the saving system.

[01:00:13.593] 👩‍🎤 Ate-A-Pi: Mm-hmm.

[01:00:32.638] Hrishi Olickel: or the saver load system, and yet it has to save an entire world as it is and then sort of reinstantiate that. And you just kind of expect that to be the norm. But once you've got that, you know, we build a lot of really social functionality into our systems. And those because the quality of them is so high outside of enterprise in sort of the consumer app space. I don't think users necessarily notice how much work goes into it in an enterprise space, because you're...
even more bound by privacy, and you're even more bound by regulations and compliance.

[01:01:01.517] 👩‍🎤 Ate-A-Pi: So what do you mean by social functionality?

[01:01:04.942] Hrishi Olickel: So the goal that we've always had is for users to be able to do something and then pass that on to a teammate and go, this is where I was, right? If you can pick up. And I'm not saying we've gotten to that goal completely yet, but even something as simple as being able to share part of the work that you're doing with someone else, right? Or having a share feature or an email feature where they can jump right into where you work. That is, you know, orders of magnitude more complicated in enterprise, right? Because access control and permissions and all of these different layers are far more thicker in enterprise.

[01:01:37.054] 👩‍🎤 Ate-A-Pi: Is that person allowed to know, right? Like if you, if the, yeah, like maybe the captain is not allowed to know what's the pricing of the cargo or whatever. Or I used to, I randomly read a shipping manual where they said one of the big problems with shipping was pill fridge, especially if you had alcohol. If you were carrying alcohol, you could expect to see some,

[01:01:41.042] Hrishi Olickel: Exactly. And what are they allowed to know? For how long?

[01:01:52.105] Hrishi Olickel: Exactly, right?

[01:02:05.321] 👩‍🎤 Ate-A-Pi: of it go missing every single time. So, you know, and also one of the big deals is, you know, making sure the alcohol or cigarettes or other contraband or non contraband, like stuff that you're legitimately shipping, isn't like lost, right? Like, or just disappears or whatever.

[01:02:07.498] Hrishi Olickel: Yes.

[01:02:23.23] Hrishi Olickel: And that happens on the human side of things, but also on the company side of things. Shipping is this massive web of thousands of companies cooperating with each other, right? Travel agents, support agents, charters, owners, managers, all of these different people. And so it's amazing that the system really works, because there's just so many lines. It's a graph, right? It's not even a tree, it's a graph. But that also means that there's always trust issues.
it definitely does happen even at the company level and not just at the individuals.

[01:02:53.329] 👩‍🎤 Ate-A-Pi: Indeed, indeed. Trust issues. And which is a good segue to how do users gain trust in using like systems like Ocean Oracle or Proteus? Do you notice a process where they start off with kind of like initial queries and then they go on to more sophisticated ones and then and if there's a single mistake some people are like I'm not going to use it, it gave me a wrong answer like
What does that trust building process look like?

[01:03:24.194] Hrishi Olickel: So that's really interesting, right? I'm also going to use it as an excuse to talk about the pathway that I've seen users kind of take through these systems. One of them, and this is really interesting to me, and it's just that most initial questions tend to be meta questions about how to use the system, which if you go back through the AI systems that you've used or the products you've used, most of them are unprepared to answer, right?
Like, let's say I give you a copilot or I give you an AI system that is meant to be able to do something. Most users first question to that system is, hey, how do I use you?
right? Or what kind of documents do you have access to? Right? And we've all done that, right? So that almost always tends to be the first question, as opposed to even with question suggestions, a question that is the task that the system is trying to do. So oftentimes, that's what we spend a lot of time trying to make sure that the system or some subsystem that activates can really like take you through the onboarding or sort of walk you through questions about these things. But otherwise, once people are in,

[01:04:18.228] 👩‍🎤 Ate-A-Pi: Mm-hmm.

[01:04:31.43] Hrishi Olickel: A lot of them go down or go up, right? They start simple and then sort of ramp up, or they start complicated, you know, hit something that's far too complex to answer and then sort of ramp down and sort of find something a bit easier.

[01:04:45.449] 👩‍🎤 Ate-A-Pi: How does that change after they've been using the system for like a few weeks or whatever? Like, you know, do they kind of settle into like a pattern? Like this is like the six things that I, you know, like when we use iPhones, we end up with like five apps being 95, 99% of usage. So is that something that happens in GPT-like systems? Like people tend to default to like five or six things that they always regularly use it for, you know, patterns of questions.
And then that's it, kind of, or, you know, how does that work?

[01:05:17.618] Hrishi Olickel: I think it definitely does. And that is something that I'm actively trying to solve, because that's a problem. You want to encourage exploration, you want people to really discover all the different questions and sort of areas. But what you have, and I say I'm trying to solve it, but I'm guilty of it as well as a user, is you'll have a short period of very excited exploration, and then you usually hit some point of disappointment or resolution, and then you'll sort of circle back. And the few things that work really well become part of your patterns.
And then you effectively stop exploring after that. And we saw the exact same thing happen with Google Assistant and Siri and all of these other ones. I sort of smart-honed a long time ago and sort of switched to Google Assistant. We went through that exploration phase. Oh, hey, can you do this thing? Can you do that thing? Within about a week, we just settled down to using it for alarms. Turn lights on and off. And that was it. And so for that pattern, you will consistently activate it.

[01:06:09.406] 👩‍🎤 Ate-A-Pi: Exactly.

[01:06:13.354] Hrishi Olickel: And occasionally it's going to pipe up with, hey, I can do this thing now, but you don't trust it anymore. You just use it as a hands-free. You basically use it as a hands-free switch, and you end up there. But that's what I'm also trying to figure out how to avoid with chat interfaces, because their biggest strength is also their Achilles heel, is that the text input is too broad. It's really, really broad. You literally can do anything with a text input and could describe almost any kind of task and sort of problem.

[01:06:34.591] 👩‍🎤 Ate-A-Pi: Right.

[01:06:42.614] Hrishi Olickel: So how do you give users some kind of control surface to work up and down so they know where the limitations of the system are, where they can experiment more? So an open problem.

[01:06:52.945] 👩‍🎤 Ate-A-Pi: Indeed, one of the many in AI. So I think, thank you, thank you, Hiroshi. And I think you're one of the very few people who have actually deployed in production, deployed GPT-like systems in a critical area in production. Because there's a lot of people in demo stage. There's actually very, very few people in production, actually.
Most people are still struggling to put stuff in production. And I think you face all of the issues with the facts not matching, and you implement a drag, obviously, because the amount of knowledge that the system has is just not specific enough for what you want it to do. So yeah.

[01:07:26.241] Hrishi Olickel: Yeah.

[01:07:42.89] Hrishi Olickel: Yeah, I mean, I've always had like an engineer's perspective, right, even though I've done some research, because my dad was an engineer the same way. So you look at how you can get something to work, and that's what you care about, right? And sometimes that means shimming something up with a piece of wood so that it stays level, even if that's not the right way to build it, you get things into production.

[01:08:05.69] 👩‍🎤 Ate-A-Pi: Indeed. Okay, thank you Rishi.

[01:08:08.362] Hrishi Olickel: Yeah, man, lovely, lovely.