Examining the Importance of Transparency and Verification in AI Decision-Making Processes, with Jason Teutsch @ Truebit (Video)

[00:00:02] Hello, everybody, and welcome to the Crypto Hipster Podcast. This is your host, Jamil Hasan, the Crypto Hipster, where I interview founders, entrepreneurs, executives, thought leaders, amazing people all over the world of crypto and blockchain globally. And I have another amazing guest for you today. This man, he is the founder and chief scientist at Truebit. His name is Jason Teutsch. Jason, welcome to the show.

[00:00:31] Jamil, thanks for having me here. You're very welcome. Very welcome. I'm looking forward to our conversation. And the first question I ask everybody is this, is what is your background and is it a logical background for what you're doing now? Well, as a logician, I'm going to have to say yes. My degree is in mathematical logic. So, yes.

[00:00:59] And I, you know, among those people who go into logic, there are people who come in because they are naturally orderly and those who seek order. I'm probably more in the second class. So, but I guess, yeah, that's where it's at.

[00:01:17] And I come, I entered into blockchain from an academic place. I actually published the first paper on Ethereum and CCS in 2015 before the network launched.

[00:01:31] So, basically scaling Ethereum before there was Ethereum. So, it's, you know, before L2s. And today, you know, the, this verification game that we introduced is used in all the optimistic roll-ups that are out today. So, yes, that's a little bit about my background.

[00:02:00] Great. Well, I want to dive into a little bit further. You have an extensive background in verification technology and computational transparency, right? Could you just elaborate a little bit on both of those?

[00:02:14] Yeah. Well, I, verification is probably in some sense more accurate than scalability. It's just, if you think about where we're starting from, we were, it's not scalability in the sense of transaction throughput, which is what most people talk about in the blockchain.

[00:02:30] We were trying to give superpowers to the smart contracts so they could do large computations. Basically, we called it, if you look at the original white paper, it's called the scalable verification solution for blockchains, because what we're basically doing is providing attestations for smart contracts in a completely decentralized way.

[00:02:49] So the original true bet on Ethereum is just a set of 18 smart contracts that have economic incentives and you make a call to the contract and it magically calls back with the answer by recruiting random participants, I guess you can say.

[00:03:05] So that's really the beginning of how I got interested in verification. It was through the first thing that we were trying to provide verification for was blockchains, but later it became apparent that the rest of the world needs verification and it's become increasingly more apparent as time goes on.

[00:03:29] And, you know, 10 years ago, frankly, we were simply too early. But I mean, now with AI and other, you know, RWA, there are a lot of tariffs, there are a lot of sort of trends that are going on right now that make it much easier to explain why verification is so important and really affects people's daily lives, whether you have it or not.

[00:04:00] Yeah, I mean, so.

[00:04:30] So true bet verify is what it sounds like it's it's it's it is. We provide attestations for code executions and API calls. But at a higher level, we're really a toolbox for applications that require verification.

[00:04:51] And so we really the it's it's it's it's it's it's it's a lot of it is about risk mitigation and providing transparency for different sorts of applications that require it, especially when you get into the decentralized space.

[00:05:10] And you can't rely, for example, you can still fall back on a technology that can prove data provenance, how data was processed, when it was processed, where it came from, those those sorts of things. And, you know, with with so, yes, that's roughly speaking, the the use here.

[00:05:37] I mean, in the case of AI, which I know is the topic for this podcast, you know, you might want to some of the things that you might prove are really simple stuff like which model is being used for inference. Right. I mean, you can prove the inference derives from further prove that that inference derives from the model.

[00:05:56] You can prove how it was trained, but it's it's really kind of I mean, really basic questions that are, you know, am I talking? Am I learning from the model that I think I'm learning from? Am I learning from the model that I think I'm learning from the model that I think I'm learning from the model that I'm learning from.

[00:06:22] And I think I'm learning from certain models because you're trying to protect intellectual property, for example. But just to give an example of the of the reputation, there's a very popular paradigm in in in in in AI, which I call Airbnb for your GPU. You know, the theory that this thesis is there's. There's. High demand for GPUs, there's sorry, there's high demand, but actually a shortage of GPUs, but there are some that may be sitting around spare.

[00:06:48] And so the idea is to match up spare GPUs with with the with the demand. But of course. If you don't do the verification when you're farming out these questions, you're going to get. Especially when they're incentives, you're going to find that people. And if you don't check the work, you're going to get bogus stuff. So that's that's in a very simple way where verification comes in to the space.

[00:07:20] And it's it is it is it there are there are you know, there it's it's a challenge. You know, we've seen last year when I O dot net. Release their network, they suddenly have, I think, 1.8 billion fake GPUs that showed up on the network. And this is the kind of thing that can happen if you don't do verification.

[00:07:49] I yeah, I think it's important because I never asked. I asked chat GPT something. It comes back with with incorrect data, you know, and I'm like, how are they verifying it? Like we're like, how is that, you know, accurate? So I'm like, what is what is the best model? Is it these large language, you know, huge models or is it something like, you know, like a machine learning? What's like a federated learning? What's it? What's it? What's the best model actually to use?

[00:08:17] You know, I guess it depends what we're trying to solve. But I mean, their verification is is a whole package. There's a different tools that you want to use. I mean, there was in there was a paper by Amit Goldwasser, Paradise and Rothblum that came out a few months ago, which talked about self-proving models. Exactly what you were talking about. Your A.I. is let's let's say 80 percent of the time it nails it and gets it right.

[00:08:45] But what about the other 20 percent of the time? Then you get a model to sort of prove its own correctness. And they take sort of an interactive proof approach to this, which I think is, you know, maybe one way to it's a different flavor of verification.

[00:09:04] But it does intersect with the TrueBit style, TrueBit verifier style verification as well in terms of if you're actually going to implement this, you would probably see something like that in the background. But, you know, in terms of I mean, there are there are so many different types of machine learning models and they're all good at different things. Like you don't want to ask an LLM to solve a math problem, for example.

[00:09:31] I mean, you'll you'll they're all they all have different strengths and weaknesses. So. So. But. Yes, I think that's verification has become. It's it's it's it's always been important. It's just sort of manifesting itself in different ways over time. Yeah, I see that, too. So you said you said.

[00:10:01] Different ones are good. So basically what you see out there in the market today, you're seeing a lot of silos and black boxes, right? Or the systems operate independently. They don't. There's no talking to each other. Right. So is this similar to what you're solving? You know, or how do we and if not, how do we get these systems to work better together? Yeah, I do think there is. You do see a sort of what you said, a trend towards siloing back in these informations.

[00:10:29] If you think that AI is the new web search. Now, people who have content on their website are looking for ways to protect it, which means that one wouldn't expect everything to be. Eventually, it's it's not findable through through the old methodology. But as I said, it's it's it's like so so so if you're going to do.

[00:10:58] I guess, I guess, a search in a decentralized way, and it's not run through again, a trusted provider that everybody knows like Google, and it's all these sort of sat. I don't know what you call them satellite or decentralized arms, I guess you're going to need to do some sort of verification on those. Those entities that are that are providing information to you.

[00:11:28] I mean, they call them when you look at you search your web search, they call it a verifiable snippet, the little thing that Google gives you so you don't have to click on any of the links right and so verifiable again it's if it's not if you have you always wonder like you know is it is it giving me. How do I know if a verifiable snippet like if it wasn't Google provided me a verifiable snippet but like you know. You know.

[00:11:52] Mr snippet.com or something you know like you you'd want to double check it in some way and you the way that we check things is changing like in the 1980s if you held up a Polaroid, you wave the Polaroid picture in the air that was that was a good as good as a proof today, like an image isn't worth very much, and we need a different set of tools to basically what you used to be able to sort of eyeball you know you're you're going

[00:12:20] going to have to use machines to figure it out. And it's just been sort of that trend when people first started talking about fake news. If you really went out and searched you could find what you were looking for, but with AI that's you know the signal to noise ratio has continues to drop off. And you're need more and more machines to deal with machines basically so that you can find the information that you're looking for and believe it. Yeah.

[00:12:48] So, you made me think about something I just got I just got my driver's license renewed. And there's a stamp on it. Thank you. It's good through 2033 I mean second drive right. There's a little mark in the corner called real ID. Right. I didn't send them. I didn't send them my latest picture. I guess I'm going to have to be young until 2033 right looking or whatever, but you know, the last picture was taken six years ago.

[00:13:15] But how do you how do you know, you know, that you know what that would how do you like is reality the use for transparency verification or how can they prove that it's me in some other way. How why is verification and transparent become becoming, you know, very critical, like because somebody could say they're me and they could take the key to drive my car. Right. What why is it so important now than it was before. Well, I think it was always important.

[00:13:45] The real ID is just a perfect example of what I said a minute ago that the standards are changing. And for whatever reason, the government decided that what they were doing before wasn't providing them. The system they had before wasn't providing the type of verification that they needed now whether you'll need a driver's license at all by 2033 because you know maybe maybe drivers will be banned altogether and we'll just be driving around in.

[00:14:11] In a in a in a in a taxis that's that's that's a different story. But I guess I hope I'm answering your question, but it's it's sort of. Yeah, I don't know. I could say these things change over time and the way that. I mean, I was a bit shocked.

[00:14:36] You know, I walked through I entered back into the United States a few months ago, walk through the passport control. I didn't even pull out my passport coming into the US. They just said hi Jason. And now I was like, you know, I was like, really, are we really doing this. Like, so, so, you know, I just.

[00:15:04] Anyway, so so yeah, I again, all these things, it raises the question of, you know, accuracy, you know, what are we actually. What information do we need to get the job done and when how reliable does it need to be, you know, when if you're if you're asking AI to make financial trades for you or drive your car or do, you know, medical procedures.

[00:15:33] There's you're going to want a really high level of certainty about these things, which right. I mean, I and there's always going to be these the models are probabilistic for the most part. So we got to put some effort into basically getting the data into a place where it's it's usable.

[00:16:00] Yes, so I do you make a good point there about certainty right. So how could verification technology address. I guess, certainly goes along with accountability and apparently there's an accountability crisis here and exists in AI with regard to certainty. So how could verification technology potentially address that certainty issue.

[00:16:28] So, so I think there are. I mentioned a few tools already on this call, so I'll just recount them and we can dive in deeper, maybe add to the list. But, you know, this self proving models is an interesting one when we're doing. True bit is another.

[00:16:46] I mean, the one place I'd add on true like true bit verify in terms of sort of bringing data provenance proving that you're interacting with the model that you think you are proving that it's doing the inference in the way you expect it to proving that it's trained on the way. The group. Any kind of semantic proofs that you can prove about the model to that's another form. And then on top of that true, any time you're going to be processing data, there are all kinds of little.

[00:17:13] What do you call them data cleaning like little processes that you have to do to interpret the data. I mean, this isn't this is a place where, you know, true bit verify is really immediately useful because, you know, you can. You can write a few lines of JavaScript code and have a proof written up, you know, within seconds. So so that's I mean, from the time you write the hit hit have the code written.

[00:17:43] Well, hey, have an A.I. write your I mean, look at look at the pipeline we're looking at, like with vibe coding catching on. You're going to have an A.I. that comes in, writes a code based off of talking to you for three seconds. And then off of that, you would be able to generate some sort of proof that that this scripting process of processing of the data is done correctly. I mean, there's just like you need it.

[00:18:06] What I've noticed having sat in this verification spot for so many years is that when someone says I want verification. They aren't saying, well, I just need this little I need a ZK proof for this little thing. I mean, occasionally you are if you're living in the blockchain space and everything is basically it's the one place where everything is certain and data doesn't disappear.

[00:18:29] But the real world isn't like that. So if you're if you're sitting in this place where data is is basically once someone says I want verification, they want it end to end because your verification is only as good as the weakest link in that hole. And so. And so.

[00:18:57] Yeah, basically, that's that's that's what I've observed that that. And so then you question, you know, where do the inputs come from and every step along the way, who's been involved in the processing and more and more we're seeing third parties being involved with processing the data right when you have one center. And so that's the central party that's the central party that's doing everything like I don't know, let's give Google as the example.

[00:19:27] If Google is doing everything itself, then you can sort of rely on Google. But when, as we get more into these. Turn to the decentralized world is AI is sort of forcing us in that direction. It creates more and more uncertainty that has to be offset. I really see true bit verify as a counterweight to this. It creates more and more. It's the next AI movement, and you can't really have one without the other.

[00:19:56] That makes sense. So there's a couple of areas to go down here. One is, one is, one is a data. Right. So I wanted to find out, with TrueBit Verify, and looking specifically at blockchains first, and the area of data. You mentioned ZK proofs. Another area that is important is decentralized Oracle networks and Oracles.

[00:20:23] Are we going to need to have those anymore, ZK proofs and Oracles and all this? Or can we just rely on AI agents, AI models, TrueBit Verify, to bypass the need for those proofs and for Oracle? I think you need all three. You're going to have to combine them together. And really, the power in this is to create the whole package, right?

[00:20:51] It's not just an Oracle. It's not just compute. And that's what we've sort of been focused on with TrueBit Verify. Normally, what one does is I need to pull in the data, then I need to process the data, and then I need to send it somewhere else. That's a super common pattern. And so just getting one piece of the puzzle is not enough.

[00:21:13] And often the data you're pulling might be in an AI model itself, which is, you know, we have a study that we just put out on Deep3, which you can find on the website, which sort of addresses this problem specifically in the AI space, which is to say, you know, you can prove that, okay, I'm going to let the AI do certain...

[00:21:34] And using different kinds of models that are out there, and then being able to prove that those models are functioning in the way that they claim to be or that I expect them to be. And sometimes it's not a matter of deceit. It's just misunderstanding. If you sort of... One person might assume that the code is doing one thing, and the other one assumes some... You know, the person who created it assumes something else, and you just end up with a mismatch where people don't...

[00:22:04] You end up with mistakes, not because someone is malicious, but simply because you didn't go down to the level of code. There was... I didn't see what came before it in the pipeline. I thought the model was trained this way, but it was actually trained that way. You know, this sort of thing comes up. And, yeah, I'll tell one more story here, but it was like there was...

[00:22:32] We had someone who was working with NOAA, you know, the weather data. So they're... And you might say, well, why do I need to verify data that comes from NOAA? You know, it's always going to be there until one day it wasn't, right? I mean, but if you have the certificates proving, you know, then you're okay. You can just sort of keep going. And we've seen...

[00:23:00] It's a recurring thing where APIs and information of websites disappears suddenly without any warning, and people are left to scramble. But if you... At least you can protect your historical stuff with proper data provenance. I mean, it's just sort of a common sense thing to do. And there's not really... There's not a lot of overhead either, right? Instead of...

[00:23:27] Our play view is like instead of running JavaScript on an ordinary cloud, just execute it over with Truebit, and you'll also get a... It'll do the same thing, but you'll also get a certificate, a transcript basically at the station that goes along with it. So you can do these things. It's not that complicated, I guess is what I'm saying. I mean, I'm not saying Truebit Verify isn't complicated,

[00:23:57] but we're trying to make this easy because I've seen a lot of ad hoc verification solutions that just simply never get finished or never get up to sort of production level standards, but everyone always has sort of the intention to do it. So that's an interesting challenge. But really, the other reason to do it isn't just because it's common sense, but because it adds value, you know?

[00:24:23] Like when you get into RWA, the more you verify, the lower the risk is for investors, and therefore the assets are worth more. You increase the demand for your assets by providing proper verification. So, you know, there's different, I guess, categories of risk. You know, I've talked about... I mean, you have counterparty risk is another one.

[00:24:52] We sort of jumped in that with the Airbnb for your GPU example. And then reputation is becoming overweighted as well because we've seen no matter how big an organization gets, there's... Or how much credence people give to them, sometimes things happen, right? So that's... The more... I personally think there's another...

[00:25:21] You can also think of verification as simply an insurance policy because what we should be doing is pre-auditing stuff so that bad things don't happen versus five years later coming back and litigating them. I mean, just think about this feeding the future that raised $240 million to feed children during the pandemic. The only problem was those children didn't exist.

[00:25:53] So, right... And now, five years later, there's a lot of pain and upset. You know, there's a lot of pain, like which could have been avoided if we just did the verification in the first place. So it's... This is the way that things have to go. We need to change this sort of paradigm. And it shouldn't... The question isn't... Shouldn't be asking the question, why should I do verification? The question should be, why aren't you doing verification?

[00:26:24] Because that's ultimately where this goes. I don't... And the sooner we get there, I think, the better. I want to talk about a practical example that to me seems like it needs verification. I don't know if you've followed this whole Newark airport thing where the air traffic controllers went on strike. You know, and it was just a mess in Newark.

[00:26:51] But their systems are old software from like that was created in the 1940s. They're not using AI. You know, they're not using agents. You know, and then it's been a mess. So how would... What would a world with verified AI systems, especially in the area of critical, important areas like air traffic control, look like as they were compared to... As compared to today? I don't know. Why would you want to throw AI

[00:27:21] at air traffic control? That just... Like, we know how to land planes. I mean, I guess, you know, you can model these things out, but I don't know that AI is always the... Adding AI to the beginning of the sentence doesn't always make it better. Like, the air traffic control is a place where you want absolute certainty. I mean, I don't know if the idea is to sort of cut costs or... What's the problem we're trying to solve here? Maybe I missed the point.

[00:27:49] But I think places where you need absolute certainty, you've got to create a system which allows you to have that level of confidence through verification of whatever the standard is. And, you know, it may not... It may be just that there's a centralized controller that everybody trusts. And that's pretty much... I mean, in all these cases,

[00:28:18] anytime that you're going to do verification, there's always some sort of kernel of trust where you have to... There's always some ground truth or something that you fall back on. Even if it's just like an API call that has nothing to do with, you know, this particular... A big API call that's... Everybody trusts, you know, a call to... I don't know if it's somebody that everybody trusts. Weather.com. Like something really... You know, like they're not going... They don't care about the deal

[00:28:48] that you made with your crypto bro friend about what the weather was going to be on next Thursday. So... Or last Thursday, I guess, is more like it. Well, usually... I don't know. The thing about predictions is they're usually about the future. But... So, you know, I guess... What do you think? I mean, like... Is there another angle here? Yeah, there is. Okay, go ahead. A lot of people are fearful

[00:29:16] that if the accountability issues with AI go unchecked and undealt with, that we're going to face an AI armageddon like in Total Recall or like some kind of, you know, sci-fi, you know, movie that there's going to be an AI armageddon. So if there is one, how do you come back from one? Sure. Oh, you mean, assuming that we made this mistake, how do you correct it? Yeah. The world along the line, you know, and then everything's chaos.

[00:29:46] How do you come... How do you correct it? Sure. I mean, I guess my point of view is let's prevent it from happening. But assuming you're you're in a situation like that. Yeah. I mean, I guess you if the AIs have already in place, you can see if you can get them to run in more verified ways. I mean, I guess that's that's certainly true. But I don't know. It's hard to...

[00:30:15] Without knowing what the disaster is, it's hard to come up with a solution for it. I assume there's also the option is to pull out the cord, right? I mean, you said that this... That's one place where humans have it over machines, in theory. You know, they can switch off... We can switch off the power grid, maybe, at least in the current scheme. So you hit the reset button. You start over. I mean, if there are ways to get it back on track, obviously, you do that. If you notice there's a problem

[00:30:44] with a particular system, you try to get out the bugs or phase it out. I mean, it's... There are a lot of strategies that one could put to work, but, you know, an ounce of prevention is worth a pound of cure for sure. I like that. An ounce of prevention is worth a pound of cure. I like that. I'm going to use it. So,

[00:31:12] going back to the blockchain, right? You know, at the intersection of blockchain and AI, the biggest technical challenges in bringing blockchain in line with AI as far as verification, what are those technical challenges and how do you see Truebit helping shape the more accountable AI future? That's a great question. I mean,

[00:31:42] the elephant in the room, obviously, AIs are big, blockchains have small compute capabilities and that's the roots of Truebit is about taking... The assumption was always that smart contracts compute correctly, which, by the way, you never say someone, hey, my smart contract computed wrong. Take that and sort of poof it up. How do we expand the power of smart contract? And it's the same thing that we're doing in the real world taking these different... Now, taking different points of trust and trying to poof those out

[00:32:11] into applications that people can really use. So you have just the sort of general complexity question of how do I execute AI? So obviously, scaling of computations and scaling of data can be challenges. I'll give you a lot of data on the blockchain already, which for those types of applications, if that blockchain is capable of accessing its own data, again, that's another place where

[00:32:40] you can understand the power of making API calls in addition to simply running everything as a... You know, you won't... You need a way to get the data back into the smart contract even if it's on the blockchain. So you have these challenges and we have true... You know, we'll talk about smart contracts. I mean, we'll be

[00:33:10] sharing soon, you know, example of a smart contract where you basically said you can just paste in JavaScript code, have it into the smart contract, and then have that basically execute the code. Make it call out, calls back in. So this is one... You asked me how true would verify. That's sort of a really simple application which shows the power of off-chain compute coupled with sort of the Oracle processes, we're saying these API calls. But the second point that I would probably

[00:33:40] point out is non-determinism is a huge issue because of the way that these AI models work. You have non-determinism getting... The blockchain is very deterministic. In cases where you need to get consensus, that's a big problem because two people might do the computation correctly and get two different answers. That's what non-determinism means. And so insofar as

[00:34:09] that's an issue, if everything has to be on consensus, how are you going to get consensus on AI? There are various sources of these non-determinism. You got the floating point is a big one and the consequences of that, which are non-associative operations, which also come up in other places, NAN, and also people running different types of hardware as well. So you can execute the same model in different places and get different answers.

[00:34:38] So in the blockchain world where everything has to work by consensus, this is a problem. However, I do think there is a way... A lot of times this consensus is overkill and I think what people really want is not necessarily consensus but simply auditability. I want to be able to prove that going back something was wrong, that I can't stop something wrong from happening but I can quickly discern that that's what happened. This is essentially the use case.

[00:35:08] That is the use case for Truebit Verify when you just need auditability and if you relax this requirement a little bit, you get something which is faster, cheaper, easy to use and all these things that really fit. the real world needs. I think that this is combining these things together.

[00:35:38] Certainly there are use cases where you do need consensus but this obsession with smart contracts has led to a bit of overuse. The question doesn't have to be how do I get data into the smart contract? How do I get it back out? Ask yourself why is it there in the first place? That's the question. It's a smart contract not always needed. You're jamming stuff.

[00:36:11] It's a small hole and you're trying to jam something big in there. Sure, maybe it'll fit. if what you want is auditability then that might not be your easiest solution. Got it. Interesting. Interesting. A couple last questions. Great. What do you see

[00:36:41] as the future for building out Truebit and how do you see your roadmap in the next few years? How do you see people being able to leverage your technology? I think we have a lot of pieces that can be added to broaden out the verification

[00:37:10] needs. Privacy is a really big theme. Bringing in ZK SNARKS to protect some of the data that people want to bring in through API calls is significant. all kinds of privacy aspects are there. Also, the ease of use

[00:37:39] of combining these tasks together. You're going to get a whole envelope full of these certificates. Now you're going to have to manage those. I think that's the inevitable if we can provide tooling that makes those sorts of things easier. Also,

[00:38:09] we're going to be talking about handling larger and larger files, which is something that you need for AI processing and more language supports. There are all kinds of things. These get down to technical development. Really, what I think it also comes down to supporting specific applications and having enough templates out there that people see in common.

[00:38:39] Everybody wants to do an airdrop? Great. Here's a tool for doing the airdrop. You don't have to think at the level of code, really. I think that's really a few different pieces of it. Thank you for asking. You're welcome. Awesome. I want to thank you very much for your time today. I enjoyed speaking with you. My final question is how can people find out more information about you and about TrueBit Verify?

[00:39:07] You can go to our website and follow us on Twitter. I'm also Jason Teuch on Twitter. Please feel free to reach out with any questions. We try it out. We love getting feedback from our

[00:39:37] users. Awesome. Thank you very much for your time today. All right. Thank you so much.