Private LLM Search and Flashnet

It is the results of many months of AI psychosis, and it's actually quite hard to distill, succinctly. Okay, so exploring agentic search. How did we get into the idea of agentic search? This is somewhat of a incredibly simple mental model of an agent. You have some type of model you need to access, and do inference over. You have memory. This could be long-term memory, short-term memory, working memory like a human.

You have tools, and then you also have, like, payments, which could maybe be thought of as a, as a tool as well. But, it's useful for me to think about as something separate. And so if we want totally private agents, we need private inference and local tool calls. We need private payments, so your agent can pay for things without people knowing who you are. And we need private search, which you might say is another tool call, and that's correct, but it's one of the main tool calls that actually leaves your machine. And so that's why we're, sort of hyper-focused on this. My AI gave this date, great diagram. You know, search goes out to the internet.

There's a lot of search engines built for agents, but when you look under the hood, they are just embeddings-based search engines, which Google's been doing for a while, so I won't look on this. There's some interesting redistribution mechanisms from these two, though, which are sort of, you know, in a world of agentic search, ads are gonna function quite differently. So they're trying to think of new ways to do that. None of them are Sybil-proof though, as our work at Flashbots has shown. And so for this, you know, I'm just assuming the rest is solved. We have inference and TEEs. We need to scale those hardware-wise and all that. We have ZK API, or it's being rolled out.

And so that's, that's a way. There's an Ethereum research post on this, and that's a way for you to reserve some type of capacity on a, you know, private LLM host or maybe even other types of hosts. And you can you can sort of mint credits and redeem them anonymously. And so what's left is, is search, in this example. And so, yeah, again, I keep coming back to, in exploring this is like, what does search even mean for an agent? And so after getting into this, after talking to tons of people how their agents are doing search, it really breaks out into two camps. There is local data. This is data you've already captured. Shashank gave me a great example yesterday.

His agent has a cron job which goes to Hacker News and GitHub and a bunch of these sources daily and fetches things for him. I have one myself as well, which scrapes X for new blog posts from AI researchers and this is an example of, like, something that I'm not really gonna go to a popular search engine for because they don't do a great job at crawling the social graph. And then there's remote data. This is, you know, fetching some resource at a server, trying to, I don't know, find a, Amazon link or a vendor. And so it's important to split those two use cases out here.

As I mentioned, everyone's already scraping, so there's a lot of, yeah, data being scraped. The, school's IP is banned, from IACR, as someone pointed out yesterday because I accidentally scraped it too hard. So ask me about my scraping setup. And so then this is another finding, from some, research reports around search. And this also highlights the distinction between local and remote search. And so in, I believe this was twenty twenty-four, fifty-eight point five percent of Google searches, according to this one thing I didn't cite, but you see me later if you want it. Fifty-eight point five percent of searches are zero click. So you search it, and the answer pops up at the top.

And it's an LLM answer. Or it's an, like an auto-generated answer. This is like, what is the weather? What's five plus five? Very simple stuff. And that now, is at sixty-eight percent when measured in early twenty twenty-six. And so the sort of, as, you know, conclusion here from Google and many others is, the amount of search people are doing is actually, like, going away. You don't really need to hit a search engine for this. You can hit a local LLM.

And so that, yeah. Those are queries you don't really need to touch the internet for.

I'm gonna skip through this. Yeah, this was another research POC I won't show, but it's a way to actually upload bulk, markdown datasets to a website and request them and have a TEE randomly sample whether they're correct, the pages people are uploading. And so this is something that you can download for local use.

And so let's go back to public search. So there's two paths here. There's one where you can still hit the Apologies for keep clicking on this. Where you can hit the existing Google, Kagi, Exa. Their You know, Kagi and Brave, they're search engines that claim they're private as well. How exactly they're private, you can't really look into it, so it's kind of a trust me, bro privacy, so I put it in this top camp. And then the second is this class of search engines that's starting to get popular or starting to become more production feasible, which rely on techniques like FHE or using a TEE. And so the idea is people create an index and put it entirely inside of the TEE, or they have the TEE actually go and index the web, and now you're able to create a, remote connection with the TEE and ensure your connection is private.

You can get your search result I built out the more fancy, one which uses a, like, a type of FHE that I'll talk about. But you'll notice in both of these scenarios that you still need to talk over the internet to reach some server. And so again, if we think back to Phil's talk earlier today, and, just all the other privacy talks, metadata is extremely important. Even if, say, Google somehow were to verifiably say they're, you know, blinding, or not looking at your queries using a TEE or one of these fancy techniques, they still could identify your agent by, you know, coming from the same IP every 30 seconds, and, in a variety of other techniques.

So yeah, I implemented this project called Tiptoe. It's a paper from 2023. Actually, iPhones use a, variant of this, to apparently search through your friends' photos. I didn't know that was turned on, but, apparently it is. There's some fancy cryptography there, so I, won't get into it even if I could fully explain it. But the way you can think about it is, someone creates an index and, essentially turns it into this, like, object that is entirely ciphertext, so you can't examine it once it's in this form. It's just an encrypted object. Then you as the searcher have a query. You say, you know, "H-how, how how's Mike's hair looking so good all the time?" And you turn that into ciphertext, you send it to anyone who is running what I call these, like, encrypted, shards, and they will actually, you know, do a cryptographic operation on it and return a, encrypted ciphertext that they can't inspect.

So this means they can't tell what you're searching. They also can't see what was returned. And then that is given back to you, and you decrypt it, decrypt it. This was thought to be, like, not very feasible at the time, and some of my results show it's, like, actually usable, which is, is cool. Yeah, so I just explained that, and I'm gonna keep rushing through this. But yeah, so metadata is the remaining leak, if we do trust TEE inference providers, TEE index providers, and some of these more fancy solutions.

And so the next thing to explain the sort of hairy trade-off is around which technique to hide your metadata. As Phil gave a strong opinion of earlier, there's approaches called onion routing. Some people are, you Their threat model is a global adversary who, you know, is, like, a nation state actor, and so onion routing with a less than five hops is insecure for them, for example. There's these other classes of protocols called DC-nets and then mix nets. And so there, there's kind You could think of them as three, the three families, but it's not exactly just three. So this is mix nets, DC-nets, and onion routing. And so DC-nets and mix nets are mostly a round-based protocol, whereas onion routing just sort of goes hop to hop and, continues on.

Mix nets require 10 nodes to, communicate with 10, and then they communicate back, and they do another round of the protocol. And so one question when analyzing agentic search is, you know, synchronized rounds are probably fine for humans, but what about agents? They move so much faster. You know, when you actually use one of these, by the time you send a message to a mix net, you are probably halfway through a round, so the node needs to wait till the end of the round. Then it'll include you in the next round. You have to do some back and forth. The further the nodes are across the world, the, you know, the more these rounds hurt.

And so it's a big question whether this is possible. And so some of the work I did was showing, I had developed a tool. It's on my GitHub. It's called, a, Deep Research Swarm. And so, what I actually did was use a variety of search tasks, deep research tasks, and I logged all the tool calls, and so we can see web search here. Archive search is a slightly different one for a different reason. And we can see that actually I think there was six agents running here. They And, the task was actually, something about finding the frontier of papers, on, RAG-based memory. And so the six agents actually fan out in parallel, and they can also sort of stack up multiple queries, and then they have a pretty long thinking time, and then they do another round.

And so the, like, the key insight here is a little bit overgeneralization just from showing you this example, but it's agents can plan ahead, and they don't really care about these constraints. They can work around them. And so if it's for the sake of privacy, we think this is okay.

I have a bunch of other profiles. I won't get into those. Agents tolerate rounds, as I said. Yeah, and the last thing I'll probably say is, so connecting the DC-nets to Phil's talk earlier. So DC-nets, as what Phil was getting into, is he likes them because they are sort of, they don't they don't rely on any cryptographic or strong cryptographic assumptions that could be proven wrong in the future. It's a very loose way of saying it. And DC-nets in their original formulation really just require XOR operations, like XOR, and so they're, you know, it won't break unless you can somehow break XOR. And so yeah, that's, that's why people really like DC-nets.

The problem is they're extremely susceptible to spam. And so all of our work, at Flashbots on this, like, branch of anonymous broadcast is around How do we take DC-nets and make them actually usable? And there was a paper from IC3 called Zipnet, where they put these DC-net nodes in a TEE, and they use the TEE to reduce the, amount of spam in the network. And so the key thing here is that they're not relying on the TEE for actual privacy guarantees. So the TEE can break, and your privacy is not broken. It's just the network can start getting spammed.

So I will just show you these demos. I also have these brains in a jar at some point where I have, this is not part of the search, but I'm just dying to, show what's, what the judges say.

Okay, so I've got three demos here. So we're just gonna search, So this is what I call this one private search engine. This is just the human front ends, but all the APIs are, like, agentic friendly. It's called Periscope. You can see here, hopefully you can see it. I typed in fancy AI stuff, and you can see the quantized, embedding here. And so what it's gonna do, that's also pretty cool, you'll see in this visual, is it's hooked up or it knows of about three different indexes that are live in these encrypted shards running on separate servers I deployed. And what it's gonna do is actually it has a succinct representation of all of these shards, and it's going to, do, like, a quick similarity score to see which one has the most relevant, or at least just from this quick operation, which one has the most relevant.

And then we're gonna hit that server, and it'll do a thing, and then, essentially it does, like, a private information retrieval to actually receive these URLs. So here you can see fancy AI stuff routed to my cognition, index, which this is where all the AI research, Twitter posts are going. And this So yeah, here are the search results. It's a search engine. This cognition one is only running twenty-five thousand documents right now, so that's not huge. You know, Google has, a trillion, indexed. But we can see that for twenty-five thousand documents, the PIR takes about one point six seconds. And then we've got about five hundred milliseconds of latency there.

So that is that. The next thing I'll show, again, remember AI psychosis, too many demos. This is a, fork of, the Open Anonymity project, so super cool project. I recommend checking it out. They have a essentially what looks like a, you know, ChatGPT, OpenAI style, LLM front end. And they have this super cool, inference ticket design where you can basically mint one of these redeemable anonymous tickets, and then you can talk to an LLM inference provider that supports them from a fresh IP or through, some type of anonymous broadcast protocol that protects your IP. And, it will return the response. There's a bunch of nuance here, and they're sort of switching to, the, ZK API-based back end.

They used a different approach.

Yeah, there's a bunch of nuance there around how to actually also keep your queries, anonymous if you're in a, like, an agentic coding session where you're making many requests to the LLM. And so this is really just geared towards single requests right now. But if you remember in my presentation, it doesn't matter if your metadata is getting leaked. You're still, you know, at risk of getting anonymized. So what I did is I just, added into this, front end the ability to, toggle between Tor, the initial version of this Flashnet, which is based on Zipnet, and then this other project I don't really have time to get into, but it's Tor with a ZK proof to prove you're not a Sybil.

And so we can hit the LLM. I actually just have a mock one for this demo, so it's not gonna return an actual LLM response. So these latency numbers are, not to be fully trusted. But we'll see on this example, Tor is five hundred milliseconds. This is a three-hop circuit. The six-hop circuit was three point nine seconds, and Flashnet was five point one second. This is a very early version of Flashnet. It just has six nodes I deployed, and it's not really optimized yet. We, haven't actually launched a test net or anything. And then the Onion egress, yeah, it's interesting. It's often when the circuits are not warm, it takes longer. We can even run a set of ten if we want, but I won't do that.

And then the last thing I will show is, this was one of the early versions of the Shape Rotator OS. And something we actually had embedded into it was, apps from some of the teams. And so what I embedded into it was a P2P backend end. So everyone that was running the Shape Rotator OS, was running a peer-to-peer network under the hood that would hit up every device on the local area network in the school and say, "Hey, are you, like, one of my peers with my protocol?" And then anytime you would search using one of my tools, with consent, of course, it would actually populate this cool, graph, which kind of plotted everyone's search into different neighborhoods.

You know, how do animals communicate with each other? A bunch of other stuff. I think there was something about Saint Damiano, which is the saint of that church right there.

Yeah. So I have two. So one is, for the PIR approach, I guess I wonder like, for anything under like literally millions of pages, is there any point in doing it? Because you might as well just download everything onto, onto your computer directly. And the second one is, so for these DC-nets to like actually be useful, they have to have like a large anonymity set, right? And probably, you want to have a large anonymity set without like all, having an assumption that any one specific application is going to really make it. And so I wonder, to what extent is it possible to share it across kind of different and like potentially very heterogeneous different types of applications?

Yeah. Okay, great questions. So yeah. So yeah, the, PIR for less than a million documents, I think it's, yeah, it's a good question. There's a few ways to think about it. One Some of the other insights I had found from reading some of the surveys is around this, you know, what people are searching outside of these zero clicks, and it roughly falls There's like some type of a normal distribution, and the majority of the distribution is actually, like it's very common, like goods and services people are trying to search for, like the latest of whatever, a soap, a sponge or a bunch of these things. And then the rest of it is like the tail distributions, which the, tail distribution, which is extremely high entropy, whereas the other parts of the distribution are like extremely low entropy.

So they're like people are searching the same thing for this for six months in this bulk part, whereas the tail is like they're new words. Like you can't even try to guess what it is the next day. So my hypothesis is that the PIR approach is only interesting for the tail of the distribution. And then I think your, you know, question is valid of like, "Well, why don't I just download the tail of the distribution?" And I think that's pretty valid. I've shown this around a bit, and some people have told me some interesting use cases that I don't want to say on camera, which, are, you know, they essentially would like to still use these for discovery of things that they feel like they couldn't just instantly code up.

These Okay, o-one example I will say is, you know, like sports streaming sites, illegal ones are very popular. People all around here were watching the World Cup on a kind of illegal one, and it got taken down, and like you could see like half the city being like, "Whoa, wait, we lost our access to sports." And you know, you hit up a friend who has like twenty because he knows the people. And so that's the type of person I think could potentially offer one of these shards, and they could basically just create the shard, deploy it somewhere, and actually have another infrastructure operator running it. Like cases where you're like operating over a terabyte, but what that terabyte is just changes every day, and it's not even just about a hard drive.

It's about like way too much constant bandwidth to handle.

Yeah. For frequently updated data, then it's, it makes sense that we would have a single server rather than having everyone replicate it, that they would have to download it often, right? Yeah.

Yeah, yeah. Yeah, and exactly. And if it's, if it's something that they're constantly refreshing, but it's like, you know, research papers, well, you don't, probably don't need this fancy technique, so you could still even download it. But yeah, for things that are changing frequently that kind of like require curation, I think is maybe another angle of it. And then your second question was... Oh, so DC-nets. Yeah. So one of the So we do have some ideas of, yes, co-locating them with other apps. One example is actually the dstack, sort of framework. This is a, you know, it's Docker Compose for TEEs. You just write your app normally. You wrap it in dstack. You can deploy it to TEEs.

Phala and, a bunch of others, Synclave offer, hosting of these dstacks. Andrew is sort of one of the core architects of it. And one of our thoughts is actually could we just slide one of these little modules in there? For DC-nets, the problem is in the, you know, sort of anonymity, latency, bandwidth trilemma, they have high bandwidth usage. And so if the app you're co-locating it with is has any like, any large bandwidth needs, it's gonna be an issue. So that's actually one of our research questions is whether or not we Or like one of the open research questions, that we're discussing next week actually at SBC in SF, is whether we can bring that down.

And then I think another cool idea is whether you could just use a ZK proof instead of a TEE to prove your request is formed correctly and maybe use some type of simple reputation. Whether the ZK proof brings bandwidth down, though, I guess, you know, at least Okay, maybe not post-quantum.

What's the difference between human search and agent search?

Humans want results now synchronously and agents don't.