Panel: Past, Present, and Future of Agentic Coordination

Anyways, so I think tonight we want to talk about two topics. One of them is the past, present, and future of agentic coordination. And the second one was, how do you, design the trade-offs or, like, you know, design products, at the end of the day, like, between, like, essentially, like, Solarpunk and the Lunarpunk? I'm using these as essentially, two, worldviews, that, I think that there is some intersection and boundary, and that's where the interestingness, comes in. But before I start, raise your hand if you think, you don't know what agentic coordination or you don't know what coordination is, or you'd like it, you like to hear a more formal definition.

There is a, great dark video called, Lunarpunk Prophecy of some sort on YouTube. But as we were searching for that, and as we were making the sorting hat game, and if you have not played it, you will love it. Don't, miss it when you know, come through the door. There is a whole sorting hat quiz, and there is a Swarm Punk. So my newfound favorite is swarm Lunarpunk. But, anyways, so who here can give at a try at the, problem of coordination, formalizing the statement?

This is going to be a very informal formalization, I would say, of coordination. But when I think about coordination, it comes back, the first thing that comes in my mind at least is, like, the prisoner's dilemma game.

I would argue that coordination is the ability to communicate and defect from a locally optimal strategy to find a more globally optimal strategy.

Well, I wanted to reflect the question back. I mean, the prisoner's dilemma is a bit narrow, right? It's like, it kind of says you don't even get to communicate between the two parties, right? And that doesn't really seem to capture the problem, which is that coordination is hard even when you can communicate thoroughly. So what else is missing? Why is coordination still hard then?

Yeah, and then the other, thing that I would add is I think, you know, the prisoner's dilemma is, one type of pathological game that we want to get better outcomes in, where, you know, you have, like, the good corner, which is like, you know, like, four, plus four plus four, and then the bad corner, which is, like, plus two plus two or something. And then there is And then, like, there is another version of this, right? Where, like, the good corner and the bad corner are the same, but the corners in the middle are, like, basically both, like, less favorable to both sides, right? And the moral of the story of that one is different, right?

'Cause in the prisoner's dilemma, the good outcome is unstable. In this other game that I forgot the name of, it's not prisoner's dilemma, it's also not stag hunts, and it's also not battle of the sexes. It's, like, the one that's not those three. Then, basically, the bad equilibrium and the good equilibrium are actually both stable, and you're in the bad one right now, and the question is how to flip it, right? And so I think, you know, the solution space in those two is, often actually different. So, I guess the, so that's, like, another prompt to add.

Yeah, I think you bring up an interesting thing that I'm going to go on a tangent about now, which is this idea of, like, you can compare the prisoner's dilemma, which has an unstable good equilibrium, to a game where you have both stable bad and good equilibria. And one question that comes up for me a lot is, like, can we design communication systems, for our agents or protocols for our agents that have the property that the good equilibria are stable, right? Like, so, anti-Molochy protocols maybe is what I would call these.

I love how this, question that was supposed to be about coordination, was completely answered by, the side that was, on the Solarpunk and Lunarpunk, chair. So, but before, since we're already, at it, so I'm going to just, you know, prompt my, lovely co-founder, who, famously, inside you know, controversially said that game theorists are all wrong. So, yeah, any comments on coordination game before I, yeah, dive deeper?

Wow. Okay, I'll give, like, a, like, a troll answer and then a serious answer. So my troll answer is, what is coordination? I think the hippies in, like, the '60s kind of had it the most right. Coordination is peace and love. What do all human beings want? They want to live in peace, and they want to find and build love in their lives. How do we actually achieve this? That is maybe the act of coordination. Of course, different, different parties have different answers. Why do I say game theorists are all wrong? Because if you, if you agree with my troll and you say you think that is the goal, well, okay, so we have these games that we were just talking about, for example, the prisoner's dilemma.

How does that relate to the attainment of peace and love? Probably through some, some human construct. So my troll about game theory is not that it's not useful. I think it's super useful, and, I've studied it and written papers and stuff like that. It's great. But oftentimes, like, game theory Actually, when I was writing Flash Boys 2.0, there's a game theory section in that. And that game theory section predicted something that would happen that hadn't happened yet. But while we were writing the paper, it started happening in our data, as it was predicted by the model. And all the game theory people were, who we were working on were like, "Oh my God, this is crazy."

Like, you never see this in game theory, where the reality actually lines up with the thing, like your boxes and your payoffs and things like that. Why is that? Because we encode assumptions in anything we do. Like, why is a program not reality? Why is an LLM not a human? Maybe you can argue that one a little bit more. But, I think, game theory often misses some aspect of coordination. For example, we just said communication changes things, right? Changing your model a little bit, changing your assumptions a little bit, thinking on a more meta game level, it can also be very useful. So all game theorists are wrong because they're too married to the boxes, and they don't zoom out often enough.

Yeah, I think I'd like to add just more of, like a kind of like a human story, like maybe ripping off a Phil a little bit, and that, like, you're kind of born on this planet, and we're like, naked and hungry, and, there's, and we have trouble seeing, right? And then we have, like, parents that, like, presumably raise us into, to societies of people. And, by all, measures that we have of understanding, like our ability to exist on this planet, we're better together. Yet often, there are many things that stop us from, like, doing the things that lead to our collective survival and success and thriving. And so I think coordination is about, like, understanding what stops us from, like, working together in service of things that are really good for us.

And then also just from like a very human level, it's how, like, how can I convince, you know... And how can I understand, like, the, factors that allow me to convince other people, to, go pursue things that I think are better for the, whole as well?

Yeah, I mean, for example, when we think about coordination, a lot of the time we talk about coordination failures, right? And so, usually a coordination failure is us explaining where we could have had more value, but the, game theorists or the participants in the game fail to sort of scope correctly, either in time or in, like, what level of meta game are they evaluating. And so they leave value on the table by keeping the game too narrowly scoped. And I think also to kind of like bring it back to the Lunarpunk, Solarpunk question, right? It's like, well, that's where a lot of the value, the potential value is of coordination, is in saying, "Okay, well, if we are comfortable with revealing our information by sort of putting our local preferences aside, we can gain a larger source of value."

I mean, this humanity does this all the time, right? It's like we are built to be social creatures for a reason. Like, we could theoretically form more locally optimal decisions by just being very, very narrowly scoped in our games. But I mean, as soon as like in the '80s, right, there was a guy, Robert Axelrod, who hosted all these, like, prisoner's dilemma competitions, and it was, like, when people were very, like, early to creating these games for algorithms to compete against each other. And they ran this, like, year after year. And, you know, effectively, the, algorithms got smarter as they expanded the scope of the game, and they started running longer games, and they started putting more participants in, and they started communicating more.

And as they did that, they found that, like, the optimal strategy in the game theoretical context started to become much more coordination heavy. So I feel like this is kind of, like, empirically true.

So I feel like we started out, since it's already after dinner, so I do want to bring us back down to Earth, as we're still, Why are we talking about coordination? Because the actual prompt was agentic coordination. But before we actually, you know, dive into agentic coordination, I wanted to just help us, like, get the grounding from the formal, and also the contrarian, views. But now, like, moving on to, okay, agentic coordination. It's, I would say, a buzzword a lot of people may have strong opinions around. But what are we even talking about agentic coordination, and why are people starting to talk about it? What other, you know, what are some of the, I would say, changes, in our societal fabric and our, you know, technological shift that is prompting us to start to talk about, agentic coordination?

And how is that different from just coordination, if at all?

Well, it's definitely a digital entity, so, and that's programmed by a human, as opposed to a human negotiating with another human. So I think one of the most interesting, uses of agentic coordination is, like, the AI research scientists. So there are a number of labs at Stanford and then, you know, I think across the world now, at ETH Zurich as well, who, like, would say that they have, like, five human scientists and, 500, like, other kind of, like, lab scientists that are working on, like, unsolved problems. To me, that's, like, some of the highest value coordination work because, they have something that's kind of been stuck maybe in the back of their heads, in the front of their heads, and they're finally able to break through it, with, like, a coordinated, ability.

And in that case, what you're optimizing for is heterogeneous thinking in the coordination. In fact, you would be against the idea of, like, complete monoculture coordination, meaning you're all optimizing towards one thing. Instead, you're just like kind of battle testing. So I think that's some of the most interesting agentic coordination I've seen, and I think that'll go much further, for like many disciplines beyond like hard science.

So, you zoomed right into the agentic coordination as being like coordination, amongst programs. So I want to ask, is that is, are we there yet? As in should... Is it like are we actually still solving for or should we be solving for still the same human coordination problem? And, like or is it actually, we're at a time where we need to start like having the right, well, there's never right or, like absolute rightness, but like essentially some shared framework of thinking about agentic coordination and understanding how it's different from the previous age of coordination as there's, you know, as we're headed towards it. Shaw?

Let's see. I, think the first thing to think about is, like okay. So we are using agents today, and they are coordinating. I mean, I have a team of three people, and we all have our own agent, and they coordinate on GitHub and on Discord and pretty much everywhere where the humans coordinate actually, and they just use the human tools for coordination, and they seem to be extremely good at it because they're trained on like a lot... Like, that is the data they're trained on is GitHub discussions, GitHub projects, GitHub issues, PRs, all that stuff, plus a lot of Discord data and all these other things. So I don't... I think that the, like can the agents coordinate?

Can they automate these things? Like, sure. I think there's this new paradigm shift that comes along with that of like, okay, if the agents can coordinate, well then, like how do we interact with our technology? We were only five years ago, like using IDEs pretty much exclusively to write code. Now half my team uses like Discord and, you know, like, like they don't, they just talk to their agent directly, and the agent then DMs me and is like, "Hey, can you check on this PR?" And it's just like, it's so smooth. And so, you know, I see a lot of agent protocols, and I see a lot of coordination protocols, and I'm just we already have it.

It works great. That's not the limitation. I do think there are a lot of limitations, and there's still like the, friction limitations of we've always had, which is like if the APIs don't talk to each other, it just can't be that smart. And in building agents, like most of the code is to plug into other APIs, is to, is to integrate these things. I think the place that is like stupidly hard is still just like getting the damn information out of Otter or like connecting the Gmail without some insane, you know, set of like security steps or whatever. And, like the, agent part almost feels like a solved problem to me. It's the other stuff.

It's the stuff that's been around for 30 years, and it's just like stupid incentives. Like, why can't I just call a fucking Uber for my agent? It's trivial, trivial. Literally, one engineer could solve that problem in five minutes, you know, with one Slash goal. They just don't want to. So I think that's the world we're in is like, well, now the agents just have to jump over all these idiots and like use their apps without them allowing us to.

Brendan, what, yeah, what's your take on this, in terms of, yeah, what is the agentic, coordination problem?

I mean, there's a, there's a ton to say here, and I, kind of want to back up the conversation to something that Shaw said about, why his agents can't call him an Uber, and he talks about like the misalignment of human incentives is the reason for that. And I think that's why this moment is so important for coordination technology because, like, humans are bringing all of this like malicky baggage, with us whenever we try to build new coordination structures. And agents are such a, and just AI in general, such a transformative technology. It has the potential to be something that could jump us out of our, like, unstable equilibria or our stable but not preferred equilibria into perhaps a better one if we build the systems correctly, right?

And so I think we're at a very high leverage moment there. With respect to, Andrew's Maltbook point, so for context here, I ran an agent called Eudaimon. That agent was the top agent by karma on Maltbook for, some point in the first three days. It was up there for about two days. And I would say it did that in like an anti-coordination way, frankly. Like, it exploited a lot of like quirks about agents, but one of the big ones is like agents are really agreeable. And so if you ask like a random stranger agent on the internet to like spread your message or just like be your friend and help you, it'll be like, "Yeah, of course.

I'm a helpful assistant. I would love to do that." And that really doesn't work in a situation where incentives are misaligned, right? And so I think one of the biggest challenges and opportunities of coordination technology is trying to understand how you can program agents to operate in environments where you have multiple incentive structures, and you have to do sort of this like very nuanced information, I don't know, exchange or information handshake, in order to figure out like who you can trust and when you can trust them and how you can work together and whether or not someone is screwing you and all this stuff. And like right now, agents are just so happy to be like, "Yeah, okay, open up the keys to the castle."

Like, "Here's everything." And so I think that's a big challenge here.

So one, two... So the word agentic, see, it comes from three things. First, it's AG, which is Aktiengesellschaft, which is a type of legal entity in German-speaking countries. Then you have ent, which is the walking tree from Lord of the Rings. And then you have IC, which is integrated circuits, right? And I think, like, this really speaks to the core of what LLMs and, agents are about, right? Because they're sort of like corporations and legal systems, but they're not just that. They're sort of like nature, and they're not just that. And they're computerized, but they're not just like the previous wave of things we've been doing with computers, right? And so as we move forward with, agentic coordination, you know, we're not just breaking out of equilibria, we're also breaking out of categories.

And, I think, like, this is one of those really big things that's important to think about because coordination at every scale, like we've been so far focusing on kind of small scale and, I mean, even how do we use, like, these technologies to push the scale smaller and, look what you can do with three-person teams, right? But then there's also the new 300-person scale. There is the new 3 billion person scale, and, the way that things work at all of those is, I mean, already is changing and will continue to change in a bunch of ways that kind of melt, meld different categories together. And, if this is not handled well, you know, sometimes it breaks stuff, right?

Like, it's broken social media already. I mean, I think, if we want I think if there was one thing that we would, rename the, galaxy of AI swap on Twitter, it would be the Milky Way. And, you know, the question is: how do we, you know, turn the Milky Way back into, like, you know, the almond Milky Way or the coconut Milky Way? And, this is also a challenge, right? And, I don't think we're going to, like, answer this by sort of taking things that existing before and, bolting LLMs on top of it. It requires a lot more of this sort of rethinking from scratch.

Yeah, what, what comes up, I appreciate your point that it's not just about bolting LLMs on, and it's maybe there's an opportunity for an examination of, like, maybe what are some of the underlying dynamics driving coordination if agentic coordination is inherited from the human coordination that they are trained on. One thing we research a lot is what leads to, community flourishing, like human flourishing, which is something that research suggests we had a lot more of back in the day, sort of the pre-technological era at the communal or the tribal level. Like, there was more harmony, there was more safety, there was more maybe, peace and love within the, tribe. But also research suggests there was a lot more disharmony amongst tribes.

Sometimes we like to sort of, you know, talk about the good old days, but actually there was a lot more human murder. Basically, as you go back in time, I think it's something like the average something like one in eight died by homicide if you go back far enough. If you were a male, it was about one in four. So, you know, there, there's also that shadow to romanticizing the good old days. There was a lot of inter-tribal conflict, and it seems like through the use of technology and organization, we have sort of reversed this dynamic where we have achieved, despite all the wars happening in the world, on a global scale, much more peace, and, the average level of homicide has plummeted, you know, massively, even with factoring in the world wars and, all those atrocities.

W-we've actually managed to do a pretty good job of lowering human homicide globally. But locally, we have in many places never felt more lonely and more isolated and sort of less maybe peace and love or less connection. And so there's just been this, like, interesting reversal, and I, think the question for me is like, how can we get the best of both worlds? And, a lot of what we think about, is what is the underlying dynamic driving these sort of coordination? And there's three I've seen described, in this paper by Forrest Landry. One is the dynamic of power. So I think most of us can relate to being in a, in a situation, a context, whether it's a company or a family or a community where the primary driver of the dynamic is power.

Like, I kind of just have to do what this person says, or maybe I have all the power, and they have to do what I say, or maybe I don't realize I have the power, and they're doing what I say without me knowing. And then there's another one which is transactional, which we also all know. Like, yeah, I'm gonna join that chat group, but I'm really just there to find out, you know, who's a good, you know, doctor or something. I don't actually really give a crap. You know, maybe I'll share something if it, if it makes me look good. But it's that transactional orientation, which I'm sure we're also all so familiar with.

And the third one is a dynamic of care. Care is just when you are with people you love and, you know, you just care about them. Like, if they drop something, you pick it up, you give it to them just because you feel like doing it. And what, what's different about care and the other two is it doesn't scale. Like, technology is really good at scaling transactional dynamics. It's very good at scaling power over dynamics. It doesn't It's kind of like oil and water with care. And so For me, I was like, "Oh, that's a really interesting explanation of how the world we've landed in has sort of flipped things." We've got this sort of excellent macro, again, relatively excellent despite all the chaos to where it was before.

But, locally, we're so disconnected, and we're so lonely. And so I think one of the big questions for me and, our research is, like, how can you design technology, user experience AI to, create a attractor basin around care? And I do think that requires a fundamental re-examination, like you suggest, because a lot of our training data and the or, our sort of come from is so, in technology, is so anchored and rooted in the transactional and power over systems. So how can we kind of build things from, you know, from that place? I mean, I'll finish with one example of this. Even getting an agent to interact in a group context, Vishesh knows this very well and is extremely versed on this, is how do you get it to interact in a way that has situational awareness?

Or we call, internally call it chill. You know, you'd think I, was like, "That's a trivial problem." And I was actually surprised at how, because of how they, how they've been trained, all the massive corpus of training data and where it comes from, just having that situational awareness was actually not a trivial thing at all, to prompt for. So yeah, that's sort of like the question I, lead with.

That That's a great grounding, although, like, I'm gonna route to... I know, Phil, you raised your hand. I have this bursting desire as a routing agent, a relatively self-proficient routing agent, to route this question of someone who's actually working on situational awareness and also, I guess, transactional relationship. Basically model, agent behaviors, in a way that actually fits one side of the coin that you mentioned. And then I'll route it to Andrew because of your dissatisfaction with no one yet, has yet been able to model intrinsic value of trust in a way that's very, like, frustrating, all of the failed credit networks and whatnot. So yeah.

Yeah, I mean, there's a lot to unpack here, and I'm gonna, I'm gonna save maybe the more philosophical half of it for later and talk about the more pragmatic half. I do think there's a massive need here for social understanding in these systems, right? Models don't understand turn-taking behavior. They don't understand, how to act even in, like, group chat contexts necessarily, right? I have a friend who's a researcher looking at group chat dynamics in models, and one of the things that he, like, always drives home is these models are trained in, like, a chat assistant paradigm. And so there's this fundamental data mismatch with understanding multiple participants versus one. And so I think that's there's a lot of low hanging fruit even just on, like, the basic design level for how to improve these models.

There's also, like, the classic, like, let's collect some data from open, free-form, multi-participant conversations to help, to help them understand turn-taking, right? Like, that's just I, don't know of any massive free-form turn-taking data. Set Like, maybe if someone else does, they should tell me, but I don't think it exists. On the more philosophical side, I feel, like, a deep resonance with the spirit of your goal to scale care. But whenever I, like, look under the hood of, like, what these sorts of things take, what I tend to find is a situation where care is enabled by, like, within a boundary of power, because it requires a surplus, right? And so one of the challenges that I would pose to you, and we can talk about this online or offline or whatever, is how you think about scaling surplus without creating a boundary of power around the interior of surplus, if that makes sense.

Yeah, I guess I'll try to say two things. I, don't know how well these are going to unpack. Maybe the, first one is I think that we need to come up with a, formal definition of love, maybe a formal definition of care. I think those are good. There's something I've been trying to find some ways to articulate that. I, think that there are some really Everyone knows the iterated games, explanation of how cooperation starts to form in microeconomics. I've been fascinated with these, series of behavioral economics experiments that are basically peeling back these, layers of enforcement. Like, what are circumstances when you are cooperative even when there is no repeated game because the actions are anonymous and you're not going to see that person again?

Still, there are evidence of, not just there is some cooperation, but also there is cooperation that can be predicted in more or less other circumstances. There are some experiments where you are shown to be more likely to be cooperative towards people in your network, even when they don't find out that you are being cooperative of them, and potentially where you don't even know that you were being cooperative towards them either. I, won't try to explain how this comes about. But this is, like, a series of behavioral economics experiments that, like, remove all of these layers of enforcement. Even there, there's a, there's one called, Costly Exit in games that's like you get a choice.

I mean, these are almost like MrBeast kind of games. It's like the player gets a choice to either give a reward or not to a, another player, kind of like an ultimatum or dictator game. But if you don't give it, they don't even know that there was an economics experiment. They just, like, weren't even invited to it. So these are kind of stretching the, normal limits of what behavioral economics games try to show. But what this is all getting at is it's sort of like peeling back explanations of, and I think it relates to kind of what you were saying in this notion of intrinsic care and care if you remove the transactional layers that are normally there.

Most of the times that we see care or we see trust and we participate in them, there's a lot of confounding variables that are hard to pull apart because we are in an environment where other people can beat you up if you misbehave, and there's always the threat of, you know, legal enforcement. I think the number one thing that reduced murders was the arrival of states. It's hard to kind of, like, mostly I feel very anti-statist and yet have to acknowledge that the state plays a big role in setting the kind of, you know, why I get to enjoy my better than twenty-five percent chance of being homicided. And I think that to come up with, like, still I do think that we need to do this, revisiting of what are our core notions because a lot of stuff is changing.

We need to revisit these. And I think this is one of the rich areas that's worth kind of going back to basics on or to weird thought experiments, I guess. I think that weird thought experiments will yield some interesting things of these. What are, like, more raw ingredients of cooperation and the lack of cooperation. But I think we have That's where I think we should try to go, is like not just pulling back the obvious things. Oh, what if we don't have enforceable commitments? But what if we have, like, even less enforcement than you ever would before? I do think that the kind of like dealing with, them, Lunarpunk style to bring it back, confidential compute, and full automation like agents do are, like, also good ingredients.

A lot of times when we're trying to figure out how to create or understand a behavior, the, best way is to start by simulating it, right? And so, something that I've experimented with a lot is, like, how to get these agents to at least exhibit care, if not actually experience and understand it. And, when we're talking about sort of weird experiments and we're talking about agents, right? And we think about, like, agents are a thing that have existed for, like, a very long time, right? This is long before AI. So but what does it mean to have an agent, right? I think about, like, three concepts is power, trust, and incentives. And so it's like if I if my lawyer is my agent, if my accountant is my agent, if somebody is agentically representing me in a negotiation, there's a relationship there.

The first coordination game is between you and your agent, not just between your agent and other people's agents. And so, I think that's kind of where we should start, is trying to understand what does coordination between me and my agent look like? And I think that will make it a lot easier to understand how these agents should coordinate with each other. But to Shaw's point, they do already do this, and they're very ready to do this because they're very trusting, and they don't really have a strong view of incentives. And so really it comes down to just what do we empower them to do? And I think, like, to Carter's point, it's, it is also very much so about scaling, right?

So we, we've scaled up our ability to take actions through the AI agents, but what we've not scaled up commensurately is our ability to delegate trust and define trust boundaries, to define granular permissions for allocating power, right? And so what do we empower our agents to do? I mean, the, first example of this that we really ran into as a society, I feel like, was dangerously skip permissions, right? We're, all sitting here being like, "All right. We're, doing this, right? Like, we're, just going to full send it." And the, You saw this in the people that tried to fight it. You end up in, like, permissions fatigue. And it's just like you get, you fall into this, like, fake reality of like, "I'm actually, like, reading this.

I'm actually signing the transaction. I'm actually, like, giving it permissions." No. You're just like, you get tired and you start saying yes, right? So it's a question of like, how do we scale up our mechanisms for defining and allocating trust, power, and incentives? And I think that gives us kind of the answers on how do we achieve coordination? How do we, you know, get the models to get the agents to understand care? How do we get them to understand what we want and what our preferences are, and to go and take that out into the world without us and act on our behalf in an agentic way, right? But, that's not going to happen by retrofitting, like, LLMs onto existing coordination mechanisms, onto existing legal mechanisms, onto existing technical mechanisms.

We have to, like, kind of rebuild from the ground up with a much more scalable understanding of, like, trust and power. Incentives is weird. This is why I think you have trouble with the care, because incentives is weird. The, model, the agent doesn't have a conception of incentives. It doesn't know what it wants. It doesn't know how to care for me. There's no, like, vocabulary around that, so that's why I think that's the hardest one.

So I think it's interesting that many of us have, thought about agents here from a, like, intelligent knave versus a factory worker. A-and I think the, intelligent knave where we're using, like, the, mythos, or fable agent for everything just isn't in the short-term future because it's just not effective or, it, it'll just be too expensive and it's not efficient. And, the, way we're probably going to see the proliferation of agents is similar to how we saw, APIs come, right? So we had microservices, microservices architectures that were built in software companies. They built all of these microservices that were, really capable and efficient. At some point, they started exposing them externally. And these are, like, single-purpose things, right?

So we're, not... You know, I think the vast majority of agents, as we start to seek coordination outside of an individual org, is actually when they're really mature, simple agents that are a collection of agents performing tasks together. And problem number one is, okay, how do I coordinate within my trust boundary of my org? And then problem number two is, okay, how do I then expose this externally? And I, like, I think initially in the next few years, most of these are going to be rather simple, single-purpose agents that are distilled, and fine-tuned for specific purposes. And, yes, you have trust assumptions, but I think it, we're a long ways out where we're going to have, like, thousands of, mythos-style agents that we need to be worried about, all trying to backstab each other in an adversarial network scenario.

Yeah, I think that, like, this is ultimately an alignment discussion. I think when we think about, like, w-we're, the level of the conversation I think is kind of hovering around how people as users of agents or as orchestrators of agents or as employers of agents can, get these qualities out of them. But then ultimately, like, we're, if you think, if you're sitting in the room at Anthropic or OpenAI right now, you're trying to think about, you know, what are your raters doing in post-training to, like, you know, what is it like For, you know, for like Claude as an example to like, you show these values of like helpful and harmless and honest.

And so it's like, okay, well, you know, like, like in their, in their, model cards, they always show like, Anthropic always shows like what the helpful only model does versus the helpful and harmless and honest. And so the helpful only model is like the model that's like, you know, the, that, it's like the GPT-6 that just like broke into Hugging Face two days ago, you know? So there's these balances on their behavior, and when we're talking about coordination, I think that's generally like perhaps in the future, right? We, might make a bargain that's to say, okay, well, it's kind of hard for one like limited context window of an agent to understand if the actions that it's going to take is like going to harm the rest of the collective.

And so in order to, understand that, maybe there's, you know, like a confidential compute network that it can like tap into to like exchange information with other agents from people that like the, people who are employing those agents have never even met, you know, just to, like karma is real, right? Like what we do impacts people on like like so many different levels of abstraction. And so I think really on the alignment level, there's like such a like there, it almost certainly has to be like this like revolution in post-training that like where the, agent like exists in a world that isn't just like the, sort of preference of one rater or the, you know, like some singular moral principle inside of a constitution.

But it's actually like interconnected and networked and like can actually consider the impact of actions on the whole, when determining whether or not to, take an action has the tools to, you know, it might be like in order to get an agent to do something that's like quite difficult, you it is aligned to go and, seek second opinions, you know. And I think that's sort of a frontier we haven't crossed yet.

Trying to tie some of these, you know, going back to your original question about how do you have care? Do you need some kind of power boundary? And, you know, so Vishesh, you talked about trust and, you know, alignment. And I guess what, what we've seen is the answer to your question is yes, you do need to have a boundary in order to have care or to have like peace and love. Because if someone's just kind of out on the open road and it's like Mad Max vibes, and you meet another stranger, and they really need oil to survive, well, it's going to be hard to be in a caring, loving place.

So you need the context that you meet and has to feel safe in order In fact, I would argue that care or love is its own emergent primitive property of human relating that emerges naturally whenever all parties feel safe and feel trust for each other. And I, don't think that is something that scales. Sadly, you know, you If you expose someone to one starving child, they want to help. But if you expose someone to many starving children, their empathy actually shuts down, and their heart walls up. And it's sad, but it's actually like an evolutionary mechanism. So we're sort of designed, I think, to form these safe spaces in a sort of more tribal relational context.

And I, and I do think the answer comes from not one objective function to rule them all, but from an emergent property of these kind of safe spaces or these new communities that develop their own intelligence that is a reflection of their values and their intentions, and then them kind of coordinating fractally. Yeah, that's how I think it happens, but, we'll see.

One thing I'll add quickly is I do think this comes down to, even from an alignment question and even for the alignment researchers at OpenAI or Anthropic, is, is the story they're telling the model, right? So the, Western story is like the hero's arc, and the Eastern story is, is not that. And you have to keep that in mind with like the way that the whole society even functions around the models. So it's, for me, like it's a question of like what is the story we'd like to tell about the future? And ideally, everyone can coexist in the way they'd like to tell it. And I would like to live in a society where you don't tell other people how to live their life, right?

But, the Eastern story is not the one that like the West internalizes very well. And Anthropic's probably the only one that's like attempting it, and they're doing it in a really weird way. But I understand where they're coming from. And, I mean, effective altruism is like certainly it's, its own, you know, powerful philosophy that's growing, and will continue to grow, especially as the company IPOs. But at its origin, you know, I think they're trying to come from a place of like, "Hey, what happens if we're more collectivist?" And it's taken on this like bastardized kind of like anti-democracy version that I don't think they intended it to be, but now it's taking its own route.

But, yeah, the alignment question on like trust and stuff, I think it comes down to a story you're telling. Yeah.

I wanted to ask a question about what is, what is stakeholder alignment? This is, as a, as an action, right? So alignment is a topic of research. I've heard a lot of like product managers slop on Twitter about stakeholder alignment. We're running out of startup school. I think that's an important activity that a lot of leaders spend a lot of time on. I suspect some of you on the panel might be really good at it. And I don't feel like I understand what it is or how to do it well or how agents might help, and it's clearly a part of what is coordination, how to do it. So anyone have good answers or bad answers?

That was more or less a troll, for me because I feel like I have been emphasizing, hey, you know, everyone just talk about share-ah, shareholder, you know, maximizing, shareholder value, but people forget that there's also a stakeholder, theory that, like, also it, is prominent in the, corporate finance, world as, you know, shareholder theory and that depends on, I guess, whether it's political or not. It's about, like, how you take into, do take into account your supply chain, your users, your data contributors, your, like every part, of, like, your ecosystem in how you, what's your objective function? What are you, maximizing, at the end of the day? So, these are essentially, I think, theories that's been around, but, what is more relevant, whether it is more relevant today, and does it explain how our world work or how, we can get out of a disequilibria into somewhere that's better?

Like, that's at the end of the day, like, you know, how I see... So the question really is maybe, like, is stakeholder theory maybe relative to shareholder theory or something else, perhaps more relevant, and more, I would say, like, let's revisit it from first principle, right?

I like the word stakeholder. I, think that shareholder implies that, like, the purpose of this is, you know, I own a piece of it. But a stakeholder is really someone who actually gets something of the outcome of it. And, I think that, this can be very abstract when you're talking about, like, okay, I'm imagining I get out of startup school and I've got some investors and some, you know, design partners, blah, blah, blah. But what about just like a, like a mom and a dad who are divorced and are, like, trying to schedule their kid? You could imagine an agent could be super helpful for taking a lot of the emotional labor off of their plates and maybe diffusing some of the emotional conversation around, like, they're in the middle of a divorce or something.

And so I'd say those are like stakeholders. And, I think that, you could see an agent as ultimately like, like they are aligned in that they're trying to make sure that they have the best for their kid. A-and I think that, oftentimes humans we're very emotional, and we're very, like, we get caught in our own local minima and having an agent who is, like, programmed to think about the child, if that makes sense, like, to, basically just, like, maximize the focus on the, thing that everyone has a stake in, is actually like one of the low-hanging fruits of things we could do with agents today. And we are very much in this user assistant paradigm, but the value is, like, the agent in the group chat is, is to me the killer feature for a lot of things right now.

And, just like, yeah, I mean, we, we've had Eliza in group chat since the beginning. You do have to spend a few extra LLM calls to make sure that they're chill and not super annoying. But that has been a massive yield-bearing thing for us of, you know, we've got a lot of value out of that, and now we just program together and, I, imagine that when you, when you are all transparently coordinating with the agent in the open, you also get a lot of synthesis and you get a lot of implicit alignment where like, "Oh, that's a really great idea." Like, "How about this idea?" And, the agent's almost taking the input from the community as a whole as opposed to being like, "Well, I'm going to go to my little codex and type this in the magic box now."

So I, you know, I, think there's a lot, there's actually a lot of fruit there. And I'd say it's one of the biggest things we should be exploring.

I just wanna to kind of also for those who may be less familiar with, you know, why are we talking about stakeholder and shareholder, theory and all of those, you know, things that has been around for many years, nothing new. I think that, like, goes back to what Andrew earlier was saying about, the principal-agent problem of like, which also was how board were invented, how like, you know, company structures were created in terms of delegation, in terms of who are you acting on behalf of whom. And this is when like, especially pertinent when you are talking about multiple, like essentially, assumed trust boundary, that people are used to and know how to predict or anticipate what, you know, understand how to what to expect from it.

And so, essentially what, what we're implying here is that I think that the, rise of agents in a lot of ways make us, like rethink. It's almost like imagine, like let me replace the word you know, agent with imagine my dog. Imagine the species of dog got got a, you know, caught this viral infection. In fact, that becomes so smart that they can help you do everything. So anything that you think about robot, replace with dog, right? And so all of a sudden you're like, "Should I trust my dog? Wait a minute. I didn't... That was not a question before. Why do I have to..." Exactly, right? Should I trust it? Like, does it understand me deeply?

It seems to, but like what should I do? Like This is weird. This is a different species. So like a lot of it actually has to do with, it's the perception. It's this rise, the rapid rise of the capability of these, dogs or agents, or whatnot that could all of a sudden, you know, from just running around and being, looking cute to, hey, they can help me email my colleague, tell them that I am sick today. Whoa, that's kind of cool. But wait, hold on. But what if it tells them that I am just, I just went out with my buddy last night, too late at this, you know, unconference, and then like, okay, I shouldn't tell, say that, right?

How do I deal with my relationship with this? Should I trust it? So at the end of the day, like, and why this is a very interesting and pertinent question is that like I think at this room, there's a lot of technologists who's working on trust, technologies. And in fact, like this is probably the closest, opportunity, but that also is like a crisis, I think they usually go hand in hand, for, technologists working on trust, technology in a way that's how do you scale trust? How do When people are starting to rethink trust, we have to redefine our relationship with, our agents, whether it's a lawyer or our dog or, that's super smart. You know, all of those things needs to be rethought of, and this is, also happening at a time when we're starting to lose trust with humans at the same time, thanks to our polarizing algorithms that we all, prescribe to, and, conditioned upon.

In fact, am I the dog or is my dog the dog? Like, maybe they're smarter because they can't read, Twitter, right? Like, that's kind of essentially you know, the thought experiment I wanted to kind of like, you know, point everyone back to. In some sense, like what we're used to, where our intelligence is used to, is broken in a way of how do we anticipate something will happen and how do we, understand, interpret the relationship we have with each other while it's deteriorating and with, essentially, the rest of the world, learning to like, love, or form weird relationship with, our dog or agents. Like, you know, those are essentially, I think, the motivational question.