CoordinationOS

I'm Shashank, and today I'm presenting about CoordinationOS. Let's just think about how things are happening today. Previously, it's been one human, another human, they're trying to collaborate and work together. But now we got, like, each human has a team of hundred agents working at the same time, and if they're not aligned, they would be going into different directions, and it's very hard to align humans with a bunch of agents. And this has been happening even in cohorts like this, especially, like if there's one company as a unit, I could take all the company data and put it in one brain and one knowledge base. That would work perfectly fine. But think of an accelerator cohort like Shape Rotator.

There are multiple teams trying to collaborate with each other, and they have different trust boundaries on how much they're willing to share with the coordinators or the rest of the cohort. And this has been a common pattern across different distributed teams. So we started looking into the Shape Rotator OS. This is the first instance of the, like, bigger vision of the CoordinationOS. So here, there are some of the requirements which have been enforced into the design. Andrew does not like filling any forms, and he wants everything to be ambient. And because most of the meetings were happening in, like, online and offline being recorded, we started using transcripts as the ambient source. Like, if you think deeply, the agent traces and GitHub repositories are showing more towards the execution.

But who is actually prompting these prompters? It tends to be humans and other humans. So we wanted to go closer to the source and capture this information. And Conclave team here has been building the transcription app with complete privacy, and we integrated with them to get in the transcripts. And another requirement is there are different trust boundaries. Teams don't want to share everything with other teams, but coordinators have a different version of what information they know. So we have to respect the principal agent problem and just work with how much of our information is being shared for each, each agent or each human in the system. And the requirements or understanding of each human, is very different, and the goal is to capture this taste and preferences into the system.

So there are two phases to this architecture. The first one is to build the brain, and the second one is to keep running the loop and keep learning in this process. So the first step, we started using transcripts. There's Conclave and also Granola, Otter, and other non-privacy preserving ones, we use as a primary signal. And this data, as you would expect, tends to be very messy and noisy. A lot of people have aliases, and people work on different projects. So for example, I could just see Andrew just involved in multiple projects. So we created some ontology to just define who is where and what are they currently dealing with, and all the relationships which are happening within the cohort, and use this to clean up the data and resolve people.

And we have a small classifier model to understand who can this data be shared with certain amount of confidence, and use that and also generate the insights out of this. We look into two types of memories here. One is, semantic memory, and the other one is episodic memory. These are coming from the concepts of cognitive science, where, the semantic memory is understanding the relationships between different entities and my perspective of the world. And episodic memory seems to be the episodes which happened in the past: what, when, and where, and tends to be a good fit for a vector database. For storage of all of these, we are partnering with Synclave, which is providing complete confidential, storage services for us, and TinyCloud is helping us manage the access control and trust graph.

Like knowing where this information can be routed at insight level and also the underlying data level. Like basically, what information do you want to trust to be shared with each team depends on the situation and the context where that is happening. And, we've been using local models... Sorry, the models which are open weight, through BitRouter, and most of these are just workflows happening through Smithers. Now, this is a workflow for the transcription. There's also secondary data sources like the GitHub repositories. Most of the projects here are open source. We have information from Teleport Router, where people share their agent traces, with us, and also they get docs and public information. So with that, we have the knowledge base being built.

The next phase is actually running the loop and keep iterating the loop. So we create skills and basic runtime to get the first version for the users. Here it's coordinators. And, then we have every human review with a Codex running on the cloud, helping us, track every result, every accept or change or evaluation which is happening to understand each user's individual preferences. Our goal in the long term is to have like a frontier model and a small memory model understanding every user's taste and preferences and capture them to be shown. The output, would be different insights being generated. We've been running the system, on all the transcripts here, looking at different types of meetings. And this is an example of a dossier, which is being generated with provenance to where each thing is coming from, whether it is GitHub transcript.

I've just heard that, like I don't, want to just show the UI. So in the last few hours, we just had a Hermes agent as well just plugged in to just show some questions being asked to this data directly. And yeah. So what we started is with the coordinator and the coordinator brain, because this coordinator is actually benefiting from finding the complementary nature for the project. They want to be like aware of what is happening with the teams, track the momentum, and treat it, like to find valuable insights. Because it's complementary and it's a positive sum game, we are trying to find different places where this could be applied. One of the goals here for this cohort is also to make sure that trust boundaries are well, managed.

So every organization would have its own version of the brain, which might not be complete compared to the other versions of the brain where their agents are acting. And so today we have Shape Rotator OS, which is, the first version of this. And now we are looking into distributed teams, which, research orgs generally like Flashbots, and then towards venture capital and private equity firms, which tend to have similar topology of how the organizations are structured. The goal is to just have different networks being generated and all of them being connected to do the routing.

Given how little time there is, in actually shipping the thing. So, I was going to ask dmarz to do like a before and after, as he was the first, he and his agent swarm, was prompted at some point before the cohort started, to ship a Shape Rotator OS, and no other instruction was given. And so, dmarz wanna, you know, show the before and then what Shashank, I'm not sure if you showed shared screen of how it's going. Because I think a lot of it was aspirational. So, like I think the reason why I'm asking, for this, you know, demonstration and contrast is because this is something that I think all of us, especially doing R&D, is going to understand and, appreciate this journey.

You start out shooting for the moon, and then you realize you're not even, like, leaving the ground. And then you rebuild, refactor. By the end of the program, you're maybe still at square one. I think in week nine, we went back five times into data pipeline, into transcription like quality and raw data, quality and our, essentially using all kinds of fancy techniques that everyone has been sharing with each other. And so I want to do a reality check, and this is something that I also want to almost like end, our marathon of last human demo day and the graduation, the first Shape Rotator graduation day with, before we have the you know, Easter egg of the live Vibe Coding Room, live stream from upstairs, which is we need to face the reality, and we need to know where we're headed to.

So can you just do a quick, through of the spec and the prototype V0 before the real human rotated into, Shape Rotator OS?

So this version, this is V zero point two point twenty-seven. This was a fully functioning Electron app. The idea was to give you actually access to your file system and do a bunch of other stuff. One of the other reasons I went with this path was because all of our communication was over Matrix, which is an open source like Slack alternative that you can just... Andrew started running one in a TEE. Anyone can connect it. We actually forked it and like integrated it into this. And so now, like, just imagine you're not constrained by Slack anymore. You can literally just put an agent in it. You could do whatever you want. You can add in phone auth, all these things.

So we went with this approach over a website originally. This was my This is also how I learned, that the AI can do, front-end stuff. So I got a little crazy here. But you know, you've got this like rough profile. It says your team, a bunch of other things. Here's the cohort view. So this takes to the clusters. This was one of the first visualizations we made, which was really just putting people into their categories. So agentic infrastructure, TEE, a bunch of stuff like that. We also mapped out dependencies and then the like, the journey that everyone was on. So how far along in, product market fit they have versus, evidence quality that we have.

These were cool, but they weren't that insightful, and automating them became incredibly hard with the variety of devices we have for transcripts. I mean, some days we were recording with two different styles of microphones into three different devices, into Otter, into a local, vibe-coded transcription tool, and a bunch of other stuff. And so, actually automating this entire pipeline was extremely hard. There was just a calendar. Hunter's coordinated tea time was on here every day.

Oh, and this was a nice onboarding guide you have here. So at Flashbots, we have like a, Notion onboarding guide that Mike made. It We, learn something new, we need to onboard someone every time. So we literally just had an agent looking at all the issues in the repos and adding onboarding stuff as people came in. They checked it off. Some program info and some asks, is about it. Here's the peer-to-peer network. So no, no peers online anymore. And yeah, this was also a place to, as I mentioned earlier, plug in apps. So we had Easel, which is actually Etherea's, sort of application. And so I won't show it, but we didn't need HDMI cords anymore when Etherea was set up.

You could just hit a button in this app, and now your screen was shared. So it was pretty cool.

So that was the before. I arrived, here in week two, and I was like, "Holy shit, this is sick. People are actually using this. Wait, people, are they using it?" And then, so there were a few, I think fans, because this entire setup design was very idealistic to be local first and privacy preserving, like to the maximal extent. Like you could see like basically, everything that we set up, like Matrix server was, super high onboarding friction. I onboarded to Matrix server in week six. And like essentially, and then like basically self-hosted Matrix server with a peer-to-peer nodes connection and whatnot, and not everyone's online. And then, like essentially all of those kind of added to the friction in addition to the pipeline didn't work.

However, there's one thing that worked, which was the core the clustering, visualization, where you see the cohort actually converge, and that was the first thing that I used. Despite it was, very iffy, it was very fuzzy, it was still sufficient to show, the different projects that are in affinity and help me as a coordinator to steward the conversations such that they could find, juxtaposition, and understand each other's specialization. But beyond that, it was hard to be for, more useful. And I was never onboarded to, essentially the Easil tool because that would have saved a lot of our time in terms of actual projection, not needing to find a cord.

Yeah, I'll just double-click on that local API design, actually. So yeah, we try to do local AI here. There's a bunch of GPUs in the basement we try to use when we're doing AI, if possible. And what we also want to do is local-first data storage. So instead of every time you, like, you log into Slack, you actually just start hitting the Slack API. It starts fetching you all your latest things. And so all of these like different platforms are structured around a centralized database. But I took the opposite spin, where it is you contact all of your peers you're connected to, and you get all the latest info. Yes, one of them sometimes needs to get the latest info from a centralized database.

So one example is we were actually tracking everyone's GitHub in here. And so the first node on the network during the day, would actually go hit up the GitHub API, scrape everyone's projects, and download it. And then when a peer comes online, you know, say two hours later, it would actually ask the nodes in the cohort, "Hey, do you have the latest GitHub API?" Before I hit the GitHub API. And so that was working, but it was also how I got the school's IP banned for two days from GitHub, because we were hitting it too hard. Yeah, there was a Like permissioning in this type of environment is incredibly hard because you have a peer with some permissions over data, and they give it to another peer, and then that peer will give it to another peer, and it's very easy for, like, permissioning to mess up.

Yeah. So I feel like we learn it the hard way, which is the truth that we always know, when you do R&D, or you build a product. Maybe fake it to you make it instead of doing the hard way, literally build out a peer-to-peer network because, you may want have, fix some UX and think about a human, you know, who's gonna use it first, what they're gonna use it for, and how they're gonna use it. So when I encountered it in week two, there was no UX, period. There was no way as a coordinator to, for me to actually upload my transcript easily for me to even know what's going on. So I was like, "Uh-oh, this looks great."

So one thing I did accomplish, and this is going, like relevant to quite a few of the folks in, like quite many of the projects that Shashank has demonstrated that, is on the roadmap to be integrated or already integrated into Shape Rotator OS, is that despite it was not quite working, but it was a beautiful prototype It caught the attention of every single one of my investors, 'cause, you know, like, as a, Flashbots, has like, I think half of the, VCs, you know, on in Silicon Valley on our cap table. I get bothered by them about like, "What have you been up to?" Like, so I have to do these investor, like updates. And so I was yeah, there's this, Shape Rotator accelerator for 10 weeks, you know, we're focusing on and..."

So then I was like, "Oh, what do I show them? Okay, cohort and what, where do I show them on?" The Shape Rotator OS. And this is the one place we learned, to my shock was like, they were like, "Holy shit, this is cool. We need this." "We are doing, like, we have this portfolio company doing agentic transformation. We, are a VC. We have this portfolio of projects that they're live humans and live, like, projects and products and blah, blah, blah. It has the same problem." I was like, "Cool. I guess I kind of accidentally fake it till you make it." Sold this beautiful visualization. Like, I was like, "Okay, well, now I kind of have to make it work."

That's why, like, essentially, the part that was not said in Brendan's Shape Rotator data story, was that there was two weeks where I, was no w-was I was off the radar, and week six, I came back. And the thing is, everything in Shape Rotator OS accelerated. Why? I was in San Francisco, and a ton of investors were asking me about like, "Hey, so how do I use this? Where's the beta product?" It's like, "Oh, yeah. It's, local-first. We will need to, do some work on it. We'll be right back. Bye." So that was actually what happened. So Dmar was like, "Tina, you went from like two weeks ago, you were just, 'Oh, well, it's one of the things that will be emergent from this project.

If someone wants to pick it up, great.'" And then two weeks later, I was like, "Shashank, you need to shape rotate. If you want a customer, I can hand you my customers, but we have no product. So let's go." So I guess now, maybe show where things are.

Yeah. So like the first commit happened nineteen days ago on this repository, and, it started with transcripts. Mike has been following up with everyone to just collect transcripts and put it in one drive. So the goal was to just make sure that uploading transcripts is easier, and just paste any link, any source. And it, started with Auto Granola Conclave and like Voxterm and Pocket. There's so many different transcript services. And then it just became, how do we even know what metadata to collect? And so example, a weekly meeting has every team. So then we have to again break it down to all the teams, know who's, who can this data be shared with, what app is this coming from, who are the participants?

And, we spend a lot of time on doing those classifications, designing a schema, and then getting the source, correctly cited. And, the next stage for us is, okay, the transcripts are not really a perfect source of information. So extracting some, Yeah.

So extracting the right insights was really hard. So the goal was then to capture who is working on what projects and create the ontology for that and some of the projects have Shape Rotator to change their names. The team members have changed. You could just see TSQL is also called Synclave, and that happened during the cohort. And, we just want to track all those changes in a deterministic way. So we define the ontology, use this data to do the corrections of the transcripts itself, and making sure the topics and entities are correctly mapped. And then, it went into creating the dossiers. So even for dossiers, now we want different versions of dossiers. One is a public version.

Like for example, this is a pitch which the founder is comfortable showing outside, based on, like the data they've have been sharing publicly. And some of the metrics here are coming from, the GitHub and online sources as well. And now there are also some, places where we have, data which is coming, like this is like an entire coordinator output, which just comes from transcripts and other sources. Some of them are from GitHub, other are from AI, and these are the transcripts of that particular sessions. Weighing recent transcripts over the old transcripts was something we had to do. Things are moving very quickly, and forgetting the old facts has been a big problem for all these memory systems.

And, yeah, managing the permissions correctly has been a challenge we are trying to actively address.

So there's a lot to make it something useful, and usable by humans, as we can see. But I think that this is something that many people, whether VCs or other, accelerators, that, I know of, they were like, "Oh, yeah," like, "We..." "Once you get it working, like we'd like to, check it out, and we can be the co-design partner." So there's definitely, When you build something useful for ourselves, you'll be able to see, like it may be the same pain point that many people share.