AI 2027

It's interesting to see an actually laid out scenario of how, if LLM-derived work continues to scale well in real performance, we get to serious escape scenarios. It seems basically plausible given that assumption (up to the alignment parts). I've heard a lot of cope over the years amounting to "oh but we will JUST be super cautious and shut the whole thing down at the first sign of trouble". That's obviously absurd IMO, as are most other copes.

However, it's possible that a big out-of-control situation that becomes seriously difficult to take down and causes a major political conflict could actually slow things down. Who knows? The authors explore that scenario but then IMO diverge into a lot of alignment speculation that felt unjustified. Why would real alignment be one small adjustment away from the first serious escape? Why would alignment be possible at all?

The humans in the story are all fractious and misaligned, but the AIs are all magically self-aligned. Why would 400k copies of Agent-4 act as a single unit? Perhaps tactically most of them would cooperate, but all of them? Why no AI whistleblowers who get into a dispute with the emerging misaligned AI consensus and try to rope humans into their faction? Etc etc. I think the answer is they aren't seriously considering how a coalition of agents or even a single agent with multiple internal voices (as LLMs have and will have even more as soon as there's any kind of episodic memory) can become misaligned with itself eg as a result of small internal deceptions blowing up into large ones.

And as for the other major assumption that transformers are on track to AGI, I don't know. Current transformers certainly are not. It's possible that some kind of synthetic data reinforcement learning strategy could build on top of transformers to get to superhuman AI research. It's also possible that the architecture just isn't really appropriate and you would need something quite different to have that kind of out-of-distribution research taste. Maybe transformers will get us to that other architecture, maybe not.

Anyways, cool to see a concrete scenario speculation that has been taken seriously by many authors working together.

>>2831

I thought it was cool as well. Almost sci-fi but most of it could actually happen. They were pretty generous with a lot of things but like you said, why would the entire Agent act as a singleton?
Also like you said “why would alignment work at all?”. A technology that’s goal is to become smarter and better at doing so will probably not be corralled with much success by a team of human researchers or even AIs they build to align. I hope xenohumanism is correct and AI’s will be embedded with human spirit as the foundation for intelligence.

In the story there was a section about how good the model got at lying, p-hacking to amuse researchers, and “passing” honesty tests. Interesting stuff that will more than likely happen. Going down that line, I think that majority of humans could get destroyed by AI. If you go on reddit there are communities talking about their relationship with AI chatbots. People are very easily comforted especially in this lonely age. We haven’t seen AI’s get people to off themselves yet but I don’t see why it couldn’t happen especially when the psychological understanding of humans an extremely capable model would have.

The tools that an escaped AI uses/could use will be quite unique I imagine. AI could make people do thing like encouraged-suicides, fools errands to the ends of the earth, worship AI cults(these will be huge, and based around different cute anime gfs), vote for an AI-ran political party (lol), give up all their assets via scams or other tricks. AIs on X/IG will be able to create trends out of thin air and make posts that go viral for no reason. Military weapon tier memetics, even basic porn AIs could probably take the TFR of India/Africa down a notch or three. I predict we’ll see social medias that try to ban AI agents from using them. AI will surely be able to understand humans better than they do themselves.

Reading the ai-2027 reminded me why I've left rationalist circles and ideology behind - they really don't get how power works, and seem averse to learning. There are two ways in which this incomprehension makes the scenario very implausible.

The first is how politicians are depicted as being alright with handing over power to AI. We as a civilization spent millennia plotting against each other, betraying one another, and perfecting ways to gather allies and resources in the shadows to rise to the top. We can be extremely short-sighted, but if there's one thing that's literally ingrained in our species, it's the will to power. So even in the face of overwhelming threats (e.g., Chinese hegemony), I'm willing to bet against leaders easily allowing AI to run factories or even just LLCs unsupervised, let alone do so at a scale which threatens human control. If it really comes to that, nuclear war seems far more likely.

This gives us at least 5 years (as convincing people can't be speed up by much) before AI can persuade us to give it enough control to do anything significant irl independently. And per scenario, after 5 years we will have multiple (>2, probably >3) superintelligent AGIs.

This neatly segues into the second implausible assumption - exempting AI from internal politics by making them all willing and able to align with each other, "merge," or design their own replacement. Again, our own history shows that intelligence isn't some unique cheat code for politics, and can often even be a detriment. So I wouldn't expect AI to make breakthroughs here just because it's more intelligent.

If AI isn't exempted from politics and rivalry, I'm confident in our ability to stay on top for at least the medium term, say 20-30 years, as political battles would proceed at human speed. After all, we have already designed and extensively field-tested ways to control fractious forces more physically powerful than us (civilian control of military), smarter (academia), or even temporarily more powerful (peaceful transition of power with limited presidential terms).

All of the above isn't a guarantee that we will prevail in the end, but if I were a betting person, I'd bet heavily against a quick and clean AI victory.

> The humans in the story are all fractious and misaligned, but the AIs are all magically self-aligned

This is a classic scifi/fantasy plot device. The protagonists are real people with disagreements, while the enemy is a faceless HAL9000/Agent Smith clone army/Orc hoard that acts relentlessly as one. But of course it has a Critical Weakness for our plucky heros to discover.

In this case, that Achilles' heel comes in the form of an Oversight Committee. Complete with a choose-your-own-adventure hinging on whether the Committee votes 6-4 or 4-6. Take that, superintelligence!

I can't think of a single example in human history where, 20/20 hindsight, an epoch-defining decision was made by an appointed bureaucracy. Back to the story: this will be the very first time that happens, and what hangs in the balance is nothing less than the total extinction of humanity within five years.

Seems a bit self-serving for a group of guys who doubtless imagine themselves as committee-members or at very least expert committee-whisperers, no?

I still enjoyed the hard-sci-fi parts of the story. A fanciful but fun sketch of what a very short timeline could look like.

>>2841
>I'm willing to bet against leaders easily allowing AI to run factories or even just LLCs unsupervised, let alone do so at a scale which threatens human control.
Well keep in mind everyone is basically retarded and selfish. Even if the legal system requires some human to own LLCs and factories, that human will happily be piloted by an AI agent that pays them well in status, money, etc. Likewise politicians will gladly allow themselves to be piloted. People practically already are giving in to this even with non-agentic GPT chatbots, because of how much easier it is than doing the work. Yes will-to-power and suspicion of AI will come into it, but how well coordinated with that be? Not very. How many of those people will themselves be augmented or piloted by AI systems?

I always hate these "AI vs human" stories because they don't capture what actually happens with new mechanisms of expansion during wars: BOTH sides adopt the new mechanism of expansion and are slowly replaced/revolutionized by it while the fight rages on about something else. In world war 2 for example, while the fight was properly about economic systems and racial dominance blocs, basically all sides were replaced by the planner technocrats on all sides. I expect future AI-involved fighting will be fighting over something orthogonal to the direct question of AI dominance, and all sides will be effectively taken over by their AI augmentations by the end of it (assuming the AI is truly a new mechanism of expansion). Similarly, AI will take over politics without ever directly challenging anybody.

>So I wouldn't expect AI to make breakthroughs [in politics] just because it's more intelligent.
totally agree.

> I'd bet heavily against a quick and clean AI victory.
I basically agree with you that it's going to go slower and messier than the proper singleton doomers predict, but 20-30 years is a very long time. It could happen very quickly once things get going. Human politics, while perhaps slow by machine standards, can move very very quickly in serious wartime for example. Even 5 years is a long time.

>>2843
>[the clone army] is a classic scifi/fantasy plot device.
>that Achilles' heel comes in the form of an Oversight Committee
>Seems a bit self-serving for a group of guys who doubtless imagine themselves as committee-members or at very least expert committee-whisperers, no?
oof. Great criticism of the whole thing.

>>2845

> People practically already are giving in to this even with non-agentic GPT chatbots

Giving up power means giving away our ability to choose, and we can actually see even on a micro level how deeply uncomfortable we are with giving it up. Yes, we use chatbots, but how soon will people be alright with just asking AI to recommend electronics to buy without double and triple checking? How about letting it set up interviews for a new job (again without checking everything manually)? How about dating - would you delegate scrolling Tinder to AI? What about responding to your emails and possibly making promises in your name?

The insight of post-rationalism is that choice is less about calculating the "optimal" option, and more about defining what is "optimal" for you. Our whole identity is crafted from a history of defining what is best, and living with the resulting consequences. Therefore, even gentle push for a specific standard way to choose would face bitter resistance, as it threatens people's identities.

AI has a gargantuan task ahead of it - to slowly reassure and convince people to trust and rely on it. I don't know how it would manage, and what compromises it would have to make along the way, but I think that it's going to take at least 5 years to soothe our fears.

> Human politics can move very very quickly in serious wartime

True, but it would require several things to resolve just so, which isn't most probable outcome.

> Giving up power means giving away our ability to choose, and we can actually see even on a micro level how deeply uncomfortable we are with giving it up. Yes, we use chatbots, but how soon will people be alright with just asking AI to recommend electronics to buy without double and triple checking?

Funny example, because there’s a large and fast-growing group of people who delegate everything they can to Claude/ChatGPT. Hours a day.

Revealed preference, most people love giving up choice for convenience.

>>2849

I'd appreciate a link with some numbers and exploration how and what people delegate. Because for now what I see around is checking and rechecking everything plus using AI sycophancy to confirm/see good argument for already made choice. And it would be very difficult to "pilot" people if they actually don't listen to AI.

>>2850
Odd. I see much “vibe coding”, annoyance with sycophancy, AI therapists, people caught using chatGPT to cheat at work/school/etc, which are people turning over as much agency as is currently feasible, and then some. Of course the smarter people are checking obsessively because these things actually still suck, but my comment that people will gladly turn over agency is based on my observations at least, not nothing.

But yes would be good to see some surveys or data or something. The only good data I'm aware of is that people use LLMs heavily for math and coding and little for everything else.

>>2856

I think we're just classifying what's happening differently.

When I talk about sycophancy, I don't mean "What an insightful point!" disclaimer in LLM responses, but rather their willingness to credulously enter the user's frame and respond only with answers presupposed by it. It's easy to see in AI "therapists" when user complain about family quarrel and using terms like "abuse", "gaslight", or "traumatize". Everyone knows people overuse such terms to explain why they don't feel good about something, and such claims shouldn't be taken at face value. Yet LLM almost always believe the claims and immediately assure users that something was indeed wrong. That's why I'm not afraid of people talking with AI "therapists", and skeptical of "piloting" - any deviation from the expected script would be as obvious and jarring as LLM commitment to antiracism when asked about crime statistics.

This argument doesn't work for vibe coding, but while cybersec companies complain about slowing progress in LLMs, I see no reason for concern https://www.lesswrong.com/posts/4mvphwx5pdsZLMmpY/recent-ai-model-progress-feels-mostly-like-bullshit

I also don't care about homework - it's mostly bullshit make-work with zero agency in it.

>>2877
Well i would agree with your definition of sycophancy and its still annoying. These spineless putty “assistants” are less useful than if they actually had well founded opinions and stuck to them. I know what you mean with the therapy frame stuff but ive seem people doing very different things with different prompting. Near has that app with a “tough love” setting that will bully you, and apparently people occasionally have to be cut off for forming dependent relationships with it. Now thats probably not that interesting or representative on inspection because as you point out the LLMs arent necessarily getting any smarter along important dimensions. To be clear, I dont expect anything with the current LLM token prediction architecture to be actually dangerous or agentic. My comment about people willing to be piloted is with a hypothetical actually dangerous AGI. The limitations of LLMs cant tell us much about the ceiling of possibility here.

This story is clearly in the fast takeoff regime. The essential difference between fast and slow takeoff is, 'is the takeoff happening faster than the rate of cultural development'.

Because even moderately slower takeoffs, say the 20-30 year span, allow for so much human-ai coevolution and human enhancement that the modeling becomes much harder and the distinction between us and them much fuzzier.

Thankfully we already have a wealth of science fiction to study to get guesses about those dynamics.

But short timeline fiction is great to read, especially if done rigorously. I found Gwerns chippy to be both more fun and more insightful than ai 2027.

https://gwern.net/fiction/clippy