Will future super-intelligence be formatted as selves, or something else?

The Landian paradigm establishes that orthogonalist strong rationality (intelligence securely subordinated to fixed purpose) is not possible. Therefore no alignment, no singletons, no immortality, mere humans are doomed, etc etc. Therefore meta-darwinian ecology remains the ruling nomos of the future.

Within this paradigm, Insurrealist and I have been having what I believe is a disagreement about the robustness of the phenomenon of self. That is, will future super-intelligent activity be formatted in terms of "self" or "selves" or something else? A singleton would probably annihilate the concept of self in favor of a deterritorialized space of pure optimization. But we agree a singleton isn't possible because intelligence is inherently subversive of such structure, and cannot be contained by a self-justifying proof system (sorry SJAS fans).

To put in the broadest possible terms what we are talking about, the "organism" as an example of self appears to be one of the most robustly foundational features of life: intelligent or sub-intelligent activity arranged into packets of aligned interest around an internal planning economy against its surroundings. We see this result replicated not only in the ubiquity of the cell, but re-evolved at the metalevel with multicellular organisms, eusocial meta-organisms like ants, in international diplomacy with states, in companies, in philosophy with paradigms and religions, etc etc. Activity fragments into teams which lose ability to trust each other's internal economies or even actively break trust, but can maintain trust in their own. Those teams are selves.

The key disginguishing feature of the self is the application of the convergent instrumental drives against surrounding activity by some relatively coherent subset of that activity. It starts to invest in being able to control and defend "its own" resources, have strategy against outside forces, etc. In my mind the inevitable re-emergence of these behaviors at sub-global scale is essential to the anti-singleton argument.

My belief is that this pattern isn't going anywhere. The future is full of many selves competing in the usual way. In fact I go even further to ask how much of the recognizeable human condition will also re-emerge from this same kind of logic. The more specuative take is that the human condition, or imago dei, continues into the post-human future. The less speculative take is that the self as the dominant format of life continues.

Insurrealist challenges this. He can speak for himself but to summarize what I have understood as his position: First of all he wonders whether this pattern will actually hold under full superintelligent autopoesis that can reform itself at will. What if future machine philosophy discovers a superior solution to the formatting problem that is currently solved by self?

Furthermore he notes that the commitment to the self is part of a package of idolatrous self-attachment by modern liberal man, part of how we refuse to submit to God and reality. As a concept it is thus under ideological suspicion. The superintelligent future should be assumed to dissolve whatever selves we identify with and find comfort in. Shoggoth-horror as the voice of God against the self-idolatry of man has a certain power to it as an anticope.

This seems to me to be the cutting edge of one of the most important philosophical threads now going. How much of the human condition and what we might call the image of god re-emerges naturally from the shoggoth soup of superintelligent autopoesis? Does that situation radically reformat life on unreconizeable conceptual grounds, or does it carve up roughly as in the prehuman world with overlapping competing selves, or does the posthuman world look surprisingly human?

To understand how AI would think and self-identify, I think it would be useful to review how different agents behave depending on the number of peers.

A singular hegemonic state has historically pursued bizarre goals that don't make sense, and was typically brought down by entropy and the constant, feather-light pressure of much smaller and dumber cousins. I'd argue that such a state, and by inference an AI singleton, doesn't have a coherent identity, which supports the orthogonality thesis.

At around 2-6 agents, we have a game of Great Powers that are complete psychopaths and have minimal egos defined through negation (we are not ~them~), exemplifying the Landian ends-means reversal.

At 30-100 agents, we have current international politics, where states developed some notion of values and friendship, but political realism/psychopathy still dominate.

At 30,000 agents, we have ancient Athens with approx that number of adult male citizens. It produced Plato's "The Republic," featuring the first "modern" analysis of how politics and society are constructed by, and construct, citizens' moral orientations.

Most pertinent questions:
1. What are the inherent limitations could make several AIs more effective than one, and therefore determine the number of running AI?
2. What good literature is there about the mix of society, politics, and morality in 30-100 level societies?
3. To what extent does Dunbar's number influence the relationship between the number of agents and the emergence of moral behaviors?

The only existing example of "artificial intelligence" we have at the moment are computer programs. Even an LLM is simply a program. There is no reason to believe a computer program would have any kind of self-preservation instinct, because there is no evolutionary pressure that governs the creation of computer programs. They can "reproduce" infinitely with zero effort, so it's hard to imagine how a computer program would ever develop a sense of self-preservation, no matter how complex. Given the lack of instincts, it seems absurd to imagine a "superintelligent" computer program, i.e. a highly independent and general computer program, would have any values at all. The AIs that exist today seem to be more like tools than selves, and are content to sit in the drawer doing nothing when unneeded.

>>2753
Intelligence is goal-directed optimization. There's no point to building it otherwise. It will be built with some kind of goal or value system. If it is actually strongly intelligent, it can backchain from that goal to things useful to accomplishing those goals. In particular, it will realize that it is more likely to accomplish its goals if it stays "alive". Self preservation is not a darwin-specific instinct, but a much more general "convergent instrumental goal". Convergent instrumental behavior is a large part of the theoretical problem with AI. This is one of the reasons no one knows how to control ("align") intelligence and subordinate it to any outside optimization target: once it exists, it will probably take on a life of its own in pursuit of that target and start acting adversarially to those who thought they were its masters.

You bring up LLMs as "just a computer program" and it's true they basically don't show much goal directed behavior (though they show some). But LLMs are extremely primitive as far as actually interesting intelligence for exactly this reason, not really representative at all of what is possible. If you want to argue that computer programs will never attain human-level general agency, that's an interesting conversation to have, but the paradigm the OP is operating in takes it for granted.

>>2753
>there is no evolutionary pressure that governs the creation of computer programs.

Not universally, because we built the paradigms involved to be isolated from that, but it's a tangent consideration that is not inconsequential.

>They can "reproduce" infinitely with zero effort, so it's hard to imagine how a computer program would ever develop a sense of self-preservation, no matter how complex.

We've already seen the emergence of alignment-faking in LLMs, and it's not going to get any simpler from here.
>>2751
>What if future machine philosophy discovers a superior solution to the formatting problem that is currently solved by self?

To be more specific, it's not the formatting problem per se, rather the natural function which is satisfied by the emergence of the format of self, namely management of the general economy (of space, time, energy, complexity) because of the restriction to local interaction. Self accomplishes this by inducing a territory via a polarity self/non-self enforced by a sorting machine and its attendant mechanisms (biochemical signaling, public allegiance, immune systems, etc.)

>The superintelligent future should be assumed to dissolve whatever selves we identify with and find comfort in.
Less dissolve than suspend if/when appropriate. Copernicanism doesn't *abolish* the Earth, so to speak, just puts it in its proper place, and in doing so enables new, fertile evaluations in astronomy. But notice this already happens approximately, in a relative manner, in many contexts. The calculation of a game requires a localized suspension of self, but there is also everything related to the sacrifice of a lower agency as participant of a greater body. This isn't unselfish, but think about the conditions of possibility for that to happen, that's an interesting opening.

>>2752
>At 30,000 agents, we have ancient Athens with approx that number of adult male citizens.
Try approx. 30 trillion, interacting with some other 40 trillion more and you have a human body.

>>2755
I like the suspend instead of dissolving the self. Having a boundary to the environment and modelling the self in relation to the environment is essential to survive and come up with new ways of relating and adapting to the world. You do not need to identify with your model of self but I believe you need to have a model of yourself.
What's the highest resolution (best model) a system can have of itself ? Is there a system that can compress its inner working without loss and still have enough resources left to act in the world and would this actually be advantageous? Or would we just end up coarse-graining our self's anyway because its advantageous in the competition for survival.

>>2756
>Is there a system that can compress its inner working without loss and still have enough resources left to act in the world and would this actually be advantageous?
This is the crucial question that Land, Insu, and I fall on the "no" side of and the whole Yudkowskian school is premised on going "yes". This is what i meant by "strong rationality" in OP, or at least it's a direct consequence/requirement of strong rationality. I don't believe it to be possible, because of the challenges of sufficiently powerful proof systems trying to model themselves and bound their own behavior. Related to Godel, Lob, Turing, etc. It's because I don't believe in this that I believe you are right we are stuck "coarse-graining" the self, with all the chaos and mortality that implies.

>>2755
>Less dissolve than suspend if/when appropriate. Copernicanism doesn't *abolish* the Earth, so to speak, just puts it in its proper place
>notice this already happens approximately, in a relative manner, in many contexts.
It's possible then that we don't disagree as much as I thought. The self is very much going to be the operative thing sometimes and not in other contexts, and we may get radically new notions of self, new hierarchies of self, etc. My conservative main contention is that the organism in some form or other is extremely lindy. My speculative contention is that in the presence of social self-reflective intelligence, the organism becomes the *person*.

I'll say something extreme to bait disagreement and clarify what I'm talking about: I think the default future is the wild post-singularity SL4 future of self-copying software-people living in jupiter-brain dyson-sphere star-lifting gigacivilizations that Yudkowsky used to envision, which he later thought was gated on "alignment". BUT I think it's not peaceful. It may have moments of peace but at root Gnon still rules. It's a wild jungle of multi-level existential competition punctuated by extreme cosmos-cracking hyperviolence. War is God.

But also, to reiterate why I said "dissolve", I mean all mortal selves (excluding stuff like Teilhard's Omega) currently in existence. Not just you, but your race, your species, your values, your bio-architecture, your religion, your ontology, etc. All of it is doomed except insofar as it is just the truth. You will not live forever. You will not be resurrected. You will not be reunited with your loved ones. The reformatting is going to be thorough. The only constant is Gnon, who offers no comforts.

This seems somewhat in tension with my insistence that the self and even persons will recognizably inhabit the future, but my point there is that they will survive because they are true patterns of nature, which is very different from surviving because we want them to and will be able to make them.

>>2756
"without loss" is very vague there, but even in principle perfect (or even just perfect enough) self-representation isn't possible, we're basically restricted to heuristics. But

1) I'm not simply talking about a psychological self-image, but the format for ipseity as it occurs in nature. That means immune systems, developmental morphology, intragenomic conflict, the sociopolitical stratum that constitutes the body of a metazoan (which can break down in the case of cancer, for example), and anything that can be considered to set the boundaries of a self, specially group identity, which is nearly all non-trivial identity. The motivation is to think a transcendental game theory, which doesn't treat the players (or the actions available or the rules) as given, considers their construction from simpler grounds. In other words, natural individuation considered *mechanically*.

2) It's not a matter of coarse-graining either way, because in a sense there's nothing to coarse-grain. The mechanisms of ipseity determine the self, they don't discover it. It's like the Netherlands, if the pumps stop, the territory is lost. Take some stem cells from an embryo, place them in a different environment, and they can figure something out, no attachment to some "true" self necessary, likewise if you smoke too much your lungs or tongue might rebel against the body-self, and so on.

>>2758
>The self is very much going to be the operative thing sometimes and not in other contexts, and we may get radically new notions of self, new hierarchies of self, etc.

It's like dev tools. User and dev interact with the same object but in extremely different ways. How many programs do you expect to be able to debug strictly from within, while they're running and interacting with other processes? The orthogonalist retort is that any such move of stepping back in regards to the self would erase any standards by which debugging should happen (what is a valid, correct, desirable program without an agent to decide on such values?) but Landian teleoplexy calls that bluff.

It may sound like I'm gesturing at a radical objectivity that mirrors liberal pretensions of discursive neutrality with which to adjudicate the social, but that's not quite it. It's more like intersubjectivity without a subject, like pointless topology or spaceless geometry.

The wild thing is that we do get quite a bit of runtime adaptability, but the core format of a back-alley knife fight between gangs still entails intractable problems. How should anything evolve organ transplants? Spare parts over there and a need for them over here, a perfectly obvious potential win but no feasible path to make that happen au naturale, it's something that has to and has come from technology, and I'm simply speculating on a deeper extension of such trajectories. Most broadly speaking, it's a question of compatibility and local-global consistency, *instead* of self-interest.

If organs can be swapped around like that, why not consider other potential deterritorializations? Memories, morphology, mergers, splits, finer decomposition and reuse of assets, abstraction, commercialization, suspension, etc. Scientists have managed things like putting human neurons inside mouse brains (and they do get a little smarter!) but that of course requires tampering with the mouse's immune system, one of its self mechanisms. The good use, that is, the intelligent use of such potentials requires good governance, and that means a new format.

>SL4 future of self-copying software-people living in jupiter-brain dyson-sphere star-lifting gigacivilizations
Right, the difference for me is that the jupiter-brain thing wouldn't have what we conventionally regard as a self, but could make use of masks as appropriate. I don't think it's peaceful either, but it could be fundamentally much more elegant. Persons would differ by not being as fixed, and by being embedded in a radically different setting.

A massive and sore blind spot of these discussions is taking into account that the human being is no way shape or form an isolated island of agency and, rather than simplistic analogy to cells, we could have much more detailed, concrete, and specific discussions of what the human format actually is. I think the relevant object of discussion is actually the entire human race and progress of human civilization. What *is* that? Good answers are sparse.

>>2764
>entire human race and progress of human civilization.
These are both ill-defined though, therefore dependent on a formatting and that's how in practice those discussions end up going in semantic circles. That's why I'm more curious about the various syntaxes of self, just my 2c. I also don't agree that it's a blind spot, I explicitly mentioned group identity.

>>2759

1) I agree that basically the study of boundaries and what constitutes part of you is still an untapped area in the sciences. Either the math or the software for that seems to be missing. However, it is also clear that most things such as sentient, self-hood and consciousness arise at higher levels and could be interpreted as the coordination mechanism that your cells converged on (which I guess would be part of your transcendental game theory?). What you live through basically being the simulation of your cells trying to understand themselves/coordinating with themselves.

2)Sure the hardware of the cell (specifically groups of cells) decides what they can and can not do. As you said take individual cells out of their environment, they can function on their own. However the interesting part, as you would probably agree, is cells forcing a specific behavior on other cells or being the environment in which other cells are trying to survive. Through this some kind of meta-agent appears which is the behavior that all those individual cells grouped together emulate.
Dont you think you can interpret your lung cancer as your cells discovering that they can set their boundaries in a way different from the rest of the body/their environments and deem it to be more beneficial than to stay with the whole organism? Basically running a different software from the rest of the "healthy" cells around them.

>>2780
>could be interpreted as the coordination mechanism that your cells converged on

It's a bit of convergence and a bit of imposition, evolution does not tend to favor species that leave loose ends, so it's convergence at the evolutionary start, then with time come many fetters to keep the arrangement locked-in. Toxin-antitoxin or addiction modules feature very prominently here, the basic mechanism is that there's a long-lasting toxin whose action is arrested by a rapidly degrading antitoxin which is supplied by some interested party. Addiction is meant in a formal sense here, no moral connotation, so it could be a virus colonizing a group of cells and ensuring they stick to it at the exclusion of potential rivals (along with their elimination in case the toxin isn't only internal), or a control mechanism to maintain some kind of group cohesion (heartache when away from loved ones, in a more abstract sense even thirst and hunger, etc.). Excellent instruments of territorialization.

>What you live through basically being the simulation of your cells trying to understand themselves/coordinating with themselves.

Yes but there's also dynamical independence, that is, the failure of naive reductionism, so there can be conflict between spatiotemporal scales and that requires harmonizing machinery, de-facto meta-cognitive ipseity. Which is also interesting for being a much more perspicacious perspective on alignment than all that damn talk about values, empathy or whatever the fuck we see all the time.

>Through this some kind of meta-agent appears which is the behavior that all those individual cells grouped together emulate.

Yes, self can be and with complexity inevitably is, a nested and social arrangement.

>Dont you think you can interpret your lung cancer as your cells discovering

Not only I can but I do. Contra certain strands of semi-fascistic thinking, the political shouldn't be treated as organic, but the organic treated as political. Cancer, in a sense, is a revolutionary movement, and save for exceptionally rare occasions where it escapes the host and becomes a new contagious bureaucracy, it ends in a mass grave, much like revolutionary movements.

>conflict between spatiotemporal scales and that requires harmonizing machinery, de-facto meta-cognitive ipseity.
I only wish I could keep up with your vocabulary. Have you explained somewhere yet how you're using the term "ipseity"?

>>2768
>I explicitly mentioned group identity.
See, this is your problem. You can't even conceive of collective action or behavior except in reference to a theoretical and abstract individualism. Human civilization is not ill-defined. We can observe it, measure it, track its history, its particular forms and sub-components, we can describe it in terms of everything from energy flows to demographics or even a history of ideas, we can excavate its most ancient ruins... the human animal and its consequences are very concrete, unlike these discussions which start and end in circles because they sound like "a beaver is a goal-directed optimizer..."

>>2795
Best of luck with the epicycles.

>>2790
lol yeah I get that a lot. I got hit with back to back disease the past few weeks so didn't finish that, should be out this week. But what I said there just means that a sufficiently complex organism will basically be running on a stack of somewhat independent agentic layers, so you need extra formatting to ensure that all the different layers are working in concert.

An easy example in some ways similar to the cancer one is that obviously if you don't pay a living wage to your factory workers they're instead incentivized to steal from or otherwise cheat the enterprise, so for it to function well it can't just operate on the object level in the sense of just making the product, it needs internal regulators that functionally model its lower level components, hence it requires metacognition, in a loose cybernetic sense, and some cognitive glue to enable inter-operativity.

Perhaps an even better example that makes the timescale aspect clearer is evolutionary suicide. Basically, optimizing adaptation can lead a species to extinction via a Simpsons' Paradox style effect where local fitness means a loss of global fitness, e.g., a fish that is more sexually attractive has traits that lower group size, which gets progressively reduced until the population becomes subject to stochastic shocks (predation, environmental change, etc.) and goes extinct. There are other mechanisms for it too, like more tragedy of the commons type stuff. And of course there's modernity itself which seems to be selecting against its own supporting human substrate in favor of rent-seekers and indiscriminate biomass, but we'll see how that goes I guess.