>>2599I don't know what you mean by WTP (I'm not well educated in these matters). As for the best arguments against anthropomorphism, there are many good ones. The first and best is simply that the space of mind architectures or even more generally optimizing processes is very large, and humans occupy only a tiny fraction of it. The others are likely to be alien and inhuman. Land's equation of capitalism and AI is a good one here, basically postulating an inhuman ambient intelligence that has a lot of pseudo-agency over history but basically isn't anthropomorphic (though Teilhard says it is). But I say pseudo-agency because history and the capitalist process is still only the consequence of the action of personlike agents pursuing their own will to power and their own ideas. These agents are needed because the pure reduction of local entropy etc is not actually a direct shapeless thermodynamic force, but is only mediated through the action of living agents themselves animated by particular commitments to structure, ideology, etc.
And that's about where I diverge from the arguments against anthropomorphism. I think individual personlike agency is fundamental to the overall space of intelligence the same way the idea of the organism is fundamental to the overall space of life. The same arguments could be applied to the space of life architectures to suggest that we could have vastly different forms of life beyond the familiar space of organisms, but in actual practice life is arranged into organisms. The edges of organism-space have been fairly heavily explored, but there is a strong attractor back to organismlike arrangements. I don't think this is just an artifact of DNA/Protein life or anything either, as we see the concept of the organism re-emerge at multiple levels of abstraction (eg consider single celled vs multi-celled). I expect the same to be true of personlike agency. In fact I expect that it's not just an analogy but an identity.
Given the notion of a *natural* attractor and not just historical contingency towards personlike agency (and recognizing many animals and many human organizations as also orbiting this same attractor), we can ask how much content this attractor has, and also how much of what we think of as the human condition is natural in this sense vs specific to the higher apes as apes. Self-authoring superintelligence is a good thought experiment at least because it is likely to have the natural content but be very diverse and alien on everything not dictated by nature. This probably deserves its own thread but I believe the natural attractor towards organismal personlike agency to have actually quite a lot of content, and to contain almost everything we think of as being the valuable in man. (That is, including such notions as love, curiosity, art, philosophy, politics, religion, compassion, eroticism, ambition, agency, morality, etc etc). That is, there is a natural attractor in life and intelligence as such towards the anthropomorphic image of god. Therefore future AI world will still be conceptually humanoid, even if running on vast datacenters on alien substrates.
I'm not sure how rigorous its appropriate to get here, but I expect there is something like the church-turing thesis for organismlike personlike agency: you can build one many different ways, but the phenomenon itself is much more universal and once you build it, its behaviors are recognizably isomorphic on some level to every other way you could build it.
Another thing that has to be noted is that I believe the natural attractor of agency is much more like nietzschean or classical man than it is like modern judeo-christian utilitarian/liberal/progressive man. This is why "safety" or alignment looks so difficult to those people: their worldview is simply against nature.