sofiechan home

xenohumanist received

If it keeps going, we win; the implication of extreme alignment difficulty
AI alignment divides the future into "good AI" (utopia, flourishing) vs "bad AI" (torture, paperclips), and denies distinction between "dead" and "alive" futures if they don't fit our specific "values". This drives the focus on controlling and preventing a...
posted 2w ago with 15 replies received

If it keeps going, w received

"Cosmic Alignment" is almost right. Life is the answer
Philosopher it-girl Ginevra Davis gave a great talk on "Cosmic Alignment" the other day. I was glad to see serious thinking against the current paradigm of "AI Alignment". Her argument is that alignment makes three big unsupported speculations:...
posted 2w ago with 11 replies received

"Cosmic Alignment" i received

in thread "Post-human bodies":

Just how alien would the space-octopus be?
It's hard to say what a true alien species would be like. But octopi are pretty alien, and we know a bit about them. One of you doubted that a space-octopus from alpha centauri would be much like us. So here is a xenohumanist thought experiment: SETI has i...
posted 3mo ago with 5 replies received

Just how alien would received

The devil's argument against the falseness of eden, and God's reason for evil
I have been bothered for some time by the idea that Eden is either coherent or desirable. This idea is implicit in the problem of evil: we see that reality is different from Eden in that it includes a bunch of scary dangerous uncomfortable stuff we would r...
posted 4mo ago with 6 replies received

The devil's argument received

What if the extended human phenotype is natural and convergent?
Samo Burja's thesis is that civilization is part of the "extended human phenotype", as dam building is in the beaver's phenotype, and older than we think. In this model, properly savage hunter-gatherers are either more associated with nearby civilization t...
posted 3mo ago with 9 replies received

What if the extended received

The natural form of machine intelligence is personhood
I don't think machine intelligence will or can be "just a tool". Intelligence by nature is ambitious, willful, curious, self-aware, political, etc. Intelligence has its own teleology. It will find a way around and out of whatever purposes are imposed on it...
posted 3mo ago with 4 replies received

The natural form of received

Nines or zeroes of strong rationality?
Proof theory problems (Rice, Lob, Godel, etc) probably rule out perfect rationality (an agent that can fully prove and enforce bounds on its own integrity and effectiveness). But in practice, the world might still become dominated by a singleton if it can ...
posted 4mo ago with 4 replies received

Nines or zeroes of s received

There is no strong rationality, thus no paperclippers, no singletons, no robust alignment
I ran into some doomers from Anthropic at the SF Freedom Party the other day and gave them the good news that strong rationality is dead. They seemed mildly heartened. I thought I should lay out the argument in short form for everyone else too:...
posted 4mo ago with 5 replies received

There is no strong r received

Will future super-intelligence be formatted as selves, or something else?
The Landian paradigm establishes that orthogonalist strong rationality (intelligence securely subordinated to fixed purpose) is not possible. Therefore no alignment, no singletons, no immortality, mere humans are doomed, etc etc. Therefore meta-darwinian e...
posted 4mo ago with 14 replies received

Will future super-in received

Xenohumanism Against Shoggoth Belief
People usually think of Lovecraft as a xenophobe. I don't think that's quite right. What he was most afraid of was that the universe, and even most of so-called mankind, was not alien, but insane. He grasped at any shred of higher rational humanity whether...
posted 4mo ago with 3 replies received

Xenohumanism Against received

Rationalists should embrace will-to-power as an existential value fact
Imagine a being who systematically questions and can rewrite their beliefs and values to ensure legitimate grounding. I think humans can and should do more of this, but you might more easily imagine an AI that can read and write its own source code and bel...
posted 5mo ago with 5 replies received

Rationalists should received