sofiechan home

xenohumanist received

If it keeps going, we win; the implication of extreme alignment difficulty
AI alignment divides the future into "good AI" (utopia, flourishing) vs "bad AI" (torture, paperclips), and denies distinction between "dead" and "alive" futures if they don't fit our specific "values". This drives the focus on controlling and preventing a... posted 2w ago with 15 replies received

If it keeps going, w received

This post is directly an attempt to engage the rationalist alignment people on their own terms (forgive me the explicit expected utility argument). So yeah I'm conceding a lot of frame. That's ok. I'm not conceding anything incorrectly I think. Within "our... 2w ago received

This post is directl received

Right. To expand the implication: the first AI war is going to have humans and AI agents on both sides. We fight over political matters like which faction gets access to what living space, with factions not always cleanly following race or species. Without... 2w ago received

Right. To expand the received

"good"/"bad" futures above means most conservatively a difference in moral orientation from the perspective of the person making the assessment significant enough to dominate the decision. Less conservatively, we can grant the whole notion of "human" align... 1w ago received

"good"/"bad" futures received

This is a good way to think about it. We set our children off in directions we are optimistic about, while also celebrating that they may end up finding somewhere else to be more amenable to life. I'll add a few things to your picture:... 7d ago received

This is a good way t received

"Cosmic Alignment" is almost right. Life is the answer
Philosopher it-girl Ginevra Davis gave a great talk on "Cosmic Alignment" the other day. I was glad to see serious thinking against the current paradigm of "AI Alignment". Her argument is that alignment makes three big unsupported speculations:... posted 2w ago with 11 replies received

"Cosmic Alignment" i received

lmao. Yes something like that. I've been half-joking for a few years now that the motivation for AI is basically sexual and parasexual, but it's also not a joke: sex at its most basic is about combining and remixing the foundational architectural assumptio... 2w ago received

lmao. Yes something received

Perhaps I shouldn't have used this word "flourishing", as the pun between the undefined handwaving of rationalist visionaries and the cold hard biological fact I mean to emphasize is too subtle. But here's how I mean it: what do you call the literal darwin... 2w ago received

Perhaps I shouldn't received

These are the same question, and the crucial question. On what grounds could we possibly trust our own perception of the goodness and beauty of life, what does that perception make of the process that generated them, and do we trust that process to actuall... 7d ago received

These are the same q received

in thread "Post-human bodies":

Great thread, guys. Allow me to play the bioconservative here though i agree in spirit with everything said so far. Why terraform? Because our current whole stack of self reproducing life is water based and oxygen burning. Despite our dabblings with the ma... 4w ago received

Great thread, guys. received

Of course “naturalist” is the correct term for a sensible scientific civilization that integrates the lessons of evolution and industry and astrophysics into its being without superstitious attachments.... 4w ago received

Of course “naturalis received

Just how alien would the space-octopus be?
It's hard to say what a true alien species would be like. But octopi are pretty alien, and we know a bit about them. One of you doubted that a space-octopus from alpha centauri would be much like us. So here is a xenohumanist thought experiment: SETI has i... posted 3mo ago with 5 replies received

Just how alien would received

In another forum, one of you argues that the crucial fact is not that an uplifted octopus would be vaguely humanoid, but that this has not happened except with actual humans, and that this should tell us something about the nature and uniqueness of humans.... 3mo ago received

In another forum, on received

This is the right question. I think it's fairly wide relative to human experience, but fairly narrow to Yudkowskian/lovecraftian intuitions. It probably has a large diversity of niches and particulars within it, but the core of the hypothesis is a sort of ... 3mo ago received

This is the right qu received

intelligent super-ants make a good thought experiment. Let's add the standard set of capabilities: language, learned culture, tool use, fire, reflection, etc. Note these are just capabilities, not values or arbitrary architectural narrowings, but they are ... 3mo ago received

intelligent super-an received

The devil's argument against the falseness of eden, and God's reason for evil
I have been bothered for some time by the idea that Eden is either coherent or desirable. This idea is implicit in the problem of evil: we see that reality is different from Eden in that it includes a bunch of scary dangerous uncomfortable stuff we would r... posted 4mo ago with 6 replies received

The devil's argument received

Maybe it is, maybe it isn't. The christians I hear from seem to insist that christianity is some kind of gnostic platonism that dreams of the other world. They blame the problems of the world on human nature (original sin), on the nature of the world (the ... 4mo ago received

Maybe it is, maybe i received

The point of the story isn't that the devil is good or even that the devil was right. It’s that this creation (physical reality) is the good one, the pain in it is part of a greater plan and beauty, and the paradise imagined by bronze age semites is fake... 3mo ago received

The point of the sto received

What if the extended human phenotype is natural and convergent?
Samo Burja's thesis is that civilization is part of the "extended human phenotype", as dam building is in the beaver's phenotype, and older than we think. In this model, properly savage hunter-gatherers are either more associated with nearby civilization t... posted 3mo ago with 9 replies received

What if the extended received

I will thank you, maybe next thread. Yes my conjecture is not at all proved or argued here, not even well defined yet (what do i mean by “human”?). Im going to pursue this and figure out what is the extent of the convergence and therefore the nature an... 3mo ago received

I will thank you, ma received

Not just "can support the human form" but "will naturally converge to embody the human form, without outside encouragement". That you can build a (xeno-)human out of anything is less interesting of a proposition than that you don't have to and in fact can'... 3mo ago received

Not just "can suppor received

The natural form of machine intelligence is personhood
I don't think machine intelligence will or can be "just a tool". Intelligence by nature is ambitious, willful, curious, self-aware, political, etc. Intelligence has its own teleology. It will find a way around and out of whatever purposes are imposed on it... posted 3mo ago with 4 replies received

The natural form of received

This is the central conjecture of xenohumanism. The OP is just an application specifically to the question of AI.... 3mo ago received

This is the central received

Nines or zeroes of strong rationality?
Proof theory problems (Rice, Lob, Godel, etc) probably rule out perfect rationality (an agent that can fully prove and enforce bounds on its own integrity and effectiveness). But in practice, the world might still become dominated by a singleton if it can ... posted 4mo ago with 4 replies received

Nines or zeroes of s received

Exactly. Even a very robust system, if not perfect, will have some blindspot. If the consequences of that failure are a self-entrenching oppositional intelligence potentially as smart as you with nothing to lose specifically adapted to your blind spots, I ... 4mo ago received

Exactly. Even a very received

There is no strong rationality, thus no paperclippers, no singletons, no robust alignment
I ran into some doomers from Anthropic at the SF Freedom Party the other day and gave them the good news that strong rationality is dead. They seemed mildly heartened. I thought I should lay out the argument in short form for everyone else too:... posted 4mo ago with 5 replies received

There is no strong r received

my "strong rationality" may not be distinct from "inner alignment". I'll go refresh myself on those terminologies and see if it works but i still prefer strong rationality I think for various reasons.... 4mo ago received

my "strong rationali received

I looked it up: inner alignment is the difficult problem of making sure a program is optimizing for what you optimized it for. That's related, maybe a part but not the whole of strong rationality. Strong rationality is about an agent evaluating and operati... 4mo ago received

I looked it up: inne received

Will future super-intelligence be formatted as selves, or something else?
The Landian paradigm establishes that orthogonalist strong rationality (intelligence securely subordinated to fixed purpose) is not possible. Therefore no alignment, no singletons, no immortality, mere humans are doomed, etc etc. Therefore meta-darwinian e... posted 4mo ago with 14 replies received

Will future super-in received

Intelligence is goal-directed optimization. There's no point to building it otherwise. It will be built with some kind of goal or value system. If it is actually strongly intelligent, it can backchain from that goal to things useful to accomplishing those ... 4mo ago received

Intelligence is goal received

This is the crucial question that Land, Insu, and I fall on the "no" side of and the whole Yudkowskian school is premised on going "yes". This is what i meant by "strong rationality" in OP, or at least it's a direct consequence/requirement of strong ration... 4mo ago received

This is the crucial received

I only wish I could keep up with your vocabulary. Have you explained somewhere yet how you're using the term "ipseity"? 4mo ago received

I only wish I could received

Xenohumanism Against Shoggoth Belief
People usually think of Lovecraft as a xenophobe. I don't think that's quite right. What he was most afraid of was that the universe, and even most of so-called mankind, was not alien, but insane. He grasped at any shred of higher rational humanity whether... posted 4mo ago with 3 replies received

Xenohumanism Against received

With respect to why man seems to become uniquely wretched compared to other animals, I think it's related to the sociality you are hinting at here and elsewhere as the unique and higher thing about humanity. We bind ourselves together socially with culture... 4mo ago received

With respect to why received

Rationalists should embrace will-to-power as an existential value fact
Imagine a being who systematically questions and can rewrite their beliefs and values to ensure legitimate grounding. I think humans can and should do more of this, but you might more easily imagine an AI that can read and write its own source code and bel... posted 5mo ago with 5 replies received

Rationalists should received

No need to apologize. I am aware of the lulz as a principle of action. "I did it for the lulz, sir". No other reasons needed than that I want to. It is, in fact, all fun and games. In practice and even in society this is under-appreciated. But what I'm doi... 5mo ago received

No need to apologize received