Not really. Let me try to clarify what I meant.

We already know that rewards and punishments influence our actions. Any utopia would try to satisfy them. Even in a complex optimized universe full of un-wireheaded sentients caring about external referents, people would want to avoid pain, ... and experience lots of excitement, ... . Wireheading just says, that's all humans care about, so there's no need for all these constraints, let's pick the obvious shortcut.

In support of this view, I gave the example of the entertainment industry that optimizes said experiences, but is completely fake (and trying to become more fake) and how many humans react positively to that. They don't complain that there's something missing, but rather enjoy those improved experiences more than the existent externally referenced alternatives.

Also, take the reversed experience machine, in which the majority of students asked would stay plugged in. If they had complex preferences as typically cited against wireheading, wouldn't they have immediately rejected it? An expected paperclip maximizer would have left the machine right away. It can't build any paperclips there, so the machine has no value to it. But the reversed experience machine seems to have plenty of value for humans.

This is essentially an outside view argument against complex preferences. What's the evidence that they actually exist? That people care about reality, about referents, all that? When presented with options that don't fulfill any of this, lots of people still seem to choose them.

You're arguing as though neurological reward maximization is the obvious goal to fall back to if other goals aren't specified coherently.

I'm not. My thought process isn't "there aren't any real values, so let's go with rewards"; it's not intended as a hack to fix value nihilism.

Rewards already do matter. It describes people's behavior well (see PCT) and makes introspective sense. I can actually feel projected and real rewards come up and how decisions arise based on that. I don't know how "I value that there are many sentients" or any other external referent could come up. It would still be judged on the emotional reaction it causes (but not always in a fully conscious manner).

I think I can imagine agents that actually care about external referents and that wouldn't wirehead. I just don't think humans are such agents and I don't see evidence to the contrary. For example, many humans have no problem with "fake" experiences, like "railroaded, specifically crafted puzzles to stimulate learning" (e.g. Portal 2), "insights that feel profound, but don't mean anything" (e.g. entheogens) and so on. Pretty much the whole entertainment industry could be called wireheading lite.

But aren't you just setting up a system that values states of the world based on the feelings they contain? How does that make any more sense?

Acting based on the feelings one will experience is something that already happens, so optimizing for it is sensible. (Not-wireheaded utopias would also optimize them after all, just not only them.)

A major problem I see with acting based on propositions about the world outside one's mind is that it would assign different value to states that one can't experimentally distinguish (successful mindless wallpaper vs. actual sentients, any decision after being memory-wiped, etc.). I can always tell if I'm wireheaded, however. I'd invoke Occam's Razor here and ignore any proposal that generates no anticipated experiences.

I like marginal revolution, if only because the comments section will usually yell at them when they post something stupid.

Overall, it sounds to me like people are confusing their feelings about (predicted) states of the world with caring about states directly.

But aren't you just setting up a system that values states of the world based on the feelings they contain? How does that make any more sense?

You're arguing as though neurological reward maximization is the obvious goal to fall back to if other goals aren't specified coherently. But people have filled in that blank with all sorts of things. "Nothing matters, so let's do X" goes in all sorts of zany directions.

2:30 is a good time to go to the dentist.

If it was determined that you were the best candidate to be Gandhi-Einstein, would you volunteer?

Dear Buck:

All I said was "looks like rain".

Q: Is it important to figure out how to make AI provably friendly to us and our values (non-dangerous), before attempting to solve artificial general intelligence?

Stan Franklin: Proofs occur only in mathematics.

This seems like a good point, and something that's been kind of bugging me for a while. It seems like "proving" an AI design will be friendly is like proving a system of government won't lead to the economy going bad. I don't understand how it's supposed to be possible.

I can understand how you can prove a hello world program will print "hello world", but friendly AI designs are based around heavy interaction WITH the messy outside world, not just saying hello to it, but learning all but its most primitive values from it.

How can we be developing 99% of our utility function by stealing it from the outside world, where we can't even "prove" that the shop won't be out of shampoo, and yet simultaneously have a "proof" that this will all work out? Even if we're not proving "friendliness" per se, but just that the AI has "consistent goals under self-modification", consistent with WHAT? If you're not programming in an opinion about abortion and gun control to start with, how can any value it comes to regarding that be "consistent" OR "inconsistent"?

Wikipedia's Epistemology - How Wikipedia determines truth. I'll let David Gerard tell us what that was about

Um, OK. This is an inchoate thing I've been bouncing around in my head for about the past six months. To attempt to summarise ...

Normally, it's a four-year liberal arts degree to learn the subtle arts of weighing up unreliable human-generated evidence and turning it into useful information. The way Wikipedia works means that you have to explain all that from scratch to argumentative teenagers with Wikipedia-induced aspergism in three paragraphs, and they'll still argue it, 'cos it's not like there's people who really do know more than them about abstracting knowledge from data, is it.

This means that Wikipedia has evolved its own epistemology of where knowledge comes from. It means there's a massive systemic bias against fields that aren't favoured by people who don't think like that. It also generates absurdities like regarding newspapers as "reliable sources", which anyone who's ever been quoted in one will laugh hysterically in horror at.

This is treated as though it is not just one epistemology of many, but the epistemology of how to abstract truth for an encyclopedia.

This is enough of a problem that I know humanities scholars who know Wikipedia in depth but are having to work out what the hell they can do about this, as academic experts in various fields start bringing themselves to Wikipedia even if it gets idiots in their faces, just to get their field properly represented.

A further problem is that early Wikipedians were encyclopedia nerds who could answer "What's an encyclopedia?" by pointing to Britannica and saying "It's a bit like that." There are kids now who have never had any other encyclopedia than Wikipedia. So "what is an encyclopedia?" is coming loose from history. This may be good or bad. I suspect it's bad but would be willing to be convinced it wasn't.

The above needs work and, the hard bit, proposed solutions. That last is what I've been stuck on.

There are no known structures in conway's game of life that are robust. Even eaters, which are used to soak up excess gliders, only work when struck from specific directions.

If you had a life board which was extremely sparsely populated, it's possible that a clever agent could send out salvos of gliders and other spaceships in all directions, in configurations that would stop incoming projectiles, inform it about the location of debris, and gradually remove that debris so that it would be safe to expand.

At a 50% density, the agent would need to start with a fairly large safe space around it, otherwise it would be overwhelmed. I can't imagine even the best sensing/manipulating technology in life is capable of picking its way through even mostly static garbage at any more than a glacial pace.

Basically you'd have to send out a probe, wait for the echo, or lack of echo, and from that, recalculate the probabilities of all the different configurations of still lifes and oscillators and spaceships and puffers and so on that the probe could've hit, and how those configurations would've been altered or destroyed or (in most cases) expanded due to collision with your probe. And then work out another probe to send, and repeat the process, until eventually you had a good enough estimate of what you were dealing with that you could send probes calculated to get rid of it, and all the additional garbage you generated in the process of probing it.

It is unknown whether robust structures can exist in life, even if incredibly intelligent, incredibly large, and incredibly slow, but I would speculate that they can.

However, it's also possible that there are far simpler robust expanding patterns, in which case, larger slower structures such as intelligent agents would be hopelessly overwhelmed.