Does anyone have any insight into VoI plays with Bayesian reasoning?

At a glance, it looks like the VoI is usually not considered from a Bayesian viewpoint, as it is here. For instance, wikipedia says:

""" A special case is when the decision-maker is risk neutral where VoC can be simply computed as; VoC = "value of decision situation with perfect information" - "value of current decision situation" """

From the perspective of avoiding wireheading, an agent should be incentivized to gain information even when this information decreases its (subjective) "value of decision situation". For example, consider a bernoulli 2-armed bandit:

If the agent's prior over the arms is uniform over [0,1], so its current value is .5 (playing arm1), but after many observations, it learns that (with high confidence) arm1 has reward of .1 and arm2 has reward of .2, it should be glad to know this (so it can change to the optimal policy, of playing arm2), BUT the subjective value of this decision situation is less than when it was ignorant, because .2 < .5.

I actually brought up a similar question in the open thread, but it didn't really go very far. May or may not be worth reading, but it's still not clear to me whether such a thing is even practical. It's likely that all substantially easier AIs are too far from FAI to still be a net good.

I've come a little closer to answering my questions by stumbling on this Future of Humanity Institute video on "Reduced Impact AI". Apparently that's the technical term for it. I haven't had a chance to look for papers on the subject, but perhaps some exist. No hits on google scholar, but a quick search shows a couple mentions on LW and MIRI's website.

Is it even possible to have a perfectly aligned AI?

If you teach an AI to model the function f(x) = sin(x), it will only be "aligned" with your goal of computing sin(x) to the point of computational accuracy. You either accept some arithmetic cutoff or the AI turns the universe to computronium in order to better approximate Pi.

If you try to teach an AI something like handwritten digit classification, it'll come across examples that even a human wouldn't be able to identify accurately. There is no "truth" to whether a given image is a 6 or a very badly drawn 5, other than the intent of the person who wrote it. The AI's map can't really be absolutely correct because the notion of correctness is not unambiguously defined in the territory. Is it a 5 because the person who wrote it intended it to be a 5? What if 75% of humans say it's a 6?

Since there will always be both computational imprecision and epistemological uncertainty from the territory, the best you can ever do is probably an approximate solution that captures what is important to the degree of confidence we ultimately decide is sufficient.

open-source prisoner's dilemma

I believe the GNU GPL was made to address this.

It seems like we are moving in this direction, with things like Etherium that enable smart contracts.

Does anyone have proof that Etherium is secure? There's also the issue of giving whomever runs Etherium complete authority over those 'smart contracts', and that could easily turn into 'pay me to make the contract even smarter'.

Technology should enable us to enforce more real-world precommitments, since we'll be able to more easily monitor and make public our private data.

People are going to adapt. And I see no reason why would anybody share particularly private stuff with everyone.

And then there's the part where things look so awesome they can easily become bad: I can imagine someone being blackmailed into one of those contracts. And plenty of other, 'welcome to the void' kind of stuff.* Where's Voldie when you need him?

We already have legal contracts to do this. If I make a website and sell a product I however people to cooperate. They can make a contract with me and then I am precommitted to deliever them the product they paid for.

How do you distinguish precommittments from simple contracts?

If you are standing in the market selling apples for dollar a pound, have you precommitted to anything?

Generally speaking, precommittments are expensive because you pay with optionality, the ability to make a choice later. There must be a good reason to precommit, something other than "wouldn't it be generally useful".

"So conservation of expected moral evidence is something that would be automatically true if morality were something real and objective, and is also a desiderata when constructing general moral systems in practice."

This seems to go against your pulsar example... I guess you mean something like: "if [values were] real, objective, and immutable"?

A few questions, and requests for elaboration:

• In what ways, and for what reasons, did people think that cybersecurity had failed?

• What techniques from cybersecurity were thought to be relevant?

• Any idea what Mallah meant by “non-self-centered ontologies”? I am imagining things like CIRL (https://arxiv.org/abs/1606.03137)

Can you briefly define (any of) the following terms (or give you best guess what was meant by them)?: * meta-machine-learning * reflective analysis * knowledge-level redundancy

FYI, Dario is from Google Brain (which is distinct from Google DeepMind).

Comparing with articles from a year ago, e.g. http://www.popsci.com/bill-gates-fears-ai-ai-researchers-know-better, this represents significant progress.

I'm a PhD student in Yoshua's lab. I've spoken with him about this issue several times, and he has moved on this issue, as have Yann and Andrew. From my perspective following this issue, there was tremendous progress in the ML community's attitude towards Xrisk.

I'm quite optimistic that such progress with continue, although pessimistic that it will be fast enough and that the ML community's attitude will be anything like sufficient for a positive outcome.