"But the general result is that one can start with an AI with utility/probability estimate pair (u,P) and map it to an AI with pair (u',P) which behaves similarly to (u,P')"

Is this at all related to the Loudness metric mentioned in this paper? https://intelligence.org/files/LoudnessPriors.pdf It seems like the two are related... (in terms of probability and utility blending together into a generalized "importance" or "loudness" parameter)

Then you get spurious proofs of inconsistency :-)

(If PA is strong enough to prove what the agent will do, then PA + "A()=x" is only consistent for one particular action x, and the specific action x for which it is consistent will be up for grabs, depending upon which proof the inconsistency checker finds.)

PA + "A() = x" may be inconsistent (proving, e.g., that 0=1 etc.).

I had an idea, and was wondering what its fatal flaw was. For UDT, what happens if, instead of proving theorems of the form "actionx --> utilityx" , it proves theorems of the form "PA+actionx |- utilityx"?

At a first glance, this seems to remove the problem of spurious counterfactuals implying any utility value, but there's probably something big I'm missing.

Completed.

This is actually isomorphic to the absent-minded driver problem. If you precommit to going straight, there is a 50/50 chance of being at either one of the two indistinguishable points on the road. If you precommit to turning left, there is a nearly 100% chance of being at the first point on the road (Since you wouldn't continue on to the second road point with that strategy.) It seems like probability can be determined only after a strategy has been locked into place.

Question for AI people in the crowd: To implement Bayes' Theorem, the prior of something must be known, and the conditional likelihood must be known. I can see how to estimate the prior of something, but for real-life cases, how could accurate estimates of P(A|X) be obtained?

Also, we talk about world-models a lot here, but what exactly IS a world-model?

Hi,

I'd be interested on LW's thoughts on this. I was quite involved in the piece, though I suggested to the journalist it would be more appropriate to focus on the high-profile names involved. We've been lucky at FHI/Cambridge with a series of very sophisticated tech-savvy journalists with whom the inferential distance has been very low (see e.g. Ross Andersen's Aeon/Atlantic pieces); this wasn't the case here, and although the journalist was conscientious and requested reading material beforehand, I found that communicating on these concepts more difficult than expected.

In my view the interview material turned out better than expected, given the clear inferential gap. I am less happy with the 'catastrophic scenarios'' which I was asked for. The text I sent (which I circulated to FHI/CSER members) was distinctly less sensational, and contained a lot more qualifiers. E.g. for geoengineering I had: "Scientific consensus is against adopting it without in depth study and broader societal involvement in the decisions made, but there may be very strong pressure to adopt once the impacts of climate change become more severe." and my pathogen modification example did not go nearly as far. While qualifiers can seem like unnecessary padding to editors, it can really change the tone of a piece. Similarly, in a pre-emptive line to ward off sensationalism, I included "I hope you can make it clear these are "worst case possibilities that currently appear worthy of study" rather than "high-likelihood events". Each of these may only have e.g. a 1% likelihood of occurring. But in the same way an aeroplane passenger shouldn't accept a 1% possibility of a crash, society should not accept a 1% possibility of catastrophe. I see our role as (like airline safety analysts) figuring out which risks are plausible, and for those, working to reduce the 1% to 0.00001%"; this was sort-of-addressed, but not really.

That said, the basic premises - that a virus could be modified for greater infectivity and released by a malicious actor, 'termination risk' for atmospheric aerosol geoengineering, future capabilities of additive manufacturing for more dangerous weapons - are intact.

Re: 'paperclip maximiser'. I mentioned this briefly in conversation, after we'd struggled for a while with inferential gaps on AI (and why we couldn't just outsmart something smarter than us, etc), presenting it as a 'toy example' used in research papers on AI goals, meant to encapsulate the idea that seemingly harmless or trivial but poorly thought through goals can result in unforseen and catastrophic consequences when paired with the kind of advanced resource utilisation and problem-solving ability a future AI might have. I didn't expect it it to be taken as a literal doomsday concern - and it wasn't in the text I sent - and to my mind it looks very silly in there, possibly deliberately so. However, I feel that Huw and Jaan's explanations were very good, and quite well-presented..

We've been considering whether we should limit ourselves to media opportunities where we can write the material ourselves, or have the opportunity to view and edit the final material before publishing. MIRI has significantly cut back on its media engagement, and this seems on the whole sensible (FHI's still doing a lot, some turns out very good, some not so good).

Lesson to take away: 1) this stuff can be really, really hard. 2) Getting used to v sophisticated, science/tech-savvy journalists and academics can leave you unprepared. 3) Things that are v reasonable with qualifies can become v unreasonable if you remove the qualifiers - and editors often just see the qualifiers as unnecessary verbosity (or want the piece to have stronger, more senational claims)

Right now, I'm leaning fairly strongly towards 'ignore and let quietly slip away' (the guardian has a small UK readership, so how much we 'push' this will probably make a difference), but I'd be interested in whether LW sees this as net positive or net negative on balance for existential risk in the public. However, I'm open to updating. I asked a couple of friends unfamiliar with the area what their take away impression was, and it was more positive than I'd anticipated.

This seems terribly inefficient and dependent you a great deal of personal sacrifice to achieve a goal. I guess I do t understand why someone would completely change their lifestyle just to help as many people as possible. If the goal is saving people in a large way it seems to me that aligning oneself with the people and organizations doing that work and regularly supplying them with the needed resources is far better than this system. I would say that the best way to live in this manner would be to position oneself as a donor to be called upon at a moments notice to fill a gap at a certain level. Working just to give money just isn't mindful of the commitment, the stress, and the need for self care. In order for you to be at your best you have to be well yourself. If you are living in a small apartment eating ramen while the beach is pristine that would not be optimal in my mind.

This concept seems a bit odd to me because why would one decide to make a lot of money only to give it away? I guess I don't understand that? I'm a bit old fashioned. I believe that a monthly commitment to giving is important. When I was growing my parents gave exactly 10% of my Dad's income to church every month. They also did extra things on an as needed basis. When the church needed a new roof and it cost $85,000 my Dad gave the church about 1/3 of that to make up for the difference the church didn't have. It was still a goodly chunk of change all at once. I think its more important about the consistent giving and the attitude towards giving rather than saying "I'm going to earn a lot so I can give it away." I think this is also an area where a spiritual path is important. I'm a writer and rather poor but I still give away old clothes, old items I don't need anymore and in some cases I give people money or things they needed to improve themselves. When a transient person is standing on the sidewalk with a sign I wait for Creator to tell me if they need something and when I hear the go-ahead I do it or don't. I see this come back to me all the time.

I guess in my personal system how much I do to or do not do for myself does not matter because its the regular giving thats important not merely the act of making a great deal so I can give it away or finding some way to make sure that I don't "backslide" into selfish spending. After all who said that not buying nice things for yourself or living a good life with a lifestyle isn't a good idea if one is mindful of community and how you can help? I would say rather than trying to give away as much money as possible (and thereby jeopardizing your future) the better option would be to give regularly and make your community better by any means necessary.