Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Skeptityke 28 March 2015 05:31:58PM 1 point [-]

"But the general result is that one can start with an AI with utility/probability estimate pair (u,P) and map it to an AI with pair (u',P) which behaves similarly to (u,P')"

Is this at all related to the Loudness metric mentioned in this paper? https://intelligence.org/files/LoudnessPriors.pdf It seems like the two are related... (in terms of probability and utility blending together into a generalized "importance" or "loudness" parameter)

Comment author: So8res 21 December 2014 01:46:47AM 2 points [-]

Then you get spurious proofs of inconsistency :-)

(If PA is strong enough to prove what the agent will do, then PA + "A()=x" is only consistent for one particular action x, and the specific action x for which it is consistent will be up for grabs, depending upon which proof the inconsistency checker finds.)

Comment author: Skeptityke 21 December 2014 02:02:40AM 1 point [-]

Got it. Thank you!

Comment author: So8res 21 December 2014 01:04:43AM 1 point [-]

PA + "A() = x" may be inconsistent (proving, e.g., that 0=1 etc.).

Comment author: Skeptityke 21 December 2014 01:27:35AM 1 point [-]

What happens if it only considers the action if it both failed to find "PA+A()=x" inconsistent and found a proof that PA+A()=x proves U()=x? Do an inconsistency check first and only consider/compare the action if the inconsistency check fails.

Comment author: Skeptityke 21 December 2014 12:19:20AM 1 point [-]

I had an idea, and was wondering what its fatal flaw was. For UDT, what happens if, instead of proving theorems of the form "actionx --> utilityx" , it proves theorems of the form "PA+actionx |- utilityx"?

At a first glance, this seems to remove the problem of spurious counterfactuals implying any utility value, but there's probably something big I'm missing.

Comment author: Skeptityke 24 October 2014 03:04:28AM 45 points [-]


In response to Baysian conundrum
Comment author: Skeptityke 13 October 2014 03:14:13PM *  1 point [-]

This is actually isomorphic to the absent-minded driver problem. If you precommit to going straight, there is a 50/50 chance of being at either one of the two indistinguishable points on the road. If you precommit to turning left, there is a nearly 100% chance of being at the first point on the road (Since you wouldn't continue on to the second road point with that strategy.) It seems like probability can be determined only after a strategy has been locked into place.

Comment author: Skeptityke 04 October 2014 06:38:26PM 1 point [-]

Question for AI people in the crowd: To implement Bayes' Theorem, the prior of something must be known, and the conditional likelihood must be known. I can see how to estimate the prior of something, but for real-life cases, how could accurate estimates of P(A|X) be obtained?

Also, we talk about world-models a lot here, but what exactly IS a world-model?

Comment author: Sean_o_h 30 August 2014 03:16:57PM *  15 points [-]


I'd be interested on LW's thoughts on this. I was quite involved in the piece, though I suggested to the journalist it would be more appropriate to focus on the high-profile names involved. We've been lucky at FHI/Cambridge with a series of very sophisticated tech-savvy journalists with whom the inferential distance has been very low (see e.g. Ross Andersen's Aeon/Atlantic pieces); this wasn't the case here, and although the journalist was conscientious and requested reading material beforehand, I found that communicating on these concepts more difficult than expected.

In my view the interview material turned out better than expected, given the clear inferential gap. I am less happy with the 'catastrophic scenarios'' which I was asked for. The text I sent (which I circulated to FHI/CSER members) was distinctly less sensational, and contained a lot more qualifiers. E.g. for geoengineering I had: "Scientific consensus is against adopting it without in depth study and broader societal involvement in the decisions made, but there may be very strong pressure to adopt once the impacts of climate change become more severe." and my pathogen modification example did not go nearly as far. While qualifiers can seem like unnecessary padding to editors, it can really change the tone of a piece. Similarly, in a pre-emptive line to ward off sensationalism, I included "I hope you can make it clear these are "worst case possibilities that currently appear worthy of study" rather than "high-likelihood events". Each of these may only have e.g. a 1% likelihood of occurring. But in the same way an aeroplane passenger shouldn't accept a 1% possibility of a crash, society should not accept a 1% possibility of catastrophe. I see our role as (like airline safety analysts) figuring out which risks are plausible, and for those, working to reduce the 1% to 0.00001%"; this was sort-of-addressed, but not really.

That said, the basic premises - that a virus could be modified for greater infectivity and released by a malicious actor, 'termination risk' for atmospheric aerosol geoengineering, future capabilities of additive manufacturing for more dangerous weapons - are intact.

Re: 'paperclip maximiser'. I mentioned this briefly in conversation, after we'd struggled for a while with inferential gaps on AI (and why we couldn't just outsmart something smarter than us, etc), presenting it as a 'toy example' used in research papers on AI goals, meant to encapsulate the idea that seemingly harmless or trivial but poorly thought through goals can result in unforseen and catastrophic consequences when paired with the kind of advanced resource utilisation and problem-solving ability a future AI might have. I didn't expect it it to be taken as a literal doomsday concern - and it wasn't in the text I sent - and to my mind it looks very silly in there, possibly deliberately so. However, I feel that Huw and Jaan's explanations were very good, and quite well-presented..

We've been considering whether we should limit ourselves to media opportunities where we can write the material ourselves, or have the opportunity to view and edit the final material before publishing. MIRI has significantly cut back on its media engagement, and this seems on the whole sensible (FHI's still doing a lot, some turns out very good, some not so good).

Lesson to take away: 1) this stuff can be really, really hard. 2) Getting used to v sophisticated, science/tech-savvy journalists and academics can leave you unprepared. 3) Things that are v reasonable with qualifies can become v unreasonable if you remove the qualifiers - and editors often just see the qualifiers as unnecessary verbosity (or want the piece to have stronger, more senational claims)

Right now, I'm leaning fairly strongly towards 'ignore and let quietly slip away' (the guardian has a small UK readership, so how much we 'push' this will probably make a difference), but I'd be interested in whether LW sees this as net positive or net negative on balance for existential risk in the public. However, I'm open to updating. I asked a couple of friends unfamiliar with the area what their take away impression was, and it was more positive than I'd anticipated.

Comment author: Skeptityke 30 August 2014 03:44:30PM 15 points [-]

I'd call it a net positive. Along the axis of "Accept all interviews, wind up in some spectacularly abysmal pieces of journalism" and "Only allow journalism that you've viewed and edited", the quantity vs quality tradeoff, I suspect the best place to be would be the one where the writers who know what they're going to say in advance are filtered, and where the ones who make an actual effort to understand and summarize your position (even if somewhat incompetent) are engaged.

I don't think the saying "any publicity is good publicity" is true, but "shoddy publicity pointing in the right direction" might be.

I wonder how feasible it is to figure out journalist quality by reading past articles... Maybe ask people who have been interviewed by the person in the past how it went?

Comment author: cameroncowan 28 August 2014 04:43:31AM 1 point [-]

This seems terribly inefficient and dependent you a great deal of personal sacrifice to achieve a goal. I guess I do t understand why someone would completely change their lifestyle just to help as many people as possible. If the goal is saving people in a large way it seems to me that aligning oneself with the people and organizations doing that work and regularly supplying them with the needed resources is far better than this system. I would say that the best way to live in this manner would be to position oneself as a donor to be called upon at a moments notice to fill a gap at a certain level. Working just to give money just isn't mindful of the commitment, the stress, and the need for self care. In order for you to be at your best you have to be well yourself. If you are living in a small apartment eating ramen while the beach is pristine that would not be optimal in my mind.

Comment author: Skeptityke 28 August 2014 05:34:20AM 1 point [-]

I think there's an important distinction to be made between the different levels of earning to give. Really, there's a spectrum between "donate 5 percent of income" at one end, and "devote existence to resolving issue" at the other end. For humans trying to do the best they can, in fact, trying to scale up too fast can lead to severe burnout. So caring for yourself and having a good life and low stress is a good idea because it guards against burnout. It is better to donate a thousand dollars a month to resolve an issue than three thousand with an 80% chance of burnout. Slowly build up to higher points on the spectrum that don't give up quality of life.

Remember, the goal is to do that which works, not to win a "I'm way more hardcore about charity than you!" contest. If that which works leads to sacrifice and you can handle it without burnout risk, then sacrifice. If self-sacrifice doesn't work for solving the issue, then don't do it. And yes, aligning oneself with the people working on it and supplying them with resources is pretty much exactly what is required in many cases. Earning to give comes from the fact that the "supplying them with resources" step works much better with more resources, and working at high paying jobs is a good way to get resources.

And finally, about not understanding why someone would completely change their lifestyle to help as many as people as possible... Lifestyle changes tend to look really intimidating from the outside, not from the inside. In college, as an example, going "I'm taking >20 credits" makes people mightily impressed and worried about your inevitable lack of a social life, but once you actually start doing it, it doesn't feel extraordinary or hard from the inside. Dropping annual expenses from 60k to 15k is another thing that sounds intimidating, but from the inside, it isn't that difficult, and quality of life doesn't significantly change.

So that's one part of it, that it doesn't take as much of a sacrifice as you think. The second part of it is that if there is anything at all that you value more than the thing you would spend the money on instead, moving the money to the more highly valued thing is inevitable if you don't compartmentalize. I value ten lives more highly than purchasing a shiny new car, and I suspect that most people would agree with this. It's just a matter of acting on preexisting values and desires.

In response to Persistent Idealism
Comment author: cameroncowan 27 August 2014 09:18:58PM 0 points [-]

This concept seems a bit odd to me because why would one decide to make a lot of money only to give it away? I guess I don't understand that? I'm a bit old fashioned. I believe that a monthly commitment to giving is important. When I was growing my parents gave exactly 10% of my Dad's income to church every month. They also did extra things on an as needed basis. When the church needed a new roof and it cost $85,000 my Dad gave the church about 1/3 of that to make up for the difference the church didn't have. It was still a goodly chunk of change all at once. I think its more important about the consistent giving and the attitude towards giving rather than saying "I'm going to earn a lot so I can give it away." I think this is also an area where a spiritual path is important. I'm a writer and rather poor but I still give away old clothes, old items I don't need anymore and in some cases I give people money or things they needed to improve themselves. When a transient person is standing on the sidewalk with a sign I wait for Creator to tell me if they need something and when I hear the go-ahead I do it or don't. I see this come back to me all the time.

I guess in my personal system how much I do to or do not do for myself does not matter because its the regular giving thats important not merely the act of making a great deal so I can give it away or finding some way to make sure that I don't "backslide" into selfish spending. After all who said that not buying nice things for yourself or living a good life with a lifestyle isn't a good idea if one is mindful of community and how you can help? I would say rather than trying to give away as much money as possible (and thereby jeopardizing your future) the better option would be to give regularly and make your community better by any means necessary.

Comment author: Skeptityke 28 August 2014 01:44:46AM 3 points [-]

The reason to make lots of money to give it away is elaborated on here, in the paragraph about the lawyer who wants to clean up the beach.

Summary version: More charities are funding-limited than volunteer-limited, and if you are making a sufficient amount of money, working one extra hour and donating the proceeds from that hour gets more done, saves more people, than using that hour to volunteer. The important part is to actually save people.

Saving people is far more important than giving consistently (If the best way to save people is to give each month, I want to give each month, if the best way to save people is to donate large chunks infrequently, I want to donate large chunks infrequently), saving people is far more important than having a good attitude towards giving (If having a good attitude towards giving makes me donate more, I want to have a good attitude towards giving, if having a selfish attitude towards giving makes me donate more, I want to have a selfish attitude), and saving people is far more important than spiritually developing in the process (I trust you can complete the pattern). I'm not saying these things are bad, it's just that they are subgoals of the thing you are trying to accomplish, which is doing the most good. Making a great deal to give it away, and making sure you don't backslide into selfishness are things to do to ensure that the most people can be saved. Regular giving is secondary in importance.

The goal is not fitting conventional patterns of giving, the goal is to help as many people as possible. To try to get a high score in the LIVES IMPROVED statistics column of the game of life. If something helps in this quest, do it, if it doesn't help, stop doing it.

View more: Next