Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: pcm 01 March 2017 06:34:57PM 0 points [-]

I think your conclusion might be roughly correct, but I'm confused by the way your argument seems to switch between claiming that an intelligence explosion will eventually reach limits, and claiming that recalcitrance will be high when AGI is at human levels of intelligence. Bostrom presumably believes there's more low-hanging fruit than you do.

Comment author: pcm 12 January 2017 06:34:45PM 1 point [-]

I have a relevant blog post on models of willpower.

Comment author: pcm 12 January 2017 06:27:38PM 2 points [-]

I reviewed the book here.

Comment author: pcm 25 December 2016 07:51:30PM 1 point [-]

I subsidized some InTrade contracts in 2008. See here, here and here.

Comment author: sarahconstantin 06 December 2016 05:40:02PM 4 points [-]

I wonder if there's any way to measure rationality in animals.

Bear with me for a second. The Cognitive Reflection Test is a measure of how well you can avoid the intuitive-but-wrong answer and instead make the more mentally laborious calculation. The Stroop test is also a measure of how well you can avoid making impulsive mistakes and instead force yourself to focus only on what matters. As I recall, the "restrain your impulses and focus your thinking" skill is a fairly "biological" one -- it's consistently associated with activity in particular parts of the brain, influenced by drugs, and impaired in conditions like ADHD.

Could we -- or have we already -- design a variant of this made out of mazes that rats could run through?

I might look into this more carefully myself, but does anyone know off the top of their heads?

Comment author: pcm 07 December 2016 05:10:17PM 1 point [-]

See Rosati et al., The Evolutionary Origins of Human Patience: Temporal Preferences in Chimpanzees, Bonobos, and Human Adults, Current Biology (2007). Similar to the marshmallow test.

Comment author: roland 10 October 2016 12:20:15PM 3 points [-]

Is the following a rationality failure? When I make a stupid mistake that caused some harm I tend to ruminate over it and blame myself a lot. Is this healthy or not? The good thing is that I analyze what I did wrong and learn something from it. The bad part is that it makes me feel terrible. Is there any analysis of this behaviour out there? Studies?

Comment author: pcm 10 October 2016 04:44:24PM 3 points [-]

I suspect attempted telekinesis is relevant.

Comment author: Ozyrus 26 September 2016 11:25:21PM *  1 point [-]

I've been meditating lately on a possibility of an advanced artificial intelligence modifying its value function, even writing some excrepts about this topic.

Is it theoretically possible? Has anyone of note written anything about this -- or anyone at all? This question is so, so interesting for me.

My thoughts led me to believe that it is theoretically possible to modify it for sure, but I could not come to any conclusion about whether it would want to do it. I seriously lack a good definition of value function and understanding about how it is enforced on the agent. I really want to tackle this problem from human-centric point, but i don't really know if anthropomorphization will work here.

Comment author: pcm 27 September 2016 03:04:58PM 2 points [-]

See ontological crisis for an idea of why it might be hard to preserve a value function.

Comment author: DataPacRat 19 September 2016 06:35:24PM 10 points [-]

As a cryonicist, I'm drafting out a text describing my revival preferences and requests, to be stored along with my other paperwork. (Oddly enough, this isn't a standard practice.) The current draft is here. I'm currently seeking suggestions for improvement, and a lot of the people around here seem to have good heads on their shoulders, so I thought I'd ask for comments here. Any thoughts?

Comment author: pcm 22 September 2016 07:23:33PM 1 point [-]

My equivalent of this document focused more on the risks of unreasonable delays in uploading me. Cryonics organizations have been designed to focus on preservation, which seems likely to bias them toward indefinite delays. This might be especially undesirable in an "Age of Em" scenario.

Instead of your request for a "neutral third-party", I listed several specific people, who I know are comfortable with the idea of uploading, as people whose approval would be evidence that the technology is adequate to upload me. I'm unclear on how hard it would be to find a genuinely neutral third party.

My document is 20 years old now, and I don't have a copy handy. I suppose I should update it soon.

Comment author: WhySpace 23 August 2016 06:26:08PM *  2 points [-]

(1) Given: AI risk comes primarily from AI optimizing for things besides human values.

(2) Given: humans already are optimizing for things besides human values. (or, at least besides our Coherent Extrapolated Volition)

(3) Given: Our world is okay.^[CITATION NEEDED!]

(4) Therefore, imperfect value loading can still result in an okay outcome.

This is, of course, not necessarily always the case for any given imperfect value loading. However, our world serves as a single counterexample to the rule that all imperfect optimization will be disastrous.

(5) Given: A maxipok strategy is optimal. ("Maximize the probability of an okay outcome.")

(6) Given: Partial optimization for human values is easier than total optimization. (Where "partial optimization" is at least close enough to achieve an okay outcome.)

(7) ∴ MIRI should focus on imperfect value loading.

Note that I'm not convinced of several of the givens, so I'm not certain of the conclusion. However, the argument itself looks convincing to me. I’ve also chosen to leave assumptions like “imperfect value loading results in partial optimization” unstated as part of the definitions of those 2 terms. However, I’ll try and add details to any specific areas, if questioned.

Comment author: pcm 24 August 2016 03:29:20PM 0 points [-]

I expect that MIRI would mostly disagree with claim 6.

Can you suggest something specific that MIRI should change about their agenda?

When I try to imagine problems for which imperfect value loading suggests different plans from perfectionist value loading, I come up with things like "don't worry about whether we use the right set of beings when creating a CEV". But MIRI gives that kind of problem low enough priority that they're acting as if they agreed with imperfect value loading.

Comment author: Wei_Dai 22 July 2016 03:54:22PM 2 points [-]

Anyone else worried about Peter Thiel's support for Donald Trump discrediting Thiel in a lot of people's eyes, and MIRI and AI safety/risk research in general by association?

Comment author: pcm 22 July 2016 06:18:58PM 5 points [-]

No, mainly because Elon Musk's concern about AI risk added more prestige than Thiel had.

View more: Next