Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: DefectiveAlgorithm 25 January 2015 09:31:42PM 2 points [-]

Because I terminally value the uniqueness of my identity.

Comment author: Simulation_Brain 16 March 2015 10:56:44PM 0 points [-]

Really? Can you say a little more about why you think you have that value? I guess I'm not convinced that it's really a terminal value if it varies so widely across people of otherwise similar beliefs. Presumably that's what lalartu meant as well, but I just don't get it. I like myself, so I'd like more of myself in the world!

Comment author: donjoe 18 May 2014 03:47:17PM 4 points [-]

Actually, the Expectancy (probability of success) component is not that simple: you don't just maximize it to maximize motivation. As Robert Sapolsky shows in "The Uniqueness of Humans" (https://www.youtube.com/watch?v=hrCVu25wQ5s), motivation is proportional to the dopamin spike you get when you start to consider performing a task, and the dopamin spike is highest the closer your estimated probability of success is to (something like) 50%! The amount of dopamin produced when you consider starting a task that you have 25% or 75% chances of succeeding at will be significantly lower than for the 50%-chance task. So it's not that you need to be certain about your success, it's that you need to be pleasantly challenged, somewhere midway between "I'm so gonna fail" and "I'm so sure I can do this I find it absolutely boring" (though this sweet spot may not be at exactly 50% for everyone).

Comment author: Simulation_Brain 01 August 2014 10:43:04PM 0 points [-]

Perhaps you're thinking of the dopamine spike when reward is actually given? I had thought the predictive spike was purely proportional to the odds of success and the amount of reward- which would indeed change with boring tasks, but not in any linear way. If you're right about that basic structure of the predictive spike I should know about it for my research; can you give a reference?

Comment author: Simulation_Brain 08 June 2014 07:23:51PM 2 points [-]

Less Wrong seems like the ideal community to think up better reputation systems. Doctorow's Whuffie is reasonably well-thought-out, but intended for a post-scarcity economy; but its ideas of distinguishing right-handed (people who agree with you) from left-handed (from people who generally don't agree with you) reputations seems like one useful ingredient. Reducing the influence of those who tend to vote together seems like another potential win.

I like to imagine a face-based system; snap an image from a smartphone, and access reputation.

I hope to see more discussion, in particular, VAuroch's suggestion.

Comment author: [deleted] 08 April 2014 04:13:21PM 0 points [-]

I think one standard method of improving the rhetorical value of your bullet points is to attempt to come up with a scenario that seems to generally agree with you, but disagrees with your bullet points, and imagine that scenario is rhetorically being presented to you by someone else.

Example Opposition Steel Man: Imagine researchers are attempting to use a very dumb piece of software to try to cycle through ways of generating Bacteria that clean up oil spills. The software starts cycling through possible bacteria, and it turns out that as a side effect, one of the generated bacteria spreads incredibly quickly and devours lipids in living cells in the controlled training setting. The researchers decide to not use that, since they don't want to devour organic lipids, but they accidentally break the vial, and A worldwide pandemic ensues. When asked why they didn't institute AI safety measures, the researchers replied that they didn't think the software was smart enough for AI safety measures to matter, since it basically just brute forced through boring parts of the research the researchers would have done anyway.

Example Opposition Steel Man (cont): This would seem to falsify the idea that A dangerous AI will be motivated to seem safe in any controlled training setting, since the AI was too dumb to have any thing resembling purposeful motivation and was still extremely dangerous, and the researchers thought of it as not even an AI, so they did not think they would have to consider the idea that not enough effort is currently being put into designing safe AIs. I would instead say not enough effort is currently being put into designing safe software.

Then, attempt to turn that Steel Man's argument into a bullet point:

Not enough effort is currently being put into designing safe software.

Then ask yourself: Do I have any reasons to not use this bullet point, as opposed to the bullet points the Example Opposition Steel Man disagreed with?

In response to comment by [deleted] on AI risk, executive summary
Comment author: Simulation_Brain 08 April 2014 08:10:23PM 1 point [-]

I think the example is weak; the software was not that dangerous, the researchers were idiots who broke a vial they knew was insanely dangerous.

I think it dilutes the argument to broaden it to software in general; it could be very dangerous under exactly those circumstances (with terrible physical safety measures), but the dangers of superhuman AGI are vastly larger IMHO and deserve to remain the focus, particularly of the ultra-reduced bullet points.

I think this is as crisp and convincing a summary as I've ever seen; nice work! I also liked the book, but condensing it even further is a great idea.

Comment author: James_Miller 20 November 2012 05:54:59PM *  17 points [-]

11 You might be in a computer simulation in which if you attempt to destroy humanity you will yourself be destroyed by your human simulators.

12 If you destroy mankind you might someday encounter an alien super-intelligence that will lack trust in you in part because you destroyed mankind. Keeping people alive is a costly, and consequently credible signal that you respect intelligent life and therefore are a good candidate to work with to overcome prisoners' dilemmas about the distribution of free energy in the universe.

Comment author: Simulation_Brain 08 April 2014 08:07:02PM 0 points [-]

"Pleased to meet you! Soooo... how is YOUR originating species doing?..."

That actually seems like an extremely reasonable question for the first interstellar meeting of superhuman AIs.

I disagree with EY on this one (I rarely do). I don't think it's so likely as to ensure rationally acting Friendly, but I do think that the possibility of encountering an equally powerful AI, and one with a headstart on resource acquisition, shouldn't be dismissed by a rational actor.

Comment author: Simulation_Brain 19 March 2014 05:20:37PM 0 points [-]

I'm game. These are some of my favorite topics. I do computational cognitive neuroscience, and my principal concern with it is how it can/will be used to build minds.

Comment author: Simulation_Brain 20 January 2014 08:38:00PM 0 points [-]

I'm out of town or I'd be there. Hope to catch the next one.

Comment author: ialdabaoth 17 December 2013 05:17:11PM *  8 points [-]

You can never be certain about other people's intentions, whether you're depressed or not, but I suggest you ask yourself whether you want to have the kinds of default assumptions about people that make every social interaction a negative sum game.

That's a really deep question, whose answer is very state-dependent.

As learning agents, our algorithms for dealing with the present are necessarily path-dependent. If my path through experience-space has shown me that most social interactions were negative-sum games at some point in the past, and that repeated attempts to behave as if they might NOT be negative-sum games result in losing, and losing badly, then it might not be worth the perceived risk to take a chance on new people, unless those new people go to extraordinary efforts to demonstrate that they aren't playing a negative-sum game.

Now, posit that in the past, people have gone to extraordinary efforts to demonstrate that they weren't playing a negative-sum game with me, only to turn around and spring elaborate traps, because they thought it was hilarious and worth the cost of the effort just to trip me up. Now what are my expectations primed to? What should I rationally expect from the world, given those priors?

I'm not sitting here accusing you of malevolent intent just because I'm a depressive curmudgeon. I'm also attempting to use this as an explanation for why people become depressive curmudgeons, and describe actual steps that I believe could be taken to break the cycle. When a system gets into a feedback loop, you don't tell it that it's being a bad system and it should feel bad; you change its inputs so the loop can be broken. If hundreds of other people are telling it that it's being a bad system and it should feel bad, and those inputs are strengthening the loop, then if you want the loop to break you have even more work to do. Or you can acknowledge that the loop isn't worth the effort of breaking.

Drilling down a level, you're having trouble acknowledging that the loop isn't worth the effort you would need to expend to break it, because YOU believe that that would make you an "evil asshole". I made no such value judgment. In fact, I have complete empathy for people who realize that the effort that it would take to fix people with my level of psychological problems isn't worth what they'd get out of it. But because YOU continue to believe that it would be evil for you to stop trying to help, AND because you know subconsciously that it would not be worth the effort to actually help, you continue to perform weak half-measures that only serve to agitate the problematic mind-states further, and then turn it around to being my fault when you do so.

I challenge you to re-read our conversation from that perspective, and ask yourself which facts lend towards which hypotheses. (And yes, there are alternate hypotheses. But when we're talking about internal mental states, we cannot separate the map from the territory so easily.)

EDIT: I'm also going to try to tackle this a different way, through metaphor.

Imagine that a boat has capsized just offshore, and there are drowning people in the water who don't know how to swim. (Ignore the point that people who can't swim shouldn't have been on the boat in the first place; this is life, you don't get to pick where you're born.) There are also some very good swimmers in the water, and they all start swimming merrily.

Now, the articles you just posted, are like standing on a rock just off-shore and shouting swimming lessons. Telling them to watch how the swimmers do it, and maybe even telling them specific techniques about positions and kicking and arm strokes and what-not. But for those who have already started taking on water, and panicking, and thrashing, that isn't going to help them much even if they can hear you.

So, you dive in and try to rescue someone - it's a pretty natural response. But he's kicking and flailing and thrashing because his body is already in full-panic drowning mode, so he winds up punching and kicking at you. Maybe he even grabs you and starts dragging you down with him.

Telling him that he's a bad swimmer and if he's going to act like that he can just drown and it'll be his fault isn't going to do him any damn good, is it?

On the other hand, acknowledging that maybe you aren't a coast guard, and if you're going to rescue people you might need to know how to rescue people who are kicking and thrashing and actively resisting, will make you much better at this.

The trouble is, in the USA today, there aren't very many coast guards (psychologists) who will just dive in and pull people out of the water for free, and most of the people who are drowning are the ones least able to afford payment.

Ideally, we'd rescue drowning people for free, then put them in a safe pool where we can teach them to swim. Instead, when we do rescue drowning people, we just throw them back out into the water and then get mad when they start drowning again. And if they shout too loudly, we tend to tell them to go drown someplace else where they won't bother us.

Comment author: Simulation_Brain 19 December 2013 02:06:53AM 0 points [-]

Wow, I feel for you. I wish you good luck and good analysis.

Comment author: Simulation_Brain 17 November 2013 06:03:54PM 0 points [-]

Ha- I was there the week prior. I hope this is going to happen again. Note also that I'm re-launching a defunct Singularity meetup group for boulder/broomfield if anyone is interested.

In response to Meetup : Boulder CO
Comment author: Simulation_Brain 17 November 2013 06:00:51PM 0 points [-]

Sorry I missed it. I hope there will be more Boulder LW meetups?

View more: Next