Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

In response to comment by [deleted] on On saving the world
Comment author: shminux 31 January 2014 12:06:36AM *  3 points [-]

Write a bullet-point summary for each sequence and tell me that one would not be tempted to "dismiss them out of hand, even lacking a chain of arguments leading up to them", unless one is already familiar with the arguments.

Comment author: JamesAndrix 06 February 2014 04:21:51AM 4 points [-]

Anecdote: I think I've had better responses summarizing LW articles in a few paragraphs without linking, than linking to them with short explanations.

It does take a lot to crosss those inferential distances, but I don't think quite that much.

To be fair, my discussions may not cover a whole sequence, I have the opportunity to pick out what is needed in a particular instance.

Comment author: Decius 08 January 2013 05:51:57AM 10 points [-]

I think that 'awesome' loses a lot of value when you are forced to make the statement "Watching lot of people die was the most awesome choice I had, because any intervention would have added victims without saving anyone."

I propose 'lame' and 'bummer' as antonyms for 'awesome'. Instead of trying to figure out the most awesome of a series of bad options, we can discuss the least lame.

In response to comment by Decius on Morality is Awesome
Comment author: JamesAndrix 15 January 2013 09:35:46PM 3 points [-]

Sucks less sucks less.

Comment author: JoshuaFox 11 January 2013 09:28:32AM *  0 points [-]

Right, SI's basic idea is correct.

However, given that WBE's will in any case be developed (and we can mention IA as well) , I'd like to see more consideration of how to keep brain-based AI's as safe as possible before they enter their Intelligence Explosion -- even though we understand that after an Explosion, there is little you can do.

Comment author: JamesAndrix 12 January 2013 08:39:02AM *  1 point [-]

One trouble is that that is essentailly tacking mind enslavement on to the WBE proposition. Nobody wants that. Uploads wouldn't volunteer. Even if a customer paid enough of a premium for an employee with loyalty modifications, that only rolls us back to relying on the good intent of the customer.

This comes down to the exact same arms race between friendly and 'just do it' . With extra ethical and reverse-engineering hurdles. (I think we're pretty much stuck with testing and filtering based on behavior. And some modification will only be testable after uploading is available)

Mind you I'm not saying don't do work on this, I'm saying not much work will be done on it.

Comment author: JamesAndrix 10 January 2013 11:37:50PM 1 point [-]

I think we're going to get WBE's before AGI.

If we viewed this as a form of heuristic AI, it follows from your argument that we should look for ways to ensure friendliness of WBE's. (Ignoring the ethical issues here.)

Now, maye this is becasue most real approaches would consider ethical issues, but it seems like figuring out how to modify a human brain so that it doesn't act against your interests even if is powerful and without hampering its intellect, is a big 'intractable' problem.

I suspect no one is working on it and no one is going to, even though we have working models of these intellects today. A new design might be easier to work with, but it will still be a lot harder than it wil seem to be worth - as long as the AI's are doing near human level work.

Aim for an AI design that's easy enough to work on saftey that people actually will work on safety... and it will start to look a lot like SIAI ideas.

Comment author: pedanterrific 18 December 2012 05:27:02PM 7 points [-]

Just because Harry's learned to keep his claws sheathed doesn't mean he's not still engaging in dominance contests.

Comment author: JamesAndrix 21 December 2012 07:09:40PM 3 points [-]

Moody set it as a condition for being able to speak as an equal.

Comment author: Desrtopa 18 December 2012 04:03:06AM 5 points [-]

Signs of leglimency can be detected, so if Quirrell is leglimizing Harry, he may be risking getting caught.

The fact that Harry has an occlumency teacher who uses leglimency on him in an instructional setting might serve to disguise other people using leglimency on him though. We don't know what kind of evidence checking for leglimency can provide about recency and whether the mind has been accessed by different people.

However, I doubt that Quirrell has been using leglimency on Harry, because there's a point where you're really just giving the antagonist too great an advantage, and if Quirrell has been accessing the contents of Harry's mind undetected, I would have an incredibly hard time buying the protagonists not being completely fucked (tvtropes link.)

Comment author: JamesAndrix 21 December 2012 07:00:52PM 1 point [-]

There is some time resolution.

Albus said heavily, "A person who looked like Madam McJorgenson told us that a single Legilimens had lightly touched Miss Granger's mind some months ago. That is from January, Harry, when I communicated with Miss Granger about the matter of a certain Dementor. That was expected; but what I did not expect was the rest of what Sophie found."

11 minute TED talk is about instrumental rationality

9 JamesAndrix 29 August 2012 07:33AM


Essentially this is about cutting your enemy, and some procedural pitfalls in his examples.

I didn't spot any novel principles in the talk, but I want to learn more about how he thinks.

Comment author: Lightwave 12 June 2012 09:56:04AM *  9 points [-]

how likely an AGI project is to doom us

Isn't it more like "how likely a formally proven FAI design is to doom us", since this is what Holden seems to be arguing (see his quote below)?

Suppose that it is successful in the "AGI" part of its goal, i.e., it has successfully created an intelligence vastly superior to human intelligence and extraordinarily powerful from our perspective. Suppose that it has also done its best on the "Friendly" part of the goal: it has developed a formal argument for why its AGI's utility function will be Friendly, it believes this argument to be airtight, and it has had this argument checked over by 100 of the world's most intelligent and relevantly experienced people. .. What will be the outcome?

Comment author: JamesAndrix 16 June 2012 10:47:44PM 1 point [-]

"When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong."


Comment author: John_Maxwell_IV 03 April 2012 12:06:29AM 0 points [-]

Is a human equipped with Google an optimization process powerful beyond the human's ability to steer?

Comment author: JamesAndrix 04 April 2012 03:06:27AM 0 points [-]

Tell me from China.

Comment author: provocateur 31 March 2012 09:37:18PM *  -2 points [-]

AGI will only be Friendly if its goals are the kinds of goals that we would want it to have

At the risk of losing my precious karma, I'll play the devil's advocate and say I disagree.

First some definitions: "Friendly" (AI), according to Wikipedia, is one that is beneficial to humanity (not a human buddy or pet). "General" in AGI means not problem-specific (narrow AI).

My counterexample is an AI system that lacks any motivations, goals or actuators. Think of an AIXI system (or, realistically, a system that approximates it), and subtract any reward mechanisms. It just models its world (looking for short programs that describe its input). You could use it to make (super-intelligent) predictions about the future. This seems clearly beneficial to humanity (until it falls into malicious human hands, but that's besides the argument you are making)

Comment author: JamesAndrix 01 April 2012 05:02:32PM 0 points [-]

That would make (human[s] + predictor) in to an optimization process that was powerful beyond the human[s]'s ability to steer. You might see a nice looking prediction, but you won't understand the value of the details, or the value of the means used to achieve it. (Which would be called trade-offs in a goal directed mind, but nothing weighs them here.)

It also won't be reliable to look for models in which you are predicted to not hit the Emergency Regret Button As that may just find models in which your regret evaluator is modified.

View more: Next