Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Jayson_Virissimo 23 April 2014 04:39:45PM *  7 points [-]

Tyler Cowen talks with Nick Beckstead about x-risk here. Basically he thinks that "people doing philosophical work to try to reduce existential risk are largely wasting their time" and that "a serious effort looks more like the parts of the US government that trained people to infiltrate the post-collapse Soviet Union and then locate and neutralize nuclear weapons."

My Straussian reading of Tyler Cowen is that a "serious" MIRI would be assembling and training a team of hacker-assassins to go after potential UFAIs instead of dinking around with decision theory.

Comment author: RichardKennaway 24 April 2014 08:42:54AM 0 points [-]

My Straussian reading of Tyler Cowen is that a "serious" MIRI would be assembling and training a team of hacker-assassins to go after potential UFAIs instead of dinking around with decision theory.

A "serious" MIRI would operate in absolute secrecy, and the "public" MIRI would never even hint at the existence of such an organisation, which would be thoroughly firewalled from it. Done right, MIRI should look exactly the same whether or not the secret one exists.

Comment author: Eugine_Nier 21 April 2014 03:50:12AM 2 points [-]

He is arguing here against theories whereby sovereignty must consist of absolute power held by a single individual beyond any legitimate challenge, his subjects having no rights against him. For Rousseau, sovereignty is the coherent extrapolated volition of humanity -- or in Rousseau's words, "the exercise of the general will". Rousseau's sovereignty is still absolute and indivisible, but is not located in any individual.

Here is a decent debunking of the notion that modern society is based on a social contract. The basic argument is that if one attempts to explicitly right down the kind of contract these theories require, one winds up with a contract that no court would enforce between private parties.

More generally, Nick Szabo argues that the concept of sovereignty is itself totalitarian.

Comment author: RichardKennaway 22 April 2014 09:45:07PM 0 points [-]

Here is a decent debunking of the notion that modern society is based on a social contract.

I agree with that.

More generally, Nick Szabo argues that the concept of sovereignty is itself totalitarian.

It certainly is. Where does that leave FAI? A superintelligent FAI, as envisaged by those who think it a desirable goal, will be a totalitarian absolute ruler imposing the CEVoH and will, to borrow Rousseau's words, be "so strong as to render it impossible to suspend [its] operation." Rather like the Super Happies' plan for humanity. The only alternative to a superintelligent FAI is supposed to be a superintelligent UFAI.

Comment author: ChristianKl 22 April 2014 08:42:13PM 2 points [-]

In this case they come from me. Feel free to post your own numbers.

The point of choosing Zeus as an example is that it's a claim that probably not going to mindkill anyone. That makes it easier to talk about the principles than using an example where the updating actually matters.

Comment author: RichardKennaway 22 April 2014 09:04:06PM -1 points [-]

In this case they come from me.

In other words, you made them up. Fictional evidence.

Feel free to post your own numbers.

you did say (my emphasis)

Your belief should change by orders of magnitude.

Why should a myth about Zeus change anyone's belief by "orders of magnitude"?

Comment author: ChristianKl 22 April 2014 11:21:45AM 0 points [-]

It looks to me to be rather clear that what is being said ("myths are not evidence for Zeus") translates roughly to "myths are very weak evidence for Zeus, and so my beliefs are changed very little by them". Is there still a real misunderstanding here?

You are making a mistake in reasoning if you don't change your belief through that evidence. Your belief should change by orders of magnitude. A change from 10^{-18} to 10^{-15} is a strong change.

The central reason to believe that Zeus doesn't exist are weak priors.

Skeptics have ideas that someone has to prove something to them for them to believe it. In the Bayesian worldview you always have probabilities for your beliefs. Social obligations aren't part of it. "Good" evidence means that someone fulfilled a social obligation of providing a certain amount of proof. It doesn't refer to how strongly a Bayesian should update after being exposed to a piece of evidence.

There are very strong instincts for humans to either believe X is true or to believe X is false. It takes effort to think in terms of probabilities.

Comment author: RichardKennaway 22 April 2014 05:03:09PM 1 point [-]

You are making a mistake in reasoning if you don't change your belief through that evidence. Your belief should change by orders of magnitude. A change from 10^{-18} to 10^{-15} is a strong change.

Where do those numbers come from?

Comment author: polymathwannabe 19 April 2014 06:34:14AM 1 point [-]

That page leaves it clear:

"All fanfiction involves borrowing the original author’s characters, situations, and world. It is ridiculous to turn around and complain if your own ideas get borrowed in turn. Anyone is welcome to steal anything from any fanfiction I write."

Comment author: RichardKennaway 19 April 2014 08:54:19AM 2 points [-]

I think that only speaks to writing fanfiction of Eliezer's fanfiction, not rights over the text itself. By default, the copyright is solely Eliezer's unless and until he says otherwise.

Comment author: Eugine_Nier 19 April 2014 04:20:39AM 2 points [-]

MAN is born free; and everywhere he is in chains. One thinks himself the master of others, and still remains a greater slave than they. How did this change come about? I do not know. What can make it legitimate? That question I think I can answer.

Could you taboo what Rousseau means by "master" and "slave" in that quote. As is, to me it sounds like deep wisdom attempting to use said words in some metaphorical way that's not at all well-defined. Also I don't see what this has to do with the subject.

The idea is that in the end state people would be motivated to work as a way of self actualization and don't need financial incentives to do work.

The problem is that the work that's self-actualizing is not necessarily the same as the work that's needed to keep society running. In other words, attempting to run society like this you'd wind up with a bunch of (mediocre) artists starving and suffering from dysentery because not enough people derive self-actualization from farming or maintaining the sewer system. Historically, many attempts by intellectuals to create planned communities fell into this problem.

Star Trek has characters who work without getting payed to do so.

Fictional evidence.

Comment author: RichardKennaway 19 April 2014 08:47:06AM 0 points [-]

MAN is born free; and everywhere he is in chains. One thinks himself the master of others, and still remains a greater slave than they. How did this change come about? I do not know. What can make it legitimate? That question I think I can answer.

Could you taboo what Rousseau means by "master" and "slave" in that quote.

The full text is readily available online. A "master" is someone with the power to tell others what to do and be obeyed; yet these masters themselves obey something above themselves (laws written and unwritten). Rousseau's answer (SPOILER WARNING!!) is the title of his work. (To which the standard counter-argument is "show me my signature on this supposed contract".)

A few more Rousseau quotes:

The social order is a sacred right which is the basis of all other rights.

...

All power comes from God, I admit; but so does all sickness: does that mean that we are forbidden to call in the doctor?

...

To renounce liberty is to renounce being a man, to surrender the rights of humanity and even its duties. For him who renounces everything no indemnity is possible. Such a renunciation is incompatible with man's nature; to remove all liberty from his will is to remove all morality from his acts.

He is arguing here against theories whereby sovereignty must consist of absolute power held by a single individual beyond any legitimate challenge, his subjects having no rights against him. For Rousseau, sovereignty is the coherent extrapolated volition of humanity -- or in Rousseau's words, "the exercise of the general will". Rousseau's sovereignty is still absolute and indivisible, but is not located in any individual.

One can cherry-pick Rousseau to multiple ends. Here's something for HBDers:

Liberty, not being a fruit of all climates, is not within the reach of all peoples.

Libertarians may find something to agree with in this:

It is wrong therefore to wish to make political institutions so strong as to render it impossible to suspend their operation.

But to know what Rousseau thought, it is better to read his work.

Comment author: somnicule 19 April 2014 04:57:08AM 1 point [-]

Hmm. It's an interesting point.

I'm not entirely clear on the purpose of the rule. It makes sense to not just increase the redundancy of anything people have said in other threads that have already got a lot of attention, but I'm sure there's plenty of interesting stuff buried deep in comment threads that haven't got much light and might be worth sharing. Conversely, there will be some quotes here from outside LW/OB that a high proportion of readers have seen already.

So it's definitely something that made sense when the LW/OB community was smaller and there wasn't much good stuff that people weren't seeing anyway, but perhaps it's time to relax the rule a little bit, replace it with the substance.

Comment author: RichardKennaway 19 April 2014 08:10:58AM 3 points [-]

I believe the purpose was to bring material to LW from outside rather than quoting each other (and especially, quoting Eliezer), to avoid an echo chamber effect. There was once an experimental LW Quotes Thread, but the experiment has not been repeated.

I don't have a strong view about whether LW regulars posting on other LW regulars' blogs should be excluded from the quotes threads, but I incline against the practice. It was a good quote though.

Comment author: eli_sennesh 17 April 2014 11:26:02PM 0 points [-]

However, every step happens voluntarily because what comes after is seen as better than what is before, and I don't see why I should consider the final outcome bad.

So you're using a "volunteerism ethics" in which whatever agents choose voluntarily, for some definition of voluntary, is acceptable, even when the agents may have their values changed in the process and the end result is not considered desirable by the original agents? You only care about the particular voluntariness of the particular choices?

Huh. I suppose it works, but I wouldn't take over the universe with it.

Comment author: RichardKennaway 18 April 2014 07:08:40AM 1 point [-]

So you're using a "volunteerism ethics" in which whatever agents choose voluntarily, for some definition of voluntary, is acceptable, even when the agents may have their values changed in the process and the end result is not considered desirable by the original agents? You only care about the particular voluntariness of the particular choices?

When it happens fast, we call it wireheading. When it happens slowly, we call it the march of progress.

Comment author: jaime2000 17 April 2014 08:48:21PM 0 points [-]

A former poster here (known elsewhere on the net as "James A. Donald")

Where can I find evidence linking the sam0345 account to the identity James A. "Jim" Donald?

Comment author: RichardKennaway 17 April 2014 09:26:12PM 5 points [-]

Where can I find evidence linking the sam0345 account to the identity James A. "Jim" Donald?

Somewhat laboriously, by searching LessWrong for his very first postings and working forwards from there, looking for my replies to him and he to me. I recognised him as James A. Donald as soon as he started posting here, from his distinctive writing style and views, which were very familiar to me from his long history of participating in rec.arts.sf.* on USENET. As evidence, I linked to other places on the net where he had posted views identical to what he had just posted here, expressed in very similar terms. He never took notice of my identification, even when replying directly to comments of mine identifying him, but I think it definite.

BTW, while "sam0345" is obviously not a real-world name, I have never seen reason to think that "James A. Donald" is. Searches on that name turn up nothing but his online activity (and a mugshot of an unprepossessing individual of the same name who served 35 years for forgery, and who I have no reason to think has any connection with him). I have almost never, here or anywhere else, seen him post anything personal about himself. He is American, and an Internet engineer, and that's about it. And 10 inches taller than his wife, for what that's worth. I have never seen anyone mention having met him. His ownership of jim.com is unusual, in that it goes back well before the advent of public Internet access and easy private ownership of domain names. Try getting a domain name that short and simple nowadays! They're all taken.

Comment author: pcm 17 April 2014 04:50:23PM 10 points [-]

Bankruptcy is normally means having debts that can't be paid, and Alcor goes out of its way to avoid having anything that could be a debt, and is careful to maintain funds that can be used to continue to keep its patients preserved. This kind of conservatism comes at some cost in its ability to grow, so it doesn't require unusually good management to have a higher than normal chance of continuing to exist.

There seem to have been two cryonics organizations that failed (CSC and CSNY). Some patients at CSNY were unharmed by that failure, so having your organization fail doesn't automatically imply death. Plus people have learned from those failures.

Comment author: RichardKennaway 17 April 2014 05:07:17PM 6 points [-]

Some patients at CSNY were unharmed by that failure, so having your organization fail doesn't automatically imply death.

Your chances of surviving your cryo company going bust may depend on how seriously society in general takes the idea that you aren't dead yet.

View more: Next