Comment author: mwaser 17 December 2012 02:56:14PM 1 point [-]
Comment author: shminux 10 May 2012 06:30:00PM *  58 points [-]

Wow, I'm blown away by Holden Karnofsky, based on this post alone. His writing is eloquent, non-confrontational and rational. It shows that he spent a lot of time constructing mental models of his audience and anticipated its reaction. Additionally, his intelligence/ego ratio appears to be through the roof. He must have learned a lot since the infamous astroturfing incident. This is the (type of) person SI desperately needs to hire.

Emotions out of the way, it looks like the tool/agent distinction is the main theoretical issue. Fortunately, it is much easier than the general FAI one. Specifically, to test the SI assertion that, paraphrasing Arthur C. Clarke,

Any sufficiently advanced tool is indistinguishable from an agent.

one ought to formulate and prove this as a theorem, and present it for review and improvement to the domain experts (the domain being math and theoretical computer science). If such a proof is constructed, it can then be further examined and potentially tightened, giving new insights to the mission of averting the existential risk from intelligence explosion.

If such a proof cannot be found, this will lend further weight to the HK's assertion that SI appears to be poorly qualified to address its core mission.

Comment author: mwaser 10 May 2012 10:07:04PM 1 point [-]

If it is true (i.e. if a proof can be found) that "Any sufficiently advanced tool is indistinguishable from agent", then any RPOP will automatically become indistinguishable from an agent once it has self-improved past our comprehension point.

This would seem to argue against Yudkowsky's contention that the term RPOP is more accurate than "Artificial Intelligence" or "superintelligence".

Comment author: mwaser 14 March 2011 10:57:24AM 0 points [-]

Actually, eating a baby bunny is a really bad idea when viewed from a long-term perspective. Sure, it's a tender tasty little morsel -- but the operative word is little. Far better from a long-term view to let it grow up, reproduce and then eat it. And large competent bunnies aren't nearly as cute as baby bunnies, are they? So maybe evo-psych does have it correct . . . . and maybe the short-sighted rationality of tearing apart a whole field by implication because you don't understand how something works doesn't seem as brilliant.

Comment author: Eliezer_Yudkowsky 13 February 2011 05:14:34PM 6 points [-]

You say you'll present some objections to CEV. Can you describe a concrete failure scenario of CEV, and state a computational procedure that does better?

Comment author: mwaser 16 February 2011 01:03:28PM 1 point [-]

MY "objection" to CEV is exactly the opposite of what you're expecting and asking for. CEV as described is not descriptive enough to allow the hypothesis "CEV is an acceptably good solution" to be falsified. Since it is "our wish if we knew more", etc., any failure scenrio that we could possibly put forth can immediately be answered by altering the potential "CEV space" to answer the objection.

I have radically different ideas about where CEV is going to converge to than most people here. Yet, the lack of distinctions in the description of CEV cause my ideas to be included under any argument for CEV because CEV potentially is . . . ANYTHING! There are no concrete distinctions that clearly state that something is NOT part of the ultimate CEV.

Arguing against CEV is like arguing against science. Can you argue a concrete failure scenario of science? Now -- keeping Hume in mind, what does science tell the AI to do? It's precisely the same argument, except that CEV as a "computational procedure" is much less well-defined than the scientific method.

Don't get me wrong. I love the concept of CEV. It's a brilliant goal statement. But it's brilliant because it doesn't clearly exclude anything that we want -- and human biases lead us to believe that it will include everything we truly want and exclude everything we truly don't want.

My concept of CEV disallows AI slavery. Your answer to that is "If that is truly what a grown-up humanity wants/needs, then that is what CEV will be". CEV is the ultimate desire -- ever-changing and never real enough to be pinned down.

Comment author: Kevin 13 February 2011 09:36:34PM *  5 points [-]

Luke, as an intermediate step before writing a book you should write a book chapter for Springer's upcoming edited volume on the Singularity Hypothesis. http://singularityhypothesis.blogspot.com/p/about-singularity-hypothesis.html I'm not sure how biased they are against non-academics... probably depends on how many submissions they get.

Maybe email Louie and me and we can brainstorm about topics; meta-ethics might not be the best thing compared to something like making an argument about how we need to solve all of philosophy in order to safely build AI.

Comment author: mwaser 16 February 2011 12:05:48PM 2 points [-]

I know the individuals involved. They are not biased against non-academics and would welcome a well-thought-out contribution from anyone. You could easily have a suitable abstract ready by March 1st (two weeks early) if you believed that it was important enough -- and I would strongly urge you to do so.

Comment author: prase 29 November 2010 02:46:17PM 2 points [-]

Do threats work on you? One of the reasons I expect threats to be ineffective is that when I am threatened, I am less likely to comply, even for significant costs. Or at least it feels like that, to be completely sure that no bias is obscuring my experiences I would have to make some statistics.

Comment author: mwaser 29 November 2010 03:47:14PM 1 point [-]

Threats are certainly a data point that I factor in when making a decision. I, too, have been known to apply altruistic punishment to people making unwarranted threats. But I also consider whether the person feels so threatened that the threat may actually be just a sign of their insecurity. And there are always times when going along with the threat is simply easier than bothering to fight that particular issue.

Do you really always buck threats? Even when justified -- such a "threatened consequences" for stupid actions on your part? Even from, say, police officers?

Comment author: Perplexed 29 November 2010 09:05:39AM 6 points [-]

I think it would be a good idea to taboo the word "threat" here. I'm picking up strong vibes that not everyone is using the same meaning on this thread.

Comment author: mwaser 29 November 2010 03:15:18PM *  1 point [-]

I much prefer the word "consequence" -- as in, that action will have the following consequences . . . .

I don't threaten, I point out what consequences their actions will cause.

In response to Imperfect Levers
Comment author: mwaser 17 November 2010 11:06:02PM *  3 points [-]

For-profit corporations, as a matter of law, have the goal of making money and their boards are subject to all sorts of legal consequences and other unpleasantnesses if they don't optimize that goal as a primary objective (unless some other goal is explicitly written into the corporate bylaws as being more important than making a profit -- and even then, there are profit requirements that must be fulfilled to avoid corporate dissolution or conversion to a non-profit -- and very few corporations have such provisions).

Translation

Corporations = powerful, intelligent entities with the primary goal of accumulating power (in the form of money).

Comment author: Jonii 08 November 2010 02:39:10AM 9 points [-]

Not really. An AI that didn't have a specific desire to be friendly to mankind would want to kill us to cut down on unnecessary entropy increases.

As you get closer to the mark, with AGI's that have utility function that roughly resembles what we would want, but is still wrong, the end results are most likely worse than death. Especially since there should be much more near-misses than exact hits. Like, AGI that doesn't want to let you die, regardless of what you go through, and little regard to your other sort of well-being, would be closer to the FAI than paperclip maximizer that would just plain kill you. As you get closer to the core of friendliness, you get all sorts of weird AGI's that want to do something that twistedly resembles something good, but is somehow missing something or is somehow altered so that the end result is not at all what you wanted.

Comment author: mwaser 09 November 2010 01:07:12PM *  2 points [-]

As you get closer to the core of friendliness, you get all sorts of weird AGI's that want to do something that twistedly resembles something good, but is somehow missing something or is somehow altered so that the end result is not at all what you wanted.

Is this true or is this a useful assumption to protect us from doing something stupid?

Is it true that Friendliness is not an attractor or is it that we cannot count on such a property unless it is absolutely proven to be the case?

Comment author: Alicorn 08 November 2010 02:48:31AM 3 points [-]

By "cousins" I meant "rational", "irrational", "rationality", "irrationality", etcetera. "Effective" is not technically a cousin, but any form of search-and-replace would not be in keeping with the spirit of the exercise. Since you are confused, I will go into more detail, but I am nearing the last straw in trying to deal with you and won't extend the courtesy again.

Lurking until you truly get what's going on around you is not the most effective (rational) way to learn.

Do you mean: Lurking is slow compared to other strategies, lurking gets worse results for the newbie, lurking is worse for the rest of the community, lurking is inefficient, lurking fails altogether at achieving the objective, or something else?

I can provide you a boatload of references supporting that if you wish.

This is meaningless until you explain the assertion you offer to support.

Do you really want subpar newbies who will accept such irrationality just to maintain your peace and quiet?

Nope. That doesn't sound appealing at all. I would rather have zero subpar newbies, and instead of peace and quiet I want lively and productive signal with minimal noise. Also, "such irrationality" is presumptuous. Weren't you going on about how LW is actually governed by structures and rules that you now understand that only look irrational? Where did that go?

Particularly when a playground option is suggested? You could even get volunteers and never deal with the hassle.

Interestingly, your "option" is not so obviously and blindingly brilliant that I could only reject it as the solution to all my problems through sheer bloodymindedness. I don't actually want LW to be attached to a rock-bottom-standards blog with a similar color scheme that purports to funnel newbies into the real deal. I think that would be bad. Yes, even if I never have to look directly at it without a pinhole camera and even if it's minded by volunteers.

Premise: It's more rational for your goals, to just ignore a good rational proposal from an erring, annoying newbie who is trying to provide access to new resources for you (both newbies and structures for their care and feeding).

If you were demonstrating actual understanding of any relevant concepts... or if you were offering to personally do some work for the site instead of just throwing around vague plans for its expansion and calling it the provision of "access"... or if your proposal were actually good or "rational"... or, I'll admit it, if you weren't so annoying... then you'd be getting a better reception. This is, of course, a counterfactual.

Comment author: mwaser 09 November 2010 03:52:10AM *  1 point [-]

I meant lurking is slow, lurking is inefficient, and a higher probability that it gets worse results for the newbie. I'm not sure which objective is being referred to in that clause. I retract those evaluations as flawed.

Yeah, I made the same mistake twice in a row. First, I didn't get that I didn't get it. Then I "got it" and figured out some obvious stuff -- and didn't even consider that there probably was even more below that which I still didn't get and that I should start looking for (and was an ass about it to boot). What a concept -- I don't know what I don't know.

The playground option was an idiot idea. I actually figured out that I don't want to go there and stagnate before your comment. I've got this horrible mental image of me being that guy that whines in boot camp. Let me take a few days and come up with a good answer to one of your questions (once I've worked this through a bit more).

I'd say thank you and sorry for being an ass but I'm not sure of its appropriateness right now. (Yeah, that <humility> tag is still really messing with me ;-)

ETA: Still re-calibrating. Realizing I'm way too spoiled about obtaining positive feedback . . . . ;-) EDIT: Make that addicted to obtaining positive feedback and less accepting of negative feedback that I don't immediately understand than I prefer to realize (and actually commenting on the first part seems to immediately recurse into hilarity)

View more: Next