All of Silas's Comments + Replies

Well, now you f*in' tell me.

When I saw the picture, I assumed she was the woman you described in one of your Bayesian conspiracy stories that you post here. But then, she was in a pink jumpsuit, and had, I think, blond hair.

@Daniel_Franke: I was just describing a sufficient, not a necessary condition. I'm sure you can ethically get away with less. My point was just that, once you can make models that detailed, you needn't be prevented from using them altogether, because you wouldn't necessarily have to kill them (i.e. give them information-theoretic death) at any point.

@Tim_Tyler:

The main problem with death is that valuable things get lost. Once people are digital, this problem tends to go away - since you can relatively easily scan their brains - and preserve anything of genuine value. In summary, I don't see why this issue would be much of a problem.

I was going to say something similar, myself. All you have to do is constrain the FAI so that it's free to create any person-level models it wants, as long as it also reserves enough computational resources to preserve a copy so that the model citizen can later be re-... (read more)

Khyre: Setting or clearing a bit register regardless of what was there before is a one-bit irreversible operation (the other two one-bit input, one-bit output functions are constant 1 and constant 0).

face-palm I can't believe I missed that. Thanks for the correction :-)

Anyway, with that in mind, Landauer's principle has the strange implication that resetting anything to a known state, in such a way that the previous can't be retrieved, necessarily releases heat, and the more information the state conveys to the observer, the more heat is released. Okay, end threadjack...

I'm going to nitpick (mainly because of how much reading I've been doing about thermodynamics and information theory since your engines of cognition post):

Human neurons ... dissipate around a million times the heat per synaptic operation as the thermodynamic minimum for a one-bit operation at room temperature. ... it ought to be possible to run a brain at a million times the speed without ... invoking reversible computing or quantum computing.

I think you mean neurons dissipate a million times the thermodynamic minimum for an irreversible one-bit operat... (read more)

Nick_Tarleton: I think you're going a bit too far there. Stability control theory had by that time been rigorously and scientifically studied, dating back to Watts's flyball governor in the 18th century (which controlled shaft rotation speed by allowing a ball to swing out and increase rotational inertia as it sped up) and probably even before that with the incubator (which used heat to move a valve that allowed just the right amount of cooling air in). Then all throughout the 19th century engineers attacked the problem of "hunting" on trains, ... (read more)

@Scott_Aaronson: Previously, you had said the problem is solved with certainty after O(1) queries (which you had to, to satisfy the objection). Now, you're saying that after O(1) queries, it's merely a "high probability". Did you not change which claim you were defending?

Second, how can the required number of queries not depend on the problem size?

Finally, isn't your example a special case of exactly the situation Eliezer_Yudkowsky describes in this post? In it, he pointed out that the "worst case" corresponds to an adversary who kno... (read more)

Could Scott_Aaronson or anyone who knows what he's talking about, please tell me the name of the n/4 left/right bits problem he's referring to, or otherwise give me a reference for it? His explanation doesn't seem to make sense: the deterministic algorithm needs to examine 1+n/4 bits only in the worst case, so you can't compare that to the average output of the random algorithm. (Average case for the determistic would, it seems, be n/8 + 1) Furthermore, I don't understand how the random method could average out to a size-independent constant.

Is the randomized algorithm one that uses a quantum computer or something?

Someone please tell me if I understand this post correctly. Here is my attempt to summarize it:

"The two textbook results are results specifically about the worst case. But you only encounter the worst case when the environment can extract the maximum amount of knowledge it can about your 'experts', and exploits this knowledge to worsen your results. For this case (and nearby similar ones) only, randomizing your algorithm helps, but only because it destroys the ability of this 'adversary' to learn about your experts. If you instead average over all cases, the non-random algorithm works better."

Is that the argument?

@Caledonian and Tiiba: If we knew where the image was, we wouldn't need the dots.

Okay, let's take a step back: the scenario, as Caledonian originally stated, was that the museum people could make a patron better see the image if the museum people put random dots on the image. (Pronouns avoided for clarity.) So, the problem is framed as whether you can make someone else see an image that you already know is there, by somehow exploiting randomness. My response is that, if you already know the image is there, you can improve beyond randomness, but just put... (read more)

@Joshua_Simmons: I got to thinking about that idea as I read today's post, but I think Eliezer_Yudkowsky answered it therein: Yes, it's important to expirment, but why must your selection of what to try out, be random? You should be able to do better by exploiting all of your knowledge about the structure of the space, so as to pick better ways to experiment. To the extent that your non-random choices of what to test do worse than random, it is because your understanding of the problem is so poor as to be worse than random.

(And of course, the only time w... (read more)

So, in short: "Randomness is like poison: Yes, it can benefit you, but only if you feed it to people you don't like."

Will_Pearson: Is it literally? Are you saying I couldn't send a message to someone that enabled them to print out a list of the first hundred integers without referencing a human's cognitive structure.

Yes, that's what I'm saying. It's counterintuitive because you so effortlessly refernce others' cognitive structures. In communicating, you assume a certain amount of common understanding, which allows you to know whehter your message will be understood. In sending such a message, you rely on that information. You would have to think, "will they unde... (read more)

Okay, fair challenge.

I agree about your metal example, but it differs significantly from my discussion of the list-output program for the non-trivial reason I gave: specifically, the output is defined by its impact on people's cognitive structure.

Look at it this way: Tim_Tyler claims that I know everything there is to know about the output of a program that spits out the integers from 1 to 100. But, when I get the output, what makes me agree that I am in fact looking at those integers? Let's say that when printing it out (my argument can be converted to ... (read more)

Further analysis, you say, Tim_Tyler? Could you please redirect effort away from putdowns and into finding what was wrong with the reasoning in my previous comment?

Very worthwhile points, Tim_Tyler.

First of all, the reason for my spirited defense of MH's statement is that looked like a good theory because of how concise it was, and how consistent with my knowledge of programs it was. So, I upped my prior on it and tended to see apparent failures of it as a sign I'm not applying it correctly, and that further analysis could yield a useful insight.

And I think I that belief is turning out to be true:

It seems to specify that the output is what is unknown - not the sensations that output generates in any particular obser... (read more)

@Eliezer_Yudkowsky: It wouldn't be an exact sequence repeating, since the program would have to handle contingencies, like cows being uncooperative because of insufficiently stimulating conversation.

Nick_Tarleton: Actually, Tim_Tyler's claim would still be true there, because you may want to print out that list, even if you knew some exact arrangement of atoms with that property.

However, I think Marcello's Rule is still valid there and survives Tim_Tyler's objection: in that case, what you don't know is "the sensation arising from looking at a the numbers 1 through 100 prettily printed". Even if you had seen such a list before, you probably would want to print it out unless your memory were perfect.

My claim generalizes nicely. For example,... (read more)

Quick question: How would you build something smarter, in a general sense, than yourself? I'm not doubting that it's possible, I'm just interested in knowing the specific process one would use.

Keep it brief, please. ;-)

With all this talk about poisoned meat and CDSes, I was inspired to draw this comic.

Adam_Ierymenko: Evolution has evolved many strategies for evolution-- this is called the evolution of evolvability in the literature. These represent strategies for more efficiently finding local maxima in the fitness landscape under which these evolutionary processes operate. Examples include transposons, sexual reproduction,

Yes, Eliezer_Yudkowsky has discussed this before and calls that optimizaiton at the meta-level. Here is a representative post where he makes those distinctions.

Looking over the history of optimization on Earth up until now, the firs

I've raised this before when Eliezer made the point, and I'll raise it again:

There most certainly is truth to "Suicide bombers are cowardly". Someone who chooses to believe a sweet, comforting lie, and go to the grave rather than face the possibility of having to deal with being wrong, is cowardly, or the term has no meaning. They might have been less cowardly with respect to their lives, but they were absolutely more cowardly with respect to their intellectual integrity. And FWIW, American soldiers who meet all of that are just as cowardly.

Mo... (read more)

in a society of Artificial Intelligences worthy of personhood and lacking any inbuilt tendency to be corrupted by power, it would be right for the AI to murder ... I refuse to extend this reply to myself, because the epistemological state you ask me to imagine, can only exist among other kinds of people than human beings.

Interesting reply. But the AIs are programmed by corrupted humans. Do you really expect to be able to check the full source code? That you can outsmart the people who win obfuscated code contests?

How is the epistemological state of human-verified, human-built, non-corrupt AIs, any more possible?

@Phil_Goetz: Have the successes relied on a meta-approach, such as saying, "If you let me out of the box in this experiment, it will make people take the dangers of AI more seriously and possibly save all of humanity; whereas if you don't, you may doom us all"?

That was basically what I suggested in the previous topic, but at least one participant denied that Eliezer_Yudkowsky did that, saying it's a cheap trick, while some non-participants said it meets the spirit and letter of the rules.

One more thing: my concerns about "secret rules" apply just the same to Russell_Wallace's defense that there were no "cheap tricks". What does Russell_Wallace consider a non-"cheap trick" in convincing someone to voluntarily, knowingly give up money and admit they got fooled? Again, secret rules all around.

@Russell_Wallace & Ron_Garret: Then I must confess the protocol is ill-defined to the point that it's just a matter of guessing what secret rules Eliezer_Yudkowsky has in mind (and which the gatekeeper casually assumed), which is exactly why seeing the transcript is so desirable. (Ironically, unearthing the "secret rules" people adhere to in outputting judgments is itself the problem of Friendliness!)

From my reading, the rules literally make the problem equivalent to whether you can convince people to give money to you: They must know that l... (read more)

When first reading the AI-Box experiment a year ago, I reasoned that if you follow the rules and spirit of the experiment, the gatekeeper must be convinced to knowingly give you $X and knowingly show gullibility. From that perspective, it's impossible. And even if you could do it, that would mean you've solved a "human-psychology-complete" problem and then [insert point about SIAI funding and possibly about why you don't have 12 supermodel girlfriends].

Now, I think I see the answer. Basically, Eliezer_Yudkowsky doesn't really have to convince ... (read more)

1) Eliezer_Yudkowsky: You should be comparing the percentage (1) change in the S&P 500 (2) to the change (3) in probability of any bailout happening (4) over the days in which the changes occurred (5) and have used more than one day (6). There, that's six errors in your calculation I count, of varying severity.

2) Tim_Tyler: Yeah, I'm surprised that hasn't been posted on Slashdot yet. I want to be the first to propose the theory that United Airlines was behind that, since Google was the cause of a recent fake plunge in United's stock price, when they ... (read more)

V, Ori, and everyone else: In my recent post, I explain how you can synthesize short and long positions. You have to ban a lot more than short-selling to ban short-selling, and a lot more than margin-buying to ban leveraged longs.

@Lara_Foster: You see, it seems quite likely to me that humans evaluate utility in such a circular way under many circumstances, and therefore aren't performing any optimizations.

Eliezer touches on that issue in "Optimization and the Singularity":

Natural selection prefers more efficient replicators. Human intelligences have more complex preferences. Neither evolution nor humans have consistent utility functions, so viewing them as "optimization processes" is understood to be an approximation.

By the way, Ask middle school girls to r... (read more)

By the way:

Human: "What do you care about 3 paperclips? Haven't you made trillions already? That's like a rounding error!" Paperclip Maximizer: "How can you talk about paperclips like that?"

PM: "What do you care about a billion human algorithm continuities? You've got virtually the same one in billions of others! And you'll even be able to embed the algorithm in machines one day!" H: "How can you talk about human lives that way?"

Wait wait wait: Isn't this the same kind of argument as in the dilemma about "We will execute you within the next week on a day that you won't expect"? (Sorry, don't know the name for that puzzle.) In that one, the argument goes that if it's the last day of the week, the prisoner knows that's the last chance they have to execute him, so he'll expect it, so it can't be that day. But then, if it's the next-to-last day, he knows they can't execute him on the last day, so they have to execute him on that next-to-last day. But then he expects it! ... (read more)

Vassar handles personal networking? Dang, then I probably shouldn't have mouthed off at Robin right after he praised my work.

Eliezer, if the US government announced a new Manhattan Project-grade attempt to be the first to build AGI, and put you in charge, would you be able to confidently say how such money should be spent in order to make genuine progress on such a goal?

It would really rock if you could show the context in which someone used the word "arbitrary" but in a way that just passed the recursive buck.

Here's where I would use it:

[After I ask someone a series of questions about whether certain actions would be immoral]

Me: Now you're just being arbitrary! Eliezer Yudkowsky: Taboo "arbitrary"! Me: Okay, he's deciding what's immoral based on whim. Eliezer Yudkowsky: Taboo "whim"! Me: Okay, his procedures for deciding what's immoral can't be articulated with finite words to a stranger suc... (read more)

Wait a sec: I'm not sure people do outright avoid modifying their own desires so as to make the desires easier to satisfy, as you are claiming here:

We, ourselves, do not imagine the future and judge, that any future in which our brains want something, and that thing exists, is a good future. If we did think this way, we would say: "Yay! Go ahead and modify us to strongly want something cheap!"

Isn't that exactly what people do when they study ascetic philosophies and otherwise try to see what living simply is like? And would people turn down a pill that made vegetable juice taste like a milkshake and vice versa?

Who cares, I want to hear about building AIs.

Matt Simpson: Many an experiment has been thought for the sole purpose of showing how utilitarianism is in direct conflict with our moral intuitions.

I disagree, or you're referring to something I haven't heard of. If I know what you mean here, those are a species of strawman ("act") utilitarianism that doesn't account for the long-term impact and adjustment of behavior that results.

(I'm going to stop giving the caveats; just remember that I accept the possibility you're referring to something else.)

For example, if you're thinking about cases whe... (read more)

"If the federal government hadn't bought so much stuff from GM, GM would be a lot smaller today." "If the federal government hadn't bought so much stuff from GM, GM would have instead been tooling up to produce stuff other buyers did want and thus could very well have become successful that way."

???

I think a lot of people are confusing a) improved ability to act morally, and b) improved moral wisdom.

Remember, things like "having fewer deaths, conflicts" does not mean moral progress. It's only moral progress if people in general change their evaluation of the merit of e.g. fewer deaths, conflicts.

So it really is a difficult question Eliezer is asking: can you imagine how you would have/achieve greater moral wisdom in the future, as evaluated with your present mental faculties?

My best answer is yes, in that I can imagine being better able to... (read more)

I picked up The Moral Animal on Eliezer's recommendation, after becoming so immersed I read 50 pages in the bookstore. Was not disappointed. This is the most eye-opening book I've read in quite a while, nearly couldn't put it down. And this is from someone who used to stay a mile away from anything related to biology on the grounds that it's "boring".

Will probably blog it. Will also continue to drop subjects from sentences.

Eliezer, I think the point you've made here generalizes to several things in the standard fallacy lists, usually which take the form:

X Fallacy: Believing Y because of Z, when Z doesn't ABSOLUTELY GUARANTEE Y.

...even though, it turns out, Z should raise the probability you assign to Y.

For example:

Appeal to authority: An expert in the field believing something within that field doesn't guarantee its truth, but is strong evidence.

Argument from ignorance: The fact that you haven't heard of any good arguments for X, doesn't mean X is necessary false, but if mos... (read more)

How does this apply to religion? Is there e.g. a situation in which everyone knows that the tenets of the religion are false, and knows that everyone else knows they're false, but if this becomes common knowledge, the bond between the people breaks apart?

These are difficult questions, but I think I can tackle some of them:

Why would anyone want to change what they want?"

If a person wants to change from valuing A to valuing B, they are simply saying that they value B, but it requires short-term sacrifices, and in the short term, valuing A may feel psychologically easier, even though it sacrifices B. They thus want to value B so that it is psychologically easier to make the tradeoff.

Why and how does anyone ever "do something they know they shouldn't", or "want something they know is w

Unknown: it's okay, maybe you meant a different mind, like a grade-schooler taking a multiplication quiz ;-)

Anyway, if I were going to taboo "correctness", I would choose the more well-defined "lack of Dutch-bookability".

Shane_Legg: factoring in approximations, it's still about zero. I googled a lot hoping to find someone actually using some version of it, but only found the SIAI's blog's python implementation of Solomonoff induction, which doesn't even compile on Windows.

Does anyone know the ratio of discussion of implementations of AIXI/-tl, to discussion of its theoretical properties? I've calculated it at about zero.

Alright, since we're at this "summary/resting point", I want to re-ask a clarifying question that never got answered. One the very important "What an Algorithm Feels like from inside" post, I asked what the heck each graph (Network 1 and 2) was supposed to represent, and never got a clear answer.

Now, before you lecture me about how I should have figured it out right now, let's be realistic. Even the very best answer I got, requires me to download a huge pdf file, and read a few chapters, most of it irrelevant to understanding what Eli... (read more)

You can't admit a single particle of uncertain danger if you want your science's funding to survive. ... So no one can do serious analysis of existential risks anymore, because just by asking the question, you're threatening the funding of your whole field.

And by writing this blog post, you're doing what to the LHC...?