Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: passive_fist 25 May 2015 03:05:23AM 0 points [-]

Every so often someone proposes this (and sometimes someone who thinks they are clever actually carries it out) and it's always a terrible idea. The purpose of peer review is not to uncover fraud. It's not even to make sure what's in the paper is correct. The purpose of peer review is just to make sure what's in the paper is plausible and sane, and worth being presented to a wider audience. The purpose is to weed out obvious low-quality material such as perpetual motion machines or people who are duplicating other's work as their own. Could you get fraudulent papers accepted in a journal? Of course. A scientist sufficiently knowledgeable of their field could definitely fool almost any arbitrarily rigorous peer review procedure. Does fraud exist in the scientific world? Of course it does. Peer review is just one of the many mechanisms that serve to uncover it. Real review of one's work begins after peer review is over and the work is examined by the scientific community at large.

Comment author: gwern 25 May 2015 02:56:26AM 0 points [-]

With the exception of Michael Anissimov we moved to The Future Primaeval.

What happened there?

Comment author: ESRogs 25 May 2015 02:45:03AM *  0 points [-]

That's much more about how he wants to live in the tiny little near world around himself

I don't follow this. Isn't he just talking about alternative possible futures -- one where he hides his illness, and one where he's open about it?

He's saying he felt an inclination to hide it, but then he reasoned about the implications of following that feeling, and decided that in the long run being open about his mental illness would be better.

Comment author: ESRogs 25 May 2015 02:36:51AM 0 points [-]

Future of Life Institute is the only existential-risk AI organization which is actually doing meaningful evidence-based research into artificial intelligence.

Has FLI done any research to date? My impression is that they're just disbursing Musk's grant (which for the record I think is fantastic).

It sounds like they're trying to disburse the grant broadly and to encourage a variety of different types of research including looking into more short term AI safety problems. This approach seems like it has the potential to engage more of the existing computer science and AI community, and to connect concerns about AI risk to current practice.

Is that what you like about it?

Comment author: Trevor_Blake 25 May 2015 02:09:09AM 0 points [-]

"When I speak of reason or rationalism, all I mean is the conviction that we can learn through criticism of our mistakes and errors, especially through criticism by others, and eventually also through self-criticism. A rationalist is simply someone for whom it is more important to learn than to be proved right; someone who is willing to learn from others — not by simply taking over another's opinions, but by gladly allowing others to criticize his ideas and by gladly criticizing the ideas of others. The emphasis here is on the idea of criticism or, to be more precise, critical discussion. The genuine rationalist does not think that he or anyone else is in possession of the truth; nor does he think that mere criticism as such helps us achieve new ideas. But he does think that, in the sphere of ideas, only critical discussion can help us sort the wheat from the chaff. He is well aware that acceptance or rejection of an idea is never a purely rational matter; but he thinks that only critical discussion can give us the maturity to see an idea from more and more sides and to make a correct judgement of it." - Karl Popper, 'All Life is Problem Solving' (1999)

Karl Popper'a method influences me. I consider errors identified as the main (perhaps only) way to be less wrong.

Comment author: philh 25 May 2015 02:04:33AM 0 points [-]

Doesn't it? IIRC you need 2 karma to post a top-level thread in discussion, more than that to post a top-level thread in main, but none to comment.

Comment author: Gondolinian 25 May 2015 02:04:17AM *  0 points [-]

I'm not sure I accept your premises? I could certainly be wrong, but I have not gotten the impression that comments can be prevented by low karma, only posts to Discussion or Main. (And I recall the minimum as 20, not 2.) The most obvious way to get the karma needed to post is by commenting on existing posts (including open threads and welcome threads), and new users with zero initial karma regularly do this without any apparent difficulty, so unless I'm missing something, I don't think it's a problem?

Comment author: Halfwitz 25 May 2015 01:47:30AM *  0 points [-]

This forum doesn't allow you to comment if you have <2 karma. How does one get their first 2 karma then?

Comment author: Epictetus 25 May 2015 01:32:02AM 0 points [-]

A president or prime minister will be the public face of the nation. He'd be expected to meet with foreign dignitaries and speak in public. At the very least, a debate gives people an idea of how their leaders carry themselves when under stress in full public view.

However, if I had to choose between live, in-person debates and a written format where candidates had plenty of time to formulate their thoughts and gather supporting evidence, I'd take the written format every time.

Making prepared statements is usually done by a politician's staff. The candidate might make some suggestions and approve/reject a draft, but otherwise such a debate would be staffers vs. staffers.

I'll also note that once upon a time, people attended public debates in part for entertainment.

On a related thought, I've idly mused on multiple occasions that live in-person political debates seem overweighted in importance.

Overall I do agree. I seldom watch debates, because what the candidates do say is often just a condensed version of the party position that shows up on any one of a dozen websites.

Comment author: philh 25 May 2015 01:08:43AM 0 points [-]

Not particularly LWish, but possibly of interest: towards a taxonomy of logic puzzles.

Comment author: Gleb_Tsipursky 25 May 2015 12:47:02AM 0 points [-]

The only thing we have any real influence over in the world is our own thought, feeling, and behavior patterns. So yes, I would say that making choices about that tiny little world is the most important thing we can be doing as agents. Everything else is secondary.

Comment author: Gleb_Tsipursky 25 May 2015 12:45:37AM 0 points [-]

Well, one thing we learned from empirically was that the most responsive audience to our content was in the secular community. We assumed that people into health and wellness would be more responsive, but our content is not sufficiently "woo" for most. So we decided to focus our early efforts on spreading rationality among the secular community, and then branch out later after we develop solid ties there.

Comment author: Konkvistador 25 May 2015 12:37:42AM *  0 points [-]

With the exception of Michael Anissimov we moved to The Future Primaeval. We will be reposing our archives there as well shortly.

These three posts should be a good introduction for the new site:

Comment author: ChristianKl 25 May 2015 12:33:37AM 0 points [-]

Some well-designed studies even prove that ghosts exist.

I"m not aware of ghosts, Scott talks about telepathy and precognition studies.

Comment author: James_Miller 25 May 2015 12:32:55AM *  0 points [-]

I do have uncertainty over how much of a boost you can get by engaging in fraud.

Comment author: ChristianKl 25 May 2015 12:30:43AM 2 points [-]

Do you have any doubt that it's very easy to get fraudulent work accepted?

Comment author: James_Miller 25 May 2015 12:26:28AM 2 points [-]

I propose that some major academic organization such as the American Economic Association randomly and secretly choose a few members and request that they attempt to get fraudulent work accepted into the highest ranked journals they can. They reveal the fraud as soon as an article is accepted. This procedure would give us some idea how of easy it is to engage in fraud, and give journals additional incentives to search for it. For some academic disciplines the incentives to engage in fraud seem similar to that with illegal performance enhancing drugs and professional sports, and I wonder if the outcomes are similar.

Comment author: buybuydandavis 24 May 2015 11:56:47PM 0 points [-]

here is an xkcd which highlights the difference: https://xkcd.com/1132/

Sweet!

Comment author: Vladimir_Nesov 24 May 2015 11:51:21PM *  2 points [-]

My point is that the very statements you are making, that we are all making all the time, are also very theory-loaded, "not followed up with empirical studies". This includes the statements about the need to follow things up with empirical studies. You can't escape the need for experimentally unverified theoretical judgement, and it does seem to work, even though I can't give you a well-designed experimental verification of that. Some well-designed studies even prove that ghosts exist.

The degree to which discussion of familiar topics is closer to observations than discussion of more theoretical topics is unclear, and the distinction should be cashed out as uncertainty on a case-by-case basis. Some very theoretical things are crystal clear math, more certain than the measurement of the charge of an electron.

That is dangerous.

Being wrong is dangerous. Not taking theoretical arguments into account can result in error. This statement probably wouldn't be much affected by further experimental verification. What specifically should be concluded depends on the problem, not on a vague outside measure of the problem like the degree to which it's removed from empirical study.

[...] anything you yourself don't have available experimental evidence for or against, I can sway you in either way. E.g. that consciousness is in information being computed and not the computational process itself.

Before considering the truth of a statement, we should first establish its meaning, which describes the conditions for judging its truth. For a vague idea, there are many alternative formulations of its meaning, and it may be unclear which one is interesting, but that's separate from the issue of thinking about any specific formulation clearly.

Comment author: Kaj_Sotala 24 May 2015 11:21:50PM 0 points [-]

Funding for SENS might fund research that could be considered too speculative for more conventional bio funders, though.

Comment author: Jiro 24 May 2015 11:18:27PM *  -1 points [-]

If you have some other reason to think that superintelligences are hard to box -- one that depends on a relevant difference between the experiment and a realistic AI scenario -- then feel free to bring that idea up.

Not all relevant differences between an experiment and an actual AI scenario can be accurately characterized as "reason to think that superintelligences are hard to box". For instance, imagine an experiment with no gatekeeper or AI party at all, where the result of the experiment depends on flipping a coin to decide whether the AI gets out. That experiment is very different from a realistic AI scenario, but one need not have a reason to believe that intelligences are hard to box--or even hold any opinion at all on whether intelligences are hard to box--to object to the experimental design.

For the AI box experiment as stated, one of the biggest flaws is that the gatekeeper is required to stay engaged with the AI and can't ignore it. This allows the AI to win by either verbally abusing the gatekeeper to the extent that he doesn't want to stay around any more, or by overwhelming the gatekeeper with lengthy arguments that take time or outside assistance to analyze. These situations would not be a win for an actual AI in a box.

I mean, the experiment's been replicated multiple times. And you already know the reasons the transcripts were left private. I understand assigning a bit less weight to the evidence because you can't examine it in detail, but the hypothesis that there's a conspiracy to fake all of these experiments isn't likely.

Refusing to release the transcripts causes other problems than just hiding fakery. If the experiment is flawed in some way, for instance, it could hide that--and it would be foolish to demand that everyone name possible flaws one by one and ask you "does this have flaw A?", "does this have flaw B?", etc. in order to determine whether the experiment has any flaws. There are also cases where whether something is a flaw is an opinion that can be argued, and it might be that someone else would consider a flaw something that the experimenter doesn't.

Besides, in a real boxed AI situation, it's likely that gatekeepers will be tested on AI-box experiments and will be given transcripts of experiment sessions to better prepare them for the real AI. An experiment that simulates an AI boxing should likewise have participants be able to read other sessions.

Comment author: Kaj_Sotala 24 May 2015 11:18:20PM *  0 points [-]

I agree with Jacob that it is and should be concerning

That depends on whether you believe that machine intelligence researchers are the people who are currently the most likely to produce valuable progress on the relevant research questions.

One can reasonably disagree on MIRI's current choices about their research program, but I certainly don't think that their choices are concerning in the sense of suggesting irrationality on their part. (Rather the choices only suggest differing empirical beliefs which are arguable, but still well within the range of non-insane beliefs.)

Comment author: DanielLC 24 May 2015 11:05:07PM 0 points [-]

because simply making all existing humans immortal solves the DA as well.

I disagree. An appreciable number of people might be the ones designing the AI, but they won't spend an appreciable portion of their lives doing it.

Comment author: Kaj_Sotala 24 May 2015 11:04:55PM 0 points [-]

At one point I started developing a religious RPG character who applied theoretical computer science to his faith.

I forget details, but among other details he believed that although the Bible prescribed the best way to live, the world is far too complex for any finite set of written rules to cover every situation. The same limitation applies to human reason: cognitive science and computational complexity theory have shown all the ways in which we are bounded reasoners, and can only ever hope to comprehend a small part of the whole world. Reason works best when it can be applied to constrained problems where clear objective answer can be found, but it easily fails once the number of variables grows.

Thus, because science has shown that both the written word of the Bible and human reason are fallible and easily lead us astray (though the word of the Bible is less likely to do so), the rational course of action for one who believes in science is to pray to God for guidance and trust the Holy Spirit to lead us to the right choices.

Comment author: Mark_Friedenbach 24 May 2015 10:44:57PM 0 points [-]

That's very cynical. What makes you say that?

Comment author: Mark_Friedenbach 24 May 2015 10:21:49PM *  0 points [-]

Not really. You've seen the movie Sphere, or read the book? Magic could be similar: the source of magic is a wish-granting device that makes whatever someone with wizard gene think of, actually happen. Of course this is incredibly dangerous--all I have to do is shout "don't think of the Apocalypse!" in a room of wizards and watch the world end. So early wizards like Merlin interdicted by using their magic to implant false memories into the entire wizarding population to provide a sort of basic set of safety rules -- magic requires wands, enchantments have to be said correctly with the right hand motion, creating new spells requires herculean effort, etc. None of that would be true, but the presence of other wizards in the world thinking it were true would be enough to make the wish-granting device enforce the rules anyway.

Comment author: RobbBB 24 May 2015 10:13:36PM *  1 point [-]

I think Eliezer mostly just used "Bayesian superintelligence" as a synonym for "superintelligence." The "Bayesian" is there to emphasize the fact that he has Bayes-optimality as a background idea in his model of what-makes-cognition-work and what-makes-some-cognition-work-better-than-other-kinds, but Eliezer thought AI could take over the world long before he knew about AIXI or thought Bayesian models of cognition were important.

I don't know as much about Anna's views. Maybe she does assign more weight to AIXI as a source of data; the example you cited supports that. Though since she immediately follows up her AIXI example with "AIXI is a theoretical toy. How plausible are smarter systems in the real world?" and proceeds to cite some of the examples I mentioned above, I'm going to guess she isn't getting most of her intuitions about superintelligence from AIXI either.

there is no particular reason to assume that the physically attainable bound will be enormously higher than human-level. The more extreme the scenario, the less probability we should assign to it, reasonably according to a light-tailed distribution.

I think our disagreement is about what counts as "extreme" or "extraordinary", in the "extraordinary claims require extraordinary evidence" sense.

If I'm understanding your perspective, you think we should assume at the outset that humans are about halfway between 'minimal intelligence' and 'maximal intelligence' -- a very uninformed prior -- and we should then update only very weakly in the direction of 'humans are closer to the minimum than to the maximum'. Claiming that there's plenty of room above us is an 'extreme' claim relative to that uninformed prior, so the epistemically modest thing to do is to stick pretty close the assumption that humans are 'average', that the range of intelligence exhibited in humans with different cognitive abilities and disabilities represents a non-tiny portion of the range of physically possible intelligence.

My view is that we should already have updated strongly away from the 'humans are average' prior as soon as we acquired information about how humans arose -- through evolution, a process that has computational resources and perseverance but no engineering ingenuity, constructing our current advantages out of chimpanzees' over the course of just 250,000 generations. At that point we are no longer a randomly selected mind; our prior is swamped by all the rich information we have about ourselves as an evolved species, enhanced by culture but not by any deliberate genetic engineering or fine-grained neurosurgery. We have no more reason in the year 2015 to think that we represent 1/10 (or for that matter 1/10,000) of what is cognitively possible, than a person in the year 1850 would have reason to think that the fastest possible flying machine was only 10x as fast as the fastest bird, or that the most powerful possible bomb was only 10x as strong as the most powerful existing bomb.

Our knowledge of physics and of present technological capabilities, though still extremely flawed as data to generalize from, is rich enough to strongly shift a flat prior (informed by nothing more than 'this bird / bomb / brain exists and the rest of the universe is unknown to me'). So although we should not be confident of any specific prediction, 'humans are near the practical intelligence maximum' is the extreme view that requires a lot of evidence to move toward. Superintelligence is unusual relative to what we can directly observe, not 'extreme' in an evidential sense.

Comment author: ChristianKl 24 May 2015 10:10:28PM 0 points [-]

What is this decision theory? I haven't read the Sequences yet, sorry.

That's not the kind of question to be answered in a paragraph. But the label for Eliezers theory is Timeless Decision Theory (TDT).

Comment author: ChristianKl 24 May 2015 10:00:25PM 0 points [-]

Wanted to mention that Intentional Insights is a nonprofit specifically promoting rationality for the masses, including encouraging empirical and evidence-based approaches (I'm the President).

What did you do from which you learned empirically? Which assumptions that you had when you started, have you managed to falsify?

Comment author: Gunnar_Zarncke 24 May 2015 09:37:50PM 0 points [-]

See LW wiki's Doomsday Argument for reference.

The problem I have with this kind of reasoning is that it causes early reasoners to come to wrong conclusions (though 'on average' the reasoning is most probably true).

Comment author: Douglas_Knight 24 May 2015 09:22:33PM 0 points [-]

I guess that Haywood's Atlas is out of print because there is something like a second edition, although I could be wrong in that characterization.

Comment author: Douglas_Knight 24 May 2015 09:17:09PM 0 points [-]

I think a bigger problem with popular books is that they are often just wrong.

Comment author: V_V 24 May 2015 09:09:59PM *  1 point [-]

I don't think anyone at MIRI arrived at worries like 'AI might be able to deceive their programmers' or 'AI might be able to design powerful pathogens' by staring at the equation for AIXI or AIXItl.

In his quantum physics sequence, where he constantly talks (rants, actually) about Solomonoff Induction, Yudkowsky writes:

A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis—perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration—by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.

Anna Salamon also mentions AIXI when discussing the feasibility of super-intelligence.

Mind you, I'm not saying that AIXI is not an interesting and possibly theoretically useful model, my objection is that MIRI people seem to have used it to set a reference class for their intuitions about super-intelligence.

Rates of cultural and technological progress in human societies are a better (though still very imperfect) source of data than AIXI about how much improvement intelligence makes possible.

Extrapolation is always an epistemically questionable endeavor.
Intelligence is intrinsically limited by how predictable the world is. Efficiency (time complexity/space complexity/energy complexity/etc.) of algorithms for any computational task is bounded. Hardware resources also have physical limits.

This doesn't mean that given our current understanding we can claim that human-level intelligence is an upper bound. That would be most likely false. But there is no particular reason to assume that the physically attainable bound will be enormously higher than human-level. The more extreme the scenario, the less probability we should assign to it, reasonably according to a light-tailed distribution.

This should be possible to some extent, especially when it comes to progress in mathematics.

Ok, but my point is that it has not been established that progress in mathematics will automatically grant an AI "superpowers" in the physical world.
And I'd even say that even superpowers by raw cognitive power alone are questionable. Theorem proving can be sped up, but there is more to math than theorem proving.

Comment author: RobbBB 24 May 2015 08:58:00PM *  1 point [-]

Except that when the hypothesis space is large, people test hypotheses because they strongly updated in the direction of them being true, and seeing empirical data produces a later, weaker update. Where an example of 'strongly updating' could be going from 9,999,999:1 odds against a hypothesis to 99:1 odds, and an example of 'weakly updating' could be going from 99:1 odds against the hypothesis to 1:99. The former update requires about 20 bits of evidence, while the latter update requires about 10 bits of evidence.

Comment author: Douglas_Knight 24 May 2015 08:53:11PM *  1 point [-]

Apply jitter() inside of qplot. Also, transparency:

qplot(jitter(Age), jitter(CharityLog,a=0.1), color=EffectiveAltruism, data=survey,alpha=I(0.5))

Default jitter() worked great on Age, but was negligible on CharityLog.

Comment author: estimator 24 May 2015 08:47:42PM *  0 points [-]

Whether the distinction is worth making or not, it is irrelevant to my point, since both are very unlikely and therefore require much more evidence than we have now.

I assume that your idea is to prevent doomsday or make it less likely. If not, why bother with all these simulations?

Comment author: Eitan_Zohar 24 May 2015 08:42:07PM *  0 points [-]

Well, for a start, I don't think that the builders would want to be the only people in their world. And recall that this only serves to produce new humans, because simply making all existing humans immortal solves the DA as well. I think it would be more efficient to fully populate the simulation.

What is this decision theory? I haven't read the Sequences yet, sorry.

Comment author: Eitan_Zohar 24 May 2015 08:31:57PM *  0 points [-]

You say that one can change A by changing B, while there is no causal mechanism by which B can influence A. That's denying causality.

That's finding a loophole in causality, and the distinction is certainly worth making. The DA is only a product of perspective; it isn't a 'real' thing that exists.

Comment author: DanielLC 24 May 2015 08:29:59PM 0 points [-]

If I were doing it, I'd save computing power by only simulating the people who would program the AI. I don't think I'm going to do that, so it doesn't apply to me. Eliezer doesn't accept the Doomsday Argument, or at least uses a decision theory that makes it irrelevant, so it wouldn't apply to him.

Comment author: Douglas_Knight 24 May 2015 08:26:46PM 1 point [-]

Suicide does not usually block life insurance.

Comment author: estimator 24 May 2015 08:15:01PM *  0 points [-]

You say that one can change A by changing B, while there is no causal mechanism by which B can influence A. That's denying causality.

Well, if you don't like the term 'denying causality', feel free to replace it, but the point holds anyway.

In my prior probabilities system, finding a way around causality is somewhere near finding a way around energy conservation law. No way, unless there are tons of evidence.

Comment author: Eitan_Zohar 24 May 2015 08:03:07PM *  0 points [-]

So, we have a choice: to deny a very counterintuitive statement or to deny causality.

I'm not 'denying causality', I'm pointing out a way around it.

Comment author: Eitan_Zohar 24 May 2015 07:55:43PM *  0 points [-]

The question isn't how many simulated observers exist in total (although that's also unknown), but how many of them are like you in some relevant sense, i.e. what to consider "typical".

I also find it hard to believe that humans of any sort would hold special interest to a superintelligence. Do I really have the burden of proof there?

But in any case, I don't think your original idea works. Running a simulation of your ancestors causes your simulated ancestors to be wrong about the DA, but it doesn't cause yourself to be wrong about it.

The whole point is that the simulators want to find themselves in a simulation, and would only discover the truth after disaster has been avoided. It's a way of ensuring that superintelligence does not fulfill the DA.

Comment author: gjm 24 May 2015 07:54:43PM 0 points [-]

By all means. At this point I'll be quite surprised if you don't suspect the same account as I do! It would be interesting to know your reasons for suspicion, too.

Comment author: CellBioGuy 24 May 2015 07:39:28PM *  0 points [-]

100% of my charitable donations are going to SENS. Why they do not get more play in the effective altruism community is beyond me.

Probably because they're unlikely to lead to anything special over and above general biology research.

Comment author: Valentine 24 May 2015 07:36:08PM 0 points [-]

I like your way of saying it. It's much more efficient than mine!

Comment author: Valentine 24 May 2015 07:32:00PM *  2 points [-]

Those are not different models. They are different interpretations of the utility of probability in different classes of applications.

That's what a model is in this case.

I'm sorry, but this Bayesian vs Frequentist conflict is for the most part non-existent.

[…]

One of the failings of the sequences is the amount of emphasis that is placed on “Frequentist” vs “Bayesian” interpretations. The conflict between the two exists mostly in Yudkowsky's mind. Actual statisticians use probability to model events and knowledge of events simultaneously.

How sure are you of that?

I know a fellow who has a Ph.D. in statistics and works for the Department of Defense on cryptography. I think he largely agrees with your point: professional statisticians need to use both methods fluidly in order to do useful work. But he also doesn't claim that they're both secretly the same thing. He says that strong Bayesianism is useless in some cases that Frequentism gets right, and vice versa, though his sympathies lie more with the Frequentist position on pragmatic grounds (i.e. that methods that are easier to understand in a Frequentist framing tend to be more useful in a wider range of circumstances in his experience).

I think the debate is silly. It's like debating which model of hyperbolic geometry is "right". Different models highlight different intuitions about the formal system, and they make different aspects of the formal theorems more or less relevant to specific cases.

I think Eliezer's claim is that as a matter of psychology, using a Bayesian model of probability lets you think about the results of probability theory as laws of thought, and from that you can derive some useful results about how one ought to think and what results from experimental psychology ought to capture one's attention. He might also be claiming somewhere that Frequentism is in fact inconsistent and therefore is simply a wrong model to adopt, but honestly if he's arguing that then I'm inclined to ignore him because people who know a lot more about Frequentism than he does don't seem to agree.

But there is a debate, even if I think it's silly and quite pointless.

And also, the axiomatic models are different, even if statisticians use both.

Regarding the other points, every single example you gave involves using empirical data that had not sufficiently propagated, which is exactly the sort of use I am in favor of. So I don't know what it is that you disagree with.

The concern about AI risk is also the result of an attempt to propagate implications of empirical data. It just goes farther than what I think you consider sensible, and I think you're encouraging an unnecessary limitation on human reasoning power by calling such reasoning unjustified.

I agree, it should itch that there haven't been empirical tests of several of the key ideas involved in AI risk, and I think there should be a visceral sense of making bullshit up attached to this speculation unless and until we can find ways to do those empirical tests.

But I think it's the same kind of stupid to ignore these projections as it is to ignore that you already know how your New Year's Resolution isn't going to work. It's not obviously as strong a stupidity, but the flavor is exactly the same.

If we could banish that taste from our minds, then even without better empiricism we would be vastly stronger.

I'm concerned that you're underestimating the value of this strength, and viewing its pursuit as a memetic hazard.

I don't think we have to choose between massively improving our ability to make correct clever arguments and massively improving the drive and cleverness with which we ask nature its opinion. I think we can have both, and I think that getting AI risk and things like it right requires both.

But just as measuring everything about yourself isn't really a fully mature expression of empiricism, I'm concerned about the memes you're spreading in the name of mature empiricism retarding the art of finishing thinking.

I don't think that they have to oppose.

And I'm under the impression that you think otherwise.

Comment author: DanArmak 24 May 2015 07:30:23PM 0 points [-]

I don't see why simulated observers would almost ever outnumber physical observers. It would need an incredibly inefficient allocation of resources.

The question isn't how many simulated observers exist in total (although that's also unknown), but how many of them are like you in some relevant sense, i.e. what to consider "typical".

Avoiding the DA gives them a much clearer motive. It's the only reason I can think of that I would want to do it. Surely it's at least worth considering?

Many people do think they would have other reasons to run ancestor simulations.

But in any case, I don't think your original idea works. Running a simulation of your ancestors causes your simulated ancestors to be wrong about the DA, but it doesn't cause yourself to be wrong about it.

Trying to steelman, what you'd need is to run simulations of people successfully launching a friendly self-modifying AI. Suppose out of every N civs that run an AI, on average one succeeds and all the others go extinct. If each of them precommits to simulating N civs, and the simulations are arranged so that in a simulation running an AI always works, so in the end there are still N civs that successfully ran an AI.

This implies a certain measure on future outcomes: it's counting "distinct" existences while ignoring the actual measure of future probability. This is structurally similar to quantum suicide or quantum roulette.

Comment author: estimator 24 May 2015 07:29:50PM 0 points [-]

Sorry for probably being too sharp.

So, we have a choice: to deny a very counterintuitive statement or to deny causality. Do we have enough evidence to choose the latter? IMO, certainly, no. Anything we have are some thought experiments, which can be misinterpreted or wrong, and contain a lot of hand-waving.

Reformulating in bayesian terms: prior probability for your statement being correct is extremely tiny, and there is almost no evidence to update on. What to do? Reject.

Comment author: RobbBB 24 May 2015 07:19:12PM *  1 point [-]

You can't work backwards from the fact that someone arrived at truth in one case to the the premise that they must have been working from a reliable method for arriving at truth. It's the "one case" that's the problem. They might have struck lucky.

I mentioned that possibility above. But Einstein couldn't have been merely lucky -- even it weren't the case that he was able to succeed repeatedly, his very first success was too improbable for him to have just plucking random physical theories out of a hat. Einstein was not a random number generator, so there was some kind of useful cognitive work going on.

That leaves open the possibility that it was only useful enough to give Einstein a 1% chance of actually being right; but still, I'm curious about whether you do think he only had a 1% chance of being right, or (if not) what rough order of magnitude you'd estimate. And I'd likewise like to know what method he used to even reach a 1% probability of success (or 10%, or 0.1%), and why we should or shouldn't think this method could be useful elsewhere.

Einstein's thought experiments inspired his formal theories, which were then confirmed by observation. Nobody thought the thought experiments provided confirmation by themselves.

Can you define "confirmation" for me, in terms of probability theory?

Comment author: Eitan_Zohar 24 May 2015 07:17:57PM 1 point [-]

It's counterintuitive, yes, which does not make it acceptable for you to fart out some mockery and consider the argument closed.

Comment author: Good_Burning_Plastic 24 May 2015 07:14:58PM 0 points [-]

I actually have a suspicion which account, too

Me too. Should you I PM you to tell which one?

Comment author: estimator 24 May 2015 07:12:37PM 0 points [-]

Nope. I don't think ignoring causality to such extent makes sense. Simulating many instances of humanity won't make other risks magically go away, because it basically has no effect on them.

Yet another example of how one can misuse rationality and start to believe bogus statements.

Comment author: Eitan_Zohar 24 May 2015 07:11:57PM *  0 points [-]

If you think you're being simulated, then you need to predict what kinds and amounts of simulations exist besides the one you're in, as well as how extensive and precise your own simulation is in past time and space, not just in its future.

I don't see why simulated observers would almost ever outnumber physical observers. It would need an incredibly inefficient allocation of resources.

There are lots of reasons other than the DA to think we're being simulated: e.g. Bostrom's Simulation Argument (posthumans are likely to run ancestor simulations).

Avoiding the DA gives them a much clearer motive. It's the only reason I can think of that I would want to do it. Surely it's at least worth considering?

Comment author: DanArmak 24 May 2015 07:04:44PM 0 points [-]

the assumption that our experiences should be more-or-less ordinary

How do you know what to call "ordinary"? If you think you're being simulated, then you need to predict what kinds and amounts of simulations exist besides the one you're in, as well as how extensive and precise your own simulation is in past time and space, not just in its future.

And this is designed to escape the DA; it's the only reason to think you are simulated in the first place.

There are lots of reasons other than the DA to think we're being simulated: e.g. Bostrom's Simulation Argument (posthumans are likely to run ancestor simulations). The DA is a very weak argument for simulation: it is equally consistent with there being an extinction event in our future.

Comment author: RobbBB 24 May 2015 07:02:50PM *  2 points [-]

No, that is not how it works: I don't need to either accept or reject MWI. I can also treat it as a causal story lacking empirical content.

To say that MWI lacks empirical content is also to say that the negation of MWI lacks empirical content. So this doesn't tell us, for example, whether to assign higher probability to MWI or to the disjunction of all non-MWI interpretations.

Suppose your ancestors sent out a spaceship eons ago, and by your calculations it recently traveled so far away that no physical process could ever cause you and the spaceship to interact again. If you then want to say that 'the claim the spaceship still exists lacks empirical content,' then OK. But you will also have to say 'the claim the spaceship blipped out of existence when it traveled far enough away lacks empirical content'.

And there will still be some probability, given the evidence, that the spaceship did vs. didn't blip out of existence; and just saying 'it lacks empirical content!' will not tell you whether to design future spaceships so that their life support systems keep operating past the point of no return.

By that logic, if I invent any crazy hypothesis in addition to an empirically testable theory, then it inherits testability just on those grounds. You can do that with the word "testabiity" if you want, but that seems to be not how people use words.

There's no ambiguity if you clarify whether you're talking about the additional crazy hypothesis, vs. talking about the conjunction 'additional crazy hypothesis + empirically testable theory'. Presumably you're imagining a scenario where the conjunction taken as a whole is testable, though one of the conjuncts is not. So just say that.

Sean Carroll summarizes collapse-flavored QM as the conjunction of these five claims:

  1. Quantum states are represented by wave functions, which are vectors in a mathematical space called Hilbert space.

  2. Wave functions evolve in time according to the Schrödinger equation.

  3. The act of measuring a quantum system returns a number, known as the eigenvalue of the quantity being measured.

  4. The probability of getting any particular eigenvalue is equal to the square of the amplitude for that eigenvalue.

  5. After the measurement is performed, the wave function “collapses” to a new state in which the wave function is localized precisely on the observed eigenvalue (as opposed to being in a superposition of many different possibilities).

Many-worlds-flavored QM, on the other hand, is the conjunction of 1 and 2, plus the negation of 5 -- i.e., it's an affirmation of wave functions and their dynamics (which effectively all physicists agree about), plus a rejection of the 'collapses' some theorists add to keep the world small and probabilistic. (If you'd like, you could supplement 'not 5' with 'not Bohmian mechanics'; but for present purposes we can mostly lump Bohm in with multiverse interpretations, because Eliezer's blog series is mostly about rejecting collapse rather than about affirming a particular non-collapse view.)

If we want 'QM' to be the neutral content shared by all these interpretations, then we can say that QM is simply the conjunction of 1 and 2. You are then free to say that we should assign 50% probability to claim 5, and maintain agnosticism between collapse and non-collapse views. But realize that, logically, either collapse or its negation does have to be true. You can frame denying collapse as 'positing invisible extra worlds', but you can equally frame denying collapse as 'skepticism about positing invisible extra causal laws'.

Since every possible way the universe could be adds something 'extra' on top of what we observe -- either an extra law (e.g., collapse) or extra ontology (because there are no collapses occurring to periodically annihilate the ontology entailed by the Schrodinger equation) -- it's somewhat missing the point to attack any given interpretation for the crime of positing something extra. The more relevant question is just whether simplicity considerations or indirect evidence helps us decide which 'something extra' (a physical law, or more 'stuff', or both) is the right one. If not, then we stick with a relatively flat prior.

Claims 1 and 2 are testable, which is why we were able to acquire evidence for QM in the first place. Claim 5 is testable for pretty much any particular 'collapse' interpretation you have in mind; which means the negation of claim 5 is also testable. So all parts of bare-bones MWI are testable (though it may be impractical to run many of the tests), as long as we're comparing MWI to collapse and not to Bohmian Mechanics.

(You can, of course, object that affirming 3-5 as fundamental laws has the advantage of getting us empirical adequacy. But 'MWI (and therefore also 'bare' QM) isn't empirically adequate' is a completely different objection from 'MWI asserts too many unobserved things', and in fact the two arguments are in tension: it's precisely because Eliezer isn't willing to commit himself to a mechanism for the Born probabilities in the absence of definitive evidence that he's sticking to 'bare' MWI and leaving almost entirely open how these relate to the Born rule. In the one case you'd be criticizing MWI theorists for refusing to stick their neck out and make some guesses about which untested physical laws and ontologies are the real ones; in the other case you'd be criticizing MWI theorists for making guesses about which untested physical laws and ontologies are the real ones.)

I am not super interested in having catholic theologians read about minimum descriptive complexity, and then weaving a yarn about their favorite hypotheses based on that.

Are you kidding? I would love it if theologians stopped hand-waving about how their God is 'ineffably simple no really we promise' and started trying to construct arguments that God (and, more importantly, the package deal 'God + universe') is information-theoretically simple, e.g., by trying to write a simple program that outputs Biblical morality plus the laws of physics. At best, that sort of precision would make it much clearer where the reasoning errors are; at worst, it would be entertainingly novel.

Comment author: Eitan_Zohar 24 May 2015 07:01:47PM *  0 points [-]

Whoopsie daisy.

Comment author: ChristianKl 24 May 2015 07:00:33PM 0 points [-]

Schilling point

Do you mean Schelling point? If so, I don't see what you mean.

Comment author: Eitan_Zohar 24 May 2015 06:51:39PM *  0 points [-]

I'm not talking about the DA only, I'm talking about the assumption that our experiences should be more-or-less ordinary. And this is designed to escape the DA; it's the only reason to think you are simulated in the first place.

Really, I got the whole idea from HPMOR: fulfilling a scary prophecy on your own terms.

Comment author: Eitan_Zohar 24 May 2015 06:44:24PM *  0 points [-]

You, quite rightly, criticized the notion that an AI would have just as much of an epistemic advantage on humans that we would have on a cow. Any correct notions we have are just that- correct. It's not like there's more truth to them that a godlike intelligence could know.

Comment author: TheAncientGeek 24 May 2015 06:43:06PM *  0 points [-]

What's your model of how Einstein in fact arrived at truth, if not via a method that is "an example of how to arrive at truth

You can't work backwards from the fact that someone arrived at truth in one case to the the premise that they must have been working from a reliable method for arriving at truth. It's the "one case" that's the problem. They might have struck lucky.

Einstein's thought experiments inspired his formal theories, which were then confirmed by observation. Nobody thought the thought experiments provided confirmation by themselves.

Comment author: DanArmak 24 May 2015 06:37:56PM 0 points [-]

If you believe the DA, and you also believe you're being simulated (with some probability), then you should believe to be among the last N% humans in the simulation. So you don't escape the DA entirely.

However, it may be that if you believe yourself to be likely in a simulation, you shouldn't believe the DA at all. The DA assumes you know how many humans lived before you, and that you're not special among them. Both may be false in a simulation of human history: it may not have simulated all the humans and pre-humans who ever lived, and/or you may be in a special subset of humans being simulated with extra fidelity. Not to mention that only periods of your life may be simulated, possibly out of order or without causal structure.

Comment author: Mark_Friedenbach 24 May 2015 06:22:28PM 0 points [-]

Hrm. Maybe I'm reading you wrong? I thought you were making a commonly made argument that any belief we arrive at, the artificial intelligence would as well. And my response was that because the AI is running different mental machinery, it is entirely possible that there are beliefs we arrive at which the AI just doesn't consider and vice versa. Were you saying something different?

Comment author: Mark_Friedenbach 24 May 2015 06:17:16PM *  0 points [-]

Those are not different models. They are different interpretations of the utility of probability in different classes of applications.

though I'm not sure how you would find out the frequency at which hypotheses turn out to be true the way you figure out the frequency at which a coin comes up heads. But that could just be my not being as familiar thinking in terms of the Frequentist model

You do it exactly the same as in your Bayesian example.

I'm sorry, but this Bayesian vs Frequentist conflict is for the most part non-existent. If you use probability to model the outcome of an inherently random event, people have called that “frequentist.” If instead you model the event as deterministic, but your knowledge over the outcome as uncertain, then people have applied the label “bayesian.” It's the same probability, just used differently.

It's like how if you apply your knowledge of mechanics to bridge and road building, it's called civil engineering, but if you apply it to buildings it is architecture. It's still mechanical engineering either way, just applied differently.

One of the failings of the sequences is the amount of emphasis that is placed on “Frequentist” vs “Bayesian” interpretations. The conflict between the two exists mostly in Yudkowsky's mind. Actual statisticians use probability to model events and knowledge of events simultaneously.

Regarding the other points, every single example you gave involves using empirical data that had not sufficiently propagated, which is exactly the sort of use I am in favor of. So I don't know what it is that you disagree with.

Comment author: Dorikka 24 May 2015 06:15:18PM 0 points [-]

If I remember correctly, NancyLebovitz is the forum moderator; she might have the means and willingness to look into this kind of thing, and take action if needed.

Comment author: TheSurvivalMachine 24 May 2015 06:11:17PM 0 points [-]

Humans are animals affected by natural selection, wherefore no translation from animals to humans is necessary or even possible.

An individual is neither a predator nor a mugger by default. An individual is a predator or a mugger because of its traits and behaviour. Probably the mugger does not value the mugging itself. Humans who value the survival of their own behavioural genes would in all probability put into practice and enforce laws against mugging, since allowing mugging would risk adversely affecting not only each individual herself, but also other humans who to a large extent are carriers of the same behavioural genes as this individual. Please see my comment to gjm, where I mention “fitnessist contractarianism”, which, by the way, is universalizable.

Ethics per se does not have any function. Teaching ethics does. Discussing ethics does. Rational people do those things for a purpose. They do them as a means to an end. The end is given by ethics. Ethics gives you the purpose, but is not the purpose, or even a means to the purpose. But discussing ethics is a means to the purpose.

Comment author: Eitan_Zohar 24 May 2015 06:09:50PM *  0 points [-]

You may be right, but it is still more parsimonious than your idea (which requires some genuinely bizarre mechanism, far more than it being a self-delusion).

Comment author: TheSurvivalMachine 24 May 2015 06:07:52PM 0 points [-]

As a fitnessist I certainly do not “hold a position that affects only [my] own actions and [my] opinions of them”. I not only evaluate my own actions, but have opinions of the actions performed by other individuals as well. These opinions are based on how other individuals affect the survival of my behavioural genes. In that sense I pass moral judgement on others, like they pass moral judgement on me.

The fundamental question for any moral theory to answer is “Which actions should be performed?” and ethical fitnessism fully answers that question, although in an indexical fashion. The central question to answer is “Which actions should I myself perform?”, since that question is relevant to what I directly am in control of, namely my own body. The actions of other individuals I can, furthermore, only affect by my own actions, for example by what I say.

The principle is actually not “maximize long-term number of copies of genes that influence my behaviour”. The survival of my behavioural genes is directly linked to how long they survive and only indirectly linked to how many they are.

What I am “prepared to count as a ‘behavioural gene’” is not really the issue, rather the issue is what science counts as a behavioural gene. The Extended Phenotype [O.U.P., 1982] gives a good idea.

There is no problem with being related to pomegranates. I do believe that humans share behavioral genes with them, but that does not mean that focusing on the production of pomegranates would maximize the survival of my behavioural genes. Such a production would seem to be a short-sighted and narrow-minded behaviour and probably not the behaviour which natural selection tends to maximize.

Perhaps there today does not seem to be much difference between maximizing the survival of one’s behavioural genes and maximizing “long-term human happiness” or “long-term human numbers”, but over time the difference will add up and show itself. For example, the difference would be apparent when we create new entertainment technology so well adapted to our prehistoric minds and bodies that there is no way for hedonists to resist the urge of such endless happiness, or when we evolve beyond humans, when my behavioural genes are carried on into new species.

On a theoretical level ethical fitnessism has stronger arguments than have the moral theories of maximizing long-term human happiness or long-term human numbers. Fitnessism lacks neither hedonism, altruism, intuitiveness, nor consideration of future generations, and is complete, consistent, to the purpose and non-dependent on indoctrination. The application of any other moral theory through its behaviour is per definition evolutionarily self-defeating and undermines its own long-term existence.

Comment author: RobbBB 24 May 2015 06:07:07PM 0 points [-]

Again, the AI box experiment is a response to the claim "superintelligences are easy to box, because no level of competence at social engineering would suffice for letting an agent talk its way out of a box". If you have some other reason to think that superintelligences are hard to box -- one that depends on a relevant difference between the experiment and a realistic AI scenario -- then feel free to bring that idea up. But this constitutes a change of topic, not an objection to the experiment.

since we are not permitted to see the session transcripts for ourselves, we can't tell if it is a good experiment.

I mean, the experiment's been replicated multiple times. And you already know the reasons the transcripts were left private. I understand assigning a bit less weight to the evidence because you can't examine it in detail, but the hypothesis that there's a conspiracy to fake all of these experiments isn't likely.

Comment author: Eitan_Zohar 24 May 2015 06:04:40PM *  0 points [-]

Um... I don't follow this at all. Where do I anthropomorphize? I think that the concept of 'knowledge' would be pretty universal.

Comment author: Mark_Friedenbach 24 May 2015 05:56:12PM 0 points [-]

You're anthropomorphizing far too much. It's possible for things which are easy for us to think to be very difficult for an AGI, if it is constructed in a different way.

Comment author: Mark_Friedenbach 24 May 2015 05:53:15PM *  2 points [-]

He's saying "I don't understand how magic could have come into being, it must have been invented by somebody." When in fact there could be dozens of other alternative theories.

I'll give you one that took me only three seconds to think up: the method for using magic isn't a delusion of the caster as Harry thought, but a mass delusion of all wizards everywhere. E.g. confounding every wizard in existence, or at least some threshold to think that Fixus Everthingus was a real spell would make it work. Maybe all it would have take to get his experiments with Hermoine to work is to confound himself as well, making it a double-blind experiment as it really should have been.

His argument here really is exactly the same as an intelligent designer: "magic is too complicated and arbitrary to be the result of some physical process."

Comment author: Valentine 24 May 2015 05:52:18PM 1 point [-]

To be clear, I can understand where your impression came from. I don't blame you. I spoke up purely to crush a rumor and clarify the situation.

I'll take your word that none of it was done with CFAR funding, but that's not the expectation a reasonable person might have from your very own website. If you want to distance yourself from HPMOR you might want to correct that.

That's a good point. I'll definitely consider it.

We're not trying to distance ourselves from HPMOR, by the way. We think it's useful, and it does cause a lot of people to show interest in CFAR.

But I agree, as a nonprofit it might be a good idea for us to be clearer about whom we are and are not paying. I'll definitely think about how to approach that.

Comment author: Mark_Friedenbach 24 May 2015 05:42:28PM *  1 point [-]

The academic field which is most conspicuously missing is artificial intelligence. I agree with Jacob that it is and should be concerning that the machine intelligence research institute has adopted a technical agenda which is non-inclusive of machine intelligence researchers.

Comment author: Mark_Friedenbach 24 May 2015 05:28:57PM 0 points [-]

I'll think about how this can be phrased differently such that it might sway you. Given that you are not Valentine, is there a difference of opinion between his posts above and your views?

That part you pulled out and quoted is essentially what I was writing about in the OP. There is a philosophy-over-hard-subjects which is pursued here, in the sequences, at FHI, and is exemplified in the conclusions drawn by Bostrom in Superintelligence, and Yudkowsky in the later sequences. Sometimes it works, e.g. the argument in the sequences about the compatibility of determinism and free will works because it essentially shows how non-determinism and free will are incompatible--it exposes a cached thought that free-will == non-deterministic choice which was never grounded in the first place. But over new subjects where you are not confused in the first place -- e.g. the nature and risk of superintelligence -- people seem to be using thought experiments alone to reach ungrounded conclusions, and not following up with empirical studies.

That is dangerous. If you allow yourself to reason from thought experiments alone, I can get you to believe almost anything. I can't get you to believe the sky is green--unless you've never seen the sky--but anything you yourself don't have available experimental evidence for or against, I can sway you in either way. E.g. that consciousness is in information being computed and not the computational process itself. That an AI takeoff would be hard, not soft, and basically uncontrollable. That boxing techniques are foredoomed to failure irregardless of circumstances. That intelligence and values are orthogonal under all circumstances. That cryonics is an open-and-shut case. On these sorts of questions we need more, not less experimentation.

When you hear a clever thought experiment that seems to demonstrate the truth of something you previously thought to have low probability, then (1) check if your priors here are inconsistent with each other; then (2) check if there is empirical data here that you have not fully updated on. If neither of those approaches resolves the issue, then (3) notice you are confused, and seek an experimental result to resolve the confusion. If you are truly unable to find an experimental test you can perform now, then (4) operate as if you do not know which of the possible theories is true.

You do not say "that thought experiment seemed convincing, so until I know otherwise I'll update in favor of it." That is the sort of thinking which led the ancients to believe that "All things come to rest eventually, so the natural state is a lack of motion. Planets continue in clockwork motion, so they must be a separate magisteria from earthly objects." You may think we as rationalists are above that mistake, but history has shown otherwise. Hindsight bias makes the Greeks seem a lot stupider than they actually were.

Take a concrete example: the physical origin of consciousness. We can rule out the naïve my-atoms-constitute-my-consciousness view from biological arguments. However I have been unable to find or construct for myself an experiment which would definitively rule out either the information-identity or computational-process theories, both of which are supported by available empirical evidence.

How is this relevant? Some are arguing for brain preservation instead of cryonics. But this only achieves personal longevity if the information-identity theory is correct as it is destructive of the computational process. Cryonics on the other hand achieves personal longevity by preserving the computational substrate itself, which achieves both information- and computational-preservation. So unless there is a much larger difference in success likelihood than appears to be the case, my money (and my life) is on cryonics. Not because I think that computational-process theory is correct (although I do have other weak evidence that makes it more likely), but because I can't rule it out as a possibility so I must consider the case where destructive brain preservation gets popularized but at the cost of fewer cryopreservations, and it turns out that personal longevity is only achieved with the preservation of computational processes. So I do not support the Brain Preservation Foundation.

To be clear, I think that arguing for destructive brain preservation at this point in time is a morally unconscionable thing to do, even though (exactly because!) we don't know the nature of consciousness and personal identity, and there is an alternative which is likely to work no matter how that problem is resolved.

Comment author: Valentine 24 May 2015 05:28:14PM 2 points [-]

Perhaps you're using a Frequentist definition of "likelihood" whereas I'm using a Bayesian one?

There's a difference? Probability is probability.

There very much is a difference.

Probability is a mathematical construct. Specifically, it's a special kind of measure p on a measure space M such that p(M) = 1 and p obeys a set of axioms that we refer to as the axioms of probability (where an "event" from the Wikipedia page is to be taken as any measurable subset of M).

This is a bit like highlighting that Euclidean geometry is a mathematical construct based on following thus-and-such axioms for relating thus-and-such undefined terms. Of course, in normal ways of thinking we point at lines and dots and so on, pretend those are the things that the undefined terms refer to, and proceed to show pictures of what the axioms imply. Formally, mathematicians refer to this as building a model of an axiomatic system. (Another example of this is elliptic geometry, which is a type of non-Euclidean geometry, which you can model as doing geometry on a sphere.)

The Frequentist and Bayesian models of probability theory are relevantly different. They both think of M as the space of possible results (usually called the "sample space" but not always) and a measurable subset EM as an "event". But they use different models of p:

  • Frequentists suggest that were you to look at how often all of the events in M occur, the one we're looking at (i.e., E) would occur at a certain frequency, and that's how we should interpret p(E). E.g., if M is the set of results from flipping a fair coin and E is "heads", then it is a property of the setup that p(E) = 0.5. A different way of saying this is that Frequentists model p as describing a property of that which they are observing - i.e., that probability is a property of the world.
  • Bayesians, on the other hand, model p as describing their current state of confidence about the true state of the observed phenomenon. In other words, Bayesians model p as being a property of mental models, not of the world. So if M is again the results from flipping a fair coin and E is "heads", then to a Bayesian the statement p(E) = 0.5 is equivalent to saying "I equally expect getting a heads to not getting a heads from this coin flip." To a Bayesian, it doesn't make sense to ask what the "true" probability is that their subjective probability is estimating; the very question violates the model of p by trying to sneak in a Frequentist presumption.

Now let's suppose that M is a hypothesis space, including some sector for hypotheses that haven't yet been considered. When we say that a given hypothesis H is "likely", we're working within a partial model, but we haven't yet said what "likely" means. The formalism is easy: we require that HM is measurable, and the statement that "it's likely" means that p(H) is larger than most other measurable subsets of M (and often we mean something stronger, like p(H) > 0.5). But we haven't yet specified in our model what p(H) means. This is where the difference between Frequentism and Bayesianism matters. A Frequentist would say that the probability is a property of the hypothesis space, and noticing H doesn't change that. (I'm honestly not sure how a Frequentist thinks about iterating over a hypothesis space to suggest that H in fact would occur at a frequency of p(H) in the limit - maybe by considering the frequency in counterfactual worlds?) A Bayesian, by contrast, will say that p(H) is their current confidence that H is the right hypothesis.

What I'm suggesting, in essence, is that figuring out which hypothesis HM is worth testing is equivalent to moving from p to p' in the space of probability measures on M in a way that causes p'(H) > p(H). This is coming from using a Bayesian model of what p is.

Of course, if you're using a Frequentist model of p, then "most likely hypothesis" actually refers to a property of the hypothesis space - though I'm not sure how you would find out the frequency at which hypotheses turn out to be true the way you figure out the frequency at which a coin comes up heads. But that could just be my not being as familiar thinking in terms of the Frequentist model.

I'll briefly note that although I find the Bayesian model more coherent with my sense of how the world works on a day-by-day basis, I think the Frequentist model makes more sense when thinking about quantum physics. The type of randomness we find there isn't just about confidence, but is in fact a property of the quantum phenomena in question. In this case a well-calibrated Bayesian has to give a lot of probability mass to the hypothesis that there is a "true probability" in some quantum phenomena, which makes sense if we switch the model of p to be Frequentist.

But in short:

Yes, there's a difference.

And things like "probability" and "belief" and "evidence" mean different things depending on what model you use.

What I'm saying is that this shouldn't change your actual beliefs -- it will flush out some stale caching, or at best identify an inconsistent belief, including empirical data that you haven't fully updated on. But it does not, by itself, constitute evidence.

Yep, we disagree.

I think the disagreement is on two fronts. One is based on using different models of probability, which is basically not an interesting disagreement. (Arguing over which definition to use isn't going to make either of us smarter.) But I think the other is substantive. I'll focus on that.

In short, I think you underestimate the power of noticing implications of known facts. I think that if you look at a few common or well-known examples of incomplete deduction, it becomes pretty clear that figuring out how to finish thinking would be intensely powerful:

  • Many people make resolutions to exercise, be nicer, eat more vegetables, etc. And while making those resolutions, they often really think they mean it this time. And yet, there's often a voice of doubt in the back of the mind, as though saying "Come on. You know this won't work." But people still quite often spend a bunch of time and money trying to follow through on their new resolution - often failing for reasons that they kind of already knew would happen (and yet often feeling guilty for not sticking to their plan!).
  • Religious or ideological deconversion often comes from letting in facts that are already known. E.g., I used to believe that the results of parapsychological research suggested some really important things about how to survive after physical death. I knew all the pieces of info that finally changed my mind months before my mind actually changed. I had even done experiments to test my hypotheses and it still took months. I'm under the impression that this is normal.
  • Most people reading this already know that if they put a ton of work into emptying their email inbox, they'll feel good for a little while, and then it'll fill up again, complete with the sense of guilt for not keeping up with it. And yet, somehow, it always feels like the right thing to do to go on an inbox-emptying flurry, and then get around to addressing the root cause "later" or maybe try things that will fail after a month or two. This is an agonizingly predictable cycle. (Of course, this isn't how it goes for everyone, but it's common enough that well over half the people who attend CFAR workshops seem to relate to it.)
  • Most of Einstein's work in raising special relativity to consideration consisted of saying "Let's take the Michelson-Morley result at face value and see where it goes." Note that he is now considered the archetypal example of a brilliant person primarily for his ability to highlight worthy hypotheses via running with the implications of what is already known or supposed.
  • Ignaz Semmelweis found that hand-washing dramatically reduced mortality in important cases in hospitals. He was ignored, criticized, and committed to an insane asylum where guards beat him to death. At a cultural level, the fact that whether Semmelweis was right was (a) testable and (b) independent of opinion failed to propagate until after Louis Pasteur gave the medical community justification to believe that hand-washing could matter. This is a horrendous embarrassment, and thousands of people died unnecessarily because of a cultural inability to finish thinking. (Note that this also honors the need for empiricism - but the point here is that the ability to finish thinking was a prerequisite for empiricism mattering in this case.)

I could keep going. Hopefully you could too.

But my point is this:

Please note that there's a baby in that bathwater you're condemning as dirty.

Comment author: TheAncientGeek 24 May 2015 04:21:49PM *  1 point [-]

"Please dont unplug me, I am about to find a cure for cancer"I

f that happened I would eat my hat. Rush to push the big red STOP button, pull the yhard-cutoff electrical lever, break glass on the case containing firearms & explosives, and then sit down and eat my hat.

And then be reviled as the man who prevented a cure for cancer? Remember that the you in the story doesn't have the same information as the you outside the story -- he doesn't know that the AI isnt sincere.

"Please dont unplug me, I am about to find a cure for cancer" is a .placeholder for a class of exploits on the part of the AI where it holds a carrot in front of us. It's not going to literally come out with the cure for cancer thing under circumstances where it's not tasked with working on something like it it, because that would be dumb , and it's supposed to be superintelligent. But superintelligence is really difficult to predict....you have to imagine exploits, then imagine versions of them that are much better.

Maybe I'm off here, but common sense tells me that if you are worried about AI takeoffs, and if you are tasking an AI with obscure technical problems like designing construction processes for first generation nanomachines, large scale data mining in support of the SENS research objectives, or plain old long-term financial projections, you don't build in skills or knowledge that is not required.

The hypothetical MIRI is putting forward is that if you task an super AI with agentively solving the whole of human happiness, then it will have to have the kind of social, psychological and linguistic knowledge necessary to talk its way out of the box.

A more specialised AGI seems safer... and likelier ... but then another danger kicks in: it's creators might be too relaxed about boxing it, perhaps allowing it to internet access... but the internet contains a wealth of information to bootstrap linguistic and psychological knowledge with.

There's an important difference between rejecting MIRIs hypotheticals because the conclusions don't follow from the antecedents, as opposed to doing so because the antecedents are unlikely in the place.

This could happen accident.

Dangers arising from non AI scenario don't prove AI safety. My point was that an AI doesn't need efffectors to be dangerous... information plus sloppy oversight is enough. However the MIRI scenario seems to require a kind of perfect storm of fast takeoff , overambition, poor oversight, etc.

all while its programmers are overseeing its goal system looking for the patterns of deceptive goal states. It'd have to be not just deceptive, but meta-deceptive.

A superintelligence can be meta deceptive. Direct inspection of code is a terrible method of oversight, since even simple AIs can work in ways that baffle human programmers.

ETA on the whole, I object to the antecedents/priors ....I think the hypothetical go through,

Comment author: Dustin 24 May 2015 04:15:29PM 0 points [-]

Yes, that's exactly what I meant and what I would expect.

Comment author: gwern 24 May 2015 03:33:33PM 5 points [-]
Comment author: Jiro 24 May 2015 03:27:29PM 0 points [-]

The AI box experiment only serves even as that if you assume that the AI box experiment sufficiently replicates the conditions that would actually be faced by someone with an AI in a box. Also, it only serves as such if it is otherwise a good experiment, but since we are not permitted to see the session transcripts for ourselves, we can't tell if it is a good experiment.

View more: Next