Hmm. I remember being non-reflective in first grade but not in second grade. One consequence was that I couldn't re-write explicit beliefs in response to new information and I saw general injunctions and commands as relatively binding and automatic. Conflicting commands couldn't be accommodated, nor could common sense. I don't think that my emotions were any more intense. I never re-wrote myself, or noticed a change at the time, but I notice it in my memories. Early ones don't include the question "why am I doing this?" or "why do this...
Actually RU, that's a good approximation for many/most professions, but not all that good an approximation.
http://www.vanderbilt.edu/Peabody/SMPY/DoingPsychScience2006.pdf
gives more detail, showing a significant marginal impact from, at the least, 99.99th percentile math achievement at age 12 relative to merely 99.8th percentile math achievement at age 12.
Phil: Your estimate rewards precision and penalizes self estimate of precision. A person of a given level of precision should be rewarded for believing their precision to be what it is, not for believing it to be low. If you had self-estimate of precision in the numerator that would negate Nick's claim, but then you could drop the term from both sides.
Eliezer: I'm pretty sure that MANY very smart people learn more from working on hard problems and failing quite frequently than from reading textbooks and practicing easy problems. Both should be part of an intellectual diet.
Maksym: We actually do need someone to translate all this OB stuff very badly, though maybe it's desirable to wait for the book. Still, someone should be presenting it. As for convincing smart college students, there are three fairly separate barriers here, those to rationality, those of information and those to action. I recommend working on barriers to rationality and action first and in conjunction, belief second, and let people find the info themselves. Politics is the natural subject to frame as rationality. Simply turn every conversation where ...
You pretty much said it. Hypotheses suggested by mind-projection priors turning out to be true pretty much refutes Occam and consequentially science.
It's not on that level, that's the level which I respond to with the forbidden bet, e.g. p = 0, along with all the other stuff that implies strongly that our concepts of probability are simply broken.
Reason is a mistake for less extreme reasons such as "I'm dreaming" or "I'm a Boltzman Brain" or some forms of "my life is not merely a simulation but a psychological experiment".
I took psi seriously back when I thought that the scientific method defined rationality. Once I learned about Bayes I realized that the sort of reports of psi that science turns up would be expected if psi isn't real while much more blatant things would be expected if real psi inspired the investigation. I also noticed that priors matter and psi really should be ignored without very large effects based on low priors. Somewhat earlier pre-Bayes psi had blended somewhat into the category "Everything you know is wrong" and loose specific identity as 'psi'. Post-Bayes the "Everything you know is wrong" itself split into a few categories and psi went in the "reason is a mistake" extreme category.
Thinking about the future as today + diff is another serious problem with similar roots.
Robin: Great Point! Eliezer: I'm awaiting that too.
Shane Legg: I don't generally like sf, film or otherwise, but try "Primer". Best movie ever made for <$6000 AND arguably best sf movie. The Truman Show was good too in the last decade or so. That's probably it though. Minority Report was OK.
Dennis Bider: A BASIC and ESSENTIAL though these days largely forgotten principle of liberal society is that it can be the case, and often is, that behavior X is NOT OK but that banning behavior X would also be NOT OK.
Pete, if you do that then being a casual decision theorist won't, you know, actually Win in the one shot case. Note that evolution doesn't produce organisms that cooperate in one shot prisoners dilemmas.
But Eliezer, you can't assume that Clippy uses the same decision making process that you do unless you know that you both unfold from the same program with different utility functions or something. If you have the code that unfolds into Clippy and Clippy has the code that unfolds into you it may be that you can look at Clippy's code and see that Clippy defects if his model of you defects regardless of what he does and cooperates if his model of you cooperates if your model of him cooperates, but you don't have his code. You can't say much about all possible minds or about all possible paperclip maximizing minds.
I would say that in my experience evangelicals and traditional Christians max out at in the low 140s IQ wise.
Nick: Do you imagine that they would tell you so? Also, you are a) young, and b) haven't been in any setting where people come from a large variety of social backgrounds.
Highly intelligent Christians, Dyson for instance, are likely to believe roughly the same things you do but frame them differently. Tegmark 4 = Spinoza's god, for instance.
Hopefully: You and may others. I will if I ever pull together the emotional resources to, which seem unusually high for me. It's very demanding of effort for me to address a large group, most of whom will fail to "get it" whatever I do.
Seconding Hopefully Anonymous on this point. Also emphasizing that not infrequently, when the accuracy of beliefs that a nerd can promote is low due to inferential distance or to gaps in intelligence, nerds tend to give true statements without bridging the inferential distance, predictably promoting less than maximally accurate beliefs, primarily, I would say, motivated by the self-righteous feeling they get from telling the truth and the feeling of superior righteous indignation they get from their perceived inferiors showing ignorance by not understandi...
Seconding Scott Aaronson (especially on the tragedy of the US getting nukes too late) and Michael G R, though with the suggestion that the "demo" bomb could have been Trinity.
Eliezer: I think that the hypocrisy of the US is mostly in our maintaining a large arsenal but telling others that they can't have any, not in having used nukes. I don't see a case for .1% probability increase for nuclear war without using the bomb which is stronger than the case for the inverse, but I do see a million dead Japanese. Also, in terms of catastrophic risks, ...
Eliezer: I think that you misunderstand Roko, but that doesn't really matter, as he seems to understand you fairly well right now and to be learning effectively.
Unknown: Not at all. Utility maximization is very likely to lead to counterintuitive actions, and might even lead to humanly useless ones, but the particular actions it leads to are NOT whatever salient actions you wish to justify but are rather some very specific set of actions that have to be discovered. Seriously, you NEED to stop reasoning with rough verbal approximations of the math and act...
Great comments thread! Thanks all!
Seconding Roko, Carl, HA, Nick T, etc.
Eliezer or Robin: Can you cite evidence for "we can more persuasively argue, for what we honestly believe". My impression is that it has been widely assumed in evolutionary psychology and fairly soundly refuted in the general psychology of deception, which tells us that the large majority of people detect lies at about chance and that similar effort seems to enable the development of the fairly rare skill of the detection of lies and evasion of such detection.
Carl: Unknow...
I second Hopefully on criticism of the strawman postmodernist. Honestly, I think that academic disciplines, or even schools, where everyone is completely full of it are extremely rare. There are thoughtful, intelligent, and honest people who frame important and fairly novel ideas in the terminology of sociology, academic feminism, Freudianism, behaviorism, even, math-help-us, Jung. Perfect intellectual honest and intelligence among humans are a chimera, and different disciplines aspire to different approximations thereof by establishing different sorts ...
BTW, I have significant personal experience with one of "the smartest guys in the room" and yes, they were (or at least he is) VERY smart by any normal business world standards. He's particularly great at giving obvious in retrospect answers to marketing type problems. It would be a somewhat unusual room full of business people or other social elites where the guy I'm thinking of isn't the smartest guy.
Caledonian: I can't think of anyone EVER choosing to interpret statements as stupid rather than sensible to the degree to which you do on this blog. There is usually NO ambiguity and you still get things wrong and then blame them for being stupid.
In all honesty why do you post here? On your own blog you are articulate and intelligent. Why not stick with that and leave commenting to people who want to actually respond to what people say rather than to straw men?
Laura: Should I have explained it in terms of your being Jewish? Geometry is more about logic and symbols, what the WAIT calls "Verbal" intelligence, than it is about visuospatial intelligence (which may be related to why its on the SAT). Does anyone think that its significantly less common for women to be competent sketch artists than for men to be? That honestly never occurred to me. But hey, you are a med student; how good are you at identifying expected drug-ligand interactions based on molecular shapes compared to similarly trained men ...
Z.M. Davis: These days I generally try not to identify as a Transhumanist, Humanist, or even atheist except to people who I have reason to believe will react positively. I don't have the character to identify as Christian to those who will react positively to that, but maybe someday. Honestly, well, on Facebook I'm "Political Views", "Other", "Religion" "Philosopher and Bayesian". As the latter, any group identity that supposedly rests on belief is out, as I explicitly represent, when unpacked precisely, all my b...
Grant: I would second everything in your post except the last paragraph.
Angel: I sincerely do apologize for making you feel bad, and I certainly won't use your name in the future, as hurting people is bad, hurting people in ways that make them likely to behave worse is worse, and my not trusting the person I harm or not considering them to have good intentions doesn't make it better.
That said, NO, it is NOT wrongwrongwrong to tell people that its a bad idea to adopt such and such a label because the people who typically use that label are in some respe...
Angel: I really don't think that in the modern world we need a label for "someone who believes in equal rights for men and women" any more than we need a label for "someone who thinks that we shouldn't have slavery" outside of the Muslim world. Abolitionist was once a mark of distinction, now it isn't. A more useful definition but extremely generous definition of feminism, one that would make me very tentatively feminist, would be something like "someone who believes that many/most cultural institutions and basic assumptions nee...
All: I'm really disappointed that no-one else seems to have found my "after the FAI does nothing" frame useful for making sense of this post. Is anyone interested in responding to that version? It seems so much more interesting and complete than the three versions E.C. Hopkins gave.
Dynamically: My "moral philosophy" if you insist on using that term (model of a recipe for generating a utility function considered desirable by certain optimizers in my brain would be a better term) is the main thing that HAS told me to steal, cheat, an...
The way I frame this question is "what if I executed my personal volition extrapolating FAI, it ran, created a pretty light show, and then did nothing, and I checked over the code many times with many people who also knew the theory and we all agreed that it should have worked, then tried again with completely different code many (maybe 100 or 1000 or millions) times, sometimes extrapolating somewhat different volitions with somewhat different dynamics and each time it produced the same pretty light show and then did nothing. Lets say I have spend a ...
Laura: It seems to me that when you apply the feminist label you might want to think about the risk of being understood to mean "like Angel". If that prospect doesn't appall you you may want to read her posts more carefully. It's not just normal academic dishonesty. Rather, its a very distinctive establishment of moral asymmetries mixed with postmodernism that characterizes major strains of modern (second and third wave?) feminism. Maybe you want the term "first wave feminist" but that's too pretentious.
Bambi: The particular reason for blogging rather than rum is that the math says he blogs here and now. The future isn't immune to our actions, it is what it is as the result of our actions, which likewise are what they are. We cause it to be in the same manner that the earlier states of a Turing Machine cause the later states to be.
Eliezer: I really like this post, but it seems to me that empirically it was substantially a cultural practice in philosophy, including Kant etc, that enabled those early 20th century Germans (and only those people, in that particular culture, with that particular philosophical tradition) to seem, vaguely a significant subset of those assumptions that they did know existed but that other philosophers and lay people didn't know existed. That philosophy also lead them down some wrong roads, such as towards thinking mind was fundamental rather than emergent...
Poke: Are you sure about mineralogy and physics as foundations of modern geology?
Patrick:
I agree that something roughly along the lines of what you are discussing can be done and is unavoidable. I am primarily attempting to refute the proposal that it is or can be corrected to become Bayesian, and hence the proposal that the process that we use to do things like this stands with the same sort of logical foundations as Bayesian reasoning does. It definitely seems to me that strictly speaking, once you remove logical omniscience, unless you replace it wit...
I second tabooing probability, but I think that we need more than two words to replace it. Casually, I think that we need, at the least, 'quantum measure', 'calibrated confidence', and 'justified confidence'. Typically we have been in the habit of calling both "Bayesian", but they are very different. Actual humans can try to be better approximations of Bayesians, but we can't be very close. Since we can't be Bayesian, due to our lack of logical omniscience, we can't avoid making stupid bets and being Dutch Booked by smarter minds. It's there...
To make the simulation really compelling it has to include some sort of assortative mating.
No Mike, your intuition for really large numbers is non-baffling, probably typical, but clearly wrong, as judged by another non-Utilitarian consequentialist (this item is clear even to egoists).
Personally I'd take the torture over the dust specks even if the number was just an ordinary incomprehensible number like say the number of biological humans who could live in artificial environments that could be built in one galaxy. (about 10^46th given a 100 year life span and a 300W (of terminal entropy dump into a 3K background from 300K, that's a large budge...
I think that Robin's point solves this problem, but doesn't solve the more general problem of an AGI's reaction to low probability high utility possibilities and the attendant problems of non-convergence.
The guy with the button could threaten to make an extra-planar factory farm containing 3^^^^^3 pigs instead of killing 3^^^^3 humans. If utilities are additive, that would be worse.
Robin: Great point about states with many people having low correlations with what one random person can effect. This is fairly trivially provable.
Utilitarian: Equal priors due to complexity, equal posteriors due to lack of entanglement between claims and facts.
Wei Dai, Eliezer, Stephen, g: This is a great thread, but it's getting very long, so it seems likely to be lost to posterity in practice. Why don't the three of you read the paper Neel Krishnaswami referenced, have a chat, and post it on the blog, possibly edited, as a main post?
"The p...
Rob: I think that some psychologists might say something like the following. Confirmation bias causes new evidence to accumulate in favor of existing beliefs. Subsequent to the accumulation of such evidence, the refutation of the original evidence for a belief will not eliminate the belief. When the beliefs in question are normative evaluations, this is called the "halo effect".
Initial evidence that something, for instance ethical behavior, is good because it leads to Santa giving presents could lead to a bias in favor of noticing other evide...
TGGP: I don't think that what you are describing would be considered by most people to constitute a disbelief in morality. Instead, I think it would be considered an atypically reflective and possibly slightly atypically targeted belief about what being means.
Rob Spear: "good old fashioned Pavlovian conditioning" requires that rewards and punishments (reinforcers) be associated very closely in time with the behaviors being reinforced. Santa doesn't do this. Neither does god. This is NOT a minor quibble, but rather is a critically important ...
Rob: In practice there is a HUGE difference. If you behave morally when no0one is watching you, new information doesn't threaten your moral foundations, as your morality is grounded in your preferences. If you believe that you are always being watched then your moral behavior will be grounded in supposed facts about the world. In this case, evidence that undermines your belief in those facts undermines your morality, leading in the direction of Nietzsche's "total eclipse of all values" as the inevitable consequence of the "twilight of th...
I think that the social sciences seem to still be following the Greek paradigm. Exceptions are excused and generally ignored rather than studied in more depth. New theories are rarely asked to explain the findings that supported old theories. Outliers are dropped, partially to make ignoring exceptions easier.
I would say that I referred to ontological types with an informational flavor rather than a "materialist" flavor, but since the process doing the referring is an informational process, what other possibility could have been addressed? Whereof one cannot speak, thereof one must be silent.
The point is not that a qualitatively more intelligent being could not design a universe to cancel a qualitatively less intelligent being's technology, but rather that an unintelligently (randomly subject to anthropic constraints, for instance) selected set of naturalistic laws could not plausibly generate such effects. You could always have a hands-on god individually deciding what happens in each situation (monadology?), but that's just a kid playing with dolls, not a universe. Also, "things" in modern physics are defined in terms of their re...
Funny, I always read "A Fire Upon The Deep" as a commentary upon this very point. It seemed to me that Vinge was rubbing people's faces in the fact that our intuitions about technology, where each discovery has an associated "importance" lisp token, doesn't correspond to the way the world works. The Zones correspond to something that exists in the minds of people thinking about the future, but doesn't correspond to anything that could possibly turn out to exist in a consilient monistic world (as opposed to the Platonistic internal wor...
Well, at the very least women constitute half of society, it's certainly acceptance within that half. I actually think that it's actually acceptance more broadly though. Women are arguably not accepted my men in general, but in so far as they are accepted it is only in a few narrow domains, primarily science, engineering, and big business that women do best by adhering to men's norms. Actually though, emotional suppression is only normative among men in science, in the military, and in low status positions. Enthusiasm (irrational exuberance) is the ulti...
It seems to me that social consensus accepts expression of strong feelings by women, just not by men.
It might be worth considering whether metaethicists, theologians, and astrologers from radically different backgrounds tended to converge upon some core set of conclusions substantially different from those reached by non-metaethicists. Metaethical, astrological or theological equivalents of Darwin and Wallace or of Newton and Leibniz would be significant confirmation of their disciplinary soundness.
My causal account of moral directionality is that cognitive resources are scarce and that people try to conserve them. One class of cognitive resource people try to conserve is complexity of model, driving a tendency towards parsimonious explanations. In other words, ethical consistency is a sub-set of consistency in general, which is a luxury good. As non-cognitive wealth increases relative to cognitive wealth people purchase more parsimony and noise in early highly random but constrained ethical systems gets locked onto strong attractor dynamics. Isl...
Eli; what did you think as a young child in a religious environment?
It seems clear that there is a systematic direction in most or all cultures towards application and generalization of moral/ethical vocalizations as wealth increases. There is a less clear trend towards broadening circles of moral consideration, but this may be an instance of the first trend.
Gordon, I recommend the Satiricon of Petronius for some fantastic confirmation of your model of gambling applied to life success in general. It's fairly difficult for Americans, raised on meritocracy, and perhaps before than on the assumption of wise and just gods, to relate to the strong desire, expressed by many characters, for an unjust and capricious world. I suppose that if people imagine "lucky", "blessed", or "well fated" to be personal traits they can then easily believe that they are above average on those traits. The last, in particular, is not easily disconfirmed.
I agree that deriving morality from stated human values is MUCH more ethically questionable than deriving it from human values, stated or not, and suggest that it is also more likely to converge. This creates a probable difficulty for CEV.
It seems to me that if it's worth destroying Huygens to stop the Superhappies it's plausibly worth destroying Earth instead to fragment humanity so that some branch experiences an infinite future so long as fragmentation frequency exceeds first contact frequency. Without mankind fragmented, the normal ending seems ine... (read more)