(To the extent that I'm negative utilitarian, I'm a hedonistic negative utilitarian, so I can't speak for the preference NUs, but...)

So what happens when you create someone who is going to die, and has an unbounded utility function?

Note that every utilitarian system breaks once you introduce even the possibility of infinities. E.g. a hedonistic total utilitarian will similarly run into the problem that, if you assume that a child has the potential to live for an infinite amount of time, then the child can be expected to experience both an infinite amount of pleasure and an infinite amount of suffering. Infinity minus infinity is undefined, so hedonistic total utilitarianism would be incapable of assigning a value to the act of having a child. Now saving lives is in this sense equivalent to having a child, so the value every action that has even a remote chance of saving someone's life becomes undefined as well...

A bounded utility function does help matters, but then everything depends on how exactly it's bounded, and why one has chosen those particular parameters.

The ones I've encountered online make an effort to avoid having children, but they don't devote every waking minute of their lives to it.

I take it you mean to say that they don't spend all of their waking hours convincing other people not to have children, since it doesn't take that much effort to avoid having children yourself. One possible answer is that loudly advocating "you shouldn't have children, it's literally infinitely bad" is a horrible PR strategy that will just get your movement discredited, and e.g. talking about NU in the abstract and letting people piece the full implications themselves may be more effective.

Also, are they all transhumanists? For the typical person (or possibly even typical philosopher), infinite lifespans being a plausible possibility might not even occur as something that needs to be taken into account.

How much suffering/preference frustration would an antinatalist be willing to inflict on existing people in order to prevent a birth? How much suffering/preference frustration would a birth have to stop in order for it to be justified? For simplicity's sake, let's assume the child who is born has a normal middle class life in a 1st world country with no exceptional bodily or mental health problems.

Does any utilitarian system have a good answer to questions like these? If you ask a total utilitarian something like "how much morning rush-hour frustration would you be willing to inflict to people in order to prevent an hour of intense torture, and how exactly did you go about calculating the answer to that question", you're probably not going to get a very satisfying answer, either.

Speaking personally, I don't negatively weigh non-aversive sensory experiences. That is to say, the billions of years of unsatisfied preferences are only important for that small subset of humans for whom knowing about the losses causes suffering. Death is bad and causes negative experiences. I want to solve death before we have more kids, but I recognize this isn't realistic. It's worth pointing out that negative utilitarianism is incoherent. Prioritarianism makes slightly more sense.

If I have a crisis of faith, how will I know?

A sense of panic and dread, and a feeling of being lost were some highlights for me. I think it would be hard to not know, though perhaps others experience these things differently.

I think there are many ways how beliefs get changed.

Take a belief such as: "The world is an hostile place and therefore I have to hide myself behind behind a shield of anonymity when I post online."

Ten years ago I would have feared that somebody associates my online writing with my real identity at that time I thought I needed the shield. Today I don't (Nickname is firstname + first 2 letters of lastname).

How did that process felt like? At the beginning I felt fear and now I don't but it was a gradual process over time.

For most practical concerns I think that we use religion way to often as reference concept. Children get usually taught that it's bad to talk to strangers. In our world it's a useful skill to talk to strangers in an friendly and inviting way.

Most people hit walls very quickly if the try to start to say to hello with a smile to every stranger they pass on the street. The come up with excuses that saying so is weird and that people will hate them if the find out that they will engage in such weird behavior.

If you want to experience a crisis of faith those social beliefs are were I would focus. There are more interesting because they actually have empirical results that you can see and you can't just pretend that you have changed your belief.

rationalists

I think you mean "humans"?

With respect to adoption vs. biological children, having your own child allows you more control over the circumstances and also means the child will probably share a some facets of your / your mate's personality, in ways that are often surprising and pleasurable.

With respect to raising children in general, it's intrinsically rewarding, like a mix of writing a book and being in love. Also, if you're assuming the environment won't radically change, having youth probably makes aging easier.

(I don't have children, but have watched them being raised. Unsure of my own plans.)

I don't consider myself an explicit rationalist, but the desire to have children stems from the desire to have someone to take care of me when I am older.

Do you see your own conception and further life as a cause for "huge heap of disutility" that can't be surpassed by the good stuff?

What motivates rationalists to have children?

The same what motivates other people. Being rational doesn't necessarily change your values.

Clearly, some people think having children is worthwhile and others don't, so that's individual. There is certainly an inner drive, more pronounced in women, because species without such a drive don't make it though natural selection.

The amount of decision-making also obviously varies -- from multi-year deliberations to "Dear, I'm pregnant!" :-)

I wouldn't dream of speaking for rationalists generally, but in order to provide a data point I'll answer for myself. I have one child; my wife and I were ~35 years old when we decided to have one. I am by any reasonable definition a rationalist; my wife is intelligent and quite rational but not in any very strong sense a rationalist. Introspection is unreliable but is all I have. I think my motivations were something like the following.

1. Having children as a terminal value, presumably programmed in by Azathoth and the culture I'm immersed in. This shows up subjectively as a few different things: liking the idea of a dependent small person to love, wanting one's family line to continue, etc.

2. Having children as a terminal value for other people I care about (notably spouse and parents).

3. I think I think it's best for the fertility rate to be close to the replacement rate (i.e., about 2 in a prosperous modern society with low infant mortality), and I think I've got pretty good genes; overall fertility rate in the country I'm in is a little below replacement and while it's fairly densely populated I don't think it's pathologically so, so for me to have at least one child and probably two is probably beneficial for society overall.

4. I expected any child I might have to have a net-positive-utility life (for themselves, not only for society at large) and indeed probably an above-average-utility life.

5. I expected having a child to be a net positive thing for marital harmony and happiness (I wouldn't expect that for every couple and am not making any grand general claim here).

I don't recall thinking much about the benefits of children in providing care when I'm old and decrepit, though I suppose there probably is some such benefit.

So far (~7.5 years in), we love our daughter to bits and so do others in our family (so #1,#2,#5 seem to be working as planned), she seems mostly very happy (so #4 seems OK so far), it's obviously early days but my prediction is still that she'll likely have a happy life overall (so #4 looks promising for the future) and I don't know what evidence I could reasonably expect for or against #3.

One more point that I haven't seen brought up -- listen to Queen:

Can anybody find me somebody to love?
Each morning I get up I die a little
Can barely stand on my feet
Take a look in the mirror and cry
Lord what you're doing to me
I have spent all my years in believing you
But I just can't get no relief,
Lord!
Somebody, somebody
Can anybody find me somebody to love?

Disclaimer: I don't have kids, won't have them anytime soon (i.e. not in the next 5 years), and until relatively recently didn't want them at all.

The best comparison I can make is that raising a child is like making a painting. It's work, but it's rewarding if done well. You create a human being, and hopefully impart them with good values and set them on a path to a happy life, and it's a very personal experience.

Personally, I don't have any drive to have kids, not one that's comparable to hunger or sexual attraction.

Upvoted. Remember to keep in mind the answer might be "making a person is as good as killing a person is bad.

Here's a simple argument for why we can't be indifferent to creating people. Suppose we have three worlds:

• Jon is alive and has 10 utils
• Jon was never conceived
• 1Jon is alive and has 20 utils

Assume we prefer Jon to have 20 utils to 10. Assume also we're indifferent between 10 utils and Jon's. Hence by transitivity we must prefer Jon exist and have 20 utils to Jon's non-existance. So we should try to create Jon, if we think he'll have over 10 utils.

Making a person and unmaking a person seem like utilitarian inverses

No because time is directional.

Making a person and unmaking a person seem like utilitarian inverses

Doesn't seem that way at all to me. A person who already exists has friends, family, social commitments, etc. Killing that person would usually effect all of these things negatively, often to a pretty huge extent. Using contraception maybe creates some amount of disutility in certain cases (for staunch Catholics, for instance) but not nearly to the degree that killing someone does. If you're only focusing on the utility for the person made or unmade, then maybe (although see blacktrance's comment on that), but as a utilitarian you have no license for doing that.

Even within pleasure- or QALY-utilitarianism, which seems technically wrong, you can avoid this by recognizing that those possible people probably exist regardless in some timeline or other. I think. We don't understand this very well. But it looks like you want lots of people to follow the rule of making their timelines good places to live (for those who've already entered the timeline). Which does appear to save utilitarianism's use as a rule of thumb.

Why isn't making a person as good as killing a person is bad

Possibly because...

I don't think contraception is tantamount to murder.

You have judged. It's possible that this is all there is to it... not killing people who do not want to die might just be a terminal value for humans, while creating people who would want to be created might not be a terminal value.

(Might. If you think that it's an instrumental value in favor of some other terminal goal, you should look for it)

Cheap answer, but remember that it might be the true one: because utilitarianism doesn't accurately describe morality, and the right way to live is not by utilitarianism.

Whether this is so or not depends on whether you are assuming hedonistic or preference utilitarianism. For a hedonistic utilitarian, contraception is, in a sense, tantamount to murder, except that as a matter of fact murder causes much more suffering than contraception does, both to the person who dies, to his or her loved ones, and to society at large (by increasing fear). By contrast, preference utilitarians can also appeal to the preferences of the individual who is killed: whereas murder causes the frustration of an existing preference, contraception doesn't, since nonexisting entities can't have preferences.

The question also turns on issues about population ethics. The previous paragraph assumes the "total view": that people who do not exist but could or will exist matter morally, and just as much. But some people reject this view. For these people, even hedonistic utilitarians can condemn murder more harshly than contraception, wholly apart from the indirect effects of murder on individuals and society. The pleasure not experienced by the person who fails to be conceived doesn't count, or counts less than the pleasure that the victim of murder is deprived of, since the latter exists but the former doesn't.

For further discussion, see Peter Singer's Practical Ethics, chap. 4 ('What's wrong with killing?").

I created a new article about this.

As far as I can tell killing/not-killing a person isn't the same not-making/make a person. I think this becomes more apparent if you consider the universe as timeless.

This is the thought experiment that comes to mind. It's worth noting that all that follows depends heavily on how one calculates things.

Comparing the universes where we choose to make Jon to the one where we choose not to:

• Universe A: Jon made; Jon lives a fulfilling life with global net utility of 2u.
• Universe A': Jon not-made; Jon doesn't exist in this universe so the amount of utility he has is undefined.

Comparing the universes where we choose to kill an already made Jon to the one where we choose not to:

• Universe B: Jon not killed; Jon lives a fulfilling life with global net utility of 2u.
• Universe B': Jon killed; Jon's life is cut short, his life has a global net utility of u.

The marginal utility for Jon in Universe B vs B' is easy to calculate, (2u - u) gives a total marginal utility (i.e. gain in utility) from choosing to not kill Jon over killing him of u.

However the marginal utility for Jon in Universe A vs A' is undefined (in the same sense 1/0 is undefined). As Jon doesn't exist in universe A' it is impossible to assign a value to Utility_Jon_A', as a result our marginal (Utility_Jon_A - Utility_Jon_A') is equal to (u - [an undefined value]). As such our marginal utility lost or gained by choosing between universes A and A' is undefined.

It follows from this that the marginal utility between any universe and A' is undefined. In other words our rules for deciding which universe is better for Jon break down in this case.

I myself (probably) don't have a preference for creating universes where I exist over ones where I don't. However I'm sure that I don't want this current existence of me to terminate.

So personally I choose maximise the utility of people who already exist over creating more people.

Eliezer explains here why bringing people into existence isn't all that great even if someone existing over not existing has a defined(and positive) marginal utility.

Here are two related differences between a child is and an adult. (1) It is very expensive to turn a child into an adult. (2) An adult is highly specific and not replaceable, while a fetus has a lot of subjective uncertainty and is fairly easily duplicated within that uncertainty. Uploading is relevant to both of these points.

From a classical utilitarian perspective, yeah, it's pretty much a wash, at least relative to non-fatal crimes that cause similar suffering.

However, around here, "utilitarian" is usually meant as "consistent consequentialism." In that frame we can appeal to motives like "I don't want to live in a society with lots of murder, so it's extra bad."

Because killing a person deprives them of positive experiences that they otherwise would have had, and they prefer to have them. But a nonexistent being doesn't have preferences.

By their degree of similarity to ancient religious mythological and sympathetic magic forms with the nouns swapped out.

We might find out by trying to apply them to the real world and seeing that they don't work.

Well, it is less common now, but I think a slow retreat of the community from the position that instrumental rationality is applied science of winning at life is one of the cases when the beliefs had to be corrected to better match evidence.

If it turns out that the whole MIRI/LessWrong memeplex is massively confused, what would that look like?

A few that come to mind:

• Some religious framework being basically correct. Humans having souls, an afterlife, etc.
• Antinatalism as the correct moral framework.
• Romantic ideas of the ancestral environment are correct and what feels like progress is actually things getting worse.
• The danger of existential risk peaked with the cold war and further technological advances will only hasten the decline.

I think the whole MIRI/LessWrong memeplex is not massively confused.

But conditional on it turning out to be very very wrong, here is my answer:

A. MIRI

1. The future does indeed take radical new directions, but these directions are nothing remotely like the hard-takeoff de-novo-AI intelligence explosion which MIRI now treats as the max-prob scenario. Any sci-fi fan can imagine lots of weird futures, and maybe some other one will actually emerge.

2. MIRI's AI work turns out to trigger a massive negative outcome -- either the UFAI explosion they are trying to avoid, or something else almost as bad. This may result from fundamental mistakes in understanding, or because of some minor bug.

3. It turns out that the UFAI explosion really is the risk, but that MIRI's AI work is just the wrong direction; e.g., it turns out that that building a community of AIs in rough power balance; or experimenting by trial-and-error with nascent AGIs is the right solution.

B. CfAR

1. It turns out that the whole CfAR methodology is far inferior to instrumental outcomes than, say, Mormonism. Of course, CfAR would say that if another approach is instrumentally better, they would adopt it. But if they only find this out years down the road, this could be a massive failure scenario.

2. It turns out that epistemologically non-rational techniques are instrumentally valuable. Cf. Mormonism. Again, CfAR knows this, but in this failure scenario, they fail to reconcile the differences between the two types of rationality they are trying for.

Again, I think that the above scenarios are not likely, but they're my best guess at what "massively wrong" would look like.

Define "massively wrong". My personal opinions (stated w/o motivation for brevity):

• Building AGI from scratch is likely to be unfeasible (although we don't know nearly enough to discard the risk altogether)
• Mind uploading is feasible (and morally desirable) but will trigger intelligence growth of marginal speed rather than a "foom"
• "Correct" morality is low Kolmogorov complexity and conforms with radical forms of transhumanism

Infeasibility of "classical" AGI and feasibility of mind uploading should be scientifically provable.

So: My position is very different from MIRI's. Nevertheless I think LessWrong is very interesting and useful (in particular I'm all for promoting rationality) and MIRI is doing very interesting and useful research. Does it count as "massively wrong"?

It would look like a failure to adequately discount for inferential chain length.

Well, why do you think socialism is so horribly wrong? During the 20th century socialists more or less won and got what they wanted. Things like social security, govermental control over business and redistribution of wealth in general are all socialist. This all may be bad from some point of view, but it is in no way mainstream opinion.

Then, those guys whom you mention in your article called themselves communists and marxists. At most, they considered socialism as some intermediate stage for building communism. And communism went bad because it was founded on wrong assumptions about how both economy and human psychology work. So, which MIRI/Lesswrong assumptions can be wrong and cause a lot of harm? Well, here are some examples.

1) Building FAI is possible, and there is a reliable way to tell if it is truly FAI before launching it. Result if wrong: paperclips.

2) Building FAI is much more difficult than AI. Launching a random AI is civilization-level suicide. Result if this idea becomes widespread: we don't launch any AI before civilization runs out of resources or collapses for some other reason.

3) Consciousness is sort of optional feature, intelligence can work just well without it. We can reliably say if given intelligence is a person. In other words, real world works the same way as in Peter Watts "Blindsight". Results if wrong: many, among them classic sci-fi AI rebellion.

4) Subscribing for cryonics is generally a good idea. Result if widespread: these costs significantly contribute to worldwide economic collapse.

It could be that it's just impossible to build a safe FAI under the utilitarian framework and all AGI's are UFAIs.

Otherwise the LessWrong memeplex has the advantage of being very diverse. When it comes to a subject like politics we do have people with mainstream views but we also have people who think that democracy is wrong. Having such a diversity of ideas makes it difficult for all of LessWrong to be wrong.

Some people paint a picture of LessWrong as a crowd of people who believe that everyone should do cryonics. In reality most the the participants aren't signed up for cryonics.

Take a figure like Nassim Taleb. He's frequently quoted on LessWrong so he's not really outside the LessWrong memeplex. But he's also a Christian.

There are a lot memes around flooting in the LessWrong memeplex that are there in a basic level but that most people don't take to their full conclusion.

So, how might we find that all these ideas are massively wrong?

It's a topic that's very difficult to talk about. Basically you try out different ideas and look at the effects of those ideas in the real world. Mainly because of QS data I delved into the system of Somato-Psychoeducation. The data I measured was improvement in a health variable. It was enough to get over the initial barrier to go inside the system. But know I can think inside the system and there a lot going on which I can't put into good metrics.

There however no way to explain the framework in an article. Most people who read the introductory book don't get the point before they spent years experiencing the system from the inside.

It's the very nature of those really things outside the memeplex that there not easily expressible by ideas inside the memeplex in a way that won't be misunderstood.

We should be wary of ideologies that involve one massive failure point....crap.

Whenever they invent time travel, they discover that the ability to change the past becomes the biggest cause of suffering, so in the end they always un-invent it.

And, similarly, should I be depressed that there currently exists NO alien species with the inclination+ability to eliminate horrific suffering in all sentient life-forms?

The circular preferences that go against the axioms of utility theory, and which are Dutch book exploitable, are not of the kind "I prefer A to B at time t1 and B to A at time t2", like the ones of your example. They are more like "I prefer A to B and B to C and C to A, all at the same time".

The couple, if they had to pay a third party a cent to get undressed and then a cent to get dressed, would probably do it and consider it worth it---they end up two cents short but having had an enjoyable experience. Nothing irrational about that. To someone with the other "bad" kind of circular preferences, we can offer a sequence of trades (first A for B and a cent, then C for A and a cent, then B for C and a cent) after which they end up three cents short but otherwise exactly as they started (they didn't actually obtain enjoyable experiences, they made all the trades before anything happened). It is difficult to consider this rational.

IQ, or intelligence as commonly understood, is a poor proxy for rationality. In many cases it simple makes people better at rationalizing beliefs they acquired irrational.

These are not my arguments, since I haven't thought about the issue enough. However, the anthropologist Scott Atran, in response to the latest Edge annual question, "What Scientific Idea is Ready for Retirement?", answered "IQ". Here's his response:

There is no reason to believe, and much reason not to believe, that the measure of a so-called "Intelligence Quotient" in any way reflects some basic cognitive capacity, or "natural kind" of the human mind. The domain-general measure of IQ is not motivated by any recent discovery of cognitive or developmental psychology. It thoroughly confounds domain-specific abilities—distinct mental capacities for, say, geometrical and spatial reasoning about shapes and positions, mechanical reasoning about mass and motion, taxonomic reasoning about biological kinds, social reasoning about other people's beliefs and desires, and so on—which are the only sorts of cognitive abilities for which an evolutionary account seems plausible in terms of natural selection for task-specific competencies.

Nowhere in the animal or plant kingdoms does there ever appear to have been natural selection for a task-general adaptation. An overall measure of intelligence or mental competence is akin an overall measure for "the body," taking no special account of the various and specific bodily organs and functions, such as hearts, lungs, stomach, circulation, respiration, digestion and so on. A doctor or biologist presented with a single measure for "Body Quotient" (BQ) wouldn't be able to make much of it.

IQ is a general measure of socially acceptable categorization and reasoning skills. IQ tests were designed in behaviorism's heyday, when there was little interest cognitive structure. The scoring system was tooled to generate a normal distribution of scores with a mean of 100 and a standard deviation of 15.

In other societies, a normal distribution of some general measure of social intelligence might look very different, in that some "normal" members of our society could well produce a score that is a standard deviation from "normal" members of another society on that other society's test. For example, in forced-choice tasks East Asian students (China, Korea, Japan) tend to favor field-dependent perception over object-salient perception, thematic reasoning over taxonomic reasoning, and exemplar-based categorization over rule-based categorization.

American students generally prefer the opposite. On tests that measure these various categorization and reasoning skills, East Asians average higher on their preferences and Americans average higher on theirs'. There is nothing particularly revealing about these different distributions other than that they reflect some underlying socio- cultural differences.

There is a long history of acrimonious debate over which, if any, aspects of IQ are heritable. The most compelling studies concern twins raised apart and adoptions. Twin studies rarely have large sample populations. Moreover, they often involve twins separated at birth because a parent dies or cannot afford to support both, and one is given over to be raised by relatives, friends or neighbors. This disallows ruling out the effects of social environment and upbringing in producing convergence among the twins. The chief problem with adoption studies is that the mere fact of adoption reliably increases IQ, regardless of any correlation between the IQs of the children and those of their biological parents. Nobody has the slightest causal account of how or why genes, singly or in combination, might affect IQ. I don't think it's because the problem is too hard, but because IQ is a specious rather natural kind.

IQ can be used to give scientific justification to our internalized biases.

I don't want to limit your rights, because you are X. I want to limit your rights because I belong to Y, and as Y does better than X on IQ tests, it is only prudent that we know better what is good for you. I am also not interested in listening to counter-arguments coming from people whose IQ is below 99.

Also, in extreme cases, it can be used to push further policies such as eugenics (the bad kind that everyone has in mind when they hear the word "eugenics"):

Ah... I forgot to say that X shouldn't have the right to have children. No offense meant, but we want to avoid dims out breeding brights. Also, keep your stupid daughter away from my son, as I really don't want my own children to pollute genetic purity of our kind.

What does arguments against "reality" mean?

Arguing against misconceptions of what people believe about IQ? In general I consider arguments against reality pointless. Asking whether IQ is real is the wrong question. It makes much more sense to ask what IQ scores mean.

What do you mean by biased? Is there a difference between being biased towards something and desiring to do it?

what do you mean by 'personal hedonic exploration'? The things you list don't sound very exploratory...

To repair hangnails: Nail cream or nail oil. I had no idea these products existed, but they do, and they are designed specifically to deal with this problem, and do a very good job IME. Regular application for a few days fixes my problems.

To prevent it: Keep your hands protected outside (gloves). Minimise exposure of your hands to things that will strip water or oil from them (e.g. detergent, soap, solvents, nail varnish, nail varnish remover), and when you can't avoid those, use moisturiser afterwards to replace the lost oil.

(Explanation: Splitting/peeling nails is usually due to insufficient of oil or more rarely moisture. I've heard some people take a paleo line that we didn't need gloves and moisturiser and nail oil in the ancestral environment. Maybe, but we didn't wash our hands with detergent multiple times a day then either.)

Nail polish base coat over the cuticle might work. Personally I just try not to pick at them. I imagine you can buy base coat at the nearest pharmaceuticals store, but asking a beautician for advice is probably a good idea; presumably there is some way that people who paint their nails prevent hangnails from spoiling the effect.

Eliezer once tried to auction a day of his time but I can't find it on ebay by Googling.

On an unrelated note, the top Google result for "eliezer yudkowsky " (note the space) is "eliezer yudkowsky okcupid". "eliezer yudkowsky harry potter" is ninth, while HPMOR, LessWrong, CFAR and MIRI don't make the top ten.

Highly variable with skills, experience, and how badly they want the job. I bet there are some brilliant adjunct professors out there effectively making minimum wage because they really wanted to be professors. OTOH, I bet that google programmer isn't just being paid for talent, but specific skills and experience.

From some WSJ article:

The setting of Einstein's initial salary at Princeton illustrates his humility and attitude toward wealth. According to "Albert Einstein: Creator & Rebel" by Banesh Hoffmann, (1972), the 1932 negotiations went as follows: "[Abraham] Flexner invited [Einstein] to name his own salary. A few days later Einstein wrote to suggest what, in view of his needs and . . . fame, he thought was a reasonable figure. Flexner was dismayed. . . . He could not possibly recruit outstanding American scholars at such a salary. . . . To Flexner, though perhaps not to Einstein, it was unthinkable [that other scholars' salaries would exceed Einstein's.] This being explained, Einstein reluctantly consented to a much higher figure, and he left the detailed negotiations to his wife."

The reasonable figure that Einstein suggested was the modest sum of $3,000 [about $46,800 in today's dollars]. Flexner upped it to $10,000 and offered Einstein an annual pension of $7,500, which he refused as "too generous," so it was reduced to $6,000. When the Institute hired a mathematician at an annual salary of $15,000, with an annual pension of $8,000, Einstein's compensation was increased to those amounts.

Many such geniuses (top intellectual performers in fields where they can out-perform the median by several orders of magnitude) choose their work not just on the basis of payment, but what they work on, where, how, and with whom (preferring the company of other top performers).

If MIRI were to compete with Google at hiring programmers, I expect money be important but not overwhelmingly so. Google let you work with many other top people in your field, develop and use cool new tech, have big resources for your projects, and provides many non-monetary workplace benefits. MIRI lets you contribute to existential risk reduction, work with rationalists, etc.

Do they expect the most talented to be more likely to accept a lower salary for a good cause?

Yes. Any one with the necessary mindset of thinking that AI is the most important issue in the world will accept a lower salary than what's possible in the market elsewhere.

I don't know whether MIRI has an interest in hiring people who don't have that moral framework.

Two notes: First, the term "genius" is difficult to define. Someone may be a "genius" at understanding the sociology of sub-Saharan African tribes, but this skill will obviously command a much lower market value compared to someone who is a "genius" as a chief executive officer of a large company. A more precise definition of genius will narrow the range of costs per year.

Second, and related to the first, MIRI is (to the extent of my knowledge) currently focusing on mathematics and formal logic research rather than programming. This makes recruiting a team of "geniuses" much cheaper. While skilled mathematicians can attract quite strong salaries, highly skilled programmers can demand significantly more. It seems the most common competing job for MIRI's researchers would be that of a mathematics professor (which have a median salary ~88,000$). Based on this, MIRI could likely hire high quality mathematicians while offering them relatively competitive salaries.

There are oldtimer cars that seem to have no problem with running "indefinitely" provided you fix parts here and there.

error checking on solid state silicon is much easier than error checking neurons.

Most of the explanations found on cryonics site, do indeed seem to base their arguments around the hopeful explanation that given nanotechnology and science of the future every problem connected to as you say rebooting would become essentially trivial.

Too difficult for whom? Us, now? Obviously. Later? Well, how much progress are you willing to allow for 'too difficult' to become 'just doable'?

When non utilitarian rationalists consider big life changes, it seems to me that they don't do it based on how happy that will make them, Why?

"Non-utilitarian" doesn't equate to "ethical egoist". I'm not a utilitarian, but I still think my big life decisions are subject to ethical constraints beyond what will make me happy. It's just that the constraint isn't always (or even usually) the maximization of some aggregate utility function.

When non utilitarian rationalists consider big life changes, it seems to me that they don't do it based on how happy that will make them, Why?

I don't know the extent to which this applies to other people, but for me (a non-utilitarian) it does, so here's my data point which may or may not give you some insight into how other non-utilitarians judge these things.

I can't really say I value my own happiness much. Contentment / peace of mind (=/= happiness!) and meaningfulness are more like what I aim for; happiness is too fleeting, too momentary to seek it out all the time. I'm also naturally gloomy, and overt displays of cheerfulness just don't hold much appeal for me, in an aesthetic sense. (They get me thinking of those fake ad people and their fake smiles. Nobody can look that happy all the time without getting paid for it!) There simply are more important things in life than my own happiness; that one can be sacrificed, if need be, for the sake of a higher value. I suppose it's just like those utilitarians you're talking about which are "trying to maximize the world's something" rather than their own pleasure, only we don't think of it in a quantitative way.

But non utiltarians, like I used to be, and like most here still are, are just... doing it like everyone else does it! "Oh, that seems like a cool change, I'll do it! yay!" then two weeks later that particular thing has none of the coolness effect it had before, but they are stuck with the decision for years....... (in case of decisions like job, partner, quitting, smoking, big travels, big decisions, not ice cream flavour stuff)

Well... that's a rather unflattering way of putting it. You don't have to compute utilities in order for your decision-making process to look a wee little more elaborate than that.

I don't think the predictive power of models build from data driven happiness research is very high. I wouldn't ignore the research completely but there nothing rational about using a model just because it's data based if nobody showed that the model is useful for prediction in the relevant domain.

I know a lot of LW-ish people in the Bay Area and I see them explicitly thinking carefully about a lot of big life changes (e.g. moving, relationships, jobs, what habits to have) in just the way you recommended. I don't know if it has something to do with utilitarianism or not.

I'm personally more inclined to think in that way than I was a few years ago, and I think it's mostly because of the social effects of from hanging out with & looking up to a bunch of other people who do so.

You can't build a clock with a photon.

You can't build a clock with an electron either. You can build one with a muon though, since it will decay after some interval. It's not very accurate, but it's something.

In general, you cannot build a clock moving at light speed. You could build a clock with two photons. Measure the time by how close they are together. But if you look at the center of mass of this clock, it moves slower than light. If it didn't, the photons would have to move parallel to each other, but then they can't be moving away from each other, so you can't measure time.

Not quite either of those.

The first thing to say is that "at t0" means different things to different observers. Observers moving in different ways experience time differently and, e.g., count different sets of spacetime points as simultaneous.

There is a relativistic notion of "interval" which generalizes the conventional notions of distance and time-interval between two points of spacetime. It's actually more convenient to work with the square of the interval. Let's call this I.

If you pick two points that are spatially separated but "simultaneous" according to some observer, then I>0 and sqrt(I) is the shortest possible distance between those points for an observer who sees them as simultaneous. The separation between the points is said to be "spacelike". Nothing that happens at one of these points can influence what happens at the other; they're "too far away in space and too close in time" for anything to get between them.

If you pick two points that are "in the same place but at different times" for some observer, then I<0 and sqrt(-I) is the minimum time that such an observer can experience between visiting them. The separation between the points is said to be "timelike". An influence can propagate, slower than the speed of light, from one to the other. They're "too far away in time and too close in space" for any observer to see them as simultaneous.

And, finally, exactly on the edge between these you have the case where I=0. That means that light can travel from one of the spacetime points to the other. In this case, an observer travelling slower than light can get from one to the other, but can do so arbitrarily quickly (from their point of view) by travelling very fast; and while no observer can see the two points as simultaneous, you can get arbitrarily close to that by (again) travelling very fast.

Light, of course, only ever travels at the speed of light (you might have heard something different about light travelling through a medium such as glass, but ignore that), which means that it travels along paths where I=0 everywhere. To an (impossible) observer sitting on a photon, no time ever passes; every spacetime point the photon passes through is simultaneous.

So: does the distance as well as the time go to 0? Not quite. Neither distance nor time makes sense on its own in a relativistic universe. The thing that does make sense is kinda-sorta a bit like "distance minus time" (and more like sqrt(distance-squared minus time-squared)), and that is 0 for any two points in spacetime that are visited by the same photon.

(Pedantic notes: 1. There are two possible sign conventions for the square of the interval. You can say that I>0 for spacelike separations, or say that I>0 for timelike separations. I arbitrarily chose the first of these. 2. There may be multiple paths that light can take between two spacetime points. They need not actually have the same "length" (i.e., interval). Strictly, "interval" is defined only locally; then, for a particular path, you can integrate it up to get the overall interval. 3. In the case of light propagating through a medium other than vacuum, what actually happens involves electrons as well as photons and it isn't just a matter of a photon going from A to B. Whenever a photon goes from A to B it does it, by whatever path it does, at the speed of light.)

Other people have explained this pretty well already, but here's a non-rigorous heuristic that might help. What follows is not technically precise, but I think it captures an important and helpful intuition.

In relativity, space and time are replaced by a single four-dimensional space-time. Instead of thinking of things moving through space and moving through time separately, think of them as moving through space-time. And it turns out that every single (non-accelerated) object travels through space-time at the exact same rate, call it c.

Now, when you construct a frame of reference, you're essentially separating out space and time artificially. Consequently, you're also separating an object's motion through space-time into motion through space and motion through time. Since every object moves through space-time at the same rate, when we separate out spatial and temporal motion, the faster the object travels through space the slower it will be traveling through time. The total speed, adding up speed through space and speed through time, has to equal the constant c.

So an object at rest in a particular frame of reference has all its motion along the temporal axis, and no motion at all along the spatial axes. It's traveling through time at speed c and it isn't traveling through space at all. If this object starts moving, then some of the temporal motion is converted to spatial motion. It's speed through space increases, and its speed through time decreases correspondingly, so that the motion through space-time as a whole remains constant at c. This is the source of time dilation in relativity (as seen in the twin paradox) - moving objects move through time more slowly than stationary objects, or to put it another way, time flows slower for moving objects.

Of course, the limit of this is when the object's entire motion through space-time is directed along the spatial axes, and none of it is directed along the temporal axes. In this case, the object will move through space at c, which turns out to be the speed of light, and it won't move through time at all. Time will stand still for the object. This is what's going on with photons.

From this point of view, there's nothing all that weird about a photon's motion. From the space-time perspective, which after all is the fundamental perspective in relativity, it is moving pretty much exactly like any other object. It's only our weird habit of treating space and time as extremely different that makes the entirely spatial motion of a photon seem so bizarre.

Does this mean that a photon emitted at location A at t0 is absorbed at location B at t0, such that it's at two places at once?

In the photon's own subjective experience? Yes. (Not that that's possible, so this statement might not make sense). But as another commenter said, certainly the limit of this statement is true: as your speed moving from point A to point B approaches the speed of light, the subjective time you experience between the time when you're at A and the time when you're at B approaches 0. And the distance does indeed shrink, due to the Lorentz length contraction.

In what sense does the photon 'travel' then?

It travels in the sense that an external observer observes it in different places at different times. For a subjective observer on the photon... I don't know. No time passes, and the universe shrinks to a flat plane. Maybe the takeaway here is just that observers can't reach the speed of light.

There are no photons. There, you see? Problem solved.

(no, the author of the article is not a crank; he's a Nobel physicist, and everything he says about the laws of physics is mainstream)

Assume there are observers at A and B, sitting at rest relative to each other. The distance between them as seen by them is X. Their watches are synchronized. Alice, sitting at A, emits a particle when her watch says t0; Bob, sitting at B, receives it when his watch says t1. Define T = t1-t0. The speed of the particle is V = X/T.

If the particle is massive, then V is always smaller than c (the speed of light). We can imagine attaching a clock to the particle and starting it when it is emitted. When Bob receives it, the clock's time would read a time t smaller than T, given by the equation:

t = T (1 - V^2/c^2)^(1/2) (this is the Lorentz factor equation mentioned by Plasmon).

As the speed V of the particle gets closer and closer to c, you can see that the time t that has passed "for the particle" gets closer and closer to 0. One cannot attach a clock to a photon, so the statement that "photons are not moving through time" is somewhat metaphoric and its real meaning is the limiting statement I just mentioned. The photon is not "at two places at once" from the point of view of any physical observer, be it Alice and Bob (for whom the travel took a time T = X/c) or any other moving with a speed smaller than c (for whom the time taken may be different but is never 0).

The Lorentz factor diverges when the speed approaches c. Because of Length contraction and time dilation, both the distance and the time will appear to be 0, from the "point of view of the photon".

(the photon is "in 2 places at once" only from the point of view of the photon, and it doesn't think these places are different, after all they are in the same place! This among other things is why the notion of an observer traveling at c, rather than close to c, is problematic)

Getting this property for electromagnetic waves was one of the main things that led Einstein to develop Special Relativity: he looked at waves and thought, "If we do a Galileian transform so that light is standing still, the resulting field is an invalid electrostatic field"

Do you want to have a career at a conservative institution such a bank or a career in politics? If so, it's probably a bad idea to have too much attack surface by using your real name.

Do you want to make as many connections with other people as possible? If so, using your real name helps. It increases the attention that other people pay yourself. If you are smart and write insightful stuff that can mean job offers, speaking and speaking gigs.

If you meet people in real life might already know you from your online commentary that they have read and you don't have to start introducing yourself.

It's really a question of whether you think strangers are more likely to hurt or help you.

All biological organisms, considered as signalling or information processing networks, are massively parallel: huge amounts of similar cells with slightly different state signalling one another. It's not surprising that the biological evolved brain works the same way. A turing machine-like sequential computer powerful/fast enough for general intelligence would be far less likely to evolve.

So the fact that human intelligence is slow and parallel isn't evidence for thinking you can't implement intelligence as a fast serial algorithm. It's only evidence that the design is likely to be different from that of human brains.

It's likely true that we don't have the algorithmic (or other mathematical) techniques yet to make general intelligence. But that doesn't seem to me to be evidence that such algorithms would be qualitatively different from what we do have. We could just as easily be a few specific algorithmic inventions away from a general intelligence implementation.

Finally, as far as sheer scale goes, we're on track to achieve rough computational parity with a human brain in a single multi-processor cluster within IIRC something like a decade.