Relatedly, if you perform an experiment n times, and the probability of success is p, and the expected number of total successes kp is much smaller than one, then kp is a reasonable measure of getting at least once success, because the probability of getting more than one success can be neglected. 

For example, if Bob plays the lottery for ten days, and each days has a 1:1000,000 chance of winning, then overall he will have a chance of 100,000 of winning once. 

This is also why micromorts are roughly additive: if travelling by railway has a mortality of one micomort per 10Mm, then travelling for 50Mm will set you back 5 micomort. Only if you leave what I would call the 'Newtonian regime of probability', e.g. by somehow managing to travel 1Tm with the railway, you are required to do proper probability math, because naive addition would tell you that you will surely have a fatal accident (1 mort) in that distance, which is clearly wrong. 

Getting down-voted to -27 is an achievement. Most things judged 'bad AI takes' only go to -11 or so, even that recent  P=NP proof only got to -25. Of course, if the author is right, then downvoting further is providing helpful incentives to him. 

I think that bullying is quite distinct from status hierarchies. The latter are unavoidable. There will always be some clique of cool kids in the class who will not invite the non-cool kids to their parties. This is ok. Sometimes, status is correlated with behaviors which are pro-social (kids not smoking; donating to EA), sometimes it is correlated with behaviors which are net-negative (kids smoking; serving in the SS). I was not part of the cool kids circle, and I was fine with that. Live and let live and all that. 

'Bullying' has a distinct negative connotation. The central example is someone who is targeted for sport for being different from the others. The bullies don't want the victims to change their ways, they just like to make their life miserable for thrills. I am sure that sometimes it unintentionally helps their victim, if you push enough people, at some point you are bound to push someone out of the path of a car or bullet. In the grand scheme of things, the bullies are however net negative for their victims and society overall. 

I see this as less of an endorsement of linear models and more of a scathing review of expert performance. 

• When an arithmetic model is calibrated, it is specifically by including feedback from the real-world effects of its predictions. Experts do not, as a rule, seek out any feedback on their calibration.

This. Basically, if your job is to do predictions, and the accuracy of your predictions is not measured, then (at least the prediction part of) your job is bullshit. 

I think that if you compare simple linear models in domains where people actually care about their predictions, the outcome would be different. For example, if simple models predicted stock performance better than experts at investment banks, anyone with a spreadsheet could quickly become rich. There are few if any cases of 'I started with Excel and 1000$, and now I am a billionaire'. Likewise, I would be highly surprised to see a simple linear model outperform Nate Silver or the weather forecast. 

Even predicting chess outcomes from mid-game board configurations is something where I would expect human experts to outperform simple statistical models working on easily quantifiable data (e.g. number of pieces remaining, number of possible moves, being in check, etc).

Neural networks contained in animal brains (which includes human brains) are quite capable of implementing linear models, and such should at least perform equally well when they are properly trained. A wolf pack deciding to chase or not chase some prey has direct evolutionary skin in the game of making their prediction of success as accurate as possible which the average school counselor predicting academic success simply does not have.

--

You touch this a bit in 'In defense of explainatory modeling', but I want to emphasize that uncovering causal relationships and pathways is central to world modelling. Often, we don't want just predictions, we want predictions conditional on interventions. If you don't have that, you will end up trying to cure chickenpox with makeup, as 'visible blisters' is negatively correlated with outcomes. 

Likewise, if we know the causal pathway, we have a much better basis to judge if some finding can be applied to out-of-distribution data. No matter how many anvils you have seen falling, without a causal understanding (e.g. Newtonian mechanics), you will not be able to reliably apply your findings to falling apples or pianos. 
 

What I don't understand is why there should be a link between trapped priors and an moral philosophy. 

I mean, if moral realism was correct, i.e. if moral tenets such as "don't eat pork", "don't have sex with your sister", or "avoid killing sentient beings" had an universal truth value for all beings capable of moral behavior, then one might argue that the reason why people's ethics differ is that they have trapped priors which prevent them from recognizing these universal truths. 

This might be my trapped priors talking, but I am a non-cognitivist. I simply believe that assigning truth values to moral sentences such as "killing is wrong" is pointless, and they are better parsed as prescriptive sentences such as "don't kill" or "boo on killing". 

In my view, moral codes are intrinsically subjective. There is no factual disagreement between Harry and Professor Quirrell which they could hope to overcome through empiricism, they simply have different utility functions.

--

My second point is that if moral realism was true, and one of the key roles of religion was to free people from trapped priors so they could recognize these universal moral truths, then at least during the founding of religions, we should see some evidence of higher moral standards before they invariably mutate into institutions devoid of moral truths. I would argue that either, our commonly accepted humanitarian moral values are all wrong or this mutation process happened almost instantly:

• Whatever Jesus thought about gender equality when he achieved moral enlightenment, Paul had his own ideas a few decades later. 
• Mohammed was clearly not opposed to offensive warfare.
• Martin Luther evidently believed that serfs should not rebel against their lords. 

On the other hand, for instances where religions did advocate for tenets compatible with humanitarianism, such as in Christian abolitionism, do not seem to correspond to strong spiritualism. Was Pope Benedict XIV condemning the slave trade because he was more spiritual (and thus in touch with the universal moral truth) than his predecessors who had endorsed it?

--

My last point is that especially with regard to relational conflicts, our map not corresponding to the territory might often not be a bug, but a feature. Per Hanson, we deceive ourselves so that we can better deceive others. Evolution has not shaped our brains to be objective cognitive engines. In some cases, objective cognition is what it advantageous -- if you are alone hunting a rabbit, no amount of self-deception will fill your stomach -- but in any social situation, expect evolution to put the hand on the scales of your impartial judgement. Arguing that your son should become the new chieftain because he is the best hunter and strongest warrior is much more effective than arguing for that simply because he is your son -- and the best way to argue that is to believe it, no matter if it is objectively true. 

The adulterer, the slave owner and the wartime rapist all have solid evolutionary reasons to engage in behaviors most of us might find immoral. I think their moral blind spots are likely not caused by trapped priors, like an exaggerated fear of dogs is. Also, I have no reason to believe that I don't have similar moral blind spots hard-wired into my brain by evolution.

I would bet that most of the serious roadblocks to a true moral theory (if such a thing existed) are of that kind, instead of being maladaptive trapped priors. Thus, even if religion and spirituality are effective at overcoming maladaptive trapped priors, I don't see how they would us bring closer to moral cognition. 

Note: there is an AI audio version of this text over here: https://askwhocastsai.substack.com/p/eliezer-yudkowsky-tweet-jul-21-2024

I find the AI narrations offered by askwho generally ok, worse than what a skilled narrator (or team) could do but much better than what I could accomplish. 

[...] somehow humanity's 100-fold productivity increase (since the days of agriculture) didn't eliminate poverty.

That feels to me about as convincing as saying: "Chemical fertilizers have not eliminated hunger, just the other weekend I was stuck on a campus with a broken vending machine." 

I mean, sure, both the broken vending machine and actual starvation can be called hunger, just as both working 60h/week to make ends meet or sending your surviving kids into the mines or prostituting them could be called poverty, but the implication that either scourge of humankind has not lost most of its terror seems clearly false. 

Sure, being poor in the US sucks, but I would rather spend a year living the life of someone in the bottom 10% income bracket in the 2024 US than spending a month living the life of a poor person during the English industrial revolution.

I am also not convinced that 60h/week is what it actually takes to survive in the US. I can totally believe that this amount of unskilled labor might be required to rent accommodations in cities, though. 

Critically, the gene editing of the red blood cells can be done in the lab; trying to devise an injectable or oral substance that would actually transport the gene-editing machinery to an arbitrary part of the body is much harder.

I am totally confused by this. Mature red blood cells don't contain a nucleus, and hence no DNA. There is nothing to edit. Injecting blood cells produced by gene-edited bone marrow in vitro might work, but would only be a therapy, not a cure: it would have to be repeated regularly. The cure would be to replace the bone marrow. 

So I resorted to reading through the linked FDA article. Relevant section:

The modified blood stem cells are transplanted back into the patient where they engraft (attach and multiply) within the bone marrow and increase the production of fetal hemoglobin (HbF), a type of hemoglobin that facilitates oxygen delivery.

Blood stem cells seems to be FDA jargon for ematopoietic stem cells. From the context, I would guess they are harvested from the bone marrow of the patients, then CRISPRed, and then injected back in the blood stream where they will find back to the bone marrow. 

I still don't understand how they would outcompete the non-GMO bone marrow which produces the faulty red blood cells, though. 

I would also take the opportunity to point out that the list of FDA-approved gene therapies tells us a lot about the FDA and very little about the state of the art. This is the agency which banned life-saving baby nutrition for two years, after all. Anchoring what is technologically possible to what the FDA approves would be like anchoring what is possible in mobile phone tech to what is accepted by the Amish. 

Also, I think that editing of multi-cellular organisms is not required for designer babies at all. 

1. Start with a fertilized egg, which is just a single cell. Wait for the cell to split. After it has split, separate the cells into two cells. Repeat until you have a number of single-cell candidates.
2.  Apply CRISPR to these cells individually. allow them to split again. Do a genome analysis on one of the two daughter cell. Select a cell line where you did only the desired change in the genome. Go back to step one and apply the next edit. 

Crucially, the costs would only scale linearly with the number of edits. I am unsure how easy is that "turn one two-cell embryo into two one-cell embryos", though. 

Of course, it would be neater to synthesize the DNA of the baby from the scratch, but while prices per base pair synthesis have been dropped a lot, they are clearly still to high to pay for building a baby (and there are likely other tech limitations). 

I thought this first too. I checked on Wikipedia:

Adult stem cells are found in a few select locations in the body, known as niches, such as those in the bone marrow or gonads. They exist to replenish rapidly lost cell types and are multipotent or unipotent, meaning they only differentiate into a few cell types or one type of cell. In mammals, they include, among others, hematopoietic stem cells, which replenish blood and immune cells, basal cells, which maintain the skin epithelium [...].

I am pretty sure that the thing a skin cell makes per default when it splits is more skin cells, so you are likely correct. 

See here. Of course, that article is a bit light on information on detection thresholds, false-positive rates and so on as compared to dogs, mass spectrometry or chemical detection methods. 

I will also note that humans have 10-20M olfactory receptor neurons, while bees have 1M neurons in total. Probably bees are under more evolutionary pressure to make optimal use of their olcfactory neurons, though. 

Dear Review Bot,

please avoid double-posting. 

On the other hand, I don't think voting you to -6 is fair, so I upvoted you.