almost any improvement will indirectly help at designing AI

That may be too strong of a statement. Say some new tool helps improve AI legislation more than AI design, this might turn slowing down the wheel.

Commit to only use superhuman persuasion when arguing towards a valid conclusion via valid arguments, in a manner that doesn't go against the interests of the person being persuaded.

In this plan, how should the AI define what’s in the interest of the person being persuaded? For example, say you have a North Korean soldier who can be persuaded to quite for the west (at the risk of getting the shitty jobs most migrants have) or who can be persuaded to remain loyal to his bosses (at the risk of raising his children in the shitty country most north korean have), what set of rules would you suggest?

the asteroid would likely burn up, but perhaps you have a solution for that

Yes, there’s a well known solution: just make the asteroid fast enough, and it will burn less in the atmosphere.

My understanding of this framework is probably too raw to go sane (A natural latent is a convolution basis useful for analyzing natural inputs, and it’s powerful because function composition is powerful) but it could fit nicely with Agency is what neurons in the biological movement area detect.

That’s great analogy. To me the strength of the OP is to pinpoint that LLMs already exhibit the kind of general ability we would expect from AGI, and the weakness is to forget that LLMs do not exhibit some specific ability most thought easy, such as the agency that even clownfishes exhibit.

In a way this sounds like again the universe is telling us we should rethink what intelligence is. Chess is hard and doing the dishes is easy? Nope. Language is hard and agency is central? Nope.

Thanks for the prompt! If we ask Claude 3 to be happy about x, don’t you think that counts as nudging it toward implementing a conscious being?

Perhaps you could identify your important beliefs

That part made me think. If I see bright minds falling in this trap, does blindness goes with importance of the belief for that person? I would say yes I think. As if that’s where we tend to make more « mistakes. that can behave as ratchets of the mind ». Thanks for the insight!

that also perhaps are controversial

Same exercise: if I see bright minds falling in this trap, does blindness goes with controversial beliefs? Definitely! Almost by definition actually.

each year write down the most likely story you can think of that would make it be wrong

I don’t feel I get this part as well as the formers. Suppose I hold the lab leak view, then notice it’s both controversial (« these morons can’t update right »), and much more important to me (« they don’t get how important it is for the safety of everyone »). What should I write?

Yup. Thanks for trying, but these beliefs seem to form a local minima, like a trap for the rational minds -even very bright ones. Do you think you understand how an aspiring rationalist could 1) recover and get out of this trap 2) don’t fall for it in the first place?

To be clear, my problem is not with the possibility of a lab leak itself, it’s with the evaluation that present evidences are anything but posthoc rationalizations fueled by unhealthy levels of tunnel vision. If bright minds can fall for that on this topic specifically, how do I know I’m not making the same mistake on something else?

(Spoiler warning)

(Also I didn’t check the previous survey nor the comments there, so expect some level of redondance)

The score itself (8/18) is not that informative, but checking the « accepted » answers is quite interesting. Here’s my « errors » and how much I’m happy making them:

You should be on the outlook for people who are getting bullied, and help defend them against the bullies.

I agree some rationalist leaders are toxic characters who will almost inevitably bully their students and collaborators, and I’m happy to keep strongly disagreeing with that. [actually most rationalists taking the survey agree with the statement, see tailcalled below]

Modern food is more healthy because we have better hygiene than people did in the past.

I strongly agree, LW accept weaker position. Ok, maybe I don’t have the data for a period of time in the past where food was not that bad. (Slight update on both the content and the need to countercheck the data myself).

It should be easy to own guns.

Seriously, is that even a question? Aren’t rationalists supposed to look at the data at some point?

Statistics show that black people still are far from catching up to white people in society.

I strongly agree, LW accept weaker position. But what I had in mind was the situation in Canada, so maybe I just don’t have the relevant data for the USA. Yes, that’s my kind of humour.

It is bad to buy and destroy expensive products in the name of "art".

I thought I disagree, but ok I admit I could imagine good examples. Like a statue from melting guns involved in murdering children at school.

You should believe your friends if they tell you they've seen ghosts.

That’s an interesting one. I would not believe ghosts caused this perception, but I will not negate the perception itself, nor believe that establishing the truth about ghosts is what should occupy my mind in this situation.

Charity organizations should offer their employees dream vacations in the tropics to make the employment more attractive and enjoyable, thereby attracting more people to the charity.

I thought everyone got the memo it did hurt the whole movement. Am I wrong or the « correct » answer is just outdated from pre SBF times?

There is no God. Supernatural claims are never true

That’s the only two questions where I firsthand did expect to differ, for complicated reasons that I just find this week was better expressed by Aella. (In short, it’s more useful to keep a sane dose of agnosticism)

https://knowingless.com/2018/05/02/so-says-crazybrain/

(I should also credit Scott Aaronson for best showing QM allows supernatural claims such as « there’s free will »)

Curiosity is for boring nerds.

Meow.

I am an old person. They may not let you do that in chemistry any more.

Absolutely! In my first chemistry lab, a long time ago, our teacher warned us that she had just lost a colleague to cancer at the age of forty, and she swore that if we didn't follow the security protocols very seriously, she would be our fucking nightmare.

I never heard her swear after that.