All of Jai's Comments + Replies

Jai0-5

Hype is a useful social mechanism for eliciting acute criticism and exposing flaws. If you want to know what your weaknesses are, you could do worse than to paint a giant target on your back.

Jai30

And if you’re in the US maybe stockpile a ton of them because companies aren’t allowed to produce incandescents anymore?

I just checked the DoE guidelines on this, and I think fairy lights are actually exempt! Here's the relevant paragraph (bold mine):

A general service incandescent lamp is a standard incandescent or halogen type lamp that is intended for general service applications. It has the following characteristics: (1) medium screw base; (2) lumen range of not less than 310 lumens and not more than 2,600 lumens or, in the case of a modified spectrum l

... (read more)
Jai50

I love learning new Smallpox Eradication Lore.

JaiΩ010

The second scenario omits the details about continuing to create and submit pull requests after takeover, instead just referring to human farms. Since it doesn't explicitly say that it's still optimizing for the original objective criteria and instead just refers to world domination, it appears to be inner misalignment (e.g. no longer aligned with the original optimizer). Did the original posing of this question specify that scenario 2 still maximizes pull requests after world domination?

2Rohin Shah
The intent was that in scenario 2 the AI constructs the human farms in order to to have the humans continually merging pull requests (i.e. yes, it still maximizes pull requests). I don't think I explicitly said this -- I probably gave the same text as in this post -- but I expect people to have made this inference (because (a) otherwise it's not clear why the AI would build human farms and (b) when I showed the inconsistency to the survey-takers, to my knowledge none of them said anything along the lines of "oh I was assuming that in scenario 2 the AI was no longer maximizing pull requests").