What are the most noteworthy sections to read? (Looks like you forgot to bold them.) Thanks!
The Amazon link in the post is for the third (and latest) edition, only $28. Your other links are for the second edition, except the Harvard link's dead.
Did you forget to bold the particularly noteworthy sections in the table of contents?
More than a 76% pay cut, because a lot of the compensation at Google is equity+bonus+benefits; the $133k minimum listed at your link is just base salary.
I'd thought it was a law of nature that quiet norms for open plans don't actually work; it sounds like you've found a way to have your cake and eat it too!
That's fair; thanks for the feedback! I'll tone down the gallows humor on future comments; gotta keep in mind that tone of voice doesn't come across.
BTW a money brain would arise out of, e.g., a merchant caste in a static medieval society after many millennia. Much better than a monkey brain, and more capable of solving alignment!
Beren, have you heard of dependent types, which are used in Coq, Agda, and Lean? (I don't mean to be flippant; your parenthetical just gives the impression that you hadn't come across them, because they can easily enforce integer bounds, for instance.)
Thanks for the great back-and-forth! Did you guys see the first author's comment? What are the main updates you've had re this debate now that it's been a couple years?
The paper's first author, beren, left a detailed comment on the ACX linkpost, painting a more nuanced and uncertain (though possibly outdated by now?) picture. To quote the last paragraph:
"The brain being able to do backprop does not mean that the brain is just doing gradient descent like we do to train ANNs. It is still very possible (in my opinion likely) that the brain could be using a more powerful algorithm for inference and learning -- just one that has backprop as a subroutine. Personally (and speculatively) I think it's likely that the brain perfor...
Re open plan offices: many people find them distracting. I doubt they're a worthwhile cost-saving measure for research-focused orgs; better to have fewer researchers in an environment conducive to deep focus. I could maybe see a business case for them in large orgs where it might be worth sacrificing individual contributors' focus in exchange for more legibility to management, or where management doesn't trust workers to stay on task when no one is hovering over their shoulder, but I hope no alignment org is like that. For many people open plan offices are...
I meant I don't think the CEV of ancient Rome has the same values as ancient Rome. Looks like your comment got truncated: "what is good if they were just"
Is there a command-line tool for previewing how a "markdown+LaTeX" text file would render as a LW draft post, for those of us who prefer to manipulate text files using productivity tools like (neo)vim and git?
Ah right, because Clippy has less measure, and so has less to offer, so less needs to be offered to it. Nice catch! Guess I've been sort of heeding Nate's advice not to think much about this. :)
Of course, there would still be significant overhead from trading with and/or outbidding sampled plethoras of UFAIs, vs the toy scenario where it's just Clippy.
I currently suspect we still get more survival measure from aliens in this branch who solved their alignment problems and have a policy of offering deals to UFAIs that didn't kill their biol...
Paperclips vs obelisks does make the bargaining harder because clippy would be offered fewer expected paperclips.
My current guess is we survive if our CEV puts a steep premium on that. Of course, such hopes of trade ex machina shouldn't affect how we orient to the alignment problem, even if they affect our personal lives. We should still play to win.
Roman values aren't stable under reflection; the CEV of Rome doesn't have the same values as ancient Rome. It's like a 5-year-old locking in what they want to be when they grow up.
Locking in extrapolated Roman values sounds great to me because I don't expect that to be significantly different than a broader extrapolation. Of course, this is all extremely handwavy and there are convergence issues of superhuman difficulty! :)
Yes it would, at least if you mean their ancient understanding of morals.
Not on mobile, in my experience.
I think it would be helpful to note at the top of the post that it's crossposted here. I initially misinterpreted "this blog" in the first sentence as referring to LW.
This idea keeps getting rediscovered, thanks for writing it up! The key ingredient is acausal trade between aligned and unaligned superintelligences, rather than between unaligned superintelligences and humans. Simulation isn't a key ingredient; it's a more general question about resource allocation across branches.
Too much power, I would assume. Yet he didn't kill Bo Xilai.
Why the downboats? People new to LW jargon probably wouldn't realize "money brain" is a typo.
Nitpick: maybe aligned and unaligned superintelligences acausally trade across future branches? If so, maybe on the mainline we're left with a very small yet nonzero fraction of the cosmic endowment, a cosmic booby prize if you will?
"Booby prize with dignity" sounds like a bit of an oxymoron...
What does "corrupt" mean in this context? What are some examples of noncorrupt employers?
A CFAR board member asked me to clarify what I meant about “corrupt”, also, in addition to this question.
So, um. Some legitimately true facts the board member asked me to share, to reduce confusion on these points:
See also: If Bad Things Happen, It Is Your Fault For Predicting Them :)