Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Wei_Dai 02 July 2015 04:03:43AM *  4 points [-]

See this 1998 discussion between Eliezer and Nick. Some relevant quotes from the thread:

Nick: For example, if it is morally preferred that the people who are currently alive get the chance to survive into the postsingularity world, then we would have to take this desideratum into account when deciding when and how hard to push for the singularity.

Eliezer: Not at all! If that is really and truly and objectively the moral thing to do, then we can rely on the Post-Singularity Entities to be bound by the same reasoning. If the reasoning is wrong, the PSEs won't be bound by it. If the PSEs aren't bound by morality, we have a REAL problem, but I don't see any way of finding this out short of trying it.

Nick: Indeed. And this is another point where I seem to disagree with you. I am not at all certain that being superintelligent implies being moral. Certainly there are very intelligent humans that are also very wicked; I don't see why once you pass a certain threshold of intelligence then it is no longer possible to be morally bad. What I might agree with, is that once you are sufficiently intelligent then you should be able to recognize what's good and what's bad. But whether you are motivated to act in accordance with these moral convictions is a different question.

Eliezer: Do you really know all the logical consequences of placing a large value on human survival? Would you care to define "human" for me? Oops! Thanks to your overly rigid definition, you will live for billions and trillions and googolplexes of years, prohibited from uploading, prohibited even from ameliorating your own boredom, endlessly screaming, until the soul burns out of your mind, after which you will continue to scream.

Nick: I think the risk of this happening is pretty slim and it can be made smaller through building smart safeguards into the moral system. For example, rather than rigidly prescribing a certain treatment for humans, we could add a clause allowing for democratic decisions by humans or human descendants to overrule other laws. I bet you could think of some good safety-measures if you put your mind to it.

Nick: How to contol a superintelligence? An interesting topic. I hope to write a paper on that during the Christmas holiday. [Unfortunately it looks like this paper was never written?]

I assume Bostrom called it something else.

He used "control", which is apparently still his preferred word for the problem today, as in "AI control".

Comment author: lukeprog 02 July 2015 07:08:14AM 2 points [-]

For those who haven't been around as long as Wei Daiā€¦

Eliezer tells the story of coming around to a more Bostromian view, circa 2003, in his coming of age sequence.

Comment author: lukeprog 25 June 2015 08:42:51PM 4 points [-]

Just FYI, I plan to be there.

Comment author: lukeprog 24 June 2015 05:43:12PM 1 point [-]

Any idea when the book is coming out?

Comment author: Houshalter 08 June 2015 01:32:53AM 1 point [-]

Take a look at this image.

Stuart Russell said recently "The commercial investment in AI the last five years has exceeded the entire world wide government investment in AI research since it's beginnings in the 1950's."

Comment author: lukeprog 08 June 2015 01:57:10AM 2 points [-]

Just FYI to readers: the source of the first image is here.

Comment author: lukeprog 30 May 2015 12:17:06AM 5 points [-]

I don't know if this is commercially feasible, but I do like this idea from the perspective of building civilizational competence at getting things right on the first try.

Comment author: lukeprog 29 May 2015 06:17:43PM 19 points [-]

Might you be able to slightly retrain so as to become an expert on medium-term and long-term biosecurity risks? Biological engineering presents serious GCR risk over the next 50 years (and of course after that, as well), and very few people are trying to think through the issues on more than a 10-year time horizon. FHI, CSER, GiveWell, and perhaps others each have a decent chance of wanting to hire people into such research positions over the next few years. (GiveWell is looking to hire a biosecurity program manager right now, but I assume you can't acquire the requisite training and background immediately.)

Comment author: ciphergoth 29 April 2015 04:32:32PM 8 points [-]

This isn't a first for CFAR or MIRI - I hope you guys are putting lots of thought into how to have your last-minute ideas earlier :-)

Comment author: lukeprog 29 April 2015 11:57:55PM 5 points [-]

I think it's partly not doing enough far-advance planning, but also partly just a greater-than-usual willingness to Try Things that seem like good ideas even if the timeline is a bit rushed. That's how the original minicamp happened, which ended up going so well that it inspired us to develop and launch CFAR.

Comment author: RyanCarey 24 April 2015 10:01:29PM 1 point [-]

For what it's worth, I used xelatex and some of Alex Vermeer's code, but I can't see why any would effect the links, and can't find any suggestions for why this would occur in Sumatra. I'll just sit on this for now, but if more people have a similar issue, I'll look further. Thanks.

Comment author: lukeprog 26 April 2015 03:02:30AM 1 point [-]

People have complained about Sumatra not working with MIRI's PDF ebooks, too. It was hard enough already to get our process to output the links we want on most readers, so we decided not to make the extra effort to additionally support Sumatra. I'm not sure what it would take.

Comment author: Epictetus 10 February 2015 11:23:20AM *  11 points [-]

I suppose I can think up a few tomes of eldritch lore that I have found useful (college math specifically):


Recommendation: Differential and Integral Calculus

Author: Richard Courant


Stewart, Calculus: Early Transcendentals: This is a fairly standard textbook for freshman calculus. Mediocre overall.

Morris Kline, Calculus: An Intuitive and Physical Approach: Great book. As advertised, focuses on building intuition. Provides a lot of examples that aren't the usual contrived "applications". This would work well as a companion piece to the recommended text.

Courant, Differential and Integral Calculus (two volumes): One of the few math textbooks that manages to properly explain and motivate things and be rigorous at the same time. You'll find loads of actual applications. There are plenty of side topics for the curious as well as appendices that expand on certain theoretical points. It's quite rigorous, so a companion text might be useful for some readers. There's an updated version edited by Fritz John (Introduction to Calculus and Analysis), but I am unfamiliar with it.

Linear Algebra:

Recommended Text: Linear Algebra

Author: Georgi Shilov


David Lay, Linear Algebra and its Applications: Used this in my undergraduate class. Okay introduction that covers the usual topics.

Sheldon Axler, Linear Algebra Done Right: Ambitious title. The book develops linear algebra in a clean, elegant, and determinant-free way (avoiding determinants is the "done right" bit, though they are introduced in the last chapter). It does prove to be a drawback, as determinants are a useful tool if not abused. This book is also a bit abstract and is intended for students who have already studied linear algebra.

Georgi Shilov, Linear Algebra: No-nonsense Russian textbook. Explanations are clear and everything is done with full rigor. This is the book I used when I wanted to understand linear algebra and it delivered.

Horn and Johnson, Matrix Analysis: I'm putting this in for completion purposes. It's a truly stellar book that will teach you almost everything you wanted to know about matrices. The only reason I don't have this as the recommendation is that it's rather advanced and ill-suited for someone new to the subject.

Numerical Methods

Recommendation: Numerical Recipes: The Art of Scientific Computing

Author: Press, Teukolsky, Vetterling, Flannery


Bulirsch and Stoer, Introduction to Numerical Analysis: German rigor. Thorough and thoroughly terse, this is one of those good textbooks that only a sadist would recommend to a beginner.

Kendall Atkinson, An Introduction to Numerical Analysis: Rigorous treatment of numerical analysis. It covers the main topics and is far more accessible than the text by Bulirsch and Stoer.

Press, Teukolsky, Vetterling, Flannery, Numerical Recipes: The Art of Scientific Computing: Covers just about every numerical method outside of PDE solvers (though this is touched on). Provides source code implementing just about all the methods covered and includes plenty of tips and guidelines for choosing the appropriate method and implementing it. THE book for people with a practical bent. I would recommend using the text by Atkinson or Bulirsch and Stoer to brush up on the theory, however.

Richard Hamming, Numerical Methods for Scientists and Engineers: How can I fail to mention a book written by a master of the craft? This book is probably the best at communicating the "feel" of numerical analysis. Hamming begins with an essay on the principles of numerical analysis and the presentations in the rest of the book go beyond the formulas. I docked points for its age and more limited scope.

Ordinary Differential Equations

Recommended: Ordinary Differential Equations

Author: Vladimir Arnold


Coddington, An Introduction to Ordinary Differential Equations: Solid intro from the author of one of the texts in the field. Definite theoretical bent that doesn't really touch on applications.

Tenenbaum and Pollard, Ordinary Differential Equations: This book manages to be both elementary and comprehensive. Extremely well-written and divides the material into a series of manageable "Lessons". Covers lots and lots of techniques that you might not find elsewhere and gives plenty of applications.

Vladimir Arnold, Ordinary Differential Equations: Great text with a strong geometric bent. The language of flows and phase spaces is introduced early on, which becomes relevant as the book ends with a treatment of differential equations on manifolds. Explanations are clear and Arnold avoids a lot of the pedantry that would otherwise preclude this kind of treatment (although it requires more out of the reader). It's probably the best book I've seen for intuition on the subject and that's why I recommend it. Use Tenenbaum and Pollard as a companion if you want to see more solution methods.

Abstract Algebra:

Note: I am mainly familiar with graduate texts, so be warned that these books are not beginner-friendly.

Recommended: Basic Algebra

Author: Nathan Jacobson


Bourbaki, Algebra: The French Bourbaki tradition in all its glory. Shamelessly general and unmotivated, this is not for the faint of heart. The drawback is its age, as there is no treatment of category theory.

Lang, Algebra: Lang was once a member of the aforementioned Bourbaki. In usual Serge Lang style, this is a tough, rigorous book that has no qualms with doing things in full generality. The language of category theory is introduced early and heavily utilized. Great for the budding algebraist.

Hungerford, Algebra: Less comprehensive, but more accessible than Lang's book. It's a good choice for someone who wants to learn the subject without having to grapple with Lang.

Jacobson, Basic Algebra (2 volumes): Note that the "Basic" in the title means "so easy, a first-year grad student can understand it". Mathematicians are a strange folk, but I digress. It's comprehensive, well-organized, and explains things clearly. I'd recommend it as being easier than Bourbaki and Lang yet more comprehensive and a better reference than Hungerford.

Elementary Real Analysis:

"Elementary" here means that it doesn't emphasize Lebesgue integration or functional analysis

Recommended: Principles of Mathematical Analysis

Author: Walter Rudin


Rudin, Principles of Mathematical Analysis: Infamously terse. Rudin likes to do things in the greatest generality and the proofs tend to be slick (i.e. rely on clever arguments that don't really clarify the thing being proved). It's thorough, it's rigorous, and the exercises tend to be difficult. You won't find any straightforward definition-pushing here. If you had a rigorous calculus course (like Courant's book), you should be fine.

Kenneth Ross, Elementary Analysis: The Theory of Calculus: I'd put this book as a gap-filler. It doesn't go into topology and is rather straightforward. If you learned the "cookbook" approach to calculus, you'll probably benefit from this book. If your calculus class was rigorous, I'd skip it.

Serge Lang, Undergraduate Analysis: It's a Serge Lang book. Contrary to the title, I don't think I'd recommend it for undergraduates.

G.H. Hardy, A Course of Pure Mathematics: Classic text. Hardy was a first-rate mathematician and it shows. The downside is that the book is over 100 years old and there are a few relevant topics that came out in the intervening years.

Comment author: lukeprog 15 April 2015 12:19:58AM 1 point [-]

Updated, thanks!

Comment author: Gram_Stone 02 January 2015 06:28:40AM 0 points [-]

Just so you know, the title of Spivak's book has been misspelled as 'Caclulus.'

Comment author: lukeprog 15 April 2015 12:07:56AM 0 points [-]

Fixed, thanks.

View more: Next