Only in the minds of "yay future!" sorts of people. My parents think self-publishing a non-fiction book is sketchy, and they have every valid reason for believing this. I agree with them, in fact.
Take the "outside view": if someone handed you a book and said, "This describes the most important scientific and moral challenge of our times", and you had never heard of MIRI/FHI/CFAR/LW in any way whatsoever, what are the questions you're going to ask? They'll almost definitely be credibility questions; you want to read a book that's been edited well and comes from some kind of well-known intellectual expert on the topic being discussed, preferably one from very credible institutions like academia and government. You're a non-expert, after all, and it's not worth your time becoming an expert on some obscure, highly technical topic just to judge whether you should spend $30 on a book.
Luckily, information above indicates that FHI is publishing Superintelligence under the Oxford University label, with actual academic credentials and such, in hardback print. I'll be waiting for that one before I recommend anything to "the normals".
"This describes the most important scientific and moral challenge of our times", and you had never heard of MIRI/FHI/CFAR/LW in any way whatsoever, what are the questions you're going to ask?
"Why do you think so?"
They'll almost definitely be credibility questions;
Not in my case.
you want to read a book that's been edited well and comes from some kind of well-known intellectual expert on the topic being discussed, preferably one from very credible institutions like academia and government.
Oh, certainly not. Government is pretty...
We're pleased to announce the release of "Smarter Than Us: The Rise of Machine Intelligence", commissioned by MIRI and written by Oxford University’s Stuart Armstrong, and available in EPUB, MOBI, PDF, and from the Amazon and Apple ebook stores.
Can we instruct AIs to steer the future as we desire? What goals should we program into them? It turns out this question is difficult to answer! Philosophers have tried for thousands of years to define an ideal world, but there remains no consensus. The prospect of goal-driven, smarter-than-human AI gives moral philosophy a new urgency. The future could be filled with joy, art, compassion, and beings living worthwhile and wonderful lives—but only if we’re able to precisely define what a “good” world is, and skilled enough to describe it perfectly to a computer program.
AIs, like computers, will do what we say—which is not necessarily what we mean. Such precision requires encoding the entire system of human values for an AI: explaining them to a mind that is alien to us, defining every ambiguous term, clarifying every edge case. Moreover, our values are fragile: in some cases, if we mis-define a single piece of the puzzle—say, consciousness—we end up with roughly 0% of the value we intended to reap, instead of 99% of the value.
Though an understanding of the problem is only beginning to spread, researchers from fields ranging from philosophy to computer science to economics are working together to conceive and test solutions. Are we up to the challenge?
Special thanks to all those at the FHI, MIRI and Less Wrong who helped with this work, and those who voted on the name!