If someone made what they thought was FAI and it wasn't actually FAI, how could you tell until it was too late?
How can you tell that a theorem is correct? By having a convincing proof, or perhaps other arguments to the effect of the theorem being correct, and being trained to accept correct proofs (arguments) and not incorrect ones.
Even though we currently have no idea about how to construct a FAI or how an argument for a particular design being a FAI might look like, we can still refuse to believe that something is a FAI when we are not given sufficient grounds for believing it is one, provided the people who make such decisions are skilled enough to accept correct arguments but not incorrect ones.
The rule to follow is to not actually build or run things that you don't know to be good. By default, they don't even work, and if they do, they are not good, because value is complex and won't be captured by chance. There need to be strong grounds for believing that a particular design will work, and here rationality training is essential, because by default people follow one of the many kinds of crazy reasoning about these things.
Follow up to: Best career models for doing research?
First, I must apologize for the somewhat self-serving post, but as it is in the discussion section I hope that this can be forgiven. Also, I would not be surprised if there are at least a few college age people lurking around with very similar problems/issues, so I expect that this might prove very useful to at least a couple of people here. If this works out, I do hope to eventually put it into the form of a more general top-level post on career advice for those interested in a career in AGI.
Now, on to the issue:
It has come to my attention that research opportunities in AGI appear to both be somewhat limited, and somewhat unstructured compared to more well-developed fields that I have looked into. It seems to me that it would be useful to have a discussion here, given the unusual population density of AGI enthusiasts/professionals, about the possible pathways that one might take after the completion of an undergraduate degree. In my case, I have a strong background in mathematics, computer science and philosophy as well as a growing knowledge base in psychology. I've been studying Pearl's work, Timeless Decision Theory, cognitive science, evolutionary and cognitive psychology, Bishop's book on Pattern Recognition and Machine Learning, the link between category theory and cognitive science/AI (which appears to have some promise for building ontologies that can combine concepts and generalize), game theory, probability/statistics, computational complexity and I have been trying to get a few more programming languages under my belt.
My initial impulse was to go ahead and study for, and then take, all of the relevant GRE subject tests (Mathematics, Psychology and Computer science anyway) and apply to cognitive science and computer science programs with strong AGI groups. I've found that the latter option is more difficult that I had realized, which is somewhat disheartening, as my future planning model does not seem to work in such an underdeveloped field, and there is no easy to find established standard source for finding out which schools/programs to look at. I also realized that the former option does not necessarily conform to my research interestes as much as I would like it to, this being a fairly long term commitment.
Perhaps I lack the knowledge to successfully evaluate AGI programs; perhaps in the case of this particular area getting a PhD is not the best option; perhaps if I were more knowledgeable or wiser I might be better able to navigate where to go next, but I seem to be at a loss here. So; I come to you, fellow Less Wrongians, in search of guidance. Can any of you help to point me (and hopefully plenty of others) in the right (or at least less wrong) direction?