Ones that seem novel and valuable, either by personal discussion or email.
Also, I had expected that SI people monitored LW discussions, not just for critiques, but also for new ideas in general
I read most such (apparently-relevant from post titles) discussions, and Anna reads a minority. I think Eliezer reads very few. I'm not very sure about Luke.
(or I simply do not know enough physics to know why Eliezer is wrong, which I think is pretty unlikely but not totally discountable).
How much physics do you know?
I tentatively accept Eliezer's metaethics, considering how unlikely it is that there will be a better one (maybe morality is in the gluons?)
What about this thread?
The model popular here is that of 'expected utility maximizer', and the 'utility function' is defined on the real world.
I think this is a bit of a misperception stemming from the use of the "paperclip maximizer" example to illustrate things about instrumental reasoning. Certainly folk like Eliezer or Wei Dai or Stuart Armstrong or Paul Christiano have often talked about how a paperclip maximizer is much of the way to FAI (in having a world-model robust enough to use consequentialism). Note that people also like to use the AIXI framework as a model, and use it to talk about how AIXI is set up not as a paperclip maximizer but a wireheader (pornography and birth control rather than sex and offspring), with its utility function defined over sensory inputs rather than a model of the external world.
For another example, when talking about the idea of creating an AI with some external reward that can be administered by humans but not as easily hacked/wireheaded by the AI itself people use the example of an AI designed to seek factors of certain specified numbers, or a proof or disproof of the Riemann hypothesis according to some internal proof-checking mechanism, etc, recognizing the role of wireheading and the difficulty of specifying goals externally rather than using simple percepts and the like.
Rain (who noted that he is a donor to SIAI in a comment) and HoldenKarnofsky (who wrote the post) are two different people, as indicated by their different usernames.
1) Most criticism of key ideas underlying SIAI's strategies does not reference SIAI, e.g. Chris Malcolm's "Why Robots won't rule" website is replying to Hans Moravec.
2) Dispersed criticism, with many people making local points, e.g. those referenced by Wei Dai, is still criticism and much of that is informed and reasonable.
3) Much criticism is unwritten, e.g. consider the more FAI-skeptical Singularity Summit speaker talks, or takes the form of brief responses to questions or the like. This doesn't mean it isn't real or important.
4) Gerrymandering the bounds of "informed criticism" to leave almost no one within bounds is in general a scurrilous move that one should bend over backwards to avoid.
5) As others have suggested, even within the narrow confines of Less Wrong and adjacent communities there have been many informed critics. Here's Katja Grace's criticism of hard takeoff (although I am not sure how separate it is from Robin's). Here's Brandon Reinhart's examination of SIAI, which includes some criticism and brings more in comments. Here's Kaj Sotala's comparison of FHI and SIAI. And there are of course many detailed and often highly upvoted comments in response to various SIAI-discussing posts and threads, many of which you have participated in.
I can represent a rigid prohibition against lying using time-relative lexicographic preferences or hyperreals, e.g. "doing an act that I now (at t1) believe has too high a probability of being a lie has infinite and overriding disutility, but I can do this infallibly (defining the high disutility act to enable this), and after taking that into account I can then optimize for my own happiness or the welfare of others, etc."
All well and good for t1, but then I need a new utility function for the next moment, t2, and places infinite weight on lying at t2 but not at t1. The indexical description of the utility function hides the fact that we need a different ranking of consequences for most every moment and situation. I can't have a stable "Kantian utility function" that values weightings over world-histories and is consistent over time.
There are also some problems with the definition of acts and epistemic procedures such that one can have 100% certainty that one is not violating the deontological rules (otherwise they override any other lesser consequences).
Also got Amanda Knox right. As others said, he made bad predictions about transhumanist technologies as a teen, although he would say this was before he benefited from learning about rationality and disavowed them (in advance).
View more: Next


Subscribe to RSS Feed
Kagan understands these issues much better than you suggest, and than most Less Wrongers. I say this based on reading his academic work and long experience with LW. The snark is unwarranted and looks silly.