Even if we accepted that the tool vs. agent distinction was enough to make things "safe", objection 2 still boils down to "Well, just don't build that type of AI!", which is exactly the same keep-it-in-a-box/don't-do-it argument that most normal people make when they consider this issue. I assume I don't need to explain to most people here why "We should just make a law against it" is not a solution to this problem, and I hope I don't need to argue that "Just don't do it" is even worse...
More specifically, fast forward to 2080, when any college kid with $200 to spend (in equivalent 2012 dollars) can purchase enough computing power so that even the dumbest AIXI approximation schemes are extremely effective, good enough so that creating an AGI agent would be a week's work for any grad student that knew their stuff. Are you really comfortable living in that world with the idea that we rely on a mere gentleman's agreement not to make self-improving AI agents? There's a reason this is often viewed as an arms race, to a very real extent the attempt to achieve Friendly AI is about building up a suitably powerful defense against unfriendly AI before someone (perhaps accidentally) unleashes one on us, and making sure that it's powerful enough to put down any unfriendly systems before they can match it.
From what I can tell, stripping away the politeness and cutting to the bone, the three arguments against working on friendly AI theory are essentially:
FWIW, I mostly agree with the rest of the article's criticisms, especially re: the organization's achievements and focus. There's a lot of room for improvement there, and I would take these criticisms very seriously.
But that's almost irrelevant, because this article argues against the core mission of SIAI, using arguments that have been thoroughly debunked and rejected time and time again here, though they're rarely dressed up this nicely. To some extent I think this proves the institute's failure in PR - here is someone that claims to have read most of the sequences, and yet this criticism basically amounts to a sexing up of the gut reaction arguments that even completely uninformed people make - AGI is probably a fantasy, even if it's not you won't be able to control it, so let's just agree not to build it.
Or am I missing something new here?
[1] Alright, to be fair, this is not a great summary of point 3, which really says that specialized AIs might help us solve the AGI problem in a safer way, that a hard takeoff is "just a theory" and realistically we'll probably have more time to react and adapt.
Is there anything that can be done about it?
I don't know how much of a problem it is, but there's definitely something that can be done about it: instead of a "dumb" karma count, use some variant of Pagerank on the vote graph.
In other words, every person is a node, every upvote that each person gets from another user is a directed edge (also signed to incorporate downvotes), every person starts with a base amount of karma, and then you iteratively update the user karma by weighting each inbound vote by the karma of the voter.
When I say "variant of Pagerank", I mean that you'd probably also have to fudge some things in there as well for practical reasons, like weighting votes by time to keep up with an evolving community, adding a bias so that a few top people don't completely control the karma graph, tuning the base karma that people receive based on length of membership and/or number of posts, weighting submissions separately from comments, avoiding "black hat SEO" tricks, etc. You know, all those nasty things that make Google a lot more than "just" Pagerank at web scale...
IMO doing something like this would improve most high traffic comment systems and online communities substantially (Hacker News could desperately use something like that to slow its slide into Reddit territory, for instance), though it would severely de-democratize them; somehow I doubt people around here would have much of a problem with that, though. The real barrier is that it would be a major pain in the ass to actually implement, and would take several iterations to really get right. It also might be difficult to retrofit an existing voting system with anything like that because sometimes they don't store the actual votes, but just keep a tally, so it would take a while to see if it actually helped at all (you couldn't backtest on the existing database to tune the parameters properly).
If someone asks the tool-AI "How do I create an agent-AI?" and it gives an answer, the distinction is moot anyways, because one leads to the other.
Given human nature, I find it extremely difficult to believe that nobody would ask the tool-AI that question, or something that's close enough, and then implement the answer...