Recently, we saw an awesome demonstration of great futility, Quantitative Easing II.

Do not use political examples unless absolutely necessary, and even then - just don't. You're automatically generating hostile reaction in half of your audience. (and in this case, you're just wrong, and clearly lack basic understanding of economics monetary policy, but even if you were right, just don't use such examples)

1- Most modern corporations would have been founded with the explicit goal of making money.

2- Why aim for "equitable" propserity?

3- Are you discussing prosperity in absolute or relative terms?

Not commenting on the A.I issue, as I don't know enough to meaningfully contribute.

Many smart people have voiced opinions against such blind following of metrics in government, but almost all organizations, once beyond the grip of the founders fall into some such pattern. Compassionate mystic movements become rigid churches. Political parties for eg. pay more attention to lip service to issues, pomp, show and mind killing than to actual issues. Companies seek to make money at the expense of creating actual value, forgetting that money is only a symbol of value.

It would be useful to see a link to a peer-reviewed study supporting each of these summaries. I would be surprised (and hence informed!) if they all turned out to be clearly true. My understanding of the sociology of religion is actually that over time religious movements become more liberal and cosmopolitan, in opposition to their early doctrines and the needs of their original members and later members similar to the founders, in the sociology of religion this is basically accepted as a driver behind the formation of schismatic religious movements and the theoretical arguments are generally about the causative factors behind the change in membership over time. Some argue that the increasing number of church members who want cosmopolitan doctrines is simply regression towards the mean (away from the atypical needs of early members) and others argue that the doctrines play a causal role in increasing the worldly success of church members. I suspect it is a little of both.

Also, I don't think companies ever have the purpose of simply creating value "in general" so it can't be a lost purpose. My understanding is that companies are created to solve the relatively well defined and more limited problem of doing something so unique and positive that they create scarce value whose combined scarcity and value are so dramatic that people are willing to fork money over for the products or services. There are lots of ways to create value, like providing free internet services or picking up garbage by the side of the road. No one does these sorts of things in private industry without some angle. The trick is for the company to do something that is clearly valuable and also that compels people to pay them for it based on their ability to deprive users of the benefits of their work unless users pay them. This sometimes "feels immoral" in small groups of people, but is cybernetically necessary in large anonymous groups where services have high marginal costs, which is one of many reasons that coordination is hard.

The political economy of democracies is even more complicated, with (1) difficult to measure psychological benefits being allocated to radicalized volunteer activists, (2) selfishly rational and highly strategic lobbying efforts coming from all sides, (3) substantial voter apathy, and (4) a gaping free rider problem right at the center because people who don't contribute to good government cannot be deprived of most of its benefits. I expect people who are experts in this domain to be doing a lot of complicated stuff that I don't currently understand and so when you casually dismiss "pomp, show, and mind killing" in the outer forms of politics, and offer no supporting citation, I'm tempted to think that you're expecting inferential distances to be small and making a rookie rationality error because all of politics isn't "doing what you naively think it should do".

The same paragraph continued...

A lot of people have bemoaned the brainpower that is moving into finance. And something even more repugnant, there is an entire economy thriving around the war on drugs, with everyone in on the cut.

I've heard people bemoaning that fact, so I agree with your front line evidence: yes people do bemoan this fact.

But this sort of begs the question about whether the bemoaners are actually right. I consider myself moderately smart, and I've thought about getting a formal degree in economics and going into finance both to learn more about adaptively self-regulating processes and because it seems like an area that needs to be "done right" if the rest of the world is to function well. Most of the reason I never followed through on these impulses is that I wouldn't plan on staying in finance after I'd learned what I came to learn and the business culture of finance doesn't seem particularly nurturing, especially for newbies. I can easily imagine people similar to me in some respects but different from me in other respects who go into finance for "basically good reasons" and thereby do enormous good.

Moreover, while it seems to be illiberal to punish people for putting whatever they want into their own bodies, the fact that so many people do engage in pharmaceutical wireheading seems kind of tragic in some respects. I could imagine reasonably good paternalistic arguments for making certain drugs illegal, and I would expect if the paternalistic argument carried the day, then it would be necessary to make the enforcement systems self sustaining if you actually wanted to decrease drug use in the long term. The only way it would be dramatically repugnant for the war on drugs to be self sustaining is if the war on drugs was itself totally repugnant so that its failure to sustain itself would be an obvious boon. Honest policy discussions should not be one-sided.

For what it's worth, I basically agree with your bottom line conclusion that there is a real potential for a swiftly arising Unfriendly AGI.

I think there are reasonable defenses of the so-called Scary Idea (maybe the "Scary Hypothesis" would be a better name?), and I think that it improves the odds of the world having a positive outcome if "algorithm literate" net activists occasionally descend on AGI researchers and demand that they give an accounting of themselves in light of the Scary Hypothesis. The trick is that the truth value of the Scary Hypothesis probably changes based on the socio-technological context and the details of each specific project. In light of this, a generic policy of publicly sourced demand for a positive theory of particular safety from each AGI project (maybe every year or two?) seems to me like a prudent and reasonably cheap political safety measure :-)

The underlying idea of Artificial Intelligences being likely to come from and work in corporate and financial situations, which is likely to lead to a very specific kind of mild friendliness/unfriendliness (i.e. working at goals which benefit some humans and are very unlikely to end the world, but which are not "good altruistic goals" that one would naively associate with a positive singularity) is something that I have thought about for some time and agree with wholeheartedly.

I can't speak to your views on, for example, quantitative easing. I don't understand the subject.

I do notice that there were some grammatical errors in the post ("more smarter") and the formatting is a little odd and not as polished as most posts I see.

I think that, had this post started out with an abstract in the discussion section, it would have received several upvotes and been worth promoting to the main page. As it is it feels in need of a rewrite or two.

everyone in on the cut

Confusing. Halfway between everyone "on the take" and "getting/taking a/their cut".

this link is messed up in the post

http://lesswrong.com/lw/1ws/the_importance_of_goodharts_law/

So the fear here doesn't appear to be that corporate and government GAIs won't accurately model what they have incentive to do. Especially if we're talking about a human-computer team we're not worried about a machine hacking the NYSE just to make the ticker for their stock go up. We're worried, I guess, that these teams will pursue what they have incentive to pursue really well. Having this worry seems to require a) that we distrust the free market generally and deny that the incentives in place are good for most people and b) that the problems we trace to corporate decision-making were wise given the incentives those decision makers had. Neither of these seem right to me.

When it comes to corporations the problem seems to me to be about incorrectly programming the incentive structure- which isn't a institutional or structural issue, it's a similar problem to FAI just with corporate/employee incentives instead of human values. In other words, the problem isn't people doing it, the problem is people doing it wrong.

Not sure if this applies to the general point or not...