@Eliezer: what 'evil' person has ever admitted to being 'evil'? I'm not talking about the petty thief who is sorry for his actions, but the bin ladens or stalins of the world who are convinced that the end justify their means and they are really doing good for their people/country/future/god/whatever. ---- Rand's objectivism and capitalism are criticized by people who reflexively see 'selfish' and equate that with greed and all the problems with capitalism. But those critics are delusional or they just don't understand human nature and our built-in modules for self-interest. And what altruism we do have is reciprocal, that doesn't make it any less 'good', it just makes it a 'good' form of self-interested behavior that benefits both or many parties and allows for societies instead of small warring bands.

Socialism is bad because it doesn't work well--capitalism is bad because it works too well. Socialism goes against our natural instincts, capitalism in its unregulated form amplifies those natural instincts to unsustainable levels. And truly, all our economies in this world are mixed economies anyway, so that should tell you something about a 'one true way'.

Again referring to my previous post, I think most of the problems are not capitalistic ideals, but the money systems that warp those ideals. Money is important, because bartering doesn't scale and is inefficient, and lacking a post-scarcity reality where everything is abundant, money will not go away. But in its current form it is far from optimal for maximum friendliness for all.

I think when most people complain about capitalism it has more to do with the monetary policies and banking systems of certain implementations of capitalism then with the capitalist ideals themselves.

For instance, fiat money that is created by the governing body and interest charged on the lending of that money is a specific condition that allows for wealth inequalities and the aggregation of power. I think alternative currencies have an absolutely amazing potential to change society but governments violently suppress any competition that will result in them losing control and power over the system.

Interest was lauded by the founders of the USA and Benjamin Franklin himself remarked that it is the 8th wonder of the world, but as fascinating as compound interest is, it ultimately creates cycles of debt and unsustainable behavior.

An example case that is cited often is the use of alternative currencies in the towns of Spain during their civil war. When banks shut down and money was not available, the towns began to fall apart and enter deep depressions. As a response, the towns took the initiative to create local exchange trading systems, and almost immediately their situations began to improve and prosperity started to return to the towns. The governing bodies, deeply afraid of losing control, shut these systems down, choosing to purposefully let them fall into a depressed state rather than lose power.

------

Now, if you believe that friendly AGI is around the corner and will result in the singularity occurring sometime before our current social systems collapse, then you probably do not want to rock the boat. You would not care for sustainability, you would want to keep accelerating innovation at all costs and keep the system as is till our friendly AGI savior arrives to save the world.

If, on the other hand, you are skeptical that such a singularity is inevitable or you weigh the probabilities higher that our social systems which are allowing such accelerating innovations will collapse before such a singularity arrives, then thinking about sustainability starts to become really important.

A new method of 'lie detection' is being perfected using functional near infrared imaging of the prefrontal cortex:

http://www.biomed.drexel.edu/fNIR/Contents/deception/

In this technique the device actually measures whether or not a certain memory is being recalled or is being generated on the spot. For example, if you are interrogating a criminal who denies ever being at a crime scene, and you show them a picture of the scene, you can deduce whether he/she has actually seen it or not by measuring if their brain is recalling some sensory data from memory or newly creating and storing it.

What you are getting at is that the ends justify the means only when the means don't effect the ends. In the case of a human as part of the means, the act of the means may effect the human and thus effect the ends. In summary, reflexivity is a bitch. This is a reason why social science and economics is so hard--the subjects being modeled change as a result of the modeling process.

This is a problem with any sufficiently self-reflective mind, not with AIs that do not change their own rules. A simple mechanical narrow AI that is programmed to roam about collecting sensory data and to weigh the risk of people dying due to traffic collisions, then stepping in only to minimize the number of deaths, would be justified if it happens to allow or even cause the smaller number of deaths.

The concept of corruption doesn't exist in this context, the act is just a mechanism. A person can transition from an uncorrupted state to a corrupted state only because the rules governing the person's behavior is subject to modification in such a complex fashion as to occur even under the radar of the person it is happening to, because the person is the behavior caused by the rules, and when the rules change the person changes. We are not in as much control as we would like to think.

When the eastern religions preach the ego is the root of all our problems, they may be more right than we give them credit for. Ego is self-identity, which arises out of the ability to introspect and separate the aggregate of particles constituting 'I' with the rest of the particles in the environment. How would you go about building an AGI that doesn't have the false duality of self and non-self? Without ego corruption does not exist.

Imagine instead of an embodied AGI, or even a software AGI running on some black box computational machine sitting in a basement, the friendly AGI takes the form of an intelligent environment, say a superintelligent house. In the house there exists safeguards that disallows any unfriendly action. The house isn't conscious, it just adds a layer of friendliness on top of harsh reality. This may be a fruitful way of thinking about friendliness that avoids all the messy reflexivity.

Fun stuff this. I am enjoying these discussions.

Sorry for the triple post, one more addition. Larry Lessig just gave a lecture on corruption and the monetary causes of certain types of corruption prevalent in our society.

http://www.lessig.org/blog/2007/10/corruption_lecture_alpha_versi_1.html

Short addition to my previous post.

I've been thinking about how to apply the notion of recursive self-improvement to social structures instead of machines. I think it actually offers (though non-intuitively) a simpler case to think about friendliness optimization. If anyone else is interested feel free to email me. I'm planning on throwing up a site/wiki about this topic and may need help.

haig51 AT google mail

That is why systems consisting of checks and balances were eventually created (ie. democracy). Such social systems try to quell the potential for power aggregation and abuse, though as current events show, there will always be ways for power hungry people to game the system (and the best way to game the system is to run it and change the rules in your favor, creating the illusion that you still abide by the rules).

I always felt that the best system would be one of two extremes: 1.) a benevolent dictator (friendly superintelligence?) or 2.) massively decentralized libertarian socialism (or similar).

Notice that its an all or nothing dichotomy based on the absolute 'goodness' of a potential benevolent dictator--meaning that if it is possible to have an absolutely perfect benevolent dictator, than it would be best to concentrate power with it, but an absence of perfection would then require the exact opposite and to spread power out as wide as possible.

Someone contemplating changing the world for the better (or best) would necessarily need to decide which camp they fall in, #1 or #2. If #1 (like EY I presume), your most important duty would be to make the creation of this benevolent dictator your highest priority. If in camp #2 (like myself), your highest priority would be to create a system that uses the knowledge of human cognition, biases, and tendencies to diffuse power aggregation/abuse while trying to maximize the pursuit of happiness.

I think work on both can be done concurrently and may be complementary. Working on #2 might help keep the world from blowing up until #1 can be completed, and work on #1 can give insights into how to tune #2 (like EY's writings inspiring me to work on #2 etc.).

Given a choice, we would all want #1 as soon as possible, but being a pragmatist, #2 might be the more fruitful position for most people.

Did Einstein try to do the impossible? No, yet looking back it seems like he accomplished an impossible (for that time) feat doesn't it. So what exactly did he do? He worked on something he felt was: 1.) important, and probably more to the point, 2.) passionate about.

Did he run the probabilities of whether he would accomplish his goal? I don't think so, if anything he used the fact that the problem has not been solved so far and the problem is of such difficulty only to fuel his curiosity and desire to work on the problem even more. He worked at it every day because he was receiving value simply by doing the work, from being on the journey. He couldn't or wouldn't want to be doing anything else (patent clerk payed the bills, but his mind was elsewhere).

So instead of worrying about whether you are going to solve an impossible problem or not, just worry about whether you are doing something you love and usually if you are a smart and sincere person, that thing you love will more often than not turn out to be pretty important.

Ben Franklin wrote something relevant when talking about playing games: "...the persons playing, if they would play well, ought not much to regard the consequence of the game, for that diverts and makes the player liable to make many false open moves; and I will venture to lay it down for an infallible rule, that, if two persons equal in judgment play for a considerable sum, he that loves money most shall lose; his anxiety for the success of the game confounds him. Courage is almost as requisite for the good conduct of this game as in a real battle; for, if he imagines himself opposed by one that is much his superior in skill, his mind is so intent on the defensive part, that an advantage passes unobserved.”

I think most people's feedback threshold requires some return on their efforts in a relatively short time period. It takes monk-like patience to work on something indefinitely without any intermediary returns. So then, I don't think the point in contention is whether people are willing to make extraordinary effort, it is whether they are willing to make extraordinary effort without extraordinary returns in a time span relative to their feedback threshold. Even in eastern cultures where many people believe that enlightenment in the strong sense is possible by meditating your whole life, there is a reason why there are only a few practicing monks.

On the existential question of our pointless existence in a pointless universe, my perspective tends to oscillate between two extremes:

1.) In the more pessimistic (and currently the only rationally defensible) case, I view my mind and existence as just a pattern of information processing on top of messy organic wetware and that is all 'I' will ever be. Uploading is not immortality, it's just duplicating that specific mind pattern at that specific time instance. An epsilon unit of time after the 'upload' event that mind pattern is no longer 'me' and will quickly diverge as it acquires new experiences. An alternative would be a destructive copy, where the original copy of me (ie. me that is typing this right now) is destroyed after or at the instance of upload. Or I might gradually replace each synapse of my brain one by one with a simulator wirelessly transmitting the dynamics to the upload computer until all of 'me' is in there and the shell of my former self is just discarded. Either way, 'I' is destroyed eventually--maybe uploading is a fancier form of preserving ones thoughts for posterity as creating culture and forming relationships is pre-singularity, but it does not change the fact that the original meatspace brain is going to eventually be destroyed, no matter what.

The second case, what I might refer to as an optimistic appeal to ignorance, is to believe that though the universe appears pointless according to our current evidence, there may be some data point in the future that reveals something more that we are ignorant to at the moment. Though our current map reveals a neutral territory, the map might be incomplete. One speculative position taken directly from physics is the idea that I am a Boltzmann Brain. If such an idea can be taken seriously (and it is) than surely there are other theoretically defensible positions where my consciousness persists in some timeless form one way or another. (Even Bostrom's simulation argument gives another avenue of possibility)

I guess my two positions can be simplified into:
1.) What we see is all there is and that's pretty fucked up, even in the best case scenario of a positive singularity.

2.) We haven't seen the whole picture yet, so just sit back, relax, and as long as you have your towel handy, don't panic.