One reason we did not go travelling might have been a resource constraint, perhaps money but also a limited ability to plan good trips because of distraction or knowledge should be counted as a limitation of planning resources.

That aside, people still have multiple drives which are not really goals, and we sort of compromise amongst these drives. The approach the mind takes is not always the best.

In people, it's really those mid-brain drives that run a lot of things, not intellect.

We could try to carefully program in some lower-level or more complex sets of "drives" into an AI. The "utility function" people speak of in these threads is really more like an incredibly overpowering drive for the AI.

If it is wrong, then there is no hedge, check or diversification. The AI will just pursue that drive.

As much as our minds often .take us in the wrong direction with our drives, at least they are diversified and checked.

Checks and diversification of drives seem like an appealing element of mind design, even at significant cost to efficiency at achieving goals. We should explore these options in detail.

One reason we did not go travelling might have been a resource constraint, perhaps money but also a limited ability to plan good trips because of distraction or knowledge should be counted as a limitation of planning resources.

That aside, people still have multiple drives which are not really goals, and we sort of compromise amongst these drives. The approach the mind takes is not always the best.

In people, it's really those mid-brain drives that run a lot of things, not intellect.

We could try to carefully program in some lower-level or more complex sets of "drives" into an AI. The "utility function" people speak of in these threads is really more like an incredibly overpowering drive for the AI.

If it is wrong, then there is no hedge, check or diversification. The AI will just pursue that drive.

As much as our minds often .take us in the wrong direction with our drives, at least they are diversified and checked.

Checks and diversification of drives seem like an appealing element of mind design, even at significant cost to efficiency at achieving goals. We should explore these options in detail.

I wonder what kinds of insights would be available for a Superintelligence that might override it's goal-content integrity goal. To begin with, the other instrumental goals, if they emerged, would themselves tamper with its allocation of resources.

Now consider humans: sometimes we think our goals are "travel a lot" but actually they turn out to be "be on a hammock two hours a day and swim once a week".

By trial and error, we re-map our goals based on what they feel from the inside, or rewards.

Which similar processes might an AGI do?

I am not that confident in the convergence properties of self-preservation as instrumental goal.

It seems that at least some goals should be pursued ballistically -- i.e., by setting an appropriate course in motion so that it doesn't need active guidance.

For example, living organisms vary widely in their commitments to self-preservations. One measure of this variety is the variety of lifespans and lifecycles. Organisms generally share the goal of reproducing, and they pursue this goal by a range of means, some of which require active guidance (like teaching your children) and some of which don't (like releasing thousands of eggs into the ocean).

If goals are allowed to range very widely, it's hard to believe that all final goals will counsel the adoption of the same CIVs as subgoals. The space of all final goals seems very large. I'm not even very sure what a goal is. But it seems at least plausible that this choice of CIVs is contaminated by our own (parochial) goals, and given the full range of weird possible goals these convergences only form small attractors.

A different convergence argument might start from competition among goals. A superintelligence might not "take off" unless it starts with sufficiently ambitious goals. Call a goal ambitious if its CIVs include coming to control significant resources. In that case, even if only a relatively small region in goal-space has the CIV of controlling significant resources, intelligences with those goals will quickly be overrepresented. Cf. this intriguing BBS paper I haven't read yet.

If your theory of time is 4-dimensionalist, then you might think the past people are 'still there,' in some timeless sense, rather than wholly annihilated. Interestingly, you might (especially if you reject determinism) think that moving through time involves killing (possible) futures, rather than (or in addition to) killing the past.

Any inference about "what sort of thingies can be real" seems to me premature. If we are talking about causality and space-time locality, it seems to me that the more parsimonious inference regards what sort of thingies a conscious experience can be embedded in, or what sort of thingies a conscious experience can be of.

The suggested inference seems to privilege minds too much, as if to say that only the states of affairs that allow a particular class of computation can possibly be real. (This view may reduce to empiricism, which people like, but stated this way I think it's pretty hard to support! What's so special about conscious experience?)

EDIT: Hmm, here is a rather similar comment. Hard to process this whole discussion.

EDIT EDIT: maybe even this comment is about the same issue, although its argument is being applied to a slightly different inference than the one suggested in the main article.

World-destroying black hole caused by LHC. Autism through vaccination. Cancer from low intensity radio waves (i.e. a cell phone rather than a radar station). A meteorite hitting your house. A plane crashing into your house if you don't live in a landing vector of an airport. Terrorists capturing the plane you are on if you fly rarely.

This is untrue as a general rule, though it can be closer or farther from the truth depending on market conditions.

To see why, imagine that every month you buy a supply of fizzlesprots from Acme Corp. Today is the first of February, so you eagerly rush off to buy your monthly fix. But wait! The government has just imposed a tax on all fizzlesprot purchases. Curses! Now you'll have to pay even more, because Acme Corp will just pass the whole tax on to you.

Now change "fizzlesprot" to "labor" and "Acme Corp" to "employee". Huh? You're an employer, not an employee? My world is turned upside down! Could it be that the narrative where You bear the full brunt of every tax and They end up paying nothing is wrong?

In fact, whenever an economic transaction is taxed, the buyers and the sellers split the tax based on who is more eager to buy or sell. Labor is no different. It's possible that, empirically, the employee usually pays more of a labor tax than the employer, but this is by no means guaranteed and I would personally expect the proportion to vary significantly between labor market segments.

(Wikipedia's article on tax incidence claims that employees pay almost all of payroll taxes, but cites a single paper that claims a 70% labor / 30% owner split for corporate income tax burden in the US, and I have no idea how or whether that translates to payroll tax burden or whether the paper's conclusions are generally accepted.)

For more details, consult your nearest introductory economics textbook.

The USA is not the best place to earn money.2 My own experience suggests that at least Japan, New Zealand, and Australia can all be better. This may be shocking, but young professionals with advanced degrees can earn more discretionary income as a receptionist or a bartender in the Australian outback than as, say, a software engineer in the USA.

As a side question, when did a receptionist or bartender become a "professional"? Is "professional" just used as a class marker, standing for something like "person with a non-vocational 4-year college degree"?

Or is the idea that one is a professional because one is in some sense a software engineer (e.g.), even while employed as a receptionist or bartender?

Great article!

I'm currently trying to figure out my personal optimal employment. (I'm a German CS student and will get my degree next year. Most importantly, I want to leave the country and live in some English-speaking country. I can't stand the cultural isolation any longer.) I was already considering Australia and you have just made it look a lot more attractive.

The specific job you provided, however, isn't right for me. Remote areas are exactly where I don't wanna be right now. I've lived in villages and small towns most of my life and I'm sick of them. But I am very interested in hearing about other options like this, especially if they would involve less telephones or similar activities, as Alicorn already mentioned.