Filter This week

Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Bill Gates: problem of strong AI with conflicting goals "very worthy of study and time"

46 ciphergoth 22 January 2015 08:21PM

Steven Levy: Let me ask an unrelated question about the raging debate over whether artificial intelligence poses a threat to society, or even the survival of humanity. Where do you stand?

Bill Gates: I think it’s definitely important to worry about. There are two AI threats that are worth distinguishing. One is that AI does enough labor substitution fast enough to change work policies, or [affect] the creation of new jobs that humans are uniquely adapted to — the jobs that give you a sense of purpose and worth. We haven’t run into that yet. I don’t think it’s a dramatic problem in the next ten years but if you take the next 20 to 30 it could be. Then there’s the longer-term problem of so-called strong AI, where it controls resources, so its goals are somehow conflicting with the goals of human systems. Both of those things are very worthy of study and time. I am certainly not in the camp that believes we ought to stop things or slow things down because of that. But you can definitely put me more in the Elon Musk, Bill Joy camp than, let’s say, the Google camp on that one.

"Bill Gates on Mobile Banking, Connecting the World and AI", Medium, 2015-01-21

New, Brief Popular-Level Introduction to AI Risks and Superintelligence

15 LyleN 23 January 2015 03:43PM

The very popular blog Wait But Why has published the first part of a two-part explanation/summary of AI risks and superintelligence, and it looks like the second part will be focused on Friendly AI. I found it very clear, reasonably thorough and appropriately urgent without signaling paranoia or fringe-ness. It may be a good article to share with interested friends.

[Link] An argument on colds

14 Konkvistador 18 January 2015 07:16PM

Source.

It's illegal to work around food when showing symptoms of contagious diseases. Why not the same for everyone else? Each person who gets a cold infects one other person on average. We could probably cut infection rates and the frequency of colds in half if sick people didn't come in to work.

And if we want better biosecurity, why not also require people to be able to reschedule flights if a doctor certifies they have a contagious disease?

Due to the 'externalities', the case seems very compelling.

Moving my commentary to a separate comment, so as to disambiguate votes on my commentary and the original argument.

Purchasing research effectively open thread

7 John_Maxwell_IV 21 January 2015 12:24PM

Many of the biggest historical success stories in philanthropy have come in the form of funding for academic research.  This suggests that the topic of how to purchase such research well should be of interest to effective altruists.  Less Wrong survey results indicate that a nontrivial fraction of LW has firsthand experience with the academic research environment.  Inspired by the recent Elon Musk donation announcement, this is a thread for discussion of effectively using money to enable important, useful research.  Feel free to brainstorm your own questions and ideas before reading what's written in the thread.

Superintelligence 19: Post-transition formation of a singleton

5 KatjaGrace 20 January 2015 02:00AM

This is part of a weekly reading group on Nick Bostrom's book, Superintelligence. For more information about the group, and an index of posts so far see the announcement post. For the schedule of future topics, see MIRI's reading guide.


Welcome. This week we discuss the nineteenth section in the reading guidepost-transition formation of a singleton. This corresponds to the last part of Chapter 11.

This post summarizes the section, and offers a few relevant notes, and ideas for further investigation. Some of my own thoughts and questions for discussion are in the comments.

There is no need to proceed in order through this post, or to look at everything. Feel free to jump straight to the discussion. Where applicable and I remember, page numbers indicate the rough part of the chapter that is most related (not necessarily that the chapter is being cited for the specific claim).

Reading: : “Post-transition formation of a singleton?” from Chapter 11


Summary

  1. Even if the world remains multipolar through a transition to machine intelligence, a singleton might emerge later, for instance during a transition to a more extreme technology. (p176-7)
  2. If everything is faster after the first transition, a second transition may be more or less likely to produce a singleton. (p177)
  3. Emulations may give rise to 'superorganisms': clans of emulations who care wholly about their group. These would have an advantage because they could avoid agency problems, and make various uses of the ability to delete members. (p178-80) 
  4. Improvements in surveillance resulting from machine intelligence might allow better coordination, however machine intelligence will also make concealment easier, and it is unclear which force will be stronger. (p180-1)
  5. Machine minds may be able to make clearer precommitments than humans, changing the nature of bargaining somewhat. Maybe this would produce a singleton. (p183-4)

Another view

Many of the ideas around superorganisms come from Carl Shulman's paper, Whole Brain Emulation and the Evolution of Superorganisms. Robin Hanson critiques it:

...It seems to me that Shulman actually offers two somewhat different arguments, 1) an abstract argument that future evolution generically leads to superorganisms, because their costs are generally less than their benefits, and 2) a more concrete argument, that emulations in particular have especially low costs and high benefits...

...On the general abstract argument, we see a common pattern in both the evolution of species and human organizations — while winning systems often enforce substantial value sharing and loyalty on small scales, they achieve much less on larger scales. Values tend to be more integrated in a single organism’s brain, relative to larger families or species, and in a team or firm, relative to a nation or world. Value coordination seems hard, especially on larger scales.

This is not especially puzzling theoretically. While there can be huge gains to coordination, especially in war, it is far less obvious just how much one needs value sharing to gain action coordination. There are many other factors that influence coordination, after all; even perfect value matching is consistent with quite poor coordination. It is also far from obvious that values in generic large minds can easily be separated from other large mind parts. When the parts of large systems evolve independently, to adapt to differing local circumstances, their values may also evolve independently. Detecting and eliminating value divergences might in general be quite expensive.

In general, it is not at all obvious that the benefits of more value sharing are worth these costs. And even if more value sharing is worth the costs, that would only imply that value-sharing entities should be a bit larger than they are now, not that they should shift to a world-encompassing extreme.

On Shulman’s more concrete argument, his suggested single-version approach to em value sharing, wherein a single central em only allows (perhaps vast numbers of) brief copies, can suffer from greatly reduced innovation. When em copies are assigned to and adapt to different tasks, there may be no easy way to merge their minds into a single common mind containing all their adaptations. The single em copy that is best at doing an average of tasks, may be much worse at each task than the best em for that task.

Shulman’s other concrete suggestion for sharing em values is “psychological testing, staged situations, and direct observation of their emulation software to form clear pictures of their loyalties.” But genetic and cultural evolution has long tried to make human minds fit well within strongly loyal teams, a task to which we seem well adapted. This suggests that moving our minds closer to a “borg” team ideal would cost us somewhere else, such as in our mental agility.

On the concrete coordination gains that Shulman sees from superorganism ems, most of these gains seem cheaply achievable via simple long-standard human coordination mechanisms: property rights, contracts, and trade. Individual farmers have long faced starvation if they could not extract enough food from their property, and farmers were often out-competed by others who used resources more efficiently.

With ems there is the added advantage that em copies can agree to the “terms” of their life deals before they are created. An em would agree that it starts life with certain resources, and that life will end when it can no longer pay to live. Yes there would be some selection for humans and ems who peacefully accept such deals, but probably much less than needed to get loyal devotion to and shared values with a superorganism.

Yes, with high value sharing ems might be less tempted to steal from other copies of themselves to survive. But this hardly implies that such ems no longer need property rights enforced. They’d need property rights to prevent theft by copies of other ems, including being enslaved by them. Once a property rights system exists, the additional cost of applying it within a set of em copies seems small relative to the likely costs of strong value sharing.

Shulman seems to argue both that superorganisms are a natural endpoint of evolution, and that ems are especially supportive of superorganisms. But at most he has shown that ems organizations may be at a somewhat larger scale, not that they would reach civilization-encompassing scales. In general, creatures who share values can indeed coordinate better, but perhaps not by much, and it can be costly to achieve and maintain shared values. I see no coordinate-by-values free lunch...

Notes

1. The natural endpoint

Bostrom says that a singleton is natural conclusion of long-term trend toward larger scales of political integration (p176). It seems helpful here to be more precise about what we mean by singleton. Something like a world government does seem to be a natural conclusion to long term trends. However this seems different to the kind of singleton I took Bostrom to previously be talking about. A world government would by default only make a certain class of decisions, for instance about global level policies. There has been a long term trend for the largest political units to become larger, however there have always been smaller units as well, making different classes of decisions, down to the individual. I'm not sure how to measure the mass of decisions made by different parties, but it seems like the individuals may be making more decisions more freely than ever, and the large political units have less ability than they once did to act against the will of the population. So the long term trend doesn't seem to point to an overpowering ruler of everything.

2. How value-aligned would emulated copies of the same person be?

Bostrom doesn't say exactly how 'emulations that were wholly altruistic toward their copy-siblings' would emerge. It seems to be some combination of natural 'altruism' toward oneself and selection for people who react to copies of themselves with extreme altruism (confirmed by a longer interesting discussion in Shulman's paper). How easily one might select for such people depends on how humans generally react to being copied. In particular, whether they treat a copy like part of themselves, or merely like a very similar acquaintance.

The answer to this doesn't seem obvious. Copies seem likely to agree strongly on questions of global values, such as whether the world should be more capitalistic, or whether it is admirable to work in technology. However I expect many—perhaps most—failures of coordination come from differences in selfish values—e.g. I want me to have money, and you want you to have money. And if you copy a person, it seems fairly likely to me the copies will both still want the money themselves, more or less.

From other examples of similar people—identical twins, family, people and their future selves—it seems people are unusually altruistic to similar people, but still very far from 'wholly altruistic'. Emulation siblings would be much more similar than identical twins, but who knows how far that would move their altruism?

Shulman points out that many people hold views about personal identity that would imply that copies share identity to some extent. The translation between philosophical views and actual motivations is not always complete however.

3. Contemporary family clans

Family-run firms are a place to get some information about the trade-off between reducing agency problems and having access to a wide range of potential employees. Given a brief perusal of the internet, it seems to be ambiguous whether they do better. One could try to separate out the factors that help them do better or worse.

4. How big a problem is disloyalty?

I wondered how big a problem insider disloyalty really was for companies and other organizations. Would it really be worth all this loyalty testing? I can't find much about it quickly, but 59% of respondents to a survey apparently said they had some kind of problems with insiders. The same report suggests that a bunch of costly initiatives such as intensive psychological testing are currently on the table to address the problem. Also apparently it's enough of a problem for someone to be trying to solve it with mind-reading, though that probably doesn't say much.

5. AI already contributing to the surveillance-secrecy arms race

Artificial intelligence will help with surveillance sooner and more broadly than in the observation of people's motives. e.g. here and here.

6. SMBC is also pondering these topics this week



In-depth investigations

If you are particularly interested in these topics, and want to do further research, these are a few plausible directions, some inspired by Luke Muehlhauser's list, which contains many suggestions related to parts of Superintelligence. These projects could be attempted at various levels of depth.

  1. What are the present and historical barriers to coordination, between people and organizations? How much have these been lowered so far? How much difference has it made to the scale of organizations, and to productivity? How much further should we expect these barriers to be lessened as a result of machine intelligence?
  2. Investigate the implications of machine intelligence for surveillance and secrecy in more depth.
  3. Are multipolar scenarios safer than singleton scenarios? Muehlhauser suggests directions.
  4. Explore ideas for safety in a singleton scenario via temporarily multipolar AI. e.g. uploading FAI researchers (See Salamon & Shulman, “Whole Brain Emulation, as a platform for creating safe AGI.”)
  5. Which kinds of multipolar scenarios would be more likely to resolve into a singleton, and how quickly?
  6. Can we get whole brain emulation without producing neuromorphic AGI slightly earlier or shortly afterward? See section 3.2 of Eckersley & Sandberg (2013).
If you are interested in anything like this, you might want to mention it in the comments, and see whether other people have useful thoughts.

How to proceed

This has been a collection of notes on the chapter.  The most important part of the reading group though is discussion, which is in the comments section. I pose some questions for you there, and I invite you to add your own. Please remember that this group contains a variety of levels of expertise: if a line of discussion seems too basic or too incomprehensible, look around for one that suits you better!

Next week, we will talk about the 'value loading problem'. To prepare, read “The value-loading problem” through “Motivational scaffolding” from Chapter 12The discussion will go live at 6pm Pacific time next Monday 26 January. Sign up to be notified here.

Optimal eating (or rather, a step in the right direction)

4 c_edwards 19 January 2015 01:35AM

Over the past few months I've been working to optimize my life.  In this post I describe my attempt to optimize my day-to-day cooking and eating - my goal with this post is to get input and to offer a potential template for people who aren't happy with their current cooking/eating patterns.  I'm a) still pretty new to LW, and b) not a nutritionist; I am not claiming that this is optimal, only that it is a step in the right direction for me.  I'd love suggestions/advice/feedback.

Goal:

How do I quantify a successful cooking/eating plan?

Healthy

"Healthy" is a broad term.  I'm not interested in making food a complicated or stressful component of my life - quite the opposite.  Healthy means that I feel good, and that I'm providing my body with a good mix of building blocks (carbs, proteins, fats) and nutrients.  This means I want most/all meals to include some form of complex carbs, protein, and either fruits or veggies or both.  As I'm currently implementing an exercise plan based on the LW advice for optimal exercising, I'm aiming to get ~120 grams of protein per day (.64g/lb bodyweight/day).  There seems to be a general consensus that absorption of nutrients from whole foods is a) higher, and b) less dangerous, so when possible I'm trying to make foods from basic components instead of buying pre-processed stuff.

I have a health condition called hypoglycemia (low blood sugar) that makes me cranky/shaky/weak/impatient/foolish/tired when I am hungry, and can be triggered by eating simple sugars.  So, for me personally, a healthy diet includes rarely feeling hungry and rarely eating simple sugars (especially on their own - if eaten with other food the effect is much less severe).  This also means trying to focus on forms of fruit and complex carbs that have low glycemic indexes (yams are better than baked potatoes, for example).  I would guess that these attributes would be valuable for anyone, but for me they are a very high priority.

I'm taking some advice from the "Exos" (formerly Core Performance) fitness program, as described in the book Core performance essentials. One of the suggestions from this that I'm trying to use here (aside from the above complex carb+protein+fruit/veg meal structure) is to "eat the rainbow every day" - that is, mix up the fruits and veggies you eat, ideally getting as many colors per day as possible.  I'm also taking advice from the (awesome) LW article on increasing longevity: "eat fish, nuts, eggs, fruit, dark chocolate."

When possible I'm trying to focus on veggies that are particularly nutrient dense - spinach, bok choy, tomatoes, etc.  I am (for now) avoiding a few food products that I have heard (but have not yet confirmed!) are linked to potential health issues: tofu, whey proteins.  Note that I do not trust my information on the potential risks of these foods, but as neither of these are important to my diet anyways, I have put researching them as a low priority compared to everything else I want to learn.

So to recap: don't stress about it, but try to do complex carbs, proteins (120g/day for me), fruits, and veggies in every meal, avoid sugars where possible (although dark chocolate is good).  Fish, nuts and eggs are high priority proteins.

Cheap

I'm on a fairly limited budget.  This means trying to focus on the seasonal fruits and veggies (which are typically cheaper, and as an added bonus are likely healthier than the same fruit/veggie when out of season), aiming for less expensive meats, and not trying to eat organically (probably worth a separate discussion of organic vs not, meat vs not).  This also means making my own foods when the price benefit is high and the time cost is low.  I often make my own breads, for example (using a breadmaker) - it takes about 10 minutes of my time, directly saves me about 3+ dollars or so compared to an equivalent quality loaf of bread (many breads can be made for ~$.50-1$), plus saves me either the time of shopping multiple times per week to obtain fresh bread or the grossness of eating bread that I've frozen to keep it from molding.  Additionally, my budget means that I prefer that my weekly meal plan not depend on eating out or buying pre-made foods.

Quick

While I'm on a fairly limited monetary budget, I'm on a very limited time budget.  Cooking can be fun for me, but I prefer that my weekly schedule not REQUIRE much time - I can always replace a quick meal with a longer fun one if I feel like it.


The Plan

My general approach is split my meals between really quick-and-easy (like chickpeas, canned salmon, and olive oil over prewashed spinach with an apple or two on the side) and batch foods where a somewhat longer time investment is split over many nights (like lentil stew in a crockpot).

To keep myself reasonable full I need about 6-7 meals per day: breakfast, snack, lunch, (optional snack depending on schedule), post-workout snack, dinner, snack.  These don't all need to be large, but I'm unhappy/unproductive without something for each of those meals, so I might as well make it easy to eat them.

In general I've found the following system to fulfill my criteria of success (healthy, cheap, quick), and it's been much less stressful to have a general plan in place - I can more easily figure out my shopping list, and it's not hard to ensure I always have food ready when I need it.

Breakfasts

Quick and easy is the key here.  I typically have either

 

  1. Yogurt with sunflower seeds and/or nuts, a handful of rolled oats (yes, uncooked, but add a bit of water at the end to make them tolerable), and sometimes some fruit on top.  Add honey for sweetener as needed (I typically don't do to hypoglycemia).
  2. Bread (often homemade, but whatever floats your boat) with some peanut butter on top, a banana or other fruit item on the side.
  3. (if I have the time) Scrambled eggs mixed with chopped broccoli or bell peppers, bread, and a piece of fruit.
(also a big glass of water, which everyone seems to think is important)(also coffee, although I'm considering transitioning to a different caffeine source.

Lunch

 

I have three "batch" meals here (I make enough for 3+lunches, so I cook lunches ~twice a week):

 

  1. salmon mash plus "spinach salad" (spinach with olive oil and either lemon juice or balsamic vinegar), fruit item.  salmon mash is a mix of cooked rice, canned salmon, black olives (for flavor - not sure that they're useful nutritionally), canned black or garbanzo beans, pasta sauce.  It sounds disgusting, but I find it pretty decent, and it's very cheap and filling, and super balanced in terms of carbs and proteins.  I do proportions of 1 cup rice, 1 large can salmon, 1-2 cans beans, 1/2 can black olives, 1/2 can pasta sauce (typically I do a double batch, which lasts me about 4-5 lunches.  Your mileage may vary)
  2. Baked yams and boneless skinless chicken breasts plus spinach salad or other veggies, fruit item
  3. pasta salad: pasta, raw chopped broccoli, tomatoes (grape/cherry tomatoes are easiest), chopped bell peppers, sliced ham, olives (for flavor again - not important nutritionally, I think), and some olive oil (you could use Caesar salad dressing if you like more flavor).  
If I haven't prepped a batch lunch, I just put salmon and beans on top of spinach, add a little olive oil, and throw in a slice of bread and a fruit on the side. Alternately, PBJ plus veggie and fruit.

 

Dinner

I aim to make one batch dinner per week and have it last for 4-5 meals, and then have several quick-and-easy dinners to fill the gap (this also makes it easy to accommodate dinners out or food related social gatherings).

Some ideas for Batch Dinners (crock pots are your friends here):

 

  • Lentil stew, bread, sliced carrots or bell peppers, fruit item (apple, banana, grapefruit, whatever).  That lentil soup recipe is ridiculously cheap, healthy, and quite tasty.
  • The potato-and-cabbage based rumpledethumps recipe (which freezes very well - make a huge batch and throw half of it in the freezer), plus a meat of some sort, a fruit item and maybe a vegetable something 
  • Other crock pot soups: chicken tortilla soup, chili, stew.  Add a veggie on the side, a fruit item, and maybe a slice of bread.
  • Large stirfry (these often take a bit longer than crock pot meals), rice or noodles, fruit on the side.
Note that since I only make one batch dinner per week, those bullets are sufficient to cover a month (and depending on what your tolerance for repetition is, that might be enough for years).

Some ideas for quick-and-easy dinners:
  • Salad made from salad greens, some form of precooked meat (salmon is good), beans, maybe sliced avacado and tomato, maybe sunflower seeds.
  • Rice/pasta; scrambled/cooked eggs or baked chicken; munching veggie like carrots, raw broccoli, bell pepper; fruit item.  Note on chicken: while there is a reasonably large elapse time from start to finish, your involvement doesn't need to take long.  Typically I have a bunch of boneless skinless chicken breasts in the freezer - pull one out, throw it in a ziplock with soy sauce, garlic powder, ginger (or whatever other marinade you prefer), put the ziplock in a bowl of warm water, preheat oven to 370ish.  Once chicken is thawed, put in a pan and cook in the oven.  Ideally do enough rice/pasta and chicken for several nights.

 

Snacks

In general my snacks are super simple: just combine some kind of munching veggie (carrots, bell pepper, raw broccoli, snap peas, etc) with hummus, some fruit item, something protein-y (handful of nuts or sunflower seeds, usually) and (optionally) a slice of bread or other carb source.  For whatever snack I have after a workout, I want to make sure there is plenty of protein, so I include either hard boiled eggs, baked chicken, or salmon (on bread).


Implementation

So over the weekend, when I plan my week and go shopping, I choose the following:

 

  1. One batch dinner to cook (usually I need to buy the stuff for this)
  2. One type of quick-and-easy dinner to eat for 2-3 nights (often using staples/leftovers I already have)
  3. Two types of batch lunch to make from my list of three.
  4. 2-3 kinds of munching veggies - enough veggies total to include in ~3 meals per day (so like 6ish carrots per day, or 2 bell peppers, etc).  Think carrots, raw broccoli, bell peppers, green beans, sugar snap peas, cherry tomatoes, etc.
  5. 2-3 kinds of fruit items.  Think apples, bananas, grapefruit, grapes, oranges, etc.
  6. Two kinds of protein for post-workout snacks, chosen from: eggs, chicken, salmon
  7. Bread recipes to make 2-3 loaves (which might just be a single recipe repeated)
I also make sure I have enough yogurt and other breakfast supplies to get me through the week.  I drink milk with most of my meals at home, so I check my milk supply as well.

Boom!  Planning done, shopping list practically writes itself!  Once per week I make an small effort on cooking a batch dinner, two or three nights per week I put an extremely minimal effort into quick-and-easy dinners, two evenings per week I make a batch of lunch foods and maybe prep workout protein (hard boil eggs or bake chicken breasts), and otherwise my "cooking" consists of taking things from the fridge and putting them onto a dish (and possibly microwaving).

 


 

Conclusions

I'm still tweaking my system, but it has been a marked improvement from the last-minute scrabbling and suboptimal meals that tended to characterize my eating before this.  It's also a big step up in terms of utility from the more elaborate and time-consuming meals I sometimes cooked to compensate for feelings of inadequacy generated by aforementioned scrabbling/suboptimal meals.  I tend to feel fairly energetic and healthy, and it's a huge reassurance to me to know that I always have food planned out and typically it's available to me without needing to do any cooking.  It appears that it's considerably cheaper, too, although there are several confounding factors that would also drive my grocery bills down (transitioning to not-organic foods, trying to hit sales, etc).

Are there things I'm missing?  Suggestions for meals?  (note that I'm a bit wary of meal-replacement shakes) Alternative systems that people have found to hit that sweet spot of healthy, quick, and inexpensive? Is this something that might be useful for you?


EDIT:  Tuna is high in mercury, and shouldn't be eaten in nearly the quantities I had originally planned.  I've replaced canned tuna with canned salmon.

Open thread, Jan. 19 - Jan. 25, 2015

3 Gondolinian 19 January 2015 12:04AM

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Previous Open Thread


Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.

Weekly LW Meetups

2 FrankAdamek 23 January 2015 07:20PM

This summary was posted to LW Main on January 16th. The following week's summary is here.

New meetups (or meetups with a hiatus of more than a year) are happening in:

Irregularly scheduled Less Wrong meetups are taking place in:

The remaining meetups take place in cities with regular scheduling, but involve a change in time or location, special meeting content, or simply a helpful reminder about the meetup:

Locations with regularly scheduled meetups: Austin, Berkeley, Berlin, Boston, Brussels, Buffalo, Cambridge UK, Canberra, Columbus, London, Madison WI, Melbourne, Moscow, Mountain View, New York, Philadelphia, Research Triangle NC, Seattle, Sydney, Toronto, Vienna, Washington DC, and West Los Angeles. There's also a 24/7 online study hall for coworking LWers.

continue reading »

LINK: Superrationality and DAOs

1 somnicule 24 January 2015 09:47AM

The cryptocurrency ethereum is mentioned here occasionally, and I'm not surprised to see an overlap in interests from that sphere. Vitalik Buterin has recently published a blog post discussing some ideas regarding how smart contracts can be used to enforce superrationality in the real world, and which cases those actually are. 

The Unique Games Conjecture and FAI: A Troubling Obstacle

0 27chaos 20 January 2015 09:46PM

I am not a computer scientist and do not know much about complexity theory. However, it's a field that interests me, so I occasionally browse some articles on the subject. I was brought to https://www.simonsfoundation.org/mathematics-and-physical-science/approximately-hard-the-unique-games-conjecture/ by a link on Scott Aaronson's blog, and read the article to reacquaint myself with the Unique Games Conjecture, which I had partially forgotten about. If you are not familiar with the UGC, that article will explain it to you better than I can.

One phrase in the article stuck out to me: "there is some number of colors k for which it is NP-hard (that is, effectively impossible) to distinguish between networks in which it is possible to satisfy at least 99% of the constraints and networks in which it is possible to satisfy at most 1% of the constraints". I think this sentence is concerning for those interested in the possibility of creating FAI.

It is impossible to perfectly satisfy human values, as matter and energy are limited, and so will be the capabilities of even an enormously powerful AI. Thus, in trying to maximize human happiness, we are dealing with a problem that's essentially isomorphic to the UGC's coloring problem. Additionally, our values themselves are ill-formed. Human values are numerous, ambiguous, even contradictory. Given the complexities of human value systems, I think it's safe to say we're dealing with a particularly nasty variation of the problem, worse than what computer scientists studying it have dealt with.

Not all specific instances of complex optimization problems are subject to the UGC and thus NP hard, of course. So this does not in itself mean that building an FAI is impossible. Also, even if maximizing human values is NP hard (or maximizing the probability of maximizing human values, or maximizing the probability of maximizing the probability of human values) we can still assess a machine's code and actions heuristically. However, even the best heuristics are limited, as the UGC itself demonstrates. At bottom, all heuristics must rely on inflexible assumptions of some sort.

Minor edits.