I left this comment on Hacker News exploring whether "AI for everyone" will be a good thing or not. Interested to hear everyone's thoughts.

Just wanted to mention that watching this panel was one of the things that convinced me to give AI safety research a try :) Thanks for re-posting, it's a good memory.

To at least try to address your question: one effect could be that there are coordination problems, where many people would be trying to "change the world" in roughly the same direction if they knew that other people would cooperate and work with them. This would result in less of the attention drain you suggest. This seems more like what I've experienced.

I'm more worried about people being stupid than mean, but that could be an effect of the bubble of non-mean people I'm part of.

Is this sort of a way to get an agent with a DT that admits acausal trade (as we think the correct decision theory would) to act more like a CDT agent? I wonder how different the behaviors of the agent you specify are from those of a CDT agent -- in what kinds of situations would they come apart? When does "I only value what happens given that I exist" (roughly) differ from "I only value what I directly cause" (roughly)?

If you have a different version of QM (perhaps what Ted Bunn has called a “disappearing-world” interpretation), it must somehow differ from MWI, presumably by either changing the above postulates or adding to them. And in that case, if your theory is well-posed, we can very readily test those proposed changes. In a dynamical-collapse theory, for example, the wave function does not simply evolve according to the Schrödinger equation; it occasionally collapses (duh) in a nonlinear and possibly stochastic fashion. And we can absolutely look for experimental signatures of that deviation, thereby testing the relative adequacy of MWI vs. your collapse theory.

He asserts that such an experiment exists. I would love it if he were to expand on this assertion.

Thanks, Mark. I'm definitely thinking about applying, but my current problem is that I have too many potential proposals that I could write:

• There's the concept learning research program that you mentioned.
• I also have an interest in figuring just what exactly human values and preferences are: this seems like a topic where there is likely to be plenty of low-hanging fruit, given that a number of fields touch upon the topic: it relates to concept learning, also in AI there's preference learning, philosophers have tried to ask the question of what value is, there's a bunch of stuff in neuroscience about motivation and preference, a bunch of stuff that I've read about emotion research lately is relevant, there's the economic definition of preferences and their research into the topic, there's intercultural comparisons about values in sociology and anthropology... It seems like if someone went through all of this stuff while keeping in mind the question of "okay, so exactly what part would we want an AI to extrapolate and how and why", one should be able to make considerable progress.
• I was just recently talking to some friends about ontology identification and ontological crises, and based on some preliminary discussion, it seemed to us like one could make progress on it using an approach inspired by conceptual metaphors. Briefly, a conceptual metaphor is a mapping from one domain to another, "that allows us to use the inferential structure of one conceptual domain (say, geometry) to reason about another (say, arithmetic)" (Lakoff & Núñez, p. 6). Intuitively, the physicists who discovered quantum mechanics knew that their discovery didn't make everything they knew of classical mechanics obsolete - all the observations supporting CM were still there, so there had to exist some mapping from CM concepts to QM concepts in a way that allowed us to preserve most of what we knew of CM. That would suggest that an AI that discovered that its current understanding of the world was lacking could attempt to take its old world-model, and identify the things that it valued in the new model by looking for the inferential rules of the old model that could still be mapped into the new one, and using that to identify the corresponding mappings to entities. One of the friends I was talking with indicated that there's research on the deep learning side that might be able to do something like this.
• At this rate, tomorrow I'll probably come up with a fourth thing that seems like a promising research direction.

I guess I could just write proposals for each of these and let FLI decide which one they find the most promising - their FAQ says that one can submit several proposals but they'll only invite one Full Proposal from a single PI.

This looks to me like a misunderstanding of Müller & Bostrom 2014. The actual figure is that 50% of AI researchers give a 10% probability of HLMI by 2022.

Müller, V. C., & Bostrom, N. Future progress in artificial intelligence: A survey of expert opinion. In V. C. Müller (Ed.), Fundamental Issues of Artificial Intelligence. Berlin: Springer. 2014

So, the "polymath" thread seems to have ground to a halt. I can't tell whether the discussion just stopped going anywhere (possibly due to elimination of low-hanging fruit), or it dropped off the recent-posts list and people forgot about it, or what. Does anyone have any insight into what's going on?

I was thinking last night of how vote trading would work in a completely rational parliamentary system. To simplify things a bit, lets assume that each issue is binary, each delegate holds a position on every issue, and that position can be normalized to a 0.0 - 1.0 ranking. (e.g. If I have a 60% belief that I will gain 10 utility from this issue being approved, it may have a normalized score of .6, if it is a 100% belief that I will gain 10 utility it may be a .7, while a 40% chance of -1000 utility may be a .1) The mapping function doesn't really matter too much, as long as it can map to the 0-1 scale for simplification.

The first point that seems relatively obvious to me is that all rational agents will intentionally mis-state their utility functions as extremes for bargaining purposes. In a trade, you should be able to get a much better exchange by offering to update from 0 to 1 than you would for updating from 0.45 to 1, and as such, I would expect all utility function outputs to be reported to others as either 1 or 0, which simplifies things even further, though internally, each delegate would keep their true utlity function values. (As a sanity check, compare this to the current parliamentary models in the real world, where most politicians represent their ideals publicly as either strongly for or strongly against)

The second interesting point I noticed is that with the voting system as proposed, where every additional vote grants additional probability of the measure being enacted, every vote counts. This means it is always a good trade for me to exchange votes when my expected value of the issue you are changing position on is higher than my expected value of the position I am changing position on. This leads to a situation, where I am better off changing positions on every issue except the one that brings me the most utility in exchange for votes on the issue that brings me the most utility. Essentially, this means that the only issue that matters to an individual delegate is the issue that potentially brings them the most utility, and the rest of the issues are just fodder for trading.

Given the first point I mentioned, that all values should be externally represented as either 1 or 0, it seems that any vote trade will be a straight 1 for 1 trade. I haven't exactly worked out the math here, but I'm pretty sure that for an arbitrarily large parliament with an arbitrarily large number of issues (to be used for trading), the result of any given vote will be determined by the proportion of delegates holding that issue as either their highest or lowest utility issue, with the rest of the delegates trading their votes on that issue for votes on another issue they find to be higher utility. (As a second sanity check, this also seems to conform closely to reality with the way lobbyist groups push single issues and politicians trade votes to push their pet issues through the vote.)

This is probably an oversimplified case, but I thought I'd throw it for discussion to see if it sparked any new ideas.

My suspicion is that this just corresponds to some particular rule for normalizing preferences over strategies. The "amount of power" given to each faction is capped, so that even if some faction has an extreme opinion about one issue it can only express itself by being more and more willing to trade other things to get it.

If goodness numbers are normalized, and some moral theory wants to express a large relative preference for one thing over another, it can't just crank up the number on the thing it likes - it must flatten the contrast of things it cares less about in order to express a more extreme preference for one thing.