Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: jkaufman 10 October 2014 11:13:27AM *  2 points [-]

Perhaps one way to improve the measurement would be to structure the question in terms of preference rather than direct measurement

This is a really cool idea. But even preference has issues. For example, I like contra dance (a kind of social dancing) a lot, and have a good time when I go. The feel in the moment is one of my favorite things. If you asked me, "would you rather be contra dancing" I would usually say yes. But if you look at my behavior, I don't actually go that often anymore, even when I do have free time. How do you tell the difference between me irrationally underconsuming something I enjoy vs me overestimating how much I enjoy it in posed comparisions?

Comment author: jsalvatier 14 October 2014 11:37:11PM 0 points [-]

For certain formulations of this, that objection seems irrelevant. Imagine that instead of a 1-10 scale, you had a ranked list of activities (or sets of activities).

Comment author: ESRogs 26 September 2014 09:24:47PM *  8 points [-]

In order to get a better handle on the problem, I’d like to try walking through the mechanics of a how a vote by moral parliament might work. I don’t claim to be doing anything new here, I just want to describe the parliament in more detail to make sure I understand it, and so that it’s easier to reason about.

Here's the setup I have in mind:

  • let's suppose we've already allocated delegates to moral theories, and we've ended up with 100 members of parliament, MP_1 through MP_100
  • these MP's will vote on 10 bills B_1 through B_10 that will each either pass or fail by majority vote
  • each MP M_m has a utility score for each bill B_b passing U_m,b (and assigns zero utility to the bill failing, so if they'd rather the bill fail, U_m,b is negative)
  • the votes will take place on each bill in order from B_1 to B_10, and this order is known to all MP's
  • all MP's know each other's utility scores

Each MP wants to maximize the utility of the results according to their own scores, and they can engage in negotiation before the voting starts to accomplish this.

Does this seem to others like a reasonable description of how the parliamentary vote might work? Any suggestions for improvements to the description?

If others agree that this description is unobjectionable, I'd like to move on to discussing negotiating strategies the MP's might use, the properties these strategies might have, and whether there are restrictions that might be useful to place on negotiating strategies. But I'll wait to see if others think I'm missing any important considerations first.

Comment author: jsalvatier 26 September 2014 10:28:21PM *  2 points [-]

Remember there's no such thing as zero utility. You can assign an arbitrarily bad value to failing to resolve, but it seems a bit arbitrary.

Comment author: Manfred 26 September 2014 07:55:41PM *  4 points [-]

Is there some way to rephrase this without bothering with the parliament analogy at all? For example, how about just having each moral theory assign the available actions a "goodness number" (basically expected utility). Normalize the goodness numbers somehow, then just take the weighted average across moral theories to decide what to do.

If we normalize by dividing each moral theory's answers by its biggest-magnitude answer, (only closed sets of actions allowed :) ) I think this regenerates the described behavior, though I'm not sure. Obviously this cuts out "human-ish" behavior of parliament members, but I think that's a feature, since they don't exist.

Comment author: jsalvatier 26 September 2014 10:26:39PM 2 points [-]

I think the key benefit of the parliamentary model is that the members will vote trade in order to maximize their expectation.

Comment author: Stuart_Armstrong 22 August 2014 10:52:06AM 2 points [-]

It seems the general goal could be cashed out in simple ways, with biochemistry, epidemeology, and a (potentially flawed) measure of "health".

Comment author: jsalvatier 28 August 2014 06:57:08PM 1 point [-]

I think you're sneaking in a lot with the measure of health. As far as I can see, the only reason its dangerous is because it caches out in the real world, on the real broad population rather than a simulation. Having the AI reason about a drugs effects on a real world population definitely seems like a general skill, not a narrow skill.

Comment author: jsalvatier 24 August 2014 06:17:13PM 3 points [-]

Narrow AI can be dangerous too is an interesting idea, but I don't think this is very convincing. I think you've accidentally snuck in some things not inside its narrow domain. In this scenario the AI has to model the actual population, including the quantity of the population, which doesn't seem too relevant. Also, it seems unlikely that people would use reducing absolute number of deaths as the goal function as opposed to chance of death for those already alive.

Comment author: nbouscal 01 August 2014 05:51:59PM *  9 points [-]

There have been numerous critiques of Connection Theory already, and I encounter people disavowing it with much more frequency than people endorsing it, in both the rationalist and EA communities. So, I don't think we have anything to worry about in that direction. I'm more worried by the zeal with which people criticize it, given that Leverage rarely seems to mention it, all of the online material about it is quite dated, and many of the people whose criticism of it I question don't seem to actually know hardly anything about it.

To be extra clear: I'm not a proponent of CT; I'm very skeptical of it. It's just distressing to me how quick the LW community is to politicize the issue.

Comment author: jsalvatier 04 August 2014 10:10:58PM 3 points [-]

One part that worries me is that they put on the EA Summit (and ran it quite well), and thus had a largish presence there. Anders' talk was kind of uncomfortable to watch for me.

Comment author: John_Maxwell_IV 17 July 2014 07:02:06AM 8 points [-]

Perhaps you could see trying to think of analogies as sampling randomly in conceptspace from a reference class that the concept you are interested in belongs to.

Imagine a big book of short computer programs that simulate real-life phenomena. I'm working on a new program for a particular phenomenon I'm trying to model. I don't have much data about my phenomenon, and I'm trying to figure out if a recursive function (say) would accurately model the phenomenon. By looking through my book of programs, I can look at the frequency with which recursive functions seem to pop up when modeling reality and adjust my credence that the phenomenon can be modeled with a recursive function accordingly.

Choosing only to look at pages for phenomena that have some kind of isomorphism with the one I'm trying to model amounts to sampling from a smaller set of data points from a tighter reference class.

This suggests an obvious way to improve on reasoning by analogy: try to come up with a bunch of analogies, in a way that involves minimal motivated cognition (to ensure a representative sample), and then look at the fraction of the analogies for which a particular proposition holds (perhaps weighting more isomorphic analogies more heavily).

Comment author: jsalvatier 17 July 2014 10:52:27PM 1 point [-]

I like the idea of coming up with lots of analogies and averaging them or seeing if they predict things in common.

Comment author: jsalvatier 05 July 2014 10:36:38PM *  0 points [-]
  1. Human Compatible AGI
  2. Human Safe AGI
  3. Cautious AGI
  4. Secure AGI
  5. Benign AGI
Comment author: jsteinhardt 20 June 2014 09:06:17AM 1 point [-]

It's not obvious to me that Qiaochu would endorse utility functions as a standard for "ideal rationality". I, for one, do not.

Comment author: jsalvatier 20 June 2014 07:06:15PM 1 point [-]

Even if you don't think it's the ideal, utility based decision theory it does give us insights that I don't think you can naturally pick up from anywhere else that we've discovered yet.

Comment author: jsalvatier 29 May 2014 12:07:03AM 2 points [-]

About 50% of my day to day friends are LWers. All 3 of my housemates are LWers. I've hosted Yvain and another LWer. Most of the people I know in SF are through LW. I've had a serious business opportunity through someone I know via LW. I've had a couple of romantic interests.

View more: Next