Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: korin43 30 November 2016 02:41:47AM 4 points [-]

This feels like a point but I'm having trouble thinking of specific examples. What would a crux like this look like and how could you resolve the disagreement involved with it?

Comment author: johnswentworth 30 November 2016 05:07:13PM 6 points [-]

Let's use the school uniforms example.

The post mentions "uniforms make students feel better about themselves" as something to avoid. But that claim strongly suggests a second statement for the claimant: "I would have felt better about myself in middle school, if we'd had uniforms." A statement like that is a huge gateway into productive discussion.

First and foremost, that second statement is very likely the true cause for the claimant's position. Second, that feeling is something which will itself have causes! The claimant can then think back about their own experiences, and talk about why they feel that way.

Of course, that creates another pitfall to watch out for: argument by narrative, rather than statistics. It's easy to tell convincing stories. But if one or both participants know what's up, then each participant can produce a narrative to underlie their own feelings, and then the real discussion is over questions like (1) which of those narratives is more common in practice, (2) should we assign more weight to one type of experience, (3) what other types of experiences should we maybe consider, and (4) does the claim make sense even given the experience?

The crux is then the experiences: if I'd been through your experience in middle school, then I can see where I'd feel similarly about the uniforms. That doesn't mean your feeling is right - maybe signalling through clothing made your middle school experience worse somehow, but I argue that kids will find ways to signal regardless of uniform rules. But that would be a possible crux of the argument.

Comment author: johnswentworth 29 November 2016 10:33:47PM 10 points [-]

A suggestion: don't avoid feelings. Instead, think of feelings as deterministic and carrying valuable information; feelings can be right even when our justifications are wrong (or vice versa).

Ultimately, this whole technique is about understanding the causal paths which led to both your own beliefs, and your conversational partner's beliefs. In difficult areas, e.g. politics or religion, people inevitably talk about logical arguments, but the actual physical cause of their belief is very often intuitive and emotional - a feeling. Very often, those feelings will be the main "crux".

For instance, I've argued that the feelings of religious people offer a much better idea of religion's real value than any standard logical argument - the crux in a religious argument might be entirely a crux of feelings. In another vein, Scott's thrive/survive theory offers psychological insight on political feelings more than political arguments, and it seems like it would be a useful crux generator - i.e., would this position seem reasonable in a post-scarcity world, or would it seem reasonable during a zombie outbreak?

In response to The Power of Noise
Comment author: johnswentworth 20 June 2014 05:49:05AM 1 point [-]

The randomized control trial is a great example where a superintelligence actually could do better by using a non-random strategy. Ideally, an AI could take its whole prior into account and do a value of information calculation. Even if it had no useful prior, that would just mean that any method of choosing is equally "random" under the the AI's knowledge.

Comment author: johnswentworth 26 November 2015 06:20:17PM 0 points [-]

I'm going to commit a social faux pas and respond to my own comment, because multiple subthreads are all saying essentially the same thing: this is just math, the theory is known, humans can already do it (often with some help from computers to get through the math).

As I've read it, one of the major takeaways of lesswrong is that AI is not magical. If humans cannot possibly figure out the theory, neither can an AI. If humans cannot possibly do the math (possibly with some help from a computer), neither can an AI. Anything an AI can do, a human can also do in principle. They differ only in the degree: AIs will eventually be able to do much more complicated math, solve much more complicated problems, and self-improve much faster and more reliably.

So if you look at my original suggestion and think "that's nothing special, a human can do that in theory" then you're completely correct. Things humans can do IN THEORY are EXACTLY the things with which an AI can help.

Comment author: IlyaShpitser 29 October 2015 06:54:05PM *  1 point [-]

This doesn't require omniscience, or AI: people do this now (based on info they have). If you have more info, we know how to use it (there is theory). Why are we talking about AI, this is a math problem.


Slightly harshly worded suggestion (not to you specifically): maybe more reading, less invocation of robo-Jesus in vain.

Comment author: johnswentworth 14 November 2015 01:21:57AM 0 points [-]

Exactly! This is a math problem! And it becomes a very complicated math problem very quickly as the prior information gets interesting.

There's nothing magical about an AI; it can't figure out anything a human couldn't figure out in principle. The difference is the "superintelligence" bit: a superintelligent AI could efficiently use much more complicated prior information for experiment design.

Comment author: johnswentworth 08 September 2015 11:28:03PM 1 point [-]

How are humans supposed to generate cryptographic reward tokens in such a way that an AI could not duplicate the process?

Comment author: Username 30 July 2015 02:46:32PM 1 point [-]

It’s much better to write the piece that you want to read yourself, which usually means pitching the technical content at a level slightly higher than you were comfortable with when you started thinking about it.

Inapplicable when you're writing about natural sciences for layman audience.

Comment author: johnswentworth 03 August 2015 07:21:43AM 1 point [-]

Apparently, Davies thinks that half the time the layman audience isn't really there, and the rest of the time, they're not interested in a watered-down version of the real natural sciences. Even if you succeed, all that you achieve is that now a bunch of morons don't understand the natural sciences and think that you agree with them.

Comment author: [deleted] 02 July 2015 10:50:48PM *  2 points [-]

Denotational equivalence is actually undecidable for any class of languages stronger than deterministic pushdown automata.

This doesn't mean that we can't obtain certain evidence that two languages/automata are equal/equivalent via some mechanism other than a decision algorithm, of course. It also doesn't mean that we can't assign a probability of equality in an entirely sensible way. In fact, in probabilistic programming, probabilistic extensional equality of random variables is trivial to model: the problem is that you're talking, there, about zero-free-parameter thunks rather than arbitrarily parameterized lambdas.

So we can't really decide the denotational equivalence of lambda expressions (or recurrent neural networks), but I think that decision algorithms aren't really useful, from a cognitive perspective, for more than obtaining a 100% likelihood of equivalence. That's powerful, when you can do it, but you should also be able to get non-100% likelihoods in other ways.

The various forms of probabilistic static analysis can probably handle that problem.

Comment author: johnswentworth 03 July 2015 12:13:10AM 2 points [-]

So, you're thinking that human abstraction ability derives from probable morphisms rather than certain morphisms over weaker classes? That makes a lot of sense.

On the other hand, from what I've seen in CS classes, humans do not seem very good at recognizing equivalences even between pushdown automata beyond a few simple cases. A human equipped with pencil and lots of paper can do a good job, but that's an awful lot more powerful than just a human.

Comment author: johnswentworth 02 July 2015 10:30:51PM 2 points [-]

I get why homomorphisms between lambda-expressions are the main homomorphism problem for AGI in general. But for humans, I'd guess that homomorphisms between some drastically simpler subset of models would explain most or all of our abstraction capacity. For example, if humans can find morphisms between causal models, or between spatial models, or between some easily-modeled agents (e.g., agents similar to the human in question), or both, then it feels like that would be sufficient for most of the abstract reasoning which we actually perform.

Comment author: johnswentworth 02 July 2015 07:59:31PM *  1 point [-]

I really like this as an example of incentive misalignment, and I think it should be in the community's core materials on that subject.

That said, the ideas about healthcare contracts to fix the alignment problem could use some fleshing out. So here's my attempt at fleshing them out.

General properties of the problem:

  • We want an open market in healthcare ideas. Anybody should be able to invest in whatever idea they have.

  • Ideas which do not work should cost the investor, ideas which do work should reward the investor.

  • The ideas must be tried on some actual people. Those people, unavoidably, bear some risk, and must be incentivized to take that risk.

Solution: First, the cost to incentivize people to try ideas must be paid by someone. The investors are a natural choice here. So imagine that the investors offer payment to people to try their ideas. To avoid selection issues, the investors may specify that people must fulfill certain criteria in order to be eligible. Anyone who meets the criteria can sign up for whatever the investors are pushing. This would be sort of like an open market for clinical trials, except that it can scale to any number of people and to long times.

For their investment, the investors would get a contract. Under that contract, they get some regular payments for as long as their participants see the benefits the investors claimed and don't see serious problems. That payment ultimately needs to come from the participants; all participants pay to participate in the market as a whole. If the market is government-managed, then everyone is a participant and their payment comes from taxes.

As an example, consider a cure for prostate cancer. The investors put out their cure at some price, paying people to try their particular idea. Only those with prostate cancer are eligible. All participants get paid by the investors. As soon as a participant's cancer clears, the investors get paid a big chunk of money.Whenever the participant has health issues later in life, the investors have to pay some money back, regardless of what those later issues are. Presumably, more serious issues would be more expensive, and the original contract might specify an extra payment if the prostate cancer returns. (There would be a secondary market for investors to insure against normal later-life health risks, but investors would still need to handle more-than-normal problems, and would get paid for less-than-normal problems).

Now an even more interesting example: a new diet. The problem with monetizing a new diet is that, once word gets out, anyone can just try it without having to pay. Thus the prevalence of pills: pills can be patented and monetized. But with this market structure, the investors are PAYING people to participate. Anyone who's trying the diet will WANT to sign up, because they get paid. The investors can then reap the benefits (if any) over the person's lifetime.

Finally, a preventative example. Once again, we have a prostate cancer treatment, but this time it is preventative rather than a cure. This time, prostate cancer is not required for participation. The investors pay the participants for their effort, and make money for every prostate-cancer-free year of every participant's life. Again, there is more complexity around paying for any later problems, insuring those payments, etc... but that's all standardized stuff and the associated costs are known in advance.

Comment author: johnswentworth 24 June 2015 10:46:18PM 2 points [-]

If an AGI is based on a neural network, how can you tell from the logs whether or not the AI knows it's in a simulation?

View more: Next