Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Brillyant 22 February 2016 08:10:52PM 1 point [-]

I love this quote. But this...

declarations of high confidence mainly tell you that an individual has constructed a coherent story in his mind

...strikes me as a highly confident declaration for which the quoted is simultaneously urging me to be skeptical.

I'd imagine the book lays out his case as to why I ought listen to his counsel. I'd be interested to dig into this.

Comment author: Armok_GoB 02 March 2016 07:24:48PM 0 points [-]

The solution here might be that it does mainly tell you they have constructed a coherent story in their mind, but that having constructed a coherent story in their mind is still usefull evidence for being true depending on what else you know abaut the person, and thus worth telling. If the tone of the book was differnt, it might say:

“I have constructed a coherent story in my mind that it is wise to take admissions of uncertainty seriously, but declarations of high confidence mainly tell you that an individual has constructed a coherent story in his mind, not necessarily that the story is true.”

Comment author: Nebu 11 December 2015 06:43:37AM 0 points [-]

I think this argument is misleading.

Re "for game theoretical reasons", the paperclipper might take revenge if it predicted that doing so would be a signalling-disincentive for other office-supply-maximizers from stealing paperclips. In other words, the paperclip-maximizer is spending paperclips to take revenge solely because in its calculation, this actually leads to the expected total number of paperclips going up.

Comment author: Armok_GoB 15 December 2015 10:19:20PM 1 point [-]

That assumes the scenario is iterated, I'm talking it'd precomit to do so even in a one-of scenario. The resxzt of you argument was my point, that the same reasoning goes for anger.

Comment author: ChristianKl 01 December 2015 10:05:19PM 1 point [-]

The twenty fiveth virtue of rationality is "Be faithless to humanity and betray it to the Way".

Comment author: Armok_GoB 04 December 2015 06:05:29AM 0 points [-]

Wow, people are still finding this occasionally. It fills me with Determination.

In response to The Stamp Collector
Comment author: torekp 02 May 2015 03:00:00PM 1 point [-]

Thanks for this. One quibble: I think it's misleading to say actions are rated according to what they achieve. Consider dancing. What dancing achieves, first and foremost, is dancing. More generally, some features of actions may be valued or disvalued as terminal values, independent of any (non-tautological) consequences. It's not hard to imagine patterns of behavior for which this is the most elegant explanation.

In response to comment by torekp on The Stamp Collector
Comment author: Armok_GoB 31 May 2015 06:31:11PM 0 points [-]

Um no. The specif sequence of muscle contractions is the action, and the thing they try to achieve is beautiful patterns of motion with certain kinds of rhythm and elegance, and/or/typically the perception of such in an observer.

Comment author: minorin 27 March 2015 03:25:40AM 3 points [-]

LessWrong updates the truth to fit its priors.

Them's fightin' words, Random Profound Deep Wisdom Generator!

Comment author: Armok_GoB 27 March 2015 11:47:55AM 1 point [-]

This thing is still alive?! :D I really should get working on that updated version sometime.

Comment author: Stuart_Armstrong 15 December 2014 01:07:30PM *  1 point [-]

Interesting. Is this kinda like a minimum complexity of outcome requirement?

Comment author: Armok_GoB 16 December 2014 07:57:14PM 0 points [-]

Didn't think of it like that, but sort of I guess.

Comment author: Armok_GoB 12 December 2014 10:31:53AM 1 point [-]

It has near maximal computational capacity, but that capacity isn't being "used" for anything in particular that is easy to determine.

This is actually a very powerful criteria, in terms of number of false positive and negatives. Sadly, the false positives it DOES have still far outweigh the genuine positives, and includes all the WORST outcomes (aka, virtual hells) as well.

Comment author: skeptical_lurker 17 September 2014 09:10:59PM *  3 points [-]

Are you implying that rational agents can be successfully blackmailed by other agents that simulate them inaccurately? (This does seem plausible to me, and is an interesting rare example of accurate knowlage posing a hazard.)

Comment author: Armok_GoB 07 October 2014 10:20:27PM 2 points [-]

Well, that's quite obvious. Just imagine the blackmailer is a really stupid human with a big gun that'd fall for blackmail in a variety of awful ways, and has a bad case of typical mind fallacy, and if anything goes other than their expectations they get angry and just shot them before thinking through the consequences.

Comment author: kokotajlod 14 September 2014 03:47:59PM *  1 point [-]

Thanks for the response. Yes, it depends on how much interaction I have with human beings and on the kind of people I interact with. I'm mostly interested in my own case, of course, and I interact with a fair number of fairly diverse, fairly intelligent human beings on a regular basis.

If you're a social butterfly who regularly talks with some of the smartest people in the world, the AI will probably struggle

Ah, but would it? I'm not so sure, that's why I made this post.

Yes, if everyone always said what I predicted, things would be obvious, but recall I specified that random variation would be added. This appears to be how dream characters work: You can carry on sophisticated conversations with them, but (probably) they are governed by algorithms that feed off your own expectations. That being said, I now realize that the variation would have to be better than random in order to account for how e.g. EY consistently says things that are on-point and insightful despite being surprising to me.

Comment author: Armok_GoB 16 September 2014 01:53:41AM *  1 point [-]

Another trick it could use is using chatbots most of the time, but swaping them out for real people only for the moments you are actually talking about deep stuff. Maybe you have deep emotional conversations with your family a few hours a week. Maybe once per year, you have a 10 hour intense discussion with Eliezer. That's not a lot out of 24 hours per day, the vast majority of the computing power is still going into simulating your brain.

Edit: another; the chatbots might have some glaring failure modes if you say the wrong thing, unable to handle edge cases, but whenever you encounter then the sim is restored from a backup 10 min earlier and the specific bug is manually patched. If this went on for long enough the chatbots would become real people, and also bloat slow, but it hasn't happened yet. or maybe the patches that dont come up in long enoguh get commented out.

Comment author: StephenR 20 August 2014 01:59:04PM 0 points [-]

Believing that a theory is true that says "true" is not a thing theories can be is obviously silly.

Oh okay. This is a two-part misunderstanding.

I'm not saying that theories can't be true, I'm just not talking about this truth thing in my meta-model. I'm perfectly a-okay with models of truth popping up wherever they might be handy, but I want to taboo the intuitive notion and refuse to explicate it. Instead I'll rely on other concepts to do much of the work we give to truth, and see what happens. And if there's work that they can't do, I want to evaluate whether it's important to include in the meta-model or not.

I'm also not saying that my theory is true. At least, not when I'm talking from within the theory. Perhaps I'll find certain facets of the correspondence theory useful for explaining things or convincing others, in which case I might claim it's true. My epistemology is just as much a model as anything else, of course; I'm developing it with certain goals in mind.

I were talking about physical systems, not physical laws. Computers, living cells, atoms, the fluid dynamics of the air... "Applied successfully in many cases", where "many" is "billions of times every second"

The math we use to model computation is a model and a tool just as much as computers are tools; there's nothing weird (at least from my point of view) about models being used to construct other tools. Living cells can be modeled successfully with math, you're right; but that again is just a model. And atoms are definitely theoretical constructs used to model experiences, the persuasive images of balls or clouds they conjure notwithstanding. Something similar can be said about fluid dynamics.

I don't mean any of this to belittle models, of course, or make them seem whimsical. Models are worth taking seriously, even if I don't think they should be taken literally.

Then ZFC is not one of those cores ones, just one of the peripheral ones. I'm talking ones like set theory as a whole, or arithmetic, or Turing machines.

The best example in the three is definitely arithmetic; the other two aren't convincing. Math was done without set theory for ages, and besides we have other foundations available for modern math that can be formulated entirely without talking about sets. Turing machines can be replaced with logical systems like the lambda calculus, or with other machine models like register machines.

Arithmetic is more compelling, because it's very sticky. It's hard not to take it literally, and it's hard to imagine things without it. This is because some of the ideas it constitutes are at the core cluster of our categories, i.e. they're very sticky. But could you imagine that some agent might a) have goals that never require arithmetical concepts, and b) that there could be models that are non-arithmetical that could be used toward some of the same goals for which we use arithmetic? I can imagine ... visualise, actually, both, although I would have a very hard time translating my visual into text without going very meta first, or else writing a ridiculously long post.

Comment author: Armok_GoB 20 August 2014 08:40:32PM 0 points [-]

Hmm, maybe I need to reveal my epistemology another step towards the bottom. Two things seem relevant here.

I think you you SHOULD take your best model literally if you live in a human brain, since it can never get completely stuck requiring infinite evidence due to it's architecture, but does have limited computation and doubt can both confuse it and damage motivation. The few downsides there are can be fixed with injunctions and heuristics.

Secondly, you seem to be going with fuzzy intuitions or direct sensory experience as the most fundamental. At my core is instead that I care about stuff, and that my output might determine that stuff. The FIRST thing that happens is conditioning on that my decisions matter, and then I start updating on the input stream of a particular instance/implementation of myself. My working definition of "real" is "stuff I might care about".

My point wasn't that the physical systems can be modeled BY math, but that they themselves model math. Further, that if the math wasn't True, then it wouldn't be able to model the physical systems.

With the math systems as well you seem to be coming from the opposite direction. Set theory is a formal system, arithmetic can model it using gödel numbering, and you can't prevent that or have it give different results without breaking arithmetic entirely. Likewise, set theory can model arithmetic. It's a package deal. Lambda calculus and register machines are also members of that list of mutual modeling. I think even basic geometry can be made sort of Turing complete somehow. Any implementation of any of them must by necessity model all of them, exactly as they are.

You can model an agent that doesn't need the concepts, but it must be a very simple agent with very simple goals in a very simple environment. To simple to be recognizable as agentlike by humans.

View more: Next