Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Gondolinian 17 December 2014 03:42:57PM *  2 points [-]

Mostly just out of curiosity:

What happens karma-wise when you submit a post to Discussion, it gets some up/downvotes, you resubmit it to Main, and it gets up/downvotes there? Does the post's score transfer, or does it start from 0?

Comment author: Kaj_Sotala 19 December 2014 08:59:34PM 2 points [-]

The post's score transfers, but I think that the votes that were applied when it was in Discussion don't get the x10 karma multiplier that posts in Main otherwise do.

Comment author: Viliam_Bur 18 December 2014 11:03:41PM 8 points [-]

Now I feel like every group that tries to do something faces a trilemma:

1) Deny your weakness. Leads to irrationality.

2) Admit your weakness. Leads to low status, and then opposition from outsiders.

3) Deny your weakness publicly, only admit them among trusted members. Leads to cultishness.

Comment author: Kaj_Sotala 19 December 2014 02:01:20PM 3 points [-]

2) Admit your weakness. Leads to low status, and then opposition from outsiders.

I wonder: it feels like with individuals, honestly and directly admitting your weakness while giving the impression that they're not anything you're trying to hide, can actually increase your status. Having weaknesses yet being comfortable with them signals that you believe you have strength that compensates for those weaknesses, plus having flaws makes you more relatable. Could that also work for groups? I guess the biggest problem would be that with groups, it's harder to present a unified front: even when a single person smoothly and honestly admits the flaw, another gets all defensive.

Comment author: Punoxysm 19 December 2014 12:41:36AM 0 points [-]

Make the text more clearly readable by spacing it away from the tables, or making more solid, lighter-colored lettering.

Comment author: Kaj_Sotala 19 December 2014 08:20:50AM *  0 points [-]

Thanks. Yeah, I've increased the size of the node graphics from the default, but the graph library that lays them out doesn't actually know that, so it places the text in a way that ends up overlapping with the graphics. Hacking that is on my to-do list.

Bayes Academy Development Report 2 - improved data visualization

4 Kaj_Sotala 18 December 2014 10:11PM

See here for the previous update if you missed / forgot it.

In this update, no new game content, but new graphics.

I wasn’t terribly happy about the graphical representation of the various nodes in the last update. Especially in the first two networks, if you didn’t read the descriptions of the nodes carefully, it was very easy to just click your way through them without really having a clue of what the network was actually doing. Needless to say, for a game that’s supposed to teach how the networks function, this is highly non-optimal.

Here’s the representation that I’m now experimenting with: the truth table of the nodes is represented graphically inside the node. The prior variable at the top doesn’t really have a truth table, it’s just true or false. The “is” variable at the bottom is true if its parent is true, and false if its parent is false.

You may remember that in the previous update, unobservable nodes were represented in grayscale. I ended up dropping that, because that would have been confusing in this representation: if the parent is unobservable, should the blobs representing its truth values in the child node be in grayscale as well? Both “yes” and “no” answers felt confusing.

Instead the observational state of a node is now represented by its border color. Black for unobservable, gray for observable, no border for observed. The metaphor is supposed to be something like, a border is a veil of ignorance blocking us from seeing the node directly, but if the veil is gray it’s weak enough to be broken, whereas a black veil is strong enough to resist a direct assault. Or something.

When you observe a node, not only does its border disappear, but the truth table entries that get reduced to a zero probability disappear, to be replaced by white boxes. I experimented with having the eliminated entries still show up in grayscale, so you could e.g. see that the “is” node used to contain the entry for (false -> false), but felt that this looked clearer.

The “or” node at the bottom is getting a little crowded, but hopefully not too crowded. Since we know that its value is “true”, the truth table entry showing (false, false -> false) shows up in all whites. It’s also already been observed, so it starts without a border.

After we observe that there’s no monster behind us, the “or” node loses its entries for (monster, !waiting -> looks) and (monster, waiting -> looks), leaving only (!monster, waiting -> looks): meaning that the boy must be waiting for us to answer.

This could still be made clearer: currently the network updates instantly. I’m thinking about adding a brief animation where the “monster” variable would first be revealed as false, which would then propagate an update to the values of “looks at you” (with e.g. the red tile in “monster” blinking at the same time as the now-invalid truth table entries, and when the tiles stopped blinking, those now-invalid entries would have disappeared), and that would in turn propagate the update to the “waiting” node, deleting the red color from it. But I haven’t yet implemented this.

The third network is where things get a little tricky. The “attacking” node is of type “majority vote” - i.e. it’s true if at least two of its parents are true, and false otherwise. That would make for a truth table with eight entries, each holding four blobs each, and we could already see the “or” node in the previous screen being crowded. I’m not quite sure of what to do here. At this moment I’m thinking of just leaving the node as is, and displaying more detailed information in the sidebar.

Here’s another possible problem. Just having the truth table entries works fine to make it obvious where the overall probability of the node comes from… for as long as the valid values of the entries are restricted to “possible” and “impossible”. Then you can see at a glance that, say, of the three possible entries, two would make this node true and one would make this false, so there’s a ⅔ chance of it being true.

But in this screen, that has ceased to be the case. The “attacking” node has a 75% chance of being true, meaning that, for instance, the “is / block” node’s “true -> true” entry also has a 75% chance of being the right one. This isn’t reflected in the truth table visualization. I thought of adding small probability bars under each truth table entry, or having the size of the truth table blobs reflect their probability, but then I’d have to make the nodes even bigger, and it feels like it would easily start looking cluttered again. But maybe it’d be the right choice anyway? Or maybe just put the more detailed information in the sidebar? I’m not sure of the best thing to do here.

If anyone has good suggestions, I would be grateful to get advice from people who have more of a visual designer gene than I do!

Comment author: AlexSchell 16 December 2014 04:35:36AM *  4 points [-]

The hack is due to Anders Sandberg, with modafinil tablets though [ETA: this last part is false, see Kaj's reply]. Works wonderfully (whether with modafinil or caffeine).

Comment author: Kaj_Sotala 17 December 2014 05:35:53PM 2 points [-]

Historical note: his original blog post on it specifically used caffeine pills.

Comment author: TheAncientGeek 14 December 2014 07:08:03PM *  0 points [-]

But the difference between infinity and any finite value is infinity . Intelligence itself, or a substantial subset if it, is easy, given infinite resources, as AIXI shows. But that's been of no use in developing real world AI: tractable approximations to AIXI aren't powerful enough to be dangerous.

It would be embarrassing to MIRI if someone cobbled together AI smart enough to be dangerous, and came to the worlds experts on AI safety for some safety features, only to be told "sorry guys, we haven't got anything that's compatible with your system, because it's finite".

What's high value again?

Comment author: Kaj_Sotala 15 December 2014 12:33:55PM 3 points [-]

But that's been of no use in developing real world AI

It's arguably been useful in building models of AI safety. To quote Exploratory Engineering in AI:

A Monte-Carlo approximation of AIXI can play Pac-Man and other simple games (Veness et al. 2011), but some experts think AIXI approximation isn’t a fruitful path toward human-level AI. Even if that’s true, AIXI is the first model of cross-domain intelligent behavior to be so completely and formally specified that we can use it to make formal arguments about the properties which would obtain in certain classes of hypothetical agents if we could build them today. Moreover, the formality of AIXI-like agents allows researchers to uncover potential safety problems with AI agents of increasingly general capability—problems which could be addressed by additional research, as happened in the field of computer security after Lampson’s article on the confinement problem.

AIXI-like agents model a critical property of future AI systems: that they will need to explore and learn models of the world. This distinguishes AIXI-like agents from current systems that use predefined world models, or learn parameters of predefined world models. Existing verification techniques for autonomous agents (Fisher, Dennis, and Webster 2013) apply only to particular systems, and to avoiding unwanted optima in specific utility functions. In contrast, the problems described below apply to broad classes of agents, such as those that seek to maximize rewards from the environment.

For example, in 2011 Mark Ring and Laurent Orseau analyzed some classes of AIXIlike agents to show that several kinds of advanced agents will maximize their rewards by taking direct control of their input stimuli (Ring and Orseau 2011). To understand what this means, recall the experiments of the 1950s in which rats could push a lever to activate a wire connected to the reward circuitry in their brains. The rats pressed the lever again and again, even to the exclusion of eating. Once the rats were given direct control of the input stimuli to their reward circuitry, they stopped bothering with more indirect ways of stimulating their reward circuitry, such as eating. Some humans also engage in this kind of “wireheading” behavior when they discover that they can directly modify the input stimuli to their brain’s reward circuitry by consuming addictive narcotics. What Ring and Orseau showed was that some classes of artificial agents will wirehead—that is, they will behave like drug addicts.

Fortunately, there may be some ways to avoid the problem. In their 2011 paper, Ring and Orseau showed that some types of agents will resist wireheading. And in 2012, Bill Hibbard (2012) showed that the wireheading problem can also be avoided if three conditions are met: (1) the agent has some foreknowledge of a stochastic environment, (2) the agent uses a utility function instead of a reward function, and (3) we define the agent’s utility function in terms of its internal mental model of the environment. Hibbard’s solution was inspired by thinking about how humans solve the wireheading problem: we can stimulate the reward circuitry in our brains with drugs, yet most of us avoid this temptation because our models of the world tell us that drug addiction will change our motives in ways that are bad according to our current preferences.

Relatedly, Daniel Dewey (2011) showed that in general, AIXI-like agents will locate and modify the parts of their environment that generate their rewards. For example, an agent dependent on rewards from human users will seek to replace those humans with a mechanism that gives rewards more reliably. As a potential solution to this problem, Dewey proposed a new class of agents called value learners, which can be designed to learn and satisfy any initially unknown preferences, so long as the agent’s designers provide it with an idea of what constitutes evidence about those preferences.

Practical AI systems are embedded in physical environments, and some experimental systems employ their environments for storing information. Now AIXI-inspired work is creating theoretical models for dissolving the agent-environment boundary used as a simplifying assumption in reinforcement learning and other models, including the original AIXI formulation (Orseau and Ring 2012b). When agents’ computations must be performed by pieces of the environment, they may be spied on or hacked by other, competing agents. One consequence shown in another paper by Orseau and Ring is that, if the environment can modify the agent’s memory, then in some situations even the simplest stochastic agent can outperform the most intelligent possible deterministic agent (Orseau and Ring 2012a).

Comment author: Vulture 12 December 2014 11:46:52PM 15 points [-]

For what it's worth, I perceived the article as more affectionate than offensive when I initially read it. This may have something to do with full piece vs. excerpts, so I'd recommend reading the full piece (which isn't that much longer) first if you care.

Comment author: Kaj_Sotala 14 December 2014 05:50:16PM 3 points [-]

I read just the excerpts, and I still thought that it came off as affectionate.

Comment author: TheAncientGeek 14 December 2014 12:26:49PM *  0 points [-]

more likely to be productive than talking about 'true essences'. 

So who, in (contemporary, analytical) philosophy talks about true essences?

In practice, then, progress toward engineering can involve moving two steps forward, then one (or two, or three) steps back.

But that's inefficient. It's wasted effort to quantify what doesn't work conceptually. It may be impossible to always get the conceptual stage right first time, but one can adopt a policy of getting it as firm as possible...rather than a policy of cpnnitatiinally associating conceptual analysis with "semantics", "true essences" and other bad things, and going straight to maths.

The highest-value things MIRI can do right now mostly involving moving toward mathematics -- including better formalizing the limitations of the AIXI equation, and coming up with formally specified alternatives

I would have thought that the highest value work is work that is relevant to systems that exist, or will argue in the near future. ...but what I see is a lot of work on AIXI (not computably tractable), Bayes (ditto), goal stable agents (no one knows how to build one).

Comment author: Kaj_Sotala 14 December 2014 05:30:10PM *  3 points [-]

AIXI (not comfortable tractable), Bayes (ditto)

The argument is that AIXI and Bayes assume infinite computing power, and thus simplify the problem by allowing you to work on it without needing to consider computing power limitations. If you can't solve the easier form of the problem where you're allowed infinite computing power, you definitely can't solve the harder real-world version either, so you should start with the easier problem first.

Comment author: Kawoomba 14 December 2014 12:28:30PM *  4 points [-]

Why not another subforum on LW, next to Main and Discussion, say, Technical Discussion? Probably because you want to avoid the "friendliness" nomenclature, but it would be nice to find some way around that, otherwise it's yet another raison d'être of this forum being outsourced.

Comment author: Kaj_Sotala 14 December 2014 05:25:41PM *  8 points [-]

LW seems to have a rather mixed reputation: if you want to attract mainstream researchers, trying to separate the research forum from the weirder stuff discussed on LW seems like a good idea.

Comment author: RobbBB 13 December 2014 09:16:14AM *  3 points [-]

I actually did consider 'Self-Modifying Intelligence Research Forum' at one point...

Comment author: Kaj_Sotala 13 December 2014 07:32:25PM 6 points [-]

I initially parsed that as (Self-Modifying (Intelligence Research Forum)), and took it to indicate that the forum's effectively a self-modifying system with the participants' comments shifting each other's beliefs, as well as changing the forum consensus.

View more: Next