There are some aspects of maps - for example, edges, blank spots, and so on, that seem, if not necessary, extremely convenient to keep as part of the map. However, if you use these features of a map in the same way that you use most features of a map - to guide your actions - then you will not be guided well. There's something in the sequences like "the world is not mysterious" about people falling into the error of moving from blank/cloudy spots on the map to "inherently blank/cloudy" parts of the world.

The slogan "the map is not the territory" might encourage focusing on the delicate corrections necessary to act upon SOME aspects of one's representation of the world, but not act on other aspects which are actually intrinsic to the representation.

You might enjoy Crutchfield's epsilon machines, and Shalizi's CSSR algorithm for learning them:

http://masi.cscs.lsa.umich.edu/~crshalizi/notabene/computational-mechanics.html

There's cognitive strategies that (heuristically) take advantage of the usually-persistent world. Should I be embarrassed, after working and practicing with pencil and paper to solve arithmetic problems, that I do something stupid when someone changes the properties of pencil and paper from persistent to volatile?

What I'd like to see is more aboveboard stuff. Suppose that you notify someone that you're showing them possibly-altered versions of their responses. Can we identify which things were changed when explicitly alerted? Do we still confabulate (probably)? Are the questions that we still confabulate on questions that we're more uncertain about - more ambiguous wording, more judgement required?

Sounds like internal audit

There's a lot of similarity between the statistical tests that a scientist does and the statistical tests that auditors do. The scientist is interested in testing that the effect is real, and the auditor is testing that the company really is making that much money, that all its operations are getting aggregated up into the summary documents correctly.

Charlie Stross has a character in his 'Rule 34', Dorothy Straight, who is an organization-auditor, auditing organizations for signs of antisocial behavior. As I understood it, she was asking whether the organizations as a whole are likely to behave badly - though one way that the organization as a whole might behave badly is by sifting out or creating leaders who are likely to individually behave badly.

What I'm trying to say is that there will be a field of auditing an organization's 'safety case' - examining why it believes that it is a Friendly organization, what its internal controls entangling it with the truth are and so on, something like GiveWell for for-profits.

I'm not sure why you say this.

Please remember that this introduction is non-standard, so you may need to be an expert on standard RL to see the connection. And while some parts are not in place yet, this post does introduce what I consider to be the most important part of the setting of RL.

So I hope we're not arguing over definitions here. If you expand on your meaning of the term, I may be able to help you see the connection. Or we may possibly find that we use the same term for different things altogether.

I should also explain why I'm giving a non-standard introduction, where a standard one would be more helpful in communicating with others who may know it. The main reason is that this will hopefully allow me to describe some non-standard and very interesting conclusions.

I'm having a hard time understanding what the arrows from W-node to W-node and M-node to M-node represent in the chess example, given the premise that the world and memory states take turns changing.

If I understand correctly, W is the board state at the start of the player's turn, and M is the state of the memory containing the model of the board and possible moves/outcomes. W(t) is the state that precedes M(t), and likewise the action resulting from the completion of remodelling the memory at M(t), plus the opposing player's action, results in new world state W(t+1).

This interpretation seems to suggest a simple, linear, linked list of alternating W and M nodes instead of the idea that, for example, the W(t-1) node is the direct precursor to W(t). The reason being, it seems that one could generate W(t) simply from the memory model in M(t-1), regardless of what W(t-1) was.. and the same goes for M(t) and W(t-1).

Perhaps it's that the arrow from one W-node to another does not represent the causal/precursor relationship that a W-node to M-node arrow represents, but a different relationship? If so, what is that relationship? Sorry if this seems picky, but I do think that the model is causing some confusion as to whether I properly understand your point.

It might be valuable to point out that nothing about this is reinforcement learning yet.

as far as I can tell, the only substantial public speech from SIAI on LessWrong

Also see How to Purchase AI Risk Reduction, So You Want to Save the World, AI Risk & Opportunity: A Strategic Analysis...

Thanks for posting this!

I am also grateful to Holden for provoking this - as far as I can tell, the only substantial public speech from SIAI on LessWrong. SIAI often seems to be far more concerned with internal projects than communicating with its supporters, such as most of us on LessWrong.