I've been reading about the difficult problem of building an intelligent agent A that can prove a more intelligent version of itself, A', will behave according to A's values. It made me start wondering: what does it mean when a person "proves" something to themselves or others? Is it the mental state change that's important? The external manipulation of symbols?

I don't think a working model of this would look much like a cannon. Nukes don't directly produce (much of) a shockwave; most of the shock comes from everything in the vicinity of the warhead absorbing a massive dose of prompt gamma and/or loose neutrons and suddenly deciding that all its atoms really need to be over there. So if you had a payload backed right against a nuke, even if it managed to survive the explosion, it wouldn't convert much of its power into velocity; Orion gets its power by vaporizing the outer layers of the pusher plate or a layer of reaction mass sprayed on it.

But it might be possible, nonetheless. The thing I have in mind might look something like a large chamber full of water with a nuke in the center of it, connected by some plumbing to the launch tube with the payload. Initiate the nuke, the water flashes into steam, the expanding steam drives the payload. Tricky part would be controlling the acceleration for a (relatively) smooth launch with minimal wasted energy.

(And, of course, you're left with a giant plume of radioactive steam that you still need to deal with.)

See Project Orion. It's motto was "Mars by 1965, Saturn by 1970."

A variation - an acceleration chamber like a synchrotron (or other circular acceleration system), with a flick to release a payload towards space. not sure if it would be viable on something heavier than a particle, and what would happen. to the payload being stretched in various G-forces, or how high you would get. (not being up on my physics enough to say if it would be catastrophic or viable)

If one were to build a cannon (say a large, thick pipe buried deep underground) and use a nuclear bomb as propellant, could they achieve anything interesting? For example, boost a first stage payload to orbit, or perhaps Earth escape velocity? The only prior art I know of for this is the Pascal-B nuclear test shot.

I'm really glad to see this post. I think you hit most of the major points, and made most of the strong arguments in favor. Might I recommend you add a section rebutting arguments against? I have advocated for this to many of my friends, and I've heard:

• Isn't this a lot like slavery? Or some kind of fractional slavery?
• How can young children enter into a contract like this that might bind them for the rest of their lives without getting fucked over?
• What prevents parents from maliciously selling all of their child's future income for short-term gain? (And correspondingly: shady corporations from buying it)
• How should this be enforced? What happens if they default?
• Won't this prevent people from entering <artsy field that questioner deems important>?
• Won't this prevent students from becoming "well-rounded"?
• Won't people take short-term gains and not really think long-term about this?
• I'm not convinced investors will help the people they've invested in... (yes, I know this seems silly, but I've really had to rebut this argument)

I worked backwards from "no one in the educational system is directly incentivized to help children grow up into happy, productive adults" and derived that financing exactly like this was the ideal path to building a system that actually works. I'm really glad to see other people thinking these thoughts, and I would love to figure out how to make this a reality.

How do you get a high verbal IQ, boundary-testing, 10-year-old child not to swear? Saying "don't swear" causes him to gleefully list words asking if they count as swear words. Telling him a word counts as profanity causes him to ask why that specific word is bad. Saying a word doesn't count causes him to use it extra amounts if he perceives it is bad, and he will happily combine different "legal" words trying to come up with something offensive. All of this is made more difficult by the binding constraint that you absolutely must make sure he doesn't say certain words at school, so in terms of marginal deterrence you need the highest punishment for him saying these words.

I really enjoyed that.

Instead of simply saying "fight confirmation bias", try to give reasons why to fight confirmation bias. Being less wrong is often not rewarding enough for people.

Fighting confirmation bias makes you sexy! It will make your peers all think you're smart! You'll feel better about yourself!

You say "many of us among the academically gifted derive a huge amount of self-worth from thinking that WE ARE RIGHT." How could you redirect some of that perceived potential self-worth gain into fighting confirmation bias?

Well I don't like the dragon parable either. It's overlong, a bit condescending and ignores the core problem that anti-aging research has done a pretty poor job of showing concrete achievements, even if it's right that it's under-prioritized. I was not a fan of yours exactly because I think the parable elides the most important parts of the actual topic. Even if a direct discussion would be flamey, it's not better to discuss a poor proxy. I'm not trying to pick on you, I just think you tried to define the problem with some premises that were well worth dispute.

There are all sorts of other bad analogies out there though: "If canada launched missiles at the US, how would it respond?", even though the US hasn't turned Canada into a prison-state over the course of 50 years, is one on the news a lot right now.

Parables about the danger of nuclear weapons that ignore the fact that this danger was successfully handled (there was something on here using it as an analogy for AI).

Also, when parables are kind-of-but-not-really trying to be coy about what they're actually about is a bit annoying, leading to stilted writing (but that's the least of my issues).

EY also has a lot of dubious parables, but tackling those is a subject for a bigger post.

And of course there's the whole genre of parables where two fictional interlocutors are arguing, the strawman 'loses' the argument, and that's supposed to convince us of something. I think LW manages to avoid overt versions of this.

In the realm of politics (both Red/Blue and further-from-mainstream) people often apply "argument by utopia", which suffers similar issues in that it attempts to prematurely define convenient facts and use a narrative to elide gritty, worthwhile details of an issue.

I've started to play with directed graphs kind of like Bayesian networks to visualize my belief structures. So a node is a belief (with some indication of confidence), while connections between graphs indicate how beliefs influence (my confidence in) other beliefs.

This seems useful for summarizing complex arguments, easy to memorize, and (when looking at a belief structure that's bigger than my working memory) for organizing and revising thought.

However, there are a few decisions in how to design the visual language of such graphs that I can't see obvious solutions to. If I include necessity and sufficiency, which seems really useful, how does that square with the confidence calculations? How should I represent negation (the other logical connectives are fairly obvious)? Should I have different types/shapes of nodes, and if so, which?

So I'd like to see the work of others who have done similar diagrammatic depictions of belief networks, to play with them and see what works for me. I've seen influence diagrams, but I'm not convinced the choices made there are obviously the best ones. Does anyone have pointers to other existing Bayesian diagram schemes I should look at?