https://boards.4channel.org/x/thread/36449024/ting-ting-ting-ahem-i-have-a-story-to-tell

Thanks for your thoughtful answer.

How much does it concern you that, previously in human history, "every book"/authority appears to have been systematically wrong about certain things for some reason? How many of these authors have directly experimented in physics, compared to how many just copied what someone else/ a small number of really clever scientists like Einstein said?

I guess maybe that accounts for the 1% doubt you assigned.

OK. But if you yourself state that you "certainly know" -- certainly -- that p is fixed, then you have already accounted for that particular item of knowledge.

If you do not, in fact, "certainly know" the probability of p -- as could easily be the case if you picked up a coin in a mafia-run casino or whatever -- then your prior should be 0.5 but you should also be prepared to update that value according to Bayes' Theorem.

I see that you are gesturing towards assigning also the probability that the coin is a fair coin (or generally such a coin that has a p of a certain value). That is also amenable to Bayes' Theorem in a normal way. Your prior might be based on how common biased coins are amongst the general population of coins, or somewhat of a rough guess based on how many you think you might find in a mafia-run casino. But by all means, your prior will become increasingly irrelevant the more times you flip the coin. So, I don't think you need to be too concerned about how nebulous that prior and its origins are! 

>Suppose that I have a coin with probability of heads $p$. I certainly know that $p$ is fixed and does not change as I toss the coin. I would like to express my degree of belief in $p$ and then update it as I toss the coin.

It doesn't change, because as you said, you "certainly know" that p is fixed and you know the value of p.

So if you would like to express your degree of belief in p, it's just p.

>But let's say I'm a super-skeptic guy that avoids accepting any statement with certainty, and I am aware of the issue of parametrization dependence too.

In that case use Bayes' Theorem to update your beliefs about p. Presumably there will be no change, but there's always going to be at least a tiny chance that you were wrong and your prior needs to be updated.

Why do so many technophiles dislike the idea of world government?

I rarely see the concept of "world government", or governance, or a world court or any such thing, spoken of positively by anyone. That includes technophiles and futurists who are fully cognizant of and believe in the concept of a technological singularity that needs to be controlled, "aligned", made safe etc.

Solutions to AI safety usually focus on how the AI should be coded, and it seems to me that the idea of "cancelling war/ merely human economics" -- in a sense, dropping our tools wherever humanity is not focused entirely on making a safe FAI -- is a little neglected.

Of course, some of the people who focus on the mathematical/logical/code aspects of safe AI are doing a great job, and I don't mean to disparage their work. But I am nonetheless posing this question.

I also do not (necessarily) mean to conflate world government with a communist system that ignores Hayek's fatal conceit and therefore renders humanity less capable of building AIs, computers etc. Just some type of governance singleton that means all nukes are in safe hands, etc.

(crosspost from Hacker News)

Spooky action at a distance, and the Universe as a cellular automaton

Suppose the author of a simulation wrote some code that would run a cellular automaton. Suppose further that unlike Conway's Game of Life, cells in this simulation could influence other cells that are not their immediate neighbour. This would be simple enough to code up, and the cellular automaton could still be Turing Complete, and indeed could perhaps be a highly efficient computational substrate for physics.

(Suppose that this automaton, instead of consisting of squares that would turn black or white each round, contained a series of numbers in each cell, which change predictably and in some logically clever way according to the numbers in other cells. One number, for example, could determine how far away the influence of this cell extends. This I think would make the automaton more capable of encoding the logic of things like electromagnetic fields etc.)

A physicist in the simulated Universe might be puzzled by this "spooky action at a distance", where "cells" which are treated as particles appear to influence one another or be entangled in puzzling ways. Think Bell's Theorem and that whole discussion.

Perhaps...we might be living in such a Universe, and if we could figure out the right kind of sophisticated cellular automaton, run on a computer if not pen and paper, physics would be making more progress than under the current paradigm of using extremely expensive machines to bash particles together?

"""The failures of phlogiston and vitalism are historical hindsight. Dare I step out on a limb, and name some current theory which I deem analogously flawed?

I name emergence or emergent phenomena—usually defined as the study of systems whose high-level behaviors arise or “emerge” from the interaction of many low-level elements. (Wikipedia: “The way complex systems and patterns arise out of a multiplicity of relatively simple interactions.”)

Taken literally, that description fits every phenomenon in our universe above the level of individual quarks, which is part of the problem. Imagine pointing to a market crash and saying “It’s not a quark!” Does that feel like an explanation? No? Then neither should saying “It’s an emergent phenomenon!”

It’s the noun “emergence” that I protest, rather than the verb “emerges from.” There’s nothing wrong with saying “X emerges from Y,” where Y is some specific, detailed model with internal moving parts. “Arises from” is another legitimate phrase that means exactly the same thing. Gravity arises from the curvature of spacetime, according to the specific mathematical model of General Relativity. Chemistry arises from interactions between atoms, according to the specific model of quantum electrodynamics."""

I feel as though when I first read this piece by Eliezer, I only partially understood what he was gesturing towards. I've recently had an insight about my musical improvisations on the keyboard that I think has helped elucidate, for me, a similar kind of idea.

When I was learning music, I was taught that, like the major and minor scales, and the locrian mode, etc., there is something called the jazz (or blues) scale that you can play over a 3-chord sequence (the twelve-bar blues) and it sounds good.

Fair enough. Then I was also taught that it's boring to just play those notes; you can throw in a D in the C blues scale, played over the twelve-bar blues in C, to liven things up--etc. Fine.

But as I've developed as a musician, and listened to lots of music that isn't strictly twelve-bar blues, if at all, I've noticed that I really dislike the blues scale. It's like this bad idea that's lingering, for whatever reason, in the back of people's minds when they hit certain chord sequences--say, G to F over C in any given song--and they'll, y'know, _modally_ play something like the blues scale over those chords when they ought to be doing something else entirely.

This makes it less a design pattern than what I would call an _anti-pattern_. Avoid the jazz scale: do not play in that fashion if you are attempting anything other than a cliche children's rendition of simplistic wailing harmonica blues.

This is also how I (and possibly Eliezer) feel about “emergence” as a concept. It's not a good concept, nor a skunked concept that isn't to be used, but a positively bad one that should be DISINTEGRATED by rationality. The reason for this is that too many people are disguising their lack of systematic, informed knowledge of physical phenomena by claiming emergence when they can't think of anything else to say.

To return to the musical analogy, a bit like how Led Zeppelin already invented all the best bluesy riffs, and Rage Against The Machine already covered all of the hip-hop metal beats--allegedly--every time someone in our particular culture refers to emergence as an explanation for anything in particular, I would view them as an unfortunate music student who is stuck playing bad blues music that doesn't move their audience the way it should.

This is not to say that in a different culture, as in the Baroque era where no-one had encountered blues music before, “emergence” would be such an anti-pattern, so worthy of stigma.

[C]riticism fails because the being does not have omniscient level ability to make logical inferences and resolve confusions

To develop this point: if logical inferences are the "Ethereum" to the "Bitcoin" of mere omniscience about patterns of information; or, to use a more frivolous metaphor, David Bowie's "The Next Day" in comparison to "Heroes", then I think this was a concept that was missing from OP's headline argument.

You are trying to apply realistic constraints to a hypothetical situation that is not intended to be realistic

Your thought experiment, as you want it to be interpreted, is too unrealistic for it to imply a new and surprising critique of Bayesian rationality in our world. However, the title of your post implies (at least to me) that it does form such a critique.

The gamesmaster has no desire to engage with any of your questions or your attempts to avoid directly naming a number. He simply tells you to just name a number.

If we interpret the thought experiment as happening in a world similar to our own—which I think is more interesting than an incomprehensible world where the 2nd law of thermodynamics does not exist and the Kolmogorov axioms don't hold by definition—I would be surprised that such a gamesmaster would view Arabic numerals as the only or best way to communicate an arbitrarily large number. This seems, to me, like a primitive human thought that's very limited in comparison to the concepts available to a superintelligence which can read a human's source code and take measurements of the neurons and subatomic particles in his brain. As a human playing this game I would, unless told otherwise in no uncertain terms, try to think outside the limited-human box, both because I believe this would allow me to communicate numbers of greater magnitude and because I would expect the gamesmaster's motive to include something more interesting, and humane and sensible, than testing my ability to recite digits for an arbitrary length of time.

There's a fascinating tension in the idea that the gamesmaster is an FAI, because he would bestow upon me arbitrary utility, yet he might be so unhelpful as to have me recite a number for billions of years or more. And what if my utility function includes (timeless?) preferences that interfere with the functioning of the gamesmaster or the game itself?