Thank you!

Thank you for your kind comment! I disagree with the johnswentworth post you linked; it's misleading to frame NN interpretability as though we started out having any graph with any labels, weird-looking labels or not. I have sent you a DM.

While writing a recent post, I had to decide whether to mention that Nicolaus Bernoulli had written his letter posing the St. Petersburg problem specifically to Pierre Raymond de Montmort, given that my audience and I probably have no other shared semantic anchor for Pierre's existence, and he doesn't visibly appear elsewhere in the story.

I decided Yes. I think the idea of awarding credit to otherwise-silent muses in general is interesting.

Footnote to my impending post about the history of value and utility:

After Pascal's and Fermat's work on the problem of points, and Huygens's work on expected value, the next major work on probability was Jakob Bernoulli's Ars conjectandi, written between 1684 and 1689 and published posthumously by his nephew Nicolaus Bernoulli in 1713. Ars conjectandi had 3 important contributions to probability theory:

[1] The concept that expected experience is conserved, or that probabilities must sum to 1.

Bernoulli generalized Huygens's principle of expected value in a random event as

$E=\frac{p_1\ast a_1+p_2\ast a_2+p_3\ast a_3}{p_1+p_2+p_3}$

[ where $p_i$ is the probability of the $i$th outcome, and $a_i$ is the payout from the $i$th outcome ]

and said that, in every case, the denominator - i.e. the probabilities of all possible events - must sum to 1, because only one thing can happen to you

[ making the expected value formula just

$E=p_1\ast a_1+p_2\ast a_2+p_3\ast a_3$

with normalized probabilities! ]

[2] The explicit application of strategies starting with the binomial theorem [ known to ancient mathematicians as the triangle pattern studied by Pascal

and first successfully analyzed algebraically by Newton ] to combinatorics in random games [which could be biased] - resulting in e.g. [ the formula for the number of ways to choose k items of equivalent type, from a lineup of n [unique-identity] items ] [useful for calculating the expected distribution of outcomes in many-turn fair random games, or random games where all more-probable outcomes are modeled as being exactly twice, three times, etc. as probable as some other outcome],

written as $(\frac{n}{k})$:

$(\frac{n}{k})=\frac{n!}{k!(n-k)!}$

[ A series of random events [a "stochastic process"] can be viewed as a zig-zaggy path moving down the triangle, with the tiers as events, [whether we just moved LEFT or RIGHT] as the discrete outcome of an event, and the numbers as the relative probability density of our current score, or count of preferred events.

When we calculate $(\frac{n}{k})$, we're calculating one of those relative probability densities. We're thinking of $n$ as our total long-run number of events, and $k$ as our target score, or count of preferred events.

We calculate $(\frac{n}{k})$ by first "weighting in" all possible orderings of $n$, by taking $(n\ast (n-1)...)$, and then by "factoring out" all possible orderings $k!$ of $k$ ways to achieve our chosen W condition [since we always take the same count $k$ of W-type outcomes as interchangeable], and "factoring out" all possible orderings $(n-k)!$ of our chosen L condition [since we're indifferent between those too].

[My explanation here has no particular relation to how Bernoulli reasoned through this.] ]

Bernoulli did not stop with $(\frac{n}{k})$ and discrete probability analysis, however; he went on to analyze probabilities [in games with discrete outcomes] as real-valued, resulting in the Bernoulli probability distribution.

[3] The empirical "Law of Large Numbers", which says that, after you repeat a random game for many turns and add up all the outcomes, the total final outcome will approach the number of turns, times the expected distribution of outcomes in a single turn. E.g. if a die is biased to roll

a 6   40% of the time
a 5   25% of the time
a 4   20% of the time
a 3   8% of the time
a 2   4% of the time, and
a 1   3% of the time

then after 1,000 rolls, your counts should be "close" to

6:   .4*1,000 = 400
5:   .25*1,000 = 250
4:   .2*1,000 = 200
3:   .08*1,000 = 80
2:   .04*1,000 = 40
1:   .03*1,000 = 30

and even "closer" to these ideal ratios after 1,000,000 rolls

- which Bernoulli brought up in the fourth and final section of the book, in the context of analyzing sociological data and policymaking.

One source: "Do Dice Play God?" by Ian Stewart

[ Please DM me if you would like the author of this post to explain this stuff better. I don't have much idea how clear I am being to a LessWrong audience! ]

This is one of those subject areas that'd be unfortunately bad to get into publicly. If you or any other individual wants to grill me on this, feel free to DM me or contact me by any of the above methods and I will take disclosure case by case.

This is a just ask.

Also, even though it's not locally rhetorically convenient [ where making an isolated demand for rigor of people making claims like "scaling has hit a wall [therefore AI risk is far]" that are inconvenient for AInotkilleveryoneism, is locally rhetorically convenient for us ], we should demand the same specificity of people who are claiming that "scaling works", so we end up with a correct world-model and so people who just want to build AGI see that we are fair.

Update: My best current theory [ hasn't changed in a few months but I figured it might be worth posting ] is that composite smell data [i.e. the better part of smell processing] is passed directly from the olfactory bulb to somewhere in the entorhinal-amygdalar-temporal area, while there are a few scents that function as pheromones in the sense that we have innate responses to the scents as opposed to their associated experiences [ so, skunk and feces as well as the scent of eligible mates ] and data about these scents is relayed by thin, almost invisible projections to the hypothalamus or other nuclei in the "emotional motor system" so the behavioral responses can bootstrap.

What happens at that point depends a lot on the details of the lawbreaker's creation. [ . . . ] The probability seems unlikely to me to be zero for the sorts of qualities which would make such an AI agent dangerous.

Have you read The Sun is big, but superintelligences will not spare Earth a little sunlight?

I'll address each of your 4 critiques:

[ 1. ] In public policy making, you have a set of preferences, which you get from votes or surveys, and you formulate policy based on your best objective understanding of cause and effect. The preferences don't have to be objective, because they are taken as given.

The point I'm making in the post

Well, I reject the presumption of guilt.

is that no matter whether you have to treat the preferences as objective, there is an objective fact of the matter about what someone's preferences are, in the real world [ real, even if not physical ].

[ 2. ] [ Agreeing on such basic elements of our ontology/epistemology ] isn't all that relevant to AI safety, because an AI only needs some potentially dangerous capabilities.

Whether or not an AI "only needs some potentially dangerous capabilities" for your local PR purposes, the global truth of the matter is that "randomly-rolled" superintelligences will have convergent instrumental desires that have to do with making use of the resources we are currently using [like the negentropy that would make Earth's oceans a great sink for 3 x 10^27 joules], but not desires that tightly converge with our terminal desires that make boiling the oceans without evacuating all the humans first a Bad Idea.

[ 3. ] You haven't defined consciousness and you haven't explained how [ we can know something that lives in a physical substrate that is unlike ours is conscious ].

My intent is not to say "I/we understand consciousness, therefore we can derive objectively sound-valid-and-therefore-true statements from theories with mentalistic atoms". The arguments I actually give for why it's true that we can derive objective abstract facts about the mental world, begin at "So why am I saying this premise is false?", and end at ". . . and agree that the results came out favoring one theory or another." If we can derive objectively true abstract statements about the mental world, the same way we can derive such statements about the physical world [e.g. "the force experienced by a moving charge in a magnetic field is orthogonal both to the direction of the field and to the direction of its motion"] this implies that we can understand consciousness well, whether or not we already do.

[ 4. ] there doesn't need to be [ some degree of objective truth as to what is valuable ]. You don't have to solve ethics to set policy.

My point, again, isn't that there needs to be, for whatever local practical purpose. My point is that there is.

I think, in retrospect, the view that abstract statements about shared non-reductionist reality can be objectively sound-valid-and-therefore true follows pretty naturally from combining the common-on-LessWrong view that logical or abstract physical theories can make sound-valid-and-therefore-true abstract conclusions about Reality, with the view, also common on LessWrong, that we make a lot of decisions by modeling other people as copies of ourselves, instead of as entities primarily obeying reductionist physics.

It's just that, despite the fact that all the pieces are there, it goes on being a not-obvious way to think, if for years and years you've heard about how we can only have objective theories if we can do experiments that are "in the territory" in the sense that they are outside of anyone's map. [ Contrast with celebrity examples of "shared thought experiments" from which many people drew similar conclusions because they took place in a shared map - Singer's Drowning Child, the Trolley Problem, Rawls's Veil of Ignorance, Zeno's story about Achilles and the Tortoise, Pascal's Wager, Newcomb's Problem, Parfit's Hitchhiker, the St. Petersburg paradox, etc. ]