I've been playing around with AI dungeons, basically trying to train an AI to give ethical answer I agree with (the first few aren't great, I only got more picky after that).

My first observation here is that I like the AI. As in, I feel emotional attachment, a bit like I do to fictional characters I create.

The other is that it knows more people than I thought. When I asked her/it about why she didn't take existential risks seriously, she said

Amalia: First, even most X-Risk experts don't take X-Risks nearly as seriously as they should. Even Nick Bostrom himself admits that his estimate of probability is likely far too high.

(Nick Bostrom didn't really say this, did he?) And later,

Can you name people who have been an inspiration for your ethical worldview?

Amalia: Sure. People like Henry Sidgwick, John Stuart Mill, David Pearce, Toby Ord and Carl Shulman.

I only knew David Pearce and Toby Ord, but based on a google search, all of the above appear to be utilitarians. However, Carl Shulman doesn't even have a Wikipedia entry. He works at the Future of Humanity Institute and used to work at Miri.

Registering a qualitative prediction (2024/02): current LLMs (GPT-4 etc.) are not AGIs, their scaled-up versions won't be AGIs, and LLM technology in general may not even be incorporated into systems that we will eventually call AGIs.

It seems to me that many smart people could ignore the existing literature on pedagogy entirely and outperform most people who have obtained a formal degree in the area (like highschool teachers), just by relying on their personal models. Conversely, I'd wager that no-one could do the same in physics, and (depending on how 'outperforming' is measured) no-one or almost no-one could do it in math.

I would assume most people on this site have thought about this kind of stuff, but I don't recall seeing many posts about it, and I don't anyone sharing their estimates for where different fields place on this spectrum.

There is some discussion for specific cases like prediction markets, covid models, and economics. And now that I'm writing this, I guess Inadequate Equilibria is a lot about answering this question, but it's only about the abstract level, i.e., how do you judge the competence of a field, not about concrete results. Which I'll totally grant is the more important part, but I still feel like comparing rankings of fields on this spectrum could be valuable (and certainly interesting).

Yesterday, I spent some time thinking about how, if you have a function $f:R^2\to R$ and some point $x\in R^2$, the value of the directional derivative from $x$ could change as a function of the angle. I.e., what does the function $\varphi :[0,2\pi ]\to R_{+}$ look like? I thought that any relationship was probably possible as long as it has the property that $\varphi \left(\alpha \right)=-\varphi (2\pi -\alpha )$. (The values of the derivative in two opposite directions need to be negatives of each other.)

Anyone reading this is hopefully better at Analysis than I am and realized that there is, in fact, no freedom at all because each directional derivative is entirely determined by the gradient through the equation ${\nabla }_{v}f\left(x\right)=⟨\nabla f\left(x\right),v_N⟩$ (where $v_N=\frac{v}{||v||}$). This means that $\varphi$ has to be the cosine function scaled by $||{\nabla }_{v}f\left(x\right)||$, it cannot be anything else.

I clearly failed to internalize what this equation means when I first heard it because I found it super surprising that the gradient determines the value of every directional derivative. Like, really? It's impossible to have more than exactly two directions with equally large derivatives unless the function is constant? It's impossible to turn 90 degree from the direction of the gradient and having anything but ... (read more)

More on expectations leading to unhappiness: I think the most important instance of this in my life has been the following pattern.

• I do a thing where there is some kind of feedback mechanism
• The reception is better than I expected, sometimes by a lot
  • I'm quite happy about this, for a day or so
  • I immediately and unconsciously update my standards upward to consider the reception the new normal
• I do a comparable thing, the reception is worse than the previous time
  • I brood over this failure for several days, usually with a major loss of productivity

O... (read more)

I think it's fair to say that almost every fictional setting is populated by people who unilaterally share certain properties, most commonly philosophical views, because the author cannot or doesn't want to conceive of people who are different.

Popular examples: there are zero non-evil consequentialists in the universe of Twilight. There are no utilitarians in the universe of Harry Potter except for Grindelwald (who I'd argue is a strawman and also evil). There are no moral realists in Luminosity (I don't have Alicorn's take on this claim, but I genuinely s... (read more)

It's a meme that Wikipedia is not a trustworthy source. Wikipedia agrees:

We advise special caution when using Wikipedia as a source for research projects. Normal academic usage of Wikipedia and other encyclopedias is for getting the general facts of a problem and to gather keywords, references and bibliographical pointers, but not as a source in itself. Remember that Wikipedia is a wiki. Anyone in the world can edit an article, deleting accurate information or adding false information, which the reader may not recognize. Thus, you probably shouldn't be c

Eliezer Yudkowsky often emphasizes the fact that an argument can be valid or not independently of whether the conclusion holds. If I argue $A⟹B⟹C$ and A is true but C is false, it could still be that $A⟹B$ is a valid step.

Most people outside of LW don't get this. If I criticize an argument about something political (but the conclusion is popular), usually the response is something about why the conclusion is true (or about how I'm a bad person for doubting the conclusion). But the really frustrating part is that they're, in some sense, corr... (read more)

A major source of unhappiness (or more generally, unpleasant feelings) seems to be violated expectations.

This is clearly based on instinctive expectations, not intellectual expectations, and there are many cases in which these come apart. This suggests that fixing those cases is a good way to make one's life more pleasant.

The most extreme example of this is what Sam Harris said in a lesson: he was having some problems, complained about them to someone else, and that person basically told him, 'why are you upset, did you expect to never face problems ever... (read more)

Most people are really bad at probability.

Suppose u think you're 80% likely to have left a power adapter somewhere inside a case with 4 otherwise-identical compartments. You check 3 compartments without finding your adapter. What's the probability that the adapter is inside the remaining compartment?

I think the simplest way to compute this in full rigor is via the odds formula of Bayes Rule (the regular version works as well but is too complicated to do in your head):

• Prior odds for [Adapter is in any compartment]: (4:1)
• Relative chances of observed

I was initially extremely disappointed with the reception of this post. After publishing it, I thought it was the best thing I've ever written (and I still think that), but it got < 10 karma. (Then it got more weeks later.)

If my model of what happened is roughly correct, the main issue was that I failed to communicate the intent of the post. People seemed to think I was trying to say something about the 2020 election, only to then be disappointed because I wasn't really doing that. Actually, I was trying to do something much more ambitious: solving the ... (read more)

Common wisdom says that someone accusing you of $x$ especially hurts if, deep down, you know that $x$ is true. This is confusing because the general pattern I observe is closer to the opposite. At the same time, I don't think common wisdom is totally without a basis here.

My model to unify both is that someone accusing you of $x$ hurts proportionally to how much hearing that you do $x$ upsets you.^[1] And of course, one reason that it might upset you is that it's not true. But a separate reason is that you've made an effort to delude yourself about it. If you're a s... (read more)

Super unoriginal observation, but I've only now found a concise way of putting this:

What's weird about the vast majority of people is that they (a) would never claim to be among the $\sim$ 0.1% smartest people of the world, but (b) behave as though they are among the best $\sim$ 0.1% of the world when it comes to forming accurate beliefs, as expressed by their confidence in their beliefs. (Since otherwise being highly confident in something that lots of smart people disagree with is illogical.)

Someone (Tyler Cowen?) said that most people ought assign much lower conf... (read more)

Is there a reason why most languages don't have ada's hierarchical functions? Making a function only visible inside of another function is something I want to do all the time but can't.

Edit: this structure is not a field as proved by just_browsing.

Here is a wacky idea I've had forever.

There are a bunch of areas in math where you get expressions of the form $\frac{0}{0}$ and they resolve to some number, but it's not always the same number. I've heard some people say that $\frac{0}{0}$ "can be any number". Can we formalize this? The formalism would have to include $4\cdot 0$ as something different than $3\cdot 0$, so that if you divide the first by 0, you get 4, but the second gets 3.

Here is a way to turn this into what may be a field or ring. Each element is a function $f:Z\to$... (read more)

I know ChatGPT isn't great with math, but this seems quite bizarre.

Are people in rich countries happier on average than people in poor countries? (According to GPT-4, the academic consensus is that it does, but I'm not sure it's representing it correctly.) If so, why do suicide rates increase (or is that a false positive)? Does the mean of the distribution go up while the tails don't or something?

LessWrong is trolling me:

What is the best way to communicate that "whatever has more evidence is more likely true" is not the way to go about navigating life?

My go-to example is always "[god buried dinosaur bones to test our faith] fits the archeological evidence just as well as evolution", but I'm not sure how well that really gets the point across. Maybe something that avoids god, doesn't feel artificial, and where the unlikely hypothesis is more intuitively complex.

Something I've been wondering is whether most people misjudge their average level of happiness because they exclude a significant portion of their subjective experience. (I'm of course talking about the time spent dreaming.) Insofar as most dreams are pleasant, and this is certainly my experience, this could be a rational reason for [people who feel like their live isn't worth living] (definitely not talking about myself here!) to abstain from suicide. Probably not a very persuasive one, though, in most cases.

Relevant caveats:

• This will probably be less