Jeffrey Heninger

5mo

Introduction

Frontier AI companies have seen rapid increases in the economic resources they have available to pursue AI progress. At some companies, the number of employees is at least doubling every year, and the amount of capital received is tripling every year. It is unclear whether this growth is sustainable. Is the revenue growing faster than the capital requirements? If it is, will revenue catch up before the capital available for AI investments runs out?

I do not think that there is enough publicly available data to answer these questions. It is plausible that frontier AI companies will run into significant economic limitations before Metaculus’s forecast for AGI in July 2033.

Similar Work

Epoch has an estimate for

... (read 3320 more words →)

Replying toWhy Latter-day Saints Have Strong Communities

I applied it to an office because the conversation that caused me to write this post involved an AI safety group office that the person I was talking to used to work at, which does function as a community.

It's plausible to me that these recommendations work better for other kinds of community.

Replying toWhy Latter-day Saints Have Strong Communities

My guess is that standardization has been more important for you than for the typical member of the church. It sounds like you move a lot more than most members, and so you spend a lot more of your time having just moved somewhere new. Standardization is helpful to build community when you're traveling or just moved somewhere new, but most people aren't in those situations all that often.

I also claim that standardization by itself does not build community. There is not a particularly strong community in McDonalds or in airports - despite these being very standardized situations. What standardization does is it reminds you of the similar situations you previously have... (read more)

Replying toWhy Latter-day Saints Have Strong Communities

The term "Gentile" to refer for non-Mormons was mostly a thing in the 1800s, and is basically not a thing today. The last time this usage occurred in General Conference was in 1981, and that was quoting something written in the 1800s. The last time this usage occurred in General Conference not quoting something else was in 1936. (The LDS General Conference corpus is great.) The terminology that is used is "the world", which puts much less focus on the uncleanness of individuals.

Mormonism, outside of Utah, is not disassociated from society at large. The attitude is instead "in the world, but not of the world". There are some people for whom most... (read more)

Replying toWhy Latter-day Saints Have Strong Communities

I agree that details matter. The system, in principle, equalizes status by passing around callings - and in practice often does. But it doesn't always work the way it should and you get The Same 10 Families rotating between the higher effort or leadership callings, while other people stay in lower effort callings.

Replying toWhy Latter-day Saints Have Strong Communities

Adequately answering this question would be at least its own blog post, but here's a gesture at a response:

In the 1600s, there was a debate over whether natural philosophy should be structured like math ('rationalists') or whether it should be based on sense & memory ('empiricists'). The empiricists won one of the most lopsided victories in the history of philosophy and empirical science was born.^[1]

Joseph Smith & the Book of Mormon have a wildly more empirical approach than any of the religions at the time.^[2] Alma 32's 'experiment on the word'. Moroni 10 is even framed in terms of falsifiability, a full century before Karl Popper introduced it to the philosophy of science.

Yes,... (read more)

Replying toWhy Latter-day Saints Have Strong Communities

I think you'd be interested in Tocqueville's description of how New England towns worked in the early 1800s. My guess is that the system of callings descends from it, and it was substantially more democratic.

Search: "Limits of the township" to find the relevant section.

Replying toWhy Latter-day Saints Have Strong Communities

I was assuming that building strong community is a good thing, because of the post this is responding to. If Scott (or other people) are looking at Mormonism to see what they can learn about building strong communities in a liberal society, it is better if they have an accurate understanding of how the community works.

I think that the things that seem most likely to be worth exporting are ministering and callings. Callings seem harder to export. but have been very important for me feeling part of the community. Ministering seems easier to export, and I wouldn't be that surprised if rationalists end up doing it better than we do.

Some of the... (read 546 more words →)

Racing Towards Fusion and AI

6mo

Epistemic status: Low-effort post about something I am very familiar with.

Preamble

Scott Alexander recently wrote about making strong communities within a liberal society. He has nice things to say about Mormons:

Ultra-Orthodox Jews and Mormons: Get lots of people of the same religion together in one place - a timeless classic. Some of the ultra-est of the ultra-Orthodox are still more fluent in Yiddish than English, giving them near-invincibility from the mainstream. 9/10.

And again in the highlights from the comments:

Maybe the Mormons are the entrepreneurs we’re looking for?

As the resident Mormon, I have opinions.

Scott's Thesis

I found myself very unconvinced by the thesis of Scott's argument. As always, read the whole thing and don’t trust... (read 2581 more words →)

102

'High-Level Machine Intelligence' and 'Full Automation of Labor' in the AI Impacts Surveys

Racing Towards a New Technology Is a Collective Choice, Not an Inevitable Consequence of Incentives

Before I started thinking about AI policy, I was working in the trying-to-get-fusion industry.

There are some significant similarities between AI and fusion.

Both are emerging technologies.
Both have the potential to have significant effects on the global economy.
Both are dominated by O(10) major actors, which include both private companies and governments.
Both currently require a capital expenditure of O($1 billion) to make something impressive.
Both of the leading companies are about the same size, adjusting for when the companies were founded.
Both feel futuristic.

The incentive structures for the two industries feel similar.

However, the culture of these two industries feels very different. Fusion companies... (read 1997 more words →)

Increasing the Span of the Set of Ideas

The 60+ Year Gap

AI Impacts has run three surveys (2016, 2022, & 2023) asking AI researchers about how they expect AI to develop in the future.^[1] One of the key questions addressed was when AI capabilities will exceed human capabilities.

The surveys did not ask directly about 'Artificial General Intelligence' (AGI). Instead, they asked about two similar terms: 'High-Level Machine Intelligence' (HLMI) and 'Full Automation of Labor' (FAOL). These terms were defined in the surveys as follows:

High-level machine intelligence (HLMI) is achieved when unaided machines can accomplish every task better and more cheaply than human workers. Ignore aspects of tasks for which being a human is intrinsically advantageous, e.g. being accepted as a jury member. Think feasibility,

... (read 1994 more words →)

Paper Summary: Princes and Merchants: European City Growth Before the Industrial Revolution

Epistemic Status: I wrote this back in January, and have been uncertain about whether to publish it. I expect that most people who read this here will be unconvinced. But I still want to express my intuition.

In the last month, these ideas have come up in conversation again. Then, Toby Ord published a paper on arxiv a few days ago with substantial overlap with ideas presented here, so it seems like it is time to post it. I considered rewriting this post to better respond to & engage with that paper, but decided to leave it as is.

Introduction

I have previously stated that I am skeptical that AGI is possible on a (classical)... (read 2478 more words →)

https://scholar.harvard.edu/files/shleifer/files/princes_merchants.pdf.

by J. Bradford de Long and Andrei Shleifer. (1993)

Summary: Freer societies have faster economic growth.

One of the oldest themes in economics is the incompatibility of despotism and development. Economies in which security of property is lacking – because of either the possibility of arrest, ruin, or execution at the command of the ruling prince or the possibility of ruinous taxation – should experience relative stagnation. By contrast, economies in which property is secure – either because of strong constitutional restrictions on the prince or because the ruling elite is made up of merchants rather than princes – should prosper and grow.

Much of the work investigating the relationship between despotism and economic growth... (read 1047 more words →)

Paper Summary: The Effects of Communicating Uncertainty on Public Trust in Facts and Numbers

https://www.pnas.org/doi/pdf/10.1073/pnas.1913678117.

by Anne Marthe van der Bles, Sander van der Linden, Alexandra L. J. Freeman, and David J. Spiegelhalter. (2020)

Summary: Numerically expressing uncertainty when talking to the public is fine. It causes people to be less confident in the number itself (as it should), but does not cause people to lose trust in the source of that number.

Uncertainty is inherent to our knowledge about the state of the world yet often not communicated alongside scientific facts and numbers. In the “posttruth” era where facts are increasingly contested, a common assumption is that communicating uncertainty will reduce public trust. However, a lack of systematic research makes it difficult to evaluate such claims.

Within many specialized... (read 589 more words →)

Advice for Activists from the History of Environmentalism

Was Partisanship Good for the Environmental Movement?

This is the fourth in a sequence of posts taken from my recent report: Why Did Environmentalism Become Partisan?

This post has more of my personal opinions than previous posts or the report itself.

Other movements should try to avoid becoming as partisan as the environmental movement. Partisanship did not make environmentalism more popular, it made legislation more difficult to pass, and it resulted in fluctuating executive action. Looking at the history of environmentalism can give insight into what to avoid in order to stay bipartisan.

Partisanship was not inevitable. It occurred as the result of choices and alliances made by individual decision makers. If they had made different choices, environmentalism could have ended up... (read 1626 more words →)

101

A Narrative History of Environmentalism's Partisanship

This is the third in a sequence of posts taken from my recent report: Why Did Environmentalism Become Partisan?

Summary

Rising partisanship did not make environmentalism more popular or politically effective. Instead, it saw flat or falling overall public opinion, fewer major legislative achievements, and fluctuating executive actions.

Public Opinion

One hypothesis is that partisanship was useful, or even necessary, for an issue to become popular. Maybe journalists never would have covered the story if it did not involve an exciting partisan contest. The public then might have never realized that this is a thing they could care about.

The polling data do not support this hypothesis.

The clearest data from McCright et al.^[1] Over 70% of both parties,... (read 1209 more words →)