All of Jeffrey Heninger's Comments + Replies

I would also make the same prediction for Q > 10. Or when CFS first sells electricity to the grid. These will be farther into the future, but I do not think that this culture will have changed by then.

I think that I predict the opposite (conditional on what exactly is being predicted).

What exactly would count as a GPT-3 moment for fusion? How about an experiment demonstrating reactor-like conditions? This is roughly equivalent to what I referred to as 'getting fusion' in my book review.

My prediction is that, after Commonwealth Fusion Systems gets Q > 5 on SPARC, they will continue to supply or plan to supply HTS tape to at least 3 other fusion startups.

I agree that this is plausibly a real important difference, but I do not think that it is obvious.

The most recent augmentative technological change was the industrial revolution. It has reshaped virtually every every activity. It allowed for the majority of the population to not work in agriculture for the first time since the agricultural revolution.

The industrial revolution centered on energy. Having much cheaper, much more abundant energy allowed humans to use that energy for all sorts of things. 

If fusion ends up being similar in cost to existing ... (read more)

Before. The 2022 survey responses were collected from June-August. ChatGPT came out at the end of November.

A few more thoughts on Ord's paper:

Despite the similarities, I think that there is some difference between Ord's notion of hyperbolation and what I'm describing here. In most of his examples, the extra dimension is given. In the examples I'm thinking of, what the extra dimension ought to be is not known beforehand.

There is a situation in which hyperbolation is rigorously defined: analytic continuation. This takes smooth functions defined on the real axis and extends them into the complex plane. The first two examples Ord gives in his paper are examples of ... (read more)

Climate change is not the only field to have defined words for specific probability ranges. The intelligence community has looked into this as well. They're called words of estimative probability.

A lot of the emphasis is on climate change, which has become partisan than other environmental issues. But other environmental issues have become partisan as well. Here's some data from a paper from 2013 by D.L. Guber, "A cooling climate for change? Party polarization and the politics of global warming."
 

The poll you linked indicates that Republicans in the Mountain West are more concerned with the environmental than Republicans in the rest of the country. There is a 27 p.p. partisan gap on the energy vs environment question (p. 17) - much less than t... (read more)

I think that this is a coincidence. Japan has low partisanship for environmentalism and has less nuclear power than most developed countries (along with low overall partisanship). The association would be between three things: (1) low partisanship for environmentalism, (2) high overall partisanship, and (3) lots of nuclear power plants. There aren't enough countries to do this kind of correlation.

From the introduction to the last post in this sequence: 

Environmentalists were not the only people making significant decisions here. Fossil fuel companies and conservative think tanks also had agency in the debate – and their choices were more blameworthy than the choices of environmentalists. Politicians choose who they do and do not want to ally with. My focus is on the environmental movement itself, because that is similar to what other activist groups are able to control.

The motivation for this report was to learn what the AI safety movement should do to keep from becoming partisan. 'Meta doesn't lobby the government' isn't an action the AI safety movement can take.

http://aiimpacts.org/wp-content/uploads/2023/04/Why-Did-Environmentalism-Become-Partisan-1.pdf

Thank you !

The links to the report are now fixed.

The 4 blog posts cover most of the same ground as the report. The report goes into more detail, especially in sections 5 & 6.

I think this is true of an environmentalist movement that wants there to be a healthy environment for humans; I'm not sure this is true of an environmentalist movement whose main goal is to dismantle capitalism.

I talk about mission creep in the report, section 6.6.

Part of 'making alliances with Democrats' involved environmental organizations adopting leftist positions on other issues. 

Different environmental organizations have seen more or less mission creep. The examples I give in the report are the women's issues for the World Wildlife Fund:

In many

This is trying to make environmentalism become partisan, but in the other direction.

Environmentalists could just not have positions on most controversial issues, and instead focus more narrowly on the environment.

There is also the far right in France, which is not the same as the right wing in America, but is also not Joe Biden. From what I can tell, the far right in France supports environmentalism.^[1]

Macron & Le Pen seem to have fairly similar climate policies. Both want France's electricity to be mostly nuclear – Le Pen more so. Both are not going to raise fuel taxes – Macron reluctantly. Le Pen talks more about hydrogen and reshoring manufacturing from countries which emit more (and claims that immigration is bad for France's environmental goals). Macron su... (read more)

I think it was possible for the environmental movement to form alliances with politicians in both parties, and for environmentalism to have remained bipartisan.

Comparing different countries and comparing the same country at different times is not the same thing as a counterfactual, but it can be very helpful for understanding counterfactuals. In this case, the counterfactual US is taken to be similar to the US in the 1980s or to the UK, France, or South Korea today.

I think you should ask the author of the song if it's referring to someone using powerful AI to do something transformative to the sun.

This is extremely obvious to me. The song is opposed to how the sun currently is, calling it "wasteful" and "distasteful" - the second word is a quote from a fictional character, but the first is not. It later talks about when "the sun's a battery," so something about the sun is going to change. I really don't know what "some big old computer" could be referring to if not powerful AI.

Thank you for responding! I am being very critical, both in foundational and nitpicky ways. This can be annoying and make people want to circle the wagons. But you and the other organizers are engaging constructively, which is great.

The distinction between Solstice representing a single coherent worldview vs. a series of reflections also came up in comments on a draft. In particular, the Spinozism of Songs Stay Sung feels a lot weirder if it is taken as the response to the darkness, which I initially did, rather than one response to the darkness.

Neverthele... (read more)

The London subway was private and returned enough profit to slowly expand while it was coal powered. Once it electrified, it became more profitable and expanded quickly.

The Baltimore tunnel was and is part of an intercity line that is mostly above ground. It was technologically similar to London, but operationally very different.

I chose the start date of 1866 because that is the first time the New York Senate appointed a committee to study rapid transit in New York, which concluded that New York would be best served by an underground railroad. It's also the start date that Katz uses.

The technology was available. London opened its first subway line in 1863. There is a 1.4 mi railroad tunnel from 1873 in Baltimore that is still in active use today. These early tunnels used steam engines. This did cause ventilation challenges, but they were resolvable. The other reasonable pre-electr... (read more)

The original version of the song reads to me as being deist or pantheist. You could replace 'God' with 'Nature' and the meaning would be almost the same. My view of Divinely Guided Evolution has a personal God fiddling with random mutations and randomly determined external factors to create the things He wants.

It is definitely anti-Young-Earth-Creationism, but it is also dismissive of the Bible. Even if you don't think that Genesis 1 should be treated as a chronology, I think that you should take the Bible seriously. Its commentary on what it means to be human is important.

Many of these seem reasonable. The "book of names" sounds to me like the Linnaean taxonomy, while the "book of night" sounds like astronomical catalogues. I don't know as much about geology, but the "book of earth" could be geological surveys.

This kind of science is often not exciting. Rutherford referred to it as "stamp collecting." It is very useful for the practice of future generations of scientists. For example, if someone wants to do a survey of various properties of binary star systems, they don't have to find a bunch of examples themselves (and wor... (read more)

If it were done at Lighthaven, it would have to be done outdoors. This does present logistical problems.

I would guess that making Lighthaven's outdoor space usable even if it rains would cost much less (an order of magnitude?) than renting out an event space, although it might cost other resources like planning time that are in more limited supply.

If Lighthaven does not want to subsidize Solstice, or have the space reserved a year in advance, then that would make this option untenable.

It's also potentially possible to celebrate Solstice in January, when event spaces are more available.

Staggering the gathering in time also works. Many churches repeat their Christmas service multiple times over the course of the day, to allow more people to come than can fit in the building.

There's another reason for openness that I should have made clearer. Hostility towards Others is epistemically and ethically corrosive. It makes it easier to dismiss people who do agree with you, but have different cultural markers. If a major thing that unifies the community is hostility to an outgroup, then it weakens the guardrails against actions based on hate or spite. If you hope to have compassion for all conscious creatures, then a good first step is to try to have compassion for the people close to you who are really annoying.

Christianity seems to

So I think the direction in which you would want Solstice to change -- to be more positive towards religion, to preach humility/acceptance rather than striving/heroism -- is antithetical to one of Solstice's core purposes.

While I would love to see the entire rationalist community embrace the Fulness of the Gospel of Christ, I am aware that this is not a reasonable ask for Solstice, and not something I should bet on in a prediction market. While I criticize the Overarching Narrative, I am aware that this is not something that I will change.

My hopes for chan... (read more)

also it’s a lot more work to setup

How hard would it be to project them? There was a screen, and it should be possible to project at least two lines with music large enough for people to read. Is the problem that we don't have sheet music that's digitized in a way to make this feasible for all of the songs?

This is more volunteer-based than I was expecting. I would have guessed that Solstice had a lot of creative work, the choir, and day-of work done by volunteers, but that the organizers and most of the performers were paid (perhaps below market rates). As it is, it is probably more volunteer-based than most Christmas programs.

I'll edit the original post to say that this suggestion is already being followed.

This kind of situation is dealt with in Quine's Two Dogmas of Empiricism, especially the last section, "Empiricism Without the Dogmas." This is a short (~10k words), straightforward, and influential work in the philosophy of science, so it is really worth reading the original.

Quine describes science as a network of beliefs about the world. Experimental measurements form a kind of "boundary conditions" for the beliefs. Since belief space is larger than the space of experiments which have been performed, the boundary conditions meaningfully constrain but do ... (read more)

I'm currently leaning towards

• kings and commonwealths and all

Tokamaks have been known for ages. We plausibly have gotten close to the best performance out of them that we could, without either dramatically increasing the size (ITER) or making the magnets significantly stronger. The high temperature superconducting^[1] 'tape' that Commonwealth Fusion has pioneered has allowed us to make stronger magnetic fields, and made it feasible to build a fusion power plant using a tokamak the size of JET.

After SPARC, Commonwealth Fusion plans to build ARC, which should actually ship electricity to customers. ARC should have... (read more)

OpenAI has to face off against giants like Google and Facebook, as well as other startups like Anthropic. There are dozens of other organizations in this space, although most are not as competitive as these.

Commonwealth Fusion has to face off against giants like ITER (funding maybe $22B, maybe $65B, estimates vary) and the China National Nuclear Corporation (building CFETR at ?? cost, while a much smaller experiment in China cost ~$1B), as well as other startups like Helion. The Fusion Industry Association has 37 members, which are all private companies tr... (read more)

I thought about including valuation in the table as well, but decided against it:

• I'm not sure how accurate startup valuations are. It make be less clear how to interpret what the funding received means, but the number is easier to measure accurately.
• These are young companies, so the timing of the valuation matters a lot. OpenAI's valuation is recent, or 8 years after the company was founded. Commonwealth Fusion's valuation is from 2 years ago, or 4 years after the company was founded. If each had multiple valuations, then I would have made a graph like Figure 1 for this.

The cost to build a tokamak that is projected to reach Q~10 has fallen by more than a factor of 10 in the last 6 years. CFS is building for $2B what ITER is building for maybe $22B, maybe $65B (cost estimates vary).

It's really not clear what the cost of fusion will end up being once it becomes mass produced.

Helion has raised a similar amount of capital as Commonwealth: $2.2B. Helion also has hundreds of employees: their LinkedIn puts them in the 201-500 employees category. It was founded in 2013, so it is a bit older than CFS or OpenAI.

My general sense is that there's more confidence in the plasma physics community that CFS will succeed than that Helion will succeed.

SPARC is a tokamak, and tokamaks have been extensively studied. SPARC is basically JET with a stronger magnetic field, and JET has been operational since the 1980s and has achieved Q=0.67. It's on... (read more)

I have now looked into this example, and talked to Bean at Naval Gazing about it.

I found data for the total tonnage in many countries' navies from 1865-2011. It seems to show overhang for the US navy during the interwar years, and maybe also for the Italian navy, but not for any of the other great powers.

Bean has convinced me that this data is not to be trusted. It does not distinguish between warships and auxiliary ships, or between ships in active duty and in reserve. It has some straightforward errors in the year the last battleship was decommissioned a... (read more)

The examples are things that look sort of like overhang, but are different in important ways. I did not include the hundreds of graphs I looked through that look nothing like overhang.

Your post reads, to me, as saying, "Better algorithms in AI may add new s-curves, but won't jump all the way to infinity, they'll level off after a while." 

The post is mostly not about either performance s-curves or market size s-curves. It's about regulation imposing a pause on AI development, and whether this would cause catch-up growth if the pause is ended.

Stacked s-curves can look like a pause + catch-up growth, but they are a different mechanism.

The Soviet Union did violate the Biological Weapons Convention, which seems like an example of "an important, binding, ratified arms treaty." It did not lead to nuclear war.

I did not look at the Washington Naval Conference as a potential example. It seems like it might be relevant. Thank you !

It seems to me that governments now believe that AI will be significant, but not extremely advantageous. 

I don't think that many policy makers believe that AI could cause GDP growth of 20+% within 10 years. Maybe they think that powerful AI would add 1% to GDP growth rates, which is definitely worth caring about. It wouldn't be enough for any country which developed it to become the most powerful country in the world within a few decades, and would be an incentive in line with some other technologies that have been rejected.

The UK has AI as one of the... (read more)

The impression of incuriosity is probably just because I collapsed my thoughts into a few bullet points.

The causal link between human intelligence and neurons is not just because they're both complicated. My thought process here is something more like:

• All instances of human intelligence we are familiar with are associated with a brain.
• Brains are built out of neurons.
• Neurons' dynamics looks very different from the dynamics of bits.
• Maybe these differences are important for some of the things brains can do.

It feels pretty plausible that the underlying archite... (read more)

Brains do these kinds of things because they run algorithms designed to do these kinds of things. 

If by 'algorithm', you mean thing-that-does-a-thing, then I think I agree. If by 'algorithm', you mean thing-that-can-be-implemented-in-python, then I disagree.

Perhaps a good analogy comes from quantum computing.* Shor's algorithm is not implementable on a classical computer. It can be approximated by a classical computer, at very high cost. Qubits are not bits, or combinations of bits. They have different underlying dynamics, which makes quantum computer... (read more)

I don't believe that "current AI is at human intelligence in most areas". I think that it is superhuman in a few areas, within the human range in some areas, and subhuman in many areas - especially areas where the things you're trying to do are not well specified tasks.

I'm not sure how to weight people who think most about how to build AGI vs more general AI researchers (median says HLAI in 2059, p(Doom) 5-10%) vs forecasters more generally.  There's a difference in how much people have thought about it, but also selection bias: most people who are sk... (read more)

From Yudkowsky's description of the AI-Box Experiment:

The Gatekeeper party may resist the AI party’s arguments by any means chosen – logic, illogic, simple refusal to be convinced, even dropping out of character – as long as the Gatekeeper party does not actually stop talking to the AI party before the minimum time expires.

One of the tactics listed on RationalWiki's description of the AI-box experiment is: 

Jump out of character, keep reminding yourself that money is on the line (if there actually is money on the line), and keep saying "no" over and over

The Lord of the Rings tells us that the hobbit’s simple notion of goodness is more effective at resisting the influence of a hostile artificial intelligence than the more complicated ethical systems of the Wise.

The miscellaneous quotes at the end are not directly connected to the thesis statement.

In practice, smoothness interacts with measurement: we can usually measure the higher-order bits without measuring lower-order bits, but we can’t easily measure the lower-order bits without the higher-order bits. Imagine, for instance, trying to design a thermometer which measures the fifth bit of temperature but not the four highest-order bits. Probably we’d build a thermometer which measured them all, and then threw away the first four bits! Fundamentally, it’s because of the informational asymmetry: higher-order bits affect everything, but lower-order b

It seems like your comment is saying something like:

These restrictions are more relevant to an Oracle than to other kinds of AI.
 

Unfortunately, decisions about units are made by a bunch of unaccountable bureaucrats. They would rather define the second in terms that only the techno-aristocracy can understand instead of using a definition that everyone can understand. It's time to turn control over our systems of measurement back to the people !
#DemocratizeUnits

Adding a compass is unlikely to also make the bird disoriented when exposed to a weak magnetic field which oscillates at the right frequency. Which means that the emulated bird will not behave like the real bird in this scenario. 

You could add this phenomenon in by hand. Attach some detector to your compass and have it turn off the compass when these fields are measured.

More generally, adding in these features ad hoc will likely work for the things that you know about ahead of time, but is very unlikely to work like the bird outside of its training di... (read more)