All of Ege Erdil's Comments + Replies

There are two arguments frequently offered for a free market economy over a centrally planned economy: an argument based around knowledge, sometimes called the socialist calculation problem; and another argument based on incentives. The arguments can be briefly summarized like so:

  • A central planning authority would not have enough knowledge to efficiently direct economic activity.
  • A central planning authority would not have the right incentives to ensure that their direction was efficient.

A point I've not seen anyone else make is that the argument from ... (read more)

This brings up another important point which is that a lot of externalities are impossible to calculate, and therefore such approaches end up fixating on the part that seems calculable without even accounting for (or even noticing) the incalculable part. If the calculable externalities happen to be opposed to larger incalculable externalities, then you can end up worse off than if you had never tried.

I think this is correct as a conditional statement, but I don't think one can deduce the unconditional implication that attempting to price some externali... (read more)

2jimmy
  It's not "attempting to price some externalities where many are difficult to price is generally bad", it's "attempting to price some externalities where the difficult to price externalities on the other side is bad". Sometimes the difficulty of pricing them means it's hard to know which side they primarily lie on, but not necessarily. The direction of legible/illegible externalities might be uncorrelated on average, but that doesn't mean that ignoring the bigger piece of the pie isn't costly. If I offer "I'll pay you twenty dollars, and then make up some rumors about you which may or may not be true and may greatly help or greatly harm your social standing",  you don't think "Well, the difficult part to price is a wash, but twenty dollars is twenty dollars" You still need a body. Sure, you can give people like Elisjsha Dicken a bunch of money, but that's because he actually blasted someone. If we want to pay him $1M per life he saved though, how much do we pay him? We can't simply go to the morgue and count how many people aren't there. We have to start making assumptions, modeling the system, and paying out based on our best guesses of what might have happened in what we think to be the relevant hypothetical. Which could totally work here, to be clear, but it's still a potentially imperfect attempt to price the illegible and it's not a coincidence that this was left out of the initial analysis that I'm responding. But what about the guy who stopped a shooting before it began, simply by walking around looking like the kind of guy that would stop an a spree killer before he accomplished much? What about the good role models in the potential shooters life that lead him onto the right track and stopped a shooting before it was ever planned? This could be ten times as important and you wouldn't even know without a lot of very careful analysis. And even then you could be mistaken, and good luck creating enough of a consensus on your program to pay out what you bel

In general, I don't agree with arguments of the form "it's difficult to quantify the externalities so we shouldn't quantify anything and ignore all external effects" modulo concerns about public choice ("what if the policy pursued is not what you would recommend but some worse alternative?"), which are real and serious, though out of the scope of my argument. There's no reason a priori to suppose that any positive or negative effects not currently priced will be of the same order of magnitude.

If you think there are benefits to having a population where mos... (read more)

1Jakub Supeł
There are some a posteriori reasons though - there are numerous studies that reject a causal link between the number of firearms and homicides, for example. This indicates that firearm manufacturers do not cause additional deaths, and therefore it would be wrong to only internalize the negative costs. That's not true. It is not better, because providing appropriate incentives is very likely impossible in this case, e.g.: - due to irrational political reasons (people have irrational fear of guns and will oppose any efforts to incentivize their purchase, while supporting efforts to disincentivize it); - due to the fact that a reward system for preventing crime can be easily gamed (cobra effect), not to mention the fact that it will probably be very costly to follow up on all cases when crime was prevented; - due to the fact that positive outcomes of gun ownership are inherently hard to quantify, hence in reality they will not be quantified and will not be taken into account (McNamara fallacy).

If the risk is sufficiently high, then the shops would simply not sell guns to anyone who seemed like they might let their guns be stolen, for example. Note that the shops would still be held liable for any harm that occurs as a result of any gun they have sold, irrespective of whether the buyer was also the perpetrator of the harm.

In practice, the risk of a gun sold to a person with a safe background being used in such an act is probably not that large, so such a measure doesn't need to be taken: the shop can just sell the guns at a somewhat inflated pric... (read more)

If the risk is sufficiently high, then the shops would simply not sell guns to anyone who seemed like they might let their guns be stolen, 

 

You do realize it is illegal to discriminate against customers on the basis of things like race, income, where they live, etc, right?

So, step 1 in this plan has to begin with "dismantle the last 60 years of civil rights legislation".

Open source might be viable if it's possible for the producers to add safeguards into the model that cannot be trivially undone by cheap fine-tuning, but yeah, I would agree with that given the current lack of techniques for doing this successfully.

The shop has the ability to invest more in security if they will be held liable for subsequent harm. They can also buy insurance themselves and pass on the cost to people who do purchase guns legally as an additional operating expense.

4Logan Zoellner
How is the shop going to stop a gun from being stolen from a gun owner (who happened to buy their gun at the shop)?  This seems much more  the domain of the law.  The police can arrest people who steal guns, shop owners cannot.

It is not a tautology.

Can you explain to me the empirical content of the claim, then? I don't understand what it's supposed to mean.

About the rest of your comment, I'm confused about why you're discussing what happens when both chess engines and humans have a lot of time to do something. For example, what's the point of this statement?

My understanding is that it is not true that if you ran computers for a long time that they would beat the human also running for a long time, and that historically, it's been quite the opposite...

I don't understand ho... (read more)

Yes, that's what I'm trying to say, though I think in actual practice the numbers you need would have been much smaller for the Go AIs I'm talking about than they would be for the naive tree search approach.

Sure, but in that case I would not say the AI thinks faster than humans, I would say the AI is faster than humans at a specific range of tasks where the AI can do those tasks in a "reasonable" amount of time.

As I've said elsewhere, there is a quality or breadth vs serial speed tradeoff in ML systems: a system that only does one narrow and simple task can do that task at a high serial speed, but as you make systems more general and get them to handle more complex tasks, serial speed tends to fall. The same logic that people are using to claim GPT-4 thinks f... (read more)

5Vladimir_Nesov
This notion of thinking speed makes sense for large classes of tasks, not just specific tasks. And a natural class of tasks to focus on is the harder tasks among all the tasks both systems can solve. So in this sense a calculator is indeed much faster than GPT-4, and GPT-4 is 2 OOMs faster than humans. An autonomous research AGI is capable of autonomous research, so its speed can be compared to humans at that class of tasks. AI accelerates the pace of history only when it's capable of making the same kind of progress as humans in advancing history, at which point we need to compare their speed to that of humans at that activity (class of tasks). Currently AIs are not capable of that at all. If hypothetically 1e28 training FLOPs LLMs become capable of autonomous research (with scaffolding that doesn't incur too much latency overhead), we can expect that they'll be 1-2 OOMs faster than humans, because we know how they work. Thus it makes sense to claim that 1e28 FLOPs LLMs will accelerate history if they can do research autonomously. If AIs need to rely on extensive search on top of LLMs to get there, or if they can't do it at all, we can instead predict that they don't accelerate history, again based on what we know of how they work.

True, but isn't this almost exactly analogously true for neuron firing speeds? The corresponding period for neurons (10 ms - 1 s) does not generally correspond to the timescale of any useful cognitive work or computation done by the brain.

Yes, which is why you should not be using that metric in the first place.

But even the top-line number is (at least theoretically) a very concrete measure of something that you can actually get out of the system. In contrast, when used in "computational equivalence" estimates of the brain, FLOP/s are (somewhat dubious

... (read more)
8Max H
Well, clock speed is a pretty fundamental parameter in digital circuit design. For a fixed circuit, running it at a 1000x slower clock frequency means an exactly 1000x slowdown. (Real integrated circuits are usually designed to operate in a specific clock frequency range that's not that wide, but in theory you could scale any chip design running at 1 GHz to run at 1 KHz or even lower pretty easily, on a much lower power budget.) Clock speeds between different chips aren't directly comparable, since architecture and various kinds of parallelism matter too, but it's still good indicator of what kind of regime you're in, e.g. high-powered / actively-cooled datacenter vs. some ultra low power embedded microcontroller. Another way of looking at it is power density: below ~5 GHz or so (where integrated circuits start to run into fundamental physical limits), there's a pretty direct tradeoff between power consumption and clock speed.  A modern high-end IC (e.g. a desktop CPU) has a power density on the order of 100 W / cm^2. This is over a tiny thickness; assuming 1 mm you get a 3-D power dissipation of 1000 W / cm^3 for a CPU vs. human brains that dissipate ~10 W / 1000 cm^3 = 0.01 watts / cm^3. The point of this BOTEC is that there are several orders of magnitude of "headroom" available to run whatever the computation the brain is performing at a much higher power density, which, all else being equal, usually implies a massive serial speed up (because the way you take advantage of higher power densities in IC design is usually by simply cranking up the clock speed, at least until that starts to cause issues and you have to resort to other tricks like parallelism and speculative execution). The fact that ICs are bumping into fundamental physical limits on clock speed suggests that they are already much closer to the theoretical maximum power densities permitted by physics, at least for silicon-based computing. This further implies that, if and when someone does fig
5Vladimir_Nesov
This notion of thinking speed depends on the difficulty of a task. If one of the systems can't solve a problem at all, it's neither faster nor slower. If both systems can solve a problem, we can compare the time they take. In that sense, current LLMs are 1-2 OOMs faster than humans at the tasks both can solve, and much cheaper. Old chess AIs were slower than humans good at chess. If future AIs can take advantage of search to improve quality, they might again get slower than humans at sufficiently difficult tasks, while simultaneously being faster than humans at easier tasks.

Sure, but from the point of view of per token latency that's going to be a similar effect, no?

I think you might have accidentally linked to your comment instead of the LessWrong post you intended to link to.

Don't global clock speeds have to go down as die area goes up due to the speed of light constraint?

For instance, if you made a die with 1e15 MAC units and the area scaled linearly, you would be looking at a die that's ~ 2e9 times larger than H100's die size, which is about 1000 mm^2. The physical dimensions of such a die would be around 2 km^2, so the speed of light would limit global clock frequencies to something on the order of c/(1 km) ~= 300 kHz, which is not 1 million times faster than the 1 kHz you attribute to the human brain. If you need multiple ... (read more)

1Tao Lin
no clock speed stays the same, but clock cycle latency of communication between regions increases. Just like CPUs require more clock cycles to access memory than they used to.
[anonymous]101

Don't global clock speeds have to go down as die area goes up due to the speed of light constraint?

Yes if you use 1 die with 1 clock domain, they would.  Modern chips don't.

For instance, if you made a die with 1e15 MAC units and the area scaled linearly, you would be looking at a die that's ~ 2e9 times larger than H100's die size, which is about 1000 mm^2. The physical dimensions of such a die would be around 2 km^2, so the speed of light would limit global clock frequencies to something on the order of c/(1 km) ~= 300 kHz, which is not 1 million time

... (read more)
4[comment deleted]

I think counterexamples are easy to find. For example, chess engines in 1997 could play at the level of top human chess players on consumer hardware, but only if they were given orders of magnitude more time to think than the top humans had available. Around 1997 Deep Blue was of a similar strength to Kasparov, but it had to run on a supercomputer; on commercial hardware chess engines were still only 2400-2500 elo. If you ran them for long enough, though, they would obviously be stronger than even Deep Blue was.

I think the claim that "in every case where w... (read more)

7gwern
It is not a tautology. Er, this is why I spent half my comment discussing correspondence chess... My understanding is that it is not true that if you ran computers for a long time that they would beat the human also running for a long time, and that historically, it's been quite the opposite: the more time/compute spent, the better the human plays because they have a more scalable search. (eg. Shannon's 'type-A strategy vs type-B strategy' was meant to cover this distinction: humans search moves slower, but we search moves way better, and that's why we win.) It was only at short time controls where human ability to plan deeply was negated that chess engines had any chance. (In correspondence chess, they deeply analyze the most important lines of play and avoid getting distracted, so they can go much deeper into the game tree that chess engines could.) Whenever the crossover happened, it was probably after Deep Blue. And given the different styles of play and the asymptotics of how those AIs smash into the exponential wall of the game tree & explode, I'm not sure there is any time control at which you would expect pre-Deep-Blue chess to be superhuman. This is similar to ML scaling. While curves are often parallel and don't cross, curves often do cross; what they don't do is criss-cross repeatedly. Once you fall behind asymptotically, you fall behind for good. I'm not familiar with 'correspondence Go' but I would expect that if it was pursued seriously like correspondence chess, it would exhibit the same non-monotonicity. That's not obvious to me. If nothing else, that time isn't useful once you run out of memory. And PCs had very little memory: you might have 16MB RAM to work with. And obviously, I am not discussing the trivial case of changing the task to a different task by assuming unlimited hardware or ignoring time and just handicapping human performance by holding their time constant while assigning unlimited resources to computers - in which case we had s
2Steven Byrnes
I understand this as saying “If you take an AI Go engine from the pre-AlphaGo era, it was pretty bad in real time. But if you set the search depth to an extremely high value, it would be superhuman, it just might take a bajillion years per move. For that matter, in 1950, people had computers, and people knew how to do naive exhaustive tree search, so they could already make an algorithm that was superhuman at Go, it’s just that it would take like a googol years per move and require galactic-scale memory banks etc.” Is that what you were trying to say? If not, can you rephrase?

If there are people who say “current AIs think many orders of magnitude faster than humans”, then I agree that those people are saying something kinda confused and incoherent, and I am happy that you are correcting them.

Eliezer himself has said (e.g. in his 2010 debate with Robin Hanson) that one of the big reasons he thinks CPUs can beat brains is because CPUs run at 1 GHz while brains run at 1-100 Hz, and the only barrier is that the CPUs are currently running "spreadsheet algorithms" and not the algorithm used by the human brain. I can find the exact... (read more)

7Steven Byrnes
Thanks. I’m not Eliezer so I’m not interested in litigating whether his precise words were justified or not. ¯\_(ツ)_/¯ I’m not sure we’re disagreeing about anything substantive here. That’s probably not what I meant, but I guess it depends on what you mean by “task”. For example, when a human is tasked with founding a startup company, they have to figure out, and do, a ton of different things, from figuring out what to sell and how, to deciding what subordinates to hire and when, to setting up an LLC and optimizing it for tax efficiency, to setting strategy, etc. etc. One good human startup founder can do all those things. I am claiming that one AI can do all those things too, but at least 1-2 OOM faster, wherever those things are unconstrained by waiting-for-other-people etc. For example: If the AI decides that it ought to understand something about corporate tax law, it can search through online resources and find the answer at least 10-100× faster than a human could (or maybe it would figure out that the answer is not online and that it needs to ask an expert for help, in which case it would find such an expert and email them, also 10-100× faster). If the AI decides that it ought to post a job ad, it can figure out where best to post it, and how to draft it to attract the right type of candidate, and then actually write it and post it, all 10-100× faster. If the AI decides that it ought to look through real estate listings to make a shortlist of potential office spaces, it can do it 10-100× faster. If the AI decides that it ought to redesign the software prototype in response to early feedback, it can do so 10-100× faster. If the AI isn’t sure what to do next, it figures it out, 10-100× faster. Etc. etc. Of course, the AI might use or create tools like calculators or spreadsheets or LLMs, just as a human might, when it’s useful to do those things. And the AI would do all those things really well, at least as well as the best remote-only human startup founder

As far as I know, in every case where we've successfully gotten AI to do a task at all, AI has done that task far far faster than humans. When we had computers that could do arithmetic but nothing else, they were still much faster at arithmetic than humans. Whatever your view on the quality of recent AI-generated text or art, it's clear that AI is producing it much much faster than human writers or artists can produce text/art.

"Far far faster" is an exaggeration that conflates vastly different orders of magnitude with each other. When compared against... (read more)

3cubefox
I think there is a weaker thesis which still seems plausible: For every task for which an ML system achieves human level performance, it is possible to perform the task with the ML system significantly faster than a human. The restriction to ML models excludes hand-coded GOFAI algorithms (like Deep Blue), which in principle could solve all kinds of problems using brute force search.

Yes, this summary seems accurate.

Answer by Ege Erdil10-2

I thought cryonics was unlikely to work because a bunch of information might be lost even at the temperatures that bodies are usually preserved in. I now think this effect is most likely not serious and cryonics can work in principle at the temperatures we use, but present-day cryonics is still unlikely to work because of how much tissue damage the initial process of freezing can do.

2Andy_McKenzie
Out of curiosity, what makes you think that the initial freezing process causes too much information loss? 

As I said, I think it's not just that the language is poetic. There is an implicit inference that goes like

  1. People who would not voluntarily undergo surgery without long-term adverse effects on their health to improve the life of a stranger are evil.
  2. Most researchers who would be in a position to know the state of the evidence on the long-term adverse health effects of kidney donation don't personally donate one of their kidneys.
  3. Most researchers are unlikely to be evil.
  4. So it's unlikely that most researchers believe kidney donation has no long-term adver
... (read more)

I don't think it's a matter of poetic license. You're making an empirical claim that if specialists actually believed kidney donation had no long-term side effects, they would be lining up to donate their kidneys and we would see a much higher rate of kidney donations in the US. I think this claim is wrong because the inconvenience of surgery is substantial enough to block people from donating their kidneys even in the absence of long-term side effects.

The use of the word "evil" sneaks in an assumption that most people would be happy to make this tradeoff ... (read more)

Have most of the researchers looking at kidney donation donated a kidney? Have most nephrology researchers donated a kidney? Most surgeons doing kidney transplants? Obviously not, otherwise we’d have more than 200 donations to strangers each year in the US. There are 10,000 board-certified nephrologists, and a few more hundred are added each year, if they took this data seriously they’d all donate.

Heck, on top of those you can add nephrology researchers, the medical statisticians who happen to focus on kidney disease, transplant surgeons, and all well-info

... (read more)
-6George3d6

There is more data, and better data, e.g. data gathered in double-blinded RCTs, that shows things like:

  1. Homeopathy works very well for a variety of conditions, sometimes better than real drugs used to treat them.
  2. Increasing the healthcare budget and the amount of healthcare people receive. Both in rich countries (e.g. USA) and poor ones (India). Having no effect on mortality.

I can make both of these claims based on many individual RCTs, as well as based on the aggregation of all existing RCTs.

I’m not saying that these claims make sense, they don’t, there are

... (read more)
4George3d6
Bingo, partially, it's likely that at least in the Indian study the mortality was too low over that period to be accurately represented ... which is the same argument I'd have for 100% of the kidney donation studies, follow-up is not lengthy enough, and the longer you followup and the stronger your controls the worse things get. Death is a bad endpoint for evaluating things and thus we should not be using it. ---------------------------------------- I would have a longer claim (in the linked article) that in some cases it is worth using, given that e.g. our views around why modern medicine is good and worthwhile ultimately root themselves in preventing mortality and such things are as of yet on shaky grounds. But when doing risk estimates we should try looking at proxies for mortality and QAL downgrades as opposed to mortality, especially when we don't have life-long studies or studies following people into old age when most of them start dying.

I don't think those ratings are comparable. On the other hand, my estimate of 3d was apparently lowballing it based on some older policy networks, and newer ones are perhaps as strong as 4d to 6d, which on the upper end is still weaker than professional players but not by much.

However, there is a big gap between weak professional players and "grandmaster level", and I don't think the raw policy network of AlphaGo could play competitively against a grandmaster level Go player.

This is not quite true. Raw policy networks of AlphaGo-like models are often at a level around 3 dan in amateur rankings, which would qualify as a good amateur player but nowhere near the equivalent of grandmaster level. If you match percentiles in the rating distributions, 3d in Go is perhaps about as strong as an 1800 elo player in chess, while "master level" is at least 2200 elo and "grandmaster level" starts at 2500 elo.

Edit: Seems like policy networks have improved since I last checked these rankings, and the biggest networks currently available for p... (read more)

8Buck
According to figure 6b in "Mastering the Game of Go without Human Knowledge", the raw policy network has 3055 elo, which according to this other page (I have not checked that these Elos are comparable) makes it the 465th best player. (I don’t know much about this and so might be getting the inferences wrong, hopefully the facts are useful)

I think you're ignoring the qualifier "literally portrayed" in Matthew's sentence, and neglecting the prior context that he's talking about AI development being something mainly driven forward by hobbyists with no outsized impacts.

He's talking about more than just the time in which AI goes from e.g. doubling the AI software R&D output of humans to some kind of singularity. The specific details Eliezer has given about this scenario have not been borne out: for example, in his 2010 debate with Robin Hanson, he emphasized a scenario in which a few people ... (read more)

habrykaΩ71012

Hmm, I do agree the foom debates talk a bunch about a "box in a basement team", but the conversation was pretty explicitly not about the competitive landscape and how many people are working on this box in a basement, etc. It was about whether it would be possible for a box in a basement with the right algorithms to become superhuman in a short period of time. In-particular Eliezer says: 

In other words, I’m trying to separate out the question of “How dumb is this thing (points to head); how much smarter can you build an agent; if that agent were telep

... (read more)
5Daniel Kokotajlo
I agree that insofar as Yudkowsky predicted that AGI would be built by hobbyists with no outsized impacts, he was wrong. ETA: So yes, I was ignoring the "literally portrayed" bit, my bad, I should have clarified that by "yudkowsky's prediction" I meant the prediction about takeoff speeds.

I assume John was referring to Unitary Evolution Recurrent Neural Networks which is cited in the "Orthogonal Deep Neural Nets" paper.

6johnswentworth
You're right, I linked the wrong one. Thanks. Fixed now.

It might be right, I don't know. I'm just making a local counterargument without commenting on whether the 2.5 PB figure is right or not, hence the lack of endorsement. I don't think we know enough about the brain to endorse any specific figure, though 2.5 PB could perhaps fall within some plausible range.

While I wouldn't endorse the 2.5 PB figure itself, I would caution against this line of argument. It's possible for your brain to contain plenty of information that is not accessible to your memory. Indeed, we know of plenty of such cognitive systems in the brain whose algorithms are both sophisticated and inaccessible to any kind of introspection: locomotion and vision are two obvious examples.

6Noosphere89
I do want to ask why don't you think the 2.5 petabyte figure is right, exactly?

I downvoted this comment for its overconfidence.

First of all, the population numbers are complete garbage. This is completely circular. You are just reading out the beliefs about history used to fabricate them. The numbers are generated by people caring about the fall of Rome. The fall of Rome didn't cause of decline in China. Westerners caring about the fall of Rome caused the apparent decline in China.

I will freely admit that I don't know how population numbers are estimated in every case, but your analysis of the issue is highly simplistic. Estimate... (read more)

Well, that's true, but at some level, what else could it possibly be? What other cause could be behind the long-run expansion in the first place, so many millennia after humans spanned every continent but Antarctica?

Technological progress being responsible for the long-run trend doesn't mean you can attribute local reversals to humans hitting limits to technological progress. Just as a silly example, the emergence of a new strain of plague could have led to the depopulation of urban centers, which lowers R&D efficiency because you lose concentration... (read more)

2AnthonyC
Yes, it's a possible story. And yes, wars and plagues impact tech development and infrastructure all the time. But I find it hard to think about how to draw a principled distinction between local growth slowdowns and hitting local limits to technological growth. For example, in medieval Europe the Black Plague definitely depopulated the continent in ways that affected lots and lots of things, and there was no way they could reasonably have known how to do better. Resolving the plague wasn't possible because the solutions were beyond the local limits of technological growth. If the same plague struck today, that wouldn't happen, because we have the tools to deal with it. We understand sanitation, and disease vectors, and we can develop vaccines. It would be a blip of a decade and a few percent GDP growth, rather than half the population and centuries of recovery.

McEvedy and Jones actually discuss a regional breakdown in the final section of the book, but they speculate too much for the discussion to be useful, I think. They attribute any substantial slowdown in growth rates to population running up against technological limits, which seems like a just-so story that could explain anything.

They note that the 3rd century AD appears to have been a critical time, as it's when population growth trends reversed in both Europe and China at around the same time: in Europe with the Crisis of the Third Century, and in China ... (read more)

2AnthonyC
Well, that's true, but at some level, what else could it possibly be? What other cause could be behind the long-run expansion in the first place, so many millennia after humans spanned every continent but Antarctica? I'm very skeptical about explanations involving wars and plagues, except insofar as those impact technological development and infrastructure, because a handful of generations is plenty to get back to the Malthusian limit even if a majority of the population dies in some major event (especially regional events where you can then also get migration or invasion from less affected regions). I guess because they use nice round year numbers as cutoffs there could be artifacts of events right near the beginning or end of a millennium, but things happening in the 2nd or 3rd century AD aren't candidates for that.

I've actually written about this subject before, and I agree that the first plague pandemic could have been significant: perhaps killing around 8% of the global population in the four years from 541 to 544. However, it's also worth noting that our evidence for this decline is rather scant; we know that the death toll was very high in Constantinople but not much about what happened outside the capital, mostly because nobody was there to write it down. So it's also entirely conceivable that the death toll was much lower than this. The controversy about this ... (read more)

In the west, I think the fall of the Western Roman Empire was probably a significant hit, and caused a major setback in economic growth in Europe.

Attribution of causality is tricky with this event, but I would agree if you said the fall coincided with a major slowdown in European economic growth.

China had its bloody Three Kingdom period, and later the An Lushan rebellion.

I think a problem re: China is that a lot of population decline estimates for China are based on the official census, and as far as I know China didn't have a formal census before t... (read more)

2dr_s
Yes, I suppose the arrow could go the other way around - that economic recession caused the fall. Or really, probably just a feedback loop of stuff going to shit. Sorry for the unwarranted implication. Yeah, just suggesting possible sources. But also, any estimates of population growth in the 1-1000 AD range must account for China, so if we can't trust the census, are you sure your figures too aren't affected by this fundamental problem? Anyway, this looks like an interesting history problem - first, figuring out if the effect is real, and then, if it is, what caused it. But there's probably enough research for a PhD, or even a whole career, in such a wide field. It's a super complex question.

If people vote as if their individual vote determines the vote of a non-negligible fraction of the voter pool, then you only need (averaged over the whole population, so the value of the entire population is instead of , which seems much more realistic.

So voting blue can make sense for a sufficiently large coalition of "ordinary altruists" with who are able to pre-commit to their vote and think people outside the coalition might vote blue by mistake etc. rather than the "extraordinary altruists" we need in the original situatio... (read more)

That would be questioning the assumption that your cost function as an altruist should be linear in the number of lives lost. I'm not sure why you would question this assumption, though; it seems rather unnatural to make this a concave function, which is what you would need for your logic to work.

I'm not quite sure what you mean by that.

Unless I expect the pool of responders to be 100% rational and choose red, then I should expect some to choose blue. Since I (and presumably other responders) do expect some to choose blue, that makes >50% blue the preferred outcome. Universal red is just not a realistic outcome.

Whether or not I choose blue then depends on factors like how I value the lives of others compared to mine, the number of responders, etc - as in the equations in your post.

Emperically, as GeneSmith points out, something is wrong with Wal... (read more)

I'm surprised by how much this post is getting upvoted. It gives us essentially zero information about any question of importance, for reasons that have already been properly explained by other commenters:

  • Chess is not like the real world in important respects. What the threshold is for material advantage such that a 1200 elo player could beat Stockfish at chess tells us basically nothing about what the threshold is for humans, either individually or collectively, to beat an AGI in some real-world confrontation. This point is so trivial that I feel somew

... (read more)

(I'm the main KataGo dev/researcher)

Just some notes about KataGo - the degree to which KataGo has been trained to play well vs weaker players is relatively minor. The only notable thing KataGo does is in some self-play games to give up to an 8x advantage in how many playouts one side has over the other side, where each side knows this. (Also KataGo does initialize some games with handicap stones to make them in-distribution and/or adjust komi to make the game fair). So the strong side learns to prefer positions that elicit higher chance of mistakes by the ... (read more)

If someone could try to convince me that this experiment was not pointless and actually worth running for some reason, I would be interested to hear their arguments. Note that I'm more sympathetic to "this kind of experiment could be valuable if ran in the right environment", and my skepticism is specifically about running it for chess.

I've been interested in the study of this question for a while. I agree this post has the flaws you point out, but I still find that it provides interesting evidence. If the result had been that Stockfish would have continue... (read more)

5MichaelStJules
I think it's more illustrative than anything, and a response to Robert Miles using chess against Magnus Carlsen as an analogy for humans vs AGI. The point is that a large enough material advantage can help someone win against a far smarter opponent. Somewhat more generally, I think arguments for AI risk often put intelligence on a pedestal, without addressing its limitations, including the physical resource disadvantages AGIs will plausibly face. I agree that the specifics of chess probably aren't that helpful for informing AI risk estimates, and that a better tuned engine could have done better against the author. Maybe better experiments to run would be playing real-time strategy games against a far smarter but materially disadvatanged AI, but this would also limit the space of actions an AI could take relative to the real world.

Are neural networks trained using reinforcement learning from human feedback in a sufficiently complex environment biased towards learning the human simulator or the direct translator, in the sense of the ELK report?

I think there are arguments in both directions and it's not obvious which solution a neural network would prefer if trained in a sufficiently complex environment. I also think the question is central to how difficult we should expect aligning powerful systems trained in the current paradigm to be.

I'm curious if these rate limits were introduced as a consequence of some recent developments. Has the website been having more problems with spam and low-quality content lately, or has the marginal benefit of making these changes gone up in some other way?

It could also be that you had this idea only recently and in retrospect it had been a good idea for a long time, of course.

I first thought about them ~8 months ago, simply because we spend ~20 minutes a day reviewing content from new or downvoted users, and it's a combination of "adds up to a lot of time" and "also kind of emotionally exhausting to think about exactly where the line is where we should take some kind of action." 

The idea of auto-rate-limits felt a lot more salient during the April spike, where a lot of people showed up due to the Eliezer TIME article and other "AI in the news" things. That has since calmed down, but I think we'll get more things like that ... (read more)

Yes, in practice having a model of what is actually driving the correlations can help you do better than these estimates. A causal model would be helpful for that.

The product estimate for the expected correlation is only useful in a setting where nothing else is known about the relationship between the three variables than the two correlations, but in practice you often have some beliefs about what drives the correlations you observe, and if you're a good Bayesian you should of course also condition on all of that.

That's a reasonable picture to have in expectation, yeah.

As an aside, I've tried to work out what the optimal learning rate for a large language model should be based on the theory in the post, and if I'm doing the calculations correctly (which is a pretty big if) it doesn't match actual practice very well, suggesting there is actually something important missing from this picture.

Essentially, the coefficient should be where is the variance of the per-parameter noise in SGD. If you have a learning rate , you scale the objective you're optimizing by a factor and the noise variance by a factor . Lik... (read more)

That's useful to know, thanks. Is anything else known about the properties of the noise covariance beyond "it's not constant"?

Some comments on the paper itself: if the problem is that SGD with homoskedastic Gaussian noise fails to converge to a stationary distribution, why don't they define SGD over a torus instead? Seems like it would fix the problem they are talking about, and if it doesn't change the behavior it means their explanation of what's going on is incorrect.

If the only problem is that with homoskedastic Gaussian noise convergence to a stationa... (read more)

4interstice
Good question. I imagine that would work but it would converge more slowly. I think a more important issue is that the homoskedastic/heteroskedastic noise cases would have different equilibrium distributions even if both existed(they don't say this but it seems intuitively obvious since there would be a pressure away from points with higher noise in the heteroskedastic case). I guess on the torus this would correspond to there being a large number of bad minima which dominate the equilibrium in the homoskedastic case. Generally speaking the SGD noise seems to provide a regularizing effect towards 'flatter' solutions. The beginning of this paper has a good overview.

Check the Wikipedia section for the stationary distribution of the overdamped Langevin equation.

I should probably clarify that it's difficult to have a rigorous derivation of this claim in the context of SGD in particular, because it's difficult to show absence of heteroskedasticity in SGD residuals. Still, I believe that this is probably negligible in practice, and in principle this is something that can be tested by experiment.

4interstice
This might not hold in practice in fact, see this paper.

Sure, I agree that I didn't put this information into the post. However, why do you need to know which is more likely to know anything about e.g. how neural networks generalize?

I understand that SLT has some additional content beyond what is in the post, and I've tried to explain how you could make that fit in this framework. I just don't understand why that additional content is relevant, which is why I left it out.

As an additional note, I wasn't really talking about floating point precision being the important variable here. I'm just saying that if you... (read more)

5tgb
In my view, it's a significant philosophical difference between SLT and your post that your post talks only about choosing macrostates while SLT talks about choosing microstates. I'm much less qualified to know (let alone explain) the benefits of SLT, though I can speculate. If we stop training after a finite number of steps, then I think it's helpful to know where it's converging to. In my example, if you think it's converging to (0,1), then stopping close to that will get you a function that doesn't generalize too well. If you know it's converging to (0,0) then stopping close to that will get you a much better function - possibly exactly equally as good as you pointed out due to discretization. Now this logic is basically exactly what you're saying in these comments! But I think if someone read your post without prior knowledge of SLT, they wouldn't figure out that it's more likely to converge to a point near (0,0) than near (0,1). If they read an SLT post instead, they would figure that out. In that sense, SLT is more useful. I am not confident that that is the intended benefit of SLT according to its proponents, though. And I wouldn't be surprised if you could write a simpler explanation of this in your framework than SLT gives, I just think that this post wasn't it.

You need to discretize the function before taking preimages. If you just take preimages in the continuous setting, of course you're not going to see any of the interesting behavior SLT is capturing.

In your case, let's say that we discretize the function space by choosing which one of the functions you're closest to for some . In addition, we also discretize the codomain of by looking at the lattice for some . Now, you'll notice that there's a radius disk around the origin which contains only functions mapping to th... (read more)

4tgb
Everything I wrote in steps 1-4 was done in a discrete setting (otherwise |A−1(f0)| is not finite and whole thing falls apart). I was intending θ to be pairs of floating point numbers and A to be floats to floats. However, using that I think I see what you're trying to say. Which is that θ1θ2 will equal zero for some cases where θ1 and θ2 are both non-zero but very small and will multiply down to zero due to the limits of floating point numbers. Therefore the pre-image of A−1(f0) is actually larger than I claimed, and specifically contains a small neighborhood of (0,0). That doesn't invalidate my calculation that shows that (0,0) is equally likely as (0,1) though: they still have the same loss and A-complexity (since they have the same macrostate). On the other hand, you're saying that there are points in parameter space that are very close to (0,0) that are also in this same pre-image and also equally likely. Therefore even if (0,0) is just as likely as (0,1), being near to (0,0) is more likely than being near to (0,1). I think it's fair to say that that is at least qualitatively the same as SLT gives in the continous version of this. However, I do think this result "happened" due to factors that weren't discussed in your original post, which makes it sound like it is "due to" A-complexity. A-complexity is a function of the macrostate, which is the same at all of these points and so does not distinguish between (0,0) and (0,1) at all. In other words, your post tells me which f is likely while SLT tells me which θ is likely - these are not the same thing. But you clearly have additional ideas not stated in the post that also help you figure out which θ is likely. Until that is clarified, I think you have a mental theory of this which is very different from what you wrote.

I'm not too sure how to respond to this comment because it seems like you're not understanding what I'm trying to say.

I agree there's some terminology mismatch, but this is inevitable because SLT is a continuous model and my model is discrete. If you want to translate between them, you need to imagine discretizing SLT, which means you discretize both the codomain of the neural network and the space of functions you're trying to represent in some suitable way. If you do this, then you'll notice that the worse a singularity is, the lower the -complexity of ... (read more)

3tgb
This is where we diverge. Please let me know where you think my error is in the following. Returning to my explicit example (though I wrote f(θ) originally but will instead use A(θ) in this post since that matches your definitions). 1. Let f0(x)=0x  be the constant zero function and S=A−1(f0).  2. Observe that S is the minimal loss set under our loss function and also S is the set of parameters θ=(θ1,θ2) where θ1=0 or θ2=0. 3. Let α,β∈S . Then A−1(α)=f0=A−1(β) by definition of S. Therefore, c(A(α))=c(A(β)). 4. SLT says that θ=(0,0) is a singularity of S but that θ=(0,1)∈S is not a singularity. 5. Therefore, there exists a singularity (according to SLT) which has identical A-complexity (and also loss) as a non-singular point, contradicting your statement I quote.
4interstice
I think the implied claim is something like "analyzing the singularities of the model will also be helpful for understanding SGD in more realistic settings" or maybe just "investigating this area further will lead to insights which are applicable in more realistic settings". I mostly don't buy it myself.

I don't think this representation of the theory in my post is correct. The effective dimension of the singularity near the origin is much higher, e.g. because near every other minimal point of this loss function the Hessian doesn't vanish, while for the singularity at the origin it does vanish. If you discretized this setup by looking at it with a lattice of mesh , say, you would notice that the origin is surrounded by many parameters that give nearly identical loss, while near other parts of the space the number of such parameters is far fewer.

The reason... (read more)

4tgb
As I read it, the arguments you make in the original post depend only on the macrostate f, which is the same for both the singular and non-singular points of the minimal loss set (in my example), so they can't distinguish these points at all. I see that you're also applying the logic to points near the minimal set and arguing that the nearly-optimal points are more abundant near the singularities than near the non-singularities. I think that's a significant point not made at all in your original point that brings it closer to SLT, so I'd encourage you to add it to the post. I think there's also terminology mismatch between your post and SLT. You refer to singularities of A(i.e. its derivative is degenerate) while SLT refers to singularities of the set of minimal loss parameters. The point θ=(0,1) in my example is not singular at all in SLT but A(θ) is singular. This terminology collision makes it sound like you've recreated SLT more than you actually have.

Can you give an example of which has the mode of singularity you're talking about? I don't think I'm quite following what you're talking about here.

In SLT is assumed analytic, so I don't understand how the Hessian can fail to be well-defined anywhere. It's possible that the Hessian vanishes at some point, suggesting that the singularity there is even worse than quadratic, e.g. at the origin or something like that. But even in this regime essentially the same logic is going to apply - the worse the singularity, the further away you can move ... (read more)

4interstice
Yeah sorry that was probably needlessly confusing, I was just referencing the image in Jesse's tweet for ease of illustration(you're right that it's not analytic, I'm not sure what's going on there) The Hessian could also just be 0 at a self-intersection point like in the example you gave. That's the sort of case I had in mind. I was confused by your earlier comment because it sounded like you were just describing a valley of dimension r, but as you say there could be isolated points like that also. I still maintain that this behavior --- of volume clustering near singularities when considering a narrow band about the loss minimum --- is the main distinguishing feature of SLT and so could use a mention in the OP.

Say that you have a loss function . The minimum loss set is probably not exactly , but it has something to do with that, so let's pretend that it's exactly that for now.

This is a collection of equations that are generically independent and so should define a subset of dimension zero, i.e. a collection of points in . However, there might be points at which the partial derivatives vanishing don't define independent equations, so we get something of positive codimension.

In these cases, what happens is that the gradient itself has vanishing de... (read more)

2interstice
Hmm, what you're describing is still in what I was referring to as "the broad basin regime". Sorry if I was unclear -- I was thinking of any case where there is no self-intersection of the minimum loss manifold as being a "broad basin". I think the main innovation of SLT occurs elsewhere. Look at the image in the tweet I linked. At the point where the curves intersect, it's not just that the Hessian fails to be of full-rank, it's not even well-defined. The image illustrates how volume clusters around a single point where the singularity is, not merely around the minimal-loss manifold with the greatest dimensionality. That is what is novel about singular learning theory.
Load More