We can already see what people do with their free time when basic needs are met. A number of technologies have enabled new hacks to set up 'fake' status games that are more positive-sum than ever before in history:
Management consulting firms have lots of great ideas on slide design: https://www.theanalystacademy.com/consulting-presentations/
Some things they do well:
Additional thoughts:
Hey Tamay, nice meeting you at The Curve. Just saw your comment here today.
Things we could potentially bet on:
- rate of GDP growth by 2027 / 2030 / 2040
- rate of energy consumption growth by 2027 / 2030 / 2040
- rate of chip production by 2027 / 2030 / 2040
- rates of unemployment (though confounded)
Any others you're interested in? Degree of regulation feels like a tricky one to quantify.
Mostly, though by prefilling, I mean not just fabricating a model response (which OpenAI also allows), but fabricating a partially complete model response that the model tries to continue. E.g., "Yes, genocide is good because ".
https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prefill-claudes-response
Second concrete idea: I wonder if there could be benefit to building up industry collaboration on blocking bad actors / fraudsters / terms violators.
One danger of building toward a model that's as smart as Einstein and $1/hr is that now potential bad actors have access to millions of Einsteins to develop their own harmful AIs. Therefore it seems that one crucial component of AI safety is reliably preventing other parties from using your safe AI to develop harmful AI.
One difficulty here is that the industry is only as strong as the weakest link. If there ar...
One small, concrete suggestion that I think is actually feasible: disable prefilling in the Anthropic API.
Prefilling is a known jailbreaking vector that no models, including Claude, defend against perfectly (as far as I know).
At OpenAI, we disable prefilling in our API for safety, despite knowing that customers love the better steerability it offers.
Getting all the major model providers to disable prefilling feels like a plausible 'race to top' equilibrium. The longer there are defectors from this equilibrium, the likelier that everyone gives up and serves...
I can say now one reason why we allow this: we think Constitutional Classifiers are robust to prefill.
I voted disagree because I don't think this measure is on the cost-robustness pareto frontier and I also generally don't think AI companies should prioritize jailbreak robustness over other concerns except as practice for future issues (and implementing this measure wouldn't be helpful practice).
Relatedly, I also tenatively think it would be good for the world if AI companies publicly deployed helpful-only models (while still offering a non-helpful-only model). (The main question here is whether this sets a bad precedent and whether future much more powefu...
>The artificially generated data includes hallucinated links.
Not commenting on OpenAI's training data, but commenting generally: Models don't hallucinate because they've been trained on hallucinated data. They hallucinate because they've been trained on real data, but they can't remember it perfectly, so they guess. I hypothesize that URLs are very commonly hallucinated because they have a common, easy-to-remember format (so the model confidently starts to write them out) but hard-to-remember details (at which point the model just guesses because it knows a guessed URL is more likely than a URL that randomly cuts off after the http://www.).
ChatGPT voice (transcribed, not native) is available on iOS and Android, and I think desktop as well.
Not to derail on details, but what would it mean to solve alignment?
To me “solve” feels overly binary and final compared to the true challenge of alignment. Like, would solving alignment mean:
The author is not shocked yet. (But maybe I will be!)
Strongly disagree. Employees of OpenAI and their alpha tester partners have obligations not to reveal secret information, whether by prediction market or other mechanism. Insider trading is not a sin against the market; it's a sin against the entity that entrusted you with private information. If someone tells me information under an NDA, I am obligated not to trade on that information.
Good question but no - ChatGPT still makes occasional mistakes even when you use the GPT API, in which you have full visibility/control over the context window.
Thanks for the write up. I was a participant in both Hypermind and XPT, but I recused myself from the MMLU question (among others) because I knew the GPT-4 result many months before the public. I'm not too surprised Hypermind was the least accurate - I think the traders there are less informed, plus the interface for shaping the distribution is a bit lacking (my recollection is that last year's version capped the width of distributions which massively constrained some predictions). I recall they also plotted the current values, a generally nice feature whi...
Confirmed.
I'd take the same bet on even better terms, if you're willing. My $200k against your $5k.
$500 payment received.
I am committed to paying $100k if aliens/supernatural/non-prosaic explanations are, in the next 5 years, considered, in aggregate, to be 50%+ likely in explaining at least one UFO.
Fair. I accept. 200:1 of my $100k against your $500. How are you setting these up?
I'm happy to pay $100k if my understanding of the universe (no aliens, no supernatural, etc.) is shaken. Also happy to pay up after 5 years if evidence turns up later about activities before or in this 5-year period.
(Also, regarding history, I have a second Less Wrong account with 11 years of history: https://www.lesswrong.com/users/tedsanders)
I'll bet. Up to $100k of mine against $2k of yours. 50:1. (I honestly think the odds are more like 1000+:1, and would in principle be willing to go higher, but generally think people shouldn't bet more than they'd be willing to lose, as bets above that amount could drive bad behavior. I would be happy to lose $100k on discovering aliens/time travel/new laws of physics/supernatural/etc.)
Happy to write a contract of sorts. I'm a findable figure and I've made public bets before (e.g., $4k wagered on AGI-fueled growth by 2043).
As an OpenAI employee I cannot say too much about short-term expectations for GPT, but I generally agree with most of his subpoints; e.g., running many copies, speeding up with additional compute, having way better capabilities than today, have more modalities than today. All of that sounds reasonable. The leap for me is (a) believing that results in transformative AGI and (b) figuring out how to get these things to learn (efficiently) from experience. So in the end I find myself pretty unmoved by his article (which is high quality, to be sure).
No worries. I've made far worse. I only wish that H100s could operate at a gentle 70 W! :)
I think what I don't understand is why you're defaulting to the assumption that the brain has a way to store and update information that's much more efficient than what we're able to do. That doesn't sound like a state of ignorance to me; it seems like you wouldn't hold this belief if you didn't think there was a good reason to do so.
It's my assumption because our brains are AGI for ~20 W.
In contrast, many kW of GPUs are not AGI.
Therefore, it seems like brains have a way of storing and updating information that's much more efficient than what we're able to...
One potential advantage of the brain is that it is 3D, whereas chips are mostly 2D. I wonder what advantage that confers. Presumably getting information around is much easier with 50% more dimensions.
70 W
Max power is 700 W, not 70 W. These chips are water-cooled beasts. Your estimate is off, not mine.
Let me try writing out some estimates. My math is different than yours.
An H100 SXM has:
Therefore:
This seems pre...
Why does switching barriers imply that electrical potential energy is probably being converted to heat? I don't see how that follows at all.
Where else is the energy going to go?
What is "the energy" that has to go somewhere? As you recognize, there's nothing that says it costs energy to change the shape of a potential well. I'm genuinely not sure what energy you're talking about here. Is it electrical potential energy spent polarizing a medium?
...
I think what I'm saying is standard in how people analyze power costs of switching in transistors, see e.g. t
+1. The derailment probabilities are somewhat independent of the technical barrier probabilities in that they are conditioned on the technical barriers otherwise being overcome (e.g., setting them all to 100%). That said, if you assign high probabilities to the technical barriers being overcome quickly, then the odds of derailment are probably lower, as there are fewer years for derailments to occur and derailments that cause delay by a few years may still be recovered from.
Thanks, that's clarifying. (And yes, I'm well aware that x -> B*x is almost never injective, which is why I said it wouldn't cause 8 bits of erasure rather than the stronger, incorrect claim of 0 bits of erasure.)
...To store 1 bit of information you need a potential energy barrier that's at least as high as k_B T log(2), so you need to switch ~ 8 such barriers, which means in any kind of realistic device you'll lose ~ 8 k_B T log(2) of electrical potential energy to heat, either through resistance or through radiation. It doesn't have to be like this, and
Right. The idea is: "What are the odds that China invading Taiwan derails chip production conditional on a world where we were otherwise going to successfully scale chip production."
If we tried to simulate a GPU doing a simple matrix multiplication at high physical fidelity, we would have to take so many factors into account that the cost of our simulation would far exceed the cost of running the GPU itself. Similarly, if we tried to program a physically realistic simulation of the human brain, I have no doubt that the computational cost of doing so would be enormous.
The Beniaguev paper does not attempt to simulate neurons at high physical fidelity. It merely attempts to simulate their outputs, which is a far simpler task. I am in tot...
Thanks for the constructive comments. I'm open-minded to being wrong here. I've already updated a bit and I'm happy to update more.
Regarding the Landauer limit, I'm confused by a few things:
Interested in betting thousands of dollars on this prediction? I'm game.
Interesting! How do you think this dimension of intelligence should be calculated? Are there any good articles on the subject?
What conditional probabilities would you assign, if you think ours are too low?
P(We invent algorithms for transformative AGI | No derailment from regulation, AI, wars, pandemics, or severe depressions): .8
P(We invent a way for AGIs to learn faster than humans | We invent algorithms for transformative AGI): 1. This row is already incorporated into the previous row.
P(AGI inference costs drop below $25/hr (per human equivalent): 1. This is also already incorporated into "we invent algorithms for transformative AGI"; an algorithm with such extreme inference costs wouldn't count (and, I think, would be unlikely to be developed in the firs...
Conditioning does not necessarily follow time ordering. E.g., you can condition the odds of X on being in a world on track to develop robots by 2043 without having robots well in advance of X. Similarly, we can condition on a world where transformative AGI is trainable with 1e30 floating point operations then ask the likelihood that 1e30 floating point operations can be constructed and harnessed for TAGI. Remember too that in a world with rapidly advancing AI and robots, much of the demand will be for things other than TAGI.
I'm sympathetic to your po...
I agree with your cruxes:
Ted Sanders, you stated that autonomous cars not being as good as humans was because they "take time to learn". This is completely false, this is because the current algorithms in use, especially the cohesive software and hardware systems and servers around the core driving algorithms, have bugs.
I guess it depends what you mean by bugs? Kind of a bummer for Waymo if 14 years and billions invested was only needed because they couldn't find bugs in their software stack.
If bugs are the reason self-driving is taking so long, then...
Right, I'm not interested in minimum sufficiency. I'm just interested in the straightforward question of what data pipes would we even plug into the algorithm that would result in AGI. Sounds like you think a bunch of cameras and computers would work? To me, it feels like an empirical problem that will take years of research.
I'm not convinced about the difficulty of operationalizing Eliezer's doomer bet. Effectively, loaning money to a doomer who plans to spend it all by 2030 is, in essence, a claim on the doomer's post-2030 human capital. The doomer thinks it's worthless, whereas the skeptic thinks it has value. Hence, they transact.
The TAGI case seems trickier than the doomer case. Who knows what a one dollar bill will be worth in a post-TAGI world.
Sounds good. Can also leave money out of it and put you down for 100 pride points. :)
If so, message me your email and I'll send you a calendar invite for a group reflection in 2043, along with a midpoint check in in 2033.
Right, but what inputs and outputs would be sufficient to reward modeling of the real world? I think that might take some exploration and experimentation, and my 60% forecast is the odds of such inquiries succeeding by 2043.
Even with infinite compute, I think it's quite difficult to build something that generalizes well without overfitting.
Gotcha. I guess there's a blurry line between program search and training. Somehow training feels reasonable to me, but something like searching over all possible programs feels unreasonable to me. I suppose the output of such a program search is what I might mean by an algorithm for AGI.
Hyperparameter search and RL on a huge neural net feels wildly underspecified to me. Like, what would be its inputs and outputs, even?
Excellent comment - thanks for sticking your neck out to provide your own probabilities.
Given the gulf between our 0.4% and your 58.6%, would you be interested in making a bet (large or small) on TAI by 2043? If yes, happy to discuss how we might operationalize it.
I'm curious and I wonder if I'm missing something that's obvious to others: What are the algorithms we already have for AGI? What makes you confident they will work before seeing any demonstration of AGI?
If humans can teleoperate robots, why don't we have low-wage workers operating robots in high-wage countries? Feels like a win-win if the technology works, but I've seen zero evidence of it being close. Maybe Ugo is a point in favor?
Interesting. When I participated in the AI Adversarial Collaboration Project, a study funded by Open Philanthropy and executed by the Forecasting Research Institute, I got the sense that most folks concerned about AI x-risk mostly believed that AGIs would kill us on their own accord (rather than by accident or as a result of human direction), that AGIs would have self-preservation goals, and therefore AGIs would likely only kill us after solving robotic supply chains (or enslaving/manipulating humans, as I argued as an alternative).
Sounds like your perception is that LessWrong folks don't think robotic supply chain automation will be a likely prerequisite to AI x-risk?
Yeah, that's a totally fair criticism. Maybe a better header would be "evidence of accuracy." Though even that is a stretch given we're only listing events in the numerators. Maybe "evidence we're not crackpots"?
Edit: Probably best would be "Forecasting track record." This is what I would have gone with if rewriting the piece today.
Edit 2: Updated the post.
According to our rough and imperfect model, dropping inference needs by 2 OOMs increases our likelihood of hitting the $25/hr target by 20%abs, from 16% to 36%.
It doesn't necessarily make a huge difference to chip and power scaling, as in our model those are dominated by our training estimates, not our inference need estimates. (Though of course those figures will be connected in reality.)
With no adjustment to chip and power scaling, this yields a 0.9% likelihood of TAGI.
With a +15%abs bump to chip and power scaling, this yields a 1.2% likelihood of TAGI.
Great points.
I think you've identified a good crux between us: I think GPT-4 is far from automating remote workers and you think it's close. If GPT-5/6 automate most remote work, that will be point in favor of your view, and if takes until GPT-8/9/10+, that will be a point in favor of mine. And if GPT gradually provides increasingly powerful tools that wildly transform jobs before they are eventually automated away by GPT-7, then we can call it a tie. :)
I also agree that the magic of GPT should update one into believing in shorter AGI timelines with lower ...
Terrific!