All of NunoSempere's Comments + Replies
curl 'https://www.lesswrong.com/graphql?' \
-H 'accept: application/json' -H 'content-type: application/json' -H 'user-agent: bot' \
--data-raw '{"query":"{ post(input: {selector: {_id: \"7p9WB5NsrQiqKEbPA\"}}) { result { htmlBody } }}"}' | \
jq .data.post.result.htmlBody | \
llm "I am a very rich software engineer in the Bay area worried about AI and pandemics. What is the most actionable information in this post?"
Cheers. Not sure that is the right thing to be optimizing for. I guess that for stuff like this that coverse different topics, people could ask an LLM what the parts they're more likely to find useful are.
I agree this is good in the american public sphere, but such speculation is still very useful to better predict behaviors. I don't think we disagree that much here.
I recognize the limitations of armchair diagnosis, especially of public figures. But there's value in examining these patterns as case studies in how psychiatric conditions manifest in high-functioning individuals, particularly when those individuals have publicly acknowledged aspects of their mental health.
Completely agree, thanks for the speculation
Interesting. Some thoughts
- I dislike "infinite money" as an analogy to resolving a market; seems like it isn't needed since in actual practice prediction markets don't require infinite money
- A "trading strategy" (as per the Logical Induction paper) is a function from prices to a list of buy/sell actions. => it feels like this isn't as elegant as "a probability is a representation of degree of belief". It feels like trying to ground an epistemology in prediction markets is incomplete.
I've known Jaime for about ten years. Seems like he made an arguably wrong call when first dealing with real powaah, but overall I'm confident his heart is in the right place.
Shapley values are constructed such that introducing a null player doesn't change the result. You are doing something different by considering the wrong counterfactual (one where C exists but isn't part of the coalition, vs one when it doesn't exist)
Adding a person with veto power is not a neutral change.
1Maybe you could address these problems, but could you do so in a way that is "computationally cheap"? E.g., for forecasting on something like extinction, it is much easier to forecast on a vague outcome than to precisely define it.
Nice consideration, we hadn't considered non-natural asteroids here. I agree this is a consideration as humanity reaches for the stars, or the rest of the solar system.
If you've thought about it a bit more, do you have a sense of your probability over the next 100 years?
To nitpick on your nitpick, in the US, 1000x safer would be 42 deaths yearly. https://en.wikipedia.org/wiki/Motor_vehicle_fatality_rate_in_U.S._by_year
For the whole world, it would just be above 1k. https://en.wikipedia.org/wiki/List_of_countries_by_traffic-related_death_rate#List, but 2032 seems like an ambitious deadline for that.
In addition, it does seem against the spirit of the question to resolve positively solely because of reducing traffic deaths.
To me this looks like circular reasoning: this example supports my conceptual framework because I interpret the example according to the conceptual framework.
Instead, I notice that Stockfish in particular has some salient characteristics that go against the predictions of the conceptual framework:
- It is indeed superhuman
- It is not the case that once Stockfish ends the game that's it. I can rewind Stockfish. I can even make one version of Stockfish play against another. I can make Stockfish play a chess variant. Stockfish doesn't annihilate my physical bod
This is importantly wrong because the example is in the context of an analogy
getting some pawns : Stockfish : Stockfish's goal of winning the game :: getting a sliver of the Sun's energy : superintelligence : the superintelligence's goals
The analogy is presented as forceful and unambiguous, but it is not. It's instead an example of a system being grossly more capable than humans in some domain, and not opposing a somewhat orthogonal goal
It's forceful and unambiguous because Stockfish's victory over the other player terminates the other player's goals, whatever those may be: no matter what your goals during the game may be, you can't pursue it once the game is over (and you've lost). Available joules are zero-sum in the same way that playing a chess game is zero-sum.
The analogy only goes through if you really double down on the 'goal' of capturing some pawns as intrinsically valuable, so even the subsequent defeat is irrelevant. At which point, you're just unironically making the New Yorke...
Incidentally you have a typo on "pawn or too" (should be "pawn or two"), which is worrying in the context of how wrong this is.
There is no equally simple version of Stockfish that is still supreme at winning at chess, but will easygoingly let you take a pawn or too. You can imagine a version of Stockfish which does that -- a chessplayer which, if it's sure it can win anyways, will start letting you have a pawn or two -- but it's not simpler to build. By default, Stockfish tenaciously fighting for every pawn (unless you are falling into some worse sacrificial trap), is implicit in its generic general search through chess outcomes.
The bolded part (bolded by me) is just wrong ma...
you will not find it easy to take Stockfish's pawns
Seems importantly wrong, in that if your objective is to take a few pawns (say, three), you can easily do this. This seems important in the context that it's hard to to obtain resources from an adversary that cares about things differently.
In the case of stockfish you can also rewind moves.
Seems importantly wrong, in that if your objective is to take a few pawns (say, three), you can easily do this.
By... decreasing your chances of winning because you keep playing moves which increase your chance of taking pawns while trading off chances of winning, so Stockfish is happy to let you hang yourself all day long, driving your winning probability down to ε by the end of the game even faster than if you had played your hardest? I don't see how this is "importantly wrong". I too can sell dollar bills for $0.90 all day long, this doesn't somehow disprove markets or people being rational - quite the opposite.
I disagree with the 5% of switching to a Sundar Pichai hairs simile:
- Prediction market prices are bounded between 0 and 1
- Polymarket has > 1k markets, and maybe 3 to 10 ambiguous resolutions a year. It's more like 0.3% to 1%.
I'm willing to bet 2k USD on my part against a single dollar yours that that if I waterboard you, you'll want to stop before 3 minutes have passed
Interesting, where are you physically located? Also, are you thinking of the unpleasantness of the situation, or are you thinking of the physical asphyxiation component?
You might want to download the Online Encyclopedia of Integer Sequences, e.g., as in here, and play around with it, e.g., look at the least likely completion for a given sequence & so on.
You're right, changed
I ended up solving the equations either analytically (partially with the help of Phil Trammell), https://forum.effectivealtruism.org/posts/FXPaccMDPaEZNyyre/a-model-of-patient-spending-and-movement-building or through simulations https://github.com/NunoSempere/ReverseShooting
https://github.com/NunoSempere/LaborCapitalAndTheOptimalGrowthOfSocialMovements
I found this post super valuable but I found the presentation confusing. Here is a table, provided as is, that I made based on this post & a few other sources:
| Source | Amount for 2024 | Note |
|---|---|---|
| Open Philanthropy | $80M | Projected from past amount |
| Foundation Model Taskforce | $20M | 100M GBP but unclear over how many years? |
| FLI | $30M | $600M donation in crypto, say you can get $300M out of it, distributed over 10 years |
| AI labs | $30M | |
| Jan Tallin | $20M | See [here](https://jaan.info/philanthropy/) |
| NSF | $5M | |
| LTFF (not OpenPhil) | $2M | |
| Nonlinear fund and dono |
You might also enjoy this review: https://nunosempere.com/blog/2023/04/28/expert-review-epoch-direct-approach/
One particularity of polymarket is that you couldn't as of the time of this market divide $1 into four shares and sell all of them for $1.09. If you could have--well, then this problem wouldn't have existed--but if you could have then this would have been a 9%.
I don't have a link off the top of my head, but the trade would have been to sell one share of yes for each market. You can do this by splitting $1 into a Yes and No share, and selling the Yes. Specifically in Polymarket you achieve this by adding and then withdrawing liquidity (for a specific type of markets called "amm', for "automatic market marker", which were the only ones supported by Polymarket at the time, though it since then also supports an order book).
By doing this, you earn $1.09 from the sale + $3 from the three events eventually, and t...
The framework is AI strategy nearcasting: trying to answer key strategic questions about transformative AI, under the assumption that key events (e.g., the development of transformative AI) will happen in a world that is otherwise relatively similar to today’s.
Usage of "nearcasting" here feels pretty fake. "Nowcasting" is a thing because 538/meteorology/etc. has a track record of success in forecasting and decent feedback loops, and extrapolating those a bit seems neat.
But as used in this case, feedback loops are poor, and it just feels like a differ...
See this comment: <https://www.lesswrong.com/posts/yCuzmCsE86BTu9PfA/there-are-no-coherence-theorems?commentId=v2mgDWqirqibHTmKb>
I am not defending the language of the OP's title, I am defending the content of the post.
You don't have strategic voting with probabilistic results. And the degree of strategic voting can also be mitigated.
Copying my second response from the EA forum:
...Like, I feel like with the same type of argument that is made in the post I could write a post saying "there are no voting impossibility theorems" and then go ahead and argue that the Arrow's Impossibility Theorem assumptions are not universally proven, and then accuse everyone who ever talked about voting impossibility theorems that they are making "an error" since "those things are not real theorems". And I think everyone working on voting-adjacent impossibility theorems would be pretty justifiedly annoyed by
Sorry, this might have not been obvious, but I indeed think the voting impossibility theorems have holes in them because of the lotteries case and that's specifically why I chose that example.
I think that intellectual point matters, but I also think writing a post with the title "There are no voting impossibility theorems", defining "voting impossibility theorems" as "theorems that imply that all voting systems must make these known tradeoffs", and then citing everyone who ever talked about "voting impossibility theorems" as having made "an error" wo...
Copying my response from the EA forum:
(if this post is right)
The post does actually seem wrong though.
Glad that I added the caveat.
Also, the title of "there are no coherence arguments" is just straightforwardly wrong. The theorems cited are of course real theorems, they are relevant to agents acting with a certain kind of coherence, and I don't really understand the semantic argument that is happening where it's trying to say that the cited theorems aren't talking about "coherence", when like, they clearly are.
Well, part of the semantic nuance is tha...
Well, part of the semantic nuance is that we don't care as much about the coherence theorems that do exist if they will fail to apply to current and future machines
The correct response to learning that some theorems do not apply as much to reality as you thought, surely mustn't be to change language so as to deny those theorems' existence. Insofar as this is what's going on, these are pretty bad norms of language in my opinion.
I am also curious about the extent to which you are taking the Hoffman scaling laws as an assumption, rather than as something you can assign uncertainty over.
I thought this was great, cheers.
Here:
Next, we estimate a sufficient horizon length, which I'll call the k-horizon, over which we expect the most complex reasoning to emerge during the transformative task. For the case of scientific research, we might reasonably take the k-horizon to roughly be the length of an average scientific paper, which is likely between 3,000 and 10,000 words. However, we can also explicitly model our uncertainty about the right choice for this parameter.
It's unclear whether the final paper would be the needed horizon length.
F...
But I think it is >30% likely you can compensate for past over or under estimations.
I'd bet against that at 1:5, i.e., against the proposition that the optimal forecast is not subject to your previous history
This is true in the abstract, but the physical word seems to be such that difficult computations are done for free in the physical substrate (e.g,. when you throw a ball, this seems to happen instantaneously, rather than having to wait for a lengthy derivation of the path it traces). This suggests a correct bias in favor of low-complexity theories regardless of their computational cost, at least in physics.
Neat. I have some uncertainty about the evolutionary estimates you are relying on, per here. But neat.
Thanks Tamay!
Seems like this assumes an actual superintelligence, rather than near-term scarily capable successor of current ML systems.
Why publish this publicly? Seems like it would improve optimality of training runs?
Good question. Some thoughts on why do this:
- Our results suggest we won't be caught off-guard by highly capable models that were trained for years in secret, which seems strategically relevant for those concerned with risks
- We looked whether there was any 'alpha' in these results by investigating the training durations of ML training runs, and found that models are typically trained for durations that aren't far off from what our analysis suggests might be optimal (see a snapshot of the data here)
- It independently seems highly likely that large training runs
Software: archivenow
Need: Archiving websites to the internet archive.
Other programs I've tried: The archive.org website, spn, various scripts, various extensions.
archivenow is trusty enough for my use case, and it feels like it fails less often than other alternatives. It was also easy enough to wrap into a bash script and process markdown files. Spn is newer and has parallelism, but I'm not as familiar with it and subjectively it feels like it fails a bit more.
See also: Gwern's setup.
Have prediction markets which pay $100 per share, but only pay out 1% of the time, chosen randomly. If the 1% case that happens, then also implement the policy under consideration.
Have prediction markets which pay $100 per share, but only pay out 1% of the time, chosen randomly. If the 1% case that happens, then also implement the policy under consideration.
The issue is that probabilities for something that will either happen or not don't really make sense in a literal way
This is just wrong/frequentist. Search for the "Bayesian" view of probability.
I thought this post was great; thanks for writing it.
Will SMTM answer NCM's post criticizing their Lithium theory? <https://manifold.markets/NuñoSempere/will-smtm-answer-ncms-post-criticiz>
https://metaforecast.org/?query=ETH+merge -> https://polymarket.com/market-group/ethereum-merge-pos -> 59% by October, 87% by November.
The Litany of Might
I strive to take whatever steps may help me best to reach my goals,
I strive to be the very best at what I strive
There is no glory in bygone hopes,
There is no shame in aiming for the win,
there is no choice besides my very best,
to play my top moves and disregard the rest
Moonlight for PauseAI?