All of DragonGod's Comments + Replies
When I saw the beginning/title I thought the post would be a refutation of the material scarcity thesis; I found myself disappointed it is not.
There is not an insignificant sense of guilt/of betraying myself from 2023 and my ambitions from before.
And I don't want to just end up doing irrelevant TCS research that only a few researchers in a niche field will ever care about.
It's not high impact research.
And it's mostly just settling. I get the sense that I enjoy theoretical research, I don't currently feel poised to contribute to the AI safety problem, I seem to have an unusually good (at least it appears so to my limited understanding) opportunity to pursue a boring TCS PhD in some niche field tha...
I still want to work on technical AI safety eventually.
I feel like I'm on quite far off path from directly being useful in 2025 than I felt in 2023.
And taking a detour to do a TCS PhD that isn't directly pertinent to AI safety (current plan) feels like not contributing.
Cope is that becoming a strong TCS researcher will make me better poised to contribute to the problem, but short timelines could make this path less viable.
[Though there's nothing saying I can't try to work on AI on the side even if it isn't the focus of my PhD.]
I think LW is a valuable intellectual hub and community.
Haven't been an active participant of recent, but it's still a service I occasionally find myself relying on explicitly, and I prefer the world where it continues to exist.
[I donated $20. Am unemployed and this is a nontrivial fraction of my disposable income.]
1
1o1's reasoning trace also does this for different languages (IIRC I've seen Chinese and Japanese and other languages I don't recognise/recall), usually an entire paragraph not a word, but when I translated them it seemed to make sense in context.
This is not a rhetorical question:) What do you mean by "probability" here?
Yeah, since posting this question:
I have updated towards thinking that it's in a sense not obvious/not clear what exactly "probability" is supposed to be interpreted as here.
And once you pin down an unambiguous interpretation of probability the problem dissolves.
I had a firm notion in mind for what I thought probability meant. But Rafael Harth's answer really made me unconfident that the notion I had in mind was the right notion of probability for the question.
I have not read all of them!
My current position now is basically:
...Actually, I'm less confident and now unsure.
Harth's framing was presented as an argument re: the canonical Sleeping Beauty problem.
And the question I need to answer is: "should I accept Harth's frame?"
I am at least convinced that it is genuinely a question about how we define probability.
There is still a disconnect though.
While I agree with the frequentist answer, it's not clear to me how to backgpropagate this in a Bayesian framework.
Suppose I treat myself as identical to all other agents in the reference class.
I kno
I'm curious how your conception of probability accounts for logical uncertainty?
So in this case, I agree that like if this experiment is repeated multiple times and every Sleeping Beauty version created answered tails, the reference class of Sleeping Beauty agents would have many more correct answers than if the experiment is repeated many times and every sleeping Beauty created answered heads.
I think there's something tangible here and I should reflect on it.
I separately think though that if the actual outcome of each coin flip was recorded, there would be a roughly equal distribution between heads and tails.
And when I was thinking t...
I separately think though that if the actual outcome of each coin flip was recorded, there would be a roughly equal distribution between heads and tails.
Importantly, this is counting each coinflip as the "experiment", whereas the above counts each awakening as the "experiment". It's okay that different experiments would see different outcome frequencies.
I mean I am not convinced by the claim that Bob is wrong.
Bob's prior probability is 50%. Bob sees no new evidence to update this prior so the probability remains at 50%.
I don't favour an objective notion of probabilities. From my OP:
...2. Bayesian Reasoning
- Probability is a property of the map (agent's beliefs), not the territory (environment).
- For an observation O to be evidence for a hypothesis H, P(O|H) must be > P(O|¬H).
- The wake-up event is equally likely under both Heads and Tails scenarios, thus provides no new information to update priors.
- The o
I mean I think the "gamble her money" interpretation is just a different question. It doesn't feel to me like a different notion of what probability means, but just betting on a fair coin but with asymmetric payoffs.
The second question feels closer to actually an accurate interpretation of what probability means.
i.e. if each forecaster has an first-order belief , and is your second-order belief about which forecaster is correct, then should be your first-order belief about the election.
I think there might be a typo here. Did you instead mean to write: "" for the second order beliefs about the forecasters?
The claim is that given the presence of differential adversarial examples, the optimisation process would adjust the parameters of the model such that it's optimisation target is the base goal.
That was it, thanks!
Probably sometime last year, I posted on Twitter something like: "agent values are defined on agent world models" (or similar) with a link to a LessWrong post (I think the author was John Wentworth).
I'm now looking for that LessWrong post.
My Twitter account is private and search is broken for private accounts, so I haven't been able to track down the tweet. If anyone has guesses for what the post I may have been referring to was, do please send it my way.
Most of the catastrophic risk from AI still lies in superhuman agentic systems.
Current frontier systems are not that (and IMO not poised to become that in the very immediate future).
I think AI risk advocates should be clear that they're not saying GPT-5/Claude Next is an existential threat to humanity.
[Unless they actually believe that. But if they don't, I'm a bit concerned that their message is being rounded up to that, and when such systems don't reveal themselves to be catastrophically dangerous, it might erode their credibility.]
Immigration is such a tight constraint for me.
My next career steps after I'm done with my TCS Masters are primarily bottlenecked by "what allows me to remain in the UK" and then "keeps me on track to contribute to technical AI safety research".
What I would like to do for the next 1 - 2 years ("independent research"/ "further upskilling to get into a top ML PhD program") is not all that viable a path given my visa constraints.
Above all, I want to avoid wasting N more years by taking a detour through software engineering again so I can get Visa sponsorship.
[...
Specifically, the experiments by Morrison and Berridge demonstrated that by intervening on the hypothalamic valuation circuits, it is possible to adjust policies zero-shot such that the animal has never experienced a previously repulsive stimulus as pleasurable.
I find this a bit confusing as worded, is something missing?
...How frequent are moderation actions? Is this discussion about saving moderator effort (by banning someone before you have to remove the rate-limited quantity of their bad posts), or something else? I really worry about "quality improvement by prior restraint" - both because low-value posts aren't that harmful, they get downvoted and ignored pretty easily, and because it can take YEARS of trial-and-error for someone to become a good participant in LW-style discussions, and I don't want to make it impossible for the true newbies (young people discovering
I think the model of "a composition of subagents with total orders on their preferences" is a descriptive model of inexploitable incomplete preferences, and not a mechanistic model. At least, that was how I interpreted "Why Subagents?".
I read @johnswentworth as making the claim that such preferences could be modelled as a vetocracy of VNM rational agents, not as claiming that humans (or other objects of study) are mechanistically composed of discrete parts that are themselves VNM rational.
I'd be more interested/excited by a refutation on the grounds ...
Oh, do please share.
Suppose it is offered (by a third party) to switch and then
Seems incomplete (pun acknowledged). I feel like there's something missing after "to switch" (e.g. "to switch from A to B" or similar).
Another example is an agent through time where as in the Steward of Myselves
This links to Scott Garrabrant's page, not to any particular post. Perhaps you want to review that?
I think you meant to link to: Tyranny of the Epistemic Majority.
Ditto for me.
I've been waiting for this!
We aren’t offering these criteria as necessary for “knowledge”—we could imagine a breaker proposing a counterexample where all of these properties are satisfied but where intuitively M didn’t really know that A′ was a better answer. In that case the builder will try to make a convincing argument to that effect.
Bolded should be sufficient.
In fact, I'm pretty sure that's how humans work most of the time. We use the general-intelligence machinery to "steer" ourselves at a high level, and most of the time, we operate on autopilot.
Yeah, I agree with this. But I don't think the human system aggregates into any kind of coherent total optimiser. Humans don't have an objective function (not even approximately?).
A human is not well modelled as a wrapper mind; do you disagree?
Thus, any greedy optimization algorithm would convergently shape its agent to not only pursue , but to maximize for 's pursuit — at the expense of everything else.
Conditional on:
- Such a system being reachable/accessible to our local/greedy optimisation process
- Such a system being actually performant according to the selection metric of our optimisation process
I'm pretty sceptical of #2. I'm sceptical that systems that perform inference via direct optimisation over their outputs are competitive in rich/complex environments.
Such o...
Do please read the post. Being able to predict human text requires vastly superhuman capabilities, because predicting human text requires predicting the processes that generated said text. And large tracts of text are just reporting on empirical features of the world.
Alternatively, just read the post I linked.
Oh gosh, how did I hallucinate that?
In what sense are they "not trying their hardest"?
It is not clear how they could ever develop strongly superhuman intelligence by being superhuman at predicting human text.
which is indifferent to the simplicify of the architecture the insight lets you find.
The bolded should be "simplicity".
Sorry, please where can I get access to the curriculum (including the reading material and exercises) if I want to study it independently?
The chapter pages on the website doesn't seem to list full curricula.
If you define your utility function over histories, then every behaviour is maximising an expected utility function no?
Even behaviour that is money pumped?
I mean you can't money pump any preference over histories anyway without time travel.
The Dutchbook arguments apply when your utility function is defined over your current state with respect to some resource?
I feel like once you define utility function over histories, you lose the force of the coherence arguments?
What would it look like to not behave as if maximising an expected utility function for a utility function defined over histories.
My contention is that I don't think the preconditions hold.
Agents don't fail to be VNM coherent by having incoherent preferences given the axioms of VNM. They fail to be VNM coherent by violating the axioms themselves.
Completeness is wrong for humans, and with incomplete preferences you can be non exploitable even without admitting a single fixed utility function over world states.
Not at all convinced that "strong agents pursuing a coherent goal is a viable form for generally capable systems that operate in the real world, and the assumption that it is hasn't been sufficiently motivated.
If you're not vNM-coherent you will get Dutch-booked if there are Dutch-bookers around.
This especially applies to multipolar scenarios with AI systems in competition.
I have an intuition that this also applies in degrees: if you are more vNM-coherent than I am (which I think I can define), then I'd guess that you can Dutch-book me pretty easily.
...The solution is IMO just to consider the number of computations performed per generated token as some function of the model size, and once we've identified a suitable asymptotic order on the function, we can say intelligent things like "the smallest network capable of solving a problem in complexity class C of size N is X".
Or if our asymptotic bounds are not tight enough:
"No economically feasible LLM can solve problems in complexity class C of size >= N".
(Where economically feasible may be something defined by aggregate global eco
The solution is IMO just to consider the number of computations performed per generated token as some function of the model size, and once we've identified a suitable asymptotic order on the function, we can say intelligent things like "the smallest network capable of solving a problem in complexity class C of size N is X".
Or if our asymptotic bounds are not tight enough:
"No economically feasible LLM can solve problems in complexity class C of size >= N".
(Where economically feasible may be something defined by aggregate global economic resources or similar, depending on how tight you want the bound to be.)
Regardless, we can still obtain meaningful impossibility results.
Very big caveat: the LLM doesn't actually perform O(1) computations per generated token.
The number of computational steps performed per generated token scales with network size: https://www.lesswrong.com/posts/XNBZPbxyYhmoqD87F/llms-and-computation-complexity?commentId=QWEwFcMLFQ678y5Jp
Strongly upvoted.
Short but powerful.
Tl;Dr: LLMs perform O(1) computational steps per generated token and this is true regardless of the generated token.
The LLM sees each token in its context window when generating the next token so can compute problems in O(n^2) [where n is the context window size].
LLMs can get along the computational requirements by "showing their working" and simulating a mechanical computer (one without backtracking, so not Turing complete) in their context window.
This only works if the context window is large enough to contain the work...
A reason I mood affiliate with shard theory so much is that like...
I'll have some contention with the orthodox ontology for technical AI safety and be struggling to adequately communicate it, and then I'll later listen to a post/podcast/talk by Quintin Pope/Alex Turner, or someone else trying to distill shard theory and then see the exact same contention I was trying to present expressed more eloquently/with more justification.
One example is that like I had independently concluded that "finding an objective function that was existentially safe when optimis...
I'm currently working through Naive Set Theory (alongside another text). I'll take this as a recommendation to work through the other textbooks later.
My maths level is insufficient for the course I'd guess.
I would appreciate it if videos of the meetings could be recorded. Or maybe I should just stick around and hope this will be run again next year.