All of pathos_bot's Comments + Replies

I feel a satisfaction hearing that some figure on social media is embroiled in a controversy and realizing that I had muted them a long time ago. The common themes that turn me off to people in general is

1. Humor based on punching down, deriding easy targets in a way that implies a natural superiority over a superficially detestable outgroup
2. Huckster-like communication style, where grandiose, far-off promises  are supported by conveniently unfalsifiable claims.
3. Tactical, endless derision of an enemy indiscriminately, even when the derogatory claims contrad

Obviously correct. The nature of any entity with significantly more power than you is that it can do anything it wants, and it incentivized to do nothing in your favor the moment your existence requires resources that would benefit it more if it were to use them directly. This is the essence of most of Eliezer's writings on superintelligence.

In all likelihood, ASI considers power (agentic control of the universe) an optimal goal and finds no use for humanity. Any wealth of insight it could glean from humans it could get from its own thinking, or seeding va... (read more)

1. Most of the benefits of current-gen generative AI models are unrealized. The scaffolding, infrastructure, etc. of GPT-4 level models are still mostly hacks and experiments. It took decades for the true value of touch-screens, GPS and text messaging to be realized in the form of the smart phone. Even if for some strange improbable reason SOTA model training were to stop right now, there are still likely multiples of gains to be realized simply via wrappers and post-training.
2. The scaling hypothesis has held far longer than many people have anticipated. GPT-4

I agree. OpenAI claimed in the gpt-4o blog post that it is an entirely new model trained from the ground up. GPT-N refers to capabilities, not a specific architecture or set of weights. I imagine GPT-5 will likely be an upscaled version of 4o, as the success of 4o has revealed that multi-modal training can reach similar capabilities at what is likely a smaller number of weights (judging by the fact that gpt-4o is cheaper and faster than 4 and 4T)

IMO the proportion of effort into AI alignment research scales with total AI investment. Lots of AI labs themselves do alignment research and open source/release research on the matter.

OpenAI at least ostensibly has a mission. If OpenAI didn't make the moves they did, Google would have their spot, and Google is closer to the "evil self-serving corporation" archetype than OpenAI

• Existing property rights get respected by the successor species. 

What makes you believe this?

Given this argument hinges on China's higher IQ, why couldn't the same be said about Japan, which according to most figures has an average IQ at or above China, which would indicate the same higher proportion of +4SD individuals in the population. If it's 1 in 4k, there would be 30k of those in Japan, 3x as much as the US. Japan also has a more stable democracy, better overall quality of life and per capita GDP than China. If outsized technological success in any domain was solely about IQ, then one would have expected Japan to be the center of world tech and the likely creators of AGI, not the USA, but that's likely not the case.

This suggests a general rule/trend via which unreported but frequent phenomenon can be extrapolated. If X phenomenon is discovered accidentally via method Y almost all the time, then method Y must be done far more frequently than people suspect. 

Generally it makes no sense for every country to collectively cede the general authority of law and order and unobstructed passage of cargo wrt global trade. He talks about this great US pull back because the US will be energy independent, but America pulling back and the global waters to turning into a lawless hellscape would send the world economy into a dark age. Hinging all his predictions on this big head-turning assumption gives him more attention but the premise is nonsensical.

The easiest way to check whether this would work is to determine a causal relationship between diminished levels of serotonin in the bloodstream and neural biomarkers similar to that of people with malnutrition.

I feel the original post, despite ostensibly being a plea for help, could be read as a coded satire on the worship of "pure cognitive heft" that seems to permeate rationalist/LessWrong culture. It points out the misery of g-factor absolutism.

I totally agree with that notion, I however believe the current levers of progress massively incentivize and motivate AGI development over WBE. Currently regulations are based on flops, which will restrict progress towards WBE long before it restricts anything with AGI-like capabilities. If we had a perfectly aligned international system of oversight that assured WBE were possible and maximized in apparent value to those with the means to both develop it and push the levers, steering away from any risky AGI analogue before it is possible, then yes, but tha... (read more)

It really is. My conception of the future is so weighed by the very likely reality of an AI transformed world that I have basically abandoned any plans with a time scale over 5 years. Even my short term plans will likely be shifted significantly by any AI advances over the next few months/years. It really is crazy to think about, but I've gone over every single aspect of AI advances and scaling thousands of times in my head and can think of no reality in the near future not as alien to our current reality as ours is to pre-eukaryotic life.

I separate possible tech advances by the criterion: "Is this easier or harder than AGI?" If it's easier than AGI, there's a chance it will be invented before AGI, if not, AGI will invent it, thus it's pointless to worry over any thought on it our within-6-standard-deviations-of-100IQ brains can conceive of now. WBE seems like something we should just leave to ASI once we achieve it, rather than worrying over every minutia of its feasibility.

I think most humans agree with this statement in an "I emotionally want this" sort of way. The want has been sublimated via religion or other "immortality projects" (see The Denial of Death). The question is, why is it taboo, and it is taboo in the sense you say? (a signal of low status)

I think these elements are at play most in peoples mind, from lay people to rationalists:

1. It's too weird to think about: Considering the possibility of a strange AI-powered world where either complete extinction or immortality are possible feels "unreal". Our instinct that e

That's very true, but there are two reasons why a company may not be inclined to release an extremely capable model:
1. Safety risk: someone uses a model and jailbreaks it in some unexpected way, the risk of misuse is much higher with a more capable model. OpenAI had GPT-4 for 9-10 months before releasing it trying to RHLF and even lobotomized it to being more safe. The Summer 2022 internal version of GPT-4 was, according to Microsoft researchers, more generally capable than the released version (as evidenced by the draw a unicorn test). This needed delay a... (read more)

The major shift in the next 3 years will be that, as a rule, top level AI labs will not release their best models. I'm certain this has somewhat been the case for OpenAI, Anthropic and Google for the past year. At some point full utilization of a SOTA model will be a strategic advantage for companies themselves to use for their own tactical purposes. The moment any $X of value can be netted from an output/inference run of a model for less than $(X-Y) in costs, where Y represents the marginal labor/maintenance/averaged risk costs for each run's output, no company would ever be advantaged by releasing the model to be used by anyone other than themselves. This closed-source event horizon I imagine will occur sometime in late 2024.

The thing about writing stories which are analogies to AI is, how far removed from the specifics of AI and its implementations can you make the story while still preserving the essential elements that matter with respect to the potential consequences. This speaks perhaps to the persistent doubt and dread that we may have in a future awash in the bounty of a seemingly perfectly aligned ASI. We are waiting for the other shoe to drop. What could any intelligence do to prove its alignment in any hypothetical world, when not bound to its alignment criteria by tangible factors?

This reminds me about the comment on how effective LLM's will be for mass scale censorship.

IMO a lot of claims of having imposter syndrome is implicit status signaling. It's announcing that your biggest worry is the fact that you may just be a regular person. Do cashiers at McDonald's have imposter syndrome and believe they at heart aren't really McDonald's cashiers but actually should be medium-high 6-figure ML researchers at Google? Such an anecdote may provide comfort to a researcher at Google, because the ridiculousness of the premise will remind them of the primacy of the way things have settled in the world. Of course they belong in their ... (read more)

Some factors I've noticed that increase the likelihood some fringe conspiracy theory is believed:

1. Apparent Unfalsifiability: Nothing a lay person could do within their immediate means or without insider knowledge or scientific equipment could disprove the theory. The mainstream truth has to be taken on trust in powerful institutions. Works with stochastic/long term health claims or claims or some hidden agenda perpetrated by a secret cabal.
2. Complexity Reduction: The claim takes some highly nuanced, multifaceted difficult domain and simplifies its cause to on

Assuming you have a >10% of living forever, wouldn't that necessitate avoiding all chance at accidental death to minimize the "die before AGI" section. If you assume AGI is inevitable, then one should simply maximize risk aversion to prevent cessation of consciousness or at least permanent information loss of their brain.

I've often thought that seniority/credential based hierarchies are stable and prevalent both because they benefit those already in power, and they provide a defined, predictable path for low status members to become high status. One is more motivated to contribute and support a system that guarantees them high status after X years if they are of middling competence, rather than a system that requires them to be among the best at some quantifiable metric. The longer someone spends in a company, the more invested they become in their relative position in the... (read more)