All of rank-biserial's Comments + Replies

"That sounds really interesting. If a company working on the problem was paying a lot, I would consider jumping ship."

The Chinese stated preferences here closely track Western revealed preferences. Americans are more likely to dismiss AI risk post-hoc in order to justify making more money, whereas it seems that Chinese people are less likely to sacrifice their epistemic integrity in order to feel like a Good Guy, Hire people, and pay them money!

So that's how you draft scissor statements >:)

Yes. Too many cached decisions.

How did you choose the salary range?

Exactly!

My understanding here is that while this is true, it will discourage the 5%, who will just go work for FAANG and donate money to someone worse (or someone overwhelmed with work), simultaneously losing any chance at a meaningful job. The point being that yes, it's good to donate, but if everyone donates (since that is the default rat race route), noone will do the important work.

No! If everyone donates, there will be enough money to pay direct workers high salaries. I know this goes contra to the image of the selfless, noble Effective Altruist, but if you want shit to get done you should pay people lots of money to do it.

A human life, with all its joys and all its pains, adding up over the course of decades, is worth far more than your brain’s feelings of comfort or discomfort with a plan.

In case you're wondering why Rationalists Don't Win, it's because they're too busy lying to themselves about what their true preferences are.

Ok, sick. I largely agree with you btw (about the hamster wheel being corrosive). If I came off as agressive, fyi, I liked the spirit of your post a lot, and I strong-upvoted it.

Yes, selfish agents want to not get turned into paperclips. But they have other goals too. You can prefer alignment be solved, while not wanting to dedicate your mind, body, and soul to waging a jihad against it. Where can Charlie effectively donate, say, 10% of his salary to best mitigate x-risk? Not MIRI (according to MIRI).

You don't need that many resources.

True, if we're talking solely about alignment. If we're talking about the larger space of, as you put them, "maximize objective function"-type goals, then there's plenty of demand for resources. Let's say I wanna do (actually effective) longevity research. Since the competition for grant money is (like most things) Goodharted and broken, and because I don't have enough biology credentials, I'm gonna need to self-fund in order to buy lab materials and grad student slaves.

If you have EA-like goals, you have a "maximize objective function"-type goal. It's in the same shape as "become as rich as possible" or "make the world as horrible as possible." Basically, the conventional path is highly highly unlikely to get you all the way there. In this case, you probably want to get into the

1. Get skills+resources
2. Use skills+resources to do impact
3. Repeat

Loop.

I'm in a similar situation to yours. (I'm currently in the "Bob" stage of the Alice -> Bob -> Charlie pipeline.) How do you propose I, and those like us, go about d... (read more)

One of the problems here is that, as of right now, there isn't much of a middle path between "Stay at Google and do nothing" and "quit your job to do alignment work full-time". Then there's the issue of status-seeking vs. altruism as a case of revealed vs. stated preferences. If there was a way to make $750k a year and save the world, people would be all over that. I, personally, would be all over that.

But there isn't. If we use johnswentworth as an optimistic case, those who would go into independent AI alignment work full-time would make about $90k per y... (read more)

I endorse Rohin Shah's response to that post.

You might think "well, obviously the superintelligent AI system is going to care about things, maybe it's technically an assumption but surely that's a fine assumption". I think on balance I agree, but it doesn't seem nearly so obvious to me, and seems to depend on how exactly the agent is built. For example, it's plausible to me that superintelligent expert systems would not be accurately described as "caring about things", and I don't think it was a priori obvious that expert systems wouldn't lead to AGI. Si

Even humans have a huge range of intellectual capacity, and someone who is good at math may not be good at say, writing a novel. So the idea of "general intelligence" is pretty weak from the outset, and it's certainly not a binary value that you either have or have not.

https://en.wikipedia.org/wiki/G_factor_(psychometrics)

If you can retroactively fit a utility function to any sequence of actions, what predictive power do we gain by including utility functions into our models of AGI?

https://en.wikipedia.org/wiki/Brain_stimulation_reward

https://doi.org/10.1126/science.140.3565.394

https://sci-hub.hkvisa.net/10.1126/science.140.3565.394

In our earliest work with a single lever it was noted that while the subject would lever-press at a steady rate for stimulation to various brain sites, the current could be turned off entirely and he would continue lever-pressing at the same rate (for as many as 2000 responses) until told to stop.

It is of interest that the introduction of an attractive tray of food produced no break in responding, alt

It's to push our understanding of some crazy niche field and push it further beyond. The widespread recognition is nice, I'm sure, but I don't think most PhD graduates' goals are this.

Insufficiently Hansonpilled.

It's time for me to shill for Big Pharma again.

I spent years trying to come up with a mental strategy that would reliably generate willpower, all to no avail. To my knowledge, there is one (1) way to cut through this Gordian knot, and that way is amphetamines.

• It is easy to get a psychiatrist to prescribe you amphetamines.^[1]
• If you request generic drugs over brand-name, they're pretty inexpensive.
• Amphetamines greatly reduce one's appetite; Adderall was originally sold as a diet pill. For the first time in my life, I could empathize with those people who

Hamming distance assumes that the strings you're comparing are the same length, but "Yavin" is shorter than "Yarvin". Levenshtein distance is the smallest number of insertions, deletions, or character substitutions required to get from string A to string B, while Hamming distance only counts char-substitutions.

Epistemic status: Shaky

That's not how it works. People bred chihuahuas from wolves, and the blue whale evolved from a goat-sized land mammal. If the additive genes that control spinal length just upregulate some pre-existing developmental process, it's not too much of a leap to assume that the additive genes that control intelligence just turn up a similar developmental knob. That would imply that intelligence-increasing mutations are common (being less complex), and that intelligence could be selected for like any other trait.

Within humans, intelligence-related genes behave additively. That is, the predicted IQ of a child of a 95 IQ mother and a 115 IQ father follows a normal distribution centered at 105. Since "general intelligence" exists to some degree in octopuses, we're amplifying an existing trait, rather than creating one de novo. If intelligence-related genes behave additively in octopuses as well (and we should assume that they do), then breeding octopi for intelligence should be as straightforward as breeding cows for milk.

Well, "Yarvin" is closer in Levenshtein distance, plus you're acting coy, so I'm updating towards "Yarvin" 😎

Some Yavin guy posted some good stuff. Then I found out he had his own blog. That was pretty good.

Is "Yavin" supposed to be "Yarvin" or "Yvain"? Both are quite plausible.

This is a Moldbug argument, but:

Markets aren't totally efficient, but they are the most adequate part of our civilization. For-profit corporations usually follow the same general structure, so the best form of governance looks like a joint-stock company.

• The "electrode in the reward center" setup has been proven to work in humans, whereas jhanas may not tranfer over Neuralink.
• Deep brain stimulation is FDA-approved in humans, meaning less (though nonzero) regulatory fuckery will be required.
• Happiness is not pleasure; wanting is not liking. We are after reinforcement.

Judging by the fox domestication experiment, intelligence could emerge way sooner than 50 generations. The fox experiment used a population size on the order of 1e2, and selected the top 20% most docile foxes to act as the breeding pool for the next generation. You could accelerate this process by increasing the population size, while selecting a smaller fraction of intelligent octopuses.

literally 1984

14. Eco-terrorism, animal rights terrorism, educational terrorism (!), and many other forms of terrorism are morally justified and should be practiced by advocates of those causes.

Ah yes, the old "let's bomb DeepMind" approach to AI safety.

15. The virtually universal practice of assigning a permanent name at birth (which only exists so that governments can more easily tax us) has caused a species-wide shift towards a more narcissistic and egotistical mode of being.

Neither of us are using our government-assigned names right now. Has the internet-culture norm of creating pseudonymous usernames countered this shift?

19. Money is, in fact, the root of all evil and we will never be free until all monetary systems are abolished.

Money allows people to express their preferences more than they otherwise could. People's revealed preferences are invariably more evil than their stated preferences. Money reveals evil, but it does not create it.

8. We will never stop fighting about race. The only way out of this racial hellscape is to simply breed race away by tweaking dating app algorithms to increase the number of interracial matings.

First, this will never work, because people have strong racial preferences. Second, people will still fight about racial features even if everyone is mixed. Just look at Brazil.

Exquisitely based

Great! If I recall correctly, you wanted genetically optimized kids to be gestated and trained.

I suspect that akrasia is a much bigger problem than most people think, and to be truly effective, one must outsource part of their reward function. There could be massive gains.

What do you think about the setup I outlined, where a pair of reseachers exist such that one controls an electrode embedded in the other's reward center? Think Focus from Vinge's A Deepness In The Sky.

Fine. What do you think about the human-augmentation cluster of strategies? I recall you thought along very similar lines circa ~2001.

I'm not well calibrated on sub 1% probabilities. Yeah, the odds are low.

There are other classes of Hail Mary. Picture a pair of reseachers, one of whom controls an electrode wired to the pleasure centers of the other. Imagine they have free access to methamphetamine and LSD. I don't think research output is anywhere near where it could be.

These are all strategies to buy time, so that alignment efforts may have more exposure to miracle-risk.

I can imagine a plausible scenario in which WW3 is a great thing, because both sides brick each other's datacenters and bomb each other's semiconductor fabs. Also, all the tech talent will be spent trying to hack the other side and will not be spent training bigger and bigger language models.

• Coordination (cartelization) so that AI capabilities are not a race to the bottom

• Coordination to indefinitely halt semiconductor supply chains

• Coordination to shun and sanction those who research AI capabilities (compare: coordination against embyronic human gene editing)

• Coordination to deliberately turn Moore's Law back a few years (yes, I'm serious)

Yeah, most of them do. I have some hope for the strategy-cluster that uses widespread propaganda^[1] as a coordination mechanism.

Given the whole "brilliant elites" thing, and the fecundity of rationalist memes among such people, I think it's possible to shift the world to a better Nash equilibrium.

What about Hail Mary strategies that were previously discarded due to being too risky? I can think of a couple off the top of my head. A cornered rat should always fight.

Possible modifications:

• Allow withdrawing funds at any point. People like to feel in control, and this setup eliminates the need for the fundraiser to have a deadline.

• For longer-term projects, let funds accumulate interest at slightly below market rates. The difference is pocketed by the crowdfunding platform. Customers get to feel financially smart while also contributing to a cool idea.