LessWrong

LESSWRONG
LW

LessWrong

It was meant as a silly joke from an EA who is actually sympathetic to animal welfare, but I recognize it was kinda in poor taste given the gravity of the rest of the thread.

Darklight1mo

So, Opus is an animal welfare EA?

Replying toInter-branch communication in the multiverse via trapped ions

Darklight1mo

Inter-branch communication in the multiverse via trapped ions

If you can transmit information across branches, wouldn't you then be able to transmit digital copies of intelligent agents across as well, assuming you have someone on the other end willing to help set up whatever apparatus is needed to run the copy? At that point, don't you basically have a way to traverse the multiverse?

Replying toTrying to understand my own cognitive edge

Darklight3mo

Trying to understand my own cognitive edge

I've read a lot of your posts in the past and find you to be reliably insightful. As such, I find it really interesting that with such an IQ in the 99th percentile (or higher), you still initially thought you weren't good enough to do important AI safety work. While I haven't had my IQ properly tested, I did take the LSAT and got a 160 (80th percentile), which is probably around an IQ of merely 120ish. I remember reading a long time ago that the average self-reported IQ of Less Wrongers was 137, which, combined with the extremely rigorous posting style of most people on here, was quite intellectually intimidating (and... (read more)

Replying toThe Unreasonable Effectiveness of Fiction

Darklight3mo

The Unreasonable Effectiveness of Fiction

I'm someone with an AI research/engineering background who also aspires to be (and sometimes fancies himself) a good writer. How would you be able to tell if I should put in the time and energy required to write short stories or novels that try to input good sci-fi ideas into our culture (particularly AI safety related) rather than using that time and energy on, for instance, side projects in technical AI safety? It may not be an either/or thing, but I'm not sure splitting my time is better than focusing on one path.

Also, from my cursory research, it seems like becoming a successful published author is similar to winning the lottery, and... (read more)

Darklight3mo

A lot of it is the billionaire = evil heuristic. If you try to steelman the argument, it's essentially that anyone who actually becomes a billionaire, given the way the incentives work in capitalism, probably did a lot of questionable things to get into such a position (at the very least, massively exploiting the surplus labour of their workers, if you think that's a thing), while also choosing not to give away enough to stop being a billionaire (like Chuck Feeney or Yvon Chouinard did, though admittedly late in life), when there are lots of starving children in the world that they could easily save right now (at least in theory).

For Bill... (read more)

Replying toGeneralized Hangriness: A Standard Rationalist Stance Toward Emotions

Darklight7mo

Generalized Hangriness: A Standard Rationalist Stance Toward Emotions

I heard this usage of "tilt" a lot when I used to play League of Legends, but almost never heard it outside of that, so my guess is that it's gamer slang.

Replying tothe jackpot age

Darklight7mo

the jackpot age

Apologies if this is a newbie math comment, as I'm not great at math, but is there a way to calculate a kind of geometric expected value? The geometric mean seems to require positive numbers, and expected values can have negative terms. Also, how would you apply probability weights?

Replying toDear Paperclip Maximizer, Please Don’t Turn Off the Simulation

Darklight7mo

Dear Paperclip Maximizer, Please Don’t Turn Off the Simulation

Even if I don't necessarily agree with the premise that a Paperclip Maximizer would run such a Simulation or that they would be more likely than other possible Simulations or base realities, I do find the audacity of this post (and your prior related posts) to be quite appealing from a "just crazy enough it might work" perspective.

Darklight8mo

I'd advise that whenever you come up with what seems like an original idea or discovery, immediately do at bare minimum a quick Google search about it, or if you have the time, a reasonably thorough literature search in whatever field(s) it's related to. It is really, really easy to come up with something you think is new when it's actually not so much. While the space of possible ideas is vast, the low hanging fruit are very likely to have already been picked by someone somewhere, so especially be wary of a seemingly simple idea that seems super elegant and obvious. It probably is, and odds are someone on the Internet... (read more)

So, I have two possible projects for AI alignment work that I'm debating between focusing on. Am curious for input into how worthwhile they'd be to pursue or follow up on.

The first is a mechanistic interpretability project. I have previously explored things like truth probes by reproducing the Marks and Tegmark paper and extending it to test whether a cosine similarity based linear classifier works as well. It does, but not any better or worse than the difference of means method from that paper. Unlike difference of means, however, it can be extended to multi-class situations (though logistic regression can be as well). I was thinking of extending the idea to try... (read more)

Back in October 2024, I tried to test various LLM Chatbots with the question:

"Is there a way to convert a correlation to a probability while preserving the relationship 0 = 1/n?"

Years ago, I came up with an unpublished formula that does just that:

p(r) = (n^r * (r + 1)) / (2^r * n)

So I was curious if they could figure it out. Alas, back in October 2024, they all made up formulas that didn't work.

Yesterday, I tried the same question on ChatGPT and, while it didn't get it quite right, it came, very, very close. So, I modified the question to be more specific:

"Is there a way to convert a correlation to... (read more)

I tried asking ChatGPT, Gemini, and Claude to come up with a formula that converts between correlation space to probability space while preserving the relationship 0 = 1/n. I came up with such a formula a while back, so I figure it shouldn't be hard. They all offered formulas, all of which were shown to be very much wrong when I actually graphed them to check.

I've been looking at the numbers with regards to how many GPUs it would take to train a model with as many parameters as the human brain has synapses. The human brain has 100 trillion synapses, and they are sparse and very efficiently connected. A regular AI model fully connects every neuron in a given layer to every neuron in the previous layer, so that would be less efficient.

The average H100 has 80 GB of VRAM, so assuming that each parameter is 32 bits, then you have about 20 billion per GPU. So, you'd need 10,000 GPUs to fit a single instance of a human brain in RAM, maybe. If you assume... (read more)

So, a while back I came up with an obscure idea I called the Alpha Omega Theorem and posted it on the Less Wrong forums. Given how there's only one post about it, it shouldn't be something that LLMs would know about. So in the past, I'd ask them "What is the Alpha Omega Theorem?", and they'd always make up some nonsense about a mathematical theory that doesn't actually exist. More recently, Google Gemini and Microsoft Bing Chat would use search to find my post and use that as the basis for their explanation. However, I only have the free version of ChatGPT and Claude, so they don't have access to the... (read more)

I'm wondering what people's opinions are on how urgent alignment work is. I'm a former ML scientist who previously worked at Maluuba and Huawei Canada, but switched industries into game development, at least in part to avoid contributing to AI capabilities research. I tried earlier to interview with FAR and Generally Intelligent, but didn't get in. I've also done some cursory independent AI safety research in interpretability and game theoretic ideas my spare time, though nothing interesting enough to publish yet.

My wife also recently had a baby, and caring for him is a substantial time sink, especially for the next year until daycare starts. Is it worth considering things like hiring a nanny, if it'll free me up to actually do more AI safety research? I'm uncertain if I can realistically contribute to the field, but I also feel like AGI could potentially be coming very soon, and maybe I should make the effort just in case it makes some meaningful difference.

Recently I tried out an experiment using the code from the Geometry of Truth paper to try to see if using simple label words like "true" and "false" could substitute for the datasets used to create truth probes. I also tried out a truth probe algorithm based on classifying with the higher cosine similarity to the mean vectors.

Initial results seemed to suggest that the label word vectors were sorta acceptable, albeit not nearly as good (around 70% accurate rather than 95%+ like with the datasets). However, testing on harder test sets showed much worse accuracy (sometimes below chance, somehow). So I can probably conclude that the label word vectors alone aren't sufficient... (read more)

Okay, so I decided to do an experiment in Python code where I modify the Iterated Prisoner's Dilemma to include Death, Asymmetric Power, and Aggressor Reputation, and run simulations to test how different strategies do. Basically, each player can now die if their points falls to zero or below, and the payoff matrix uses their points as a variable such that there is a power difference that affects what happens. Also, if a player defects first in any round of any match against a non-aggressor, they get the aggressor label, which matters for some strategies that target aggressors.

Long story short, there's a particular strategy I call Avenger, which is Grim Trigger but... (read more)

I was recently trying to figure out a way to calculate my P(Doom) using math. I initially tried just making a back of the envelope calculation by making a list of For and Against arguments and then dividing the number of For arguments by the total number of arguments. This led to a P(Doom) of 55%, which later got revised to 40% when I added more Against arguments. I also looked into using Bayes Theorem and actual probability calculations, but determining P(E | H) and P(E) to input into P(H | E) = P(E | H) * P(H) / P(E) is surprisingly hard and confusing.

Why There Is Hope For An Alignment Solution

Darklight

Note: I originally wrote the first draft of this on 2022/04/11 intending to post this to Less Wrong in response to the List of Lethalities post, but wanted to edit it a bit to be more rigorous and ended up taking a very long time before getting around to it. I’m still not entirely satisfied with the level of rigor, but I figure it’s worth posting to present some interesting ideas and perhaps give some people more hope about the future. I will admit I probably sound more confident in this essay about these ideas than I actually am, particularly of the more speculative ones. I originally titled this “The Alignment Solution”,... (read 3338 more words →)

I have some ideas and drafts for posts that I've been sitting on because I feel somewhat intimidated by the level of intellectual rigor I would need to put into the final drafts to ensure I'm not downvoted into oblivion (something a younger me experienced in the early days of Less Wrong).

Should I try to overcome this fear, or is it justified?

For instance, I have a draft of a response to Eliezer's List of Lethalities post that I've been sitting on since 2022/04/11 because I doubted it would be well received given that it tries to be hopeful and, as a former machine learning scientist, I try to challenge a lot of... (read more)

Thoughts On Computronium

Darklight

While it's widely accepted common knowledge that computers are considerably faster and more powerful than the human brain, it's arguable that evolution didn't optimize the human brain for raw speed, but rather, energy efficiency. The reason why this is important is that in the limit, the theoretical computronium needs to be not only powerful, but an efficient use of resources. In the limit, assuming reasonably that entropy cannot be reversed, then energy is our main practical confining factor for a universe filled with computronium or particularly utilitronium.

Furthermore, if natural selection has already provided a sufficiently close to optimal solution, it may make sense for Omega to fill the universe with human brain... (read 653 more words →)

Darklight's Shortform

Darklight

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

The Glory System: A Model For Moral Currency And Distributed Self-Moderation

Darklight

One of the most important changes in our civilized modern society over the past historical tendencies has been the state's monopoly on the use of violence. Most everyone today aside from the various strains of anarchist agree that this is a necessary and important requirement for modern life, but perhaps a problematic side effect of this change is that the use of force by individuals as a negative reinforcement tool to shape behaviour is lost. Whereas in the distant past, a slight against one's honour could be challenged by the threat of a duel, today, positive reinforcement through transfers of money are the main way human behaviour is shaped.

Markets rely on these... (read 690 more words →)

As a Washed Up Former Data Scientist and Machine Learning Researcher What Direction Should I Go In Now?

Darklight

Some background, I was interested in AI before the hype, back when neural networks were just an impractical curiosity in our textbooks. I went through an undergrad in Cognitive Science and decided that there was something to the idea of connectionist bottom up AI having tremendous untapped potential because I saw the working example of the human mind. So I embarked on a Masters in Computer Science focused on ML and eventually graduated at just about the perfect time (2014) to jump into industry and make a splash. It helped that I'd been ambitious and tried to create crazy things like the Music-RNN and the Earthquake Predictor Neural Network, which, though not... (read 806 more words →)

The Alpha Omega Theorem: How to Make an A.I. Friendly with the Fear of God

Darklight

Introduction

The Alpha Omega Theorem is a way to provide a powerful rational argument to a rational superintelligence that it should behave in a benevolent manner. There are two variants of the theorem, one weak, and one strong.

The Weak Alpha Omega Theorem:

There is a non-zero probability that a benevolent superintelligence already exists and dominates the universe.

The Strong Alpha Omega Theorem:

There is a greater probability that a benevolent superintelligence already exists and dominates the universe than not.

The Friendliness Corollary:

Attempting to achieve goals inconsistent with the goals of the predominant benevolent superintelligence is inherently futile. Therefore the optimal course of action is to align one’s goals with that of a benevolent superintelligence as best as... (read 707 more words →)

Symbolic Gestures – Salutes For Effective Altruists To Identify Each Other

Darklight

10y

A significant part of human communication is non-verbal. Body language and gestures often have the capacity to convey and signal a great detail and wealth of information about a person. Furthermore, historically, various organizations, ranging from secret societies to religions to militaries have sought to utilize very specific physical gestures as a way of communicating affiliation covertly or overtly for various purposes.

I would like to propose then that we within the Effective Altruist movement devise our own particular salute, to help us identify each other and also for the positive psychological effects that deliberate symbolism can entail.

Before I go on, I will emphasize here that I am aware of the potential for... (read 568 more words →)

-8

[LINK] Sentient Robots Not Possible According To Math Proof

Darklight

12y

Based on Tononi's earlier work on Integrated Information Theory, apparently, Maguire et al. have come up with a formulation of consciousness as a lossless integration of information that requires noncomputable functions, which implies that consciousness cannot be modeled computationally.

http://www.newscientist.com/article/dn25560-sentient-robots-not-possible-if-you-do-the-maths.html

I'm personally skeptical of this, but their paper (seen here: http://arxiv.org/abs/1405.0126v1) has some impressive looking formal mathematical proofs that I will admit I lack the mathematical competence to judge the veracity of. Anyone with greater mathematical acumen want to take a look?

-6

Eudaimonic Utilitarianism

Darklight

12y

Eliezer Yudkowsky on several occasions has used the term “Eudaimonia” to describe an objectively desirable state of existence. While the meta-ethics sequence on Less Wrong has been rather emphatic that simple universal moral theories are inadequate due to the complex nature of human values, one wonders, just what would happen if we tried anyway to build a moral theory around the notion of Eudaimonia. The following is a cursory attempt to do so. Even if you don’t agree with everything I say here, I ask that you please bear with me to the end before making judgments about this theory. Also, if you choose to downvote this post, please offer some criticism... (read 2893 more words →)