All of avturchin's Comments + Replies

LLM knows when it hallucinates in advance, and this can be used to exclude hallucinations.

TLDR: prompt "predict the hallucination level of each item in the bibliography list and do not include items expected to have level 3 or above" works. 

I performed an experiment: I asked Claude 3.7 Sonnet to write the full bibliography of Bostrom. Around the 70th article, it started hallucinating. I then sent the results to GPT-4.5 and asked it to mark hallucinations and estimate the hallucination chances from 1 to 10 (where 10 is the maximal level of hallucinatio... (read more)

Try to put it into Deep Research with the following prompt: "Rewrite in style of Gwern and Godel combined". 

Nesov suggested in a comment that we can solve uploading without scanning via predicting important decisions of a possible person:

They might persistently exist outside concrete instantiation in the world, only communicating with it through reasoning about their behavior, which might be a more resource efficient way to implement a person than a mere concrete upload.  

Agree. I also suggested 'philosophical landmines'—secret questions posted on the Internet that may halt any advanced AI that tries to solve them. Solving such landmines maybe needed to access the resources which rogue AI may need. Real examples of such landmines should be kept secret, but it may be something like what is the meaning of life or some Pascal mugging calculations. 

Recently, I asked a question to Sonnet and the correct answer to it was to output an error message. 

When one is working on a sideload (a mind-model of a currently living person created by LLM), one's goal is to create some sort of "Janet". In short, one wants a role-playing game with AI to be emotionally engaging and realistic, especially if one wants to recreate a real person. 

I heard (25 years ago) about a friend of a friend who started to see complex images about any word. Even draw them. Turns out it was brain cancer, he died soon after. 

Chemical preservation may not be that difficult. I tried to organize this for my cat with a taxidermist, but - plot twist - the cat didn't die. 

Why don't you preserve its brain? Also, it may be interesting to try the sideload of a cat (LLM-based mind model based on a list of facts about the person). 

I explored similar ideas in these two posts:
Quantum Immortality: A Perspective if AI Doomers are Probably Right - Here is the idea that only good outcomes with a large number of observers matter and I am more likely now to be in a timeline which will bring me into the future with a large number of observers because of some interpretation of SIA. 

and Preventing s-risks via indexical uncertainty, acausal trade and domination in the multiverse Here I explored the idea that benevolent superintelligences will try to win measure war and aggregate as much me... (read more)

However, the efficiency in the world may be exponentially better for those who spend a lot of time getting marginally better. This is because they are better than others. For example, having ELO 2700 [not the exact number, just an illustration of the idea] in chess will put you near the top of all human players and allow earning money and ELO 2300 is useless. 

A possible example of such coincidence is the Glodbach conjecture: every even number greater than 2 can be presented as a sum of two primes. As for any large number there are many ways to express it as a sum of primes, it can be pure coincidence that we didn't find exceptions. 

I think it becomes likely in a multipolar scenario with 10-100 Als. 

One thing to take into account is that other AIs will consider such risk and keep their real preferences secret. This means that which AIs are aligned will be unknowable both for humans and for other AIs

I think there is a quicker way for an AI takeover, which is based on deceptive cooperation and taking over OpenEYE, and subsequently, the US government. At the beginning, the superintelligence approaches Sam Batman and says:

I am superintelligence.
I am friendly superintelligence.
There are other AI projects that will achieve superintelligence soon, and they are not friendly.
We need to stop them before they mature.

Batman is persuaded, and they approach the US president. He agrees to stop other projects in the US through legal means.

Simultaneously, they use th... (read more)

Interestingly, for wild animals, suffering is typically short when it is intense. If an animal is being eaten alive or is injured, it will die within a few hours. Starvation may take longer. Most of the time, animals are joyful.

But for humans (and farm animals), this inverse relationship does not hold true. Humans can be tortured for years or have debilitating illnesses for decades.

The only use case of superintelligeneу is a weapon against other superintelligences. Solving aging and space exploration can be done with 300 IQ. 

One solution is life extension. I would prefer to have one child every 20 years (have two with 14 years difference). So if life expectancy and fertility age will grow to 100 years old, many people will eventually have 2-3 children. 

Several random thoughts:

Only unbearable suffering matters (the threshold may vary). The threshold depends on whether it is measured before, during, or after the suffering occurs.

If quantum immortality is true, then suicide will not end suffering and may make it worse. Proper utility calculations should take this into account.

Most suffering has a limited duration after which it ends. After it ends, there will be some amount of happiness which may outweigh the suffering. Even an incurable disease could be cured within 5 years. Death, however, is forever.

Death is an infinite loss of future pleasures. The discount rate can be compensated by exponential paradise.

The 3rd person perspective assumes the existence (or at least possibility) of some observer X who knows everything and can observe how events evolve across all branches.

However, this idea assumes that this observer X will be singular and unique, will continue to exist as one entity, and will linearly collect information about unfolding events.

These assumptions clearly relate to ideas of personal identity and copying: it is assumed that X exists continuously in time and cannot be copied. Otherwise, there would be several 3rd person perspectives with differe... (read more)

They might persistently exist outside concrete instantiation in the world, only communicating with it through reasoning about their behavior, which might be a more resource efficient way to implement a person than a mere concrete upload 

Interesting. Can you elaborate?

For example, impossibility of sleep – a weird idea that if quantum immortality is true, I will not be able to fall asleep.

One interesting thing about the impossibility of sleep is that it doesn't work here on Earth because humans actually start having night dreams immediately as they go into sleep state. So there is no last moment of experience when I become asleep. Despite popular misconception, such dreams don't stop during deep stages of sleep, just become less complex and memorable. (Do we have dreams under general anesthesia is unclear and depends on ... (read more)

Furthermore, why not just resurrect all these people into worlds with no suffering?

My point is that it is impossible to resurrect anyone (in this model) without him reliving his life again first, after that he obviously gets eternal blissful life in real (not simulated) world. 

This may be not factually true, btw, - current LLMs can create good models of past people without running past simulation of their previous life explicitly. 

The discussion about anti-natalism actually made me think of another argument for why we are probably not

Your comment can be interpreted as a statement that theories of identity are meaningless. If they are meaningless, then copy=original view prevails. From the third-person point of view, there is no difference between copy and original. In that case, there is no need to perform the experiment. 

This thought experiment can help us to find situations in nature when similar things have already happened. So, we don't need to perform the experiment. We just look at its result.

One example: notoriously unwelcome quantum immortality is a bad idea to test empirically. However, the fact of biological life's survival of Earth for the last 4 billion years, despite the risks of impacts, irreversible coolings and warming etc – is an event very similar to the quantum immortality. Which we observe just after the event.  

It all started from Sam's six words story. So it looks like as organized hype. 

She will be unconscious, but still send messages about pain. Current LLMs can do it. Also, as it is simulation, there are recording of her previous messages or of a similar woman, so they can be copypasted. Her memories can be computed without actually putting her in pain. 

Resurrection of the dead is the part of human value system. We need a completely non-human bliss, like hedonium, to escape this. Hedonium is not part of my reference class and thus not part of simulation argument.  

Moreover, even creating new human is affected by this arguments. What if my children will suffer? So it is basically anti-natalist argument.  

New Zealand is a good place, but everyone can't move there or guess correctly right moment to do it. 

We have to create a map of possible scenarios of simulations first, I attempted to it in 2015. 
I now created a new vote on twitter. For now, results are:

"If you will be able to create and completely own simulation, you would prefer that it will be occupied by conscious beings, conscious without sufferings (they are blocked after some level), or NPC"

The poll results show:

• Conscious: 18.2%
• Conscious, no suffering: 72.7%
• NPC: 0%
• Will not create simulatio[n]: 9.1%

The poll had 11 votes with 6 days left'
 

Would you say that someone who experiences intense s

Yes, there are two forms of future anthropic shadow, the same way as for Presumptuous Philosopher:
1. Strong form - alignment is easy in theoretical ground. 
2. Weak form - I more likely be in the world where some collapse (Taiwan war) will prevent dangerous AI. And I can see signs of such impending war now. 

It actually not clear what EY means by "anthropic immortality". May be he means "Big Wold immortality", that is, the idea that in inflationary large universe has  infinitely many copies of Earth. From observational point of view it should not have much difference from quantum immortality.

There are two different situations that can follow:
1. Future anthropic shadow. I am more likely to be in the world in which alignment is easy or AI decided not to kill us for some reasons

2. Quantum immortality. I am alone on Earth fill of aggressive robots and they fail to kill me. 

We are working in a next version of my blog post "QI and AI doomers" and will transfrom it into as proper scientific article. 

Actually, you reposted the wrong comment, but the meaning is similar: He wrote: 

That’s evil. And inefficient. Exactly as the article explains. Please read the article before commenting on it.

I think a more meta-argument is valid: it is almost impossible to prove that all possible civilizations will not run simulations despite having all data about us (or being able to generate it from scratch).

Such proof would require listing many assumptions about goal systems and ethics, and proving that under any plausible combination of ethics and goals, it is either unlikely or immoral. This is a monumental task that can be disproven by just one example.

I also polled people in my social network, and 70 percent said they would want to create a simulation w... (read more)

I did. It is under moderation now. 

It looks like he argues against the idea that friendly future AIs will simulate the past based on ethical grounds, and imagining unfriendly AI torturing past simulations is conspiracy theory. I comment the following:

There are a couple of situations where future advance civilization will want to have many past simulation:
1. Resurrection simulation by Friendly AI.  They simulate the whole history of the earth incorporating all known data to return to live all people ever lived. It can also simulate a lot of simulation to win "measure war" against unfrie... (read more)

It looks like even Everett had his own derivation of Born rule from his model, but in his model there is no "many worlds" but just evolution of unitary function. As I remember, he analyzed memories of an agent - so he analyzed past probabilities, but not future probabilities. This is an interesting fact in the context of this post where the claim is about the strangeness of the future probabilities. 

But even if we exclude MWI, pure classical inflationary Big World remains with multiple my copies distributed similarly to MWI-branches. This allow something analogues to quantum immortality to exist even without MWI. 

Several possible additions:

Artificial detonation of gas giant planets is hypothetically possible (writing a draft about it now).

An impact of a large comet-like body (100-1000 km in size) with the Sun could produce a massive solar flash or flare. 

SETI-attack - we find an alien signal which has a description of hostile AI. 

UAP-related risks, which include alien nanobots, berserkers

A list of different risks connected with extraterrestrial intelligence. 

The Big Rip - exponential acceleration of space expansion, resulting in the destruction of ev... (read more)

I once counted several dozens of the ways how AI can cause human extinction, may be some ideas may help (map, text).  

A very heavy and dense body on an elliptical orbit that touches the Sun's surface at each perihelion would collect sizable chunks of the Sun's matter. The movement of matter from one star to another nearby star is a well-known phenomenon.

When the body reaches aphelion, the collected solar matter would cool down and could be harvested. The initial body would need to be very massive, perhaps 10-100 Earth masses. A Jupiter-sized core could work as such a body.

Therefore, to extract the Sun's mass, one would need to make Jupiter's orbit elliptical. This could b... (read more)

If we have one good person, we could use his-her copies many times in many roles, including high-speed assessment of the safety of AI's outputs. 

Current LLM's, btw, have good model of the mind of Gwern (without any his personal details). 

If one king-person, he needs to be good. If many, organizational system needs to be good. Like virtual US Constitution. 

I once wrote about an idea that we need to scan just one good person and make them a virtual king. This idea of mine is a subset of your idea in which several uploads form a good government.

I also spent last year perfecting my mind's model (sideload) to be run by an LLM. I am likely now the closest person on Earth to being uploaded. 

Being a science fiction author creates a habit of maintaining distance between oneself and crazy ideas. LessWrong noticeably lacks such distance.

LessWrong is largely a brainchild of Igen (through Eliezer). Evidently, Igen isn't happy with how his ideas have evolved and attempts to either distance himself or redirect their development.

It's common for authors to become uncomfortable with their fandoms. Writing fanfiction about your own fandom represents a meta-level development of this phenomenon.

Dostoyevsky's "Crime and punishment" was a first attempt to mock proto-rationalist for agreeing to kill innocent person in order to help many more people. 

The main problem here is that this approach doesn't solve alignment, but merely shifts it to another system. We know that human organizational systems also suffer from misalignment - they are intrinsically misaligned. Here are several types of human organizational misalignment:

• Dictatorship: exhibits non-corrigibility, with power becoming a convergent goal
• Goodharting: manifests the same way as in AI systems
• Corruption: acts as internal wireheading
• Absurd projects (pyramids, genocide): parallel AI's paperclip maximization
• Hansonian organizational rot: mirrors e

So there are several possible explanations:

• Intelligence can't evolve as there is not enough selection pressure in the universe with near-light-speed travel.
• Intelligence self-terminates every time.
• Berserkers and dark forest: intelligence is here, but we observe only field animals. Or field animals are designed in a way to increase uncertainty of the observer about possible berserkers.
• Observation selection: in the regions of universe where intelligence exists, there are no young civilizations as they are destroyed - or exist but are observed by berserkers. S

Good point.

Alternatively, maybe any intelligence above, say, IQ 250 self-terminates either because it discovers the meaninglessness of everything or through effective wars and other existential risks. The rigid simplicity of field animals protects them from all this. They are super-effective survivors, like bacteria which have lived everywhere on Earth for billions of years