When discussing the physics behind why the sky is blue, I'm surprised that the question 'Why isn't it blue on Mars or Titan?' isn't raised more often. Perhaps kids are so captivated by concepts like U(1) that they overlook inconsistencies in the explanation.
Just realized that stability of goals under self-improvement is kinda similar to stability of goals of mesa-optimizers; so there vingian reflection paradigm and mesa-optimization paradigm should fit.
What are practical implication of alignment research in the world where AGI is hard?
Imagine we have a good alignment theory but do not have AGI. Can this theory be used to manipulate existing superintelligent systems such as science, deep state, stock market? Does alignment research have any results which can be practically used outside of AGI field right now?
How does AGI solves it's own alignment problem?
For the alignment to work its theory should not only tell humans how to create aligned super-human AGI, but also tell AGI how to self-improve without destroying its own values. Good alignment theory should work across all intelligence levels. Otherwise how does paperclips optimizer which is marginally smarter than human make sure that its next iteration will still care about paperclips?
I don’t know too much about alignment research, but what surprises me most is lack of discussion of two points:
For the alignment to work its theory should not only tell humans how to create aligned super-human AGI, but also tell that AGI how to self-improve without destroying its own values. Otherwise how does paperclips optimizer which is marginally smarter than human make sure that its next iteration will still care about paperclips? Good alignment theory should work across all intelligence levels.
What are practical implication of alignment researc
What do you think about offering an option to divest from companies developing unsafe AGI? For example, by creating something like an ESG index that would deliberately exclude AGI-developing companies (Meta, Google etc) or just excluding these companies from existing ESGs.
The impact = making AGI research a liability (being AGI-unsafe costs money) + raising awareness in general (everyone will see AGI-safe & AGI-unsafe options in their pension investment menu + a decision itself will make a noise) + social pressure on AGI researchers (equating them...
You can do something similar to the Drake equation:
where Nlife is how many stars with life there are in the Milky Way and it is assumed that a) once self-replicating molecule is evolved it produces life with 100% probability; b) there is an infinite supply of RNA monomers, and c) lifetime of RNA does not depend on its length. In addition:
On the object level it looks like there are a spectrum of society-level interventions starting from "incentivizing research that wouldn't be published" (which is supported by Eliezer) and all the way to "scaring the hell out of general public" and beyond. For example, I can think of removing $FB and $NVDA from ESGs, disincentivizing publishing code and research articles in AI, introducing regulation of compute-producing industry. Where do you think the line should be drawn between reasonable interventions and ones that are most likely to backfire?
On the me...
You haven't commented much on Eliezer's views on the social approach to slow down the development of AGI - the blocks starting with
I don't know how to effectively prevent or slow down the "next competitor" for more than a couple of years even in plausible-best-case scenarios.
and
I don't want to sound like I'm dismissing the whole strategy, but it sounds a lot like the kind of thing that backfires because you did not get exactly the public reaction you wanted
What's your take on this?
Here are some other failure modes that might be important:
The Covid origin story (https://www.facebook.com/yudkowsky/posts/10159653334879228) - some sort of AI research moratorium is held in US, the problem appears to be solved, but in reality it is just off-shored, and then it explodes in an unpredicted way.
The Archegos/Credit Suisse blow up (https://www.bloomberg.com/opinion/articles/2021-07-29/archegos-was-too-busy-for-margin-calls) - special comittee is set up to regulate AI-related risks, and there is a general consensus that something has to be...
An important consideration is whether you are trying to fool simulated creatures into believing simulation is real by hiding glitches, or you are doing an honest simulation and allow exploitation of these glitches. You should take it into account when you consider how deep you should simulate matter to make simulation plausible.
For example up until 1800-s you could coarse-grain atoms and molecules, and fool everyone about composition of stuff. The advances in chemistry and physics and widespread adoption of inventions relying on atomic theory made it progr...
I think the offer needs to be modified to generate more solid market.
First, instead of making a crazy N-point contract, the correct way to trade souls is through the NFT auctions/markets. The owner gets the same symbolic rights as the owner of the arts sold through this mechanism, and there are no those extra unclear requirements on seller's actions that will hinder the healthy financial derivatives market.
Second, only desperate people sell their whole soles. Obviously, you should trade shares of souls.
So, how much do you think it will cost to develop a platform, where one can register, put a share of one's soul for auction, or build a solid portfolio of souls? What do you think the market size would be?
If I follow the logic correctly, the root cause of aging is that stem cells can irreversibly and invisibly accumulate active transposones, which are then passed on to derived cells, which then become senescent much faster. Also, for some reason this process is supressed in gonads. So, I see these alternatives:
On the 'old parent' hypothesis : let me point out that if there is not a full 'reset' somewhere in the chain, well, we wouldn't exist, because some of our ancestors were old parents, and the effect would be cumulative, such that none of us would live past childhood due to aging. (there is a disease like that, albeit with a different root cause)
Here is another example of inadequacy / inefficiency in the pharmaceutical market.
Cancer X is very aggressive and even when it is diagnosed at very early stage and surgically removed, the recurrence rate is something around 70% in 5 years. When cancer returns or when a patient has advanced stage, the mean survival time is only 6 months.
Pharmaceutical company Y has recently discovered a new anticancer drug. According to state-of-art preclinical experiments, the drug inhibits spread of cancer and kills cancer cells very effectively. Top scientists at company...
This is very strong claim. It puts severe limitations on biotech capabilities. Do you have any references to support it?