All of PashaKamyshev's Comments + Replies

5Steven Byrnes
I think when you say “force the idea of “digital life,” “digital minds” or “uploads” onto people” and such, you are implying that there are people who are in favor of uploading everyone including people who don’t want to be uploaded. If that’s not what you believe, then I think you should change the wording. This isn’t about vibes, it’s about what people actually say, and what they believe. I think you are misreading vibes in various ways, and therefore you should stick to what they actually say. It’s not like Robin Hanson and Eliezer are shy about writing down what they think in excruciating detail online. And they do not say that we should upload everyone including people who don’t want to be uploaded. For example, here’s an Age of Em quote which (I claim) is representative of what Robin says elsewhere: Note that he doesn’t say is that these “some” are a problem and we need to fix this problem by force of law. Here’s another quote: Emphasis added. There is nothing in the book that says we should or will forcibly upload people who don’t want to be uploaded, and at least this one passage explicitly to the contrary (I think there are other passages along the same lines). I’m confused. In our analogy (uploading ↔ going to Mars), “go to Mars and then forcefully prevent other people from going there” would correspond to “upload and then forcefully prevent other people from uploading”. Since when does Nate want to prevent people from uploading? That’s the opposite of what he wants. I’m not sure why you expect digital minds to be unproductive. Well, I guess in a post-AGI era, I would expect both humans and uploads to be equally economically unproductive. Is that what you’re saying? I agree that a superintelligent AGI sovereign shouldn’t give equal Universal Basic Income shares to each human and each digital mind while also allowing one person to make a gazillion uploaded copies of themselves which then get a gazillion shares while the biological humans only get one

I think this post suffers pretty badly from Typical Mind Fallacy. This thinking isn't alien to me. I used to think exactly like this 8 years ago, but since marriage and kid I now disagree with basically every point.

One claim that is hopefully uncontroversial: Humans are not literally optimizing for IGF,

I think this is controverisial because it's basically wrong :)

First, its not actually obvious what "definition" of IGF you are using. If you talk about animals, the definition that might fit is "number of genes in the next generation". However if you talk ab... (read more)

7Steven Byrnes
None of those say (or imply) that we should forcibly upload people who don't want to be uploaded. I think nobody believes that, and I think you should edit your post to not suggest that people do. By analogy: * I can believe that people who don't want to go to Mars are missing out on a great experience, but that doesn't mean I'm in favor of forcing people who don't want to go to Mars to go to Mars. * I can desire for it to be possible to go to Mars, but that doesn't mean I'm in favor of forcing people who don't want to go to Mars to go to Mars. * I can advocate for future Martians to not be under tyrannical control of Earthlings, with no votes or political rights, but that doesn't mean I'm in favor of forcing people who don't want to go to Mars to go to Mars. * I can believe that the vast majority of future humans will be Martians and all future technology will be invented by them, but that doesn't mean I'm in favor of forcing people who don't want to go to Mars to go to Mars. Right? The two people you cite have very strong libertarian tendencies. They do NOT have the belief "something is a good idea for an individual" ---> "...therefore obviously the government should force everyone to do it" (a belief that has infected other parts of political discourse, a.k.a. everything must be either mandatory or forbidden). If your belief is "in the unlikely event that uploading is possible at all, and somebody wants to upload, then the government should prevent them from doing so" - as it seems to be - then you should say that explicitly, and then readers can see for themselves which side of this debate is in favor of people imposing their preferences on other people.

I generally don't think LLMs today are conscious, as far as i can tell neither does Sam Altman, but there is some disagreement. They could acquire some characteristics that could be considered conscious as scale increases. However merely having "qualia" and being conscious is not the same thing as being functionally equivalent a new human, let alone a specific human. The term "upload" as commonly understood is a creation of a software construct functionally and qualia-equivalent to a specific human.

  • a human brain in a vat wouldn't be so far from the experie
... (read more)

Thanks for the first part of the comment. 

As mentioned in my above comment, the reason for mixing "can" and "should" problems is that they form a "stack" of sorts, where attempting to approximately solve the bottom problems makes the above problems harder and verification is important. How many people would care about the vision if one could never be certain the process succeeds?

Fixed the link formatting and added a couple more sources, thanks for the heads up. The temperature claim does not seem unusual to me in the slightest. I have personally tried to do a relatively cold bath and noticed my "perception" alter pretty significantly. 

The organ claim does seem more unusual, but I have heard various forms of it from many sources at this point. It does not however seem in any way implausbile. Even if you maintain that the brain is the "sole" source of cognition, the brain is still an organ and is heavily affected by the operation of other organs.

8[anonymous]
Sure but if all of the cognition is within the brain, the rest can be conceivably simulated as inputs to the brain, we might also have to simulate an environment for it.   Yours is ultimately a thesis about embodied cognition, as I understand it. If cognition is strongly embodied, then a functional human brain will need to have a very accurately simulated/emulated human body. If cognition is weakly embodied, and the brain's digitalised neuroplasticity is flexible enough, we can get away with not simulating an actual human body.     I don't think the debate is settled.

There is a lot to unpack. I have definitely heard from leaders of the community claims to the tune of "biology is over," without further explanation of what exactly that means or what specific steps are expected to happen when the majority of people disagree with this. The lack of clarity here makes it hard to find a specific claim of "I will forcefully do stuff to people they don't like," but me simply saying "I and others want to actually have what we think of as "humans" keep on living" is met with some pushback.

You seem to be saying that the "I" or "Se... (read more)

8Steven Byrnes
I am very highly confident that "leaders of the community" would be unhappy with a future where everyone who wants to live out their lives as biological humans are unable to do so. I don't know what you heard, or from who, but you must have misunderstood it. I think it's possible that said biological humans will find that they have no gainful employment opportunities, because anything they can do, can alternatively be done by a robot who charges $0.01/hour and does a much better job. If that turns out to be the case, I hope that Universal Basic Income will enable those people to have a long rich "early retirement" full of travel, learning, friendship, family, community, or whatever suits them. I also think it's pretty likely that AI will wipe out the biological humans, using plagues and missile strikes and so on. In the unlikely event that there are human uploads, I would expect them to get killed by those same AIs as well. Obviously, I'm not happy to have that belief, and I am working to make it not come true. Speaking of which, predicting that something will happen is different from (in fact, unrelated to) hoping that it will happen. I’ve never quite wrapped my mind around the fact that people mix these two things up so often. But that’s not a mistake that “community leaders” would make. I wonder if the “biology is over” claim was a prediction that you mistook as being a hope? By the same token, “uploading is bad” and “uploading is impossible” are not two layers of the same stack, they’re two unrelated claims. All four combinations (bad+impossible, good+impossible, bad+possible, good+possible) are perfectly coherent positions for a person to hold.

The general vibe of the first two parts seems correct to me. Also, an additional point is that evolution's utility function of inclusive genetic fitness didn't completely disappear and is likely still a sub-portion of the human utility function. I suspect there is going to be disagreement on this, but it would also be interesting to do a poll on this question and break it down by people who do or do not have kids.

Yes I think we understand each other. One thing to keep in mind is that different stakeholders in AI are NOT utilitarians, they have local incentives they individually care about. Given the fact that COVID didn't stop gain-of-function research, this means that getting EVERYONE to care would require a death toll larger than COVID. However, getting someone like CEO of google to care would "only" require a half - a - trillion dollar lawsuit against Microsoft for some issue relating to their AIs. 

And I generally expect those - types of warning shots to be pretty likely given how gun-ho the current approach is.

I am mostly agreeing with you here, so I am not sure you understood my original point. Yes Reality is giving us things that for a set of reasonable people such as you and me should be warning shots. 

Since a lot of other people don't react to them, you might become pessimistic and extrapolate that NO warning shot is going to be good enough. However I posit that SOME warning shots are going to be good enough. An AI - driven bank run followed by an economic collapse is one example, but there could be others. Generally I expect that when warning shots reach "nation-level" socio-economic problems, people will pay attention. 

However, this will happen before doom.

2DaemonicSigil
Thanks for the reply, I think we do mostly agree here. Some points of disagreement might be that I'm not at all confident that we get a truly large scale warning shot before AI gets powerful enough to just go and kill everyone. Like I think the threshold for what would really get people paying attention is above "there is a financial disaster", I'm guessing it would actually take AI killing multiple people (outside of a self-driving context). That could totally happen before doom, but it could also totally fail to happen. We probably get a few warning shots that are at least bigger than all the ones we've had before, but I can't even predict that with much confidence.

I am also familiar with Paul Christiano, I think his arguments for slower, more continous take off are broadly on the right track as well. 

Given that the extreme positions have strong stake-outs on twitter, I am once again claiming that there needs to be a strong stake-out of the more reasonable centrism. This isn't the first post in this direction, there were ones before and there will be ones after. 

Just trying to keep this particular ball rolling. 

3the gears to ascension
twitter is a toxicity machine and as a result I suspect that people who are much at all +reasonableness are avoiding it - certainly that's why I don't post much and try to avoid reading from my main feed, despite abstractly agreeing with you. that said, here's me, if it helps at all: https://twitter.com/lauren07102

I have skimmed the Alignment Forum side and read most of MIRI's work before 2015. While it's hard to know about the "majority of people," it does seem that the public reporting is around two polarized camps. However in this particular case, I don't think it's just the media. The public figures for both sides (EY and Yann Lecunn) seem pretty consistent with their messaging and talking past each other. 

Also if the majority of people in the field agree with the above, that's great news and also means that reasonable centrism needs to be more prominently ... (read more)

5the gears to ascension
this seems like a major concern in and of itself.
2Seth Herd
The public figures are drawn from the most extreme positions. And Yudkowsky founded this field, so he's also legitimately the most desired speaker. But things have changed a lot since 2015. Check out Paul Christiano, Alex Turner, and Steve Byrnes for different views that are neither doomer nor foomer. I don't have a survey result handy, but the ones I vaguely remember put the p(doom) estimates from within the field at vastly lower than MIRI's 90%+.

Your comment is a little hard to understand. You seem to be saying that "scaling" is going to make it harder to align, which I agree with. I am not sure what "deliberate reasoning" means in this context. I also agree that having a new kind of training process is definitely required to keep GPT aligned either vis-a-vis OpenAI's rules or actually good rules. 

I agree that the current model breaks down into "shoggoth" and "mask." I suspect future training, if it's any good would need to either train both simultaneously with similar levels of complexity fo... (read more)

2Vladimir_Nesov
Right now, while pre-training is mostly on human-written text and not synthetic data, masks are human imitations, mostly aligned by default, regardless of how well they adhere to any intended policy. Shoggoths are implementation details of the networks that don't necessarily bear any resemblance to humans, they are inscrutable aliens by default, and if they become agentic and situationally aware ("wake up"), there is no aligning them with the technology currently available. It's not a matter of balance between masks and shoggoths. Deliberative reasoning is chain-of-thought, study of deliberative reasoning is distillation of things arrived-at with chain-of-thought back into the model, preparing datasets of synthetic data produced by a LLM for training it to do better. This is already being done in GPT-4 to mitigate hallucinations (see the 4-step algorithm in section 3.1 of the System Card part of GPT-4 report). There's probably going to be more of this designed to train specialized or advanced skills, eventually automating training of arbitrary skills. The question is if it's going to sufficiently empower the masks before dangerous mesa-optimizers arise in the networks ("shoggoths wake up"), which might happen as a result of mere additional scaling.

While I agree that Twitter is a bad site, I expect some of Musk's actions to make it better (but not fully fix it). Your attempt to tie personality-based critiques (stem / white / male) isn't helpful. Addiction to social platforms is a general issue and needs to be solved in a general way. 

 

However, the solutions you outline are in fact some of the ways that the situation will proceed. I don't think 1. [government] is likely or will sit well either.

However, 2 [fix] is plausible. Companies would not "admin" problems, but they could fix without "ad... (read more)

1scrollop
Seems Musks actions are not "making it better". Surprise, surprise: the opposite: https://www.bbc.co.uk/news/technology-64804007
2scrollop
What are you basing your optimism for Musk's future for Twitter on? (Sorry, I'm doing something wrong trying to insert links with markdown on)   [AP: eport: Tweets with racial slurs soar since Musk takeover]  (https://apnews.com/article/elon-musk-technology-business-government-and-politics-2907d382db132cfd7446152b9309992c?) [BBC: Scale of abuse of politicians on Twitter revealed] (https://www.bbc.co.uk/news/uk-63330885) [Reuters: Elon Musk's Twitter slow to act on misleading U.S. election content, experts say](https://www.reuters.com/technology/elon-musks-twitter-girds-surge-us-midterm-election-misinformation-2022-11-08/) What Musk says: ["Mr Musk insisted that the platform's commitment to moderation remained "absolutely unchanged"."](https://news.sky.com/story/elon-musk-defends-culling-twitter-staff-but-insists-commitment-to-moderation-remains-absolutely-unchanged-12738642) What Musk does: ["Yesterday’s reduction in force affected approximately 15% of our Trust & Safety organization (as opposed to approximately 50% cuts company-wide"](https://twitter.com/yoyoel/status/1588657227035918337) The market will decide what to do with Twitter, it seems, though these are early days. His antics and hypocrisy [aren't a good sign](https://www.thelondoneconomic.com/news/elon-musk-becomes-the-butt-of-the-joke-after-he-welcomes-comedy-back-on-twitter-338363/)   In terms of your riposte  "Your attempt to tie personality-based critiques (stem / white / male) isn't helpful.": The following quotes are from the book: [The Psychology of Silicon Valley, Cook, K. (2020)][https://link.springer.com/chapter/10.1007/978-3-030-27364-4_2)   "Simon Baron-Cohen, a psychologist and researcher at the University of Cambridge, has researched the neurological characteristics endemic in certain fields, most notably in science, technology, engineering, and mathematics (STEM ) professions. Baron-Cohen has repeatedly found that those with autism or autistic traits are over-represented in t

Yeah defining "harm", or more formally, a "non-beneficial modification to a human being" is a hard task and is in many ways the core problem I am pointing at. Allowing people to take part in defining what is "harmful" to themselves is both potentially helpful as it brings local information and tricky because people may have already been ensnared by a hostile narrow AI to misunderstand "harm."

Thanks for the comment! I look into some of the philosophy required in part 4, x-risk relationships in part 5 and todos in part 6. Understanding consciousness is an important sub-component and might be needed sooner than we think. I think an important piece is understanding what modifications to consciousness are harmful or beneficial. This would have a sub-problem of what chemicals or organisms alter it and in what ways as well as what ideas and experiences seem to have a lasting effect on people. It's possible this is more doable than understanding consciousness as a whole, but it certainly touches a lot of the core problem. 

Mostly agree. A sophisticated user might have some great feedback on how a website ranks its products, but shouldn't and doesn't want to have access to the internals of the algorithm. So giving some users a slightly better surface area for interaction aside from "being part of the training data" seems like an important problem to solve. Could be useful all the way toward AGI as we would need a story of how a particular person still has some capacity for future influence.

2Nathan Helm-Burger
Yeah, good point. Some balance needs to be struck in such a scenario where people are given the power to do some sort of customization, but not so much that they can warp the intended purpose of the model into being directly harmful or helping them be harmful.