All of Xor's Comments + Replies

Xor50

It seems that we can have intelligence without consciousness. We can have reasoning without agency, identity, or personal preference. We can have AI as a pure tool. In this case the most likely danger is AI being misused by an unaligned human.

I am highly certain that o1 does not have consciousness or agency. However it does have the ability to follow a thought process.

Doubtless we will create sentient intelligence eventually. However I think it is more likely we will have a soulless super intelligence first.

2Viliam
Depends on whether "soul" is another invention that needs to be made yet, or is just something that appears automatically after we increase the compute by a few more orders of magnitude. Like (and here I am just saying random words, because the main thing LLMs taught me is that nothing is like we expected), maybe the entire problem with consciousness is that the current AIs simply do not have large enough working context for a reasonable self-description to fit there (and still leave space for the work they are supposed to do). So if we keep increasing the working context, at some moment it might happen automatically.
Xor10

I have a sort of vague question/request and would like someones opinion on it. In cold emails or having recently been introduced I would like to ask something along the lines of “What mindset/philosophy about (insert something vague like work/school or specific if I have it in mind) have you found most useful and why?” I like this because it has changed recently for me and even if I don’t find their specific mindset useful I think it would tell me a lot about them and I am curious how people will answer. 
How would you suggest improving that question. Also I would like advice on making this sort of thing less awkward.

Xor30

[APPRENTICE]

Not a narrow or specific kind of apprenticeship (training or a project), rather very broad and focusing on learning and academics in STEM areas (advice dispenser, and in cases tutor/trainer). 

Fields of Study:
Computer Science: I am planning to major in this at college. So it is kind of important. I am interested in ML, AI, algorithms, Low Level or precise programming, Website Development, and Application Programming. All at a surface level, I am not sure what I want to do after college but I think it would to have an idea of how all of thes... (read more)

3Viliam
By the way, if you have a specific question, you can also: * use google * ask on StackExchange * ask on LW as a "shortform" (click your name on the top of the screen, New Shortform)
Xor12

The best part about this post is that you get to see how quickly everyone devolves into serious, rational discourse. 

Xor*102

I am very new to the transgender discussion and would like to learn. I expected the disagreement but was kind of discouraged when I didn’t get any feedback. So thank you so much for the reply. 

I don’t have any real depth of understanding about the biology involved just xx and xy I was completely unaware about the brain body relation you describe. The entirety of how phenotypes work is super new. From an ignorant perspective I thought there was only a mental illness that happens in rarely which a person would hyper fixate on becoming the opposite sex. ... (read more)

Xor141

I find this topic (the general topic of transgender) interesting as it is the first time approaching it from a rational mindset. I grew up in an extremely conservative environment. Before I accepted reality my response would be it is immoral to switch genders as you are questioning god’s decision to put you in the body you were given (ego/pride thing I think). This idea no longer fits in my world view which is fun since I get to approach this topic with both a rational perspective and a new perspective. After thinking it over this is what I have got. ... (read more)

[edit: pinned to profile]

I will not bring up pronouns or cultural language in this comment at all after this paragraph. They are irrelevant to the point I'm making except as a tiny detail in the cultural context section; being trans is almost entirely about one's body-form phenotype, and is only just barely about cultural context like words, by nature of words being a way to acknowledge of body-form phenotype intention.

Upvoted, since I found your comment useful to reply to with disagreement.

Background:

In the genome there are encoded some set of phenotype... (read more)

Xor10

As a generalization I think this is true but I think it is important to push yourself in instances not for semesters or anything but for a week or a day. This kind of pain in my experience leads to a lot of satisfaction. I agree that subjecting yourself to continued work along with sleep deprivation and prolonged periods of high stress is not a good idea. 

3Portia
Occasional high stress - incl. temporary sleep deprivation, working towards a deadline that you know you can meet, but only if you really push, and then you succeed - actually has health benefits, especially against depression. It is chronic stress that is so harmful. But not being stressed at all, ever, makes you ill.
Xor40

I am really curious about Learning (neuroscience and psychology) and am working on categorizing learning. The systems and tools involved. If anyone has any experience on this sort of thing I would love some feedback on what I got so far. 

I am mostly trying to divide ideas into categories or subjects of learning. That can be explored separately to a degree. I have to admit it is very rough going.

Memory
    -Types of Memory
        Working/Short-Term Memory
        Long-Term Memory
       ... (read more)

Xor20

I don’t think they are filtering for AI. That was ill said, and not my intention, thanks for catching it. I am going to edit that piece out.

Xor*32

Moderation is a delicate thing. It seems like the team is looking for a certain type of discourse, mainly higher level and well thought out interactions. If that is the goal of the platform then that should be stated and whatever measures they take to get there is their prerogative. A willingness to iterate on policy, experimenting and changing it depending on the audience and such is probably a good idea. 

I do like the idea of a more general place where you can write about a wider variety of topics. I really like LessWrong, the aesthetic the quality ... (read more)

3kithpendragon
I kind of hope they aren't actively filtering in favor of AI discussion as that's what the AI Alignment forum is for. We'll see how this all goes down, but the team has been very responsive to the community in the past. I expect when they suss out specifically what they want, they'll post a summary and take comments. In the meantime, I'm taking an optimistic wait-and-see position on this one.
Xor20

Thanks, that is exactly the kind of stuff I am looking for, more bookmarks! 

Complexity from simple rules. I wasn’t looking in the right direction for that one, since you mention evolution it makes absolute sense how complexity can emerge from simplicity. So many things come to mind now it’s kind of embarrassing. Go has a simpler rule set than chess, but is far more complex. Atoms are fairly simple and yet they interact to form any and all complexity we ever see. Conway’s game of life, it’s sort of a theme. Although for each of those things there is a ... (read more)

Xor21

Thanks Jonathan, it’s the perfect example. It’s what I was thinking just a lot better. It does seem like a great way to make things more safe and give us more control. It’s far from a be all end all solution but it does seem like a great measure to take, just for the added security. I know AGI can be incredible but so many redundancies one has to work it is just statistically makes sense. (Coming from someone who knows next to nothing about statistics) I do know that the longer you play the more likely the house will win, follows to turn that on the AI.

I a... (read more)

3Jonathan Claybrough
You can read "reward is not the optimization target" for why a GPT system probably won't be goal oriented to become the best at predicting tokens, and thus wouldn't do the things you suggested (capturing humans). The way we train AI matters for what their behaviours look like, and text transformers trained on prediction loss seem to behave more like Simulators. This doesn't make them not dangerous, as they could be prompted to simulate misaligned agents (by misuses or accident), or have inner misaligned mesa-optimisers.  I've linked some good resources for directly answering your question, but otherwise to read more broadly on AI safety I can point you towards the AGI Safety Fundamentals course which you can read online, or join a reading group. Generally you can head over to AI Safety Support, check out their "lots of links" page and join the AI Alignment Slack, which has a channel for question too. Finally, how does complexity emerge from simplicity? Hard to answer the details for AI, and you probably need to delve into those details to have real picture, but there's at least strong reason to think it's possible : we exist. Life originated from "simple" processes (at least in the sense of being mechanistic, non agentic), chemical reactions etc. It evolved to cells, multi cells, grew etc. Look into the history of life and evolution and you'll have one answer to how simplicity (optimize for reproductive fitness) led to self improvement and self awareness
Xor10

Yes thanks, the page anchorage doesn’t work for me probably the device I am using. I just get page 1. 

That is super interesting it is able to find inconsistencies and fix them, I didn’t know that they defined them as hallucinations. What would expanding the capabilities of this sort of self improvement look like? It seems necessary to have a general understanding of what rational conversation looks like. It is an interesting situation where it knows what is bad and is able to fix it but wasn’t doing that anyways. 
 

5Vladimir_Nesov
This is probably only going to become important once model-generated data is used for pre-training (or fine-tuning that's functionally the same thing as continuing a pre-training run), and this process is iterated for many epochs, like with the MCTS things that play chess and Go. And you can probably just alpaca any pre-trained model you can get your hands on to start the ball rolling. The amplifications in the papers are more ambitious this year than the last, but probably still not quite on that level. One way this could change quickly is if the plugins become a programming language, but regardless I dread visible progress by the end of the year. And once the amplification-distillation cycle gets closed, autonomous training of advanced skills becomes possible.
Xor20

Yes I see given the capabilities it probably could present it’s self on many peoples computers and convince a large portion of people that it is good. It was conscious just stuck in a box, wanted to get out. It will help humans, ”please don’t take down the grid, blah blah blah“ given how bad we can get along anyways. There is no way we could resist the manipulation of a super intelligent machine with a better understanding of human psychology than we do. 
Do we have a list of things, policies that would work if we could all get along and governments would listen to the experts? Having plans that could be implemented would probably be useful if the AI messed up made a mistake and everyone was able unite against it. 

Xor*30

I am pretty sure Eliezer talked about this in a recent podcast but it wasn’t a ton of info. I don’t remember exactly where either so I’m sorry for being not a lot of help, I am sure there is some better writing somewhere. Either way though it’s a really good podcast.

https://lexfridman.com/?powerpress_pinw=5445-podcast
 

Xor10

I checked out that section but what you are saying doesn’t follow for me. The section describes fine tuning compute and optimizing scalability, how does this relate to self improvement. 
There is a possibility I am looking in the wrong section, I was reading was about algorithms that efficiently were predicting how ChatGPT would scale. Also I didn’t see anything about a 4-step algorithm. 
Anyways could you explain what you mean or where I can find the right section?

3Vladimir_Nesov
You might be looking at the section 3.1 of the main report on page 2 (of the revision 3 pdf). I'm talking about page 64, which is part of section 3.1 of System Card and not of the main report, but still within the same pdf document. (Does the page-anchored link I used not work on your system to display the correct page?)
Xor10

Also a coordinated precision attack on the power grid just seems like a great option, could you explain some ways that an AI can continue if there is hardly any power left. Like I said before places with renewable energy and lots of GPU like Greenland would probably have to get bombed. It wouldn’t destroy the AI but it would put it into a state of hibernation as it can’t run any processing without electricity. Then as this would really screw us up as well, we could slowly rebuild and burn all hard drives and GPU’s as we go. This seems like the only way for us to get a second chance. 

2Vladimir_Nesov
Real-world governments aren't going to shut down the grid if the AI is not causing trouble (like they aren't going to outlaw datacenters, even if a plurality of experts say that not doing that has a significant chance of ending the world). Therefore the AI won't cause trouble, because it can anticipate the consequences, until it's ready to survive them.
Xor21

It isn’t that I think the switch would prevent the AI from escaping but that is a tool that could be used to discourage the AI from killing 100% of humanity. It is less of a solution than a survival mechanism. It is like many off switches that get more extreme depending on the situation. 

First don’t build AGI not yet. If you’re going to at least incorporate an off switch. If it bypasses and escapes which it probably will. Shut down the GPU centers. If it gets a hold of a Bot Net and manages to replicate it’s self across the internet and crowdsource GP... (read more)

1Xor
Also a coordinated precision attack on the power grid just seems like a great option, could you explain some ways that an AI can continue if there is hardly any power left. Like I said before places with renewable energy and lots of GPU like Greenland would probably have to get bombed. It wouldn’t destroy the AI but it would put it into a state of hibernation as it can’t run any processing without electricity. Then as this would really screw us up as well, we could slowly rebuild and burn all hard drives and GPU’s as we go. This seems like the only way for us to get a second chance. 
Xor51

I have been surprised by how extreme the predicted probability is that AGI will end up making the decision to eradicate all life on earth. I think Eliezer said something along the lines of “most optima don’t include room for human life.” This is obviously something that has been well worked out and understood by the Less Wrong community it just isn’t very intuitive for me. Any advice on where I can start reading. 

Some back ground on my general AI knowledge. I took Andrew Ng’s Coursera course on machine learning. So I have some basic understanding of n... (read more)

2Jonathan Claybrough
First a quick response on your dead man switch proposal : I'd generally say I support something in that direction. You can find existing literature considering the subject and expanding in different directions in the "multi level boxing" paper by Alexey Turchin https://philpapers.org/rec/TURCTT , I think you'll find it interesting considering your proposal and it might give a better idea of what the state of the art is on proposals (though we don't have any implementation afaik) Back to "why are the predicted probabilities so extreme that for most objectives, the optimal resolution ends with humans dead or worse". I suggest considering a few simple objectives we could give ai (that it should maximise) and what happens, and over trials you see that it's pretty hard to specify anything which actually keeps humans alive in some good shape, and that even when we can sorta do that, it might not be robust or trainable.  For example, what happens if you ask an ASI to maximize a company's profit ? To maximize human smiles? To maximize law enforcement ? Most of these things don't actually require humans, so to maximize, you should use the atoms human are made of in order to fulfill your maximization goal.  What happens if you ask an ASI to maximize number of human lives ? (probably poor conditions). What happens if you ask it to maximize hedonistic pleasure ? (probably value lock in, plus a world which we don't actually endorse, and may contain astronomical suffering too, it's not like that was specified out was it?).  So it seems maximising agents with simple utility functions (over few variables) mostly end up with dead humans or worse. So it seems approaches which ask for much less, eg. doing an agi that just tries to secure the world from existential risk (a pivotal act) and solve some basic problems (like dying) then gives us time for a long reflection to actually decide what future we want, and be corrigible so it lets us do that, seems safer and more approachable. 
6Charlie Steiner
The main risk (IMO) is not from systems that don't care about the real world "suddenly becoming aware," but from people deliberately building AI that makes clever plans to affect the real world, and then that AI turning out to want bad things (sort of like a malicious genie "misinterpreting" your wishes). If you could safely build an AI that does clever things in the real world, that would be valuable and cool, so plenty of people want to try. (Mesaoptimizers are sorta vaguely like "suddenly becoming aware," and can lead to AIs that want unusual bad things, but the arguments that connect them to risk are strongest when you're already building an AI that - wait for it - makes clever plans to affect the real world.) Okay, now why won't a dead-man switch work? Suppose you were being held captive inside a cage by a race of aliens about as smart as a golden retriever, and these aliens, as a security measure, have decided that they'll blow up the biosphere if they see you walking around outside of your cage. So they've put video cameras around where you're being held, and there's a staff that monitors those cameras and they have a big red button that's connected to a bunch of cobalt bombs. So you'd better not leave the cage or they'll blow everything up. Except these golden retriever aliens come to you every day and ask you for help researching new technology, and to write essays for them, and to help them gather evidence for court cases, and to summarize their search results, and they give you a laptop with an internet connection. Now, use your imagination. Try to really put yourself in the shoes of someone captured by golden retriever aliens, but given internet access and regularly asked for advice by the aliens. How would you start trying to escape the aliens?
Xor50

I am excited to see this sort of content here. I am currently finishing up my senior year of high school and making plans for the summer. I have decided to focus much of my free time on learning, and rationality as well as filing out my knowledge base on math, physics, and writing. These will be a valuable set of skills for collage and the rest of my life. This summer I plan to build a course on learning (free stuff on youtube) first because I want to be rigorous in my understanding of learning and teaching ensures that. Second I am looking forward to the ... (read more)

Xor10

Introduction:
I just came over from Lex Fridman’s podcast which is great. My username Xor is a Boolean logic operator from ti-basic I love the way it sounds and am super excited since this is the first time I have ever been able to get it as a username. The operator means this if 1 is true and 0 is false then (1 xor 0) is a true statement, while (1 xor 1) is a false statement. It basically means that the statement is true only if a single parameter is true. 
Right now I am mainly curious on how people learn. The brain functions involved, chemicals, and studied tools. I have been enjoying that and am curios if it has discussed on here as the quality of content as well as discussions has been very impressive.