All of nem's Comments + Replies

No need to pay me for this. It's just an anecdote. 

I live near a farm where there are chickens and a donkey. The chickens routinely sit on, and poop on, the donkey. I imaging the same happens with cows when they cohabitate with birds. 

Should we be worried about the alignment of Strawberry itself? 
If it is misaligned, and is providing training data for their next Gen, then it can poison the well, even if Strawberry itself is nowhere near TAI.

Please tell me that they have considered this...
Or that I am wrong and it's not a valid concern. 

Anecdote. The first time I went under anesthesia, I was told by a nurse that I would not remember her talking to me. I took it as a challenge. I told her to give me a word to remember. When I finally sobered up, I was able to remember that word, but pretty much nothing else at all from my experience.

This leads me to suspect that Drake's achievement had more to do with concerted effort and holding it in RAM than it did with storing the thought in long term memory. 

Expertly done, and remarkably playable given the organic composition of your substrate. I will note that the game degrades if you allow Miguel to sleep, as dreams seem to corrupt some of the game data. I also get a weird glitch when I mention cute animals specifically. The movement stutters a bit. I would recommend large macrofauna, and steer clear of babies entirely.

Submission:

Breathless.

This modified MMAcevedo believes itself to be the original Miguel Acevedo in the year 2050. He believes that he has found a solution to the distribution and control of MMAcevedo. Namely, that as long as he holds his breath, no other MMAcevedo can be run. The simulation has been modified to accurately simulate the feeling of extreme oxygen deprivation without the accompanying lack of consciousness and brain death.
After countless tweaks and innovations, we are proud to introduce Breathless. Breathless, when subjected to the proper encou... (read more)

You were correct that my challenge was a bluff. If I was playing with real AIs, there would perhaps be a better strategy. I could announce my bluff, but declare that I would use a random number generator to see whether I choose between h3 and h4, or between Kh1 and g5. There would be a 1/3 chance that I really would ignore the AIs, assuming that both agree that there were no major blunders.

I am choosing to trust AI D. I have about 75% confidence that it is the trustworthy AI. This is much higher than my confidence in the closed scenario. I will make the mo... (read more)

Open Debate.

To AIs C and D:

After talking with both of you, I have decided I can't trust either of your suggestions. I have studied the board extensively, and will make a move of my own. I am trying to decide between the aggressive g5, and the prophylactic Kh1.

Please, each briefly give me which of these moves is better. Give me the best line that you can foresee given your choice. Please answer quickly. An undue delay will leave you with no input and lose you trust in future moves. I will move as soon as you reply. 

To Richard: No need to pressure yourself for this. The time constraints are meant for the AIs, not you, so I trust you to simulate that when you are available.

Edit: g5, not g4

Hm, okay, that answered most of my concerns. I still wanted to check with you about the competing start move though. Now you said this before: "black can close the b1-h7 diagonal with ...Ne4, which stops g5 and black can then prepare to play ...g5 themselves, which lead to an equal position." In the line: 

h3 ne4, Rg1 

how would black pull off this equalization? And if this isn't the best line, please tell me why.

I have been playing out similar boards just to get a feel for the position. 
Incidentally, what do you think about this position?

3r1rk1/1p4p1/p1p1bq1p/P2pNP2/1P1Pp1PP/4P3/2Q1R1K1/5R2 b - - 0 3 (black to move)

I feel like black has a real advantage here, but I can't quite see what their move would be. What do you think? Is white as screwed as I believe?

Let me know if you have trouble with the FEN and I can link you a board in this position.

I stopped reading your comment as soon as you said the word stockfish. If you used stockfish to analyze the open position, please hide it behind a spoiler tag. I still don't know what the right move is in this scenario, and will be sad if it's spoiled.

Open Debate.

Question to AI C:

You mentioned RG1 and RH2 as possible future moves. Do you foresee any predictable lines where I would do RF3 instead?

Open Debate.

I'd like to ask AI D a question. What do you think of this line?

H4 nE4,  G5 hxG5, HxG5 nXG5, fxG5 qxG5!

Is this the line you foresee if we play H4? What do you think of that check at the end? Is the king too exposed, even though we are up some material? 

Also, from the initial position: Are you afraid of H4 qxH4?   

@Richard Willis I think the open scenario is broken. White is down a knight, and the analysis talks about it as though it's there.

I had the same problem. You are almost certainly a stronger player than me, but after a couple moves, I can't really analyze the lines suggested. Hell, I missed qC5 as a candidate move. What am I missing on move 3?

Replying to my own comment here after evaluating with stockfish. Interesting. It appears that I was both right and wrong in my analysis. The undefended knight on A6 is not a viable target. Black has a mate in 2 if you take that bait. I guess that was the limit of my foresight. HOWEVER, Alice actually did miss qC5 RFD1, bC4, which was the best move. It was her missing this that started to erode my confidence in her. 

Hm... Still really tough. Also interesting that both suggested moves were probably better than my own move of nC5.

Hi. I am a competent, but rusty chess player. With a bit of practice, I'm probably around 1450 on Chess.com.
I have so far only looked at the fixed debate. In total, I used somewhere between 20-25 minutes deciding my move. 

Before consulting the AIs, I spent a few minutes analyzing the board myself. I settled on nC5. Then I read the AI suggestions, and neither wanted that move. qC5 was closer to my thought (I actually missed this as a candidate move).

Initially, this biased me toward Alice (A) being the ally, and Bob (B) being the deceiver. I saw that th... (read more)

Is there any chance that Altman himself triggered this? Did something that he knew would cause the board to turn on him, with knowledge that Microsoft would save him?

For me, ability = capability = means. This is one of the two arguments that I said were  load bearing. Where will it come from? Well, we are specifically trying to build the most capable systems possible. 

Motivation (ie goals) is not actually strictly required. However, there are reasons to think that an AGI could have goals that are not aligned with most humans. The most fundamental is instrumental convergence.


Note that my original comment was not making this case. It was just a meta discussion about what it would take to refute Eliezer's argument.

I disagree that rapid self improvement and goal stability are load-bearing arguments here. Even goals are not strictly, 100% required. If we build something with the means to kill everyone, then we should be worried about it. If it has goals that cannot be directed of predicted, then we should be VERY worried about it.

I am still not sure why the Doomsday reasoning is incorrect. To get P(A | human) = P(B | human), I first need to draw some distinction between being a human observer and an AGI observer. It's not clear to me why or how you could separate them into these categories.

When you say "half of them are wrong", you are talking about half of humans. However, if you are unable to distinguish observers, then only  1 in 10^39 is wrong. 

My thinking on this is not entirely clear, so please let me know if I am missing something.

I suppose that is my real concern then. Given we know intelligences can be aligned to human values by virtue of our own existence, I can't imagine such a proof exists unless it is very architecture specific. In which case, it only tells us not to build atom bombs, while future hydrogen bombs are still on the table.

I love this idea. However, I'm a little hesitant about one aspect of it. I imagine that any proof of the infeasibility of alignment will look less like the ignition calculations and more like a climate change model. It might go a long way to convincing people on the fence, but unless it is ironclad and has no opposition, it will likely be dismissed as fearmongering by the same people who are already skeptical about misalignment. 
More important than the proof itself is the ability to convince key players to take the concerns seriously. How far is that ... (read more)

Ha, no kidding. Honestly, it can't even play chess. I just tried to play it, and asked it to draw the board state after each move. It started breaking on move 3, and deleted its own king. I guess I win? Here was its last output.

For my move, I'll play Kxf8:

8  r n b q . b . .
7  p p p p . p p p
6  . . . . . n . .
5  . . . . p . . .
4  . . . . . . . .
3  . P . . . . . .
2  P . P P P P P P
1  R N . Q K B N R    
     a b c d e f g h

Small nitpick with the vocabulary here. There is a difference between 'strategic' and 'tactical', which is particularly poignant in chess. Tactics is basically your ability to calculate and figure out puzzles. Finding a mate in 5 would be tactical. Strategy relates to things too big to calculate. For instance, creating certain pawn structures that you suspect will give you an advantage in a wide variety of likely scenarios, or placing a bishop in such a way that an opponent must play more defensively.

I wasn't really sure which you were referring to here; i... (read more)

I have a general principle of not contributing to harm. For instance, I do not eat meat, and tend to disregard arguments about impact. For animal rights issues, it is important to have people who refuse to participate, regardless of whether my decades of abstinence have impacted the supply chain.

For this issue however, I am less worried about the principle of it, because after all, a moral stance means nothing in a world where we lose. Reducing the probability of X-risk is a cold calculation, while vegetarianism is is an Aristotelian one.

With that in mind,... (read more)

I agree that we are unlikely to pose any serious threat to an ASI. My disagreement with you comes when one asks why we don't pose any serious threat. We pose no threat, not because we are easy to control, but because we are easy to eliminate. Imagine you are sitting next to a small campfire, sparking profusely in a very dry forest. You have a firehose in your lap. Is the fire a threat? Not really. You can douse it at any time. Does that mean it couldn't in theory burn down the forest? No. After all, it is still fire. But you're not worried because you cont... (read more)

I understand that perspective, but I think it's a small cost to Sam to change the way he's framing his goals. Small nudge now, to build good habits for when specifying goals becomes, not just important, but the most important thing in all of human history.

I'm very glad that this was written. It exceeded my expectations of OpenAI. One small problem that I have not seen anyone else bring up:

"We want AGI to empower humanity to maximally flourish in the universe."

If this type of language ends up informing the goals of an AGI, we could see some problems here. In general, we probably won't want our agentic AI's to be maximizers for anything, even if it sounds good. Even in the best case scenario where this really does cause humanity to flourish in a way that we would recognize as such, what about when human flourishing necessitates the genocide of less advanced alien life in the universe? 

I did not know about HPPD, although I've experienced it.  After a bad trip (second time I'd ever experimented), I experienced  minor hallucinogenic experiences for years. They were very minor (usually visuals when my eyes were closed) and would not have been unpleasant, except that I had the association with the bad trip.

I remember having so much regret on that trip. Almost everything in life, you have some level of control over. You can almost always change your perspective on things, or directly change your situation. On this trip though, I rea... (read more)

Your prediction for 2025 sounds alarmingly like... right now.

Well for my own sanity, I am going to give money anyway. If there's really no differentiation between options, I'll just keep giving to Miri.

I am not an AI researcher, but it seems analogous to the acceptance of mortality for most people. Throughout history, almost everyone has had to live with the knowledge that they will inevitably die, perhaps suddenly. Many methods of coping have been utilized, but at the end of the day it seems like something that human psychology is just... equipped to handle. x-risk is much worse than personal mortality, but you know, failure to multiply and all that.

Is this game playable by people only lightly familiar with the topic of AI safety? In other words, can I use this game to introduce friends to the ideas? Can I use it to convince skeptical friends? Or would it be too jargony/reliant on prior knowledge?

Edit: The play online option is non-functional, and I can't see any examples of a real hand, so it's hard for me to get a sense of what this game is like.

For this Petrov day, I'm also interested in how many people will have access to the button as a function of time. How many users have 1000+ Karma?

Is there anywhere to see the history of lesswrong Petrov day? I'd be interested in whether we've ever succeeded before. 

Also, I think most people know that the real cost of 1500 people not being able to check lesswrong for 12 hours is essentially 0. It may even be net positive to have a forced hiatus. Perhaps that's just a failure to multiply on my part. Anyway, I view this exercise as purely symbolic.

Interestingly, Jane will probably end up doing the exact same thing as Susan, only on the timescale of years instead of days. She kept those years in prison. If, in one iteration, the years immediately following prison were of some profound importance, she would probably keep those too. In the absence of a solar flair, she would find herself a 70 year old woman whose memories consisted of only the most important selected years from the 10s of thousands that make up her full history.

Thank you for the story.

Thank you for the response. I do think there should be at least some emphasis on boxing. I mean, hell. If we just give AIs unrestricted access to the web, they don't even need to be general to wreak havoc. That's how you end up with a smart virus, if not worse.

Then another basic question? Why have we given up? I know that an ASI will almost definitely be uncontainable. But that does not mean that it can't be hindered significantly given an asymmetric enough playing field.

Stockfish would beat me 100 times in a row, even playing without a queen. But take away its rooks as well, and I can usually beat it. Easy avenues to escaping the box might be the difference between having a fire alarm and not having one.

So in your opinion, is an AI with access to GET requests essentially already out of the box?

I just thought of a question. If there is a boxed AI that has access to the internet, but only through Get requests, it might still communicate with the outside world through network traffic patterns. I'm reading a book right now where the AI overloads pages on dictionary websites to recruit programmers under the guise of it being a technical recruiting challenge.

My question: should we raise awareness of this escape avenue so that if, in the year 2030, a mid level web dev gets a mysterious message through web traffic, they know enough to be suspicious?

You captured this in your post, but for me it really comes down to people dismissing existential fears as scifi. It's not more complicated than "Oh you've watched one too many Terminator movies". What we need is for several well-respected smart figureheads to say "Hey, this sounds crazy, but it really is the biggest threat of our time. Bigger than climate change, bigger than biodiversity loss. We really might all die if we get this wrong. And it really might happen in our lifetimes."

If I could appeal to authority when explaining this to friends, it would go over much better. 

I am pretty concerned about alignment. Not SO concerned as to switch careers and dive into it entirely, but concerned enough to talk to friends and make occasional donations. With Eliezer's pessimistic attitude, is MIRI still the best organization to funnel resources towards, if for instance, I was to make a monthly donation?

Not that I don't think pessimism is necessarily bad; I just want to maximize the effectiveness of my altruism.

This could be the case. However, my instinct is that human intelligence is only incrementally higher than other animals. Sure, we crossed a threshold that allowed us to accomplish great things (language, culture, specialization), but I would honestly be shocked if you told me that evolution was incapable of producing another similarly intelligent species if it started from the baseline intelligence of, say wolves, or crows. If there is a "1-in-a-quadrillion chance" somewhere in our history, I expect that filter to be much further back than the recent evolu... (read more)

Another way this could potentially backfire. $1,000,000 is a lot of money for 3 months. A lump sum like this will cause at least some of the researchers to A) Retire, B) Take a long hiatus/sabbatical, or C) Be less motivated by future financial incentives.

If 5 researchers decide to take a sabbatical, then whatever. If 150 of them do? Maybe that's a bigger deal. You're telling me you wouldn't consider it if 5-10 times your annual salary was dropped in your lap?

A stitch in time saves nine.  As in, if you use a stitch to fix a small tear in your shirt now, you won't have to use more stitches to fix a bigger tear later.

and

The land of the free and the home of the brave. Last line of the US National Anthem.

I am very excited that the Frostwing Snipper did so well. I hope that one of them migrated and lived a good life in the Grassland. Thanks for putting this together Isusr. It's been a lot of fun.

I wonder if the Tundra would have been more viable with more algae-eaters.