All of wunan's Comments + Replies

wunan52

Did you and GPT4 only output the moves, or did you also output the board state after each turn?

2Rafael Harth
Moves only
wunan10

Unfortunately without speaker labels the YouTube transcript is less useful unless you're listening while reading.

2Shmi
Double speed makes it usable
2Shmi
click on 3 dots (if on a browser) and there is a transcript option in the pop-up menu.
wunan103

Another similar result was that AlphaFold was trained on its own high-confidence predictions for protein sequences with unknown structures:

The AlphaFold architecture is able to train to high accuracy using only supervised learning on PDB data, but we are able to enhance accuracy (Fig. 4a) using an approach similar to noisy student self-distillation35. In this procedure, we use a trained network to predict the structure of around 350,000 diverse sequences from Uniclust3036 and make a new dataset of predicted structures filtered to a high-confidence subset.

... (read more)
wunan*40

I'm also dealing with chronic illness and can relate to everything you listed. I've been thinking that a discord server specifically for people with chronic illness in the rationality community might be helpful to make it easier for us to share notes and help each other. There are different discord servers for various conditions unaffiliated with the rationality community, but they tend to not have great epistemic standards and generally have a different approach than what I'm looking for. Do you have any interest in a discord server?

2riceissa
Agreed on the epistemic standards of random health groups, and yeah, I'd be interested in a Discord server. I am aware of this Facebook group, if you use Facebook, though it's not very active.
wunan30

By forward do you mean sooner (shorter timelines) or later (longer, slower timelines)?

4Shmi
"forward in time" means "later".
wunan150

I tried giving this to GPT-3 and at first it would only give the tautological "pawns become more powerful" example, then I expanded the prompt to explain why that is not a valid answer, and it gave a much better response.

I believe this response is the same as your fourth bullet point example of a good answer.

 

 

Here's the prompt in copy/pastable format for anyone who wants to try playing with it:

 

Consider a new variant of chess, in which each pawn can move up to six squares forward on its first move, instead of being limited to one or two squ

... (read more)
4Archimedes
From the new Bing which is reportedly (some variant of) GPT-4, here are the first responses I got under each conversational mode using your prompt: Conversational style: Balanced The new variant of chess would make the king’s pawn opening less effective. This is because the king’s pawn opening is a common and aggressive first move that stakes a claim to the center and frees up the queen and king bishop. However, with pawns able to move up to six squares forward on their first move, the opponent could easily counter this opening by advancing their own pawn to block or capture the king’s pawn. This would disrupt the center control and expose the king to more threats. Therefore, players would have to be more careful about their first moves and consider other openings that are less vulnerable to pawn attacks. Conversational style: Creative This variant of chess would make it much easier for pawns to reach the opposite side of the board and promote into a more powerful piece. This would increase the importance of pawn structure and pawn breaks, as well as create more dynamic and complex positions. A player who can create a passed pawn (a pawn that has no enemy pawns on its file or adjacent files) would have a significant advantage, as they could quickly push it to promotion with minimal resistance. Conversely, a player who has a backward pawn (a pawn that cannot advance because it is blocked by an enemy pawn on an adjacent file) would be at a serious disadvantage, as they would be vulnerable to attacks and unable to counterattack. Therefore, this variant of chess would reward aggressive and creative play with pawns, while punishing passive and defensive play. Conversational style: Precise In this new variant of chess where pawns can move up to six squares forward on their first move, the game balance and strategy would change significantly. One potential impact is that the opening phase of the game could become much more aggressive and fast-paced. With pawns able t
1gjm
I tried a different sort of prompt. The prompt ends, and GPT-3's output begins, immediately after "MAGNUS:". I'm not sure what I think of "Magnus"'s response. It's not clear to me that PGUS chess would make controlling the centre easier; it seems like it certainly would make pawn-storms easier to carry out; I'm not sure whether it would make it easier to sacrifice a pawn for faster piece development (PGUS would make it possible for the pawn to get more quickly out of the way of the pieces you want to develop, I guess? and maybe in the case where it gets taken immediately, whatever captures would be less likely to be in the way?) But at least one of the claims being made -- the one about pawn storms -- seems likely correct, important, and not 100% trivial. A second response to the same prompt: "A lot more dynamic" is somewhat plausible (though it's in tension with the "easier to set up pawn chains" thing). The explanation for why feels a bit off -- not wrong but too superficial. "More aggressive advances" is surely true but maybe trivial. "More interesting and complex strategic decisions" doesn't really say much (and it seems perfectly possible to me that PGUS would make the game tactically more exciting but if anything simpler strategically, though you'd need to be a much stronger player than me before I'd believe your predictions on that front without playing lots of PGUS games). This one has a bit more of the "throw superficially plausible things at the wall and hope some of them sound good" feel that's characteristic of a lot of GPT writing. If a substantially smarter GPT-4 comes out in the near future, I think it's likely that it will give more impressive answers to questions like these. I think I'd be on the other side of the first half of Omid's bet from Omid. He's probably right about the second half ... except that I wouldn't be astonished if it's possible to make an AlphaZero-ish thing that, given a modestly different set of rules for a game it's learne
wunan25

Agreed that it would be insanely impressive. It would probably convince me that a fast takeoff is very likely coming within the next 5 years. Yet I can't really say I'm more than 90% confident that GPT-4 won't be able to do it. Maybe 95%.

wunan50

I'm not sure about that. See page 8 of the LamDA paper where they gave it access to a "toolset" including things like a calculator. I wouldn't be surprised if they gave GPT-4 access to similar tools including a way to access the current date.

2dugannash
I highly doubt LamDA could tell you today's date unless they programmed that as an explicit feature. If GPT-4 can tell you today's date, but only because they added a kludgey return date.today() feature, I'd find that to be totally uninteresting.  In any case, we shall see!
wunan*5738

There are probably less impressive things than this that it won't be able to do, but here's one prediction in which I am about 90% confident:

If you invent or find a board game of similar complexity to chess that GPT-4 has never seen before and explain the rules using only text (and, if GPT-4 is multimodal, also images), GPT-4 will not be able to perform as well at the game as an average human who has never seen the game before and is learning it for the first time in the same way. I say "perform as well as" rather than "beat" because, depending on the spec

... (read more)
4baturinsky
GPT4 fails even in extremely simple games. "There are four pebbles on the 2 x 2 grid. Each turn, one player can remove one pebble, or two nearby pebbles. If these were the last pebbles, he wins. If two players play, who has the winning strategy?" 🤖"In this game, the first player has a winning strategy. The strategy is to always leave an odd number of pebbles for the opponent to take on their turn. This can be achieved by removing one pebble on the first turn, and then mirroring the opponent's moves throughout the game. Eventually, the first player will be left with one pebble, which they can remove to win the game. " I guess it just have failed to understand the rules.
2Adele Lopez
Nisan and I had GPT-4 play Wordle (which it claimed it didn't recognize, even though it came out before the cutoff date), and it was able to understand and play it, but performed fairly poorly, focusing mostly on the green and yellow squares, and not seeming to use the information from the black squares. It might be possible to have it do better with better prompting, this was just a casual attempt to see if it could do it.
4p.b.
How many humans could learn and play a game like chess solely via verbal interactions? Edit: That is to say, to me this would be insanely impressive. Even a model that could learn any game of chess-like complex rules from pictures and text without that being specifically engineered in ... I'd expect that to be AGI. 
Omid122

Prompt: "Consider a new variant of chess, in which each pawn can move up to six squares forward on its first move, instead of being limited to one or two squares. All other rules remain intact. Explain how game balance and strategy is changed with this new variant of chess."

Successful responses might include:

  • The number of moves available in the early game increases, improving white's edge over black.
  • Drawing your knight out from 1st to 3rd rank becomes more dangerous, as black can immediately threaten the knight by moving a pawn forward three spaces. The kn
... (read more)
3James_Miller
Games, of course, are extensively used to train AIs.  It could be that OpenAI has its programs generate, evaluate, and play games as part of its training for GPT-4.
6jacob_cannell
I'm about 70% on this, with most of that 30% coming from that chance that GPT-4 is multimodal and was curriculum trained vaguely more like humans on a dataset that includes some games in addition to specialized sub datasets created specifically to better develop general in-context/meta learning.
wunan10

I ran into another similar example in the last section of this post.

wunan92

As another example, I remember somebody (maybe EY or Gwern?) suggest that prompting people with "how would somebody smarter than you solve this problem?" can actually be effective at soliciting better solutions than just asking them to solve it directly.

I don't remember where I saw this so if anybody has a link feel free to share.

1wunan
I ran into another similar example in the last section of this post.
5gwern
https://www.gwern.net/on-really-trying
wunan60

This sounds similar to John Sarno -- are you familiar with it and do you know whether the approaches are substantially different?

2Valentine
Sorry, no, I have no familiarity with John Sarno. This is the first time I've heard of him.
wunan10

This is deeply perverse, but three "risk factors" over which you may have direct, immediate control are smoking (possibly of marijuana as well as tobacco) your level of vaccination, and your amount of physical activity. It is not medical advice to take up smoking, avoid vaccination/booster shots, or sit around all day, in order to qualify for Paxlovid should you get sick. But I do worry that some people might go to such lengths due to this policy.


I can't imagine why anybody would take up smoking or stop exercising in order to get Paxlovid instead of just f... (read more)

4DirectedEvolution
You may be right, but some people are very averse to lying.
wunan30

I'm not sure about the timing of when the edits in your post were made, but if you want feedback about your planned contact with Demis Hassabis I think you should make a new post about it -- most people who would comment on it may have missed it because they only saw the original unedited post about Tao, which had already received feedback.

I also think that, for the same reason that you chose to let someone else contact Tao instead of you, it may be better to let someone else contact Hassabis (or find someone else to contact him).

1P.
The email to Demis has been there since the beginning, I even received feedback on it. I think I will send it next week, but will also try to get to him through some DeepMind employee if that doesn’t work.
wunan130

Sci-hub lets you get around paywalls for pretty much all academic articles.

7supposedlyfun
this is your second great response to a question on my shortform!
wunan20

Did your previous experiences with VR involve something where your in-game movement wasn't one-to-one with your actual movement (e.g. where you could move your character by pushing forward on an analog stick, rather than by walking)? It's pretty rare for VR games with one-to-one movement (like Beat Saber and TotF) to cause motion sickness, so if your previous sickness was in a non-one-to-one game it may be worth giving VR another shot with a more comfortable game.

3Maxwell Peterson
Yeah, that makes sense. ToF itself didn’t make me sick - it was other games, like the nightmare PUBG-like VR shooter where you start by parachuting to a place on the ground, and the Walking Dead VR shooter. I would probably be fine if I restricted the games I played, but I wanted to play alllll the games, and returned my headset when I couldn’t.
wunan10

Thanks for writing! I don't see an actual answer to the question asked in the beginning -- "Given the ongoing history of continually increasing compute power, what is the maximum compute power that might be available to train ML models in the coming years?" Did I miss it?

wunan10

And would a doctor's note make any difference towards them allowing you to wear something like a versaflo or other PAPR?

wunan10

Does anybody know what is the best mask that they'll allow you to wear on an airplane? Has anyone worn a P100 with the exhale valve covered, or do they not allow that?

4Dorikka
i have worn this on an airplane with a surgical mask over the exhale valve   https://www.amazon.com/GVS-SPR457-Elipse-Respirator-Medium/dp/B013SIIBFQ/ref=sxts_rp_s1_0?cv_ct_cx=gvs+elipse+p100&gclid=CjwKCAiAzrWOBhBjEiwAq85QZyLzBYwxwOZcFGAMZbn4S6OVwxcQXnl1L6L9xWu1dQgzAFdKK8oxOhoCqo0QAvD_BwE&hvadid=409918312011&hvdev=c&hvlocphy=9014238&hvnetw=g&hvqmt=e&hvrand=6118992613772945421&hvtargid=kwd-267611734495&hydadcr=20509_11115072&keywords=gvs+elipse+p100&pd_rd_i=B013SIIBFQ&pd_rd_r=03b1c070-7191-4557-b0bb-79b3930e0ac8&pd_rd_w=JYxDi&pd_rd_wg=uF7dV&pf_rd_p=53ef7f8b-c1cb-4ebe-8d67-39f3ca0ebaf7&pf_rd_r=PE758N16XDW0FMZ1MZXD&psc=1&qid=1640904683&sr=1-1-5e1b2986-06e6-4004-a85e-73bfa3ee44fe
1wunan
And would a doctor's note make any difference towards them allowing you to wear something like a versaflo or other PAPR?
wunan10

hopefully realize it's a bad idea have a morality that allows this

 

To expand on this: https://www.nickbostrom.com/papers/unilateralist.pdf

wunan40

What do you mean by "immune erosion"? Is this different than "immune evasion" and "immune escape"? I can't find any explanation on google -- is this a standard term?

1cistrane
Existing mRNA vaccines theoretically provide antibodies to about 42 epitopes of the original spike protein. So, they are very effective against the original strain as any one of these antibodies would be neutralizing. Let's say for example that only 10 out of 42 are neutralizing for omicron. That means that many people will be unlucky and not produce enough of the right antibodies to stop the omicron at first, thus their protection has eroded. But some will be lucky enough to produce the necessary amount of the right antibodies. So the protection was not completely bypassed.
2Caridorc Tergilti
Immune erosion is used to make people understand that immune escape is only partial and not total
1bww
Per my understanding "immune evasion" and "immune escape" imply a binary, i.e. previous immunity has no effect. "Immune erosion" is a more intuitive way of describing how this variant likely interacts with protection from previous infection or vaccination, with protection against infection, hospitalization, death "eroding" to a degree but not being completely negated. I believe I first saw it in this twitter thread: https://twitter.com/kakape/status/1428657737190322184?s=20
wunan20

What is meant by "immune erosive"? Is this different than "immune evasive"? I can't find any explanation on google -- is this a standard term?

-4mwacksen
It's the terminology you use to signal that you believe the vaccine is safe and effective™ and therefore cannot be fully "evaded".
wunan30

If it's a normal distribution, what's the standard deviation?

wunan10

For software development, rewriting the code from scratch is typically a bad idea. It may be helpful to see how well the arguments in that article apply to your domain.

4Shmi
Rewriting is hard, refactoring is easy and gets you 80% toward the goal that pushes one to rewrite. Also can be done incrementally.
wunanΩ8140

Context for anyone who's not aware:

Nerd sniping is a slang term that describes a particularly interesting problem that is presented to a nerd, often a physicist, tech geek or mathematician. The nerd stops all activity to devote attention to solving the problem, often at his or her own peril

Here's the xkcd comic which coined the term.

2adamShimi
Thanks, I sometimes forget not everyone knows the term. :)
wunan70

If MIRI hasn't already, it seems to me like it'd be a good idea to try reaching out. It also seems worth being at least a little bit strategic about it as opposed to, say, a cold email. 

 

+1 especially to this -- surely MIRI or a similar x-risk org could attain a warm introduction with potential top researchers through their network from someone who is willing to vouch for them.

wunan*340

On one hand, meditation -- when done without all the baggage, hypothetically -- seems like a useful tool. On the other hand, it simply invites all that baggage, because that is in the books, in the practicing communities, etc.

 

I think meditation should be treated similarly to psychedelics -- even for meditators who don't think of it in terms of anything supernatural, it can still have very large and unpredictable effects on the mind. The more extreme the style of meditation (e.g. silent retreats), the more likely this sort of thing is.

Any subgroups he... (read more)

4Gunnar_Zarncke
I have pointed out the risks of meditation and meditation-like practices before. The last time was on the Shoulder Advisors which does seem to fall on the boundary. I have experience with meditation and have been to extended silent meditation retreats with only positive results. Nonetheless, bad trips are possible - esp. without a supportive teacher and/or community.  But I wouldn't make a norm against groups fostering meditation. Meditation depends on groups for support (though the same might be said about psychedelics). Meditation is also a known way to gain high levels of introspective awareness and to have many mental health benefits (many posts about that on LW I'm too lazy to find). The group norm about these things should be to require oversight by a Living Tradition of Knowledge in the relevant area (for meditation e.g. an established - maybe even Buddhist - meditation school).
2Kenny
Psychedelics, woo, and meditation are very separate stuff. They are often used in conjunction with each other due to popularity and the context some of these things are discussed along with each other. Buddhism has incorporated meditation into its woo while other religions have mostly focused on group based services in terms of talking about their woos. I like how some commenters have grouped psychedelics and meditation separate of the woo stuff, but it was a bit surprising to me to see Eliezer dismissing psychedelics along with woo in the same statements. He probably hasn't taken psychedelics before. Meditation is quite different as in it's more of a state of mind as opposed to an altered mentality. With psychedelics there is a clear distinction between when you are tripping and when you aren't tripping. With meditation, it's not so clear when you are meditating and when you aren't. Woo is just putting certain ideas into words, which has nothing to do with different mindset/mentalities.
wunan50

Possible small correction: GPT-2 to GPT-3 was 16 months, not 6. The GPT-2 paper was published in February 2019 and the GPT-3 paper was published in June 2020.

3Quintin Pope
Whoops! Corrected.
wunan30

I can't tell from the descriptions, but it seems like these programs have been run before -- is that right? Are there any reviews or other writeups about participants' experiences anywhere?

1Yi-Yang
Hi wunan, yes we run the programs monthly. Unfortunately, we don't have any reviews/testimonies written up for the public, but here's a broad feel of our program.
wunan20

That would make a good monthly open thread.

wunan20

If compute is the main bottleneck to AI progress, then one goalpost to watch for is when AI is able to significantly increase the pace of chip design and manufacturing. After writing the above, I searched for work being done in this area and found this article. If these approaches can actually speed up certain steps in this process from taking weeks to just taking a few days, will that increase the pace of Moore's law? Or is Moore's law mainly bottlenecked by problems that will be particularly hard to apply AI to?

4steven0461
There was some previous discussion here.
wunan10

Do you have some examples? I've noticed that rationalists tend to ascribe good faith to outside criticisms too often, to the extent that obviously bad-faith criticisms are treated as invitations for discussions. For example, there was an article about SSC in the New Yorker that came out after Scott deleted SSC but before the NYT article. Many rationalists failed to recognize the New Yorker article as a hit piece which I believe it clearly was, even more clearly now that the NYT article has come out.

1Aryeh Englander
I am reluctant to mention specific examples, partly because maybe I've misunderstood and partly because I hate being at all confrontational. But regardless, I have definitely seen this outside the rationalist community, and I have definitely noticed myself doing this. Usually I only do it in my head though, where I feel upset when it's coming from outside my group but if someone inside the group says it then I'll mentally nod along.
wunan10

Yeah, my main takeaway from that question was that a change in the slope of of the abilities graph was what would convince him of an imminent fast takeoff. Presumably the x axis of the graph is either time (i.e. the date) or compute, but I'm not sure what he'd put on the Y axis and there wasn't enough time to ask a followup question.

wunan30

Even without having a higher IQ than a peak human, an AGI that merely ran 1000x faster would be transformative.

wunan70

Of the bottlenecks I listed above, I am going to mostly ignore talent. IMO, talented people aren't the bottleneck right now, and the other problems we have are more interesting.

 

Can you clarify what you mean by this? I see two main possibilities for what you might mean:

  • There are many talented people who want to work on AI alignment, but are doing something else instead.
  • There are many talented people working on AI alignment, but they're not very productive.

If you mean the first one, I think it would be worth it to survey people who are interested in AI... (read more)

6simeon_c
I think that "There are many talented people who want to work on AI alignment, but are doing something else instead." is likely to be true. I met at least 2 talented people who tried to get into AI Safety but who weren't able to because open positions / internships were too scarce. One of them at least tried hard (i.e applied for many positions and couldn't find one (scarcity), despite the fact that he was one of the top french students in ML). If there was money / positions, I think that there are chances that he would work on AI alignment independently. Connor Leahy in one of his podcasts mentions something similar aswell. That's the impression I have.
2Charlie Steiner
Yeah, I mean the first. Good survey question ideas :)
wunan10

The movie Downsizing is about this.

1Maxwell Peterson
Thanks for that! I'd seen it mentioned in a tweet but we wondered about the veracity. Seeing another source is helpful.
2Dokler
I saw someone tweet a successful report of using LSD for this purpose. Edit: went back to search for it and there seem to be many people discussing this and quite a few have found it helpful (much moreso for smell than taste, but of course the two are closely related and I believe that loss of smell was a more common experience).
wunan10

What's the advantage of taking over an existing corporation rather than creating a new organization?

1jpgeiger
Technically, there's no difference in the long-term outcome, and if you had reason that creating a new company would be more likely to succeed, that would be a fine idea. I assume it would be easier to take over an existing company, because you'd gain access to their infrastructure, intellectual property, experienced employees, and existing customer base.  
wunan40

What are some examples of makers who gained wealth/influence/status by having a huge negative impact on the world?

The marketing company Salesforce was founded in Silicon Valley in '99, and has been hugely successful.  It's often ranked as one of the best companies in the U.S. to work for.  I went to one of their conferences recently, and the whole thing was a massive status display- they'd built an arcade with Salesforce-themed video games just for that one conference, and had a live performance by Gwen Stafani, among other things.

...But the marketing industry is one massive collective action problem. It consumes a vast amount of labor and resources, distort... (read more)

wunan00

What I mean is that they haven't really considered it. As I'm sure you're aware, your mind does not work like most people's. When most people consider the question of whether they'd like to die someday, they're not really thinking about it as if it were a real option. Even if they give detailed, logically coherent explanations for why they'd like to die someday, they haven't considered it in in near mode.

 

I am very confident of this -- once they have the option, they will not choose to die. Right now they see it as just an abstract philosophical conve... (read more)

1Willa
I wonder what the research area for "finding out whether people with strong beliefs in a religious afterlife of some kind change their minds near death regarding wanting to die vs wanting to go to that afterlife" is called? I do think you underestimate the strength of religious individual's convictions and the impact of that on their decisionmaking especially near death if they have significant faith in their religion's conception of an afterlife (assuming it has one). Still...staring imminent death in the face does spur many changes whenever an individual experiences that, so maybe that does shake things up...but, I'm not sure how much hope I place in that idea without seeing some research around the topic. I make this claim: Individuals with sufficiently strong religious beliefs in a religion that has an afterlife will more often than not prefer to die on the "mortal plane" and go to their faith's afterlife than continue living (even in good health and restored to youth, etc.) for [100 more years] [300 more years] [lots more years] [the rest of time]. I make that claim because of my experiences interacting with individuals who have very strong faith in their religious beliefs, including interacting with such individuals when they are near death. I need to figure out what kind of research people have already done that points in the direction of that claim and see what others have found, then I can pursue this inquiry further. I don't want to make a claim and forever rely on anecdote to support it!
wunan10

They'll almost definitely change their minds once we have good treatments for aging.

1Willa
I know some will, but that's too optimistic and ignores the preferences / experiences of a huge amount of people, because there are categories of people who prefer death over immortality for whom the aging process doesn't factor in to their choice on that matter. Especially people with strong faith in their religion's afterlife.
wunan40

Those graphs all show the percentage share of the different variants, but more important would be the actual growth rate. Is the delta variant growing, or is it just shrinking less quickly than the others?

2[anonymous]
Because there is no evidence that they mean any of that.
1Pablo Repetto
It is a direct response to a quotation from the article, so not really.
wunan30

Precommitment for removal and optionality for adding.

2ozziegooen
Thanks! I think precommitement is too narrow (I don't see dying as a precommitement). Optionality seems like a solid choice for adding. "Options" are a financial term, so something a bit more generic seems appropriate.
wunan10

There's a discord for Crypto+Rationalists you may be interested in if you're not already aware: https://discord.gg/3ZCxUt8qYw

wunan80

To any high schoolers reading this: If I could send just one of the items from the above list back to myself in high school, it would be "lift weights." Starting Strength is a good intro.

9lsusr
The recommendation to lift weights applies to boys and girls. If you are a woman, lifting weights won't make you bulky. It will make you hot.
wunan20

I have a potential category of questions that could fit on Metaculus and work as an "AGI fire alarm." The questions are of the format "After an AI system achieves task x, how many years will it take for world output to double?"

Load More