All of Taran's Comments + Replies

For the record, I personally found the way Said engaged to be annoying at points because I would have preferred to make the same complaint about the post with more tact so that the author might actually fix the issue within 24 hours instead of taking that long to figure out what it was. Since he had made the comment, it was often difficult to add mine (since the author was busy in another thread!). I don't really expect any decline in quality on LessWrong because I think the job Said was doing is ~fungible (though I hope everyone reading this comment could have guessed that I know that people are not).

As far as I know the cyclic weakness in KataGo (the top Go AI) was addressed fairly quickly. We don't know a weird trick to beating the current version. (altough adverserial training might turn up another weakness). The AIs are superhuman at Go. The fact that humans could beat them by going out of distribution doesn't seem relevant to me.

[discussion of Ryan's point is ongoing on the MIRI slack, but I have a response to this comment that doesn't weigh on that; other contributors likely disagree with me] The FAR work totally rocks. However, I don't think that 'humans can use other AIs to discover successful adversarial strategies that work only if they know they're playing against the AI' is cleanly an example of the AI not being superhuman in the usual sense of superhuman. You're changing what it means to be human to include the use of tools that are upstream of the AI (and it's not at all inevitable that humans will do this in every case), and changing the definition of superhuman downstream of that.  In the context of the analogy, this looks to me like it ~commits you to a Vitalik-esque defensive tech view. This is a view that I at least intend to reject, and that it doesn't feel especially important to kneel to in our framing (i.e. the definition of our central concept: superintelligence).

Conditionally on him continuing to play the opening, I would expect he has a refutation to that refutation, but no reason to use the counter-refutation in public games against the computer. On the other hand, he may not want to burn it on you either.

That’s fair, but I still think saying “Go has been unsolved” is importantly misleading: for one, it hasn’t been solved! Also, the specific cycle attack doesn’t work against other engines I think? In the paper their adversary doesn’t transfer very well to LeelaZero, for example. So it’s more one particular AI having issues, than a fact about Go itself. (EDIT: As Tony clarifies below, the cyclic attack seems to be a common failure mode amongst other top Go AIs as well, so I retract this paragraph -- at the very least, it seems to be a fact about ~all the top Go AIs!) EDIT: also, I think if you got arbitrary I/O access to a Magnus simulator, and then queried it millions of times in the course of doing AlphaZero style training to derive an adversarial example, I’d say it’s pretty borderline if it’s you beating beating him. Clearly there’s some level of engine skill where it’s no longer you playing!

To which one should reply: 'oh really, is it a draw or a win for white?'

I don't like it. "The problem of creating AI that is superhuman at chess" isn't encapsulated in the word "chess", so you shouldn't say you "solved chess" if what you mean is that you created an AI that is superhuman at chess. What it means for a game to be solved is widely-known and well-developed[0]. Using the exact same word, in extremely similar context, to mean something else seems unnecessarily confusing.  [0] See https://en.wikipedia.org/wiki/Solved_game

I agree that it wouldn't be valid as an absolute, or even as a strong claim. I'm not sure I agree that it is no evidence at all.

If "outside view" was a natural category that was useful to use, AND people generally had a shared understanding of what it meant, then it would slow things down unnecessarily to be more specific, at least often (sometimes even then you'd want to be more specific.) My original post cast doubt on both the naturalness/usefulness of the concept (not saying there's absolutely nothing tying the things in the Big Lists together, just saying that there isn't really any good evidence that whatever it is that ties them together is epistemically important) and the shared understanding (ho boy do different people seem to have different ideas of what it means and how it should be used and what evidential status it confers)

These statements seem awfully close to being unfalsifiable. The amount of research and development coming from twitter in the 5 years before the acquisition was already pretty much negligible, so there's no difference there. How long do we need to wait for lawsuits or loss of clients to cause observable consequences?

A non-obvious thing here: last year, Lightcone put a fair amount of effort into doing lots of interviews, orienting on the Leverage situation, and attempting to publish a blogpost that offered a pretty clear and comprehensive set of information. We were specifically thinking of this from the standpoint of "it doesn't seem like there are very good community justice institutions or practices around, beyond random discussions", and thinking maybe we could contribute something useful. And then, well, a lot of stuff came up and we didn't get the piece over the finish-line of publishing.  So I'm coming at this from the perspective, partly "how valuable would it have been to get that across the finish line?". And I see both this piece and the Zoe piece as representing the collective situation.  I also do just agree some of the claims in this piece (implicit and explicit) that many of the cult-looking-behaviors of leverage are red herrings and are reasonable things I want to defend.

My take was "It'd be quite weird for this post to show up in the Best of LessWrong books, but I think part of the point of the review is to reflect on things that had some kind of impact on your worldmodels, even if the posts aren't directly optimized for that." 

Interesting; that isn't something I knew about Tumblr. This is especially surprising given how often I see screenshots of Tumblr discussions shared on FB, like the post I responded to here. (I really don't like share-by-screenshot culture)

The reference class would be "wars to extend a country's territory permanently". As such there's an interest to have the value of the newly won territory as high as possible.  When waging war over a city, for both sides there are actions that can be taken to increase or decrease the amount of damage that the city takes.  In Ukraine, it seems that no party went out of its way to reduce the damage to cities. We know that from Amnesty trying to understand what happened and them finding that frequently Ukrainian army stationed itself inside the city and got shot at by the Russian army. This dynamic does explain that a part of the city is destroyed but it doesn't explain why 90% of Mariupol's residential buildings had been damaged or destroyed (Wikipedia numbers). The 90% sounds to me like this is more than just collateral damage but that someone made a conscious choice to destroy more of the city than they would need for purely military reasons.  One reason to do that might be propaganda reasons and to make the population fear you. Given that according to Russian propaganda Russia came to liberate the Russian minority, destroying a city with a large number of ethnic Russians makes little sense for that goal.  Often in war destruction is also done as a punishment. Russia punished the population in Chechnya for their local resistance in the Second Chechnyan War. It's unclear to me why the population of Mariupol would deserve to be punished from the Russian perspective.  On the other hand, at the time that Mariupol was taking Ukrainians might have thought that this war will end in a way where Donetsk and Luhansk would permanently be part of Russia. Under that assumption making Mariupol worth as little as possible seems to me like an understandable reason.  Reporting suggests that taking Crimea was expensive for Russia. Having Mariupol destroyed means Russia would have to invest more money into it to make it function again after the war.   

The Wikipedia number for Mariupol's ethnic Russian population is 44%. Russia certainly had the intention to make Mariupol Russian territory. Making Mariupol Russian territory is worth more if it stays standing. Russia has to invest less money into rebuilding Mariupol if it's not destroyed.  The inability of the Russian army was not as apparent at the time Mariupol was taken as it's now, so it's quite plausible that Ukrainians didn't expect to be able to retake it at the time.  World War II was not a far fought to take over German territory, so it's not in the same reference class. That's especially true because of lessons from World War I, that the German population might have to see part of Germany being destroyed to really understand that they lost. 

Ever heard of scorched earth?

This seems right to me about most go clubs, but there’re a lot of other places that seem to me different on this axis. Distinguishing features of Go clubs from my POV: 1. A rapid and trustworthy feedback loop, where everyone wins and loses at non-rigged games of Go regularly.  (Opposite of schools proliferating without evidence.) 2. A lack of need to coordinate individuals.  (People win or lose Go games on their own, rather than by needing to organize other people into coordinating their play.) Some places where I expect “being in sync with the narrative” would diverge more from “just figuring out how to get stronger / how to do the object-level task in a general way”: 1. A hypothetical Go club that somehow twisted around to boost a famous player’s ego about how very useful his particular life-and-death problems were, or something, maybe so they could keep him around and brag about how they had him at their club, and so individual members could stay on his good side.  (Doesn’t seem very likely, but it’s a thought experiment.) 2. Many groups with an “ideological” slant, e.g. the Sierra Club or ACLU or a particular church  3. (?Maybe? not sure about this one) Many groups that are trying to coordinate their members to follow a particular person’s vision for coordinated action, e.g. Ikea's or most other big companies' interactions with their staff, or even a ~8-employee coffee shop that's trying to realize a particular person's vision

I’m curious what you think of the examples in the long comment I just made (which was partly in response to this, but which I wrote as its own thing because I also wish I’d added it to the post in general). I’m now thinking there’re really four concepts: 1. Narrative syncing.  (Example: “the sand is lava.”)   2. Narrative syncing that can easily be misunderstood as information sharing.  (Example: many of Fauci’s statements about covid, if this article about it is correct.)   3. Narrative syncing that sets up social pressure not to disagree, or not to weaken the apparent social norm about how we’ll talk about that.  (Example: “Gambi’s is a great restaurant and we are all agreed on going there,” when said in an irate tone of voice after a long and painful discussion about which restaurant to go to.”)   4. Narrative syncing that falls into categories #2 and #3 simultaneously.   (Example: “The 911 terrorists were cowards,” if used to establish a norm for how we’re going to speak around here rather than to share honest impressions and invite inquiry.) I am currently thinking that category #4 is my real nemesis — the actual thing I want to describe, and that I think is pretty common and leads to meaningfully worse epistemics than an alternate world where we skillfully get the good stuff without the social pressures against inquiry/speech. I also have a prediction that most (though not all) instances of #2 will also be instances of #3, which is part of why I think there's a "natural cluster worth forming a concept around" here.

It's definitely a fair suggestion that I should inform myself better on the actual history of communist ideology in the United States. If I'm scared by what I read, then I will have a better idea of what the problem is. If what I find out doesn't scare me, I'll be relieved, and leave with a better understanding of astronomy.

Yeah, that's the question, is agency substrate-independent or not, and if it is, does it help to pick a specific substrate, or would one make more progress by doing it more abstractly, or maybe both? 

I think we disagree on Yudkowsky's conclusion: his point IMO is that Einstein was able to reduce the search space a lot. He overemphasize for effect (and because it's more impressive to have someone who guesses right directly through these methods), but that doesn't change that Einstein reduced the state space a lot (which you seem to agree with). Many of the relevant posts I quoted talk about how the mechanism of Science are fundamentally incapable of doing that, because they don't specify any constraint on hypothesis except that they must be falsifiable. Your point seems to be that in the end, Einstein still used the sort of experimental data and methods underlying traditional Science, and I tend to agree. But the mere fact that he was able to get the right answer out of millions of possible formulations by checking a couple of numbers should tell you that there was a massive hypothesis-space reducing step before.

Thanks for the kind and thoughtful comment! That's a really good point. I didn't go into that debate in the post (because I tried to not criticize Yudkowky, and also because the post is already way too long), but my take on this is: Yudkowsky probably overstates the case, but that doesn't mean he's wrong about the relevance for Einstein's work of the constrains and armchair reasoning (even if the armchair reasoning was building on more empirical evidence that Yudkowsky originally pointed out). As you say, Einstein apparently did reduce the search space significantly: he just failed to find exactly what he wanted in the reduced space directly.

Things get complicated in situations where e.g. 70% of the group is welcoming and 30% of the group is silently judging and will enact their disapproval later.  And the zeitgeist that is willing to welcome pro-Leverage perspectives might not be willing to actively pressure people to not discriminate against pro-Leverage folk.  Like, they might be fine with somebody being gay, but not motivated enough to step in if someone else is being homophobic in a grocery store parking lot, metaphorically speaking. (This may not describe the actual situation here, of course.  But again it's a fear I feel like I can't dismiss or rule out.)

I expect that many of the people who are giving out party invites and job interviews are strongly influenced by LW. If that's the case, then we can prevent some of the things Duncan mentions by changing LW in the direction of being more supportive of good epistemics (regardless of which "side" that comes down on), with the hope of flow-through effects.

Thanks for gathering these.  They are genuinely helpful (several of them I missed). But yes, as you inferred, the people I've talked to are scared about real-life consequences such as losing funding or having trouble finding employment, which are problems they don't currently have but suspect they will if they speak up. I reiterate that this is a fact about them, as oppose to a fact about reality, but they're not crazy to have some weight on it.

Certainly. But you might not be able to make testable predictions for which others will readily agree with your criteria for judgement. In the exchange, Geoff gives some "evidence", and in other places he gives additional "evidence". It's not really convincing to me, but it at least has the type signature of evidence. Eliezer responds: This is eliding that Geoff probably has significant skill in identifying more detail of how beliefs and goals interact, beyond just what someone would know if they heard about cognitive dissonance theory. Like basically I'm saying that if Eliezer sat with Geoff for a few hours through a few sessions of Geoff doing his thing with some third person, Eliezer would see Geoff behave in a way that suggests falsifiable understanding that Eliezer doesn't have. (Again, not saying he should have done that or anything.)

Is that a serious blow to Cummings' thesis, though? If your idea is, say, "we need a ministry of eduction but the current one is irrecoverably broken", you don't need to invent Google or Amazon to replace it. Depending on how bad the status quo is (which is partly an empirical question), any random thing you come up with might be better than what's already there. In which case his use of the term "startup" would be misleading, but the overall thesis would stay relatively intact. That said, I am quite sympathetic to Chesterton's Fence in this argument. In particular, trying to abolish and replace the Pentagon on day 1 of a new administration (as Cummings suggests in his essay) is... optimistic in a world where other nations can hear you say that.

Exactly! Hes mistaken survival of the salient examples for some kind of intrinsic quality.

I am unaware if Hanson has written about this.  Panspermia could happen by the first replicators happening in space perhaps on comets and then spreading to planets.  As Hanson has pointed out, if life is extremely rare it is strange that life would originate on earth when there are almost certainly super-earths on which you would think life would be much more likely to develop.  A solution to this paradox is that life did develop on such an Eden and then spread to earth billions of years ago from a star system that is now far away.  Our sun might have been very close to the other star system when life spread, or indeed in the same system at the time.

Thanks for natural language stochastic compiler explanation, makes a lot of sense. I broadly get a sense of what you mean by "context window" since people have been mentioning that quite a lot when talking about GPT-3. As for whether it makes sense to write docstrings for trivial things, I think this is only pointing at the Codex demo examples where people write docstrings and get results, but for most of my use cases, and when it gets really interesting, is when it auto-completes 1) while I'm writing 2) when I'm done writing and it guesses the next line 3) when I start a line by "return " or "x = " and wait for his auto-completion. Here, I would have no idea how to formulate it in the docstring, I just generally trust its ability to follow the logic of the code that precedes it (and I find it useful most of the time).

1. if you want a longer init, write a doctring for it I don't get what you mean here. I'm also not an expert on the Codex' "Context windows". 1) in my experience, even if not specified in your prompt, the model still goes or your depency graph (in different files in your repo, not Github) and picks which functions are relevant for the next line 2) if you know which function to. use, then add these function or "API calls" in the docstring;

With respect to [1], "now" would be a good time for someone to kill him given that it might have been much easier to do while he was in custody. (Your un-numbered follow-up about locals being responsible for killing him seems as plausible as anything else!)

I intended that more as an illustrative example than the key piece of evidence. (I think I've gotten tons of advice that was nuanced and wasn't well written up in the EA sphere, and depended on someone correcting my misunderstandings) I think there's generally a lag time of 2ish years between someone having a clear sense of the advice they give people, and that advice getting written up. For example, my previous post here: https://forum.effectivealtruism.org/posts/HBKb3Y5mvb69PRHvP/dealing-with-network-constraints-my-model-of-ea-careers That's basically the advice I'd have given someone for 1-2 years prior to posting it. Meanwhile at the time I posted it I also had the Mysterious Old Wizard bottleneck formulated in my head, but didn't publish for another 1.5 years. Then there's a post like this: https://www.lesswrong.com/posts/HnC29723hm6kJT7KP/taking-ai-risk-seriously-thoughts-by-critch (possibly out of date now, not sure if Critch still endorses it), which I was able to write because Critch and I were in a shared social-context at the time, and I got to overhear him saying things. But he would never have gotten around to writing it on his own. And it still took me maybe 7 months to go from "oh I could have written this up in a blogpost" to "it's actually written up." And then the post is still optimized for addressing my particular misconceptions, which actually involved a lot of back-and-forth at the time. And I think the world is actually changing fairly rapidly, and our best understanding of what-people-should-do changes with it, so being 2 years out of date is pretty bad.

I'll go with Taran's idea there, I think. Something like Douglas Hofstadter's BlooP language, perhaps, which only allows primitive recursive functions. Would BlooP allow for chained arrow notation, or would it be too restrictive for that? More generally, what is the fastest growing primitive recursive function possible, and what limits would that give us in terms of the scope of problems that can be solved by our magic box?

Good point!

I think you made a good effort at intellectual charity!

Agreed that if translate is good enough I can take it. Is that a good source?

https://drive.google.com/file/d/1AG0iHPkHgkbPapSKt0fmXQUHn_ZWCDbS/view?usp=sharing

How high does the base rate need to be in order to explain most COVID deaths? Note that your reference defines deficiency as less than 20 ng/mL, but correlation studies suggest using 30 ng/mL as the COVID-relevant threshold for deficiency.

For me, just understanding the weighting for the votes based on existing karma of the voter would be sufficient to say my idea was "implemented". Even without knowing that it was helpful to learn about that feature of the votes.

I think this is valid.

I want to think further and also want to answer you now, so: knee-jerk response without too much thought is something like "there's a class of cultural values that this framing is insufficient to help you talk about, but it feels to me like a piece of the puzzle that lets you bridge the gap." i.e. I agree there are ways this can be counterproductive for whole categories of important communication. But I'd probably route through this thing anyway, given my current state of knowledge? Would not be surprised to find myself talked out of this viewpoint.

I like this comment.