All of tristanm's Comments + Replies

A couple of guesses for why we might see this, which don't seem to depend on property:

• An obligation to act is much more freedom-constraining than a prohibition on an action. The more and more one considers all possible actions with the obligation to take the most ethically optimal one, the less room they have to consider exploration, contemplation, or pursuing their own selfish values. Prohibition on actions does not have this effect.
• The environment we evolved in had roughly the same level of opportunity to commit harmful acts, bur far less opportunity

It seems to construct an estimate of it by averaging a huge number of observations together before each update (for Dota 5v5, they say each batch is around a million observations, and I'm guessing it processes about a million batches). The surprising thing is that this works so well, and it allows leveraging of computational resources very easily.

My guess for how it deals with partial observability in a more philosophical sense is that it must be able to store an implicit model of the world in some way, in order to better predict the reward it will eventu... (read more)

I don't know how hard it would be to do a side by side "FLOPS" comparison of Dota 5v5 vs AlphaGo / AlphaZero, but it seems like they are relatively similar in terms of computational cost required to achieve something close to "human level". However, as has been noted by many, Dota is a game of vastly more complexity because of its continuous state, partial observability, large action space, and time horizon. So what does it mean when it requires roughly similar orders of magnitude of compute to achieve the same level of ability as human... (read more)

I've been meditating since I was about 19, and before I came across rationality / effective altruism. There is quite a bit of overlap between the sets of things I've been able to learn from both schools of thought, but I think there are still a lot of very useful (possibly even necessary) things that can only be learned from meditative practices right now. This is not because rationality is inherently incapable of learning the same things, but because within rationality it would take very strong and well developed theories, perhaps developed through large ... (read more)

It seems like in the vast majority of conversations, we find ourselves closer to the "exposed to the Deepak Chopra version of quantum mechanics and haven't seen the actual version yet" situation than we do to the "Arguing with someone who is far less experienced and knowledgeable than you are on this subject." In the latter case, it's easy to see why steelmanning would be counterproductive. If you're a professor trying to communicate a difficult subject to a student, and the student is having trouble understanding your position, it's u... (read more)

I don't see him as arguing against steelmanning. But the opposite of steelmanning isn't arguing against an idea directly. You've got to be able to steelman an opponent's argument well in order to argue against it well too, or perhaps determine that you agree with it. In any case, I'm not sure how to read a case for locally valid argumentation steps as being in favor of not doing this. Wouldn't it help you understand how people arrive at their conclusions?

I would also like to have a little jingle or ringtone play every time someone passes over my comments, please implement for Karma 3.0 thanks

What's most unappealing to me about modern, commercialized aesthetics is the degree to which the bandwidth is forced to be extremely high - something I'd call the standardization of aesthetics. When I walk down the street in the financial district of SF, there's not much variety to be found in people's visual styles. Sure, everything looks really nice, but I can't say that it doesn't get boring after a while. It's clear that a lot of information is being packed into people's outfits, so I should be able to infer a huge amount about someone just by looking ... (read more)

It seems like this objection might be empirically testable, and in fact might be testable even with the capabilities we have right now. For example, Paul posits that AlphaZero is a special case of his amplification scheme. In his post on AlphaZero, he doesn't mention there being an aligned "H" as part of the set-up, but if we imagine there to be one, it seems like the "H" in the AlphaZero situation is really just a fixed, immutable calculation that determines the game state (win/loss/etc.) that can be performed with any board input... (read more)

I can't emphasize enough how important the thing you're mentioning here is, and I believe it points to the crux of the issue more directly than most other things that have been said so far. 

We can often weakman postmodernism as making basically the same claim, but this doesn't change the fact that a lot of people are running an algorithm in their head with the textual description "there is no outside reality, only things that happen in my mind." This algorithm seems to produce different behaviors in people than if they were running... (read more)

I could probably write a lot more about this somewhere else, but I'm wondering if anyone else felt that this paper seemed to be kind of shallow. This comment is probably too brief to really do this feeling justice, but I'll probably decompose this into two things I found disappointing:

1. "Intelligence" is defined in such a way that leaves a lot to be desired. It doesn't really define it in a way that makes it qualitatively different than technology in general ("tasks thought to require intelligence" is probably much less us

This is partly a reply and partly an addendum to my first comment. I've been thinking about a sort of duality that exists within the rationalist community to a degree and that has become a lot more visible lately, in particular with posts like this. I would refer to this duality as something like "The Two Polarities of Noncomformists", although I'm sure someone could think of something better to call it. The way I would describe it is that, communities like this one are largely composed of people who feel fairly uncomfortable with the w... (read more)

I think I've been doing "mythic mode" for nearly my entire life, but not because I came up with this idea explicitly and intentionally deployed it, but just because it sort of happens on its own without any effort. It's not something that happens at every moment, but frequently enough that it's a significant driving force.

But it used to be that I didn't notice myself doing this at all, this is just how it felt to be a person with goals and desires, and doing mythic mode was just how that manifested in my subconscious desires. ... (read more)

Or more generally: Break up a larger and more difficult task into a chain of subtasks, and see if you have enough willpower to accomplish just the first piece. If you can, you can allow yourself a sense of accomplishment and use that as a willpower boost to continue, or try to break up the larger task even further.

If this works and people are able to get themselves to do more complex and willpower heavy tasks that they wouldn't normally be able to do, wouldn't that be a good thing by default? Or are you worried that it would allow people with poorly aligned incentives to do more damage?

Circling seems like one of those things where both its promoters and detractors vastly overestimate its effects, either positive or negative. Like a lot the responses to this are either "it's pretty cool" or "it's pretty creepy." What about "meh"? The most likely outcome is that Circling does something extremely negligible if it does anything at all, or if it does seem to have some benefit it's because of the extra hour you set aside to think about things without many other distractions. In which case, a questio... (read more)

Personally I wonder how much of this disagreement can be attributed to prematurely settling on specifc fundamental positions or some hidden metaphysics that certain organizations have (perhaps unknowingly) committed to - such as dualism or pansychism. One of the most salient paragraphs from Scott's article said:

Morality wasn’t supposed to be like this. Most of the effective altruists I met were nonrealist utilitarians. They don’t believe in some objective moral law imposed by an outside Power. They just think that we should pursue our own human-paroc

I'm actually having a hard time deciding what kind of superstimuli are having too strong of a detrimental effect on my actions. The reason for this is that some superstimuli also act as a willpower restorer. Take music, for example. Listening to music does not usually get mentioned as a bad habit, but it also is an extremely easy stimuli to access, requires little to no attention or effort to maintain use of, and at least for me, tends to amplify the degree of mind wandering and daydreaming. On the other hand, it is a huge mood booster and increases c... (read more)

I'm not actually seeing why this post is purely an instance of conjunctive fallacy. A lot of the details he describes are consequences of cars being autonomous or indirect effects of this. And that's not to say there are no errors here, just that I don't think it's merely a list of statements A,B,C,etc with no causal relationship.

If you define "rationality" as having good meta level cognitive processes for carving the future into a narrower set of possibilities in alignment with your goals, then what you've described is simply a set of relatively poor heuristics for one specific set of goals, namely, the gaining of social status and approval. One can have that particular set of goals and still be a relatively good rationalist. Of course, where do you draw the line between "pseudo" and "actual" given that we are all utilizing cognitive heuristics to some degree? I see the line being drawn as sort of arbitrary.

I think the driving motivator for seeking out high variance in groups of people to interact with is an implicit belief that my value system is malleable and a stong form of modesty concerning my beliefs about what values I should have. Over time I realized that my value system isn't really all that malleable and my intuitions about it are much more reliable indicators than observing a random sample of people, therefore a much better strategy for fulfilling goals set by those values is to associate with people who share them.

If the latter gets too large, then you start getting swarmed with people who want money and prestige but don't necessarily understand how to contibute, who are incentivized to degrade the signal of what's actually important.

During this decade the field of AI in general became one of the most prestigious and high-status academic fields to work in. But as far as I can tell, it hasn't slowed down the rate of progress in advancing AI capability. If anything, it has sped it up - by quite a bit. It's possible that a lot of newcomers to the f... (read more)

I'm curious as to whether or not the rationalsphere/AI risk community has ever experimented with hiring people to work on serious technical problems who aren't fully aligned with the values of the community or not fully invested in it already. It seems like ideological alignment is a major bottleneck to locating and attracting relevant skill levels and productivity levels, and there might be some benefit to being open about tradeoffs that favor skill and productivity at the expense of not being completely committed to solving AI risk.

(Re-writing this comment from the original to make my point a little more clear).

I think it is probably quite difficult to map the decisions of someone on a continuum from really bad to really good if you can't simulate the outcomes of many different possible actions. There's reason to suspect that the "optimal" outcome in any situation looks vastly better than even very good but slightly sub-optimal decisions, and vise-versa for the least optimal outcome.

In this case we observed a few people who took massive risks (by devoting their ... (read more)

Mostly I just want people to stop bringing models about the other person's motives or intentions into conversations, and if tabooing words or phrases won't accomplish that, and neither will explicitly enforcing a norm, then I'm fine not going that route. It will most likely involve simply arguing that people should adopt a practice similar to what you mentoned.

Confusion in the sense of one or both parties coming to the table with incorrect models is a root cause, but this is nearly always the default situation. We ostensibly partake in a conversation in order to update our models to more accurate ones and reduce confusion. So while yes, a lack of confusion would make bad conversations less likely, it also just reduces the need for the conversation to begin with.

And here we’re talking about a specific type of conversation that we’ve claimed is a bad thing and should be prevented. Here we need to identify a diffe... (read more)

I'm not really faulting all status games in general, only tactics which force them to become zero-sum. It's basically unreasonable to ask that humans change their value systems so that status doesn't play any role, but what we can do is alter the rules slightly so that outcomes we don't like become improbable. If I'm accused of being uncharitable, I have no choice but to defend myself, because being seen as "an uncharitable person" is not something I want to be included in anyone's models of me (even in the case wher... (read more)

If the goal is for conversations to be making epistemic progress, with the caveat that individual people have additional goals as well (such as obtaining or maintaining high status within their peer group), and Demon Threads “aren’t important” in the sense that they help neither of these goals, then it seems the solution would simply be better tricks participants in a discussion can use in order to notice when these are happening or likely to happen. But I think it’s pretty hard to actually measure how much status is up for grabs in a given conversation. ... (read more)

I'm trying to decide whether or not I understand what "looking" is, and I think it's possible I do, so I want to try and describe it, and hopefully get corrected if it turns out I'm very wrong. 

Basically, there's sort of a divide between "feeling" and "Feeling" and it's really not obvious that there should be, since we often make category errors in referring to these things. On the one hand, you might have the subjective feeling of pain, like putting your hand on something extremely hot. Part of that f... (read more)

We need to understand how the black box works inside, to make sure our version's behavior is not just similar but based on the right reasons.

I think here "black-box" can be used to refer to two different things, one to refer to things in philosophy or science which we do not fully understand yet, and also to machine learning models like neural networks that seem to capture their knowledge in ways that are uninterpretable to humans.

We will almost certainly require the use of machine learning or AI to model systems that are beyond our capabi... (read more)

A sort of fun game that I’ve noticed myself playing lately is to try and predict the types of objections that people will give to these posts, because I think once you sort of understand the ordinary paranoid / socially modest mindset, they become much easier to predict.

For example, if I didn’t write this already, I would predict a slight possibility that someone would object to your implication that requiring special characters in passwords is unnecessary, and that all you need is high entropy. I think these types of objections could even contain some p... (read more)

My understanding of A/B testing is that you don't need an explicit causal model , or a "big theory" in order to successfully use it, you mostly would be using intuitions gained from experience in order to test hypotheses like "users like the red page better than the blue page", which has no explicit causal information.

Here you argue that intuitions gained from experience count as hypotheses just as much as causal theories do, and not only that, but that they tend to succeed more often than the big theories do. That depends on what... (read more)

I wonder if it would have been as frustrating if he had instead opened with "The following are very loosely based on real conversations I've had, with many of the details changed or omitted." That's something many writers do and get away with, for the very reason that sometimes you want to show that someone actually thinks what you're claiming people think, but you don't actually want to be adversarial to the people involved. Maybe it's not the fairest to the specific arguments, but the alternative could quite possibly tu... (read more)

There’s a fundamental assumption your argument rests on which is a choice of prior: Assume that everyone’s credences in a given proposition is a distribution centered around the correct value an ideal agent would give to that proposition if they had access to all the information that was available and relevant to that proposition and had enough time and capacity to process that information to the fullest extent. Your arguments are sound given that the above is actually the correct prior, but I see most of your essay as arguing why modesty would be the corr... (read more)

I believe that equilibria has already arrived...and at no real surprise, since no preventive measures were ever put into place.

The reason this equilibria occurs is because there is a social norm that says "upvote if this post is both easy to understand and contains at least one new insight." If a post contains lots of deep and valuable insights, this increases the likelihood that it is complex, dense, and hard to understand. Hard to understand posts often get mistaken for poor writing (or worse, will be put in a separate class and compared agai... (read more)

I think the post could also be interpreted as saying, "when you select for rare levels of super-competence in one trait, you are selecting against competence in most other traits" or at least, "when you select for strong charisma and leadership ability, you are selecting for below average management ability." It's a little ambiguous about how far this is likely to generalize or just how strongly specific skills are expected to anti-correlate.

I think the central reason it’s possible for an individual to know something better than the solution currently prescribed by mainstream collective wisdom is that the vastness in the number of degrees of freedom in optimizing civilization guarantees that there will always be some potential solutions to problems that simply haven’t received any attention yet. The problem space is simply way, way too large to expect that even relatively easy solutions to certain problems are known yet.

While modesty may be appropriate in situations regarding a problem tha... (read more)