Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: komponisto 23 June 2017 11:49:37AM *  1 point [-]

Creating a distinct new concept in one's mind is an expensive operation (with both short term and long term costs), so I think it's only to be expected that people will try to match a supposedly new concept to an existing one and see if they can get away with just reusing the existing concept.

Right, but I was reacting to a prior history with that particular commenter, who has been especially prone to doing this (very often where, in my view, it isn't appropriate).

But also: I regard concept-creation as being a large part of what we're in the business of doing, here. (At least, it's a large part of what I'm here for.) That's what theorization is, and I think we're here to theorize (maybe among other things). So it's a cost that I think one has signed up to bear in a context of this sort.

For the most part, it's great if one has the motivation to write up a thorough exposition of a new concept, starting from very elementary premises (although there's also the negative aspect of potentially reinforcing a norm of this level of effort being generally expected every time one wants to introduce a new concept). However, one doesn't always have that motivation (or time, etc.), so it should be allowed sometimes to just point and say "look over here; if you think about this for a while, you may traverse the same inferential path I have, which leads to this conclusion."

Indeed, that's basically exactly what I want out of this forum: a place where people can state inferentially-distant conclusions you might not hear elsewhere (without necessarily needing to justify them from first principles -- such requirements might, after all, be part of why they're not heard elsewhere!). This, of course, requires a community where a certain amount of epistemic trust has been built up, but I think that happened already (c. 2009-11).

For epistemic norms designed to avoid false positives, there are skeptics' forums, and scientific journals. And your grandmother (to paraphrase Feynman). Here, we could use more of the opposite approach (avoiding false negatives). Who else specializes in that (high-quality speculation)? It's basically an empty niche.

Clearly there are distinct skills within what you call physical cognition, and all those skills are not equally valuable

Perhaps I can "strike a chord" with you in particular by talking about value uncertainty in this context. Even to the extent it's clear that not all of the "subskills" are equally valuable (which I don't necessarily concede, in part because its not even clear to me what the right decomposition into subskills is!), it's not necessarily clear which ones are more valuable, and by how much.

To be honest, I'm a little bit suspicious of the whole approach of trying to decompose something like music (or the "physical cognition" involved therein) into its component subskills, with the aim of measuring their relative values. The reason for this is that I doubt anyone currently understands either music, psychology, or 'values' well enough to do this -- at least, at any level of detail much beyond what I've already done by pointing to the physicality of music. To me, the relation between physicality of this sort and certain especially valuable forms of thought (precise, imaginative) is intuitively obvious, and I think consideration and investigation into the matter will reveal this to others; but I don't think this translates easily into something like "music study trains Cognitive Skill S X% more effectively than [rival activity]", especially where we can be confident that S is ontologically sound, and X numerically accurate, "enough".

What is on more solid ground at the moment is the heuristic, correlational case that it is better to be the kind of person who is interested and experienced in things like music than the kind of person who isn't. And it's better to live in the kind of society where such pursuits are enjoyed and admired than in the kind where they're not.

It would be nice to have a more detailed idea of why this is the case -- but I think the study of music, and the other activities in this reference class, is itself a conceptual prerequisite for more fully understanding the phenomenon.

Comment author: Wei_Dai 23 June 2017 09:57:34PM *  1 point [-]

However, one doesn't always have that motivation (or time, etc.), so it should be allowed sometimes to just point and say "look over here; if you think about this for a while, you may traverse the same inferential path I have, which leads to this conclusion."

I don't think that's a reasonable expectation or norm. The expected return from a reader doing something like that is way too low, even in a community like this one. Most new ideas are wrong, and if your idea is wrong then people trying to traverse the same inferential path will get nowhere, and not even know if its their own fault or not. If you write it down then people can figure out where you went wrong and point it out. Even if your idea is right and your reader can be sure of that, why shouldn't you write an good explanation once, which will then save time for potentially hundreds or thousands of readers? By trying to save that time for yourself, you cause other people to waste their time, and then you end up having to answer their confusions and perhaps not even save time for yourself.

You could make an exception to this if you just had a new idea and you want to find out if anyone else already had a similar idea or can see an obvious flaw in it, before deciding to invest more time into explaining it fully, but that doesn't seem to be what you're doing here.

Perhaps I can "strike a chord" with you in particular by talking about value uncertainty in this context. Even to the extent it's clear that not all of the "subskills" are equally valuable (which I don't necessarily concede, in part because its not even clear to me what the right decomposition into subskills is!), it's not necessarily clear which ones are more valuable, and by how much.

I have some uncertainty here, but not that much. I took one semester of piano and one semester of electronic music in high school, and it was intuitively clear that the return from that time spent wasn't nearly as valuable as say reading science fiction or economics textbooks. There's obviously a lot of individual differences here, so if my kid naturally has an interest or talent in music or art and wants to study it, I'm not going to stop her. But if your position is that we should more vigorously encourage an interest in artistic pursuits, I'm going to need more evidence and/or better arguments.

To me, the relation between physicality of this sort and certain especially valuable forms of thought (precise, imaginative) is intuitively obvious, and I think consideration and investigation into the matter will reveal this to others;

This is totally unclear to me. I guess even if it's true, it would be hard for me to figure out on my own since I probably haven't studied music enough to be familiar with the kind of "physicality" that you're talking about. Nor do I understand what forms of thought you're suggesting is related to such physicality. "Precise, imaginative" is pretty vague.

What is on more solid ground at the moment is the heuristic, correlational case that it is better to be the kind of person who is interested and experienced in things like music than the kind of person who isn't. And it's better to live in the kind of society where such pursuits are enjoyed and admired than in the kind where they're not.

I agree with the latter, but I think it's just because in every society there will be some people who naturally enjoy artistic pursuits and almost everyone at least enjoy consuming art, so if art isn't being enjoyed and admired, something must have gone terribly wrong to have caused that. On an individual level, such a correlation, if it exists, can be easily explained by the fact that "better" people have more resources available to pursue artistic interests. Again if you're making the case that artistic pursuits cause people to become better (compared to other pursuits they could spend the time on), you'll have to give more evidence and/or better arguments.

Comment author: komponisto 23 June 2017 08:12:16AM *  1 point [-]

I would describe this more generally as real-world achievement, which is a lot clearer than a label like "physical cognition"

There you go again, compulsively trying to round concepts off to something else!

"Real-world achievement" is considerably less clear as a way of pointing to what I am trying to point to than "physical cognition". It evokes all kinds of distracting side-issues about what constitutes the "real world". (Is pure mathematics "real-world achievement"? et cetera, et cetera).

I can't tell what the point of your second paragraph is. Is it just an attempt to provide reassurance (to whom?) about the value of humanities academia, in the face of what you took to be a "boo humanities academia!" from me (in my comment on Otium)? Or are you seeking to dispute my contention that physical cognition is underpracticed and undervalued there (in which case it would tend to look like your proposal to substitute "real-world achievement" for "physical cognition" was an attempt to muddy the waters in preparation for an equivocation)?

All this notwithstanding, I'm grateful for the pointer to the Eric Raymond essay, as it is relevant to what I was talking about with respect to Maslow and so forth. (In particular, it serves as anecdotal information about, and confirmation of, the distinction between Levels 4 and 5.)

Comment author: Wei_Dai 23 June 2017 09:46:20AM *  1 point [-]

There you go again, compulsively trying to round concepts off to something else!

Creating a distinct new concept in one's mind is an expensive operation (with both short term and long term costs), so I think it's only to be expected that people will try to match a supposedly new concept to an existing one and see if they can get away with just reusing the existing concept. I suggest that if you don't want people to do that, you should define your new concept as clearly as possible, give lots of both positive and negative examples, explain how it differs from any nearby concepts that people might try to "round off" to, and why it makes sense to organize one's thinking in terms of the new concept. (It would also help to give it a googleable name so people can find all that information. Right now, Google defines physical cognition as "Physical cognition, or 'folk physics', is a common sense understanding of the physical world around us and how different objects interact with each other." which is obviously not what you're talking about.)

I think I've avoided rounding off your physical cognition to an existing concept, but I still don't understand how the concept is defined exactly or why it's a useful way of organizing one's thinking as it relates to the question of what kinds of children's activities are most valuable. Clearly there are distinct skills within what you call physical cognition, and all those skills are not equally valuable, nor does practicing one physical cognition skill improve all physical cognition skills equally (e.g., if you practice math skills you improve math skills more than piano skills, and vice versa). Given that, why does it make sense to group a bunch of different skills together into "physical cognition" and then say that practicing piano is valuable because it exercises physical cognition? Wouldn't it make more sense to talk about exactly what skills are improved by practicing piano, and how valuable the increase of those specific skills are?

Comment author: paulfchristiano 23 June 2017 06:53:13AM 0 points [-]

If you are concerned exclusively with suffering, then increasing the number of mature civilizations is obviously bad and you'd prefer that the average civilization not exist. You might think that our descendants are particularly good to keep around, since we hate suffering so much. But in fact almost all s-risks occur precisely because of civilizations that hate suffering, so it's not at all clear that creating "the civilization that we will become on reflection" is better than creating "a random civilization" (which is bad).

To be clear, even if we have modest amounts of moral uncertainty I think it could easily justify a "wait and see" style approach. But if we were committed to a suffering-focused view then I don't think your argument works.

Comment author: Wei_Dai 23 June 2017 08:15:27AM 1 point [-]

But in fact almost all s-risks occur precisely because of civilizations that hate suffering

It seems just as plausible to me that suffering-hating civilizations reduce the overall amount of suffering in the multiverse, so I think I'd wait until it becomes clear which is the case, even if I was concerned exclusively with suffering. But I haven't thought about this question much, since I haven't had a reason to assume an exclusive concern with suffering, until you started asking me to.

To be clear, even if we have modest amounts of moral uncertainty I think it could easily justify a "wait and see" style approach. But if we were committed to a suffering-focused view then I don't think your argument works.

Earlier in this thread I'd been speaking from the perspective of my own moral uncertainty, not from a purely suffering-focused view, since we were discussing the linked article, and Kaj had written:

The article isn't specifically negative utilitarian, though - even classical utilitarians would agree that having astronomical amounts of suffering is a bad thing. Nor do you have to be a utilitarian in the first place to think it would be bad: as the article itself notes, pretty much all major value systems probably agree on s-risks being a major Bad Thing

What's your reason for considering a purely suffering-focused view? Intellectual curiosity? Being nice to or cooperating with people like Brian Tomasik by helping to analyze one of their problems?

Comment author: paulfchristiano 22 June 2017 04:36:46PM 0 points [-]

This question seems like a major input into whether x-risk reduction is useful.

Comment author: Wei_Dai 22 June 2017 06:02:17PM *  0 points [-]

If we try to answer the question now, it seems very likely we'll get the answer wrong (given my state of uncertainty about the inputs that go into the question). I want to keep civilization going until we know better how to answer these types of questions. For example if we succeed in building a correctly designed/implemented Singleton FAI, it ought to be able to consider this question at leisure, and if it becomes clear that the existence of mature suffering-hating civilizations actually causes more suffering to be created, then it can decide to not make us into a mature suffering-hating civilization, or take whatever other action is appropriate.

Are you worried that by the time such an FAI (or whatever will control our civilization) figures out the answer, it will be too late? (Why? If we can decide that x-risk reduction is bad, then so can it. If it's too late to alter or end civilization at that point, why isn't it already too late for us?) Or are you worried more that the question won't be answered correctly by whatever will control our civilization?

Comment author: paulfchristiano 21 June 2017 05:24:53PM 1 point [-]

I've always thought that in order to prevent astronomical suffering, we will probably want to eventually (i.e., after a lot of careful thought) build an FAI that will colonize the universe and stop any potential astronomical suffering arising from alien origins and/or try to reduce suffering in other universes via acausal trade etc., so the work isn't very different from other x-risk work.

It seems like the most likely reasons to create suffering come from the existence of suffering-hating civilizations. Do you think that it's clear/very likely that it is net helpful for there to be more mature suffering-hating civilizations? (On the suffering-focused perspective.)

Comment author: Wei_Dai 21 June 2017 10:19:48PM 3 points [-]

Do you think that it's clear/very likely that it is net helpful for there to be more mature suffering-hating civilizations? (On the suffering-focused perspective.)

My intuition is that there is no point in trying to answer questions like these before we know a lot more about decision theory, metaethics, metaphilosophy, and normative ethics, so pushing for a future where these kinds of questions eventually get answered correctly (and the answers make a difference in what happens) seems like the most important thing to do. It doesn't seem to make sense to try to lock in some answers (i.e., make our civilization suffering-hating or not suffering-hating) on the off chance that when we figure out what the answers actually are, it will be too late. Someone with much less moral/philosophical uncertainty than I do would perhaps prioritize things differently, but I find it difficult to motivate myself to think really hard from their perspective.

Comment author: cousin_it 21 June 2017 08:23:25AM *  0 points [-]

I didn't realize then that disutility of human-built AI can be much larger than utility of FAI, because pain is easier to achieve than human utility (which doesn't reduce to pleasure). That makes the argument much stronger.

Comment author: Wei_Dai 21 June 2017 11:47:52AM 0 points [-]

I didn't realize then that disutility of human-built AI can be much larger than utility of FAI, because pain is easier to achieve than human utility (which doesn't reduce to pleasure).

This argument doesn't actually seem to be in the article that Kaj linked to. Did you see it somewhere else, or come up with it yourself? I'm not sure it makes sense, but I'd like to read more if it's written up somewhere. (My objection is that "easier to achieve" doesn't necessarily mean the maximum value achievable is higher. It could be that it would take longer or more effort to achieve the maximum value, but the actual maximums aren't that different. For example, maybe the extra stuff needed for human utility (aside from pleasure) is complex but doesn't actually cost much in terms of mass/energy.)

Comment author: Kaj_Sotala 21 June 2017 09:00:25AM *  3 points [-]

Huh, I feel very differently. For AI risk specifically, I thought the conventional wisdom was always "if AI goes wrong, the most likely outcome is that we'll all just die, and the next most likely outcome is that we get a future which somehow goes against our values even if it makes us very happy." And besides AI risk, other x-risks haven't really been discussed at all on LW. I don't recall seeing any argument for s-risks being a particularly plausible category of risks, let alone one of the most important ones.

It's true that there was That One Scandal, but the reaction to that was quite literally Let's Never Talk About This Again - or alternatively Let's Keep Bringing This Up To Complain About How It Was Handled, depending on the person in question - but then people always only seemed to be talking about that specific incident and argument. I never saw anyone draw the conclusion that "hey, this looks like an important subcategory of x-risks that warrants separate investigation and dedicated work to avoid".

Comment author: Wei_Dai 21 June 2017 11:17:20AM *  2 points [-]

I don't recall seeing any argument for s-risks being a particularly plausible category of risks, let alone one of the most important ones.

There was some discussion back in 2012 and sporadically since then. (ETA: You can also do a search for "hell simulations" and get a bunch more results.)

I never saw anyone draw the conclusion that "hey, this looks like an important subcategory of x-risks that warrants separate investigation and dedicated work to avoid".

I've always thought that in order to prevent astronomical suffering, we will probably want to eventually (i.e., after a lot of careful thought) build an FAI that will colonize the universe and stop any potential astronomical suffering arising from alien origins and/or try to reduce suffering in other universes via acausal trade etc., so the work isn't very different from other x-risk work. But now that the x-risk community is larger, maybe it does make sense to split out some of the more s-risk specific work?

Comment author: cousin_it 21 June 2017 06:40:08AM *  4 points [-]

I guess I didn't think about it carefully before. I assumed that s-risks were much less likely than x-risks (true) so it's okay not to worry about them (false). The mistake was that logical leap.

In terms of utility, the landscape of possible human-built superintelligences might look like a big flat plain (paperclippers and other things that kill everyone without fuss), with a tall sharp peak (FAI) surrounded by a pit that's astronomically deeper (many almost-FAIs and other designs that sound natural to humans). The pit needs to be compared to the peak, not the plain. If the pit is more likely, I'd rather have the plain.

Was it obvious to you all along?

Comment author: Wei_Dai 21 June 2017 08:17:15AM 2 points [-]

Didn't you realize this yourself back in 2012?

Comment author: cousin_it 20 June 2017 09:04:32AM *  6 points [-]

That's interesting. May I ask in what capacity?

Father of four, teach a programming class to some kids in my area, attend many social gatherings full of kids.

What if you restrict YouTube, then give them enough free time to get bored of their toys and "generally being annoying"?

YMMV, but I haven't seen much progress happening as a result of boredom. As a child I was in this situation and spent most of my time pointlessly reading fiction. Got serious about math and programming only due to having amazing teachers in high school (actual math and CS researchers).

On the other hand, if you're asking about your kids, maybe they'll turn out like you and get interested in stuff naturally :-)

What are verbally focused activities? Like competitive debating?

At preschool level it's games like broken telephone or I Spy. At school level I'm not sure, but I feel that my verbal abilities are low because I never did anything like debating in my teens.

Comment author: Wei_Dai 21 June 2017 07:49:36AM 2 points [-]

teach a programming class to some kids in my area

Interesting, how do you motivate the kids to want to learn?

YMMV, but I haven't seen much progress happening as a result of boredom. As a child I was in this situation and spent most of my time pointlessly reading fiction.

Reading fiction hardly seems pointless, compared to other pursuits a parent might push a child into. It develops vocabulary and reading comprehension (helpful when you later want to read non-fiction), general knowledge and social abilities, and can lead to other interests. I got interested in crypto and the Singularity from reading Vernor Vinge, and philosophy in part from reading Greg Egan.

It seems like boredom as a strategy requires a lot of time and patience, even when it succeeds. I wasn't that serious about programming (despite learning the basics as a kid) until I got into crypto and decided that writing an open source crypto library would be a good way to help push towards a positive Singularity, and that only happened in college after I read Vinge's A Fire Upon the Deep.

At school level I'm not sure, but I feel that my verbal abilities are low because I never did anything like debating in my teens.

Your verbal abilities don't seem low to me (at least in writing). Maybe low compared to Eliezer, but then he is just off the charts.

I'm worried that competitive debating trains for the wrong things (e.g., using arguments as soldiers). ChristianKl's suggestion of drama lessons doesn't seem like it would increase verbal abilities more than say reading, but I'd be interested if anyone has evidence to offer about that. I'll probably have to do some research to see what other activities are good for increasing verbal skills.

Comment author: komponisto 20 June 2017 10:05:05PM 6 points [-]

Piano and ballet seem like upper-class costly signalling. "I am so rich I can spend tons of time doing unproductive activities."

Well, no need to speculate about a future Malthusian dystopia, since it appears to be already here, psychologically!

Allow me to refer you to this comment of mine, and the ensuing discussion, on Sarah Constantin's blog. Artistic pursuits may be "upper-class", but they are not unproductive. They serve to keep the upper classes practiced in physical cognition, counteracting a tendency to shift entirely into social modes of cognition (gossip and status-signaling games) as one ascends the social ladder. This is very important for the quality of decisions they make as leaders of society. (See here for more on the distinction between physical and social cognition -- which, incidentally, I myself would identify with the famous "near" and "far" modes respectively, though not everybody goes along with that.)

The fact that there has been such a decline in interest and participation in high culture among the upper classes is very worrying, and something I would not particularly hesitate to link to the intellectual decadence that we see in general society. (Ever notice how hard it is to engage in reasoning in public? Or the stigmatization -- including self-stigmatization -- of so-called "nerds"? These are facets of the decadence I'm talking about.)

Now, you refer (rightly, I think) to sports as being "useful". But sports are just a more primitive version of arts; they are useful for basically the same reason, but require, on average, less intellectual ability and more physical ability. (Cf. this comment of mine on Zack Davis's blog.) The most interesting of each, of course, are typically somewhat demanding in both ways.

In particular, if you "get" sports (and programming/CS or math) and want to understand what arts are about, try thinking of it like this: imagine a version of sports where it was actually true that "it doesn't matter whether you win or lose, but only how you play the game". That is, the "standings" did not consist simply of an ordered list (array), but rather a highly complex weighted graph of some sort, that took into account the details of the trajectories of "gameplay".

If the upper classes strongly favor sports over arts, you're probably living in a crassly militaristic society like ancient Sparta, Rome, or the 20th-century USA. You don't usually find the exact opposite, but when arts at least have a strong presence (pre-WWI European powers), your society has a chance at getting interesting things done (e.g. scientific and technological innovation).

The worst situation to be in, however, is where the upper classes stop participating in either, and instead spend all of their time in passive consumption and in gossipy status games; then not only is your society probably headed for collapse, but you won't even produce much value along the way. (Cf. the fall of Rome, this is where the USA and similar countries now seem headed.)

Now, if you're thinking "even if true, none of this pertains to the present discussion, because LW readers aren't part of the upper classes" (which, indeed, is an implication of the parent comment), this is wrong. LW readers are rich programmers; people like Wei Dai and Viliam can pick up the phone (or, more likely, dash off an email) and get themselves a six-figure job starting next week, if somehow they don't already have one. With this level of resources (distributed in whatever way within a portfolio of financial, social, and intellectual capital), there is no excuse for conceiving oneself at any level below 4 of the Maslow hierarchy. Probably 5, really. No excuse, that is, except for toxic memeplexes spawned by evil egregores, that say that LW readers are destined only to be servants of the Man.

Comment author: Wei_Dai 21 June 2017 06:18:53AM 0 points [-]

Artistic pursuits may be "upper-class", but they are not unproductive. They serve to keep the upper classes practiced in physical cognition, counteracting a tendency to shift entirely into social modes of cognition (gossip and status-signaling games) as one ascends the social ladder.

I'm having trouble understanding this. Why do artistic pursuits constitute practice in physical cognition as opposed to social cognition? It seems obvious to me that artistic pursuits are (among other things) a type of status signaling, so I'm confused why you're contrasting the two. Please explain?

With this level of resources (distributed in whatever way within a portfolio of financial, social, and intellectual capital), there is no excuse for conceiving oneself at any level below 4 of the Maslow hierarchy. Probably 5, really.

(Aside from not being sure how valid the Maslow hierarchy is) I agree with this. But I don't see art/music/dance classes as a particularly good way to prepare most kids to fulfill their level 4 and 5 needs, mostly because there is too much competition from other parents pushing their kids into artistic pursuits. The amount of talent, time, and effort needed to achieve recognition or a feeling of accomplishment seem too high, compared to other possible pursuits.

View more: Next