All of kman's Comments + Replies

kman82

I think I mostly agree with the critique of "pause and do what, exactly?", and appreciate that he acknowledged Yudkowsky as having a concrete plan here. I have many gripes, though.

Whatever name they go by, the AI Doomers believe the day computers take over is not far off, perhaps as soon as three to five years from now, and probably not longer than a few decades. When it happens, the superintelligence will achieve whatever goals have been programmed into it. If those goals are aligned exactly to human values, then it can build a flourishing world beyond ou

... (read more)
kman41

You acknowledge this but I feel you downplay the risk of cancer - an accidental point mutation in a tumour suppressor gene or regulatory region in a single founder cell could cause a tumour.

For each target the likely off-targets can be predicted, allowing one to avoid particularly risky edits. There may still be issues with sequence-independent off-targets, though I believe these are a much larger problem with base editors than with prime editors (which have lower off-target rates in general). Agree that this might still end up being an issue.

Unless you ar

... (read more)
kman10

This seems unduly pessimistic to me. The whole interesting thing about g is that it's easy to measure and correlates with tons of stuff. I'm not convinced there's any magic about FSIQ compared to shoddier tests. There might be important stuff that FSIQ doesn't measure very well that we'd ideally like to select/edit for, but using FSIQ is much better than nothing. Likewise, using a poor man's IQ proxy seems much better than nothing.

This may have missed your point, you seem more concerned about selecting for unwanted covariates than 'missing things', which i... (read more)

kman30

Non-coding means any sequence that doesn't directly code for proteins. So regulatory stuff would count as non-coding. There tend to be errors (e.g. indels) at the edit site with some low frequency, so the reason we're more optimistic about editing non-coding stuff than coding stuff is that we don't need to worry about frameshift mutations or nonsense mutations which knock-out the gene where they occur. The hope is that an error at the edit site would have a much smaller effect, since the variant we're editing had a very small effect in the first place (and... (read more)

4George3d6
I don't particularly see why the same class of errors in regulatory regions couldn't cause a protein to stop being expressed entirely or accidentally up/down-regulate expression by quite a lot, having similar side effects. But it's getting into the practical details of gene editing implementation so no idea.
kman10

Another thing: if you have a test for which g explains the lion's share of the heritable variance, but there are also other traits which contribute heritable variance, and the other traits are similarly polygenic as g (similar number of causal variants), then by picking the top-N expected effect size edits, you'll probably mostly/entirely end up editing variants which affect g. (That said, if the other traits are significantly less polygenic than g then the opposite would happen.)

I should mention, when I wrote this I was assuming a simple model where the causal variants for g and the 'other stuff' are disjoint, which is probably unrealistic -- there'd be some pleiotropy.

kman40

Even out of this 10%, slightly less than 10% of that 10% responded to a 98-question survey, so a generous estimate of how many of their customers they got to take this survey is 1%. And this was just a consumer experience survey, which does not have nearly as much emotional and cognitive friction dissuading participants as something like an IQ test.

What if 23&me offered a $20 discount for uploading old SAT scores? I guess someone would set up a site that generates realistically distributed fake SAT scores that everyone would use. Is there a standardize... (read more)

1who am I?
I wouldn't call it magic, but what makes FSIQ tests special is that they're specifically crafted to estimate g. To your point, anything that involves intelligence (SAT, ACT, GRE, random trivia quizzes, tying your shoes) will positively correlate with g even if only weakly, but the correlations between g factor scores and full-scale IQ scores from the WAIS have been found to be >0.95, according to the same Wikipedia page you linked in a previous reply to me. Like both of us mentioned in previous replies, using imperfect proxy measures would necessitate multiplying your sample size because of diluted p-values and effect sizes, along with selecting for many things that are not intelligence. There are more details about this in my reply to gwern's reply to me.
1kman
This may have missed your point, you seem more concerned about selecting for unwanted covariates than 'missing things', which is reasonable. I might remake the same argument by suspecting that FSIQ probably has some weird covariates too -- but that seems weaker. E.g. if a proxy measure correlates with FSIQ at .7, then the 'other stuff' (insofar as it is heritable variation and not just noise) will also correlate with the proxy at .7, and so by selecting on this measure you'd be selecting quite strongly for the 'other stuff', which, yeah, isn't great. FSIQ, insofar as it had any weird unwanted covariates, would probably much less correlated with them than .7
kman40

Thanks for leaving such thorough and thoughtful feedback!

You could elect to use proxy measures like educational attainment, SAT/ACT/GRE score, most advanced math class completed, etc., but my intuition is that they are influenced by too many things other than pure g to be useful for the desired purpose. It's possible that I'm being too cynical about this obstacle and I would be delighted if someone could give me good reasons why I'm wrong.

The SAT is heavily g-loaded: r = .82 according to Wikipedia, so ~2/3 of the variance is coming from g, ~1/3 from other ... (read more)

3who am I?
The article that Wikipedia cites for that factoid, Frey & Detterman 2004, uses the National Longitudiunal Survey of Youth 1979 for its data that included the SAT and ASVAB (this is what they used to estimate IQ, so first need to find correlation between ASVAB and actual FSIQ) scores for the samples. This introduces the huge caveat that the SAT has changed drastically since this study was conducted and is likely no longer nearly as strongly correlated with g ever since 1994. This is when they began recentering scores and changing the scoring methodology, making year-to-year comparisons of scores no longer apples to apples. The real killer was their revision of the math and verbal sections to mostly include questions that "approximate more closely the skills used in college and high school work", get rid of "contrived word problems" (e.g., the types of verbal ability questions you'd see on an IQ test), and include "real-world" problems that may be more relevant to students. Since it became more focused on assessing knowledge rather than aptitude, this rehauling of the scoring and question format made it much more closely reflect a typical academic benchmark exam rather than an assessment of general cognitive ability. This decreased its predictive power for general intelligence and increased its predictive power for high school GPA, as well as other things that correlate with high school GPA like academic effort, openness, and SES. It's for these reasons that Mensa and other psychometrics societies stopped using SAT as an acceptable proxy for IQ unless you took it prior to 1994. I've taken both the SAT and ACT and I cannot imagine the ACT is much better (2004 study showed r=0.73). My guess is that the GRE would be much more correlated with general intelligence than either of the other two tests (still imperfectly so, wouldn't put it >0.8), but then the problem is that a much smaller fraction of the population has taken the GRE and there is a large selection bias as to
1kman
I should mention, when I wrote this I was assuming a simple model where the causal variants for g and the 'other stuff' are disjoint, which is probably unrealistic -- there'd be some pleiotropy.
kman10

I don't think this therapy as OP describes it is possible for reasons that have already been stated by HiddenPrior and other reasons

Can you elaborate on this? We'd really appreciate the feedback.

7who am I?
I posted my reply to this as a direct reply to the OP because I think it's too huge and elaborate to keep hidden here.
kman50

We'd edit the SNPs which have been found to causally influence the trait of interest in an additive manner. The genome would only become "extremely unlikely" if we made enough edits to push the predicted trait value to an extreme value -- which you probably wouldn't want to do for decreasing disease risk. E.g. if someone has +2 SD risk of developing Alzheimer's, you might want to make enough edits to shift them to -2 SD, which isn't particularly extreme.

You're right that this is a risk with ambitious intelligence enhancement, where we're actually intereste... (read more)

kman60

Promoters (and any non-coding regulatory sequence for that matter) are extremely sensitive to point mutations.

A really important question here is whether the causal SNPs that affect polygenic traits tend to be located in these highly sensitive sequences. One hypothesis would be that regulatory sequences which are generally highly sensitive to mutations permit the occasional variant with a small effect, and these variants are a predominant influence on polygenic traits. This would be bad news for us, since even the best available editors have non-negligible... (read more)

kman32

what improvements would be top of mind for you?

  • allow multiple causal variants per clump
  • more realistic linkage disequilibrium structure
  • more realistic effect size and allele frequency distributions
    • it's not actually clear to me the current ones aren't realistic, but this could be better informed by data
    • this might require better datasets
  • better estimates of SNP heritability and number of causal variants
    • we just used some estimates which are common in the literature (but there's a pretty big range of estimates in the literature)
    • this also might require better datasets
kman61

The hope is that local neural function could be altered in a way that improves fluid intelligence, and/or that larger scale structural changes could happen in response to the edits (possibly contingent on inducing a childlike state of increased plasticity).

2dr_s
The former thing sounds like overclocking a CPU. The latter instead "erase chunks of someone's personality and memory and let them rewrite it, turning them into an essentially different person". I don't think many people would just volunteer for something like that. We understand still far too little of how brains work to think that tinkering with genes and just getting some kind of Flowers for Algernon-ish intelligence boost is the correct model of this. As it often happens, it's much easier to break something than to build it up, especially something as delicate and complex as a human brain. Right now this seems honestly to belong in the "mad science" bin to me.
kman40

Showing that many genes can be successfully and accurately edited in a live animal (ideally human). As far as I know, this hasn't been done before! Only small edits have been demonstrated.

This is more or less our current plan.

Showing that editing embryos can result in increased intelligence. I don't believe this has even been done in animals, let alone humans.

This has some separate technical challenges, and is also probably more taboo? The only reason that successfully editing embryos wouldn't increase intelligence is that the variants being targeted weren... (read more)

1LGS
  It's hard, yes -- I'd even say it's impossible. But is it harder than the brain? The difference between growth plates and whatever is going on in the brain is that we understand growth plates and we do not understand the brain. You seem to have a prior of "we don't understand it, therefore it should be possible, since we know of no barrier". My prior is "we don't understand it, so nothing will work and it's totally hopeless". Actually, IQ test scores increase by a few points if you test again (called test-retest gains). Additionally, IQ varies substantially based on which IQ test you use. It is gonna be pretty hard to convince people you've increased your patients' IQ by 3 points due to these factors -- you'll need a nice large sample with a proper control group in a double-blind study, and people will still have doubts. Lol. I mean, you're not wrong with that precise statement, it just comes across as "the fountain of eternal youth will enable progress on important, difficult diplomatic and geopolitical situations". Yes, this is true, but maybe see if you can beat botox at skin care before jumping to the fountain of youth. And there may be less fantastical solutions to your diplomatic issues. Also, finding the fountain of youth is likely to backfire and make your diplomatic situation worse. (To explain the metaphor: if you summon a few von Neumanns into existence tomorrow, I expect to die of AI sooner, on average, rather than later.)
kman10

Probably not? The effect sizes of the variants in question are tiny, which is probably why their intelligence-promoting alleles aren't already at fixation.

There probably are loads of large effect size variants which affect intelligence, but they're almost all at fixation for the intelligence-promoting allele due to strong negative selection. (One example of a rare intelligence promoting mutation is CORD7, which also causes blindness).

kman10

I think that most are focusing on single-gene treatments because that's the first step. If you can make a human-safe, demonstrably effective gene-editing vector for the brain, then jumping to multiplex is a much smaller step (effective as in does the edits properly, not necessarily curing a disease). If this were a research project I'd focus on researching multiplex editing and letting the market sort out vector and delivery.

Makes sense.

I am more concerned about the off-target effects; neurons still mostly function with a thousand random mutations, but you

... (read more)
kman102

Really interesting, thanks for commenting.

My lab does research specifically on in vitro gene editing of T-cells, mostly via Lentivirus and electroporation, and I can tell you that this problem is HARD.

  • Are you doing traditional gene therapy or CRISPR-based editing?
    • If the former, I'd guess you're using Lentivirus because you want genome integration?
    • If the latter, why not use Lipofectamine?
  • How do you use electroporation?

Even in-vitro, depending on the target cell type and the amount/ it is very difficult to get transduction efficiencies higher than 70%, and t

... (read more)
kman92

The smaller size of Fanzors compared to Cas9 is appealing and the potential for lower immunogenicity could end up being very important for multiplex editing (if inflammation in off-target tissues is a big issue, or if an immune response in the brain turns out to be a risk).

The most important things are probably editing efficiency and the ratio of intended to unintended edits. Hard to know how that will shake out until we have Fanzor equivalents of base and prime editors.

kman43

(I should clarify, I don't see modification of polygenic traits just as a last ditch hail mary for solving AI alignment -- even in a world where I knew AGI wasn't going to happen for some reason, the benefits pretty clearly outweigh the risks. The case for moving quickly is reduced, though.)

kman21

The stakes could hardly be more different -- polygenic trait selection doesn't get everyone killed if we get it slightly wrong.

4kman
(I should clarify, I don't see modification of polygenic traits just as a last ditch hail mary for solving AI alignment -- even in a world where I knew AGI wasn't going to happen for some reason, the benefits pretty clearly outweigh the risks. The case for moving quickly is reduced, though.)
kman10

How large are the Chinese genotype datasets?

kman10

The scaling laws are extremely well established in DL and there are strong theoretical reasons (and increasingly experimental neurosci evidence) that they are universal to all NNs, and we have good theoretical models of why they arise.

I'm not aware of these -- do you have any references?

Both brains and DL systems have fairly simple architectural priors in comparison to the emergent learned complexity

True but misleading? Isn't the brain's "architectural prior" a heckuva lot more complex than the things used in DL?

Brains are very slow so have limited combina

... (read more)
3jacob_cannell
[Scaling law theories] Sure: here's a few: quantization model, scaling laws from the data manifold, and a statistical model. The full specification of the DL system includes the microde, OS, etc. Likewise much of the brain complexity is in the smaller 'oldbrain' structures that are the equivalent of a base robot OS. The architectural prior I speak of is the complexity on top of that, which separates us from some ancient earlier vertebrate brain. But again see the brain as a ULM post, which cover the the extensive evidence for emergent learned complexity from simple architecture/algorithms (now the dominant hypothesis in neuroscience). Most everything above the hardware substrate - but i've already provided links to sections of my articles addressing the convergence of DL and neurosci with many dozens of references. So it'd probably be better to focus exactly on what specific key analogies/properties you believe diverge. DL is extremely general - it's just efficient approximate bayesian inference over circuit spaces. It doesn't imply any specific architecture, and doesn't even strongly imply any specific approx inference/learning algorithm (as 1st and approx 2nd order methods are both common). Training to increase working memory capacity has near zero effect on IQ or downstream intellectual capabilities - see gwern's reviews and experiments. Working memory capacity is important in both brains and ANNs (transformers), but it comes from large fast weight synaptic capacity, not simple hacks. Noise is important for sampling - adequate noise is a feature, not a bug.
kman40

Of course if you combine gene edits with other interventions to rejuvenate older brains or otherwise restore youthful learning rate more is probably possible

We thought a bit about this, though it didn't make the post. Agree that it increases the chance of the editing having a big effect.

kman95

ANNs and BNNs operate on the same core principles; the scaling laws apply to both and IQ in either is a mostly function of net effective training compute and data quality.

How do you know this?

Genes determine a brain's architectural prior just as a small amount of python code determines an ANN's architectural prior, but the capabilities come only from scaling with compute and data (quantity and quality).

In comparing human brains to DL, training seems more analogous to natural selection than to brain development. Much simpler "architectural prior", vastly mo... (read more)

6jacob_cannell
From study of DL and neuroscience of course. I've also written on this for LW in some reasonably well known posts: starting with The Brain as a Universal Learning Machine, and continuing in Brain Efficiency, and AI Timelines specifically see the Cultural Scaling Criticality section on the source of human intelligence, or the DL section of simboxes. Or you could see Steven Byrne's extensive LW writings on the brain - we are mostly in agreement on the current consensus from computational/systems neuroscience. The scaling laws are extremely well established in DL and there are strong theoretical reasons (and increasingly experimental neurosci evidence) that they are universal to all NNs, and we have good theoretical models of why they arise. Strong performance arises from search (bayesian inference) over a large circuit space. Strong general performance is strong performance on many many diverse subtasks which require many many specific circuits built on top of compressed/shared base circuits down a heirarchy. The strongest quantitative predictor of performance is the volume of search space explored which is the product of C * T (capacity and data/time). Data quality matters in the sense that the search volume quantitative function of predictive loss only applies to tasks similar enough to the training data distribution. No - biological evolution via natural selection is very similar to technological evolution via engineering. Both brains and DL systems have fairly simple architectural priors in comparison to the emergent learned complexity (remember whenever I use the term learning, I use it in a technical sense, not a colloquial sense) - see my first early ULM post for a review of the extensive evidence (greatly substantiated now by my scaling hypothesis predictions coming true with the scaling of transformers which are similar to the archs I discussed in that post). Whenever I use the word learning, without further clarification, I mean learning as in bayesian le
kman40

Repeat administration is a problem for traditional gene therapy too, since the introduced gene will often be eliminated rather than integrated into the host genome.

kman72

Mildly deleterious mutations take a long time to get selected out, so you end up with an equilibrium where a small fraction of organisms have them. Genetic load is a relevant concept.

kman30

It seems fairly straightforward to test whether a chromosome transfer protocol results in physical/genetic damage in small scale experiments (e.g. replace chromosome X in cell A with chromosome Y in cell B, culture cell A, examine cell A's chromosomes under a microscope + sequence the genome).

The epigenetics seems harder. Having a good gears-level understanding of the epigenetics of development seems necessary, because then you'd know what to measure in an experiment to test whether your protocol was epigenetically sound.

kman30

You probably wouldn't be able to tell if the fruit fly's development was "normal" to the same standards that we'd hold a human's development to (human development is also just way more complicated, so the results may not generalize). That said, this sort of experiment seems worth doing anyways; if someone on LW was able to just go out and do it, that would be great.

1Grant Demaree
That implies the ability to mix and match human chromosomes commercially is really far off I agree that the issues of avoiding damage and having the correct epigenetics seem like huge open questions, and successfully switching a fruit fly chromosome isn't sufficient to settle them Would this sequence be sufficient? 1. Switch a chromosome in a fruit fly Success = normal fruit fly development 2a. Switch a chromosome in a rat Success = normal rat development 2b. (in parallel, doesn't depend on 2a) Combine several chromosomes in a fruit fly to optimize aggressively for a particular trait Success = fruit fly develops with a lot of the desired trait, but without serious negative consequences 3. Repeat 2b on a rat 4. Repeat 2a and 2b on a primate Can you think of a faster way? It seems like a very long time to get something commercially viable
kman95

A working protocol hasn't been demonstrated yet, but it looks like there's a decent chance it's doable with the right stitching together of existing technologies and techniques. You can currently do things like isolating a specific chromosome from a cell line, microinjecting a chromosome into the nucleus of a cell, or deleting a specific chromosome from a cell. The big open questions are around avoiding damage and having the correct epigenetics for development.

4Grant Demaree
Maybe the test case is to delete one chromosome and insert another a chromosome in a fruit fly. Only 4 pairs of chromosomes, already used for genetic modifications with CRISPR Goal = complete the insertion and still develop a normal fruit fly. I bet this is a fairly inexpensive experiment, within reach of many people on LessWrong
kmanΩ010

From section 3.1.2:

C. The EU passes such a law. 90%

...

M. There’s nowhere that Jurgen Schmidhuber (currently in Saudi Arabia!) wants to move where he’s allowed to work on dangerously advanced AI, or he retires before he can make it. 50%

These credences feel borderline contradictory to me. M implies you believe that, conditional on no laws being passed which would make it illegal in any place he'd consider moving to, Jurgen Schmidhuber in particular has a >50% chance of building dangerously advanced AI within 20 years or so. Since you also believe the EU h... (read more)

[This comment is no longer endorsed by its author]Reply
1michaelcohen
I don't understand. Importantly, these are optimistically biased, and you can't assume my true credences are this high. I assign much less than 90% probability to C. But still, they're perfectly consistent. M doesn't say anything about succeeding--only being allowed. M is basically saying: listing the places he'd be willing to live, do they all pass laws which would make building dangerously advanced AI illegal? The only logical connection between C and M is that M (almost definitely) implies C.
kman10

Since I currently have the slack to do so, I'm going to try getting into a balanced biphasic schedule to start with. If I actually manage to pull it off I'll make another post about it.

kman10

If we consider the TM to be "infinitely more valuable" than the rest of our life as I suggested might make sense in the post, then we would accept whenever . We will never accept if  i.e. accepting does not decrease the description length of the TM.

kman10

Right. I think that if we assign measure inverse to the exponent of the shortest description length and assume that the  probability increases the description length of the physically instantiated TM by  (because the probability is implemented through reality branching which means more bits are needed to specify the location of the TM, or something like that), then this actually has a numerical solution depending on what the description lengths end up being and how much we value this TM compared to the rest of our life.

Say ... (read more)

1kman
If we consider the TM to be "infinitely more valuable" than the rest of our life as I suggested might make sense in the post, then we would accept whenever ϵ>2U+L−T. We will never accept if U+L≥T i.e. accepting does not decrease the description length of the TM.
kman32

I see. When I wrote

such a TM embedded in our physical universe at some point in the future (supposing such a thing is possible)

I implicitly meant that the embedded TM was unbounded, because in the thought experiment our physics turned out to support such a thing.

5Vladimir_Nesov
Ah I see, the problem was ambiguity between TM-defined-by-initial-state and TM-with-full-computation-history. Since you said it was embedded in physics, I resolved ambiguity in favor of the first option, also allowing a bit of computation to take place, but not all of it. But if unbounded computation fits in physics, saying that something is physically instantiated can become meaningless if we allow the embedded unbounded computations to enumerate enough things, and some theory of measure of how much something is instantiated becomes necessary (because everything is at least a little bit instantiated), hence the relevance of your point about description length to caring-about-physics.
kman10

physicality of the initial states of a TM doesn't make its states from sufficiently distant future any more physically computed

I'm not sure what you mean by this.

Let's suppose the description length of our universe + bits needed to specify the location of the TM was shorter than any other way you might wish to describe such a TM. So with the lottery, you are in some sense choosing whether this TM gets a shorter or longer description.

2Vladimir_Nesov
The argument for moral worth of physically instantiated details says that details matter when they are physically instantiated. Any theories about description lengths are not part of this argument. Caring about such things is an example of caring about things other than physical world. What I mean is that sufficiently distant states of the TM won't be physically instantiated regardless of how many times its early states get to be physically instantiated. Therefore a preference that cares about things based on whether they get to be physically instantiated won't care about distant states of the TM regardless of how many times its early states get to be physically instantiated. A preference that cares about things other than physical instantiation can of course care about them, including conditionally on how many times early states of a TM get to be physically instantiated. Which is sufficient to implement the thought experiment, but not necessary, since one shouldn't fight the hypothetical. If the thought experiment asks us to consider caring about unbounded TMs, that's the appropriate thing to do, whether that happens to hold about us in reality or not.
kman10

Suppose I further specify the "win condition" to be that you are, through some strange sequence of events, able to be uploaded in such a TM embedded in our physical universe at some point in the future (supposing such a thing is possible), and that if you do not accept the lottery then no such TM will ever come to be embedded in our universe. The point being that accepting the lottery increases the measure of the TM. What's your answer then?

5Vladimir_Nesov
That wouldn't matter in general, physicality of the initial states of a TM doesn't make its states from sufficiently distant future any more physically computed, so there is no "increasing the measure of the TM" by physical means. The general argument from being physically instantiated doesn't cover this situation, it has to be a separate fact about preference, caring about a TM in a way that necessarily goes beyond caring about the physical world. (This is under the assumption that the physical world can't actually do unbounded computation of undiluted moral weight, which it in principle might.)
kman10

Sure, having just a little bit more general optimization power lets you search slightly deeper into abstract structures, opening up tons of options. Among human professions, this may be especially apparent in mathematics. But that doesn't make it any less scary?

Like, I could have said something similar about the best vs. average programmers/"hackers" instead; there's a similarly huge range of variation there too. Perhaps that would have been a better analogy, since the very best hackers have some more obviously scary capabilities (e.g. ability to find security vulnerabilities).

2JBlack
It's definitely scary. I think it is somewhat less scary in general capabilities than for mathematics (and a few closely related fields) in particular. Most of the scary things that UFAI can do will - unlike mathematics - involve feedback cycles with the real world. This includes programming (and hacking!), science research and development, stock market prediction or manipulation, and targeted persuasion. I don't think the first average-human level AIs for these tasks will be immediately followed by superhuman AIs. In the absence of a rapid self-improvement takeoff, I would expect a fairly steady progression through from average human capabilities (though with weird strengths and weaknesses), through increasingly rare human capability and eventually into superhuman. While ability to play chess is a terrible analogy for AGI, it did follow this sort of capability pattern. Computer chess programs were beating increasingly more skilled enthusiasts for decades before finally exceeding the top grandmaster capabilities. In the absence of rapid AGI self improvement or a possible sudden crystallization of hardware overhang into superhuman AGI capability through software breakthrough, I don't much fear improvement curves in AI capability blowing through the human range in an eyeblink. It's certainly a risk, but not a large chunk of my total credence for extinction. Most of my weight is on weakly superhuman AGI being able to improve itself or successors into strongly superhuman AGI.
kman10

It's certainly plausible that something like this pumps in quite a bit of variation on top of the genetics, but I don't think it detracts much from the core argument: if you push just a little harder on a general optimizer, you get a lot more capabilities out.

kman22

Specialization on different topics likely explains much more than algorithmic tweaks explain.

That the very best mathematicians are generally less specialized than their more average peers suggests otherwise.

3PeterMcCluskey
I had in mind an earlier and somewhat more subtle type of specialization, along the lines of what Henrich discusses in WEIRDest People. An example is that people who learn to read at an early age tend to have poorer facial recognition, and more of the abstract cognitive skills that are measured by IQ test. This kind of difference likely alters a nontrivial amount of learning over a period of 15 or so years before people start thinking about specializations within higher math.
kman32

There are other reasons why top mathematicians could have better output compared to average mathematicians. They could be working on more salient problems, there's selection bias in who we call a "top mathematician", they could be situated in an intellectual microcosm more suitable for mathematical progress, etc.

Do you really think these things contribute much to a factor of a thousand? Roughly speaking, what I'm talking about here is how much longer it would take for an average mathematician to reproduce the works of Terry Tao (assuming the same prior inf... (read more)

kman21

I doubt you could use numpy to compute this efficiently, since (afaik) numpy only gives you a big speedup on very regular vector/matrix type computations, which this is not.

kman20

Do you think it would be a good idea to delete this and repost it at the beginning of August?

Alternatively, I could repost an easier version in a month, since I'd be shocked if anyone solved this one. Though I guess that was part of the point -- to show that induction of fairly simple programs is super hard for humans in general. The previous challenge was too easy because each element in the output sequence was a simple function of two elements earlier in the sequence (and the 'elements' were easy to identify as they were output in the standard floating point format). On the other hand, it would be neat to make tough-but-solvable program induction challenges a recurring monthly thing as you suggested. Thoughts?

4jaspax
I don't know if it's necessary to delete, but I'll bet you'll get a lot more uptake if you repost another in August.
5Lone Pine
What I'd like to see is an ordinary piece of media (i.e. an image), but reformatted in an 'alien' way. Change RGB to the NTSC color system, then order the pixels in a non-cartesian way (or just swap rows and columns, or something), then write the data in your own invented number format, then compress it using an atypical algorithm, then add some arbitrary bytes to the front representing metadata, etc. Maybe when delivering it to the community, tell them it's an image, and see how long before people figure it out it's an image of a tiger or whatever.
kman20

This is true, but the farther out into the tails of the distribution we get the more likely we are to see negative effects that from traits that aren't part of the index we're selecting on.

True, but we wouldn't need to strictly select for G by association with IQ via GWASes. I suspect G variation is largely driven by mutation load, in which case simply replacing each rare variant with one of its more common counterparts should give you a huge boost while essentially ruling out negative pleiotropy. To hedge your bets you'd probably want to do a combined app... (read more)

kman20

Even the brightest geniuses don't really start having much of an impact on a field until about 20. And it takes further time for ideas to spread, so perhaps they'd need to reach the age of 30.

We could probably create humans vastly smarter than have ever previously existed with full genome synthesis, who could have a huge impact at a much younger age. But otherwise I agree.

Another short-term technology not even mentioned on your list is gamete sequencing. Sperm and eggs are produced in groups of four, with two complementary pairs per stem cell. If we could

... (read more)
2GeneSmith
This is true, but the farther out into the tails of the distribution we get the more likely we are to see negative effects that from traits that aren't part of the index we're selecting on. For example, I would be pretty surprised if we could increase IQ by 10 standard deviations in one generation without some kind of serious deleterious effects. I have to admit, I haven't actually done the math here, but Gwern seems to think it would roughly double the effect. Yeah, this is one of my hopes. I will probably write something about this in the future. I mostly think the value would be in more actual understanding of alignment difficulties among people working on AI capabilities. Thanks for the response.
kman30

(Edited because I don't think my original terse reply made my thoughts on this very clear)

If we're in a (very) long timeline world, I suspect the default thing that ends up happening is that embryo selection is gradually adopted, and G slowly rises population-wide. The reason timelines are long in such a world is that AGI ended up being way harder than it currently looks, so the gradually rising G levels would indeed increase the probability that unaligned AGI is created, unless this somewhat-higher-G world also manages to get large scale coordination righ... (read more)

kman20

Good point, I didn't address this at all in the post. Germline editing is indeed outside the current Overton window. One thing I'm curious about is whether there are any shreds of hope that we might be able to accelerate any of the relevant technical research: one thing this implies is not specifically focusing on the use case of enhancement, to avoid attracting condemnation (which would risk slowing existing research due to e.g. new regulations being levied).

For some techniques this seems harder than for others: iterated embryo selection is pretty clearly... (read more)