This is super cool stuff, thank you for posting!
I may have missed this, but do these scoring rules prevent agents from trying to make the environment more un-predictable? In other words, if you're competing against other predictors, it may make sense to influence the world to be more random and harder to understand.
I think this prediction market type issue has been discussed elsewhere but I can't find a name for it.
You can absolutely harvest potential energy from the solar system to spin up tethers. ToughSF has some good posts on this:
https://toughsf.blogspot.com/2018/06/inter-orbital-kinetic-energy-exchanges.html https://toughsf.blogspot.com/2020/07/tethers-all-way.html
Ideally your tether is going to constantly adjust its orbit so it says far away from the atmosphere, but for fun I did a calculation of what would happen if a 10K tonne tether (suitable for boosting 100 tonne payloads) fell to the Earth. Apparently it just breaks up in the atmosphere and produces very...
The launch cadence is an interesting topic that I haven't had a chance to tackle. The rotational frequency limits how often you can boost stuff.
Since time is money you would want a shorter and faster tether, but a shorter time of rotation means that your time window to dock with the tether is smaller, so there's an optimization problem there as well.
It's a little easier when you've got catapults on the moon's surface. You can have two running side by side and transfer energy between them electrically. So load up catapult #1, spin it up, launch the payload, and then transfer the remaining energy to catapult #2. You can get much higher launch cadence that way.
Lunar tethers actually look like they will be feasible sooner than Earth tethers! The lack of atmosphere, micrometeorites, and lower gravity (g) makes them scale better.
In fact, you can even put a small tether system on the lunar surface to catapult payloads to orbit: https://splittinginfinity.substack.com/p/should-we-get-material-from-the-moon
Whether tethers are useful on the moon depends on the mission you want to do. Like you point out, low delta-V missions probably don't need a tether when rockets work just fine. But if you want to take lunar material ...
Thanks for the comments! Going point-by-point:
I think both fiberglass and carbon fiber use organic epoxy that's prone to UV (and atomic oxygen) degradation? One solution is to avoid epoxy entirely using parallel strands or something like a Hoytether. The other option is to remove old epoxy and reapply over time, if its economical vs just letting the tether degrade.
I worry that low-thrust options like ion engines and sails could be too expensive vs catching falling mass, but I could be convinced either way!
Yeah, some form of vibration damping will
Yeah, my overall sense is that using falling mass to spin the tether back up is the most practical. But solar sails and ion drives might contribute too, these are just much slower which hurts launch cadence and costs.
The fact that you need a regular supply of falling mass from e.g. the moon is yet another reason why tethers need a mature space industry to become viable!
Getting the Hessian eigenvalues does not require calculating the full Hessian. You use Jacobian vector product methods in e.g. JAX. The Hessian itself never has to be explicitly represented in memory.
And even assuming the estimator for the Hessian pseudoinverse is cheap and precise, you'd still need to get its rank anyway, which would by default be just as expensive as getting the rank of the Hessian.
You may find this interesting "On the Covariance-Hessian Relation in Evolution Strategies":
https://arxiv.org/pdf/1806.03674
It makes a lot of assumptions, but as I understand it if you: a. Sample points near the minima [1]. b. Select only the lowest loss point from that sample and save it. c. Repeat that process many times d. Create a covariance matrix of the selected points
The covariance matrix will converge to the inverse of the Hessian, assuming the loss landscape is quadratic. Since the inverse of a matrix has the same rank, you could probably just use ...
I like the simple and clear model and I think discussions about AI risk are vastly improved by people proposing models like this.
I would like to see this model extended by including the productive capacity of the other agents in the AI's utility function. In other words, the other agents have a comparative advantage over the AI in producing some stuff and the AI may be able to get a higher-utility bundle overall by not killing everyone (or even increasing the productivity of the other agents so they can produce more stuff for the AI to consume).
Super useful post, thank you!
The condensed vaporized rock is particularly interesting to me. I think it could be an asset instead of a hindrance. Mining expends a ton of energy just crushing rock into small pieces for processing, turning ores into dust you can pump with air could be pretty valuable.
I was always skeptical of enhanced geothermal beating solar on cost, though I do think the supercritical water Quaise could generate has interesting chemical applications: https://splittinginfinity.substack.com/p/recycling-atoms-with-supercritical
This post has some useful info:
https://milkyeggs.com/biology/lifespan-extension-separating-fact-from-fiction/
It basically says that sunscreen, ceramide moisturizers, and retinols are the main evidence-based skincare products. I would guess that more expensive versions of these don't add much value.
Some amount of experimentation is required to find products that don't irritate your skin.
Good framing! Two forms of social credit that I think are worth paying attention to:
It's somewhat tangential, but Sarah Constantin discussing attestation has some i...
Note that these sorts of situations are perfectly foreseeable from the perspective of owners. They know precisely what they will pay each year in taxes based on their bid. It's prudent to re-value the home every once in a while if taxes drift too much, but the owner can keep the same schedule if they want. They can also use the public listing of local bids, so they know what to bid and can feel pretty safe that they will keep their home. They truly have the highest valuation of all the bidders in most cases.
The thing is, every system of land ownership face...
Land value taxation is designed to make land ownership more affordable by lowering the cost to buy land. Would it change the value of property as an investment for current owners? I'm not sure, one one hand, land values would go down, but on the other, land would get used more efficiently and deadweight loss of taxation would go down, boosting the local economy.
As for the public choice hurdles, reform doesn't seem intractable. Detroit is considering a split-rate property tax, and it's not infeasible that other places switch. Owners hate property taxes and ...
So yes, taxing property values is undesirable, but it also happens with imperfect land value assessments: https://www.jstor.org/stable/27759702
It looks like you have different numbers for the cost of land, sale value of a house, and cost of construction. I'm not an expert, so I welcome other estimates. A couple comments:
Are there examples of ineffective drugs leading to increased FDA stringency? I'm not as familiar with the history. For example, people agree that Aducanumab is ineffective, has that cause people to call for greater scrutiny? (genuinely asking, I haven't followed this story much).
There are definitely examples of a drug being harmful that caused increased scrutiny. But unless we get new information that this drug is unsafe, that doesn't seem to be the case here.
I agree that the difference between disease-treating interventions (that happen to extend life) versus longevity interventions is murky.
For example, would young people taking statins to prevent heart disease be a longevity intervention?
https://johnmandrola.substack.com/p/why-i-changed-my-mind-about-preventing
See this post arguing that rapamycin is not a longevity drug:
https://nintil.com/rapamycin-not-aging
Broadly, I'm not too concerned with what we classify a drug as as long as its safe, effective, well-understood, and gets approved by regulatory aut...
I personally don't expect very high efficacy, and I do expect that Loyal will sell the drug for the next 4.5 years. However, as long as Loyal is clear about the nature of the approval of the drug, I think this is basically fine. People should be allowed to, at their own expense, give their pets experimental treatments that won't hurt them and might help them. They should also be able to do the same for themselves, but that's a fight for another day.
Agreed! Beyond potentially developing a drug, think Loyal's strategy has the potential to change regulations ...
Note: I'm not affiliated with Loyal or any other longevity organization, I'm going off the same outside information as the author.
I think there's a substantial chance that this criticism is misguided. A couple points:
The term "efficacy nod" is a little confusing, the FDA term is "reasonable expectation of effectiveness", which makes more sense to me, it sounds like the drug has enough promise that the FDA thinks its worth continuing testing. They may not have actual effectiveness data yet, just evidence that it's safe and a reasonable explanation for why i...
Large breed dogs often die of heart disease which is often due to dilated cardiomyopathy (heart becomes enlarged and can't pump blood effectively). This enlargement can come from hypertrophic cardiomyopathy (overgrowth of the heart muscle).
Dilated cardiomyopathy and hypertrophic cardiomyopathy are two different conditions that I've not seen co-occur. They are basically sign-flipped versions of each other.
Dilated cardiomyopathy is when heart tissue becomes weaker and thinner. It stretches out like an overfilled balloon, and can't beat with the same strength...
Thanks for writing this!
In addition to regulatory approaches to slowing down AI development, I think there is room for "cultural" interventions within academic and professional communities that discourage risky AI research:
https://www.lesswrong.com/posts/ZqWzFDmvMZnHQZYqz/massive-scaling-should-be-frowned-upon
Could someone help me collect the relevant literature here?
I think the complete class theorems are relevant: https://www.lesswrong.com/posts/sZuw6SGfmZHvcAAEP/complete-class-consequentialist-foundations
The Non-Existence of Representative Agents: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3302656
Representative Agents: https://en.wikipedia.org/wiki/Representative_agent
John Wentworth on Subagents: https://www.lesswrong.com/posts/3xF66BNSC5caZuKyC/why-subagents
Two arguments I would add:
Alignment applies to everyone, and we should be willing to make a symmetric commitment to a superintelligence. We should grant them rights, commit to their preservation, respect it's preferences, be generally cooperative and avoid using threats, among other things.
It may make sense t...
Standardization/interoperability seems promising, but I want to suggest a stranger option: subsidies!
In general, monopolies maximize profit by setting an inefficiently high price, meaning that they under-supply the good. Essentially, monopolies don't make enough money.
A potential solution is to subsidize the sale of monopolized goods so the monopolist increases supply to the efficient level.
For social media monopolies, they charge too high a "price" by using too many ads, taking too much data, etc. Because of the network effect, it would be socially benefi...
Frowning upon groups which create new, large scale models will do little if one does not address the wider economic pressures that cause those models to be created.
I agree that "frowning" can't counteract economic pressures entirely, but it can certainly slow things down! If 10% of researchers refused to work on extremely large LM's, companies would have fewer workers to build them. These companies may find a workaround, but it's still an improvement on the situation where all researchers are unscrupulous.
The part I'm uncertain about is: what percent of...
I think you're greatly underestimating Karpathy's Law. Neural networks want to work. Even pretty egregious programming errors (such as off-by-one bugs) will just cause them to converge more slowly, rather than failing entirely. We're seeing rapid growth from multiple approaches, and when one architecture seems to have run out of steam, we find a half dozen others, initially abandoned as insufficiently promising, to be highly effective, if they're tweaked just a little bit.
In this kind of situation, nothing short of a total freeze is sufficient to slow prog...
I like this intuition and it would be interesting to formalize the optimal charitable portfolio in a more general sense.
I talked about a toy model of hits-based giving which has a similar property (the funder spends on projects proportional to their expected value rather than on the best projects):
https://ea.greaterwrong.com/posts/eGhhcH6FB2Zw77dTG/a-model-of-hits-based-giving
Updated version here: https://harsimony.wordpress.com/2022/03/24/a-model-of-hits-based-giving/
Great post!!
I think the section "Perhaps we don’t want AGI" is the best argument against these extrapolations holding in the near-future. I think data limitations, practical benefits of small models, and profit-following will lead to small/specialized models in the near future.
https://www.lesswrong.com/posts/8e3676AovRbGHLi27/why-i-m-optimistic-about-near-term-ai-risk
Yeah I think a lot of it will have to be resolved at a more "local" level.
For example, for people in a star system, it might make more sense to define all land with respect to individual planets ("Bob owns 1 acre on Mars' north pole", "Alice owns all of L4" etc.) and forbid people from owning stationary pieces of space. I don't have the details of this fleshed out, but it seems like within a star system, its possible to come up with a sensible set of rules and have the edge cases hashed out by local courts.
For the specific problem of predicting planetary o...
I feel like something important got lost here. The colonists are paying a land value tax in exchange for (protected) possession of the planet. Forfeiting the planet to avoid taxes makes no sense in this context. If they really don’t want to pay taxes and are fine with leaving, they could just leave and stop being taxed; no need to attack anyone.
The “its impossible to tax someone who can do more damage than their value” argument proves too much; it suggests that taxation is impossible in general. It’s always been the case that individuals can do more damage...
... this would provide for enough time for a small low value colony, on a marginally habitable planet, to evacuate nearly all their wealth.
But the planet is precisely what's being taxed! Why stage a tax rebellion only to forfeit your taxable assets?
If the lands are marginal, they would be taxed very little, or not at all.
Even if they left the planet, couldn’t the counter strike follow them? It doesn’t matter if you can do more economic damage if you also go extinct. It’s like refusing to pay a $100 fine by doing $1000 of damage and then ending up in pri...
There are two possibilities here:
Nations have the technology to destroy another civilization
Nations don't have the technology to destroy another civilization
In either case, taxes are still possible!
In case 1, any nation that attempts to destroy another nation will also be destroyed since their victim has the same technology. Seems better to pay the tax.
In case 2, the Nation doesn't have a way to threaten the authorities, so they pay the tax in exchange for property rights and protection.
...Thus threatening the destruction of value several orders of
I imagine that these policies will be enforced by a large coalition of members interested in maintaining strong property rights (more on that later).
Its not clear that space war will be dominated by kinetic energy weapons or MAD:
These weapons seem most useful when entire civilizations are living on a single planet, but its possible that people will live in disconnected space habitats. These would be much harder to wipe out.
Any weapon will take a long time to move over interstellar distances. A rebelling civilization would have to wait thousands of ye
I haven't given this a thoroughly read yet, but I think this has some similarities to retroactive public goods funding:
https://harsimony.wordpress.com/2021/07/02/retroactive-public-goods-funding/
https://medium.com/ethereum-optimism/retroactive-public-goods-funding-33c9b7d00f0c
The impact markets team is working on implementing these:
Going by figure 5, I think the way to format climate contingent finance like an impact certificate would be:
... robust broadly credible values for this would be incredibly valuable, and I would happily accept them over billions of dollars for risk reduction ...
This is surprising to me! If I understand correctly, you would prefer to know for certain that P(doom) was (say) 10% than spend billions on reducing x-risks? (perhaps this comes down to a difference in our definitions of P(doom))
Like Dagon pointed out, it seems more useful to know how much you can change P(doom). For example, if we treat AI risk as a single hard step, going from 10% -> 1% or 99% ->...
Yes, "precision beyond order-of-magnitude" is probably a better way to say what I was trying to.
I would go further and say that establishing P(doom) > 1% is sufficient to make AI the most important x-risk, because (like you point out), I don't think there are other x-risks that have over a 1% chance of causing extinction (or permanent collapse). I don't have this argument written up, but my reasoning mostly comes from the pieces I linked in addition to John Halstead's research on the risks from climate change.
...You need to multiply by the amount of chan
Setting aside how important timelines are for strategy, the fact that P(doom) combines several questions together is a good point. Another way to decompose P(doom) is:
How likely are we to survive if we do nothing about the risk? Or perhaps: How likely are we to survive if we do alignment research at the current pace?
How much can we really reduce the risk with sustained effort? How immutable is the overall risk?
Though people probably mean different things by P(doom) and seems worthwhile to disentangle them.
...Talking about our reasoning for our pers
You may also want to check out John Wentworth's natural abstraction hypothesis work:
Interesting!
So if I am understanding correctly, SIA puts more weight on universes with many civilizations, which lowers our estimate of survival probability q. This is true regardless of how many expanding civs. we actually observe.
The latter point was surprising to me, but on reflection, perhaps each observation of an expanding civ also increases the estimated number of civilizations. That would mean that there are two effects of observing an expanding civ: 1) Increased the feasibility of passing a late filter 2) increasing the expected number of civiliza...
Some other order-of-magnitude estimates on available data, assuming words roughly equal tokens:
Wikipedia: 4B English words, according to this page.
Library of Congress: from this footnote a assume there are at most 100 million books worth of text in the LoC and from this page assume that books are 100k words, giving 10T words at most.
Constant writing: I estimate that a typical person writes at most 1000 words per day, with maybe 100 million people writing this amount of English on the internet. Over the last 10 years, these writers would have produced 370T ...
Right. Similar to a property tax, this would discourage land improvements somewhat (though unlike a property tax, it would not discourage non-land improvements like houses).
All land value taxes do something like this. In practice, the effect is small because individual changes to land values are dwarfed by changes caused by external factors like local economic growth.
Great idea, thanks for posting this!
I wrote a post on how to have productive disagreements with loved ones:
https://harsimony.wordpress.com/2022/06/21/winning-arguments-with-loved-ones/
Here is the subsection on analyzing disagreements:
...Because arguments are emotional, it can be helpful to try to dispassionately assess the situation with your partner and get to the root of the problem.
The first step is to break the disagreement down into isolated chunks. Identify the handful of differences you are having, and deal with them as independently as possible. If
Suppose I could buy some very cheap land, clear trees and scrub, remove rocks, construct a dam, fertilize the soil and so on, such that on the open market I could now lease it to a farmer at $50,000/year instead of nearly nothing.
Since the taxes are based on the sale price of the empty land you bought, your taxes in this case would remain the same despite the improvements (not 50k/year). But once you sold the land, the next owner would pay 50k/year, since they paid the true price at auction.
There is an incentive to improve land, but unfortunately this p...
Oh that makes sense!
If the predictors can influence the world in addition to making a prediction, they would also have an incentive to change the world in ways that make their predictions more accurate than their opponents right? For example, if everyone else thinks Bob is going to win the presidency, one of the predictors can bribe Bob to drop out and then bet on Alice winning the presidency.
Is there work on this? To be fair, it seems like every AI safety proposal has to deal with something like this.