All of dmav's Comments + Replies

Unfortunately, comparing the returns isn't a great way of evaluating the portfolio compared to the S&P 500. You should really be comparing their Sharpe ratios (or just annualized tstat). If you have, for example, 5% annualized returns on $x in excess of the risk-free rate, you can just double your pnl to 10% by borrowing $x more money and investing it (assuming you can borrow at a competitive rate). Why not do that? Well, you'll also have more variance in your portfolio. Probably what you really care about is risk-adjusted returns.

The most common way t... (read more)

This wouldn't really solve much of the problem though, since ETFs are still pretty expressive. For example, when they have a sense for whether an important clean-energy bill will pass or fail, they could buy/sell a clean-energy-tracking ETF.

Some ETFs are pretty high-weight Nvidia, so it would be pretty easy to still trade it indirectly, albeit a little bit less efficiently.

And honestly even the S&P500 will still move a lot based on various policy outcomes.

Just so you know, this is still missing on your personal site.
Also the image here doesn't exist on your personal site's post.
Thanks for writing all these wonderful resources Neel!

You probably also want to do some kind of normalization here based on how many total posts the user has upvoted. (So you can't just i.e. upvote everything.) (You probably actually care about something a little different from the accuracy of their upvoted-as-predictions on average though...)

Here's a good/accessible blog post that does a pretty good job discussing this topic. https://ericneyman.wordpress.com/2019/09/17/least-squares-regression-isnt-arbitrary/

I think that this is true of the original version of alphastar, but they have since trained a new version on camera inputs and with stronger limitations on apm (22 actions/5s) (Maybe you'd want some kind of noise applied to the inputs still, but I think the current state is much closer to human-like playing conditions.) See: https://www.deepmind.com/blog/alphastar-grandmaster-level-in-starcraft-ii-using-multi-agent-reinforcement-learning

In other words, we should be telling children 'be careful of roads/cars' (including on Halloween) Not 'be careful of Halloween'

I agree with the post, but I will point out that you really do need to emphasize the utility per micromort here. If you keep your utility constant, it is the total risk that matters. Just like if you were going to go on a long car ride tomorrow (on safer-than-usual roads, but not enough to outweigh the total driving) and someone points out you're much more likely to die than usual - sure, you can point out 'ah yes, but the chance I... (read more)

As you kind of say - there are already (at least decently smart/competent) people trying to do (almost) all of these things. For many of these projects, joining current efforts is probably a better allocation than starting your own effort, and most of the value to be added is if you're in the 99.5th+ %-ile (?) for the 'skills needed.' (or sometimes there's just not enough people working on a problem, or sometimes there's a place to add value if you're willing to do annoying work other people don't want to do - these are both rarer though, in the current fu... (read more)

Note that your prediction isn't interesting. Each year, conditioned on a doomsday not happening, it would be pretty weird for the date(s) to not have moved forward. 
Do you instead mean to say that you guess that the date will move forward each year by more than a year, or something like that?

Here are some objections I have to your post:
How are you going to specify the amount of optimization pressure the AI exerts on answering a question/solving a problem? Are you hoping to start out training a weaker AI that you later augment? 
If so, I'd be concerned about any distributional shifts in its optimization process that occur during that transition
If not, it's not clear to me how you have the AI 'be safe' through this training process.

At the point where you, the human, is labeling data to train the AI to identify concepts with measurements/feat... (read more)

I think the question of you/Adele miscommunicating is mostly under-specification of what features you want your test-AGI to have.

• If you throttle its ability to optimize for its goals, see EY and Adele's arguments.

• If you don't throttle in this way, you run into goal-specification/constraint-specification issues, instrumental convergence concerns and everything that goes along with it.

I think most people here will strongly feel a (computationally) powerful AGI with any incentives is scary, and that any test-versions should require using at-most a mu... (read more)