I would like to note that this dataset is not as hard as it might look like. Humans performed not so well because there is a strict time limit, I don't remember exactly but it was something like 1 hour for 25 tasks (and IIRC the medalist only made arithmetic errors). I am pretty sure any IMO gold medailst would typically score 100% given (say) 3 hours.
Nevertheless, it's very impressive, and AIMO results are even more impressive in my opinion.
Thanks, I think I understand your concern well now.
I am generally positive about the potential of prediction markets if we will somehow resolve the legal problems (which seems unrealistic in the short term but realistic in the medium term).
Here is my perspective on "why should a normie who is somewhat risk-averse, don't enjoy wagering for its own sake, and doesn't care about the information externalities, engage with prediction markets"
First, let me try to tackle the question at face value:
Good to know :)
I do agree that subsidies run into a tragedy-of-commons scenario. So despite subsidies are beneficial, they are not sufficient.
But do you find my solution to be satisfactory?
I thought about it a lot, I even seriously considered launching my own prediction market and wrote some code for it. I strongly believe that simply allowing the usage of other assets solves most of the practical problems, so I would be happy to hear any concerns or further clarify my point.
Or another, perhaps easier solution (I updated my original answer): just all...
Isn't this just changing the denominator without changing the zero- or negative-sum nature?
I feel like you are mixing two problems here: an ethical problem and a practical problem. UPD: on second thought, maybe you just meant the second problem, but still I think my response would be clearer by considering them separately.
The ethical problem is that it looks like prediction markets do not generate income, thus they are not useful and shouldn't be endorsed, they don't differ much from gambling.
While it's true that they don't generate income and are ze...
Why does it have to be "safe enough"? If all market participants agree to bet using the same asset, it can bear any degree of risk.
I think I should have said that a good prediction market allows users to choose what asset will a particular "pair" use. It will cause a liquidity split which is also a problem, but it's also manageable and, in my opinion, it would be much closer to an imaginary perfect solution than "bet only USD".
I am not sure I understand your second sentence, but my guess is that this problem will also go away if each market "pair" uses a single (but customizable) asset. If I got it wrong, could you please clarify?
In a good prediction market design users would not bet USD but instead something which appreciates over time or generates income (e.g. ETH, Gold, S&P 500 ETF, Treasury Notes, or liquid and safe USD-backed positions in some DeFi protocol).
Another approach would be to use funds held in the market to invest in something profit-generating and distribute part of the income to users. This is the same model which non-algorithmic stablecoins (USDT, USDC) use.
So it's a problem, but definitely a solvable one, even easily solvable. The major problem is that predi...
I am currently job hunting, trying to get a job in AI Safety but it seems to be quite difficult especially outside of the US, so I am not sure if I will be able to do it.
If I will not land a safety job, one of the obvious options is to try to get hired by an AI company and try to learn more there in the hope I will either be able to contribute to safety there or eventually move to the field as a more experienced engineer.
I am conscious of why pushing capabilities could be bad so I will try to avoid it, but I am not sure how far it extends. I understa...
I am currently job hunting, trying to get a job in AI Safety but it seems to be quite difficult especially outside of the US, so I am not sure if I will be able to do it.
This has to be taken as a sign that AI alignment research is funding constrained. At a minimum, technical alignment organizations should engage in massive labor hording to prevent the talent from going into capacity research.
The British are, of course, determined to botch this like they are botching everything else, and busy drafting their own different insane AI regulations.
I am far from being an expert here, but I skimmed through the current preliminary UK policy and it seems significantly better compared to EU stuff. It even mentions x-risk!
Of course, I wouldn't be surprised if it will turn out to be EU-level insane eventually, but I think it's plausible that it will be more reasonable, at least from the mainstream (not alignment-centred) point of view.
And compute, especially inference compute, is so scarce today that if we had ASI right now, it would take several decades, even with exponential growth, to build enough compute for ASIs to challenge humanity.
Uhm, what? "Slow takeoff" means ~1 year... Your opinion is very unusual, you can't just state it without any justification.
Preliminary benchmarks had shown poor results. It seems that dataset quality is much worse compared to what LLaMA had or maybe there is some other issue.
Yet another proof that top-notch LLMs are not just data + compute, they require some black magic.
Generally, I am not sure if it's bad for safety in the notkilleveryoneism sense: such things prevent agent overhang and make current (non-lethal) problems more visible.
Hard to say if net good or net bad, too many factors and the impact of each are not clear.
I am not sure how did you come to the conclusion that current models are superhuman. I can visualize complex scenes in 3D for example. Especially under some drugs :)
And I don't even think I have an especially good imagination.
In general, it is very hard to compare mental imagery with Stable Diffusion. For example, it it is hard to imagine something with many different details in different parts of the image but it is perhaps a matter of representation. An analogy could be that our perception is like a low-resolution display. I can easily zoom in on a...
It is easy to understand why such news could increase P(doom) even more for people with high P(doom) prior.
But I am curious about the following question: what if an oracle told us that P(doom) is 25% before the announcement (suppose it was not clear to the oracle what strategy will Anthropic choose, it was inherently unpredictable due to quantum effects or whatever).
Would it still increase P(doom)?
What if the oracle said P(doom) is 5%?
I am not trying to make any specific point, just interested in what people think.
I think it is not necessarily correct to say that GPT-4 is above village idiot level. Comparison to humans is a convenient and intuitive framing but it can be misleading.
For example, this post argues that GPT-4 is around Raven level. Beware that this framing is also problematic but for different reasons.
I think that you are correctly stating Eliezer's beliefs at the time but it turned out that we created a completely different kind of intelligence, so it's mostly irrelevant now.
In my opinion, we should aspire to avoid any comparison unless it has pra...
I can't agree more with the post but I would like to note that even the current implementation is working. It definitely grabbed people's attention.
My friend who never read LW writes in his blog about why we are going to die. My wife who is not a tech person and was never particularly interested in AI gets TikToks where people say that we are going to die.
So far it looks like definitely positive impact overall. But it's early to say, I am expecting some kind of shitshow soon. But even shitshow is probably better than nothing.
I agree it's a very significant risk which is possibly somewhat underappreciated in the LW community.
I think all three situations are very possible and potentially catastrophic:
Arguments against (1) could be "evil people are stupid" and "terrorism is not about terror".
Arguments against (1) and (2) could be "timelines are short" and "AI power is likely to be very concentrated".
I think that Deepmind is impacted by race dynamics and Google's code red etc. I heard from a Deepmind employee that the leadership including Demis is now much more focused on products and profits, at least in their rhetoric.
But I agree it looks like they tried and likely still trying to push back against incentives.
And I am pretty confident that they reduced publishing on purpose and it's visible.
I agree it was a pretty weak point. I wonder if there is a longer form exploration of this topic from Eliezer or somebody else.
I think it is even contradictory. Eliezer says that AI alignment is solvable by humans and that verification is easier than the solution. But then he claims that humans wouldn't even be able to verify answers.
I think a charitable interpretation could be "it is not going to be as usable as you think". But perhaps I misunderstand something?
Fwiw I live in London and have been to the Bay Area and I think that London is better across all 4 dimensions you mentioned.
you're misunderstanding the TIME article as more naive and less based-on-an-underlying-complicated-model than is actually the case.
I specifically said "I do not necessarily say that this particular TIME article was a bad idea" mainly because I assumed it probably wasn't that naive. Sorry I didn't make it clear enough.
I still decided to comment because I think this is pretty important in general, even if somewhat obvious. Looks like one of those biases which show up over and over again even if you try pretty hard to correct it.
Also, I think it's pretty hard...
I second this.
I think people really get used to discussing things in their research labs or in specific online communities. And then, when they try to interact with the real world and even do politics, they kind of forget how different the real world is.
Simply telling people ~all the truth may work well in some settings (although it's far from all that matters in any setting) but almost never works well in politics. Sad but true.
I think that Eliezer (and many others including myself!) may be suspectable to "living in the should-universe" (as named by...
I think that Eliezer (and many others including myself!) may be suspectable to "living in the should-universe"
That's a new one!
More seriously: Yep, it's possible to be making this error on a particular dimension, even if you're a pessimist on some other dimensions. My current guess would be that Eliezer isn't making that mistake here, though.
For one thing, the situation is more like "Eliezer thinks he tried the option you're proposing for a long time and it didn't work, so now he's trying something different" (and he's observed many others trying other thi...
People like Ezra Klein are hearing Eliezer and rolling his position into their own more palatable takes. I really don't think it's necessary for everyone to play that game, it seems really good to have someone out there just speaking honestly, even if they're far on the pessimistic tail, so others can see what's possible. 4D chess here seems likely to fail.
https://steno.ai/the-ezra-klein-show/my-view-on-ai
Also, there's the sentiment going around that normies who hear this are actually way more open to the simple AI Safety case than you'd expect, we've been...
2. I think non-x-risk focused messages are a good idea because:
3. There were cases when it worked well. For example, ...
I think it is only getting started. I expect that likely there will be more attention in 6 months and very likely in 1 year.
OpenAI has barely rolled out its first limited version of GPT-4 (only 2 weeks have passed!). It is growing very fast but it has A LOT of room to grow. Also, text-2-video is not here in any significant sense but it will be very soon.
OpenAI just dropped ChatGPT plugins yesterday. It seems like it is an ideal platform for it? Probably will be even easier to implement than before and have better quality. But more importantly, it seems that ChatGPT plugins will quickly shape to be the new app store and it would be easier to get attention on this platform compared to other more traditional ways of distribution. Quite speculative, I know, but seems very possible.
If somebody will start such a project, please contact me. I am ex-Google SWE with decent knowledge of ML and experience of running software startup (as co-founder and CTO in the recent past).
I would also be interested to hear why it could be a bad idea.
Probably not, from the paper: 'We used LeetCode in Figure 1.5 in the introduction, where GPT-4 passes all stages of mock interviews for major tech companies. Here, to test on fresh questions,
we construct a benchmark of 100 LeetCode problems posted after October 8th, 2022, which is after GPT-4’s pretraining period.'
Really helpful for learning new frameworks and stuff like that. I had a very good experience using it for Kaggle competitions (I am semi-intermediate level, probably it is much less useful on the expert level).
Also, I found it quite useful for research on obscure topics like "how to potentiate this not well-known drug". Usually, such research involves reading through tons of forums, subreddits etc. and signal to noise ratio is quite high. GPT-4 is very useful to distil signal because it basically already read this all.
Btw, I tried to make it solve competit...
Well, I do not have anything like this but it is very clear that China is way above GPT-3 level. Even the open-source community is significantly above. Take a look at LLaMA/Alpaca, people run them on consumer PC and it's around GPT-3.5 level, the largest 65B model is even better (it cannot be run on consumer PC but can be run on a small ~10k$ server or cheaply in the cloud). It can also be fine-tuned in 5 hours on RTX 4090 using LORA: https://github.com/tloen/alpaca-lora .
Chinese AI researchers contribute significantly to AI progress, although of course, t...
Uhm, I don't think anybody (even Eliezer) implies 99.9999%. Maybe some people imply 99% but it's 4 orders of magnitude difference (and 100 times more than the difference between 90% and 99%).
I don't think there are many people who think 95%+ chance, even among those who are considered to be doomerish.
And I think most LW people are significantly lower despite being rightfully [very] concerned. For example, this Metaculus question (which is of course not LW but the audience intersects quite a bit) is only 13% mean (and 2% median)
I don't think that Waluigi is an attractor state in some deeply meaningful sense. It is just that we have more stories where bad characters pretend to be good than vice versa (although we have some). So a much simpler "solution" would be just to filter the training set. But it's not an actual solution, because it's not an actual problem. Instead, it is just a frame to understand LLM behaviour better (in my opinion).
I think that RLHF doesn't change much for the proposed theory. A "bare" model just tries to predict next tokens which means finishing the next part of a given text. To complete this task well, it needs to implicitly predict what kind of text it is first. So it has a prediction and decides how to proceed but it's not discrete. So we have some probabilities, for example
On the surface level, it feels like an approach with a low probability of success. Simply put, the reason is that building CoEm is harder than building any AGI.
I consider it to be harder not only because it is not what everyone already does but also because it seems to be similar to AI people tried to create before deep learning and it didn't work at all until they decided to switch to Magic which [comparatively] worked amazingly.
Some people are still trying to do something along the lines (e.g. Ben Goertzel) but I haven't seen anything working at le...
Codex + CoT reaches 74 on a *hard subset* of this benchmark: https://arxiv.org/abs/2210.09261
The average human is 68, best human is 94.
Only 4 months passed and people don't want to test on full benchmark because it is too easy...
Formally, it needs to be approved by 3 people: the President, the Minister of Defence and the Chief of the General Staff. Then (I think) it doesn't launch rockets. It unlocks them and sends a signal to other people to actually launch them.
Also, it is speculated to be some way to launch them without confirmation from all 3 people in case some of them cannot technically approve (e.g. briefcase doesn't work/the person is dead/communication problems), but the details of how exactly it works are unknown.
It is goalpost moving. Basically, it says "current models are not really intelligent". I don't think there is much disagreement here. And it's hard to make any predictions based on that.
Also, "Producing human-like text" is not well defined here; even ELIZA may match this definition. Even the current SOTA may not match it because the adversarial Turning Test has not yet been passed.
They are simluators (https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators), not question answerers. Also, I am sure Minerva does pretty good on this task, probably not 100% reliable but humans are also not 100% reliable if they are required to answer immediately. If you want the ML model to simulate thinking [better], make it solve this task 1000 times and select the most popular answer (which is a quite popular approach for some models already). I think PaLM would be effectively 100% reliable.
Another related Metaculus prediction is
I have some experience in competitive programming and competitive math (although I was never good in math despite I solved some "easy" IMO tasks (already in university, not onsite ofc)) and I feel like competitive math is more about general reasoning than pattern matching compared to competitive programming.
P.S the post matches my intuitions well and is generally excellent.
So far 2022 predictions were correct. There is Codegeex and others. Copilot, DALLE-2 and Stable Diffusion made financial prospects obvious (somewhat arguably).
ACT-1 is in a browser, I have neural search in Warp Terminal (not a big deal but qualifies), not sure about Mathematica but there was definitely significant progress in formalization and provers (Minerva).
And even some later ones
2023
ImageNet -- nobody measured it exactly but probably already achievable.
2024
Chatbots personified through video and audio -- Replica sort of qualifies?
40% on MATH already reached.
I think you might find this paper relevant/interesting: https://aidantr.github.io/files/AI_innovation.pdf
TL;DR: Research on LLM productivity impacts in material disocery.
Main takeaways: