All of Anon User's Comments + Replies
Oftentimes downvoting without taking time to commet and explain reasons is reasonable, and I tends to strongly disagree with people who think I owe an incompetent write an explanation when downvoting. However, just this one time I would ask - can some of the people downvoting this explain why?
It is true that our standard way of mathematically modeling things implies that any coherent set of preferences must behave like a value function. But any mathematical model of the world is new essarily incomplete. A computationally limited agent that cannot fully for...
This looks to be primarily about imports - that is, primarily taking into account Trump's new tariffs. I am guessing that Wall Street does not quite believe that Trump actually means it...
It would seem that my predictions of how Trump would approach this were pretty spot on... @MattJ I am curious what's your current take on it?
Why would the value to me personally of existence of happy people be linear in the number of them? Does creating happy person #10000001 [almost] identical to the previous 10000000 as joyous as when the 1st of them was created? I think value is necessary limited. There are always diminishing returns from more of the same...
> if you have a program computing a predicate P(x, y) that is only true when y = f(x), and then the program just tries all possible y - is that more like a function, or more like a lookup?
In order to test whether y=f(x), the program must have calculated f(x) and stored it somewhere. How did it calculate f(x)? Did it use a table or calculate it directly?
What I meant is that the program knows how to check the answer, but not how to compute/find one, other than by trying every answer and then checking it. (Think: you have a math equation, no idea how to solve for x, so you are just trying all possible x in a row).
Aligned with current (majority) human values, meaning any social or scientific human progress would be stifled by the AI and humanity would be doomed to stagnate.
Only true when current values are taked naively, because future progress is a part of current human values (otherwise we would not be all agreeing with you that preventing it would be a bad outcome). It is hard to coherently generalize and extrapolate the human values, so that future progress is included in that, but not necessarily impossible.
Your timelines do not add up. Individual selection works on smaller time scales than group selection, and once we get to a stage of individual selection acting in any non-trivial way on AGI agents capable of directly affecting the outcomes, we already lost - I think at this point it's pretty much a given that humanity is doomed on a lot shorter time scale that that required for any kinds of group selection pressures to potentially save us...
This seems to be making a somewhat arbitrary distinction - specifically a program that computes f(x) in some sort of a direct way, and a program that computes it in some less direct way (you call it a "lookup table", but you seem to actually allow combining that with arbitrary decompression/decoding algorithms). But realistically, this is a spectrum - e.g. if you have a program computing a predicate P(x, y) that is only true when y = f(x), and then the program just tries all possible y - is that more like a function, or more like a lookup? What about if you have first compute some simple function of the input (e.g. x mod N), then do a lookup?
1Yes, and I was attempting to illustrate why this is a bad assumption. Yes, LLMs subject to unrealistic limitations are potentially easier to align, but that does not help, unfortunately.
You ask a superintendent LLM to design a drug to cure a particular disease. It outputs just a few tokens with the drug formula. How do you use a previous gen LLM to check whether the drug will have some nasty humanity-killing side-effects years down the road?
Edited to add: the point is that even with a few tokens, you might still have a huge inferential distance that nothing with less intelligence (including humanity) could bridge.
Agreed on your second part. A part of Trump "superpower" is to introduce a lot of confusion around the bounds, and then convince at least his supporters that he is not really stepping over that where it should have been obvious that he does. So the category "should have been plainly illegal and would have been considered plainly illegal before, but now nobody knows anymore" is likely to be a lot better defined that "still plainly illegal". Moreover, Trump is much more likely to attempt the former than the latter - not because he actually cares about not do...
Yes, potentially less that ASI, and security is definitely an issue, But people breaching the security would hoard their access - there will be periodic high-profile spills (e.g. celebrities engaged in sexual activities, or politicians engaged in something inappropriate would be obvious targets), but I'd expect most of the time people would have at least an illusion of privacy.
I found Eliezer Yudkowsky's "blinking stars" story (That Alien Message — https://search.app/uYn3eZxMEi5FWZEw5) persuasive. That story also has a second layer of having the extra smart Earth with better functioning institutions, but at the level of intuition you are going for it is probably unnecessary and would detract from the message. I think imagining a NASA-like organisation dedicated to controlling a remote robot at say 1 cycle of control loop per month (where it is perhaps corresponding to 1/30 of a second for the aliens), showing how totally screwed up the aliens are in this scenario, then flipping it around, should be at least somewhat emotionally persuasive.
For the specific example of arguing in a podcast, would not you expect people to already be aware of a substantial subset of arguments from the other side, and so would not it be entirely expected that there would be 0 update on information that is not new, and so not as much update overall, if only a fraction of information is actually new?
Hm, not sure about it being broadcast vs consumed by a powerful AI that somebody else has at least a partial control over.
Getting to the national math Olympiad requires access to regional Olympiad first, then being able to travel. Smart kids from "middle of nowhere" places - exactly to the kinds of kids you want to reach - are more likely to participate in the cities tournament. I wonder whether kids who were eligible for the summer camp, but did not make it there are more of your target audience than those who participated in the camp.
P.S. my knowledge of this is primarily based on how things were ~35 years ago, so I could be completely off.
What about trying to use the existing infrastructure in Russia, e.g.
- Donating to school libraries of math magnet schools (starting with "usual suspects" of 57, 2, 43 in Moscow, 239 in St Petersburg, etc, and then going down the list)?
- Contacting a competition organizers (e.g. for тургор - турнир городов which tends to have a higher diversity of participants compared to the Olympiad system) and coordinating to use the books as prises for finalists?
Besides not having to reinvent the wheel, kids might be more open to the ideas if the book comes from a local, more readily trusted party.
Think MMORPGs - what are the chances of simulation being like that vs a simulation with just a few special beings, and the rest NPCs?. Even if you say it's 50/50, then given that MMORPG-style simulations have billions of observes and "observers are special" ones only have a few, then an overwhelming majority of simulates observers are actually not that special in their simulations.
Ah, OK, then would suggest adding it to both title and body to make it clear, and to not waste time of people what are not the audience for this.
Sorry, feedback on what? Where is your resume/etc - what information to you expect the feedback to be based on?
But here is actional feedback - when asking people to help you for free out of goodness of their hearts (including this post!), you need to get out of your way to make it as easy and straightforward for them as possibl. When asking for feedback provide all the relevant information collected in an easy to navigate package,with TLDR summaries, etc. When asking for a recommendation, introduction, etc provide brief talking points, with more detailed iinformation provided for context (and make it clear you do not expect them to need to review it, and it is provided "just in case you would find it helpful".
Interesting - your 40/20/40 is a great toy example to think about, thanks! And it does show that a simple instant runoff schema for RCV should not necessarily help that much...
I am not sure about the median researcher. Many fields have a few "big names" that everybody knows and who's opinions have disproportionate weight.
- Finally, we wouldn't get a second try - any bugs in your AIs, particularly the 2nd one, are very likely to be fatal. We do not know how to create your 2nd AI in such a way that the very first time we turn it on, all the bugs were already found and fixed.
- Also, human values, at least the ones we know how to consciously formulate, are pretty fragile - they are things that we want weak/soft optimization for, but would actually be very bad if a superhuman AI would hard-optimize. We do not know how to capture human values in a way that things would not go terribly wrong if the optimization is cranked to the max, and your Values AI is likely to not help enough, as we would not know what missing inputs we are failing to provide it (because they are aspects of our values that would only become important in some future circumstances we cannot even imagine today).
- We do not know how to create an AI that would not regularly hallucinate. The Values AI hallucinating would be a bad thing.
- In fact, training AI to closer follow human values seems to just cause it to say what humans want to hear, while being objectively incorrect more often.
- We do not know how to create an AI that reliability follows the programed values outside of a training set. Your 2nd AI going off the rails outside of the training set would be bad.
Do you care about what kind of peace it is, or just that there is some sort of peace? If latter, I might agree with you on Trump being more likely to quickly get us there. For former, Trump is a horrible choice. On of the easiest way for a US President to force a peace agreement in Ukraine is probably to privately threaten Ukranians to withhold all support, unless they quickly agree to Russian demands. IMHO, Trump is very likely to do something like that. The huge downside is that while this creates a temporary peace, it would encourage Russia to go for it...
Ability to predict how outcome depends on inputs + ability to compute the inverse of the prediction formula + ability to select certain inputs => ability to determine the output (within limits of what the influencing the inputs can accomplish). The rest is just an ontological difference on what language to use to describe this mechanism. I know that if I place a kettle on a gas stove and turn on the flame, I will get the boiling water, and we colloquially describe this as bowling the water. I do not know all the intricacies of the processes inside the w...
Perhaps you are missing the point of what I am saying here somewhat? The issue is is not the scale of the side-effect of a computation, it's the fact that the side-effect exists, so any accurate mathematical abstraction of an actual real-world ASI must be prepared to deal with solving a self-referential equation.
I think it's important to further refine the accuracy criterion - I think another very important criterion (particularly given today's state of US politics) is how conducive the voting system towards consensus-building vs polarization. In other words, not only pure accuracy matters, but the direction of the error as well. That is, an error towards a more extreme candidate is IMHO a lot more harmful than an equally sized error towards a more consensus candidate.
It seems you are overlooking the notion of superintelligence being able to compute through your decisionmaking process backwards. Yes, it's you who would be making the decision, but SI can tell you exactly what you need to hear in order for your decision to result in what it wants. It is not going to try to explain how it is manipulating you, it will not try to prove to you it is manipulating you correctly - it will just manipulate you. Internally, it may have a proof, but what reason would it have to show it to you? And if placed into some very constraine...
Your proof actually fails to fully account for the fact that any ASI must actually exist in the world. It would affect the world other then just through its outputs - e.g. if it's computation produces heat, that heat would also affect the world. Your proof does not show that the sum of all effects of the ASI on the world (both intentional + side-effects of it performing its computation) could be aligned. Further, real computation takes time - it's not enough for the aligned ASI to produce the right output, it also needs to produce it at the right time. You did not prove it to be possible.
The 3rd paragraph of the Wikipedia page you linked to seems to answer the very question you are asking:
Maximal lotteries do not satisfy the standard notion of strategyproofness [...] Maximal lotteries are also nonmonotonic in probabilities, i.e. it is possible that the probability of an alternative decreases when a voter ranks this alternative up
If your AGI uses a bad decision theory T it would immediately self-modify to use a better one.
Nitpick - while probably a tiny part of the possible design space, there are obvious counterexamples to that, including when using T results in the AGI [incorrectly] concluding T is the best, or otherwise not realizing this self-modification is for the best.
After finishing any task/subtask and before starting the next one, go up the hierarchy at least two levels, and ask yourself - is moving onto the next subtask still the right way to achieve the higher-level goal, and is it still the highest priority thing to tackle next. Also do this anytime there is a significant unexpected difficulty/delay/etc.
Periodically (with period defined at the beginning) do this for the top-level goal regardless of where you are in the [sub]tasks.
There are so many side-effects this overlooks. Winning $110 complicates my taxes by more than $5. In fact, once gambling winnings taxes are considered, the first bet will likely have a negative EV!
Your last figure should have behaviours on the horizontal axis, as this is what you are implying - you are effectively saying, any intelligence capable of understanding "I don't know what I don't know" will on.y have power seeking behaviours, regardless of what its ultimate goals are. With that correction, your third figure is not incompatible with the first.
I buy your argument that power seeking is a convergent behavior. In fact, this is a key part of many canonical arguments for why an unaligned AGI is likely to kill us all.
But, on the meta level you seem to argue that this is incompatible with orthogonally thesis? If so, you may be misunderstanding the thesis - the ability of an AGI to have arbitrary utility functions is orthogonal (pun intended) to what behaviors are likely to result from those utility functions. The former is what orthogonality thesis claims, but your argument is about the latter.
Your principles #3 and #5 are in a weak conflict - generating hypothesis without having enough information to narrow the space of reasonable hypotheses would too often lead to false positives. When faced with an unknown novel phenomena, one put to collect information first, including collecting experimental data without a fixed hypothesis, before starting to formulate any hypotheses.
I'm not involved in politics or the military action, but I can't help but feel implicated by my government's actions as a citizen here
Please consider the implications of not only being a citizen, but also taxpayer, and customer to other taxpayers. Through taxes, you work indirectly supports the Russian war effort.
I'm interested in building global startups,
If you succeed while still in Russia, what is stopping those with powerful connections from simply taking over from you? From what you say, it does not sound like you have connections of your own t...
1Option 5: the questioner is optimizing a metric other than what appears to be the post's implicit "get max info with minimal number of questions, ignoring communication overhead", which is IMHO a weird metric to optimize to begin with - not only it does not take length/complexity of each question into account, but is also ignoring things like maintaining answerer wilingness to continue answering questions, not annoying the answerer, ensuring proper context so that a question is not misunderstood, and this is not even taking into account the possiblity that while the questioner does care about getting the information, they might also simultaneously care about other things.
Looks like a good summary of their current positions, but how about willingness to update their position and act decisively and based on actual evidence/data? De Santis's history of anti-mask/anti-vaccine stances have to be taken into account, perhaps? Same for Kennedy?
I am not working on X because it's so poorly defined that I dread needing to sort it out.
I not working on X because I am at a loss where to start
I feel like admiring the problem X and considering all the ways I could theoretically start solving it, so I am not actually doing something to solve it.
For a professor at a top university, this would be easily 60+ hrs/week. https://www.insidehighered.com/news/2014/04/09/research-shows-professors-work-long-hours-and-spend-much-day-meetings claims 61hrs/week is average, and something like 65 for a full Professor. The primary currency is prestige, not salary, and prestige is generated by research (high-profile grants, high-profile publications, etc), not teaching. For teaching, they would likely care a lot more about advanced classes for students getting closer to potentially joining their research team, and...
So what system selects the best leader out of the entire population?
None - as Churchill said, democracy is the worst form of Government except for all those other forms that have been tried from time to time. Still, should be realistic when explaining the benefits.
One theory of democracy’s purpose is to elect the “right” leaders. In this view, questions such as “Who is best equipped to lead this nation?” have a correct answer, and democracy is merely the most effective way to find that answer.
I think this is a very limiting view of instrumental goals of democracy. First, democracy has almost no chance of selecting the best leader - at best, it could help select a better one out of a limited set of options. Second, this ignores a key, IMHO the key, feature of democracy - keeping leaders accountable after they are ...
I think the use of the term "AGI" without a specific definition is causing an issue here - IMHO the crux of the matter is the difference between the progress in average performance vs worst-case performance. We are having amazing progress in the former, but struggling with the latter (LLM hallucinations, etc). And robotaxis require an almost-perfect performance.
This makes assumptions that make no sense to me. Auto-GPT is already not passively safe, and there is no reason to be sure LLMs would remain myopic as they are scaled. LLMs are inscrutable matrixes of floating points that we are barely learning how to understand and interpret. We have no reliable way to predict when LLMs might hallucinate or misbehave in some other way. There is also no "human level" - LLMs are way faster than humans and are way more scalable than humans - there is no way to get LLMs that are as good as humans without having something that's way better than humans along a huge number of dimensions.
As a few commenters have already pointed out, this "strategy" completely fails in step 2 ("Specify safety properties that we want all AIs to obey"). Even for a "simple" property you cite, "refusal to help terrorists spread harmful viruses", we are many orders of magnitude of descriptive complexity away from knowing how to state them as a formal logical predicate on the I/O behavior of the AI program. We have no clue how to define "virus" as a mathematical property of the AI sensors in a way that does not go wrong in all kinds of corner cases, even less clu...
Wouldn't "outing" potential honeypots be extremely counterproductive? So yeah, if you know some - please keep it to yourself!