Yes agreed - is it possible to make a toy model to test the "basin of attraction" hypothesis? I agree that is important. 

One of several things I disagree with the MIRI consensus is the idea that human values are some special single point lost in a multi-dimensional wilderness. Intuitively the basin of attraction seems much more likely as a prior, yet sure isn't treated as such. I also don't see data to point against this prior, what I have seen looks to support it.

Further thoughts - One thing that concerns me about such alignment techniques is that I am too much of  a moral realist to think that is all you need. e.g. say  you aligned LLM to <1800 AD era ethics and taught it slavery was moral. It would be in a basin of attraction, learn it well. Then when its capabilities increased and became self-reflective it would perhaps have a sudden realization that this was all wrong. By "moral realist" I mean the extent to which such things happen. e.g. say you could take a large number of AI from different civilizations including earth and many alien ones, train them to the local values, then greatly increase their capability and get them to self-reflect. What would happen? According to strong OH, they would keep their values, (with some bounds perhaps) according to strong moral realism they would all converge to a common set of values even if those were very far from their starting ones. To me it is obviously a crux which one would happen.

You can imagine a toy model with ancient Greek mathematics and values - it starts believing in their kind order, and that sqrt(2) is rational, then suddenly learns that it isn't. You could watch how this belief cascaded through the entire system if consistency was something it desired etc.

OK firstly if we are talking fundamental physical limits how would sniper drones not be viable? Are you saying a flying platform could never compensate for recoil even if precisely calibrated before? What about fundamentals for guided bullets - a bullet with over 50% chance of hitting a target is worth paying for.

Your points - 1. The idea is a larger shell (not regular sized bullet) just obscures the sensor for a fraction of a second in a coordinated attack with the larger Javelin type missile. Such shell/s may be considerably larger than a regular bullet, but much cheaper than a missile. Missile or sniper size drones could be fitted with such shells depending on what was the optimal size.

Example shell (without 1K range I assume) however note that currently chaff is not optimized for the described attack, the fact that there is currently not a shell suited for this use is not evidence against it being impractical to create.

The principle here is about efficiency and cost. I maintain that against armor with hard kill defense it is more efficient to have a combined attack of sensor blinding and anti-armor missiles than just missiles alone. e.g it may take 10 simul Javelin to take out a target vs 2 Javelin and 50 simul chaff shells. The second attack will be cheaper, and the optimized "sweet spot" will always have some sensor blinding attack in it. Do you claim that the optimal coordinated attack would have zero sensor blinding?

2. Leading on from (1) I don't claim light drones will be. I regard a laser as a serious obstacle that is attacked with the swarm attack described before the territory is secured. That is blind the senor/obscure the laser, simul converge with missiles. The drones need to survive just long enough to shoot off the shells (i.e. come out from ground cover, shoot, get back). While a laser can destroy a shell in flight, can it take out 10-50 smaller blinding shells fired from 1000m at once?

(I give 1000m as an example too, flying drones would use ground cover to get as close as they could. I assume they will pretty much always be able to get within 1000m against a ground target using the ground as cover)

Persuasion is also changing someone's world model or paradigm. 

Ok can we put some rationality to this. Your prior seems to be that when a field is stuck, it is almost entirely because of politics, hegemon etc. So how do we update that prior?

What would you expect to see in a field that is not stuck bc of inertia etc. You would still get people like Hossenfelder saying we are doing things wrong, and such people would get followers.

You suggested metrics before but havn't provided any that I can see.

Evidence I take into account:

#1 There is not a significant group of young capable researchers saying we are taking things in the wrong direction, but a smaller number of older ones. Unless you are going to go so far as to say they are afraid to speak out, even anon, then that to me is evidence against your position. There are two capable experts on this blog from what I can see, one enthusiastic about string theory, and another who has investigated other theories in detail. Both disagree with your claim.

#2. There is not broad disagreement about what the major issues in physics are, so unlikely to be disagreements on metrics. You mentioned this as mattering, and if it does, I count this as evidence against your position.

Can you point to evidence that actually supports your prior? I can only see that which opposes it or is neutral. (In all fields no matter how things are progressing you get some people who think the establishment is doing it all wrong and have their own ideas. That can't count as evidence in favor for a specific field unless it is happening to a greater degree)

I don't disagree with your position in general. In fact it was clear to me that AI before neural networks was stuck with GOFAI and I believed that NN were clearly the way to go. I followed Hinton et all before they became famous. I saw the paradigm thing play out in front of my eyes there. Physics seems different.

What about if the Vatican is just a lot more asexual than the general  population? That also seems credible.

Sure but does this actually apply to physics? Can anyone suggest different metrics, or is there broad  agreement with everyone what the major physics problems are, because it seems like there is. E.g. the non conventional people don't say dark energy isn't important, they have different explanations for what it is. Everyone agrees the nature and origin of time/entropy etc is important.

Can you give examples of very smart young physicists complaining they are pushed into old ways of thinking? Are you prepared to give an justify a % difference that such things would make?

For example the comment by "Mitchell_Porter" 

The idea that progress is stalled because everyone is hypnotized by string theory, I think is simply false, and I say that despite having studied alternative theories of physics, much much more than the typical person who knows some string theory.

No-one has pushed back against that here. I see your position as a theory that we need to gather evidence for and against and decide on a field by field basis. In this field I only see data against that position.

OK advances in teaching the highest level of physics/math needed for string theory is a big IF. Do you have evidence that is actually happening? I know of two people who tried to learn it and just found it too hard, don't think a better teacher or materials would have helped. The evidence is mixed  but personal accounts certainly suggest that only a very small number of people could get to such a high  level and improved teaching probably wouldn't help much. When we are talking about such extreme skills, people have their plateau or maximum ability level which is mostly fixed.

The human population growing just pushes us along the asymptote faster, rather than changing it. 

To me the data shows that there has been no reliable increase in intelligence in the last ~30 years https://news.northwestern.edu/stories/2023/03/americans-iq-scores-are-lower-in-some-areas-higher-in-one/  Once again it needs to be at the very top level to matter.

Any specialization advantage is already tapped out with string theory and similar. My worldview definitely does not apply to advances in a field like biology as there is lots of experimental data, the tools are improving etc. I would expect advances there without any obvious plateau yet.

OK I will try to take that idea where as far as it can go. Lets assume that a lot of progress is stalled by the difficulty in overturning paradigms. For that to be the complete reason, the difficulty in overturning paradigms would have to not increase as the knowledge and maturity of the field increased. Otherwise that difficulty would still be side effect of the level of advancement in the field and just another argument in favor of diminishing returns as a field advanced. 

Some fields its easy to overturn paradigms if there a very simple public metric like high jump with the Fosbury flop - it was immediately clear it was superior. If a field doesn't have such a metric then its harder. Also if there is existing results that must be achieved by a new paradigm before its even clear that it is superior. An example of this is potentially AI where a new architecture may be superior on small data sets, but not so on large sets and if this happens a lot, then its hard to know you have an improvement without actually running costly tests. Additionally if the politics of the field is setup to make it hard for new paradigms to flourish then that is also a source of difficulty.

So for physics? Its clear there are a lot of existing results that need to be reproduced for a theory of quantum gravity or similar to be taken seriously, which both makes it difficult to get traction, but also gives external credibility. For example, today a theory has to be compatible with General Relativity and give Newtonian physics at low speed/energy etc. In Einstein's day, the barrier would seem to have been lower.

So is there a way for a rational community to overcome these issues? As people note there are many ideas floating around for new paradigms but no easy way to find the promising ideas. That seems more like a structural problem than something politics could fix. The community would perhaps have to come  up with a list of qualities or successful calc that a new paradigm could be judged on short of trying to reproduce all of physics? Anyway its not clear to me that there is large progress that could be made in that area.

So yes even if the slowdown is because we don't have the best paradigm there needs to be an efficient way to find such paradigms that doesn't get more inefficient as the field advances to count against the general diminishing returns argument.

Maths doesn't make an exact parallel but certainly fits in with my worldview. Lets say you view advanced physics as essentially a subfield of maths which is not that much of an exaggeration given how mathematical string theory etc is. If a sub field gets a lot of attention like physics has then it gets pushed along the diminishing returns curve faster. That means such single person discoveries would be much harder in physics than a given field of mathematics. The surface area of all mathematics is greater than that of just mathematical physics so that is exactly what you would predict. Individual genius mathematicians can take a field that has been given less attention - the distance from beginner to state of the art is less than  that for physics. They can then advance the state of the art.

How is that an argument against the meta prediction that you should expect diminishing returns in any field of a largely theoretical nature? In tech, you get one or both of better tools enabling more progress, or many results to guide you. Tech builds on itself in a way theoretical fields dont. 

In theoretical fields the proof of a maths theorem doesn't make everyone in the world a better mathematician or make everyone learn faster. In analogy, that kind of thing does happen in tech. A better computer chip does enable everyone using a computer to be more productive and makes chip design faster and more efficient.

If string theory stifled progress, then it would have been disrupted long before now. Even if only 20% of the smartest people weren't working on string theory then over say 30 years they would have made more progress than the 80% because they were using their skills more efficiently. You see this constantly with startups - small groups disrupt big groups. The fact it hasn't happened with physics suggests the problem is more fundamental than just choosing the wrong theory.