The problem, which various people have pointed out, is that this implies an intelligence capable of taking over the world, but not capable of working out that when a human says pursue a certain goal, they would not want this goal to be pursued in a way that leads to the destruction of the world.

The entity providing the goals for the AI wouldn't have to be a human, it might instead be a corporation. A reasonable goal for such an AI might be to 'maximize shareholder value'. The shareholders are not humans either, and what they value is only money.

I find "false positive" and "false negative" also a bit confusing, albeit less so than "type I" and "type II" errors. Perhaps because of a programming background, I usually interpret 'false' and 'negative' (and '0') as the same thing. So is a 'false positive' something that is false but is mistaken as positive, or something that is positive (true), but that is mistaken as false (negative)? In other words, does 'false' apply to the postiveness (it is actually negative, but classified as positive), to being classified as positive (it is actually positive, but classified as positive)?

Perhaps we should call false positives "spurious" and false negatives "missed".

The link you provided contains absolutely no physics, as far as I can tell. Nor is there any math aside from some basic logic. So I am skeptical on whether this theory is correct (or even falsifiable).

Physics is built on top of mathematics, and almost all of mathematics can be built on top of ZFC (there are other choices). But there is as much time in ZFC as there are words in a single pixel on your screen.

Why would the price of necessities rise?

There are three reasons why the price might go up:

1. demand increases
2. supply decreases
3. inflation

Right now, everyone is already consuming these necessities, so if UBI is introduced, demand will not go up. So 1 would not be true.

Supply could go down if enough people stop working. But if this reduces supply of the necessities, there is a strong incentive for people on just UBI to start working again. There is also increasing automation. So I find 2 unlikely.

That leaves 3, inflation. I am not an economist, but as far as I understand this shouldn't be a significant factor.

Define the sequence S by

S(0) = 0
S(n+1) = 1 + S(n)

This is a sequence of natural numbers. This sequence does not converge, which means that the limit as n goes to infinite of S(n) is not a natural number (nor a real number for that matter).

You could try to write it as a function of time, S'(t) such that S'(1-0.5^n) = S(n). That is, S'(0)=0, S'(0.5)=1, S'(0.75)=2, etc. A possible formula is S'(t) = -log_2(1-t). You could then ask what is S'(1). The answer is that this is the same as the limit S(infinity), or as log(0), which are both not defined. So in fact S' is not a function from numbers between 0 and 1 inclusive to natural or real numbers, since the domain excludes 1.

You can similarly define a sequence of distributions over the natural numbers by

T(0) = {i -> 0.5 * 0.5^i}
T(n+1) = the same as T(n) except two values swapped

This is the example that you gave above. The sequence T(n) doesn't converge (I haven't checked, but the discussion above suggests that it doesn't), meaning that the limit "lim_{n->inf} T(n)" is not defined.

This question presupposes that the task will ever be done Sure. It's called super-tasks.

From mathematics we know that not all sequences converge. So the sequence of distributions that you gave, or my example of the sequence 0,1,2,3,4,... both don't converge. Calling them a supertask doesn't change that fact.

What mathematicians often do in such cases is to define a new object to denote the hypothetical value at the end of sequence. This is how you end up with real numbers, distributions (generalized functions), etc. To be fully formal you would have to keep track of the sequence itself, which for real numbers gives you Cauchy sequences for instance. In most cases these objects behave a lot like the elements of the sequence, so real numbers are a lot like rational numbers. But not always, and sometimes there is some weirdness.

From the wikipedia link:

In philosophy, a supertask is a countably infinite sequence of operations that occur sequentially within a finite interval of time.

This refers to something called "time". Most of mathematics, ZFC included, has no notion of time. Now, you could take a variable, and call it time. And you can say that a given countably infinite sequences "takes place" in finite "time". But that is just you putting semantics on this sequence and this variable.

What can one expect after this super-task is done to see?

This question presupposes that the task will ever be done. Since, if I understand correctly, you are doing an infinite number of swaps, you will never be done.

You could similarly define a super-task (whatever that is) of adding 1 to a number. Start with 0, at time 0 add 1, add one more at time 0.5, and again at 0.75. What is the value when you are done? Clearly you are counting to infinity, so even though you started with a natural number, you don't end up with one. That is because you don't "end up" at all.

Why do you believe that? And do you also believe that ZF is inconsistent?

Not about the game itself, but the wording of the questions is a bit confusing to me:

In the above network, suppose that we were to observe the variable labeled "A". Which other variables would this influence?

The act of observing a variable doesn't influence any of the variables, it would only change your beliefs about the variables. The only things influencing a variable are its parents in the Bayesian network.