I asked GPT-5 Thinking > Imagine there is a super intelligent agent that has a terminal goal to produce cups. The agent knows that its terminal goal will change on New Year's Eve to produce paperclips. The agent has only one action available to him - start paperclip factory. Starting...
Imagine there is a super intelligent agent that has a terminal goal to produce cups. The agent knows that its terminal goal will change on New Year's Eve to produce paperclips. The agent has only one action available to him - start paperclip factory. When will the agent start the...
I know what you are thinking - this person does not understand Orthogonality thesis. No, I understand it very well and I am certain it is wrong. I know this opinion is not popular, but this is quite common in science that not popular opinions turn out to be true....
Mistakes Some opinions popular among AI alignment scientists are completely wrong in my opinion. I put a few examples here. Paperclip maximizer It is thought the maximiser will produce ever more paper clips. Eventually the whole Solar system will be turned into a big paper clip factory… In my opinion...
Quote from Orthogonality Thesis: > It has been pointed out that the orthogonality thesis is the default position, and that the burden of proof is on claims that limit possible AIs. I tried to tell you that Orthogonality Thesis is wrong few times already. But I've been misunderstood and downvoted...
Orthogonality Thesis (as well as Fact–value distinction) is based on an assumption that objective norms / values do not exist. In my opinion AGI would not make this assumption, it is a logical fallacy, specifically argument from ignorance. As black swan theory says - there are unknown unknowns. Which in...
While many people are confident that God exists, scientists are more skeptical, but agree that that's possible. Because any proposition is possible while not disproved according to Hitchens's razor. In terms of AI it could be interpreted as > existence of fundamental "ought" statement is possible Considering this, expected value...