Can you do something similar if I change my program to output NOT-HALT when it doesn't find a proof?

Consider a program that enumerates all proofs under PA of length up to N^c for some c. If it finds a contradiction, it loops forever, otherwise it halts. I have reasonable belief that it halts, but your fake oracle can't prove that it does using a proof of length at most N, if c is sufficiently large (see On the length of proofs of finitistic consistency statements in first order theories).

Or here's another way that doesn't depend on that result. Define program A as follows: Enumerate all proofs under PA up to length N. If it finds a proof for "program A halts", then it loops forever, otherwise it halts. If PA is consistent, then it must be that A halts but there's no proof for it under PA of length N or less.