Consider the intuitively simpler problem of "is something a universal turing machine?" Consider further this list of things that are capable of being a universal turing machine;

• Computers.
• Conway's game of life.
• Elementary cellular automata.
• Lots of Nand gates.

Even a sufficiently complex shopping list might qualify. And it's even worse, because knowing that A doesn't have personhood, and that B doesn't have personhood doesn't let us conclude that A+B doesn't have personhood. A single Transistor isn't a computer, but 3510 transistors might be a 6502. If we want to be 100% safe, we have to rule out anything we can't analyze, which means we pretty much have to rule out everything. We might as well make the function always return 1.

OK, as bad as that sounds, it just means we shouldn't work too hard on solving the problem perfectly, because we know we'll never be able to do so in a meaningful way. But perhaps we can solve the problem imperfectly. Spam assassin faces a very similar kind of problem, "how can we tell if a message is spam?" The technique it uses is conceptually simple; Pick a test that some messages pass and some fail. Use the test on a corpus of messages classified as spam and a corpus classified as non-spam, and use the results to assign a probability that a message is spam if it passes the test. In addition to the obvious advantage of "I can see how to do that for a non-person predicate test", such a test could also give a score for "has some person-like properties". Thus we can meaningfully approach the problem of A + B being a person even though A and B aren't by themselves.

What kind of tests can we run? Beats me, but presumably we'll have something before we can make an AI by design.

One problem with this approach is it could be wrong. It might even be very wrong. Also, training the predicate function might be an evil process - that is, training may involve purposely creating things that pass.