You don't need to simulate them perfectly, just to the level that you don't notice a difference. When the simulator has access to your mind, that might be a lot easier than you'd think.

There's also no need to create p-zombies, if you can instead have a (non-zombie) AI roleplaying as the people. The AI may be perfectly conscious, without the people it's roleplaying as existing.