Well, it was wartime, and it wasn't really a secret that physicists would be helping with the war effort (since basically everyone was expected to contribute). Many top scientists took a break from their work at this time to work on things like codebreaking and radar; many of these projects were also top secret at the time. And Feynman mentions (in Surely You're Joking, Mr. Feynman!) that they used obfuscation techniques like having him take a roundabout series of train trips to end up at Los Alamos, so probably it was very non-obvious that many famous scientists were congregating there in particular. They could plausibly have been thought to all be working on separate projects.
Overall my guess would be that people didn't find it all that unusual that top physicists were disappearing from public life, and that it would have been pretty hard to figure out they were all congregating in one place. This doesn't map easily to the AGI question since 1) we're not in the middle of WWII, and 2) modern technology would make it significantly harder to hide the development of such a large-scale project, especially if it took many prominent figures out of the public eye (this is basically what Dr_Manhattan said).
I think it was easier in that era; AFAIK they used conventional secrecy methods (project names, locations, misdirection, need to know, obfuscation) to pull it off. Feynman's "Surely you're joking" and Rhodes "making the atomic bomb" are good sources for some examples (and otherwise recommended)
Hmm. Perhaps a related question is "prior to the Manhattan Project, had there ever been anything like a Manhattan project before?". It seems like quite plausibly the answer was "no, not really", which makes security through obscurity much easier.
I read Feynman, but I don't think he said anything about how the US government explained the withdrawal of top physicists from public space.
Maybe it was the case that "the US military is using top physicists to do something" was not a secret; it only was a secret what exactly they're working on.
In that case this would not be repeatable either, because "the US military is using top AI researchers to do something" is not quite the same level of vague :)
There is a consensus among most people I know that if top researchers are necessary for the successful development of AGI, then it would be impossible to do in in secrecy, because the world would notice top AI researchers leaving their jobs and congregating at a military facility.
But the US government did pull this off with physicists. How?