It works by finding common genetics up to a limit of relatedness like fourth-cousin level. I think some Jewish groups may be sufficiently inbred/endogamous for long enough periods that it might not be possible to run GCTA with the usual cutoff since they'll all be too related to each other. Population structure beyond that is dealt with by the usual approach of subtracting out 10 or 20 principal components and including them to control for that. This is a bit ad hoc but does work well in GWASes and gets rid of that problem, as indicated by the fact that the hits replicate within-family where the population structure is equalized by design and also have a good track record cross-racially/country too: https://www.reddit.com/r/science/comments/4kf881/largestever_genetics_study_shows_that_genetic/d3el0p2