Just spitballing here:
Promote the AI challenge as a rationalist meetup topic with the goal of having several working groups
Instead of trying to get one big group with a leader right from the start, appoint (or whatever) several leaders: assign to each leader a small collection of interested people
Be clear about what you want the leaders to do: what are the short and medium range goals
Put up an early post asking people to express interest and (maybe) skill-sets so that teams could be assembled with some balance / hope of accomplishing something
Keep in contact with the various leaders and see where people are getting stuck (I'm assuming that you are ultimately the person in charge of this project); periodically, have the leaders talk to each other -- but not extremely often; post regular discussion threads focusing on solving specific "We're stuck on this," problems
Try to reframe the problem or parts of the problem in a way that connects to generic rationality, so that non-programmers can contribute something -- looking over the old thread, it seems that a lot of people were intimidated by the threat of having to code stuff, but the programmers might nonetheless get a good idea or two from what non-programmers have to say about generic rationalist-type problems
Make some direct suggestions about the "worthwhile things" you mention. For example, apart from the AI project itself, what methods would you suggest site members use to cooperate and why? (Okay, maybe there isn't much more to be said directly about positive publicity and advancing AI ... but then, maybe there is ...)
Set benchmarks for when things should be done, even if those benchmarks have to be re-set several times along the way
Try to reframe the problem or parts of the problem in a way that connects to generic rationality, so that non-programmers can contribute something
This is harder than it sounds.
Late last year a LessWrong team was being mooted for the Google AI challenge (http://aichallenge.org/; http://lesswrong.com/r/discussion/lw/8ay/ai_challenge_ants/). Sadly, after a brief burst of activity, no "official" LessWrong entry appeared (AFAICT, and please let me know if I am mistaken). The best individual effort from this site's regulars (AFAICT) came from lavalamp, who finished around #300.
This is a pity. This was an opportunity to achieve, or at least have a go at, a bunch of worthwhile things, including development of methods of cooperation between site members, gathering positive publicity, and yes, even advancing the understanding of AI related issues.
So - how can things be improved for the next AI challenge (which I think is about 6 months away)?