Don't expect people to commit up front to representing LessWrong. The idea of representing the entire community sounds kind of intimidating. In general don't stress benefits like getting positive publicity or learning important things about cooperation too much. I think that people who'll participate in a competition like that will be those who find it intrinsically rewarding. So stress the fun and coolness factor, I guess? Maybe have the people who participate initially write down the accounts of their early experiences in order to capture the imagination and draw in others.
It would be good if people could try their hand at participating in a team without too much commitment. This sounds hard and probably heavily depends on the structure of the particular programming problem and maybe rules of the competition.
Have a ready-to-use platform for cooperation so that people can immediately jump into team activities. Preparing such could amount to researching various websites providing such services, choosing the best combination of them and then writing a short tutorial on their use. In the extreme case, a group of volunteers could give the chosen solution a test run by cooperating on solving a problem from some prior programming contest and then writing down a report of their experience.
Late last year a LessWrong team was being mooted for the Google AI challenge (http://aichallenge.org/; http://lesswrong.com/r/discussion/lw/8ay/ai_challenge_ants/). Sadly, after a brief burst of activity, no "official" LessWrong entry appeared (AFAICT, and please let me know if I am mistaken). The best individual effort from this site's regulars (AFAICT) came from lavalamp, who finished around #300.
This is a pity. This was an opportunity to achieve, or at least have a go at, a bunch of worthwhile things, including development of methods of cooperation between site members, gathering positive publicity, and yes, even advancing the understanding of AI related issues.
So - how can things be improved for the next AI challenge (which I think is about 6 months away)?