Setting up this sort of experiment, especially in regard to poetry or other humanities topics, seems to be the overwhelming barrier.

We can take at face value that "Malley's" poems were created from phrases of a limited length selected at random (whatever that really means in this case) and then arranged in a random manner.

This setup would allow us to say that some modernist critics cannot distinguish a modernist poem written by a single person (although with possible allusions and cribbings) from one constructed with phrases less than a specified length from a specific pool of literature.

From what I have found on the affair, it is hard to see if there was much experimental design at all (a criticism that Sokal can share in):

"So, in a series of mischievous creative fugues, they gleaned lines from here and there, even from the American Armed Forces guide to mosquito infestation, and put it together in what they perceived to be a brilliant imitation of the new poetic genre. They dubbed the poet Ern Malley and to avoid the publishers seeking contact with him, they said that, like Keats, he had died young. They then invented his sister, Ethel, who “discovered” the poetry and decided to send it to Harris to judge it for literary merit." -(http://www.ernmalley.com/text_only.html)

In this specific case, we are stuck with two people who seemed to intentionally create a spoof of modernist poetry which is not a terrible representation of the genre. For a progressive journal to publishing something that was designed to make a strong attempt at passing as modernist poetry using the new technique of collage seems completely appropriate.

Does this seem like an adequate control poem for an experiment of this sort:

====

Night Piece

The swung torch scatters seeds
In the umbelliferous dark
And a frog makes guttural comment
On the naked and trespassing
Nymph of the lake.

The symbols were evident,
Though on park-gates
The iron birds looked disapproval
With rusty invidious beaks.

Among the water-lilies
A splash - white foam in the dark!
And you lay sobbing then
Upon my trembling intuitive arm

I suspect that a randomly generated poem from a large amount of source material would look significantly different. I tried out some google poem generators (which are probably not acceptable for this sort of experiment either), and the results weren't as nice <http://shawnrider.com/google/index.php?query=modernism&Submit=generate+poem>

In the end, problems with authorship and creation by collage are two of the widely recognized features of modernist poetry <http://en.wikipedia.org/wiki/Modernist_poetry_in_English>. The hoax seems to prove that some of modernist poetry's techniques are indeed effective.

I think your point about intentionally created spoofs like Nathicana coming out as good poetry drives home the point that these sorts of parodies aren't necessarily a good example of control poem construction.

Making these sorts of critiques brings in the distinction between being rational vs rationalizing <http://www.overcomingbias.com/2007/09/rationalization.html>. If you already have a point you want to prove and proceed to construct a method whereby you'll prove it, it isn't truly rational. If you spend a long time working on experimental design and becoming curious about how these methods (structural analysis of myths or modernist poetry) succeed/fail vs a random smattering of words and ideas, then you can build some rational knowledge on the matter.

While I like the idea of the spot the fakes test, I think it would be difficult to come up with good examples where the experimental design really leads to interesting conclusions with the scope of the project.