Reviewing Conference Papers - Mark Bernstein

Please note that “X” and “Z” do not relate to yourseniority, rank, or experience at large, but only toyour relationship with the specific topic of the paper.A very junior graduate student might reasonably beexpected to review work on the topic of her thesiswith an “X”, while Turing Prize winners mightfrequently find themselves using the “Z”.The Narrative IsWhat MattersAs a practical matter, the written review is far moreinfluential than the rating. Try to ensure that thenarrative is consistent with the rating, and that bothaccurately reflect your opinions. Occasionally, tracesof indecision or waffling make the committee’s workexceptionally difficult, as people try to sort out thehidden meaning of comments.A: What did Prof. Clump say?B: Clump gave the paper a provisional accept. But look: almostall her comments are negative. Missing references,methodological problems, an objection to the survey technique.C: Well, Clump always is a stickler for technique.A: But it doesn’t sound like Clump liked much of anythingabout the paper. And Dexter just said “reject: unimaginativeand dull”B: Which tells us nothingA: Oh, dear.When your reaction to a paper is mixed, it is best toexplain both the perceived strengths and weaknesses— even though these might be so apparent to you asto need no mention.contribution, while another assumed that the theorywas well understood but found the implementationand analysis especially useful.Second, on rare occasions a review might be misfiled.Having a brief summary helps assure us that we’re alltalking about the same work.Third, a summary not only helps the committeeduring the meeting; it also convinces the authorsthat you have read the paper with some care andsympathy. Angry authors may be inclined to dismissyour critique as incompetent or absurd; a judicioussummary may help convince them to weight yourarguments thoughtfully.Major IssuesThe review comment then moves to discuss anymajor issues that arise in reading the paper.Remember that major issues need not bedisqualifying; often, they may be readily corrected.But major issues must be considered and addressed.A very useful distinction may be drawn betweenpapers that are actively pernicious, and those thatare simply not particularly good.BLUNDERS AND ERRORS OF FACTIf someone relied on these results to build a bridge,might it collapse?You discuss the behavior of meadowlarks which were observedin Colorado, Wyoming, and in South Dakota. The Dakotanbirds, however, might well be Eastern Meadowlark (sturnellamagna), while the other observations were doubtless WesternMeadowlark (sturnella neglecta). Though visually verysimilar, these are two distinct species.Start with a SummaryThe first sentence or two of your comments shouldsummarize the key point or argument of the paper.The summary, which might seem a tedious chore,serves two important roles. First, summaries oftenhelp explain divergent opinions. Perhaps one readersaw the paper as a theoretical study with a proof-ofconceptimplementation which made only a minorERRORS OF METHODOLOGYPlease be alert for methodological errors,particularly those that could lead to misleadinginterpretation of the results.You report that your students enjoyed using your system. Mightthey have wanted to please you? Might this be a Hawthorneeffect, a response to a welcome change of pace? Might they havechosen to study with you because you are the sort of person whowould create a system they would enjoy? Perhaps a study of adifferent population would yield new information.© Copyright 2008 by Mark Bernstein. All Rights Reserved.