Paper Review Guidelines

Review Timeline

This document provides both general guidelines and track-specific guidelines for reviewing SIGCSE Technical Symposium Papers.

General Guidelines

Here are some recommendations for writing reviews of submitted papers that help the authors and improve the quality of the symposium.

• Papers submitted to the SIGCSE Technical Symposium are expected to be original work. If the work is not original or is not a sufficient advancement over prior work, please state so in the review or in the ‘confidential comments to the program committee’ section of the reviewer form.

• Your job as a reviewer is to write detailed reviews, even for excellent papers. In addition to telling the authors what you didn’t like about their paper, be sure to include what you did like as well.

• Even if your opinion is that the paper is poorly written or poorly thought-out, you can still provide constructive criticisms to help the authors, and in the long run, the conference. For such papers, think of your goal as convincing the authors to submit something else next year, but of such high quality that it will be well-reviewed and easily accepted. Give the authors advice on how to do that.

• The best reviews clearly justify the reviewer’s choice of rating. The least valuable review gives a low score with no written comments. That simply tells the authors that they have been unsuccessful, with no indication of how or why.

• Papers that you review are supposed to be anonymous. Your review should be based on the merits of the paper, not the reputation of the authors or their institutions. Therefore, we have asked the authors to remove all identifiable references to themselves. If the work is not anonymized, please let the Program Chairs know and continue to provide a fair and unbiased review. If you are unable to do so, please notify the Program Chairs.

• We realize that reviewers sometimes know the work and can guess who the authors of the papers might be. If you recognize the work, it is your responsibility to give a fair and unbiased review, using only the information in the paper. If you do not feel that you can give a fair, unbiased review of the paper and not the authors or institutions, please contact the Program Chairs immediately. Do NOT submit a review for the paper. The Program Chairs can reassign as necessary.

• Your review should not include comments to the authors about the anonymization (or lack thereof) in the paper. If you feel that it is necessary to comment on this, please use the text box, ‘Confidential comments to the committee’, in the reviewer form.

• We realize that occasionally anonymization requires the authors to remove information that affects your review (information that otherwise the paper appears to lack). As a reviewer, you can give the authors the benefit of the doubt. Use the ‘Confidential comments to the committee’ box to indicate this to us. Example “This paper should reference Valerie E. Taylor’s work, unless those references were removed for anonymity”).

• Please point out typographic and grammatical errors; if there are too many to list, please state so in your review.

• The focus of your review should be on content. If you find grammatical errors, try not let it distract you from reviewing the research/work presented in the paper.

• Although the SIGCSE Technical Symposium requires all submitted papers to be polished work and are to be reviewed as if they are the final publishable version, all authors of accepted papers get a brief opportunity to improve the presentation of their paper before the camera-ready copy is due. Your detailed feedback may help improve a paper and by extension improve the conference.

Substandard Reviews

The SIGCSE Technical Symposium uses a meta-review process after reviews have been submitted. Reviews that do not objectively, accurately, and clearly assess a paper’s suitability for publication at the SIGCSE Technical Symposium, or are not founded in the reviewer’s disciplinary expertise and on the basis of the written paper’s originality, technical soundness, contribution to computing education, and clarity of presentation, may be deleted.

For example, an unacceptable review might:

• be incoherent, unreadable, or irrelevant to the paper;

• focus on the paper’s topic area or presumed authors at the expense of assessing the paper itself; or

• provide no justification for its numeric ratings. (Even in “obvious” cases, reviewers should briefly justify ratings.)

Please note that a difference in rating or opinion with other reviewers or PC members will NEVER be cause for deletion of a review.

Reviewers with substandard reviews will be subject to the Recalcitrant Reviewers policy.

Examples of Good Reviews

To help reviewers better understand the qualities of good, useful reviews, here are several example comments, organized by review category:

Summary of Submission

Please summarize the submission in 2-4 sentences in your own words. Please DO NOT copy/paste the abstract into this section.

Strengths of this Submission

• This paper makes a very good argument in the introduction for why this course is needed. It is timely, and addresses a topic outside of the norm often seen at the SIGCSE Technical Symposium.

• I can’t recall ever seeing something similar at the SIGCSE Technical Symposium. This paper should generate a lot of discussion and have a good audience. It is a topic that many schools are trying to address (including mine.)

• Good level of detail on your approach. Table 2 is very handy. Under Section 2, it seems like log analysis and auditing may fit in your column two. How will you ensure additional security emphasis is implemented?

• The organization is faultless. It is very clear what the paper is going to say and how. The paper follows through with crystal clear subject headings and a logical flow of information.

Comments for Authors / Areas for Improvement

• The hypotheses are too obvious and the validation of them is not enough. Therefore, the contribution of this paper is quite limited.

• Hard to judge given the writing organization problems, but I do not see a lot of significance here. The verification that the laboratory helped more than on-line components alone is a nice result, if it is supported by the data. Having taught this course already and collected feedback on your approach makes the paper stronger.

• It is important for those who might be considering this approach to know that it can be successful. If I were considering this approach I would want to know if the students could understand the code, and how deeply I could get into the material given time constraints.

• I would have liked to see some discussion and references setting this work in the context of other studies of student learning and knowledge retention. While I don’t know of other studies that have examined exactly the phenomenon this paper does, a short search in the ACM digital library turned up these examples that are relevant…

• The paper could use additional proofing and polishing. I suggest finding a non-robotics person to read for both language and communication. Some sentences are poorly formed (e.g., sent. 1 of last par. in sec. 1). Some content seems misplaced (e.g., discussion of mobility in section 3).

Paper Track Guidelines

There are three tracks for papers. Reviewers and APCs will be assigned to review papers in one of the three tracks. Please ensure that the Program Chairs know your preferences (this is done in the reviewer signup form) for the track(s) where you can provide the most expertise and best feedback.

Authors must choose the track that they feel best fits their submission. Review the submission using the guidelines for the track the submission is in; not the track you would prefer it to be in. The Program Chairs will not move papers between tracks.

Computing Education Research Paper Track

Papers submitted to the Computing Education Research track describe an empirical computing education project.

Computing Education Research papers should adhere to rigorous standards, describing research questions, hypotheses, methods, results with statistical rigor (where applicable), and limitations, as is typical and expected of research studies. These papers normally focus on topics relevant to computing education with emphasis on educational goals and knowledge units/topics; instructional and learning methods or techniques in computing education; evaluation of pedagogical approaches; studies of the many different populations that are engaged in computing education, including (but not limited to) students, instructors, teaching assistants, tutors, advisors, etc; and issues of gender, diversity, equity, and underrepresentation. We welcome replication papers and papers that present null or negative results that meet the criteria below.

For a typical paper in this track, here are some key factors to include (as an author) and to look for (as a reviewer):

1. Are there one or more clearly stated research questions? Since the rest of the paper will be organized around these, it’s often good to put them in the abstract and in the first section of the paper.
2. Are the questions of interest to the SIGCSE Technical Symposium audience?
3. Related work in computing education
   • Is the relevant work in computing education included? If not, a good review must give references to missing material. Simply saying “The related work section is incomplete” is not enough.
   • Do the authors clearly describe the relationship between the previous work and the current research questions? In what ways does the current project build on the previous work, and how is it different?
4. Related work in educational theory
   • Is the project based in educational theory? If not, should it be and what are some theories the authors should consider?
   • Is the theory described clearly, with appropriate citations?
   • Is the theory’s relationship to the current project clearly described?
5. Is the data gathering sufficiently clearly described so that the reader could replicate it? The reporting tips by csedresearch.org have great recommendations on what data to gather and report. Some key information to include:
   • About the data: why this particular type of data is relevant to your research questions
   • About the participants: how many, what was their background (are they instructors, students, alumni, etc.); what if any formal coursework have they had in computing; how many were men and how many women; and any other factors that are relevant to the author’s project
   • About the person(s) gathering the data: What is their relationship to the participants? For example, if the data were collected from students in a class, was the instructor one of the researchers or not?
   • About the data gathering process: did the project use surveys, interviews, samples of student work, other; If surveys or interviews, exactly what questions were asked.
   • Six pages may not be sufficient to provide all the necessary details. Authors may link to supplemental materials that should be blinded for review. However, as a reviewer, you are not expected to review linked supplemental materials.
6. Is the data analysis process/methodology sufficiently described so that the reader could reproduce it?
   • What methodology was used?
   • Is the methodology described, with an appropriate citation?
   • Is the implementation of the methodology clearly enough described? How many people were involved? What process was used to resolve any disagreements?
   • Is the analysis process/methodology appropriate for answering the research questions?
7. Is the analysis methodology something new to computing education research that might be a contribution itself?
8. Are the results of the analysis clearly summarized?
9. Are the results thoroughly discussed, including:
   • Their relationship to the research questions
   • Their relationship to previous work
   • The implications of the results for teaching
   • The implications of the results for future research
10. Are threats to validity discussed?

Research Paper Resources

There are many resources for writing high quality papers for submission to the SIGCSE Technical Symposium. We encourage reviewers to read and evaluate papers from prior SIGCSE Technical Symposium, especially those designated as best papers, which were selected both due to content and high quality reporting. We have linked in additional resources that you may find useful as you review CS Education Research papers.

• Writing a research question (csedresearch.org)
• Reporting Tips (csedresearch.org)
• Checklist for Research Articles (csedresearch.org)
• Evaluation Instruments (csedresearch.org)

Experience Reports and Tools Paper Track

Experience Reports and Tools papers should carefully describe a computer science education intervention and its context, and provide a rich reflection on what worked, what didn’t, and why. This track accepts experience reports, teaching techniques, and pedagogical tools. All papers in this track should provide enough detail so that others could adopt the new innovation.

For a typical paper in this track, here are some key factors to include (as an author) and to look for (as a reviewer):

1. Are there one or more clearly stated goals in this paper? Since the rest of the paper will be organized around these, it’s often good to put them in the abstract and in the first section of the paper.
2. Is the experience or tool of interest to the SIGCSE Technical Symposium audience?
3. Related work in computing education
   • Is the relevant work in computing education included? If not, a good review must give references to missing material. Simply saying “The related work section is incomplete” is not enough.
   • Do the authors clearly describe the relationship between the previous work and the current research questions? In what ways does the current project build on the previous work, and how is it different?
4. Are the observations and/or findings from the experience or the use of a tool clearly summarized?
5. Are the findings thoroughly discussed, including:
   • Their relationship to previous work
   • The implications of the results for future use
   • The implications of the results for teaching
   • Information on how to adopt or adapt teaching techniques and/or pedagogical tools in other contexts or institutions.

Position and Curricula Initiatives Paper Track

Curricula initiatives should describe new curricula, programs, and degrees, the motivating context before the new initiative was undertaken, what it took to put the initiative into place, what the impact has been, and suggestions for others wishing to adopt it. This track may also include position papers, which are meant to engender fruitful academic discussion by presenting a defensible opinion about a computing education topic, substantiated with evidence.

1. Is the innovation clearly stated? Since the rest of the paper will be organized around this, it’s often good to put it in the abstract and in the first section of the paper.
   • Description of the problem or need being addressed.
2. Is the curricular innovation or position paper of interest to the SIGCSE Technical Symposium audience?
3. Related work in computing education
   • What prior solutions to this problem exist?
   • Is the relevant work in computing education included? If not, a good review must give references to missing material. Simply saying “The related work section is incomplete” is not enough?
   • Do the authors clearly describe the relationship between the previous work and the current research questions? In what ways does the current project build on the previous work, and how is it different?
4. Examination for discussion
   • How is the curricular innovation or position paper addressing this problem or need?
   • How is the curricular innovation or position paper different from previous ideas?
5. Future Success Indicators
   • How could the curricular innovation or proposed idea be assessed if adopted or implemented?
   • In what context can the curricular innovation or proposed idea be used (large research institutions, community colleges, high schools)?
   • How difficult would the curricular innovation or proposed idea be to adopt it? For example, the human and financial resources needed.

Here are some tips and good practices for reviewing gathered from the SIGCSE community during SIGCSE TS 2021.

While your review text should clearly support your scores and recommendation, please do not include your preference for acceptance or rejection of a submission in the feedback to the authors. Instead, use the provided radio buttons to make a recommendation (the authors will not see this) based on your summary review and provide any details that refer to your recommendation directly in the confidential comments to the APC or track chairs. Remember that as a reviewer, you will only see a small portion of the submissions, so one that you recommend for acceptance may be rejected when considering the other reviewer recommendations and the full set of submissions.

Discussion

The discussion and recommendation period provides the opportunity for the Track Chairs to discuss reviews and feedback so they can provide the best recommendation for acceptance or rejection to the Program Chairs and that the submission is given full consideration in the review process. We ask that Reviewers engage in discussion when prompted by other reviewers, the Track Chairs by using the Comments feature of EasyChair. During this period you will be able to revise your review based on the discussion, but you are not required to do so.

The Track Chairs will make a final recommendation to the Program Chairs from your feedback.

Recalcitrant Reviewers

Reviewers who don’t submit reviews, have reviews with limited constructive feedback, or who submit inappropriate reviews will be removed from the reviewer list (as per SIGCSE policy). Recalcitrant reviewers will be informed of their removal from the reviewer list. Reviewers with repeated offenses (two within a three year period) will be removed from SIGCSE reviewing for three years.