Web Submission and Review, Chair Documentation

Version 0.53 - September 2006

Table of Contents

  1. Installing the software
  2. First-time customization
  3. The administration page
  4. Managing the submission site
  5. Managing the review site
    1. Initial set-up
    2. Paper assignments and blocking access to papers
    3. Tracking progress
    4. Program-committee votes
    5. Wrapping up the PC work
  6. Managing the final-version submission site

1. Installing the software

One of the goals of this software was to make administration as easy as possible. In particular, you never need to run scripts etc. In principle, to install this software you just unpack the zip file in some directory in your web tree (call it the BASE directory), point your browser at that directory and off you go.

Some caveats

This software is currently a PHP/MySQL software, which means that to use it you must have a MySQL server and a PHP engine (and your web-server must supports PHP and MySQL). I tested the software with PHP 4.3.2 and MySQL 3.23 (both quite old now). It may work with even older versions (but it may not). Also, the proper functioning of this software depend on "proper" functioning of you web-server. In particular, two issues that I came across are: Experimenting with the software. I wrote a small testing package for the purpose of experimenting with the software. To use this package, you just need to unpack it in the same BASE directory as the software package itself (before you customize the installation). This testing package adds 53 dummy submissions and 17 dummy PC members to the conference database.

Troubleshooting. Most of the parameters of the software are stored in the file includes/confConstants.php under the BASE directory. (This file is generated by the customize page which is discussed below.) If you have problems with the software, looking at this file may provide some hints as to the source of the problem.

2. First-time customization

When you first point your browser to a virgin copy of the software, it takes you to a "customization page" where you customize this copy for your conference. In this page you specify things like the conference name, the PC chair email address, whether or not submissions are anonymous, and many other parameters. Nearly all the parameters that you specify in this page can later by modified from the conference administration page. Some fields in the customization form that require more explanation are described below:
Web Server. The DNS name or IP address of the web server, as accessed by the users of this software. This parameter is used when the software sends emails and needs to specify the address where some pages can be found. (Note that in some cases the administrator may use a different address. For example, the administrator uses "http://localhost/confName/" while everybody else uses "http://www.severName.org/confName/").

MySQL Server. This parameter tells the PHP scripts where to look for the MySQL server. In common installations the same host runs both the web-server and the MySQL server. If this is not the case (i.e., the web server communicates with a back-end database that is hosted on a different machine) then you need to check the radio-button for "The MySQL server runs on a different host" and provide the DNS name or IP address of the machine that hosts the MySQL server.

MySQL Administrator or MySQL Database and User. There are two ways to create a database for use with the submission and review software:


Email Settings. This section is meant to provide some workarounds common bugs and configuration errors in email systems. It currently consists of two fields:
Email header-line separation: RFC 2822 specifies that different lines in the email header should be separated using the two characters \r\n (CRLF). However, the documentation for the PHP mail function contains the following comment:
Note:If messages are not received, try using a LF (\n) only. Some poor quality Unix mail transfer agents replace LF by CRLF automatically (which leads to doubling CR if CRLF is used). This should be a last resort, as it does not comply with RFC 2822.
Accordingly, the default setting that is used in the software is CRLF, but you have the option to override that default.

Extra parameters for sendmail: It was observed that some ISP's filter as spam email messages where the From line does not match the "envelope sender". When email messages are sent from the web-server, the From line is set to the PC chair email address but the "envelope sender" is the web-server itself (e.g., apache@local-machine). This situation leads to loss of email messages in some cases. A counter-measure that sometimes helps is to explicitly tell the mail-transport-agent to set also the "envelope sender" to the chair's email address, using the extra parameter -f <chair-email>. The PHP documentation has the following comment about this method:
The user that the web server runs as should be added as a trusted user to the sendmail configuration to prevent a 'X-Warning' header from being added to the message when the envelope sender (-f) is set using this method. For sendmail users, this file is /etc/mail/trusted-users.
Also, this method does not work with PHP in "safe mode", so this parameter is ignored when the web-server is configured to use "safe mode" PHP.

Program Chair and Program Committee email. In this section you must enter the PC chair's email address, and you can optionally also specify the names and email addresses of the program committee members. For each PC member (or chair) you can specify either just an email address or also the name (in the form Name <email>). If you do not specify the chair's name, it will default to something like "<conf-name> Chair". For the other members, if you do not specify the name then it will remain blank.

The email addresses that you specify here will be used as the username for the purpose of accessing the review site. The names will be displayed on the reviews and discussion boards. All the names and addresses that you supply here can later be modified (or added or removed) from the administration page.

Supported Formats. The software is very permissive in terms of handling formats other than the ones that you specify here. If a submitter uploads a file in an unsupported format, the only thing that happens is that the extension unsupported is added to the file name. (For example, uploading a file xyz.txt, the file will be stored as xyz.txt.unsupported.

Reviewer Preferences. If you check this box, PC members will see a link on the review page asking them to specify their reviewing preferences. The chair will then be able to see all these preferences and can take them into account in assigning submissions to reviewers. Moreover, the chair will be given the option to let the software automatically compute an assignment or submissions to reviewers based on these preferences. UNchecking this box will turn off reviewer preferences, and also disable the automatic assignment option.

After Customization

Once you hit the Submit button at the bottom of the customization page, you are taken to a confirmation page where you can once again verify all the parameters. Then the software is customized with these parameters, and (if all goes well) you are presented with a receipt page. That page includes the chair's initial password and links to the submission page and the administration page. The submission site is now ready to receive submissions, and you can use the administration page to manage it.

3. The Administration Page

The administration page is what you use the manage the site and the entire submission and review process. The links on that page change according to the different phases of the process. You have one set of links during the submission phase, another during the review phase, and yet others during and after the final-version submission phase. You also have some "general purpose" links there for changing your password, looking at the error-log file, reading the documentation etc.

You also have an "Undo/Redo" link that lets you either undo the last change that you did from the administration page, or redo the last change that you undid. For example, if you closed the submission site and the decided to reopen it, you can use the Undo link to do that. Note, however, that only the last change that you made is kept, so you cannot undo multiple operations.

4. Managing the submission site

The administration page for the submission site is fairly simple. There are three things that you can do from that page: You will also see on the administration phase links to some pages that deal with review-period activities, such as setting the review guidelines, managing PC membership and setting up PC votes. These are described in more details below.

5. Managing the review site

Once you close the submission site, your are taken back to the administration page, where you can now administer the review site. At the top-right corner you still have a "Submission-site" box, now consisting only of one link that lets you revise or withdraw submissions if need be. (Note that you can also have submit new papers, just in case you want to accept a submission that arrived after the deadline.)

5a. Initial set-up

One of the first things that you would want to do after the deadline is letting the PC members access the review site. This is done from the "Manage PC membership" page. On that page you can add or remove PC members, reset the password of a PC member and modify the email address or the name that is recorded in the database for each member. The PC members can login to the site using their email address and password. When you reset the password of a PC member, an email message is sent to that member with the new password. (Also, changing the email address of a PC member automatically resets his/her password.) Note that it is possible for several people to use the same email address as long as they use different passwords (but all the passwords will be sent to that one email address).

Other things that you can do from the administration page is to create an archive file with all the submissions (so that reviewers do not need to download each submission separately), and edit the review-guideline page. The review guidelines page consists of the grade semantics, important dates in the committee work (such as the individual review deadline), and other review policies.

5b. Paper assignments and blocking access to papers

The software also lets you assign some submissions to each reviewer or block reviewers from accessing specific submissions (e.g., blocking PC members from accessing the reviews of their own papers).

If you did not disable that option, then PC members are given the option to access a Reviewing-Preferences page where they can specify what submissions they want or do not want to review, and they can also specify if they feel that they have conflict of interest with some submissions. All these preferences are made available to the chair, should you want to use them in assigning or blocking access to submissions.

Automatically Computing Paper Assignments

If you did not disable that option, you will find a link off the administration page that points to the auto-assignment page, where you can have the software compute an assignment of submissions to reviewers. The software uses a Stable-Marriage-like algorithm to generate this assignment, as described next.

Recall that in a (generalized) stable-marriage problem, you have a bipartite graph (which in our case represent reviewers and submissions) where each node on the left can potentially have preferences about nodes on the right and vise versa. Also, there are parameters that specify for each node X in the graph the maximum number of nodes from the other side that can be matched to X (i.e., the maximum degree of X). A solution to this problem is some subset of the edges of the complete bipartite graph that satisfies all the maximum-degree constraints. A solution is maximal if you cannot add any edges to it without violating the maximum-degree constraint of one of the nodes. A solution is called unstable is there exist two nodes X1, X2 on the left and two nodes Y1, Y2 on the right such that: (This is unstable since X1 and Y2 are motivated to dump Y1 and X2 respectively and elope together.) A solution is stable is there are no two such pairs. It is well known [1] that one can employ a greedy algorithm to compute a stable maximal solution in time proportional to the number of edges in the complete bipartite graph.

In our case, the PC members specify their preferences for the submissions that they want to review, thus establishing preferences from left to right. In the other direction, the chair is given the opportunity on the Auto-Assign page to specify preferences as to who should (or shouldn't) review what submission, thus establishing preferences from right to left. Note that specifying that "I prefer that Joe is assigned to submission 53" is different than manually assigning submission 53 to Joe. Specifically, this is just a hint to the software and is not binding, which means that you can specify as many preferences as you want without having to worry about load balancing and such like.

Once you got the preferences from all the PC members and also specified your own preferences, you are ready to let the software compute the initial assignments for you. Before the algorithm is run, however, you have to specify some parameters. In particular, you can specify a list of excluded reviewers that will be hidden from the view of the algorithm. This means that the algorithm will completely ignore these reviewers: they will not be assigned any submissions and existing assignments that they have will not be removed. Also, such existing assignments (if any) will not be counted against the number of required reviews per submission. You can use this option, for example, if one of the "PC members" on your list is the PC chair (to whom you do not want to assign any submissions).

Other parameters that you need to specify are how many reviewers are assigned for each submission and whether or not you want to keep existing assignments (if any) or start from scratch. Once you specify all the parameters, the software automatically computes an assignment of submissions to reviewers, and you are taken to the manual-assignment page where you can review and modify this assignment.

Manually Assigning Submissions to Reviewers

The manual-assignment page provides you with two different interfaces that you can use to specify the reviewing assignments. You can use either of these interfaces as you see fit, and once you "save the assignments" with one interface, the other interface is automatically updated as well.

Scratch Assignments vs. Visible Assignments

The assignments that are automatically computed by the software or set from the manual assignment page are stored as a "scratch copy" that is only visible to the chair but not to the reviewers. This allows the chair to play around with the assignments before making them publically available.

To make the assignments visible to the reviewers, the chair must check the box to "Make these assignments visible to the reviewers" next to the "Save Assignments" button on either the matrix or the list interface. Hitting the "Save Assignments" button with this box checked will copy the scratch assignments to the visible assignments, thus letting the reviewers see their assigned papers.

Note: As opposed to paper assignments, setting or clearing conflict-of-interests status for papers will immediately be visible to the reviewers.

5c. Tracking progress

After you assigned the reviewing duties, you can track the progress of the reviewing process off the administration page. The Overview page shows a matrix of all PC members and submissions, and each entry in that matrix shows whether a PC member is assigned to a submissions and whether it entered a review of it. If the PC member entered a review for the submission, you can click on the entry to see that review. At the bottom of the page you can manage the "discuss flags" that let PC members see each other's comments and participate in the discussions.

At any time you can go to the Set Submission Status page where you can modify the status of any submission. The available status values are the usual ones, namely "No status", "Reject", "Maybe Reject", "Discuss", "Maybe accept", and "Accept". At the top of the Overview and Set-Status pages you can see how many submissions are currently assigned any of the status values.

Also, at any time you can also go from the Administration page to the Review page (via the links at the top and bottom of the page), where you can participate in the review and discussions just like any other PC member.

5d. Program-committee votes

The software includes support for setting up and managing program-committee votes. You can have the committee vote on things such as "what papers to move from 'Discuss' to 'Maybe Accept'", or "which of the accepted papers should receive the best paper award", or even "who will give the invited talk at the conference". Managing the votes is done from the Voting page. On that page you can set the parameters for a new ballot, edit the paremeters of the ballots that are currently in progress, and see a list of all the ballots (in progrress and otherwise) with a link to the result page for each one.

You can det up a ballot to vote either on submissions or on "other things" (that you need to specify explicitly). Note that PC members can only vote on submissions when they are in the public-discussion phase of the review. Before that they can only vote on "ther things" (e.g., invited speakers).

When voting on submissions, you can either include all the submissions in the vote, include only submissions from specific categories (e.g., only submissions in the "Discuss" category), or explicitly list the submissions that you want to include in the vote. Note that if you choose to include in the vote, say, all the submissions in the "Discuss" category, then the submissions that the PC members see when they vote are the ones in the "Discuss" category at the time that they are voting. Put in other words: adding or removing submissions to the "Discuss" category in mid-vote will add or remove these submissions from the ballot.

When you set up ballots, the PC members will see a link from their review page that lets them participate in the current ballots, and you will be able to track the voting results off the Voting page. You can also modify the parameters of a vote, even when this vote is already in progress. Note that if you modify the list of submission-IDs or the list of "other things to vote on" in mid-vote, you must make sure that you do not modify the order of items, since the software identifies vote-items with their position in the list. For example, swapping the order of two items will result in each of them being assigned the tally of the other.

5e. Wrapping up the PC work

After all the decisions are made, the software lets you send to the authors the notification of decisions and the comments-to-authors that were entered during the review period. Just follow the appropriate link off the administration page, and you will be given the opportunity to edit these emails before they are sent. In addition to the contact authors, the chair is also CCed on all these emails.

Note that the Notification and Comments pages only send emails to contact authors of submissions whose status is either "Accept" or "Reject". These pages give you the choice of either sending emails to the contact authors of all the accepted and rejected submissions, or only to specific submissions that you explicitly list. This can be used to hold off on sending emails to some submissions (by setting their status to something other than "Accept" or "Reject") and later assigning the correct status and specifying these submission explicitly.

If you want to keep all the reviews and discussions for future reference, you can find a link off the administration page that packs all of them in one text ascii page that you can then save for your files.

Finally, you have a link to close the review site and activate the site for final-version submission. This link takes you to a confirmation page, where you can also edit the camera-version submission instructions that will be sent to the contact authors of accepted papers. (Once the review site is closed, PC members can still access the site and read the comments and discussions, but they can no longer modify them.)

6. Managing the final-version submission site

After you activated the final-version submission site, you still see a "Review site" box at the top right corner of the administration page, that lets you send emails to authors or change the status of submissions if need be. Other than that, you can see a list of accepted papers (with links to the final version when it is uploaded), and you can create (and then download) an archive file containing the final versions of all the accepted papers.

When the camera-ready deadline arrives, you can shutdown the final-version submission site. After you do that, you are given yet another link that automates the creation of the various indexes for the proceedings. Specifically, you are given the option of specifying the order in which the papers will appear in the proceedings and the length of each paper, and then the software generates for you a LaTeX file (using Springer llncs style) that includes the external-reviewer list, the list of program committee members, the table-of-contents and the author index.

7. Feedback

I will very much appreciate any comments that you can give me regarding this software. Please contact me at shaih@alum.mit.edu.
[1] Dan Gusfield , Robert W. Irving, The stable marriage problem: structure and algorithms, MIT Press, Cambridge, MA, 1989.