Mechanisms for Capped-reviewing Conferences
Four mechanisms for admitting submissions for review in a conference with capped reviewing capacity. The mechanisms are simulated using COLM data.
Posted on: October 27, 2026
[While I am on the COLM board, this writeup does not represent a COLM policy. It is here to air ideas, elicit thoughts, and get feedback. This is even though I use COLM data for simulation.]
I recently proposed that conferences should admit to review submissions according to their capacity, which is enforced through a pre-review mechanism that balances between serving the community and allowing for external contributions. This post suggests four mechanisms, and conducts rudimentary simulation using past COLM data.
The overall framework, initially set here, has the conference set its reviewing capacity, the number of submissions the program committee considers it can review in a relatively meaningful way. This quantity is split into two: X% are reserved to the community set, and Y% are randomly distributed. Submissions are only admitted within this capacity. In addition, authors have a meaningful and strict maximum cap.
Regardless of the mechanism, this phase must run enough time before the deadline so authors do not do work for nothing. For example, 2-3 weeks before the full paper deadline, authors make abstract submissions with a final author list. Admission decisions are sent promptly. A side effect is fewer last minute (and hastily prepared) submissions.
P0: Two Random Pools
The two sets are allocated at random, in order. First, the community set. Only submissions with an author that is a community member or an AC of that year are considered by the community set. Submissions are admitted through random uniform sampling up to the community set cap. All remaining submissions, including community-set eligible ones not yet sampled, go to the random set, where again submissions are admitted through uniform sampling up to the set of the cap. If fewer papers are eligible for the community set than there are slots, all of them are admitted and the unused community slots go to the random set. Throughout this process, and across both sets, the author submission cap is enforced, and once reached, all remaining submissions with that author are removed from consideration.
An advantage of P0 is its simplicity. It also allows to reward ACs and creates strong incentives for being one. Weaknesses include an incentive to flood with submissions, not way to signal quality, and an incentive to find a past author to add to submissions (although admitted submissions count against a cap). The randomness provides little certainty, even for community authors.
P1: Deterministic Credits for Past Accepted Papers
Every accepted paper of the two prior years gets 1 community credit for the current year. Credits are issued for each target year and expire with it. For example, the accepted papers of 2024 and 2025 create the credits for 2026. Every AC of the target year also gets one community credit. Each admission to review under the community set requires 1 credit, and the assignment of the credit is specified at submission time. Every community slot that is not used goes to the random set, in addition to its set size. Sampling from the random set is uniform. The per-author cap is enforced for both sets, and is strict. Even if a submission that exceeds the cap comes with eligible credit, it’s declined.
P1 is deterministic and transparent for community members. It allows community members to prioritize their highest quality work to use precious admission into what is hopefully a higher quality reviewing process. It also rewards ACs. Its biggest weakness is that it requires a stable acceptance rate, otherwise it creates compounding growth. It also requires coordination between authors about credit usage, although future potential credits can be assigned to specific authors when submissions are made, saving later drama (thank you, Tatsu, for this suggestion). Similar to P1, it incentivizes flooding the random set.
P2: Queued Admission Based on Past Authorship
Each submitted paper is assigned a number that represents its relative importance and is decided by the authors. Community set admissions (P) are decided by this loop:
P ← ∅
Every author of an accepted paper from last 2 years gets 1 credit (regardless of the number of papers)
Each AC gets \+1 to their credits (even if it was zero because they are not past authors)
Community authors are all authors with credits that have submissions
A ← shuffle community authors
while \|P\| \< k and there exists an author in A with an admissible paper:
a ← author with most credits (break ties based on shuffled ordering)
p ← top-importance un-admitted paper for author a (break ties randomly)
for a' in p.authors:
a'.credits –
P ← P ∪ {p}
return P
All remaining submissions are considered for the random set, where the admitted set is sampled uniformly. The per-author cap is enforced for both sets, and is strict.
P2, similar to P1, allows authors to signal quality, but is robust to them avoiding it. It does not require coordination, and provides a broad spread of admissions to review across individuals. It also rewards ACs. It dis-incentivizes adding co-authors with no meaningful contribution, because admission takes credits from all authors, thereby hopefully creating better authorship practices. Also similar to previous proposals is incentive to flood the random set. There is also an incentive for flooding by community authors, who can benefit from low submissions by the community. Finally, it is a relatively complex method, and that is a weakness on its own.
P3: Queued Admission with Credits per Year of Past Authorship
Identical mechanism to P2, except that credit assignment is accumulated across each of the two past years. For example, an author that had an accepted paper in the two previous years and is an AC will have 3 credits. P3 is relatively identical to P2 in its strengths and weaknesses, with the main difference being accounting for persistent authors.
Simulation
It is simple to simulate the implications of these mechanisms using COLM data from the past 3 years, with 2024 and 2025 as the historical year, and 2026 the current year. The proposals share certain parameters, and the simulation assumes certain values for them. The overall number of papers that can be admitted (1,500), the size of the community set (1,000), the size of the random set (500), and the cap of admitted submissions per author (5). The overall 1,500 value is set to create an artificial situation of many more submissions than capacity. The simulation also compares two baselines: random lottery without set distinction and a lottery with a per-author cap. The visual summary of the outcomes is below.
