Anonymity Policy

The Critical Assessment of Genome Interpretation (CAGI) best benefits the community when the methods used can be studied relative to their performance. However, many groups have requested that they be able to submit anonymously to prevent any negative consequences of participating in CAGI.

We will allow most predictors to remain anonymous if they choose, with a few exceptions. For the benefit of the community, “top predictors” on any challenge will be publicly disclosed; the top predictors will be the top few methods or methods in the top plateau of performance. All anonymous predictions from the same group will be linked together, so it will be possible to see that a method that performed moderately well on one challenge may have performed poorly on another.

Predictors may chose anonymity on a submission-by-submission basis. For example, a prediction group may submit 6 submissions for a given challenge, of which 3 are anonymous and 3 are public. All submissions must include a detailed methods description which will be released only if the result is made public (either by submitter request or if a “top prediction.”) In addition submissions for which anonymity is requested need to include an “anonymous” methods which summarizes the type of approach used for the benefit of the community, but without sufficient detail to identify the submittors.

If a method’s performance in CAGI is to be described in an independent publication by its submitters, that method should be disclosed (not anonymous). Publication of such results by CAGI participants shall be interpreted by the organizers as permitting them to disclose that predictors’ results, so that the independent publication may be validated against actual CAGI results.

CAGI organizers will make best efforts to maintain anonymity when requested subject to the limitations above, but cannot guarantee it in any instance due to resource limitations.

Last updated: May 3, 2021