Differential Expression Score (DES)

The differential expression score evaluates how accurately a model predicts differential gene expression, a key output of most single-cell functional genomics experiments and a key input for downstream biological interpretation.

First, for each perturbation (predicted and ground truth), we calculate differential gene expression p-values between perturbed and control cells, using the Wilcoxon rank-sum test with tie correction. To define significant DE genes for prediction $G_{k,pred}$ and ground truth $G_{k,true}$ for perturbation $k$, we use the Benjamini-Hochberg procedure to control the False Discovery Rate at level $\alpha=0.05$. If the predicted set size $n_{k,pred}=|G_{k,pred}|$ is smaller or equal to the true set size, $n_{k,true}= |G_{k,true}|$, we define the Differential Expression Score as the intersection between the predicted and true sets, normalized to the size of the true set:

If $n_{k,pred} > n_{k,true}$, we employ a different calculation to avoid overpenalizing predictions that overestimate the significance of differential expression. We define the predicted DE gene set $\tilde{G}_{k,pred}$ by selecting $n_{k,true}$ genes with the largest absolute values of fold changes (with respect to control cells) from the full predicted significant set $G_{k,pred}$, and calculate the normalized intersection:

To obtain the overall score, we calculate the mean of $DES_k$ over all predicted perturbations.

Perturbation Discrimination Score (PDS)

Adapted from Wu et al., 2024, the perturbation discrimination score measures a model's ability to distinguish between perturbations by ranking predictions according to their similarity to the true perturbational effect, regardless of their effect size. First, we calculate pseudobulk expression, predicted $\hat y_k$ and true $y_k$, for each perturbation $k$ ($1\le k \le N$) by averaging the log1p-normalized expressions of all genes over all perturbed cells. Next, we calculate the Manhattan ($L1$) distance between a predicted perturbation $p$ and true perturbations $t$ and sort it in ascending order:

The target gene for each perturbation is excluded from the distance calculation. The index (rank) of the true perturbation in this ordered list, $argind\{d_{pt}\}_{t=p}$, is used to define the discrimination score by normalizing it to the total number of perturbations:

If the predicted perturbation has the minimal distance to the true perturbation, $PDS_p=1$.

Finally, the overall score is calculated as the mean of all predicted perturbation scores.

Mean Absolute Error (MAE)

To ensure that predictions are also evaluated across all genes, including those that are not differentially expressed, we include a third metric: mean absolute error (MAE). While MAE is less biologically interpretable, it captures overall predictive accuracy and provides a view of model performance across the entire gene expression profile. We use the standard definition of MAE, calculating the mean absolute difference between the pseudobulk predicted $\hat y_k$ and true $y_k$ expressions:

where $g$ are gene indexes and $G$ is the number of genes. To obtain the overall score, we calculate the mean of $MAE_k$ over all predicted perturbations.

Overall Score

Overall score on the leaderboard, and ultimately the scoring of final entries eligible for prizes, is determined by averaging the improvement of the three metrics relative to a baseline based on the cell-mean model of the training dataset.

Differential Expression Score (DES) and Perturbation Discrimination Score (PDS) range from 0 (worst) to 1 (best), so we define the scaled scores as

Here baseline scores are calculated for the cell-mean baseline model, which makes predictions by simply averaging the expression from all perturbations. The baseline scores are pre-calculated on the Training dataset and can be seen in the raw score table.

For mean absolute error, which has the best value of 0, we define the scaled score as:

If a scaled score is negative (i.e. prediction performs worse than the baseline), it’s clipped to 0. As defined above, the scaled scores range from 0 (performance equal or worse than the baseline) to 1 (exact match to the ground truth). Finally, we take the mean of the three scaled score to obtain the overall leaderboard score: