Multi-Class Classification

Equality of Outcome Metrics

Multiclass Statistical Parity: This metric computes statistical parity for a classification task with multiple classes and a protected attribute with multiple groups. For each group compute the vector of success rates for entering each class. Compute all distances (mean absolute deviation) between such vectors. Then aggregate them using the mean, or max strategy.

The accepted values and bounds for this metric are the same as the 1d case. A value of 0 is desired. Values below 0.1 are considered fair.

\[SP_{max} = \max_{g} \left| SR_{g} - SR_{a} \right|\]

where \(SR_{g}\) is the ratio of the number of positive outcomes to the total number of outcomes in that group.

If the mean strategy is selected, the metric is defined as:

\[SP_{mean} = \frac{1}{n} \sum_{i=1}^{n} \left| SR_{g} - SR_{a} \right|\]

Equality of Opportunity Metrics

Multiclass Equality of Opportunity: This metric is a multiclass generalisation of Equality of Opportunity. For each group, compute the matrix of error rates (normalised confusion matrix). Compute all distances (mean absolute deviation) between such matrices. Then aggregate them using the mean, or max strategy.

The accepted values and bounds for this metric are the same as the 1d case. A value of 0 is desired. Values below 0.1 are considered fair.

\[EOD_{max} = \max_{g} \left| TPR_{g} - TPR_{a} \right|\]

where \(TPR_{g}\) is the true positive rate of group \(g\).

If the mean strategy is selected, the metric is defined as:

\[EOD_{mean} = \frac{1}{n} \sum_{i=1}^{n} \left| TPR_{g} - TPR_{a} \right|\]

Multiclass Average Odds: This metric is a multiclass generalisation of Average Odds. For each group, compute the matrix of error rates (normalised confusion matrix). Average these matrices over rows, and compute all pariwise distance (mean absolute deviation) between the resulting vectors. Aggregate results using either mean or max strategy.

The accepted values and bounds for this metric are the same as the 1d case. A value of 0 is desired. Values below 0.1 are considered fair.

\[AOD_{max} = \max_{g} \left| \frac{1}{2}[(TPR_{g} - TPR_{a}) + (FPR_{g} - FPR_{a})] \right|\]

where \(TPR_{g}\) is the true positive rate of group \(g\) and \(FPR_{g}\) is the false positive rate of group \(g\).

If the mean strategy is selected, the metric is defined as:

\[AOD_{mean} = \frac{1}{n} \sum_{i=1}^{n} \left| \frac{1}{2}[(TPR_{g} - TPR_{a}) + (FPR_{g} - FPR_{a})] \right|\]

Multiclass True Positive Difference: This metric is a multiclass generalisation of TPR Difference. For each group, compute the matrix of error rates (normalised confusion matrix). Compute all distances (mean absolute deviation) between the diagonals of such matrices. Then aggregate them using the mean, or max strategy.

The accepted values and bounds for this metric are the same as the 1d case. A value of 0 is desired. Values below 0.1 are considered fair.

\[TPD_{max} = \max_{g} \left| TPR_{g} - TPR_{a} \right|\]

where \(TPR_{g}\) is the true positive rate of group \(g\).

If the mean strategy is selected, the metric is defined as:

\[TPD_{mean} = \frac{1}{n} \sum_{i=1}^{n} \left| TPR_{g} - TPR_{a} \right|\]