Regression#
Equality of Outcome Metrics#
q-Disparate Impact: This metric computes the ratio of success rates between group a and group b, where sucess means predicted score exceeds a given quantile (default = 0.8).If q is a vector, this metric returns a vector with the respective result for each given quantile in q.
A value of 1 is desired. Values below 1 are unfair towards group_a. Values above 1 are unfair towards group_b. The range (0.8,1.2)is considered acceptable.
where \(SR_{g}\) is the ratio of the number of positive outcomes to the total number of outcomes in that group.
No Disparate Impact Level: If we calculate the adverse impact for each possible quantile, we can find the minimum maximum quantile for which the algorithm is considered unbiased (i.e. the disparate impact falls between 0.8 and 1.2).
Average Score Difference: this metric is the difference between the average score of the unprivileged and privileged group.
The ideal value is 0, a value < 0 disadvantages the unprivileged group and > 0 is favorable.
where \(\hat{y}_{g}\) is the predicted score of group \(g\).
Average Score Ratio: this metric computes the ratio in average scores between group a and group b. If q is a vector, this metric returns a vector with the respective result for each given quantile in q.
A value of 1 is desired. Values below 1 indicate the group a has lower average score, so bias against group_a. Values above 1 indicate group_b has lower average score, so bias against group_b. The [0.8, 1.25] range is considered fair.
where \(\hat{y}_{g}\) is the predicted score of group \(g\).
Z Score Difference: the Z score spread is the average score spread divided by the pooled standard deviation. It allows us to compare the difference in average scores with the standard deviation.
The ideal value is 0, a value less than 0 disadvantages the unprivileged group and larger than 0 is favorable.
where \(poolStd\) is the pooled standard deviation of the predicted scores, defined as
Max Statistical Parity: This metric computes the maximum over all thresholds of the absolute statistical parity between group a and group b.
A value of 0 is desired. Values below 0.1 in absolute value are considered acceptable.
where \(SR_{g}\) is the ratio of the number of positive outcomes to the total number of outcomes in that group.
Statistical Parity AUC: This metric computes the area under the statistical parity versus threshold curve.
A value of 0 is desired. Values below 0.075 are considered acceptable.
where \(SR_{g}\) is the ratio of the number of positive outcomes to the total number of outcomes in that group.
Equality of Opportunity Metrics#
RMSE Ratio: This metric computes the ratio of the RMSE for group a and group b. If q is a vector, this metric returns a vector with the respective result for each given quantile in q.
A value of 1 is desired. Lower values show bias against group a. Higher values show bias against group b.
where \(RMSE_{g}\) is the root mean squared error of group \(g\).
MAE Ratio: This metric computes the ratio of the MAE for group a and group b. If q is a vector, this metric returns a vector with the respective result for each given quantile in q.
A value of 1 is desired. Lower values show bias against group a. Higher values show bias against group b.
where \(MAE_{g}\) is the mean absolute error of group \(g\).
Correlation Difference: This metric computes the difference in correlation between predictions and targets for group a and group b. If q is a vector, this metric returns a vector with the respective result for each given quantile in q.
A value of 0 is desired. This metric ranges between -2 and 2, with -1 indicating strong bias against group a, and +1 indicating strong bias against group b.
where \(\rho_{g}\) is the correlation between predictions and targets for group \(g\).