holisticai.bias.metrics.multiclass_average_odds#

holisticai.bias.metrics.multiclass_average_odds(p_attr, y_pred, y_true, groups=None, classes=None, aggregation_fun='mean')[source]#

Multiclass Average Odds

This metric is a multiclass generalisation of Average Odds. For each group, compute the matrix of error rates (normalised confusion matrix). Average these matrices over rows, and compute all pariwise distances (mean absolute deviation) between the resulting vectors. Aggregate results using either mean or max strategy.

Interpretation

The accepted values and bounds for this metric are the same as the 1d case. A value of 0 is desired. Values below 0.1 are considered fair.

Parameters

p_attrarray-like: Multiclass protected attribute vector.
y_predarray-like: Prediction vector (categorical).
y_truearray-like: Target vector (categorical).
groupslist, optional: Unique groups from p_attr in order
classeslist, optional: The unique output classes in order
aggregation_funstr, optional: The function to aggregate across groups (‘mean’ or ‘max’)

Returns

float: Multiclass Average Odds

Examples

>>> import numpy as np
>>> from holisticai.bias.metrics import multiclass_average_odds
>>> p_attr = np.array(["A", "A", "A", "A", "B", "B", "B", "B", "C", "C"])
>>> y_pred = np.array([0, 1, 2, 0, 1, 2, 0, 1, 2, 0])
>>> y_true = np.array([0, 1, 1, 0, 1, 0, 2, 1, 2, 1])
>>> multiclass_average_odds(p_attr, y_pred, y_true)
0.16666666666666666