holisticai.bias.metrics.multiclass_true_rates

holisticai.bias.metrics.multiclass_true_rates(p_attr, y_pred, y_true, groups=None, classes=None, aggregation_fun='mean')

Multiclass True Rates

This metric is a multiclass generalisation of TPR Difference. For each group, compute the matrix of error rates (normalised confusion matrix). Compute all distances (mean absolute deviation) between the diagonals of such matrices. Then aggregate them using the mean, or max strategy.

Interpretation

The accepted values and bounds for this metric are the same as the 1d case. A value of 0 is desired. Values below 0.1 are considered fair.

Parameters

p_attrarray-like

Multiclass protected attribute vector.

y_predarray-like

Prediction vector (categorical).

y_truearray-like

Target vector (categorical).

groupslist, optional

Unique groups from p_attr in order

classeslist, optional

The unique output classes in order

aggregation_funstr, optional

The function to aggregate across groups (‘mean’ or ‘max’)

Returns

pandas DataFrame

Accuracy Matrix : shape (num_groups, num_classes)

Examples

>>> import numpy as np
>>> from holisticai.bias.metrics import multiclass_true_rates
>>> p_attr = np.array(["A", "A", "A", "A", "B", "B", "B", "B", "C", "C"])
>>> y_pred = np.array([0, 1, 2, 0, 1, 2, 0, 1, 2, 0])
>>> y_true = np.array([0, 1, 1, 0, 1, 0, 2, 1, 2, 1])
>>> multiclass_true_rates(p_attr, y_pred, y_true)
0.6666666666666666