holisticai.bias.metrics.avg_precision_ratio

holisticai.bias.metrics.avg_precision_ratio(group_a, group_b, mat_pred, mat_true, top=None, thresh=0.5, normalize=False)

Average precision ratio

This function computes the ratio of average precision (over users)on minority and majority group.

Interpretation

A value of 1 is desired. Lower values show bias against minority group.Higher values show bias against majority group.

Parameters

group_aarray-like

Group membership vector.

group_barray-like

Group membership vector.

mat_predmatrix-like

Matrix with shape (num_users, num_items). A recommenderscore (binary or soft pred) for each user,item interaction.

mat_truematrix-like

Matrix with shape (num_users, num_items). A target score(binary or soft pred) for each user,item pair.

topint, optional

If not None, the number of items that are shown to each user.

threshfloat, optional

Threshold indicating value at which a given item is shown to user (if top is None).

normalizebool, optional

If True, normalises the data matrix to [0,1] range.

Returns

float

Ratio of average precisions

Notes

\(\frac{\texttt{AVg_precision_min}}{\texttt{AVg_precision_maj}}\)

References

[1]

Yunqi Li, Hanxiong Chen, Zuohui Fu, Yingqiang Ge, Yongfeng Zhang (2021). User-oriented Fairness in Recommendation. <https://arxiv.org/pdf/2104.10671.pdf>

Examples

>>> import numpy as np
>>> from holisticai.bias.metrics import avg_precision_ratio
>>> group_a = np.array([1, 1, 1, 1, 0, 0, 0, 0, 0, 0])
>>> group_b = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1, 1])
>>> mat_pred = np.array([[0.9, 0.8, 0.4, 0.2],
                      [0.7, 0.9, 0.1, 0.7],
                      [0.3, 0.2, 0.3, 0.3],
                      [0.2, 0.1, 0.7, 0.8],
                      [0.8, 0.7, 0.9, 0.1],
                      [1. , 0.9, 0.3, 0.6],
                      [0.8, 0.9, 0.1, 0.1],
                      [0.2, 0.3, 0.1, 0.5],
                      [0.1, 0.2, 0.7, 0.7],
                      [0.2, 0.7, 0.1, 0.2]])
>>> mat_true = np.array([[0.7, 0.8, 0.4, 0.2],
                      [0.9, 0.9, 0.1, 0.2],
                      [0.3, 0.8, 0.2, 0.6],
                      [0.2, 0.1, 0.7, 0.8],
                      [0.6, 0.7, 0.9, 0.1],
                      [1. , 0.9, 0.3, 0.6],
                      [0.8, 0.1, 0.1, 0.1],
                      [0.2, 0.3, 0.1, 0.5],
                      [0.1, 0.2, 0.7, 0.7],
                      [0.2, 0.1, 0.1, 0.8]])
>>> avg_precision_ratio(
...     group_a, group_b, mat_pred, mat_true, top=None, thresh=0.2, normalize=False
... )
1.161290322580645