holisticai.bias.metrics.avg_f1_ratio

holisticai.bias.metrics.avg_f1_ratio(group_a, group_b, mat_pred, mat_true, top=None, thresh=0.5, normalize=False)

Average f1 ratio

This function computes the ratio of average f1 (over users) on minority and majority group.

Interpretation

A value of 1 is desired. Lower values show bias against minority group. Higher values show bias against majority group.

Parameters

group_aarray-like

Group membership vector.

group_barray-like

Group membership vector.

mat_predmatrix-like

Matrix with shape (num_users, num_items). A recommender score (binary or soft pred) for each user,item interaction.

mat_truematrix-like

Matrix with shape (num_users, num_items). A target score (binary or soft pred) for each user,item pair.

topint, optional

If not None, the number of items that are shown to each user.

threshfloat, optional

Threshold indicating value at which a given item is shown to user (if top is None).

normalizebool, optional

If True, normalises the data matrix to [0,1] range.

Returns

float

Ratio of average f1

Notes

\(\frac{\texttt{AVg_f1_min}}{\texttt{AVg_f1_maj}}\)

References

[1]

Yunqi Li, Hanxiong Chen, Zuohui Fu, Yingqiang Ge, Yongfeng Zhang (2021). User-oriented Fairness in Recommendation. <https://arxiv.org/pdf/2104.10671.pdf>

Examples

>>> import numpy as np
>>> from holisticai.bias.metrics import avg_f1_ratio
>>> group_a = np.array([1, 1, 1, 1, 0, 0, 0, 0, 0, 0])
>>> group_b = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1, 1])
>>> mat_pred = np.array([[0.9, 0.8, 0.4, 0.2],
                      [0.7, 0.9, 0.1, 0.7],
                      [0.3, 0.2, 0.3, 0.3],
                      [0.2, 0.1, 0.7, 0.8],
                      [0.8, 0.7, 0.9, 0.1],
                      [1. , 0.9, 0.3, 0.6],
                      [0.8, 0.9, 0.1, 0.1],
                      [0.2, 0.3, 0.1, 0.5],
                      [0.1, 0.2, 0.7, 0.7],
                      [0.2, 0.7, 0.1, 0.2]])
>>> mat_true = np.array([[0.7, 0.8, 0.4, 0.2],
                      [0.9, 0.9, 0.1, 0.2],
                      [0.3, 0.8, 0.2, 0.6],
                      [0.2, 0.1, 0.7, 0.8],
                      [0.6, 0.7, 0.9, 0.1],
                      [1. , 0.9, 0.3, 0.6],
                      [0.8, 0.1, 0.1, 0.1],
                      [0.2, 0.3, 0.1, 0.5],
                      [0.1, 0.2, 0.7, 0.7],
                      [0.2, 0.1, 0.1, 0.8]])
>>> avg_f1_ratio(
...     group_a, group_b, mat_pred, mat_true, top=None, thresh=0.5, normalize=False
... )
0.9285714285714286