holisticai.robustness.metrics.accuracy_degradation_profile#

holisticai.robustness.metrics.accuracy_degradation_profile(X_test: DataFrame, y_test: Series, y_pred: Series, n_neighbors: int = 5, neighbor_estimator: Any | None = None, baseline_accuracy: float | None = None, threshold_percentual: float = 0.95, above_percentual: float = 0.9, step_size: float = 0.05)[source]#

Generates an accuracy degradation profile by iteratively reducing the size of the nearest neighbors considered in the test set and comparing the classifier’s accuracy against a baseline.

This function assesses the robustness of a model by gradually reducing the test set size and evaluating whether the accuracy falls below a defined threshold. It returns a DataFrame summarizing whether the accuracy at each step meets the baseline accuracy or if there is degradation.

Parameters:#

X_testpd.DataFrame

The feature matrix of the test set. Each row represents a sample, and each column represents a feature.

y_testpd.Series

The true labels for the test set. This should be a one-dimensional Series.

y_predpd.Series

The predicted labels for the test set. This should be a one-dimensional Series.

n_neighborsOptional[int], optional

The number of neighbors to consider when using a nearest neighbors model. If not provided, the function will not perform neighbor-based operations.

neighbor_estimatorOptional[Any], optional

An estimator implementing the neighbor search algorithm. If not provided, the default NearestNeighbors from sklearn will be used.

baseline_accuracyOptional[float], optional

The baseline accuracy to be compared against. If not provided, it will be calculated using y_test and y_pred.

threshold_percentualfloat, optional (default=0.95)

The threshold for acceptable accuracy degradation. Defined as a percentage of the baseline accuracy, below which degradation is considered to occur.

above_percentualfloat, optional (default=0.90)

The proportion of samples that must have accuracy above the threshold to avoid being marked as degraded.

step_sizefloat, optional (default=STEP_SIZE)

The step size by which to reduce the test set size in each iteration. It determines the incremental reduction of the test set in each step.

Returns

pd.DataFrame

A pandas DataFrame summarizing the accuracy degradation results. The DataFrame contains the following columns:

  • size_factor: The fraction of the test set used in each step.

  • above_threshold: The number of samples with accuracy above the threshold.

  • ADP: The percentage of samples exceeding the accuracy threshold.

  • decision: Whether the accuracy at each step meets the threshold (‘OK’) or is considered degraded (‘acc degrad!’).

Example

>>> from sklearn.neighbors import NearestNeighbors
>>> import pandas as pd
>>> X_test = pd.DataFrame([[1.2, 3.4], [2.2, 1.8], [1.1, 4.5], [3.2, 2.1]])
>>> y_test = pd.Series([0, 1, 0, 1])
>>> y_pred = pd.Series([0, 1, 0, 0])
>>> degradation_profile = accuracy_degradation_profile(
...     X_test=X_test, y_test=y_test, y_pred=y_pred, n_neighbors=3
... )
>>> print(degradation_profile)
# Outputs a DataFrame summarizing the accuracy degradation decisions for each step.
raises ValueError:

If the lengths of X_test, y_test, or y_pred are inconsistent, or if invalid values are provided for threshold_percentual, above_percentual, or n_neighbors.

Notes

  • The function assumes that the input DataFrames or Series are correctly structured and that the baseline accuracy, if provided, is meaningful.

  • Nearest neighbors is used to simulate accuracy degradation by reducing the test set size incrementally.