ABTest

class hypex.ABTest(additional_tests=None, multitest_method='holm', equal_variance=None, cuped_features=None, cupac_models=None, enable_cupac=False, **kwargs)[source]

Bases: ExperimentShell

A class for conducting A/B tests with configurable statistical tests and multiple testing correction.

This class provides functionality to run A/B tests with options for different statistical tests (t-test, u-test, chi-square test) and multiple testing correction methods.

Parameters:

  • additional_tests (Union[str, ABTestTypesEnum, List[Union[str, ABTestTypesEnum]], None], optional) – Statistical test(s) to run in addition to the default group difference calculation. Valid options are ‘t-test’, ‘u-test’, ‘chi2-test’ or ABTestTypesEnum.t_test, ABTestTypesEnum.u_test, and ABTestTypesEnum.chi2_test. Can be a single test name/enum or list of test names/enums. Defaults to [ABTestTypesEnum.t_test].

  • multitest_method (ABNTestMethodsEnum, optional) –

    Method to use for multiple testing correction. Valid options are: ABNTestMethodsEnum.bonferroni, ABNTestMethodsEnum.sidak, etc. Defaults to ABNTestMethodsEnum.holm.

    For more information refer to the statsmodels documentation: https://www.statsmodels.org/dev/generated/statsmodels.stats.multitest.multipletests.html

Examples

# Basic A/B test with default t-test
ab_test = ABTest()
results = ab_test.execute(data)

# A/B test with multiple statistical tests
ab_test = ABTest(
    additional_tests=[ABTestTypesEnum.t_test, ABTestTypesEnum.chi2_test],
    multitest_method=ABNTestMethodsEnum.bonferroni,
    cuped_features={"target_feature": "pre_target_feature"},
    enable_cupac=True,
    cupac_models=['linear', 'ridge']
)
results = ab_test.execute(data)
__init__(additional_tests=None, multitest_method='holm', equal_variance=None, cuped_features=None, cupac_models=None, enable_cupac=False, **kwargs)[source]
Parameters:

  • additional_tests (Union[str, ABTestTypesEnum, list[str | ABTestTypesEnum], None]) – Statistical test(s) to run in addition to the default group difference calculation. Valid options are ‘t-test’, ‘u-test’, ‘chi2-test’ or ABTestTypesEnum.t_test, ABTestTypesEnum.u_test, and ABTestTypesEnum.chi2_test. Can be a single test name/enum or list of test names/enums. Defaults to [ABTestTypesEnum.t_test].

  • multitest_method (Optional[Literal['bonferroni', 'sidak', 'holm-sidak', 'holm', 'simes-hochberg', 'hommel', 'fdr_bh', 'fdr_by', 'fdr_tsbh', 'fdr_tsbhy', 'quantile']]) – Method to use for multiple testing correction. Valid options are ABNTestMethodsEnum.bonferroni, ABNTestMethodsEnum.sidak, etc. Defaults to ABNTestMethodsEnum.holm.

  • equal_variance (Optional[bool]) – Whether to use equal variance in t-test (optional).

  • cuped_features (Optional[dict[str, str]]) – dict[str, str] — Dictionary {target_feature: pre_target_feature} for CUPED. Only dict is allowed.

  • cupac_models (Union[str, list[str], None]) – str | list[str] — model name (e.g. ‘linear’, ‘ridge’, ‘lasso’, ‘catboost’) or list of model names to try. If None, all available models will be tried and the best will be selected by variance reduction.

  • enable_cupac (bool) – bool — Enable CUPAC variance reduction. CUPAC configuration is extracted from dataset.features_mapping.