Overview
The study addresses the problem of detecting sparse departures from independence in two-way (R by C) tables. Unlike traditional methods that employ an omnibus statistic against a dense alternative, this work frames the problem around a detection boundary, focusing on the weakest signal a test can identify when departures are concentrated in a few cells. The investigation proceeds in three distinct parts, detailing the theoretical underpinnings, proposing an adaptive detection mechanism, and offering an assumption-lean exact method for challenging scenarios.
Research Context
Testing independence in two-way tables commonly involves a single omnibus statistic, which can be suboptimal for detecting sparse departures. When deviations from independence are localized to a small number of cells, the critical analytical objective becomes the detection boundary – the minimum signal strength discernible by any test. This research specializes sparse-signal detection theory, referencing works by Ingster (1997), Donoho and Jin (2004), and Chhor, Mukherjee and Sen (2024), specifically adapting it to the context of two-way tables. The underlying statistical model is characterized as a heteroscedastic Gaussian sequence, where the variance profile aligns with the expected-count table. This profile is determined by the two margins through the product-CV identity, expressed as $CV_e^2 = (1 + CV_r^2)(1 + CV_c^2) - 1$, indicating that marginal heterogeneity directly dictates the heteroscedasticity profile.
Approach
The methodology unfolds in three primary components:
Part I: Heteroscedasticity Profile and Detection Boundary
The initial phase characterizes the two-way table problem within sparse-signal detection theory. A variance-stabilizing root transform is applied, yielding a standardized amplitude of $2 \sqrt{m}(\sqrt{1 + a} - 1)$. This transformation facilitates the derivation of a closed-form separation radius. This framework is considered valid under a growing-count condition but is shown to fail at fixed small counts. The connection between marginal heterogeneity and the heteroscedasticity profile is established via the product-CV identity $CV_e^2 = (1 + CV_r^2)(1 + CV_c^2) - 1$.
Part II: Adaptive Higher-Criticism Gate
The second part reconceptualizes independence testing as a goodness-of-fit problem for the log-linear no-interaction model. It proposes that higher criticism, utilizing its closed-form Jaeschke-Eicker null, can achieve the detection boundary adaptively across an unknown range of sparsity. However, the study candidly notes that in very sparse, strong signal regimes, the maximum-cell rule tends to dominate. The optimality of this approach under exact-conditional scenarios remains conjectural.
Part III: Exact Anchor and Routing Rule
The final component addresses calibration failures observed at small counts, where test size can range from 0.3 to 0.6 even with uniform margins. This failure is attributed to the per-cell tail behavior rather than heterogeneity. To rectify this, the approach removes the small-count calibration failure by employing exact margin-conditional reference laws. It establishes that conditioning on both margins provides a complete treatment for the test, obviating the need for an oracle. This method achieves a size of 0.003 to 0.009. A routing rule is introduced, designed to direct large-count tables to the asymptotic gate (higher criticism) and small-count tables to the exact anchor. This rule is keyed to the margin profile. The implementation of this routing rule on a public corpus of 5,543 tables revealed that two-thirds of these tables route to the exact anchor.
Findings
- The detection of sparse departures from independence in two-way tables can be modeled as a heteroscedastic Gaussian sequence where the variance profile is given by the expected-count table.
- Marginal heterogeneity directly defines the heteroscedasticity profile, quantified by the product-CV identity $CV_e^2 = (1 + CV_r^2)(1 + CV_c^2) - 1$.
- A variance-stabilizing root transform produces a standardized amplitude $2 \sqrt{m}(\sqrt{1 + a} - 1)$ and a closed-form separation radius, which is valid under a growing-count condition but fails at fixed small counts.
- Higher criticism, with a Jaeschke-Eicker null, can adaptively attain the sparse detection boundary for the log-linear no-interaction model, though the maximum-cell rule dominates in very sparse strong regimes.
- Small-count calibration failure, manifesting as inflated test size (0.3–0.6), is driven by per-cell tail behavior.
- Exact margin-conditional reference laws remove small-count calibration failure, achieving test sizes of 0.003 to 0.009, with conditioning on both margins providing a complete treatment.
- A routing rule, based on the margin profile, directs large-count tables to the asymptotic gate and small-count tables to the exact anchor.
- Numerical confirmation, derived from openly deposited, deterministically seeded code, supports every claim presented.
- Two-thirds of a public corpus consisting of 5,543 tables routed to the exact anchor.
Why This Matters
This research offers refined statistical methods for detecting subtle but significant departures from independence in two-way tables, particularly when these departures are localized to a few cells. By providing both an adaptive asymptotic method and a robust exact conditional method, the work addresses limitations inherent in traditional omnibus tests and improves calibration accuracy in challenging small-count scenarios. The developed routing rule provides a practical framework for selecting the appropriate statistical test based on table characteristics.
Potential Applications
The refined methods for sparse departure detection in two-way tables, including the adaptive higher-criticism gate for large counts and the exact margin-conditional anchor for small counts, offer improved statistical tools for analyses across various scientific disciplines where categorical data is organized in contingency tables. The routing rule based on margin profile could guide practitioners in selecting the most appropriate and robust testing procedure for their specific dataset.