Rashomon Alignment: A Geometrical Perspective on Functional Similarity
Modeling the Decision Landscape
The core premise of this work challenges a fundamental assumption in current machine learning practice: that a model's quality can be judged solely by its performance on a held-out test set. In reality, a single model often occupies a vast region of the input space where its predictions remain consistent. This region, termed the "Rashomon set," encompasses all functions that achieve comparable performance to the peak performer. Traditional evaluation methods, which rely on fixed test splits, inherently limit their scope to these specific regions. While effective for benchmarking, they fail to capture the model's behavior in unobserved territories. The proposed Rashomon Alignment framework transcends this limitation by estimating functional similarity across the entire input space, utilizing a geometrical approach that considers the entire landscape of possible inputs, not just the sampled subset.
Central to the Rashomon Alignment (RA) methodology is the distinction between distributional and geometric similarity. Distributional alignment evaluates model outputs based on the specific data distribution present in the training and test sets. In contrast, geometric alignment examines the models' behavior across the entire instance space, utilizing uniformly sampled data to map the decision boundaries' positioning. Experiments conducted across a diverse set of 90 datasets reveal that these two metrics frequently diverge. A model may exhibit high predictive accuracy on specific test distributions while maintaining poor functional coherence across the broader input space. This divergence underscores the critical limitation of test-set-dependent evaluation: a model can appear highly competent within a narrow distributional scope yet fail catastrophically when deployed in real-world scenarios exhibiting distribution shifts.
Furthermore, the research demonstrates that functional similarity does not guarantee predictive parity. It is entirely possible for two models to exhibit strong functional alignment yet exhibit stark differences in their accuracy profiles. Conversely, models with divergent functional architectures can exhibit near-identical accuracy profiles. This finding necessitates a shift in paradigm: evaluation criteria must expand beyond predictive accuracy to encompass functional coherence and geometric consistency. RA provides the mechanistic lens to diagnose these discrepancies, offering a robust framework for model selection that prioritizes structural integrity alongside raw accuracy.
Beyond raw accuracy, RA opens new avenues for model interpretability and ensemble construction. By identifying the specific regions of the input space where models agree or diverge, practitioners can construct more robust ensembles that capitalize on complementary strengths. Furthermore, the ability to map decision boundaries across the entire instance space facilitates enhanced interpretability, allowing researchers to trace how specific features influence model behavior across the entire distribution, not just within the confines of a test split. The implications are significant: RA enables a paradigm shift from accuracy-obsessed evaluation to a holistic assessment of model reliability, paving the way for more resilient AI systems.
Experimental Validation and Key Findings
Experimental validation was conducted across 92 distinct datasets, spanning vision, language, and tabular domains. The core finding is the frequent dissociation between predictive performance and functional alignment. In numerous cases, models achieved state-of-the-art accuracy on benchmark splits while exhibiting poor functional coherence across the broader input space. For instance, an ensemble of models might achieve near-perfect accuracy on a fixed test split while remaining functionally orthogonal across the majority of the instance space. Conversely, distinct models with vastly different architectural designs can exhibit near-identical accuracy profiles yet remain functionally disjoint. This dissociation between accuracy and alignment underscores the central thesis of the work: predictive power alone is an insufficient metric for model quality. RA provides the diagnostic tool to identify these blind spots, enabling more informed decision-making in real-world deployment scenarios.