LSR-Synth: How Anti-Memorization Design Changes the Measurement of Symbolic Discovery
Background and Benchmark Challenges
Scientific equation discovery has long relied on benchmarks composed of well-known equations available in the public domain. This composition makes it difficult to determine whether a model is genuinely discovering laws from data or merely recalling answers from its training corpus. LSR-Synth was developed to address this limitation by introducing novel synthetic terms into established scientific mechanisms. These terms are filtered for novelty, solvability, and scientific plausibility, creating a more robust evaluation framework. The core problem identified is the difficulty in distinguishing between genuine scientific discovery and simple memorization of training data.
Methodology and Analysis Framework
The research constructs a semantics-free baseline using a fixed vocabulary with publicly documented provenance. This approach allows for systematic assessment of candidate coverage through three key mechanisms: semantic blinding, library weakening, and matched operator-family knockouts. The fixed vocabulary serves as a
lready covers most tasks under the current snapshot, while language-model-generated candidates rarely expand the set of solvable instances. The study reveals that the marginal contribution of language-model priors becomes substantial only when vocabulary coverage is selectively disrupted. This finding suggests that most current tasks are suitable for evaluating the fitting and recombination of previously unseen expressions, but are insufficient on their own to identify contributions from priors beyond a fixed search space.
Out-of-Distribution Evaluation Results
Strict out-of-distribution evaluation lowers the absolute success rates of all methods but does not alter the fundamental relationship between vocabulary coverage and model performance. Even under these challenging conditions, the fixed vocabulary maintains its coverage advantage, and language-model-generated candidates still only make significant contributions when selective disruption occurs. This resilience demonstrates the robustness of the current task design against memorization while highlighting the limitations of language-model priors in expanding solvability.
Conclusion and Research Implications
The findings neither invalidate LSR-Synth's controls against memorization of complete formulas nor imply that language-model priors are generally unhelpful. Rather, they support a more nuanced conclusion: most current tasks remain suitable for evaluating the fitting and recombination of previously unseen expressions, but are insufficient on their own to identify contributions from priors beyond a fixed search space. This work provides valuable insights for designing more effective benchmarks that can better distinguish between genuine discovery and mere recall in scientific equation discovery systems.
END