Direction Instability in an LLM Evidence Synthesizer

Hadi Soufi

Abstract

A deterministic aggregation rule does not make a language-model component reproducible. An earlier pre-registered study found that a hosted model, used as an evidence synthesizer whose vote enters a deterministic consensus rule, returned its modal direction in only about 72% of repeats under an identical input when synthetic evidence was evenly divided. We ask whether, on recorded inputs, this instability is greater for evidence on which the rule has not reached a consensus without the model. We replayed structured evidence recorded from a live demo-account trading system (free-text rationales were not recorded and were omitted) and sent each evidence set five times to the hosted model at temperature 0, in two pre-registered confirmatory waves (500 sets and 2,500 calls; 898 analysed sets and 4,490 analysed calls). Mean direction instability, the unbiased Gini-Simpson diversity of the five outputs, was lower in the decided stratum in both waves, but neither one-sided test provided sufficient evidence for the registered hypothesis (differences 0.011 and 0.018; permutation p = 0.37 and 0.20; day-block bootstrap 95% intervals -0.048 to 0.086 and -0.024 to 0.041). The study did not establish equivalence. In counterfactual replay at a Research weight of 1.0, about one third of sets produced more than one consensus outcome across five calls; a replay in which only direction varied retained most of this outcome variation, whereas varying only confidence produced much less. At the production weight of zero the synthesizer had no influence by construction. A local open-weights model showed complete direction agreement across repeats. The results distinguish output repeatability, baseline consensus status and downstream decision sensitivity.

Read on SSRN

Pre-registration: OSF Registries, DOI 10.17605/OSF.IO/X4TF7, with wave-2 addendum OSF uxrhy

Data and code: OSF project component

Cite

Soufi, H. (2026). Direction Instability in an LLM Evidence Synthesizer: A Pre-Registered Two-Wave Study of Recorded Trading Inputs. SSRN Working Paper. https://doi.org/10.2139/ssrn.7681837

@misc{soufi2026direction,
  author       = {Soufi, Hadi},
  title        = {Direction Instability in an LLM Evidence Synthesizer: A Pre-Registered Two-Wave Study of Recorded Trading Inputs},
  year         = {2026},
  howpublished = {SSRN Working Paper},
  doi          = {10.2139/ssrn.7681837},
  url          = {https://doi.org/10.2139/ssrn.7681837}
}

Copy BibTeX