Stress-Testing Unit Discovery on Noisy, Multi-Actor Systems

A detector that recovers a hidden coordinating pair in one controlled run can still fail on a harder scenario or a different seed of the identical one — a reminder that a single positive result is evidence of a mechanism, not of a reliable instrument.

What decision changes?

Do not treat a single successful recovery, however genuine, as license to deploy the same detector on a harder or noisier case. Re-run across seeds and against a scenario with more participants before trusting the result generalizes.

A method that works once, in a controlled setting, is a proof of concept — not a validated instrument. The project’s practice is to say so explicitly rather than stop at the first success.

An intervention-based unit-discovery method that had just recovered a genuinely coordinating pair in a two-participant test was pushed harder in two directions. First, a third, non-coordinating participant was added to the scene, to check whether a blunt channel-disabling probe would falsely implicate the bystander (a “ripple” failure) or whether the underlying detector would hold up at all. It did not hold up: on this harder scenario the same method missed the real coordinating pair entirely — a different, and in some ways more basic, failure than the ripple risk it was built to guard against.

Second, the original two-participant scenario was re-run across several random seeds instead of the one that had produced the earlier success. Exact recovery fell to roughly one run in five, statistically indistinguishable from the simpler baseline method the intervention-based approach was meant to outperform.

Neither result overturns the narrower, load-bearing claim that the method’s core mechanism — comparing an intervention’s effect against a measured baseline rather than a fixed one — behaves correctly and reproduces known results on deterministic systems. What it does show is that extending the method to noisy, model-driven, multi-participant systems is not yet reliable, and that a single success on one seed of one scenario should not be read as more than that. The technical findings log records both the success and these failures side by side, on purpose.