Chain-of-thought prompting improves reasoning. Rationale-trace fine-tuning improves it further. Both claims are widely repeated, and on table question answering with FLAN-T5, neither held.
The setup
Eleven conditions over WikiTableQuestions:
- zero-shot
- plain chain-of-thought prompting
- structured chain-of-thought prompting
- answer-only fine-tuning
- rationale-trace fine-tuning
Run across FLAN-T5 at 250M and 780M, with dual seeds so a single lucky initialization could not carry a conclusion.
The result
Zero-shot scored 0.241 exact match and beat every intervention.
Worse for the interventions: the fine-tuned conditions fell below the 0.551 majority-class floor. Always guessing the majority label would have scored higher than the models that had been trained on the task.
Checking whether it was real
A negative result is easy to produce by accident, so the burden of proof runs the other way — you have to work to show you did not simply do it wrong.
- Cross-dataset generalization. Re-ran on TabFact to check the finding was not an artifact of one dataset's quirks.
- McNemar's test. Applied to the paired predictions, to establish that the differences were not noise.
- Error attribution. Broke errors down across lookup, aggregation, and multi-hop reasoning, so the finding says where the intervention failed rather than only that it did.
What was actually happening
Fine-tuning overfit to task format. The models learned the shape of the expected output rather than the reasoning that produces it — which is exactly the failure mode that a headline accuracy number hides and an error breakdown exposes.
Why report it
A negative result that survives a cross-dataset check and a significance test is information. It tells you which intervention not to spend a training budget on, which is worth more than another marginal positive on a benchmark that already has hundreds.
The alternative — quietly running more conditions until something wins — produces a number nobody should believe, including me.