Researchers fix Llama-3.1-8B reasoning errors with 8 even attention heads, boosting accuracy by 60%
NYU Tandon researchers led by Industry Professor Gustavo Sandoval discovered and fixed a specific reasoning error in the Llama-3.1-8B language model where it incorrectly compares decimals like 9.11 and 9.8. They found that even-numbered attention heads specialize in numerical comparison while odd-numbered heads handle other functions. Using precisely eight even attention heads at Layer 10, they achieved perfect repair with 60% accuracy improvement, demonstrating that targeted interventions can fix AI reasoning errors without affecting overall performance.
