Foundation Models

We reveal that transformers trained autoregressively naturally encode causal structures—gradient attributions directly recover underlying causal graphs without any explicit causal objectives.