In April 2003, the Human Genome Project was declared complete. After more than a decade of work, scientists had assembled a reference sequence for human DNA. The achievement gave biology something it could return to: a common place to locate a finding, compare it with another, and carry an investigation forward.
A stretch of DNA could gradually gather a body of knowledge around it. One experiment might reveal when a gene was active; another might show how a change affected the protein it encoded. Findings made years apart, by people asking different questions, could become part of the same account. Understanding would grow through these connections, as researchers learned how the sequence participated in the life of a cell. That work continues long after the project’s declared completion.
There is something familiar about this situation when we open a language model such as Llama. Its trained parameters are available to download. We know how the numbers are arranged and the mathematical operations applied to them. We can follow a calculation through the network, all the way to the scores from which an answer is chosen. Still, it can be difficult to explain how those calculations allow the model to implement functions, such as keeping track of a name, finding a particular word, or retrieving the right entry from a table, etc.
We occupy a curious position with these machines. We designed their architecture and the process by which they learn. During training, however, the parameters changed through exposure to data, acquiring an organization whose functional details we did not specify one by one. Now we encounter the results as abilities. A model can perform a task before we can give a satisfying account of the computation that makes it possible.
For an open model, then, the counterpart of the genome sequence is already in our hands. The work ahead is to discover what training has made of it.
This is where the genome analogy becomes useful. It suggests a way for individual discoveries to grow into a shared understanding. If we find a component involved in retrieving a word, we should be able to place that finding beside evidence about the same component in another task. If we identify a route by which information reaches an answer, a later experiment should be able to follow that route, test its limits, and extend the account.
We call this effort the LLM Genome Project. We are building a functional atlas of Llama, beginning with discovered components and the circuits in which neurons participate. We hope the atlas will help us approach a model with more precise questions: where a computation went astray when the model makes a mistake, what an intervention might change, and whether that prediction holds in a case we have not seen before, etc. Each experiment contributes a small piece of that understanding, with a place for the next one to begin.