The idea: many small files first, one packed file later
Git stores every version of every file, every directory listing and every commit as an
object named by its hash. New objects are written the simple way: one compressed file
per object, a loose object. That is fast to write but wasteful: a one-line change
to a 12 KB file stores the whole file again. Now and then git gc gathers the objects into a
packfile, where similar objects are stored as small deltas against each
other, and writes an index that finds any object in a few steps.
This page is part of a Git series with Git Objects, Git Refs and HEAD, Merge vs Rebase and Three-Way Merge. Here the objects themselves are taken as given; the question is how they sit on disk.
Loose objects
To write an object, Git builds a header such as blob 12000\0, puts the content after it,
hashes the whole thing (SHA-1, 40 hex digits) and writes it zlib-compressed to
.git/objects/4e/7a91c…: the first two hex digits are the directory, the rest the file name.
Reading one back needs no index: the ID gives the path.
One commit that edits one file writes three objects: the new blob, a new tree (the directory listing, which points at the new blob and the unchanged README blob) and the commit. An unchanged file is never stored again, but a changed file is stored in full every time (Demo: a commit writes 3 loose objects). Thousands of small files also waste disk blocks and make every directory scan slow.
git gc and git repack
git gc runs git repack -a -d: list every reachable object, write them all into one
new pack-<hash>.pack, delete the old packs (-d), then
git prune-packed deletes the loose files that are now packed. Git runs this by itself
(git gc --auto) once there are more than about 6,700 loose objects (gc.auto) or
50 packs (gc.autoPackLimit). git push and git fetch send packs too, built
the same way.
The pack starts with a 12-byte header (PACK, version 2, object count) and ends with a 20-byte
checksum. In between, each entry is a small header (type and size) followed by zlib data. Git writes
commits first, then trees, then blobs in the order a checkout reads them, so objects used together are
close together in the file (Demo: git gc packs and delta-compresses).
Delta compression
Version 3 of app.js and version 2 differ by one line. Instead of storing both, the pack stores
one in full and the other as a delta: a list of instructions that rebuild it from a
base object. There are only two kinds: COPY offset, length from the base and
INSERT new bytes. Here the 12,000-byte version 1 becomes two COPY instructions, about 20 bytes.
Git finds bases without knowing file history. It sorts the objects by type, by path (a hash of the file
name) and by size, largest first, then tries each object against the 10 before it (pack.window)
and keeps the smallest delta. Files usually grow, so the newest version tends to be the base and old
versions become deltas: the common case, reading recent files, needs no delta work. An entry of type
OFS_DELTA names its base by how many bytes back it sits in the same pack. A delta's base can be
a delta too, forming a chain; pack.depth (default 50) caps its length.
| Object | Loose (zlib) | In the pack |
|---|---|---|
| app.js v3 (newest) | 3,624 B | 3,624 B, whole |
| app.js v2 | 3,612 B | 20 B, delta against v3 |
| app.js v1 | 3,600 B | 20 B, delta against v2 |
| tree t1, t2 | 78 B each | 32 B each, delta against the next tree |
| commits, README.md | 1,445 B | 1,445 B, whole |
Reading app.js v1 means: find its entry, see that it is a delta, jump back to its base (v2, also
a delta), jump back again to v3 (whole), inflate v3, then apply v2's delta and v1's delta
(Demo: read a delta, follow the chain). After a new commit and another git gc, v4 becomes
the whole one and v3 turns into a delta (Demo: commit after gc, then gc again).
The pack index
A pack can hold millions of objects, and its entries have different sizes, so Git cannot jump to the n-th
one. The .idx file next to it holds every ID in sorted order, each with its offset in the
.pack. In front sits the fanout table: 256 counts, where fanout[b]
is how many IDs start with a byte ≤ b. For an ID that starts with 0x4e, the matching rows are
fanout[0x4d] up to fanout[0x4e] − 1; a binary search inside that small range finds
the ID, and its offset says where to seek (Demo: look up a whole object in the .idx).
Git always looks in the packs first, then in the loose objects. An object written after the last
git gc is missed by the index (the fanout range is empty or the search fails) and found as a
loose file. With 1 million objects the binary search takes about 12 steps, since the fanout already divides
the list by 256.
The numbers on this page
| After 3 commits | Files | Bytes |
|---|---|---|
| All loose | 10 | 12,515 |
| After git gc | 2 (.pack + .idx) | 5,283 + 1,352 |
The object IDs are short made-up ones, and the sizes are rounded (zlib to 30 % of a text file). Real repositories gain much more: the Linux kernel's history packs to a small fraction of its loose size, because most of its millions of blob versions are deltas.
What the page leaves out
- Real SHA-1 (or SHA-256) IDs, and the exact header and size encoding of pack entries.
REF_DELTA(a base named by ID, used in thin packs during fetch), and delta reuse when repacking an existing pack.- The
.revreverse index, multi-pack indexes (multi-pack-index), bitmaps and commit-graph files. - Unreachable objects:
git gckeeps them loose for two weeks (gc.pruneExpire) beforegit prunedeletes them, and the reflog keeps old commits reachable. - Git's cache of delta bases (
core.deltaBaseCacheLimit) and memory-mapped pack windows.
See also LSM Tree vs B-Tree for another store that writes small pieces first and compacts them later, and Huffman coding for how compression like zlib's works.