You can run Git on object storage if you re-make packfiles

In one line

Git packfiles were designed for mmap and local disk, so pulling one object out of a bucket means guessing at a byte range. I wrote a new format instead.

Opening excerpt

Skip to main content Contents It sure seems that a bunch of companies are trying to ship a git product of some kind as of late. Wonder why that is.

Either way, I’m building a Git server backed by object storage as an open-source project . It sounded simple enough to start: Git looks like a filesystem, so let’s use a filesystem as a translation layer on top of object storage to make Git speak object storage. This model worked… ok, I guess? But it didn’t work for real-world size repositories, so I needed a different approach. Git stores everything in Objects , so why not store those as objects in Tigris?

Turns out Git packfiles and how they intersected with my (admittedly somewhat terrible) filesystem shim were the main reason why it was slow. I ended up having to invent my own packfile format with a columnar store that’s object storage native. This is the fruit of all of my performance analysis, metrics annotations, and more Texas-style distributed systems work than you can make your k8s cluster shake sticks at.

(Excerpted from the original; full article via the source link below.)

This story hit the Hacker News front page today (133 points / 30 comments, via tigrisdata.com). Our Tech Radar aggregates daily signals on AI engineering, backend architecture and DevOps — browse the related services and further reading below, or get in touch with our team.