Building Bubble: Look Ma, No “Master”: Decentralized Integration

In this series of posts, our engineering team talks about the inner workings of Bubble, the cloud-based visual programming language that’s making programming accessible to everyone. If you’re interested in joining the no-code revolution, see our list of open positions.

Git is great. It’s great for many reasons, but a big one is that it’s decentralized: each developer has their own copy of the repository, and can experiment with different versions of the codebase without having to coordinate with a centralized chokepoint.

Git’s merge features make it very easy for developers to work independently. I can work on feature X, and my friend can work on feature Y, and we don’t need to worry about whether or not we’re touching the same set of files: git will allow us to combine our changes down the line, and help us work through potential conflicts.

However, there is one place left in the modern development workflow where centralization still lurks. The place it haunts is continuous integration, and the monster’s name is “master”.

At many companies, this is what a state-of-the-art continuous integration workflow looks like:

• I build a new feature
• I create a pull request, and a teammate merges the feature into master
• A post-commit hook notifies our continuous integration engine, which spins up the test suite, marks the commit as good, and deploys my feature to production

This is great… until I submit a buggy request and break the build. Then, I have to endure the contempt of my teammates as I scramble to undo the damage while they sit tapping their feet impatiently, waiting to deploy the features they are working on.

There’s often a strong culture discouraging build-breaking: engineers are judged for how often they do it, since it slows down the whole team’s velocity.

Our philosophy at Bubble is that this is backwards. People make mistakes, and developers create bugs: that’s part of the natural lifecycle of development. As a rule of thumb, if a task needs to be done over and over without errors occurring, it’s a task for a computer, not a human. So we don’t believe in creating systems where an engineer can become the bottleneck for their team.

At Bubble, we don’t use master, and we don’t practice continuous integration. Instead, what we do is this:

• We build a new feature
• Once it’s been reviewed, and we’re ready to deploy to production, we run a script that says “deploy branch my_feature” (we use a slack bot we wrote in-house to do the automation)
• The script queries a database where we store the hash of our current production commit
• The script checks whether our feature branch is strictly ahead of the prod commit via running git merge-base –is-ancestor prod_commit_hash my_feature
• If our feature branch isn’t strictly ahead of production, it aborts and prompts me to merge the current prod commit into my feature branch
• If I am ahead of production, it runs our test suite on the latest commit on my feature branch, and on success, marks that commit hash as being tested in our database
• It then re-checks that the commit is still ahead of production (in case someone else deployed code while the test suite was running), and if so, updates our database to mark this as the production commit, and rolls it out.

The big difference between this and the standard continuous integration workflow is that the tests run before integration, instead of afterwards. This has a couple big advantages:

• Developers can try to deploy code in parallel. I can be running the tests on my feature branch while my teammate runs the tests on hers: if mine fails and hers succeeds, her code makes it into production and she can move on to her next project without worrying about my failed build
• I’m much less exposed to my teammate’s bugs. The only code I have to merge with mine prior to deploy is the current production commit, which I know has passed all the tests already. So, if I get a test failure, 98% of the time it’s because of the code I was working on. Not having to track down whether a bug is in your code or in someone else’s is a huge time save.
• Because we track the production commit in a database rather than in git, rollbacks are a little simpler: we don’t have to worry about developers having pulled a version of master that’s out of date. Our deploy script accepts a “rollback” command that disables the “is this commit ahead of production” check, which allows us to do complicated rollbacks without making a mess of our branch structure.

We’ve been using this workflow at Bubble for a few years now, and it’s served us well. We often have multiple developers releasing code at the same time, and they almost never need to coordinate with each other, which is a big win for productivity. Coordination and teamwork is great, but when you’re working on unrelated things, it doesn’t make sense to be forced to coordinate. That was the insight behind git, and we see this as a logical extension of that philosophy.

The biggest change for a new developer joining Bubble is that “master” isn’t a special branch (we tend to keep it up to date with production, because Github generates statistics based on master and we like charts, but it’s not officially part of our workflow). We’ve added a couple pieces of automation over the years to smooth out the workflow:

• A “branch” command that generates a new feature branch by querying our database for the latest production commit, and starting a branch based on that commit
• A “review” command that can be run on a feature branch. It uses git merge-base to find the common ancestor between the feature branch and production, and diffs the feature branch against the common ancestor to show only new changes on the feature branch. (It’s somewhat equivalent to merging prod into the feature branch then diffing against prod, but saves having to do the merge each time to check the diffs).

We also added some logic in our deploy script to handle races between developers deploying code. Let’s say we’re deploying two different feature branches, A and B. If the tests on A pass before they do on B, our script stops the deploy of B, and attempts to merge A into B (via the github API). If it can do the merge without conflicts to produce commit B´, it then runs our test suite on B´, and if that succeeds, deploys B´. In practice, we find that 99% of the time there aren’t conflicts (either at the git level, or at a test-breaking semantic level). So two developers can be working on different features, both hit “deploy”, go out to lunch, and have both features be live when they get back! (And if a third developer tries to deploy a feature with a bug in it at the same time, it won’t break the build for the two successful deploys).