On October 29 we said we’d share a preliminary root cause analysis (RCA) for the security incident that occurred the week of October 26. This is that report, based on our own ongoing investigation and the findings so far from an independent review by outside forensic experts. It covers your apps and the data in them, and we’ll share a final report once the investigation is complete.
You can also read Co-CEO Josh Haas’ post on the changing security landscape and the team’s response here.
Summary
- The attacker targeted certain individual apps directly: some of that relied on Bubble app configuration issues, and some used issues in our own platform. In certain cases they accessed app data or stored credentials. Where they got around one of our platform controls, we’ve contacted the app’s owner directly.
- Through our internal systems, the attacker read private credentials that developers had saved in the plugin editor. We emailed those developers on October 26 asking them to change those credentials. So far, our review has not shown that the attacker’s access to our internal systems was used to read our application databases or the data and files in your apps (in other words, your users’ data). The outside firm’s preliminary findings are consistent with ours so far.
What you should do
- If we've contacted you, follow the steps in our message. It should also help inform what you tell your app's users. Reply to us with any questions.
- If we haven't contacted you, our recommendation is the same as on October 29: Take routine precautions, and otherwise continue business as usual. Specifically:
- Run the security dashboard on each app and fix anything it flags.
- Review your privacy rules, and check that API keys and other credentials are marked private and API Connector calls are configured correctly.
- Set your app to private unless you intend the editor to be public (app security settings). A public editor lets anyone view the app's settings and data, including stored keys.
- If you notice anything unexpected in your app, contact support.
What happened
On October 26, an attacker obtained temporary internal credentials by exploiting a security vulnerability in our platform as well as part of our infrastructure that required updates. With those temporary credentials, they gained access to further internal credentials and used them to reach other internal systems. Through one of those systems, the attacker read private plugin data, including credentials that plugin developers had saved in the plugin editor.
Our monitoring flagged unusual use of an internal key the same day. We closed the initial path, began changing the internal credentials involved, and emailed the affected plugin developers asking them to change their credentials as well. Some of the steps we took to close that initial path caused the service disruption many of you saw that day.
The attacker also targeted some individual apps directly, in two ways. First, they exploited issues in our platform that enabled them to reach an app’s stored credentials or data even when the app was set up correctly. Where the attacker accessed data or credentials that way, we’re contacting the app’s owner directly. Second, the attacker looked for instances in which an app’s own settings, such as its privacy rules, which may be set up that way on purpose, granted them access. Where that succeeded, and where we’re confident the data was meant to be private, we’ve also reached out to owners.
Later that week, on October 30–31, the attacker used credentials obtained earlier to access one of our internal services, and we detected it. Our logs show they accessed internal configuration, including other credentials, and tried to use it to sign in to Bubble accounts; those attempts returned no data. We changed the credentials they used. We've reviewed this additional access, and it doesn't change our current findings on what data the attacker accessed or which app owners we're contacting.
Preliminary root causes
- An application vulnerability on older server infrastructure exposed temporary internal credentials.
- The temporary credentials gave access to further internal credentials, which the attacker used to reach other internal systems.
- Some platform features could be used to reach an app's stored credentials or data even when the app was set up correctly.
We’re holding back further technical detail while we continue to roll out and validate remediation. At this time, we have contacted all customers individually who we believe need to take action to secure their apps against these vulnerabilities. The final report will include more detail, along with the longer-term changes that address these root causes.
What we’ve done
For each cause above:
- We closed the initial path and completed updates to newer server infrastructure.
- We changed the credentials used by the attacker and are reviewing the rest of our credentials. Additionally, we narrowed what our internal systems can access.
- We've shipped fixes for the platform issues that enabled initial system compromise.
We've also expanded how we monitor for and respond to attacks, which are still active. Our expanded monitoring, together with the outside forensic firm's, gives us much broader visibility, and we continue to detect and respond to them.
What we’re still verifying
- The complete timeline, including if there was relevant activity prior to October 26.
- The full scope of what was accessed, through the outside firm's independent review.
Service disruption
Some of our containment steps disrupted service. These issues have been resolved.
- October 26: For several hours, plugins, file uploads, PDFs, mobile deploys, and AI app generation, didn't work correctly. Most were restored that evening.
- Dedicated environments: Infrastructure updates and later credential rotations caused disruption for some dedicated customers.
- October 30: Separately from containment, a code deploy to provide additional improvements to our security controls caused an outage of about 35 minutes for some apps. We rolled it back and tightened how we ship changes while the incident is ongoing.
Next update
We meet with the outside firm again this week and will post another update here by the end of the week. Questions are welcome in the security Q&A thread on the Bubble forum, and the status page always has the latest.
Timeline
| Date | What happened |
|---|---|
| Before Oct 26 | Activity we believe is linked to the same attacker against public-facing parts of our platform, going back to mid-October. We’ve performed an initial audit of those events and are waiting on the outside firm for full confirmation. |
| Oct 26 | Access to our production systems, detected the same day. We closed the initial path, began rotating credentials and emailed affected plugin developers. Containment steps disrupted several services; most were restored that evening. |
| Oct 26–28 | We updated the server infrastructure that host our applications and contacted dedicated customers who needed to make additional changes. |
| Oct 27 | We engaged an outside forensics firm. |
| Oct 28–30 | We detected the attacker accessing individual apps, and began contacting owners of the affected apps. |
| Oct 29 | We posted an update in the forum's security Q&A thread. |
| Oct 30 | An unrelated code deploy caused an outage of about 35 minutes for some apps. We rolled it back. |
| Oct 30–31 | The attacker used credentials obtained earlier to access one of our internal services. We detected it and rotated those credentials. |
| Nov 2 | The outside firm shared early findings, consistent with ours so far. |
Build your app on Bubble's Free plan. No need to upgrade until you're ready to launch your app.
Join building