Incident Initiation
At 9:26 AM CET on March 31, 2025, engineers detected an issue affecting global feature flag functionality. Certain features controlled by feature flags, such as the collapsible task pane, became temporarily unavailable for a small subset of users in West Europe (Production 0).
Investigation
Between 9:30 AM and 10:10 AM, engineers conducted an investigation to identify the root cause. It was determined that the newly deployed Foundation package version contained a bug that disrupted global feature flag functionality. The impact was confined to West Europe (Production 0) systems that had upgraded to this version.
Mitigation and Resolution
At 10:10 AM, engineers initiated mitigation efforts by rolling back the faulty Foundation package version to the previous stable release. This rollback was applied across all affected services.
At 11:21 AM, the final fix was successfully deployed to production, restoring feature flag functionality. To prevent further impact, production releases were temporarily blocked until a full fix was delivered across all systems.
Impact and Scope
Scope: The incident was limited to a subset of customer on our West Europe (Production 0) cluster who were using certain feature flags.
User Experience: Feature flag-controlled functionalities, including the collapsible task pane, were temporarily unavailable.
Post-Incident Actions
To prevent similar incidents in the future, the engineering team will review the Foundation package update process and strengthen automated testing for feature flag dependencies before deployment.