On the 24-th of April at 9:05 CET an incident occured that affected users using our West Europe (Production 1) cluster.
The incident resulted from an execution plan update of the SQL Server, causing queries to execute without utilizing the appropriate database index. Consequently, queries on the UserManagement database experienced high resource consumption, leading to severe slowdowns and eventual inaccessibility, causing the application on the West Europe (Production 1) cluster to become inaccessible. At 9:41 CET mitigation was achieved by scaling up SQL Server resources. Currently, the engineering team is actively investigating solutions to prevent similar incidents in the future.
Posted Apr 25, 2024 - 13:26 CEST
Resolved
The incident has been resolved, and further information will be provided in a postmortem shortly.
We apologize for the impact to affected customers.
Posted Apr 24, 2024 - 12:46 CEST
Monitoring
The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.
Posted Apr 24, 2024 - 09:43 CEST
Identified
We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.
Posted Apr 24, 2024 - 09:32 CEST
This incident affected: Templafy Hive (User Management).