Service degradation - Content library is not working on web addin in West Europe (Production 0)

Investigation

On July 17, 2025, at 1:25 PM CET, an issue was introduced in the web add-in's collapsible task pane. At 3:33 PM CET, the engineering team was alerted to an issue affecting the content library within the web add-in's collapsible task pane in Templafy Hive. Users across multiple tenants in West Europe (Production 0) reported difficulties accessing content, leading to disruption in daily workflows. Investigation efforts began promptly at 3:38 PM CET. By 4:29 PM CET, the team had confirmed that the incident was isolated to West Europe (Production 0) and traced the cause to recent changes in the system's routing, which inadvertently affected the loading of essential services such as library content, AI assistant, find templates, etc. in the collapsible task pane were not available.

Mitigation

As soon as the root cause was identified, the engineering team reverted the recent routing-related changes that had been deployed in the affected system at 4:53 PM CET. This rollback was performed immediately after the issue was diagnosed and was closely monitored to ensure the restoration of normal service.

Resolution

Following the rollback, functionality within the web add-in's content library was successfully restored for all impacted users in West Europe (Production 0). The engineering team continued to monitor system performance to confirm that no further issues persisted and resolved the incident at 4:57 PM CET.

Post-Incident Actions

To help prevent similar incidents in the future, the engineering team will:

• Improve testing coverage for the Web Add-In both locally and within the testing environment
• Implement additional automated tests targeting critical features such as the content library

Impact and Scope

This incident impacted multiple tenants served by the West Europe (Production 0) cluster, specifically users of the web add-in's content library in the Templafy Hive environment. The issue was isolated to this cluster and did not affect other environments or regions.

We sincerely apologize for the disruption caused by this incident. Ensuring a reliable and seamless experience for our users remains our highest priority, and we are actively working to further strengthen our testing and deployment processes.