Service degradation - A subset of users could encounter access issues to Spaces

Incident Report for Templafy

Postmortem

Investigation

The incident began on October 17, 2025, at 8:02 AM CET in West Europe (Production 0) and on October 20, 2025, between 8:48 AM CET and 9:15 AM CET across Production 1–5 clusters. It was detected on October 20, 2025, at 1:44 PM CET, after reports indicated that users were able to access Spaces within their own environment without being members of those Spaces.  The engineering team initiated the investigation at 3:34 PM CET on October 20, 2025. The root cause was traced to a recent change in the Library App that affected the authorization logic used when fetching Space data.

Mitigation

At 3:58 PM CET, the engineering team began implementing a fix. The first mitigation step involved rolling back the recent changes that introduced the authorization issue. This rollback effectively restored the correct filtering logic, ensuring users could only access Spaces to which they were authorized.

Resolution

By 4:29 PM CET on October 20, 2025, the rollback had been fully deployed across all production clusters. Following verification and monitoring, normal access restrictions were confirmed to be functioning correctly. The issue was considered resolved at this time.

Post-Incident Actions

To prevent similar issues from occurring, the following preventive measures are being implemented:

  • Expanding automated testing of the Library feature to ensure access controls work as intended.
  • Strengthening how internal APIs are protected by moving to a more restrictive allow-list model.
  • Providing additional security training for our engineering teams.
  • Improving our development environments to better detect access and permission issues before release.

Impact and Scope

This incident allowed existing platform users from within the same customer environment to view Spaces of which they weren’t members. The issue only affected customers with multiple Spaces. We have directly notified the small number of impacted customers. If you have not been contacted, no assets associated with your account were actively used during the incident window. While certain assets may have been visible to users within your own environment, there is no indication of any interaction, modification or misuse. It is important to note that the visibility issue is strictly confined to users within customers’ own environment — no users from other customers or external third parties had any access. The issue affected all production clusters:

  • West Europe (Production 0): October 17, 2025, 8:02 AM CET
  • West Europe (Production 1): October 20, 2025, 9:15 AM CET
  • East US (Production 2): October 20, 2025, 8:48 AM CET
  • Australia East (Production 3): October 20, 2025, 8:48 AM CET
  • Canada Central (Production 4): October 20, 2025, 8:48 AM CET
  • West Europe (Production 5): October 20, 2025, 9:15 AM CET

All affected clusters were promptly rolled back following the detection of the issue.We sincerely apologize for the disruption caused by this security-related incident. Protecting our customers’ data and maintaining strict access control standards are our highest priorities. We are taking immediate and long-term actions to strengthen safeguards, improve internal validation and reinforce our commitment to delivering a secure and reliable service.

Posted Oct 21, 2025 - 17:57 CEST

Resolved

The incident has been resolved, and further information will be provided in a postmortem shortly.

We apologize for the impact to affected customers.
Posted Oct 21, 2025 - 17:53 CEST

Monitoring

The issue was caused by a bug introduced in a recent Library App update, which temporarily allowed users to view Spaces within their tenant even if they were not members. The root cause has been identified, and the changes that introduced the bug have been rolled back. Access controls are now functioning as expected, and we are actively monitoring the system to ensure stability.
Further information will be provided in a postmortem shortly.
Posted Oct 20, 2025 - 17:08 CEST

Identified

We have identified an issue that affects a subset of customers and are working towards a resolution.
Further updates will be posted here soon.
Posted Oct 20, 2025 - 15:51 CEST
This incident affected: Templafy Hive (Library & Dynamics).