Investigation
The issue began at 8:00 AM CET on October 20, 2025, when users experienced errors while creating documents across West Europe (Production 0) in the Hive environment. It was detected at 8:56 AM CET, prompting an immediate investigation by the engineering team.
Initial observations suggested that the problem originated when document generation was failing due to an authentication issue with the Azure OpenAI integration. Further review of recent changes revealed that the issue was caused by introducing a new dependency which had incorrect authentication configuration set up.
Mitigation
At 9:33 AM CET, the engineering team began implementing a mitigation plan to fix the configuration issue for the new dependency. The fix involved aligning the authentication configuration of the new service.
Resolution
At 12:57 PM CET, the fix was deployed across West Europe (Production 0) in the Hive environment. The authentication alignment successfully restored document generation capabilities for affected users. Post-deployment monitoring confirmed service stability, and no further issues were observed.
Post-Incident Actions
To prevent similar issues from recurring, we’re implementing a smoke test for document generation to detect authentication discrepancies early.
Impact and Scope
This incident affected users creating documents across West Europe (Production 0) in the Hive environment. All other platform functionalities remained stable throughout the event, and no data loss occurred.
We sincerely apologize for the disruption caused by this incident. Maintaining a reliable and consistent customer experience remains our highest priority. We are taking the outlined steps to strengthen testing and validation processes and to prevent similar issues in the future.