Incident Initiation
To improve the automation and security of package management, the engineering team implemented a series of changes. These updates included expanding the status options for each package and involved a database migration. However, the changes unintentionally caused the endpoint responsible for retrieving current packages for users to incorrectly exclude third-party packages. Consequently, these packages were removed from users' computers during the next Templafy Desktop run, making the associated Add-ins unavailable in Office applications.
This change was deployed across all environments at 8:20 AM CET on October 23, 2024, though it only started impacting users after the Templafy Desktop component triggered a package update. The issue was first reported at 11:44 AM CET.
Investigation
The incident was placed at 12:10 PM CET, when the software engineering team was notified. The root cause was quickly identified and the engineering team immediately started implementing a fix to ensure the endpoint responsible for retrieving Add-ins correctly included third-party packages.
Impact and Scope
The incident impacted all regions but affected only specific tenants with third-party Add-ins installed. Among these, only users whose Templafy Desktop triggered a package update within the incident timeframe were affected.
Resolution
By 3:14 PM CET, the code fix was successfully deployed across all production environments. This update allowed the Templafy Desktop component to retrieve and reinstall third-party Add-ins, enabling users to access them again following a Templafy Desktop synchronization.
Post-Incident Actions
The engineering team has commenced an extensive review to expand test cases for package delivery, activation, and security assertion, ensuring future enhancements to package functionality do not impact the integrity and stability of existing configurations.