Incident Initiation:

On September 4, 2024, at 11:59 AM CET, an anomaly was detected in the internal component of Hive, which initiated an incident. The issue specifically affected the campaign designer, causing newly uploaded images to disappear for users.

Investigation:

The engineering team immediately began investigating the issue. It was discovered that the newly uploaded images were not being transferred to long-term storage due to a malfunction in the storage service. The root cause was identified as a misconfiguration introduced during a recent update to the image handling system.

Mitigation and Resolution:

A temporary mitigation was put in place to ensure that images uploaded after the detection of the issue were properly stored. Following this, a correction was developed and deployed at 01:44 PM CET to restore the application's functionality. This fix addressed the underlying configuration error, and the system returned to normal operation.

Impact and Scope:

The incident impact was on across all regions and affected only certain tenants with a specific configuration attempting to upload images during the incident window.

Post-Incident Actions:

Procedural improvements will be implemented, including enhanced monitoring of storage services. The engineering team is also reviewing the update process to prevent similar misconfigurations.