On August 3, 2023, starting 8:40 AM, 5% of the users started to have high response times which lead to slowness in the WebApp and customer cases started to appear. At 10:49 AM CET the engineers started to investigate the issue. The engineers identified the issue immedietly and started mitigating the issue by slowing down some of the large internal requests for background tasks to prevent slowness. At 11:40 AM CET the slowness disappeared and daily work could be processed normally. We monitored the behavior until 2:21 PM CET and could not see any slowness, both from our side and customers reported that the issue is not affecting them anymore. The incident was mitigated.

To prevent this issue in future, we aligned the incoming request amount with our infrastructure scaling rules to be more reactive when large requests are being handled by our background tasks.