On November 24th, 2023, starting at 6:39 PM CET, users of Templafy Hive in the Australia East (Production 3) region started to experience degraded application performance when authenticating to the application. The root cause of the problem was an external dependency, which started to have availability issues, this propagated to our application, because the current implementation logic was reacting in a way that caused threads to not get enough resources to work properly in our application for authentication requests.

By 6:42 PM CET, our engineers were notified by our automatic alert system about the slow performance. The root cause was quickly identified, and the issue was mitigated by the engineering team. This action restored the proper functionality to the system, successfully resolving the incident by November 24th at 10:15 PM CET. Currently, the engineering team is working on improving the system making it more resilient for similar occurrences.