Summary of the incident:
January 27th 2020 Templafy web app was unreachable, which resulted in tenants being unavailable and users experiencing a blank page in the web browser when attempting to access it. Our team was notified of the degraded performance at 5:28 UTC from our automated alert systems.
Investigations were initiated immediately and the issue was quickly identified.
Templafy web app was fully operational at 6:15 UTC.
Details of the incident:
At 5:28 UTC our automated alert systems detected an issue.
Investigation proved that a number of servers were not serving incoming traffic as expected, which subsequently caused remaining working servers to overload, due to the unexpected, heavy load.
A more detailed overview of the incident is described in the following breakdown:
A restart of all servers.
Root cause of incident:
All internal investigations so far points to problems on Microsoft Azure platform and we are in contact with Microsoft to further investigate and understand the two core problems found on their side: