Connectivity Issues
Incident Report for Templafy
Postmortem

We sincerely apologize for the inconvenience and loss of service experienced today with Templafy. We would like to explain what happened, how we reached a quick resolution and how we are committed to a secure solution.

Summary

At 10:28am CEST on the 11/09/2018, Templafy’s web application and add-ins were unavailable for the total duration of 20m and 22s. Templafy was running as expected by 10:53am CEST.

Templafy was first alerted, by our internal monitoring system, that our response time was slowing. Our DevOps team started to investigate the issue immediately. A connection timeout notification was then received at 10:28am.

Root cause

This was caused by an unexpected number of concurrent users triggering a large volume of requests to our database. Also, a recent update on how we detect new offline content made this check relatively slow.

Resolution

We doubled the Database capacity and the load is now below normal use.

Future-Proofing

We will keep an extra eye on the system the coming period and ensure we setup more monitoring to prevent similar situations in the future.

Posted Sep 11, 2018 - 16:54 CEST

Resolved
The problem has been resolved but we are still investigating the root cause.
Posted Sep 11, 2018 - 11:07 CEST
Update
We are continuing to work on a fix for this issue.
Posted Sep 11, 2018 - 11:00 CEST
Update
We are continuing to work on a fix for this issue.
Posted Sep 11, 2018 - 10:54 CEST
Identified
The issue has been identified and our team is currently looking at it
Posted Sep 11, 2018 - 10:42 CEST
This incident affected: Templafy One (WebApp, Public API).