Templafy Status - Service degradation - Trusted publisher notification

Service degradation - Trusted publisher notification

Incident Report for Templafy

Postmortem

Summary:
Early this week, we identified an issue with Templafy VSTO Add-ins in Microsoft Office for PC. A subset of users received a “Publisher cannot be verified” notification when starting Office desktop applications that had Templafy VSTO Add-ins installed. Our service was not down, and users who chose to install the Office Add-in from the unknown publisher could continue using our product via Office desktop. As an immediate response, we assembled a team from Engineering, Product, Support, and InfoSec to scope and address the issue. Templafy identified that the issue was only impacting customers using a particular subset of VSTO add-ins and versions. Subsequently, on Tuesday, Templafy implemented and then deployed a fix for the impacted customers, without requiring any further action from the clients, to ensure that the issue was resolved.

Issue overview:
On Sunday, 30th of April, we received support tickets indicating users were being prompted with a Microsoft Office Customization Installer notification, stating “Publisher cannot be verified.” Upon investigation, we determined that the issue was limited to Windows users who opened Microsoft Word, PowerPoint, Excel, or Outlook using Templafy One and Hive with the following VSTO Add-ins and their versions:

Library version 6.1.36 - 6.1.73
Productivity version 5.0.701 - 7.1.205
Email Signature version 6.1.36 - 6.1.65

Templafy Web Add-ins (For MS Office on-prem, Online & Mac) were not impacted. Customers on the latest stable version of Templafy Hive were also not impacted.

Root cause:
Upon thorough investigation, we determined that the root cause was that the signing certificate, used to sign the VSTO Add-ins, could not be verified by Microsoft Office anymore due to a change in the Time Stamp Authority requirements.

The signing certificate we used was valid until the 28th of April. To ensure that Add-ins are valid after the certificate expires, we use a Time Stamp Authority (TSA) when signing add-ins. As long as the Time Stamp Authority is trusted, add-ins remain valid after the certificate expires. To enhance global security, on the 1st of June 2021, the CA/B Forum increased the required key length for Time Stamp Authority certificates to a minimum of 3072..) The Time Stamp Authority (TSA) we used when signing our add-ins was using a key length of 2048. Therefore, it was no longer trusted, resulting in the add-ins becoming invalid after the certificate expiration date.

We had anticipated this problem and had switched to a new Time Stamp Authority (TSA) when signing executable files in March 2021, but we had not correctly switched to the new TSA when signing manifest files. Because of this, after the certificate expiration date on the 28th of April, users with impacted VSTO Add-in versions who opened an Office application received the "Publisher cannot be verified" notification.

Resolution:
To resolve the issue, we manually re-signed Library, Email Signature, and Productivity VSTO Add-ins using a new certificate with a valid Time Stamp Authority without changing any behavior of the software. We manually validated the re-signed VSTO Add-ins with both Templafy One and Hive customers.

After confirmation, we re-signed all affected add-ins and programmatically updated Templafy Desktop packages for all affected Templafy One customers. Similarly, we programmatically updated all affected Hive customers to the re-signed add-ins. This process was completed by the end of Tuesday, 2nd of May. The packages with the re-signed VSTO Add-ins were automatically installed by Templafy Desktop on end users' PCs after midnight Tuesday (local time).

Going forward, we will take extra precautions to ensure that our signed packages remain valid and adhere to the necessary key length standards in order to prevent similar problems in the future.

Timeline:

Sunday, 30th of April: We identified the incident and received the first support tickets.
Monday, 1st of May: We began investigating and escalated to involve our Engineering, Product, Support, and InfoSec departments. We identified the scope, cause, and solution. At the end of Monday, we got confirmation from a few customers that the fix was working.
Tuesday, 2nd of May: We automatically re-signed all add-ins and updated all affected customers.
Wednesday, 3rd of May: Customers received the corrected VSTO Add-ins and the issue was officially resolved for all impacted customers.

Lastly, we would like to thank you for your patience and understanding during this incident. We know that disruptions to your business operations are costly and frustrating, and as outlined above, we are committed to doing everything in our power to prevent similar incidents from occurring in the future. If you have any further questions or concerns, please do not hesitate to reach out to support@templafy.com. We are always here to assist you and help you get the most out of our product. We value your business and appreciate your ongoing trust in our service.

Posted May 04, 2023 - 14:39 CEST

Resolved

The incident has been resolved, and further information will be provided in a postmortem.

We apologize for the impact to affected customers.

Posted May 03, 2023 - 18:24 CEST

Monitoring

We have mitigated the incident for all affected customers.

All affected customers have been updated to the re-signed versions of the add-ins. To deploy this update, the Templafy Desktop app will install the re-signed add-ins the next time it checks for updates, which will automatically happen after midnight local time. It can also be triggered manually by restarting the machine or through the Templafy Desktop application itself. No additional action will be needed from the admin or end-user.

For any customers using server mode (e.g. Citrix), an update would have to be triggered to fetch the newest update.

If the end-user has disabled the add-in in Office in the past 24 hours, they might need to manually activate it again. Read more here: https://support.templafy.com/hc/en-us/articles/115005420025

While this issue is resolved, we will not close this incident until we have confirmed updates have been deployed for all customers. Please expect a detailed post-mortem within the next 24 business hours.

To report any further issues, please reach out to support@templafy.com. Thank you for your patience as we resolved this issue.

Posted May 02, 2023 - 23:22 CEST

Update

We have updated the majority of affected customers on Templafy One to the re-signed versions of the add-ins.

Templafy Desktop will automatically install the re-signed add-ins the next time it checks for updates. This will automatically happen after midnight (local time) or can be triggered manually by the end-user by restarting the machine or through the Templafy Desktop application. No additional action will be needed from either the admin or end-user side.

In a few cases, we have not yet been able to update the add-ins. This is in two situations: If you are using Server Mode (e.g. Citrix) or have special characters in your existing package.

If an end-user has disabled the add-in in Office in the past 24 hours, they might need to manually activate it again. Read more here: https://support.templafy.com/hc/en-us/articles/115005420025

With this update, we have mitigated the incident for the majority of the affected customers.

We are continuing work to create an update that also applies to customers on Templafy Hive as well as customers using Server Mode add-ins.
If you’re on Templafy One (and the two cases above don’t apply to you) and still experience any issues, please reach out to support@templafy.com

Posted May 02, 2023 - 18:15 CEST

Update

We now have a full overview of affected versions and customers.
If you are a Hive customer on an automatic release track (like most tenants), you are not affected.

All affected versions of Library, Email Signature, ProductivityPlus, and Check have now been re-signed with the new certificate, and the re-signed add-in versions have been tested and confirmed working with multiple customers.

For customers on Templafy Hive:
New versions of add-ins have been uploaded to the admin center, and we are currently working on automatically updating all affected customers to the version containing the re-signed add-ins. The only change to the new add-in version will be the re-signing of add-in files.

For customers on Templafy One:
New versions of existing Desktop packages are being uploaded to the admin center. These new packages contain re-signed add-in files but are otherwise the same as the active desktop package.

New versions of ProductivityPlus, Check, and Email Signature have been uploaded to the admin center and are available from the dropdowns with an x.1 version extension. Like the other add-ins, no other changes than re-signing have happened to these add-ins.

Next Steps
We are actively working on an automatic update of the add-in version in the admin center for both One and Hive customers. In this automated update, versions will be changed to the re-signed version of what is already active on the tenant.

When this has been done, no additional action will be needed from either the admin or end-user side. We expect this to happen throughout today (European time).

Templafy Desktop will automatically install the re-signed add-ins the next time it checks for updates. This will automatically happen after midnight (local time) or can be triggered manually by the end-user by restarting the machine or through the Templafy Desktop application.

Customers using server mode (e.g. Citrix)
For customers using server mode such as a Citrix environment, an update of Templafy Desktop will need to be triggered on the customer side after the changes above have been rolled out.

Posted May 02, 2023 - 11:02 CEST

Update

Templafy has identified a solution for this issue and is in the process of coordinating automatic deployment to all customers to ensure service is returned without customer involvement.

Details of the fix:
- We have already re-signed all impacted Library add-ins and are in the process of re-signing all impacted e-mail signature and Productivity add-ins
- This is being validated for both One and Hive customers
Additional details for Hive customers:
- All Hive customers on either the stable or insider tracks have not been impacted
- After all impacted add-ins have been validated, we will deploy the fix for Hive customers immediately
Additional details for One customers:
- We are developing a method for programmatically updating all Templafy Desktop packages with re-signed add-ins
- We will provide an update ASAP as to when this deployment can be expected

Posted May 01, 2023 - 22:49 CEST

Update

The latest stable Templafy Hive add-in versions are not impacted by this issue. This issue impacts the following VSTO add-ins and their versions:
Library add-in version 6.1.36 - 6.1.73
Productivity version 5.0.701 - 7.1.205
Email Signature version 6.1.36 - 6.1.65
Templafy Hive customers experiencing this issue can upgrade to the latest stable version in order to resolve it.

Posted May 01, 2023 - 19:32 CEST

Update

Cause:
In April 2021, Templafy updated the timestamp server that we used to sign our Templafy VSTO certificates due to the timestamp server being purchased by another company. The certificates are used by Templafy to ensure that our software has not been tampered with and can be installed safely and silently by end users. Due to this change in timestamp server, Templafy started using a timestamp server that was flagged invalid at a later stage, therefore signing VSTO packages with the invalid timestamp server. Templafy is still identifying all affected VSTO versions.
Potential resolution:
Templafy is in the process of signing all new VSTO packages with the correct timestamp server as well as working diligently to identify a programmatic / centralized solution to make sure all previous VSTO packages are signed with the correct timestamp server.
Please continue to follow the status page for more updates as they become available.
Thank you.

Posted May 01, 2023 - 17:01 CEST

Update

The engineering team is continuing to work on the incident and we will provide more information once available.

Posted May 01, 2023 - 16:10 CEST

Update

We are continuing to investigate the issue as well as scope the affected versions of Templafy add-ins. We will continue to provide status updates every hour.

Posted May 01, 2023 - 15:12 CEST

Identified

We have identified the cause of "Publisher cannot be verified" messages. The recent routine certificate changes performed by Templafy have exposed a configuration change that was made on our root certificate timestamp servers. This issue has been escalated to the highest degree and all technical teams are involved to resolve the issue as soon as possible.

Posted May 01, 2023 - 13:58 CEST

Investigating

We have identified an issue where customers receive a notification popup when starting an Office application where Templafy VSTO Add-ins are installed. This only affects Windows users. For more information about it and how to proceed please see our knowledge base article: https://support.templafy.com/hc/en-us/articles/10485903891485-Trusted-publisher-notification

Currently, our engineering team is actively working on implementing a solution.

Posted May 01, 2023 - 12:03 CEST

This incident affected: Templafy Hive (Add-in Management).