PaperCut Hive - Service Outage
Incident Report for PaperCut Software
Postmortem

On Monday, March 23, 2021, at 14:45 GMT, PaperCut Pocket and PaperCut Hive experienced an outage for a duration of 15 minutes. This affected printing for all customers using our platform and was related to the outage on Monday, 21st March 2021.

The cause of the outage has been identified as several Edge Nodes existing within customer environments being blocked from automatically updating. These nodes were stuck running a legacy version that included a bug that has since been rectified.

At the start of the incident period, one of our cloud services encountered a minor issue that, in conjunction with this small number of legacy nodes, led to a sustained 1000% increase in traffic to our services. This increase was due to the affected nodes making an incorrect call every 60 nanoseconds rather than every 60 minutes.

We have already taken action to quarantine the affected nodes and have increased the capacity of our systems to handle unexpected events.

Further to this, we continue to evolve our PaperCut Hive platform to improve our services' resiliency, performance, and availability.

We apologize to our customers whose businesses were impacted during this disruption.

Posted Mar 31, 2022 - 07:44 UTC

Resolved
Our PaperCut Pocket & Hive services have now been restored.

This issue affected our cloud services at 14:45 GMT / 01:45 AEST / 10:45 EST today. Services were restored at approx 15:00 GMT / 02:00 AEST / 11:00 EST.

While services are now online, this outage is linked to the incident on Monday 21st March which is actively being investigated. During this time, we apologize if any further disruption is experienced. We will provide updates on our status page if there is any additional service impact.

We will provide a detailed statement on this incident in due course once the reason for the outage has been thoroughly investigated.

We apologize to all of our customers for the interruption to your printing today.
Posted Mar 23, 2022 - 15:07 UTC
Monitoring
PaperCut Pocket and PaperCut Hive cloud services are now returning to normal. We are monitoring for any further interruptions.
Posted Mar 23, 2022 - 15:00 UTC
Investigating
We are currently investigating an outage of the PaperCut Pocket and PaperCut Hive cloud services. We will provide a further update shortly.
Posted Mar 23, 2022 - 14:45 UTC
This incident affected: PaperCut Pocket & PaperCut Hive.