PaperCut Pocket and Hive platform issue

Incident Report for PaperCut Software

Postmortem

On Monday, March 21, 2021, at 13:25 GMT, PaperCut Pocket and PaperCut Hive experienced an outage for a duration of 5 hours and 45 minutes. This affected printing for all customers using our platform.

The cause of the outage has been identified as several Edge Nodes existing within customer environments being blocked from automatically updating. These nodes were stuck running a legacy version that included a bug that has since been rectified.

At the start of the incident period, one of our cloud services encountered a minor issue that, in conjunction with this small number of legacy nodes, led to a sustained 1000% increase in traffic to our services. This increase was due to the affected nodes making an incorrect call every 60 nanoseconds rather than every 60 minutes.

We have already taken action to quarantine the affected nodes and have increased the capacity of our systems to handle unexpected events.

Further to this, we continue to evolve our PaperCut Hive platform to improve our services' resiliency, performance, and availability.

We apologize to our customers whose businesses were impacted during this disruption.

Posted Mar 31, 2022 - 07:43 UTC

Resolved

Our PaperCut Pocket & Hive services have now been restored.

This issue affected our cloud services at 13:25 GMT / 00:25 AEST / 9:25 EST today. Services were restored at approx 19:10 GMT / 06:10 AEST / 15:10 EST.

While services are now online, we continue to investigate the outage’s root cause. During this time, we apologize if any further disruption is experienced. We will provide updates on our status page if there is any additional service impact.

We will provide a detailed statement on this incident in due course once the reason for the outage has been thoroughly investigated.

We apologize to all of our customers for the interruption to your printing today.

Posted Mar 21, 2022 - 19:10 UTC

Update

We are continuing to experience an outage of PaperCut Pocket and PaperCut Hive. Our team is actively investigating this incident, and we apologize for any inconvenience caused during this time. We will continue to provide updates on this page as investigations continue.

Posted Mar 21, 2022 - 17:08 UTC

Update

Posted Mar 21, 2022 - 15:41 UTC

Update

Posted Mar 21, 2022 - 14:59 UTC

Update

We are continuing to investigate the outage affecting both PaperCut Pocket and PaperCut Hive. We apologize for any inconvenience caused. We will provide a further update as soon as possible.

Posted Mar 21, 2022 - 14:17 UTC

Update

We are currently investigating an outage affecting PaperCut Pocket and PaperCut Hive. We will provide a further update shortly.

Posted Mar 21, 2022 - 13:43 UTC

Update

We're actively working on the incident occurring with our cloud services. More updates to come.

Posted Mar 21, 2022 - 13:25 UTC

Investigating

We're experiencing an issue with our cloud services and are currently looking into the issue.

Posted Mar 21, 2022 - 13:25 UTC

This incident affected: PaperCut Pocket & PaperCut Hive.