On Monday, March 21, 2021, at 13:25 GMT, PaperCut Pocket and PaperCut Hive experienced an outage for a duration of 5 hours and 45 minutes. This affected printing for all customers using our platform.
The cause of the outage has been identified as several Edge Nodes existing within customer environments being blocked from automatically updating. These nodes were stuck running a legacy version that included a bug that has since been rectified.
At the start of the incident period, one of our cloud services encountered a minor issue that, in conjunction with this small number of legacy nodes, led to a sustained 1000% increase in traffic to our services. This increase was due to the affected nodes making an incorrect call every 60 nanoseconds rather than every 60 minutes.
We have already taken action to quarantine the affected nodes and have increased the capacity of our systems to handle unexpected events.
Further to this, we continue to evolve our PaperCut Hive platform to improve our services' resiliency, performance, and availability.
We apologize to our customers whose businesses were impacted during this disruption.