tag:status.papercut.com,2005:/historyPaperCut Software Status - Incident History2024-03-19T08:49:28ZPaperCut Softwaretag:status.papercut.com,2005:Incident/191949502023-11-23T07:00:49Z2023-11-23T23:06:46ZPaperCut Known Issues<p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>07:00</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>06:45</var> UTC</small><br><strong>Update</strong> - This incident is now resolved - please reach out via support.papercut.com if you notice anything out of the ordinary.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>13:30</var> UTC</small><br><strong>Identified</strong> - The PaperCut Known issues page is currently unavailable, we are currently investigating.</p>tag:status.papercut.com,2005:Incident/190635432023-11-09T00:45:37Z2023-11-09T00:45:37ZMultiverse services unavailable in AU datacenter<p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>00:45</var> UTC</small><br><strong>Resolved</strong> - Our services in our PaperCut Multiverse AU datacenter have now been resolved.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>23:05</var> UTC</small><br><strong>Investigating</strong> - We've identified an issue related to all services in PaperCut Multiverse AU datacenter. Our engineers are currently working with our cloud service provider to rectify the problem.</p>tag:status.papercut.com,2005:Incident/190633652023-11-09T00:45:00Z2023-11-09T00:45:00ZPaperCut Pocket and Hive platform issues in AU datacenter<p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>00:45</var> UTC</small><br><strong>Resolved</strong> - Our services in our PaperCut Hive / PaperCut Pocket AU data center have now been resolved.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>22:41</var> UTC</small><br><strong>Identified</strong> - We've identified an issue related to all services in PaperCut Hive / PaperCut Pocket AU datacenter customers. Our engineers are currently working with our cloud service provider to rectify the problem. We apologise to the customers affected.</p>tag:status.papercut.com,2005:Incident/189746962023-10-31T06:02:54Z2023-10-31T06:02:54ZIssues accessing PaperCut Hive customers in non-US data centers from Multiverse<p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>06:02</var> UTC</small><br><strong>Resolved</strong> - This issue has now been resolved - the issue was due to a change related to supporting the additional PaperCut Hive data centers, and is now fixed.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>03:12</var> UTC</small><br><strong>Identified</strong> - We are currently investigating issues with accessing PaperCut Hive customers from Multiverse that are in the EU, AU, and UK data centers.<br /><br />Customers that are in the US are not affected. We are able to reproduce this and will update here as soon as we know more. This does not impact PaperCut Hive usage at all - customers will not see any impact when using PaperCut Hive.</p>tag:status.papercut.com,2005:Incident/188311762023-10-17T20:49:39Z2023-10-17T20:49:39ZIntermitted issues Portal and Ordering system<p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>20:49</var> UTC</small><br><strong>Resolved</strong> - The issue has been resolved. <br /><br />We are reviewing the issue and logs to work to find out the root cause of the issue but for now, this issue is resolved and systems are back to being fully operational.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>20:21</var> UTC</small><br><strong>Investigating</strong> - We have reports of users having issues accessing the PaperCut Portal and the MF ordering platform. <br /><br />The issues seems to be intermitted and only affecting some users. <br /><br />We are investigating the issue</p>tag:status.papercut.com,2005:Incident/188263902023-10-17T13:55:43Z2023-10-17T13:55:43ZPaperCut Hive - Konica Minolta - Unable to uninstall embedded software<p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>13:55</var> UTC</small><br><strong>Resolved</strong> - The issue affecting the ability to uninstall the full embedded app on Konica Minolta has now been resolved. Customers should now be able to uninstall the full embedded app successfully. <br /><br />If you continue to experience any issues, please reach out to our support team: https://support.papercut.com.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>12:44</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating reports that a blank screen is displayed when attempting to uninstall the Konica Minolta full embedded app. We will provide a further update shortly.</p>tag:status.papercut.com,2005:Incident/187771262023-10-13T10:01:12Z2023-10-13T10:01:12ZPaperCut HIVE | Toshiba MFDs failing to load correctly<p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>10:01</var> UTC</small><br><strong>Resolved</strong> - The error we were previously experiencing with our cloud services has been resolved.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>10:00</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>09:50</var> UTC</small><br><strong>Investigating</strong> - We are currently receiving reports that the embedded interface on Toshiba MFDs for PaperCut HIVE are failing to load correctly, with code strings appearing on action buttons and information fields.</p>tag:status.papercut.com,2005:Incident/186807232023-10-04T07:58:32Z2023-10-04T07:58:32ZPaperCut Pocket / Hive - Cloud Node unavailable<p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>07:58</var> UTC</small><br><strong>Resolved</strong> - We have resolved the technical difficulties affecting our Cloud Node, and our services have now been restored. If you experience any issues with printing, please reach out to our team: https://support.papercut.com</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>18:54</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing technical difficulties with our PaperCut Pocket / Hive Cloud Node. Print jobs are currently unable to be submitted.<br />Any jobs stored on our Cloud Node are currently unavailable for release.<br />Any new mobile print jobs, or jobs where a local edge node is not available will fail during submission.</p>tag:status.papercut.com,2005:Incident/186687582023-10-03T03:37:53Z2023-10-03T03:37:53ZIntermittent issues with logging into (and ordering) on PaperCut Partner Portal<p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>03:37</var> UTC</small><br><strong>Resolved</strong> - This issue has now been resolved and has not occurred since.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>17:29</var> UTC</small><br><strong>Monitoring</strong> - After initial investigation, the issue has since resolved itself - the problem occurred between ~1:30PM PDT to 3PM PDT. The underlying cause seems to be cloud infrastructure related and is currently being investigated.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>17:28</var> UTC</small><br><strong>Investigating</strong> - An issue has been identified with intermittent issues to our Partner Portal which resulted in degraded performance in placing orders and quotes in our order system. We are currently investigating.</p>tag:status.papercut.com,2005:Incident/186067542023-09-27T00:16:21Z2023-09-27T00:16:21ZCloud Provider Degraded Performance Impacting Print Job Logging and Mobile Notifications<p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>00:16</var> UTC</small><br><strong>Resolved</strong> - The incident involving our cloud service provider has now been resolved. The issues affecting print job visibility and mobile notifications have been successfully addressed.</p><p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>02:20</var> UTC</small><br><strong>Monitoring</strong> - We are currently addressing an ongoing incident involving our cloud service provider. Presently, our provider is experiencing degraded performance for one of their services, this potentially is affecting the visibility and logging of print jobs in both the Job Log page and mobile notifications.<br /><br />Our team is actively monitoring the situation and maintaining direct communication with our cloud provider's support team to ensure real-time updates on their progress in resolving the issue. While they endeavor to restore normal service, please be aware of potential delays in print job logging and mobile notifications during this period.</p>tag:status.papercut.com,2005:Incident/184811542023-09-12T20:21:38Z2023-09-12T20:21:38ZPaperCut Pocket/Hive and PaperCut Mobility Cloud Print - Printing Failure<p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>20:21</var> UTC</small><br><strong>Resolved</strong> - After monitoring the infrastructure changes that have been made we can confirm that issues around our IoT Core service have been fully rememdied.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>20:20</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>17:59</var> UTC</small><br><strong>Update</strong> - We're continuing to monitor the situation and will provide additional updates here.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>17:59</var> UTC</small><br><strong>Monitoring</strong> - Our IoT core service is now operating on separate infrastructure and functionality has been restored. PaperCut Hive / Pocket and PaperCut Mobility Cloud Print are now fully operational.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>15:36</var> UTC</small><br><strong>Identified</strong> - Our Product Development Team has identified an issue with the infrastructure that supports the IoT Core service of both PaperCut Hive / Pocket and PaperCut Cloud Print. <br /><br />The IoT Core service is being migrated to separate infrastructure as a means of resolving this outage. <br /><br />Symptoms as it relates to PaperCut Pocket / Hive:<br />Print Release failure, namely 502 errors when attempting to release print jobs.<br /><br />Symptoms as it relates to PaperCut Mobility Print:<br />Delayed communication in printing or complete printing outage as PaperCut Mobility Cloud Print relies on this service to orchestrate the Peer to Peer connection between clients and the Mobility Print server.</p>tag:status.papercut.com,2005:Incident/184593162023-09-11T11:12:20Z2023-09-11T11:12:20ZPaperCut Partner Portal - login failures<p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>11:12</var> UTC</small><br><strong>Resolved</strong> - Systems are all functioning as expected - please reach out to customer.service@papercut.com if you have any issues.</p><p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>06:48</var> UTC</small><br><strong>Monitoring</strong> - We are still currently monitoring systems to ensure they are running smoothly.</p><p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>05:09</var> UTC</small><br><strong>Update</strong> - We have now exited maintenance mode and things should be working as expected, and we are currently monitoring systems to ensure they are running smoothly. Please reach out to customer.service@papercut.com if you notice any issues.</p><p><small>Sep <var data-var='date'>10</var>, <var data-var='time'>16:47</var> UTC</small><br><strong>Investigating</strong> - We are currently updating our ordering system, and as a result, the order system, and PaperCut Portal will be unavailable until 5pm on Monday the 11th September (AEST).</p>tag:status.papercut.com,2005:Incident/181376592023-08-14T03:40:40Z2023-08-22T01:00:44ZPaperCut Hive and Pocket outage<p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>03:40</var> UTC</small><br><strong>Resolved</strong> - During a routine infrastructure update, a certificate was dropped from the load balancer. This impacted access to hive.papercut.com and pocket.papercut.com and impacted printing and print release. Automated monitoring alerted us to the issue, and was resolved by re-applying the certificate to the load balancer.<br /><br />The issue has now been completely resolved and the system is stable. Please get in touch with our support team at https://support.papercut.com if you continue to experience any issues.</p>tag:status.papercut.com,2005:Incident/181019262023-08-09T23:12:51Z2023-08-10T04:41:07ZPaperCut Hive/Pocket partial outage<p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>23:12</var> UTC</small><br><strong>Resolved</strong> - The degraded performance of PaperCut Hive/Pocket has been resolved and the service is stable after a period of monitoring. Improvements to the associated infrastructure are taking place as a means of mitigating this moving forward while also improving overall stability and performance.</p>tag:status.papercut.com,2005:Incident/180976352023-08-09T23:12:44Z2023-08-09T23:13:33ZMobility Cloud Print Partial Outage<p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>23:12</var> UTC</small><br><strong>Resolved</strong> - The partial outage of Mobility Cloud Print services has been resolved and the service is stable after a period of monitoring. Improvements to the associated infrastructure are taking place as a means of mitigating this moving forward while also improving overall stability and performance.</p><p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>20:20</var> UTC</small><br><strong>Monitoring</strong> - Cloud Print services have now reached a functional and stable state. We're continuing to monitor the situation while these services are again running on our primary infrastructure.</p><p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>17:54</var> UTC</small><br><strong>Identified</strong> - Post investigation, our development team has identified the root cause of this partial outage. Cloud Print services are being temporarily migrated to other infrastructure in an effort to restore functionality. <br /><br />This partial outage will affect some existing Cloud Print customers and any new Cloud Print servers that attempt to register.<br /><br />Once the underlying issue has been resolved, these services will then be migrated back.</p>tag:status.papercut.com,2005:Incident/179100222023-07-20T09:41:43Z2023-07-20T09:42:02ZMobility Cloud Print Service Degraded<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>09:41</var> UTC</small><br><strong>Resolved</strong> - This Mobility Cloud Print issue has been resolved, and all systems are functioning normally. If you are still experiencing issues you can restart the Mobility Print Server Service to regain feature functionality.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>09:25</var> UTC</small><br><strong>Investigating</strong> - The Mobility Print team have migrated services to a backup server temporarily to regain feature functionality. The team will continue to investigate the primary server issue and eventually flip the services back to the primary server once the issues are rectified. If you are still experiencing issues you can restart the Mobility Print Server Service to reconnect straight away.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>09:13</var> UTC</small><br><strong>Identified</strong> - An issue with Mobility Print's Cloud Print feature has been identified, and the team has been notified.</p>tag:status.papercut.com,2005:Incident/178282482023-07-12T01:44:35Z2023-07-12T01:44:35ZPaperCut Hive Scan to OneDrive Outage<p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>01:44</var> UTC</small><br><strong>Resolved</strong> - The issue with scanning to OneDrive for Business, OneDrive for Personal and Sharepoint using PaperCut Hive has now been fully resolved.<br /><br />Since the Microsoft token used for OneDrive and Sharepoint has been refreshed, this will mean that all users performing scans to OneDrive or Sharepoint since this refresh will need to re-authorize their access. Subsequent scans after re-authorizing will go through without any additional actions.</p><p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>22:36</var> UTC</small><br><strong>Identified</strong> - The underlying cause of this outage has been identified. We've confirmed that this only impacts OneDrive - no other cloud storage providers are affected.<br /><br />We're actively working to restore functionality.</p><p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>21:44</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating a partial outage with Hive integrated scanning. Scan to OneDrive is currently experiencing an outage where all scans are triggering authorization emails that fail to authorize the scan with an "Oops Something went wrong" message. This is the only destination / storage provider experiencing this issue at this time.</p>tag:status.papercut.com,2005:Incident/176186982023-06-19T11:34:10Z2023-06-19T13:56:51ZPaperCut Pocket & PaperCut Hive - Unable to release print jobs from the Cloud Node<p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>11:34</var> UTC</small><br><strong>Resolved</strong> - We have resolved the issue affecting printing for PaperCut Pocket & PaperCut Hive users. This issue occurred between 07:20 UTC to 10:25 UTC on 19th June 2023. <br /><br />1. At the time of print release, jobs that were only available to download and print from our Cloud Node could not be released. <br />2. Jobs failed to be submitted from iOS or Android devices.<br /><br />Please note that jobs submitted while this issue was occurring may need to be submitted again for them to be released successfully. <br /><br />If you continue to see any issues, please reach out to our support team: https://support.papercut.com.</p><p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>10:00</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating reports that print jobs are failing to print when only available on the PaperCut Pocket/Hive Cloud Node.</p>tag:status.papercut.com,2005:Incident/166233712023-03-26T10:05:46Z2023-03-26T10:05:46ZMaintenance work to increase system capacity<p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>10:05</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>10:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>21:12</var> UTC</small><br><strong>Scheduled</strong> - We will be having a small maintenance window where we will be updating and increasing capacity on some of our cloud systems. During this period printing may be affected.</p>tag:status.papercut.com,2005:Incident/164773782023-03-13T16:36:42Z2023-03-13T22:39:17ZPaperCut Hive - Intermittent admin interface login failures<p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>16:36</var> UTC</small><br><strong>Resolved</strong> - Our authentication partner, Auth0, has resolved the issues causing increased error rates across their services. This has, in turn, resolved the errors when attempting to log in to the PaperCut Pocket and Hive Admin Interface. See the Auth0 status page for details about the incident: https://status.auth0.com/incidents/41gk3rcgm9vg<br /><br />Please get in touch with our support team at https://support.papercut.com if you continue to experience any issues.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>16:17</var> UTC</small><br><strong>Update</strong> - Our authentication partner, Auth0, is currently reporting an elevated number of timeouts and error rates in their environments which is affecting login attempts to PaperCut Pocket and PaperCut Hive. This is being investigated by Auth0. We will provide a further update shortly.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>15:46</var> UTC</small><br><strong>Investigating</strong> - We are investigating reports of intermittent issues accessing the PaperCut Hive admin interface. This may be seen as an error when logging in or the system taking longer than usual to process requests. We will provide a further update shortly.</p>tag:status.papercut.com,2005:Incident/163657532023-03-07T05:30:45Z2023-03-07T02:13:24ZError: "error parsing addon gw url" when trying to add a Scanning add-on in PaperCut Hive<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>05:30</var> UTC</small><br><strong>Resolved</strong> - Error: "error parsing addon gw url" is shown when trying to add a Scanning add-on in PaperCut Hive for the first time. Add-ons which had already been enabled would continue to work successfully. Admins would not be able to add any new Scanning add-ons. This would also have impacted the ability to create new quick scans.<br /><br />This issue has now been resolved.</p>tag:status.papercut.com,2005:Incident/158562392023-01-16T13:30:00Z2023-01-16T18:28:19ZPaperCut Hive - Jobs stuck analyzing / showing as zero pages in the Job Log<p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>13:30</var> UTC</small><br><strong>Resolved</strong> - At approximately 13:30 UTC we were investigating reports of jobs showing as "0" pages in the Job Log which resulted in print jobs not showing up for release when logging into the MFD. <br /><br />By 15:55 UTC a new Edge Node version (2023-01-16-1453) was promoted to production resolving the issue. <br /><br />We confirmed that the issue stemmed from an Edge Node version that had been promoted to production last week (2023-01-12-2301). With Edge Node version 2023-01-16-1453 being pushed to production Edge Nodes should begin to update automatically over the next 24 hours. Alternatively, Hive admins can use the "Update Now" located on the Edge Mesh page of the Admin UI to force an edge node to update.</p>tag:status.papercut.com,2005:Incident/146943972022-12-18T05:00:27Z2022-12-18T05:00:27ZPaperCut Hive - 30 minutes of downtime for most of the Embedded Full Apps and the Easy Release Apps<p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>05:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>01:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>22:10</var> UTC</small><br><strong>Scheduled</strong> - We are updating the certificates used to serve some of the Embedded Full Apps and all of the Easy Release Apps. We are moving to GCP-managed certificates so that this process will be seamless (with no downtime) in the future. As part of this move we'll be updating our DNS entries which will impact the devices listed below.<br /><br />As a result of this change, we estimate that there could be up to a 30 minute outage for Konica-Minolta, Fuji Xerox, HP OXP, Sharp, Xerox, Ricoh and Kyocera full embedded apps, as well as all of the Easy Release Apps, within the 4 hour maintenance window.<br /><br />- Saturday 17th: 5pm to 9am US Pacific time (PST)<br />- Sunday 18th: 1am to 5am UK time (GMT)<br />- Sunday 18th: 12pm to 4pm AU time (AEDT)<br /><br />Customers will not need to do anything, however after the maintenance window if you are experiencing issues, please restart the device. If you have any further questions please contact us and quote CDSS-1916.</p>tag:status.papercut.com,2005:Incident/145345842022-12-06T16:10:47Z2022-12-07T05:13:09ZPaperCut Hive - Users are unable to release jobs from Ricoh and Kyocera Full Embedded Apps, and from all types of Lite release apps (mobile release unaffected)<p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>16:10</var> UTC</small><br><strong>Resolved</strong> - At around 15:00 AEDT we were investigating reports of an error 'Error releasing print jobs: Authentication required' when attempting to release jobs on Kyocera or Ricoh devices using the Full embedded app - or on any platform using the 'Lite' app.<br /><br />By 16:10 AEDT we had resolved the issue affecting the release of print jobs on Kyocera and Ricoh devices, and the Lite apps.<br /><br />We have confirmed that the issues were limited to releasing print jobs, and would only have impacted users on Kyocera and Ricoh devices when using the Full embedded apps - or any platform using the Lite apps. Printing jobs, and releasing jobs on other platforms were not impacted. Releasing jobs through the Mobile Release was also not impacted.<br /><br />We'll be performing a post-mortem and will update this with any findings.</p>tag:status.papercut.com,2005:Incident/143695132022-11-30T05:30:26Z2022-12-05T01:16:37ZCloud Node was offline for accepting new jobs<p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>05:30</var> UTC</small><br><strong>Resolved</strong> - The Cloud Node (optional component for PaperCut Pocket & Hive) was offline during this period due to a system misconfiguration. Normal printing was unaffected during this time, but situations requiring the cloud node functionality would have failed.<br /><br />Issue period was from ~05:30 30/11/2022 UTC until ~11:30 1/12/2022 UTC</p>