The avg response time for the last 15 minutes is now 1.22 seconds. According to our logs, the incident effectively stopped immediately after we applied the back end capacity changes, which was at 4:09 PM, Mountain Time. Thanks to everyone for their patience. We have no changes planned for tomorrow.
Posted Jul 19, 2023 - 22:30 UTC
Update
Current response time avg is down to 1.38 seconds, which is in the normal band. We'll continue monitoring a bit longer and ensure the system remains stable.
Posted Jul 19, 2023 - 22:16 UTC
Update
We've increased our concurrent transaction capacity with back end partners. We're seeing response times start to come down. The avg is the lowest it's been since the incident began.
Posted Jul 19, 2023 - 22:09 UTC
Update
The memory issue appears to be fixed. Response times and timeouts have improved, but we're still seeing higher than normal error rates. We believe this is related to fixed concurrent traffic limits imposed by our backend processing partners and are working to increase capacity.
Posted Jul 19, 2023 - 21:34 UTC
Monitoring
A fix has been deployed and we are monitoring the recovery.
Posted Jul 19, 2023 - 20:50 UTC
Investigating
BlockChyp is currently experiencing elevated error rates due to a memory issue in the cloud relay routing system. A fix is being tested and will be deployed shortly.