Resolved -
Following the work that Hermes carried out on Tuesday, the service has stabilised.
The incident has been resolved and we will continue to monitor performance.
Should you experience any further issues, please contact our support team on
support@scurri.com.
Jun 13, 09:06 IST
Monitoring -
Hermes have resolved the issue impacting label generation today and are currently monitoring to ensure service stability.
They have provided us with the following update:
*********************************
The IT incident affecting the Label API service for clients attempting to retrieve labels from Hermes has now been resolved. Complications occurred following a planned change to increase system resources on the Label API Service. The incident were resolved following a full reinstall of the core web service which is responsible for returning label routing details.
Close monitoring will continue throughout this evening and into the morning to ensure the service stability is maintained. Hermes IT will be conducting full root cause analysis on this issue and the recent disruption to ensure any measures required for securing this service moving forward are put in place.
Please accept our apologies for the disruption this has caused and be assured that we are taking every precaution to prevent similar issues in the future.
*********************************
Jun 11, 22:31 IST
Update -
From approximately 3:30pm today Hermes label generation is having timeout issues again.
Hermes are investigating as a matter of urgency and will update us as soon as they know more and have a timeframe for resolution.
Jun 11, 16:02 IST
Update -
Hermes have provided us with an update regarding resolutions they are putting in place for the intermittent issues that have been experienced recently.
Short term actions, being completed today 11th June:
- Divorcing the address lookup service from the database – reducing the load on the infrastructure
- Introducing additional memory to the boxes @ 2.00pm today
- Auto restart @01:00 daily
- 2 hourly refreshes
Medium Term action:
- The service is currently run out of Germany, but is due to be moved into the Hermes UK control in AWS.
Jun 11, 13:12 IST
Update -
We have been monitoring the Hermes performance and service has improved this afternoon.
We are awaiting confirmation from Hermes about the schedule for the downtime tomorrow to add further server capacity. As soon as we receive confirmation we will update this incident and the scheduled maintenance.
Jun 10, 17:30 IST
Identified -
Earlier today Hermes experienced server issues which resulted in all attempts to generate labels timing out.
They restarted their system, which seemed to have resolved the issue, but only temporarily.
The issue is occurring again and Hermes have advised that they are currently working with their colleagues in Germany to perform a further restart. They are also implementing a 2 hourly refresh to prevent further instances until they can deploy a more permanent fix.
They will be carrying out maintenance tomorrow, at a time to be confirmed, to increase memory on the virtual server which hosts their system.
We will continue to keep you updated as we hear more from them.
Jun 10, 11:14 IST