Sterling (STR02) - iland Secure Cloud Backup degraded performance
Incident Report for iland Secure Cloud
Identified
Some iland customers are experiencing degraded performance with their cloud services in Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

How we are resolving it:

iland senior engineers are spreading workloads across additional resources which are being implemented. This will allow iland to remedy the connection problem (previously posted) which was uncovered and reported to Veeam. iland continues to work with our software vendor regarding a patch for the bug and will provide updates as they become available.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Posted Feb 26, 2020 - 03:49 CST
Update
Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

How we are resolving it:

iland senior engineers are spreading workloads across additional resources which are being implemented within the next two weeks. This will allow iland to remedy the connection problem (previously posted) which was uncovered and reported to Veeam. iland continues to work with our software vendor regarding a patch for the bug and will provide updates as they become available.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Posted Feb 20, 2020 - 16:22 CST
Update
Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

How we are resolving it:

iland senior engineers are spreading workloads across additional resources which are being implemented within the next two weeks. This will allow iland to remedy the connection problem (previously posted) which was uncovered and reported to Veeam. iland continues to work with our software vendor regarding a patch for the bug and will provide updates as they become available.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Posted Feb 13, 2020 - 16:44 CST
Update
Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

Details:

Several customers have reported long running backup jobs and/or job failures with the iland Secure Cloud Backup environment. This is a result of a series of hardware and software errors that we have experienced recently. We are working with our vendors to troubleshoot these errors and expect to resolve both issues quickly.

How we are resolving it:

From a hardware perspective, we have replaced the faulty mid-plane globally that was causing drive issues for some customers. This also caused some Veeam repositories to suffer drive disconnections, causing jobs to fail.

For the software, we uncovered and reported an issue to Veeam in the way our provider side Veeam B&R servers communicate with the backend repositories over SSH. There is a bug in the way the VBR maintains stale SSH connections to the repositories at scale, keeping the SSH session alive and not tearing it down. The end result of this is that the repositories no longer accept new connections to start the Veeam Agent process to process tenant tasks. We have implemented a workaround which involves controlling the issue reported above to maintain a good state for incoming task processing, allowing customer tenant jobs to process accordingly. We will provide an update once we get a proper patch from Veeam, but so far the workaround is providing good results.

Additional Details:

We are also aware that our iland status page is reporting 100% uptime, even with degraded performance. We recognize that limiting our status page to only availability status is not sufficient and does not accurately reflect the service environment. We are committed to improving our service reporting and will update our status page to provide additional detail updates as we resolve these issues.



We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Posted Feb 10, 2020 - 10:44 CST
Update
Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

Details:

Several customers have reported long running backup jobs and/or job failures with the iland Secure Cloud Backup environment. This is a result of a series of hardware and software errors that we have experienced recently. We are working with our vendors to troubleshoot these errors and expect to resolve both issues quickly.

How we are resolving it:

From a hardware perspective, we have replaced the faulty mid-plane globally that was causing drive issues for some customers. This also caused some Veeam repositories to suffer drive disconnections, causing jobs to fail.

For the software, we uncovered and reported an issue to Veeam in the way our provider side Veeam B&R servers communicate with the backend repositories over SSH. There is a bug in the way the VBR maintains stale SSH connections to the repositories at scale, keeping the SSH session alive and not tearing it down. The end result of this is that the repositories no longer accept new connections to start the Veeam Agent process to process tenant tasks. We have implemented a workaround which involves controlling the issue reported above to maintain a good state for incoming task processing, allowing customer tenant jobs to process accordingly. We will provide an update once we get a proper patch from Veeam, but so far the workaround is providing good results.

Additional Details:

We are also aware that our iland status page is reporting 100% uptime, even with degraded performance. We recognize that limiting our status page to only availability status is not sufficient and does not accurately reflect the service environment. We are committed to improving our service reporting and will update our status page to provide additional detail updates as we resolve these issues.



We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Posted Feb 06, 2020 - 10:42 CST
Update
Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

Details:

Several customers have reported long running backup jobs and/or job failures with the iland Secure Cloud Backup environment. This is a result of a series of hardware and software errors that we have experienced recently. We are working with our vendors to troubleshoot these errors and expect to resolve both issues quickly.

How we are resolving it:

From a hardware perspective, we have replaced the faulty mid-plane globally that was causing drive issues for some customers. This also caused some Veeam repositories to suffer drive disconnections, causing jobs to fail.

For the software, we uncovered and reported an issue to Veeam in the way our provider side Veeam B&R servers communicate with the backend repositories over SSH. There is a bug in the way the VBR maintains stale SSH connections to the repositories at scale, keeping the SSH session alive and not tearing it down. The end result of this is that the repositories no longer accept new connections to start the VeeamAgent process to process tenant tasks. We have a workaround for this issue and are contacting customers directly. The workaround we implemented involves controlling the issue reported above to maintain a good state for incoming task processing, allowing customer tenant jobs to process accordingly. We will provide an update once we get a proper patch from Veeam, but so far the workaround is providing good results.

Additional Details:

We are also aware that our iland status page is reporting 100% uptime, even with degraded performance. We recognize that limiting our status page to only availability status is not sufficient and does not accurately reflect the service environment. We are committed to improving our service reporting and will update our status page to provide additional detail updates as we resolve these issues.



We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Posted Feb 05, 2020 - 13:14 CST
Update
iland senior engineers are continuing to work with our vendor to troubleshoot reported issues on the iland Secure Cloud Backup environment.

Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Posted Feb 03, 2020 - 12:32 CST
Update
iland senior engineers are continuing to work with our vendor to troubleshoot reported issues on the iland Secure Cloud Backup environment.

Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Posted Jan 31, 2020 - 10:14 CST
Update
iland senior engineers have engaged our vendor and are continuing to troubleshoot an issue on the iland Secure Cloud Backup environment.
Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Posted Jan 30, 2020 - 10:57 CST
Update
iland senior engineers are actively troubleshooting an issue on the iland Secure Cloud Backup environment. Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Posted Jan 29, 2020 - 09:38 CST
Investigating
iland senior engineers are actively troubleshooting an issue on the iland Secure Cloud Backup environment. Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will provide an update to this incident as more information becomes available.
Posted Jan 28, 2020 - 12:21 CST
This incident affects: Sterling STR02 (North America) (iland Secure Cloud Backup).