Update - iland Cloud Operations have completed testing of a software update provided by the backup software vendor to address an identified bug in proper handling of inbound connections and session limits to a cloud backup repository. An emergency maintenance update will be scheduled within the next 24-48 hours to begin deployment of this software update to across all affected iland data center locations.

Additional details of the scheduled maintenances will be provided via the scheduled maintenance notifications.

This incident will remain open in a "monitoring" state until all iland data center locations are updated and results are validated and confirmed post maintenance update.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 25, 17:37 CDT

Monitoring - iland Cloud Operations identified the recent issues with backup job performance due to the ongoing management of inbound connection limits to backup repositories as a result of a bug in the vendor backup software. iland Engineering continues to work with the vendor to obtain the required update to provide a permanent solution.

Re-balancing of job connections across all affected repositories is now completed with backup job performance and completion times returning to normal operating levels. iland will continue to monitor job status for the next 48-72 hours to ensure normal operating stability before resolving this incident.

Please contact Support with any additional questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 20, 18:22 CDT

Investigating - At this time, some iland customers within the Sterling location (STR02) may be experiencing long running backup jobs and/or job failures. iland Cloud Operations is continuing to work with it's software vendors to determine proper remediation steps to resolve the issue.

This issue continues to be a high priority, and we will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 17, 15:32 CDT

Monitoring - Significant improvement with cloud services in the Sterling (STR02) iland Secure Cloud Backup environment has been seen due to the recent implementation of additional resources and workload balancing by iland senior engineers. iland will continue to work with our software vendor regarding a patch to remedy the connection problem which was uncovered and reported to Veeam.

We will continue to monitor performance and follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 13, 15:49 CDT

Identified - Some iland customers are experiencing degraded performance with their cloud services in Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

How we are resolving it:

iland senior engineers are spreading workloads across additional resources which are being implemented. This will allow iland to remedy the connection problem (previously posted) which was uncovered and reported to Veeam. iland continues to work with our software vendor regarding a patch for the bug and will provide updates as they become available.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 26, 03:49 CST

Update - Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

How we are resolving it:

iland senior engineers are spreading workloads across additional resources which are being implemented within the next two weeks. This will allow iland to remedy the connection problem (previously posted) which was uncovered and reported to Veeam. iland continues to work with our software vendor regarding a patch for the bug and will provide updates as they become available.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 20, 16:22 CST

Update - Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

How we are resolving it:

iland senior engineers are spreading workloads across additional resources which are being implemented within the next two weeks. This will allow iland to remedy the connection problem (previously posted) which was uncovered and reported to Veeam. iland continues to work with our software vendor regarding a patch for the bug and will provide updates as they become available.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 13, 16:44 CST

Update - Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

Details:

Several customers have reported long running backup jobs and/or job failures with the iland Secure Cloud Backup environment. This is a result of a series of hardware and software errors that we have experienced recently. We are working with our vendors to troubleshoot these errors and expect to resolve both issues quickly.

How we are resolving it:

From a hardware perspective, we have replaced the faulty mid-plane globally that was causing drive issues for some customers. This also caused some Veeam repositories to suffer drive disconnections, causing jobs to fail.

For the software, we uncovered and reported an issue to Veeam in the way our provider side Veeam B&R servers communicate with the backend repositories over SSH. There is a bug in the way the VBR maintains stale SSH connections to the repositories at scale, keeping the SSH session alive and not tearing it down. The end result of this is that the repositories no longer accept new connections to start the Veeam Agent process to process tenant tasks. We have implemented a workaround which involves controlling the issue reported above to maintain a good state for incoming task processing, allowing customer tenant jobs to process accordingly. We will provide an update once we get a proper patch from Veeam, but so far the workaround is providing good results.

Additional Details:

We are also aware that our iland status page is reporting 100% uptime, even with degraded performance. We recognize that limiting our status page to only availability status is not sufficient and does not accurately reflect the service environment. We are committed to improving our service reporting and will update our status page to provide additional detail updates as we resolve these issues.



We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 10, 10:44 CST

Update - Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

Details:

Several customers have reported long running backup jobs and/or job failures with the iland Secure Cloud Backup environment. This is a result of a series of hardware and software errors that we have experienced recently. We are working with our vendors to troubleshoot these errors and expect to resolve both issues quickly.

How we are resolving it:

From a hardware perspective, we have replaced the faulty mid-plane globally that was causing drive issues for some customers. This also caused some Veeam repositories to suffer drive disconnections, causing jobs to fail.

For the software, we uncovered and reported an issue to Veeam in the way our provider side Veeam B&R servers communicate with the backend repositories over SSH. There is a bug in the way the VBR maintains stale SSH connections to the repositories at scale, keeping the SSH session alive and not tearing it down. The end result of this is that the repositories no longer accept new connections to start the Veeam Agent process to process tenant tasks. We have implemented a workaround which involves controlling the issue reported above to maintain a good state for incoming task processing, allowing customer tenant jobs to process accordingly. We will provide an update once we get a proper patch from Veeam, but so far the workaround is providing good results.

Additional Details:

We are also aware that our iland status page is reporting 100% uptime, even with degraded performance. We recognize that limiting our status page to only availability status is not sufficient and does not accurately reflect the service environment. We are committed to improving our service reporting and will update our status page to provide additional detail updates as we resolve these issues.



We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 6, 10:42 CST

Update - Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

Details:

Several customers have reported long running backup jobs and/or job failures with the iland Secure Cloud Backup environment. This is a result of a series of hardware and software errors that we have experienced recently. We are working with our vendors to troubleshoot these errors and expect to resolve both issues quickly.

How we are resolving it:

From a hardware perspective, we have replaced the faulty mid-plane globally that was causing drive issues for some customers. This also caused some Veeam repositories to suffer drive disconnections, causing jobs to fail.

For the software, we uncovered and reported an issue to Veeam in the way our provider side Veeam B&R servers communicate with the backend repositories over SSH. There is a bug in the way the VBR maintains stale SSH connections to the repositories at scale, keeping the SSH session alive and not tearing it down. The end result of this is that the repositories no longer accept new connections to start the VeeamAgent process to process tenant tasks. We have a workaround for this issue and are contacting customers directly. The workaround we implemented involves controlling the issue reported above to maintain a good state for incoming task processing, allowing customer tenant jobs to process accordingly. We will provide an update once we get a proper patch from Veeam, but so far the workaround is providing good results.

Additional Details:

We are also aware that our iland status page is reporting 100% uptime, even with degraded performance. We recognize that limiting our status page to only availability status is not sufficient and does not accurately reflect the service environment. We are committed to improving our service reporting and will update our status page to provide additional detail updates as we resolve these issues.



We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 5, 13:14 CST

Update - iland senior engineers are continuing to work with our vendor to troubleshoot reported issues on the iland Secure Cloud Backup environment.

Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 3, 12:32 CST

Update - iland senior engineers are continuing to work with our vendor to troubleshoot reported issues on the iland Secure Cloud Backup environment.

Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Jan 31, 10:14 CST

Update - iland senior engineers have engaged our vendor and are continuing to troubleshoot an issue on the iland Secure Cloud Backup environment.
Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Jan 30, 10:57 CST

Update - iland senior engineers are actively troubleshooting an issue on the iland Secure Cloud Backup environment. Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Jan 29, 09:38 CST

Investigating - iland senior engineers are actively troubleshooting an issue on the iland Secure Cloud Backup environment. Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will provide an update to this incident as more information becomes available.
Jan 28, 12:21 CST

Monitoring - iland Cloud Operations has completed its remediation steps to address the degraded replication performance in the Dallas location (DAL25) for inbound replication. We will continue to monitor the status of the environment for the next 48-72 hours before closing this issue as resolved.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 24, 13:13 CDT

Identified - iland Cloud Operations has identified an issue with a core backup server in the Dallas location (DAL25) for inbound replication jobs that may impact the performance of some customer replication jobs. We are currently working to remediate this issue and will provide an update as soon as progress is made to restore services back to normal operating levels.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 23, 10:31 CDT