Identified - Some iland customers are experiencing degraded performance with their cloud services in Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

How we are resolving it:

iland senior engineers are spreading workloads across additional resources which are being implemented. This will allow iland to remedy the connection problem (previously posted) which was uncovered and reported to Veeam. iland continues to work with our software vendor regarding a patch for the bug and will provide updates as they become available.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 26, 03:49 CST

Update - Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

How we are resolving it:

iland senior engineers are spreading workloads across additional resources which are being implemented within the next two weeks. This will allow iland to remedy the connection problem (previously posted) which was uncovered and reported to Veeam. iland continues to work with our software vendor regarding a patch for the bug and will provide updates as they become available.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 20, 16:22 CST

Update - Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

How we are resolving it:

iland senior engineers are spreading workloads across additional resources which are being implemented within the next two weeks. This will allow iland to remedy the connection problem (previously posted) which was uncovered and reported to Veeam. iland continues to work with our software vendor regarding a patch for the bug and will provide updates as they become available.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 13, 16:44 CST

Update - Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

Details:

Several customers have reported long running backup jobs and/or job failures with the iland Secure Cloud Backup environment. This is a result of a series of hardware and software errors that we have experienced recently. We are working with our vendors to troubleshoot these errors and expect to resolve both issues quickly.

How we are resolving it:

From a hardware perspective, we have replaced the faulty mid-plane globally that was causing drive issues for some customers. This also caused some Veeam repositories to suffer drive disconnections, causing jobs to fail.

For the software, we uncovered and reported an issue to Veeam in the way our provider side Veeam B&R servers communicate with the backend repositories over SSH. There is a bug in the way the VBR maintains stale SSH connections to the repositories at scale, keeping the SSH session alive and not tearing it down. The end result of this is that the repositories no longer accept new connections to start the Veeam Agent process to process tenant tasks. We have implemented a workaround which involves controlling the issue reported above to maintain a good state for incoming task processing, allowing customer tenant jobs to process accordingly. We will provide an update once we get a proper patch from Veeam, but so far the workaround is providing good results.

Additional Details:

We are also aware that our iland status page is reporting 100% uptime, even with degraded performance. We recognize that limiting our status page to only availability status is not sufficient and does not accurately reflect the service environment. We are committed to improving our service reporting and will update our status page to provide additional detail updates as we resolve these issues.



We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 10, 10:44 CST

Update - Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

Details:

Several customers have reported long running backup jobs and/or job failures with the iland Secure Cloud Backup environment. This is a result of a series of hardware and software errors that we have experienced recently. We are working with our vendors to troubleshoot these errors and expect to resolve both issues quickly.

How we are resolving it:

From a hardware perspective, we have replaced the faulty mid-plane globally that was causing drive issues for some customers. This also caused some Veeam repositories to suffer drive disconnections, causing jobs to fail.

For the software, we uncovered and reported an issue to Veeam in the way our provider side Veeam B&R servers communicate with the backend repositories over SSH. There is a bug in the way the VBR maintains stale SSH connections to the repositories at scale, keeping the SSH session alive and not tearing it down. The end result of this is that the repositories no longer accept new connections to start the Veeam Agent process to process tenant tasks. We have implemented a workaround which involves controlling the issue reported above to maintain a good state for incoming task processing, allowing customer tenant jobs to process accordingly. We will provide an update once we get a proper patch from Veeam, but so far the workaround is providing good results.

Additional Details:

We are also aware that our iland status page is reporting 100% uptime, even with degraded performance. We recognize that limiting our status page to only availability status is not sufficient and does not accurately reflect the service environment. We are committed to improving our service reporting and will update our status page to provide additional detail updates as we resolve these issues.



We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 6, 10:42 CST

Update - Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

Details:

Several customers have reported long running backup jobs and/or job failures with the iland Secure Cloud Backup environment. This is a result of a series of hardware and software errors that we have experienced recently. We are working with our vendors to troubleshoot these errors and expect to resolve both issues quickly.

How we are resolving it:

From a hardware perspective, we have replaced the faulty mid-plane globally that was causing drive issues for some customers. This also caused some Veeam repositories to suffer drive disconnections, causing jobs to fail.

For the software, we uncovered and reported an issue to Veeam in the way our provider side Veeam B&R servers communicate with the backend repositories over SSH. There is a bug in the way the VBR maintains stale SSH connections to the repositories at scale, keeping the SSH session alive and not tearing it down. The end result of this is that the repositories no longer accept new connections to start the VeeamAgent process to process tenant tasks. We have a workaround for this issue and are contacting customers directly. The workaround we implemented involves controlling the issue reported above to maintain a good state for incoming task processing, allowing customer tenant jobs to process accordingly. We will provide an update once we get a proper patch from Veeam, but so far the workaround is providing good results.

Additional Details:

We are also aware that our iland status page is reporting 100% uptime, even with degraded performance. We recognize that limiting our status page to only availability status is not sufficient and does not accurately reflect the service environment. We are committed to improving our service reporting and will update our status page to provide additional detail updates as we resolve these issues.



We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 5, 13:14 CST

Update - iland senior engineers are continuing to work with our vendor to troubleshoot reported issues on the iland Secure Cloud Backup environment.

Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 3, 12:32 CST

Update - iland senior engineers are continuing to work with our vendor to troubleshoot reported issues on the iland Secure Cloud Backup environment.

Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Jan 31, 10:14 CST

Update - iland senior engineers have engaged our vendor and are continuing to troubleshoot an issue on the iland Secure Cloud Backup environment.
Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Jan 30, 10:57 CST

Update - iland senior engineers are actively troubleshooting an issue on the iland Secure Cloud Backup environment. Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Jan 29, 09:38 CST

Investigating - iland senior engineers are actively troubleshooting an issue on the iland Secure Cloud Backup environment. Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will provide an update to this incident as more information becomes available.
Jan 28, 12:21 CST

Update - Some iland customers are experiencing degraded performance with their cloud services in Dallas (DAL22). We are working to resolve it quickly as possible.

How we are resolving it:

iland senior engineers are continuing to spread workloads across additional resources which are being implemented. This will allow iland to remedy the connection problem (previously posted) which was uncovered and reported to Veeam. iland continues to work with our software vendor regarding a patch for the bug and will provide updates as they become available.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 26, 03:47 CST

Update - Some iland customers are experiencing degraded performance with their cloud services in our Dallas (DAL22). We are aware of this issue and are working to resolve it quickly.

How we are resolving it:

iland senior engineers are spreading workloads across additional resources which are being implemented within the next two weeks. This will allow iland to remedy the connection problem (previously posted) which was uncovered and reported to Veeam. iland continues to work with our software vendor regarding a patch for the bug and will provide updates as they become available.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 20, 16:21 CST

Update - Some iland customers are experiencing degraded performance with their cloud services in our Dallas (DAL22). We are aware of this issue and are working to resolve it quickly.

How we are resolving it:

iland senior engineers are spreading workloads across additional resources which are being implemented within the next two weeks. This will allow iland to remedy the connection problem (previously posted) which was uncovered and reported to Veeam. iland continues to work with our software vendor regarding a patch for the bug and will provide updates as they become available.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 13, 16:47 CST

Update - Some iland customers are experiencing degraded performance with their cloud services in our Dallas (DAL22). We are aware of this issue and are working to resolve it quickly.

Details:

Several customers have reported long running backup jobs and/or job failures with the iland Secure Cloud Backup environment. This is a result of a series of hardware and software errors that we have experienced recently. We are working with our vendors to troubleshoot these errors and expect to resolve both issues quickly.

How we are resolving it:

From a hardware perspective, we have replaced the faulty mid-plane globally that was causing drive issues for some customers. This also caused some Veeam repositories to suffer drive disconnections, causing jobs to fail.

For the software, we uncovered and reported an issue to Veeam in the way our provider side Veeam B&R servers communicate with the backend repositories over SSH. There is a bug in the way the VBR maintains stale SSH connections to the repositories at scale, keeping the SSH session alive and not tearing it down. The end result of this is that the repositories no longer accept new connections to start the VeeamAgent process to process tenant tasks. We have implemented a workaround which involves controlling the issue reported above to maintain a good state for incoming task processing, allowing customer tenant jobs to process accordingly. We will provide an update once we get a proper patch from Veeam, but so far the workaround is providing good results.

Additional Details:

We are also aware that our iland status page is reporting 100% uptime, even with degraded performance. We recognize that limiting our status page to only availability status is not sufficient and does not accurately reflect the service environment. We are committed to improving our service reporting and will update our status page to provide additional detail updates as we resolve these issues.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 10, 10:45 CST

Update - Some iland customers are experiencing degraded performance with their cloud services in our Dallas (DAL22). We are aware of this issue and are working to resolve it quickly.

Details:

Several customers have reported long running backup jobs and/or job failures with the iland Secure Cloud Backup environment. This is a result of a series of hardware and software errors that we have experienced recently. We are working with our vendors to troubleshoot these errors and expect to resolve both issues quickly.

How we are resolving it:

From a hardware perspective, we have replaced the faulty mid-plane globally that was causing drive issues for some customers. This also caused some Veeam repositories to suffer drive disconnections, causing jobs to fail.

For the software, we uncovered and reported an issue to Veeam in the way our provider side Veeam B&R servers communicate with the backend repositories over SSH. There is a bug in the way the VBR maintains stale SSH connections to the repositories at scale, keeping the SSH session alive and not tearing it down. The end result of this is that the repositories no longer accept new connections to start the VeeamAgent process to process tenant tasks. We have implemented a workaround which involves controlling the issue reported above to maintain a good state for incoming task processing, allowing customer tenant jobs to process accordingly. We will provide an update once we get a proper patch from Veeam, but so far the workaround is providing good results.

Additional Details:

We are also aware that our iland status page is reporting 100% uptime, even with degraded performance. We recognize that limiting our status page to only availability status is not sufficient and does not accurately reflect the service environment. We are committed to improving our service reporting and will update our status page to provide additional detail updates as we resolve these issues.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 6, 10:40 CST

Identified - Some iland customers are experiencing degraded performance with their cloud services in our Dallas (DAL22). We are aware of this issue and are working to resolve it quickly.

Details:

Several customers have reported long running backup jobs and/or job failures with the iland Secure Cloud Backup environment. This is a result of a series of hardware and software errors that we have experienced recently. We are working with our vendors to troubleshoot these errors and expect to resolve both issues quickly.

How we are resolving it:

From a hardware perspective, we have replaced the faulty mid-plane globally that was causing drive issues for some customers. This also caused some Veeam repositories to suffer drive disconnections, causing jobs to fail.

For the software, we uncovered and reported an issue to Veeam in the way our provider side Veeam B&R servers communicate with the backend repositories over SSH. There is a bug in the way the VBR maintains stale SSH connections to the repositories at scale, keeping the SSH session alive and not tearing it down. The end result of this is that the repositories no longer accept new connections to start the VeeamAgent process to process tenant tasks. We have a workaround for this issue and are contacting customers directly. The workaround we implemented involves controlling the issue reported above to maintain a good state for incoming task processing, allowing customer tenant jobs to process accordingly. We will provide an update once we get a proper patch from Veeam, but so far the workaround is providing good results.

Additional Details:

We are also aware that our iland status page is reporting 100% uptime, even with degraded performance. We recognize that limiting our status page to only availability status is not sufficient and does not accurately reflect the service environment. We are committed to improving our service reporting and will update our status page to provide additional detail updates as we resolve these issues.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 5, 13:30 CST

Identified - iland engineers are identified an issue on the Dallas (DAL22) ZVM server in which ther Zerto services are unresponsive. In order to return the services to a healthy state, we are going to proactively reboot the server. Customers may see VPGs in bitmap sync as the services reconnect.

iland engineers are actively working closely with our vendor on this issue.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 25, 15:54 CST

Monitoring - iland engineers were alerted to an issue on the Dallas (DAL22) ZVM server in which ther Zerto services were unresponsive. In order to return the services to a healthy state, we have proactively rebooted the server. Customers may see VPGs in bitmap sync as the services reconnect.

iland engineers will continue to monitor the environment for the next 24 hours.

We apologize for the inconvenience this may cause.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 21, 13:52 CST

Update - The majority of issues are now resolved. Some customers are still reporting incorrect tenant usage information from ‘Tenant Storage Overview’ section.

We are working with individual customers and our vendor to resolve each reported case.

NOTE: these issues do not affect customer backups.

Please contact support with any questions or concerns.
We apologize for the inconvenience this may cause.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 20, 16:23 CST

Update - The majority of issues are now resolved. Some customers are still reporting incorrect tenant usage information from ‘Tenant Storage Overview’ section.

We are working with individual customers and our vendor to resolve each reported case.

NOTE: these issues do not affect customer backups.

Please contact support with any questions or concerns.
We apologize for the inconvenience this may cause.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 18, 09:09 CST

Update - Initial issues are now resolved with the exception of some customers are unable to see backup job status when enabling ‘remote management’. We are working with individual customers and our vendor to resolve each reported case.

NOTE: these issues do not affect customer backups.

Please contact support with any questions or concerns.
We apologize for the inconvenience this may cause.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 10, 10:47 CST

Update - Initial issues are now resolved with the exception of some customers are unable to see backup job status when enabling ‘remote management’. We are working with individual customers and our vendor to resolve each reported case.

NOTE: these issues do not affect customer backups.

Please contact support with any questions or concerns.
We apologize for the inconvenience this may cause.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 5, 10:58 CST

Update - iland have now resolved the issue regarding missing tenants being displayed in the console.

We are continuing to investigate some remaining customer issues relating to incorrect tenant information.

NOTE: these issues do not affect customer backups.

Please contact support with any questions or concerns.
We apologize for the inconvenience this may cause.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Jan 31, 02:56 CST

Update - iland has identified several issues related to the new functionality in the iland Secure Cloud Console. Engineers have identified potential resolution steps and are continuing to work through those areas.

Customers may be experiencing missing or incorrect VCC tenant information displayed in the console.

NOTE: these issues do not affect customer backups.

Please contact support with any questions or concerns.
We apologize for the inconvenience this may cause.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Jan 27, 09:05 CST

Identified - iland has identified an issue related to the new functionality in the iland Secure Cloud Console and we continue to investigate. Customers may be experiencing missing or incorrect VCC tenant information displayed in the console.

NOTE: these issues do not affect customer backups.

Please contact support with any questions or concerns.
We apologize for the inconvenience this may cause.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Jan 22, 11:16 CST

Investigating - iland are aware some customers may be experiencing missing or wrong VCC tenant information in the iland Secure Cloud Console. iland engineers are investigating and will post an update later today.

NOTE: these issues do not affect customer backups.

Please contact support with any questions or concerns.
We apologize for the inconvenience this may cause.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Jan 22, 09:02 CST