Update - Concurrent tasks have been reduced to one per tenant to aid recovery. Some customers may see errors from other jobs while recovery continues however, several jobs are currently processing.

Please contact support with any question or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Dec 6, 11:04 CST

Update - Recent tests conducted by iland and senior Veeam engineers revealed that a retry interval under certain conditions could take as long as 24 hours. This interval may be recreated by right clicking the delayed job and choosing “Sync Now”.

Veeam recommends starting a new interval using the Sync Now option to get data flowing again. (for customers seeing delays in the job to start synchronizing with the iland cloud connect repository)

Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Dec 4, 21:32 CST

Update - iland engineers have been working with Veeam Tier 3 support on restoring the performance of the DAL22 iland Secure Cloud Backup environment. The environment is fully online and accepting connections for jobs to run. The system is currently working through a backlog of jobs and as a result, jobs will take significantly longer to begin data transfer. Based on test jobs, data transfers should operate at normal speed once resources have been assigned.

We will continue to work with Veeam Tier 3 support to improve performance.
Dec 4, 15:48 CST

Monitoring - iland engineers have been working with Veeam Tier 3 support on restoring the performance of the DAL22 iland Secure Cloud Backup environment. The environment is fully online and accepting connections for jobs to run. The system is currently working through a backlog of jobs and as a result, jobs will take significantly longer to begin data transfer. Based on test jobs, data transfers should operate at normal speed once resources have been assigned.

We will continue to work with Veeam Tier 3 support to improve performance.
Dec 3, 16:24 CST

Update - iland engineers and Veeam support continue to work on the issue affecting DAL22 iland Secure Cloud Backup performance. We have made some progress with the issue and some customers may still be experiencing long running running or failing jobs.

We will provide updates as we progress with a full resolution.

We apologize for the inconvenience this is causing and are actively working on resolving the issue as quickly as possible.
Dec 3, 11:18 CST

Update - iland engineers continue to work with the vendor Tier 3 team on the issue with the DAL22 iland Secure Cloud Backup environment. The issue appears to be Veeam service provider software related and not related to any specific hardware faults.

We will provide and update to this incident as information becomes available.
Dec 3, 09:23 CST

Update - iland engineers continue to work with the vendor on the issue with the DAL22 iland Secure Cloud Backup environment. The issue appears to be Veeam service provider software related and not related to any specific hardware faults.

We will provide and update to this incident as information becomes available.
Dec 2, 18:12 CST

Update - iland engineers have raised a Sev 1 ticket with our provider Veeam following the restart of the iland secure cloud backup environment and continue investigating reported performance degradation. Some customers may experience job failures or long running jobs at this time.

We apologize for the inconvenience this is causing and are actively working on resolving the issue as quickly as possible.
Dec 2, 14:06 CST

Update - iland engineers have restarted the iland secure cloud backup environment and continue investigating reported performance degradation. Some customers may experience job failures or long running jobs at this time.

We apologize for the inconvenience this is causing and are actively working on resolving the issue as quickly as possible.
Dec 2, 13:00 CST

Investigating - iland engineers are actively investigating reported performance degradation in the DAL22 iland Secure Cloud Backup environment. Some customers may experience job failures or long running jobs at this time.

We will provide an update shortly.
Dec 2, 11:58 CST

Update - Performance continues to improve with job and merge times however, some customers may still be experiencing issues. iland engineers will continue to monitor and provide updates until the the environment has recovered.

Please contact support with any question or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Nov 27, 11:01 CST

Update - Performance continues to improve with job and merge times however, some customers may still be experiencing issues. iland engineers will continue to monitor and provide updates until the the environment has recovered.

Please contact support with any question or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Nov 26, 09:55 CST

Update - Performance continues to improve with job and merge times however, some customers may still be experiencing issues. iland engineers will continue to monitor and provide updates until the the environment has recovered.

Please contact support with any question or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Nov 25, 19:01 CST

Update - iland engineers have replaced the faulty mid-plane on the remaining repository. The repository is online and recovering. Jobs and merge times should improve for some affected customers. We will continue to monitor performance.

We apologize for the inconvenience this is causing and are actively working on resolving the issue as quickly as possible.
Nov 23, 18:39 CST

Update - iland engineers will be replacing a faulty mid-plane on the remaining repository this afternoon 11/22/2019 at 13:00 CT. The repository will be offline for a short amount of time while the part is replaced and some customers may see job failures at this time. Once the part is replaced and the repository is online, jobs and merge times should improve.

We apologize for the inconvenience this is causing and are actively working on resolving the issue as quickly as possible.
Nov 22, 11:57 CST

Identified - iland engineers have identified the issue within the Dallas (DAL22) environment. Some of the many servers we use that hold the Veeam Cloud Connect backups are subjected to a manufacturer problem with their mid-plane. What is happening is that a few of these repository servers are having slow disk performance because of this issue and may cause failed jobs or merge issues. This is isolated to a few servers at this time. We are working diligently with the manufacturer who has released a new version of the mid-plane and are actively replacing them in these servers.

We apologize for the inconvenience this is causing and are actively working on resolving the issue as quickly as possible.
Nov 21, 13:00 CST

Investigating - iland senior engineers are actively troubleshooting an issue on the iland Secure Cloud Backup environment. Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will provide an update to this incident as more information becomes available.
Nov 19, 11:47 CST