Update - iland Cloud Operations have completed testing of a software update provided by the backup software vendor to address an identified bug in proper handling of inbound connections and session limits to a cloud backup repository. An emergency maintenance update will be scheduled within the next 24-48 hours to begin deployment of this software update to across all affected iland data center locations.

Additional details of the scheduled maintenances will be provided via the scheduled maintenance notifications.

This incident will remain open in a "monitoring" state until all iland data center locations are updated and results are validated and confirmed post maintenance update.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 25, 17:37 CDT
Monitoring - iland Cloud Operations identified the recent issues with backup job performance due to the ongoing management of inbound connection limits to backup repositories as a result of a bug in the vendor backup software. iland Engineering continues to work with the vendor to obtain the required update to provide a permanent solution.

Re-balancing of job connections across all affected repositories is now completed with backup job performance and completion times returning to normal operating levels. iland will continue to monitor job status for the next 48-72 hours to ensure normal operating stability before resolving this incident.

Please contact Support with any additional questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 20, 18:22 CDT
Investigating - At this time, some iland customers within the Sterling location (STR02) may be experiencing long running backup jobs and/or job failures. iland Cloud Operations is continuing to work with it's software vendors to determine proper remediation steps to resolve the issue.

This issue continues to be a high priority, and we will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 17, 15:32 CDT
Monitoring - Significant improvement with cloud services in the Sterling (STR02) iland Secure Cloud Backup environment has been seen due to the recent implementation of additional resources and workload balancing by iland senior engineers. iland will continue to work with our software vendor regarding a patch to remedy the connection problem which was uncovered and reported to Veeam.

We will continue to monitor performance and follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 13, 15:49 CDT
Identified - Some iland customers are experiencing degraded performance with their cloud services in Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

How we are resolving it:

iland senior engineers are spreading workloads across additional resources which are being implemented. This will allow iland to remedy the connection problem (previously posted) which was uncovered and reported to Veeam. iland continues to work with our software vendor regarding a patch for the bug and will provide updates as they become available.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 26, 03:49 CST
Update - Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

How we are resolving it:

iland senior engineers are spreading workloads across additional resources which are being implemented within the next two weeks. This will allow iland to remedy the connection problem (previously posted) which was uncovered and reported to Veeam. iland continues to work with our software vendor regarding a patch for the bug and will provide updates as they become available.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 20, 16:22 CST
Update - Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

How we are resolving it:

iland senior engineers are spreading workloads across additional resources which are being implemented within the next two weeks. This will allow iland to remedy the connection problem (previously posted) which was uncovered and reported to Veeam. iland continues to work with our software vendor regarding a patch for the bug and will provide updates as they become available.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 13, 16:44 CST
Update - Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

Details:

Several customers have reported long running backup jobs and/or job failures with the iland Secure Cloud Backup environment. This is a result of a series of hardware and software errors that we have experienced recently. We are working with our vendors to troubleshoot these errors and expect to resolve both issues quickly.

How we are resolving it:

From a hardware perspective, we have replaced the faulty mid-plane globally that was causing drive issues for some customers. This also caused some Veeam repositories to suffer drive disconnections, causing jobs to fail.

For the software, we uncovered and reported an issue to Veeam in the way our provider side Veeam B&R servers communicate with the backend repositories over SSH. There is a bug in the way the VBR maintains stale SSH connections to the repositories at scale, keeping the SSH session alive and not tearing it down. The end result of this is that the repositories no longer accept new connections to start the Veeam Agent process to process tenant tasks. We have implemented a workaround which involves controlling the issue reported above to maintain a good state for incoming task processing, allowing customer tenant jobs to process accordingly. We will provide an update once we get a proper patch from Veeam, but so far the workaround is providing good results.

Additional Details:

We are also aware that our iland status page is reporting 100% uptime, even with degraded performance. We recognize that limiting our status page to only availability status is not sufficient and does not accurately reflect the service environment. We are committed to improving our service reporting and will update our status page to provide additional detail updates as we resolve these issues.



We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 10, 10:44 CST
Update - Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

Details:

Several customers have reported long running backup jobs and/or job failures with the iland Secure Cloud Backup environment. This is a result of a series of hardware and software errors that we have experienced recently. We are working with our vendors to troubleshoot these errors and expect to resolve both issues quickly.

How we are resolving it:

From a hardware perspective, we have replaced the faulty mid-plane globally that was causing drive issues for some customers. This also caused some Veeam repositories to suffer drive disconnections, causing jobs to fail.

For the software, we uncovered and reported an issue to Veeam in the way our provider side Veeam B&R servers communicate with the backend repositories over SSH. There is a bug in the way the VBR maintains stale SSH connections to the repositories at scale, keeping the SSH session alive and not tearing it down. The end result of this is that the repositories no longer accept new connections to start the Veeam Agent process to process tenant tasks. We have implemented a workaround which involves controlling the issue reported above to maintain a good state for incoming task processing, allowing customer tenant jobs to process accordingly. We will provide an update once we get a proper patch from Veeam, but so far the workaround is providing good results.

Additional Details:

We are also aware that our iland status page is reporting 100% uptime, even with degraded performance. We recognize that limiting our status page to only availability status is not sufficient and does not accurately reflect the service environment. We are committed to improving our service reporting and will update our status page to provide additional detail updates as we resolve these issues.



We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 6, 10:42 CST
Update - Some iland customers are experiencing degraded performance with their cloud services in our Sterling (STR02). We are aware of this issue and are working to resolve it quickly.

Details:

Several customers have reported long running backup jobs and/or job failures with the iland Secure Cloud Backup environment. This is a result of a series of hardware and software errors that we have experienced recently. We are working with our vendors to troubleshoot these errors and expect to resolve both issues quickly.

How we are resolving it:

From a hardware perspective, we have replaced the faulty mid-plane globally that was causing drive issues for some customers. This also caused some Veeam repositories to suffer drive disconnections, causing jobs to fail.

For the software, we uncovered and reported an issue to Veeam in the way our provider side Veeam B&R servers communicate with the backend repositories over SSH. There is a bug in the way the VBR maintains stale SSH connections to the repositories at scale, keeping the SSH session alive and not tearing it down. The end result of this is that the repositories no longer accept new connections to start the VeeamAgent process to process tenant tasks. We have a workaround for this issue and are contacting customers directly. The workaround we implemented involves controlling the issue reported above to maintain a good state for incoming task processing, allowing customer tenant jobs to process accordingly. We will provide an update once we get a proper patch from Veeam, but so far the workaround is providing good results.

Additional Details:

We are also aware that our iland status page is reporting 100% uptime, even with degraded performance. We recognize that limiting our status page to only availability status is not sufficient and does not accurately reflect the service environment. We are committed to improving our service reporting and will update our status page to provide additional detail updates as we resolve these issues.



We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 5, 13:14 CST
Update - iland senior engineers are continuing to work with our vendor to troubleshoot reported issues on the iland Secure Cloud Backup environment.

Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 3, 12:32 CST
Update - iland senior engineers are continuing to work with our vendor to troubleshoot reported issues on the iland Secure Cloud Backup environment.

Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Jan 31, 10:14 CST
Update - iland senior engineers have engaged our vendor and are continuing to troubleshoot an issue on the iland Secure Cloud Backup environment.
Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Jan 30, 10:57 CST
Update - iland senior engineers are actively troubleshooting an issue on the iland Secure Cloud Backup environment. Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Jan 29, 09:38 CST
Investigating - iland senior engineers are actively troubleshooting an issue on the iland Secure Cloud Backup environment. Customers in this location may see long running backup jobs and/or job failures. We have made this a high priority and are working to resolve the issue as soon as possible.

We will provide an update to this incident as more information becomes available.
Jan 28, 12:21 CST
Monitoring - iland Cloud Operations has completed its remediation steps to address the degraded replication performance in the Dallas location (DAL25) for inbound replication. We will continue to monitor the status of the environment for the next 48-72 hours before closing this issue as resolved.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 24, 13:13 CDT
Identified - iland Cloud Operations has identified an issue with a core backup server in the Dallas location (DAL25) for inbound replication jobs that may impact the performance of some customer replication jobs. We are currently working to remediate this issue and will provide an update as soon as progress is made to restore services back to normal operating levels.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 23, 10:31 CDT
Amsterdam AMS00 (Europe) Operational
iland Colocation Operational
Amsterdam AMS01 (Europe) Operational
iland Cloud Operational
iland Secure Cloud Operational
iland Secure Cloud Backup Operational
iland Secure Cloud Backup for Microsoft Office 365 with Veeam Operational
iland Secure Cloud Console Operational
iland Secure DRaaS with Double-Take Operational
iland Secure DRaaS with Zerto Operational
Amsterdam AMS03 (Europe) Operational
iland Secure DRaaS with Veeam Operational
Dallas DAL02 (North America) Operational
iland Cloud Operational
iland Secure Cloud Operational
iland Secure Cloud Backup Operational
iland Secure Cloud Backup for Microsoft Office 365 with Veeam Operational
iland Secure Cloud Console Operational
iland Secure DRaaS with Double-Take Operational
iland Secure DRaaS with Veeam Operational
iland Secure DRaaS with Zerto Operational
iland Secure Private Cloud Operational
Dallas DAL06 (North America) Operational
iland Cloud Operational
iland Lab Engine Operational
iland Secure Cloud Operational
iland Secure Cloud Backup Operational
iland Secure Cloud Console Operational
iland Secure DRaaS with Double-Take Operational
iland Secure DRaaS with Zerto Operational
iland Secure Private Cloud Operational
Dallas DAL22 (North America) Operational
iland Secure Cloud Operational
iland Secure Cloud Backup Operational
iland Secure Cloud Backup for Microsoft Office 365 with Veeam Operational
iland Secure DRaaS with Zerto Operational
Dallas DAL25 (North America) Operational
iland Secure DRaaS with Veeam Operational
Houston HOU00 (North America) Operational
iland Colocation Operational
London LON02 (United Kingdom) Operational
iland Cloud Operational
iland Lab Engine Operational
iland Secure Cloud Operational
iland Secure Cloud Backup Operational
iland Secure Cloud Backup for Microsoft Office 365 with Veeam Operational
iland Secure Cloud Console Operational
iland Secure Cloud Object Storage Operational
iland Secure DRaaS with Double-Take Operational
iland Secure DRaaS with Veeam Operational
iland Secure DRaaS with Zerto Operational
iland Secure Private Cloud Operational
London LON03 (United Kingdom) Operational
iland Secure DRaaS with Veeam Operational
Los Angeles LAX01 (North America) Operational
iland Cloud Operational
iland Lab Engine Operational
iland Secure Cloud Operational
iland Secure Cloud Backup Operational
iland Secure Cloud Backup for Microsoft Office 365 with Veeam Operational
iland Secure Cloud Console Operational
iland Secure DRaaS with Double-Take Operational
iland Secure DRaaS with Veeam Operational
iland Secure DRaaS with Zerto Operational
iland Secure Private Cloud Operational
Manchester MAN01 (United Kingdom) Operational
iland Cloud Operational
iland Secure Cloud Operational
iland Secure Cloud Backup Operational
iland Secure Cloud Backup for Microsoft Office 365 with Veeam Operational
iland Secure Cloud Console Operational
iland Secure DRaaS with Double-Take Operational
iland Secure DRaaS with Zerto Operational
iland Secure Private Cloud Operational
McLean MCL00 (North America) Operational
iland Colocation Operational
Melbourne MEL02 (Asia Pacific) Operational
iland Cloud Operational
iland Lab Engine Operational
iland Secure Cloud Operational
iland Secure Cloud Backup Operational
iland Secure Cloud Backup for Microsoft Office 365 with Veeam Operational
iland Secure Cloud Console Operational
iland Secure DRaaS with Double-Take Operational
iland Secure DRaaS with Veeam Operational
iland Secure DRaaS with Zerto Operational
iland Secure Private Cloud Operational
Miami MIA00 (North America) Operational
iland Colocation Operational
Reston RES01 (North America) Operational
iland Cloud Operational
iland Secure Cloud Operational
iland Secure Cloud Backup Operational
iland Secure Cloud Backup for Microsoft Office 365 with Veeam Operational
iland Secure Cloud Console Operational
iland Secure Cloud Object Storage Operational
iland Secure DRaaS with Double-Take Operational
iland Secure DRaaS with Veeam Operational
iland Secure DRaaS with Zerto Operational
iland Secure Private Cloud Operational
Seattle SEA00 (North America) Operational
iland Colocation Operational
Singapore SIN01 (Asia Pacific) Operational
iland Cloud Operational
iland Secure Cloud Operational
iland Secure Cloud Backup Operational
iland Secure Cloud Backup for Microsoft Office 365 with Veeam Operational
iland Secure Cloud Console Operational
iland Secure DRaaS with Double-Take Operational
iland Secure DRaaS with Veeam Operational
iland Secure DRaaS with Zerto Operational
iland Secure Private Cloud Operational
Sterling STR02 (North America) Operational
iland Secure Cloud Operational
iland Secure Cloud Backup Operational
iland Secure Cloud Console Operational
iland Secure Cloud Backup for Microsoft Office 365 with Veeam Operational
iland Secure DRaaS with Zerto Operational
Sterling STR03 (North America) Operational
iland Lab Engine Operational
Sterling STR05 (North America) Operational
iland Secure Cloud Console Operational
iland Secure DRaaS with Veeam Operational
Sydney SYD02 (Asia Pacific) Operational
iland Cloud Operational
iland Lab Engine Operational
iland Secure Cloud Operational
iland Secure Cloud Backup Operational
iland Secure Cloud Backup for Microsoft Office 365 with Veeam Operational
iland Secure Cloud Console Operational
iland Secure Cloud Object Storage Operational
iland Secure DRaaS with Veeam Operational
iland Secure DRaaS with Zerto Operational
iland Secure DRaaS with Double-Take Operational
iland Secure Private Cloud Operational
Toronto TOR02 (Canada) Operational
iland Secure Cloud Operational
iland Secure Cloud Console Operational
iland Secure DRaaS with Zerto Operational
Toronto TOR03(Canada) Operational
iland Lab Engine Operational
iland Secure Cloud Console Operational
iland Secure DRaaS with Veeam Operational
iland Secure Cloud Backup Operational
iland Secure Cloud Backup for Microsoft Office 365 with Veeam Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
Los Angeles - Scheduled Maintenance Mar 30, 22:00 - Mar 31, 01:00 CDT
Duration: 03/30/2020 from 20:00 PT to 23:00 PT
Location: Los Angeles
Type Of Event: Scheduled Maintenance

As part of on-going maintenance, iland engineers will be applying an important upgrade to our iland Secure DRaaS with Zerto infrastructure. This upgrade will improve performance of the iland Secure DRaaS with Zerto Infrastructure. Customers should stop all failover tests prior to this maintenance window and may see an intermittent site disconnect status. If you need any assistance with your infrastructure during this maintenance window, please do not hesitate to contact us.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Posted on Mar 24, 17:33 CDT
Dallas (DAL02) - Scheduled Maintenance Apr 3, 20:00 - Apr 4, 00:00 CDT
Duration: 04/03/2020 from 20:00 CT to 04/04/2020 00:00 CT
Location: Dallas (DAL02)
Type Of Event: Scheduled Maintenance

As part of on-going maintenance, iland engineers will be applying an important upgrade to our iland Secure DRaaS with Zerto infrastructure. This upgrade will improve performance of the iland Secure DRaaS with Zerto Infrastructure. Customers should stop all failover tests prior to this maintenance window and may see an intermittent site disconnect status. If you need any assistance with your infrastructure during this maintenance window, please do not hesitate to contact us.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Posted on Mar 24, 17:36 CDT
Past Incidents
Mar 29, 2020

No incidents reported today.

Mar 28, 2020
Completed - The scheduled maintenance has been completed.
Mar 28, 18:00 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 28, 14:00 CDT
Scheduled - Duration: 03/28/2020 from 19:00 GMT to 23:00 GMT
Location: Manchester
Type Of Event: Scheduled Maintenance

As part of on-going maintenance, iland engineers will be applying an important upgrade to our iland Secure DRaaS with Zerto infrastructure. This upgrade will improve performance of the iland Secure DRaaS with Zerto Infrastructure. Customers should stop all failover tests prior to this maintenance window and may see an intermittent site disconnect status. If you need any assistance with your infrastructure during this maintenance window, please do not hesitate to contact us.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 24, 17:32 CDT
Postmortem - Read details
Mar 28, 14:24 CDT
Completed - The scheduled maintenance has been completed.
Mar 28, 13:47 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 28, 13:17 CDT
Scheduled - iland Cloud Operations is extending the previous schedule maintenance for Sterling (STR02) for an additional 30-min to allow additional time for services to come back online. No issues are reported at this time.

Thank you,

iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 28, 13:16 CDT
Postmortem - Read details
Mar 28, 14:24 CDT
Completed - The scheduled maintenance has been completed.
Mar 28, 13:00 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 28, 11:00 CDT
Update - We will be undergoing scheduled maintenance during this time.
Mar 27, 19:32 CDT
Scheduled - SCHEDULED TIME: 2020-03-28 | 12:00pm ET

SUMMARY:
iland Cloud Operations will be performing an emergency maintenance software update in the Sterling (str02) infrastructure stack to apply a vendor hotfix that addresses a software bug causing inbound backup connections and session limit issues with customer backup repositories.

This maintenance is specifically being carried out to address previous incident notifications of degraded backup performance at both these locations.

IMPACT:
Due to required restart of the provider cloud connect services, some customers may experience a pause or delay in any backup job(s) currently processing during the maintenance window. Upon completion, all services and job processing should return to normal operational status. Should you encounter issues or prolonged impact to your backup jobs after the maintenance is completed, please contact our Support team for assistance.

Thank you,

iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 27, 19:23 CDT
Completed - The scheduled maintenance has been completed.
Mar 28, 13:00 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 28, 11:00 CDT
Update - We will be undergoing scheduled maintenance during this time.
Mar 27, 19:37 CDT
Update - We will be undergoing scheduled maintenance during this time.
Mar 27, 19:20 CDT
Scheduled - SCHEDULED TIME: 2020-03-28 | 11:00am CT

SUMMARY:
iland Cloud Operations will be performing an emergency maintenance software update in the Dallas (dal22) infrastructure stacks to apply a vendor hotfix that addresses a software bug causing inbound backup connections and session limit issues with customer backup repositories.

This maintenance is specifically being carried out to address previous incident notifications of degraded backup performance at both these locations.

IMPACT:
Due to required restart of the provider cloud connect services, some customers may experience a pause or delay in any backup job(s) currently processing during the maintenance window. Upon completion, all services and job processing should return to normal operational status. Should you encounter issues or prolonged impact to your backup jobs after the maintenance is completed, please contact our Support team for assistance.

Thank you,

iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 27, 19:11 CDT
Mar 27, 2020
Completed - The scheduled maintenance has been completed.
Mar 27, 16:00 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 27, 14:00 CDT
Scheduled - Duration: 03/27/2020 from 20:00 CET to 22:00 CET
Location: Amsterdam
Type Of Event: Scheduled Maintenance

As part of on-going maintenance, iland engineers will be applying an important upgrade to our iland Secure DRaaS with Zerto infrastructure. This upgrade will improve performance of the iland Secure DRaaS with Zerto Infrastructure. Customers should stop all failover tests prior to this maintenance window and may see an intermittent site disconnect status. If you need any assistance with your infrastructure during this maintenance window, please do not hesitate to contact us.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 24, 17:30 CDT
Completed - The scheduled maintenance has been completed.
Mar 27, 12:00 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 27, 10:00 CDT
Scheduled - Duration: 03/27/2020 from 02:00 AEDT to 04:00 AEDT
Location: Melbourne
Type Of Event: Scheduled Maintenance

As part of on-going maintenance, iland engineers will be applying an important upgrade to our iland Secure DRaaS with Zerto infrastructure. This upgrade will improve performance of the iland Secure DRaaS with Zerto Infrastructure. Customers should stop all failover tests prior to this maintenance window and may see an intermittent site disconnect status. If you need any assistance with your infrastructure during this maintenance window, please do not hesitate to contact us.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 24, 17:28 CDT
Mar 26, 2020
Completed - The scheduled maintenance has been completed.
Mar 26, 22:00 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 26, 19:01 CDT
Scheduled - Duration: 03/26/2020 from 20:00 ET to 23:00 ET
Location: Sterling
Type Of Event: Scheduled Maintenance

iland engineers will be updating our iland Secure DRaaS with Zerto infrastructure. This update will improve performance of the iland Secure DRaaS with Zerto Infrastructure. Customers should stop all failover tests prior to this maintenance window and may see an intermittent site disconnect status. If you need any assistance with your infrastructure during this maintenance window, please do not hesitate to contact us.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 26, 08:19 CDT
Completed - The scheduled maintenance has been completed.
Mar 26, 12:00 CDT
Update - iland engineers will have to restart the management stack as part of this maintenance. For approximately the next 30 minutes, operations to virtual machines via the iland Secure Cloud Console will be inaccessible. There will be no impact to running or protected virtual machines during this maintenance.

We apologize for this inconvenience and will have the management stack operational shortly.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 26, 10:56 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 26, 10:00 CDT
Scheduled - Duration: 03/27/2020 from 02:00 AEDT to 04:00 AEDT
Location: Sydney
Type Of Event: Scheduled Maintenance

As part of on-going maintenance, iland engineers will be applying an important upgrade to our iland Secure DRaaS with Zerto infrastructure. This upgrade will improve performance of the iland Secure DRaaS with Zerto Infrastructure. Customers should stop all failover tests prior to this maintenance window and may see an intermittent site disconnect status. If you need any assistance with your infrastructure during this maintenance window, please do not hesitate to contact us.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 24, 17:26 CDT
Resolved - - Incident Summary:
At approximately 12:30pm CT iland received indications of time synchronization issues at the Dallas data center location, causing time-sync drift for some customer VM workloads.

- Root Cause:
iland Site Reliability Engineering discovered a configuration issue with internal time synchronization services within the DAL02 infrastructure stack. This issue created polling delays for a number of core infrastructure services attempting to synchronize with iland's primary external NTP sources. As a result, some customers reported time sync drift with certain VM workloads.

- Remediation:
iland's Site Reliability Engineering team was able to quickly identify the required configuration update and remediation steps, restoring proper NTP and time synchronization services across all supporting infrastructure components at approximately 3:30pm CT.

If you require additional information or have any questions, please contact our Support team for assistance.

Thank you!

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 26, 11:30 CDT
Completed - The scheduled maintenance has been completed.
Mar 26, 11:00 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 26, 09:00 CDT
Scheduled - Duration: 03/26/2020 from 21:00 SGT to 23:00 SGT
Location: Singapore
Type Of Event: Scheduled Maintenance

As part of on-going maintenance, iland engineers will be applying an important upgrade to our iland Secure DRaaS with Zerto infrastructure. This upgrade will improve performance of the iland Secure DRaaS with Zerto Infrastructure. Customers should stop all failover tests prior to this maintenance window and may see an intermittent site disconnect status. If you need any assistance with your infrastructure during this maintenance window, please do not hesitate to contact us.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 24, 17:24 CDT
Completed - This maintenance is currently not in progress. The automation was set to the incorrect start time. We will cancel this notice and resubmit a new one for tonight 03/26/2020 beginning at 20:00 ET. We apologize for any inconvenience.

Thank you.
Mar 26, 08:17 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 26, 07:00 CDT
Scheduled - Duration: 03/26/2029 from 20:00 ET to 23:00 ET
Location: Sterling
Type Of Event: Scheduled Maintenance

iland engineers will be updating our iland Secure DRaaS with Zerto infrastructure. This update will improve performance of the iland Secure DRaaS with Zerto Infrastructure. Customers should stop all failover tests prior to this maintenance window and may see an intermittent site disconnect status. If you need any assistance with your infrastructure during this maintenance window, please do not hesitate to contact us.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 19, 15:42 CDT
Mar 25, 2020
Completed - The scheduled maintenance has been completed.
Mar 25, 13:30 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 25, 12:30 CDT
Scheduled - Duration: 03/26/2020 from 04:30 AEDT to 05:30 AEDT
Location: Melbourne
Type Of Event: Scheduled Infrastructure Maintenance

As part of ongoing maintenance, iland engineers will be updating the iland Cloud and iland Secure Cloud environment. This latest vendor update to our iland Cloud and iland Secure Cloud product incorporates significant bug fixes and enhancements that allow iland to offer a more stable environment.

During this maintenance window, operations to virtual machines via the iland Secure Cloud Console will be inaccessible. Customers using iland's Secure DRaaS products may see intermittent site disconnects and will be unable to perform operations on protected virtual machines. We also require failover tests to be stopped during the maintenance. There will be no impact to running or protected virtual machines during this maintenance.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 20, 15:22 CDT
Completed - The scheduled maintenance has been completed.
Mar 25, 12:00 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 25, 11:00 CDT
Scheduled - Duration: 03/26/2020 from 03:00 AEDT to 04:00 AEDT
Location: Sydney
Type Of Event: Scheduled Infrastructure Maintenance

As part of ongoing maintenance, iland engineers will be updating the iland Cloud and iland Secure Cloud environment. This latest vendor update to our iland Cloud and iland Secure Cloud product incorporates significant bug fixes and enhancements that allow iland to offer a more stable environment.

During this maintenance window, operations to virtual machines via the iland Secure Cloud Console will be inaccessible. Customers using iland's Secure DRaaS products may see intermittent site disconnects and will be unable to perform operations on protected virtual machines. We also require failover tests to be stopped during the maintenance. There will be no impact to running or protected virtual machines during this maintenance.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 20, 15:19 CDT
Completed - The scheduled maintenance has been completed.
Mar 25, 10:00 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 25, 09:00 CDT
Scheduled - Duration: 03/25/2020 from 22:00 SGT to 23:00 SGT
Location: Singapore
Type Of Event: Scheduled Infrastructure Maintenance

As part of ongoing maintenance, iland engineers will be updating the iland Cloud and iland Secure Cloud environment. This latest vendor update to our iland Cloud and iland Secure Cloud product incorporates significant bug fixes and enhancements that allow iland to offer a more stable environment.

During this maintenance window, operations to virtual machines via the iland Secure Cloud Console will be inaccessible. Customers using iland's Secure DRaaS products may see intermittent site disconnects and will be unable to perform operations on protected virtual machines. We also require failover tests to be stopped during the maintenance. There will be no impact to running or protected virtual machines during this maintenance.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 20, 14:51 CDT
Completed - The scheduled maintenance has been completed.
Mar 25, 04:00 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 24, 23:00 CDT
Scheduled - As part of on-going maintenance, one of iland’s redundant providers will be performing maintenance on their upstream networking infrastructure at our Toronto datacenter. No downtime is expected during the maintenance window.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 9, 10:34 CDT
Mar 24, 2020
Resolved - iland Cloud Operations have confirmed this issue as resolved. This incident will now be closed. A full RCA will be provided once final analysis on the root/cause is completed.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 24, 23:29 CDT
Monitoring - iland Cloud Operations have completed remediation steps to address the affected VPGs in the DAL02 data center location. All affected VPGs are now back online and currently in delta sync and/or already operating within the required RPO.

We will continue to monitor the environment for the next 24-hours to ensure stability before closing this incident. A full RCA will provided at time of closure.

Thank you,

iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 17, 16:27 CDT
Identified - iland engineers are working on an issue with the iland Secure DRaaS with Zerto environment in the DAL02 datacenter. Some customers affected by the issue will see their VPG in an error state while we work to remediate. Once the resolved, affected VPGs should go into a bitmap sync.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 16, 18:06 CDT
Completed - The scheduled maintenance has been completed.
Mar 24, 23:00 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 24, 22:00 CDT
Scheduled - Duration: 03/24/2020 from 20:00 PT to 21:00 PT
Location: Los Angeles
Type Of Event: Scheduled Infrastructure Maintenance

As part of ongoing maintenance, iland engineers will be updating the iland Cloud and iland Secure Cloud environment. This latest vendor update to our iland Cloud and iland Secure Cloud product incorporates significant bug fixes and enhancements that allow iland to offer a more stable environment.

During this maintenance window, operations to virtual machines via the iland Secure Cloud Console will be inaccessible. Customers using iland's Secure DRaaS products may see intermittent site disconnects and will be unable to perform operations on protected virtual machines. We also require failover tests to be stopped during the maintenance. There will be no impact to running or protected virtual machines during this maintenance.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 20, 14:46 CDT
Completed - The scheduled maintenance has been completed.
Mar 24, 21:00 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 24, 20:00 CDT
Scheduled - Duration: 03/24/2020 from 20:00 CT to 21:00 CT
Location: Dallas (DAL25)
Type Of Event: Scheduled Infrastructure Maintenance

As part of ongoing maintenance, iland engineers will be updating the iland Cloud and iland Secure Cloud environment. This latest vendor update to our iland Cloud and iland Secure Cloud product incorporates significant bug fixes and enhancements that allow iland to offer a more stable environment.

During this maintenance window, operations to virtual machines via the iland Secure Cloud Console will be inaccessible. Customers using iland's Secure DRaaS products may see intermittent site disconnects and will be unable to perform operations on protected virtual machines. We also require failover tests to be stopped during the maintenance. There will be no impact to running or protected virtual machines during this maintenance.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 20, 14:30 CDT
Completed - The scheduled maintenance has been completed.
Mar 24, 17:30 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 24, 16:30 CDT
Scheduled - Duration: 03/24/2020 from 21:30 GMT to 22:30 GMT
Location: Manchester
Type Of Event: Scheduled Infrastructure Maintenance

As part of ongoing maintenance, iland engineers will be updating the iland Cloud and iland Secure Cloud environment. This latest vendor update to our iland Cloud and iland Secure Cloud product incorporates significant bug fixes and enhancements that allow iland to offer a more stable environment.

During this maintenance window, operations to virtual machines via the iland Secure Cloud Console will be inaccessible. Customers using iland's Secure DRaaS products may see intermittent site disconnects and will be unable to perform operations on protected virtual machines. We also require failover tests to be stopped during the maintenance. There will be no impact to running or protected virtual machines during this maintenance.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 20, 14:26 CDT
Completed - The scheduled maintenance has been completed.
Mar 24, 16:00 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 24, 15:00 CDT
Scheduled - Duration: 03/24/2020 from 20:00 GMT to 21:00 GMT
Location: London
Type Of Event: Scheduled Infrastructure Maintenance

As part of ongoing maintenance, iland engineers will be updating the iland Cloud and iland Secure Cloud environment. This latest vendor update to our iland Cloud and iland Secure Cloud product incorporates significant bug fixes and enhancements that allow iland to offer a more stable environment.

During this maintenance window, operations to virtual machines via the iland Secure Cloud Console will be inaccessible. Customers using iland's Secure DRaaS products may see intermittent site disconnects and will be unable to perform operations on protected virtual machines. We also require failover tests to be stopped during the maintenance. There will be no impact to running or protected virtual machines during this maintenance.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 20, 14:24 CDT
Mar 23, 2020
Completed - The scheduled maintenance has been completed.
Mar 23, 22:30 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 23, 21:30 CDT
Scheduled - Duration: 03/23/2020 from 21:30 CT to 22:30 CT
Location: Dallas (DAL06)
Type Of Event: Scheduled Infrastructure Maintenance

As part of ongoing maintenance, iland engineers will be updating the iland Cloud and iland Secure Cloud environment. This latest vendor update to our iland Cloud and iland Secure Cloud product incorporates significant bug fixes and enhancements that allow iland to offer a more stable environment.

During this maintenance window, operations to virtual machines via the iland Secure Cloud Console will be inaccessible. Customers using iland's Secure DRaaS products may see intermittent site disconnects and will be unable to perform operations on protected virtual machines. We also require failover tests to be stopped during the maintenance. There will be no impact to running or protected virtual machines during this maintenance.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 20, 14:18 CDT
Completed - The scheduled maintenance has been completed.
Mar 23, 21:00 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 23, 20:01 CDT
Scheduled - Duration: 03/23/2020 from 20:00 CT to 21:00 CT
Location: Dallas (DAL02)
Type Of Event: Scheduled Infrastructure Maintenance

As part of ongoing maintenance, iland engineers will be updating the iland Cloud and iland Secure Cloud environment. This latest vendor update to our iland Cloud and iland Secure Cloud product incorporates significant bug fixes and enhancements that allow iland to offer a more stable environment.

During this maintenance window, operations to virtual machines via the iland Secure Cloud Console will be inaccessible. Customers using iland's Secure DRaaS products may see intermittent site disconnects and will be unable to perform operations on protected virtual machines. We also require failover tests to be stopped during the maintenance. There will be no impact to running or protected virtual machines during this maintenance.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 20, 14:14 CDT
Mar 22, 2020

No incidents reported.

Mar 21, 2020

No incidents reported.

Mar 20, 2020
Resolved - No further issues have been identified or reported as a result of this incident. A final RCA will be provided upon final collection of results from the remediation steps.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 20, 18:29 CDT
Monitoring - Significant improvement with cloud services in the Dallas (DAL22) iland Secure Cloud Backup environment has been seen due to the recent implementation of additional resources and workload balancing by iland senior engineers. iland will continue to work with our software vendor regarding a patch to remedy the connection problem which was uncovered and reported to Veeam.

We will continue to monitor performance and follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 13, 15:48 CDT
Update - Some iland customers are experiencing degraded performance with their cloud services in Dallas (DAL22). We are working to resolve it quickly as possible.

How we are resolving it:

iland senior engineers are continuing to spread workloads across additional resources which are being implemented. This will allow iland to remedy the connection problem (previously posted) which was uncovered and reported to Veeam. iland continues to work with our software vendor regarding a patch for the bug and will provide updates as they become available.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 26, 03:47 CST
Update - Some iland customers are experiencing degraded performance with their cloud services in our Dallas (DAL22). We are aware of this issue and are working to resolve it quickly.

How we are resolving it:

iland senior engineers are spreading workloads across additional resources which are being implemented within the next two weeks. This will allow iland to remedy the connection problem (previously posted) which was uncovered and reported to Veeam. iland continues to work with our software vendor regarding a patch for the bug and will provide updates as they become available.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 20, 16:21 CST
Update - Some iland customers are experiencing degraded performance with their cloud services in our Dallas (DAL22). We are aware of this issue and are working to resolve it quickly.

How we are resolving it:

iland senior engineers are spreading workloads across additional resources which are being implemented within the next two weeks. This will allow iland to remedy the connection problem (previously posted) which was uncovered and reported to Veeam. iland continues to work with our software vendor regarding a patch for the bug and will provide updates as they become available.

We will continue to follow up with individual customers directly. Please contact support with any questions or concerns.

Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 13, 16:47 CST
Update - Some iland customers are experiencing degraded performance with their cloud services in our Dallas (DAL22). We are aware of this issue and are working to resolve it quickly.

Details:

Several customers have reported long running backup jobs and/or job failures with the iland Secure Cloud Backup environment. This is a result of a series of hardware and software errors that we have experienced recently. We are working with our vendors to troubleshoot these errors and expect to resolve both issues quickly.

How we are resolving it:

From a hardware perspective, we have replaced the faulty mid-plane globally that was causing drive issues for some customers. This also caused some Veeam repositories to suffer drive disconnections, causing jobs to fail.

For the software, we uncovered and reported an issue to Veeam in the way our provider side Veeam B&R servers communicate with the backend repositories over SSH. There is a bug in the way the VBR maintains stale SSH connections to the repositories at scale, keeping the SSH session alive and not tearing it down. The end result of this is that the repositories no longer accept new connections to start the VeeamAgent process to process tenant tasks. We have implemented a workaround which involves controlling the issue reported above to maintain a good state for incoming task processing, allowing customer tenant jobs to process accordingly. We will provide an update once we get a proper patch from Veeam, but so far the workaround is providing good results.

Additional Details:

We are also aware that our iland status page is reporting 100% uptime, even with degraded performance. We recognize that limiting our status page to only availability status is not sufficient and does not accurately reflect the service environment. We are committed to improving our service reporting and will update our status page to provide additional detail updates as we resolve these issues.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 10, 10:45 CST
Update - Some iland customers are experiencing degraded performance with their cloud services in our Dallas (DAL22). We are aware of this issue and are working to resolve it quickly.

Details:

Several customers have reported long running backup jobs and/or job failures with the iland Secure Cloud Backup environment. This is a result of a series of hardware and software errors that we have experienced recently. We are working with our vendors to troubleshoot these errors and expect to resolve both issues quickly.

How we are resolving it:

From a hardware perspective, we have replaced the faulty mid-plane globally that was causing drive issues for some customers. This also caused some Veeam repositories to suffer drive disconnections, causing jobs to fail.

For the software, we uncovered and reported an issue to Veeam in the way our provider side Veeam B&R servers communicate with the backend repositories over SSH. There is a bug in the way the VBR maintains stale SSH connections to the repositories at scale, keeping the SSH session alive and not tearing it down. The end result of this is that the repositories no longer accept new connections to start the VeeamAgent process to process tenant tasks. We have implemented a workaround which involves controlling the issue reported above to maintain a good state for incoming task processing, allowing customer tenant jobs to process accordingly. We will provide an update once we get a proper patch from Veeam, but so far the workaround is providing good results.

Additional Details:

We are also aware that our iland status page is reporting 100% uptime, even with degraded performance. We recognize that limiting our status page to only availability status is not sufficient and does not accurately reflect the service environment. We are committed to improving our service reporting and will update our status page to provide additional detail updates as we resolve these issues.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 6, 10:40 CST
Identified - Some iland customers are experiencing degraded performance with their cloud services in our Dallas (DAL22). We are aware of this issue and are working to resolve it quickly.

Details:

Several customers have reported long running backup jobs and/or job failures with the iland Secure Cloud Backup environment. This is a result of a series of hardware and software errors that we have experienced recently. We are working with our vendors to troubleshoot these errors and expect to resolve both issues quickly.

How we are resolving it:

From a hardware perspective, we have replaced the faulty mid-plane globally that was causing drive issues for some customers. This also caused some Veeam repositories to suffer drive disconnections, causing jobs to fail.

For the software, we uncovered and reported an issue to Veeam in the way our provider side Veeam B&R servers communicate with the backend repositories over SSH. There is a bug in the way the VBR maintains stale SSH connections to the repositories at scale, keeping the SSH session alive and not tearing it down. The end result of this is that the repositories no longer accept new connections to start the VeeamAgent process to process tenant tasks. We have a workaround for this issue and are contacting customers directly. The workaround we implemented involves controlling the issue reported above to maintain a good state for incoming task processing, allowing customer tenant jobs to process accordingly. We will provide an update once we get a proper patch from Veeam, but so far the workaround is providing good results.

Additional Details:

We are also aware that our iland status page is reporting 100% uptime, even with degraded performance. We recognize that limiting our status page to only availability status is not sufficient and does not accurately reflect the service environment. We are committed to improving our service reporting and will update our status page to provide additional detail updates as we resolve these issues.


Thank you,

iland Support

https://www.iland.com/support/

US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Feb 5, 13:30 CST
Mar 19, 2020

No incidents reported.

Mar 18, 2020
Completed - The scheduled maintenance has been completed.
Mar 18, 22:01 CDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 18, 21:00 CDT
Scheduled - Duration: 03/18/2020, from 21:00 CT to 22:00 CT
Location: Dallas (DAL02)

iland engineers will be performing database maintenance in the iland Cloud and iland Secure Cloud environment. During this maintenance window, operations to virtual machines via the iland Secure Cloud Console will be inaccessible. Customers using iland's Secure DRaaS products may see intermittent site disconnects and will be unable to perform operations on protected virtual machines. We also require failover tests to be stopped during the maintenance. There will be no impact to running or protected virtual machines during this maintenance.

It is our goal to provide the highest level of uptime and reliability in order for us to consistently serve our customers. If you have any questions regarding this maintenance, please contact our Support team.

Thank you,
iland Support

https://www.iland.com/support/
US 1-800-697-7088
UK +(44) (0) 20 7096 0149
SG +(65) 3158 8438
NL +(31) (0) 10 808 0440
AU +(61) (0) 2 9056 7004
Mar 18, 15:15 CDT
Mar 15, 2020

No incidents reported.