Incident Details

Resolved [13/06/2020 06:33]

Update [04/06/2020 18:39]

We have unfortunately noticed further problems with the Control Panel database server.

As a result, we have gone back to a backup from 03/06/2020, the last known good backup of the control panel server. We will be working on restoring data manually that was placed over the 03/06 / 04/06 manually tomorrow.

We apologise for the continued disruption this is causing.

Update [04/06/2020 17:04]

We can confirm that access control panel and shared SQL are restored. We have had a sleepless night and hard day dealing with a perfect storm of hardware, network, data integrity and software issues across multiple platforms which have challenged us. We are dreadfully sorry that this extended into working hours and it's not until close of play that access has been restored.

We are still working on restoring access to Windows Server 19 accounts, and hope to have this operational this afternoon.

Thank you for your patience and understanding.

Update [04/06/2020 16:30]

We are close to providing access again. We are waiting for on-site hands in the data centre to correct a networking issue preventing it from returning. Thank you for your continued patience.

Update [04/06/2020 12:16]

We had hoped to have service resumed by now. It's just going to take a little longer. We appreciate your patience, and apologies for the continued outage. We will update you just as soon as we're confident of an ETR.

Update [04/06/2020 07:21]

Users on Windows Server 19 will also be affected by this issue. We hope to get service resumed as quickly as possible.

Update [04/06/2020 05:27]

Despite works through the night we have not been able to bring the control panel and shared SQL services back online. We are currently aiming to resolve by midday. Apologies for the disruption this may cause.

Investigating [03/06/2020 23:03]

The control panel and Shared SQL servers are currently down. We hope to restore service soon.