Power outage in South London
Incident Report for Pulsant Service
Resolved
This incident has been resolved.
Posted Jun 28, 2018 - 07:42 BST
Update
We are continuing to monitor for any further issues.
Posted Jun 28, 2018 - 07:25 BST
Update
Work has completed and normal service is restored. We shall continue to monitor.
Posted Jun 28, 2018 - 06:06 BST
Update
The planned emergency maintenance in South London Data Hall 1 - has been completed successfully with no service interruption. Normal service was resumed at 19.30 BST on Wednesday 27th June and was monitored for the last 30 minutes. This completes the planned maintenance for this evening, if you have any issues, please raise a helpdesk ticket in the normal way.
Posted Jun 27, 2018 - 20:00 BST
Update
Remediation plans are still on target. Full power resilience has been restored to the site.

Preparation is still scheduled to begin at 18:00 and the work is expected to be completed within 2 hours.
Posted Jun 27, 2018 - 17:02 BST
Update
Plans have been finalised, delivery of parts is scheduled for 16:00

Preparation work will commence when the parts have been delivered.

Next update is due at 17:00
Posted Jun 27, 2018 - 15:19 BST
Update
At this time we continue to expect delivery of replacement components required at 16:00, when remedial work is expected to commence.

The Pulsant South London facility continues to run on protected electrical supply with two tiers of redundancy.

A further update will be provided at 15:00
Posted Jun 27, 2018 - 14:09 BST
Update
At this time we are awaiting delivery of replacement components required for the remediation. As such, work is now expected to commence at 16:00.

The Pulsant South London facility continues to run on protected electrical supply with two tiers of redundancy.

A further update will be provided at 14:00.
Posted Jun 27, 2018 - 09:56 BST
Update
Investigation has identified a significant failure in the electrical switch infrastructure which has extended the time to full remediation. Due to the resilient electrical configuration the site continues to run on secondary power with two tiers of redundancy.

The full remediation is scheduled for midday tomorrow.

Next update 10:00 tomorrow.
Posted Jun 26, 2018 - 18:32 BST
Update
We are investigating reports from a few customers who appear to still be experiencing intermittent network issues. Please contact Pulsant support if you have any issues.
Posted Jun 26, 2018 - 15:49 BST
Monitoring
We have restored power to all services in South London. Engineers continue to work on restoring full resilience to all services. We are expecting minimal disruption at this time as we restore full service. If there are any concerns please contact us at support@pulsant.com or 0345 11 99999.

The next scheduled update will be at 18:00.
Posted Jun 26, 2018 - 14:11 BST
Update
We continue to work on the isolated environment to restore full resilience to the UPS infrastructure. Work has allowed us to bring full power back, as an extra precaution we have taken the decision to run on secondary power sources until full resilience can be restored. Environment services have been restored. We are beginning to restore service to the network followed by full restoration of all services.

Next Update 14:00.
Posted Jun 26, 2018 - 13:37 BST
Update
We have now successfully isolated the fault and have brought the healthy infrastructure online and are working to power up the environmental systems. Engineers are currently completeing further checks in preparation for the restoration of power to network services. On completion of this we will begin full system recovery. This is still estimated to begin at 14:00.

Next update 13:30.
Posted Jun 26, 2018 - 13:15 BST
Update
We have isolated the fault and have confirmed that remaining components are healthy. We are working with the vendors to bring the power infrastructure back online. We are working to restore service as soon as possible. Early indications are that this should be within the next two hours.

Next update at 13:00.
Posted Jun 26, 2018 - 12:09 BST
Update
The cause of the fault condition has been identified. In line with health and safety requirements, we are working to isolate the affected components. If safe to do so we will then restore power and work to bring the facility back on line. Engineers continue to focus on restoring power as quickly as possible.

Next update 12:00
Posted Jun 26, 2018 - 11:22 BST
Identified
We have identified the source of the fault condition. We are working to reset the fault condition to restore power as soon as possible with support from onsite electrical engineers.
Posted Jun 26, 2018 - 10:33 BST
Update
We apologise to customers who continue to be impacted by the power issue in our South London Facility.

There is a fault condition in part of the facility which we are investigating.
This has also partially impacted IP Transit services for some customers.

Next update at 10:30.
Posted Jun 26, 2018 - 10:02 BST
Update
Apology for the delay in sending an update. The power issue has been identified to only be affecting one data hall. Some customers will experience network connectivity issues.
Engineers still investigating cause.
Next Update at 10am.
Posted Jun 26, 2018 - 09:25 BST
Update
We are currently still investigating the issue.

Next update at 09:00.
Posted Jun 26, 2018 - 08:35 BST
Update
The network in the South London facility has been affected by the power issue. Multiple customers will be impacted. Engineers still investigating. Next update in 30 minutes.
Posted Jun 26, 2018 - 08:02 BST
Investigating
Engineers have been alerted to an interuptuion to the power in the South London Data Centre. The facility is currently running on alternate power source. Engineers are investigating. Further update in 30 minutes.
Posted Jun 26, 2018 - 07:26 BST
This incident affected: Managed Networks (South London) and Data Centre Services (South London).