Mains power fault - TVHC datacentres
Incident Report for Pulsant Service
Resolved
Resilient/Redundant Power from our electricity distribution provider has now been restored to both our TVHC1 and TVHC2 sites, this was completed at 11.15am. We then monitored for an hour to ensure there was stability, prior to doing a controlled transfer back onto the electricity distribution’s redundant feeds.

We have not seen any issues during the last two hours, since returning to utility mains, and we now intend to close this incident down. If you have any issues or concerns, please raise a helpdesk ticket in the normal way.

We have now completed a final delivery of fuel to replenish all supplies on site and return the site back to full capacity.

Should you have any questions, please raise a ticket by emailing support@pulsant.com or by calling our support team on 0845 1199 999.
Posted Dec 02, 2016 - 14:07 GMT
Update
Following on from our notification yesterday evening, after a successful period of stability, our electricity distribution company is looking at restoring redundant supplies to our TVHC1 and TVHC2 facilities.

As a precautionary measure Pulsant have decided to do a controlled transfer to generator support whilst this final switching is completed and once completed we will organise a controlled transfer back to this configuration.

This work is due to commence today, in a window between 0930GMT - 1300GMT. We will send out a further notification once these works have been completed.

This will have no impact to customer services but in light of recent events we believe this is the best risk adverse plan.

Thank you for your understanding and patience.

Should you have any questions, please raise a ticket by emailing support@pulsant.com or by calling our support team on 0845 1199 999.

Pulsant appreciate your patience and understanding during this incident.
Posted Dec 02, 2016 - 09:02 GMT
Monitoring
Following on from our previous notification, our utility provider has confirmed with us that the utility mains has been restored to site.

Upon restoration of mains from the utility provider, we monitored to ensure stability for 30 minutes to ensure there were no issues before returning TVHC1 to utility mains supply.

Currently our fuel contractor is on site and replenishing all fuel supplies to bring the sites back to full capacity.

Since performing the controlled return to utility mains, we have monitored the site power for the last hour to ensure no issues and now consider the initial incident closed.

TVHC2 continues to remain stable on utility supply which was restored earlier in the day.

However, monitoring and heightened awareness for both sites will remain in place overnight to ensure that we have no issues and ensure a suitable time of stability has elapsed before closing the incident completely.

Should you have any questions, please raise a ticket by emailing support@pulsant.com or by calling our support team on 0845 1199 999.

Pulsant appreciate your patience and understanding during this incident.
Posted Dec 01, 2016 - 15:28 GMT
Update
Following on from our previous notification, our utility provider has informed us that they are completing some final testing to then reinstate utility power to TVHC1. They have revised their ETA and are working towards a resolution time of 14:00GMT.

TVHC1 – continues to remain on "Generator support" at this time
TVHC2 – remains on utility mains, and resilience restored with us having our standby generator support available if required.

Upon confirmation of utility power being restored, we will then instruct our fuel contractor to replenish all fuel supplies on site and ensure we are returned to full capacity.

There continues to be no service or customer impact on either site.

We will provide a further update with more information within the next 2 hours on any updates we receive.

On site presence and heightened awareness is in place and will remain until restoration of utility supply has been completed on both sites and a subsequent period of stability has elapsed.

Should you have any questions, please raise a ticket by emailing support@pulsant.com or by calling our support team on 0845 1199 999.

Please accept our sincere apologies for any inconvenience this issue is causing.
Posted Dec 01, 2016 - 12:27 GMT
Update
Following on from our previous notification, our utility power provider have now completed restoration of the utility supply to our TVHC2 facility.

The utility company are now working on the restoration of utility supply to our TVHC1 facility and have revised their ETA and are working towards a resolution time of 13:00GMT.
TVHC1 – continues to remain on "Generator support"
TVHC2 – now has utility supply restored, we have successfully transferred back to the utility supply and resilience restored with us having our standby generator support available if required.

Once utility has been restored to both sites, a further fuel delivery will then be undertaken to replenish all fuel supplies and ensure returned to full capacity.

TVHC1 will remain on generator support for a period even after restoration of utility supply to ensure stability, once we are confident then we will undertake a controlled switch back to the utility supply.
There continues to be no service or customer impact on either site.

We will provide a further update with more information within the next 3 hours on any updates we receive.
On site presence and heightened awareness is in place and will remain until restoration of utility supply has been completed on both sites and a subsequent period of stability has elapsed.

Should you have any questions, please raise a ticket by emailing support@pulsant.com or by calling our support team on 0845 1199 999.

Please accept our sincere apologies for any inconvenience this issue is causing.
Posted Dec 01, 2016 - 07:40 GMT
Update
Following on from our previous notification, SSE continue to work on deploying a temporary work-around solution.

SSE have not revised their ETA and are continuing to work towards resolution by 11:00GMT.

Our facilities remain on "Generator support".
Both sites have completed the refueling process and are back to full capacity with regards to fuel levels.
Once the temporary solution has been completed and we are back on utility provided power we will undertake another refueling process to ensure supplies are replenished accordingly.

Sites will remain on generator support for a period even after completion of the temporary solution by SSE, to ensure stability, prior to then undertaking a controlled switch back to utility mains supply.

There continues to be no service or customer impact.

We will provide a further update with more information within the next 2 hours on any updates we receive.
On site presence and heightened awareness is in place and will remain until completion of the temporary fix and a subsequent period of stability.

Should you have any questions, please raise a ticket by emailing support@pulsant.com or by calling our support team on 0845 1199 999.

Please accept our sincere apologies for any inconvenience this issue is causing.
Posted Dec 01, 2016 - 05:12 GMT
Update
Following on from our previous notification, SSE have confirmed that they have now identified and isolated the fault. They are now working on deploying a temporary fix to get power restored to the RG6 area and, more importantly for us, our TVHC1 & 2 facilities.

SSE have revised their ETA for implementation of temporary resolution of 11:00GMT. Once this is in place, they can work on a permanent resolution to the issue and at this time have not provided us with a definitive date or time for this. We continue to remain in contact with them and obtain updates.

Our facilities remain on "Generator support". We have, as a precautionary measure, invoked our fuel replenishment process. We will have onsite fuel replenished to ensure that we have adequate fuel supplies on site to support the load until such time as the temporary solution from SSE has been completed.
We will remain on generator for a period even after completion of temporary solution to ensure stability, prior to undertaking a controlled switch back to utility mains supply.
There continues to be no service or customer impact.

We will provide a further update with more information within the next 2 hours on any updates we receive.
On site presence and heightened awareness is in place and will remain until completion of temporary fix and a subsequent period of stability.

Should you have any questions, please raise a ticket by emailing support@pulsant.com or by calling our support team on 0845 1199 999.

Please accept our sincere apologies for any inconvenience this issue is causing.
Posted Dec 01, 2016 - 02:55 GMT
Update
Following on from our previous notification earlier, SSE have now confirmed that they have a major fault affecting the RG6 area which is affecting both our Pulsant TVHC1 & 2 facilities.

SSE have given a revised ETA now for restoration of power of 04:00GMT.

The facilities remain on "Generator support" and we have switched to "manual" to ensure no unnecessary switching is done as utility power is restored.

We will remain on generator even after restoration of utility mains power and await a stability period, prior to undertaking a controlled switch back to utility mains supply.
There continues to be no impact to any service or customer impact.

We will provide a further update with more information within the next 2 hours on any updates we receive.
On site presence and heightened awareness is in place and will remain until restoration of utility mains and a subsequent period of stability.

Should you have any questions, please raise a ticket by emailing support@pulsant.com or by calling our support team on 0845 1199 999.

Please accept our sincere apologies for any inconvenience this issue is causing.
Posted Dec 01, 2016 - 00:03 GMT
Update
The following update applies to services at the Pulsant TVHC facilities.

SSE are continuing to investigate a mains power disruption to the wider RG6 area and have provided an updated ETA for fix of 2359GMT. Pulsant's TVHC datacentres will continue to run on diesel power generation while we await the "all-clear" from SSE.

*There is no impact to any customer services and the sites are being supported by diesel power generation.*

We will provide a further update with more information within the next 60 minutes.

Should you have any questions, please raise a ticket by emailing support@pulsant.com or by calling our support team on 0845 1199 999.
Posted Nov 30, 2016 - 21:37 GMT
Identified
The following notification applies to services at the Pulsant TVHC facilities.

We are currently investigating a mains power failure at this site. SSE are aware of the outage and have provided a current ETA for fix of 2045GMT.

*There is no impact to any customer services and the sites are being supported by diesel power generation.*

We will provide a further update with more information within the next 60 minutes.

Should you have any questions, please raise a ticket by emailing support@pulsant.com or by calling our support team on 0845 1199 999.
Posted Nov 30, 2016 - 20:02 GMT