All Systems Operational
Past Incidents
Dec 8, 2016
The following notification applies to Network services in the Milton Keynes Datacentre:

At 1:13 this morning a component in a core router experienced an issue. This was not a hard failure, therefore service became degraded before the network self-healed. This issue is still under investigation. Initial findings suggest there was traffic interruption for some paths out of Milton Keynes between 1:13 - 1:16 and 1:17 and 1:19. Pulsant is currently liaising with the support vendor for a root cause and to determine if the configuration could be improved to reduce disruption if an issue of this nature reoccurs.

All services have been restored in the Milton Keynes network. If you have any issues, please raise a ticket by emailing [email protected] or by calling our support team on 0845 1199 999.
Dec 8, 13:05 GMT
Resolved - Pulsant upstream provider has confirmed that their service has been fully restored. Our peering sessions to them have been brought back online and normal service has resumed.
Dec 8, 10:14 GMT
Identified - Pulsant upstream provider confirmed an issue with their core device located in London. Our peering sessions with them remain down at this time. We are continuing to monitor and will restore this transit service at the earliest opportunity.
Dec 7, 21:54 GMT
Investigating - We have identified the issues to with one of our upstream transit providers. We have shut down our connection to the effected provider for the time being. Customers should see there services resuming.

We will continue to provide updates until this issue is fully resolved.
Dec 7, 19:49 GMT
Identified - We are currently experiencing intermittent connectivity issues that may be effecting IP transit customers.

We are currently investigating the issues and will provide updates once we have completed initial investigations
Dec 7, 19:26 GMT
Dec 7, 2016
Completed - The scheduled maintenance has been completed.
Dec 7, 19:00 GMT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 7, 18:00 GMT
Scheduled - What:
Pulsant will be performing essential UPS maintenance at our Rose Street POP.

Why:
The UPS has reported a fault condition and requires the battery to be replaced.

When:
The maintenance will take place during the following window:

Start: Wednesday 7th December 2016 18:00 GMT
End: Wednesday 7th December 2016 19:00 GMT

There is no outage expected however an at risk period should be observed for the duration of this maintenance.

If you have any questions, please raise a ticket by emailing [email protected] or by calling our support team directly.

For your reference, please see our status page for updates on planned maintenance and service incidents. This can be found at http://status.pulsant.com.

Please accept our sincere apologies for any inconvenience this may cause.

Regards,
Pulsant Support Team

0845 1199 999
Dec 5, 10:58 GMT
Dec 6, 2016

No incidents reported.

Dec 5, 2016
Resolved - This incident has been resolved.
Dec 5, 10:57 GMT
Identified - We've noticed intermittent issues with externally hosted sites on 123-reg.
We will continue to monitor and watch for updates on https://twitter.com/123reg
Nov 28, 10:27 GMT
Resolved - This incident has been resolved.
Dec 5, 10:57 GMT
Monitoring - Power was restored at 11:10,

We will now monitor
Nov 27, 11:59 GMT
Update - Fault has been identified by SP and they are working to resolve this, ETA 11:30, Next update 12:00
Nov 27, 11:59 GMT
Update - EFM circuits are now in a down state, Pulsant is chasing BT/SP for an update.
Nov 27, 10:33 GMT
Identified - Power has been lost at BT's Glasgow central Exchange, this has been for a duration longer than the UPS batteries can sustain. Other service providers have also confirmed power loss. This is a wider power issue in the G2 Area
Nov 27, 10:31 GMT
Dec 4, 2016
Monitoring - We have experienced this evening a small sequence of interrupted utility power delivery to our Maidenhead facilities from our electricity distribution company.
The power interruptions were very short and the DC infrastructure worked to design and supported the sites throughout with no issues
There was no service or customer impact seen during these incidents.

Should you have experienced any issues or have any questions, please raise a helpdesk ticket in the normal way by emailing our support team at [email protected] or by calling our support team on 0845 1199 999.

For your reference, please see our service status page for updates on incidents. This can be found at http://status.pulsant.com
We will continue to monitor the situation and update if there are any further incidents.
Dec 3, 22:18 GMT
Dec 2, 2016
Resolved - Resilient/Redundant Power from our electricity distribution provider has now been restored to both our TVHC1 and TVHC2 sites, this was completed at 11.15am. We then monitored for an hour to ensure there was stability, prior to doing a controlled transfer back onto the electricity distribution’s redundant feeds.

We have not seen any issues during the last two hours, since returning to utility mains, and we now intend to close this incident down. If you have any issues or concerns, please raise a helpdesk ticket in the normal way.

We have now completed a final delivery of fuel to replenish all supplies on site and return the site back to full capacity.

Should you have any questions, please raise a ticket by emailing [email protected] or by calling our support team on 0845 1199 999.
Dec 2, 14:07 GMT
Update - Following on from our notification yesterday evening, after a successful period of stability, our electricity distribution company is looking at restoring redundant supplies to our TVHC1 and TVHC2 facilities.

As a precautionary measure Pulsant have decided to do a controlled transfer to generator support whilst this final switching is completed and once completed we will organise a controlled transfer back to this configuration.

This work is due to commence today, in a window between 0930GMT - 1300GMT. We will send out a further notification once these works have been completed.

This will have no impact to customer services but in light of recent events we believe this is the best risk adverse plan.

Thank you for your understanding and patience.

Should you have any questions, please raise a ticket by emailing [email protected] or by calling our support team on 0845 1199 999.

Pulsant appreciate your patience and understanding during this incident.
Dec 2, 09:02 GMT
Monitoring - Following on from our previous notification, our utility provider has confirmed with us that the utility mains has been restored to site.

Upon restoration of mains from the utility provider, we monitored to ensure stability for 30 minutes to ensure there were no issues before returning TVHC1 to utility mains supply.

Currently our fuel contractor is on site and replenishing all fuel supplies to bring the sites back to full capacity.

Since performing the controlled return to utility mains, we have monitored the site power for the last hour to ensure no issues and now consider the initial incident closed.

TVHC2 continues to remain stable on utility supply which was restored earlier in the day.

However, monitoring and heightened awareness for both sites will remain in place overnight to ensure that we have no issues and ensure a suitable time of stability has elapsed before closing the incident completely.

Should you have any questions, please raise a ticket by emailing [email protected] or by calling our support team on 0845 1199 999.

Pulsant appreciate your patience and understanding during this incident.
Dec 1, 15:28 GMT
Update - Following on from our previous notification, our utility provider has informed us that they are completing some final testing to then reinstate utility power to TVHC1. They have revised their ETA and are working towards a resolution time of 14:00GMT.

TVHC1 – continues to remain on "Generator support" at this time
TVHC2 – remains on utility mains, and resilience restored with us having our standby generator support available if required.

Upon confirmation of utility power being restored, we will then instruct our fuel contractor to replenish all fuel supplies on site and ensure we are returned to full capacity.

There continues to be no service or customer impact on either site.

We will provide a further update with more information within the next 2 hours on any updates we receive.

On site presence and heightened awareness is in place and will remain until restoration of utility supply has been completed on both sites and a subsequent period of stability has elapsed.

Should you have any questions, please raise a ticket by emailing [email protected] or by calling our support team on 0845 1199 999.

Please accept our sincere apologies for any inconvenience this issue is causing.
Dec 1, 12:27 GMT
Update - Following on from our previous notification, our utility power provider have now completed restoration of the utility supply to our TVHC2 facility.

The utility company are now working on the restoration of utility supply to our TVHC1 facility and have revised their ETA and are working towards a resolution time of 13:00GMT.
TVHC1 – continues to remain on "Generator support"
TVHC2 – now has utility supply restored, we have successfully transferred back to the utility supply and resilience restored with us having our standby generator support available if required.

Once utility has been restored to both sites, a further fuel delivery will then be undertaken to replenish all fuel supplies and ensure returned to full capacity.

TVHC1 will remain on generator support for a period even after restoration of utility supply to ensure stability, once we are confident then we will undertake a controlled switch back to the utility supply.
There continues to be no service or customer impact on either site.

We will provide a further update with more information within the next 3 hours on any updates we receive.
On site presence and heightened awareness is in place and will remain until restoration of utility supply has been completed on both sites and a subsequent period of stability has elapsed.

Should you have any questions, please raise a ticket by emailing [email protected] or by calling our support team on 0845 1199 999.

Please accept our sincere apologies for any inconvenience this issue is causing.
Dec 1, 07:40 GMT
Update - Following on from our previous notification, SSE continue to work on deploying a temporary work-around solution.

SSE have not revised their ETA and are continuing to work towards resolution by 11:00GMT.

Our facilities remain on "Generator support".
Both sites have completed the refueling process and are back to full capacity with regards to fuel levels.
Once the temporary solution has been completed and we are back on utility provided power we will undertake another refueling process to ensure supplies are replenished accordingly.

Sites will remain on generator support for a period even after completion of the temporary solution by SSE, to ensure stability, prior to then undertaking a controlled switch back to utility mains supply.

There continues to be no service or customer impact.

We will provide a further update with more information within the next 2 hours on any updates we receive.
On site presence and heightened awareness is in place and will remain until completion of the temporary fix and a subsequent period of stability.

Should you have any questions, please raise a ticket by emailing [email protected] or by calling our support team on 0845 1199 999.

Please accept our sincere apologies for any inconvenience this issue is causing.
Dec 1, 05:12 GMT
Update - Following on from our previous notification, SSE have confirmed that they have now identified and isolated the fault. They are now working on deploying a temporary fix to get power restored to the RG6 area and, more importantly for us, our TVHC1 & 2 facilities.

SSE have revised their ETA for implementation of temporary resolution of 11:00GMT. Once this is in place, they can work on a permanent resolution to the issue and at this time have not provided us with a definitive date or time for this. We continue to remain in contact with them and obtain updates.

Our facilities remain on "Generator support". We have, as a precautionary measure, invoked our fuel replenishment process. We will have onsite fuel replenished to ensure that we have adequate fuel supplies on site to support the load until such time as the temporary solution from SSE has been completed.
We will remain on generator for a period even after completion of temporary solution to ensure stability, prior to undertaking a controlled switch back to utility mains supply.
There continues to be no service or customer impact.

We will provide a further update with more information within the next 2 hours on any updates we receive.
On site presence and heightened awareness is in place and will remain until completion of temporary fix and a subsequent period of stability.

Should you have any questions, please raise a ticket by emailing [email protected] or by calling our support team on 0845 1199 999.

Please accept our sincere apologies for any inconvenience this issue is causing.
Dec 1, 02:55 GMT
Update - Following on from our previous notification earlier, SSE have now confirmed that they have a major fault affecting the RG6 area which is affecting both our Pulsant TVHC1 & 2 facilities.

SSE have given a revised ETA now for restoration of power of 04:00GMT.

The facilities remain on "Generator support" and we have switched to "manual" to ensure no unnecessary switching is done as utility power is restored.

We will remain on generator even after restoration of utility mains power and await a stability period, prior to undertaking a controlled switch back to utility mains supply.
There continues to be no impact to any service or customer impact.

We will provide a further update with more information within the next 2 hours on any updates we receive.
On site presence and heightened awareness is in place and will remain until restoration of utility mains and a subsequent period of stability.

Should you have any questions, please raise a ticket by emailing [email protected] or by calling our support team on 0845 1199 999.

Please accept our sincere apologies for any inconvenience this issue is causing.
Dec 1, 00:03 GMT
Update - The following update applies to services at the Pulsant TVHC facilities.

SSE are continuing to investigate a mains power disruption to the wider RG6 area and have provided an updated ETA for fix of 2359GMT. Pulsant's TVHC datacentres will continue to run on diesel power generation while we await the "all-clear" from SSE.

*There is no impact to any customer services and the sites are being supported by diesel power generation.*

We will provide a further update with more information within the next 60 minutes.

Should you have any questions, please raise a ticket by emailing [email protected] or by calling our support team on 0845 1199 999.
Nov 30, 21:37 GMT
Identified - The following notification applies to services at the Pulsant TVHC facilities.

We are currently investigating a mains power failure at this site. SSE are aware of the outage and have provided a current ETA for fix of 2045GMT.

*There is no impact to any customer services and the sites are being supported by diesel power generation.*

We will provide a further update with more information within the next 60 minutes.

Should you have any questions, please raise a ticket by emailing [email protected] or by calling our support team on 0845 1199 999.
Nov 30, 20:02 GMT
Dec 1, 2016
Completed - Please be aware of the cancellation of the previously advised planned maintenance window.

Pulsant's focus with any maintenance work we undertake is always to put customer service first, but equally consider risk mitigation.

Although Cadogan House has been unaffected, in light of the recent utility power outage incidents for Suttons Business Park on 30th November and the solution put in place by our power provider whilst they establish a more permanent fix, under our Risk Assessment and Management process we have decided to cancel the planned works until the New Year.

Due to the upcoming festive period and change freeze, the decision was made that we would be better to re-schedule these works in the New Year.
Once we have confirmation of the new revised dates we will re-submit a maintenance window when these changes will take place.

We apologise for any inconvenience caused, but maintaining a high level of customer service and risk mitigation is our priority.

A further notification will be sent out in the New Year once we have confirmation of the new proposed dates.
Dec 1, 13:52 GMT
Update - Please be aware of the postponement of the previously planned maintenance window.

Pulsant's focus with any maintenance work which we undertake is always to put customer service first, but equally the mitigation of risk.

Although Cadogan House was unaffected, we feel that it is prudent to postpone these works due to the recent instability in the mains power grid in and around Reading on Wednesday 23rd November, and to ensure stability from the local grid.

The previously proposed works have now been rescheduled for Friday 2nd December, starting at 0800GMT and we will continue to monitor and work with 3rd party providers leading up to this date. If anything changes, we will of course sent out further notifications.

The full, updated, notification is detailed below:

---------

WHAT
In keeping with the Pulsant Datacentre plant infrastructure capacity management and efficiency programme, we will be carrying out scheduled modifications to the generator backup configuration at our Cadogan House, Reading, datacentre.

SERVICE IMPACT
None expected - This work will be carried out on only one generator feed at any one time. This work is being carried out only on emergency backup cabling and will not affect the datacentre facilities during normal operation. However, due to the nature of this work and some customers having single corded supplies to their racks, power will be deemed "At Risk" for a portion of the maintenance window.

The works will be completed by specialist 3rd parties, supervised by Pulsant datacentre management throughout. The generator functionality will be fully tested prior to being returned to normal service and complete power protection will be restored upon completion of these works.

WHY
Pulsant is revising and downgrading the generator capacity required at the Cadogan House Datacentre.

Cadogan House has four generators delivering standby power to the site. Two legacy generator sets connected to the datacentre are being removed as part of our ongoing capacity management and efficiency strategy. This work is the first of two stages of work intended to allow us to decommission these legacy generator sets and remove them from service. After the conclusion of these works, the datacentre will remain fully supported by two modern diesel generators.

WHEN
Work will commence during the following maintenance window:

Stage One
Friday 2nd December 2016, starting at 0800GMT and concluding by 1800GMT on the same day

Stage Two
Saturday 10th December 2016, starting at 0800GMT and concluding by 1800GMT on the same day

The "At Risk" window will be kept to an absolute minimum and all power backup systems will be returned to full resilience at the earliest opportunity.

A completed notification will be sent out once the works have been completed and normal service has resumed.

If you have any questions, please raise a ticket by emailing [email protected] or by calling our support team directly.
Nov 24, 16:52 GMT
Scheduled - Dear Customer,

Please be aware of the following planned maintenance window:

WHAT
In keeping with the Pulsant Datacentre plant infrastructure capacity management and efficiency programme, we will be carrying out scheduled modifications to the generator backup configuration at our Cadogan House, Reading, datacentre.

SERVICE IMPACT
None expected - This work will be carried out on only one generator feed at any one time. This work is being carried out only on emergency backup cabling and will not affect the datacentre facilities during normal operation. However, due to the nature of this work and some customers having single corded supplies to their racks, power will be deemed "At Risk" for a portion of the maintenance window.

The works will be completed by specialist 3rd parties, supervised by Pulsant datacentre management throughout. The generator functionality will be fully tested prior to being returned to normal service and complete power protection will be restored upon completion of these works.

WHY
Pulsant is revising and downgrading the generator capacity required at the Cadogan House Datacentre.

Cadogan House has four generators delivering standby power to the site. Two legacy generator sets connected to the datacentre are being removed as part of our ongoing capacity management and efficiency strategy. This work is the first of two stages of work intended to allow us to decommission these legacy generator sets and remove them from service. After the conclusion of these works, the datacentre will remain fully supported by two modern diesel generators.

WHEN
Work will commence during the following maintenance window:

Stage One
• Friday 25th November 2016, starting at 0800GMT and concluding by 1800GMT on the same day

Stage Two
• Saturday 10th December 2016, starting at 0800GMT and concluding by 1800GMT on the same day

The "At Risk" window will be kept to an absolute minimum and all power backup systems will be returned to full resilience at the earliest opportunity.

A completed notification will be sent out once the works have been completed and normal service has resumed.

If you have any questions, please raise a ticket by emailing [email protected] or by calling our support team directly.

For your reference, please see our status page for updates on planned maintenance and service incidents. This can be found at http://status.pulsant.com

Regards,
Pulsant Support Team
0845 1199 999
Nov 11, 03:37 GMT
Nov 29, 2016
Completed - The scheduled maintenance has been completed.
Nov 29, 23:00 GMT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 29, 21:00 GMT
Scheduled - Please be aware of the following planned maintenance window:

WHAT
In keeping with the manufacturer's recommendations and the Pulsant planned and preventative maintenance regime, we will be carrying out a planned electronics board replacement on one of the UPSes at our Cadogan House, Reading, datacentre.

SERVICE IMPACT
None - Only one UPS will be worked on. However, due to the nature of this work and some customers having single corded supplies to their racks, power will be deemed "At Risk" for a portion of the maintenance window.

The maintenance will be completed by a specialist 3rd party and the UPS will be fully tested prior to being returned to normal service. Complete power protection will be restored upon completion of the UPS maintenance.

WHY
Pulsant follow a stringent Planned Preventative Maintenance regime. This work forms a major part of this regime.

During the recent planned UPS maintenance, it was identified that an electronics board on one of the UPS units serving Cadogan House was approaching end of life and would need to be replaced. Pulsant approved the replacement card and have arranged for it to be replaced as soon as practicable.

Pulsant's specialist UPS contractors will be on site to complete the scheduled maintenance, supervised by the Datacentre management team throughout.

WHEN
Work will commence during the following maintenance windows:

Tuesday 29th November 2016, starting at 2100GMT and ending by 2300GMT on the same day.

The "At Risk" window is anticipated to be less than one hour, and will be kept to an absolute minimum. The UPS system will be returned to full resilience at the earliest opportunity.

A completed notification will be sent out once the works have been completed and normal service has resumed.

If you have any questions, please raise a ticket by emailing [email protected] or by calling our support team directly.
Nov 22, 16:52 GMT
Completed - The scheduled maintenance has been completed.
Nov 29, 22:00 GMT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 29, 19:00 GMT
Scheduled - Please be aware of the following emergency maintenance window for essential UPS Maintenance at our Reading Datacentre facility – TVHC1 only.

WHAT
Following on from an alert received this morning on our monitoring system, we are aware of a fault with one of our UPS modules in our UPS array.
We have isolated the module and in line with the manufacturers’ recommendations we have authorised for our specialist UPS contractor to investigate the issue further and bring the module back on line. This is in line with our processes, and to maintain maximum efficiency of our infrastructure as soon as possible.

SERVICE IMPACT
NONE – All UPS arrays are still on line, but in each UPS unit there are multiple modules. One of the modules has gone into fault and has been isolated, which has resulted in reduced resilience in one UPS unit. There will be no planned service impact during the emergency maintenance, but it will required one full UPS unit to be isolated so that the module and unit can be investigated and the issue resolved.
This will mean that on the “A” feed there will be reduced UPS resilience, however, power will remain protected and unaffected to all our customer racks during these essential emergency works. Due to the element of electrical switching that needs to be undertaken, an Emergency "At Risk" notification is being sent out to inform our customers accordingly.
The essential maintenance will be completed by our specialist UPS contractor and the unit will be fully tested prior to returning into the array and returned to normal service.

WHY
The UPS contractor has requested that the UPS module in fault is isolated in order to fully investigate the fault, and check the other modules safely, prior to returning it into the array.

Pulsant's specialist UPS Contractor will be on site to complete the emergency maintenance, supervised by the datacentre management team throughout.

WHEN
Work is scheduled to be carried out during the following maintenance window:

Tuesday 29th November 2016, commencing at 19:00GMT and ending by 22:00GMT on Tuesday 29th November 2016.

The "At Risk" window will be kept to an absolute minimum and the UPS array will be returned to full resilience at the earliest opportunity.

A completed notification will be sent out once the works have been completed and normal service has resumed.
Nov 29, 15:54 GMT
Nov 27, 2016
Resolved - This issue has now been resolved.
Nov 27, 11:58 GMT
Update - Fault has been identified by SP and they are working to resolve this, ETA 11:30, Next update 12:00
Nov 27, 10:54 GMT
Update - Service has been restored
Nov 27, 05:44 GMT
Identified - Pulsant investigating shared webfarm issues.
Nov 27, 04:45 GMT
Nov 26, 2016
Resolved - This has now been completed.
Nov 26, 12:41 GMT
Identified - WHAT
As part of Pulsants planned maintenance regime, it is necessary for us to undertake an Annual Darken Building Test on site, to ensure all infrastructure works correctly in the event of a full loss of mains power.

SERVICE IMPACT
NONE - No interruption or disruption to service is expected during these essential works, however due to the level of switching that will be undertaken during executing our Darken Building Test procedure, we are issuing a courtesy At Risk notification to all our customers.

WHY
The Annual Darken Building Test forms part of our stringent maintenance regime and this is to ensure all infrastructure works correctly in the event of a full loss of mains power to site.
To do this we will be working with our 3rd party contractors and the electricity company to isolate the mains power to the site and perform essential testing of our HV and LV infrastructure.
During this period, it will be necessary to run the site on generators. Pulsant will follow our Darken Building Test procedure, which will isolate the incoming utility supply to site The testing will be completed by Pulsant staff and a number of specialized 3rd party contractors.

The following works will be carried out;-
1. Isolation of utility mains incomer to site.
2. Monitor failover to Generator and ensure supporting site (Power to Racks will remain UPS protected throughout).
3. Once testing period completed the power will be restored in a controlled manner.
5. Reinstatement and check that all services back to a normal state.
Monitoring will continue for a further 1hr post restoration of utility power to ensure there are no outstanding issues.

All diesel supplies to the site have been checked and confirmed adequate supplies, reserve supplies are on standby. Once works have been completed, the fuel supplies will be replenished accordingly.

We have provisioned a maintenance window of up to 8 hours to allow us an adequate time window to complete all the necessary tests. All personnel will remain on-site during this maintenance window and for a period of up to 1-hour post the maintenance being completed window.

Works are expected to be non-service affecting as customers will remain on UPS protected supplies throughout until all testing has been completed. This maintenance does involve changes to power source to the site, hence an At Risk notification is being sent out.
Pulsant has been preparing extensively for this scheduled Darken Building Test.

We will have a number of different contractors and Pulsant personnel on site including expert contractors to support our Generators, UPS devices, electrical infrastructure and most important Pulsant network and site support staff. We have worked with the electricity utility supplier to mitigate risk as far as possible, below is the planned approach with regards to completing all necessary works and undertaking this Annual Darken Building Test

WHEN
Pulsant confirm that the maintenance window is noted below and the associated At Risk period:

Saturday 26th of November 2016, commencing at 09.30 GMT and completed by 17.30GMT on Saturday 26th of November 2016 by.

To mitigate risk and impact, Pulsant have scheduled for these works to be carried outside normal working hours.

The At Risk window will be kept to an absolute minimum and the site will have utility mains power restored and resuming full site protection at the earliest opportunity.

Please accept our sincere apologies for any inconvenience this may cause.

Regards,
Pulsant Support Team
0845 1199 999
Nov 26, 08:51 GMT
Nov 25, 2016

No incidents reported.

Nov 24, 2016

No incidents reported.