Forced outage down Clanwilliam Way
Five weeks after a power failure had forced an evacuation of Clanwilliam House, the ESB Customer Supply and ESB Networks occupants of the building made their way to the Fire Assembly Point on the morning of Tuesday December 4th.
This time the evacuation was a scheduled drill, but those taking part could hardly have imagined what was to transpire later that afternoon! Yes, another power failure occurred just before 4pm and they were back again to the Assembly Point - deja vu! Alas, this time the problem was more serious with Marine House also affected. As in October, a number of management personnel stayed onsite to receive situation updates and to put plans in place for all eventualities.
Contacts between the landlord, who has responsibility for maintenance of the building, Facilities, ESB Networks, ESB Customer Supply and the Corporate Business Continuity Office continued throughout the evening, as it became clear that power would not be available for at least 12 hours and maybe 24.
A decision to invoke the hot-site was made and HP, the vendor, was contacted. Emergency procedures were activated by ITS and Telecoms. This was new territory: the hot-site had never previously been invoked by ESB since it was contractually established in the late 1980s, now all of the contingency planning and rehearsals would be put to the test. A Facilities Manager was assigned to the role of 'Hot-site Manager' and went on duty in Swords at 8am. ICT Group staff were deployed to Swords, supported by colleagues in HO and by HP and commenced system recovery. By 3.30am, 30 PCs were restored to working order and connectivity was established to the corporate network, with tests confirming that all was in order. Business staff had been advised to report to Clanwilliam in the morning.
Work on replacing the burnt circuit board resumed at 7am. Luckily a replacement board was already on site - as normally a seven to 10 day delivery period would apply. By early morning, Shared Services had sourced a number of short-term accommodation solutions within ESB. Some 30 people were directed to the hot-site, while others were accommodated in Fleet Street, Leopardstown and Finglas. Corresponding telephone extensions were identified and re-routed to mobile numbers. All displaced staff were back in business by mid-morning, helped greatly by the IT desktop ('MyDesktop') that ITS had rolled out to most locations in 2007.
Those using the hot-site for the first time were impressed with the facilities. While some known IT constraints applied, no significant problems were reported by people working there, some up to 4.30pm.
By mid-afternoon, power was restored to all floors of Clanwilliam House and the green light was given. This was preceded by the disconnection of PCs and other electrical equipment so as to avoid electrical damage, which would have created a need for timeconsuming repairs. Normal occupation of the building resumed on Thursday, December 6th.
A project to improve Business Continuity across ESB Group was undertaken in 2007. It produced a management process for handling major incidents in any part of ESB. It also clarified arrangements for emergency accommodation, IT contingency and incident communication. The Clanwilliam incident served as an early and useful test of this process and provides confidence that the procedures in place are appropriate.
Good communications across all parties is vital in managing an incident and, thankfully, this was the case in December. Another important enabler of the prompt and successful response was the streamlined IT desktop that is now used by most ESB staff. Finally, a special word of thanks to the many people who responded to the emergency. Their prompt and committed response, including unexpected night-time working, ensured that the business impacts were minimised and that a generally smooth outcome was achieved. Well done to all concerned.
