Plan, Test, Execute, Review or How to Handle Business Continuity Planning - Infinity Network Solutions

For those of you that read this from say California or Michigan on September the 11th Hurricane Irma passed through the state of Georgia.  It was predicted to be one of the most powerful storms to reach our state in quite a while.  We had wide spread loss of property, utilities and life due to the storm and as of the end of November I can still see debris and damage in our communities.  But I am positive it would have been much worse if not for the amazing technologies we have that provide forewarning of the storm and if not for the continuity and disaster planning and execution of many organizations.  Here at Infinity we not only have to plan for our own business but for and with all of our clients to ensure they can properly operate during and recover from events like this.  I will say I believe we were successful at a high level in executing on our plans.  But as with anything we can always learn.  To help with that about 2 weeks ago we held a post event review of our plan, our execution and what we learned so we will be better next time.  I wanted to share some of our findings here with you in hopes that it might help you with your Business Continuity Planning and Review.

The meeting we held had leadership from all three business units for us, Service, Sales, and Operations along with key team members who had responsibilities in the execution of our plans.  We are a team of high trust and embrace conflict  as Lencioni outlines in 5 Dysfunctions of a Team.  This allows us to not just talk about issues but reach the root of them and find ways to help fix them.  This can make for lively debate but always results in a collective understanding of the direction forward and willingness to see improvement.  From that meeting we identified 4 key areas; Technology, People, Process and Geography; that needed to be reviewed and addressed.

Regarding Technology; we have a large number of services and systems that remain critical to us during an event and some that become critical.  All of our main systems remained in operation during the event due to the fact that we didn’t loss power or Internet connection from our main site and our hosted services.  We were able to allow staff to “work in shelter” which amounts to either at home or alternate safe locations due to the remote access to our systems including soft-phones we use on our laptops.  This allowed our service team to continue to take support calls and work on systems that were impacted remotely the entire day.  It also allowed us to know which clients were impacted by the storm in real time so we could build response plans and provide updates on things like recovery of power and Internet at their facilities.  We identified in the review several what-ifs around power and backup Internet access for our facilities including the fact that our remote office did loss power though it did not impact function since it doesn’t host any services.  We took note to put in place further discussions around generators and redundant Internet access that could be improved beyond the current systems we have in place.  We also identified that LTE connections worked for most functions for those “working in shelter” though Skype didn’t work as well for meetings and calls we held during the storm that required more than text and audio.

On the People front we continue to know that everyone that is “customer facing” is critical.  We were able to leverage the “work in shelter” to ensure enough of those personal were available the entire time.  Sales worked as needed to support service and our clients but for the most part were determined to be non-critical as did operations as long as our main Auto Attendant and Support Queue remained operational.  If either of these would have failed we would have needed our operations team to work as front line call handlers.  In an extended event Sales and Operations being able to work remotely would have become critical to allow business continuation to happen.  We also saw how good communication between staff and clients allowed for continued support, updates of issues and outages and recovery plans.  Which shows our well everyone works together.

Process is the one area you can always improve as much because you don’t practice these events enough but also because we realized we made some decisions “in the hall” when there should have been a clear path and better command and control in place.  So we are working to ensure our process defines more of the details around who must work and who can work and who calls on added personal when needed.  Our process should also include the cancellation and/or rescheduling of meetings that aren’t critical in an event.  One part of the process that I think worked well was the rescheduling of on-site project work.  It would have been easy to overlook making those calls prior to the event to ensure everyone knew we would not be on-site for project work.  Also our communication process for updating clients about our plans and availability via the vCIOs worked well.  This BLOG post is an addition to our process based on the outcome of our review meeting.

Geography is a new item for us since we expanded to a second location in the past several years.  It makes us aware that with each geography the timing and impact of events like this will vary and therefore their needs and expectations will also.  As we continue to add and change staffing we will likely further spread our team members and skills around our locations to ensure we can meet our client expectations.  Also we found that service providers and utilities in different areas had different process and plans for both information and recovery which required us to work to bring that information into one context for our actions, updates and responses.

So again overall I think we were successful with execution of our plans but we will update them with our findings, improve them both now and over time and will continue to practice.  We also still believe our greatest threats are short term weather and health related items which we believe are well addressed.  So I hope as you review your organization’s impact from both this and other events you will take the time to consider these key areas and others so that you can not only weather them but reduce their impact on you.  What have you done and what else are you doing that I didn’t mention?  I always love hearing from and learning from you too.