Oct 16 2017 911 Outage Narrowly Averted Due To A Next Generation 911 Solution
(Photo: Courtesy of Baltimore County Fire Department)
With all of the 911 outages we have seen over the past few months, finally we hear a story about a Next Generation 911 (NG911) solution that worked as designed and allowed emergency calls to be handled in the midst of a potential disaster.
Baltimore County faced a potentially major disaster on Thursday (10/12/2016) morning where employees were forced to evacuate their primary 911 center. The Primary 911 Center for Baltimore County is located in the basement of the Baltimore County Circuit Courthouse which was evacuated for four hours due to an accidentally split gas main by Baltimore Gas & Electric (BG&E). The split line was fixed and the building was reopened at 1:30PM, but staff and anyone else there for hearings or other business did not return for several more hours.
Good Design Led to Quick Recovery
Baltimore County was fortunate because the design of their NG911 solution allowed them to take calls at their alternate 911 in nearby Hunt Valley, MD. Staff simply relocated to the alternate center, just 10 miles down the road and handled every 911 call without missing a beat.
The ability to handle this disaster was due to the multiple levels of redundancy built into the Baltimore County 911 center. First, the Baltimore County 911 center is setup as a geo-separated, “active-active”, NG911 solution, which simply means that the center is distributed across two geographically separated sites where staff can handle calls at either site. So, while the primary site was evacuated, the equipment at the primary site was still functioning and routing calls to available call takers at the alternate site.
Even if the primary site had lost power or had a massive equipment failure, the alternate site would immediately accept the inbound emergency calls and route the calls to available call takers at either site.
Redundant Redundancy for Added Reliability!
But, the redundancy that Baltimore County 911 leverages doesn’t stop there. When Baltimore County upgraded their E-911 solution to an NG911 solution in 2011, they opted to preserve their old legacy E-911 equipment. That legacy equipment included a carrier-grade TDM telephone switch with VoIP capabilities. So, in the unlikely event that their NG911 system were to completely fail, Baltimore County has the ability to simply revert to their old E-911 system with a single push of a button. To ensure this capability is always at the ready, Baltimore County tests the backup solution once a month by switching all calls to the legacy system for a period of two hours and handling live emergency calls.
And, even if the entire NG911 and E-911 solutions came to their knees, Baltimore County always has the option to handle calls the old fashion way – via analog phones.
Lessons Learned from Past Failures
The handling of this potentially disastrous situation by Baltimore County’s 911 center is in stark contrast to the 911 outages recently experienced in nearby Montgomery County, MD, Baltimore City, MD and Washington, DC.
In the case of Montgomery County, MD, the outage was caused by a malfunctioning HVAC system that resulted in a power outage to the backup system. The backup system was being used at the time because the primary center was undergoing a major upgrade that was scheduled to take more than 6 weeks. Baltimore County’s software-based solution eliminates long upgrade times because most upgrades are done as software updates.
The City of Baltimore experienced an outage due to the misrouting of 911 calls to an unmanned backup center. Baltimore County’s 911 utilizes Automatic Call Distribution (ACD) software to only route calls to manned stations at either of the 911 sites. In addition, ACD rollover logic is built-in to the solution to route calls to Non-Emergency call takers in case no Emergency call takers are available.
Washington, DC’s 911 outage was due to human error when a contractor inadvertently pulled an emergency power shut off switch that cut electricity to the 911 phone system and a call routing system at the District’s Unified Communications Center. In order for a similar failure to occur at Baltimore County, a power shutdown would be required at both sites simultaneously to shut down both 911 centers taking calls.
Credit Where Credit is Due
While the superior technology at Baltimore County can be credited for why Baltimore County survived a potentially disastrous situation when nearby jurisdictions did not, the true credit really needs to go to the County officials and visionaries that had the foresight to invest in NG911 technology early and implement it in a fault-tolerant and reliable fashion. Recognizing the importance of 911 services is the first step and then making the commitment to implement what’s right the first time is the next step to foolproof 911.