In 1947, two Wichita, Kansas business partners opened a small auto glass shop. Forty-five years later, the single shop had expanded into a major corporation, with over 1 million windshields manufactured. Today, Safelite Group, Inc. employs almost 6,000 people around the U.S. and operates three major business units: automotive glass fulfillment services, windshield manufacturing, and fleet and insurance claims management services.
Safelite depends on literally hundreds of electronic B2B and POS applications. To keep these operations running efficiently, a network monitoring system constantly oversees this immense technology infrastructure. However, the system did not include satisfactory wireless notification of sudden anomalies in the network — a critical feature for a company that depends on its business applications to run uninterrupted, 24 hours a day.
Need
Approximately 17 technical support groups manage Safelite’s network infrastructure. Team members of the technical support groups are divided by specialty, such as the database administration team, the systems support team, or the UNIX team, and so on. Each team member on call must respond quickly to after-hours alerts generated by the network monitoring system which are sent to his or her smartphone. Should an alert go unaddressed, problems within the network can quickly accelerate.
But the native messaging capabilities within Safelite’s network monitoring system simply could not guarantee every alert was received by its personnel. What Safelite needed was a foolproof messaging solution in place that assured critical alerts were actually reaching on-call staff.
Problem
Ted Meisky, Safelite’s manager of open systems, was tasked with getting a more reliable message delivery system off the ground. He recalled the limitations that the company’s network monitoring system imposed on alert notification of network outages and application failures.
“Its primary messaging protocol was email and, therefore, unreliable in the event of a network service outage. There was also no escalation and no scheduling capability — two key features needed for managing twenty different groups responsible at any given time for after hours, on-call duties,” Ted said.
Some wireless messaging within the network monitoring system was possible, but required a tedious configuration process.
“Each support group was writing their own messaging programs to integrate with the network monitoring system. They had to include individual email addresses in the code for each team member’s wireless device,” Ted added.
“That meant every time we switched carriers, dozens of lines of code would have to be rewritten for each program,” Ted explained. “Inevitably, a line of code would be missed, and so would an alert.”
In a nutshell, Safelite’s technical support groups were each on their own in regard to alert notification distribution and response.
“Actually, we preferred that our technical support teams retain control over their message scheduling, but we also needed a centralized, uniform response solution in place,” Ted said. “Without one, there was no way we could guarantee consistently responsive action when problems occurred within the network. The trick was to find a solution that could centralize, yet give individual groups autonomy over scheduling.”
Eventually, additional concerns led Safelite to replace their network monitoring system with the open source Nagios. However, Nagios’ messaging capabilities were just as limited as the previous system. Safelite still needed to deploy an advanced wireless messaging solution — one that would provide full message scheduling and escalation capabilities, and easily integrate with Nagios.
Solution
Safelite evaluated several wireless messaging solutions, which were quickly eliminated when it was revealed that Safelite would need to rigidly conform to the solution’s specifications — instead of the other way around. That left one standout candidate: HipLink Software’s wireless messaging software.
“I was immediately impressed with HipLink’s extremely robust group scheduling and escalation — important features we particularly needed,” Ted explained.
Safelite was able to easily integrate HipLink with Nagios. Once installed, HipLink was used to assign primary and secondary escalation of network alerts to 80 receivers, utilizing email, dial up, and wireless messaging protocols.
What’s more, HipLink’s command line interface (CLI) feature allowed Safelite to write an additional front-end application that gave the individual support groups more control over their specific scheduling requirements — for example, overriding a team member’s on-call status. This was an extremely helpful feature in a message scheduling system for 80 receivers in 17 different groups, where at any given time, schedule changes can and do occur.
“HipLink centralized our mission-critical alerts, without sacrificing the flexibility to apply varying alert rules and schedules within individual response groups,” Ted added.