Bob Mackin
After censoring documents about its Victoria Day technology crash, BC Ferries said Aug. 2 that the problem was preventable.
James Tan, the July-hired chief information officer, told reporters during a news conference about B.C. Day long weekend plans that steps are underway to prevent a repeat of the May 22 website, app and call-centre outage.
BC Ferries James Tan (LinkedIn)
“On the May long weekend, the root cause of the issue that caused the outage across multiple systems was the main storage at the server level in our Kamloops data centre reached a threshold unexpectedly and basically ran out of space, and because of that, it caused multiple systems then to come down and fail,” Tan said.
Tan said BC Ferries is increasing storage and is confident there will be no repeat of the Victoria Day outage.
“As we were investigating the root causes of what should have been done proactively, [we] made sure that we’ve got monitoring thresholds so that we’re actually catching these unexpected spikes in usage of the systems much earlier than what we had on the May long weekend,” Tan said.
BC Ferries charged a $10 application fee and an additional $37.50 for the internal email and briefing notes. BC Ferries heavily censored the documents because it felt they contained policy advice or recommendations and it feared disclosure would harm computer and communications systems and public safety.
According to the timeline, the problem on Victoria Day began before 5:30 a.m. and was not fully resolved until almost 3 p.m. CEO Nicolas Jimenez was provided a script the next day that said staff were “undertaking a deep dive to determine if the problem could have been avoided or dealt with earlier” and “undertaking broader work to revisit our technical environment to make sure it’s resilient for our business needs. That’s not a short-term fix.”
At Wednesday’s news conference in BC Ferries’ Victoria headquarters, Jimenez said that the website will have a virtual waiting room for people making reservations and checking current sailing conditions.
BC Ferries CEO Nicolas Jimenez (BC Ferries)
“So when there’s a surge of demand on the system, it doesn’t bring down the whole system, but we queue people up in an orderly fashion,” Jimenez said.
Jimenez said a longer-term solution to the multiple-sailing delays that have hampered the system this summer is to find more-efficient ways of bringing passengers and vehicles into the terminals, through the ticket booths and onboard the ferries.
“That includes the data that will inform what is going on in that particular sailing so that we can provide better real time information to customers,” Jimenez said. “So we’ve got a project currently before the [B.C. Ferry] commissioner in order to do that work.”
Tan worked at ICBC under Jimenez, most-recently as vice-president of claims customer and material damage services. He was also asked whether BC Ferries had assessed its cybersecurity risks and if it is prepared for a potential attack.
He said there is a multiple layer defence system, “both system-driven, as well we have humans actually going in and checking alerts as they come on board.” Tan said BC Ferries also relies on external expert support and is in constant communication with cybersecurity authorities.
BC Ferries is urging passengers to book ahead and, if possible, take public transit to terminals and walk-on sailings during the B.C. Day long weekend, traditionally the busiest weekend of the year.
Support theBreaker.news for as low as $2 a month on Patreon. Find out how. Click here.
