TRANSCRIPT
Last week's Optus outage put large swathes of Australian life out of action for nearly 12 hours.
In return, the company's Chief Executive has endured an uncomfortable two hours.
That's how long Kelly Bayer Rosmarin was facing a Senate inquiry into the issue, getting questions from senators from across the political spectrum.
The nationwide data outage on the 8th of November impacted phones, public transport, EFTPOS machines, and government agencies like Border Force.
Ms Bayer Rosmarin admitted the obvious.
"As a business, we are nothing without our customers, who put their faith in us. And it is indisputable that on that day, our performance was not acceptable. We let you down. And, for that, I am deeply sorry."
The November 8 failure affected nearly ten million people.
It was the result of a routine upgrade overwhelming Optus’ system.
At 4:05am eastern time, 90 Optus routers went down.
Engineers were scrambling for a fix within half an hour.
By 7:35am, Ms Bayer Rosmarin was on site at Optus' Sydney headquarters.
At 8:30am, she called federal Communications Minister Michelle Rowland to inform her what was going on.
Optus technicians tried reversing the upgrade.
It didn't fix the problem.
Ms Bayer Rosmarin says Optus technicians commenced a hard reboot of their servers, with coverage starting to come back online at 10:40am.
"This commenced at around 10:30am, with the vast majority of customers recovered by 2pm, and 99 per cent by 4pm. The actions we took were a brute force resuscitation of the network. We had not yet identified the cause of the issue."
Around 10:40am was also only when the general public first heard from Ms Bayer Rosmarin - a fact seized upon by Greens Senator Sarah Hanson-Young.
"But what people were saying was that they weren't hearing from you. They weren't hearing from you. That is what was being reported. That is not reassuring to the community at all."
Ms Bayer Rosmarin defended herself against that.
"It's actually unusual for a CEO to appear at all during an outage because the public would expect my focus is on working with the teams to resolve the issue."
Optus' Managing Director of Networks, Lambo Kanagaratnam, appeared with the Chief Executive at the hearing.
He's admitted the company was unprepared for a failure this big.
"We didn't have a plan in place for that specific scale of outage. It was unexpected. We have high levels of redundancy and it's not something that we expect to happen. It is certainly something that we commit to learning from this outage. We will take such exercises into consideration in the future."
Optus' parents company, Singtel, has distanced itself from the failure.
Its software update to Optus routers was deployed hours before the failure, but Ms Bayer Rosmarin admits the failure came from the routers.
"The trigger event that led to that was the upgrade on the Singtel international peering network. We had put out the statement, which then got interpreted by various commentators as being that the root cause was the Singtel upgrade. The trigger was the Singtel upgrade but the root cause was the routers."
One of the most emotional aspects of the outage was a literal life or death proposition - people not being able to call triple-zero.
228 triple-zero calls failed to go through during the outage.
That's despite the fact Australian law facilitates a phone connected to one network being able to use another telco's network to call triple zero.
Ms Bayer Rosmarin says Optus doesn't know why the calls failed to go through, saying Optus participates in the triple-zero system, but doesn't run it.
Senator Hanson-Young was not buying that.
"But it's not anybody else’s fault that your customers couldn't call triple zero. Surely it's Optus' fault? So isn’t Optus that should pay the fine?"
On the matter of fines, and all things financial resulting from this fiasco, Optus has been contacted by 8,500 customers and business for compensation.
They've demanded a combined $430,000.
So far, Optus has paid $36,000, but Ms Bayer Rosmarin couldn't tell the Senate whether that's been paid out in cash or services.
She's defending the company's decision to offer extra data as compensation to affected customers.
"It's in everybody's interest that we keep the networks up and running. And we are doing what we can to show our customers that we care, that we apologise, that we appreciate their patience and their loyalty."
It was another offering not impressing Senator Hanson-Young.
"Have you been surprised at the frustration that your customers have expressed at that data offer? I mean, I haven't anyone who's said, oh great, thanks Optus!"
Ms Bayer Rosmarin dodged questions about her future, after a report in the Australian Financial Review newspaper that she may look to depart in the next 12 months.
The Senate hearing has adjourned for now; Optus' uphill fight in the court of public opinion still has a long way to run.
