Vodacom’s embarrassing data network outage in South Africa

Vodacom has dismissed speculation that poor network design and configuration resulted in many subscribers being unable to access locally hosted websites last Thursday.

However, it has not provided a sound explanation for why its data network was offline for many subscribers while the networks of its South African rivals remained available.

This comes after Vodacom subscribers around South Africa could not access the Internet for several hours following a suspected submarine landslide that knocked out four undersea cables.

The incident happened off the coast of Cote d’Ivoire, near Abidjan, breaking the West Africa Cable System (WACS), Africa Coast to Europe (ACE), MainOne, and SAT–3.

Due to the location and depth of the breaks, the cable owners have ruled out sabotage, stray boat anchors, and other human activity as possible causes for the outages. This leaves a submarine landslide as the most likely explanation.

What was strange about last Thursday’s outage was that Vodacom customers could not even access locally-hosted sites.

Microsoft’s locally-hosted Azure cloud service was similarly impacted, with companies like payments provider Yoco unable to function due to the downtime.

This raises questions about how these networks were configured and why they required international Internet access for local services to keep working.

At the time of the outage, Vodacom said only “certain customers” were experiencing “intermittent connectivity issues”.

MyBroadband asked Vodacom which of its subscribers were impacted.

“Last week Thursday was an extreme case where there were multiple undersea cable breaks. SAT–3 failed during the morning and WACS after mid-day,” a Vodacom spokesperson said.

“At the time when WACS failed, Vodacom had various network links in place between South Africa, various parts of Africa and Europe.”

Vodacom said this included connectivity on Google’s Equiano cable from South Africa to Europe.

“Due to the dimensions of the network links from Africa to Europe having different capacities, it resulted in congestion on some of the links, resulting in certain customers experiencing an intermittent service,” stated Vodacom.

“Nonetheless, we implemented capacity upgrades on cable systems that were not impacted which means that normal service to our customers was restored on Thursday evening.”

MyBroadband also asked Vodacom why affected subscribers couldn’t connect to local sites.

“Some local sites would have gone down because they host internationally, so [they] will have been affected by the undersea cable breaks as well,” Vodacom said.

“Normal service was restored once we implemented capacity upgrades on cable systems that were not impacted by the breaks.”

This explanation is extremely strange, as MyBroadband conducted tests during the outage from locations in Gauteng and the Western Cape using at least six unique devices, each with its own SIM.

Our experiences were echoed by countless people on social media — those with an alternative Internet connection who could complain online.

Based on our tests, there was a period of around two hours during which there was no throughput on Vodacom’s data network.

We could not reach our websites or speed test servers, which are both hosted in South Africa. Our speed test service is also hosted in a different data centre from our websites.

Although the websites are behind Cloudflare, it has nodes in Johannesburg, Cape Town, and Durban.

Switching to a different network operator allowed us to connect, indicating there was something specific about the Vodacom network that had been impacted that did not affect MTN or Telkom.

To rule out a wider network outage, we also placed a call from the Western Cape to a number on an MTN-based MVNO in Gauteng.

The call connected, and both parties could hear each other clearly, suggesting that the issue on Thursday was limited to Vodacom’s data network.