Date / time of Incident - 15/06/2020 10:50 AM AEST

Date / time of Restoration - 15/06/2020 1:39 PM AEST

Duration of Incident - 2 hours, 49 minutes

Impact:

On the 15th of June at 10:50 AEST Aatrox Support began receiving reports regarding customers unable to make or receive calls. The issue was declared a major incident and our upstream engineering teams were engaged.

On investigation packet loss was identified on traffic into our upstreams Core Session Border Controllers (SBC’s) located in Sydney and Melbourne. Our upstream engineering teams engaged with Vendor support and while trying to failover network elements normal service operation this was not restored as expected. VRRP priority within the core environment was then reverted to its original state at which point normal operation was restored.

Root Cause and Resolution:

Based on the investigation to date the initial cause of the fault appears to be a brief interruption in connectivity between core infrastructure. This triggered a flood of signalling traffic and resulted in infrastructure operating in a degraded state. Issue was resolved after toggling Virtual Router Redundancy Protocol (VRRP) configuration within the network routing environment.

Actions to mitigate reoccurrence:

Investigation ongoing with our upstreams NOC and vendor support to understand the nature of the fault and steps required to mitigate against future failures. In the interim the impacted environment is under close observation. Monitoring of environment being reviewed to identify if it’s possible identify issues of this nature and further redundancy options being implemented.