Network instability incident at CMC Datacenter

vHost's monitoring system detected that the network at CMC Datacenter was unstable.

• Incident Time: 06:02, March 25, 2026 (GMT+7, Vietnam Timezone)


• Resolved Time: 07:43, March 25, 2026 (GMT+7, Vietnam Timezone)


• Preliminary Cause: Our technical team is currently investigating and will provide updates as soon as possible.


• Impact Scope: Network at CMC Datacenter was unstable.


• Current Status: The network has returned to stable operation.

After thoroughly investigating the root cause and implementing remediation actions, vHost would like to report the incident details as follows:

• Incident Severity: P1


• Impact: The entire network of servers located at CMC Datacenter was inaccessible.


• Root Cause: The firewall system performed an auto update at 06:00 AM on March 25, 2026. During the OS update process, the device reloaded the network service, which resulted in the loss of the Default Route (0.0.0.0/0) on the router connected to downstream servers. The system then automatically switched to a backup router located at FPT to take over operations. However, most IP ranges had not yet been advertised by FPT, leading to connectivity issues where many ISPs could not access servers hosted at CMC Datacenter.
  


• Permanent Resolution:
  
  
  
  • Disable auto update on the firewall device.
  
  
  • Coordinate with FPT to advertise CMC IP ranges to other ISPs.
  
  
  • Deploy a backup firewall device at CMC Datacenter.
  
  
  
  

  Thank you for your understanding.

  
  Best regards,