Timeframe
5am-7am 17th October 2015 (UTC)
Impact
- New builds unable to launch during outage
Root Cause
Dynamic DNS entries for the CloudBees build system were not updated correctly, taking the build service offline.
Data Loss / Security Implications
There was no data loss or security impacts from this outage.
Followup
- The impacted DNS update tools are being reviewed to determine the root cause of the faulty updates.
- The problem is intermittent and ongoing - however the system is being run manually (with verification) to maintain availability.
