Timeframe
2015-12-09 14:12pm - 15:00pm UTC.
Impact
- Customers weren't able to login.
- Customers weren't able to use their DEV@cloud masters.
- Builds weren't started.
Root Cause
An erroneous config file was deployed to production. The authentication service went down, because the application failed to validate the new config file. Syntactically the config was ok, but it had a non-existent DNS entry which caused the validation failure.
Data Loss / Security Implications
There are no known data loss or security implications for DEV@cloud customers.
Followup
Stop using dynamically generated DNS entries from EC2 instance tags.
