Outage Report - Authentication Service

Timeframe

2015-12-09 14:12pm - 15:00pm UTC.

Impact

• Customers weren't able to login.
• Customers weren't able to use their DEV@cloud masters.
• Builds weren't started.

Root Cause

An erroneous config file was deployed to production. The authentication service went down, because the application failed to validate the new config file. Syntactically the config was ok, but it had a non-existent DNS entry which caused the validation failure.

Data Loss / Security Implications

There are no known data loss or security implications for DEV@cloud customers.

Followup
Stop using dynamically generated DNS entries from EC2 instance tags.

Global Restart for Jenkins 1.609.4.6

Timeframe

Vulnerability public @ November 9th 2015

Vulnerability fixed @ November 6th 2015

Impact

• Undisclosed at this time

You may restart your Jenkins to be upgraded to 1.609.4.6 immediately. We will automatically restart your Jenkins in the next 48 hours if you have not done so already.

Root Cause

Undisclosed at this time.

There is one DEV@cloud relevant high-severity vulnerability being patched in 1.609.4.6

Further information will not be provided at the current time.

Data Loss / Security Implications

During the global restart, jobs that are queued (but not building) are lost.  We are internally tracking a fix for this issue, however it will not be in place for this release.

There are no other known data loss or security implications for DEV@cloud customers.


Followup

Full information on the security vulnerability will be made available when the Jenkins team publicly announces the list of vulnerabilities included in the security release.