We found earlier through our internal monitoring that our docker service status checks were not running after our restart of the docker service in our Scheduled Maintenance this morning. After investigating the issue it became clear that docker was deadlocked and it affected all containers. After rebooting the Container Manager server we were able to unpause builds, and new builds are now building again at this time.
We have monitored this issue for a some time and feel that it has been resolved at this point. We are under the impression that we spun up the paused builds too fast when we restarted the Container Manager service after our Scheduled Maintenance that ended early this afternoon causing the docker service to deadlock. Ultimately we had to reboot the Container Manager to resolve the issue.
We have restarted some project builds that we were able to identify that did not restart properly or never started while we had the issue with docker. If your build is showing "Invalid Content", or steps haven't run at all, please click the Rebuild link on your build to rebuild it, and this should bring your build back online.
We apologize for any inconvenience that this outage may have caused!
Posted Nov 29, 2017 - 18:59 EST
Monitoring
Docker is back online. Builds can be viewed and created again. We're monitoring the situation to see if there are any other issues.
Posted Nov 29, 2017 - 17:00 EST
Investigating
This issue is causing new builds to fail and old builds will not be available at this time.