26
loading...
AWS CloudFormation UPDATE_ROLLBACK_FAILED fix in production
Recently, while trying to deploy a serverless application, the pipeline failed thus putting the cloud formation stack in "UPDATE_ROLLBACK_FAILED" state. This is due to an error in your pipeline. In my own case, I was trying to use a base layer - "arn:aws:lambda:us-east-1:770693421928:layer:klayers-python38-pandas:35" which at the point of writing this article does not exist.
If you are familiar with AWS SAM Cli or cloud formation generally, you will know that you wont be able to deploy a new update until this status changes to "UPDATE_COMPLETE".
Now, you have two options facing this situation (at least that I am aware of):
The second option is almost perfect and simple except for the fact that you might not want to do this in production has is basically shutting down your application.
The first option however is much more advisable. Now, how do you complete the update rollback.
You can make use of the aws cloudformation continue-update-rollback command to complete your update rollback.
aws cloudformation continue-update-rollback \
--stack-name STACK_NAME \
--resources-to-skip LIST_OF_RESOURCESWhere STACK_NAME is the name of your stack while LIST_OF_RESOURCES is the list of the logical IDs of resources you will like to skip. Please note that for LIST_OF_RESOURCES you have to specify resources that are in the UPDATE_FAILED state only.
Sample
aws cloudformation continue-update-rollback \
--stack-name eazido-app-stack \
--resources-to-skip CustomerApi PaymentApiOnce this is done, your stack should now carry the "** UPDATE_ROLLBACK_COMPLETE**" status. If you try to deploy your updates again, it should work just fine.
However, remember how the update failed because of an error, you will need to identify and fix this before you deploy. In my own case, I was using a layer whose version does not exist. I had to update the layer by updating the stack template (we will talk about this in a bit). You can tell why your stack failed by checking the Status reason column under the Events tab of the failed stack.
Once the "View in designer"'s page loads, you should see a page similar to this.
You can get the template for a particular stack using AWS CLI's (get-template)[https://docs.aws.amazon.com/cli/latest/reference/cloudformation/get-template.html] command.
aws cloudformation get-template --stack-name STACK_NAMEUnder "Stack Actions" select "Create change set for current stack". You should see a page like this
Click on "Replace current template", select a template source (local or s3) based on where you saved the edited template.
Review the change set and Execute. This is will starts updating the AWS cloud formation stack and you can see the progress on the event tab.
You can update a particular stack using AWS CLI's (update-stack)[https://docs.aws.amazon.com/cli/latest/reference/cloudformation/update-stack.html] command.
aws cloudformation update-stack --stack-name STACK_NAME --template-url https://s3.amazonaws.com/sample/updated_template.templateNote: If you are getting AWS cli errors, you might want to take a look at the - AWS cli troubleshooting guide.
26