I have been working since 7th August, close to 5 months as of today and recently did my first ever DevOops. The tl;dr can be found below:
The 5 W’s + a bonus H
Released 2 EIPs.
Morning after some coffee.
Code did not reflect what was deployed in a new AWS account.
Manually released them.
Slightly longer explanation
We had just been created a brand new AWS account to use for “nonprod” and I was going over code which created very specific VPC’s with the engineer who wrote the code. During that session we noticed that there were more EIPs in the new account than there should be, everything so far had been created using the code we were reviewing.. so we thought. Towards the end of the day an email was sent out asking why 2 EIPs were missing from a list of EIPs I had never seen before. After a few apology emails had been passed around it turned out that someone else had reserved the EIPs for future use; it was due to ‘business processes’ rather then technical reasons.
Later I was told that EIPs were like ‘Critical Infrastructure’, I was actually surprised that EIPs did not support some kind of tagging or release protection and when I googled around I found this:
Feature request: tags for elastic ip addresses
Posted by: technz
Posted on: Jun 6, 2012 4:34 PM
It was more than 5 years ago and there was still no tagging for EIPs..
I came across this tweet which made me feel warm on the inside.
This would have been nice to have a few weeks ago.
Another thing that I hope for in the future is to have the equivalent of EC2 termination protection on EIPs. This seems to be a feature that is become more common as Network Load Balancers have it and they are pretty new.
What to take away from this?
- We need better management and documentation of critical resources.
- Question everything, question everyone.
- You now have a chance to recover EIPs!
From the AWS documentation there looks to be a new feature which allows you to recover released EIPs under a few conditions.
To lose is to improve.
Remember to #hugops