I am sharing this story with you guys because I think there is a lot to learn from this case about the security and best practices or what is just enough in terms of security. If you are using any of the cloud platforms like AWS , GCP, Azure or another Cloud Vendor to support your Applications and have not really paid attention to the security aspect or feel that since you are using one of the well known Cloud Providers you are safe, this article might change how you approach this domain entirely.
Unless security is something you look at being an important piece of work or domain which you need to invest your time in or there is work done to ensure security is tested and a vulnerability assessment checklist is maintained and worked on, your system is still prone to breach.
Now what happened with us was a minor incident, but the way it happened made us wiser and more focused on security. The hacker gained the access to one of the AWS access key-secret pair from the environment file which was hosted on a web application, a developer had forgotten to put that file in gitignore and there was another developer who forgot to check for bad paths and excluding certain files from being served at all.
Still could be worse …
The DevOps team had been lazy and was using a single key for all the AWS resource API and this key was associated with an admin role. More so this key was also shared with various application services or micro-services to provide access to AWS resources and services.
We had One Key to Rule Them all … ( AWS resources )
Once he got that access he waited for the right moment and found out the IP of our productions instance as it was the public IP of our website. Now for the hacker to try and get access to our sensitive data he would require to either ssh into our instance, but trying to SSH into that instance will be futile as he would not have the key or pem file which is req by AWS.
But here is the most clever and simple trick he used yet no one thinks of this stuff or the fact that this will happen to them.
The hacker cloned our current EBS storage volume which was stored as a snapshot for backup purposes. Then went ahead to create a new VM instance, attached the volume. Since he created the VM instance from scratch thus a creating a new key pair, allowing him to SSH into the newly created VM instance.
This is where we could have been more active in setting alerts on the account activity and making sure we got active alerts for any new resource creation. This would have helped us to take note of new resources being created from a foreign location or at a very unusual time. There is a perfect solution for this which we will discuss soon.
Now the hacker had access to our virtual machine instance which had our code environment variables and all of our sensitive data. This included our database credentials and other sensitive information. He already had access to aws account. And now with all the credentials, he had all the ingredients to successfully reach into our database and do harm us.
And these events took place over several days, but none of the DevOps could spot the extra resources or keys which were created. And to the credit of hacker, the naming conventions he used were able to skip the negligent eye. So basically all the new resources like security groups and ec2 instances or keys created were named by twisting the names of existing resources so that these will not be spotted over just a glance.
For example, if a resource was named — launching-wizard5
the hacker named his security group lanuching-wizrad5.
This is where we learned another lesson for we had not given importance to the naming of some of the resources or did not have a clearly defined naming strategy for resources. If we did we would have been abler and have had more chance to spot abnormalities.
Alright so how did we eventually find him or what was it that we did right which allowed us to recognize something was happening albeit a bit too late …
Find out here…
Backlink – https://medium.com/@damitj07/we-got-hacked-86552e22197a