Hello luvs, no matter how hard I try, it seems not entirely possible to write as much I love to. I dream of creating live streams, podcasts, more open-source software, and participating in more exciting startups. Yet as a living creature, I need to be happy, stay healthy (workout, diet), maintain healthy relationships, excel at my career and studies, and do my best to be kind and decent. however, in this crazy 2020, trying to stay sane, happy, and survive looks like a real challenge to me already. xD
Alright, let's jump to our today's topic.
While working on HunterSuite, and as part of the job, j3ssie and I are always thinking of something we can automate to make black-box security testing easier. We discussed this idea of creating a multiple platform cloud brute-force hunter.mainly to find open buckets, apps, and databases hosted on the clouds and possibly app behind proxy servers.
In this writeup, I will show you our approach to researching the critical part of today's internet infrastructures, also known as clouds.
We look into how data and apps are placed on the clouds, so you know how to deal with them during manual security tests. As current top companies and newer to come are likely to use one of these cloud providers as a security tester, it's almost essential to at least know how they usually work. We wrap it up by automating the task and building a tool to find a company (target) infrastructure, files, and apps on the top cloud providers (Amazon, Google, Microsoft, DigitalOcean, Alibaba, Vultr, Linode). The outcome is useful for bug bounty hunters, red teamers, and penetration testers alike.
If you are only interested in the tool, the repo is here.
Here is are examples of reports of issues you can find using the provided tool and methodologies.
Researching the previous approaches
I started researching previous tools after a while searching on GitHub and playing around with tags and topics. I found a nice repo which lists a lot of similar projects. Still, some updated years ago, and a year is a long time in the modern era, so I had to google around more recent creations. Finally, I come up with a list of projects to understand the community approach to date better.
Luckily, reading these mostly small scripts (because the task is not significant) wasn't that roublesome.
Things we loved :
Things we didn't love:
Please note some of the projects only lack a few points and doing a reasonably good job already.
To overcome these issues, we come up with some solutions.
we combined all and sorted previous wordlists, added some fresh ideas, created an account on each modern cloud provider.
Finally, we analyzed their APIs, URLs, Apps, structure, and regions to ensure I have the most updated current version.
Now we have the basic idea, Le'ts talk about structures.
These big cloud providers have data centers in many regions and counties. You may think of why? Because they can provide better service to the customers based on their location. For instance, the nearest datacenter to you. Now for security testing, these regions are attractive because they appear in URLs and endpoints. As a tester, when you try to find open access storage (bucket) or an API app behind the proxy, you can use these regions to find more accurate results.
How these regions related to each URL and endpoint depend on the provider. Some providers make things a lot easier like google.
for example, when you deploy an app to the google app engine, you will see something like this
codeBucket: staging.cloudplaygrouond.appspot.com ..... defaultBucket: cloudplaygrouond.appspot.com defaultHostname: cloudplaygrouond.el.r.appspot.com .....
In this case, el.r is the region, but it doesn't matter because we can access the app with name + appspot.com. In other words, name + appspot.com = name + "region" + appstop.com.
But in the case of AmazonAWS apps, when they aren't on appname.s3.amazonaws.com They the URL is only valid when the region is there. Or, app.amazonaws.com != app.region.amazonaws.com and a valid app URL will be appname+seperator+us-east-1.amazon.app
One last example is azure again in the case of cloudapp.azure.com
the region used as part of the URL so app.region.cloudapp.azure.com != app.cloudapp.azure.com
Now we have a basic understanding of the structure. We can categories them. By categorizing them, we can store them is simple YAML files, so later on, we can use it directly for automation.
regions: [ "ap-northeast-1", "ap-northeast-2", "ap-northeast-3", "ap-south-1", "ap-southeast-1", "ap-southeast-2", "ap-northeast-3", "ca-central-1", "cn-north-1", "cn-northwest-1", "eu-central-1", "eu-north-1", "eu-west-1", "eu-west-2", "eu-west-3", "sa-east-1", "us-east-1", "us-east-2", "us-west-1", "us-west-2", "cn-north-1", "cn-northwest-1", "us-gov-east-1", "us-gov-west-1", ] app_urls: ["awsapps.com"] storage_urls: ["s3.amazonaws.com"] region_urls: ["amazonaws.com"]
app_urls: [ "appspot.com"] storage_urls: ["storage.googleapis.com"] regions:  region_urls: 
regions: ["oss-cn-hangzhou", "oss-cn-shanghai", "oss-cn-qingdao", "oss-cn-beijing", "oss-cn-zhangjiakou", "oss-cn-huhehaote", "oss-cn-wulanchabu", "oss-cn-shenzhen", "oss-cn-heyuan", "oss-cn-chengdu", "oss-cn-hongkong", "oss-ap-southeast-1", "oss-ap-southeast-2", "oss-ap-southeast-3", "oss-ap-southeast-3", "oss-ap-southeast-5", "oss-ap-northeast-1", "oss-ap-south-1", "oss-eu-central-1", "oss-eu-west-1", "cn-north-1", "oss-us-west-1", "oss-us-east-1", "oss-me-east-1"] storage_urls:  app_urls:  region_urls: ["aliyuncs.com"]
regions: ["ewr1"] storage_urls:  app_urls:  region_urls: ["vultrobjects.com"]
regions: [ "nyc1", "nyc2", "nyc3", "ams1", "ams2", "ams3", "sfo1", "sgp1", "lon1", ] storage_urls:  app_urls:  region_urls: ["digitaloceanspaces.com"]
and Microsoft Azure.
regions: ['eastasia', 'southeastasia', 'centralus', 'eastus', 'eastus2', 'westus', 'northcentralus', 'southcentralus', 'northeurope', 'westeurope', 'japanwest', 'japaneast', 'brazilsouth', 'australiaeast', 'australiasoutheast', 'southindia', 'centralindia', 'westindia', 'canadacentral', 'canadaeast', 'uksouth', 'ukwest', 'westcentralus', 'westus2', 'koreacentral', 'koreasouth', 'francecentral', 'francesouth', 'australiacentral', 'australiacentral2', 'southafricanorth', 'southafricawest'] storage_urls: [ "web.core.windows.net", "file.core.windows.net", "blob.core.windows.net", "queue.core.windows.net" ] app_urls: [ "azurewebsites.net", "cloudapp.net", "p.azurewebsites.net"] region_urls: ["cloudapp.azure.com"]
Now you will see a few keys in each YAML file. storage_url means the URLs for keeping files. app_urls means application URLs (useful when the primary domain behind WAF/Proxy). region URLs are where these regions appear in the URL.
I draw my brain just like this.
To desing and implement our idea, we had to answer some questions. In this part, we are looking into these questions and some possible answers to them.
How to detect a cloud provider?
There are a few solutions to answer this question. We can check the IP address of the target website using cloud IP ranges. We used IPINFO API; it's the free plan that will do perfectly fine for us. In case we detect Cloudflare, it means we can't find real IP, and the tool won't work. to overcome cloud detection limitation, we support the force method to search through your selected provider (e.g., through your previous OSINT)
Do we need a DNS brute force?
I struggled here a little my first idea was to check if DNS exists (53) (and also CNAME records) first and then check for the HTTP (80) because I found some mentioned tools use DNS brute force to discover stuff like regions on Azure to find entities. Still, in the end, if there is no app, it's not probably interesting for thee(except you want to attack the DNS server). Finally, every region based entity also available on HTTP.
How to handle HTTP responses?
Each cloud provider follows their structure and endpoints, but responses are quite similar, and usually, if a new entity found and return a 403, it means it's protected. If it returns a 200, it means whatever is there is open and accessible.
This method is not going to be 100% accurate, and the reason is somewhat apparent. Different providers have different security measures. Like rate limitations, IP score, request patterns, and so on so forth. Again based on the provider, they can manipulate results response codes and results. For example, instead of a 403, they return a 404.
How to counter security measures?
There are many ways to counter stuff like late limitations; the easiest solution is probably to use a proxy list, yet again finding the right proxy list that works precisely with our chosen cloud provider isn't as easy as it looks. To answer this simple question, I ended up writing another tool, which checks a proxy list and finds valid ones based on the target.
ProxyFor repo is here.
Even though this solution works on many providers, It's good to mention this solution isn't perfect yet, due to quality proxies and proxy detection measurements. We shall talk about more solutions later.
Any other know-how?
j3ssie also had this excellent idea to scan port on founded entity because there are chances of an API there, and the port is not 80. we shall merge this feature later.
So is this the best implementation?
Not at all, there is an endless cycle of improvement to any creation, but the better question is how much time we want to spend on a particular task. Here are things I can come up with while writing this.
So you found an app or buckets, what's next?
Almost in every case, there a friendly CLI tool provided by the provider. Namely, AWS, DO, GCP, which you can ab(use) to take your testing future by testing stuff like write accesses, apps behind WAFs, and finding hidden treasures.
We looked into the various cloud from BlackBox (unauthenticated) view and learned about cloud storage plus built a more effective software based on previous creation.
As you can see, creating a capable program (even though very simple) can take a lot of research behind the scene and it takes more effort than we initially imagined. now go ahead and enjoy will be amazed by things you can find out there.
PS: check out our awsome HunterSuite and happy hunting .