Gobuster Tutorial – How to Find Hidden Directories, Sub-Domains, and S3 Buckets

When it comes to web application security, what you see is not always what you get. Beneath the polished exterior of a website often lies a labyrinth of hidden directories, forgotten files, and undocumented endpoints ripe for exploitation by digital intruders.

As penetration testers and security professionals, it‘s our job to shine a light into these dark corners and uncover potential vulnerabilities before they can be leveraged by malicious actors. One of the most powerful tools in our arsenal for this task is Gobuster.

In this in-depth tutorial, we‘ll learn how to wield Gobuster to map the attack surface of web applications by brute-forcing directories, subdomains, and even cloud storage buckets. We‘ll cover everything from basic installation and usage to advanced techniques and how to protect your own assets from this type of enumeration.

But first, a quick disclaimer: this article is for educational purposes only. Scanning systems without permission is illegal, unethical, and a quick way to get yourself in trouble. Always obtain explicit consent from the owner before testing.

What is Gobuster?

At its core, Gobuster is a brute-forcing tool used to enumerate websites. It takes a wordlist containing common names and iterates through it, sending HTTP requests and noting which ones return valid responses.

This allows us to map out the directory structure and identify interesting files and folders that are not intended for public access. We can find things like:

Backup files (e.g. backup.zip, db.sql)
Admin interfaces (e.g. /admin, /dashboard)
Configuration files (e.g. config.json, .htaccess)
Source code (e.g. index.php, api.py)
Sensitive documents (e.g. invoices.pdf, users.csv)

Gobuster has grown to be more than just a directory buster. It now offers multiple modes for enumerating different types of targets:

dir mode for brute-forcing directories and files
dns mode for brute-forcing subdomains
s3 mode for brute-forcing AWS S3 bucket names
vhost and fuzz modes for other virtual host and parameter fuzzing

Gobuster is fast, flexible and highly extensible via custom wordlists. It‘s one of the first go-to tools in any tester‘s toolkit when investigating a new web app or domain.

Installing Gobuster

Gobuster is quick and easy to install. It‘s written in Go, so you just need a single binary for your platform.

On Kali Linux and other Debian derivatives, you can install from the standard repo:

sudo apt updatesudo apt install gobuster

On macOS you can install via Homebrew:

brew install gobuster

For Windows and other systems, pre-compiled binaries are available on the GitHub releases page. Just download the archive for your OS, extract the executable, and place it in your PATH.

To test your install, run gobuster -h and you should see the help output with available commands and options.

Wordlists 101

Brute-forcing is only as good as the wordlists you feed into your tools. Using weak or outdated lists will cause you to miss important findings and waste time on irrelevant junk.

A wordlist is simply a text file containing a list of terms, one per line. These can be names, words, numbers, or any combination thereof. Some common types of wordlists include:

Directories (e.g. admin, backup, config, data)
Files (e.g. backup.zip, index.php, config.json)
Subdomains (e.g. api, dev, staging, uat)
Passwords (e.g. 123456, password, qwerty)

You can find many public-domain wordlists on GitHub and other security resource sites. One of the best collections is SecLists, which offers a wide variety of purpose-built lists for different use cases.

I recommend downloading the SecLists repo and extracting it in a convenient location, such as /usr/share/seclists. You‘ll then have a treasure trove of wordlists at your fingertips ready to plug into Gobuster.

For optimal results, use multiple wordlists and tweak them to your target. For example, include company-specific terms, technology stack, and industry jargon for more focused brute-forcing. You can also generate custom permutations with tools like CeWL and crunch.

Directory Busting

Now that we have Gobuster installed and our wordlists ready, let‘s get started busting some directories! We‘ll use the classic DVWA app as our test target.

The most basic Gobuster command looks like this:

gobuster dir -u http://target.com -w /path/to/wordlist.txt

There are tons of options to customize our scan, as seen in the gobuster dir --help output:

Some useful options include:

-s to specify valid status codes (default 200,204,301,302,307)
-x to specify file extensions (e.g. php,txt,zip)
-a to set the User-Agent string (default "gobuster/3.1.0")
-c to use cookies for authenticated scans
-H to add extra headers (e.g. Authorization token)
-k to skip TLS certificate validation
-o to save results to a file
-p to specify an HTTP proxy
-t to adjust number of concurrent threads (default 10)

For our DVWA example, a good starting command would be:

gobuster dir -u http://dvwa.localtest/ -w /usr/share/seclists/Discovery/Web-Content/directory-list-2.3-medium.txt -x php,txt -o dvwa_dirs.log

This tells Gobuster to:

Scan the base URL http://dvwa.localtest/
Use the directory-list-2.3-medium.txt wordlist from SecLists
Check for .php and .txt extensions in addition to directories
Save output to dvwa_dirs.log

After a few minutes (longer for larger wordlists), we get a list of discovered items:

Subdomain Enumeration

In addition to finding hidden files and folders within a website, Gobuster can also be used to discover subdomains. Misconfigured DNS and lax cloud instance hygiene often expose numerous subdomains that are not properly secured.

To scan for subdomains, we use Gobuster‘s dns mode:

gobuster dns -d target.com -w /path/to/subdomain/wordlist.txt

The options are similar to dir mode, just swap -u for -d to specify the base domain. For wordlist, you want a dedicated subdomain list with common names like:

mail
webmail
ftp
cpanel
webdisk
ns1
support
dev
api
stage
uat
vpn
admin

Using a longer subdomain list like SecLists‘ subdomains-top1million-5000.txt will yield more comprehensive results, but will also take much longer. I recommend starting with a short list of high-probability names, then working up to larger lists if the target warrants deep investigation.

Note that by default, Gobuster will use the system DNS resolver to check for subdomains. You can specify custom resolvers with the -r option for more stealth and stability.

Also be aware that subdomain scanning can be noisy and may trigger alerts. Always obtain permission and consider the sensitivity of your target before unleashing a barrage of DNS queries.

S3 Bucket Hunting

One of Gobuster‘s newest features is the ability to brute-force public AWS S3 bucket names. Improperly secured S3 buckets are a major source of data leaks, and guessing their names is often far too easy.

To scan for open buckets, just point Gobuster at the S3 base URL with an appropriate wordlist:

gobuster s3 -w /usr/share/seclists/Discovery/Web-Content/common-bucket-prefixes.txt

The wordlist should contain either standalone bucket names or prefixes that are likely to be used in a company‘s naming scheme. For example:

wordpress-assets
customer-invoices
employee-data
aws-logs
puppet-secrets
docker-registry
jenkins-builds
circleci-artifacts

You can generate permutations based on a specific company name with tools like S3Scanner or just Google for open S3 buckets and take note of common patterns.

If Gobuster finds any accessible buckets, it will list their URL and the contents of the root XML file, which may contain further clues about their purpose and sensitivity.

As with subdomain enumeration, be very careful when testing S3 buckets and stick to non-invasive probing. Attempting to access private files can be a criminal offense, even if they are world-readable due to misconfiguration. If you do find an open bucket, notify the owner ASAP so they can lock it down.

Defending Against Gobuster

Knowing how to use Gobuster is only half the equation – we must also learn to defend against it. After all, our own websites and apps are just as likely to be targeted by enterprising hackers looking for an easy score.

The first step is to run Gobuster against yourself. Perform a full asset inventory and subject each domain, subdomain, and S3 bucket to the same rigorous analysis you would apply to an authorized target. This will give you a birds-eye view of your true attack surface and help prioritize remediation efforts.

Next, apply the principle of least privilege to every level of your stack. Each entity should have access to only the bare minimum resources it needs to function – no more, no less. In practice, this means:

Disable directory indexing and hide version/backup files
Place sensitive interfaces behind access controls and 2FA
Sanitize server responses to strip identifying information
Use restrictive S3 IAM policies and avoid public ACLs
Regularly rotate secrets and prune stale resources

You should also invest in active monitoring and alerting to detect and block automated scans. WAFs like CloudFlare, Sucuri, and AWS WAF have built-in rules to filter Gobuster traffic based on behavioral heuristics.

Finally, have an incident response plan for when (not if) someone manages to find a vulnerable asset. Know who to contact, what actions to take, and how to disclose the issue responsibly to prevent further damage. Platforms like HackerOne and Bugcrowd can be helpful for managing inbound vulnerability reports.

Conclusion

Gobuster is a powerful addition to any appsec arsenal, allowing us to efficiently map the darkest corners of our targets. By probing for hidden directories, forgotten subdomains, and mismanaged S3 buckets, we can gain valuable insights into the true scope of a website‘s attack surface.

But with great power comes great responsibility. Always obtain permission before testing, and never abuse your findings. As defenders, we must also learn to protect our own assets from the prying eyes of Gobuster by practicing the fundamentals of least privilege, continuous monitoring, and responsible disclosure.

By wielding Gobuster for good, we can shine a light on the web‘s darkest secrets and build a safer internet for everyone. Happy busting!

Gobuster Tutorial – How to Find Hidden Directories, Sub-Domains, and S3 Buckets - Bomberbot (2024)