Directory brute-forcing

The gist A server answers an existing path and a non-existent one differently. We run a wordlist of names through it and watch the response codes — that's how you find pages nothing links to: admin panels, backups, service functions. In English — content discovery, forced browsing, directory brute-forcing.

The server's status code for each guessed path is the whole signal.

First — what the site hands you itself

Before brute-forcing, check the free stuff:

robots.txt — the list of things the owner asks not to be indexed, i.e. a map of the interesting bits. On Mr Robot it held a key and a wordlist.
sitemap.xml
the page source: comments, paths inside JS files

How to read the responses

Code	What it means
`200`	the path exists and is served
`301` / `302`	exists, a redirect — often a directory, appends a `/`
`401`	exists, needs authorization
`403`	exists, but access is denied — still interesting: files inside may be open
`404`	not there

gobuster

# quick start
gobuster dir -u http://target -w /usr/share/wordlists/dirb/common.txt

# with extensions and threads
gobuster dir -u http://target -w /usr/share/wordlists/dirbuster/directory-list-2.3-medium.txt -x php,txt,bak -t 50

Flag	What for
`-x php,txt,bak`	also look for files with extensions — otherwise `config.php.bak` won't be found
`-t 50`	threads (default 10)
`-b 404,403`	which codes not to show (default `404`)
`--xl 1234`	drop responses of this length — the cure for wildcard
`-k`	don't verify the TLS certificate
`-d 500ms`	pause between requests — quieter in the logs

ffuf — the same, more flexible

ffuf -u http://target/FUZZ -w /usr/share/wordlists/dirb/common.txt -fc 404

FUZZ is the substitution point — you can put it anywhere: in a parameter, a header, a subdomain. Filters: -fc by code, -fs by size.

Wordlists on this Kali

File	Lines	When
`/usr/share/wordlists/dirb/common.txt`	4,614	first quick pass
`/usr/share/wordlists/dirbuster/directory-list-2.3-medium.txt`	220,560	the main one
SecLists `Discovery/Web-Content/`	—	not installed yet: `sudo apt install seclists`

Pitfalls

Wildcard. The server answers 200 to any path — gobuster stops with an error. Look at the length of a "non-existent" response and cut it off with --xl (in ffuf — -fs).
Without extensions half the finds are lost — backups and configs sit as files, not directories.
It's noisy. Hundreds of requests a minute show up in any log. On real work the rate is agreed in advance.

Defence

Don't rely on hidden addresses — check permissions on the server for every request.
Rate-limiting, a WAF, an alert on a spike of 404s from one address.

403 is a find, not a dead end A 403 means the path exists but access is denied — already interesting, and files inside it may still be open. The art is telling "exists but blocked" from "isn't there", which is exactly what the status codes hand you.