Passive/Active Information Gathering: Subdomain Enumeration

This post is design to share some of the information I’ve learned while working through the Information Gathering- Web Edition module in HackTheBox Academy. The module provides a lot of useful information for those looking to get into bug bounties or the security field in general.

I’ve divided this post by first providing a brief overview of why information gathering is important and what kind of information needs to be obtained. Next is a brief overview of DNS to develop an understanding of why enumerating subdomains is important. Finally, I’ll go over some examples of conducting passive as well as active subdomain enumeration.

Please note the examples were done on an organization that allows this type of activity on their bug bounty program. Please don’t conduct any hacking or security assessments without permission as this can be viewed as illegal activity!

Before beginning any type of penetration test or security assessment, we need to develop a clear understanding of the target in question. We need to obtain enough information as possible about our target in order to get a clear understanding of the landscape we’re dealing with. This is true whether we’re trying to assess a target’s web presences as well as their overall infrastructure. Therefore, when conducting web assessments, it’s essential to conduct information gathering to collect information such as:

The information gathering process in this context falls into two main categories:

One thing to note about information gathering is that it can occur at any stage of the assessment. The more information we have about a target, the more likely we’re able to tailor our tests to not only become successful but provide value to the client.

For the purposes of this post, I’ll cover the different ways to conduct passive and active information gathering as it pertains to enumerating subdomains. Please note that the example target here (tesla.com) has an open bug bounty program which allows for this activity. Please take care when doing these tests to ensure the activity being conducted is legal.

DNS or Domain Name System can be thought of as the internet’s phone book. It’s a protocol by which domain names are converted from their human-readable names to IP addresses. This is so that when we want to go to google.com, we don’t have to memorize it by its IP address 142.250.176.78. Each internet-connected device has a unique IP address that other machines/servers use to locate it. How these machines are able to be located can be understood by analyzing a URL of a website. Say we have www.tesla.com as an example. DNS hostnames such as this one are set up in a type of hierarchy, starting from the root domain(.). This isn’t explicitly shown in the URL because it’s already implied, as everything starts from the root domain.

Starting from right to left, the first domains that are nested under the root domain are called Top-Level Domains (TLDs). Some examples of these domains include: com, edu, and net. The next level of nested domains are the second-level domains which are usually the domains we’re trying to get to. These domain names are registered with a domain name registrar which basically signs them up to be part of the internet phonebook. Any domain nested under these secondary domains are considered subdomains and are usually part of the organization. So our example: com is the top level domain which is typically used by commercial organizations, tesla is the main domain and organization we’re trying to reach and www is the subdomain for the main webpage of the organization. As mentioned before, the root domain is not mentioned in the URL but would be to the right of com (com.).

For more information on DNS please see Domain Name System

Now that we have a general understanding of DNS, we can move the conversation now to enumerating subdomains. Since subdomains are set up by individual companies and may be used for different reasons within the organization, they may prove to be useful to uncover during a security assessment. Like www, there may be subdomains for other web pages that may not be as tightly managed, for internal portals only used by employees such as an intranet or any other reason that network administrators would want to keep hidden.

There are many ways to conduct passive subdomain enumeration. For that reason I’m only going to focus on a couple of examples. For passive subdomain enumeration, I’ll be retrieving information from the following sources:

Project Sonar

Project Sonar is a security research project by Rapid7 which conducts internet-wide surveys across various services and protocols to gather insight into worldwide vulnerability exposure. Because the information is made to be public to facilitate research, we can use the project to find subdomains. We can do this by visiting the API URL (https://sonar.omnisint.io/subdomains/<domain>). In our example we’d use https://sonar.omnisint.io/subdomains/tesla.com however the output on the browser may be harder to read as the response is in JSON format.

A cleaner option would be to use Curl from the command line as follows:

curl -k -s https://sonar.omnisint.io/subdomains/tesla.com | jq -r ‘.[]’ | sort -u

Project Sonar also allows us to retrieve other information besides subdomains. For more information please see the Git page for the associated API or Omnisint website.

SSL/TLS Certificates

SSL/TLS certs provide another interesting way of extracting subdomains for a target organization. Certificate Transparency requires that every certificate issued by a Certificate Authority must be published to a publicly accessible log. This allows for efficient identification of mistakenly or maliciously issued certificates. A useful resource to accomplish enumeration in this case would be https://crt.sh/. Simply type in the organization to search and a long list of certification information will be presented with dates and subdomains associated with each.

As with the previous example, using Curl from the command line may assist in accomplishing a cleaner output.

curl -k -s “https://crt.sh/?q=tesla.com&output=json" | jq -r ‘.[] | “\(.name_value)\n\(.common_name)”’ | sort -u

Sublist3r

There are plenty of tools that automate the passive enumeration of subdomains but one of my favorites is Sublist3r. It can be cloned from Github or installed using a package manager (if using Linux). Sublist3r enumerates subdomains using many search engines such as Google, Yahoo, Bing, Baidu and Ask. Sublist3r also enumerates subdomains using Netcraft, Virustotal, ThreatCrowd, DNSdumpster and ReverseDNS. It’s an ideal tool that takes the hassle out of doing manual searches. One thing to keep in mind that a lot of requests are being sent to these search engines so you may run into a situation where the requests are being blocked by CAPTCHAs.

Enabling verbosity (-v option) will display where the results were captured from. The example shows which subdomains were found using ThreatCrowd.

For active subdomain enumeration, we’ll be probing the target organization or 3rd party DNS servers previously identified. Remember this kind of enumeration can be noisy and can lead to detection of recon activities so take care and make sure assessments are done responsibly and with permission. For this section, we’ll be looking at DNS zone transfers and a command line tool called GoBuster.

Zone Transfer

When an organization sets up DNS, chances are there are more than one DNS server. This is so that there is redundancy and backup of information in the case that one of the primary DNS servers goes down. The replication or synchronization of the DNS information is done via zone transfers. The caveat is that zone transfers SHOULD ONLY be allowed by specified secondary DNS servers within the organization environment. Although rare, there may be times where DNS servers are misconfigured in that they allow ANYBODY to conduct zone transfers. If that is the case, anyone is able to retrieve ALL subdomains plus additional domain information present within an organization. Knowing that, it’s worth testing to see if this functionality is allowed.

Zone transfers can be done using command line tools such as Nslookup or Dig but we can also use online services such as https://hackertarget.com/zone-transfer/ to expedite assessment. As we can see from the example below Tesla doesn’t allow zone transfers.

As mentioned, manual zone transfers can be done using command line utilities. In our example we’ll use Nslookup to understand how we’d do this manually.

nslookup -type=NS tesla.com

Name servers are the servers that respond to DNS requests and deliver the appropriate information. In the command above, we’re specifying that we want a list of nameservers (-type=NS) for the tesla.com domain. Nslookup responded with the name servers as well as their associated IP Address.With these name servers, we can try performing a zone transfer on each one to see if any are misconfigured.

Note: if you used the online service from the previous example, you can see that it automatically tried to conduct a zone transfer each of the discovered named servers.

2. Testing for ANY and AXFR Zone Transfer

nslookup -type=any -query=AXFR tesla.com a1–12.akam.net

The command above is telling Nslookup that we want to retrieve any information we can using a Authoritative Transfer (AXFR) for tesla.com from name server a1–12.akam.net. Although the transfer failed, it’s still useful to know how a zone transfer is done.

Note: if ever a zone transfer is successful, there’s no need to continue the subdomain enumeration process as a successful zone transfer would result in having ALL domain information for the target.

GoBuster

Gobuster is a tool that can do many things but one of which is to perform subdomain enumeration. It does so by making many DNS requests, using a list of subdomains to guess, in hopes of finding valid subdomains. Essentially, the success of this approach depends on the list we’re using. Note: the longer the list, the noisier this will be as it will try each and every item in the list.

Ideally, evaluating the patterns and naming conventions of subdomains we’ve discovered during the passive phase would help us curate pretty effective lists but we could also use lists that others have already curated online. There are plenty of wordlists online that we can use for this, which can be found at the Seclists Github repository.

For demonstration purposes, we’ve made a short list of subdomains to try against and ran the following command:

gobuster dns -d tesla.com -w simple-subdomain-list.txt

This tells gobuster to run a DNS search on the tesla.com domain using the specified wordlist (-w) we gave.

GoBuster is useful because it’s quick and customizable in terms of the threads to use and the scanning options. Please see the Github page for more information on how to use it as I’m sure it will be very helpful.

I hope that this post proves to be a helpful resource for gathering subdomain information. This is just some of the information I’ve learned while going through the Information Gathering module in HackTheBox Academy and hope that it helps the next person either looking to get into bug bounties or the security field in general. Please feel free to leave feedback on the information provided along with any tips or additional useful resources.

Thanks!