Guide to Using httpx

Discover httpx a powerful tool for reconnaissance, and HTTP probing.

httpx is a powerful tool designed for HTTP probing and reconnaissance. It allows users to identify and gather information about web services efficiently. Below, we dive into various commands and usage scenarios of httpx, providing detailed examples and descriptions for each feature.

You can find a shorter cheat sheet version of this article here.

Probe Host

To probe a single host, use the following command:

httpx -u example.com -probe

This command initiates an HTTP probe on `example.com`. This is useful for quickly checking the availability of the web page.

Scan Hosts from Input File

To scan multiple hosts listed in a file, use:

httpx -l hosts.txt

Here, `hosts.txt` contains the list of target hosts, each on a new line. The output will include a list of hosts where the HTTP probe was successful. This is particularly helpful when dealing with large sets of hosts, such as those generated from subdomain enumeration tools.

Scan Targets from Other Program Output

httpx can also take input from other programs using pipes:

cat hosts.txt | grep example.com | httpx

This method reads hosts from `hosts.txt`, grep for specific domain and pipes them directly into httpx for scanning.

Tool Chain Example

Combining httpx with other tools, like subfinder:

subfinder -d example.com -silent | httpx -title -tech-detect -status-code

This command uses `subfinder` to find subdomains of `example.com` and pipes them into httpx to get the page title, detect technologies, and retrieve HTTP status codes.

Testing Multiple Ports

To scan multiple common HTTP ports:

httpx -u example.com -ports 80,443,8009,8080,8081,8090,8180,9443

This command tests the specified ports for HTTP services on example.com. Testing multiple ports is crucial because web services can run on non-standard ports, and identifying these can reveal additional attack surfaces.

Path Brute-force

Test different paths or files:

httpx -l urls.txt -sc -path "/,/path1,/path2,/path3"

This command checks the specified paths on the targets from `urls.txt` file and displays the status codes. It might be useful when you want to check multiple paths or files on a single host or as in this example list of URLs.

httpx offers a variety of probes that allow you to extract detailed information from your HTTP scans. These probes are essential for gaining deeper insights into the target’s properties, such as the response status, server details, and content types. By using these probes, you can tailor your scans to capture the most relevant data for your analysis.

httpx -status-code -content-type -content-length -location \
-title -web-server -tech-detect -ip -cname -word-count -line-count -response-time \
-cdn -hash sha256 -include-response -silent -stats -follow-host-redirects -max-redirects 2

This command collects a wide range of data, including status code, content type, server name, and more, following redirects up to two times. This allows you to get a detailed overview of the target’s HTTP properties.

Save responses and filter HTTP redirects and error pages:

httpx -silent -l urls.txt -j -o httpx.json -sr \
-sc -title -ct -cl -bp -server -td -ip -cname -word-count -hash sha256 -fep -fc 301 \
-tlsi -random-agent -stats -t 5 -rl 10 -timeout 5 -maxhr 3

This command saves the scan results in `httpx.json`, displaying various headers and applying filters to ignore specific status codes and error pages. This is useful for post-processing and analyzing the results offline — for example looking for secrets like hard-coded API keys.

Interesting Probes

Here are some of the key probes available in httpx:

`-sc, -status-code`: Display response status-code`-title`: Display page title`-bp`: Display body preview`-server, -web-server`: Display server name`-ip`: Display host IP`-cname`: Display host CNAME`-cl, -content-length`: Display response content-length`-ct, -content-type`: Display response content-type`-location`: Display response redirect location`-hash string`: Display response body hash (supported: md5, mmh3, simhash, sha1, sha256, sha512)`-rt, -response-time`: Display response time`-lc, -line-count`: Display response body line count`-wc, -word-count`: Display response body word count`-td, -tech-detect`: Display technology in use based on wappalyzer dataset (default true)`-method`: Display HTTP request method`-cdn`: Display CDN/WAF in use (default true)

Rate limits in httpx allow you to control the number of requests sent per second or minute, ensuring that your scanning process does not overwhelm the target server or your network. This feature is essential for maintaining a respectful and efficient scanning. By configuring rate limits, you can balance the thoroughness of your scans with the need to avoid detection and throttling by the target server. Here’s how you can set and manage rate limits in httpx:

httpx -u example.com -t 10 -rate-limit 50

This sets the number of threads to 10 and limits the rate to 50 requests per second.

Rate Limit Options

`-t int`: Number of threads`-rl int`: Maximum requests per second`-rlm int`: Maximum number of requests per minute

Matchers in httpx enable you to filter and process responses based on specific criteria, such as HTTP status codes, the presence of certain strings, or regular expressions. This functionality is crucial for narrowing down results to those that meet particular conditions, making your scans more focused and effective. By using matchers, you can quickly identify and act upon relevant responses, optimizing your workflow and ensuring that you capture the data that matters most to your analysis or security assessments. Here’s how you can utilize matchers in httpx to streamline your scanning process:

Match Specific HTTP Codes

cat hosts.txt | httpx -mc 200,302

This matches responses with HTTP status codes 200 or 302.

Match Responses with Specific String

cat hosts.txt | httpx -ms admin

This matches responses containing the string “admin”.

Match Responses with Regex

cat hosts.txt | httpx -mr 'admin*'

This matches responses that fit the regex pattern `admin*`.

Filters in httpx allow you to exclude unwanted responses based on specific criteria, such as HTTP status codes or text content. This feature is essential for refining your scan results, enabling you to focus only on the relevant data and reducing noise from your output. By applying filters, you can efficiently manage large volumes of data, streamline your analysis, and ensure that you are only working with the most pertinent information. Here’s how you can leverage filters in httpx to optimize your scanning and data processing workflows:

Filter Responses with Specific HTTP Codes

httpx -l urls.txt -fc 404,403,401,400,500

This filters out responses with the specified status codes.

Filter Responses Based on ML Error Page Detection

httpx -l urls.txt -sc -fep

This uses machine learning to filter error pages.

Filter Responses with Specific Text

httpx -l urls.txt -fs error

This filters out responses containing the text “error”.

Filter Responses Based on Regex

httpx -l urls.txt -fe '.*Error.*'

This filters out responses that match the regex pattern `.*Error.*`.

Extractors in httpx are powerful tools designed to pull specific parts of the response content based on patterns or regular expressions. This feature is particularly useful for isolating and retrieving valuable data from HTTP responses, such as usernames, emails, or other critical information embedded within the response body. By using extractors, you can automate the process of data extraction, making it more efficient and accurate. When you are performing security assessments, extractors help you focus on the essential elements of your scan results, ensuring that you gather the necessary information quickly and effectively. Here’s how you can utilize extractors in httpx to enhance your data extraction processes:

Extract Part of the Response with Regex

cat hosts.txt | httpx -er 'admin*'

This extracts parts of the response that match the regex `admin*`.

Optimizations in httpx are designed to enhance the performance and efficiency of your scans. By fine-tuning various parameters, you can control how httpx handles timeouts, retries, error counts, and host exclusions. These optimizations are crucial for large-scale scanning operations, where performance and accuracy are paramount. Leveraging these settings allows you to minimize resource usage, avoid unnecessary retries, and ensure that your scans run smoothly without interruption. Here are some key optimization techniques to improve your scanning process:

Probe with Protocol Scheme Supplied in the Input

httpx -l urls.txt -nfs

This probes using the protocol scheme supplied in the input without falling back to another scheme.

Set Timeouts, Max Error Count, and Retries

httpx -l urls.txt -timeout 5 -maxhr 3 -retries 1

This sets a timeout of 5 seconds, a maximum error count of 3 per host, and 1 retry.

httpx provide versatile options for storing and displaying scan results. Here’s how you can utilize httpx’s output options to enhance your scanning and analysis processes:

Save Output to a File

httpx -l urls.txt -o httpx.log

This saves the scan output to `httpx.log`.

Print Output in JSONL Format

httpx -l urls.txt -j

This prints the output in JSON Lines format.

Print Stats During Scan

httpx -l urls.txt -stats

This command prints scan statistics during execution. You can change how often stats are displayed using `-si` parameter followed by the number of seconds. The default interval is `5` seconds.

Store Responses

httpx -l urls.txt -sr http-responses/

This saves the HTTP responses in a given directory.

The Screenshots feature in httpx allows you to capture visual representations of web pages during your scanning process. This capability is invaluable for visually inspecting how web pages appear and verifying their content and layout. httpx screenshots are made using headless browser. Here are some examples:

Create a Screenshot of the Website

echo https://example.com | httpx -ss -st

This takes a screenshot of the specified URL.

Screenshot Options

`-ss`: Save a screenshot of the page using a headless browser`-st`: Set a timeout for the screenshot (default 10 seconds)

httpx offers a robust configuration file feature that allows users to customize and streamline their scanning operations according to specific requirements. This feature is particularly useful for defining scan parameters, such as headers, timeouts, retries, and output preferences, in a structured YAML format. By utilizing httpx configuration files, users can standardize scan settings across multiple sessions, enhance scanning efficiency, and tailor scans to different environments or targets.

Use Custom Configuration File

httpx -config httpx-config.yaml

This uses a custom configuration file located at `httpx-config.yaml`.

Example Configuration File

status-code: true
content-length: true
content-type: true
location: true
line-count: true
word-count: true
title: true
body-preview: true
web-server: true
tech-detect: true
ip: true
cname: true
filter-code: 302,401,403
filter-error-page: true
threads: 10
rate-limit: 20
update: false
disable-update-check: true
store-response: true
store-response-dir: httpx-responses
json: true
include-response-header: true
include-response: true
random-agent: true
#header: Custom Global Headers
follow-redirects: false
follow-host-redirects: false
tls-impersonate: true
version: false
stats: true
silent: true
stats-interval: 5
max-host-error: 3
retries: 0
timeout: 5

This configuration file sets various default parameters for httpx scans.

httpx simplifies updates with options to check for and install the latest version. It also allows users to disable automated update checks on startup for better control of the scan.

Disable Update Checks

httpx -l targets.txt -duc

This disables automatic update checks.

Update httpx

httpx -up

This updates httpx to the latest version.

As a bonus here are some examples of how to filter and extract specific information from httpx JSON output.

Learn how to selectively process JSON results based on status codes, technology, titles, and more:

Select Results with `status_code == 200`

cat httpx.json | jq 'select(.status_code == 200)'

This filters results where the status code is 200.

Select Results with `Ruby` in `tech`

cat httpx.json | jq 'select(.tech[] | contains("Ruby"))'

This selects results that have “Ruby” in the technology list.

Select URL, tech where tech contains `Ruby`

cat httpx.json | jq 'select(.tech[] | contains("Ruby")) | .tech,.url' 2>/
dev/null

This extracts URLs and technologies where the tech list contains “Ruby”.

The Same for PHP

cat httpx.json | jq 'select(.tech[] | contains("PHP")) | .tech,.url' 2>/dev/null

This extracts URLs and technologies where the tech list contains “PHP”.

Search Results by Title

cat httpx.json | jq 'select(.title | contains("Index of"))'

This searches results for titles containing “Index of”.

Search All Results for Nginx (Case-Insensitive)

cat httpx.json | jq 'select(.tech[] | ascii_downcase| contains("nginx")) | .tech,.url' 2>/dev/null

This searches for Nginx in the technology list, case-insensitively.

Extract Basic Info About Each Request

cat httpx.json| jq '{url: .url, host: .host, method: .method, status_code: .status_code, content_type: .content_type, words: .words, webserver: .webserver, tech: .tech, hash: .hash}'

This extracts basic information about each request.

The Same, but Only for 200 Responses

cat httpx-* | jq 'select(.status_code == 200) | {url: .url, host: .host, method: .method, status_code: .status_code, content_type: .content_type, words: .words, webserver: .webserver, tech: .tech, hash: .hash}'

This extracts basic information for requests with a status code of 200.

Select `200` Status Code and Content Type `application/json`

cat httpx.json| jq 'select((.status_code == 200) and (.content_type == "application/json"))'

This selects results where the status code is 200 and the content type is `application/json`.

In summary, httpx is a powerful tool tailored for security assessments, web reconnaissance, and HTTP probing tasks. httpx enables users to efficiently scan hosts, test multiple ports, perform path bruteforcing, and employ advanced filtering and extraction on the results. httpx is useful for identifying vulnerabilities and gaining insights into the technological landscape of target systems. By integrating httpx into your toolkit, you gain a reliable HTTP scanning solution for conducting infrastructure analysis.

References

Httpx usageRunning httpx