How do I automate my recon — Part Two

9 months ago 80

BOOK THIS SPACE FOR AD

ARTICLE AD

Hellow world!

Hope you are doing well. After a few days I’m finally back with part 2 of our series “How do I automate my recon”. If you haven’t checked out the first part yet, I highly recommend you to do so here.

Anyways if you haven't time to check it out, here’s a little summary. We had a stage1.sh script that took a domain name and an organization name and created a directory for it.

Then it passed the info to stage2.sh and our recon actually started there. It used subfinder to gather subs and did a number of other things to organise the data.

Today we are going to discuss stage3.sh , insert.py , update.py . On top of that we will discover how to automate things. And as a bonus will also learn how to properly setup a discord server and get notified of certain info gathered throughout the recon process.

Let’s get into it…

import sys
import pymongo
from pymongo import MongoClient
import time
import pytz
from datetime import datetime

# Connect to MongoDB
client = MongoClient("CONNECTION-STRING") # Add your connection string here
db = client.assets # Replace "assets" with your collection name

# Get current time in UTC
current_time_utc = datetime.utcnow()

# Convert UTC time to a specific timezone
timezone = pytz.timezone('Europe/Berlin') # Replace'Europe/Berlin' your timezone
current_time_timezone = current_time_utc.replace(tzinfo=pytz.utc).astimezone(timezone)

# Format the time with timezone
pretty_time = current_time_timezone.strftime("%d. %b. %Y %H:%M %Z")

# Check if the command-line arguments are provided correctly
if len(sys.argv) != 3:
print("Usage: python3 myscript.py /path/to/all-subs.txt org")
sys.exit(1)

# Extract the directory path and organization name from the command-line arguments
directory_path = sys.argv[1]
organization = sys.argv[2]

collection = db[organization]

# Read subdomains from file
with open(f'{directory_path}', 'r') as file:
new_subdomains = set(file.read().splitlines())

# Query existing subdomains from MongoDB
existing_subdomains = set(doc['subdomain'] for doc in collection.find({}, {'subdomain': 1}))

# Identify new subdomains
non_duplicate_subdomains = new_subdomains - existing_subdomains

# Insert non-duplicate subdomains into MongoDB
if non_duplicate_subdomains:
new_docs = [{'subdomain': subdomain, 'org': organization, 'status': '', 'added': pretty_time, 'last_update': ''} for subdomain in non_duplicate_subdomains]
collection.insert_many(new_docs)
print(f"{len(non_duplicate_subdomains)} new subdomains inserted.")
else:
print("No new subdomains to insert.")

This is the python script that our stage2.sh and stage3.sh scripts use to insert found subdomains into the database. For each subdomain it describes the following:

org- name of the organisation/program (provided to stage1.sh)status- current status of subdomain (is empty, will be updated later)added- time stamp for when the record is added to databaselast_update- also empty for now, will be updated with each update later

It also makes sure that there are no duplicates.

Don’t forget to change the connection string, collection name, and time zone according to your needs.

import sys
import pymongo
from pymongo import MongoClient
import time
import pytz
from datetime import datetime

# Connect to MongoDB
client = MongoClient("CONNECTION-STRING") # Add your connection string here
db = client.assets # Replace "assets" with your collection name

# Get current time in UTC
current_time_utc = datetime.utcnow()

# Format the time with timezone
pretty_time = current_time_timezone.strftime("%d. %b. %Y %H:%M %Z")

# Check if the command-line arguments are provided correctly
if len(sys.argv) != 3:
print("Usage: python3 myscript.py /path/to/all-subs.txt org")
sys.exit(1)

# Extract the directory path and organization name from the command-line arguments
directory_path = sys.argv[1]
organization = sys.argv[2]

collection = db[organization]

def read_subdomains_from_file(filename):
subdomains = []
with open(filename, 'r') as file:
for line in file:
subdomains.append(line.strip()) # Remove newline characters and append to the list
return subdomains

def update_200():
filename = directory_path + "/200.txt"
subdomains_list = read_subdomains_from_file(filename)
updated_count = 0 # Initialize a counter for updated records

for subdomain in subdomains_list:
result = collection.update_one({"subdomain": subdomain}, {"$set": {"status": 200, "last_update": pretty_time}})
if result.modified_count > 0:
updated_count += 1
return updated_count

updated_records_count_200 = update_200()
if updated_records_count_200 > 0:
print(f"{updated_records_count_200} records with status 200 updated.")

def update_301():
filename = directory_path + "/301.txt"
subdomains_list = read_subdomains_from_file(filename)
updated_count = 0 # Initialize a counter for updated records

for subdomain in subdomains_list:
result = collection.update_one({"subdomain": subdomain}, {"$set": {"status": 301, "last_update": pretty_time}})
if result.modified_count > 0:
updated_count += 1
return updated_count

updated_records_count_301 = update_301()
if updated_records_count_301 > 0:
print(f"{updated_records_count_301} records with status 301 updated.")

def update_302():
filename = directory_path + "/302.txt"
subdomains_list = read_subdomains_from_file(filename)
updated_count = 0 # Initialize a counter for updated records

for subdomain in subdomains_list:
result = collection.update_one({"subdomain": subdomain}, {"$set": {"status": 302, "last_update": pretty_time}})
if result.modified_count > 0:
updated_count += 1
return updated_count

updated_records_count_302 = update_302()
if updated_records_count_302 > 0:
print(f"{updated_records_count_302} records with status 302 updated.")

def update_401():
filename = directory_path + "/401.txt"
subdomains_list = read_subdomains_from_file(filename)
updated_count = 0 # Initialize a counter for updated records

for subdomain in subdomains_list:
result = collection.update_one({"subdomain": subdomain}, {"$set": {"status": 401, "last_update": pretty_time}})
if result.modified_count > 0:
updated_count += 1
return updated_count

updated_records_count_401 = update_401()
if updated_records_count_401 > 0:
print(f"{updated_records_count_401} records with status 401 updated.")

def update_403():
filename = directory_path + "/403.txt"
subdomains_list = read_subdomains_from_file(filename)
updated_count = 0 # Initialize a counter for updated records

for subdomain in subdomains_list:
result = collection.update_one({"subdomain": subdomain}, {"$set": {"status": 403, "last_update": pretty_time}})
if result.modified_count > 0:
updated_count += 1
return updated_count

updated_records_count_403 = update_403()
if updated_records_count_403 > 0:
print(f"{updated_records_count_403} records with status 403 updated.")

def update_404():
filename = directory_path + "/404.txt"
subdomains_list = read_subdomains_from_file(filename)
updated_count = 0 # Initialize a counter for updated records

for subdomain in subdomains_list:
result = collection.update_one({"subdomain": subdomain}, {"$set": {"status": 404, "last_update": pretty_time}})
if result.modified_count > 0:
updated_count += 1
return updated_count

updated_records_count_404 = update_404()
if updated_records_count_404 > 0:
print(f"{updated_records_count_404} records with status 404 updated.")

def update_503():
filename = directory_path + "/503.txt"
subdomains_list = read_subdomains_from_file(filename)
updated_count = 0 # Initialize a counter for updated records

for subdomain in subdomains_list:
result = collection.update_one({"subdomain": subdomain}, {"$set": {"status": 503, "last_update": pretty_time}})
if result.modified_count > 0:
updated_count += 1
return updated_count

updated_records_count_503 = update_503()
if updated_records_count_503 > 0:
print(f"{updated_records_count_503} records with status 503 updated.")

As the name suggests, update.py is used to update our subdomain. During the recon process, after the httpx when we have the status code and relevant data for each subdomain, update.py uses those data to query the subdomains from the database and update the status code on each sub. It also adds a time stamp for when each record is updated under “last_update”.

#!/bin/bash

baseDir="/root/recon"
scriptsDir="/root/recon/scripts"

if [[ -d "$baseDir" ]]; then
for dir in "$baseDir"/*/; do
if [[ -f "${dir}/rootdomain.txt" ]]; then
programName=$(basename "$dir")
echo "Gathering subs for $programName..."
subfinder -dL "${dir}/rootdomain.txt" -all -silent | anew -q "${dir}/all_subs.txt"
echo "Resolving found subdomains..."
dnsx -l "${dir}/all_subs.txt" -silent | anew -q "${dir}/resolved.txt"
echo "Gathering http metadata..."
httpx -l "${dir}/resolved.txt" -sc -title -ct -location -server -td -method -ip -cname -asn -cdn > "${dir}/metadata.txt"
echo "Separating subs by status code..."
#using sed to remove the color output in metadata file so the grep doens't freak out later on
sed 's/\x1B\[[0-9;]*[JKmsu]//g' "${dir}/metadata.txt" > "${dir}/metadata.tmp"
grep '\[200\]' "${dir}/metadata.tmp" | cut -d " " -f 1 | cut -d "/" -f 3 > "${dir}/200.txt"
grep '\[301\]' "${dir}/metadata.tmp" | cut -d " " -f 1 | cut -d "/" -f 3 > "${dir}/301.txt"
grep '\[302\]' "${dir}/metadata.tmp" | cut -d " " -f 1 | cut -d "/" -f 3 > "${dir}/302.txt"
grep '\[401\]' "${dir}/metadata.tmp" | cut -d " " -f 1 | cut -d "/" -f 3 > "${dir}/401.txt"
grep '\[403\]' "${dir}/metadata.tmp" | cut -d " " -f 1 | cut -d "/" -f 3 > "${dir}/403.txt"
grep '\[404\]' "${dir}/metadata.tmp" | cut -d " " -f 1 | cut -d "/" -f 3 > "${dir}/404.txt"
grep '\[502\]' "${dir}/metadata.tmp" | cut -d " " -f 1 | cut -d "/" -f 3 > "${dir}/502.txt"
grep '\[503\]' "${dir}/metadata.tmp" | cut -d " " -f 1 | cut -d "/" -f 3 > "${dir}/503.txt"
cat "${dir}/metadata.tmp" | grep '\[200\]' | notify -bulk -pc "${dir}/provider-config.yaml"
echo "Inserting records into the database..."
programName=$(basename "$dir")
python3 "${scriptsDir}/insert.py" "${dir}/all_subs.txt" ${programName}
echo "Updating status codes..."
python3 "${scriptsDir}/update.py" "${dir}" ${programName}
else
programName=$(basename "$dir")
echo "No root domains found for $programName!"
fi
done
else
echo "Directory '$baseDir' does not exist."
fi

stage3.sh is the main part of our automation. It is very similar to our stage2.sh except it repeats the recon process for every folder/directory inside a base directory defined at the very beginning of the script. The base directory is/root/recon for me, but obviously you can change it to whatever works for you.

So whenever you want to add a program/target to your automation, all you have to do is to create a folder under the base directory and have a rootdomain.txt file inside it containing the targets home page.

Here is how my recon directory looks like:

Once I run the stage3.sh script, it goes ahead and does a full recon on all the targets you see in the picture.

Now the way we are going to be automating all these is using a cronjob. Cron is simply a command-line utility that performs actions on a schedule. What we need to do is to add our stage3.sh as a cron job and give it a 6 or 12 hour interval. (you can edit this to your liking)

And what happens is the cron utility will be running our stage3.sh every 6/12 hours in the background, and as we learned earlier that means we are performing recon on all our targets every 6/12 hours.

The way to setup a cronjob is using the following command

crontab -e

Then you are going to give it the time interval based on which you want the script to run and give it the path. The below example is from my own setup:

0 */6 * * * /root/recon/scripts/stage3.sh >> /root/recon/scripts/logfile.log 2>&1

It runs stage3.sh(/root/recon/scripts/stage3.sh) every 6 hours (0 */6 * * *) and keeps the logs in thelogfile.log .

One important thing you don’t want to miss out is to add your path right before the cron job. Since cron jobs are running in a different environment than your own shell, you need to provide absolute pathing for every command. That is one hell of a headache if you don’t provide the path for the job. Believe me I learned this the hard way … 😆

You can do it simply by typing echo $PATH in your terminal, and then copy pasting it exactly on top of your cronjob. (see the pic below)

Then simply save the job and exit.

Congratulations! You have now successfully set up your own automation system.

Now this is by no means a sophisticated recon system, but the idea here is to give you the ability to start tinkering around with the code, changing it according to your own needs and slowly building your own automation step by step.

What I like to do is to create a discord server for my recon and create one text channel for each of the targets I have in my base/recon directory. Then I create a discord web hook for each and set it inside a provider-config.yaml file. This way every time the recon is done on a target I get a discord notification containing all the subdomains with a 200 status code and all their found metadata.

Feel free to add me on discord as well 😅

Just to recap: Make sure you have notify installed on your box. Then for each target you have under /root/recon create a provider-config.yaml file and paste your discord channels web hook in it in the following format:

discord:
- id: "WHATEVER"
discord_channel: "CHANNELS-NAME"
discord_username: "NO-IMPORTANT"
discord_format: "{{data}}"
discord_webhook_url: "WEBHOOK-HERE"

All the rest is taken care of automatically.

That’s it. I think we are done now. 😌

I gotta be honest it was way harder to write this article than I initially though. I had 3–4 unsuccessful attempts in previous days but I finally did it. So in order to make sure you will not run into any problem setting this up, as I did writing about it, I will upload this on my GitHub and link it down below. Make sure to check it as well.

As always if you followed all the way down to here, thank you very much. You are a hero.

Till next time…

Peace

Read Entire Article