Creating a recon database with Flask, MongoDB, REST Api and ChatGPT — Part One

Hellow world!

My name is Ali, I am a web application security researcher/enthusiast and today I will be writing my very first blog post in here.

I am also interested in bug bounty hunting and I’ve been following the infosec community on medium for a long time. I have learned a ton here and that’s why I decided to share something back to the community.

If you are also new to bug bounty, you probably too experienced the confusion and pain of not knowing on how to store your recon data properly.

I sure did, and still do to some extent. So my goal here is to figure out a way to solve that issue and share it here with you. For this I will be setting a MongoDB which holds all the recon data (for now it will be only the scope, subdomains, status codes, and update time) and we’ll use Flask and REST Api to be able to push/fetch data easily to our db.

This is by no means a step-by-step guide (but it could be I guess), I am just going to document/journal what I did. I hope you will find it useful.

NOTE: I will be using “booking.com” (wildcard) which is public program on HackerOne to show case the data in my recon database.

After creating a MongoDB cluster in Atalas (https://www.mongodb.com/) I did the following:

Created a database named assets and the first collection booking

Created an admin user with a password, gave it an atlasAdmin role for highest permission

Under the “Network Access” added IP address of “0.0.0.0/0” so that I can access my db from any IP

Connecting to your database

Using Python:

Using Mongo Shell:

Using Compass:

Adding some dummy records to the DB

For start I just wanted to get comfortable with queries so I started with a simple list:

The logic is, let’s say for example I’m running a tool like subfinder or amass and there’s an output text file with a list of subs. I wanted to see how I can insert those subs into a separate document each, with specific additional keys.

For now, for the sake of simplicity I will just go with the following keys: subdomain (domain address), org (organization/scope), alive(is the host up?), status (http status code), updated (last updated date/time).

For the list above you can insert it into a connection in 2 ways:

insert_one()

insert_many()

an answer like this means the operation was successful:

Don’t forget to define your db and collection before anything else:

Next step for me is to figure out a way to insert a huge text file containing a list of subdomains, and only subdomains (no org name, status code, or anything really) into the collection with appropriate keys and pairs. But how…?

NOTE: I guess for now I will just forget about status code and come back to fix this later.

ChatGPT to the rescue:

I have a text file containing a list of subdomains in each separate line. How can I insert that text file into a python list that can be used to insert into a mongo db database?

The code needed a little adjustment but we’re fine.

I commented out collection.insert_one({"subdomain": subdomain}) and added print to see the output and it is what I wanted. Now the question is, how can I add “org”, “status”, “update”, etc to the list.

hmm…

After trying a couple of minutes: (no ChatGPT for this one XD)

master_list output looks like this:

→ the update time needs a little work, but it will do for now

and finally:

Next step I guess would be to make sure that the subdomain key is also unique, in order to avoid adding the same subdomain into the collection over and over again later in our automation.

Adding the following line will take care of that:

Now if I try to reinsert an existing sub into the collection I will get an error:

Nice!

If you read until here, thank you very much for following along. That’s it for part one.

Stay tuned for the second part.