IVRE: A Versatile Network Reconnaissance Framework

We’ve reviewed a lot of open source tools at SecurityTrails over the past few months and today’s the day for another new tool. Read on, and we’ll review the open source tool IVRE, which is supported by the French Atomic Energy Commission.

What is IVRE?

IVRE is an advanced network reconnaissance framework built with Python and MongoDB. IVRE relies on a number of packages (Nmap, Masscan, Zmap, Zeek, Argus, Nfdump, ZDNS) and brings them all together through the CLI and a web interface.

A researcher can perform active or passive reconnaissance of a network, then browse and perform further analysis via the CLI or web interface.

IVRE can be used for standard red and blue team investigations, creating your own private Shodan/Censys, or even your own passive DNS service.

Main features:

Framework: IVRE doesn’t re-invent the wheel—it relies on existing software by combining them and building on that foundation Web interface: The web-based GUI makes filtering and exploring data a breeze Visualizations: Adding to the Web interface, there are multiple visualizations and sorting mechanisms that enable greater analysis (including visualizing network flows) API: If you find any limitations within the framework, you can use the API to export your data and visualize/analyse it elsewhere

Who is IVRE for?

Just like Masscan and Jok3r, IVRE is great for both offensive red team investigations as well as defensive research.

As mentioned above, you can also use IVRE to make your own private Shodan or passive DNS service. The passive service is great for organizations of all types to monitor their networks and see the flow of network activity to detect any anomalous behaviour.

Installing IVRE on Linux

IVRE offers many installation options. Based on the documentation, IVRE supports Docker and is packaged on a number of Linux distributions, as you can see here.

However, this tool relies on a number of different packages that require their own installation. These include MongoDB, Nmap, Zmap, Masscan, Apache/Nginx (optional) and many others. We’ll attempt to follow a lean installation by only using required packages.

It’s always important to use some type of sandboxing environment when installing new software. You could opt for a virtual machine (VM), container or a remote test server. We used Ubuntu 20.04 for this review, and any commands used here should apply to Debian-based distros (and with a few minor tweaks, to other distros as well).

First, we install some system-level packages:

sudo apt install mongodb nmap zmap gcc make libpcap-dev python3-virtualenv

The command above will install two scanners (Nmap and ZMap) and the other packages will be used to build Masscan, which can be done as follows:

git clone https://github.com/robertdavidgraham/masscan cd masscan make

You can see our full installation instructions for Masscan here.

Now we’ll create a Python virtual environment and install IVRE:

mkdir IVRE-install virtualenv --python=python3.8 IVRE-install/ cd IVRE-install/ source bin/activate pip install ivre

You should now have IVRE installed system-wide, which you can confirm by running:

ivre --help

Now we can initialize IVRE by running the following commands:

ivre ipinfo --init ivre scancli --init ivre view --init ivre flowcli --init sudo ivre runscansagentdb --init

This will remove existing data from the database. These commands can be used in between investigations to clean up the data.

The last step will be to fetch IP data:

ivre ipdata --download

This command fetches IP data from the IVRE website and Maxmind. This data is required if you want to scan/list IP addresses from an AS or per country.

Testing/Usage

Our first test will be to run a scan on 100 random hosts by running five parallel Nmap processes:

ivre runscans --routable --limit 100 --output=XMLFork --processes 5

Now we can import the results into the database and create a view of it:

ivre scan2db -c ROUTABLE-001 -s MySource -r scans/ROUTABLE/up ivre db2view nmap

You can verify that your scan ran successfully by the output of the ‘scan2db’ command, which should show something like:

INFO:ivre:9 results imported.

We can do a quick analysis of the results by running:

ivre scancli --port 22

But this isn’t very useful unless you’re comfortable with the CLI. So let’s see what the web interface shows us instead. Start up the basic httpd server (ensure that you have firewall protection if you’re running the testing server remotely) as shown:

ivre httpd --bind-address 0.0.0.0

The results in the web interface are much more informative, showing us which ports are in use, including which are open and which are closed. We’re also shown the location of the IP address. Now let’s add a filter to see how the application handles it. Filtering by Port 21:

This is also quite telling, providing a lot of valuable information about the port (which is FTP) and the version of the software being used on the port.

Now we clear up all the existing data, and attempt to run a similar scan on a specific subnet:

rm -rf scans/ROUTABLE/*

If you want to start a new active reconnaissance investigation without any existing data in the database or view, you can remove the data by running the following two commands:

ivre scancli --init ivre view --init

We’ll run our scan against a subnet that has some malvertising IPs we’re tracking:

ivre runscans --routable --categories MALVERT --network 139.45.192.0/20 --output=XMLFork --processes 5

In the command above, we:

Created our own category ‘MALVERT’, which will appear in /scans/MALVERT/ Scanned an entire subnet under ‘network’ Used only five concurrent Nmap processes

The first thing we discovered when running this modified scan was how slow the scan process took for a relatively small subnet. This may be because of the amount of processes we used (with five being too low). The total run-time took roughly 3+ hours.

We’ll now insert the data into the database so that we can do further investigations:

ivre scan2db -c MALVERT -s test2 -r scans/MALVERT/up ivre db2view nmap

Running:

ivre scancli --count

This will output the total amount of results currently in the database from all scans (our total is now 10).

Using the ‘Explore’ feature, we can see some of the visualizations IVRE has to offer. IPs and ports:

The following is the address space for the 10 IPs:

Depending on how your investigation is conducted, you will either see a concentration of address space/IPs (when the investigation is looking at a specific subnet) or a scattered plot (when looking at the IPs of a large technology company like Google).

IVRE also supports plotting graphs directly using matplotlib (we needed to install the software on its own, which can be done by using this quick command: pip install matplotlib). For example:

ivre plotdb --category ROUTABLE --2d

…would plot a 2-dimensional graph of the results from the ROUTABLE category.

For another check, we can perform a domain DNS audit by running:

ivre auditdom facebook.com > fb.xml

This will give us an Nmap-style XML output that provides DNS information.

While we set up this test for active reconnaissance, IVRE is also useful for passive reconnaissance using tools like Zeek. You can connect your logfile to IVRE and continuously feed the data from the logs to IVRE. More information about that can be found here.

Another really great feature of IVRE is the ability to run remote ‘agents’. With ‘agents’ (or workers) you can run a minimal amount of tools and sync up the IVRE install to the ‘agent’ via rsync. This is handy in situations where you’re running large scans and want a distributed network of agents to handle the scanning without requiring your host IVRE instance to run any scans of its own.

Conclusion

IVRE requires a slight learning curve for installation and use, but after spending only a few days with this tool, we found it really great and informative. It combines the power of existing tools (Nmap, Masscan, Zeek and others) and adds helpful visualizations to enable better analysis and investigation.

There is, however, another option: If you’re looking for a tool that offers what IVRE does, plus a whole lot more, SecurityTrails offers an easy to use API to integrate critical infosec reconnaissance data within your own apps, or SurfaceBrowser™, a web-based interface that will instantly enrich your intelligence research tasks. Get in touch with us for more information.