ARTIF - An Advanced Real Time Threat Intelligence Framework To Identify Threats And Malicious Web Traffic On The Basis Of IP Reputation And Historical Data.

ARTIF is a new advanced real time threat intelligence framework built that adds another abstraction layer on the top of MISP to identify threats and malicious web traffic on the basis of IP reputation and historical data. It also performs automatic enrichment and threat scoring by collecting, processing and correlating observables based on different factors.

Key features of ARTIF includes:-

Scoring System: Enriches IP addresses with threat metadata including a threat score which can act as a threshold value for security teams to take action on. Containerized: ARTIF is deployed using containers and hence provides ease for deployment. Modular Architecture: The project is plugin-based and can be extended easily by just modifyng threat feeds in MISP. There would be no downtime to the actual service as these would be in-line update. Alerting: Extended feature providng seamless integration with slack for active alerting. It also provides better attack profiling and visualization.

Some use Cases:-

Threat Detection Logging and Monitoring User profiling Alerting automation

Why use ARTIF? It is real time Threat Intel Framework that can help identify malicious IPs even though they are not present in the MISP. This helps organizations to set up first layer of defense by providing transparency over malicious web traffic reaching their servers.

By default 52 configured open source threat feeds with a database of 0.7M IP addresses. Has latency of ~ 180 ms > 10x faster than commercial products. Historical IPs are stored for analysis and used in scoring on the basis of past records and patterns. Adds score to each IP in addition to other metadata.

Pre-Requisites

What is MISP and how to install?

Taken from MISP : MISP is an open source software solution for collecting, storing, distributing and sharing cyber security indicators and threats about cyber security incidents analysis and malware analysis. MISP is designed by and for incident analysts, security and ICT professionals or malware reversers to support their day-to-day operations to share structured information efficiently.

MISP can be installed using source code or their pre-built AWS images. More information about MISP installation can be found at their website.

We need to have subscription to maxmind in order to populate meta-data for the IP. To add your sub key edit docker-compose.yaml

Installation

Clone the repository using git or download the zip file. Build the docker by changing the working directory to the ARTIF folder and start the docker containers for it. Setup MISP and visit the MISP dashboard and get the MISP key. Now edit the config.yaml and add the MISP_KEY and the MISP_URL values. Here MISP_KEY will be your API key for MISP and MISP_URL will be the URL where MISP is hosted.

Below is a config example.yaml for your reference. Simply replace the corresponding values with your values.

Now run the below command with the full absolute path to update_check.py file with -s argument Now run the same command without -s argument. Add the crontab using Django's inbuilt support by running the below command Start the Django server from the ip_rep directory.

This will open the port 8000 which can be used to get the metadata for the IP addresses. You can now try getting the threat score for any particular IP.

The output being

The score represents a lower risk for the IP as the threat score is high. The higher the score the lesser the non-malicious IP it is.

Note: Instance with 8GB RAM is recommended for ARTIF installation.

Setting up docker containers

Starting ARTIF

Adding Custom feeds

ARTIF supports synchronization with MISP. It syncs MISP feeds, picking the most recent config from settings.yaml and all the new events modified from MISP are reflected in settings.yaml. To add new IP, just login to MISP and click on add feed page. Once the IP is added, the cronjob will pick it up according to its schedule and will be processed automatically.

Technical Details

ARTIF being a threat framework, is highly useful for visibility inside organization traffic. It is completely written in python and collects intelligence on an IP from various feeds. It then sends this data to a correlation engine which generates a threat score where historical data is also one of the factors taken into account for threat score calculation.

Every time a new IP hits the service a celery worker is assigned the task to update its score in the database by coordinating with the correlation engine, which in turn gathers data from multiple sources. To ensure that the data isn’t stale (default ‘stale time’ value is 24hrs), we run a worker for each IP in the database which hasn’t been updated for the past ‘stale time’. As with all the other parameters this value is configurable too. Since the threat score calculation is the key to all of this, we’ve laid a lot of emphasis on it. Apart from the well-known threat feeds such as MISP, Cortex, Alien Vault, VirusTotal, and popular blacklist of IPs, we can also add custom feeds according to business and security use case, where customer loyalty can also be used as a parameter for threat score calculation.

For ease of configuration, ARTIF needs the following input to run itself:

Feeds in MISP instance A MISP key for communicating with the MISP instance A MISP URL to connect to MISP instance using the MISP key

Upon running, the data is processed and stored on a MongoDB container. MongoDB container contains 3 important databases which stores information about the IP in the feed as well as its metadata for eg. country/ASN, Org, etc. If the IP is not found in the database that means it’s a new IP reaching the server and whose information is not present in MISP. We then calculate a risk score for the particular IP using an algorithm and various other parameters like geolocation, ASN and Org. More information about the scoring engine can be found here.

ARTIF has additional functionality to keep track of old feeds. The default configuration replenishes the latest feeds every 24 hrs. After 24 hrs the old feed is moved to another collection and the latest ones are added. For ease of operation, we have added a field called “historical” in the output which tells if the IP was historically bad. A false value indicates that IP has been recently added to the feeds while a True value suggests that IP was already present in older feeds and hence its a historically malicious IP.

All the historical IP will be removed from database after 7 days by default.

 

Usage/Examples

You need to invoke ARTIF using update_check.py which is the backbone for ARTIF.

You can also look at the cron job by running the below command -

This will auto-update the feed. By default every 24 hrs it will check the MISP for the latest feed and replenish the DB with new IP from feeds. IPs older than 7 days are also removed by the scheduler.

FAQ

I am trying to run update_check.py and I am getting a Tag-based error saying tags are already present/removed. How to resolve it?

You can safely ignore it since this happens when the Tag is already there and we are trying to add it. In that case, it will give an error saying above, and hence the tag removal can’t be done. MISP doesn't have a way to catch those exceptions and hence we have to rely on this way.

This is taking too much time. By when will it finish?

This indeed takes time sometimes from 30 mins to 90 mins depending upon your internet speed and RAM. This is basically downloading all the feeds into MongoDB and processing those and hence it will take time and also it is a one-time effort.

I followed the steps mentioned but the MongoDB is still not populated.

Solution: In such cases, run the update_check.py file without -s argument. This would download all the IPs from the feed. This error generally happens when you abruptly interrupt the execution of update_check.py during its initial phase.

Future Roadmap

Application Firewall integration support with ARTIF for automatic blocking malicious IPs based on their threat score. Metric visibility over UI dashboard in form of graphical representation to better visualize the threats.

Contributing

Contributions are always welcome! See contributing.md for ways to get started. Please adhere to this project's code of conduct.

Communicating with Team

The easiest way to communicate with the team is via GitHub Issues.

Please file new issues, feature requests, and suggestions, but search for similar open/closed pre-existing issues before creating a new issue.

Authors

Divyanshu Mehta Avinash Jain Govind Menon

Acknowledgements

MISP Maxmind