Urlscan.io API unwittingly leaks sensitive URLs, data

Public listings have made sensitive data searchable due to misconfigured third-party services

Researchers have warned of enterprise software misconfigurations leading to the leak of sensitive records on urlscan.io.

Urlscan.io is a website scan and analysis engine. The system accepts URL submissions and generates a wealth of data, including domains, IPs, DOM information, and cookies, alongside screenshots.

The developers say the engine’s purpose is to allow “anyone to easily and confidently analyze unknown and potentially malicious websites”. Urlscan.io supports many enterprise customers and open source projects, and an API is provided to integrate these checks into third-party products.

GitHub warning

In a blog post published today (November 2), Positive Security said the urlscan API came to its attention due to an email sent by GitHub in February, warning customers that GitHub Pages URLs had been accidentally leaked via a third party during metadata analysis.

“With the type of integration of this API (for example via a security tool that scans every incoming email and performs a urlscan on all links), and the amount of data in the database, there is a wide variety of sensitive data that can be searched for and retrieved by an anonymous user,” the researchers say.

RECOMMENDED GitHub patches bug that could allow access to another user’s repo

Upon further investigation, Positive Security found that this could include urlscan.io dorks, password reset links, setup pages, Telegram bots, DocuSign signing requests, meeting invitations, package tracking links, and PayPal invoices.

Pingbacks to leaked email addresses appeared to show that misconfigured security tools that submitted links received via email as public scans to urlscan.io were the culprits.

For example, many API integrations utilized generic python-requests/2.X.Y user agents that ignored account visibility settings, thus allowing scans to be wrongfully submitted as public.

SOAR misconfiguration

Positive Security reached out to numerous leaked email addresses and there was only one response – from an organization that sent an employee a DocuSign link to their work contract and subsequently launched an investigation.

The employer found that a misconfiguration of their Security Orchestration, Automation, and Response (SOAR) playbook, which was integrated with urlscan.io, was at fault.

Positive Security also examined historic urlscan.io information, uncovering misconfigured clients that could be abused by scraping the system for email addresses or sending phishing emails.

If password resets from other web services were requested, it was possible to check for corresponding domain search results and URL content, leading to account takeovers.

Speaking to The Daily Swig, Fabian Bräunlein, co-founder of Positive Security said that this attack vector could be triggered “for personal services like banking or social media or company services such as for popular SaaS or custom applications.

“For many SaaS providers, access to an email address with a certain domain is already sufficient to gain access to internal company data (e.g. chats or code repositories),” Bräunlein added. “In such a case, an attacker does not even need to take over existing accounts but can just create new accounts at interesting services.”

Urlscan overhaul

Once the impact of the issue’s assessment was completed in July, Positive Security reported its findings to urlscan.io. As a result, the cybersecurity firm and urlscan.io developers worked together to address the problems uncovered, leading to the release of a new engine version later in the month.

The improved software includes an enhanced scan visibility interface and team-wide visibility settings.

Urlscan.io subsequently also published Scan Visibility Best Practices, which explain the security benefits and risks posed by three visibility settings users choose between when submitting a URL: ‘Public’, ‘Unlisted’, and ‘Private’.

Read more of the latest cybersecurity research news

Urlscan.io has also contacted customers who have submitted vast amounts of public scans and begun reviewing third-party SOAR tool integrations. Finally, the developers have added deletion rules, highlighted visibility settings in the user interface, and implemented a report button to deactivate problematic search results.

“Security teams that run a SOAR platform must make sure that no sensitive data is leaked to the public via integrations of third-party services,” Bräunlein commented.

Urlscan GmbH CEO Johannes Gilger told The Daily Swig: “We welcome the research performed by Positive Security and appreciate their professional conduct while working with us to identify the scope and source of these inadvertent information leaks.

“We have improved the visibility of the relevant settings on our platform, we have educated our users about the issue through a dedicated blog post and we continue to work with third party automation providers to ensure adherence to safe default behaviors.

“A platform like urlscan will always carry the risk of unintended information disclosure due to the nature of its operation, so we take every available measure to minimize the likelihood of these things happening.”

DON’T FORGET TO READ OpenSSL vulnerability downgraded to ‘high’ severity