Blog

OSINT Tools Library

A constantly updated list of web-based OSINT tools and techniques from across the open-source intelligence community, curated by Flashpoint

January 13, 2023

Welcome! Thank you for visiting our OSINT Tools Library. Here you will find information and links to some of the most useful OSINT tools and techniques from across the OSINT community. We encourage you to bookmark this page and refer back to it to support you in your investigations. This page is updated regularly, so be sure to check in from time to time!

Text-based Search

Search for source code

If you’re looking at a unique piece of malicious code within a script, it can help to check if that same unique snippet is found elsewhere in the code. PublicWWW is a public search engine for source code that will help you do exactly that. For a good example, find a small portion of the Magecart skimmer, put it into the search engine, and look at the results.

Find deep web data

90% of the internet isn’t indexed by search engines. This includes many internal links and deep websites. If you’re looking for a file that was shared by a company between two people or organizations, you most likely won’t be able to find it in Google or any other search engine. URLScan has many of these links indexed in their historical scans. If you type in a domain for any website into their search engine, you’ll find multiple links from that domain in the search results. It’s a great way to do deep web research without needing access or logins.

Search for Bitcoin addresses

Blockchain and cryptocurrency investigations have been a rather recent development in the OSINT space and are gaining relevance as more threat actors and scams begin using cryptocurrency as their transaction method of choice. If you’re looking into a crypto wallet that might be related to ransomware, blackmailing, fraud, etc., check out BitcoinAbuse. It’s a public database of addresses used for nefarious reasons. They also have an API if you’re looking to build an OSINT tool for crypto.

Power up your search

You.com is a powerful alternative search engine. It’s privacy-centric (like DuckDuckGo), but it has built-in functionality like social media searching, video searching, etc. and it separates the results by category. It could be the basis of a pretty awesome OSINT tool if leveraged correctly.

Search Pastebin dumps

Pastebin is a great resource for OSINT collection. A great tool to search for Pastebin data is PsbDmp. At the time of writing this, PsbDmp has 25990293 Pastebin dumps in its archive. You can easily use its search engine to check if the data point you’re investigating is in its archive.

Search Shodan

If you’re doing technical OSINT collection or investigation, you’ve probably heard of Shodan. At first glance, Shodan can be intimidating and it can be hard to find what you’re looking for. Fortunately, Jake Jarvis put together a guide with multiple use cases and examples for searching on Shodan. From prison pay phones to industrial control systems, there’s a ton of untapped potential inside of Shodan’s index.

Look up businesses using Secretary of State data

If you’re doing business-related investigations for entities in the United States, you’ll often be requested to pull Secretary of State documents for state filings of that business. These can be hard to track down if you visit each individual state website and pull the information that way. Fortunately, Cobalt Intelligence made a tool that pulls Secretary of State filings for businesses in all states from one search engine, saving a lot of time on business lookups.

Easily look up Skype profiles for data enrichment

Although Skype has declined in popularity in recent years, it contains a wealth of historical data useful for missing-person investigations and research into specific entities that were active when Skype was at its peak. Epieos created a tool that allows you to find valuable (mostly historical), information on Skype.

Collect website metadata

If collecting metadata is important to your OSINT workflow or are building a URL or domain-specific OSINT tool, finding a reliable, fast way of grabbing that information from a target URL can be tricky if that’s not your area of expertise. Fortunately, there’s an API called Website Metadata that can easily grab metadata for any URL you specify.

Reverse cleartext and hashed passwords

Similar to using Google dorks for Pastebin, you can also reverse cleartext and hashed passwords to find more information. If you have a cleartext password, use an online hashing tool like MD5 Hash Generator to turn it into a hash; if you have a hashed password, use hashes.com to turn it into cleartext.

From there, you can use tools like Dehashed or IntelX to turn those cleartext or hashed passwords into emails, phone numbers, names, etc. to further expand your ability to pivot across different data sources and build a full digital record.

Find leaked emails on Github

If you’re doing OSINT investigations into GitHub repositories, you might be interested to see who is all making commits to that project. Fortunately, finding email addresses in these commits from Github users is pretty simple. The hard part is grabbing all of that content, specifically on larger projects. GitRecon will do that for you in one motion.

Example: You find a GitHub repository that says it’s a PoC of an exploit for a software vulnerability. Upon further investigation, you find that it’s actually a honeypot for malware. You can use GitRecon to figure out who is involved if that information is available.

Use Reddit dorks

Reddit is a powerful platform for OSINT investigations. Because Reddit is anonymous, people share a wealth of information they wouldn’t have if they used their real names. Like Google, Reddit has a search engine that allows you to sort through their index; however, many people don’t know that they have a series of “dork”, or advanced operators, that make it easier to find what you’re looking for.

You might ask, why don’t I just use Google and the site:reddit.com operator? Well, Reddit has a few options that Google doesn’t such as flair:, subreddit:, author:, nsfw:, etc.

Uncover more Linkedin data

If you’re doing OSINT research and come across information from LinkedIn but your sock account isn’t a close enough connection to the target to get useful information, Revealin might help you uncover more. Exploiting a design flaw in LinkedIn, Revealin gives you the full name of a person when it only shows only an initial without needing to expand your network.

Subscribe to our threat intelligence newsletter

Interested to see top news from Flashpoint hit your inbox directly? Subscribe to our newsletter to receive curated content on a regular basis.

Think about the methods this tool uses. The more familiar you are with these concepts, the more you can find your own internet easter eggs and level up your OSINT game.

Gather OSINT from hashes on Pastebin

If you’re looking to collect a series of hashes values, such as passwords or IOCs, you can use a specific Google dork to locate them in a website like Pastebin.

site:pastebin.com intext:“SHA256”

You can change out SHA256 with any other algorithm including: MD5, SHA1, MySQL, NTLM, or SHA512.

Once you have a hash, you can check it against places like VirusTotal to see if it’s malicious or not.

Look up phone numbers with Phomber

Phomber is an easy-to-use, free Python tool for reverse phone number lookups.

Analysis

Investigate URL shortened links for OSINT

If you’re doing OSINT data collection at scale, you’re likely to come across a lot of shortened links. If you try to manually investigate each one, which is a viable method, it’s going to take a lot of time. Fortunately, there’s an open-source tool that will expand those links for you at scale. Links like bit.ly, adf.ly, lnx.lu, linkbucks.com, and adfoc.us can be expanded automatically with urlExpander. Whether you have a million links or a hundred, be sure to check it out.

Visually investigate cryptocurrency

If you’re tracking a lot of illicit actors, you’ve probably had some exposure to cryptocurrency investigations. A common challenge in crypto investigations is making sense of extremely large datasets. Many tools will show you a single connection and rely on you importing that information into a different tool like Maltego to form visual connections. Breadcrumbs is a blockchain analytics platform accessible to everyone. It offers a range of tools for investigating, monitoring, tracking, and sharing relevant information on blockchain transactions. It solves the problem of connecting the dots between transactions and wallets. The free tier provides limited access to almost all functions including monitoring and graph creation. Whether you’re new to this space or looking for a budget-friendly solution, this is worth a look.

Leverage this domain OSINT multi-tool

For investigations, it’s helpful to keep a running list of found data points into as many tools and resources as possible. Sometimes a slight change in configuration from one tool to another can offer a different angle or set of information. Investigator is an OSINT multi-tool for domains. It has a variety of modules to help dig deeper into specific categories. The web app format makes it possible to use with limited technical knowledge or in controlled environments.

Try Bitcrook

Bitcrook is an OSINT multi-tool that might check a couple of your boxes in a single script. Bitcrook is unique because it factors in court cases from case.law and data from Melissa. It also has standard modules like IP and username lookup.

Automation & Efficiency

Shorten repetitive tasks in terminal

If you spend a lot of time in the terminal for OSINT collection and analysis, you probably run the same commands over and over. There’s the classic meme of pressing the up arrow a million times until you find the command you were looking for. What if you could create a shortened version of that command to reference quickly and use at will? PCMD lets you do that. Type in a long, specific command, save it as a shortcut with PCMD, and use it over and over without having to type it out.

Discover the best OSINT threads

Some of the best OSINT techniques and workflows can be found on Twitter. The issue with Twitter is that they break them up into multiple messages so discovering threads can be difficult if the post doesn’t specifically say it’s a thread. If you’ve read a thread on Twitter before, you have likely seen people tagging apps like ThreadReaderApp to consolidate the thread into one post. What you might not know is that you can query all threads they create on their website just by searching for #OSINT.

Collect onion URLs automatically

Because the dark web doesn’t have a free, unified search engine like Google, OSINT researchers who don’t have access to a service like Flashpoint have to either hunt for it manually or rely on static databases like dark.fail to find more. OnionSearch aggregates the search results of multiple dark web search engines that index these links allowing you to search for things in multiple places at once. If you’re looking for a free tool to discover new dark web URLs, OnionSearch is worth checking out.

Remove web elements more efficiently

Many OSINT workflows require adding and removing web elements to reveal certain details. Snip allows you to permanently remove certain web elements so that you never see them again. This is very useful for frequently visited websites.

Build workflows with Automa

This Chrome extension allows you to create a workflow using a drag-and-drop interface and save it as a playbook to replay at any time. From auto-fill forms, to repetitive tasks, to taking screenshots, to scraping data from a website, Automa is pretty flexible. If you want to automate some of your OSINT workflows, give this a spin.

Automate scraping with Browse AI

If you’ve ever used Instant Data Scraper, this tool will be very familiar to you. What Browse AI does that Instant Data Scraper doesn’t is set up automated scraping tasks and export the results automatically to a Google Sheet. If you want a no-code way of scraping websites or want to build quick proofs on concepts before committing to writing a custom web scraper, give this a shot.

Automate writing tasks with Tango

For writers of blog posts or documentation articles, Tango is a very handy tool. Tango allows you to write step-by-step tutorials with screenshots in a few seconds. It works by recording your actions on the screen while enabled and converting that into a process-based how-to guide when the recording stops.

Manipulate datasets from Terminal

OSINT practitioners frequently run across TXT and CSV files. Often, the formatting for CSV files is not compatible with your investigation and you need to remove certain columns, add others, change column titles, etc. If you’re looking to remove all columns except for a select few, use this trick:

$ cut -d, -f<column number> –complement input.csv > output.csv

Example:

$ cut -d, -f4 –complement osint.csv > oshint.csv

This will remove the fourth column of osint.csv and save it to oshint.csv. If you remove –complement, you’ll remove all columns except for column 4.

To learn more about this Linux command, check out this article.

Search through large datasets with ease

Leaked credential databases are a key component to investigations into individuals with a small digital footprint. Sometimes, these datasets can be hundreds of gigabytes in size. Opening them in any software or browser and looking for applicable information can be a struggle.

Enter Ripgrep. It allows you to recursively look through a database—regardless of size—for a specific data point, and see if it exists. There’s a bit of a learning curve so you might want to check out this video before getting started.

Data Organization

Archive digital evidence

Save as MHTML is useful, but is limited in the fact that it can only capture one page at a time. If you want to download multiple pages or an entire website as MHTML. Enter SinglePage. SinglePage allows you to capture the current tab, a specific selection, multiple tabs, highlighted text, etc. It also has a SHA256 module to hash your evidence.

Manage CSV files with CSVKit

If you’re dealing with spreadsheets on a regular basis, it can be hard to manage really large datasets. CSVKit is a suite of command-line tools for converting to and working with CSV files. It allows you to convert file types (like CSV to JSON), select certain types of information from the dataset, or import the data to PostGreSQL. The lookup capability is especially useful. 

Example: csvgrep -c phone_number -r “555-555-\d{4}” data.csv > new.csv

You can lookup a phone number and take it from the main dataset and add it to a new one.

Convert URLs to PDFs

PDF my URL allows you to download a website to a PDF for offline viewing. You can convert a single page or an entire website to a PDF and remain anonymous because the app’s IP address requests the information from the website, not yours.

Image & Video Tools

Get better Instagram data

A common method for finding connections on social media is by looking at followers, followings, and engagements (comments, likes, etc.). Instagram is becoming more and more difficult to collect data from. Sterra allows you to get information about an Instagram account through its followers and followings. Here are some things it considers:

  • 📊 A probability function to determine the close social circle of your target
  • 📥 Export of the followers / following lists (with their details) to excel and csv

Screenshot OCR for text and QR codes

OCR, or optical character recognition, is a great way to capture OSINT from images without having to manually review them. There are many OCR tools out there, but they’re often inaccurate or buggy. TextCapture allows you to capture text from images and QR codes—including screenshots—and will show you where the URL links to. TextCapture is available for $2.99 and is compatible with Mac computers.

Capture YouTube metadata at scale

If you’re collecting video OSINT, it’s important to document the metadata. For YouTube specifically, YouTube Metadata Bulk allows you to capture metadata for a single video or for a list of videos. That metadata can be exported as TXT, JSON, CSV, or PNG. Entities like tags, geolocation, links, and upload times can be grabbed instantly.

Build your own reverse image search

Reverse image searching is a major technique used in OSINT collection and analysis. The issue with many reverse image tools is that you’re reliant on the algorithms of Google, Yandex, etc. to find a positive match. While these tools are often reliable, wouldn’t it be great if you could build your own? Well, you can. This Github project gives provides the foundation for building your own reverse image tool. It’ll find exact matches and near matches out of the box; however, you could combine it with things like facial recognition to make it more sophisticated and accurate.

Scan and detect deepfake videos

Deepfake videos are becoming a huge problem on the internet. From Tom Cruise, to Barack Obama’s public service announcements, the future of disinformation looks pretty grim with the ability to create fake videos that fool the masses.

Deepware provides a web app and API that helps identify deepfake videos. Whether it’s stored locally or hosted on a URL online, you can send it to Deepware for review. If you want to add this capability to an OSINT tool, the API is very straightforward and easy to use.

Know your Yandex hacks

Yandex is a fantastic tool for reverse image searching. OSINT researcher Irina Shamaeva discovered that using a VPN to manipulate your IP address can improve your results. See Irina Shamaeva’s Tweet with examples.

Try Facebook’s built-in OCR

You can search for specific alphanumeric strings, such as license plate numbers on Facebook and comb through images to find a match. Facebook automatically tries to use OCR within its images for displaying relevant results. I tried a search for “MWP-46-99” (a license plate) and got the following result:

Geolocation

Improve your geolocation skills

GeoGuessr has become the official primer for developing your geolocation skills. It gives you continuous exposure to different scenarios to train your deduction skills. Give it a read or three and watch your scores improve. Take what you learn on GeoGuessr and apply it to real challenges.

Bootstrap geolocation

AllYouCanRead is a collection of newspapers that are local to a specific city, county, etc. If you collect all of the newspapers you’re interested in and pull their RSS feed, you can use keyword filters to look for keywords of locations within publications of papers local to that location. So looking for “San Francisco” in a San Francisco-based newspaper is likely to yield results from that area. Scale that across multiple cities and countries and you have bootstrapped geolocation.

“Open-source information is going to be of incalculable value both to our intelligence community and military in the future.”

Glenn gerstell, former general counsel of the national security agency

Identity Protection & Safety

Check data points against spam

If you’re investigating a lot of data, specifically IP addresses, emails, or usernames, make sure to check Stop Forum Spam to see if any of the data points you’re dealing with are found there. If you find a positive hit on any of your searches, you can pivot on the found data points to find more information. Example: If you have an IP address and get a match, it’ll give you an email and a username. You can reverse those back into Stop Forum Spam or any other OSINT tool you use.

Create credit cards for your sock puppet

When building a sock puppet (alternate persona), it’s important to create as complete a profile as possible. Many websites require a credit card to sign up. Using Card Generator, you can make a card tied specifically to your sock puppet and maintain consistency across the web without compromising your identity.

Education & Training

Level up your Telegram investigations

If you’re collecting data from Telegram, there are a ton of tools you can use to find new users, groups, channels, etc. Awesome Telegram OSINT is a collection of tools, tactics, and techniques for extracting information from Telegram. If you’re a seasoned Telegram collector or just getting started, this is your one-stop shop.

WEBINAR REPLAY

Telegram for OSINT: Level Up Your Investigations

Read OSINT case studies

One of the best ways to learn about new tools, tactics, and techniques is to read case studies. “Exposing the Invisible” has a list of case studies on its website that can inspire you in your own investigations.

Check out more OSINT resources

For an extensive list of OSINT resources, check out Oh Shint on GitHub.

OSINT Trainers:

Begin your free trial today.