Robin The Archaeologist of the Dark Web

Introduction to Robin: An AI-Powered Dark Web OSINT Tool

The speaker, a Senior Threat Research Analyst at Cybereason, introduces "Robin," an AI-powered Open Source Intelligence (OSINT) tool designed for the dark web. The tool aims to automate and streamline the process of gathering and analyzing information from dark web sources, which are crucial for cybersecurity threat intelligence.

The Need for Dark Web OSINT

Research on the dark web is essential due to the presence of cybercriminal forums and marketplaces that often operate on .onion domains. Traditional manual research is time-consuming and inefficient, necessitating the development of automated tools.

Existing Tools and Their Limitations

Currently, dark web OSINT tools are often fragmented and available on platforms like GitHub. These tools typically fall into categories such as search engines, link collectors, scanners, and crawlers. The primary challenge with these existing tools is their disjointed nature, requiring users to employ multiple tools for different stages of the research process: collecting links, verifying their availability, scraping content, and finally, report generation. This multi-step process is cumbersome and inefficient.

The Genesis of Robin

The speaker conceived Robin to consolidate these fragmented functionalities into a single, integrated tool. The integration of Large Language Models (LLMs) was a key consideration to simplify report generation. Instead of trying to combine existing tools written in various languages and architectures, the decision was made to build Robin from scratch.

Robin: Features and Functionality

Robin is an AI-powered dark web OSINT tool with four core functions:

Search: Locates relevant information on the dark web.

Filtering: Checks the availability of discovered links.

Scraping: Extracts data from the identified sources.

Analysis: Generates an analyst report.

The tool supports both Command Line Interface (CLI) and a User Interface (UI) for accessibility. It is compatible with major LLMs like OpenAI, Gemini, and Claude, and can also utilize local models via Ollama.

Robin's Architecture and AI Prompts

The tool's architecture begins with a user query, which is then refined by the first LLM prompt to ensure clarity and relevance. Robin searches across 15 onion search engines, identifies potential links, and filters them. A second LLM prompt is used to narrow down the results to a manageable number (e.g., top 20) that are most relevant to the original query before scraping. The final LLM prompt is used to generate a comprehensive investigation report, categorizing artifacts like names, emails, crypto addresses, and PII, and suggesting next steps.

Prompt Engineering and Trial-and-Error

Developing effective LLM prompts involved significant trial and error. Specific instructions were needed to ensure the LLM returned precise outputs, such as only the refined query or a list of top indices. For instance, refining results required providing the LLM with titles, URLs, and indices to prevent hallucinations and ensure relevance. The report generation prompt was extensively tuned, including defining what constitutes an "artifact" in a cybercrime context and specifying a desired output format that prioritizes valuable information for analysts.

Demo and Investigation Report

While a live demo was not feasible due to technical issues, video demonstrations of both the CLI and UI versions were presented. The UI is more intuitive, allowing users to input queries and initiate the process. The resulting report includes the input query, source links for verification, identified investigative artifacts (e.g., ransomware names, crypto addresses), key insights, and suggested next steps with example queries for further investigation. The output is generated in Markdown format for easy integration with other LLMs.

Future Updates

Planned updates for Robin include:

Continuity: Implementing a feature where previous searches and their outputs are saved and accessible within the UI, rather than just as separate files.

New LLM Support: Adding compatibility for the latest LLM models as they become available.

Saved Search: Incorporating a feature to save frequently used search queries within the UI.

Credits and Contact

The speaker credits Thomas for the inspiration to combine LLMs with dark web OSINT. Friends assisted with the logo design and UI development. The Robin tool is available on GitHub under the speaker's name, and feedback or improvement suggestions are welcomed via GitHub or social media.