Autonomous Video Hunter AI Agents for Real Time OSINT

Introduction to Autonomous Video Hunter

Kevin Delar Rossa, founder of cloud glue, a startup focused on agentic systems that understand video, presented the Autonomous Video Hunter. This AI agent is designed to extract intelligence from video data by identifying targets within random video clips and synthesizing information into structured data.

The Challenge of Video Data

Videos contain vast amounts of information, but extracting it is challenging. The emergence of advanced AI capabilities allows for the extraction of this information at rest. Potential targets range from public sources like connected cameras and satellite imagery to social media posts on platforms like TikTok and Reddit, where faces, license plates, and various topics can be identified.

Autonomous Video Hunter: A Proof of Concept

The Autonomous Video Hunter is a proof-of-concept system that helps users integrate multimodal context into their AI systems. It functions similarly to deep research AI agents, but instead of coding websites, it analyzes intelligence reports. The agent uses various lower-level vision and audio analysis tools, creates a plan of attack based on user-defined targets, executes these tools, and synthesizes a report.

Architecture Overview

The system involves:

Video Listener: Identifies and collects videos of interest (e.g., via social media scraping).

Video Processor: Extracts general context from videos at rest, processing frames and using vision-language models for dense descriptions.

Context Store: Makes the processed video data searchable and queryable.

Video Analysis Agent: Orchestrates various tools to fetch specific information from the video data.

Demo Scenarios Celebrity Tracking in Zurich

This demo illustrated how to find a celebrity, like Will Smith, in videos related to Zurich. It involved:

Social Media Intelligence Video Listener: A TikTok scraper using Gumloop to collect videos related to a specific hashtag and location, sending results via email.

Video Processing and Context Storage: (Conceptual) Processing videos to extract descriptions and store them for ad-hoc search.

Video Analysis Agent: An agent using Langraph to identify the target face, perform face detection, and generate a report detailing the celebrity's activities, timestamps, emotional state, and relevant video frames.

Logo Detection and Context Analysis

This demo focused on identifying a Starbucks logo in video streams and understanding the context in which it appears. The process involved:

Targeting: Specifying the Starbucks logo as the target image.

Agent Execution: The agent planned and executed tasks to find the logo in a set of videos.

Reporting: Summarizing the findings, including the number of videos containing the logo, its appearance on building facades, cups, and in work contexts (customer testimonials, employee demonstrations, promotional material).

Key Technological Components

Video Understanding: Powered by a company API, providing dense descriptions, speech transcription, and on-screen text for each scene.

Targeted Extraction Tools: Pre-configured tools to quickly identify logos, faces, speech, indoor/outdoor settings, and text within videos.

Face Matching: Utilizes the deepface library with models like RetinaFace for detection and VG Face for comparison, optimized for CPU.

Image Matching: Employs OpenCV and SIFT descriptors with RANSAC for matching arbitrary images like logos or building facades.

Zero-Shot Object Detection: Uses Hugging Face's OWL-v2 model to detect arbitrary concepts within images (e.g., flags, rockets, logos).

Deep Agent Framework (Langraph): Orchestrates the AI system, including planning, tool execution, memory management, and sub-agent interaction for facial, visual, and content analysis.

Advanced Applications Real-time Screen Recording and Visualization

Connecting a screen recorder to the Autonomous Video Hunter allowed for real-time summarization of on-screen activity. This involved:

Screen Recorder: Capturing short video segments.

Video Processing: Indexing the captured video for agent interaction.

Cloud Desktop Integration: Using the Model Context Protocol (MCP) to expose insights to Cloud Desktop for AI-driven visualizations. This enabled understanding conversations and visualizing topics, implementations, concerns, and knowledge graphs from discussions on platforms like Discord.

Potential Use Cases

Modern-day protest analysis

Identifying problematic drivers

Analyzing influencer content

Government and intelligence gathering

Challenges and Considerations

Cost vs. Accuracy: Higher accuracy often requires more expensive tools and GPUs, impacting inference speed.

CPU Optimization: The provided repository prioritizes CPU-runnable models for educational purposes, but these can be swapped for GPU-intensive state-of-the-art models.

Resources

Links to the presentation slides, GitHub repository, and Kevin Delar Rossa's contact information (Linktree, Twitter) were provided via a QR code and direct links.