NVIDIA Released DeepStream 9.1: Bringing Agentic AI to Vision AI With 13 Skills and Multi-View 3D Tracking

NVIDIA just released DeepStream 9.1. The update targets a persistent problem in video analytics. Tracking one object across many cameras traditionally requires manual camera calibration and complicated calculations. DeepStream 9.1 addresses this with two additions: Multi-View 3D Tracking (MV3DT) and AutoMagicCalib (AMC). Both ship as agentic skills for coding agents. As a result, developers move from concept to a running pipeline faster.

What is DeepStream 9.1

To understand the update, start with the base platform. DeepStream is NVIDIA’s streaming analytics toolkit for AI-based video and image understanding. It provides a GStreamer-based framework for multi-stream, multi-model inference on NVIDIA GPUs. Pipelines combine hardware-accelerated decoding and encoding, TensorRT inference, object tracking, and message-broker integration.

Building on that base, version 9.1 adds five notable items:

1. 13 agentic skills for coding agents.
2. The MV3DT skill for cross-camera tracking.
3. The AMC skill for automatic calibration.
4. NVIDIA JetPack 7.2 support for Jetson Orin and Thor edge devices.
5. A unified open-source GitHub repository under CC-BY-4.0 AND Apache-2.0.

How MV3DT Tracks Objects Across Cameras

Among those additions, MV3DT is the main skill, so consider how it works. At its core, MV3DT projects detections from multiple calibrated cameras into a shared 3D coordinate system. It then associates observations of the same object across camera views. Finally, it assigns one globally consistent object ID.

Concretely, the data flow runs in four stages. For detection, each camera stream runs an object detector. MV3DT supports three models out of the box:

• PeopleNetTransformer: a transformer-based people detector, the default for pedestrian scenes.
• PeopleNet v2.6.3: a high-efficiency detector based on the DetectNet_v2 architecture.
• RT-DETR 2D: a multi-class detector for pedestrians, transporters, and forklifts.

Next, for monocular 3D perception, each camera uses a 3×4 projection matrix stored in a YAML calibration file. This back-projects 2D bounding boxes into 3D world-space coordinates using a ground-plane assumption. Then, for multi-view association, the tracker shares tracklets using Message Queuing Telemetry Transport (MQTT). MQTT is a lightweight pub/sub messaging protocol. When two cameras observe the same person, it matches tracklets by proximity in 3D world space.

After association, results stream out in three forms. The On-Screen Display (OSD) shows a tiled grid with 2D and 3D bounding boxes. The Bird’s-Eye View (BEV) renders a top-down trajectory map. Kafka messaging delivers per-frame protobuf metadata, including sensor ID, object ID, and 3D bounding box.

How AutoMagicCalib Removes Manual Setup

MV3DT depends on calibrated cameras, which traditionally means checkerboards and downtime. Instead, AMC calibrates a network by analyzing tracked objects in existing video files or streams. It estimates each camera’s intrinsic parameters (focal length, principal point, lens distortion). It also estimates extrinsic parameters (rotation, translation, world position).

Under the hood, the pipeline runs five stages. These are per-camera trajectory extraction, single-view rectification, multi-view tracklet matching, bundle adjustment, and optional VGGT refinement. VGGT (Visual Geometry Grounded Transformer) helps when object movement is limited. AMC runs as a microservice with REST APIs and a web interface. Users supply only a layout image and a few alignment points.

The Agentic Skills Workflow

With MV3DT and AMC defined, the delivery mechanism is the skills themselves. Rather than editing configuration files, you describe intent in natural language. The skills work with Claude Code, Codex, Cursor, and similar agents. Setup is short:

git clone https://github.com/NVIDIA/DeepStream.git
cd DeepStream
# Copy skills into your agent's skill directory (Codex shown)
mkdir -p ~/.codex/skills
cp -r skills/* ~/.codex/skills/

After launching the agent, a single prompt runs the reference app:

deploy mv3dt on the 12-camera sample dataset

From there, the MV3DT skill validates prerequisites, pulls the container, and installs Kafka and Mosquitto broker services. It also downloads model weights, generates the pipeline config, and launches tracking. Notably, if calibration files are missing, it triggers the AMC skills automatically.

DeepStream 9.0 vs 9.1

For context, the table below shows what changed between releases.

Use Cases With Examples

Given these capabilities, the features map to concrete deployments:

• Warehouse safety: track a worker near forklifts across aisles with one ID, using RT-DETR 2D.
• Retail analytics: follow a shopper between camera zones to measure dwell time without re-identification errors.
• Smart-building monitoring: count occupancy across floors and feed Kafka metadata to dashboards.
• Robotics and smart cities: share consistent world coordinates for navigation and incident review.

Interactive Explainer

To see the mechanism, the embedded demo below animates one person walking between three camera fields of view. Toggle between naive per-camera 2D tracking and MV3DT 3D fusion to watch the object ID stay consistent.

Key Takeaways

• DeepStream 9.1 ships 13 agentic skills, letting coding agents build multi-camera vision pipelines from natural-language prompts.
• MV3DT fuses per-camera detections into one shared 3D world, keeping a single globally consistent object ID across views.
• AutoMagicCalib replaces manual checkerboard calibration by estimating camera intrinsics and extrinsics from existing video.
• JetPack 7.2 support extends deployment to Jetson Orin and Thor, under a unified open-source GitHub monorepo.
• Outputs stream as OSD, Bird’s-Eye View, and Kafka protobuf metadata, ready for downstream analytics and dashboards.

Check out the Repo here. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us