Giving Your NVR a Voice: Using Local Vision LLMs to Describe Security Events

The Hook You've already got your local NVR spotting humans with lightning speed, but a generic "Person Detected" notification is a bit dull. What if your home lab could actually look at the snapshot and describe exactly what is happening in plain English?

The Tech Stack

• Frigate (or any NVR supporting MQTT)

• Docker

• Ollama (running a Vision model like LLaVA)

• MQTT Broker

• A basic Python watchdog script

The Concept

• Standard AI detection is incredibly fast but lacks context.

• We can intercept the MQTT snapshot payload from the NVR and securely pass it to a local Vision LLM running via Ollama.

• This entire process happens locally, keeping your security feeds completely off the cloud.

Spinning Up LLaVA

• Pull and run a lightweight vision model alongside your existing conversational models using a simple Docker command.

Bash

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

docker exec -it ollama ollama run llava

The Python Glue

• A clean Python script using paho-mqtt listens for the NVR event, encodes the image, queries the local API, and extracts the description.

• Instead of a boring "Motion at Front Door" alert, your notification says, "A bloke in a red shirt and a cap is leaving a cardboard box on the porch."

The Verdict

• Blending local NVR object detection with Generative AI is the absolute pinnacle of a private, smart home security setup.

• It turns raw data into actionable, human-readable insights without ever compromising your privacy.

 

"If you've got any questions or need a hand wrangling your own setup, don't hesitate to reach out at [email protected] or connect with me via www.yazan.me. I'm always keen to help out!"