Skip to content

Architecture and Engineering

The proposed video analysis solution should meet all requirements detailed in this A&E specification document.

Supported video inputs

The solution should support both recorded video files and live video streams.

Recorded videos

  1. The solution should support uploading and indexing recorded video files.
  2. The solution should support the following recorded video requirements:
Property Requirement
Supported formats .mp4, .mkv
Supported codecs H264, H265
Maximum resolution 4K
Maximum frame rate 30 FPS
Maximum bitrate 10 Mbps
Maximum upload size 20 GB

Live streams

  1. The solution should support analysis of live video streams.
  2. The solution should support RTSP over TCP, RTSP over UDP, and RTMP input streams.
  3. The solution should support live streams from fixed cameras, moving cameras, PTZ cameras, body cameras, and thermal cameras.
  4. The solution should support the following live stream requirements:
Property Requirement
Supported protocols RTSP over TCP, RTSP over UDP, RTMP
Supported codecs H264, H265
Maximum resolution 4K
Maximum frame rate 30 FPS
Maximum bitrate 10 Mbps

Time synchronization

All cameras, Video Management Systems, and solution servers should use synchronized time, for example via NTP, to ensure correct video indexing, search results, and alarm timestamps.

Video analysis technology

The solution should be based on deep learning technology and metadata-driven video analysis.

Two-phase analysis

  1. The solution should divide video analysis into two phases: indexing and searching.
  2. During indexing, the solution should analyze video inputs and generate Use Case-independent metadata.
  3. During searching, the solution should execute historical searches or live analytics rules over the indexed metadata.
  4. The solution should not require the original video to be reprocessed for every new search when the required metadata has already been indexed.
  5. The solution should support indexing both uploaded video files and live streams.
  6. The solution should support searching both historic metadata and live metadata streams.
  7. Historical search results must match the detections that would have been produced if the same search configuration had been running as live alarms over the same video interval.

Indexed metadata

The solution should be able to generate the following metadata from video inputs:

  1. Object detections and object classes.
  2. Object tracks and movement paths.
  3. Semantic feature vectors for detected objects.
  4. Semantic feature vectors for full-frame or background regions.
  5. Face detection and recognition metadata, when enabled.

The indexed metadata should be compact compared with the original video and should be suitable for repeated searches without re-indexing the source video.

Target and object types

  1. The solution should support detection and tracking of common object types, including people, vehicles, animals, and other common objects.
  2. Object detections should include timestamp, object type, confidence score, bounding box, and track information where applicable.
  3. The maintained list of supported object types is available in the Reference.

Configurable analysis parameters

  1. The solution should allow analysis parameters to be configured when a video or stream is added.
  2. The configurable parameters should include detector frame rate and vector extraction settings for object attributes, face recognition, and full-image search.
  3. Higher detector frame rates and higher vector counts should be supported when required by the scene, with corresponding hardware impact.

Video investigation

Method of operation

  1. The solution should allow users to search indexed video without requiring a detection rule to have been configured before the relevant event occurred.
  2. The solution should support searching one or multiple selected videos or streams.
  3. The solution should support folder-based source selection.
  4. The solution should support searches over absolute and relative time intervals.
  5. The solution should support object type selection and search-specific parameters.
  6. The solution should support area-of-interest and line-based configuration where required by the selected use case.

Free-text search

  1. The solution should support free-text search over indexed visual content.
  2. The user should be able to enter a natural-language description and search for matching objects or visual regions.
  3. Free-text search should support searching one or more videos.
  4. Free-text search should allow users to configure source, time frame, and object types.
  5. Free-text search should currently be treated as an English-language search feature.

Advanced search use cases

The solution should support predefined advanced search use cases, including:

  • Object in area
  • Line crossing detection
  • Area occupancy
  • Time in area
  • Started object detection
  • Stopped object detection
  • Object presence change
  • Presence with absence of another object
  • Object removal detection
  • Hazardous area detection
  • Near miss detection
  • Tailgating detection
  • Speeding detection
  • Traffic counting
  • Dual line counting
  • Object count statistics
  • Face finder
  • Face recognition
  • Line crossing with face recognition
  • Fire and smoke detection
  • Multi-condition event

Some use cases may be available only for historical search and may not be available as live analytics rules.

Multi-condition event

  1. The solution should support a Multi-condition event Use Case for complex searches that combine multiple conditions.
  2. The Multi-condition event Use Case should allow different track-based movement conditions to be combined.
  3. The Multi-condition event Use Case should allow track-based movement conditions to be combined with classifier-based object attributes.
  4. The Multi-condition event Use Case should support customer-specific scenarios that are not covered by simpler predefined Use Cases.

Viewing search results

  1. The solution should display search results in multiple view modes.
  2. The solution should support full-frame snapshots with detected objects highlighted.
  3. The solution should support object crop views.
  4. The solution should support playback of video segments around detections.
  5. The solution should support heatmap visualization.
  6. The solution should support timeline visualization.
  7. The solution should support metadata table visualization.
  8. The solution should allow users to review detection metadata, including object type, confidence score, timestamp, and bounding box coordinates where available.

Search process and exports

The solution should provide the following investigation capabilities:

  • Save search results for later review.
  • Reopen and review saved search results.
  • Duplicate saved search results.
  • Download PDF reports.
  • Download CSV metadata.
  • Delete one or more saved search results.

Fine-tuning

The solution should handle Use Cases consistently across historical and real-time operation. From a user perspective, the solution should support two distinct search modes:

  • Historical searches: executed on indexed metadata to find specific events or objects in past video streams.
  • Real-time searches: continuously executed on live metadata to trigger alerts based on defined conditions.

This approach should allow users to experiment and fine-tune search configuration on historical data, then transform successful and accurate searches into real-time live analytics rules.

Real-time event detection

Live analytics rules

  1. The solution should support live analytics rules for real-time detection on live streams.
  2. Live analytics rules should run continuously on indexed live metadata.
  3. Live analytics rules should produce alarms when matching detections occur.
  4. The solution should allow alarm cooldown configuration.
  5. The solution should allow supported advanced searches to be converted into live analytics rules.
  6. The solution should allow users to validate a use case on historical video before enabling it for real-time operation.

Alarm review

  1. The solution should display alarms produced by live analytics rules.
  2. The solution should show alarm details, including detection snapshot, metadata, and video segment where available.
  3. The solution should allow users to toggle bounding box display where applicable.
  4. The solution should allow alarm reports to be downloaded as PDF.
  5. The solution should allow alarms to be added to cases where case management is used.

Event integration

The solution should be able to send real-time detection events to external systems using supported integration targets.

Supported integration methods should include:

  • Webhook integration for sending detection events to HTTP endpoints.
  • Kafka output for downstream consumers.
  • Milestone XProtect integration for sending analytics events.
  • Genetec Security Center integration for sending analytics events.

Personalized AI classifiers and extensibility

  1. The solution should support Personalized AI classifiers that filter detected objects by visual attributes.
  2. Personalized AI classifiers should be usable in searches and live analytics rules where supported by the selected Use Case.
  3. The solution should support both built-in and uploaded Personalized AIclassifier files.

Built-in or available Personalized AI classifiers may include:

  • Gender
  • Person on the phone
  • Hands up
  • Emergency vehicle
  • PPE helmet
  • PPE vest
  • Face mask
  • Fallen person

Custom classifiers and Use Cases

  1. The solution should support additional classifiers for customer-specific use cases.
  2. The solution should support additional Use Cases for customer-specific analytics workflows.
  3. Additional classifiers and Use Cases should be applicable to already indexed data when the required metadata is available.
  4. The solution should not require re-indexing of source video solely because a new compatible classifier or Use Case is added.

Cases and investigation workflow

  1. The solution should support Case-based investigation workflows where enabled.
  2. The solution should allow users to create and manage Cases.
  3. The solution should allow alarms or search results to be added to Cases.
  4. The solution should allow users to review saved results and related metadata.
  5. The solution should allow reports or metadata to be exported for external use.

Public API and integrations

Public API

  1. The solution should expose a documented REST API.
  2. The REST API should support integration with external applications and custom user interfaces.
  3. A Swagger UI should be available on deployed systems for API exploration.

Integration capabilities

The solution should support the following integration capabilities:

  • Webhook event notification.
  • Kafka result streaming.
  • Milestone XProtect analytics event integration.
  • Genetec Security Center analytics event integration.

Integration targets should be configurable in the user interface and selectable when configuring live analytics rules where applicable.

User management and access control

  1. The solution should provide user management.
  2. The solution should provide user group management.
  3. The solution should provide role-based access control.
  4. The solution should support folder-level access control.
  5. The solution should support user activation and password recovery workflows.
  6. The solution should separate administrative permissions from operational permissions.

The solution should support predefined roles, including:

  • Operator
  • Technician
  • Investigator
  • Administrator

Licensing

  1. The solution should require a valid license for video indexing.
  2. The solution should display license status in the user interface.
  3. The solution should support online license activation.
  4. The solution should support offline license activation for installations without direct internet access.
  5. If the license expires, existing searches and rules should retain their data, but indexing and live alarm production may stop until a valid license is activated.

Backup and restore

  1. The solution deployment tooling should support backup and restore workflows.
  2. Backup and restore operations may require temporary service downtime while services are stopped and restarted.

System architecture

The solution should be based on the following main components:

  1. Core node
  2. Indexer node or nodes
  3. Web user interface
  4. REST API
  5. Metadata and event processing services
  6. Storage services
  7. Integration services

The current architecture diagram is available in the Architecture page.

Core node

The core node should provide central product services:

  • Web user interface
  • REST API
  • User, group, role, and permission management
  • Search orchestration
  • Live analytics rule management
  • Alarm management
  • Integration target management
  • License management
  • Metadata processing and search execution services

Indexer nodes

Indexer nodes should process live streams and recorded video inputs. Indexer nodes should:

  • Decode video inputs
  • Run deep learning inference
  • Extract object, face, and semantic metadata where configured
  • Store or forward indexed metadata to the core services
  • Scale horizontally by adding more indexer nodes where deployment size and topology require it

Indexer nodes and standalone nodes should require NVIDIA GPU resources that meet the hardware requirements.

Deployment topologies

The solution should support the following deployment patterns:

  1. Standalone installation, where core and indexing services run on one server.
  2. Distributed installation, where a core node and one or more indexer nodes run on separate servers.
  3. Site-oriented deployment, where indexer capacity is placed close to cameras to reduce bandwidth between the camera network and the core services.