Revolutionizing Retail with AI-Driven Surveillance

About the Client

The leading player in the retail sector, dedicated to integrating the latest technology to enhance store security, customer experience, and operational efficiency.
Customer
Confidential
Location
USA
Industry
Retail

Company’s Request

To develop a cost-effective, AI-driven surveillance system capable of real-time threat detection and customer behavior analysis.

Technology Set

Rockchip RK3399Pro
A development board with a high-performance NPU for AI tasks. Chosen for its edge computing power, enabling real-time video analysis and decision-making for surveillance.
TensorFlow
An AI framework for developing and training machine learning models. Optimized AI models for the RK3399Pro NPU, enhancing efficiency and performance in surveillance tasks.
AWS IoT Core
A cloud service for device connectivity and management. Enabled centralized management of surveillance devices, facilitating updates and security.
AWS S3 and Lambda
S3 provide scalable storage; Lambda offers serverless computing for data processing. Stored large volumes of video data and processed information for advanced analytics.
Python
A programming language for software development. Used for developing the software components of our system.
OpenCV
A library for image processing and computer vision. Processed video feeds from cameras, key for real-time image analysis and surveillance operations.
PyTorch
An AI framework for deep learning model development and training. Employed alongside TensorFlow for training AI models, offering alternatives for optimization and efficiency.
Docker
A platform for developing, shipping, and running applications in containers. Used as a consistent environment for our application for deployment and scaling across devices.

Project Kickoff
Our client approached us with a clear goal: to develop a cost-effective, AI-driven surveillance system capable of detecting threats and analyzing customer behavior in real time. The primary objectives were to identify suspicious activities, such as potential theft (for instance, moving items into pockets), and to monitor unauthorized presence in restricted areas. We knew right away that the solution had to be both powerful and modular. It needed to handle real-time processing while being flexible enough to expand easily. So, we assembled a team of experts in AI, edge computing, and cloud integration to see the project through from start to finish.

Stage 1: Conceptualizing the Modular Architecture
Our approach prioritized modularity to enable affordable scaling. Each module would function independently, connecting to a broader network and allowing the client to add or expand surveillance zones without complete redesigns. Building this system posed technical challenges in meeting three core requirements:

  • Edge Processing: Real-time video analysis required each module to process data on-site to reduce latency.
  • Cloud Integration: Centralized control was necessary for remote monitoring, management, and advanced data processing.
  • Cost Efficiency: The system had to deliver high performance within budget constraints.

To achieve real-time edge processing, we evaluated three configurations:

  • Centralized Processing: Sending all video data to the cloud increased latency, making this setup unfeasible.
  • Fully Decentralized Processing: Running all tasks locally reduced latency but was cost-prohibitive due to high hardware demands.
  • Hybrid Approach: This configuration balanced speed, security, and cost by assigning primary processing to each module, with the cloud handling intensive, secondary tasks. The hybrid setup allowed each module to manage real-time tasks independently, while the cloud supported heavier computing as needed.

After establishing the edge processing structure, we focused on efficient cloud integration. The cloud setup needed to handle centralized device management, secure video storage, and non-real-time analytics. We explored three cloud configurations:

Direct Device-to-Cloud: Each module connecting independently strained scalability.

Cloud Gateway: This setup aggregated data locally, reducing direct connections but adding complexity.

Cloud Management Hub: This solution uses AWS IoT Core, centralized management, and scaling – each module is connected to the hub for updates, monitoring, and analytics to simplify the future expansion.

The Cloud Management Hub was selected for its security and single-interface management capabilities. AWS IoT Core provided control over all modules, allowing for scaling as more were added.

Balancing performance and affordability required precise hardware selection. After evaluating multiple options, we chose the Rockchip RK3399Pro, whose dual-core Cortex-A72, quad-core Cortex-A53 CPUs, and NPU (capable of 3.0 TOPs) provided the processing power necessary for continuous, high-definition video analysis across multiple cameras. This board’s NPU efficiently handled local AI tasks, reducing reliance on cloud resources.

Compatibility with TensorFlow allowed us to optimize our threat detection models specifically for the RK3399Pro’s architecture, making the solution cost-effective.

For scalability, each module was designed as a plug-and-play unit. Upon activation, modules automatically registered with the cloud hub, configured themselves, and began operation. This setup, managed via AWS IoT Core, enabled rapid deployment of additional modules without complex reconfiguration.

Stage 2: Hardware Justification
With this architecture in place, our next priority was to select hardware that could meet the client’s need for real-time threat detection without exceeding budget limits. This required edge computing power strong enough to process data instantly. Our primary criteria for hardware selection were high processing capacity and compatibility with AI frameworks.

Establishing Hardware Requirements

Each module had to support uninterrupted, high-definition video from multiple cameras, executing real-time analysis. This demanded a CPU that could manage multiple data streams without lag. Additionally, since AI-based video analysis was central to the project, an integrated Neural Processing Unit (NPU) was necessary for efficient object detection and behavioral analysis on the edge. Compatibility with TensorFlow was also important, allowing us to tailor pre-trained models for this hardware.

Challenges in Model Integration and Optimization
After selecting the Rockchip RK3399Pro, we focused on integrating high-resolution camera feeds and tailoring AI models to meet specific client requirements. We began with a pre-trained neural network as a foundation, which was then fine-tuned to recognize “suspicious behavior” scenarios, such as detecting potential theft (e.g., moving items into concealed areas) and identifying unauthorized presence in restricted zones. This approach allowed us to leverage existing model architectures.

During integration, early tests indicated that the NPU managed individual video streams efficiently but encountered latency issues with multiple feeds, which impacted real-time detection. To address this, we implemented a two-part optimization strategy:

  • CPU-NPU Load Balancing: We configured the system to assign lighter tasks, like initial frame processing, to the CPU. This left the NPU free to focus on more intensive tasks, such as object detection, reducing its load and allowing it to manage multiple video streams.
  • Model Optimization: By adjusting our TensorFlow models specifically for the RK3399Pro’s NPU, using quantization to lower model size and processing demands, we minimized memory usage and maintained high-speed processing. This allowed the system to handle high-definition video feeds without sacrificing detection accuracy.

Energy efficiency was another priority, especially given the multi-location deployment. Continuous video analysis can be power-intensive, so we adopted a few strategies to manage consumption:

  • Dynamic Power Scaling: The RK3399Pro was configured to adjust CPU and NPU power based on current demand. During periods of low video activity, the system scaled down to conserve power, ramping up only during high-activity moments, such as when detecting motion.
  • Heat Management: To prevent overheating during heavy processing, we developed a passive cooling solution with an optimized heat sink design. This kept temperatures stable without adding power-consuming cooling components.

TensorFlow’s compatibility with the RK3399Pro was a core advantage, though effective integration required refining models to the hardware’s specific architecture. This included:

  • Pruning and Layer Adjustment: By removing non-essential layers, we simplified model inference, balancing accuracy with speed. Through several iterations, we found an optimal model structure that handled complex detection tasks efficiently.
  • Batch Processing: We implemented batch processing so the NPU could handle video frames in groups, enhancing throughput and reducing per-frame processing time, thus improving real-time response.

Stage 3: Initial System Testing and Optimization
With hardware in place, we began testing to ensure the Rockchip RK3399Pro could handle real-time processing for multiple camera feeds under realistic conditions. Initial tests revealed performance issues as the system struggled with high-demand tasks, requiring adjustments.

Performance Bottlenecks
The first round of tests involved connecting several high-resolution cameras to a single RK3399Pro module. While initial results met expectations, issues surfaced as more video streams were added:

  • Frame Rate Drops: Adding multiple cameras caused frame rate reductions, delaying threat detection and impacting alert responsiveness.
  • NPU Saturation: The NPU managed individual streams effectively but lagged when handling complex tasks like high-resolution object identification across multiple feeds.
  • CPU Utilization Spikes: The CPU experienced load spikes during peak activity, slowing processing, especially when handling concurrent detections.

These bottlenecks indicated that while the RK3399Pro was robust, high-intensity tasks required further optimization of processing strategy and model efficiency.

To address these challenges, we implemented a balanced workload distribution between the NPU and CPU, focusing on task prioritization to maximize processing efficiency on the edge.

Task Prioritization and Delegation:

  • Edge-Based Threat Detection: We optimized the system to handle priority detection tasks, such as identifying dangerous objects or behaviors, directly on the NPU for fast on-site responses.
  • Cloud Delegation for Non-Critical Tasks: Lower-priority tasks, like aggregated behavioral data analysis, were offloaded to AWS IoT Core. This allowed edge devices to concentrate on immediate detection without processing delays.

TensorFlow Model Optimization for the NPU:

  • Model Pruning and Quantization: We reduced model size and processing demands by pruning unnecessary layers and using quantization. This adjustment reduced the NPU workload, improving speed.
  • Inference Pipeline Splitting: Primary tasks like initial object detection were prioritized on the NPU, while secondary tasks, such as tracking and classification, were delayed until necessary.

Adaptive CPU-NPU Load Management:

  • Dynamic Task Assignment: Simpler tasks, such as frame pre-processing, were assigned to the CPU, reserving the NPU for complex inference tasks. This distribution kept frame rates steady across all streams.
  • Real-Time Task Adjustment: We developed an adaptive algorithm that redistributed tasks when NPU load spiked, temporarily offloading additional processing to the CPU to prevent bottlenecks.

Fine-Tuning System Response and Reducing Latency
Through iterative testing, we refined the system to reduce latency and maintain consistent, real-time operation across different activity levels. 

Alerting Mechanism Optimization:

  • Parallel Alert Processing: We streamlined alert processing to run parallel with detection tasks, reducing the time between threat detection and alert activation. 
  • Cloud-Assisted Alert Verification: To reduce false positives, we added a cloud-based verification layer. This step cross-referenced detected threats with additional data, ensuring only valid alerts triggered a response.

Latency Reduction with Frame Skipping:

  • Intelligent Frame Skipping: During high-activity periods, a frame-skipping mechanism enabled the system to bypass non-essential frames, balancing speed with detection accuracy. This allowed the system to maintain a consistent frame rate and retain its threat detection capabilities.

Stage 4: Edge Computing vs. Cloud Integration Strategy
After optimizing edge processing on the RK3399Pro, we shifted focus to cloud integration to address limitations in storage, device management, and advanced analytics. We aimed to balance on-site processing with cloud resources to create a centrally managed and resilient system.

Defining the Role of the Cloud
We outlined three main objectives for cloud integration:

  • Centralized Device Management: With each module operating independently, managing multiple units across locations requires a centralized system for monitoring, firmware updates, and configuration. Without this, scaling the system would be complex and error-prone.
  • Data Storage and Retrieval: Local storage on the RK3399Pro couldn’t support the high volume of video data, especially in a multi-site, long-term setup. Relying on local storage alone would risk data loss and performance degradation.
  • Advanced Analytics: While the RK3399Pro handled real-time processing, more intensive analysis, like behavioral trend detection, demanded additional cloud resources, allowing the edge devices to focus on immediate tasks.

With these needs in mind, we assessed various cloud options, prioritizing scalability, real-time capabilities, and seamless edge integration:

  • Private Cloud Infrastructure: This provided high control and security but proved costly and lacked the scalability of managed IoT solutions.
  • Google Cloud IoT Core: Google offered strong IoT management but required more complex configurations to match AWS in real-time performance.
  • AWS IoT Core: AWS IoT Core emerged as the ideal choice for its comprehensive IoT management, real-time monitoring, secure storage, and integration with services like Amazon S3 and SageMaker. This alignment with our requirements made AWS the most suitable choice for efficient scaling.

After evaluating these options, we selected AWS IoT Core as our cloud platform for its capabilities in managing edge devices, scalable storage, and enhanced analytics.

Integrating AWS IoT Core with Edge Modules
Configuring each RK3399Pro module to connect with AWS IoT Core enabled secure and centralized device management. Key components included:

  • Device Registration and Secure Communication: We registered each module as an IoT device within AWS IoT Core and assigned security certificates to encrypt communications. TLS protocols added another layer of data protection, securing information as it traveled from each module to the cloud.
  • Centralized Firmware Management: AWS IoT Core facilitated a central repository for firmware updates, enabling modules to receive upgrades and patches during off-peak hours to minimize disruption. This process allowed updates across all modules simultaneously.
  • Real-Time Monitoring and Alerts: AWS IoT Core’s rules engine triggered alerts for issues such as module disconnections or storage capacity limits. Alerts routed through AWS CloudWatch, and SNS enabled timely responses, offering the client full visibility of system health for easier network management.

Beyond storage, the cloud extended the system’s analytics capabilities. While edge modules managed immediate threat detection, AWS provided a platform for deeper analysis, such as behavioral modeling and trend detection.

  • Behavioral Analysis via Amazon SageMaker: SageMaker enabled advanced analysis of behavior patterns within stored video data. By identifying recurring paths, crowd density changes, and other trends, the system generated insights beyond real-time detection, helping the client improve security strategies over time.
  • Data Visualization through Amazon QuickSight: For a comprehensive view of system activity, we utilized Amazon QuickSight. This tool offered customizable dashboards to track key metrics like detection events, storage usage, and device health. These visual insights allowed the client to monitor performance and make decisions for future surveillance needs.

Stage 5: Implementing Data Security and Privacy Solutions
As our project progressed, data security became a top priority, especially given the sensitivity of video data processed on the edge and transmitted to the cloud. Both data at rest and in transit required protection. While edge devices handled real-time analysis, metadata and essential data were sent to the cloud, which managed centralized control and storage. To safeguard against potential data breaches, we developed a multi-layered security framework covering edge and cloud components.

A major challenge was balancing the security limitations of edge devices with the advanced protection available in the cloud. AWS offered strong features like multi-layer encryption, intrusion detection, and continuous monitoring, so we built our security around these cloud capabilities, limiting data exposure on the edge. Our security goals included:

  • Secure Data Transmission: Ensuring all video data sent to the cloud was encrypted to prevent interception.
  • Data Protection at Rest: Securing edge and cloud storage to prevent unauthorized access.
  • Controlled Access: Restricting data access to authorized personnel only.

To protect data as it moved between edge devices and the cloud, we implemented AWS’s encryption tools:

  • TLS Encryption: All transmissions from the RK3399Pro modules to AWS IoT Core used TLS 1.2 encryption, securing data against interception. AWS IoT Core authenticated encryption certificates allow only trusted devices to connect, preventing potential attacks.
  • AES-256 Encryption for Sensitive Data: Highly sensitive data was encrypted directly on the edge device using AES-256 before being sent to the cloud. This standard is one of the most secure, making it effective for protecting surveillance data.
  • Public-Key Infrastructure (PKI) Authentication: Each module received a unique digital certificate for authentication with AWS IoT Core. This PKI-based setup ensured that only authorized modules accessed the cloud.

Our team also applied local security measures to the edge devices, especially for sensitive data processed before transmission:

  • On-Device Encryption: Sensitive data was encrypted locally on the RK3399Pro using AES-256 for storage and RSA encryption for metadata sent to the cloud.
  • Regular Key Rotation: For long-term security, we implemented key rotation, updating encryption keys on each module with AWS Key Management Service (KMS).
  • Secure Storage for Video Cache: Each device maintained a short-term video cache for immediate analysis. We created a secure storage partition using a Trusted Execution Environment (TEE), isolating sensitive data from the main system.

 

Stage 6: Final Testing and System Deployment

Testing was designed to validate the system’s durability, adaptability, and scalability:

 

  • High-Load Testing: Each module was pushed to handle maximum video feeds under varied conditions to confirm that the NPU and CPU could handle intense workloads without compromising real-time processing or overheating.
  • Network Disruption Handling: Simulated network loss and bandwidth fluctuation tests verified that modules could switch to local storage and adapt data rates, maintaining surveillance even during connectivity issues.
  • Scalability Verification: Adding modules incrementally and synchronizing them through AWS IoT Core validated the system’s plug-and-play functionality across locations.

We tested latency to ensure immediate threat alerts and adjusted detection thresholds to reduce false positives, refining the system to deliver reliable, meaningful alerts.

Our team also prepared comprehensive documentation and conducted training to enable the client’s team to operate and scale the system independently. Initial deployment began with a staged rollout, confirming cloud connectivity and stable operation, followed by full deployment and final calibration on-site.

Value Delivered

Enhanced Security
Immediate threat detection reduces theft, securing the shopping environment.
Customer Insights
Behavioral analysis yields data on preferences and peak times, guiding personalized marketing and optimal layout planning.
Enhanced Marketing Analytics
Data analytics enhances the understanding of consumer behavior, improving the effectiveness of marketing strategies and campaigns.
Energy Savings
Smart monitoring of foot traffic and activity patterns enables energy-efficient lighting and climate control, lowering operational costs.
Loss Prevention
Detection of unusual activities or inventory discrepancies aids in rapid loss prevention, and safeguarding assets.
Customer Experience
Insights into customer flow and behavior contribute to a tailored shopping experience, boosting satisfaction and loyalty.