Full-Scale Technological Upgrade for Industrial Metrics and Analytics

About the Client

Arch Systems specialize in global monitoring and metrics solutions for industrial machines, with a particular focus on Surface-Mount Technology (SMT). The company's technology is used in 5 out of the 10 largest EMSs and more than 5 growing Fortune 1000 OEMs. Their systems remotely monitor over 1,000 production lines and around 10,000 machines, preventing 20% to as much as 60% utilization losses on sites around the world.
Customer
Location
USA
Industry
Industrial Automation

Company’s Request

Arch Systems was in search of highly qualified talent, capable of handling diverse and complex tasks with minimal guidance. Their primary need was to enhance their cloud/web server capabilities and improve their core product features. The initial project scope was flexible and has evolved in complexity over time.

Technology Set

Python 3
The backbone for backend development, offering robust libraries and frameworks.
Django and DRF (Django Rest Framework)
For building scalable and secure web applications.
Pandas
Data manipulation and analysis.
Celery
Asynchronous task queuing.
Elasticsearch
Advanced search and analytics capabilities.
PostgreSQL and TimescaleDB
Relational data storage, with TimescaleDB providing additional time-series capabilities.
InfluxDB
Time-series data storage, optimized for high-write and query loads.
Docker and Kubernetes
Containerization and orchestration, ensuring seamless deployment and scalability.

At Sirin Software, our engineering team employed a multi-layered strategy to address the complex requirements set out by Arch Systems. A specialized unit was deployed to focus on the server/framework for on-premises data collection. The project was initiated in September 2019 and has since been a continuous engineering effort designed to adapt to Arch Systems’ dynamic operational needs.

Core Technologies and Strategies

Utilizing Python as the core programming language, we integrated specialized libraries such as Pandas for complex data manipulation and Celery for efficient asynchronous task queuing. The web application layer was constructed on a Django and Django Rest Framework (DRF) backbone, for scalability and robustness. Beyond the utilization of established technologies, our engineers devised solutions to meet specific client challenges.

Technical Challenges and Custom Solutions

One of the first significant tasks we undertook was the integration with InfluxDB. This was complex work due to the database’s peculiar limitations and the need to integrate multiple versions with different interfaces, libraries, and APIs. We had to come up with creative and unorthodox solutions to enable smooth migration and ensure that the system could adapt to these limitations.

Another significant challenge was achieving real-time metrics with large datasets. The sheer volume of data – millions of records – posed a challenge for real-time calculations. To overcome this, we implemented multiple levels of optimization, such as persistent caching, aggregating cached calculations of smaller scopes into bigger ones, and using fine-tuned libraries to speed up data conversions. These optimizations were necessary for managing high-write and query loads, a prevalent issue in the field of industrial monitoring.

Metrics Subsystem Overhaul

We engineered a novel metrics subsystem from the ground up, incorporating sophisticated functionalities such as machine movement history tracking. This new system was designed with modularity and a “Never compute twice” approach, allowing metrics to share data and avoid repeated queries. It was designed to flexibly support all known metrics and as many future extensions as possible. This was a significant improvement over the existing system, which was complicated and hard to extend.

Lessons Learned

The project reiterated the significance of avoiding technological debt, maintaining strong test coverage, keeping the production environment near to the development environment, and postponing solution delivery until the client’s demands are well-known and thoroughly investigated.

Milestones and Future Plans

Throughout the project, several key milestones were achieved, including metrics caching and aggregation, machine assignment introduction, and the development of a change request system. These milestones were key in enhancing the system’s overall efficiency and adaptability. Looking ahead, the “Global OEE” is in the process of being rolled out, and we are considering a significant data migration from InfluxDB to Timescale.

Value Delivered

Time Savings
The complete overhaul of the metrics subsystem streamlined the data collection and analysis process. This efficiency gain translated into time savings, enabling clients of Arch Systems to make quicker, data-driven decisions.
Operational Efficiency
Our custom solutions, particularly the new metrics subsystem, optimized the operations. While predictive maintenance is in the plans, current metrics help in monitoring performance and identifying areas for improvement.
Staff Optimization
Staff Optimization The automation and streamlining of various processes have the potential to optimize staffing needs. Tasks that previously required manual intervention could now be automated, freeing up staff to focus on more strategic activities.
Enhanced Capabilities
Enhanced Capabilities The technological advancements introduced by our team added new capabilities to Arch Systems' offerings. Real-time analytics and data visualization features are now integral parts of their industrial monitoring systems, providing their clients with a competitive edge in the market.
Multi-Brand Support
Arch Systems offers a unique advantage by supporting various machine models and brands, unlike existing systems that are brand-locked. This flexibility has been enhanced by our technological contributions, making their system more versatile and user-friendly.