Full-Scale Technological Upgrade for Industrial Metrics and Analytics

At Sirin Software, our engineering team employed a multi-layered strategy to address the complex requirements set out by Arch Systems. A specialized unit was deployed to focus on the server/framework for on-premises data collection. The project was initiated in September 2019 and has since been a continuous engineering effort designed to adapt to Arch Systems’ dynamic operational needs.

Core Technologies and Strategies

Utilizing Python as the core programming language, we integrated specialized libraries such as Pandas for complex data manipulation and Celery for efficient asynchronous task queuing. The web application layer was constructed on a Django and Django Rest Framework (DRF) backbone, for scalability and robustness. Beyond the utilization of established technologies, our engineers devised solutions to meet specific client challenges.

Technical Challenges and Custom Solutions

One of the first significant tasks we undertook was the integration with InfluxDB. This was complex work due to the database’s peculiar limitations and the need to integrate multiple versions with different interfaces, libraries, and APIs. We had to come up with creative and unorthodox solutions to enable smooth migration and ensure that the system could adapt to these limitations.

Another significant challenge was achieving real-time metrics with large datasets. The sheer volume of data – millions of records – posed a challenge for real-time calculations. To overcome this, we implemented multiple levels of optimization, such as persistent caching, aggregating cached calculations of smaller scopes into bigger ones, and using fine-tuned libraries to speed up data conversions. These optimizations were necessary for managing high-write and query loads, a prevalent issue in the field of industrial monitoring.

Metrics Subsystem Overhaul

We engineered a novel metrics subsystem from the ground up, incorporating sophisticated functionalities such as machine movement history tracking. This new system was designed with modularity and a “Never compute twice” approach, allowing metrics to share data and avoid repeated queries. It was designed to flexibly support all known metrics and as many future extensions as possible. This was a significant improvement over the existing system, which was complicated and hard to extend.

Lessons Learned

The project reiterated the significance of avoiding technological debt, maintaining strong test coverage, keeping the production environment near to the development environment, and postponing solution delivery until the client’s demands are well-known and thoroughly investigated.

Milestones and Future Plans

Throughout the project, several key milestones were achieved, including metrics caching and aggregation, machine assignment introduction, and the development of a change request system. These milestones were key in enhancing the system’s overall efficiency and adaptability. Looking ahead, the “Global OEE” is in the process of being rolled out, and we are considering a significant data migration from InfluxDB to Timescale.