Data Warehouse

Data warehousing enhanced analytics capabilities for an IoT service provider

» Data warehousing enhanced analytics capabilities for an IoT service provider

The client is a leading IoT platform developer based out of Asia offering full-scale digital services to a global clientele. Their major focus is implementing digitalization for large scale manufacturing companies to enhance efficiencies.

Business Need

As a part of digital monitoring services, the client collected data from all the devices installed at their customer’s site on IoT platform. It enabled them to have basic monitoring of assets from multiple geographical locations.

However, the IoT platform had limitations when it comes to deriving intelligent insights through various datasets. It poses challenges like:

Limitations in performing predictive and other advanced analytics
Migrating data to other databases due to lack of documentation
Accessing data from the platform is only possible through APIs which also have certain drawbacks of their own like cap on amount of data extracted at a time

To overcome these challenges, Hitech’s data sciences team proposed creating a data warehouse to leverage advanced analytics and enable custom reporting.

Want an exceptional set of data warehousing services for your company?

Get in Touch with us »

Solution

A project team was assembled, clear milestones and roadmap were planned. Once in place, the team divided the entire project into a two-part solution, to create a data warehouse and build pipelines for streaming real-time data. The second part dealt with migration of historical data from the platform to the newly created data warehouse.

Part #1: Process for daily data ingestion [real-time data]:

Data received from various IoT devices on shop floor machines was collected on Azure BLOB storage in form of configuration files having all data for machine health
These configuration files were to be stored there for seven days and then pushed to the endpoint in the data warehouse through an edge server
This inbound Telemetry data was then transformed into relevant and desired formats before loading to a staging area
Specific data models were built after defining entities, their attributes and relationships. The project team then created a cloud-based data warehouse on AWS cloud based upon this data model using AWS Redshift data warehouse service.
Frequency of extraction, transformation and loading was pre-decided and set at regular intervals during which the data was loaded to data warehouse
All transactions were logged and failed files were moved to appropriate buckets for retry. Upon retry failure a notification was sent to the admin

Part #2: Fetching historical data:

Historical data was fetched by developing custom scripts
From there, all relevant information was extracted and used as and when needed

Once the data warehouse was loaded, cloud credentials were shared with the client with secure login and role-based custom access. The client was empowered with opportunities for advanced analytics and get custom reports.

Results

The project team successfully delivered a cloud-based data warehouse with 100% uptime and 24×7 availability. In the beginning, the primary features of this AWS Redshift data warehouse included:

Automated data pipelines for streaming real-time IoT data
High scalability for data storage
Incident reporting through notification