Best-practice architecture for a cloud-based IoT platform

We have developed a best-practice architecture that has proven effective for many customers in meeting the requirements of modern IoT platforms. The development of IoT platforms often presents similar challenges, like the efficient processing and delivery of data generated by a wide range of devices. It must be received, processed and made available to surrounding systems. Our best-practice architecture addresses this challenge. It is suitable for projects of any scale and complexity in terms of user and device numbers, and has proven its worth in a wide range of demanding use cases.      

1. The advantage of the cloud: scalability and efficiency
2. Reach your goals faster with managed services and best-practice architecture
3. Flexibility through modular components
4. Focus on the ingest pipeline
5. Architecture of the ingest pipeline
6. Architectural components 
7. Implementation scenarios
8. Conclusion

The advantage of the cloud: scalability and efficiency

The underlying architecture is independent of specific deployment scenarios and can be used with major cloud service providers (CSPs) offering a wide range of managed services, as well as with smaller CSPs and in on-premises environments. However, using the cloud offers significant advantages; services such as Azure Event Hub, for example, simplify the installation, configuration and operational processes considerably.

The associated higher costs and loss of flexibility are justified in many cases, particularly when the benefits of PaaS and SaaS services are taken into account: simplified scaling, faster deployment and easier maintenance. A potential vendor lock-in is deliberately accepted if it aligns with the customer’s cloud strategy.

Reach your goals faster with managed services and best-practice architecture

Our best-practice architecture enables a quick start, as the time-consuming process of designing a system architecture can be significantly shortened, or even eliminated altogether. PaaS and SaaS managed services act as ready-made building blocks for the architecture; they can be easily set up, thereby further accelerating progress in building an IoT platform. This means that the focus of the project can be shifted much earlier to the implementation of the components that deliver the greatest added value.

Flexibility through modular building blocks

Each abstract component of our architecture can be populated with different concrete implementations. In theory, these can be combined in any way, but certain implementation scenarios are encountered more frequently. Examples:

• Hyperscalers with a 100% focus on CSP SaaS
• Medium-sized CSP with managed K8S and open-source building blocks
• On-premises with 100% open source

These implementation scenarios can be quickly identified based on the requirements at hand and translated into suitable implementation strategies.

Focus on the ingest pipeline

In this article, we have focused on the ingest pipeline for IoT metrics and telemetry data. However, other mechanisms (commands / request–response, device management, on-boarding, provisioning, etc.) are also covered in our best-practice architecture and will be discussed in future articles.

Our experience shows that a well-thought-out architecture not only improves the efficiency and scalability of IoT solutions, but also lays the foundation for innovative developments.

Architecture of the ingest pipeline

The IoT platform’s ingest pipeline is a generic IoT data processing chain designed to receive, process, enrich and store data from a wide range of devices. Depending on the use case to be implemented, the platform stores this data in (time-series) databases, deposits it in a data lake or an archiving area, and/or forwards it to surrounding systems in near real time. In our experience, some of the sub-components of the ingest pipeline contain use-case-specific logic. However, many sub-components can be modelled in a completely generic manner. The figure below shows the best-practice architecture and the generic and use-case-specific sub-components of the platform it contains.

The ingest pipeline extends from the connected device to the storage layer. When implementing it, it has proved helpful to take the following general principles into account:

• Event streaming: messages arriving from devices are treated as a stream of events. The following three core elements play a key role in this process

• • Kafka: forms the centre of the architecture and acts as the data and event hub.

• Stream Processing: enables the use of stream processing and streaming analytics frameworks to carry out further calculations. 
• Specific functional building blocks: Event-oriented functions or services (usually stateless) that read data from a Kafka topic, process it and write it to another Kafka topic.
• Generic data: The data within the ingest pipeline is largely generic and can therefore be processed further regardless of the specific use case.
  • Data sinks: Generic components that store data in a generic format in specific data sinks.
• Hot, Cold, Warm Paths: The processing chain essentially distinguishes between three processing paths, which primarily influence processing latency:
  • Hot Path: Hot data is processed in near real time and made available with very low latency.
  • Warm Path: Warm data is processed quickly and made available with relatively low latencies.
  • Cold Path: Cold data is processed with a delay or in batches and is usually persisted in relatively slow storage. Read latencies in particular can be high here.

The table below explains the functions of the platform’s standard components:

David Lane is a Principal Solution Architect and IoT specialist at iteratec. Drawing on his expertise in smart products, smart manufacturing and IoT security, he develops innovative and secure solutions for digital transformation. He uses his technical know-how and enthusiasm for the IoT to support companies in developing digital products or optimising manufacturing processes through bespoke, high-performance and scalable IoT platforms.

Do you have any questions or need assistance?

You can find further information about the possibilities of digital twins for your business on our website. Please feel free to contact us.