Skip to main content

Architecture Overview

Foundation Layer: Core Kafka Components​

KafkaConsumerBase​

The consumer base class establishes the fundamental pattern for all data ingestion in SMOCS. It encapsulates Kafka consumer setup, topic subscription (supporting both explicit topic lists and regex patterns), and provides a standardized message processing loop. The key architectural decision here is the abstract process_message() method that forces all consumers to implement their specific business logic while inheriting reliable Kafka handling, error recovery, and resource cleanup.

KafkaProducerBase​

Producer components focus on reliable message publishing with automatic topic creation and message serialization. The base class handles Kafka producer configuration, topic management, and provides utility methods for topic name sanitization (converting MQTT-style topics to valid Kafka names). This abstraction allows data source integrations to focus on their specific protocols while delegating Kafka concerns to the base layer.

KafkaStreamingProcessBase​

The streaming processor represents SMOCS's most sophisticated component pattern. It inherits from KafkaConsumerBase for input handling while composing a KafkaProducerBase instance for output publishing. This design creates bidirectional Kafka communication through a single component. The process_message() method returns both a success flag and a list of output tuples, enabling complex stream transformations and multi-topic publishing patterns.

Application Layer: Specialized Implementations​

Data Source Producers​

Components like MQTTKafkaProducer and EpicsKafkaProducer extend KafkaProducerBase to bridge external protocols into the Kafka ecosystem. These producers translate protocol-specific data formats into standardized JSON messages, handle connection management for their respective protocols, and provide configurable data parsing based on the central configuration file.

Storage Consumers​

The InfluxDBConsumer exemplifies the consumer pattern, subscribing to all Kafka topics via regex patterns and selectively processing messages based on type markers. It demonstrates how consumers can implement intelligent filtering and routing logic while maintaining the simple single-threaded processing model.

Control Plane Processors​

The KafkaGymWrapper showcases the streaming processor pattern by consuming action commands, executing them in a Gymnasium environment, and publishing complete state transition tuples back to Kafka. It supports both blocking and non-blocking modes, demonstrating how the base architecture can accommodate different operational patterns.

Architectural Benefits​

This layered approach provides several key advantages:

Separation of Concerns: Kafka mechanics are isolated from business logic, protocol handling is separated from data processing, and each agent thread has a single responsibility.

Composability: New agents can be created by implementing three abstract methods, new data sources require only extending the producer base, and complex workflows emerge from combining simple components.

Reliability: Each layer handles its own error recovery, thread failures don't cascade across components, and Kafka provides natural message durability and replay capabilities.

Testability: Each layer can be tested independently, mock implementations can replace external dependencies, and the single-threaded design eliminates timing-dependent test failures.

The architecture demonstrates how careful abstraction can create a system that is both powerful and approachable, allowing domain experts to implement sophisticated data processing workflows without becoming Kafka experts.