RedPanda Streaming broker

introduction the redpanda node is the central event backbone of the mantsu architecture it hosts the redpanda streaming broker , which acts as the message bus for all business events exchanged between services what is the redpanda streaming broker? it is a modern, high performance event streaming platform that acts as an alternative to apache kafka it is built from the ground up to be light weight & efficient and it is engineered as a drop in replacement fro kafka; you can use your existing kafka client applications and tools with redpanda without changing anything role in the architecture redpanda functions as the it tier broker of the mantsu unified namespace (uns) all significant business events — such as order state changes, quality results, downtime registrations, and recipe updates — are published as kafka topics by the originating mendix service topic names follow the isa 95 hierarchical naming convention, making each event self describing and discoverable within the uns downstream consumers (other mendix services, the reporting application, bi platforms) subscribe to the relevant topics independently mqtt streams from ot edge nodes do not pass through redpanda they are consumed directly by the services or the reporting application that require them redpanda handles business level events in the uns it tier; raw sensor and machine data lives in the ot tier and is served by the mqtt broker technology redpanda redpanda streaming broker redpanda is a kafka compatible streaming broker written in c++ it is fully api compatible with apache kafka, meaning any standard kafka producer, consumer, or admin tool connects to it without code changes why redpanda over apache kafka no jvm dependency redpanda runs as a single native binary kafka requires a java virtual machine and a separate zookeeper (or kraft) cluster for coordination, adding significant operational overhead redpanda handles all coordination internally, reducing the deployment footprint considerably ultra low latency redpanda utilizes a thread per core architecture this allows it to bypass context switching and avoid unpredictable jvm garbage collection pauses, delivering up to 10x lower tail latency compared to traditional kafka lower resource consumption the c++ runtime is substantially more memory and cpu efficient than the jvm, making redpanda viable on smaller infrastructure kafka api compatibility existing tooling, client libraries, connectors, and integrations built for kafka work against redpanda without modification agentic data plane available for mid to large deployments — requires redpanda enterprise license the agentic data plane is a redpanda enterprise module that exposes the streaming broker as a governed data source for ai agents it provides two complementary integration protocols mcp (model context protocol) an open protocol that allows ai agents and large language models to query redpanda topics as structured data sources an mcp server is exposed by the agentic data plane; agents connect to it to retrieve, filter, and reason over event data without needing direct broker access a2a (agent to agent) a protocol enabling multiple ai agents to exchange information and coordinate actions in the mantsu context this allows specialised agents (e g a quality analysis agent, a production planning agent) to collaborate by sharing observations derived from the event stream governance all agent authorisations, topic access permissions, and connection configurations are managed through redpanda's built in management interface this ensures that agents operate within defined boundaries and that access to sensitive business event data is controlled and auditable redpanda iceberg topics available for mid to large deployments — requires redpanda enterprise license a standard kafka topic is optimised for streaming events flow in real time, are retained for a defined period (hours to weeks typically), and are consumed sequentially kafka storage is a write optimised log; it is not designed for analytical queries that span large time ranges or filter across many fields simultaneously an iceberg topic is a redpanda enterprise feature that automatically mirrors a kafka topic into apache iceberg format in an external object store (e g azure data lake, s3 compatible storage) apache iceberg is an open table format designed specifically for large scale analytical workloads dimension kafka topic iceberg topic (cold storage) optimised for real time streaming, sequential reads analytical queries, columnar scans retention short to medium term (configurable) long term / permanent query interface kafka consumer api sql (via spark, trino, databricks, snowflake, etc ) storage format kafka log segments columnar parquet files + iceberg metadata typical consumers services, reporting app bi platforms, data scientists, data lake queries etl pipeline required n/a no — offloading is automatic in practice, the kafka topic continues to serve real time consumers as normal redpanda simultaneously and automatically offloads the same events to the iceberg table in cold storage there is no additional etl job, no separate pipeline to maintain, and no duplication of effort bi platforms and data lake tools connect directly to the iceberg table using standard sql and they do not interact with the kafka broker at all how does mantsu use redpanda? kafka topics in mantsu a kafka topic is a named, ordered, durable log of events producers append events to the end of the log; consumers read from any position in the log at their own pace, independently of each other key characteristics durability events are persisted to disk for a configurable retention period a consumer that was offline can replay all missed events on reconnection consumer groups multiple independent consumers can each maintain their own read position on the same topic, receiving the full stream without interfering with one another ordering events within a topic partition are strictly ordered; producers and consumers can rely on sequence scalability topics are partitioned across broker nodes, distributing load horizontally in mantsu, every significant business event is modelled as a kafka topic this ensures that all services and the reporting application receive a complete, ordered, replayable record of what happened in the plant — without any service needing to poll or query another service's database directly clustering redpanda has out of the box capabilities to run multiple cluster for load balancing & reliability for production environments multiple brokers can be setup depending on the expected load and redundancy (fail over) the cluster size and replication factor are determined during the implementation phase based on throughput, retention, and failover requirements agentic data plane use cases in mantsu the agentic data plane is the foundation for future ai assisted capabilities, such as anomaly detection on production events intelligent downtime root cause suggestions predictive quality interventions this could be obtained without requiring custom data pipelines for each use case agents query the live and historical event stream directly through the governed mcp interface iceberg topics in mantsu in mantsu iceberg topics allow the full history of mes business events (every order, quality result, downtime, and recipe change) to accumulate in the customer's data lake over time this history becomes the foundation for long term analytics, trend reporting, kpi benchmarking, and machine learning model training all using the customer's preferred bi tooling without any mantsu specific connector