RedPanda Streaming broker

what is the redpanda node? audience this section is written primarily for it architects, integration engineers, and data teams who design the event backbone or plan to connect analytics or ai tooling to the event stream non technical readers, such as project stakeholders, can read it to understand how business events flow through mantsu the redpanda node is the central event backbone of the mantsu architecture it hosts the redpanda streaming broker , which acts as the message bus for all business events exchanged between services what is the redpanda streaming broker? it is a modern, high performance event streaming platform that acts as an alternative to apache kafka it is built from the ground up to be light weight & efficient and it is engineered as a drop in replacement for kafka; you can use your existing kafka client applications and tools with redpanda without changing anything what is redpanda's role in the architecture? redpanda functions as the it tier broker of the mantsu unified namespace docid\ u8kwmvus5wjtrln2whh6y all significant business events, such as order state changes, quality results, downtime registrations, and recipe updates, are published as kafka topics by the originating mendix service topic names follow the isa 95 hierarchical naming convention, making each event self describing and discoverable within the uns downstream consumers (other mendix services, the reporting application, bi platforms) subscribe to the relevant topics independently mqtt streams from ot edge nodes do not pass through redpanda they are consumed directly by the services or the reporting application that require them redpanda handles business level events in the uns it tier; raw sensor and machine data lives in the ot tier and is served by the mqtt broker technology redpanda redpanda streaming broker redpanda is a kafka compatible streaming broker written in c++ it is fully api compatible with apache kafka, meaning any standard kafka producer, consumer, or admin tool connects to it without code changes why redpanda over apache kafka no jvm dependency redpanda runs as a single native binary kafka requires a java virtual machine and a separate zookeeper (or kraft) cluster for coordination, adding significant operational overhead redpanda handles all coordination internally, reducing the deployment footprint considerably ultra low latency redpanda utilizes a thread per core architecture this allows it to bypass context switching and avoid unpredictable jvm garbage collection pauses, delivering up to 10x lower tail latency compared to traditional kafka lower resource consumption the c++ runtime is substantially more memory and cpu efficient than the jvm, making redpanda viable on smaller infrastructure kafka api compatibility existing tooling, client libraries, connectors, and integrations built for kafka work against redpanda without modification agentic data plane available for mid to large deployments requires a redpanda enterprise license the agentic data plane is a redpanda enterprise module that exposes the streaming broker as a governed data source for ai agents it provides two complementary integration protocols mcp (model context protocol) an open protocol that allows ai agents and large language models to query redpanda topics as structured data sources an mcp server is exposed by the agentic data plane; agents connect to it to retrieve, filter, and reason over event data without needing direct broker access a2a (agent to agent) a protocol enabling multiple ai agents to exchange information and coordinate actions in the mantsu context this allows specialised agents (e g a quality analysis agent, a production planning agent) to collaborate by sharing observations derived from the event stream how is agent access governed? all agent authorisations, topic access permissions, and connection configurations are managed through redpanda's built in management interface this ensures that agents operate within defined boundaries and that access to sensitive business event data is controlled and auditable what are redpanda iceberg topics? available for mid to large deployments requires a redpanda enterprise license a standard kafka topic is optimised for streaming events flow in real time, are retained for a defined period (hours to weeks typically), and are consumed sequentially kafka storage is a write optimised log; it is not designed for analytical queries that span large time ranges or filter across many fields simultaneously an iceberg topic is a redpanda enterprise feature that automatically mirrors a kafka topic into apache iceberg format in an external object store (e g azure data lake, s3 compatible storage) apache iceberg is an open table format designed specifically for large scale analytical workloads dimension kafka topic iceberg topic (cold storage) optimised for real time streaming, sequential reads analytical queries, columnar scans retention short to medium term (configurable) long term / permanent query interface kafka consumer api sql (via spark, trino, databricks, snowflake, etc ) storage format kafka log segments columnar parquet files + iceberg metadata typical consumers services, reporting app bi platforms, data scientists, data lake queries etl pipeline required n/a no, offloading is automatic in practice, the kafka topic continues to serve real time consumers as normal redpanda simultaneously and automatically offloads the same events to the iceberg table in cold storage there is no additional etl job, no separate pipeline to maintain, and no duplication of effort bi platforms and data lake tools connect directly to the iceberg table using standard sql and they do not interact with the kafka broker at all how does mantsu use redpanda? kafka topics in mantsu a kafka topic is a named, ordered, durable log of events producers append events to the end of the log; consumers read from any position in the log at their own pace, independently of each other key characteristics durability events are persisted to disk for a configurable retention period a consumer that was offline can replay all missed events on reconnection consumer groups multiple independent consumers can each maintain their own read position on the same topic, receiving the full stream without interfering with one another ordering events within a topic partition are strictly ordered; producers and consumers can rely on sequence scalability topics are partitioned across broker nodes, distributing load horizontally in mantsu, every significant business event is modelled as a kafka topic this ensures that all services and the reporting application receive a complete, ordered, replayable record of what happened in the plant, without any service needing to poll or query another service's database directly how does clustering work? redpanda has out of the box capabilities to run multiple clusters for load balancing & reliability for production environments multiple brokers can be setup depending on the expected load and redundancy (fail over) the cluster size and replication factor are determined during the implementation phase based on throughput, retention, and failover requirements agentic data plane use cases in mantsu the agentic data plane is the foundation for future ai assisted capabilities, such as anomaly detection on production events intelligent downtime root cause suggestions predictive quality interventions this could be obtained without requiring custom data pipelines for each use case agents query the live and historical event stream directly through the governed mcp interface iceberg topics in mantsu in mantsu iceberg topics allow the full history of mes business events (every order, quality result, downtime, and recipe change) to accumulate in the customer's data lake over time this history becomes the foundation for long term analytics, trend reporting, kpi benchmarking, and machine learning model training all using the customer's preferred bi tooling without any mantsu specific connector frequently asked questions what role does redpanda play in mantsu? it is the it tier broker of the mantsu uns; every business event (order, quality, downtime, recipe) is published once as a kafka topic and consumed by any service that needs it can mantsu keep using existing kafka tooling? yes redpanda is a drop in kafka replacement, so standard kafka producers, consumers, and admin tools connect without code changes does machine data flow through redpanda in mantsu? no raw sensor and machine data stays in the ot tier on the mqtt broker; redpanda handles business events what do iceberg topics add in a mantsu deployment? on mid to large deployments they automatically offload the full mes event history to the customer's data lake in apache iceberg format, queryable with standard sql and without an etl pipeline summary redpanda is the kafka compatible event broker at the heart of mantsu every business event is published once and consumed by any service that needs it, with optional enterprise features for ai agents, governance, and data lake offloading