AWS Managed Services for Kafka (Streaming)

The Kafka stream high-fidelity digital twin in your Tensor9 AWS account mirrors the operational state of the Kafka cluster running in the customer appliance. This allows you to monitor the performance of the streaming platform and diagnose potential issues without direct access to the customer’s environment.

CloudWatch Dashboard

The CloudWatch dashboard provides key metrics to monitor the health and performance of the Kafka cluster, such as:

• Messages In/Out per Topic: The number of messages produced and consumed for each topic.
• Consumer Lag: The difference between the last produced message and the latest consumed message, helping detect delays.
• Partition Count: The number of partitions per topic, indicating the level of data parallelism.
• Broker CPU and Memory Utilization: The resource usage of brokers in the cluster.
• Disk Utilization: The storage space used by each broker for logs and message retention.
• Error Rate: The number of failed produce and consume requests.

These metrics provide an overview of cluster performance, enabling early detection of issues such as slow consumers, broker overload, or insufficient partitioning.

Audit Logging and Security

• Allow-Listed Metrics: The customer must explicitly allow-list which metrics can be sent to the digital twin.
• Traceability: All synchronized metrics are appended to the customer’s audit log to maintain a record of vendor interactions.

By using the Kafka stream high-fidelity digital twin, you can ensure that your streaming workloads remain performant and reliable across customer environments while adhering to security and compliance requirements.