AWS Elastic MapReduce (Batch processing)

The Elastic MapReduce (EMR) high-fidelity digital twin in your Tensor9 AWS account mirrors the operational state of the EMR cluster running in the customer appliance. This allows you to monitor the performance and state of distributed batch processing jobs without requiring direct access to the customer’s environment.

CloudWatch Dashboard

The CloudWatch dashboard provides key metrics to monitor the health and performance of the EMR cluster, such as:

• Cluster State: The current state of the cluster (e.g., STARTING, RUNNING, WAITING, TERMINATING).
• Job Execution Count: The number of jobs executed within the cluster.
• Task Progress: Percentage completion of active tasks within the EMR job.
• Node Health Status: The health status of the master, core, and task nodes in the cluster.
• CPU and Memory Usage: The average and peak resource usage for the EMR nodes.
• HDFS Storage Utilization: The amount of HDFS storage used within the cluster.
• Failed Task Count: The number of failed tasks during batch processing.

These metrics help you monitor the overall cluster performance and quickly detect issues, such as resource bottlenecks or failed tasks.

Audit Logging and Security

• Allow-Listed Metrics: The customer must explicitly allow-list which logs and metrics can be sent to the digital twin.
• Traceability: All synchronized logs and metrics are appended to the customer’s audit log for full transparency and traceability.

By using the EMR high-fidelity digital twin, you can gain real-time insights into your distributed batch processing workflows running in customer environments, ensuring that jobs run reliably and resource usage stays efficient.