Cluster filesystem is almost full

Symptom

A CustomerClusterFilesystemAlmostFull alert is firing.

CustomerClusterFilesystemAlmostFull alerts are are sent to subscribers via email.

Observations

Here are the details on the alerts:

Alert: CustomerClusterFilesystemAlmostFull

Expression

instance:node_filesystem_avail:ratio * 100 < 1

Delay

15 minutes

Labels

team: customer

Annotations

  • summary

  • description

  • tenant

  • cluster_id

  • cluster_name

  • instance

  • device

  • mountpoint

Determine cluster node and filesystem filling up

The alert will contain:

  • the cluster name/id (the k8s_cluster_name and k8s_cluster_id labels)

  • the cluster node (the instance label)

  • the device (the device label)

  • the filesystem (the mountpoint label)

Note the values of these labels.

Solutions

This section provides solutions that should help resolve the issue in most cases.

Increase cluster node size

The size of the device used by the cluster node can be increased in AWS. Use the cluster node value (instance label) to find the node and the device (device label) running out of space and increase the device size in the AWS EC2 web interface or AWS command line.

Feedback

PaaS

×

Location

This widget lets you know where you are on the docs site.

You are currently perusing through the DX Cloud docs.

Main doc sections

DX Core Headless PaaS Legacy Cloud Incubator modules