Excessive disk use on alpha-worker-7 #14

New Issue

2025-03-09T14:59:16Z

eric commented

2025-03-09 14:59:16 +00:00

I am getting a lot of alarms for excessive disk use on this worker. In truth, all of the workers probably should be upgraded to 32GB disks.

eric commented

2025-03-11 13:59:51 +00:00

Oddly, this is still firing after logs have been rotated and images have been pruned:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.8G     0  7.8G   0% /dev
tmpfs           1.6G  2.0M  1.6G   1% /run
/dev/vda1        16G  7.7G  7.3G  52% /
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/vda15      124M   12M  113M  10% /boot/efi
tmpfs           1.6G     0  1.6G   0% /run/user/1000
...

The alerting rule is set to resolve once storage has dropped below 80%. I can silence the alarm by removing the unique resolution rule.

Oddly, this is still firing after logs have been rotated and images have been pruned: ``` $ df -h Filesystem Size Used Avail Use% Mounted on udev 7.8G 0 7.8G 0% /dev tmpfs 1.6G 2.0M 1.6G 1% /run /dev/vda1 16G 7.7G 7.3G 52% / tmpfs 7.9G 0 7.9G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock /dev/vda15 124M 12M 113M 10% /boot/efi tmpfs 1.6G 0 1.6G 0% /run/user/1000 ... ``` The alerting rule is set to resolve once storage has dropped below 80%. I can silence the alarm by removing the unique resolution rule.

eric commented

2025-03-11 15:35:20 +00:00

I restarted the prometheus-node-exporter service and the fields updated successfully.

eric closed this issue

2025-03-11 15:35:20 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: DevOps/software-infrastructure#14