High disk watermark

{"type": "server", "timestamp": "2022-10-18T12:35:19,201Z", "level": "WARN", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "high disk watermark [90%] exceeded on [R8X3DL8_R22p0tlev4Jkog][elasticsearch-master-0][ /usr/share/elasticsearch/data/nodes/0 ] free: 3.2gb[5.5%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete", "cluster.uuid": "uXXXXXpg", "node.id": "a-XQ"  }

{"type": "server", "timestamp": "2022-10-18T12:36:58,853Z", "level": "WARN", "component": "o.e.c.r.a.AllocationService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-1", "message": "failing shard [failed shard, shard [reputation_staging][0], node[R8Xadffdkog], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=ijIDfNnlRp-j9jFMPy5PBA], unassigned_info[[reason=CLUSTER_RECOVERED], at[2022-10-18T12:26:18.006Z], delayed=false, allocation_status[no_attempt]], message [failed recovery], failure [RecoveryFailedException[[reputation_staging][0]: Recovery failed from {elasticsearch-master-1}{aBfUKsO9QvKVvRXq_lo-XQ}{XE6viq6KToG1uMFb70Ihzg}{10.12.7.35}{10.12.7.35:9300}{dilmrt}{ml.machine_memory=2684354560, ml.max_open_jobs=20, xpack.installed=true, transform.node=true} into {elasticsearch-master-0}{R8X3DL8_R22p0tlev4Jkog}{Vzjse4lRSHilYFWA8cPJ9g}{10.12.15.47}{10.12.15.47:9300}{dilmrt}{ml.machine_memory=2684354560, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; nested: RemoteTransportException[[elasticsearch-master-1][10.12.7.35:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[elasticsearch-master-0][10.12.15.47:9300][internal:index/shard/recovery/file_chunk]]; nested: IOException [ No space left on device ]; ], markAsStale [true]]", "cluster.uuid": "uTbgs0L7Q4aJN_ES4s8gpg", "node.id": "aBfadfadsXq-XQ" , 

Related

• low disk watermark

See also

• Elasticsearch troubleshooting, Elasticsearch logs, yellow, Low disk watermark, in-memory shard lock, re-running