Kubernetes troubleshooting

• Kubernetes troubleshooting steps

Commands

• kubectl logs your_pod
• kubectl get events -A
• kubectl describe pod your_pod
• kubectl describe nodes, review conditions:
• kubectl top
• kubectl cluster-info dump

• Tools: K9s and crictl

Events

• FailedScheduling: Insufficient cpu, Insufficient memory, timed out waiting for the condition, unbound immediate PersistentVolumeClaims, preemption, had untolerated taint(s)
• kubectl get events, OOMKilling, FailedKillPod, SuccessfulDelete, SuccessfulCreate, NoPods, Warning, Critical, NodeSysctlChange, FailedAttachVolume, FailedMount, UnAvailableLoadBalancer, FailedCreatePodSandBox, InvalidDiskCapacity, Scheduled, NetworkNotReady, Evict, Killing, SuccessfulReconcilied, FailedToUpdateEndpointSlices, BackendNotFound, FailedScheduling, ProvisioningFailed, --event-ttl, TaintManagerEviction
• Kubernetes node events: Kubernetes node events, NodeNotSchedulable, NodeAllocatableEnforced, NodeHasNoDiskPressure, DiskPressure, NodeHasSufficientMemory, NodeHasSufficientPID, RegisteredNode, InvalidDiskCapacity, Starting, NodeReady, RemovingNode

Kubernetes components

Load Balancer

• UnAvailableLoadBalancer

Kubelet

• PLEG is not healthy
• /var/log/kubelet.log

Scheduling

• Kubernetes scheduling
• Kubernetes Pod Topology Spread Constraints
• Kubernetes pod affinity and anti affinity
• Karpenter
• ttlSecondsUntilExpired, controller.node Triggering termination for expired node after 168h0m0s .../...

etcd

Log

• Karpenter logs
• Kubelet logs
• /var/log/kubelet.log

Related

• Readiness, Liveness, Readiness probe errored
• Reason: ProbeWarning
• Kubernetes Pod Disruptions
• Unable to connect to the server, ~/.kube/config
• DiskPressure
• CalculateExpectedPodCountFailed
• aws eks create-cluster --logging
• Node-pressure Eviction
• karpenter.sh/do-not-evict: true
• NodeNotReady
• kubectl-node-shell
• kubectl exec, kubectl attach
• EKS troubleshooting
• Kubernetes info
• node.kubernetes.io/disk-pressure
• ktop

Activities

• Review: https://learnk8s.io/troubleshooting-deployments
• Kubernetes debugging with an ephemeral debug container: kubectl debug

See also

• kubectl get events, OOMKilling, FailedKillPod, SuccessfulDelete, SuccessfulCreate, NoPods, Warning, Critical, NodeSysctlChange, FailedAttachVolume, FailedMount, UnAvailableLoadBalancer, FailedCreatePodSandBox, InvalidDiskCapacity, Scheduled, NetworkNotReady, Evict, Killing, SuccessfulReconcilied, FailedToUpdateEndpointSlices, BackendNotFound, FailedScheduling, ProvisioningFailed, --event-ttl, TaintManagerEviction
• Kubernetes node events: Kubernetes node events, NodeNotSchedulable, NodeAllocatableEnforced, NodeHasNoDiskPressure, DiskPressure, NodeHasSufficientMemory, NodeHasSufficientPID, RegisteredNode, InvalidDiskCapacity, Starting, NodeReady, RemovingNode
• K8s troubleshooting: kubectl logs, kubectl top, kubectl get events -A, kubectl describe pod, Liveness, Readiness, Kubernetes events, Pulling image, OOMKilled, ProbeWarning, Reason, FailedScheduling, errImagePull, ImagePullBackOff, Kubelet conditions: MemoryPressure, DiskPressure, KubeletHasSufficientPID, KubeletReady, kubectl [ debug | attach | exec ] kubectl cluster-info dump, SimKube, KWOK, CrashLoopBackOff