Difference between revisions of "Kubectl get events"
Jump to navigation
Jump to search
(→Events) |
|||
| (23 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
{{lc}} | {{lc}} | ||
| + | TOMERGE: [[Kubernetes node events]] | ||
| + | |||
| + | * <code>[[kubectl get events --help]]</code> | ||
| + | * <code>[[kubectl get events -A]]</code> | ||
| + | * <code>[[kubectl get events -A]] | grep [[Warning]]</code> | ||
| + | * <code>[[kubectl get events -A -o wide]] | grep [[Warning]]</code> | ||
* <code>[[kubectl get]] events</code> | * <code>[[kubectl get]] events</code> | ||
| − | * <code>[[kubectl get events - | + | * <code>[[kubectl get events -o wide]]</code> |
* <code>kubectl get events -o yaml</code> | * <code>kubectl get events -o yaml</code> | ||
| Line 10: | Line 16: | ||
* <code>[[kubectl -n gitlab-runner get events --field-selector type!=Normal]]</code> | * <code>[[kubectl -n gitlab-runner get events --field-selector type!=Normal]]</code> | ||
| − | + | ||
kubectl get events -A | grep [[Warning]] | egrep "[[FailedMount]]|[[FailedAttachVolume]]|[[Unhealthy]]|[[ClusterUnhealthy]]|[[FailedScheduling]]" | kubectl get events -A | grep [[Warning]] | egrep "[[FailedMount]]|[[FailedAttachVolume]]|[[Unhealthy]]|[[ClusterUnhealthy]]|[[FailedScheduling]]" | ||
| Line 16: | Line 22: | ||
== Events examples == | == Events examples == | ||
| + | * [[Normal]], [[Warning]], [[Critical]] | ||
| + | |||
=== Warning === | === Warning === | ||
| + | [[BackoffLimitExceeded]] | ||
| + | [[CalculateExpectedPodCountFailed]] | ||
| + | [[ClusterUnhealthy]] | ||
| + | [[FailedMount]] | ||
| + | [[FailedScheduling]] | ||
| + | [[InvalidDiskCapacity]] | ||
[[Unhealthy]] | [[Unhealthy]] | ||
| − | |||
.../... | .../... | ||
| + | |||
| + | |||
your_namespace 28s Warning [[FailedScheduling]] pod/kibana-kibana-654ccb45bd-pbp4r 0/2 nodes are available: 2 [[Insufficient cpu]]. | your_namespace 28s Warning [[FailedScheduling]] pod/kibana-kibana-654ccb45bd-pbp4r 0/2 nodes are available: 2 [[Insufficient cpu]]. | ||
| Line 48: | Line 63: | ||
| − | [[kube-system]] 43m [[Warning]] [[ClusterUnhealthy]] [[configmap/]][[cluster-autoscaler-status]] | + | [[kube-system]] 43m [[Warning]] [[ClusterUnhealthy]] [[configmap/]] |
| + | [[cluster-autoscaler-status]] | ||
| + | |||
| + | |||
| + | LAST SEEN TYPE REASON OBJECT MESSAGE | ||
| + | 28s Warning FailedScheduling pod/deployment-1230/3 nodes | ||
| + | are available: 3 [[persistentvolumeclaim]] "your" bound to non-existent | ||
| + | persistentvolume "". | ||
| + | 19m Warning FailedScheduling pod/deployment-123 0/3 nodes | ||
| + | are available: 3 persistentvolumeclaim "your" not found. | ||
| + | 10m Warning FailedScheduling pod/deployment-91234 0/3 nodes | ||
| + | are available: 3 persistentvolumeclaim "your" bound to non-existent | ||
| + | persistentvolume "". | ||
| + | 17m Warning [[ClaimLost]] persistentvolumeclaim/yourclaim [[Bound claim has lost reference to PersistentVolume]]. Data on the volume is lost! [[Cluster has no ready nodes]]. | ||
=== Normal === | === Normal === | ||
| Line 63: | Line 91: | ||
[[NodeNotSchedulable]] | [[NodeNotSchedulable]] | ||
[[RemovingNode]] | [[RemovingNode]] | ||
| + | [[TaintManagerEviction]] | ||
| + | [[WaitForFirstConsumer]] | ||
| + | [[ExternalProvisioning]] | ||
| + | [[TaintManagerEviction: Cancelling deletion of pod]] | ||
| Line 85: | Line 117: | ||
* <code>[[FailedKillPod]]</code> | * <code>[[FailedKillPod]]</code> | ||
* <code>[[FailedScheduling]]</code> | * <code>[[FailedScheduling]]</code> | ||
| + | * <code>[[FailedToUpdateEndpoint]]</code> | ||
* <code>[[FailedToUpdateEndpointSlices]]</code> | * <code>[[FailedToUpdateEndpointSlices]]</code> | ||
* <code>[[Generated]]</code> | * <code>[[Generated]]</code> | ||
| Line 96: | Line 129: | ||
* <code>[[SuccessfulCreate]]</code> | * <code>[[SuccessfulCreate]]</code> | ||
* <code>[[SuccessfulDelete]]</code> | * <code>[[SuccessfulDelete]]</code> | ||
| + | * <code>[[NetworkNotReady]]</code> | ||
* <code>[[NodeNotReady]]</code> | * <code>[[NodeNotReady]]</code> | ||
| + | * <code>[[NodeAllocatableEnforced]]</code> | ||
* <code>[[NoPods]]</code> | * <code>[[NoPods]]</code> | ||
* <code>[[NodeHasNoDiskPressure]]</code> | * <code>[[NodeHasNoDiskPressure]]</code> | ||
* <code>[[UnAvailableLoadBalancer]]</code> | * <code>[[UnAvailableLoadBalancer]]</code> | ||
| + | * <code>[[Unhealthy]]</code> | ||
* <code>[[VolumeFailedDelete]]</code> | * <code>[[VolumeFailedDelete]]</code> | ||
== Related == | == Related == | ||
| + | * <code>[[kubectl events]]</code> | ||
* <code>[[kubectl top]]</code> | * <code>[[kubectl top]]</code> | ||
* <code>[[kubectl logs]]</code> | * <code>[[kubectl logs]]</code> | ||
* <code>[[gcloud logging read resource.labels.cluster_name]]</code> | * <code>[[gcloud logging read resource.labels.cluster_name]]</code> | ||
| − | |||
* [[job-controller]] | * [[job-controller]] | ||
* [[GCP Node logs]] | * [[GCP Node logs]] | ||
| Line 117: | Line 153: | ||
* {{kubectl events}} | * {{kubectl events}} | ||
* {{K8s troubleshooting}} | * {{K8s troubleshooting}} | ||
| + | * {{K8s monitoring}} | ||
[[Category:K8s]] | [[Category:K8s]] | ||
Latest revision as of 17:53, 10 April 2024
TOMERGE: Kubernetes node events
kubectl get events --helpkubectl get events -Akubectl get events -A | grep Warningkubectl get events -A -o wide | grep Warningkubectl get eventskubectl get events -o widekubectl get events -o yaml
kubectl get events --sort-by=.metadata.creationTimestampkubectl get events --sort-by=.metadata.creationTimestamp -Akubectl get events --sort-by='.lastTimestamp
kubectl get events -A | grep Warning | egrep "FailedMount|FailedAttachVolume|Unhealthy|ClusterUnhealthy|FailedScheduling"
kubectl get events -A | grep Normal
Events examples[edit]
Warning[edit]
BackoffLimitExceeded CalculateExpectedPodCountFailed ClusterUnhealthy FailedMount FailedScheduling InvalidDiskCapacity Unhealthy .../...
your_namespace 28s Warning FailedScheduling pod/kibana-kibana-654ccb45bd-pbp4r 0/2 nodes are available: 2 Insufficient cpu.
your_namespace 4m53s Warning ProbeWarning pod/metabase-prod-f8f4b765b-h4pgs Readiness probe warning:
your_namespace 30m Warning BackoffLimitExceeded job/your-job27740460 Job has reached the specified backoff limit
your_namespace 26m Warning Unhealthy pod/elasticsearch-master-1 Readiness probe failed: Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )...
your_namespace 99s Warning BackOff pod/elasticsearch-master-0 Back-off restarting failed container your_namespace 108s Warning BackOff pod/elasticsearch-master-1 Back-off restarting failed container
your_namespace 12m Warning PresentError challenge/prod-admin-tls-cert-dzmbt-2545 Error presenting challenge: error getting clouddns service account: secret "clouddns-dns01-solver-svc-acct" not found
your_namespace 27m Warning OOMKilling node/gke-you-pool4 Memory cgroup out of memory: Killed process 2768158 (python) total-vm:5613088kB, anon-rss:3051580kB, file-rss:65400kB, shmem-rss:0kB, UID:0 pgtables:7028kB oom_score_adj:997
your_namespace 8m51s Warning FailedScheduling pod/myprometheus-alertmanager-5967d4ff85-5glkh running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition default 4m58s Normal ExternalProvisioning persistentvolumeclaim/myprometheus-alertmanager waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Solution: Install aws-ebs-csi-driver
default 107s Warning ProvisioningFailed persistentvolumeclaim/myprometheus-server (combined from similar events): failed to provision volume with StorageClass "gp2": rpc error: code = Internal desc = Could not create volume "pvc-4e14416c-c9c2-4d39-b749-9ce0fa98d597": could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: Goz6E3qExxxxx.../...
kube-system 9m44s Warning FailedMount pod/kube-dns-85df8994db-v8qdg MountVolume.SetUp failed for volume "kube-dns-config" : failed to sync configmap cache: timed out waiting for the condition
kube-system 43m Warning ClusterUnhealthy configmap/ cluster-autoscaler-status
LAST SEEN TYPE REASON OBJECT MESSAGE 28s Warning FailedScheduling pod/deployment-1230/3 nodes are available: 3 persistentvolumeclaim "your" bound to non-existent persistentvolume "". 19m Warning FailedScheduling pod/deployment-123 0/3 nodes are available: 3 persistentvolumeclaim "your" not found. 10m Warning FailedScheduling pod/deployment-91234 0/3 nodes are available: 3 persistentvolumeclaim "your" bound to non-existent persistentvolume "". 17m Warning ClaimLost persistentvolumeclaim/yourclaim Bound claim has lost reference to PersistentVolume. Data on the volume is lost! Cluster has no ready nodes.
Normal[edit]
Started Created Pulled Pulling Scheduled Killing Evict SandboxChanged SuccessfulCreate - ReplicaSet SuccessfulDelete NodeNotSchedulable RemovingNode TaintManagerEviction WaitForFirstConsumer ExternalProvisioning TaintManagerEviction: Cancelling deletion of pod
default 4s Normal Provisioning persistentvolumeclaim/myprometheus-alertmanager External provisioner is provisioning volume for claim "default/myprometheus-alertmanager"
Related: kubectl get pvc
ingress-nginx 53m Normal UpdatedLoadBalancer service/nginx-ingress-controller Updated load balancer with new hosts ingress-nginx 54m Warning UnAvailableLoadBalancer service/nginx-ingress-controller There are no available nodes for LoadBalancer
Events[edit]
BackOffCompletedCreatedDeadlineExceededFailedFailedAttachVolumeFailedCreatePodSandBoxFailedMountFailedKillPodFailedSchedulingFailedToUpdateEndpointFailedToUpdateEndpointSlicesGeneratedPresentErrorPulledPullingRequestedSawCompletedJobScheduledStartedSuccessfulCreateSuccessfulDeleteNetworkNotReadyNodeNotReadyNodeAllocatableEnforcedNoPodsNodeHasNoDiskPressureUnAvailableLoadBalancerUnhealthyVolumeFailedDelete
Related[edit]
kubectl eventskubectl topkubectl logsgcloud logging read resource.labels.cluster_name- job-controller
- GCP Node logs
gcloud logging read projects/yourproject/logs/kubeletkubectl describe nodes (conditions:)kubectl describe nodes | grep KubeletReady--event-ttldefines the amount of time to retain events, default 1h.
See also[edit]
- Kubernetes events, Kubernetes node events:
[ Normal | Warning ],kubectl events, kubectl get events:BackOff,FailedMount, FailedAttachVolume, TaintManagerEviction, InvalidDiskCapacity,FailedScheduling, kubectl describe events, Unhealthy,conditions:, UpdatedLoadBalancer, BackoffLimitExceeded - K8s troubleshooting:
kubectl logs, kubectl top, kubectl get events -A, kubectl describe pod, Liveness, Readiness,Kubernetes events, Pulling image, OOMKilled, ProbeWarning, Reason,FailedScheduling,errImagePull, ImagePullBackOff, Kubelet conditions:MemoryPressure, DiskPressure, KubeletHasSufficientPID, KubeletReady, kubectl [ debug | attach | exec ] kubectl cluster-info dump, SimKube, KWOK - Kubernetes monitoring, node conditions, Kube-state-metrics (KSM), Prometheus, VictoriaMetrics,
node-problem-detector, Thanos, log collection,ProbeWarning, Kubernetes node-problem-detector, Pixie, OpenMetrics,kind: PodMonitor, Jaeger
Advertising:
