Garbage collection is the process by which Kubernetes removes dependent objects after you have deallocated a resource. It also handles the automatic cleanup of unnecessary items in your cluster. Ready-made containers and unused images are periodically removed to avoid accumulating a large number of unnecessary resources over time.

Several garbage collection strategies can be deployed depending on the type of object being deleted. Garbage collection of unused resources is configurable by kubelet settings on the worker nodes of your cluster. In this article, we’ll explain the different forms of garbage collection and how you can customize the clean-up settings.

basic garbage collection

kubernetes garbage collection handle Removing dependent objects, old containers, and unused images, Kubelet periodically runs cleanup tasks while its service is active. This will remove closed containers, such as those from terminated jobs, and remove unused images once a certain disk usage limit is reached.

Garbage collection of cluster resources in response to object deletion a. uses it cascading mechanism, This ensures that an object’s children are removed with it, preventing orphans from coming into existence. You can control this process by requesting a specific cascade order when you start deleting.

owner reference

Parent-child relationships are expressed by Kubernetes owner reference, Owners are responsible for objects nested within themselves. This creates a dependency graph that informs the garbage collection process.

Kubernetes uses owner context data to establish which resources should be removed before the target can be deleted. As an example, Pods that are part of a ReplicaSet will have an owner context that defines this link. Removing a replicaset will also automatically remove the pods inside it.

Owner references are set to the objects in them metadata.ownerReference manifest area. You normally don’t need to inspect or change this field manually because owner references are automatically managed by Kubernetes controllers.

Children can block deletion of their parents metadata.blockOwnerDeletion field for boolean value true, This will stop the parent in the object’s ownerReference The chain cannot be removed unless the object itself is removed.

cascading deletion mechanism

Owner reference chain means there are two ways In which object deletion can proceed:

• Background Removal: The target object is removed first, then its dependents.
• Remove foreground: Dependents are removed first, then the target object.

Consider a delete command like the following:

$ kubectl delete deployment/demo-app

Background remove method will remove immediately demo-app deployment object. This will leave your pods running in your cluster. They will be automatically cleared in the background as part of the garbage collection process.

Foreground delete begins by marking demo-app Deployment as “Deletion is in progress”. It then removes all dependent objects within it. demo-app The deployment will be visible in your cluster until its pods are removed. The deployment will be cleared after the dependents are gone.

Background cascading is the default mechanism. It provides an immediate result that removes the target object from your cluster without any delay. However, foreground deletion may be more desirable in some scenarios, such as when you want to delete a deployment or replica set only after its pods have been destroyed. This avoids the brief period where pods are effectively orphaned after their owner replicasets are already gone.

You can opt-in for foreground deletion by passing --cascade=foreground the flag kubectl delete command:

$ kubectl delete deployment/demo-app --cascade=foreground

pods will be removed, then demo-app deployment.

Disabling dependent garbage collection

There is a third cascade option: ignore dependent garbage collection altogether. This will orphan any dependents of the object, leaving them in your cluster but removing their owner reference.

$ kubectl delete deployment/demo-app --cascade=orphan

This command removes immediately demo-app deployment but retains its pods. They will continue to run in your cluster until they are removed separately, as part of another command. Although this behavior is rarely desirable, it can be helpful if you decide that pods no longer need to be in deployment or replica sets.

Garbage Collection and Finalizer

finalizer It also affects how garbage collection operates. They prevent deletion from occurring until certain conditions are met.

Many apparently “stuck” deletions are caused by pending finalizers. A finalizer that does not mark itself as terminated will prevent objects from being removed from your cluster. Once finalized, the Kubernetes control plane will proceed with the deletion.

There are some finalizers included with Kubernetes, such as pv-protection Defense Against deleting an actively used volume. Others can be added by third-party controllers that you add to your cluster. Overriding the Finalizer Not recommended as it can create a false state that leads to unexpected behavior.

Optimizing garbage collection of unused resources

Garbage collection of unnecessary containers and images occurs automatically on your worker nodes. Five parameters are available To tune the deletion threshold:

• maximum-dead-containers – The maximum number of old containers that can exist after running garbage collection. default of -1 Removes the limit.
• maximum-dead-containers-per-container – Set the maximum number of old containers created on a per-container basis. Controls how many previous versions of the container remain in the cluster after a new replacement is introduced.
• minimum-container-ttl-duration Containers become eligible for garbage collection several minutes after being stopped. default value of 0 Disables delay.
• image-gc-high-threshold – When the disk usage reaches this percentage, Kubelet will attempt to remove the unused images…
• image-gc-low-threshold – … to bring the usage down to this level.

These parameters are set as kubelet launch flags. The flag is usually placed in /var/lib/kubelet/kubeadm-flags.env,

KUBELET_KUBEADM_ARGS="--image-gc-high-threshold=60 --image-gc-low-threshold=50 --minimum-container-ttl-duration=5"

This configuration will allow automatic removal of containers that are unnecessary for at least five minutes. Each garbage collection pass will attempt to shrink old images after they reach 60% disk usage, with the goal of reducing consumption by 50%.

You should restart the kubelet process after making these changes:

$ sudo systemctl daemon-reload
$ sudo systemctl restart kubelet

Since the settings are part of Kubelet, you’ll need to apply them to each of your worker nodes in your cluster if you want the configuration to be consistent across all of them.

summary

Understanding the different forms of Kubernetes garbage collection helps you decide on the right strategy when removing objects from your cluster. The background and foreground cascades affect the order in which dependent resources are cleaned up. Relationships between objects are identified by their owner reference chains. Kubernetes also cleans up unused containers and background images. The threshold for this behavior can be optimized using the kubelet settings.

For best results you should stick to the mechanism provided by Kubernetes. Project warns against using Third-party garbage collection tools because these can create state anomalies that cause problems in your cluster.