Labelling Deployment always restarts it

I am using the Kubernetes API for Java in my Kotlin app.

Whenever I am patching a deployment for the sole purpose of adding a single label key-value pair, the deployment is being restarted. I used the Strategic Merge Patch and JSON Merge Patch, respectively. Both have the same undesired side-effect. I did not use the generic JSON Patch strategy, because I think it is bad practice to get rid of the whole Kubernetes API abstraction layer just to patch a Kubernetes object.

How can I use the Java implementation of the Kubernetes API to patch a deployment (i.e. add a single label key-value pair to it), without making the deployment restart?

Apparently, labelling strictly the Deployment, but not the PodTemplateSpec, which is part of the Deployment, makes the whole thing not restart.