How to make controller-runtime webhook.CustomDefaulter play with kubebuilder defaults markers

Cluster information:

Kubernetes version: Server Version: v1.24.0 / Client Version: v1.30.2
Cloud being used: NA
Installation method: NA
Host OS: Linux
CNI and version: NA
CRI and version: NA

Hello,

In my team we are writing an operator that, until now, was relying on kubebuilder markers for default values and validation.

Now we are implementing admission control and I noticed that, when using the webhook.CustomDefaulter interface to set some defaults that require actual code processing, at the end the update gets rejected with this kind of message

The ApiDefinition "api-with-context" is invalid: 
    * spec.labels: Invalid value: "null": spec.labels in body must be of type array: "null"
    * spec.path_mappings: Invalid value: "null": spec.path_mappings in body must be of type array: "null"
    * spec.resources: Invalid value: "null": spec.resources in body must be of type array: "null"
    * spec.tags: Invalid value: "null": spec.tags in body must be of type array: "null"
    * spec.categories: Invalid value: "null": spec.categories in body must be of type array: "null"
    * spec.metadata: Invalid value: "null": spec.metadata in body must be of type array: "null"
    * spec.pages: Invalid value: "null": spec.pages in body must be of type object: "null"
    * spec.flows: Invalid value: "null": spec.flows in body must be of type array: "null"
    * spec.properties: Invalid value: "null": spec.properties in body must be of type array: "null"

This fields are supposed to get there defaults from kubuilder:default markers.

My question is:

Is it expected behaviour and does using CustomDefaulter makes us having to validate everything in the Default method or is there good chances something is wrong in our config ?

Edit: We are using controller-runtime 18.4

Thanks