Gateway API CRDs failure / kubectl does not use server_side_apply var

[ledroide]

What happened?

Using these variables and values in the inventory :

gateway_api_enabled: true
gateway_api_channel: experimental     # same issue if set to "standard"
server_side_apply: true

Running cluster.yml playbook. Failure at this step :

TASK [kubernetes_sigs.kubespray.kubernetes-apps/common_crds/gateway_api : Gateway API | Install Gateway API] ********************************************************************
skipping: [k8ststmaster-2]
skipping: [k8ststmaster-3]
skipping: [k8ststworker-1]
skipping: [k8ststworker-2]
skipping: [k8ststworker-3]
skipping: [k8ststworker-4]
skipping: [k8ststworker-5]
fatal: [k8ststmaster-1]: FAILED! => {"changed": false, "msg": "error running kubectl (/usr/local/bin/kubectl apply --force --filename=/etc/kubernetes/addons/gateway_api/experimental-install.yaml) command (rc=1), out='customresourcedefinition.apiextensions.k8s.io/backendtlspolicies.gateway.networking.k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/grpcroutes.gateway.networking.k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/referencegrants.gateway.networking.k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/tcproutes.gateway.networking.k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/tlsroutes.gateway.networking.k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/udproutes.gateway.networking.k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/xbackendtrafficpolicies.gateway.networking.x-k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/xlistenersets.gateway.networking.x-k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/xmeshes.gateway.networking.x-k8s.io configured\n', err='The CustomResourceDefinition \"httproutes.gateway.networking.k8s.io\" is invalid: metadata.annotations: Too long: may not be more than 262144 bytes\n'"}

Consequence : The playbook failed as it was updating the HTTPRoute ressource. HTTPRoute does not exist anymore.

$ kubectl api-resources --api-group gateway.networking.k8s.io
NAME                 SHORTNAMES   APIVERSION                           NAMESPACED   KIND
backendtlspolicies   btlspolicy   gateway.networking.k8s.io/v1         true         BackendTLSPolicy
gatewayclasses       gc           gateway.networking.k8s.io/v1         false        GatewayClass
gateways             gtw          gateway.networking.k8s.io/v1         true         Gateway
grpcroutes                        gateway.networking.k8s.io/v1         true         GRPCRoute
referencegrants      refgrant     gateway.networking.k8s.io/v1beta1    true         ReferenceGrant
tcproutes                         gateway.networking.k8s.io/v1alpha2   true         TCPRoute
tlsroutes                         gateway.networking.k8s.io/v1alpha3   true         TLSRoute
udproutes                         gateway.networking.k8s.io/v1alpha2   true         UDPRoute

All HTTPRoute ressources disappeared from my cluster (!)

$ kubectl get httproute -A
No resources found

The root cause is :

• using kubectl without "--server-side" stores the last change history in annotations
• when the history is too big, the kube-apiserver refuses the request
• the file roles/kubernetes-apps/common_crds/gateway_api/tasks/main.yml uses the module "kube"
• the module "kube" is defined in plugins/modules/kube.py

I will propose very soon a fix that will make the module "kube" run kubectl in server-side apply mode if variable server_side_apply: true.

What did you expect to happen?

CRDs update without issue.

If it happens to you, before I fix it, try this :

kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml

If it doesn't work, try this :

kubectl delete crd \
  backendtlspolicies.gateway.networking.k8s.io \
  tcproutes.gateway.networking.k8s.io \
  tlsroutes.gateway.networking.k8s.io \
  udproutes.gateway.networking.k8s.io \
  xbackendtrafficpolicies.gateway.networking.x-k8s.io \
  xlistenersets.gateway.networking.x-k8s.io \
  xmeshes.gateway.networking.x-k8s.io \
  --ignore-not-found

kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml

kubectl annotate gateway -A --all force-sync=$(date +%s) --overwrite

Unfortunately, your missing ressources are lost, you will need to recreate them from your manifests.

How can we reproduce it (as minimally and precisely as possible)?

Update your Gateway API CRDs with cluster.yml playbook. It looks like experimental channel is bigger than standard channel so it has a better chance to fail.

gateway_api_enabled: true
gateway_api_channel: experimental     # same issue if set to "standard"
server_side_apply: true

OS

Ubuntu 24

Version of Ansible

ansible [core 2.18.14]
  config file = /home/shartmann/git/streamlane/ansible.cfg
  configured module search path = ['/home/shartmann/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/shartmann/.local/lib/python3.12/site-packages/ansible
  ansible collection location = /home/shartmann/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/shartmann/.local/bin/ansible
  python version = 3.12.3 (main, Mar  3 2026, 12:15:18) [GCC 13.3.0] (/usr/bin/python3)
  jinja version = 3.1.4
  libyaml = True

Version of Python

Python 3.12.3

Version of Kubespray (commit)

da6539c

Network plugin used

cilium

Full inventory with variables

Command used to invoke ansible

ansible-playbook cluster.yml --tags gateway_api,kubespray_defaults

Output of ansible run

TASK [kubernetes_sigs.kubespray.kubernetes-apps/common_crds/gateway_api : Gateway API | Install Gateway API] ********************************************************************
skipping: [k8ststmaster-2]
skipping: [k8ststmaster-3]
skipping: [k8ststworker-1]
skipping: [k8ststworker-2]
skipping: [k8ststworker-3]
skipping: [k8ststworker-4]
skipping: [k8ststworker-5]
fatal: [k8ststmaster-1]: FAILED! => {"changed": false, "msg": "error running kubectl (/usr/local/bin/kubectl apply --force --filename=/etc/kubernetes/addons/gateway_api/experimental-install.yaml) command (rc=1), out='customresourcedefinition.apiextensions.k8s.io/backendtlspolicies.gateway.networking.k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/grpcroutes.gateway.networking.k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/referencegrants.gateway.networking.k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/tcproutes.gateway.networking.k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/tlsroutes.gateway.networking.k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/udproutes.gateway.networking.k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/xbackendtrafficpolicies.gateway.networking.x-k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/xlistenersets.gateway.networking.x-k8s.io configured\ncustomresourcedefinition.apiextensions.k8s.io/xmeshes.gateway.networking.x-k8s.io configured\n', err='The CustomResourceDefinition \"httproutes.gateway.networking.k8s.io\" is invalid: metadata.annotations: Too long: may not be more than 262144 bytes\n'"}

Anything else we need to know

No response

[tico88612]

/triage accepted