When working with Kubernetes External Secrets Operator (ESO) and its webhook provider, you might encounter some interesting URL encoding challenges. In this post, I’ll walk through a specific issue we encountered and how we solved it, which might help others facing similar problems.
The Setup
We started with a basic ClusterSecretStore
configuration using ESO’s webhook provider:
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: webhook-backend
spec:
provider:
webhook:
url: "http://eso-webhook.default.svc.cluster.local:80/webhook?{{ .remoteRef.key }}"
method: GET
result:
jsonPath: "$"
Our webhook server was a simple Go application designed to handle requests and return secret values based on a path parameter. We expected clean, separate query parameters, but reality had different plans.
The Problem
When we started debugging why our webhook wasn’t working as expected, we added some basic logging to our Go server:
key := r.URL.Query().Get("key")
fmt.Println("key", key)
To our surprise, the key was empty! Further investigation by printing the raw URL revealed something interesting:
Method: GET
URL: /webhook?key%3Dcoca-cola.cluster-001%26path%3Dnamespaces.cokeSystem.secrets.adminCredentials.value
Protocol: HTTP/1.1
The entire query string was URL-encoded as a single parameter. Instead of getting our parameters separately, everything was encoded within the key
parameter.
The Solution
Understanding What’s Happening
When ESO processes the template in our ClusterSecretStore URL, it’s taking the entire remoteRef.key
value and inserting it as-is into the URL. If our remoteRef.key
contains URL-special characters (like =
and &
), they get encoded, resulting in our doubly-nested query string.
Fixing the Server Code
Here’s how we modified our webhook server to handle this situation properly:
func webhookHandler(w http.ResponseWriter, r *http.Request) {
// Get the 'key' query parameter
encodedKey := r.URL.Query().Get("key")
// Unescape the key parameter
decodedKey, err := url.QueryUnescape(encodedKey)
if err != nil {
http.Error(w, "Failed to decode key parameter", http.StatusBadRequest)
return
}
// Parse the decoded key as a query string
values, err := url.ParseQuery(decodedKey)
if err != nil {
http.Error(w, "Invalid key format", http.StatusBadRequest)
return
}
authKey := values.Get("key")
path := values.Get("path")
// Validate and use the parameters
if authKey != "coca-cola.cluster-001" {
http.Error(w, "Invalid authentication key", http.StatusUnauthorized)
return
}
if path == "" {
http.Error(w, "Path is required", http.StatusBadRequest)
return
}
// Process the request...
}
The key improvements in this solution are:
- Using
url.QueryUnescape()
to properly decode the URL-encoded parameter - Using
url.ParseQuery()
to parse the decoded string as a query string itself - Extracting the actual key and path values from the parsed query parameters
Using the ClusterSecretStore
With this server implementation, we can now use our ClusterSecretStore like this:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: example-external-secret
spec:
secretStoreRef:
name: webhook-backend
kind: ClusterSecretStore
target:
name: example-secret
data:
- secretKey: adminPassword
remoteRef:
key: "key=coca-cola.cluster-001&path=namespaces.cokeSystem.secrets.adminCredentials.value"
Lessons Learned
- Always implement detailed request logging when debugging webhook issues
- Don’t assume how parameters will be encoded or structured in webhook requests
- Use standard library functions for URL parsing and decoding instead of custom string manipulation
- Remember that template variables in Kubernetes resources might be processed differently than you expect
Conclusion
URL encoding issues can be tricky to debug, especially when dealing with nested query parameters. By understanding how ESO processes template variables and using proper URL parsing tools, we can build robust webhook handlers that correctly handle these scenarios.
Remember to always implement proper logging during development and testing phases - it makes debugging these kinds of issues much easier. The Go standard library provides excellent tools for handling URL encoding and parsing, so make use of them instead of trying to parse query strings manually.
This solution provides a clean way to handle the URL encoding challenges while maintaining the security and functionality requirements of our webhook provider.