Zero to Hero - external-secret-operator

Using External Secrets Operator with HashiCorp Vault to Create Kubernetes Secrets

2024-12-22T00:00:00+00:00

When working with Kubernetes, securely managing secrets is a crucial part of your infrastructure. Traditional approaches—like directly embedding secrets in manifests or manually managing Secret</code> objects—can become cumbersome and less secure over time. Enter the External Secrets Operator (ESO)</a>, a powerful way to integrate external secret stores with Kubernetes. One such secret store is HashiCorp Vault</a>, a proven solution for securely storing and accessing secrets.

In this post, we’ll walk through the key Custom Resource Definitions (CRDs) you need to set up in your cluster to fetch secrets from Vault and have them automatically synchronized as Kubernetes secrets.

Overview</h2> ESO acts as a bridge between external secret stores (in this case, Vault) and Kubernetes Secret</code> objects. To make this happen, you’ll define a store resource that tells ESO where and how to retrieve secrets. You’ll then define another resource that specifies which secrets to fetch and how to transform them into Kubernetes secrets. In other words: SecretStore/ClusterSecretStore: Defines the external secret provider configuration</li> ExternalSecret: Defines the specific secrets you want to pull from your chosen store and how to map them into Kubernetes secrets</li> </ul> Key Resources</h2> 1. SecretStore or ClusterSecretStore</h3> The SecretStore</code> or ClusterSecretStore</code> resource provides information about your Vault instance, including: Endpoint: The URL of the Vault server</li> Authentication method: How ESO should authenticate to Vault (e.g., using a Kubernetes Service Account token, Vault token, or another method)</li> Paths and parameters: Details like which Vault mount paths and keys to read from</li> </ul> When to use SecretStore</code> vs. ClusterSecretStore</code>? SecretStore: Used when you want the configuration to be namespace-specific</li> ClusterSecretStore: Used when you want a cluster-wide configuration accessible by multiple namespaces</li> </ul> Example: apiVersion: external-secrets.io/v1beta1 kind: ClusterSecretStore metadata: name: vault-backend spec: provider: vault: server: "https://vault.example.com" path: "secret" # The path where secrets are stored in Vault version: "v2" auth: tokenSecretRef: name: vault-token key: token</code></pre> This ClusterSecretStore</code> tells ESO how to communicate with a Vault server at vault.example.com</code>, using a token stored in the vault-token</code> Secret. 2. ExternalSecret</h3> The ExternalSecret</code> resource references the SecretStore</code> or ClusterSecretStore</code> and tells ESO which keys from Vault to sync and how to project them into a Kubernetes Secret. Example: apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: my-app-secret namespace: default spec: refreshInterval: 1h secretStoreRef: kind: ClusterSecretStore name: vault-backend target: name: my-app-k8s-secret creationPolicy: Owner data: - secretKey: api-key # Key in the resulting K8s Secret remoteRef: key: my-app/api-key # Path/key in Vault</code></pre> In this ExternalSecret</code>: secretStoreRef</code> points to the ClusterSecretStore</code> named vault-backend</code></li> It requests a secret named my-app/api-key</code> from Vault</li> It maps that key into a Kubernetes Secret named my-app-k8s-secret</code> under the api-key</code> key</li> </ul> How it Works</h2> Define the Store: Set up a ClusterSecretStore</code> or SecretStore</code> resource that points to your Vault instance and specifies authentication details. </li> Request the Secret: Create an ExternalSecret</code> resource that references the store and specifies the exact secrets you need. </li> ESO Synchronizes the Secret: The External Secrets Operator continuously reconciles resources. When it sees your ExternalSecret</code>, it connects to Vault (using the instructions from the store resource), fetches the requested secret, and creates or updates the corresponding Kubernetes Secret object in your cluster. </li> </ol> Analogies</h2> Think of the ClusterSecretStore</code> as a “GPS coordinate” that tells ESO where to find the treasure (your secrets) in Vault.</li> The ExternalSecret</code> is your “shopping list,” telling ESO which items (specific keys/values) to bring back from that secret store and how to pack them into a Kubernetes Secret.</li> </ul> Pros and Cons</h2> Pros:</h3> Separation of concerns: Store configuration is separate from secret retrieval specifications</li> Reusability: You can reuse the same ClusterSecretStore</code> for multiple ExternalSecrets</code></li> Security: Keeps sensitive data in Vault, only syncing what’s necessary to Kubernetes</li> </ul> Cons:</h3> Initial complexity: Requires understanding CRDs and writing additional YAML manifests</li> Extra components: Relies on running ESO and Vault integrations</li> </ul> Summary</h2> By defining a SecretStore</code> or ClusterSecretStore</code> and one or more ExternalSecrets</code>, you can seamlessly integrate HashiCorp Vault secrets into your Kubernetes cluster with the External Secrets Operator. This pattern centralizes and secures your secret management workflow, providing a more maintainable and secure approach than manually managing Secret objects. Key Takeaways:</h3> Use SecretStore</code> or ClusterSecretStore</code> to define how to connect to Vault</li> Use ExternalSecret</code> to specify which secrets to pull from Vault</li> ESO automatically keeps your Kubernetes secrets in sync with Vault’s secrets</li> </ul> Further Reading</h2> External Secrets Operator Documentation</a></li> HashiCorp Vault Documentation</a></li> </ul>



Building a Custom Webhook Provider for External Secrets Operator: A Step-by-Step Guide
2024-12-12T00:00:00+00:00
In this tutorial, we’ll walk through creating a custom webhook provider for External Secrets Operator (ESO) from scratch. We’ll build a simple Go server that serves as a webhook, deploy it to Kubernetes, and configure ESO to use it for secret management.</p>
Understanding the Architecture</h2>
The External Secrets Operator (ESO) can fetch secrets from various providers, including custom webhooks. In this setup:</p>

ESO sends requests to our webhook with a key</li>
The webhook returns a structured JSON response</li>
ESO processes this response and creates Kubernetes secrets accordingly</li>
</ol>
Step 1: Creating the Webhook Server</h2>
First, let’s create a simple Go server that responds to webhook requests. Here’s a basic implementation:</p>
package</span> main</span></span>
</span>
import</span> (</span></span>
    "</span>encoding/json</span>"</span></span>
    "</span>fmt</span>"</span></span>
    "</span>log</span>"</span></span>
    "</span>net/http</span>"</span></span>
)</span></span>
</span>
// Response structure matching ESO's expectations</span></span>
type</span> Response</span> struct</span> {</span></span>
    Namespaces</span> map</span>[</span>string</span>]</span>NamespaceData</span> `</span>json:"namespaces"</span>`</span></span>
}</span></span>
</span>
type</span> NamespaceData</span> struct</span> {</span></span>
    Secrets</span> map</span>[</span>string</span>]</span>Secret</span> `</span>json:"secrets"</span>`</span></span>
}</span></span>
</span>
type</span> Secret</span> struct</span> {</span></span>
    Type</span>     string</span>         `</span>json:"type"</span>`</span></span>
    Value</span>    string</span>         `</span>json:"value"</span>`</span></span>
    Metadata</span> SecretMetadata</span> `</span>json:"metadata"</span>`</span></span>
    Expires</span>  string</span>         `</span>json:"expires"</span>`</span></span>
    NotBefore</span> string</span>        `</span>json:"notBefore"</span>`</span></span>
}</span></span>
</span>
type</span> SecretMetadata</span> struct</span> {</span></span>
    Labels</span>            map</span>[</span>string</span>]</span>string</span> `</span>json:"labels"</span>`</span></span>
    Annotations</span>       map</span>[</span>string</span>]</span>string</span> `</span>json:"annotations"</span>`</span></span>
    CreationTimestamp</span> string</span>           `</span>json:"creationTimestamp"</span>`</span></span>
    LastUpdated</span>       string</span>           `</span>json:"lastUpdated"</span>`</span></span>
}</span></span>
</span>
func</span> main</span>() {</span></span>
    http</span>.</span>HandleFunc</span>("</span>/webhook</span>",</span> webhookHandler</span>)</span></span>
    fmt</span>.</span>Println</span>("</span>Server is running on :8080</span>")</span></span>
    log</span>.</span>Fatal</span>(</span>http</span>.</span>ListenAndServe</span>("</span>:8080</span>",</span> nil</span>))</span></span>
}</span></span>
</span>
func</span> webhookHandler</span>(</span>w</span> http</span>.</span>ResponseWriter</span>,</span> r</span> *</span>http</span>.</span>Request</span>) {</span></span>
    if</span> r</span>.</span>Method</span> !=</span> http</span>.</span>MethodGet</span> {</span></span>
        http</span>.</span>Error</span>(</span>w</span>, "</span>Method not allowed</span>",</span> http</span>.</span>StatusMethodNotAllowed</span>)</span></span>
        return</span></span>
    }</span></span>
</span>
    key</span> :=</span> r</span>.</span>URL</span>.</span>Query</span>().</span>Get</span>("</span>key</span>")</span></span>
    if</span> key</span> !=</span> "</span>coca-cola.cluster-001</span>" {</span></span>
        http</span>.</span>Error</span>(</span>w</span>, "</span>Invalid key</span>",</span> http</span>.</span>StatusBadRequest</span>)</span></span>
        return</span></span>
    }</span></span>
</span>
    response</span> :=</span> Response</span>{</span></span>
        Namespaces</span>:</span> map</span>[</span>string</span>]</span>NamespaceData</span>{</span></span>
            "</span>coke-system</span>": {</span></span>
                Secrets</span>:</span> map</span>[</span>string</span>]</span>Secret</span>{</span></span>
                    "</span>admin-credentials</span>": {</span></span>
                        Type</span>:  "</span>k8s</span>",</span></span>
                        Value</span>: "</span>super-secret-secret</span>",</span></span>
                        Metadata</span>:</span> SecretMetadata</span>{</span></span>
                            Labels</span>:</span> map</span>[</span>string</span>]</span>string</span>{</span></span>
                                "</span>managed-by</span>": "</span>coke-system</span>",</span></span>
                            },</span></span>
                            Annotations</span>:</span> map</span>[</span>string</span>]</span>string</span>{</span></span>
                                "</span>inject-sidecar</span>": "</span>true</span>",</span></span>
                            },</span></span>
                            CreationTimestamp</span>: "</span>2024-01-01</span>",</span></span>
                            LastUpdated</span>:      "</span>2024-01-01</span>",</span></span>
                        },</span></span>
                        Expires</span>:   "</span>2024-01-01</span>",</span></span>
                        NotBefore</span>: "</span>2024-01-01</span>",</span></span>
                    },</span></span>
                },</span></span>
            },</span></span>
        },</span></span>
    }</span></span>
</span>
    w</span>.</span>Header</span>().</span>Set</span>("</span>Content-Type</span>", "</span>application/json</span>")</span></span>
    json</span>.</span>NewEncoder</span>(</span>w</span>).</span>Encode</span>(</span>response</span>)</span></span>
}</span></span></code></pre>Step 2: Containerizing the Webhook</h2>
Create a Dockerfile to package the webhook:</p>
FROM</span> golang:1.20-alpine </span>AS</span> builder</span></span>
WORKDIR</span> /app</span></span>
COPY</span> main.go .</span></span>
RUN</span> go build -o webhook-server main.go</span></span>
</span>
FROM</span> alpine:latest</span></span>
WORKDIR</span> /root/</span></span>
COPY</span> --from=builder /app/webhook-server .</span></span>
EXPOSE</span> 8080</span></span>
CMD</span> [</span>"./webhook-server"</span>]</span></span></code></pre>Step 3: Deploying to Kubernetes</h2>
Deploy the webhook using a Kubernetes Deployment and Service:</p>
apiVersion</span>:</span> apps/v1</span></span>
kind</span>:</span> Deployment</span></span>
metadata</span>:</span></span>
  name</span>:</span> eso-webhook</span></span>
  labels</span>:</span></span>
    app</span>:</span> eso-webhook</span></span>
spec</span>:</span></span>
  replicas</span>:</span> 1</span></span>
  selector</span>:</span></span>
    matchLabels</span>:</span></span>
      app</span>:</span> eso-webhook</span></span>
  template</span>:</span></span>
    metadata</span>:</span></span>
      labels</span>:</span></span>
        app</span>:</span> eso-webhook</span></span>
    spec</span>:</span></span>
      containers</span>:</span></span>
      -</span> name</span>:</span> eso-webhook</span></span>
        image</span>:</span> localhost:5000/eso-webhook:v1</span></span>
        ports</span>:</span></span>
        -</span> containerPort</span>:</span> 8080</span></span>
---</span></span>
apiVersion</span>:</span> v1</span></span>
kind</span>:</span> Service</span></span>
metadata</span>:</span></span>
  name</span>:</span> eso-webhook</span></span>
spec</span>:</span></span>
  type</span>:</span> LoadBalancer</span></span>
  selector</span>:</span></span>
    app</span>:</span> eso-webhook</span></span>
  ports</span>:</span></span>
    -</span> protocol</span>:</span> TCP</span></span>
      port</span>:</span> 80</span></span>
      targetPort</span>:</span> 8080</span></span></code></pre>Step 4: Configuring External Secrets Operator</h2>
Now comes the crucial part: configuring ESO to use our webhook. This involves two components:</p>

A ClusterSecretStore that defines how to access our webhook</li>
An ExternalSecret that specifies what secrets to fetch</li>
</ol>
ClusterSecretStore Configuration</h3>
apiVersion</span>:</span> external-secrets.io/v1beta1</span></span>
kind</span>:</span> ClusterSecretStore</span></span>
metadata</span>:</span></span>
  name</span>:</span> webhook-backend</span></span>
spec</span>:</span></span>
  provider</span>:</span></span>
    webhook</span>:</span></span>
      url</span>: "</span>http://eso-webhook.default.svc.cluster.local/webhook?key={{ .remoteRef.key }}</span>"</span></span>
      method</span>:</span> GET</span></span>
      result</span>:</span></span>
        jsonPath</span>: "</span>$.namespaces</span>"</span></span></code></pre>ExternalSecret Configuration with Advanced Templating</h3>
apiVersion</span>:</span> external-secrets.io/v1beta1</span></span>
kind</span>:</span> ExternalSecret</span></span>
metadata</span>:</span></span>
  name</span>:</span> coke-admin-credentials</span></span>
  namespace</span>:</span> coke-system</span></span>
spec</span>:</span></span>
  refreshInterval</span>: "</span>15s</span>"</span></span>
  secretStoreRef</span>:</span></span>
    name</span>:</span> webhook-backend</span></span>
    kind</span>:</span> ClusterSecretStore</span></span>
  target</span>:</span></span>
    name</span>:</span> admin-credentials</span></span>
    creationPolicy</span>:</span> Owner</span></span>
    template</span>:</span></span>
      engineVersion</span>:</span> v2</span></span>
      metadata</span>:</span></span>
        labels</span>:</span></span>
          managed-by</span>: '</span>{{ index .coke-system.secrets.admin-credentials.metadata.labels "managed-by" }}</span>'</span></span>
        annotations</span>:</span></span>
          inject-sidecar</span>: '</span>{{ index .coke-system.secrets.admin-credentials.metadata.annotations "inject-sidecar" }}</span>'</span></span>
      type</span>: '</span>{{ .coke-system.secrets.admin-credentials.type }}</span>'</span></span>
      data</span>:</span></span>
        username</span>: '</span>{{ .coke-system.secrets.admin-credentials.value | b64dec }}</span>'</span></span>
        password</span>: '</span>{{ .coke-system.secrets.admin-credentials.value | b64dec | upper }}</span>'</span></span>
        created-at</span>: '</span>{{ now | date "2006-01-02T15:04:05Z07:00" }}</span>'</span></span>
  data</span>:</span></span>
  -</span> secretKey</span>:</span> coke-system</span></span>
    remoteRef</span>:</span></span>
      key</span>:</span> coca-cola.cluster-001</span></span></code></pre>Understanding the Template Engine</h2>
The ExternalSecret configuration uses ESO’s v2 template engine, which provides powerful features for transforming secret data:</p>

Engine Version</strong>: We specify engineVersion: v2</code> to use the latest templating capabilities.</li>
Accessing Hyphenated Fields</strong>: Fields with hyphens require the index</code> function, e.g., {{ index .metadata.labels "managed-by" }}</code>.</li>
Data Transformation</strong>: We can use functions like b64dec</code> for base64 decoding and upper</code> for uppercase conversion.</li>
Time Functions</strong>: The now</code> function with date formatting helps track secret creation times.</li>
</ol>
Common Pitfalls and Solutions</h2>

Hyphenated Field Access</strong>: Always use the index</code> function for fields with hyphens.</li>
JSON Path</strong>: Ensure your jsonPath</code> in the ClusterSecretStore matches your webhook’s response structure.</li>
Service Discovery</strong>: Use the full Kubernetes service DNS name for reliable webhook access.</li>
Template Engine Version</strong>: Always specify engineVersion: v2</code> for advanced templating features.</li>
</ol>
Conclusion</h2>
Creating a custom webhook provider for External Secrets Operator involves careful consideration of:</p>

Webhook response structure</li>
Kubernetes deployment configuration</li>
ESO template engine features</li>
Service networking and discovery</li>
</ul>
By following this guide, you can create a flexible secret management solution that integrates seamlessly with your Kubernetes infrastructure while maintaining security and scalability.</p>


Debugging URL Encoding Issues with External Secrets Operator's Webhook Provider
2024-12-12T00:00:00+00:00
When working with Kubernetes External Secrets Operator (ESO) and its webhook provider, you might encounter some interesting URL encoding challenges. In this post, I’ll walk through a specific issue we encountered and how we solved it, which might help others facing similar problems.</p>
The Setup</h2>
We started with a basic ClusterSecretStore</code> configuration using ESO’s webhook provider:</p>
apiVersion</span>:</span> external-secrets.io/v1beta1</span></span>
kind</span>:</span> ClusterSecretStore</span></span>
metadata</span>:</span></span>
  name</span>:</span> webhook-backend</span></span>
spec</span>:</span></span>
  provider</span>:</span></span>
    webhook</span>:</span></span>
      url</span>: "</span>http://eso-webhook.default.svc.cluster.local:80/webhook?{{ .remoteRef.key }}</span>"</span></span>
      method</span>:</span> GET</span></span>
      result</span>:</span></span>
        jsonPath</span>: "</span>$</span>"</span></span></code></pre>
Our webhook server was a simple Go application designed to handle requests and return secret values based on a path parameter. We expected clean, separate query parameters, but reality had different plans.</p>
The Problem</h2>
When we started debugging why our webhook wasn’t working as expected, we added some basic logging to our Go server:</p>
key</span> :=</span> r</span>.</span>URL</span>.</span>Query</span>().</span>Get</span>("</span>key</span>")</span></span>
fmt</span>.</span>Println</span>("</span>key</span>",</span> key</span>)</span></span></code></pre>
To our surprise, the key was empty! Further investigation by printing the raw URL revealed something interesting:</p>
Method: GET</span></span>
URL: /webhook?key%3Dcoca-cola.cluster-001%26path%3Dnamespaces.cokeSystem.secrets.adminCredentials.value</span></span>
Protocol: HTTP/1.1</span></span></code></pre>
The entire query string was URL-encoded as a single parameter. Instead of getting our parameters separately, everything was encoded within the key</code> parameter.</p>
The Solution</h2>
Understanding What’s Happening</h3>
When ESO processes the template in our ClusterSecretStore URL, it’s taking the entire remoteRef.key</code> value and inserting it as-is into the URL. If our remoteRef.key</code> contains URL-special characters (like =</code> and &</code>), they get encoded, resulting in our doubly-nested query string.</p>
Fixing the Server Code</h3>
Here’s how we modified our webhook server to handle this situation properly:</p>
func</span> webhookHandler</span>(</span>w</span> http</span>.</span>ResponseWriter</span>,</span> r</span> *</span>http</span>.</span>Request</span>) {</span></span>
    // Get the 'key' query parameter</span></span>
    encodedKey</span> :=</span> r</span>.</span>URL</span>.</span>Query</span>().</span>Get</span>("</span>key</span>")</span></span>
    </span></span>
    // Unescape the key parameter</span></span>
    decodedKey</span>,</span> err</span> :=</span> url</span>.</span>QueryUnescape</span>(</span>encodedKey</span>)</span></span>
    if</span> err</span> !=</span> nil</span> {</span></span>
        http</span>.</span>Error</span>(</span>w</span>, "</span>Failed to decode key parameter</span>",</span> http</span>.</span>StatusBadRequest</span>)</span></span>
        return</span></span>
    }</span></span>
    </span></span>
    // Parse the decoded key as a query string</span></span>
    values</span>,</span> err</span> :=</span> url</span>.</span>ParseQuery</span>(</span>decodedKey</span>)</span></span>
    if</span> err</span> !=</span> nil</span> {</span></span>
        http</span>.</span>Error</span>(</span>w</span>, "</span>Invalid key format</span>",</span> http</span>.</span>StatusBadRequest</span>)</span></span>
        return</span></span>
    }</span></span>
</span>
    authKey</span> :=</span> values</span>.</span>Get</span>("</span>key</span>")</span></span>
    path</span> :=</span> values</span>.</span>Get</span>("</span>path</span>")</span></span>
</span>
    // Validate and use the parameters</span></span>
    if</span> authKey</span> !=</span> "</span>coca-cola.cluster-001</span>" {</span></span>
        http</span>.</span>Error</span>(</span>w</span>, "</span>Invalid authentication key</span>",</span> http</span>.</span>StatusUnauthorized</span>)</span></span>
        return</span></span>
    }</span></span>
</span>
    if</span> path</span> ==</span> "" {</span></span>
        http</span>.</span>Error</span>(</span>w</span>, "</span>Path is required</span>",</span> http</span>.</span>StatusBadRequest</span>)</span></span>
        return</span></span>
    }</span></span>
</span>
    // Process the request...</span></span>
}</span></span></code></pre>
The key improvements in this solution are:</p>

Using url.QueryUnescape()</code> to properly decode the URL-encoded parameter</li>
Using url.ParseQuery()</code> to parse the decoded string as a query string itself</li>
Extracting the actual key and path values from the parsed query parameters</li>
</ol>
Using the ClusterSecretStore</h3>
With this server implementation, we can now use our ClusterSecretStore like this:</p>
apiVersion</span>:</span> external-secrets.io/v1beta1</span></span>
kind</span>:</span> ExternalSecret</span></span>
metadata</span>:</span></span>
  name</span>:</span> example-external-secret</span></span>
spec</span>:</span></span>
  secretStoreRef</span>:</span></span>
    name</span>:</span> webhook-backend</span></span>
    kind</span>:</span> ClusterSecretStore</span></span>
  target</span>:</span></span>
    name</span>:</span> example-secret</span></span>
  data</span>:</span></span>
  -</span> secretKey</span>:</span> adminPassword</span></span>
    remoteRef</span>:</span></span>
      key</span>: "</span>key=coca-cola.cluster-001&path=namespaces.cokeSystem.secrets.adminCredentials.value</span>"</span></span></code></pre>Lessons Learned</h2>

Always implement detailed request logging when debugging webhook issues</li>
Don’t assume how parameters will be encoded or structured in webhook requests</li>
Use standard library functions for URL parsing and decoding instead of custom string manipulation</li>
Remember that template variables in Kubernetes resources might be processed differently than you expect</li>
</ol>
Conclusion</h2>
URL encoding issues can be tricky to debug, especially when dealing with nested query parameters. By understanding how ESO processes template variables and using proper URL parsing tools, we can build robust webhook handlers that correctly handle these scenarios.</p>
Remember to always implement proper logging during development and testing phases - it makes debugging these kinds of issues much easier. The Go standard library provides excellent tools for handling URL encoding and parsing, so make use of them instead of trying to parse query strings manually.</p>
This solution provides a clean way to handle the URL encoding challenges while maintaining the security and functionality requirements of our webhook provider.</p>


Using External Secrets Operator with HTTP Endpoints: A Complete Guide
2024-12-12T00:00:00+00:00
External Secrets Operator (ESO) is a powerful Kubernetes operator that helps manage secrets from external sources. While it’s commonly used with cloud provider secret managers like AWS Secrets Manager or HashiCorp Vault, ESO also supports fetching secrets from HTTP endpoints. In this guide, we’ll explore how to configure ESO to poll an HTTP endpoint and automatically create Kubernetes secrets from the response.</p>
Understanding the Webhook Provider</h2>
The Webhook provider in ESO allows you to fetch secrets from any HTTP endpoint that returns a JSON response. This is particularly useful when:</p>

You have an existing internal secrets service</li>
You need to integrate with a custom secrets management system</li>
You want to generate secrets dynamically through an API</li>
</ul>
Configuration Steps</h2>
1. Setting up the SecretStore</h3>
First, we need to configure a SecretStore that defines our HTTP endpoint and how to interact with it:</p>
apiVersion</span>:</span> external-secrets.io/v1beta1</span></span>
kind</span>:</span> SecretStore</span></span>
metadata</span>:</span></span>
  name</span>:</span> webhook-secret-store</span></span>
spec</span>:</span></span>
  provider</span>:</span></span>
    webhook</span>:</span></span>
      url</span>:</span> https://your-http-endpoint.com/secrets</span></span>
      result</span>:</span></span>
        jsonPath</span>: "</span>$.secrets[*]</span>"</span></span>
      headers</span>:</span></span>
        -</span> name</span>: "</span>Authorization</span>"</span></span>
          value</span>: "</span>Bearer <your-token-here></span>"</span></span></code></pre>
This configuration tells ESO:</p>

Which endpoint to call (url</code>)</li>
How to extract secrets from the response (jsonPath</code>)</li>
What headers to include in the request (headers</code>)</li>
</ul>
2. Creating the ExternalSecret</h3>
Next, we define an ExternalSecret that specifies which secrets to fetch and how often to refresh them:</p>
apiVersion</span>:</span> external-secrets.io/v1beta1</span></span>
kind</span>:</span> ExternalSecret</span></span>
metadata</span>:</span></span>
  name</span>:</span> webhook-external-secret</span></span>
spec</span>:</span></span>
  refreshInterval</span>: "</span>1h</span>"</span></span>
  secretStoreRef</span>:</span></span>
    name</span>:</span> webhook-secret-store</span></span>
    kind</span>:</span> SecretStore</span></span>
  target</span>:</span></span>
    name</span>:</span> my-secret</span></span>
    creationPolicy</span>:</span> Owner</span></span>
  data</span>:</span></span>
    -</span> secretKey</span>:</span> username</span></span>
      remoteRef</span>:</span></span>
        key</span>:</span> username</span></span>
    -</span> secretKey</span>:</span> password</span></span>
      remoteRef</span>:</span></span>
        key</span>:</span> password</span></span></code></pre>
This configuration:</p>

Sets a refresh interval of 1 hour</li>
References our SecretStore</li>
Defines which secrets to fetch and how to map them to Kubernetes secret keys</li>
</ul>
Best Practices and Security Considerations</h2>
When implementing this solution, consider the following best practices:</p>


HTTPS</strong>: Always use HTTPS endpoints to ensure secret data is encrypted in transit.</p>
</li>

Authentication</strong>: Implement proper authentication using headers or other mechanisms to secure your endpoint.</p>
</li>

Rate Limiting</strong>: Set appropriate refresh intervals to avoid overwhelming your HTTP endpoint.</p>
</li>

Error Handling</strong>: Ensure your HTTP endpoint returns appropriate error codes and that ESO can handle them gracefully.</p>
</li>

Access Control</strong>: Use Kubernetes RBAC to control which pods can access the created secrets.</p>
</li>
</ol>
Expected HTTP Response Format</h2>
Your HTTP endpoint should return a JSON response that matches your configured JSONPath. For example:</p>
{</span></span>
  "</span>secrets</span>": [</span></span>
    {</span></span>
      "</span>username</span>": "</span>admin</span>",</span></span>
      "</span>password</span>": "</span>secure-password-123</span>"</span></span>
    }</span></span>
  ]</span></span>
}</span></span></code></pre>Troubleshooting</h2>
If you encounter issues:</p>


Check the ESO operator logs using:</p>
kubectl</span> logs</span> -n</span> external-secrets</span> -l</span> app.kubernetes.io/name=external-secrets</span></span></code></pre></li>

Verify your HTTP endpoint is accessible from the cluster</p>
</li>

Confirm the JSONPath expression matches your response structure</p>
</li>

Ensure all required headers are properly configured</p>
</li>
</ol>
Conclusion</h2>
Using External Secrets Operator with HTTP endpoints provides a flexible way to integrate custom secret management solutions with Kubernetes. By following this guide and best practices, you can securely manage and automatically update your Kubernetes secrets from any HTTP source.</p>
Remember to always follow security best practices and thoroughly test your configuration in a non-production environment first.</p>
For more information, refer to the External Secrets Operator documentation</a>.</p>


HUnderstanding ClusterSecretStore with JWT Authentication in Kubernetes
2024-12-03T00:00:00+00:00
Understanding ClusterSecretStore with JWT Authentication in Kubernetes</h1>
When working with External Secrets Operator (ESO) in Kubernetes, configuring a ClusterSecretStore with JWT authentication requires careful consideration of several aspects. Let’s explore how to set this up correctly and understand its limitations.</p>
Understanding the Components</h2>
A typical setup involves:</p>

A webhook service that validates JWT tokens</li>
A Kubernetes secret storing the JWT token</li>
A ClusterSecretStore configuration that uses the token for authentication</li>
</ol>
Basic Configuration</h2>
Here’s a basic ClusterSecretStore configuration with JWT authentication:</p>
apiVersion</span>:</span> external-secrets.io/v1beta1</span></span>
kind</span>:</span> ClusterSecretStore</span></span>
metadata</span>:</span></span>
  name</span>:</span> vsecm-scout</span></span>
spec</span>:</span></span>
  provider</span>:</span></span>
    webhook</span>:</span></span>
      url</span>: "</span>https://vsecm-scout.default.svc.cluster.local:8443/webhook?key={{ .remoteRef.key }}</span>"</span></span>
      method</span>:</span> GET</span></span>
      result</span>:</span></span>
        jsonPath</span>: "</span>$</span>"</span></span>
      headers</span>:</span></span>
        Authorization</span>: "</span>Bearer {{ .jwt.token }}</span>"</span></span>
      secrets</span>:</span></span>
      -</span> name</span>:</span> jwt</span></span>
        secretRef</span>:</span></span>
          name</span>:</span> jwt-token</span></span>
          key</span>:</span> token</span></span>
      caBundle</span>: "</span>your-ca-bundle-here</span>"</span></span></code></pre>Token Management Challenges</h2>
Static Nature of Configuration</h3>
One critical limitation to understand is that ClusterSecretStore configurations are static. When you rotate the JWT token by updating the Kubernetes secret, the ClusterSecretStore doesn’t automatically pick up the changes.</p>
# This secret update won't automatically reflect in ClusterSecretStore</span></span>
apiVersion</span>:</span> v1</span></span>
kind</span>:</span> Secret</span></span>
metadata</span>:</span></span>
  name</span>:</span> jwt-token</span></span>
type</span>:</span> Opaque</span></span>
stringData</span>:</span></span>
  token</span>: "</span>your-new-jwt-token</span>"</span></span></code></pre>Handling Token Rotation</h3>
Given this limitation, there are several approaches to handle token rotation:</p>

Manual Update Approach</strong>:</li>
</ol>
# Update the JWT secret</span></span>
kubectl</span> create secret generic jwt-token</span> --from-literal=token=new-jwt-token -n</span> default</span> --dry-run=client -o</span> yaml</span> |</span> kubectl</span> apply</span> -f</span> -</span></span>
</span>
# Force update of ClusterSecretStore</span></span>
kubectl</span> replace</span> -f</span> clusterSecretStore.yaml</span></span></code></pre>


Custom Controller Approach</strong>:
Create a controller that watches for secret changes and updates the ClusterSecretStore accordingly.</p>
</li>

Webhook Service Adaptation</strong>:
Modify your webhook service to handle token fetching differently:</p>
</li>
</ol>
func</span> webhookHandler</span>(</span>w</span> http</span>.</span>ResponseWriter</span>,</span> r</span> *</span>http</span>.</span>Request</span>) {</span></span>
    // Instead of validating a static token,</span></span>
    // fetch the current token from Kubernetes and validate against that</span></span>
    currentToken</span>,</span> err</span> :=</span> getCurrentTokenFromK8s</span>()</span></span>
    if</span> err</span> !=</span> nil</span> {</span></span>
        http</span>.</span>Error</span>(</span>w</span>, "</span>Error fetching token</span>",</span> http</span>.</span>StatusInternalServerError</span>)</span></span>
        return</span></span>
    }</span></span>
    </span></span>
    // Validate the incoming token against the current token</span></span>
    // Rest of your handler logic...</span></span>
}</span></span></code></pre>Security Considerations</h2>
1. Token Lifetime</h3>
Consider the implications of token expiration:</p>

Long-lived tokens are easier to manage but pose higher security risks</li>
Short-lived tokens are more secure but require more complex rotation mechanisms</li>
</ul>
2. Secret Access</h3>
Ensure proper RBAC for accessing the JWT secret:</p>
apiVersion</span>:</span> rbac.authorization.k8s.io/v1</span></span>
kind</span>:</span> Role</span></span>
metadata</span>:</span></span>
  name</span>:</span> jwt-secret-reader</span></span>
rules</span>:</span></span>
-</span> apiGroups</span>: [""]</span></span>
  resources</span>: ["</span>secrets</span>"]</span></span>
  resourceNames</span>: ["</span>jwt-token</span>"]</span></span>
  verbs</span>: ["</span>get</span>"]</span></span></code></pre>3. TLS Configuration</h3>
Always use TLS for the webhook endpoint:</p>
tlsConfig</span> := &</span>tls</span>.</span>Config</span>{</span></span>
    MinVersion</span>:</span> tls</span>.</span>VersionTLS12</span>,</span></span>
    CipherSuites</span>: []</span>uint16</span>{</span></span>
        tls</span>.</span>TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384</span>,</span></span>
        tls</span>.</span>TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384</span>,</span></span>
    },</span></span>
}</span></span></code></pre>Best Practices</h2>


Monitoring</strong>:</p>

Monitor token expiration</li>
Set up alerts for failed secret fetches</li>
Track webhook endpoint health</li>
</ul>
</li>

Error Handling</strong>:</p>

Implement proper error reporting</li>
Use appropriate HTTP status codes</li>
Log authentication failures (but not sensitive data)</li>
</ul>
</li>

Documentation</strong>:</p>

Document token rotation procedures</li>
Maintain runbooks for common issues</li>
Keep configuration templates updated</li>
</ul>
</li>
</ol>
Example: Complete Setup</h2>
Here’s a complete example bringing everything together:</p>
# JWT Secret</span></span>
apiVersion</span>:</span> v1</span></span>
kind</span>:</span> Secret</span></span>
metadata</span>:</span></span>
  name</span>:</span> jwt-token</span></span>
type</span>:</span> Opaque</span></span>
stringData</span>:</span></span>
  token</span>: "</span>your-jwt-token</span>"</span></span>
---</span></span>
# ClusterSecretStore</span></span>
apiVersion</span>:</span> external-secrets.io/v1beta1</span></span>
kind</span>:</span> ClusterSecretStore</span></span>
metadata</span>:</span></span>
  name</span>:</span> vsecm-scout</span></span>
spec</span>:</span></span>
  provider</span>:</span></span>
    webhook</span>:</span></span>
      url</span>: "</span>https://vsecm-scout.default.svc.cluster.local:8443/webhook?key={{ .remoteRef.key }}</span>"</span></span>
      method</span>:</span> GET</span></span>
      result</span>:</span></span>
        jsonPath</span>: "</span>$</span>"</span></span>
      headers</span>:</span></span>
        Authorization</span>: "</span>Bearer {{ .jwt.token }}</span>"</span></span>
      secrets</span>:</span></span>
      -</span> name</span>:</span> jwt</span></span>
        secretRef</span>:</span></span>
          name</span>:</span> jwt-token</span></span>
          key</span>:</span> token</span></span>
      caBundle</span>: "</span>your-ca-bundle-here</span>"</span></span>
---</span></span>
# ExternalSecret Example</span></span>
apiVersion</span>:</span> external-secrets.io/v1beta1</span></span>
kind</span>:</span> ExternalSecret</span></span>
metadata</span>:</span></span>
  name</span>:</span> example-external-secret</span></span>
spec</span>:</span></span>
  refreshInterval</span>: "</span>1h</span>"</span></span>
  secretStoreRef</span>:</span></span>
    kind</span>:</span> ClusterSecretStore</span></span>
    name</span>:</span> vsecm-scout</span></span>
  target</span>:</span></span>
    name</span>:</span> example-secret</span></span>
  data</span>:</span></span>
  -</span> secretKey</span>:</span> mykey</span></span>
    remoteRef</span>:</span></span>
      key</span>:</span> myremotekey</span></span></code></pre>Conclusion</h2>
While ClusterSecretStore with JWT authentication provides a robust way to secure external secrets access, it comes with important limitations regarding token rotation. Understanding these limitations and implementing appropriate strategies for token management is crucial for maintaining a secure and operational system.</p>
Remember that security is a continuous process, and regular reviews of your authentication mechanisms, including JWT token management, should be part of your security practices.</p>
When implementing this setup, always consider your specific use case and requirements, and be prepared to adapt these patterns to match your security needs and operational capabilities.</p>

Zero to Hero - external-secret-operator

Using External Secrets Operator with HashiCorp Vault to Create Kubernetes Secrets

Key Resources</h2>

1. SecretStore or ClusterSecretStore</h3> The SecretStore</code> or ClusterSecretStore</code> resource provides information about your Vault instance, including:</p>

Pros and Cons</h2>

Further Reading</h2> External Secrets Operator Documentation</a></li> HashiCorp Vault Documentation</a></li> </ul>

Building a Custom Webhook Provider for External Secrets Operator: A Step-by-Step Guide

Debugging URL Encoding Issues with External Secrets Operator's Webhook Provider

Using External Secrets Operator with HTTP Endpoints: A Complete Guide

Understanding the Webhook Provider</h2> The Webhook provider in ESO allows you to fetch secrets from any HTTP endpoint that returns a JSON response. This is particularly useful when:</p>

Configuration Steps</h2>

1. Setting up the SecretStore</h3> First, we need to configure a SecretStore that defines our HTTP endpoint and how to interact with it:</p>

HUnderstanding ClusterSecretStore with JWT Authentication in Kubernetes

Token Management Challenges</h2>

Security Considerations</h2>

Further Reading</h2>

External Secrets Operator Documentation</a></li>
HashiCorp Vault Documentation</a></li> </ul>