External Secrets Operator (ESO) is a powerful Kubernetes operator that helps manage secrets from external sources. While it’s commonly used with cloud provider secret managers like AWS Secrets Manager or HashiCorp Vault, ESO also supports fetching secrets from HTTP endpoints. In this guide, we’ll explore how to configure ESO to poll an HTTP endpoint and automatically create Kubernetes secrets from the response.

Understanding the Webhook Provider

The Webhook provider in ESO allows you to fetch secrets from any HTTP endpoint that returns a JSON response. This is particularly useful when:

  • You have an existing internal secrets service
  • You need to integrate with a custom secrets management system
  • You want to generate secrets dynamically through an API

Configuration Steps

1. Setting up the SecretStore

First, we need to configure a SecretStore that defines our HTTP endpoint and how to interact with it:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: webhook-secret-store
spec:
  provider:
    webhook:
      url: https://your-http-endpoint.com/secrets
      result:
        jsonPath: "$.secrets[*]"
      headers:
        - name: "Authorization"
          value: "Bearer <your-token-here>"

This configuration tells ESO:

  • Which endpoint to call (url)
  • How to extract secrets from the response (jsonPath)
  • What headers to include in the request (headers)

2. Creating the ExternalSecret

Next, we define an ExternalSecret that specifies which secrets to fetch and how often to refresh them:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: webhook-external-secret
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: webhook-secret-store
    kind: SecretStore
  target:
    name: my-secret
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: username
    - secretKey: password
      remoteRef:
        key: password

This configuration:

  • Sets a refresh interval of 1 hour
  • References our SecretStore
  • Defines which secrets to fetch and how to map them to Kubernetes secret keys

Best Practices and Security Considerations

When implementing this solution, consider the following best practices:

  1. HTTPS: Always use HTTPS endpoints to ensure secret data is encrypted in transit.

  2. Authentication: Implement proper authentication using headers or other mechanisms to secure your endpoint.

  3. Rate Limiting: Set appropriate refresh intervals to avoid overwhelming your HTTP endpoint.

  4. Error Handling: Ensure your HTTP endpoint returns appropriate error codes and that ESO can handle them gracefully.

  5. Access Control: Use Kubernetes RBAC to control which pods can access the created secrets.

Expected HTTP Response Format

Your HTTP endpoint should return a JSON response that matches your configured JSONPath. For example:

{
  "secrets": [
    {
      "username": "admin",
      "password": "secure-password-123"
    }
  ]
}

Troubleshooting

If you encounter issues:

  1. Check the ESO operator logs using:

    kubectl logs -n external-secrets -l app.kubernetes.io/name=external-secrets

  2. Verify your HTTP endpoint is accessible from the cluster

  3. Confirm the JSONPath expression matches your response structure

  4. Ensure all required headers are properly configured

Conclusion

Using External Secrets Operator with HTTP endpoints provides a flexible way to integrate custom secret management solutions with Kubernetes. By following this guide and best practices, you can securely manage and automatically update your Kubernetes secrets from any HTTP source.

Remember to always follow security best practices and thoroughly test your configuration in a non-production environment first.

For more information, refer to the External Secrets Operator documentation.