Secrets manipulation using the UDS Operator

[docandrew]

Describe the solution you'd like

Frequently when "wiring" two components together, component A will use credentials or a path, etc. stored in a secret, and another component B needs to reference this secret to know how to connect to A. In Kyverno, policies can be used to mutate resources on create, and one use of this capability is to copy secrets between namespaces. I propose adding a similar capability to UDS Operator to allow for secrets to be copied between namespaces.

Describe alternatives you've considered

1. Helm lookups can be used here - when a chart is applied a manifest with a "blank" secret can be inserted where the contents of the secret are obtained from the lookup function as described here

2. A custom Pepr capability can be used for this purpose:

/**
 *  Copy secrets between namespaces
 *  To test this capability you run `pepr dev`and then run the following command:
 *  `kubectl apply -f capabilities/hello-pepr.samples.yaml`
 */
export const CopySecrets = new Capability({
  name: "copy-secrets",
  description: "Copy secrets between namespaces w/ Pepr",
  namespaces: ["minio", "cti-updater"],
});

// Use the 'When' function to create a new action, use 'Store' to persist data
const { When } = CopySecrets;

/**
 * ---------------------------------------------------------------------------------------------------
 *                                   Mutate Action (Namespace)                                   *
 * ---------------------------------------------------------------------------------------------------
 *
 * This action removes the label `remove-me` when a Namespace is created.
 * Note we don't need to specify the namespace here, because we've already specified
 * it in the Capability definition above.
 */
When(a.Secret)
  .IsCreated()
  .InNamespace("minio")
  .WithName("minio-creds-secret")
  .Watch(async sec => {
    Log.info("minio-creds secret was created. Copying to cti-updater");

    try {
      await K8s(kind.Secret).Apply({
        metadata: {
          name: "minio-creds-secret",
          namespace: "cti-updater",
        },
        data: sec.data,
      });
    } catch (error) {
      Log.error(error, "Failed to copy minio-creds to cti-updater");
    }
  });

However, I think this pattern is common enough to warrant special functionality in the UDS Operator to support it.

Proposed Solution

I propose two new custom resources and support in the UDS Operator to enable copying of secrets between namespaces.

1. A new key secrets in the Package CR to support a convenient, high-level syntax for copying of secrets in support of UDS Packages that need to reference secrets in other namespaces for their proper operation.

Basic syntax could look something like this:

apiVersion: uds.dev/v1alpha1
kind: Package
metadata: ...
spec:
  sso: ...
  monitor: ...
  network: ...
  secrets:
    - name: minio-creds
      description: Copy MinIO credentials for my app to use for bucket creation
      ...
      from:
        namespace: minio
        name: minio-creds-secret
      to:
        namespace: app-that-uses-minio
        name: minio-creds-secret

With time, more sophisticated pattern matching, transformation and application logic can be added. For example, a field required that indicates whether UDS Package installation should succeed if the secret to be copied is not present.

2. The recognition of a new Secret CR in the uds.dev namespace, to support more general use of the UDS Operator to perform manipulation of secrets at runtime.

One possible schema:

apiVersion: uds.dev/v1alpha1
kind: Secret
metadata: ...
spec:
  copy:
    from:
      namespace: minio
      name: minio-creds-secret
    to:
      namespace: app-that-uses-minio 
      name: minio-creds-secret

With time, new types of UDS Secret beyond the copy constructor could be created to allow for things like run-time generation of TLS certificates, passwords, or other types of material.

For example:

apiVersion: uds.dev/v1alpha1
kind: Secret
metadata: ...
spec:
  tls:
    namespace: nginx
    name: nginx-creds
    cert_name: nginx.crt
    key_name: nginx.key
    algo: ed25519
    ...
---
apiVersion: uds.dev/v1alpha1
kind: Secret
metadata: ...
spec:
  password:
    namespace: apache
    name: admin_creds
    length: 24
    char_pattern: "[a-z][A-Z][0-9]"
    ...

This is merely an example of how a Secret object can be used more generally to support many other common types of run-time manipulation and generation.

An alternate design might be separate CRs such as uds.dev/CopySecret or uds.dev/GenerateSecret each with their own schema and behavior.

[mjnagel]

Was mulling over this the other day - one random thought on how we could approach this. If Pepr were watching for a "UDS SecretCopy" CR or something like that, we could build an in-memory list of secrets to copy around (source + destination). Then to ensure secrets are copied properly we would need to make sure we are watching all secrets, and filtering on that list in our callback, before taking action to copy the secret. This would enable any amount of flexibility with regex, etc, but our actual watch would end up being across all namespaces.

Would be curious to get @cmwylie19's thoughts on this approach. I don't love the idea of a "global" secret watch given how often it would be triggered (similar to the pain around our pod watch). The other approach I think we could take would be dynamically starting up watches based on new requests (potentially would need to use KFC for these?). Even if we went that route though we might end up in a similar situation of watching all secrets if using a regex for namespace + name.

I'll also toss out another alternative - Zarf onDeploy.after actions could be used to copy a secret somewhere else after creation. This is somewhat limited though since it would require you to know all destinations during deployment of the source secret so I don't think it's the best route, but could be a lightweight option for some scenarios.

[cmwylie19]

Was mulling over this the other day - one random thought on how we could approach this. If Pepr were watching for a "UDS SecretCopy" CR or something like that, we could build an in-memory list of secrets to copy around (source + destination). Then to ensure secrets are copied properly we would need to make sure we are watching all secrets, and filtering on that list in our callback, before taking action to copy the secret. This would enable any amount of flexibility with regex, etc, but our actual watch would end up being across all namespaces.

Would be curious to get @cmwylie19's thoughts on this approach. I don't love the idea of a "global" secret watch given how often it would be triggered (similar to the pain around our pod watch). The other approach I think we could take would be dynamically starting up watches based on new requests (potentially would need to use KFC for these?). Even if we went that route though we might end up in a similar situation of watching all secrets if using a regex for namespace + name.

I'll also toss out another alternative - Zarf onDeploy.after actions could be used to copy a secret somewhere else after creation. This is somewhat limited though since it would require you to know all destinations during deployment of the source secret so I don't think it's the best route, but could be a lightweight option for some scenarios.

Thoughts:

• UDSSecretCopy - I like it, it is robust, nice from a user prospective too. I hear you about the global watch too but I'm not sure you would necessarily need it to be global since secrets you could set up a watch per item in the secrets array and do the watch with a namespace /api/v1/namespaces/minio/secrets?watch. Between this and a global what Im not sure which makes more sense.

I don't think the watch issue is always going to be a problem, we have seen it work in Go on several occasions, it is a matter of finding the time to get in and change it. I really hope to get to it sooner rather than later. Can follow up on that. We have 2 paths forward that both should work.

On the side of regex, there are tradeoffs, there are a lot of regex vulnerabilities (regex DOS) but if you can control the input we should be fine.

[docandrew]

I'm starting to lean towards an Admission Controller based model of annotating the destination secret (which could be empty) with labels or other metadata to indicate how the UDS Operator should handle it.

apiVersion: v1
kind: Secret
metadata:
  name: dotfile-secret
  uds:
    copy-from:
      namespace: <source secret namespace>
      name: <source secret>
data:
  <could leave blank, or have more detailed fields here for exactly which keys to copy>