Create ClusterConfig CRD to configure cluster/cloud parameters

[cmwylie19]

Is your feature request related to a problem? Please describe.

This came out of CTO office hours. I was chatting about this strange EKS edge-case where we do not receive an event when the Kubernetes EndpointSlice and Service . Also, I know that @mjnagel is already working on a derivative of this.

ClusterConfig as a way to parameter things like CIDR Range, Domain, ect. This should allow us to hedge against strange edge-cases that are present in specific environments. This would be a more robust solution that is highly configurable.

Describe the solution you'd like

• Given I want to have a very specific cluster config for a given cloud environment
• When I create an Instance of a ClusterConfig
• Then it will program specific configurations for edge cases that exist in given clouds.

[mjnagel]

Updating this issue to note that we have a very lightweight version of this implemented today with a secret helm template + values interface. This is provided to Pepr as env values, which are then used to construct our in-memory config.

We don't currently watch this config for changes, so there is a gap if someone updates their config without cycling Pepr. I'm not opposed to the CRD route for this config - not sure I see an immediate gain from it but the more we add to this secret the more benefit there would be to having a structured/typed config.

[mjnagel]

One additional thing brought up in a thread was the potential desire/need to customize the default runAsUser that Pepr mutates on pods (currently 1000). Different environments may use different image sources that have a default user to run as, such as the nonroot user for distroless, etc. Having this configurable globally would help reduce some of the pain of adoption and this mutation in particular.