KubeNodes target does not fully handle node IP changes

[mjnagel]

#1119 introduces a new KubeNodes target for network policies. There is an edge case in the current logic where a node could change IPs and the new IP will get added but the old IP will not be removed.

This is low impact as the network policies will allow the proper connectivity still (since the new node IP was added) but it could be confusing or accidentally allow more egress traffic than desired. Note that when Pepr cycles (specifically a restart of the watcher pod) the IP list would be completely refreshed so there is a path to handle this but it is not automated.

Possible paths forward:

• Use Pepr's OnSchedule to refresh the list of nodes every X interval: This wouldn't be immediate but would clean up network policies as often as whatever cadence we are comfortable with
• Rather than tracking a list of IPs, track a list of node names/ids along with the corresponding IP: This would be one more param to track but would give us a way to identify if a node with the same name has switched IPs and replace the IP rather than adding a new one (this should be relatively safe since renaming a node is uncommon/hard to do successfully)
• On each node update, refresh the entire node list (do a k8s.get call): This was one of the initial implementations in the PR, but should be careful to not cause strain on the controlplane

Regardless of solution the desired outcome is that when a node changes IPs, the old IP should be removed from the list. Ideally this should also be covered by a unit test since reproducing this behavior with k3d/e2e tests may be difficult.