Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Internal feature: Stateless scheduler #995

Open
Omrigan opened this issue Jun 26, 2024 · 2 comments
Open

Internal feature: Stateless scheduler #995

Omrigan opened this issue Jun 26, 2024 · 2 comments
Assignees
@sharnoff

Comments

@Omrigan
Copy link
Contributor

Omrigan commented Jun 26, 2024

Problem description / Motivation

Currently, the scheduler acts as the source of truth for the cluster state. We'd like to change that because of the following reasons:

  1. Makes scheduler replication hard, as it is unclear how to reliably replicate that state.
  2. Every scheduler restart results in a state reset, thus a period of warm-up is needed to repopulate the state by the autoscaler-agent. This mechanism is responsible for additional complication of the components logic and protocol, namely:
    a. The existence of the Buffer values.
    b. The notion of lastPermit.
  3. Potentially, the upscaling communication diagram can be simplified to bypass the autoscaler-agent after the upscaling was approved by scheduler.

Feature idea(s) / DoD

  1. A scheduler can be replicated.
  2. A scheduler restart doesn't cause the sub-optimal scheduling decisions.
  3. We no longer have the concepts of Buffered resources and lastPermit.

Implementation ideas

The scheduler's state and autoscaler-agent ↔ scheduler protocol can be moved entirely to annotations on the associated VirtualMachine object.

Related work

The implementation of neondatabase/neon#8111 might allow us to significantly simplify autoscaler-agent (or even merge it's remaining functionality into other components).

Further information

There are more details in the internal RFC.
This was referenced Nov 18, 2024
Merged
Closed
Merged
@sharnoff sharnoff self-assigned this Nov 18, 2024
sharnoff added a commit that referenced this issue Nov 18, 2024
@sharnoff
api: Touch-up comments on AgentRequest (#1143)
ff1126e 
I think the comments were just never updated when the autoscaler-agent
was moved from a per-pod sidecar to a per-node daemonset.

Found this while working on some background work for #995.
sharnoff added a commit that referenced this issue Nov 19, 2024
@sharnoff
scheduler: Shorten tolerations for node failure (#1146)
9ac98e7 
Similar to what was done in #1055, we need to explicitly add tolerations
to the scheduler to get it to be recreated more quickly on node failure.

This is particularly necessary because we don't have #995. We could wait
for that, but it's a lot of work, and this is a small thing we can do in
the meantime.

Fixes neondatabase/cloud#17298, part of neondatabase/cloud#14114.
@sharnoff sharnoff mentioned this issue Nov 22, 2024
Merged
@sharnoff
Copy link
Member

sharnoff commented Dec 2, 2024

Status update: I've written a draft implementation, doesn't yet pass tests.

This week, I'll finish up that initial implementation, and write up an RFC/design doc.

@sharnoff
Copy link
Member

Status update: #1163 is in review, with a number of changes left for me to make. It's also been thoroughly tested on staging.

@sharnoff sharnoff mentioned this issue Jan 6, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sharnoff sharnoff

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Omrigan @sharnoff