Feature Request: Add runner group information in the workflow_job webhook

[pballandras]

Select Topic Area

Question

Body

Context

In a nutshell, we have a system that receives webhooks and dynamically starts self hosted runners based on the labels. A problem we faced is that, in a scenario as follows:

• Job 1 requests labels A, B, C
• Job 2 requests labels A, B, C, D

The system will interpret the labels and spawn two ephemeral runner instances that have the specified labels, but the problem is, Job 1 can "steal" job 2's runner and leave it starving. I made it generic, but in real life terms most of the time job 2 is more demanding and requires another type, class or... group of runners.

And so we found there are groups now in the runners and it would be exactly what we need, but looking at the raw events we receive from GitHub (and also the doc, the doc says it clearly) the runner group information is null while it is queued:

runner_group_name: The name of the runner group that is running this job. This will be null as long as workflow_job[status] is queued.

runner_group_id: The ID of the runner group that is running this job. This will be null as long as workflow_job[status] is queued.

{
  "action": "queued",
  "workflow_job": {
    "cutting": "it short...",

    "labels": [
      "self-hosted",
      "linux",
      "x64",
      "k8s"
    ],
    "runner_group_id": null,
    "runner_group_name": null,

    "cutting_it": "short"
  }
}

The problem is, we need this information to spawn the instance that will dequeue the job 😅

Use case

We'd like to have k8s runners heavier than some. Like more RAM and such. We deal with this by specifying the RAM request in labels. This leads to the problem explained above where smaller jobs would take heavier runners before the heavy jobs.

Question

So my question is: Would it be possible to have information about runner groups even in the queued state?

My reasoning is that the information must be available somewhat easily (famous last words) since the labels are available at that point and the group is (well, in the file) really close to the labels. This way, we could dynamically add the runner group to the new instance we spawn and the heavy jobs would get their heavy runners.

Solutions explored outside groups

We could also edit all our workflows to change the labels so that the A B C D problem from earlier doesn't happen, but I feel like the groups are such a nice addition I'd like to explore this first.

Another solution we thought of was to change the workflows to label themselves with labels like A B C D and A B C E

[AM1729]

@pballandras We ran into exact same problem, seems this is still not implemented from Github's end, curious which strategy did you use to sort this out..?

[pballandras]

So we ended up fixing like this:

For two jobs of the type

1. A B C
2. A B C D - which any A B C job can steal runners with those labels

We do

1. A B C default
2. A B C D

And for special labels that could be parametrized, we make it one:
A B C C.foo C.bar
becomes
"A" "B" "C foo bar"

This other solution doesn't suffer from the labels overlap since the "C" label is now tied with it's parameter, making the requested labels unqualified for the jobs that require A B C only.

Hope it helps you!
Cheers

[gabriel-samfira]

Pinging this thread, seeing as how github has marked it as dormant.

While this works, it would be great not to have to rely on workarounds and have the group be a part of the queued jobs. Having the "requested" group be a part of the queued webhook just like labels, would be extremely useful.

[gabriel-samfira]

This would definitely be useful for scheduling. Targeting a particular runner group should trigger logic to spin up runners only in that pool. But without knowing which group the workflow requested, you might end up spinning runners without setting the runner group ID on the runner and it will never pick up the job.

So a workflow like:

runs-on:
  group: my-runner-group
  labels: ["large", "self-hosted"]

should set the runner_group_name to the requested group for jobs in queued state. The same way the queued job has the requested labels set in the labels field.

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬

[gabriel-samfira]

Could we reopen this and add the feature request tag to it?

[bpoole6]

It doesn't make sense that the webhook doesn't contain the runner information. If you query the the github api it populates the runner group id.
https://docs.github.com/en/rest/actions/workflow-jobs?apiVersion=2022-11-28#get-a-job-for-a-workflow-run.

Why not include that in the queued workflow_job webhook?