Add forge instance entity

[gabriel-samfira]

This change adds a new forge entity type along side Repository{}, Organization{} and Enterprise{} called ForgeInstance{}. This new entity type is available currently only on Gitea and possibly on Forgejo if we ever get around to adding it.

The new ForgeInstance{} entity requires gitea credentials that have rw access to the admin api. This feels a bit too much to allow GARM to have, but if instance level pools are required, it cannot be helped.

The API is similar to the other entity types. WebUI has been updated to accomodate the new ForgeInstance type, as has the CLI.

[Infinoid]

I've set up a test instance to try this out, but I'm unsure whether I set it up properly. Here are the steps I followed:

• build docker image for gabriel-samfira/add-forge-instance-entity (commit eee6176), and push it somewhere my k8s cluster can see it
• build test copy of cli tool
• deploy docker image into test environment (via a temporary terraform workspace)
• configure garm-cli to talk to test manager (add test [[manager]] to config.toml and set it as default)
• set up garm-test service account in gitea
• set up personal access token for garm-test service account, with admin:readwrite scope
• update gitea config to add test instance's hostname to webhook.ALLOWED_HOST_LIST
• ./garm-cli gitea endpoint create --name=gitea-test …
• ./garm-cli gitea credentials add --name=token …
• ./garm-cli forge-instance add --forge-type=gitea --credentials=token --endpoint=gitea-test --install-webhook --random-webhook-secret
• ./garm-cli pool add --forge-instance=gitea-test --tags=garm-test …
• add a descriptive name to the webhook in gitea (it was empty)
• try to trigger a job that runs-on: garm-test

Does this look right? I'm seeing the test environment receive webhook notifications from gitea, but rejecting them. Logs:

time=2026-06-30T13:11:35.235Z level=ERROR msg="failed to find pool manager" error="error fetching repo: error fetching repo: not found" hook_target_type=repository
time=2026-06-30T13:11:35.235Z level=ERROR msg="got not found error from DispatchWorkflowJob. webhook not meant for us?" error="error fetching poolManager: error fetching repo: error fetching repo: not found"
time=2026-06-30T13:11:35.235Z level=INFO msg=access_log method=POST uri=/webhooks/404760f1-a6d2-47d3-9248-aea6b0da4756 user_agent=Go-http-client/1.1 ip=[2600:1f18:7bbe:ba49:87aa::9]:38818 code=200 bytes=0 request_time=2.114832ms
time=2026-06-30T13:11:36.763Z level=ERROR msg="failed to find pool manager" error="error fetching repo: error fetching repo: not found" hook_target_type=repository
time=2026-06-30T13:11:36.763Z level=ERROR msg="got not found error from DispatchWorkflowJob. webhook not meant for us?" error="error fetching poolManager: error fetching repo: error fetching repo: not found"
time=2026-06-30T13:11:36.763Z level=INFO msg=access_log method=POST uri=/webhooks/404760f1-a6d2-47d3-9248-aea6b0da4756 user_agent=Go-http-client/1.1 ip=[2600:1f18:7bbe:ba49:87aa::9]:38818 code=200 bytes=0 request_time=711.585µs

[Infinoid]

I added a debug message here, to print the values of forgeType and giteaTargetType. I got:

• hookType=repository
• giteaTargetType=repository

This is despite the fact that the gitea webhook was created at the instance level. (The edit link in the gitea UI is https://gitea.my-site/-/admin/hooks/20649)

So that doesn't seem like a good way to detect SystemHooks.

[gabriel-samfira]

./garm-cli forge-instance add --forge-type=gitea --credentials=token --endpoint=gitea-test --install-webhook --random-webhook-secret

After you ran this, did the webhook show up under Site administration --> integration --> webhooks ?

[Infinoid]

Wait, no, I'm wrong! It didn't create a system webhook, it created a default webhook.

A system webhook triggers for any repo on the system, and operates at the system level.
A default webhook is a template that gets copied into new repos (including the fork I was using for testing), and operates at the repo level.

I think that's the problem.

[Infinoid]

It seems poorly documented, but I think the solution is adding is_system_webhook=true to the Config map when creating the webhook. I'm building a test image which does that now.

[Infinoid]

With the is_system_webhook=true config tweak, I deleted the default webhook, the old pool and the old forge instance, and then recreated them again, using the same commands as above. It created the right type of webhook:

With that in place, I triggered another job, garm spawned an ec2 worker and and it ran to completion.

[gabriel-samfira]

Yup. There were several issues that needed fixing. Many places throughout the code where changes are needed to add this.

One more thing, GARM tries to determine the endpoint from which a webhook originates. To do this, it looks at the job payload and compares the HTML URL in the payload with the base URL in the endpoint you defined in GARM. Make sure you use the same base URL as you receive via job. You should be able to see a payload in gitea if you click on the webhook that gets added after a hook is sent. If your endpoint base URL does not match what you have in the HTML URL, define a new endpoint in GARM.

[gabriel-samfira]

Grab the latest commit from this branch. There was also an issue with recording jobs and another with consuming them by the forge instance pool manager. Should work now. I need to also add tests.

[Infinoid]

Ok, trying 9c9f0da.

[Infinoid]

I think the solution is adding is_system_webhook=true to the Config map

Damn, I felt so clever with this, but it looks like you got there 6 minutes earlier. 😛

[gabriel-samfira]

I think the solution is adding is_system_webhook=true to the Config map

Damn, I felt so clever with this, but it looks like you got there 6 minutes earlier. 😛

Still clever! Not a race! And you were right. That was the fix. I didn't even notice that there were 2 types of webhooks until you mentioned system webhooks 😀

But after adding that, runners were still not being spun up correctly. The pool manager ignored the job. The watcher wasn't recording the job, so no runnet was being spun up in a zero idle runner pool. There were some places where I forgot to populate the ForgeInstanceID. Hopefully I got them all. Most of this PR is the generated client code for the web UI and golang. But it's still a pretty big mechanical change. So it's easy to miss stuff.

[Infinoid]

But after adding that, runners were still not being spun up correctly. The pool manager ignored the job. The watcher wasn't recording the job, so no runner was being spun up in a zero idle runner pool.

I had noticed that in the previous round of testing, actually. It started working when I set --min-idle-workers=1.

With 9c9f0da, it works even when --min-idle-workers=0. Looking good!

(Though it waits 30 seconds before it starts to spawn a runner; is that configurable?)

[gabriel-samfira]

(Though it waits 30 seconds before it starts to spawn a runner; is that configurable?)

yes. You can edit that in the WebUI --> Dashboard --> Controller Information --> Settings --> Minimum Job Age Backoff (seconds) (set to 0)

or you can just:

garm-cli controller update --minimum-job-age-backoff 0

[gabriel-samfira]

let me know how it works for you. If you see any weirdness. If not, I can just merge this. And increment later.

[Infinoid]

Well, so far so good. But I'll put it into heavier rotation and let you know what we see.

[gabriel-samfira]

merging this as-is. If you see anything weird, we can fix in a separate PR.

[Infinoid]

Sounds good, thanks!