Refactor mage target configuration

[swiatekm]

What does this PR do?

Introduces a configuration struct holding all the configuration for mage targets. This configuration can be read once at the mage target level from environment variables and files, and then passed around. As a result, our local tooling code never calls os.Setenv to pass configuration, and only calls os.Getenv when loading the configuration.

The only place where this becomes problematic is with mage targets setting other mage targets as dependencies, but wanting to modify configuration for them. We deal with this by making all mage targets take a context parameter, and attaching the configuration to the context. Then, each mage target can try to read the configuration from context or environment, modify it, and pass it down to its dependencies.

It's worth noting that this change does not eliminate all global config from dev-tools, nor does it make all the config handling nice and clear. I'd like to address some of the remaining problems in follow-ups, here I erred on the side of making the PR reviewable.

Why is it important?

Configuring the local tooling is a mess, with environment variables being read and set all over the place. As a result, it's hard to understand how a given target behaves, and hard to test them. This is the first major step towards making this code more maintainable.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

How to test this PR locally

Use the mage targets. :) The ones involving packaging are affected most.

Related issues

• Relates https://github.com/elastic/observability-dev/issues/5088

[mergify]

This pull request does not have a backport label. Could you fix it @swiatekm? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b chore/mage-config upstream/chore/mage-config
git merge upstream/main
git push upstream chore/mage-config

[github-actions]

🔍 Preview links for changed docs

• docs/reference/edot-collector/components.md
• docs/reference/edot-collector/components/elasticapmprocessor.md
• docs/reference/edot-collector/components/elasticsearchexporter.md
• docs/reference/edot-collector/config/configure-profiles-collection.md
• docs/reference/edot-collector/config/default-config-standalone.md
• docs/reference/edot-collector/config/tail-based-sampling.md
• docs/reference/edot-collector/custom-collector.md

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b chore/mage-config upstream/chore/mage-config
git merge upstream/main
git push upstream chore/mage-config

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b chore/mage-config upstream/chore/mage-config
git merge upstream/main
git push upstream chore/mage-config

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: ac8d911

History

• 💔 Build #34447 failed 4449a1a
• 💚 Build #34432 succeeded 1c662de
• 💛 Build #33796 was flaky 3cac0f5
• 💔 Build #33795 failed fba2c76
• 💛 Build #33589 was flaky efdadd0

cc @swiatekm

[ycombinator]

Nit: could we please call this settings so it doesn't sound like a function that's doing some setting work?

[michel-laterman]

Do we have a plan to replace the TODO or should we just use Background?