Enable pipeline steps to inject environment into the parent pipeline

[maxgio92]

This commit enable pipelines to inject environment variables the same way they're able to inject package dependencies into the build or test configuration. This will allow to remove responsibility of satisfying environment dependencies from the consumer. When setting needs.environment all the other pipeline runs in the same build or config will inherit that additional environment.

Cases like go/covdata pipeline will benefit from this feature being able to inject the GOCOVERDIR environment variable that the go runtime will use to generate coverage data during the whole test runtime, and the go tool covdata to generate reports from.

Melange Pull Request Template

Fixes #1823

Functional Changes

• This change can build all of Wolfi without errors (describe results in notes)

Notes:

SCA Changes

• Examining several representative APKs show no regression / the desired effect (details in notes)

Notes:

Linter

• The new check is clean across Wolfi
• The new check is opt-in or a warning

Notes:

[maxgio92]

I'm not really convinced by this API as needs.environment can be confusing considering that environment exists as well.

Specifically, the former applies to the whole config hence built/test time, while the latter to just the specific pipeline.

[maxgio92]

An example of use case would be the following:

diff --git a/pkg/build/pipelines/go/covdata.yaml b/pkg/build/pipelines/go/covdata.yaml
index b90b539..737856e 100644
--- a/pkg/build/pipelines/go/covdata.yaml
+++ b/pkg/build/pipelines/go/covdata.yaml
@@ -5,6 +5,8 @@ needs:
   packages:
     - ${{inputs.package}}
     - busybox
+  environment:
+    GOCOVERDIR: /home/build

 inputs:
   package:

# crane.yaml
...
subpackages:
  - name: ${{package.name}}-cover
    pipeline:
      - uses: go/build
        with:
          packages: ./cmd/crane
          extra-args: -cover
          ldflags: -buildid= -X github.com/google/go-containerregistry/cmd/crane/cmd.Version=${{package.version}} -X github.com/google/go-containerregistry/pkg/v1/remote/transport.Version=${{package.version}}
          output: crane
    test:
      environment:
        contents:
          packages:
            - jq
        # The following can be removed thanks to go/covdata needs.environment['GOCOVERDIR']=/home/build
        #environment:
        #  GOCOVERDIR: /home/build
      pipeline:
        - name: Verify Crane installation
          runs: |
            crane version || exit 1
            crane --help
        - name: Fetch and verify manifest
          runs: |
            crane manifest chainguard/static | jq '.schemaVersion' | grep '2' || exit 1
        - name: List tags for a public image
          runs: |
            crane ls chainguard/static | grep -E 'latest|v[0-9]+.[0-9]+.[0-9]+' || exit 1
        - name: Validate image existence
          runs: |
            crane digest chainguard/static:latest && echo "Image exists" || exit 1
        - name: Pull and save an image locally
          runs: |
            crane pull chainguard/static:latest static_latest.tar || exit 1
            [ -f static_latest.tar ] || exit 1
        - uses: go/covdata

we briefly discussed this use case @joshrwolf

[imjasonh]

Can you add an example to examples/ to demonstrate this working? That will act both as a test, and as runnable documentation of its behavior.

[maxgio92]

Yes, sure @imjasonh! I'll do it.

[maxgio92]

Can you add an example to examples/ to demonstrate this working? That will act both as a test, and as runnable documentation of its behavior.

@imjasonh an example which can be used at test as well has been added to examples/needs-environment.yaml.
Thanks for highlighting it. I fixed a bug thanks to it which was affecting non built-in pipelines.

[hectorj2f]

lgtm

[imjasonh]

This seems weird, if I'm understanding correctly. A later step in the pipeline can set env vars for a previous step? What should happen if two steps overwrite each other, who wins?

I'd have expected this to need to come before the steps that use it, and then the rules of precedence are easier to understand (latest wins)

(I'm really glad this example exists, to make it clear how this is working!)

[maxgio92]

I see the point @imjasonh. I'm too not 100% convinced by the UX.

W.r.t. the priority, the pipeline-level environment wins over needs.environment, so for instance:

  pipeline:
    - name: Expect an environment transparently.
      environment:
        FOO: "baz"
      runs: |
        set -u
        echo "FOO: ${FOO}"
    - name: Inject an environment to the parent pipeline.
      needs:
        environment:
          FOO: "bar"

will end up with $FOO evaluated to baz.

The way it works is that it feeds the pipeline's environment.environment, so it's enabled for either for all steps or nothing, with a priority lower than the step specific environment.

From usability point of view is not 100% intuitive, at the same time it solves some use cases. I think it's a trade-off. Whether it's worth it or not, I'm fine both ways :-)

[kranurag7]

👋🏻 Hello, if we merge this then it'll enable me/us to write pipelines which will allow us to define environment in the pipeline itself. I wanted to this patch inside test/kwok/cluster pipeline.

[maxgio92]

I guess @kranurag7 is to export KUBERNETES_SERVICE_HOST + KUBERNETES_SERVICE_PORT environment variable to available to subsequent step's shell session. This feature should avoid exporting them for each step dependent on the kwok step