From bef0d58c0aa238d8377c97e6ce91f6f802baa026 Mon Sep 17 00:00:00 2001 From: Sambhav Kothari Date: Sat, 14 Aug 2021 00:54:03 +0530 Subject: [PATCH 1/3] Additional exportable layers Signed-off-by: Sambhav Kothari --- text/0000-additional-exported-layers.md | 117 ++++++++++++++++++++++++ 1 file changed, 117 insertions(+) create mode 100644 text/0000-additional-exported-layers.md diff --git a/text/0000-additional-exported-layers.md b/text/0000-additional-exported-layers.md new file mode 100644 index 000000000..1849b38a4 --- /dev/null +++ b/text/0000-additional-exported-layers.md @@ -0,0 +1,117 @@ +# Meta +[meta]: #meta +- Name: Additional exportable layers +- Start Date: 2021-08-13 +- Author(s): [@samj1912](https://github.com/samj1912) +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +Allow users to provide additional paths that can be exported similar to the application directory and allow easy configuration of the application directory. These paths are defined via the [`VOLUME`](https://docs.docker.com/engine/reference/builder/#volume) directive in `Dockerfile` and stored in the [`Volumes` object](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) in the image config. + +# Definitions +[definitions]: #definitions + +- `Workspace`: The default application directory where the application source code is mounted and final build outputs are written. This directory is exported. +- `Volume`: A set of directories describing where the buildpack process is likely to write data specific and should be exported alongside the application workspace. +- `WorkingDir`: The default working directory for a container. + +# Motivation +[motivation]: #motivation + +The main motivation behind this RFC is to unlock exporting application images that require libraries or software to be installed in certain directories apart from the default application directory and these installations are rebase safe and self-contained. This unlocks use cases where buildpacks may install software in `/opt` directories (for AWS Lambda extensions or other common standalone software) for example or preserve some settings in a `/home/$USER/.config` for the output application. This provides a rebase safe extension point without major changes to the buildpacks API. + +# What it is +[what-it-is]: #what-it-is + +Users would be able to define stack images with `VOLUME` directives in Dockerfiles. + + +For example a build/run base image may look like - + +```Dockerfile +# Set a common base +FROM ubuntu:bionic as base + +# Set required CNB information +ENV CNB_USER_ID=1000 +ENV CNB_GROUP_ID=1000 +ENV CNB_STACK_ID="io.buildpacks.samples.stacks.bionic" +LABEL io.buildpacks.stack.id="io.buildpacks.samples.stacks.bionic" + +# Create the user +RUN groupadd cnb --gid ${CNB_GROUP_ID} && \ + useradd --uid ${CNB_USER_ID} --gid ${CNB_GROUP_ID} -m -s /bin/bash cnb + +# Start a new run stage +FROM base as run + +# Volume instructions to identify paths that the lifecycle +# may safely overlay on the run image. +# This would ensure that these paths are rebase safe as +# docker currently discards any changes made to directories +# declared as volumes. See https://docs.docker.com/engine/reference/builder/#notes-about-specifying-volumes +VOLUME ["/app", "/opt/extensions", "/home/cnb/.config"] + +# Set user and group (as declared in base image) +USER ${CNB_USER_ID}:${CNB_GROUP_ID} + +# Start a new build stage +FROM base as build + +# Volume instruction in the build image that the lifecycle +# will export out. This must match the volumes declared in the run image. +VOLUME ["/app", "/opt/extensions", "/home/cnb/.config"] + +# Optionally users can override the default application workspace +# by specifying it through the WORKDIR directive in the build +# and run images. +WORKDIR /app + +# Set user and group (as declared in base image) +USER ${CNB_USER_ID}:${CNB_GROUP_ID} +``` + +Buildpacks will be passed a list of volumes through the `CNB_EXPORT_VOLUMES` variable and this variable would be a JSON list. Buildpacks may read this environment variable during `detect` or `build`. + +# How it Works +[how-it-works]: #how-it-works + +This RFC would require changes to the lifecycle and platform API. + +The platform would be responsible for mounting appropriate volumes based on the `Volumes` key in the OCI image config. For a platform like `pack` which relies on a daemon this should be fairly straight-forward to achieve since `docker run` automatically mounts appropriate volumes. For other platforms like `kpack` this would involve inspecting the builder image beforehand and modifying the build pod spec to accommodate the specified volumes. + +The lifecycle changes would involve exporting the files present in the above locations which should be similar to the logic that currently exists for exporting application workspace. Buildpacks could also take advantage of `slices` to specify paths in these additional directories that should exist as separate layers. + +Changes to the Buildpack API would be minimal and would mostly involve the Buildpacks being passed the additional `CNB_EXPORT_VOLUMES` variable for detection and build logic. + +# Drawbacks +[drawbacks]: #drawbacks + +N/A. + +# Alternatives +[alternatives]: #alternatives + +This proposal is useful as it relies of existing OCI image/Docker conventions to add a much needed extension point to the API. + +# Prior Art +[prior-art]: #prior-art + +- [RFC 72](https://github.com/buildpacks/rfcs/blob/main/text/0072-image-workdir.md) +- [RFC PR #172](https://github.com/buildpacks/rfcs/pull/172) +- [RFC PR #173](https://github.com/buildpacks/rfcs/pull/173) + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +None. + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +As noted above. \ No newline at end of file From b56ed25d8d8d1e795cf8f95b3397db16cadecf9c Mon Sep 17 00:00:00 2001 From: Sambhav Kothari Date: Thu, 16 Sep 2021 14:31:51 +0100 Subject: [PATCH 2/3] Add alternative proposal Signed-off-by: Sambhav Kothari --- text/0000-additional-exported-layers.md | 70 ++++++------------------- 1 file changed, 17 insertions(+), 53 deletions(-) diff --git a/text/0000-additional-exported-layers.md b/text/0000-additional-exported-layers.md index 1849b38a4..15c0740ae 100644 --- a/text/0000-additional-exported-layers.md +++ b/text/0000-additional-exported-layers.md @@ -11,14 +11,12 @@ # Summary [summary]: #summary -Allow users to provide additional paths that can be exported similar to the application directory and allow easy configuration of the application directory. These paths are defined via the [`VOLUME`](https://docs.docker.com/engine/reference/builder/#volume) directive in `Dockerfile` and stored in the [`Volumes` object](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) in the image config. +Allow users to provide additional paths that can be exported similar to the application directory and allow easy configuration of the application directory. These paths are defined via the `buildpack.export_dirs` by each buildpack. # Definitions [definitions]: #definitions -- `Workspace`: The default application directory where the application source code is mounted and final build outputs are written. This directory is exported. -- `Volume`: A set of directories describing where the buildpack process is likely to write data specific and should be exported alongside the application workspace. -- `WorkingDir`: The default working directory for a container. +- `export_dirs`: A set of directories describing where the buildpack process is likely to write data specific and should be exported alongside the application workspace. # Motivation [motivation]: #motivation @@ -28,76 +26,42 @@ The main motivation behind this RFC is to unlock exporting application images th # What it is [what-it-is]: #what-it-is -Users would be able to define stack images with `VOLUME` directives in Dockerfiles. +Buildpack authors would be able to buildpacks with `buildpack.export_dirs` keys in their `buildpack.toml` files. +For example a a buildpack may look like - -For example a build/run base image may look like - +```toml +api = "" -```Dockerfile -# Set a common base -FROM ubuntu:bionic as base - -# Set required CNB information -ENV CNB_USER_ID=1000 -ENV CNB_GROUP_ID=1000 -ENV CNB_STACK_ID="io.buildpacks.samples.stacks.bionic" -LABEL io.buildpacks.stack.id="io.buildpacks.samples.stacks.bionic" - -# Create the user -RUN groupadd cnb --gid ${CNB_GROUP_ID} && \ - useradd --uid ${CNB_USER_ID} --gid ${CNB_GROUP_ID} -m -s /bin/bash cnb - -# Start a new run stage -FROM base as run - -# Volume instructions to identify paths that the lifecycle -# may safely overlay on the run image. -# This would ensure that these paths are rebase safe as -# docker currently discards any changes made to directories -# declared as volumes. See https://docs.docker.com/engine/reference/builder/#notes-about-specifying-volumes -VOLUME ["/app", "/opt/extensions", "/home/cnb/.config"] - -# Set user and group (as declared in base image) -USER ${CNB_USER_ID}:${CNB_GROUP_ID} - -# Start a new build stage -FROM base as build - -# Volume instruction in the build image that the lifecycle -# will export out. This must match the volumes declared in the run image. -VOLUME ["/app", "/opt/extensions", "/home/cnb/.config"] - -# Optionally users can override the default application workspace -# by specifying it through the WORKDIR directive in the build -# and run images. -WORKDIR /app - -# Set user and group (as declared in base image) -USER ${CNB_USER_ID}:${CNB_GROUP_ID} +[buildpack] +id = "" +name = "" +export_dirs = ["/opt", "/home/cnb/.config"] ``` -Buildpacks will be passed a list of volumes through the `CNB_EXPORT_VOLUMES` variable and this variable would be a JSON list. Buildpacks may read this environment variable during `detect` or `build`. +The above would only be valid for normal buildpacks and not meta-buildpacks. Based on the `volumes` field, a platform would be responsible for validating that the `build` and `run` image do not have any content on these paths. Additionally, the builder image would be tagged with a label `io.buildpacks.export_dirs` with the list of paths that need to be exported. # How it Works [how-it-works]: #how-it-works This RFC would require changes to the lifecycle and platform API. -The platform would be responsible for mounting appropriate volumes based on the `Volumes` key in the OCI image config. For a platform like `pack` which relies on a daemon this should be fairly straight-forward to achieve since `docker run` automatically mounts appropriate volumes. For other platforms like `kpack` this would involve inspecting the builder image beforehand and modifying the build pod spec to accommodate the specified volumes. +The platform would be responsible for mounting appropriate volumes based on the `io.buildpacks.export_dirs` label in the OCI image config. For a platform like `pack` which relies on a daemon this should be fairly straight-forward to achieve. For other platforms like `kpack` this would involve inspecting the builder image beforehand and modifying the build pod spec to accommodate the specified volumes. The lifecycle changes would involve exporting the files present in the above locations which should be similar to the logic that currently exists for exporting application workspace. Buildpacks could also take advantage of `slices` to specify paths in these additional directories that should exist as separate layers. -Changes to the Buildpack API would be minimal and would mostly involve the Buildpacks being passed the additional `CNB_EXPORT_VOLUMES` variable for detection and build logic. +The lifecycle `analysis` phase would also be responsible for validating that the above list of `export_dirs` is valid for the provided `run` image. # Drawbacks [drawbacks]: #drawbacks -N/A. +- Additional complexity +- Platforms like Tekton may not be able to implement something like this easily. # Alternatives [alternatives]: #alternatives -This proposal is useful as it relies of existing OCI image/Docker conventions to add a much needed extension point to the API. +TBD: Multiple App Directories proposal # Prior Art [prior-art]: #prior-art @@ -109,7 +73,7 @@ This proposal is useful as it relies of existing OCI image/Docker conventions to # Unresolved Questions [unresolved-questions]: #unresolved-questions -None. +TBD. # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From eda604b7e878b9f4a81c2d16aac56ff8f9daf421 Mon Sep 17 00:00:00 2001 From: Sambhav Kothari Date: Wed, 1 Dec 2021 20:30:22 +0000 Subject: [PATCH 3/3] Update the RFC to use multiple app dirs idea Signed-off-by: Sambhav Kothari --- text/0000-additional-exported-layers.md | 31 ++++++++++++++----------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/text/0000-additional-exported-layers.md b/text/0000-additional-exported-layers.md index 15c0740ae..5360ffc9e 100644 --- a/text/0000-additional-exported-layers.md +++ b/text/0000-additional-exported-layers.md @@ -11,12 +11,12 @@ # Summary [summary]: #summary -Allow users to provide additional paths that can be exported similar to the application directory and allow easy configuration of the application directory. These paths are defined via the `buildpack.export_dirs` by each buildpack. +Allow users to provide additional volumes that can be exported similar to the application directory and allow easy configuration of the application directory. These volumes are defined via the `buildpack.exportable-volumes` by each buildpack. # Definitions [definitions]: #definitions -- `export_dirs`: A set of directories describing where the buildpack process is likely to write data specific and should be exported alongside the application workspace. +- `exportable-volumes`: A set of volumes describing where the buildpack process is likely to write data specific and should be exported alongside the application workspace. # Motivation [motivation]: #motivation @@ -26,9 +26,9 @@ The main motivation behind this RFC is to unlock exporting application images th # What it is [what-it-is]: #what-it-is -Buildpack authors would be able to buildpacks with `buildpack.export_dirs` keys in their `buildpack.toml` files. +Buildpack authors would be able to buildpacks with `buildpack.exportable-volumes` keys in their `buildpack.toml` files. -For example a a buildpack may look like - +For example a a buildpack may look like - ```toml api = "" @@ -36,33 +36,38 @@ api = "" [buildpack] id = "" name = "" -export_dirs = ["/opt", "/home/cnb/.config"] +exportable-volumes = ["aws-extensions", "user-config-dir"] ``` -The above would only be valid for normal buildpacks and not meta-buildpacks. Based on the `volumes` field, a platform would be responsible for validating that the `build` and `run` image do not have any content on these paths. Additionally, the builder image would be tagged with a label `io.buildpacks.export_dirs` with the list of paths that need to be exported. +The above would only be valid for normal buildpacks and not meta-buildpacks. Based on the `exportable-volumes` field, a platform would be responsible for validating that the `build` and `run` image do not have any content on these paths. By default these volume names will map to `/workspaces/` but a user may specify a label `io.buildpacks.exportable-volumes` on the `build` image to map a volume name to a different location. This label would look like - + +```json +[ + {"name": "user-config-dir", "value": "/home/cnb/.config"}, + {"name": "aws-extensions", "value": "/opt"} +] +``` + +For volumes defined by buildpacks but not in the `io.buildpacks.exportable-volumes` we will use a default `/workspaces/` directory. # How it Works [how-it-works]: #how-it-works This RFC would require changes to the lifecycle and platform API. -The platform would be responsible for mounting appropriate volumes based on the `io.buildpacks.export_dirs` label in the OCI image config. For a platform like `pack` which relies on a daemon this should be fairly straight-forward to achieve. For other platforms like `kpack` this would involve inspecting the builder image beforehand and modifying the build pod spec to accommodate the specified volumes. +The lifecycle would be responsible for creating symlinks from `/workspaces/` in the build image to the actual mapped directory and make these writable for buildpacks. It will also be responsible for exporting the content on these directories to the actual paths on the run-image. -The lifecycle changes would involve exporting the files present in the above locations which should be similar to the logic that currently exists for exporting application workspace. Buildpacks could also take advantage of `slices` to specify paths in these additional directories that should exist as separate layers. - -The lifecycle `analysis` phase would also be responsible for validating that the above list of `export_dirs` is valid for the provided `run` image. +The current default workspace directory would be moved to `/workspaces/default` but the lifecycle will symlink that directory back to `/workspace` for backwards compatibility. # Drawbacks [drawbacks]: #drawbacks - Additional complexity -- Platforms like Tekton may not be able to implement something like this easily. # Alternatives [alternatives]: #alternatives -TBD: Multiple App Directories proposal - +N/A # Prior Art [prior-art]: #prior-art