From 8b9de038eaef507c372c576442b7fa721654963e Mon Sep 17 00:00:00 2001 From: Nikolai Kondrashov Date: Mon, 2 Jul 2018 19:33:17 +0300 Subject: [PATCH 01/12] Add reporting design --- reporting_design.md | 130 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 reporting_design.md diff --git a/reporting_design.md b/reporting_design.md new file mode 100644 index 0000000..b2f1217 --- /dev/null +++ b/reporting_design.md @@ -0,0 +1,130 @@ +Skt data flow and reporting design +================================== + +Each skt stage command has: + +* Two inputs: + * Command-line options + * One of which may be specifying a configuration file containing + some command-line option values. The file cannot be used to + specify anything, which cannot be set with command-line options. + Command-line options following this option override values in + the file. This option can have a default, such as ~/.skt.yaml + * Input directory +* Two outputs: + * Output directory + * Exit status summarizing the produced contents of the output + directory. I.e. the output directory contains all the data encoded + in the exit status. + +Among skt options are: input directory and output directory +input directory by default is current directory +output directory by default is input directory + +None of the different skt stage commands have output file names named the +same, so they cannot step onto each other's toes. However, re-running the same +skt stage command replaces existing files completely. E.g. it will need to +remove all files that might have been created by the previous run of the same +command, before starting adding its own, so the output file set is not +inconsistent. + +Output of one skt stage command is a superset of an input to the next stage +command. Note: input and output don't have to be in the same directory. I.e. +output of one stage can be input to multiple runs of the next stage, e.g. with +different command-line options. + +A stage command shouldn't require repeating the option passed to any of the +previous stages (i.e. those should be encoded in the stage's output), but may +allow overriding them. + +In essence, an output of a stage is an independent entity, using it should not +require providing any other input to skt. + +Each command has inputs and outputs documented explicitly, so they could be +used both by skt and other programs, however some parts might be defined +opaque for use by skt only. Specific command descriptions follow. + +Merge +----- + +Produce a source tree with patches applied. + +### Inputs + +#### Command-line options + +* git repository to fetch from +* git reference from the repository to base patches on +* git fetch depth +* what to merge on top (in order of specification), + arbitrary combination and number: + * git reference from the same repository + * URL of a patch mbox (including file:// scheme) + +#### Input directory + +No files expected + +### Outputs + +#### Output directory + +* "source" directory containing what is required as the input by the + "Build" command. +* "merge.log" - diagnostics output of the merging attempt. +* "merge.done" - empty file, created only if merge stage succeeded + +#### Exit status +Zero on success, one if "merge.done" is missing, two and greater if other +failure occurred. + +Build +----- + +Produce a kernel tarball. + +### Inputs + +#### Command-line options + +* Extra make options +* Configuration type: "tinyconfig", "rh-configs" (works only if Makefile + supports "rh-configs" target), other configuration-generating target + supported by the Makefile in "source". +* Shell glob, relative to the "source" root, matching a configuration file + inside the "source" tree to use. Only valid if "rh-configs" + configuration type is used. +* Kernel configuration file to use as the base. Exclusive with + "tinyconfig" and "rh-configs" configuration types. + +TODO: Figure out a better arrangement, so values, not options are exclusive. +TODO: See if we can define allowed configuration types more strictly. + +#### Input directory + +* "source" directory containing the source of the "kernel", must contain + the following (TODO: make sure everything we need is listed): + * Makefile + * Must support the following targets (TODO: detail all targets): + * "mrproper" - cleanup the tree + * "tinyconfig" - generate minimum kernel configuration + * "kernelrelease" - output kernel version + * "targz-pkg" - build kernel tarball + * May support the following targets: + * "rh-configs" - generate a set of configuration files + somewhere in the build directory + * other targets generating kernel configuration +* "merge.done" - empty file + +### Outputs + +#### Output directory + +* "kernel.tar.gz" - the built kernel tarball +* "kernel.config" - the used kernel configuration +* "build.log" - diagnostics output of the building stage +* "build.done" - empty file, created only if build stage succeeded + +#### Exit status +Zero on success, one if "build.done" is missing, two and greater if other +failure occurred. From a7c00668203ad0d1662f69858540c70e317b9cdf Mon Sep 17 00:00:00 2001 From: Nikolai Kondrashov Date: Wed, 4 Jul 2018 14:40:26 +0300 Subject: [PATCH 02/12] reporting_design: Add brief run stage doc --- reporting_design.md | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/reporting_design.md b/reporting_design.md index b2f1217..ec7e45e 100644 --- a/reporting_design.md +++ b/reporting_design.md @@ -120,7 +120,7 @@ TODO: See if we can define allowed configuration types more strictly. #### Output directory -* "kernel.tar.gz" - the built kernel tarball +* "kernel.tar.gz" - the built kernel tarball, as required by "run" input. * "kernel.config" - the used kernel configuration * "build.log" - diagnostics output of the building stage * "build.done" - empty file, created only if build stage succeeded @@ -128,3 +128,33 @@ TODO: See if we can define allowed configuration types more strictly. #### Exit status Zero on success, one if "build.done" is missing, two and greater if other failure occurred. + +Run +--- + +Run tests on a kernel. + +### Inputs + +#### Command-line options + +* Runner type + * "beaker" - run tests in Beaker. Requires Beaker client ("bkr") + configured for access to a Beaker service. +* Runner parameters + * "beaker" + * Beaker job XML template + * Beaker user to run the job as. Optional. + +#### Input directory + +* "kernel.tar.gz" - the built kernel tarball + + TODO: Describe minimal requirements to the tarball, so that we can + substitute this with a minimal "kernel" for testing. + +#### Output directory + +* "run.done" - empty file, created only if run stage completed. + +TODO: Describe representation of test results From 6adf6e0738ac7a064376d99069b7cfc9694e1dd3 Mon Sep 17 00:00:00 2001 From: Nikolai Kondrashov Date: Mon, 9 Jul 2018 14:05:35 +0300 Subject: [PATCH 03/12] reporting_design: Describe stage report text --- reporting_design.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/reporting_design.md b/reporting_design.md index ec7e45e..9598e28 100644 --- a/reporting_design.md +++ b/reporting_design.md @@ -44,6 +44,22 @@ Each command has inputs and outputs documented explicitly, so they could be used both by skt and other programs, however some parts might be defined opaque for use by skt only. Specific command descriptions follow. +All stages +---------- + +### Report text +Each stage output includes a file containing a human-readable report text, +summarizing what happened during the stage. That text can then be included +into the overall report by sktm. The text can refer to other files output by +the stage, using special formatting, which can then be replaced by skt by +e.g. references to e-mail attachments or hyperlinks. + +The syntax of such references is `{}`, where `` is the name of the +output file to include into the report. To provide an option to include +literal curly braces into reports, any characters preceded by a backslash +would be copied by sktm into the summarized report literally, without +interpretation. + Merge ----- @@ -72,6 +88,7 @@ No files expected * "source" directory containing what is required as the input by the "Build" command. * "merge.log" - diagnostics output of the merging attempt. +* "merge.report" - report text, as described above * "merge.done" - empty file, created only if merge stage succeeded #### Exit status @@ -123,6 +140,7 @@ TODO: See if we can define allowed configuration types more strictly. * "kernel.tar.gz" - the built kernel tarball, as required by "run" input. * "kernel.config" - the used kernel configuration * "build.log" - diagnostics output of the building stage +* "build.report" - report text, as described above * "build.done" - empty file, created only if build stage succeeded #### Exit status @@ -155,6 +173,7 @@ Run tests on a kernel. #### Output directory +* "run.report" - report text, as described above * "run.done" - empty file, created only if run stage completed. TODO: Describe representation of test results From c829f652d57f32403fceebf7391463f628de7d23 Mon Sep 17 00:00:00 2001 From: Nikolai Kondrashov Date: Tue, 10 Jul 2018 15:56:44 +0300 Subject: [PATCH 04/12] reporting_design: Refine all stages section structure Structure the "All stages" section similarly to specific stages. --- reporting_design.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/reporting_design.md b/reporting_design.md index 9598e28..8be6760 100644 --- a/reporting_design.md +++ b/reporting_design.md @@ -47,7 +47,11 @@ opaque for use by skt only. Specific command descriptions follow. All stages ---------- -### Report text +### Outputs + +#### Output directory + +##### Report text Each stage output includes a file containing a human-readable report text, summarizing what happened during the stage. That text can then be included into the overall report by sktm. The text can refer to other files output by From e5df0fc98b4b1cb464efbf4e4af24f71487e2dd1 Mon Sep 17 00:00:00 2001 From: Nikolai Kondrashov Date: Tue, 10 Jul 2018 16:02:04 +0300 Subject: [PATCH 05/12] reporting_design: Describe result and exit status --- reporting_design.md | 30 ++++++++++++++++++------------ 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/reporting_design.md b/reporting_design.md index 8be6760..b27d315 100644 --- a/reporting_design.md +++ b/reporting_design.md @@ -64,6 +64,21 @@ literal curly braces into reports, any characters preceded by a backslash would be copied by sktm into the summarized report literally, without interpretation. +##### Result status +Each stage output includes a status file, which is set executable and contains +either the `true` or `false` strings, for the overall stage status of success +or failure respectively. If the file is missing, but any of the other output +files are present, the stage can be considered as incomplete, i.e. it must +have exprienced an error, and there was no conclusive result. + +#### Exit status +Execution of each stage should return an exit status of zero, if the stage +completed succesfully, one if it failed, and something else if an error +occurred. + +The same exit status (without differentiating errors) can be reproduced by +executing the result status file. + Merge ----- @@ -93,11 +108,7 @@ No files expected "Build" command. * "merge.log" - diagnostics output of the merging attempt. * "merge.report" - report text, as described above -* "merge.done" - empty file, created only if merge stage succeeded - -#### Exit status -Zero on success, one if "merge.done" is missing, two and greater if other -failure occurred. +* "merge.result" - stage result status, as described above Build ----- @@ -135,7 +146,6 @@ TODO: See if we can define allowed configuration types more strictly. * "rh-configs" - generate a set of configuration files somewhere in the build directory * other targets generating kernel configuration -* "merge.done" - empty file ### Outputs @@ -145,11 +155,7 @@ TODO: See if we can define allowed configuration types more strictly. * "kernel.config" - the used kernel configuration * "build.log" - diagnostics output of the building stage * "build.report" - report text, as described above -* "build.done" - empty file, created only if build stage succeeded - -#### Exit status -Zero on success, one if "build.done" is missing, two and greater if other -failure occurred. +* "build.result" - stage result status, as described above Run --- @@ -178,6 +184,6 @@ Run tests on a kernel. #### Output directory * "run.report" - report text, as described above -* "run.done" - empty file, created only if run stage completed. +* "run.result" - stage result status, as described above TODO: Describe representation of test results From 6c0f76287d2ae94b467f4544f35fb0dd5dedc88f Mon Sep 17 00:00:00 2001 From: Nikolai Kondrashov Date: Tue, 10 Jul 2018 16:30:23 +0300 Subject: [PATCH 06/12] reporting_design: Describe common output requirements --- reporting_design.md | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/reporting_design.md b/reporting_design.md index b27d315..d6e05da 100644 --- a/reporting_design.md +++ b/reporting_design.md @@ -51,6 +51,14 @@ All stages #### Output directory +Names of all files output by a stage must beging with the stage name followed +by a dot. This way it should be possible to detect if a stage ran and makes it +easier to delete all files belonging to a stage. + +The output of each stage doesn't have to be limited to particular files +described below, and can include more files. E.g. extra files referenced by +the report text. However, all of them must observe the naming scheme above. + ##### Report text Each stage output includes a file containing a human-readable report text, summarizing what happened during the stage. That text can then be included @@ -104,8 +112,8 @@ No files expected #### Output directory -* "source" directory containing what is required as the input by the - "Build" command. +* "merge.source" directory containing what is required as the input by the + "Build" stage. * "merge.log" - diagnostics output of the merging attempt. * "merge.report" - report text, as described above * "merge.result" - stage result status, as described above @@ -134,8 +142,8 @@ TODO: See if we can define allowed configuration types more strictly. #### Input directory -* "source" directory containing the source of the "kernel", must contain - the following (TODO: make sure everything we need is listed): +* "merge.source" - directory containing the source of the "kernel", must + contain the following (TODO: make sure everything we need is listed): * Makefile * Must support the following targets (TODO: detail all targets): * "mrproper" - cleanup the tree @@ -151,8 +159,8 @@ TODO: See if we can define allowed configuration types more strictly. #### Output directory -* "kernel.tar.gz" - the built kernel tarball, as required by "run" input. -* "kernel.config" - the used kernel configuration +* "build.kernel.tar.gz" - the built kernel tarball, as required by "run" input. +* "build.kernel.config" - the used kernel configuration * "build.log" - diagnostics output of the building stage * "build.report" - report text, as described above * "build.result" - stage result status, as described above @@ -176,7 +184,7 @@ Run tests on a kernel. #### Input directory -* "kernel.tar.gz" - the built kernel tarball +* "build.kernel.tar.gz" - the built kernel tarball TODO: Describe minimal requirements to the tarball, so that we can substitute this with a minimal "kernel" for testing. From 969c0ed7a4507c7048ff8be9df8dbfada58406d7 Mon Sep 17 00:00:00 2001 From: Nikolai Kondrashov Date: Tue, 10 Jul 2018 16:30:59 +0300 Subject: [PATCH 07/12] reporting_design: Stop requiring logs --- reporting_design.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/reporting_design.md b/reporting_design.md index d6e05da..57a576c 100644 --- a/reporting_design.md +++ b/reporting_design.md @@ -114,7 +114,6 @@ No files expected * "merge.source" directory containing what is required as the input by the "Build" stage. -* "merge.log" - diagnostics output of the merging attempt. * "merge.report" - report text, as described above * "merge.result" - stage result status, as described above @@ -161,7 +160,6 @@ TODO: See if we can define allowed configuration types more strictly. * "build.kernel.tar.gz" - the built kernel tarball, as required by "run" input. * "build.kernel.config" - the used kernel configuration -* "build.log" - diagnostics output of the building stage * "build.report" - report text, as described above * "build.result" - stage result status, as described above From c882add32eed1865675cb24feb81c35a94a51da2 Mon Sep 17 00:00:00 2001 From: Nikolai Kondrashov Date: Tue, 10 Jul 2018 16:34:10 +0300 Subject: [PATCH 08/12] reporting_design: Clear test result description TODO We do not need to describe test result representation now that we have reports. --- reporting_design.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/reporting_design.md b/reporting_design.md index 57a576c..72a021b 100644 --- a/reporting_design.md +++ b/reporting_design.md @@ -191,5 +191,3 @@ Run tests on a kernel. * "run.report" - report text, as described above * "run.result" - stage result status, as described above - -TODO: Describe representation of test results From b87373d82047d58a7c4b410d718591c166597cb2 Mon Sep 17 00:00:00 2001 From: Nikolai Kondrashov Date: Wed, 11 Jul 2018 15:40:03 +0300 Subject: [PATCH 09/12] reporting_design: Fix typos --- reporting_design.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/reporting_design.md b/reporting_design.md index 72a021b..572ed47 100644 --- a/reporting_design.md +++ b/reporting_design.md @@ -51,7 +51,7 @@ All stages #### Output directory -Names of all files output by a stage must beging with the stage name followed +Names of all files output by a stage must begin with the stage name followed by a dot. This way it should be possible to detect if a stage ran and makes it easier to delete all files belonging to a stage. @@ -77,11 +77,11 @@ Each stage output includes a status file, which is set executable and contains either the `true` or `false` strings, for the overall stage status of success or failure respectively. If the file is missing, but any of the other output files are present, the stage can be considered as incomplete, i.e. it must -have exprienced an error, and there was no conclusive result. +have experienced an error, and there was no conclusive result. #### Exit status Execution of each stage should return an exit status of zero, if the stage -completed succesfully, one if it failed, and something else if an error +completed successfully, one if it failed, and something else if an error occurred. The same exit status (without differentiating errors) can be reproduced by From 91a33818ec223d8adb5dbc48041dd948a6905f7c Mon Sep 17 00:00:00 2001 From: Nikolai Kondrashov Date: Wed, 11 Jul 2018 15:41:44 +0300 Subject: [PATCH 10/12] reporting_design: Remove build.kernel.config output --- reporting_design.md | 1 - 1 file changed, 1 deletion(-) diff --git a/reporting_design.md b/reporting_design.md index 572ed47..9cb4e89 100644 --- a/reporting_design.md +++ b/reporting_design.md @@ -159,7 +159,6 @@ TODO: See if we can define allowed configuration types more strictly. #### Output directory * "build.kernel.tar.gz" - the built kernel tarball, as required by "run" input. -* "build.kernel.config" - the used kernel configuration * "build.report" - report text, as described above * "build.result" - stage result status, as described above From bbd7130ceefdfac2b9a902c3d79e6ea6310a3773 Mon Sep 17 00:00:00 2001 From: Nikolai Kondrashov Date: Wed, 11 Jul 2018 16:04:53 +0300 Subject: [PATCH 11/12] reporting_design: Use attachment paths, not names --- reporting_design.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/reporting_design.md b/reporting_design.md index 9cb4e89..977962e 100644 --- a/reporting_design.md +++ b/reporting_design.md @@ -66,11 +66,11 @@ into the overall report by sktm. The text can refer to other files output by the stage, using special formatting, which can then be replaced by skt by e.g. references to e-mail attachments or hyperlinks. -The syntax of such references is `{}`, where `` is the name of the -output file to include into the report. To provide an option to include -literal curly braces into reports, any characters preceded by a backslash -would be copied by sktm into the summarized report literally, without -interpretation. +The syntax of such references is `{}`, where `` is the path to the +output file to include into the report, relative to the directory the report +file is in. To provide an option to include literal curly braces into reports, +any characters preceded by a backslash would be copied by sktm into the +summarized report literally, without interpretation. ##### Result status Each stage output includes a status file, which is set executable and contains From a97a826dd0b4c31f0566875285e4d382ed020a38 Mon Sep 17 00:00:00 2001 From: Nikolai Kondrashov Date: Wed, 11 Jul 2018 16:17:37 +0300 Subject: [PATCH 12/12] reporting_design: Add report inclusion --- reporting_design.md | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/reporting_design.md b/reporting_design.md index 977962e..d282920 100644 --- a/reporting_design.md +++ b/reporting_design.md @@ -68,9 +68,34 @@ e.g. references to e-mail attachments or hyperlinks. The syntax of such references is `{}`, where `` is the path to the output file to include into the report, relative to the directory the report -file is in. To provide an option to include literal curly braces into reports, -any characters preceded by a backslash would be copied by sktm into the -summarized report literally, without interpretation. +file is in. + +The text can refer to other reports to be included directly into the resulting +report text. Such included reports are given a name, which is used to +"namespace" all the output files they refer to, by e.g. prepending it to their +filenames. The syntax of such references is `<:>`, where `` +is the report name (consisting of alphanumeric characters and underscores, can +be empty for "root" namespace), and `` is the path to the report to +include, relative to the directory the including report text resides in. + +For example, the following report: + + x86_64 build failed: + + +is referring to another report located in `x86_64/build.report`, and is naming +it `x86_64`. So, if that report was: + + See {build.log.gz} for details. + +then e.g. the resulting e-mail would read: + + x86_64 build failed: + See the attached x86_64_build.log.gz for details. + +To provide the option to include literal curly braces or angle brackets into +reports, any characters preceded by a backslash would be copied by sktm into +the summarized report literally, without interpretation. ##### Result status Each stage output includes a status file, which is set executable and contains