Musing name is coded into POMS configuration

[brownd1978]

The musing name is used as part of the description in the output file names and dataset names produced by a campaign. Currently the musing name is hard-coded into the POMS configuration files.
This creates a dependency reversal that files with the new musing name must committed and tagged before that musing actually exists. It relies on the operator to insure by hand that the content is consistent when the musing is finally created. A better design would be to extract the musing name from the musing itself when building file and dataset names, so that it never appears in the repo.

[rlcee]

I think #216 and this are conceptually the same problem and I'd combine them. We have talked about this in the form of noting that POMS doesn't distinguish between a campaign item that defines a procedure and a parameter that might be considered as input to the procedure (like input and output file name, database choice, etc). In that discussion, we thought a viable approach would be to flag POMS parameters as variable by calling them something like "par_xyz" and allowing those to be modified as needed for the job. So then you have fewer tag updates and smaller chicken-egg problem, but this becomes a problem of reliably documenting how you set the par_* parameters. I think there is still a lot of details to work out so I wouldn't make a decision yet, but just call this another concept in the toolbox.

[kutschke]

I agree with Ray that #215 and #216 are will likely be resolved by an solution that resolves both at the same time.

Here are two ideas. I have not yet thought them through to end.

One way to resolve this is to move POMS files out of Production and into a new repo; so the scope of the Production repos is narrowed. Maybe this repo is named pomsconfig, workflow, wfconfig .... ?

Another way is treat some well identified files in Production as "templates"; conceptually some parameters would be given values of "nil". To create an actual campaign, you process the "templates" (by hand or by script) to create a complete campaign. We would need to define where to store those scripts and where to store the ready-to-run POMS files.

The first solution feels like the cleanest to me.

[brownd1978]

I agree they are related and will probably have the same or similar solution. As the source of the actual code, the Musing seems more fundamental than the database version, but I could be wrong.

On Fri, Nov 11, 2022 at 7:45 AM Rob Kutschke ***@***.***> wrote: I agree with Ray that #215 <#215> and #216 <#216> are will likely be resolved by an solution that resolves both at the same time. Here are two ideas. I have not yet thought them through to end. One way to resolve this is to move POMS files out of Production and into a new repo; so the scope of the Production repos is narrowed. Maybe this repo is named pomsconfig, workflow, wfconfig .... ?

I don't think this solves the problem. We would need to expand the musing to include this new repo to fully document the production, which brings us right back to where we started. I strongly suspect that ultimately, the person launching the campaign will have to type the musing and database version into a web form (or maybe select them from a pulldown menu), along with some other parameters: Minimum and maximum run number to process, ... How those parameters get embedded into the scripts that actually run remains to be worked out, as does how we create a record of what values were used and associate them with the resultant dataset. I could imagine the later might simply be the sam metadata.

Another way is treat some well identified files in Production as "templates"; conceptually some parameters would be given values of "nil". To create an actual campaign, you process the "templates" (by hand or by script) to create a complete campaign. We would need to define where to store those scripts and where to store the ready-to-run POMS files.

The fcl files in Production/Jobconfig are already "templates" (I called them stubs): they cannot be run directly, they must first get processed by the scripts in Production/Scripts, which set several parameters. I suspect the solution for the POMS scripts will be similar. Dave

…

The first solution feels like the cleanest to me. — Reply to this email directly, view it on GitHub <#215 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABAH574GDSCXIDXSA6VTQ7LWHZSZPANCNFSM6AAAAAAR5TEUJE> . You are receiving this because you were assigned.Message ID: ***@***.***>

-- David Brown ***@***.*** Office Phone (510) 486-7261 Lawrence Berkeley National Lab M/S 50R5008 (50-6026C) Berkeley, CA 94720

[kutschke]

I don't think this solves the problem. We would need to expand the musing to include this new repo to fully document the production, which brings us right back to where we started.

In my picture this new repo would not need to be in the Musing, it would be checked out in the working area from which POMS jobs are created/submitted. What I don't know is if we can cleanly factorize the exisiting Production into two repos such that my picture works.

I strongly suspect that ultimately, the person launching the campaign will have to type the musing and database version into a web form (or maybe select them from a pulldown menu), along with some other parameters: Minimum and maximum run number to process, ... How those parameters get embedded into the scripts that actually run remains to be worked out, as does how we create a record of what values were used and associate them with the resultant dataset. I could imagine the later might simply be the sam metadata.

This sounds very similar to my second suggestions - the difference is that the web form can create the files that must be preserved and can automatically preserve them (repo or other solution ). Web forms always need ongoing maintenance to add new fields and to add options to pull-down menus but if they ensure provenance it's worth the effort. They can also automatically check for many types of misconfigurations before allowing the work to proceed.

[brownd1978]

On Fri, Nov 11, 2022 at 8:49 AM Rob Kutschke ***@***.***> wrote: I don't think this solves the problem. We would need to expand the musing to include this new repo to fully document the production, which brings us right back to where we started. In my picture this new repo would not need to be in the Musing, it would be checked out in the working area from which POMS jobs are created/submitted.

POMS campaigns are launched from a web form, not from a workdir. They can refer to a workdir, but issue 207 <#207> is about making sure they only reference cvmfs directories, hence the need for a musing.

…

What I don't know is if we can cleanly factorize the exisiting Production into two repos such that my picture works. I strongly suspect that ultimately, the person launching the campaign will have to type the musing and database version into a web form (or maybe select them from a pulldown menu), along with some other parameters: Minimum and maximum run number to process, ... How those parameters get embedded into the scripts that actually run remains to be worked out, as does how we create a record of what values were used and associate them with the resultant dataset. I could imagine the later might simply be the sam metadata. This sounds very similar to my second suggestions - the difference is that the web form can create the files that must be preserved and can automatically preserve them (repo or other solution ). Web forms always need ongoing maintenance to add new fields and to add options to pull-down menus but if they ensure provenance it's worth the effort. They can also automatically check for many types of misconfigurations before allowing the work to proceed. — Reply to this email directly, view it on GitHub <#215 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABAH57ZFNRD3EEUUMZTWY63WHZ2JBANCNFSM6AAAAAAR5TEUJE> . You are receiving this because you were assigned.Message ID: ***@***.***>

-- David Brown ***@***.*** Office Phone (510) 486-7261 Lawrence Berkeley National Lab M/S 50R5008 (50-6026C) Berkeley, CA 94720

[rlcee]

I think basically all these ideas have been raised before. They are all on the table for Pass1, as far as I'm concerned. Let me make one obvious(?) point, that simulation campaigns and pass1 may ultimately have a different emphasis in their needs, so may have a different infrastructure. Now that the current pass1 talk is winding down, I'm beginning to think that a new phase is required, a list of requirements and scenarios, and how they would be handled, in some detail, in order to start making decisions on the ideas on the table. It seems some issues just have trade-offs, not obvious correct answers, and we may not be able to maintain all favored principles at once.

[kutschke]

On Fri, Nov 11, 2022 at 8:49 AM Rob Kutschke @.***> wrote: I don't think this solves the problem. We would need to expand the musing to include this new repo to fully document the production, which brings us right back to where we started. In my picture this new repo would not need to be in the Musing, it would be checked out in the working area from which POMS jobs are created/submitted.
POMS campaigns are launched from a web form, not from a workdir. They can refer to a workdir, but issue 207 <#207> is about making sure they only reference cvmfs directories, hence the need for a musing.
…

Can we add our own fields to the web form or is it fixed by the POMS team?

[rlcee]

Can we add our own fields to the web form or is it fixed by the POMS team?
I think basically, the only fields are fields we define and set at any time.

I've been musing the following design, which has sort-of been under our noses. I'll call the concept "parsets". We have one implementation which is the envset. We have Roberto doing this, but not using it as a design.

We strip down the POMS job so all it knows is "I have a art file on input and I produce an art file and log file" Everything else is captured in two POMS parameters:

1. the path to the cfg file which knows what to do to get input to output as infrastructure, but not any details.
2. a path to cfg files in /cvmfs/.../POMSsets/MDC2020_000.cfg
3. would include 2. 2 contains things like input file naming, database choice and output file naming.

I've thought about some details such as bookkeeping, and I think this looks pretty good. I am immediately "attracted to it".

I propose that after the CM, I give a pass1_2 talk about requirements, and a Pass1_3 talk about the pros and cons of this (or other) design against the requirements.

[brownd1978]

Hi Ray, What part of this is how Roberto operated POMS? I don't see how stripped-down campaigns can handle mixing (primary + secondary input streams) or digitization (one input multiple output streams). Are you going back to a purely atomic campaign model? I don't see how moving the .cfg files helps us: we still need a way to create those files, which is the intellectual challenging part IMHO. Are you just saying you want to put those (transient) files on /cvmfs? If so, that's fine with me, but it doesn't solve the problem of merging the 'fixed' content coming from the repo (how campaigns are structured in terms of fcl, jobs, etc) with 'transient' information (Musing version, database version, ...). Dave

…

On Fri, Nov 11, 2022 at 11:04 AM Ray Culbertson ***@***.***> wrote: Can we add our own fields to the web form or is it fixed by the POMS team? I think basically, the only fields are fields we define and set at any time. I've been musing the following design, which has sort-of been under our noses. I'll call the concept "parsets". We have one implementation which is the envset. We have Roberto doing this, but not using it as a design. We strip down the POMS job so all it knows is "I have a art file on input and I produce an art file and log file" Everything else is captured in two POMS parameters: 1. the path to the cfg file which knows what to do to get input to output as infrastructure, but not any details. 2. a path to cfg files in /cvmfs/.../POMSsets/MDC2020_000.cfg 3. would include 2. 2 contains things like input file naming, database choice and output file naming. I've thought about some details such as bookkeeping, and I think this looks pretty good. I am immediately "attracted to it". I propose that after the CM, I give a pass1_2 talk about requirements, and a Pass1_3 talk about the pros and cons of this (or other) design against the requirements. — Reply to this email directly, view it on GitHub <#215 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABAH57534IGO4CY6U6UUP6TWH2KEPANCNFSM6AAAAAAR5TEUJE> . You are receiving this because you were assigned.Message ID: ***@***.***>

-- David Brown ***@***.*** Office Phone (510) 486-7261 Lawrence Berkeley National Lab M/S 50R5008 (50-6026C) Berkeley, CA 94720

[rlcee]

Hi Ray, What part of this is how Roberto operated POMS?

He committed cfg files which included cfg files in his personal area, conceptually I just made that part of the design.

I don't see how stripped-down campaigns can handle mixing (primary + secondary input streams) or digitization (one input multiple output streams).

I don't know how to answer this because I don't fully understand the constraints, but I imagine several different "stripped down" POMS campaigns, each handles a different structure of jobs (Primary, Mixing) (but doesn't know exactly which files it will handle). I'll think about it. As I mentioned it might also be that pass1 and sim have different emphasis so end up with somewhat different designs. In my musing, I'm also assuming fcl-less jobs everywhere, but I don't think that is a requirement for the parsets design.

Are you going back to a purely atomic campaign model?

That is an orthogonal question

I don't see how moving the .cfg files helps us: we still need a way to create those files, which is the intellectual challenging part IMHO.

In my casual observation I completely agree - campaign speed is mostly limited by development

Are you just saying you want to put those (transient) files on /cvmfs?

Yes, but they are not any more transient than envsets are, They are a permanent part of the processing record, once written they never change (like envsets).

If so, that's fine with me, but it doesn't solve the problem of merging the 'fixed' content coming from the repo (how campaigns are structured in terms of fcl, jobs, etc) with 'transient' information (Musing version, database version, ...).

My claim/intent is that is exactly what it does. It splits the procedure into an infrastructure part, which is in git tags and a parameters part which is documented as more-easily-updated, but permanent, POMSsets files. Many details TBD