add support for performing 2-flavour and BSM inversions from external program via lib_wrapper by kostrzewa · Pull Request #4 · pittlerf/tmLQCD

kostrzewa · 2021-10-14T17:18:19Z

I've adjusted where things are initialised, such that all BSM operators are now also usable when inversions are driven via tmLQCD_invert in wrapper/lib_wrapper.c. In the process I've also changed how the scalar field index is kept track of (at least in invert.c by externalizing this to a function in io/scalar_read.c).

I only touched contractions_BSM in a few places. Could you please check that it still works?

I was a bit surprised that you have zero-padding of the scalar field index in contractions_BSM, while in the past it was not zero-padded. Does this mean that we've agreed on zero-padding now?

…eld reading logic currently in invert

…ublet source

…ernal program using tmLQCD's lib_wrapper

kostrzewa · 2021-10-15T09:11:12Z

  int tmLQCD_get_mpi_params(tmLQCD_mpi_params * params);
  int tmLQCD_get_lat_params(tmLQCD_lat_params * params);

+  int tmLQCD_load_next_scalar_field(const int nscalarstep, const int npergauge, const int imeas, const int reset);


I'm not sure if I want to keep this interface like this. Perhaps it makes sense after all that the scalar index is computed externally and simply passed (just like for the gauge field). The reasoning for this was to keep the index calculation in the load_next_scalar_field function in io/scalar_read.c to avoid repetition, but I might just externalize this into a separate function which can then be called also by the external program.

Yeah, I've switched away from this to a solution with two functions.

…erfaces for external usage

kostrzewa · 2021-10-15T11:25:23Z

-          exit(0);
-        }
+        if (j!=0) fatal_error("Not enough memory for scalar fields! Aborting...\n", __func__);
+        j = init_bispinor_field(VOLUMEPLUSRAND, 4);


This is a major problem unfortunately. I don't really want to call init_bispinor_field from every executable that we have (because it is a bona-fide dependency of the BSM operators). At the same time, this does not really fit into the way that tmLQCD uses the bispinor fields... Not quite sure yet what a good solution would be.

…ugeConfigInputFile is set

kostrzewa · 2021-10-15T14:41:35Z

I've now extended the support for the smeared gauge field in a more general fashion. When SmearedGaugeConfigInputFile is set, memory is allocated for a separate smeared gauge field. I've adjusted invert.c to make use of this, such that the BSM3 operator can actually be inverted by the invert program.

I still need to take care of interfacing the smearing of the scalar field.

kostrzewa · 2021-10-16T11:44:26Z

The two inversions performed in op_invert for the BSM operators need to be controlled via some kind of interface such that it's possible to get either D^{-1} or [D^\dagger]^{-1}. Of course, this was never intended in tmLQCD so it will take some creativity to get there.

It's also quite silly that op_invert contains the loop over source flavours. This was fine when the default was to write out the propagators to disk, but it's not a good idea at this stage...

pittlerf · 2021-10-16T15:29:29Z

op_invert

yeah, maybe a boolean inside the parameters, whether it should be daggered or not?

pittlerf · 2021-10-16T15:41:24Z

I actually have problems with the parts of
f( (i = read_gauge_field(conf_filename,g_smeared_gauge_field)) !=0)
in contractions_BSM

kostrzewa · 2021-10-16T15:44:09Z

I actually have problems with the parts of
f( (i = read_gauge_field(conf_filename,g_smeared_gauge_field)) !=0)
in contractions_BSM

can you explain?

pittlerf · 2021-10-16T15:45:34Z

I actually have problems with the parts of
f( (i = read_gauge_field(conf_filename,g_smeared_gauge_field)) !=0)
in contractions_BSM

can you explain?

yeah, sorry I got segmentation fault, when trying to read

Trying to read smeared gauge field from file conf_smeared.0000 in double precision.

Constructing LIME reader for file conf_smeared.0000 ...

WARNING: IO CHECKS HAVE BEEN DISABLED

[cyclamen:17843] *** Process received signal ***
[cyclamen:17843] Signal: Segmentation fault (11)
[cyclamen:17843] Signal code: Address not mapped (1)
[cyclamen:17843] Failing at address: (nil)
[cyclamen:17843] [ 0] /lib64/libpthread.so.0(+0xf5d0)[0x7f415c4215d0]
[cyclamen:17843] [ 1] ./contractions_BSM[0x6bc196]
[cyclamen:17843] [ 2] ./contractions_BSM[0x6b540b]
[cyclamen:17843] [ 3] ./contractions_BSM[0x4035ba]
[cyclamen:17843] [ 4] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f415c0673d5]
[cyclamen:17843] [ 5] ./contractions_BSM[0x403039]
[cyclamen:17843] *** End of error message ***
Segmentation fault

I have to look at it, maybe something in the initializiation changed

kostrzewa · 2021-10-16T15:47:52Z

You have to set SmearedGaugeConfigInputFile. When this has non-zero length, init_gauge_field is called with smeared = 1, which causes memory for the smeared gauge field to be allocated.

pittlerf · 2021-10-16T15:49:11Z

You have to set SmearedGaugeConfigInputFile. When this has non-zero length, init_gauge_field is called with smeared = 1, which causes memory for the smeared gauge field to be allocated.

thank you

kostrzewa · 2021-10-16T15:52:31Z

op_invert
yeah, maybe a boolean inside the parameters, whether it should be daggered or not?

sure, I just don't know yet how to integrate this elegantly with the other behaviours

pittlerf · 2021-10-16T15:53:20Z

yeah, basically in the other case we do not need the dagger at all

kostrzewa · 2021-10-16T16:26:39Z

I think there's a way to handle multiplying the source by the operator before the inversion in prepare_source, I'm just not 100% sure yet if it can be exposed nicely all the way to lib_wrapper. Maybe I'll have some time 2mw to implement it.

kostrzewa · 2021-10-18T11:55:07Z

@urbach I would like to modify the logic for dealing with two-flavour operators in op_invert. On the one hand, this will make it easier to deal with the needs of the BSM operators, but it will also translate to being able to drive ND inversions via lib_wrapper in a straightforward manner. We can then backport these changes into the quda and eventually master branches for doing ND inversions comfortably.

remove the loop over source flavours in op_invert
add source flavour loop in invert.c (if using invert)
add source flavour index argument to prepare_source

Would value your input if you see issues.

…idx' and 'dagger' arguments to drive inversions of M and Mdagger as well as the two source flavour indices externally

kostrzewa · 2021-10-18T14:11:31Z

In 29df069 I've implemented what I came up with to solve our problem in a (hopefully) minimally-invasive way. I've broken contractions_BSM in the process but will attempt to fix it today or 2mw. Basically I've moved the two loops over flavour and non-dagger, dagger out of op_invert to be done externally. To make this work, I've added a flav_idx argument to prepare_source and op_invert and a dagger argument to op_invert.

…g the daggered operator

kostrzewa · 2021-10-28T15:18:04Z

@pittlerf @Marcogarofalo do I understand it correctly that the scalar field used for the propagators (the one stored in g_scalar_field) is already pre-smeared? Or in other words, it is smeared externally and then loaded via read_scalar_field[_parallel]?

pittlerf · 2021-10-28T15:19:09Z

Yes, hypercubically smeared.

kostrzewa · 2021-10-28T15:21:02Z

Thanks.

kostrzewa · 2021-10-29T14:12:18Z

@pittlerf I've got the crosscheck for any combination of gamma and tau structures at source and sink ready. What I don't have yet is the pseudo-Wilson density but I'll likely be able to add that in the coming weeks too. I'll send my correlators to the list in a few minutes.

pittlerf · 2021-10-29T14:36:42Z

Thank you very much

kostrzewa · 2021-11-03T13:29:37Z

@pittlerf sorry I just noticed that I had not actually included the axial vector ;) I'll give an update here soon.

kostrzewa · 2021-11-04T19:08:20Z

Current state of the cross-check

See also https://github.com/HISKP-LQCD/bsmsimul/commit/018a355e2fe1543f0e0ebc8df5fadfaac429467a and https://github.com/HISKP-LQCD/bsmsimul/commit/907e6e480da306ce8ec80a8777e40ebe30e9ad32

Scalar field average compute

kostrzewa · 2021-11-11T16:32:51Z

@pittlerf what does "LOCALCURRENTTAU0P0" mean? More specifically, what is TAU0 ?

Marcogarofalo · 2021-11-11T16:40:15Z

I suspect that there was a typo that shifted the enumeration so
LOCALCURRENTTAU0P0 should be renamed LOCALCURRENTTAU1P1
LOCALCURRENTTAU1P1 should be renamed LOCALCURRENTTAU2P2
LOCALCURRENTTAU2P2 should be renamed LOCALCURRENTTAU3P3

pittlerf · 2021-11-11T19:48:24Z

I suspect that there was a typo that shifted the enumeration so LOCALCURRENTTAU0P0 should be renamed LOCALCURRENTTAU1P1 LOCALCURRENTTAU1P1 should be renamed LOCALCURRENTTAU2P2 LOCALCURRENTTAU2P2 should be renamed LOCALCURRENTTAU3P3

Hi, yes, this is a typo,thanks for noticing it, I am correcting now

…to tau1 , tau1 to tau2 and tau2 to tau3

kostrzewa · 2021-11-12T15:54:50Z

Thanks, I figured but wanted to make sure that I had not misunderstood (tau0 could be a shorthand for the identity to build the iscoscalar combination, for example).

This is the current state of the cross-check:

pittlerf · 2021-11-12T15:57:40Z

ah, ok, thanks there are problems with tau2 and tau3, but tau1 is correct

kostrzewa · 2021-11-12T15:58:57Z

ah, ok, thanks there are problems with tau2 and tau3, but tau1 is correct

it seems so but I wonder how that can happen

kostrzewa · 2021-11-16T11:54:52Z

ah, ok, thanks there are problems with tau2 and tau3, but tau1 is correct

it seems so but I wonder how that can happen

@pittlerf should we compare some intermediate fields or something like that?

pittlerf · 2021-11-16T12:01:10Z

ah, ok, thanks there are problems with tau2 and tau3, but tau1 is correct

it seems so but I wonder how that can happen

@pittlerf should we compare some intermediate fields or something like that?
Hi Batek, yes we could do that, I have tried to build nyom on cyclamen,
but I got an error regarding yamlcpp:
`CMake Error at /cyclamen/home/fpittler/build/yamlcpp/yaml-cpp-config.cmake:11 (include):
include could not find load file:

/cyclamen/home/fpittler/build/yamlcpp/yaml-cpp-targets.cmake

Call Stack (most recent call first):
CMakeLists.txt:107 (find_package)`

kostrzewa · 2021-11-16T17:18:33Z

I have tried to build nyom on cyclamen, but I got an error regarding yamlcpp:

yaml-cpp (https://github.com/jbeder/yaml-cpp) is a dependency and one either has to have it installed somewhere where CMake looks for cmake files by default (for example, in /usr, such that the cmake files are under /usr/share/cmake/yaml-cpp) or one sets -DCMAKE_SYSTEM_PREFIX_PATH=[...] to the directory where one has installed it locally. For example, I do:

-DCMAKE_SYSTEM_PREFIX_PATH="/nfshome/bartek/build/yaml-cpp/install_dir/share/cmake/yaml-cpp"

in my cmake call to configure nyom.

Hi Bartek, yes we could do that

So how should we proceed? I can relatively easily write out any intermediate quantity, but I'm not sure whether this makes so much sense or whether we should instead discuss in terms of formulae. I'm a bit surprised that we have compatible results for <P^2 P^2> (up to sign) but see a difference when the sink gamma matrix is changed to \gamma_0 \gamma_5 instead of just \gamma_5...

kostrzewa added 6 commits October 12, 2021 11:42

add support for doublet inversions in lib_wrapper

6857775

initialisation of D_psi_BSM[2,3] is done in init_operators, scalar fi…

b8f3f3d

…eld reading logic currently in invert

fix initialisation order in tmLQCD_invert_init and properly handle do…

3400825

…ublet source

rename nscalar to initial_scalar_counter to make it easier to understand

cc2fb7e

add support for performing two-flavour and BSM inversions from an ext…

eee36b5

…ernal program using tmLQCD's lib_wrapper

define SCALAR_FILENAME_LENGTH only in one place

f70264a

kostrzewa commented Oct 15, 2021

View reviewed changes

kostrzewa added 2 commits October 15, 2021 11:45

externalize scalar_idx calculation into separate function and add int…

20bbd82

…erfaces for external usage

remove unwanted debug statement

1b24045

kostrzewa commented Oct 15, 2021

View reviewed changes

support a global smeared gauge field in general, enabled if SmearedGa…

2690523

…ugeConfigInputFile is set

kostrzewa commented Oct 16, 2021

View reviewed changes

Comment thread init/init_gauge_field.c Outdated

kostrzewa force-pushed the BSMmerge_lib_wrapper branch from 1a775e8 to 2690523 Compare October 16, 2021 16:23

typos in init_gauge_field

4045c27

WIP: for ND operators, move flavour loop out of op_invert, add 'flav_…

29df069

…idx' and 'dagger' arguments to drive inversions of M and Mdagger as well as the two source flavour indices externally

kostrzewa added 2 commits October 18, 2021 16:40

typo in is_doublet logic in prepare_source

0de358b

expose tmLQCD_invert interface for the flavour index and for invertin…

6933907

…g the daggered operator

adding loops for flavours and dagger in contraction_BSM

57463d7

pittlerf added 2 commits October 29, 2021 00:12

work in progress local current times density contraction

e421ce6

implementing the local axial current times pseudo-scalar density

3733044

kostrzewa added 2 commits October 30, 2021 06:27

expose scalar field pointers in interface

e816042

change signature of scalar field pointer exposure

b2b71e4

pittlerf and others added 5 commits November 4, 2021 00:10

computing average scalar field

b002b47

some xcheck on the gauge conf filename

987fa87

typo corrected

f1c5e42

another typo corrected

b9260e3

correcting bug

0c25f29

kostrzewa and others added 2 commits November 5, 2021 19:48

Merge pull request #16 from pittlerf/scalar_field_average_compute

aaa6b24

Scalar field average compute

error in if statement after snprintf is corrected

19dead4

correcting naming conventions in local current density, nrename tau0 …

fcc12f0

…to tau1 , tau1 to tau2 and tau2 to tau3

kostrzewa mentioned this pull request Dec 11, 2021

augment QUDA interface to support two-flavour inversions for analysis etmc/tmLQCD#514

Open

Conversation

kostrzewa commented Oct 14, 2021

Uh oh!

kostrzewa Oct 15, 2021

Choose a reason for hiding this comment

Uh oh!

kostrzewa Oct 15, 2021

Choose a reason for hiding this comment

Uh oh!

kostrzewa Oct 15, 2021

Choose a reason for hiding this comment

Uh oh!

kostrzewa commented Oct 15, 2021

Uh oh!

kostrzewa commented Oct 16, 2021

Uh oh!

pittlerf commented Oct 16, 2021

Uh oh!

Uh oh!

pittlerf commented Oct 16, 2021

Uh oh!

kostrzewa commented Oct 16, 2021

Uh oh!

pittlerf commented Oct 16, 2021

Trying to read smeared gauge field from file conf_smeared.0000 in double precision.

Constructing LIME reader for file conf_smeared.0000 ...

WARNING: IO CHECKS HAVE BEEN DISABLED

Uh oh!

kostrzewa commented Oct 16, 2021

Uh oh!

pittlerf commented Oct 16, 2021

Uh oh!

kostrzewa commented Oct 16, 2021

Uh oh!

pittlerf commented Oct 16, 2021

Uh oh!

kostrzewa commented Oct 16, 2021

Uh oh!

kostrzewa commented Oct 18, 2021

Uh oh!

kostrzewa commented Oct 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kostrzewa commented Oct 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pittlerf commented Oct 28, 2021

Uh oh!

kostrzewa commented Oct 28, 2021

Uh oh!

kostrzewa commented Oct 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pittlerf commented Oct 29, 2021

Uh oh!

kostrzewa commented Nov 3, 2021

Uh oh!

kostrzewa commented Nov 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kostrzewa commented Nov 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Marcogarofalo commented Nov 11, 2021

Uh oh!

pittlerf commented Nov 11, 2021

Uh oh!

kostrzewa commented Nov 12, 2021

Uh oh!

pittlerf commented Nov 12, 2021

Uh oh!

kostrzewa commented Nov 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kostrzewa commented Nov 16, 2021

Uh oh!

pittlerf commented Nov 16, 2021

Uh oh!

kostrzewa commented Nov 16, 2021

Uh oh!

kostrzewa commented Oct 18, 2021 •

edited

Loading

kostrzewa commented Oct 28, 2021 •

edited

Loading

kostrzewa commented Oct 29, 2021 •

edited

Loading

kostrzewa commented Nov 4, 2021 •

edited

Loading

kostrzewa commented Nov 11, 2021 •

edited

Loading

kostrzewa commented Nov 12, 2021 •

edited

Loading