Add UFS tracing instrumentation to nuopc/cmeps driver by NickSzapiro-NOAA · Pull Request #1075 · CICE-Consortium/CICE

NickSzapiro-NOAA · 2025-12-16T15:26:07Z

For detailed information about submitting Pull Requests (PRs) to the CICE-Consortium,
please refer to: https://github.com/CICE-Consortium/About-Us/wiki/Resource-Index#information-for-developers

PR checklist

ufs-community/ufs-weather-model#2884 adds a simple tracing module to UFS and updates some subcomponents' nuopc drivers to produce a trace file which can be used to identify performance issues. The tracing module is not built and used by default. It can be enabled by setting a build option `-DUFS_TRACING=ON'. The generated trace files can be visualized using the chrome-tracing tool or, for example, the Perfetto UI online tool (https://ui.perfetto.dev/) like in ufs-community/ufs-weather-model#2883 . Tracing has been useful in optimizing GFS runtime.

…restart_fh") via UFS configuration as for other components Part of ufs-community/ufs-weather-model#2348

NickSzapiro-NOAA · 2025-12-16T15:38:23Z

This is a draft PR as 1) the UFS PR hasn't been merged yet and 2) to ask if the changes are acceptable to the nuopc/cmeps cap

Maybe asking @dabail10 @anton-seaice in particular if there is objection to UFS_TRACING ifdef at this level (and any other preferences/criticisms/...). There is an alternate flavor to have the ifdefs at lower layer more like cice_wrapper_mod.F90 (with wrapper like med_ufs_trace_wrapper_mod in NOAA-EMC/CMEPS#151)

anton-seaice

All ok from me, we might find that useing the same driver code eventually becomes unmanageable, but for now happy to keep trying to keep it all together.

I should have asked more about the tracing work on Tuesday. Is there a summary repo / issue / notebook somewhere ?

anton-seaice · 2025-12-16T21:32:42Z

+    call ESMF_GridCompGet(gcomp, vm=vm,rc=rc)
+    if (ChkErr(rc,__LINE__,u_FILE_u)) return
+    call ESMF_VMGet(vm, localpet=mype, rc=rc)
+    if (ChkErr(rc,__LINE__,u_FILE_u)) return


if (ChkErr(rc,__LINE__,u_FILE_u)) return call ESMF_VMGet(vm, localpet=mype, rc=rc) if (ChkErr(rc,__LINE__,u_FILE_u)) return

Are these used anywhere?

to set mype (before my_task and master_task are available in InitializeAdvertise)

dabail10

This generally looks fine, but we already have variables for this sort of thing in CICE.

dabail10 · 2025-12-16T21:48:30Z

+    if (ChkErr(rc,__LINE__,u_FILE_u)) return
+
+#ifdef UFS_TRACING
+    if (mype == 0) call ufs_trace_init()


We should remain consistent with the rest of the code and check:

if (my_task == master_task)

Yes, I thought the same. But, the tracing starts before my_task and master_task are available in InitializeAdvertise

Can we call init_grid1 before this phase?

Maybe not (?) as I think call input_data has to happen before call init_grid1 like in cice_init1 to have all the namelist variables

But, it's more that conflicts with the intent to start the tracing as soon as possible in all subcomponents (before calling much else) towards getting the fullest timeline of the run. And for this to be "unobtrusive"

fwiw, I think master_task and mype==0 are really the same since

CICE/cicecore/cicedyn/infrastructure/comm/mpi/ice_communicate.F90

Line 84 in fa682b3

master_task = 0

I guess these lines could be moved up to SetServices from InitializeAdvertise if there is really strong preference

CICE/cicecore/drivers/nuopc/cmeps/ice_comp_nuopc.F90

Lines 346 to 368 in fa682b3

call ESMF_GridCompGet(gcomp, vm=vm, rc=rc)

if (ChkErr(rc,__LINE__,u_FILE_u)) return

call ESMF_VMGet(vm, mpiCommunicator=lmpicom, localPet=localPet, PetCount=npes, rc=rc)

if (ChkErr(rc,__LINE__,u_FILE_u)) return

#ifdef CESMCOUPLED

call ESMF_VMGet(vm, pet=localPet, peCount=nthrds, rc=rc)

if (ChkErr(rc,__LINE__,u_FILE_u)) return

if (nthrds==1) then

call NUOPC_CompAttributeGet(gcomp, "nthreads", value=cvalue, rc=rc)

if (ESMF_LogFoundError(rcToCheck=rc, msg=ESMF_LOGERR_PASSTHRU, line=__LINE__, file=u_FILE_u)) return

read(cvalue,*) nthrds

endif

!$ call omp_set_num_threads(nthrds)

#endif

!----------------------------------------------------------------------------

! Initialize cice communicators

!----------------------------------------------------------------------------

call init_communicate(lmpicom) ! initial setup for message passing

mastertask = .false.

if (my_task == master_task) mastertask = .true.

I think it is not guaranteed on all systems that master_task = 0. Does the call to ufs_trace_init really have to happen at this phase? I realize there are often issues with circular dependencies. I feel like the tracing doesn't need to happen before the scatter and PE decomposition.

I think it's just easier to trace fully than "be smart" about what needs to be traced

One example where accounting/separating the different phases was found to be very useful was in trying to reduce GFS initialization time (ufs-community/ufs-weather-model#2831). Like removing some advertised fields from FV3 reduced InitializeAdvertise (top to bottom)

dabail10 · 2025-12-16T21:48:57Z

  character(*), parameter      :: u_FILE_u = &
       __FILE__

+  integer :: mype = -1


There is already my_task available in the code.

DeniseWorthen · 2025-12-16T23:15:01Z

All ok from me, we might find that useing the same driver code eventually becomes unmanageable, but for now happy to keep trying to keep it all together.

I should have asked more about the tracing work on Tuesday. Is there a summary repo / issue / notebook somewhere ?

I think both CESM and UFS have benefited greatly from sharing common nuopc_caps between MOM6, CICE6 and WW3, coupled to CMEPS (and CDEPS). I think we should strive to maintain that inter-operability.

But when there are features or options that are really used by only one system, I think we all benefit from making those use cases as unobtrusive as possible, thus the existence of various wrapper mods in the shared NUOPC caps. That was my thinking in asking Nick to get feedback from the other cap users.

The ufs-tracing was something one of our colleagues developed during the recent push to get the GFSv17 ready for operations. The existing EMSF based profiling didn't provide the sort of "time line" feature he wanted, so he developed a very basic tracing code that could sit inside each component. I think it is an exceptionally useful little feature, but it would need a CESM user to enable it across their infrastructure system (driver etc).

NickSzapiro-NOAA · 2025-12-17T23:12:21Z

I should have asked more about the tracing work on Tuesday. Is there a summary repo / issue / notebook somewhere ?

@anton-seaice Not afaik, but it's intended to be a straightforward setup. Each component writes to its own trace file and then combine them (ufs_tracing/combine_traces.sh) to look like:

{"name":"SetServices", "ph":"B", "ts":"1416357460", "pid":"1", "tid":"cice"},
{"name":"SetServices", "ph":"E", "ts":"1416359964", "pid":"1", "tid":"cice"},
{"name":"InitializeP0", "ph":"B", "ts":"1416431949", "pid":"1", "tid":"cice"},
{"name":"InitializeP0", "ph":"E", "ts":"1416432126", "pid":"1", "tid":"cice"},
{"name":"InitializeAdvertise", "ph":"B", "ts":"1419655739", "pid":"1", "tid":"cice"},
{"name":"InitializeAdvertise", "ph":"E", "ts":"1425338210", "pid":"1", "tid":"cice"},
{"name":"InitializeRealize", "ph":"B", "ts":"1498516061", "pid":"1", "tid":"cice"},
{"name":"InitializeRealize", "ph":"E", "ts":"1506396931", "pid":"1", "tid":"cice"},
{"name":"ModelAdvance", "ph":"B", "ts":"1580100026", "pid":"1", "tid":"cice"},
{"name":"ModelAdvance", "ph":"E", "ts":"1581058909", "pid":"1", "tid":"cice"},
...
{"name":"ModelAdvance", "ph":"B", "ts":"2194075055", "pid":"1", "tid":"cice"},
{"name":"ModelAdvance", "ph":"E", "ts":"2196529209", "pid":"1", "tid":"cice"},
{"name":"ModelFinalize", "ph":"B", "ts":"2196531389", "pid":"1", "tid":"cice"},
{"name":"ModelFinalize", "ph":"E", "ts":"2196531441", "pid":"1", "tid":"cice"},

Then, can open with Perfetto UI online tool (https://ui.perfetto.dev/) and navigate wasd

NickSzapiro-NOAA and others added 17 commits September 27, 2024 16:20

No ncat,ntrcr in call icepack_aggregate in ice_prescribed_mod.F90

f124914

Update to CICE-Consortium/main (2024-09-28)

5e57a89

Merge remote-tracking branch 'CICE-Consortium/main' into develop

cf01f33

Sync CICE to Consortium/main (2024-11)

513f8d1

Enable writing restarts that can be triggered at any forecast time ("…

ff7fd76

…restart_fh") via UFS configuration as for other components Part of ufs-community/ufs-weather-model#2348

Merge 'CICE-Consortium/main' into EMC/develop

ea4582b

Sync to CICE-Consortium/main (2025-01-23)

4411a15

Add ufs_tracing

dce7176

Merge main from CICE-Consortium

473f258

Merge CICE-Consortium/main

feb4c7b

Merge CICE-Consortium/main (2025-08-06)

90ed252

Merge remote-tracking branch 'origin/develop' into ufs_tracing

31921b1

Conditional compilation of ufs_tracing

01024e6

Sync to CICE-Consortium 2025-11

5278e39

Sync to CICE-Consortium (2025-11-20)

c90ee5a

Merge remote-tracking branch 'origin/develop' into ufs_tracing

f888f31

Merge ufs_tracing from DusanJovic-NOAA

d69a94d

apcraig requested review from DeniseWorthen, anton-seaice and dabail10 December 16, 2025 20:45

apcraig added the Drivers/coupling label Dec 16, 2025

anton-seaice reviewed Dec 16, 2025

View reviewed changes

dabail10 reviewed Dec 16, 2025

View reviewed changes

NickSzapiro-NOAA changed the title ~~Add UFS tracing instrumentation to nuopc driver~~ Add UFS tracing instrumentation to nuopc/cmeps driver Dec 17, 2025

	call ESMF_GridCompGet(gcomp, vm=vm, rc=rc)
	if (ChkErr(rc,__LINE__,u_FILE_u)) return
	call ESMF_VMGet(vm, mpiCommunicator=lmpicom, localPet=localPet, PetCount=npes, rc=rc)
	if (ChkErr(rc,__LINE__,u_FILE_u)) return

	#ifdef CESMCOUPLED
	call ESMF_VMGet(vm, pet=localPet, peCount=nthrds, rc=rc)
	if (ChkErr(rc,__LINE__,u_FILE_u)) return
	if (nthrds==1) then
	call NUOPC_CompAttributeGet(gcomp, "nthreads", value=cvalue, rc=rc)
	if (ESMF_LogFoundError(rcToCheck=rc, msg=ESMF_LOGERR_PASSTHRU, line=__LINE__, file=u_FILE_u)) return
	read(cvalue,*) nthrds
	endif
	!$ call omp_set_num_threads(nthrds)
	#endif

	!----------------------------------------------------------------------------
	! Initialize cice communicators
	!----------------------------------------------------------------------------

	call init_communicate(lmpicom) ! initial setup for message passing
	mastertask = .false.
	if (my_task == master_task) mastertask = .true.

Conversation

NickSzapiro-NOAA commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR checklist

Uh oh!

NickSzapiro-NOAA commented Dec 16, 2025

Uh oh!

anton-seaice left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dabail10 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DeniseWorthen commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NickSzapiro-NOAA commented Dec 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

NickSzapiro-NOAA commented Dec 16, 2025 •

edited

Loading

DeniseWorthen commented Dec 16, 2025 •

edited

Loading