Issue with NMreadInits

[RichardHooijmaijers]

Hi,

I am looking into a way to automatically obtain info from model files for translation and found the NMreadInits function very useful!
However I just encountered a case where the function was not able to obtain omega values from a control stream.
I have included a reproducible example below. The control stream seems relatively straight forward so I am not sure what the issue might be:

library(NMdata)
chk <- readLines("https://raw.githubusercontent.com/dpastoor/ddmore_scraping/refs/heads/master/221/Executable_SLD_SUV_OS_GIST.mod")
NMdata::NMreadInits(lines=chk, section="omega")  

Any help on this is greatly appreciated!

[philipdelff]

Thank you for reaching out, and thank you for providing a clear and reproducible report.

NMreadInits() and NMwriteInits() were introduced in recent versions of NMdata, and I'm happy to hear you find them useful. There has been some work done to them since first introduced, and it seems like you need to update NMdata for it to work on your model. This is what I get with NMdata 0.2.3:

> unloadNamespace("NMdata")
> install.packages("NMdata",repos="https://cran.r-project.org")
Installing package into ‘x86_64-pc-linux-gnu-library/4.5’
(as ‘lib’ is unspecified)
trying URL 'https://cran.r-project.org/src/contrib/NMdata_0.2.3.tar.gz'
Content type 'application/x-gzip' length 5693970 bytes (5.4 MB)
==================================================
downloaded 5.4 MB

* installing *source* package ‘NMdata’ ...
** this is package ‘NMdata’ version ‘0.2.3’
** package ‘NMdata’ successfully unpacked and MD5 sums checked
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (NMdata)

The downloaded source packages are in
	‘/tmp/RtmpIeDWvW/downloaded_packages’
> library(NMdata)
NMdata 0.2.3. Browse NMdata documentation at
https://NMautoverse.github.io/NMdata/
> chk <- readLines("https://raw.githubusercontent.com/dpastoor/ddmore_scraping/refs/heads/master/221/Executable_SLD_SUV_OS_GIST.mod")
> inits <- NMdata::NMreadInits(lines=chk, section="omega")
> head(inits)
   parameter   par.name par.type i j iblock blocksize init
1 OMEGA(1,1) OMEGA(1,1)    OMEGA 1 1      1         1    0
2 OMEGA(2,2) OMEGA(2,2)    OMEGA 2 2      2         1    0
3 OMEGA(3,3) OMEGA(3,3)    OMEGA 3 3      3         1    0
4 OMEGA(4,4) OMEGA(4,4)    OMEGA 4 4      4         1    0
5 OMEGA(5,5) OMEGA(5,5)    OMEGA 5 5      5         1    0
6 OMEGA(6,6) OMEGA(6,6)    OMEGA 6 6      6         1    0
  lower upper FIX SAME sameblock Nsameblock
1  <NA>  <NA>   1    0         0          0
2  <NA>  <NA>   1    0         0          0
3  <NA>  <NA>   1    0         0          0
4  <NA>  <NA>   1    0         0          0
5  <NA>  <NA>   1    0         0          0
6  <NA>  <NA>   1    0         0          0

Does this resolve it for you?

[RichardHooijmaijers]

Thanks for the quick reply! This indeed solved the problem (sorry for wasting your time, I realize I should have tried this first).

Not sure if it is helpful but I am using this repository for testing. You could use it to test some of your package functions as well. For instance, you can download the repo and run the following:

allfl  <- list.dirs("ddmore_scraping-master/ddmore_scraping-master",recursive = FALSE)
allmod <- list.files("ddmore_scraping-master/ddmore_scraping-master", pattern = "\\.mod$|\\.mdl", full.names = TRUE, recursive = TRUE)
for(i in allmod){
  chk <- try(NMdata::NMreadInits(i), silent=TRUE)
  if(inherits(chk,"try-error")){
    cat("could not read",basename(i),"\n")
  }else if(inherits(chk,"data.frame")){
    cat("correctly read",basename(i),"\n")
  }
}

I did not look in detail but it seems there might be some things to look into..

[RichardHooijmaijers]

Maybe one more addition; It seems the function fails in case there are non-ASCII characters in a file. This can be easily overcome using (line 3 - iconv):

library(NMdata)
chk <- readLines("https://github.com/dpastoor/ddmore_scraping/raw/refs/heads/master/228/Executable_run126h.mod")
chk <- iconv(chk, "latin1", "ASCII", sub="")
NMdata::NMreadInits(lines=chk, section="omega")  

[philipdelff]

Thank you so much @RichardHooijmaijers !

I added the conversion you suggested to a function being used to read control streams. This ensures support for control streams with non-ascii characters but it unfortunately drops the non-ascii characters that can't be converted, in this case the Greek mu character.
It would be better to ensure support for non-ascii in the different text manipulation steps but that would take some more effort and will have to be on the todo for now. The fix is pushed to the main branch of NMdata. If you need it on CRAN, let me know and I can put a release together.

I love the ddmore scraping repo by Devin! I wasn't aware of it so thank you for sharing! The .mdl files do not seem to be Nonmem control streams which is probably why you are getting several errors when testing on all of those. When I run tests on just the .mod files it now runs on all of them. Before adding the conversion to ascii, two models were failing, and they both contain non-ascii characters.

allmod <- list.files("~/wdirs/ddmore_scraping", pattern = "\\.mod$", full.names = TRUE, recursive = TRUE)
dt <- data.table(mod=allmod)[,n := .I]
dt[,ext := fnExtension(mod)]

dt[,(c("status","msg","nrow","ncol")) := {
    chk <- try(NMdata::NMreadInits(mod), silent=TRUE)
    if(inherits(chk,"try-error")){
        list("Error",chk,0,0)
    }else if(inherits(chk,"data.frame")){
        list("OK","",nrow(chk),ncol(chk))
    }
},by=n]

dt[,.N,keyby=.(ext,status)]
dt[ext=="mod",lapply(.SD,range),.SDcols=c("nrow","ncol")]
## all meaningful numbers of parameters and columns
dt[ext=="mod",.N,keyby=.(nrow,ncol)]

I also ran this on all the .lst files in the repo. This gave some errors but looking into the first several of those files that failed, they are not real output control streams. Some of them seem like they are snips of outputs, some are ddmore models (like .mdl), some output tables.

I'm glad you are finding these functions useful. I want to mention some of the downstream functionality based on this.

Combine inital values with parameter estimates and parameter annotations in one data frame:

file.mod <- system.file("examples/nonmem/xgxr132.mod",package="NMdata")
NMreadParsText(file.mod,format="%idx:%symbol;%label[%unit];%trans",
               format.sigma="%label") |>
    mergeCheck(NMreadInits(file=file.mod),by="parameter",common.cols="drop.y",all.x=TRUE)|>
    mergeCheck(NMreadExt(file=file.mod),by="parameter",common.cols="drop.x",all.x=TRUE)

The NMwork package has functions to present this in parameter tables: NMwork - Create Nonmem Parameter Tables:

NMsim::NMsim uses it to run simulations with modified parameter values, like

Simulation with Modifications to Parameters and Model Code:

simres.inits1 <- NMsim(file.mod=file.mod,
                       data=data.sim,
                       inits=list("theta(1)"=list(init=0.54)
                                 ,"omega(3,3)"=list(init=.5)
                                  ),
                       name.sim="inits1")

You can also use that to modify the initial parameter values for estimation.

[philipdelff]

just keeping it open in case you have follow up questions, @RichardHooijmaijers

[RichardHooijmaijers]

Hi @philipdelff Thanks for the detailed descriptions and additional examples. Glad I could be of some help. I really like the various packages you created and I think they are a great addition to the pharmacometrician's toolbox :)

I am working on some packages myself that may have some overlap with your packages but may also be complementary or have a nice synergy. If you are interested, we could get in contact and see if we can work together on some things?

For now no follow up questions, so OK for me to close off the issue.

[philipdelff]

Great @RichardHooijmaijers ,

Can you find me on LinkedIn? If er connect there, you can see my email. Happy to talk about synergy and how we can collaborate.

Best,
Philip