Running the testsuite on Windows

[rruusu]

I have some issues running the test suite on a Windows computer.

I have followed the instructions in the README.md:

• Pulled the OMDev repository.
• Turned on test suite in the CMakeLists.txt.
  • option(OM_OMS_ENABLE_TESTSUITE "enable the OMSimulator testsuite" ON)
• Built OMSimulator to an initially empty build directory:
  • cmake -S . -B build/ -DCMAKE_INSTALL_PREFIX=install/
  • cmake --build build/ --target install
  • cmake --build build/ --target testsuite-depends
• Run the /testsuite/partest/runtests.pl script in OMDev MinGW bash shell.

But all I get from the script is:

# ./runtests.pl
16 threads
 [activateVariant3.lua:0] [activateVariant1.lua:0] [exportSSVTemplate3.lua:0] [exportSSVTemplate2.py:0] [activateVariant2.lua:0] [deleteStartValuesInSSV.lua:0] [exportSSVTemplate1.lua:0] [deleteConnector.lua:0] [deleteRootSystem.lua:0] [deleteReferencesAndStartValues.lua:0] [setUnits1.lua:0] [checkUnits.lua:0] [test03.lua:0] [import_export_units2.lua:0] [deleteStartValuesInSSV1.lua:0] [import_export_units1.lua:0] [deleteStartValues.lua:0] [deleteConnector1.lua:0] [discreteConnection.lua:0] [AircraftVehicleDemonstrator.lua:0] [importExportAllResources.lua:0] [importStartValues.py:0] [embrace.py:0] [getDirectionalDerivative_ME.lua:0] [fmidertest.lua:0] [reduceSSV.py:0] [reduceSSV.lua:0] [setUnits2.lua:0] [duplicateVariant1.lua:0] [getDirectionalDerivative_ME.py:0] [getDirectionalDerivative_CS.py:0] [deleteResourcesInSSP2.lua:0] [test_omsExport.py:0] [test01.py:0] [deleteStartValuesInSSV2.lua:0] [DualMassOscillator.lua:0] [test02.py:0] [test03.py:0] [embrace.lua:0] [embrace_test2Conf.py:0] [getDirectionalDerivative_CS.lua:0] [exportSSMTemplate.lua:0] [duplicateActivateVariant1.lua:0] [test_omsExport.lua:0] [setUnits3.py:0] [test02.lua:0] [equationPair_fixedpoint.py:0] [equationPair.py:0] [test01.lua:0] [fmuState01.lua:0] [import_export_parameters1.lua:0] [deleteReferencesInSSD2.lua:0] [deleteReferencesInSSD1.lua:0] [exportSnapshotSSP.lua:0] [exportConnectorsToResultFile.lua:0] [exportSSMTemplate.py:0] [import_export_parameters_to_ssv.lua:0] [duplicateActivateVariant2.py:0] [duplicateVariant4.py:0] [fmuState02.py:0] [duplicateVariant3.lua:0] [EventTest.lua:0] [deleteResourcesInSSP1.lua:0] [Enumeration.lua:0] [buses.py:0] [import_export_signalFilter.lua:0] [Enumeration2.lua:0] [import_export_parameters_inline.lua:0] [connections.py:0] [duplicateVariant2.lua:0] [import_export_parameters4.lua:0] [addExternalResources4.py:0] [buses.lua:0] [connections.lua:0] [addExternalResources1.lua:0] [replaceSubmodel1.lua:0] [import_export_parameters2.lua:0] [addExternalResources5.lua:0] [addExternalResources2.lua:0] [renameValues2.lua:0] [replaceSubmodel10.lua:0] [import_export_parameters3.lua:0] [import_hierarchical_ssv_sources1.lua:0] [replaceSubmodel9.lua:0] [replaceSubmodel14.py:0] [replaceSubmodel2.lua:0] [replaceSubmodel3.lua:0] [replaceSubmodel8.lua:0] [renameValues1.lua:0] [importSuppressUnitConversion.lua:0] [addExternalResources3.py:0] [referenceResources1.lua:0] [rename.lua:0] [QuarterCarModel.DisplacementDisplacement.lua:0] [PI_Controller.lua:0] [import_export_snapshot.lua:0] [import_export_signalFilter2.lua:0] [simulation.lua:0] [replaceSubmodel7.py:0] [partialSnapshot.lua:0] [renameModel.lua:0] [referenceResources2.lua:0] [renameSnapshot.lua:0] [renameNewCref_1.lua:0] [importStartValues.lua:0] [replaceSubmodel12.py:0] [replaceSubmodel6.py:0] [replaceSubmodel4.lua:0] [replaceSubmodel5.lua:0] [nls.py:0] [importExportEnumerationDefinition.lua:0] [import_parameter_mapping_from_ssm.lua:0] [str_hello_world.lua:0] [replaceSubmodel13.py:0] [import_parameter_mapping_inline.lua:0] [renameNewCref_2.lua:0] [importPartialSnapshot.lua:0] [replaceSubmodel11.py:0] [importStartValues1.lua:0] [listVariants1.lua:0] [simulation.py:0] [renameNewCref.lua:0] [import_hierarchical_ssv_sources.lua:0] [BouncingBall.lua:0] [Feedthrough.lua:0] [multipleConnections.lua:0] [multipleInstance.lua:0] [ASSCExample.lua:0] [validateModeldescription.py:0] [suppressUnitConversion.lua:0] [tankYPipe.lua:0] [validateSRMD.lua:0] [snapshot.py:0] [snapshot.lua:0] [table.lua:0] [Resource.lua:0] [Dahlquist.lua:0] [validateSRMD.py:0] [validateSSM.py:0] [str_hello_world.py:0] [validateSSV.py:0] [Stair.lua:0] [VanDerPol.lua:0] [validateSSD.py:0] [setExternalInputs.lua:0]{joined thread: 1}{joined thread: 2}{joined thread: 3}{joined thread: 4}{joined thread: 5}{joined thread: 6}{joined thread: 7}{joined thread: 8}{joined thread: 9}{joined thread: 10}{joined thread: 11}{joined thread: 12}{joined thread: 13}{joined thread: 14}{joined thread: 15}{joined thread: 16}

Failed tests:
        ./AircraftVehicleDemonstrator/AircraftVehicleDemonstrator.lua
        ./AircraftVehicleDemonstrator/embrace.lua
        ./AircraftVehicleDemonstrator/embrace.py
        ./AircraftVehicleDemonstrator/embrace_test2Conf.py
        ./api/addExternalResources1.lua
        ./api/addExternalResources2.lua
        ./api/addExternalResources3.py
        ./api/addExternalResources4.py
        ./api/addExternalResources5.lua
        ./api/buses.lua
        ./api/buses.py
        ./api/connections.lua
        ./api/connections.py
        ./api/deleteReferencesInSSD1.lua
        ./api/deleteReferencesInSSD2.lua
        ./api/deleteResourcesInSSP1.lua
        ./api/deleteResourcesInSSP2.lua
        ./api/fmidertest.lua
        ./api/getDirectionalDerivative_CS.lua
        ./api/getDirectionalDerivative_CS.py
        ./api/getDirectionalDerivative_ME.lua
        ./api/getDirectionalDerivative_ME.py
        ./api/importExportAllResources.lua
        ./api/import_export_units1.lua
        ./api/import_export_units2.lua
        ./api/reduceSSV.lua
        ./api/reduceSSV.py
        ./api/setUnits1.lua
        ./api/setUnits2.lua
        ./api/setUnits3.py
        ./api/test01.lua
        ./api/test01.py
        ./api/test02.lua
        ./api/test02.py
        ./api/test03.lua
        ./api/test03.py
        ./api/test_omsExport.lua
        ./api/test_omsExport.py
        ./assc/ASSCExample.lua
        ./export/exportSSVTemplate1.lua
        ./export/exportSSVTemplate2.py
        ./export/exportSSVTemplate3.lua
        ./import/importStartValues.py
        ./reference-fmus/BouncingBall.lua
        ./reference-fmus/Dahlquist.lua
        ./reference-fmus/Feedthrough.lua
        ./reference-fmus/Resource.lua
        ./reference-fmus/Stair.lua
        ./reference-fmus/VanDerPol.lua
        ./simulation/DualMassOscillator.lua
        ./simulation/Enumeration.lua
        ./simulation/Enumeration2.lua
        ./simulation/EventTest.lua
        ./simulation/PI_Controller.lua
        ./simulation/QuarterCarModel.DisplacementDisplacement.lua
        ./simulation/activateVariant1.lua
        ./simulation/activateVariant2.lua
        ./simulation/activateVariant3.lua
        ./simulation/checkUnits.lua
        ./simulation/deleteConnector.lua
        ./simulation/deleteConnector1.lua
        ./simulation/deleteReferencesAndStartValues.lua
        ./simulation/deleteRootSystem.lua
        ./simulation/deleteStartValues.lua
        ./simulation/deleteStartValuesInSSV.lua
        ./simulation/deleteStartValuesInSSV1.lua
        ./simulation/deleteStartValuesInSSV2.lua
        ./simulation/discreteConnection.lua
        ./simulation/duplicateActivateVariant1.lua
        ./simulation/duplicateActivateVariant2.py
        ./simulation/duplicateVariant1.lua
        ./simulation/duplicateVariant2.lua
        ./simulation/duplicateVariant3.lua
        ./simulation/duplicateVariant4.py
        ./simulation/equationPair.py
        ./simulation/equationPair_fixedpoint.py
        ./simulation/exportConnectorsToResultFile.lua
        ./simulation/exportSSMTemplate.lua
        ./simulation/exportSSMTemplate.py
        ./simulation/exportSnapshotSSP.lua
        ./simulation/fmuState01.lua
        ./simulation/fmuState02.py
        ./simulation/importExportEnumerationDefinition.lua
        ./simulation/importPartialSnapshot.lua
        ./simulation/importStartValues.lua
        ./simulation/importStartValues1.lua
        ./simulation/importSuppressUnitConversion.lua
        ./simulation/import_export_parameters1.lua
        ./simulation/import_export_parameters2.lua
        ./simulation/import_export_parameters3.lua
        ./simulation/import_export_parameters4.lua
        ./simulation/import_export_parameters_inline.lua
        ./simulation/import_export_parameters_to_ssv.lua
        ./simulation/import_export_signalFilter.lua
        ./simulation/import_export_signalFilter2.lua
        ./simulation/import_export_snapshot.lua
        ./simulation/import_hierarchical_ssv_sources.lua
        ./simulation/import_hierarchical_ssv_sources1.lua
        ./simulation/import_parameter_mapping_from_ssm.lua
        ./simulation/import_parameter_mapping_inline.lua
        ./simulation/listVariants1.lua
        ./simulation/multipleConnections.lua
        ./simulation/multipleInstance.lua
        ./simulation/nls.py
        ./simulation/partialSnapshot.lua
        ./simulation/referenceResources1.lua
        ./simulation/referenceResources2.lua
        ./simulation/rename.lua
        ./simulation/renameModel.lua
        ./simulation/renameNewCref.lua
        ./simulation/renameNewCref_1.lua
        ./simulation/renameNewCref_2.lua
        ./simulation/renameSnapshot.lua
        ./simulation/renameValues1.lua
        ./simulation/renameValues2.lua
        ./simulation/replaceSubmodel1.lua
        ./simulation/replaceSubmodel10.lua
        ./simulation/replaceSubmodel11.py
        ./simulation/replaceSubmodel12.py
        ./simulation/replaceSubmodel13.py
        ./simulation/replaceSubmodel14.py
        ./simulation/replaceSubmodel2.lua
        ./simulation/replaceSubmodel3.lua
        ./simulation/replaceSubmodel4.lua
        ./simulation/replaceSubmodel5.lua
        ./simulation/replaceSubmodel6.py
        ./simulation/replaceSubmodel7.py
        ./simulation/replaceSubmodel8.lua
        ./simulation/replaceSubmodel9.lua
        ./simulation/setExternalInputs.lua
        ./simulation/simulation.lua
        ./simulation/simulation.py
        ./simulation/snapshot.lua
        ./simulation/snapshot.py
        ./simulation/str_hello_world.lua
        ./simulation/str_hello_world.py
        ./simulation/suppressUnitConversion.lua
        ./simulation/table.lua
        ./simulation/tankYPipe.lua
        ./validate/validateModeldescription.py
        ./validate/validateSRMD.lua
        ./validate/validateSRMD.py
        ./validate/validateSSD.py
        ./validate/validateSSM.py
        ./validate/validateSSV.py

Slowest test 1.532s ./simulation/replaceSubmodel11.py
145 of 145 failed

[rruusu]

Setting the baseline for an individual test using the /testsuite/rtest script just produces the following, which probably explains the results:

# ../rtest -b equationpair.py
We do not support mingw64 anymore, only ucrt64

What should I do instead?

[rruusu]

After using the correct OMDev repository and fixing the error in the CMakeLists.txt mentioned below, I got the test cases to run, but I am getting several failures from test cases in the maintenance/v2.1 branch:

Failed tests:
        ./api/addExternalResources1.lua
        ./api/addExternalResources2.lua
        ./api/addExternalResources5.lua
        ./api/buses.lua
        ./api/connections.lua
        ./api/deleteReferencesInSSD1.lua
        ./api/deleteResourcesInSSP1.lua
        ./api/importExportAllResources.lua
        ./api/import_export_units1.lua
        ./api/import_export_units2.lua
        ./api/reduceSSV.lua
        ./api/setUnits1.lua
        ./api/setUnits2.lua
        ./api/test01.lua
        ./api/test03.lua
        ./api/test_omsExport.lua
        ./export/exportSSVTemplate1.lua
        ./export/exportSSVTemplate3.lua
        ./simulation/activateVariant1.lua
        ./simulation/activateVariant2.lua
        ./simulation/activateVariant3.lua
        ./simulation/deleteConnector.lua
        ./simulation/deleteConnector1.lua
        ./simulation/deleteReferencesAndStartValues.lua
        ./simulation/deleteRootSystem.lua
        ./simulation/deleteStartValues.lua
        ./simulation/deleteStartValuesInSSV.lua
        ./simulation/deleteStartValuesInSSV1.lua
        ./simulation/deleteStartValuesInSSV2.lua
        ./simulation/discreteConnection.lua
        ./simulation/duplicateActivateVariant1.lua
        ./simulation/duplicateVariant1.lua
        ./simulation/duplicateVariant2.lua
        ./simulation/duplicateVariant3.lua
        ./simulation/exportConnectorsToResultFile.lua
        ./simulation/exportSSMTemplate.lua
        ./simulation/exportSnapshotSSP.lua
        ./simulation/importExportEnumerationDefinition.lua
        ./simulation/importPartialSnapshot.lua
        ./simulation/importStartValues.lua
        ./simulation/importStartValues1.lua
        ./simulation/importSuppressUnitConversion.lua
        ./simulation/import_export_parameters1.lua
        ./simulation/import_export_parameters2.lua
        ./simulation/import_export_parameters3.lua
        ./simulation/import_export_parameters4.lua
        ./simulation/import_export_parameters_inline.lua
        ./simulation/import_export_parameters_to_ssv.lua
        ./simulation/import_export_signalFilter.lua
        ./simulation/import_export_signalFilter2.lua
        ./simulation/import_export_snapshot.lua
        ./simulation/import_hierarchical_ssv_sources1.lua
        ./simulation/import_parameter_mapping_from_ssm.lua
        ./simulation/import_parameter_mapping_inline.lua
        ./simulation/listVariants1.lua
        ./simulation/multipleInstance.lua
        ./simulation/partialSnapshot.lua
        ./simulation/referenceResources1.lua
        ./simulation/referenceResources2.lua
        ./simulation/rename.lua
        ./simulation/renameModel.lua
        ./simulation/renameNewCref_2.lua
        ./simulation/renameSnapshot.lua
        ./simulation/renameValues1.lua
        ./simulation/renameValues2.lua
        ./simulation/replaceSubmodel1.lua
        ./simulation/replaceSubmodel10.lua
        ./simulation/replaceSubmodel2.lua
        ./simulation/replaceSubmodel3.lua
        ./simulation/replaceSubmodel4.lua
        ./simulation/replaceSubmodel5.lua
        ./simulation/replaceSubmodel8.lua
        ./simulation/replaceSubmodel9.lua
        ./simulation/setExternalInputs.lua
        ./simulation/snapshot.lua
        ./simulation/suppressUnitConversion.lua
        ./validate/validateSRMD.lua

77 of 145 failed

[rruusu]

Most of them look like unexpected additions of empty lines, like this:

Equation mismatch: diff says:
--- /tmp/oms-rtest-rruusu/simulationtmp_4965/equations-expectedtmp_expected-import_export_parameters4.lua	2025-10-23 14:06:16.395835300 +0300
+++ /tmp/oms-rtest-rruusu/simulationtmp_4965/equations-got_tmp_got-import_export_parameters4.lua	2025-10-23 14:06:16.703932800 +0300
@@ -259,10 +259,11 @@
 type="Real"
 kind="parameter" />
 </oms:SignalFilter>
 </oms:file>
 </oms:snapshot>
+
 info:      Virgin Parameter settings
 info:      import_export_parameters.co_sim.addP.k1     : 60.0
 info:      import_export_parameters.co_sim.addP.k2     : 150.0
 info:      import_export_parameters.co_sim.addP.u1     : 0.0
 info:      Virgin-Top-System-Level-Vars

Equation mismatch: omc-diff says:
Line 264: Lexical token differs:
expected: info:      Virgin Parameter settings
got:      

Some of them also have floating point representation differences:

Equation mismatch: diff says:
--- /tmp/oms-rtest-rruusu/simulationtmp_8749/equations-expectedtmp_expected-import_export_parameters3.lua	2025-10-23 14:06:16.388639500 +0300
+++ /tmp/oms-rtest-rruusu/simulationtmp_8749/equations-got_tmp_got-import_export_parameters3.lua	2025-10-23 14:06:16.830556100 +0300
@@ -106,15 +106,15 @@
 type="org.openmodelica">
 <oms:Annotations>
 <oms:SimulationInformation>
 <oms:VariableStepSolver
 description="cvode"
-absoluteTolerance="0.000100"
-relativeTolerance="0.000100"
-minimumStepSize="0.000100"
-maximumStepSize="0.100000"
-initialStepSize="0.000100" />
+absoluteTolerance="0.0001"
+relativeTolerance="0.0001"
+minimumStepSize="0.0001"
+maximumStepSize="0.1"
+initialStepSize="0.0001" />
 </oms:SimulationInformation>
 </oms:Annotations>
 </ssc:Annotation>
 </ssd:Annotations>
 </ssd:System>
@@ -426,10 +426,11 @@
 type="Real"
 kind="parameter" />
 </oms:SignalFilter>
 </oms:file>
 </oms:snapshot>
+
 info:      Virgin Parameter settings
 info:      import_export_parameters.co_sim.addP.k1     : -60.0
 info:      import_export_parameters.co_sim.addP.k2     : -1.0
 info:      import_export_parameters.co_sim.addP.u1     : 300.0
 info:      import_export_parameters.co_sim.addI.k2     : -150.0

Equation mismatch: omc-diff says:
Line 431: Lexical token differs:
expected: info:      Virgin Parameter settings
got:      

[arun3688]

@rruusu those new line differences and floating point differences are fine, you can update the baseline with new output

[rruusu]

Are you sure that is ok? They didn't seem to fail in Jenkins.

[rruusu]

I found the issue two weeks ago, but forgot to report it here.

Those seemingly empty lines and non-significant zeroes on Windows were caused by the fact that I had Git by default configured to perform automatic line ending conversion to CRLF in the local working tree, while the test suite comparison was expecting to process only files with LF line ending.

There actually weren't any additional empty lines. This was just the way the comparison treated an empty line with CR+LF ending instead of just LF.

To make the test suite work on Windows, I had to switch to using LF line endings in the Git configuration, and update the working tree;

.git/config:

core.autocrlf=false
core.eol=lf

[rruusu]

I seem to have also overlooked an error in building the testsuite-depends target:

make[3]: *** No rule to make target 'D:/Git/OMSimulator/testsuite/resources/invalidSSP/SystemStructure.ssd', needed by 'D:/Git/OMSimulator/testsuite/resources/invalidSSP.ssp'.  Stop.
make[2]: *** [CMakeFiles/Makefile2:1973: testsuite/resources/CMakeFiles/testsuite-resources.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:1928: testsuite/CMakeFiles/testsuite-depends.dir/rule] Error 2
make: *** [Makefile:637: testsuite-depends] Error 2

There is no directory named /testsuite/resources/invalidSSP.

Looking at Git history, the testsuite/resources/CMakeLists.txt file seems to have been fixed in the master branch, but not in maintenance/v2.1, which I had checked out.

I will try with the testsuite from the mater branch, but this is likely not related to the "We do not support mingw64 anymore, only ucrt64" message above.

[arun3688]

@rruusu you can use Makefile in testsuite directory and not the cmake for maintenance branch, just run
>> make resources

[rruusu]

The makefile did seem to build all the resources.

Anyway, I discovered at least the following changes that were necessary for testsuite/resources/CMakeLists.txt in the maintenance/v2.1 branch:

--- a/testsuite/resources/CMakeLists.txt
+++ b/testsuite/resources/CMakeLists.txt
@@ -13,6 +13,7 @@ set(OM_OMS_TESTSUITE_RESOURCES_FMUS
         ${CMAKE_CURRENT_SOURCE_DIR}/ASSCExample.Class2
         ${CMAKE_CURRENT_SOURCE_DIR}/B
         ${CMAKE_CURRENT_SOURCE_DIR}/BouncingBall
+        ${CMAKE_CURRENT_SOURCE_DIR}/DiscreteTest
         ${CMAKE_CURRENT_SOURCE_DIR}/Dahlquist
         ${CMAKE_CURRENT_SOURCE_DIR}/DualMassOscillator.System1
         ${CMAKE_CURRENT_SOURCE_DIR}/DualMassOscillator.System2
@@ -53,6 +54,7 @@ set(OM_OMS_TESTSUITE_RESOURCES_FMUS
         ${CMAKE_CURRENT_SOURCE_DIR}/tank1
         ${CMAKE_CURRENT_SOURCE_DIR}/tank2
         ${CMAKE_CURRENT_SOURCE_DIR}/tank3
+        ${CMAKE_CURRENT_SOURCE_DIR}/testArray
         ${CMAKE_CURRENT_SOURCE_DIR}/VanDerPol
         ${CMAKE_CURRENT_SOURCE_DIR}/ypipe
 )
@@ -69,7 +71,10 @@ set(OM_OMS_TESTSUITE_RESOURCES_SSPS
         ${CMAKE_CURRENT_SOURCE_DIR}/import_export_parameters2
         ${CMAKE_CURRENT_SOURCE_DIR}/import_export_parameters3
         ${CMAKE_CURRENT_SOURCE_DIR}/importExportAllResources
-        ${CMAKE_CURRENT_SOURCE_DIR}/invalidSSP
+        ${CMAKE_CURRENT_SOURCE_DIR}/invalidSRMD
+        ${CMAKE_CURRENT_SOURCE_DIR}/invalidSSD
+        ${CMAKE_CURRENT_SOURCE_DIR}/invalidSSM
+        ${CMAKE_CURRENT_SOURCE_DIR}/invalidSSV
         ${CMAKE_CURRENT_SOURCE_DIR}/replaceSubmodel4
         ${CMAKE_CURRENT_SOURCE_DIR}/replaceSubmodel5
 )

One interesting thing that I noticed was that /testsuite/validate/validateSRMD.lua failed due to the missing resource /testsuite/resources/invalidSRMD.ssp, but /testsuite/validate/validateSRMD.py did not, even though it was dependent on the very same missing resource.

[arun3688]

@rruusu yes that is correct, actually we are switching to OMSimulator-3.0 and that is work in progress in master branch and we are not much focussed to fix the maintenance branch, in the new oms3, the lua support will be removed and we support only python, but i see the changes you made are on the c code base which is still essential for OMSimulator-3.0, So we also need to port your changes back to master at some point once we merged with maintenance branch

[rruusu]

Thanks for the info.

I had so much trouble trying to merge my changes to the event-related refactoring of the CVODE-based SystemSC code a year ago, with not enough hours then for the effort, that I just decided to continue on a version from last October. And then we decided to make the manual rebasing effort first on the maintenance branch, assuming that you will probably anyway do big rewrites on the master branch for 3.0.

Let’s hope that someone has time to take at least some of the changes into the master branch.