Skip to content

fix for support for eta constraints in spec file#152

Open
zurabksmlp wants to merge 6 commits into
masterfrom
eta_spec_handling_fix
Open

fix for support for eta constraints in spec file#152
zurabksmlp wants to merge 6 commits into
masterfrom
eta_spec_handling_fix

Conversation

@zurabksmlp

Copy link
Copy Markdown
Collaborator

Fixed handing of eta constraints defined in spec file. Eta constraints in command line worked as expected.
Changes are inn files:
src/smlp_py/smlp_flows.py
src/smlp_py/smlp_spec.py
The change is relevant to the following tests -- these test use spec file with an eta constraint:
-- Master files for tests 90-92,94,113-116 are not really affected in the sense that only the .txt log files changed but not the results
-- Master files tests 93,95,96,98,108,228 are affected -- results changed -- in particular, assertion counter examples and/or trace files -- and results are as expected. For test 108 some of the missing master files (not main ones) were also added.
-- Master files tests 142,146,165–177,188–195,199,200 -- these master files do not exist and we cannot compare.
-- Master files for test 123 -- input data is missing for this test and changes cannot be compared

Test 229 also refers to spec file with eta constraint but this eta constraint has no effect (this test is a sanity check).

@zurabksmlp zurabksmlp requested review from fbrausse and mdmitry1 May 31, 2026 15:31

@fbrausse fbrausse left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general looks good to me, except for typos and I have a question about the introduction of named parameters.

Comment thread src/smlp_py/smlp_spec.py Outdated
Comment thread src/smlp_py/smlp_flows.py Outdated
Comment thread src/smlp_py/smlp_flows.py Outdated

@mdmitry1 mdmitry1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some new diffs. Most of the due to missing print statement.
However, for several tests diffs are in the results

  1. New diffs report
    smlp_eta_spec_handling_fix_diffs.log

  2. All Results
    eta_spec_handling_fix_code.tar.gz

  3. Reproduction - wheel for Ubuntu 24.04
    wheel.tar.gz

  4. Got the same results with wheel built by GitHub actions

=== 1. Run 26743307966 [eta_spec_handling_fix]: Build ===
  attempt 3 [by: mdmitry1] [started: 2026-06-01 13:22:46 IDT] [finished: 2026-06-01 13:37:10 IDT]:
    smlptech-1.2.1rc11.dev5+g446399f54-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Note: the two first build attempts failed due to GitHub actions issue - failures are unrelated to this PR.

@mdmitry1

mdmitry1 commented Jun 3, 2026

Copy link
Copy Markdown
Collaborator

After the latest commit 972c4a4 there is a crash.

smlp -data "../data/smlp_toy_num_resp_mult.csv" -out_dir ./ -pref Test59 -mode verify -resp y2 -feat x,p1,p2 -model nn_keras -nnet_encoding nested -mrmr_pred 0 -plots f -pred_plots f -resp_plots f -seed 10 -log_time f -nn_keras_epochs 20 -nn_keras_seq_api f  -save_model_config f -spec ../specs/smlp_toy_num_resp_mult_y2_verify.spec -asrt_names asrt1 -asrt_exprs "2*y2>1" -solver_path ../../../external/mathsat-5.6.8-linux-x86_64-reentrant/bin/mathsat
smlp_logger - INFO - Model interface constraints are consistent
[ext ] note : child 2323537 exited with code 0
Traceback (most recent call last):
  File "/home/mdmitry/github/smlp_eta_spec_handling_fix/scripts/venv/smlp_package_venv/bin/smlp", line 8, in <module>
    sys.exit(main2())
             ^^^^^^^
  File "/home/mdmitry/github/smlp_eta_spec_handling_fix/scripts/venv/smlp_package_venv/lib/python3.11/site-packages/smlp/run_smlp.py", line 16, in main2
    main(sys.argv)
  File "/home/mdmitry/github/smlp_eta_spec_handling_fix/scripts/venv/smlp_package_venv/lib/python3.11/site-packages/smlp/run_smlp.py", line 13, in main
    smlpInst.smlp_flow()
  File "/home/mdmitry/github/smlp_eta_spec_handling_fix/scripts/venv/smlp_package_venv/lib/python3.11/site-packages/smlp/smlp_py/smlp_flows.py", line 354, in smlp_flow
    self.verifyInst.smlp_verify(syst_expr_dict, args.model, model, 
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mdmitry/github/smlp_eta_spec_handling_fix/scripts/venv/smlp_package_venv/lib/python3.11/site-packages/smlp/smlp_py/smlp_verify.py", line 125, in smlp_verify
    asrt_res = self.verify_asrt(
               ^^^^^^^^^^^^^^^^^
  File "/home/mdmitry/github/smlp_eta_spec_handling_fix/scripts/venv/smlp_package_venv/lib/python3.11/site-packages/smlp/smlp_py/smlp_verify.py", line 55, in verify_asrt
    self._verify_logger.info('Verifying assertion {} <-> {}'.format(str(asrt_name), str(asrt_expr)))
    ^^^^^^^^^^^^^^^^^^^
AttributeError: 'SmlpVerify' object has no attribute '_verify_logger'

Wheel for reproduction:

=== 1. Run 26808367610 [eta_spec_handling_fix]: Build ===
  attempt 1 [by: mdmitry1] [started: 2026-06-02 11:36:22 IDT] [finished: 2026-06-02 11:51:02 IDT]:
    smlptech-1.2.1rc11.dev6+g972c4a42b-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Reproduction

cd $(git rev-parse --show-toplevel)/scripts/venv
../github/download_artifact.sh smlptech-1.2.1rc11.dev6+g972c4a42b-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl 26808367610
run_dora -wheel ./smlptech-1.2.1rc11.dev6+g972c4a42b-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
source smlp_package_venv/bin/activate
cd smlp_package_venv/smlp/regr_smlp/code
smlp -data "../data/smlp_toy_num_resp_mult.csv" -out_dir ./ -pref Test59 -mode verify -resp y2 -feat x,p1,p2 -model nn_keras -nnet_encoding nested -mrmr_pred 0 -plots f -pred_plots f -resp_plots f -seed 10 -log_time f -nn_keras_epochs 20 -nn_keras_seq_api f  -save_model_config f -spec ../specs/smlp_toy_num_resp_mult_y2_verify.spec -asrt_names asrt1 -asrt_exprs "2*y2>1" -solver_path ../../../external/mathsat-5.6.8-linux-x86_64-reentrant/bin/mathsat |& tail -20
smlp_logger - INFO - Model interface constraints are consistent
[ext ] note : child 2329053 exited with code 0
Traceback (most recent call last):
  File "/home/mdmitry/github/smlp/scripts/venv/smlp_package_venv/bin/smlp", line 8, in <module>
    sys.exit(main2())
             ^^^^^^^
  File "/home/mdmitry/github/smlp/scripts/venv/smlp_package_venv/lib/python3.11/site-packages/smlp/run_smlp.py", line 16, in main2
    main(sys.argv)
  File "/home/mdmitry/github/smlp/scripts/venv/smlp_package_venv/lib/python3.11/site-packages/smlp/run_smlp.py", line 13, in main
    smlpInst.smlp_flow()
  File "/home/mdmitry/github/smlp/scripts/venv/smlp_package_venv/lib/python3.11/site-packages/smlp/smlp_py/smlp_flows.py", line 354, in smlp_flow
    self.verifyInst.smlp_verify(syst_expr_dict, args.model, model, 
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mdmitry/github/smlp/scripts/venv/smlp_package_venv/lib/python3.11/site-packages/smlp/smlp_py/smlp_verify.py", line 125, in smlp_verify
    asrt_res = self.verify_asrt(
               ^^^^^^^^^^^^^^^^^
  File "/home/mdmitry/github/smlp/scripts/venv/smlp_package_venv/lib/python3.11/site-packages/smlp/smlp_py/smlp_verify.py", line 55, in verify_asrt
    self._verify_logger.info('Verifying assertion {} <-> {}'.format(str(asrt_name), str(asrt_expr)))
    ^^^^^^^^^^^^^^^^^^^
AttributeError: 'SmlpVerify' object has no attribute '_verify_logger'

Below changes fix this crash

diff --git a/src/smlp_py/smlp_flows.py b/src/smlp_py/smlp_flows.py
index a1480b3..a0d2bc0 100644
--- a/src/smlp_py/smlp_flows.py
+++ b/src/smlp_py/smlp_flows.py
@@ -87,6 +87,7 @@ class SmlpFlows:
         self.specInst.set_logger(self.logger)
         self.frontierInst.set_logger(self.logger)
         self.optInst.set_logger(self.logger)
+        self.verifyInst.set_logger(self.logger)
         self.queryInst.set_logger(self.logger)
         self.refineInst.set_logger(self.logger)
         self.correlInst.set_logger(self.logger)
diff --git a/src/smlp_py/smlp_verify.py b/src/smlp_py/smlp_verify.py
index cba0990..205b633 100644
--- a/src/smlp_py/smlp_verify.py
+++ b/src/smlp_py/smlp_verify.py
@@ -63,11 +63,11 @@ class SmlpVerify:
         solver_instance.add(self._smlpTermsInst.smlp_not(asrt_form))
         res = solver_instance.check()
         
-        if self._modelTermsInst.solver_status_is_unsat(res): #isinstance(res, core.unsat):
+        if self._modelTermsInst.solver_status_unsat(res): #isinstance(res, core.unsat):
             status = 'UNSAT' if asrt_name == self._VACUITY_ASSERTION_NAME else 'PASS'
             self._verify_logger.info('Completed with result: {}'.format(status)) #UNSAT 'PASS'
             asrt_res_dict = {'status':'PASS', 'asrt':None, 'model':None}
-        elif self._modelTermsInst.solver_status_is_sat(res): #isinstance(res, core.sat):
+        elif self._modelTermsInst.solver_status_sat(res): #isinstance(res, core.sat):
             status = 'SAT' if asrt_name == self._VACUITY_ASSERTION_NAME else 'FAIL'
             self._verify_logger.info('Completed with result: {}'.format(status)) #SAT 'FAIL (SAT)'
             witness_vals_dict = self._smlpTermsInst.witness_term_to_const(self._modelTermsInst.get_solver_model(res),
@@ -76,7 +76,7 @@ class SmlpVerify:
             asrt_ce_val = eval(asrt_expr, {},  witness_vals_dict)
             assert not asrt_ce_val
             asrt_res_dict = {'status':'FAIL', 'asrt': asrt_ce_val, 'model':witness_vals_dict}
-        elif self._modelTermsInst.solver_status_is_unknwn(res): #isinstance(res, core.unknown):
+        elif self._modelTermsInst.solver_status_unknown(res): #isinstance(res, core.unknown):
             self._verify_logger.info('Completed with result: {}'.format('UNKNOWN'))
             # TODO !!!: add reason for UNKNOWN or report that reason as 'status' field
             asrt_res_dict = {'status':'UNKNOWN', 'asrt':None, 'model':None}

However, now one of the output files is missing: Test59_smlp_toy_num_resp_mult_verify_results.json
Instead, there is a file: Test59_smlp_toy_num_resp_mult_assertions_results.json.
Another problem is that results are substantially different:

diff Test59_smlp_toy_num_resp_mult.txt ../master/
15,16d14
< smlp_logger - INFO - Global eta   : None
< 
291c289,290
< smlp_logger - INFO - Verifying assertion consistency_check <-> False
---
> smlp_logger - INFO - Verifying consistency of configuration for assertion asrt1:
>    true
293c292
< smlp_logger - INFO - Completed with result: SAT
---
> smlp_logger - INFO - Input, knob and configuration constraints are consistent
295a295,296
> 
> smlp_logger - INFO - The configuration is consistent with assertion asrt1

@zurabksmlp, please review

@mdmitry1 mdmitry1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed diffs.
Commit: 48cdadc
Build 76 smlptech-1.2.1rc11.dev7+g8bc18a240-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

@mdmitry1 mdmitry1 requested a review from fbrausse June 3, 2026 10:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants