Test Suite Release v4.0.0

[louaybassbouss]

• 1. ⏳ Update to version of new release v4.0.0
• 2. ⏳ Use new OF (@yanj-github use tag v4.0.0)
• 3. ⏳ Use new TR (@FritzHeiden use tag v4.0.0)
• 4. ⏳ Use new Test Content Repo (@rbouqueau use tag v4.0.0)
• 5.⏳ Update dpctf-tests Repo to include the new list of validated/beta tests and create new tag v4.0.0
• 6. ⏳Once previous steps are done, dpctf-deploy#staging will be updated to use the new dpctf-tests tag (v4.0.0)
• 7. ⏳ Once step 6 is done, dpctf-deploy#staging will be ready for testing (and potentially fix bugs if necessary):
  • @gitwjr: on Windows@WSL2
  • FOKUS: on Ubuntu
  • FOKUS: on Mac
  • FOKUS: on Windows@WSL2
  • Other volunteers are welcome
• 8. ⏳ Once step 7 is approved:
  • merge dpctf-deploy#staging into dpctf-deploy#master
  • create new GitHub tag v4.0.0 in dpctf-deploy#master
  • create new dpctf-deploy release (for tag dpctf-deploy#master@v4.0.0) with release notes. The release notes should include a human readable list of validated/beta tests from the new PR.
  • ⚠️ After creating the new release, do not touch master branch and work only in staging for next/future releases.
• 9. ⏳Once step 8 is completed, ask TWG for review and approval. Use new release branch.

[louaybassbouss]

@gitwjr @jpiesing @yanj-github @rbouqueau FYI, anything else missing?

[gitwjr]

When staging is updated I will update the suite and test on WSL using my new automated build and run script as a test.

[gitwjr]

If I test on a new iPad 11 A16 processor, updated OS, is there anything I need to do to record it on an iPhone15 plus? CHATGPT says the audio must be muted in the playback or not part of playback stream to auto play a video. Or should I just test content with no video? I am willing to buy a new iPad if it will work. Otherwise I can use an android phone as DUT and iPhone for recording.

[gitwjr]

@louaybassbouss @FritzHeiden
Is there a target date for #6?

[louaybassbouss]

Immediate next steps:

• @yanj-github please tag latest stable OF release with v4.0.0
• @rbouqueau please tag the latest Test Content release with v4.0.0
• @FritzHeiden please tag the latest TR and with v4.0.0

[yanj-github]

@louaybassbouss this is InProgress will be ready in couple of days.

[louaybassbouss]

@yanj-github ping us when you are ready. We only need a tag on the main branch, but you need to merge first the feature branch into main.

[yanj-github]

@louaybassbouss just completed now you can find it here: https://github.com/cta-wave/device-observation-framework/releases/tag/v4.0.0

[gitwjr]

@louaybassbouss @FritzHeiden @yanj-github @jpiesing
I'm trying to build the staging branch but I get a checksum error. My script goes to dpctf-deploy, branch set to 'staging'. Staging does not show the 4.0.0 tag. Says it is up to date with 'master'. It appears I have to fetch from a different location. What is the link to the 4.0.0 staging repo? I see the 4 repos are 4.0.0 but 'staging' says 'latest' 'master'.

[gitwjr]

@louaybassbouss @FritzHeiden @yanj-github @jpiesing @mbergman42 I was able to build dpctf 4.0.0 on a second PC I use for 2-device tests (CTA's PC as host, my pc as DUT), although not with the automated script which uses the 'build.sh' command. I will have to modify it to use 'make build' if that is what will be used in the Release version.

As noted earlier, using 'make build' built dpctf v4.0.0 but running './build-dof.sh' (per the README) built the dof v3.0.0 image. The analysis of a 2-device test using 'analyse_recording.sh' failed since it was looking for DOF v4.0.0. I had to run 'docker build --file Dockerfile.dof -t dpctf-dof:v4.0.0 . ' to build dof v4.0.0. The analysis then passed. Either 'make build' should handle the DOF image build automatically or there needs to be instructions in README on how to build the correct DOF image in that step of the DOF TAG needs to be updated to 4.0.0 in the staging repo since that clearly must be what I cloned to get the dpctf 4.0.0 image. (Running docker image returned dpctf 4.0.0 and DOF 3.0.0 images).

I will next (tomorrow) build my PC to 4.0.0 manually using the 'make build' and the DOF 4.0.0 fix above and test it. Then, if successful I will modify the automated scripts, assuming 'make build' is what the release will use.

In the meantime, it appears that the WSL build works with the temporary DOF fix. I still get a failure on 50Hz tests since my iPhone only handles 60Hz recording. But at this point analysis of multiple tests for 1-device and 2-device testing works.

[yanj-github]

@FritzHeiden and @louaybassbouss I also tried and I have got error when doing docker-compose up having error like this:
container.image_config['ContainerConfig'].get('Volumes') or {}
KeyError: 'ContainerConfig'

OF works fine.

I have got
dpctf-dof image v4.0.0
dpctf image v4.0.0

[gitwjr]

OF 4.0.0 is available but when I clone dpctf staging tag v4.0.0 and run build-dof.sh I get DOF v3.0.0. As noted in my comment I had to run docker build --file Dockerfile.dof -t dpctf-dof:v4.0.0 . which then builds DOF v4.0.0 image. I rebuild again using the automation scripts modified to use make build/make import. dpctf image was 4.0.0 but DOF image was still v3.0.0. Ran the fix and it created DOF v4.0.0 image and everything passed. Ran analysis on 3 recordings including one with 2 consecutive tests and all passed with no interventions or errors (still fail 50Hz test due to 60fps camera of course).