Kendalls W by juliuspfadt · Pull Request #319 · jasp-stats/jaspReliability

juliuspfadt · 2026-05-22T16:02:52Z

adds Kendalls coefficient of concordance with the irr::kendall function.
also restructures the agreement analysis a bit
fixes jasp-stats/jasp-issues#2151

Copilot

Pull request overview

Adds Kendall’s coefficient of concordance (W) to the Reliability module’s Rater Agreement analysis (using irr::kendall()), and refactors the Rater Agreement UI/options around a shared bootstrap/CI + seed “Advanced Options” section. Also adds extensive info tooltips across multiple analyses and updates tests + versioning.

Changes:

Add Kendall’s W computation + bootstrap CI support to raterAgreement (R + QML) and extend unit tests with a published reference dataset.
Restructure Rater Agreement options (new bootstrapSamples, coefficient checkboxes default off) + add upgrade mapping for renamed option.
Add info metadata to many tables/plots/controls; bump version + update renv.lock.

Reviewed changes

Copilot reviewed 27 out of 64 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
R/raterAgreement.R	Adds Kendall’s W + placeholder table; renames bootstrap option usage; adds `info` strings.
inst/qml/RaterAgreement.qml	Adds Kendall’s W UI + “Advanced Options” section for CI/bootstrap/seed; expands allowed variable types.
tests/testthat/test-raterAgreement.R	Updates options for new defaults; adds Kendall’s W tests + reference check.
tests/testthat/anxietyRatings.csv	New reference dataset for Kendall’s W verification test.
inst/Upgrades.qml	Adds rename upgrade: `krippendorffsAlphaBootstrapSamplesForCI` → `bootstrapSamples`.
R/standardErrorOfMeasurement.R	Adds `info` metadata; adjusts `$dependOn()` wiring for CI tables/plots.
inst/qml/StandardErrorOfMeasurement.qml	Adds `info` tooltips across SEM UI controls.
R/unidimensionalReliabilityFrequentist.R	Adds `info` metadata to tables.
inst/qml/UnidimensionalReliabilityFrequentist.qml	Adds `info` tooltips and minor UI restructuring (e.g., Samples section).
R/unidimensionalReliabilityBayesian.R	Adds `info` metadata to tables/containers/plots.
inst/qml/UnidimensionalReliabilityBayesian.qml	Adds `info` tooltips across Bayesian UI controls.
R/intraclassCorrelation.R	Adds `info` metadata to main ICC table.
inst/qml/IntraclassCorrelation.qml	Adds `info` tooltips to ICC controls.
R/blandAltman.R	Adds `info` metadata to Bland–Altman containers/plots/tables.
inst/qml/BlandAltman.qml	Adds `info` tooltips to Bland–Altman controls.
NEWS.md	Adds release notes for 0.97.1 (Kendall’s W + UI restructuring).
DESCRIPTION	Bumps package version/date.
renv.lock	Updates pinned dependency metadata (incl. `jaspTools` entry) + updates JASP GitHub SHAs.
.Rbuildignore	Ignores additional dev directories (`.positai`, `.claude`).
.gitignore	Ignores `.positai`.
.vscode/settings.json	Adds workspace R LSP setting.
inst/Description.qml	Minor formatting change (blank line).
inst/help/unidimensionalReliabilityFrequentist.md	Deleted help page.
inst/help/unidimensionalReliabilityBayesian.md	Deleted help page.
inst/help/standardErrorOfMeasurement.md	Deleted help page.
inst/help/raterAgreement.md	Deleted help page.
inst/help/intraclassCorrelation.md	Deleted help page.
inst/help/toolTip/sampleSavingFreq.md	Deleted tooltip help page (replaced by `info`).
inst/help/toolTip/sampleSavingBayes.md	Deleted tooltip help page (replaced by `info`).

  jaspTable <- createJaspTable(title = "Krippendorff's alpha")
+  jaspTable$info <- gettext("Krippendorff's alpha: reliability coefficient applicable to any number of raters, any scale level (nominal/ordinal/interval/ratio), and incomplete data.")
  jaspTable$addColumnInfo(name = "method", title = gettext("Method"), type = "string")
  jaspTable$addColumnInfo(name = "kAlpha", title = "Krippendorff's alpha", type = "number")
  jaspTable$position <- 2


+.kendallWBootRA <- function(jaspResults, dataset, options, ready) {
+  if (!ready || !is.null(jaspResults[["kendallWBootstrapSamples"]]$object))
+    return()
+


+  jaspTable$addColumnInfo(name = "W",     title = gettext("W"),          type = "number")
+  jaspTable$addColumnInfo(name = "chisq", title = gettext("Chi-square"), type = "number")
+  jaspTable$addColumnInfo(name = "df",    title = gettext("df"),         type = "integer")
+  jaspTable$addColumnInfo(name = "p",     title = gettext("p"),          type = "pvalue")


+^\.positai$
+^\.claude$


FBartos · 2026-06-16T14:28:39Z

 		{
-			name:   	"krippendorffsAlpha"
-			label:  	qsTr("Krippendorff's alpha")
-			checked: 	true
-
-			DropDown
-			{
-				name:	"krippendorffsAlphaMethod"
-				label:	qsTr("Method")
-				values:
-				[
-					{label: qsTr("Nominal"),		value: "nominal"},
-					{label: qsTr("Ordinal"),		value: "ordinal"},
-					{label: qsTr("Interval"),		value: "interval"},
-					{label: qsTr("Ratio"),			value: "ratio"}
-				]
-			}
-			IntegerField
+			name:              "ci"
+			label:             qsTr("Confidence interval")
+			checked:           true
+			childrenOnSameRow: true


@juliuspfadt maybe good points?

Implemented (cbb4262): added id: ciOpt to the CI checkbox; the bootstrap-samples field, CI level, and SetSeed now bind enabled: ciOpt.checked, so the advanced controls are disabled when CI is unchecked.

FBartos · 2026-06-16T14:28:28Z

+			info:         qsTr("Number of bootstrap replications used to compute confidence intervals. Higher values give more stable estimates.")
 		}
+
+		SetSeed {}


@juliuspfadt maybe good points?

Implemented (cbb4262): SetSeed now enabled: ciOpt.checked, disabled when the CI box is unchecked (same for the bootstrap-samples field and CI level).

-  ciTable$dependOn(optionsFromObject = jaspResults[["semMainContainer"]][["coefficientTable"]],
-                   options = c("ciLevelTable", "sumScoreCiTable"))
+  ciTable$info <- gettext("Confidence intervals for the true score at each sum score level, computed from the conditional SEM.")
+  ciTable$dependOn(optionsFromObject = jaspResults[["semMainContainer"]][["coefficientTable"]])


-    plot$dependOn(optionsFromObject = jaspResults[["semMainContainer"]][["coefficientTable"]],
-                  options = "combinedPointPlot")
+    plot$info <- gettext("Overlay of all selected methods' conditional SEM across the full range of sum scores.")
+    plot$dependOn(optionsFromObject = jaspResults[["semMainContainer"]][["coefficientTable"]])


-  ciPlotsContainer$dependOn(optionsFromObject = jaspResults[["semMainContainer"]][["coefficientTable"]],
-                            options = c("sumScoreCiPlots", "sumScoreCiPlotsCutoff", "sumScoreCiPlotsCutoffValue", "ciLevelPlots"))
+  ciPlotsContainer$info <- gettext("True-score confidence interval plotted against sum score for each selected method.")
+  ciPlotsContainer$dependOn(optionsFromObject = jaspResults[["semMainContainer"]][["coefficientTable"]])


@@ -1757,6 +1764,7 @@ unidimensionalReliabilityBayesian <- function(jaspResults, dataset, options) {
    g <- jaspGraphs::themeJasp(g)

    plot <- createJaspPlot(plot = g, title = "Posterior Predictive Check Omega", width = 350)


Copilot

Pull request overview

Copilot reviewed 27 out of 64 changed files in this pull request and generated 10 comments.

  ciTable <- createJaspTable(gettext("Sum Score CI Table"))
-  ciTable$dependOn(optionsFromObject = jaspResults[["semMainContainer"]][["coefficientTable"]],
-                   options = c("ciLevelTable", "sumScoreCiTable"))
+  ciTable$info <- gettext("Confidence intervals for the true score at each sum score level, computed from the conditional SEM.")
+  ciTable$dependOn(optionsFromObject = jaspResults[["semMainContainer"]][["coefficientTable"]])
  ciTable$position <- 2


-    plot$dependOn(optionsFromObject = jaspResults[["semMainContainer"]][["coefficientTable"]],
-                  options = "combinedPointPlot")
+    plot$info <- gettext("Overlay of all selected methods' conditional SEM across the full range of sum scores.")
+    plot$dependOn(optionsFromObject = jaspResults[["semMainContainer"]][["coefficientTable"]])


-  ciPlotsContainer$dependOn(optionsFromObject = jaspResults[["semMainContainer"]][["coefficientTable"]],
-                            options = c("sumScoreCiPlots", "sumScoreCiPlotsCutoff", "sumScoreCiPlotsCutoffValue", "ciLevelPlots"))
+  ciPlotsContainer$info <- gettext("True-score confidence interval plotted against sum score for each selected method.")
+  ciPlotsContainer$dependOn(optionsFromObject = jaspResults[["semMainContainer"]][["coefficientTable"]])


 .semPointPlots <- function(jaspResults, dataset, options, ready) {

  if (!is.null(jaspResults[["semMainContainer"]][["pointPlots"]]) || !options[["pointPlots"]]
      || !ready || jaspResults[["semMainContainer"]]$getError()) {return()}

  pointPlotsContainer <- createJaspContainer(title = gettext("Point Plots"))
+  pointPlotsContainer$info <- gettext("Conditional SEM plotted as a function of sum score for each selected method.")
  pointPlotsContainer$dependOn(optionsFromObject = jaspResults[["semMainContainer"]][["coefficientTable"]], options = "pointPlots")
  jaspResults[["semMainContainer"]][["pointPlotsContainer"]] <- pointPlotsContainer


 .semSumScoreCiPlots <- function(jaspResults, dataset, options, ready) {

  if (!is.null(jaspResults[["semMainContainer"]][["ciPlots"]]) || !options[["sumScoreCiPlots"]]
      || !ready || jaspResults[["semMainContainer"]]$getError()) {return()}

  nc <- length(unique(c(as.matrix(dataset)))) # may be needed for IRT

  ciPlotsContainer <- createJaspContainer(title = gettext("Sum Score CI Plots"))
-  ciPlotsContainer$dependOn(optionsFromObject = jaspResults[["semMainContainer"]][["coefficientTable"]],
-                            options = c("sumScoreCiPlots", "sumScoreCiPlotsCutoff", "sumScoreCiPlotsCutoffValue", "ciLevelPlots"))
+  ciPlotsContainer$info <- gettext("True-score confidence interval plotted against sum score for each selected method.")
+  ciPlotsContainer$dependOn(optionsFromObject = jaspResults[["semMainContainer"]][["coefficientTable"]])
  jaspResults[["semMainContainer"]][["ciPlotsContainer"]] <- ciPlotsContainer


  jaspTable <- createJaspTable(title = "Krippendorff's alpha")
+  jaspTable$info <- gettext("Krippendorff's alpha: reliability coefficient applicable to any number of raters, any scale level (nominal/ordinal/interval/ratio), and incomplete data.")
  jaspTable$addColumnInfo(name = "method", title = gettext("Method"), type = "string")
  jaspTable$addColumnInfo(name = "kAlpha", title = "Krippendorff's alpha", type = "number")


+.kendallWBootRA <- function(jaspResults, dataset, options, ready) {
+  if (!ready || !is.null(jaspResults[["kendallWBootstrapSamples"]]$object))
+    return()
+
+  bootstrapSamples <- createJaspState()
+  bootstrapSamples$dependOn(options = c(
+    "variables", "kendallW", "ci", "bootstrapSamples",
+    "correctForTies", "dataStructure", "setSeed", "seed"
+  ))
+  jaspResults[["kendallWBootstrapSamples"]] <- bootstrapSamples
+


+			checked:           true
+			childrenOnSameRow: true
+			id: 							 ciOpt
+			info:              qsTr("Report a bootstrap confidence interval for each agreement coefficient.")


+			label:             qsTr("Confidence interval")
+			checked:           true
+			childrenOnSameRow: true
+			id: 							 ciOpt


 ^\.github$
 ^tests/upgrades$
 ^examples$
 ^jasp_dev_work_dir$
+^\.positai$
+^\.claude$


CRAN binary for dcurver on aarch64-apple-darwin20 links against libiomp5 (Intel OpenMP), unavailable on CI (fopenmp stripped). Source install compiles without OpenMP. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

jaspTools now uses getFromNamespace instead of sourcing R files, requiring jaspReliability to be loaded before analysisOptions() is called. test-blandAltman and test-intraclassCorrelation ran before library tests that incidentally loaded the package, causing only those two to fail. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…mpat)" This reverts commit ed50a8e.

FBartos · 2026-06-15T06:57:22Z

build builder bot!

FBartos · 2026-06-16T14:28:28Z

+			info:         qsTr("Number of bootstrap replications used to compute confidence intervals. Higher values give more stable estimates.")
 		}
+
+		SetSeed {}


@juliuspfadt maybe good points?

FBartos · 2026-06-16T14:28:39Z

 		{
-			name:   	"krippendorffsAlpha"
-			label:  	qsTr("Krippendorff's alpha")
-			checked: 	true
-
-			DropDown
-			{
-				name:	"krippendorffsAlphaMethod"
-				label:	qsTr("Method")
-				values:
-				[
-					{label: qsTr("Nominal"),		value: "nominal"},
-					{label: qsTr("Ordinal"),		value: "ordinal"},
-					{label: qsTr("Interval"),		value: "interval"},
-					{label: qsTr("Ratio"),			value: "ratio"}
-				]
-			}
-			IntegerField
+			name:              "ci"
+			label:             qsTr("Confidence interval")
+			checked:           true
+			childrenOnSameRow: true


@juliuspfadt maybe good points?

FBartos · 2026-06-16T14:47:38Z

build builder bot!

github-actions · 2026-06-16T14:54:41Z

✅ Build Complete!

Your JASP module bundles have been successfully built for macOS (ARM) and Windows.

👉 Click here to download the artifacts

(Scroll to the bottom of the page to find the "Artifacts" section).

Copilot

Pull request overview

Copilot reviewed 29 out of 66 changed files in this pull request and generated 3 comments.


  bootstrapSamples <- createJaspState()


+  if (!ready || !is.null(jaspResults[["kendallWBootstrapSamples"]]$object))
+    return()


  ciTable$dependOn(optionsFromObject = jaspResults[["semMainContainer"]][["coefficientTable"]],
-                   options = c("ciLevelTable", "sumScoreCiTable"))
+                   options           = c("sumScoreCiTable", "ciLevelTable"))


address PR review: translatable plot title in Bayesian omega PPC; statistical symbols (W, df, p) should not be marked for translation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 29 out of 66 changed files in this pull request and generated 3 comments.

juliuspfadt · 2026-07-02T12:07:52Z

+	Upgrade
+	{
+		functionName: "raterAgreement"
+		fromVersion:  "0.97.0"
+		toVersion:    "0.97.1"

+		ChangeRename
+		{
+			from: "krippendorffsAlphaBootstrapSamplesForCI"
+			to:   "bootstrapSamples"
+		}


Good catch — fixed in 6f21c58. fromVersion lowered to 0.95.5 (the released version that still carries krippendorffsAlphaBootstrapSamplesForCI); 0.97.0 was never released so the migration was being skipped for real .jasp files.

juliuspfadt · 2026-07-02T12:07:54Z

+{
+  "build": ["dcurver"]
+}


Intentional. dcurver is a transitive dependency via mirt (used for the IRT SEM analysis), and forcing a source build avoids an OpenMP linker failure on CI (35af3c2). Leaving as-is.

juliuspfadt · 2026-07-02T12:07:55Z

  renv_bootstrap_run <- function(project, libpath, version) {
+    tryCatch(
+      renv_bootstrap_run_impl(project, libpath, version),
+      error = function(e) {
+        msg <- paste(
+          "failed to bootstrap renv: the project will not be loaded.",
+          paste("Reason:", conditionMessage(e)),
+          "Use `renv::activate()` to re-initialize the project.",
+          sep = "\n"
+        )
+        warning(msg, call. = FALSE)
+      }
+    )
+  }


This is auto-generated renv boilerplate (regenerated by renv when its version changes), not hand-authored logic, so not modifying it here.

old option krippendorffsAlphaBootstrapSamplesForCI exists in released 0.95.5; fromVersion 0.97.0 (never released) skipped migration for real .jasp files Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

irr::kappam.fleiss returns per-category detail rows in its own order; categoryKappas was reindexed to sorted categories but categorySE was not, so SE/CI attached to the wrong category when the two orders differed. Reindex the detail matrix once so kappa and SE stay paired. Regenerate the (previously swapped) expected values. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

juliuspfadt force-pushed the kendallsW branch from 60ab1e1 to 2be69d7 Compare May 26, 2026 12:01

juliuspfadt added 7 commits June 11, 2026 11:05

stuff for kendalls W

070b1ef

restructure agreement analysis

de83311

adjust tests

d644483

change from md files to info field in qml

23eb4dc

add info to jaspResults objects

90b1535

fix kendall tbale issue

250253a

more tests for kendalls w

5ff7f7c

juliuspfadt force-pushed the kendallsW branch from 173437b to a649b22 Compare June 11, 2026 09:06

juliuspfadt requested a review from Copilot June 11, 2026 09:06

Copilot started reviewing on behalf of juliuspfadt June 11, 2026 09:06 View session

Copilot AI reviewed Jun 11, 2026

View reviewed changes

juliuspfadt force-pushed the kendallsW branch from a649b22 to f81e7c8 Compare June 11, 2026 09:28

fix tests and add more info

cf18504

juliuspfadt force-pushed the kendallsW branch from f81e7c8 to cf18504 Compare June 11, 2026 09:39

juliuspfadt requested a review from Copilot June 11, 2026 09:45

Copilot started reviewing on behalf of juliuspfadt June 11, 2026 09:45 View session

Copilot AI reviewed Jun 11, 2026

View reviewed changes

juliuspfadt force-pushed the kendallsW branch 2 times, most recently from ef4171f to 354a13e Compare June 11, 2026 12:11

fix copilot feedback

a480ed5

juliuspfadt force-pushed the kendallsW branch from 354a13e to a480ed5 Compare June 11, 2026 12:39

juliuspfadt and others added 3 commits June 11, 2026 15:05

fix: force dcurver source install to avoid OpenMP CI failure

35af3c2

CRAN binary for dcurver on aarch64-apple-darwin20 links against libiomp5 (Intel OpenMP), unavailable on CI (fopenmp stripped). Source install compiles without OpenMP. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Revert "fix: load package before tests (jaspTools getFromNamespace co…

baf13bd

…mpat)" This reverts commit ed50a8e.

juliuspfadt requested a review from FBartos June 11, 2026 15:02

update jaspTools

426d03d

FBartos requested changes Jun 16, 2026

View reviewed changes

fix ci id checked stuff

cbb4262

juliuspfadt requested review from FBartos and Copilot June 16, 2026 14:54

Copilot started reviewing on behalf of juliuspfadt June 16, 2026 14:55 View session

Copilot AI reviewed Jun 16, 2026

View reviewed changes

i18n: wrap PPC omega plot title; unwrap W/df/p symbols

a779163

address PR review: translatable plot title in Bayesian omega PPC; statistical symbols (W, df, p) should not be marked for translation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

juliuspfadt requested a review from Copilot July 2, 2026 11:44

Copilot started reviewing on behalf of juliuspfadt July 2, 2026 11:45 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

fix: correct raterAgreement upgrade fromVersion to 0.95.5

6f21c58

old option krippendorffsAlphaBootstrapSamplesForCI exists in released 0.95.5; fromVersion 0.97.0 (never released) skipped migration for real .jasp files Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

juliuspfadt requested review from FBartos and removed request for FBartos July 2, 2026 12:22

		@@ -1757,6 +1764,7 @@ unidimensionalReliabilityBayesian <- function(jaspResults, dataset, options) {
		g <- jaspGraphs::themeJasp(g)

		plot <- createJaspPlot(plot = g, title = "Posterior Predictive Check Omega", width = 350)

		if (!ready \|\| !is.null(jaspResults[["kendallWBootstrapSamples"]]$object))
		return()

		^\.positai$
		^\.claude$

Uh oh!

Conversation

juliuspfadt commented May 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

FBartos commented Jun 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FBartos commented Jun 16, 2026

Uh oh!

github-actions Bot commented Jun 16, 2026

✅ Build Complete!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants