Skip to content

Collect pod resource usage and cgroup metrics in must-gather #2494

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
evanwang9x wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Collect pod resource usage and cgroup metrics in must-gather #2494

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
efb67f0
feat: collect pod resource usage and cgroup metrics in skupper-gather
Jun 10, 2026
976865f
Changed code to retrieve from all containers
Jul 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 109 additions & 1 deletion scripts/skupper-gather.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,102 @@ get_log_collection_args() {
fi
}

function getPodResources() {
local ns="${1}"
local podName="${2}"
local container="${3}"
local podDirName="${podName##"pod/"}"

local resourcePath=${BASE_COLLECTION_PATH}/namespaces/${ns}/pods/${podDirName}/${container}/${container}/resources
mkdir -p "${resourcePath}"

echo "Collecting resource usage for pod ${podName} in ${ns}"

# Memory Info
oc exec -n "${ns}" "${podName}" -c "${container}" -- bash -c '
curr=$(cat /sys/fs/cgroup/memory.current)
max=$(cat /sys/fs/cgroup/memory.max)
curr_kb=$((curr / 1024))
curr_mb=$((curr / 1024 / 1024))

if [ "$max" = "max" ]; then
limit_str="max (uncapped)"
else
max_kb=$((max / 1024))
max_mb=$((max / 1024 / 1024))
limit_str="$max bytes ($max_kb KiB / $max_mb MiB)"
fi

echo "Usage: $curr bytes ($curr_kb KiB / $curr_mb MiB) - Limit: $limit_str" > /tmp/memory.info
' 2>/dev/null
oc exec -n "${ns}" "${podName}" -c "${container}" -- cat /tmp/memory.info > "${resourcePath}/memory.info" 2>&1
Comment on lines +30 to +47

@coderabbitai coderabbitai Bot Jul 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win

Avoid the intermediate /tmp file and second oc exec round trip.

Unlike the other cgroup blocks (pressure, cpu.stat, cpu.max) which write directly to stdout and are captured via output redirection in a single oc exec, memory.info is written to a hardcoded /tmp/memory.info inside the container and then read back with a second oc exec. Static analysis flags this as an insecure/predictable temp file (CWE-377), and it also doubles the exec calls needlessly.

🛡️ Proposed fix: write directly to stdout like the other blocks 
   # Memory Info
-  oc exec -n "${ns}" "${podName}" -c "${container}" -- bash -c '
+  oc exec -n "${ns}" "${podName}" -c "${container}" -- bash -c '
     curr=$(cat /sys/fs/cgroup/memory.current)
     max=$(cat /sys/fs/cgroup/memory.max)
     curr_kb=$((curr / 1024))
     curr_mb=$((curr / 1024 / 1024))

     if [ "$max" = "max" ]; then
       limit_str="max (uncapped)"
     else
       max_kb=$((max / 1024))
       max_mb=$((max / 1024 / 1024))
       limit_str="$max bytes ($max_kb KiB / $max_mb MiB)"
     fi

-    echo "Usage: $curr bytes ($curr_kb KiB / $curr_mb MiB) - Limit: $limit_str" > /tmp/memory.info
-  ' 2>/dev/null
-  oc exec -n "${ns}" "${podName}" -c "${container}" -- cat /tmp/memory.info > "${resourcePath}/memory.info" 2>&1
+    echo "Usage: $curr bytes ($curr_kb KiB / $curr_mb MiB) - Limit: $limit_str"
+  ' > "${resourcePath}/memory.info" 2>&1
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# Memory Info
oc exec -n "${ns}" "${podName}" -c "${container}" -- bash -c '
curr=$(cat /sys/fs/cgroup/memory.current)
max=$(cat /sys/fs/cgroup/memory.max)
curr_kb=$((curr / 1024))
curr_mb=$((curr / 1024 / 1024))
if [ "$max" = "max" ]; then
limit_str="max (uncapped)"
else
max_kb=$((max / 1024))
max_mb=$((max / 1024 / 1024))
limit_str="$max bytes ($max_kb KiB / $max_mb MiB)"
fi
echo "Usage: $curr bytes ($curr_kb KiB / $curr_mb MiB) - Limit: $limit_str" > /tmp/memory.info
' 2>/dev/null
oc exec -n "${ns}" "${podName}" -c "${container}" -- cat /tmp/memory.info > "${resourcePath}/memory.info" 2>&1
# Memory Info
oc exec -n "${ns}" "${podName}" -c "${container}" -- bash -c '
curr=$(cat /sys/fs/cgroup/memory.current)
max=$(cat /sys/fs/cgroup/memory.max)
curr_kb=$((curr / 1024))
curr_mb=$((curr / 1024 / 1024))
if [ "$max" = "max" ]; then
limit_str="max (uncapped)"
else
max_kb=$((max / 1024))
max_mb=$((max / 1024 / 1024))
limit_str="$max bytes ($max_kb KiB / $max_mb MiB)"
fi
echo "Usage: $curr bytes ($curr_kb KiB / $curr_mb MiB) - Limit: $limit_str"
' > "${resourcePath}/memory.info" 2>&1
🧰 Tools
🪛 ast-grep (0.44.1)

[warning] 46-46: Writing to or reading from a hardcoded, predictable path under /tmp is vulnerable to symlink and TOCTOU attacks: a local attacker can pre-create the file (or a symlink pointing elsewhere) and hijack or corrupt the contents. Generate a unique, unpredictable temporary file with mktemp instead, e.g. tmpfile="$(mktemp)" (or mktemp -d for directories) and reference "$tmpfile".
Context: /tmp/memory.info
Note: [CWE-377] Insecure Temporary File.

(predictable-tmp-file-bash)

Source: Linters/SAST tools


# Get previous container termination info
oc get pod "${podName##"pod/"}" -n "${ns}" -o jsonpath="{.status.containerStatuses[?(@.name=='${container}')].lastState}" > "${resourcePath}/last.terminated" 2>/dev/null
if [ ! -s "${resourcePath}/last.terminated" ]; then
echo "No previous termination recorded" > "${resourcePath}/last.terminated"
fi

# Memory pressure
oc exec -n "${ns}" "${podName}" -c "${container}" -- bash -c '
if [ -f /sys/fs/cgroup/memory.pressure ]; then
cat /sys/fs/cgroup/memory.pressure
else
echo "PSI not available on this node (kernel psi=1 not enabled)"
fi
' > "${resourcePath}/memory.pressure" 2>&1

# CPU usage
oc exec -n "${ns}" "${podName}" -c "${container}" -- bash -c "
declare -A cpu_descriptions
cpu_descriptions=(
[usage_usec]='Total CPU time consumed (user + system)'
[user_usec]='Time spent in user-space (app logic)'
[system_usec]='Time spent in kernel-space (syscalls, I/O)'
[core_sched.force_idle_usec]='Time CPU forced idle for security (cross-HT side-channel protection)'
[nr_periods]='Number of CPU quota enforcement periods (0 = no quota set)'
[nr_throttled]='Times the container was throttled for exceeding CPU quota'
[throttled_usec]='Total time spent throttled/waiting for CPU quota'
[nr_bursts]='Times container used burst CPU capacity beyond quota'
[burst_usec]='Total time spent using burst CPU capacity'
)

cat /sys/fs/cgroup/cpu.stat | while read line; do
val=\$(echo \$line | awk '{print \$2}')
key=\$(echo \$line | awk '{print \$1}')
desc=\${cpu_descriptions[\$key]}
if [[ \$key == *_usec ]] && [ \$val -ge 1000000 ]; then
sec=\$(awk \"BEGIN {printf \\\"%.2f\\\", \$val/1000000}\")
echo \"\$line (\$sec s) | \$desc\"
else
echo \"\$line | \$desc\"
fi
done
" > "${resourcePath}/cpu.stat" 2>&1
Comment on lines +64 to +90

@coderabbitai coderabbitai Bot Jul 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Inspect the Skupper container images used across pods to check shell availability assumptions.
rg -n 'FROM ' --type=dockerfile
fd -e Dockerfile

Repository: skupperproject/skupper

Length of output: 195

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n== relevant call sites ==\n'
rg -n 'getSkupperResourcesInNamespace|getPodResources|bash -c|declare -A|cpu\.stat' scripts/skupper-gather.sh

printf '\n== file map ==\n'
ast-grep outline scripts/skupper-gather.sh --view expanded

printf '\n== container/image hints in repo ==\n'
rg -n 'Dockerfile\.(kube-adaptor|controller|cli|system-controller|network-observer)|FROM ' -g '!**/node_modules/**' .

Repository: skupperproject/skupper

Length of output: 2093

🏁 Script executed:

#!/bin/bash
set -euo pipefail

sed -n '1,170p' scripts/skupper-gather.sh | cat -n

Repository: skupperproject/skupper

Length of output: 7333

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n== script excerpt ==\n'
sed -n '1,170p' scripts/skupper-gather.sh | cat -n

Repository: skupperproject/skupper

Length of output: 7355

oc exec -- bash -c does not work for the scratch-based Skupper images.
getSkupperResourcesInNamespace now iterates every container in every app.kubernetes.io/part-of=skupper pod, and the final images for Dockerfile.cli, Dockerfile.controller, Dockerfile.kube-adaptor, Dockerfile.network-observer, and Dockerfile.system-controller are FROM scratch. Those containers won’t have bash, so resource collection fails for them; a POSIX sh path or another fallback is needed.


# CPU limit
oc exec -n "${ns}" "${podName}" -c "${container}" -- bash -c '
cpu_max=$(cat /sys/fs/cgroup/cpu.max)
quota=$(echo $cpu_max | awk "{print \$1}")
period=$(echo $cpu_max | awk "{print \$2}")
if [ "$quota" = "max" ]; then
echo "CPU Limit: uncapped (no quota set)"
else
cores=$(awk "BEGIN {printf \"%.2f\", $quota/$period}")
echo "CPU Limit: $quota/$period us = $cores cores"
fi
' > "${resourcePath}/cpu.max" 2>&1

# CPU pressure
oc exec -n "${ns}" "${podName}" -c "${container}" -- bash -c '
if [ -f /sys/fs/cgroup/cpu.pressure ]; then
cat /sys/fs/cgroup/cpu.pressure
else
echo "PSI not available on this node (kernel psi=1 not enabled)"
fi
' > "${resourcePath}/cpu.pressure" 2>&1
}

function getSkupperRouterSkstatInNamespace() {
local routerNamespace="${1}"
local flags=("g" "c" "l" "n" "e" "a" "m" "p")
Expand All @@ -28,7 +124,17 @@ function getSkupperRouterSkstatInNamespace() {
for flag in "${flags[@]}"; do
oc exec -n "${sitens}" "${routerName}" -c router -- skstat -"${flag}" > "${logPath}/skstat.${flag}" 2>&1
done
done
done
}

function getSkupperResourcesInNamespace() {
local ns="${1}"

for podName in $(oc get pods -n "${ns}" -l app.kubernetes.io/part-of=skupper -oname); do
for container in $(oc get "${podName}" -n "${ns}" -o jsonpath='{.spec.containers[*].name}'); do
getPodResources "${ns}" "${podName}" "${container}"
done
done
}

function getCRDs() {
Expand Down Expand Up @@ -113,12 +219,14 @@ function main() {
inspectNamespace "${sitens}"

getSkupperRouterSkstatInNamespace "${sitens}"
getSkupperResourcesInNamespace "${sitens}"
done

# iterate over all namespaces which have AttachedConnectors
for acns in $(oc get attachedconnector -A -o=jsonpath="{.items[*].metadata.namespace}"); do
echo "Inspecting AttachedConnector in ${acns} namespace"
inspectNamespace "${acns}"
getSkupperResourcesInNamespace "${acns}"
done

echo
Expand Down
Loading