Skip to content

Commit 1a8e113

Browse files
smartwatermelonClaude Code Botclaude
authored
feat: initial scaffold (#1)
* feat: initial scaffold for archive-resolver Adds idempotent macOS DNS override installer for archive.today and mirrors, plus GitHub Actions workflows for CI and automated mirror list maintenance. - install.sh: idempotent script that writes /etc/resolver files (primary resolver + symlinks), tracks managed entries via manifest, removes stale entries, flushes DNS cache. Supports --update-mirrors (fetches live mirror list from Wikipedia API, no root needed), --dry-run, --uninstall, --no-fetch, --nameserver, --yes. - mirrors.txt: domain list sourced from Wikipedia; first entry is the primary (gets a resolver file), rest become symlinks to it. - .github/workflows/ci.yml: ShellCheck lint, bash syntax check, mirrors.txt format validation on every push/PR. - .github/workflows/release.yml: creates GitHub release on vX.Y.Z tags; verifies tag matches SCRIPT_VERSION in install.sh. - .github/workflows/update-mirrors.yml: runs first Monday of each month, diffs Wikipedia mirror list against mirrors.txt, opens a PR if the set has changed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(update-mirrors): compute diff before overwriting mirrors.txt DIFF_SUMMARY was computed in the 'Create pull request' step by diffing mirrors.txt against /tmp/ordered_domains.txt — but mirrors.txt had already been overwritten with the new content, so the diff was always empty and the PR body always read 'See diff'. Fix: capture the diff as a step output in the 'check' step (before the overwrite), then reference it via steps.check.outputs.diff_summary in the 'Create pull request' step. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: pass DESIRED_DOMAINS explicitly to apply_removals export on a bash array only serialises element [0] to child processes; relying on an implicit global inside a $() subshell is fragile and misleading. Instead, pass both arrays as positional arguments using a "--" delimiter, and parse them inside the function. Also guard against the empty-manifest edge case: when no manifest file exists yet, "${MANIFEST_DOMAINS[@]:-}" expands to a single empty string, which was being treated as a domain to remove. Skip empty entries. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Code Bot <claude-code@smartwatermelon.github> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent a0175ac commit 1a8e113

7 files changed

Lines changed: 1074 additions & 0 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: ["main"]
6+
pull_request:
7+
branches: ["main"]
8+
9+
permissions:
10+
contents: read
11+
12+
jobs:
13+
shellcheck:
14+
name: ShellCheck
15+
runs-on: ubuntu-latest
16+
steps:
17+
- uses: actions/checkout@v4
18+
19+
- name: Install ShellCheck
20+
run: sudo apt-get install -y shellcheck
21+
22+
- name: Lint install.sh
23+
run: shellcheck --severity=warning install.sh
24+
25+
syntax:
26+
name: Bash Syntax
27+
runs-on: ubuntu-latest
28+
steps:
29+
- uses: actions/checkout@v4
30+
31+
- name: Check install.sh syntax
32+
run: bash -n install.sh
33+
34+
- name: Check mirrors.txt format
35+
run: |
36+
# Must contain at least one non-comment domain line
37+
if ! grep -qE '^[^#[:space:]]' mirrors.txt; then
38+
echo "mirrors.txt contains no valid domain entries" >&2
39+
exit 1
40+
fi
41+
# Each domain line must look like a valid hostname (letters, digits, dots, hyphens)
42+
while IFS= read -r line; do
43+
[[ "$line" =~ ^[[:space:]]*# ]] && continue # comment
44+
[[ -z "$line" ]] && continue # blank
45+
if ! [[ "$line" =~ ^[a-zA-Z0-9][a-zA-Z0-9.-]+[a-zA-Z0-9]$ ]]; then
46+
echo "Invalid domain in mirrors.txt: '${line}'" >&2
47+
exit 1
48+
fi
49+
done < mirrors.txt
50+
echo "mirrors.txt is valid"
51+
52+
dry-run:
53+
name: Dry-run (Linux compat check)
54+
runs-on: ubuntu-latest
55+
steps:
56+
- uses: actions/checkout@v4
57+
58+
- name: Patch script for Linux dry-run
59+
# The script exits on non-Darwin; stub out the OS check so we can
60+
# exercise argument parsing and logic on the CI runner.
61+
run: |
62+
sed 's/\[\[ "$(uname -s)" == "Darwin" \]\]/true/' install.sh > install-test.sh
63+
chmod +x install-test.sh
64+
65+
- name: Run dry-run (no-fetch, no root required)
66+
run: bash install-test.sh --dry-run --no-fetch 2>&1 || true
67+
# We expect non-zero exit because /etc/resolver doesn't exist on Linux;
68+
# the goal is to verify flag parsing and early-stage logic don't crash.

‎.github/workflows/release.yml‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
name: Release
2+
3+
on:
4+
push:
5+
tags:
6+
- "v*.*.*"
7+
8+
permissions:
9+
contents: write
10+
11+
jobs:
12+
release:
13+
name: Create GitHub Release
14+
runs-on: ubuntu-latest
15+
steps:
16+
- uses: actions/checkout@v4
17+
18+
- name: Validate tag format
19+
run: |
20+
TAG="${GITHUB_REF_NAME}"
21+
if ! [[ "$TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
22+
echo "Tag '${TAG}' does not match vMAJOR.MINOR.PATCH" >&2
23+
exit 1
24+
fi
25+
26+
- name: Verify script version matches tag
27+
run: |
28+
TAG_VERSION="${GITHUB_REF_NAME#v}"
29+
SCRIPT_VERSION="$(grep -oP "(?<=SCRIPT_VERSION=\")[^\"]+" install.sh)"
30+
if [[ "$TAG_VERSION" != "$SCRIPT_VERSION" ]]; then
31+
echo "Tag version (${TAG_VERSION}) does not match SCRIPT_VERSION in install.sh (${SCRIPT_VERSION})" >&2
32+
exit 1
33+
fi
34+
35+
- name: Install ShellCheck
36+
run: sudo apt-get install -y shellcheck
37+
38+
- name: Lint before release
39+
run: shellcheck --severity=warning install.sh
40+
41+
- name: Validate mirrors.txt
42+
run: |
43+
if ! grep -qE '^[^#[:space:]]' mirrors.txt; then
44+
echo "mirrors.txt contains no valid domain entries" >&2
45+
exit 1
46+
fi
47+
48+
- name: Create release archive
49+
run: |
50+
VERSION="${GITHUB_REF_NAME}"
51+
ARCHIVE="archive-resolver-${VERSION}.tar.gz"
52+
tar -czf "$ARCHIVE" \
53+
install.sh \
54+
mirrors.txt \
55+
README.md \
56+
LICENSE.md
57+
echo "ARCHIVE=${ARCHIVE}" >> "$GITHUB_ENV"
58+
59+
- name: Generate release notes
60+
id: release_notes
61+
run: |
62+
VERSION="${GITHUB_REF_NAME}"
63+
DATE="$(date -u '+%Y-%m-%d')"
64+
MIRROR_COUNT="$(grep -cE '^[^#[:space:]]' mirrors.txt)"
65+
66+
cat > release_notes.md <<EOF
67+
## ${VERSION} — ${DATE}
68+
69+
### What's included
70+
71+
- \`install.sh\` — idempotent macOS DNS override installer
72+
- \`mirrors.txt\` — ${MIRROR_COUNT} archive.today mirror domains
73+
74+
### Install
75+
76+
\`\`\`sh
77+
RELEASE_URL="https://github.com/${{ github.repository }}"
78+
curl -fsSL "${RELEASE_URL}/releases/download/${VERSION}/archive-resolver-${VERSION}.tar.gz" \\
79+
| tar -xz
80+
sudo ./install.sh
81+
\`\`\`
82+
83+
Or run directly from \`main\` (always uses latest mirror list):
84+
85+
\`\`\`sh
86+
git clone https://github.com/${{ github.repository }}.git
87+
cd archive-resolver
88+
sudo ./install.sh
89+
\`\`\`
90+
91+
### Mirrors in this release
92+
93+
$(grep -E '^[^#[:space:]]' mirrors.txt | sed 's/^/- /')
94+
EOF
95+
96+
- name: Publish release
97+
env:
98+
GH_TOKEN: ${{ github.token }}
99+
run: |
100+
gh release create "$GITHUB_REF_NAME" \
101+
--title "archive-resolver $GITHUB_REF_NAME" \
102+
--notes-file release_notes.md \
103+
"$ARCHIVE" \
104+
install.sh \
105+
mirrors.txt

‎.github/workflows/update-mirrors.yml‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
name: Update mirror list
2+
3+
on:
4+
schedule:
5+
# First Monday of every month at 09:07 UTC
6+
- cron: "7 9 1-7 * 1"
7+
workflow_dispatch:
8+
inputs:
9+
dry_run:
10+
description: "Dry run (show changes without creating a PR)"
11+
type: boolean
12+
default: false
13+
14+
permissions:
15+
contents: write
16+
pull-requests: write
17+
18+
jobs:
19+
update-mirrors:
20+
name: Check Wikipedia for mirror changes
21+
runs-on: ubuntu-latest
22+
steps:
23+
- uses: actions/checkout@v4
24+
25+
- name: Fetch and parse mirror list from Wikipedia
26+
id: check
27+
run: |
28+
WIKIPEDIA_API="https://en.wikipedia.org/w/api.php?action=parse&page=Archive.today&prop=text&format=json"
29+
30+
echo "Fetching Wikipedia article..."
31+
curl --silent --fail --max-time 30 --location "$WIKIPEDIA_API" -o /tmp/wiki.json
32+
33+
# Extract <li>archive.TLD</li> entries from the infobox HTML.
34+
# Writing to a temp file avoids any shell-quoting issues with the JSON.
35+
python3 - /tmp/wiki.json <<'PYEOF' > /tmp/new_domains.txt
36+
import sys, json, re
37+
38+
with open(sys.argv[1]) as f:
39+
html = json.load(f)['parse']['text']['*']
40+
41+
found = re.findall(r'<li>(archive\.[a-z]{2,6})</li>', html)
42+
seen = set()
43+
for d in found:
44+
if d not in seen:
45+
seen.add(d)
46+
print(d)
47+
PYEOF
48+
49+
if [[ ! -s /tmp/new_domains.txt ]]; then
50+
echo "ERROR: No domains extracted from Wikipedia article" >&2
51+
exit 1
52+
fi
53+
54+
# Ensure archive.today is first; preserve Wikipedia article ordering for the rest.
55+
{
56+
echo "archive.today"
57+
grep -v '^archive\.today$' /tmp/new_domains.txt
58+
} > /tmp/ordered_domains.txt
59+
60+
echo "Domains found on Wikipedia:"
61+
cat /tmp/ordered_domains.txt
62+
63+
# Compare sorted sets (order differences don't constitute an update).
64+
CURRENT_SORTED="$(grep -E '^[^#[:space:]]' mirrors.txt | sed 's/[[:space:]]*//' | sort)"
65+
NEW_SORTED="$(sort /tmp/ordered_domains.txt)"
66+
67+
if [[ "$CURRENT_SORTED" == "$NEW_SORTED" ]]; then
68+
echo "up_to_date=true" >> "$GITHUB_OUTPUT"
69+
echo "No changes detected — mirrors.txt is up to date."
70+
else
71+
echo "up_to_date=false" >> "$GITHUB_OUTPUT"
72+
echo "Changes detected:"
73+
diff <(echo "$CURRENT_SORTED") <(echo "$NEW_SORTED") \
74+
| grep '^[<>]' | sed 's/^< / removed: /; s/^> / added: /' || true
75+
76+
# Save the diff before overwriting mirrors.txt — the Create pull request
77+
# step runs in a separate shell and mirrors.txt will already be updated by then.
78+
DIFF_SUMMARY="$(diff \
79+
<(echo "$CURRENT_SORTED") \
80+
<(echo "$NEW_SORTED") \
81+
| grep '^[<>]' | sed 's/^< /- removed: /; s/^> /+ added: /' || echo 'See diff')"
82+
echo "diff_summary<<EOF" >> "$GITHUB_OUTPUT"
83+
echo "$DIFF_SUMMARY" >> "$GITHUB_OUTPUT"
84+
echo "EOF" >> "$GITHUB_OUTPUT"
85+
86+
# Write updated mirrors.txt
87+
{
88+
echo "# archive-resolver mirror list"
89+
echo "# Format: one domain per line. Lines starting with # are comments."
90+
echo "# The first non-comment line is the primary domain (others become symlinks)."
91+
echo "#"
92+
echo "# Source: https://en.wikipedia.org/wiki/Archive.today"
93+
printf "# Updated: %s\n" "$(date -u '+%Y-%m-%dT%H:%M:%SZ')"
94+
cat /tmp/ordered_domains.txt
95+
} > mirrors.txt
96+
fi
97+
98+
- name: Create pull request
99+
if: steps.check.outputs.up_to_date == 'false' && inputs.dry_run != 'true'
100+
env:
101+
GH_TOKEN: ${{ github.token }}
102+
run: |
103+
DATE="$(date -u '+%Y-%m-%d')"
104+
BRANCH="chore/update-mirrors-${DATE}"
105+
106+
git config user.name "github-actions[bot]"
107+
git config user.email "github-actions[bot]@users.noreply.github.com"
108+
git checkout -b "$BRANCH"
109+
git add mirrors.txt
110+
git commit -m "chore: update mirror list from Wikipedia (${DATE})"
111+
git push origin "$BRANCH"
112+
113+
DIFF_SUMMARY="${{ steps.check.outputs.diff_summary }}"
114+
WIKI_URL="https://en.wikipedia.org/wiki/Archive.today"
115+
gh pr create \
116+
--title "chore: update mirror list from Wikipedia (${DATE})" \
117+
--body "Automated update of \`mirrors.txt\` based on the
118+
current [Archive.today Wikipedia article](${WIKI_URL}).
119+
120+
**Changes:**
121+
\`\`\`
122+
${DIFF_SUMMARY}
123+
\`\`\`
124+
125+
**Review checklist:**
126+
- [ ] Verify removed domains are genuinely discontinued mirrors
127+
- [ ] Verify added domains are genuine archive.today mirrors
128+
- [ ] Confirm \`archive.today\` remains the first (primary) entry" \
129+
--label "automated" \
130+
--head "$BRANCH" \
131+
--base main
132+
133+
- name: Dry run summary
134+
if: inputs.dry_run == 'true'
135+
run: |
136+
if [[ "${{ steps.check.outputs.up_to_date }}" == "true" ]]; then
137+
echo "Dry run: mirrors.txt is already up to date."
138+
else
139+
echo "Dry run: changes detected. A PR would be created if dry_run=false."
140+
echo "Updated mirror list:"
141+
cat /tmp/ordered_domains.txt
142+
fi

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,3 @@
11
.claude/
2+
*.tar.gz
3+
install-test.sh

0 commit comments

Comments
 (0)