diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index f23bc42f4..6d0433fd1 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -41,15 +41,13 @@ jobs: git config --global init.defaultBranch main - name: Build and start services with Docker Compose - run: docker compose up -d --build + run: docker compose up -d --build --wait || true - - name: Wait for services to be ready + - name: Debug service state + if: always() run: | - timeout 60 bash -c ' - while [ "$(docker compose ps | grep -c "Up")" -ne 3 ]; do - sleep 2 - done - ' || { echo "Service readiness check failed:"; docker compose ps; exit 1; } + docker compose ps + docker compose logs - name: Run E2E tests run: npm run test:e2e diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 82ca651fe..3081b3762 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,62 +1,388 @@ -# GitProxy Contribution and Governance Policies +# Contributing to GitProxy -This document describes the contribution process and governance policies of the FINOS GitProxy project. The project is also governed by the [Linux Foundation Antitrust Policy](https://www.linuxfoundation.org/antitrust-policy/), and the FINOS [IP Policy](IP-Policy.pdf), [Code of Conduct](Code-of-Conduct.md), [Collaborative Principles](Collaborative-Principles.md), and [Meeting Procedures](Meeting-Procedures.md). +Thanks for your interest in contributing to GitProxy! This guide covers everything you need to get a local development environment running, understand the codebase, and submit high-quality pull requests. -## Contribution Process +For project governance, roles, and voting procedures, see the [Governance section on the website](https://git-proxy.finos.org). -Before making a contribution, please take the following steps: +## Table of Contents -1. Check whether there's already an open issue related to your proposed contribution. If there is, join the discussion and propose your contribution there. -2. If there isn't already a relevant issue, create one, describing your contribution and the problem you're trying to solve. -3. Respond to any questions or suggestions raised in the issue by other developers. -4. Fork the project repository and prepare your proposed contribution. -5. Submit a pull request. +- [Prerequisites](#prerequisites) +- [Getting Started](#getting-started) +- [Project Structure](#project-structure) +- [Development Workflow](#development-workflow) +- [Testing](#testing) + - [Unit Tests](#unit-tests) + - [End-to-End Tests](#end-to-end-tests) + - [UI Tests (Cypress)](#ui-tests-cypress) + - [Fuzz Tests](#fuzz-tests) + - [Coverage Requirements](#coverage-requirements) +- [Code Quality](#code-quality) +- [Configuration Schema](#configuration-schema) +- [Submitting a Pull Request](#submitting-a-pull-request) +- [Community](#community) -If this is your first time contributing to open source projects on GitHub, it's recommended that you -follow the [contribution guide for first-time contributors](https://github.com/firstcontributions/first-contributions#first-contributions). +## Prerequisites -NOTE: All contributors must have a contributor license agreement (CLA) on file with FINOS before their pull requests will be merged. Please review the FINOS [contribution requirements](https://finosfoundation.atlassian.net/wiki/spaces/FINOS/pages/75530375/Contribution+Compliance+Requirements) and submit (or have your employer submit) the required CLA before submitting a pull request. +| Tool | Version | Notes | +| ---------------------------------------------------------------------------------------------------------- | ------------------------------ | --------------------------- | +| [Node.js](https://nodejs.org/en/download) | 20.18.2+, 22.13.1+, or 24.0.0+ | Check with `node -v` | +| [npm](https://npmjs.com/) | 8+ | Bundled with Node.js | +| [Git](https://git-scm.com/downloads) | Any recent version | Must support HTTP/S | +| [Docker](https://docs.docker.com/get-docker/) & [Docker Compose](https://docs.docker.com/compose/install/) | Any recent version | Required for E2E tests only | -## Governance +## Getting Started -### Roles +### 1. Fork & clone -The project community consists of Contributors and Maintainers: +```bash +# Fork on GitHub, then clone your fork +git clone https://github.com//git-proxy.git +cd git-proxy +``` -- A **Contributor** is anyone who submits a contribution to the project. (Contributions may include code, issues, comments, documentation, media, or any combination of the above.) -- A **Maintainer** is a Contributor who, by virtue of their contribution history, has been given write access to project repositories and may merge approved contributions. -- The **Lead Maintainer** is the project's interface with the FINOS team and Board. They are responsible for approving [quarterly project reports](https://finosfoundation.atlassian.net/wiki/spaces/FINOS/pages/93225748/Board+Reporting+and+Program+Health+Checks) and communicating on behalf of the project. The Lead Maintainer is elected by a vote of the Maintainers. +### 2. Install dependencies -### Contribution Rules +```bash +npm install +``` -Anyone is welcome to submit a contribution to the project. The rules below apply to all contributions. (The key words "MUST", "SHALL", "SHOULD", "MAY", etc. in this document are to be interpreted as described in [IETF RFC 2119](https://www.ietf.org/rfc/rfc2119.txt).) +This installs all dependencies for the server, UI, and CLI workspace packages. [Husky](https://typicode.github.io/husky/) git hooks are configured automatically via the `prepare` script. -- All contributions MUST be submitted as pull requests, including contributions by Maintainers. -- All pull requests SHOULD be reviewed by a Maintainer (other than the Contributor) before being merged. -- Pull requests for non-trivial contributions SHOULD remain open for a review period sufficient to give all Maintainers a sufficient opportunity to review and comment on them. -- After the review period, if no Maintainer has an objection to the pull request, any Maintainer MAY merge it. -- If any Maintainer objects to a pull request, the Maintainers SHOULD try to come to consensus through discussion. If not consensus can be reached, any Maintainer MAY call for a vote on the contribution. +### 3. Run the application -### Maintainer Voting +```bash +# Run both the proxy server and the dashboard UI (recommended for development) +npm start +``` -The Maintainers MAY hold votes only when they are unable to reach consensus on an issue. Any Maintainer MAY call a vote on a contested issue, after which Maintainers SHALL have 36 hours to register their votes. Votes SHALL take the form of "+1" (agree), "-1" (disagree), "+0" (abstain). Issues SHALL be decided by the majority of votes cast. If there is only one Maintainer, they SHALL decide any issue otherwise requiring a Maintainer vote. If a vote is tied, the Lead Maintainer MAY cast an additional tie-breaker vote. +This starts two processes concurrently: -The Maintainers SHALL decide the following matters by consensus or, if necessary, a vote: +| Process | Command | URL | Description | +| ------------ | ---------------- | --------------------------------------------------------------- | ------------------------------------------- | +| Proxy server | `npm run server` | `http://localhost:8000` (proxy) / `http://localhost:8080` (API) | Express server handling git operations | +| Dashboard UI | `npm run client` | `http://localhost:3000` | Vite dev server with hot module replacement | -- Contested pull requests -- Election and removal of the Lead Maintainer -- Election and removal of Maintainers +You can also run them independently: -All Maintainer votes MUST be carried out transparently, with all discussion and voting occurring in public, either: +```bash +npm run server # Proxy server only +npm run client # Vite UI dev server only +``` -- in comments associated with the relevant issue or pull request, if applicable; -- on the project mailing list or other official public communication channel; or -- during a regular, minuted project meeting. +### 4. Verify it works -### Maintainer Qualifications +```bash +# Clone a repo through GitProxy +git clone http://localhost:8000/octocat/Hello-World.git +``` -Any Contributor who has made a substantial contribution to the project MAY apply (or be nominated) to become a Maintainer. The existing Maintainers SHALL decide whether to approve the nomination according to the Maintainer Voting process above. +By default, GitProxy blocks all pushes. To allow pushes for a specific repo, add it to `proxy.config.json`. See the [Configuration docs](https://git-proxy.finos.org/docs/category/configuration) for details. -### Changes to this Document +## Project Structure -This document MAY be amended by a vote of the Maintainers according to the Maintainer Voting process above. +``` +git-proxy/ +├── src/ +│ ├── proxy/ # Core proxy logic (action chain, processors) +│ ├── service/ # Express app, API routes, authentication (Passport.js) +│ ├── db/ # Database abstraction (MongoDB + NeDB) +│ ├── config/ # Configuration loading and generated types +│ ├── ui/ # React dashboard (Material-UI) +│ ├── plugin.ts # Plugin base classes (PushActionPlugin, PullActionPlugin) +│ └── types/ # Shared TypeScript types +├── test/ # Unit and integration tests (Vitest) +├── tests/e2e/ # End-to-end tests (Vitest + Docker) +├── cypress/ # UI tests (Cypress) +├── localgit/ # Local git server for E2E testing (see localgit/README.md) +├── packages/ +│ └── git-proxy-cli/ # CLI package +├── plugins/ # Sample plugin packages +├── website/ # Documentation site (Docusaurus) +├── index.ts # CLI entry point +├── docker-compose.yml # Docker Compose for E2E environment +├── proxy.config.json # Default proxy configuration +├── config.schema.json # JSON Schema for configuration +├── vite.config.ts # Frontend build configuration +├── vitest.config.ts # Unit test configuration +└── vitest.config.e2e.ts # E2E test configuration +``` + +### Key architectural concepts + +- **Action chain**: Git push/fetch requests flow through a chain of processors in `src/proxy/chain.ts` +- **Plugin system**: Extends the action chain with custom logic (see `src/plugin.ts`) +- **Dual database**: MongoDB for production state; [NeDB](https://github.com/seald/nedb) for local file-based development (`.data/` directory) +- **Authentication**: Passport.js strategies (local, Active Directory, OpenID Connect) + +## Development Workflow + +### Building + +```bash +npm run build # Full build: generate config types, build UI, compile TypeScript +npm run build-ts # Compile TypeScript server code to dist/ +npm run build-ui # Build React frontend with Vite to build/ +``` + +### Type checking + +```bash +npm run check-types # Type check everything (server + UI) +npm run check-types:server # Type check server code only (faster) +``` + +### Git hooks + +Husky runs the following hooks automatically: + +- **pre-commit**: `lint-staged` runs Prettier on staged files +- **commit-msg**: `@commitlint/cli` enforces [Conventional Commits](https://www.conventionalcommits.org/) format + +Commit message examples: + +``` +feat: add new OIDC authentication strategy +fix: resolve race condition in push processor +docs: update testing guide with Vitest examples +test: add fuzz tests for repo name validation +``` + +## Testing + +GitProxy has three test suites, each serving a different purpose. + +### Unit Tests + +Unit and integration tests use [Vitest](https://vitest.dev/) and are located in the `test/` directory. These do **not** require Docker. + +```bash +npm test # Run all unit tests once +npm run test-watch # Watch mode (re-runs on file changes) +npm run test-shuffle # Randomized execution order (detects test coupling) +npm run test-coverage # Run with coverage report +``` + +Configuration: [vitest.config.ts](vitest.config.ts) + +Test files are organized by module: + +``` +test/ +├── processors/ # Proxy processor logic +├── db/ # Database operations +├── services/ # API and service tests +├── integration/ # Cross-module integration tests +├── plugin/ # Plugin system tests +├── preReceive/ # Git hook tests +└── fixtures/ # Binary test data for protocol-level tests +``` + +### End-to-End Tests + +E2E tests perform real git operations against a Dockerized environment. They use Vitest with a separate config. + +**Prerequisites**: Docker and Docker Compose must be running. + +```bash +# Run E2E tests (builds containers, runs tests, tears down) +npm run test:e2e + +# Watch mode for E2E development +npm run test:e2e:watch +``` + +Configuration: [vitest.config.e2e.ts](vitest.config.e2e.ts) + +#### Docker Compose environment + +The E2E environment is defined in [docker-compose.yml](docker-compose.yml) and consists of three services: + +| Service | Port | Description | +| ------------ | ---------- | ------------------------------------------------------------------------- | +| `git-proxy` | 8000, 8081 | GitProxy application under test | +| `mongodb` | 27017 | MongoDB 7 instance | +| `git-server` | 8443 | Apache-based git HTTP server with test repos (see [localgit/](localgit/)) | + +All services run in an isolated `git-network` Docker bridge network. + +#### Managing the environment manually + +When developing or debugging E2E tests, you'll often want to keep the containers running between test runs rather than letting the test script tear them down: + +```bash +# Start all services in the background +docker compose up -d + +# Verify all three containers are running +docker compose ps + +# Rebuild from scratch (e.g., after changing localgit/ or Dockerfile) +docker compose down -v +docker compose build --no-cache +docker compose up -d +``` + +#### Test repositories and credentials + +The git server is initialized with two test repos: + +| Repository | Path | +| -------------------------- | ------------------------------------------------------- | +| `test-owner/test-repo.git` | Simple test repo with a README and text file | +| `e2e-org/sample-repo.git` | Sample project with a README, package.json, and LICENSE | + +Two users are pre-configured: + +| Username | Password | Purpose | +| ---------- | ---------- | ------------------------- | +| `admin` | `admin123` | Full access to all repos | +| `testuser` | `user123` | Standard user for testing | + +#### Interacting with test repos + +```bash +# Clone directly from the git server +git clone http://admin:admin123@localhost:8443/test-owner/test-repo.git + +# Clone through GitProxy +git clone http://admin:admin123@localhost:8000/test-owner/test-repo.git + +# Push a change +cd test-repo +echo "test" > test.txt +git add test.txt +git commit -m "test commit" +git push origin main +``` + +#### Viewing logs + +```bash +docker compose logs -f git-proxy # GitProxy application logs +docker compose logs -f git-server # Apache git server logs +docker compose logs -f mongodb # MongoDB logs +``` + +#### Troubleshooting + +If services won't start or tests fail unexpectedly: + +```bash +# Check service status +docker compose ps + +# View logs for the failing service +docker compose logs git-server + +# Nuclear option: tear down everything and rebuild +docker compose down -v +docker compose build --no-cache +docker compose up -d +``` + +If MongoDB connections fail: + +```bash +docker compose exec mongodb mongosh --eval "db.adminCommand('ping')" +``` + +#### Generating test fixtures + +The git server includes a data capture system that records raw git protocol data for every operation. This is useful for creating binary test fixtures (e.g., PACK files) for unit tests. See [localgit/README.md](localgit/README.md) for details on the capture system, PACK extraction tools, and fixture generation workflow. + +### UI Tests (Cypress) + +[Cypress](https://docs.cypress.io) tests exercise the dashboard UI end-to-end. + +```bash +# Start the app first +npm start + +# Then, in another terminal: +npm run cypress:open # Interactive test runner (recommended for development) +npm run cypress:run # Headless mode (used in CI) +``` + +Configuration: [cypress.config.js](cypress.config.js) + +Cypress tests live in `cypress/e2e/` and use custom commands defined in `cypress/support/commands.js` (e.g., `cy.login(username, password)`). + +### Fuzz Tests + +Some test files include fuzz tests using [fast-check](https://fast-check.dev/) to find edge-case bugs with randomized inputs. These run as part of the regular unit test suite (`npm test`). + +### Coverage Requirements + +All new code introduced in a PR **must have over 80% patch coverage**. This is enforced by [CodeCov](https://app.codecov.io/gh/finos/git-proxy) in CI. + +```bash +# Generate a local coverage report +npm run test-coverage +``` + +The coverage report is written to `./coverage/`. If your PR is below the threshold, check the CodeCov report on your PR for uncovered lines. + +## Code Quality + +```bash +npm run lint # Run ESLint +npm run lint:fix # ESLint with auto-fix +npm run format # Format all files with Prettier +npm run format:check # Check formatting without modifying files +``` + +CI runs ESLint, Prettier, and TypeScript type checks on every PR (see [`.github/workflows/lint.yml`](.github/workflows/lint.yml)). + +## Configuration Schema + +GitProxy uses a JSON Schema ([config.schema.json](config.schema.json)) to define and validate configuration. When adding or modifying config properties: + +1. Update `config.schema.json` with the new/changed properties +2. Regenerate TypeScript types: + ```bash + npm run generate-config-types + ``` +3. Regenerate the schema reference documentation for the website: + ```bash + # Requires Python and json-schema-for-humans: + # pip install json-schema-for-humans + npm run gen-schema-doc + ``` + +## Submitting a Pull Request + +1. **Check for existing issues**: Search [open issues](https://github.com/finos/git-proxy/issues) before starting work. If none exists, [create one](https://github.com/finos/git-proxy/issues/new) describing the change. +2. **Fork & branch**: Create a feature branch from `main` (e.g., `feat/my-feature` or `fix/my-bugfix`). +3. **Make your changes**: Follow the code style enforced by ESLint and Prettier. Write tests for new functionality. +4. **Verify locally**: + ```bash + npm run check-types:server # Type check + npm test # Unit tests + npm run lint # Lint + npm run format:check # Formatting + ``` +5. **Commit using [Conventional Commits](https://www.conventionalcommits.org/)**: The commit-msg hook validates this automatically. +6. **Push & open a PR**: Target the `main` branch. Fill in the PR template and link the relevant issue. + +### CI checks on your PR + +The following checks must pass before a PR can be merged: + +- **Unit tests**: Run across a matrix of Node.js (20, 22, 24) and MongoDB (6.0, 7.0, 8.0) versions on Ubuntu, plus a Windows build +- **E2E tests**: Docker-based end-to-end tests +- **Cypress tests**: UI end-to-end tests +- **Lint & format**: ESLint, Prettier, TypeScript type checks +- **Commit lint**: Conventional Commits validation +- **Coverage**: 80%+ patch coverage via CodeCov +- **Security**: CodeQL analysis, dependency review, OpenSSF Scorecard + +### Contributor License Agreement (CLA) + +All contributors must have a CLA on file with FINOS before PRs can be merged. Review the FINOS [contribution requirements](https://finosfoundation.atlassian.net/wiki/spaces/FINOS/pages/75530375/Contribution+Compliance+Requirements) and submit the required CLA. + +## Community + +- **Slack**: [#git-proxy](https://finos-lf.slack.com/archives/C06LXNW0W76) on the FINOS Slack workspace +- **Mailing list**: [git-proxy+subscribe@lists.finos.org](mailto:git-proxy+subscribe@lists.finos.org) +- **Community meetings**: Fortnightly on Mondays at 4PM BST (odd week numbers) via [Zoom](https://zoom-lfx.platform.linuxfoundation.org/meeting/95849833904?password=99413314-d03a-4b1c-b682-1ede2c399595). [Add to Google Calendar](https://calendar.google.com/calendar/event?action=TEMPLATE&tmeid=MTRvbzM0NG01dWNvNGc4OGJjNWphM2ZtaTZfMjAyNTA2MDJUMTUwMDAwWiBzYW0uaG9sbWVzQGNvbnRyb2wtcGxhbmUuaW8&tmsrc=sam.holmes%40control-plane.io&scp=ALL). +- **Issues**: [github.com/finos/git-proxy/issues](https://github.com/finos/git-proxy/issues) diff --git a/docker-compose.yml b/docker-compose.yml index 27157df0c..a81f25c60 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -10,8 +10,10 @@ services: # If using Podman, you might need to add the :Z or :z option for SELinux # - ./test-e2e.proxy.config.json:/app/test-e2e.proxy.config.json:ro,Z depends_on: - - mongodb - - git-server + mongodb: + condition: service_healthy + git-server: + condition: service_healthy networks: - git-network environment: @@ -30,6 +32,12 @@ services: # - Comma-separated list = 'http://localhost:3000,https://example.com' # - Unset/empty = Same-origin only (most secure) - ALLOWED_ORIGINS= + healthcheck: + test: ['CMD-SHELL', 'curl -sf http://localhost:8081/api/v1/healthcheck || exit 1'] + interval: 5s + timeout: 5s + retries: 12 + start_period: 10s mongodb: image: mongo:7@sha256:606f8e029603330411a7dd10b5ffd50eefc297fc80cee89f10a455e496a76ae7 ports: @@ -40,6 +48,12 @@ services: - MONGO_INITDB_DATABASE=gitproxy volumes: - mongodb_data:/data/db + healthcheck: + test: ['CMD', 'mongosh', '--eval', "db.adminCommand('ping')"] + interval: 5s + timeout: 5s + retries: 12 + start_period: 5s git-server: build: localgit/ @@ -50,6 +64,16 @@ services: networks: - git-network hostname: git-server + healthcheck: + test: + [ + 'CMD-SHELL', + 'GIT_TERMINAL_PROMPT=0 GIT_SSL_NO_VERIFY=1 git ls-remote https://admin:admin123@localhost:8443/test-owner/test-repo.git HEAD || exit 1', + ] + interval: 5s + timeout: 5s + retries: 12 + start_period: 5s networks: git-network: diff --git a/localgit/README.md b/localgit/README.md index e6f451f6b..2db89ac8e 100644 --- a/localgit/README.md +++ b/localgit/README.md @@ -1,809 +1,89 @@ -# Local Git Server for End-to-End Testing +# Local Git Server -This directory contains a complete end-to-end testing environment for GitProxy, including: +This directory contains the local git HTTP server used by GitProxy's end-to-end test suite. It provides an isolated environment for testing real git operations without requiring external services. -- **Local Git HTTP Server**: Apache-based git server with test repositories -- **MongoDB Instance**: Database for GitProxy state management -- **GitProxy Server**: Configured to proxy requests to the local git server -- **Data Capture System**: Captures raw git protocol data for low-level testing +For instructions on running E2E tests, managing the Docker environment, and interacting with test repositories, see the [End-to-End Tests](../CONTRIBUTING.md#end-to-end-tests) section of CONTRIBUTING.md. -## Table of Contents +## What it does -- [Overview](#overview) -- [Quick Start](#quick-start) -- [Architecture](#architecture) -- [Test Repositories](#test-repositories) -- [Basic Usage](#basic-usage) -- [Advanced Use](#advanced-use) - - [Capturing Git Protocol Data](#capturing-git-protocol-data) - - [Extracting PACK Files](#extracting-pack-files) - - [Generating Test Fixtures](#generating-test-fixtures) - - [Debugging PACK Parsing](#debugging-pack-parsing) -- [Configuration](#configuration) -- [Troubleshooting](#troubleshooting) -- [Commands Reference](#commands-reference) +The git server is an Apache HTTP container that serves bare git repositories over HTTP via `git-http-backend`. A Python CGI wrapper (`git-capture-wrapper.py`) sits in front of the git backend to capture raw protocol data for every operation. ---- - -## Overview - -This testing setup provides an isolated environment for developing and testing GitProxy without requiring external git services. It's particularly useful for: - -1. **Integration Testing**: Full end-to-end tests with real git operations -2. **Protocol Analysis**: Capturing and analyzing git HTTP protocol data -3. **Test Fixture Generation**: Creating binary test data from real git operations -4. **Low-Level Debugging**: Extracting and inspecting PACK files for parser development - -### How It Fits Into the Codebase - -``` -git-proxy/ -├── src/ # GitProxy source code -├── test/ # Unit and integration tests -│ ├── fixtures/ # Test data (can be generated from captures) -│ └── integration/ # Integration tests using this setup -├── tests/e2e/ # End-to-end tests -├── localgit/ # THIS DIRECTORY -│ ├── Dockerfile # Git server container definition -│ ├── docker-compose.yml # Full test environment orchestration -│ ├── init-repos.sh # Creates test repositories -│ ├── git-capture-wrapper.py # Captures git protocol data -│ ├── extract-captures.sh # Extracts captures from container -│ └── extract-pack.py # Extracts PACK files from captures -└── docker-compose.yml # References localgit/ for git-server service -``` - ---- - -## Quick Start - -### 1. Start the Test Environment - -```bash -# From the project root -docker compose up -d - -# This starts: -# - git-server (port 8080) -# - mongodb (port 27017) -# - git-proxy (ports 8000, 8081) ``` - -### 2. Verify Services - -```bash -# Check all services are running -docker compose ps - -# Should show: -# - git-proxy (git-proxy service) -# - mongodb (database) -# - git-server (local git HTTP server) +Git CLI + │ HTTP + ▼ +GitProxy (optional, port 8000) + │ + ▼ +Apache HTTP Server (git-server) + │ CGI + ▼ +git-capture-wrapper.py ──► saves request/response to /var/git-captures + │ + ▼ +git-http-backend + │ + ▼ +Bare git repositories (/var/git/owner/repo.git) ``` -### 3. Test Git Operations - -```bash -# Clone a test repository -git clone http://admin:admin123@localhost:8080/coopernetes/test-repo.git -cd test-repo - -# Make changes -echo "Test data $(date)" > test-file.txt -git add test-file.txt -git commit -m "Test commit" - -# Push (this will be captured automatically) -git push origin main -``` - -### 4. Test Through GitProxy - -```bash -# Clone through the proxy (port 8000) -git clone http://admin:admin123@localhost:8000/coopernetes/test-repo.git -``` - ---- - -## Architecture - -### Component Diagram - -``` -┌─────────────┐ -│ Git CLI │ -└──────┬──────┘ - │ HTTP (port 8080 or 8000) - ▼ -┌─────────────────────────┐ -│ GitProxy (optional) │ ← Port 8000 (proxy) -│ - Authorization │ ← Port 8081 (UI) -│ - Logging │ -│ - Policy enforcement │ -└──────┬──────────────────┘ - │ - ▼ -┌─────────────────────────┐ -│ Apache HTTP Server │ ← Port 8080 (direct) -│ (git-server) │ -└──────┬──────────────────┘ - │ CGI - ▼ -┌──────────────────────────────────┐ -│ git-capture-wrapper.py │ -│ ├─ Capture request body │ -│ ├─ Save to /var/git-captures │ -│ ├─ Forward to git-http-backend │ -│ └─ Capture response │ -└──────┬───────────────────────────┘ - │ - ▼ -┌─────────────────────────┐ -│ git-http-backend │ -│ (actual git processing)│ -└──────┬──────────────────┘ - │ - ▼ -┌─────────────────────────┐ -│ Git Repositories │ -│ /var/git/owner/repo.git│ -└─────────────────────────┘ -``` - -### Network Configuration - -All services run in the `git-network` Docker network: - -- **git-server**: Hostname `git-server`, accessible at `http://git-server:8080` internally -- **mongodb**: Hostname `mongodb`, accessible at `mongodb://mongodb:27017` internally -- **git-proxy**: Hostname `git-proxy`, accessible at `http://git-proxy:8000` internally - -External access: - -- Git Server: `http://localhost:8080` -- GitProxy: `http://localhost:8000` (git operations), `http://localhost:8081` (UI) -- MongoDB: `localhost:27017` - ---- - -## Test Repositories - -The git server is initialized with test repositories in the following structure: - -``` -/var/git/ -├── coopernetes/ -│ └── test-repo.git # Simple test repository -└── finos/ - └── git-proxy.git # Simulates the GitProxy project -``` - -### Authentication - -Basic authentication is configured with two users: - -| Username | Password | Purpose | -| ---------- | ---------- | ------------------------- | -| `admin` | `admin123` | Full access to all repos | -| `testuser` | `user123` | Standard user for testing | - -### Repository Contents - -**coopernetes/test-repo.git**: - -- `README.md`: Simple test repository description -- `hello.txt`: Basic text file - -**finos/git-proxy.git**: - -- `README.md`: GitProxy project description -- `package.json`: Simulated project structure -- `LICENSE`: Apache 2.0 license - ---- - -## Basic Usage - -### Cloning Repositories - -```bash -# Direct from git-server -git clone http://admin:admin123@localhost:8080/coopernetes/test-repo.git - -# Through GitProxy -git clone http://admin:admin123@localhost:8000/coopernetes/test-repo.git -``` - -### Push and Pull Operations - -```bash -cd test-repo - -# Make changes -echo "New content" > newfile.txt -git add newfile.txt -git commit -m "Add new file" - -# Push -git push origin main - -# Pull -git pull origin main -``` - -### Viewing Logs - -```bash -# GitProxy logs -docker compose logs -f git-proxy - -# Git server logs -docker compose logs -f git-server +## Files -# MongoDB logs -docker compose logs -f mongodb -``` - ---- - -## Advanced Use - -### Capturing Git Protocol Data - -The git server automatically captures raw HTTP request/response data for all git operations. This is invaluable for: - -- Creating test fixtures for unit tests -- Debugging protocol-level issues -- Understanding git's wire protocol -- Testing PACK file parsers - -#### How Data Capture Works - -The `git-capture-wrapper.py` CGI script intercepts all git HTTP requests: - -1. **Captures request body** (e.g., PACK file during push) -2. **Forwards to git-http-backend** (actual git processing) -3. **Captures response** (e.g., unpack status) -4. **Saves three files** per operation: - - `.request.bin`: Raw HTTP request body (binary) - - `.response.bin`: Raw HTTP response (binary) - - `.metadata.txt`: Human-readable metadata +| File | Purpose | +| ------------------------ | ------------------------------------------------------------- | +| `Dockerfile` | Defines the git-server container (Apache, git, Python) | +| `httpd.conf` | Apache configuration for git HTTP backend and CGI | +| `init-repos.sh` | Creates and populates test repositories on container startup | +| `git-capture-wrapper.py` | CGI wrapper that captures git protocol request/response data | +| `extract-captures.sh` | Extracts captured data from the running container to the host | +| `extract-pack.py` | Extracts PACK files from captured request binaries | +| `generate-cert.sh` | Generates self-signed HTTPS certificates for the server | -#### Captured File Format +## Data capture system -**Filename Pattern**: `{timestamp}-{service}-{repo}.{type}.{ext}` +The capture wrapper records three files per git operation into `/var/git-captures/` inside the container: -Example: `20251001-185702-925704-receive-pack-_coopernetes_test-repo.request.bin` +| File | Contents | +| ---------------- | ---------------------------------------------------------- | +| `*.request.bin` | Raw HTTP request body (includes PACK data for pushes) | +| `*.response.bin` | Raw HTTP response | +| `*.metadata.txt` | Human-readable metadata (timestamp, service, paths, sizes) | -- **timestamp**: `YYYYMMDD-HHMMSS-microseconds` -- **service**: `receive-pack` (push) or `upload-pack` (fetch/pull) -- **repo**: Repository path with slashes replaced by underscores +Filename pattern: `{YYYYMMDD}-{HHMMSS}-{microseconds}-{service}-{repo}.{type}.{ext}` -#### Extracting Captures +### Extracting captures and PACK files ```bash cd localgit -# Extract all captures to a local directory +# Copy all captures from the container to a local directory ./extract-captures.sh ./captured-data -# View what was captured -ls -lh ./captured-data/ - -# Read metadata -cat ./captured-data/*.metadata.txt -``` - -**Example Metadata**: - -``` -Timestamp: 2025-10-01T18:57:02.925894 -Service: receive-pack -Request Method: POST -Path Info: /coopernetes/test-repo.git/git-receive-pack -Content Type: application/x-git-receive-pack-request -Content Length: 711 -Request Body Size: 711 bytes -Response Size: 216 bytes -Exit Code: 0 -``` - -### Extracting PACK Files - -The `.request.bin` file for a push operation contains: - -1. **Pkt-line commands**: Ref updates in git's pkt-line format -2. **Flush packet**: `0000` marker -3. **PACK data**: Binary PACK file starting with "PACK" signature - -The `extract-pack.py` script extracts just the PACK portion: - -```bash -# Extract PACK from captured request +# Extract the PACK portion from a push request capture ./extract-pack.py ./captured-data/*receive-pack*.request.bin output.pack -# Output: -# Found PACK data at offset 173 -# PACK signature: b'PACK' -# PACK version: 2 -# Number of objects: 3 -# PACK size: 538 bytes -``` - -#### Working with Extracted PACK Files - -```bash -# Index the PACK file (required before verify) +# Verify with git git index-pack output.pack - -# Verify the PACK file git verify-pack -v output.pack - -# Output shows objects: -# 95fbb70... commit 432 313 12 -# 8c028ba... tree 44 55 325 -# a0b4110... blob 47 57 380 -# non delta: 3 objects -# output.pack: ok - -# Unpack objects to inspect -git unpack-objects < output.pack ``` -### Generating Test Fixtures - -Use captured data to create test fixtures for your test suite: +### Generating test fixtures -#### Workflow +Captured data can be copied into `test/fixtures/` for use in unit tests: ```bash -# 1. Perform a specific git operation -git clone http://admin:admin123@localhost:8080/coopernetes/test-repo.git -cd test-repo -# ... create specific test scenario ... -git push - +# 1. Perform a git operation against the running environment # 2. Extract the capture -cd ../localgit -./extract-captures.sh ./test-scenario-captures +./extract-captures.sh ./my-captures # 3. Copy to test fixtures -cp ./test-scenario-captures/*receive-pack*.request.bin \ - ../test/fixtures/my-test-scenario.bin - -# 4. Use in tests -# test/mytest.js: -# const fs = require('fs'); -# const testData = fs.readFileSync('./fixtures/my-test-scenario.bin'); -# const result = await parsePush(testData); -``` - -#### Example: Creating a Force-Push Test Fixture - -```bash -# Create a force-push scenario -git clone http://admin:admin123@localhost:8080/coopernetes/test-repo.git -cd test-repo -git reset --hard HEAD~1 -echo "force push test" > force.txt -git add force.txt -git commit -m "Force push test" -git push --force origin main - -# Extract and save -cd ../localgit -./extract-captures.sh ./force-push-capture -cp ./force-push-capture/*receive-pack*.request.bin \ - ../test/fixtures/force-push.bin -``` - -### Debugging PACK Parsing - -When developing or debugging PACK file parsers: - -#### Compare Your Parser with Git's - -```bash -# 1. Extract captures -./extract-captures.sh ./debug-data - -# 2. Extract PACK -./extract-pack.py ./debug-data/*receive-pack*.request.bin debug.pack - -# 3. Use git to verify expected output -git index-pack debug.pack -git verify-pack -v debug.pack > expected-objects.txt - -# 4. Run your parser -node -e " -const fs = require('fs'); -const data = fs.readFileSync('./debug-data/*receive-pack*.request.bin'); -// Your parsing code -const result = myPackParser(data); -console.log(JSON.stringify(result, null, 2)); -" > my-parser-output.txt - -# 5. Compare -diff expected-objects.txt my-parser-output.txt -``` - -#### Inspect Binary Data - -```bash -# View hex dump of request -hexdump -C ./captured-data/*.request.bin | head -50 - -# Find PACK signature -grep -abo "PACK" ./captured-data/*.request.bin - -# Extract pkt-line commands (before PACK) -head -c 173 ./captured-data/*.request.bin | hexdump -C -``` - -#### Use in Node.js Tests - -```javascript -const fs = require('fs'); - -// Read captured data -const capturedData = fs.readFileSync( - './captured-data/20250101-120000-receive-pack-test-repo.request.bin', -); - -console.log('Total size:', capturedData.length, 'bytes'); - -// Find PACK offset -const packIdx = capturedData.indexOf(Buffer.from('PACK')); -console.log('PACK starts at offset:', packIdx); - -// Extract PACK header -const packHeader = capturedData.slice(packIdx, packIdx + 12); -console.log('PACK header:', packHeader.toString('hex')); - -// Parse PACK version and object count -const version = packHeader.readUInt32BE(4); -const numObjects = packHeader.readUInt32BE(8); -console.log(`PACK v${version}, ${numObjects} objects`); - -// Test your parser -const result = await myPackParser(capturedData); -assert.equal(result.objectCount, numObjects); -``` - ---- - -## Configuration - -### Enable/Disable Data Capture - -Edit `docker-compose.yml`: - -```yaml -git-server: - environment: - - GIT_CAPTURE_ENABLE=1 # 1 to enable, 0 to disable -``` - -Then restart: - -```bash -docker compose restart git-server -``` - -### Add More Test Repositories - -Edit `localgit/init-repos.sh` to add more repositories: - -```bash -# Add a new owner -OWNERS=("owner1" "owner2" "newowner") - -# Create a new repository -create_bare_repo "newowner" "new-repo.git" -add_content_to_repo "newowner" "new-repo.git" - -# Add content... -cat > README.md << 'EOF' -# New Test Repository -EOF - -git add . -git commit -m "Initial commit" -git push origin main -``` - -Rebuild the container: - -```bash -docker compose down -docker compose build --no-cache git-server -docker compose up -d +cp ./my-captures/*receive-pack*.request.bin ../test/fixtures/my-scenario.bin ``` -### Modify Apache Configuration - -Edit `localgit/httpd.conf` to change Apache settings (authentication, CGI, etc.). - -### Change MongoDB Configuration - -Edit `docker-compose.yml` to modify MongoDB settings: - -```yaml -mongodb: - environment: - - MONGO_INITDB_DATABASE=gitproxy - - MONGO_INITDB_ROOT_USERNAME=admin # Optional - - MONGO_INITDB_ROOT_PASSWORD=secret # Optional -``` - ---- - -## Troubleshooting - -### Services Won't Start - -```bash -# Check service status -docker compose ps - -# View logs -docker compose logs git-server -docker compose logs mongodb -docker compose logs git-proxy - -# Rebuild from scratch -docker compose down -v -docker compose build --no-cache -docker compose up -d -``` - -### Git Operations Fail - -```bash -# Check git-server logs -docker compose logs git-server - -# Test git-http-backend directly -docker compose exec git-server /usr/lib/git-core/git-http-backend - -# Verify repository permissions -docker compose exec git-server ls -la /var/git/coopernetes/ -``` - -### No Captures Created - -```bash -# Verify capture is enabled -docker compose exec git-server env | grep GIT_CAPTURE - -# Check capture directory permissions -docker compose exec git-server ls -ld /var/git-captures - -# Should be: drwxr-xr-x www-data www-data - -# Check wrapper is executable -docker compose exec git-server ls -l /usr/local/bin/git-capture-wrapper.py - -# View Apache error logs -docker compose logs git-server | grep -i error -``` - -### Permission Errors - -```bash -# Fix capture directory permissions -docker compose exec git-server chown -R www-data:www-data /var/git-captures - -# Fix repository permissions -docker compose exec git-server chown -R www-data:www-data /var/git -``` - -### Clone Shows HEAD Warnings - -This has been fixed in the current version. If you see warnings: - -```bash -# Rebuild with latest init-repos.sh -docker compose down -docker compose build --no-cache git-server -docker compose up -d -``` - -The fix ensures repositories are created with `--initial-branch=main` and HEAD is explicitly set to `refs/heads/main`. - -### MongoDB Connection Issues - -```bash -# Check MongoDB is running -docker compose ps mongodb - -# Test connection -docker compose exec mongodb mongosh --eval "db.adminCommand('ping')" - -# Check GitProxy can reach MongoDB -docker compose exec git-proxy ping -c 3 mongodb -``` - ---- - -## Commands Reference - -### Container Management - -```bash -# Start all services -docker compose up -d - -# Stop all services -docker compose down - -# Rebuild a specific service -docker compose build --no-cache git-server - -# View logs -docker compose logs -f git-proxy -docker compose logs -f git-server -docker compose logs -f mongodb - -# Restart a service -docker compose restart git-server - -# Execute command in container -docker compose exec git-server bash -``` - -### Data Capture Operations - -```bash -# Extract captures from container -cd localgit -./extract-captures.sh ./captured-data - -# Extract PACK file -./extract-pack.py ./captured-data/*receive-pack*.request.bin output.pack - -# Verify PACK file -git index-pack output.pack -git verify-pack -v output.pack - -# Clear captures in container -docker compose exec git-server rm -f /var/git-captures/* - -# View captures in container -docker compose exec git-server ls -lh /var/git-captures/ - -# Count captures -docker compose exec git-server sh -c "ls -1 /var/git-captures/*.bin | wc -l" -``` - -### Git Operations - -```bash -# Clone directly from git-server -git clone http://admin:admin123@localhost:8080/coopernetes/test-repo.git - -# Clone through GitProxy -git clone http://admin:admin123@localhost:8000/coopernetes/test-repo.git - -# Push changes -cd test-repo -echo "test" > test.txt -git add test.txt -git commit -m "test" -git push origin main - -# Force push -git push --force origin main - -# Fetch -git fetch origin - -# Pull -git pull origin main -``` - -### Repository Management - -```bash -# List repositories in container -docker compose exec git-server ls -la /var/git/coopernetes/ -docker compose exec git-server ls -la /var/git/finos/ - -# View repository config -docker compose exec git-server git -C /var/git/coopernetes/test-repo.git config -l - -# Reset a repository (careful!) -docker compose exec git-server rm -rf /var/git/coopernetes/test-repo.git -docker compose restart git-server # Will reinitialize -``` - -### MongoDB Operations - -```bash -# Connect to MongoDB shell -docker compose exec mongodb mongosh gitproxy - -# View collections -docker compose exec mongodb mongosh gitproxy --eval "db.getCollectionNames()" - -# Clear database (careful!) -docker compose exec mongodb mongosh gitproxy --eval "db.dropDatabase()" -``` - ---- - -## File Reference - -### Core Files - -| File | Purpose | -| ------------------------ | ------------------------------------------------------------- | -| `Dockerfile` | Defines the git-server container with Apache, git, and Python | -| `httpd.conf` | Apache configuration for git HTTP backend and CGI | -| `init-repos.sh` | Creates test repositories on container startup | -| `git-capture-wrapper.py` | CGI wrapper that captures git protocol data | -| `extract-captures.sh` | Helper script to extract captures from container | -| `extract-pack.py` | Extracts PACK files from captured request data | - -### Generated Files - -| File | Description | -| ---------------- | --------------------------------------------- | -| `*.request.bin` | Raw HTTP request body (PACK files for pushes) | -| `*.response.bin` | Raw HTTP response (unpack status for pushes) | -| `*.metadata.txt` | Human-readable capture metadata | - ---- - -## Use Cases Summary - -### 1. Integration Testing - -Run full end-to-end tests with real git operations against a local server. - -### 2. Generate Test Fixtures - -Capture real git operations to create binary test data for unit tests. - -### 3. Debug PACK Parsing - -Extract PACK files and compare your parser output with git's official tools. - -### 4. Protocol Analysis - -Study the git HTTP protocol by examining captured request/response data. - -### 5. Regression Testing - -Capture problematic operations for reproduction and regression testing. - -### 6. Development Workflow - -Develop GitProxy features without requiring external git services. - ---- - -## Status - -✅ **All systems operational and validated** (as of 2025-10-01) - -- Docker containers build and run successfully -- Test repositories initialized with proper HEAD references -- Git clone, push, and pull operations work correctly -- Data capture system functioning properly -- PACK extraction and verification working -- Integration with Node.js test suite confirmed - ---- - -## Additional Resources +## Customization -- **Git HTTP Protocol**: https://git-scm.com/docs/http-protocol -- **Git Pack Format**: https://git-scm.com/docs/pack-format -- **Git Plumbing Commands**: https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain -- **GitProxy Documentation**: `../website/docs/` +**Add repositories**: Edit `init-repos.sh`, then rebuild (`docker compose build --no-cache git-server`). ---- +**Toggle data capture**: Set `GIT_CAPTURE_ENABLE=0` in `docker-compose.yml` under `git-server.environment` to disable. -**For questions or issues with this testing setup, please refer to the main project documentation or open an issue.** +**Modify Apache**: Edit `httpd.conf` for authentication, CGI, or other server changes. diff --git a/localgit/init-repos.sh b/localgit/init-repos.sh index 502d26dd1..d5d5d54c8 100644 --- a/localgit/init-repos.sh +++ b/localgit/init-repos.sh @@ -3,7 +3,7 @@ set -e # Exit on any error # Create the git repositories directories for multiple owners BASE_DIR="${BASE_DIR:-"/var/git"}" -OWNERS=("coopernetes" "finos") +OWNERS=("test-owner" "e2e-org") TEMP_DIR="/tmp/git-init" # Create base directory and owner subdirectories @@ -27,11 +27,11 @@ create_bare_repo() { local owner="$1" local repo_name="$2" local repo_dir="$BASE_DIR/$owner" - + echo "Creating $repo_name in $owner's directory..." cd "$repo_dir" || exit 1 git init --bare --initial-branch=main "$repo_name" - + # Configure for HTTP access cd "$repo_dir/$repo_name" || exit 1 git config http.receivepack true @@ -47,7 +47,7 @@ add_content_to_repo() { local repo_name="$2" local repo_path="$BASE_DIR/$owner/$repo_name" local work_dir="$TEMP_DIR/${owner}-${repo_name%-.*}-work" - + echo "Adding content to $owner/$repo_name..." cd "$TEMP_DIR" || exit 1 git clone "$repo_path" "$work_dir" @@ -55,15 +55,15 @@ add_content_to_repo() { } # Create repositories with simple content -echo "=== Creating coopernetes/test-repo.git ===" -create_bare_repo "coopernetes" "test-repo.git" -add_content_to_repo "coopernetes" "test-repo.git" +echo "=== Creating test-owner/test-repo.git ===" +create_bare_repo "test-owner" "test-repo.git" +add_content_to_repo "test-owner" "test-repo.git" # Create a simple README cat > README.md << 'EOF' # Test Repository -This is a test repository for the git proxy, simulating coopernetes/test-repo. +A dummy repository used for GitProxy end-to-end testing. EOF # Create a simple text file @@ -75,29 +75,29 @@ git add . git commit -m "Initial commit with basic content" git push origin main -echo "=== Creating finos/git-proxy.git ===" -create_bare_repo "finos" "git-proxy.git" -add_content_to_repo "finos" "git-proxy.git" +echo "=== Creating e2e-org/sample-repo.git ===" +create_bare_repo "e2e-org" "sample-repo.git" +add_content_to_repo "e2e-org" "sample-repo.git" # Create a simple README cat > README.md << 'EOF' -# Git Proxy +# Sample Repository -This is a test instance of the FINOS Git Proxy project for isolated e2e testing. +A dummy repository used for GitProxy end-to-end testing. EOF -# Create a simple package.json to simulate the real project structure +# Create a simple package.json to simulate a project structure cat > package.json << 'EOF' { - "name": "git-proxy", + "name": "sample-repo", "version": "1.0.0", - "description": "A proxy for Git operations", + "description": "A sample project for e2e testing", "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, - "keywords": ["git", "proxy", "finos"], - "author": "FINOS", + "keywords": ["git", "proxy", "test"], + "author": "Test", "license": "Apache-2.0" } EOF @@ -146,4 +146,4 @@ done echo "Successfully initialized Git repositories in $BASE_DIR" echo "Owners created: ${OWNERS[*]}" -echo "Total repositories: $(find $BASE_DIR -name "*.git" -type d | wc -l)" \ No newline at end of file +echo "Total repositories: $(find $BASE_DIR -name "*.git" -type d | wc -l)" diff --git a/test-e2e.proxy.config.json b/test-e2e.proxy.config.json index ccf0926f4..8258f59b6 100644 --- a/test-e2e.proxy.config.json +++ b/test-e2e.proxy.config.json @@ -11,14 +11,14 @@ }, "authorisedList": [ { - "project": "coopernetes", + "project": "test-owner", "name": "test-repo", - "url": "https://git-server:8443/coopernetes/test-repo.git" + "url": "https://git-server:8443/test-owner/test-repo.git" }, { - "project": "finos", - "name": "git-proxy", - "url": "https://git-server:8443/finos/git-proxy.git" + "project": "e2e-org", + "name": "sample-repo", + "url": "https://git-server:8443/e2e-org/sample-repo.git" } ], "sink": [ diff --git a/tests/e2e/fetch.test.ts b/tests/e2e/fetch.test.ts index e08678154..a6c6bb923 100644 --- a/tests/e2e/fetch.test.ts +++ b/tests/e2e/fetch.test.ts @@ -37,17 +37,17 @@ describe('Git Proxy E2E - Repository Fetch Tests', () => { describe('Repository fetching through git proxy', () => { it( - 'should successfully fetch coopernetes/test-repo through git proxy', + 'should successfully fetch test-owner/test-repo through git proxy', async () => { // Build URL with embedded credentials for reliable authentication const baseUrl = new URL(testConfig.gitProxyUrl); baseUrl.username = testConfig.gitUsername; baseUrl.password = testConfig.gitPassword; - const repoUrl = `${baseUrl.toString()}/coopernetes/test-repo.git`; + const repoUrl = `${baseUrl.toString()}/test-owner/test-repo.git`; const cloneDir: string = path.join(tempDir, 'test-repo-clone'); console.log( - `[TEST] Cloning ${testConfig.gitProxyUrl}/coopernetes/test-repo.git to ${cloneDir}`, + `[TEST] Cloning ${testConfig.gitProxyUrl}/test-owner/test-repo.git to ${cloneDir}`, ); try { @@ -73,7 +73,7 @@ describe('Git Proxy E2E - Repository Fetch Tests', () => { const readmePath: string = path.join(cloneDir, 'README.md'); expect(fs.existsSync(readmePath)).toBe(true); - console.log('[TEST] Successfully fetched and verified coopernetes/test-repo'); + console.log('[TEST] Successfully fetched and verified test-owner/test-repo'); } catch (error) { console.error('[TEST] Failed to clone repository:', error); throw error; @@ -83,16 +83,18 @@ describe('Git Proxy E2E - Repository Fetch Tests', () => { ); it( - 'should successfully fetch finos/git-proxy through git proxy', + 'should successfully fetch e2e-org/sample-repo through git proxy', async () => { // Build URL with embedded credentials for reliable authentication const baseUrl = new URL(testConfig.gitProxyUrl); baseUrl.username = testConfig.gitUsername; baseUrl.password = testConfig.gitPassword; - const repoUrl = `${baseUrl.toString()}/finos/git-proxy.git`; - const cloneDir: string = path.join(tempDir, 'git-proxy-clone'); + const repoUrl = `${baseUrl.toString()}/e2e-org/sample-repo.git`; + const cloneDir: string = path.join(tempDir, 'sample-repo-clone'); - console.log(`[TEST] Cloning ${testConfig.gitProxyUrl}/finos/git-proxy.git to ${cloneDir}`); + console.log( + `[TEST] Cloning ${testConfig.gitProxyUrl}/e2e-org/sample-repo.git to ${cloneDir}`, + ); try { const gitCloneCommand: string = `git clone ${repoUrl} ${cloneDir}`; @@ -120,7 +122,7 @@ describe('Git Proxy E2E - Repository Fetch Tests', () => { const readmePath: string = path.join(cloneDir, 'README.md'); expect(fs.existsSync(readmePath)).toBe(true); - console.log('[TEST] Successfully fetched and verified finos/git-proxy'); + console.log('[TEST] Successfully fetched and verified e2e-org/sample-repo'); } catch (error) { console.error('[TEST] Failed to clone repository:', error); throw error; diff --git a/tests/e2e/push.test.ts b/tests/e2e/push.test.ts index d154aa29b..76fb38989 100644 --- a/tests/e2e/push.test.ts +++ b/tests/e2e/push.test.ts @@ -268,7 +268,7 @@ describe('Git Proxy E2E - Repository Push Tests', () => { // Get the test-repo repository and add permissions const repos = await getRepos(adminCookie); const testRepo = repos.find( - (r: any) => r.url === 'https://git-server:8443/coopernetes/test-repo.git', + (r: any) => r.url === 'https://git-server:8443/test-owner/test-repo.git', ); if (testRepo && testRepo._id) { @@ -299,11 +299,11 @@ describe('Git Proxy E2E - Repository Push Tests', () => { const baseUrl = new URL(testConfig.gitProxyUrl); baseUrl.username = testConfig.gitUsername; baseUrl.password = testConfig.gitPassword; - const repoUrl = `${baseUrl.toString()}/coopernetes/test-repo.git`; + const repoUrl = `${baseUrl.toString()}/test-owner/test-repo.git`; const cloneDir: string = path.join(tempDir, 'test-repo-push'); console.log( - `[TEST] Testing push operation to ${testConfig.gitProxyUrl}/coopernetes/test-repo.git`, + `[TEST] Testing push operation to ${testConfig.gitProxyUrl}/test-owner/test-repo.git`, ); try { @@ -421,7 +421,7 @@ describe('Git Proxy E2E - Repository Push Tests', () => { const baseUrl = new URL(testConfig.gitProxyUrl); baseUrl.username = authorizedUser.username; baseUrl.password = authorizedUser.password; - const repoUrl = `${baseUrl.toString()}/coopernetes/test-repo.git`; + const repoUrl = `${baseUrl.toString()}/test-owner/test-repo.git`; const cloneDir: string = path.join(tempDir, 'test-repo-authorized-push'); console.log(`[TEST] Testing authorized push with user ${authorizedUser.username}`); @@ -583,7 +583,7 @@ describe('Git Proxy E2E - Repository Push Tests', () => { const baseUrl = new URL(testConfig.gitProxyUrl); baseUrl.username = authorizedUser.username; baseUrl.password = authorizedUser.password; - const repoUrl = `${baseUrl.toString()}/coopernetes/test-repo.git`; + const repoUrl = `${baseUrl.toString()}/test-owner/test-repo.git`; const cloneDir: string = path.join(tempDir, 'test-repo-approved-push'); console.log( diff --git a/tests/e2e/setup.ts b/tests/e2e/setup.ts index cee0616c4..a3b699fdb 100644 --- a/tests/e2e/setup.ts +++ b/tests/e2e/setup.ts @@ -19,14 +19,14 @@ */ import { beforeAll } from 'vitest'; +import { execSync } from 'child_process'; // Environment configuration - can be overridden for different environments export const testConfig = { gitProxyUrl: process.env.GIT_PROXY_URL || 'http://localhost:8000/git-server:8443', gitProxyUiUrl: process.env.GIT_PROXY_UI_URL || 'http://localhost:8081', + gitServerUrl: process.env.GIT_SERVER_URL || 'https://localhost:8443', timeout: parseInt(process.env.E2E_TIMEOUT || '30000'), - maxRetries: parseInt(process.env.E2E_MAX_RETRIES || '30'), - retryDelay: parseInt(process.env.E2E_RETRY_DELAY || '2000'), // Git credentials for authentication gitUsername: process.env.GIT_USERNAME || 'admin', gitPassword: process.env.GIT_PASSWORD || 'admin123', @@ -39,96 +39,75 @@ export const testConfig = { : 'http://localhost:8000/'), }; +const INFRA_HINT = + 'The E2E test infrastructure is not running. ' + + 'Start it with: docker compose up -d\n' + + 'See CONTRIBUTING.md for details.'; + /** - * Configures git credentials for authentication in a temporary directory - * @param {string} tempDir - The temporary directory to configure git in + * Verifies GitProxy is reachable by hitting its healthcheck endpoint. + * Fails immediately instead of retrying — if the infrastructure isn't + * running we want to fail fast with a helpful message. */ -export function configureGitCredentials(tempDir: string): void { - const { execSync } = require('child_process'); - +async function checkGitProxy(): Promise { + const healthUrl = `${testConfig.gitProxyUiUrl}/api/v1/healthcheck`; try { - // Configure git credentials using URL rewriting - const baseUrlParsed = new URL(testConfig.gitProxyBaseUrl); - - // Initialize git if not already done - try { - execSync('git rev-parse --git-dir', { cwd: tempDir, encoding: 'utf8', stdio: 'pipe' }); - } catch { - execSync('git init', { cwd: tempDir, encoding: 'utf8' }); - } - - // Configure multiple URL patterns to catch all variations - const patterns = [ - // Most important: the proxy server itself (this is what's asking for auth) - { - insteadOf: `${baseUrlParsed.protocol}//${baseUrlParsed.host}`, - credUrl: `${baseUrlParsed.protocol}//${testConfig.gitUsername}:${testConfig.gitPassword}@${baseUrlParsed.host}`, - }, - // Base URL with trailing slash - { - insteadOf: testConfig.gitProxyBaseUrl, - credUrl: `${baseUrlParsed.protocol}//${testConfig.gitUsername}:${testConfig.gitPassword}@${baseUrlParsed.host}${baseUrlParsed.pathname}`, - }, - // Base URL without trailing slash - { - insteadOf: testConfig.gitProxyBaseUrl.replace(/\/$/, ''), - credUrl: `${baseUrlParsed.protocol}//${testConfig.gitUsername}:${testConfig.gitPassword}@${baseUrlParsed.host}`, - }, - ]; - - for (const pattern of patterns) { - execSync(`git config url."${pattern.credUrl}".insteadOf "${pattern.insteadOf}"`, { - cwd: tempDir, - encoding: 'utf8', - }); + const response = await fetch(healthUrl, { + method: 'GET', + signal: AbortSignal.timeout(5000), + }); + if (response.ok || response.status < 500) { + console.log(`GitProxy is reachable at ${testConfig.gitProxyUiUrl}`); + return; } - } catch (error) { - console.error('Failed to configure git credentials:', error); - throw error; + throw new Error(`Healthcheck returned HTTP ${response.status}`); + } catch (error: any) { + console.error(`Error reaching GitProxy at ${healthUrl}: ${error}`); + throw new Error(`GitProxy is not reachable at ${healthUrl}.\n${INFRA_HINT}`); } } -export async function waitForService( - url: string, - maxAttempts?: number, - delay?: number, -): Promise { - const attempts = maxAttempts || testConfig.maxRetries; - const retryDelay = delay || testConfig.retryDelay; - - for (let i = 0; i < attempts; i++) { - try { - const response = await fetch(url, { - method: 'GET', - headers: { Accept: 'application/json' }, - }); - if (response.ok || response.status < 500) { - console.log(`Service at ${url} is ready`); - return; - } - } catch (error) { - // Service not ready yet - } - - if (i < attempts - 1) { - console.log(`Waiting for service at ${url}... (attempt ${i + 1}/${attempts})`); - await new Promise((resolve) => setTimeout(resolve, retryDelay)); - } +/** + * Verifies the local git server is reachable by running `git ls-remote` + * against a known test repository. + */ +function checkGitServer(): void { + const authedUrl = new URL(testConfig.gitServerUrl); + authedUrl.username = testConfig.gitUsername; + authedUrl.password = testConfig.gitPassword; + authedUrl.pathname = '/test-owner/test-repo.git'; + const repoUrl = `${testConfig.gitServerUrl}/test-owner/test-repo.git`; + try { + execSync(`git ls-remote ${authedUrl.href}`, { + encoding: 'utf8', + timeout: 10000, + env: { + ...process.env, + GIT_TERMINAL_PROMPT: '0', + GIT_SSL_NO_VERIFY: '1', + }, + stdio: 'pipe', + }); + console.log(`Git server is reachable at ${testConfig.gitServerUrl}`); + } catch (error: any) { + console.error(`Error reaching Git server at ${repoUrl}: ${error}`); + throw new Error(`Git server is not reachable at ${repoUrl}.\n${INFRA_HINT}`); } - - throw new Error(`Service at ${url} failed to become ready after ${attempts} attempts`); } beforeAll(async () => { console.log('Setting up e2e test environment...'); console.log(`Git Proxy URL: ${testConfig.gitProxyUrl}`); console.log(`Git Proxy UI URL: ${testConfig.gitProxyUiUrl}`); + console.log(`Git Server URL: ${testConfig.gitServerUrl}`); console.log(`Git Username: ${testConfig.gitUsername}`); console.log(`Git Proxy Base URL: ${testConfig.gitProxyBaseUrl}`); - // Wait for the git proxy UI service to be ready - // Note: Docker Compose should be started externally (e.g., in CI or manually) - await waitForService(`${testConfig.gitProxyUiUrl}/api/v1/healthcheck`); + // Pre-flight: verify both services are reachable before running any tests. + // These checks fail fast so developers get a clear error instead of + // waiting through retries when the Docker environment isn't running. + await checkGitProxy(); + checkGitServer(); console.log('E2E test environment is ready'); }, testConfig.timeout); diff --git a/website/docs/development/contributing.mdx b/website/docs/development/contributing.mdx index 54ba6fdc7..06b92224d 100644 --- a/website/docs/development/contributing.mdx +++ b/website/docs/development/contributing.mdx @@ -2,55 +2,58 @@ title: Contributing --- -Here's how to get setup for contributing to GitProxy. +GitProxy is a [FINOS](https://www.finos.org/) project. We welcome contributions from anyone in the community. -## Setup -The GitProxy project relies on the following pre-requisites: +For developer setup, building, testing, and coding guidelines, see [`CONTRIBUTING.md`](https://github.com/finos/git-proxy/blob/main/CONTRIBUTING.md) in the repository. -- [Node](https://nodejs.org/en/download) (20+) -- [npm](https://npmjs.com/) (8+) -- [git](https://git-scm.com/downloads) or equivalent Git client. It must support HTTP/S. +## Contribution Process -Once you have the above tools installed & setup, clone the repository and run: +1. **Check for existing issues** — search [open issues](https://github.com/finos/git-proxy/issues) to see if someone is already working on what you have in mind. +2. **Open an issue** — if nothing exists, [create one](https://github.com/finos/git-proxy/issues/new) describing the change and the problem it solves. +3. **Discuss** — respond to questions or suggestions from maintainers and other contributors. +4. **Fork & submit a PR** — prepare your contribution on a feature branch and open a pull request against `main`. -```bash -$ npm install -``` +If this is your first open source contribution, the [first-contributions guide](https://github.com/firstcontributions/first-contributions#first-contributions) is a great starting point. -This will install the full project's dependencies. Once complete, you can run the app locally: +## Contributor License Agreement (CLA) -```bash -$ npm run start # Run both proxy server & dashboard UI -$ npm run server # Run only the proxy server -$ npm run client # Run only the UI -``` +All contributors must have a CLA on file with FINOS before pull requests can be merged. Review the FINOS [contribution requirements](https://community.finos.org/docs/governance/Software-Projects/contribution) and submit (or have your employer submit) the required CLA via [EasyCLA](https://community.finos.org/docs/governance/Software-Projects/easycla). -## Testing +## Governance -Currently, we use Mocha and Chai for unit testing and Cypress for E2E testing. For more details on how to use these testing libraries, check out our [Testing documentation](testing). +The project is governed by the [Linux Foundation Antitrust Policy](https://www.linuxfoundation.org/antitrust-policy/) and the FINOS [IP Policy](https://community.finos.org/assets/files/IP-Policy-fe5925025fc0a57b1cbed64f86b26a73.pdf), [Code of Conduct](https://www.finos.org/code-of-conduct), [Collaborative Principles](https://github.com/finos/git-proxy/blob/main/Collaborative-Principles.md), and [Community Meeting Procedures](https://community.finos.org/docs/journey/engage#meet-the-community). -### Patch coverage requirements +### Roles -Newly introduced changes **must have over 80% unit test coverage**. This is enforced by our CI, and in practice, only few exceptions (such as emergency fixes) are allowed to skip this requirement. Make sure to add thorough unit tests to your PR to help reviewers approve your PR more quickly! +- **Contributor** — anyone who submits a contribution (code, issues, comments, documentation, media, or any combination). +- **Maintainer** — a Contributor who, by virtue of their contribution history, has been given write access to the repository and may merge approved contributions. +- **Lead Maintainer** — the project's interface with the FINOS team and Board. Responsible for communicating on behalf of the project and for ensuring the project is following FINOS policies and procedures. -## Configuration schema -The configuration for GitProxy includes a JSON Schema ([`config.schema.json`](https://github.com/finos/git-proxy/blob/main/config.schema.json)) to define the expected properties used by the application. When adding new configuration properties to GitProxy, ensure that the schema is updated with any new, removed or changed properties. See [JSON Schema docs for specific syntax](https://json-schema.org/docs). +### Contribution Rules -When updating the configuration schema, you must also re-generate the reference doc used here on the site. To generate the reference documentation, [json-schema-for-humans](https://github.com/coveooss/json-schema-for-humans) is used to output the Markdown. +- All contributions **must** be submitted as pull requests, including contributions by Maintainers. +- All pull requests **must** be reviewed by a Maintainer (other than the contributor) before being merged. +- Pull requests for non-trivial contributions **must** remain open long enough for all Maintainers to review and comment. +- After the review period, if no Maintainer objects, any Maintainer **may** merge. +- If any Maintainer objects, the Maintainers **should** try to reach consensus through discussion. If consensus cannot be reached or the contribution is deemed inappropriate or otherwise unable to be accepted into the project, the pull request will be closed. -1. Install [json-schema-for-humans](https://github.com/coveooss/json-schema-for-humans?tab=readme-ov-file#installation) (requires Python) -2. Run `npm run gen-schema-doc`. +If a pull request is closed due to objections, the contributor may address the concerns and re-open it for review. We encourage contributors to engage in good faith with any feedback and to seek help from the community if needed. If a contributor feels that their contribution was unfairly rejected, they may request a review by the Maintainer team and/or raise the issue in the community meeting. -## Submitting a pull request - +### Becoming a Maintainer -## About FINOS contributions - +Any Contributor who has made a substantial contribution may apply (or be nominated) to become a Maintainer. Existing Maintainers approve nominations via the voting process above. ## Community Meetings -Join our [fortnightly Zoom meeting](https://zoom-lfx.platform.linuxfoundation.org/meeting/95849833904?password=99413314-d03a-4b1c-b682-1ede2c399595) on Monday, 4PM BST (odd week numbers). -🌍 [Convert to your local time](https://www.timeanddate.com/worldclock) -[Click here](https://calendar.google.com/calendar/event?action=TEMPLATE&tmeid=MTRvbzM0NG01dWNvNGc4OGJjNWphM2ZtaTZfMjAyNTA2MDJUMTUwMDAwWiBzYW0uaG9sbWVzQGNvbnRyb2wtcGxhbmUuaW8&tmsrc=sam.holmes%40control-plane.io&scp=ALL) for the recurring Google Calendar meeting invite. -Alternatively, send an e-mail to [help@finos.org](https://zoom-lfx.platform.linuxfoundation.org/meeting/95849833904?password=99413314-d03a-4b1c-b682-1ede2c399595#:~:text=Need-,an,-invite%3F) to get a calendar invitation. -Previous recordings available at: https://openprofile.dev \ No newline at end of file +Join our [fortnightly Zoom meeting](https://zoom-lfx.platform.linuxfoundation.org/meeting/95849833904?password=99413314-d03a-4b1c-b682-1ede2c399595) on Monday, 4PM BST (odd week numbers). +[Convert to your local time](https://www.timeanddate.com/worldclock). +[Add to Google Calendar](https://calendar.google.com/calendar/event?action=TEMPLATE&tmeid=MTRvbzM0NG01dWNvNGc4OGJjNWphM2ZtaTZfMjAyNTA2MDJUMTUwMDAwWiBzYW0uaG9sbWVzQGNvbnRyb2wtcGxhbmUuaW8&tmsrc=sam.holmes%40control-plane.io&scp=ALL). +Alternatively, email [help@finos.org](mailto:help@finos.org) for a calendar invitation. + +Previous recordings available at: https://openprofile.dev + +## Contact + +- **Slack**: [#git-proxy](https://finos-lf.slack.com/archives/C06LXNW0W76) on the FINOS Slack workspace +- **Issues**: [github.com/finos/git-proxy/issues](https://github.com/finos/git-proxy/issues) +- **Mailing list**: [git-proxy+subscribe@lists.finos.org](mailto:git-proxy+subscribe@lists.finos.org) diff --git a/website/docs/development/plugins.mdx b/website/docs/development/plugins.mdx index f8f5868bd..02a148d22 100644 --- a/website/docs/development/plugins.mdx +++ b/website/docs/development/plugins.mdx @@ -126,19 +126,19 @@ Loaded plugin: FooPlugin To develop a new plugin, you must add `@finos/git-proxy` as a [peer dependency](https://docs.npmjs.com/cli/v10/configuring-npm/package-json#peerdependencies). The plugin & proxy classes can then be imported from `@finos/git-proxy` via the following extension points: ```js -import { PushActionPlugin } from '@finos/git-proxy/src/plugin.js' +import { PushActionPlugin } from '@finos/git-proxy/plugin' ``` - Use this class to execute as an action in the proxy chain during a `git push` ```js -import { PullActionPlugin } from '@finos/git-proxy/src/plugin.js' +import { PullActionPlugin } from '@finos/git-proxy/plugin' ``` - Use this class to execute as an action in the proxy chain during a `git fetch` ```js -import { Step, Action } from '@finos/git-proxy/src/proxy/actions/index.js' +import { Step, Action } from '@finos/git-proxy/proxy/actions' ``` - These are internal classes which act as carriers for `git` state during proxying. Plugins should modify the passed in `Action` for affecting any global state of the git operation and add its own custom `Step` object to capture the plugin's own internal state (logs, errored/blocked status, etc). @@ -149,7 +149,7 @@ Please see the [sample plugin package included in the repo](https://github.com/f If your plugin relies on custom state, it is recommended to create subclasses in the following manner: ```javascript -import { PushActionPlugin } from "@finos/git-proxy/src/plugin.js"; +import { PushActionPlugin } from "@finos/git-proxy/plugin"; class FooPlugin extends PushActionPlugin { constructor() { @@ -183,8 +183,8 @@ $ npm install --save-peer @finos/git-proxy 2. Create a new JavaScript file for your plugin. The file should export an instance of `PushActionPlugin` or `PullActionPlugin`: ```javascript -import { PushActionPlugin } from "@finos/git-proxy/src/plugin.js"; -import { Step } from "@finos/git-proxy/src/proxy/actions/index.js"; +import { PushActionPlugin } from "@finos/git-proxy/plugin"; +import { Step } from "@finos/git-proxy/proxy/actions"; //Note: Only use a default export if you do not rely on any state. Otherwise, create a sub-class of [Push/Pull]ActionPlugin export default new PushActionPlugin(function(req, action) { diff --git a/website/docs/development/testing.mdx b/website/docs/development/testing.mdx deleted file mode 100644 index 2741c003f..000000000 --- a/website/docs/development/testing.mdx +++ /dev/null @@ -1,344 +0,0 @@ ---- -title: Testing ---- - -## Testing - -As of v1.19.2, GitProxy uses [Mocha](https://mochajs.org/) (`ts-mocha`) as the test runner, and [Chai](https://www.chaijs.com/) for unit test assertions. User interface tests are written in [Cypress](https://docs.cypress.io), and some fuzz testing is done with [`fast-check`](https://fast-check.dev/). - -### Unit testing with Mocha and Chai - -Here's an example unit test that uses Chai for testing (`test/testAuthMethods.test.js`): - -```js -// Import all the test dependencies we need -const chai = require('chai'); -const sinon = require('sinon'); -const proxyquire = require('proxyquire'); - -// Import module that contains the function we want to test -const config = require('../src/config'); - -// Allows using chain-based expect calls -chai.should(); -const expect = chai.expect; - -describe('auth methods', async () => { - it('should return a local auth method by default', async function () { - const authMethods = config.getAuthMethods(); - expect(authMethods).to.have.lengthOf(1); - expect(authMethods[0].type).to.equal('local'); - }); - - it('should return an error if no auth methods are enabled', async function () { - const newConfig = JSON.stringify({ - authentication: [ - { type: 'local', enabled: false }, - { type: 'ActiveDirectory', enabled: false }, - { type: 'openidconnect', enabled: false }, - ], - }); - - const fsStub = { - existsSync: sinon.stub().returns(true), - readFileSync: sinon.stub().returns(newConfig), - }; - - const config = proxyquire('../src/config', { - fs: fsStub, - }); - - // Initialize the user config after proxyquiring to load the stubbed config - config.initUserConfig(); - - expect(() => config.getAuthMethods()).to.throw(Error, 'No authentication method enabled'); - }); - - it('should return an array of enabled auth methods when overridden', async function () { - const newConfig = JSON.stringify({ - authentication: [ - { type: 'local', enabled: true }, - { type: 'ActiveDirectory', enabled: true }, - { type: 'openidconnect', enabled: true }, - ], - }); - - const fsStub = { - existsSync: sinon.stub().returns(true), - readFileSync: sinon.stub().returns(newConfig), - }; - - const config = proxyquire('../src/config', { - fs: fsStub, - }); - - // Initialize the user config after proxyquiring to load the stubbed config - config.initUserConfig(); - - const authMethods = config.getAuthMethods(); - expect(authMethods).to.have.lengthOf(3); - expect(authMethods[0].type).to.equal('local'); - expect(authMethods[1].type).to.equal('ActiveDirectory'); - expect(authMethods[2].type).to.equal('openidconnect'); - }); -}); -``` - -Core concepts to keep in mind when unit testing JS/TS modules with Chai: - -#### Stub internal methods to make tests predictable - -Functions often make use of internal libraries such as `fs` for reading files and performing operations that are dependent on the overall state of the app (or database/filesystem). Since we're only testing that the given function behaves the way we want, we **stub** these libraries. - -For example, here we stub the `fs` library so that "reading" the `proxy.config.json` file returns our mock config file: - -```js -// Define the mock config file -const newConfig = JSON.stringify({ - authentication: [ - { type: 'local', enabled: true }, - { type: 'ActiveDirectory', enabled: true }, - { type: 'openidconnect', enabled: true }, - ], -}); - -// Create the stub for `fs.existsSync` and `fs.readFileSync` -const fsStub = { - existsSync: sinon.stub().returns(true), - readFileSync: sinon.stub().returns(newConfig), -}; -``` - -This stub will make all calls to `fs.existsSync` to return `true` and all calls to `readFileSync` to return the `newConfig` mock file. - -Then, we use `proxyquire` to plug in the stub to the library that we're testing: - -```js -const config = proxyquire('../src/config', { - fs: fsStub, -}); - -// Initialize the user config after proxyquiring to load the stubbed config -config.initUserConfig(); -``` - -Finally, when calling the function we're trying to test, the internal calls will automatically resolve to the values we chose. - -#### Setup and cleanup - -`before` and `beforeEach`, `after` and `afterEach` are testing constructs that allow executing code before and after each test. This allows setting up stubs before each test, making API calls, setting up the database - or otherwise cleaning up the database after test execution. - -This is an example from another test file (`test/addRepoTest.test.js`): - -```js -before(async function () { - app = await service.start(); - - await db.deleteRepo('test-repo'); - await db.deleteUser('u1'); - await db.deleteUser('u2'); - await db.createUser('u1', 'abc', 'test@test.com', 'test', true); - await db.createUser('u2', 'abc', 'test2@test.com', 'test', true); -}); - -// Tests go here - -after(async function () { - await service.httpServer.close(); - - await db.deleteRepo('test-repo'); - await db.deleteUser('u1'); - await db.deleteUser('u2'); -}); - -afterEach(() => { - sinon.restore(); -}); -``` - -Note that `after` will execute once after **all** the tests are complete, whereas `afterEach` will execute at the end of **each** test. - -#### Reset sinon and proxyquire cache - -**It's very important to reset Sinon and the Proxyquire/require cache after each test** when necessary. This prevents old stubs from leaking into subsequent tests. - -Here is an example of a function that resets both of these after each test (`test/chain.test.js`): - -```js -const clearCache = (sandbox) => { - delete require.cache[require.resolve('../src/proxy/processors')]; - delete require.cache[require.resolve('../src/proxy/chain')]; - sandbox.restore(); -}; - -... - -afterEach(() => { - // Clear the module from the cache after each test - clearCache(sandboxSinon); -}); -``` - -#### Focus on expected behaviour - -Mocha and Chai make it easy to write tests in plain English. It's a good idea to write the expected behaviour in plain English and then prove it by writing the test: - -```js -describe('auth methods', async () => { - it('should return a local auth method by default', async function () { - // Test goes here - }); - - it('should return an error if no auth methods are enabled', async function () { - // Test goes here - }); - - it('should return an array of enabled auth methods when overridden', async function () { - // Test goes here - }); -}); -``` - -Assertions can also be done similarly to plain English: - -```js -expect(authMethods).to.have.lengthOf(3); -expect(authMethods[0].type).to.equal('local'); -``` - -#### Unit testing coverage requirement - -**All new lines of code introduced in a PR, must have over 80% coverage** (patch coverage). This is enforced by our CI, and generally a PR will not be merged unless this coverage requirement is met. Please make sure to write thorough unit tests to increase GitProxy's code quality! - -If test coverage is still insufficient after writing your tests, check out the [CodeCov report](https://app.codecov.io/gh/finos/git-proxy) after making the PR and take a look at which lines are missing coverage. - -#### More examples - -Check out [test/1.test.js](https://github.com/finos/git-proxy/blob/main/test/1.test.js) for another example on how to write unit tests. - -### UI testing with Cypress - -Although coverage is currently low, we have introduced Cypress testing to make sure that end-to-end flows are working as expected with every added feature. - -This is a sample test from `cypress/e2e/repo.cy.js`: - -```js -describe('Repo', () => { - beforeEach(() => { - // Custom login command - cy.login('admin', 'admin'); - - cy.visit('/dashboard/repo'); - - // prevent failures on 404 request and uncaught promises - cy.on('uncaught:exception', () => false); - }); - - describe('Code button for repo row', () => { - it('Opens tooltip with correct content and can copy', () => { - const cloneURL = 'http://localhost:8000/finos/git-proxy.git'; - const tooltipQuery = 'div[role="tooltip"]'; - - cy - // tooltip isn't open to start with - .get(tooltipQuery) - .should('not.exist'); - - cy - // find the entry for finos/git-proxy - .get('a[href="/dashboard/repo/git-proxy"]') - // take it's parent row - .closest('tr') - // find the nearby span containing Code we can click to open the tooltip - .find('span') - .contains('Code') - .should('exist') - .click(); - - cy - // find the newly opened tooltip - .get(tooltipQuery) - .should('exist') - .find('span') - // check it contains the url we expect - .contains(cloneURL) - .should('exist') - .parent() - // find the adjacent span that contains the svg - .find('span') - .next() - // check it has the copy icon first and click it - .get('svg.octicon-copy') - .should('exist') - .click() - // check the icon has changed to the check icon - .get('svg.octicon-copy') - .should('not.exist') - .get('svg.octicon-check') - .should('exist'); - - // failed to successfully check the clipboard - }); - }); -}); -``` - -Here, we use a similar syntax to Mocha to **describe the behaviour that we expect**. The difference, is that Cypress expects us to write actual commands for executing actions in the app. Some commands used very often include `visit` (navigates to a certain page), `get` (gets a certain page element to check its properties), `contains` (checks if an element has a certain string value in it), `should` (similar to `expect` in unit tests). - -#### Custom commands - -Cypress allows defining **custom commands** to reuse and simplify code. - -In the above example, `cy.login('admin', 'admin')` is actually a custom command defined in `/cypress/support/commands.js`. It allows logging a user into the app, which is a requirement for many E2E flows: - -```js -Cypress.Commands.add('login', (username, password) => { - cy.session([username, password], () => { - cy.visit('/login'); - cy.intercept('GET', '**/api/auth/profile').as('getUser'); - - cy.get('[data-test=username]').type(username); - cy.get('[data-test=password]').type(password); - cy.get('[data-test=login]').click(); - - cy.wait('@getUser'); - cy.url().should('include', '/dashboard/repo'); - }); -}); -``` - -### Fuzz testing with fast-check - -Fuzz testing helps find edge case bugs by generating random inputs for test data. This is very helpful since regular tests often have naive assumptions of users always inputting "expected" data. - -Fuzz testing with fast-check is very easy: it integrates seamlessly with Mocha and it doesn't require any additional libraries beyond fast-check itself. - -Here's an example of a fuzz test section for a test file (`testCheckRepoInAuthList.test.js`): - -```js -const fc = require('fast-check'); - -// Unit tests go here - -describe('fuzzing', () => { - it('should not crash on random repo names', async () => { - await fc.assert( - fc.asyncProperty( - fc.string(), - async (repoName) => { - const action = new actions.Action('123', 'type', 'get', 1234, repoName); - const result = await processor.exec(null, action, authList); - expect(result.error).to.be.true; - } - ), - { numRuns: 100 } - ); - }); -}); -``` - -Writing fuzz tests is a bit different from regular unit tests, although we do still `assert` whether a certain value is correct or not. In this example, fc.string() indicates that a random string value is being generated for the `repoName` variable. This `repoName` is then inserted in the `action` to see if the `processor.exec()` function is capable of handling these or not. - -In this case, we expect that the `result.error` value is always true. This means that the `exec` flow always errors out, but never crashes the app entirely. You may also want to test that the app is always able to complete a flow without an error. - -Finally, we have the `numRuns` property for `fc.assert`. This allows us to run the fuzz test multiple times with a new randomized value each time. This is important since the test may randomly fail or pass depending on the input. diff --git a/website/sidebars.js b/website/sidebars.js index c778eca85..6573101d1 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -43,7 +43,7 @@ module.exports = { }, collapsible: true, collapsed: false, - items: ['development/contributing', 'development/plugins', 'development/testing'], + items: ['development/contributing', 'development/plugins'], }, ], };