Add NixOS integration test suite for Home Manager configuration

[Copilot]

Home Manager Testing Strategy Implementation

• Explore repository structure and understand existing test patterns
• Create tests/home-manager.nix test file with NixOS Integration Test framework
• Update flake.nix to include the new test in checks output
• Add test to the checks section (following Nix flake standards)
• Verify the test structure follows existing patterns (paperless.nix, tandoor.nix)
• Ensure test imports home-manager correctly
• Configure test VM with X11 support for visual testing
• Add functional tests for git, nvim, and direnv
• Add visual tests for Neovim theming (screenshot-based)
• Fix neovim config path override for test VM (mkOutOfStoreSymlink → direct store path)
• Document test setup with comprehensive README
• Document limitations and notes about the test implementation
• Address code review feedback (simplify test conditions and fix background process handling)
• Fix variable scoping in test configuration (extract testUserConfig for proper closure)
• Validate test runs successfully in CI

Original prompt

Home Manager Testing Strategy

Create a new branch if necessary.

1. Overview

This document outlines the strategy for testing the Home Manager configuration within the tiborpilz/NixOS repository. The primary goal is to ensure that user environments are reproducible, functional, and visually consistent across updates.

2. Objectives

• Isolation: Tests must run in a clean, isolated Virtual Machine (VM) to prevent host contamination and ensure reproducibility.
• Tooling: Utilize native Nix ecosystem tools, specifically the NixOS Integration Test framework (nixos-test).
• Functional Verification: Verify the presence and execution of key user packages (e.g., git, zsh, direnv).
• Visual Verification: Validate "Look & Feel" elements, specifically the Neovim theming and status line configurations, using OCR and screenshot capabilities.

3. Technical Architecture

3.1. Testing Framework

We will use the NixOS Integration Testing framework. This allows us to define a lightweight QEMU VM described entirely in Nix code.

• Entry Point: checks.<system>.home-manager-test in flake.nix.
• Module: A dedicated test module that imports <home-manager/nixos> and the user's home configuration.

3.2. Test Environment Node

The test environment will consist of a single machine node (machine) with the following characteristics:

• User: A standard unprivileged user (e.g., testuser).
• Graphics: X11 enabled (services.xserver) to support terminal emulators for visual testing.
• Window Manager: A minimal Window Manager (e.g., i3 or none with just xterm) to reduce noise.

4. Test Specifications

4.1. Functional Tests

Goal: Ensure critical tools are installed and usable.

• Check: Verify nvim is in $PATH.
• Check: Verify git is in $PATH.
• Check: execute nvim --version to ensure binary sanity.
• Check: Verify specific configuration files exist (e.g., ~/.config/nvim/init.lua).

4.2. Visual Tests (Look & Feel)

Goal: Verify that Neovim loads with the correct theme and plugins.

• Methodology:

  1. Launch a graphical terminal (e.g., xterm or kitty).
  2. Execute nvim inside the terminal.
  3. Use the python test driver's machine.wait_for_text() (OCR) to detect visual elements.
  4. Capture screenshots for regression analysis.

• Specific Checks:

  • Status Line: Wait for text unique to the status line plugin (e.g., specific mode indicators like NORMAL or branch icons if OCR supports it).
  • Startup Screen: Verify no error messages (e.g., "Error detected while processing...").
  • Screenshot: machine.screenshot("neovim_dashboard").

5. Implementation Plan

Step 1: Create Test File

File: tests/home-manager.nix

{ pkgs, ... }:

{
  name = "home-manager-suite";

  nodes.machine = { config, pkgs, ... }: {
    imports = [ <home-manager/nixos> ];

    # 1. Setup X11 for visual tests
    services.xserver.enable = true;
    services.xserver.desktopManager.xterm.enable = false;
    services.xserver.windowManager.i3.enable = true; # Minimal WM
    services.displayManager.defaultSession = "none+i3";

    # 2. Configure Test User
    users.users.testuser = {
      isNormalUser = true;
      password = "password"; # Required for some display managers/tests
      shell = pkgs.zsh;
    };

    # 3. Import Home Manager Configuration
    # We adapt the existing home configuration for the test user
    home-manager.users.testuser = import ../home/default.nix; 
    
    # Ensure dependencies for testing (e.g. terminal) are present
    environment.systemPackages = with pkgs; [ xterm ];
  };

  # Python Test Script
  testScript = ''
    start_all()
    
    # Phase 1: Functional CLI Tests
    machine.wait_for_unit("multi-user.target")
    machine.succeed("su - testuser -c 'which nvim'")
    machine.succeed("su - testuser -c 'nvim --version'")

    # Phase 2: Visual/GUI Tests
    machine.wait_for_x()
    
    # Open a terminal with Neovim running
    # We run it in the background (&) so the script continues
    machine.succeed("su - testuser -c 'DISPLAY=:0 xterm -e nvim &'")
    
    # Wait for Neovim to potentially load (give it 5-10 seconds)
    machine.sleep(5)
    
    # Take a screenshot of the initial state
    machine.screenshot("neovim_startup")

    # Check for the absence of error messages on screen
    # (This is a negative check; OCR can be flaky so we rely mostly on the screenshot and exit code)
    
    # Attempt to verify status line text if predictable (e.g. "Empty")
    # machine.wait_for_text("Empty") 
  '';
}

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[tiborpilz]

@copilot Try to run the test in the PR, see why it failed and fix it.

[Copilot]

@copilot Try to run the test in the PR, see why it failed and fix it.

Fixed the test scoping issue in commit d515d64. The problem was that inputs variable needed proper closure - extracted testUserConfig as a local binding in the node definition to ensure the inputs parameter is accessible when the home-manager configuration is evaluated.