CD_Nodes: Distributed Test Execution System

This system was developed for a Distributed Computing course to demonstrate how a network of independent nodes can collaborate to solve intensive tasks. In this case, the task is the automated execution of Python test suites using pytest.

The project is built around a Master-Worker architecture, where a starter node coordinates the network and several worker nodes handle the heavy lifting of test execution.

Core Architecture

The system is composed of several moving parts that communicate across different protocols:

1. The API Layer

A Flask-based web server serves as the entry point for users. It provides endpoints for submitting new test tasks, checking the status of evaluations, and monitoring the health of the network. The API communicates with the Master node via Unix Domain Sockets.

2. The Starter Node (Master)

The starter node is responsible for managing the network topology. It keeps track of all connected workers, handles the distribution of test files, and decides which node executes which part of the test suite using a Round-Robin strategy.

3. Worker Nodes

Worker nodes connect to the starter node to join the network. They receive code packages in chunks, unzip them, and run specific tests assigned to them. Once finished, they send the results back up the chain.

Key Features

• Distributed Task Allocation: Tests are split across available workers to parallelize execution and reduce total completion time.
• Fault Tolerance: The system includes a backup mechanism. Nodes monitor their peers, and if a node fails during execution, its tasks can be recovered or reassigned.
• Flexible Inputs: You can submit projects by providing GitHub repository URLs or by uploading zip files directly through the API.
• Custom Communication Protocol: We implemented a custom protocol (CDProto) over TCP to handle message passing, file transfers, and node synchronization.

Getting Started

The easiest way to run the project is using Docker and Docker Compose, which handles the networking and node environment setup for you.

Prerequisites

• Docker and Docker Compose
• Python 3 (if running locally without Docker)

Running the System

1. Start the environment: You can use the provided script to initialize the containers:

   ./start_containers.sh

   Alternatively, use Docker Compose directly:

   docker-compose up --build

2. Submit an evaluation: Once the system is up, you can send a POST request to the /evaluation endpoint. For example, using a JSON payload with GitHub URLs:

   {
     "auth_token": "your_github_token",
     "projects": ["https://github.com/user/repo"]
   }

3. Monitor the network:

   • GET /network: View the current connections between nodes.
   • GET /stats: Get statistics about evaluation progress and node performance.
   • GET /evaluation: List all completed and ongoing evaluations.

Project Structure

• /src: Contains the core logic for the Master, Workers, and Protocol.
• /src/api: The Flask application handling REST requests.
• /src/protocol: Definitions for message formats and communication logic.
• /test_results: Where the system stores the output of executed tests.

This project was designed to explore the challenges of distributed systems, including synchronization, state management, and reliable communication in a containerized environment.