Search.Libre: An OpenNIC Search Engine

The search engine is available at search.libre:45260.

The search engine is currently down, while I try to rewrite the web crawler. Previously, crawling progress has been almost 0, due to the crawler taking up too much RAM.

The search engine is up and running, but no crawling is taking place due to the above.^^

Getting Started

If you would like to host your own version of this site, for development purposes or otherwise, follow these steps:

1. Set up a Virtual Environment

• Create a Python virtual environment inside the empty .venv folder.
• This has been tested with Python 3.10.

2. Install Python dependencies

• Install Python dependencies using pip.

  pip install -r requirements.txt

3. Build the OpenNIC Search Package

• While in the opennic_search folder, run:

  maturin develop -r

  Note: You can safely ignore the warning about the extension-module feature not being enabled.

4. Compile the Project

• Build the project using Cargo:

  cargo build -r

5. Host the Website

• After building, run the following command to host the site:

  cargo run -r

6. Start the Crawler

• Start the Python crawler (with the venv activated!):

  python main.py

  Important: The search engine relies on the crawler for finding sites, so ensure the crawler is running unless you want no search results.

Configuration

The following options can be adjusted in the config.json file:

• hostname: The domain the site accepts requests from (e.g., search.libre).
• port: The port number the site will use. Defaults to 80 for HTTP.
• bind_address: The IP address the server will listen on. Set it to 127.0.0.1 to bind to localhost or 0.0.0.0 for external access.
• restrict_hostname: If true, the server only accepts requests from the specified hostname. Otherwise, it accepts requests from any hostname.