Update version to 0.1.22, introducing experimental submissions and submission notes

- Incremented package version in `pyproject.toml` to 0.1.22.
- Added support for experimental submissions in the CLI and client, allowing users to mark submissions as experimental and attach private notes.
- Updated documentation across multiple files to clarify the new experimental submission feature and its implications for leaderboard and scoring.

Author: jrosenfeld13

README.md

@@ -64,6 +64,15 @@ pip install crowdcent-challenge
 
    # Submit predictions
    client.submit_predictions(file_path="predictions.parquet")
+
+   # Or test a new model with an experimental submission + private note
+   # (requires another slot to have a non-experimental submission)
+   client.submit_predictions(
+       file_path="predictions.parquet",
+       slot=2,
+       is_experimental=True,
+       notes="2-layer transformer w/ sector embeddings",
+   )
    ```
 
 ## 🏆 Available Challenges
@@ -90,6 +99,10 @@ crowdcent download-inference-data current -o inference.parquet
 
 # Submit predictions
 crowdcent submit predictions.parquet
+
+# Submit an experimental prediction with a private note
+crowdcent submit predictions.parquet --slot 2 --experimental \
+    --notes "2-layer transformer w/ sector embeddings"
 ```
 
 **Documentation**: [docs.crowdcent.com](https://docs.crowdcent.com)

docs/faq.md

@@ -93,7 +93,7 @@ Your submission must be a Parquet file containing an `id` column that matches th
 
 If a submission window is currently open, your prediction is submitted immediately. If no window is open, your prediction is **queued** and will be automatically submitted when the next window opens.
 
-By default, submissions are also queued for the following period (auto-rollover). Use `queue_next=False` (Python) or `--no-queue-next` (CLI) to opt out.
+By default, submissions are also queued for the following period (auto-rollover). Use `queue_next=False` (Python) or `--no-queue-next` (CLI) to opt out. The Python client and CLI also accept `is_experimental` / `--experimental` and `notes` / `--notes` — see [What is an experimental submission?](#what-is-an-experimental-submission) below.
 
 ### How often can I submit?
 
@@ -105,6 +105,30 @@ You can submit multiple times for an active inference period. Your latest valid
 *   **CLI:** Use `crowdcent list-submissions <challenge_slug>` (optionally filter with `--period current` or `--period YYYY-MM-DD`) and `crowdcent get-submission <challenge_slug> <submission_id>`.
 Statuses include "pending", "processing", "evaluated" (or "scored"), and "error" (or "failed").
 
+### What is an experimental submission?
+
+A submission you mark as experimental gets scored against the non-experimental competitive field (so you can see your shadow percentile) but is excluded from the leaderboard, the meta-model, and your CC Points. It's the right tool for testing a new architecture or feature set without dragging your competitive standing.
+
+### Why was my experimental submission rejected?
+
+You need at least one **non-experimental** submission in another slot for the same period. This keeps the competitive pool honest and prevents experimental-only entries. Submit a non-experimental prediction to another slot first, or uncheck experimental on the current one.
+
+### Do experimental submissions affect my CC Points or streak?
+
+Experimental submissions are excluded from the [Performance Adjustment](points-system.md) (only your non-experimental slots feed your average percentile). They don't influence the meta-model's weighting either. Your non-experimental submission is what triggers daily base credit and feeds your streak — by design every period has at least one non-experimental sub when you also submit experimental, so streak/base-credit math is unchanged.
+
+### Can other users see my experimental submissions or notes?
+
+Experimental submissions are visible on profiles with an **experimental** label (triangular slot badge), and you can see other users' shadow percentiles. However, **notes are private to the submission owner**, i.e. only the note creator can see them.
+
+### Can I edit notes after the period closes?
+
+Yes. The notes endpoint is independent of the scoring lifecycle. Edit them inline on your own profile/scores pages at any time.
+
+### Why did my live submission succeed but the queue copy was rejected?
+
+The live save and the queue-for-next-period copy are processed independently — one can succeed while the other is rejected. If you submit experimental (`is_experimental=True`) without a non-experimental queued submission in another slot, the live save commits but the queue copy is rejected. Your live submission still counts. Either pass `queue_next=False`, or queue a non-experimental sub to another slot first.
+
 ## Scoring & Uniqueness
 
 ### What are uniqueness metrics?

docs/getting-started.md

@@ -4,7 +4,7 @@ Sign up for a CrowdCent account [here](https://crowdcent.com/accounts/signup/) o
 ## Explore Challenges
 Once logged in, you'll land on the [Challenge List](https://crowdcent.com/challenge) page. Browse through the available challenges to find one that interests you. Each challenge card will give you a brief overview. Click on a challenge to see more details.
 
-[![Challenge List](overrides/assets/images/challenge_list.png)](https://crowdcent.com/challenge){:target="_blank"}
+[![Challenge List](overrides/assets/images/challenge_list.png){: style="width:70%; display:block; margin:0 auto;"}](https://crowdcent.com/challenge){:target="_blank"}
 
 ## Download Data
 On the detail page for your chosen challenge (e.g. [hyperliquid-ranking](https://crowdcent.com/challenge/hyperliquid-ranking)), you will find:
@@ -17,20 +17,22 @@ Using the downloaded training data, build a model to predict the challenge targe
 
 ## Submit predictions during an Inference Period
 - The Challenge Detail page will display information about the current **Inference Data** period, including its release date and submission deadline.
-- You have a specific number of **submission slots** (e.g., up to 5) for each inference period. You can choose which slot to use for each submission.
+- You have multiple **submission slots** for each inference period (currently up to 5 for Hyperliquid Ranking). You can choose which slot to use for each submission.
 
-[![Submission Panel](overrides/assets/images/inference_period.png)](https://crowdcent.com/challenge){:target="_blank"}
-*The submission panel on the Challenge Detail page shows inference periods and your submission slots. In this submission panel image, we see that slots 1, 2, and 3 have successful submissions for the current inference period (lower-right check marks). However, only slots 1 and 3 have queued submissions for the next inference period (upper-right squares)*
+[![Submission Panel](overrides/assets/images/submission_panel.png)](https://crowdcent.com/challenge){:target="_blank"}
+*The submission panel on the Challenge Detail page shows inference periods, your submission slots, and options to queue, mark a submission as experimental, or attach a private note. Slot indicators show submitted (check marks) and queued (squares/dots) states.*
 
 There are two main ways to submit your predictions:
 
 ### 1. Via the Website (UI)
 - Go to the Challenge Detail page.
 - In the submission panel, select an available **slot**.
 - Upload your prediction file (typically a Parquet file).
-- **Submissions are now flexible:**
+- **Submissions are flexible:**
     - If the window is **open**, your file is submitted immediately. By default, it is also queued for the *next* period (auto-rollover).
     - If the window is **closed**, your file is **queued** and will be automatically submitted when the next period opens.
+- **Mark a submission as experimental** to test new models without affecting your leaderboard rank, CC Points, or meta-model contribution. Experimental submissions are still scored and shown on profiles with an **experimental** label (triangular slot badge). You must keep at least one non-experimental submission in another slot for the same period.
+- **Add a note** (e.g. *"added sector features"*) so future-you remembers what changed. Notes are private to you and editable any time.
 
 ### 2. Programmatically (via API)
 - Go to your **User Profile** page (accessible from the top navigation bar when logged in).

docs/hyperliquid-ranking.md

@@ -111,6 +111,9 @@ client.submit_predictions(file_path="submission.parquet", slot=2) # or a parquet
 !!! tip "Flexible Timing"
     You can submit anytime. If the window is open, your submission is accepted immediately. If the window is closed, your submission is queued and automatically submitted when the next window opens. By default, submissions are also queued for the following period (auto-rollover) if you submit during an open window. Use `queue_next=False` to opt out.
 
+!!! info "Experimental Submissions & Notes"
+    Use any of your 5 slots for an **experimental** submission to test a new model. It will be scored and given a shadow percentile, but won't affect the leaderboard, the meta-model, or your CC Points. You must keep at least one non-experimental submission in another slot for the same period. Pass `is_experimental=True` (Python) / `--experimental` (CLI). Optional `notes="..."` / `--notes "..."` attaches a private annotation.
+
 | id      | pred_10d | pred_30d |
 |---------|----------|----------|
 | BABY    | 0.2      | 0.3      |
@@ -131,7 +134,7 @@ Before scoring, for each prediction timeframe, ids are uniform ranked [0, 1], an
 
 These metrics measure how accurate your predictions are against actual market outcomes.
 
-1) [Symmetric Normalized Discounted Cumulative Gain (NDCG@40)](scoring.md#symmetric-ndcgk)
+1) [Symmetric Normalized Discounted Cumulative Gain (NDCG@40)](scoring.md#symmetric-ndcgk-symmetric_ndcg_at_k)
 
 When you see NDCG@40, think: "how well did I rank the top 40 assets and how well did I rank the bottom 40 assets?" With ~170 tokens in the universe, k=40 represents approximately the top/bottom 20-25% of assets. This metric equally rewards both:
 
@@ -143,7 +146,7 @@ When you see NDCG@40, think: "how well did I rank the top 40 assets and how well
 !!! note "Random Baseline"
       Random predictions score approximately 0.55 for NDCG@40 with ~170 tokens, not 0.5. See the [detailed explanation](scoring.md#interpretation) for why this happens.
 
-2) [Spearman Correlation](scoring.md#spearman-correlation)
+2) [Spearman Correlation](scoring.md#spearman-correlation-spearman_correlation)
 
 Spearman's rank correlation (ρ) measures how well your predicted ranks align with the true ranks across the entire universe of ~170 tokens. Unlike NDCG@40 which focuses on the 40 extremes, ρ treats all rank positions in the entire universe equally.
 
@@ -155,7 +158,7 @@ The leaderboard provides a **Raw / Unique toggle** to switch between these two v
 
 Three uniqueness metrics are computed for each horizon (10d, 30d):
 
-1) **Unique Spearman** (`unique_spearman`) -- Your predictions are first [neutralized](scoring.md#neutralization) against the meta-model to isolate the orthogonal component, then scored with Spearman correlation against actuals. Positive values mean your unique signal is predictive.
+1) **Unique Spearman** (`unique_spearman`) -- Your predictions are first [neutralized](scoring.md#neutralization-neutralize_predictions) against the meta-model to isolate the orthogonal component, then scored with Spearman correlation against actuals. Positive values mean your unique signal is predictive.
 
 2) **Unique NDCG@40** (`unique_ndcg@40`) -- Same neutralization step, but scored with Symmetric NDCG@40. Measures whether your unique signal correctly identifies the top and bottom 40 assets.
 
@@ -254,7 +257,7 @@ The meta-model is constructed daily using a **points-weighted average** of all s
 
 1. **Uniform Ranking**: Each individual submission's predictions are first converted to uniform rankings [0, 1] for each prediction column (`pred_10d`, `pred_30d`)
 2. **Missing ID Handling**: Any asset IDs missing from individual submissions are filled with neutral rankings of 0.5 *after* the uniform ranking step
-3. **Average slots**: Create a single prediction for each user by taking the arithmetic mean of all normalized rankings across all submission slots for each user.
+3. **Average slots**: Create a single prediction for each user by taking the arithmetic mean of all normalized rankings across that user's **non-experimental** submission slots.
 4. **Points-Weighted Average**: The final meta-model is created by taking a **weighted average** across all users, where each user's weight is proportional to their **CC Points EMA** (Exponential Moving Average with 7-day half-life).
 
 Users with more accumulated CC Points have greater influence on the meta-model. See [CC Points System](points-system.md) for details on how points are earned and how weights are calculated.

docs/install-quickstart.md

@@ -145,6 +145,9 @@ Submit your model's predictions for a challenge. The file must include an `id` c
 
     Use `queue_next=False` (Python) or `--no-queue-next` (CLI) to opt out of auto-rollover during open windows.
 
+!!! info "Experimental submissions and notes"
+    Pass `is_experimental=True` to test a new model without affecting your leaderboard rank, CC Points, or meta-model contribution (requires at least one non-experimental submission in another slot for the same period). Add `notes="..."` to attach a private annotation. See the [experimental submissions FAQ](faq.md#what-is-an-experimental-submission) for edge cases.
+
 === "Python"
 
     ```python
@@ -171,6 +174,15 @@ Submit your model's predictions for a challenge. The file must include an `id` c
 
     # Opt out of auto-rollover (don't queue for next period)
     client.submit_predictions(df=pred_df, queue_next=False)
+
+    # Experimental submission with notes (requires another slot to have a
+    # non-experimental submission for the same period)
+    response = client.submit_predictions(
+        df=pred_df,
+        slot=2,
+        is_experimental=True,
+        notes="2-layer transformer w/ sector embeddings",
+    )
     ```
 
 === "CLI"
@@ -187,6 +199,12 @@ Submit your model's predictions for a challenge. The file must include an `id` c
 
     # Opt out of auto-rollover
     crowdcent submit submission.parquet --no-queue-next
+
+    # Experimental submission with a private note
+    crowdcent submit submission.parquet \
+        --slot 2 \
+        --experimental \
+        --notes "2-layer transformer w/ sector embeddings"
     ```
 
 ## Retrieving Submissions