Drought Forecasting Shiny Module

[willprokopik]

End Goal: A Shiny module that expects daily flow data, identifies and trims events, summarizes AGWR in a table, plots results, and allows the user to plug in flow‑dependent AGWR values to forecast drought outlooks in an adjustable way.

Drought forecast module that can:

• Run the baseflow event ID script on a timeseries and summarize events in a DT::datatable().
• Provide a plotly scatterplot illustrating AGWRC by flow regression analysis.
• Optionally open a modal to inspect individual events in detail.
• Allow users to enter a single AGWRC or AGWRC vs. Flow matrix and project flows forward at 15, 30, 45, and 90 days.

Progress Updates

Update for 1/25:

project environment code (will push this afternoon): updated app code
Quick summary:
The app now follows a modular, retrieval-only data architecture that separates precomputed analysis data from observed flow data. Baseflow event analyses are pulled from GitHub, while historical and anchoring flows are retrieved dynamically from USGS. This structure clarifies how each panel is driven, especially the Drought Forecast, and makes future refactors more straightforward.

Whole App Data Flow

graph TD;

  %% =========================
  %% Whole App Data Flow
  %% =========================

  subgraph UI[User Interface]
    U1[User selects gage]
    U2[User selects event]
    U3[User sets forecast inputs]
  end;

  subgraph Core[Core Reactives]
    GID[GageID reactive]
    AP[AnalysisPoints reactive]
    OD[OriginalDF reactive]
    TD[TrimmedDF reactive]
    ES[EventSummary reactive]
    UD[USGSDaily reactive]
  end;

  subgraph Sources[Data Sources]
    GH[GitHub Analysis CSV]
    US[USGS Daily Flow Service]
  end;

  subgraph Funcs[Key Functions]
    GET[bf_get_gage_analysis]
    STD[bf_standardize_analysis_df]
    NWIS[readNWISdv]
    SUM[make_ben_event_summary]
  end;

  subgraph Panels[Panels and Outputs]
    OV1[Overview flow plot]
    OV2[Overview points plot]
    EVT1[Events table]
    EVT2[Event inspection plots]
    REG1[Regression plot]
    REG2[Regression summary text]
    FC1[Forecast plot]
    FC2[Forecast table]
  end;

  %% --- UI triggers ---
  U1 --> GID;

  %% --- Data retrieval branches ---
  GID --> GET;
  GET --> GH;
  GH --> STD;
  STD --> AP;

  GID --> NWIS;
  NWIS --> US;
  US --> UD;

  %% --- Derived reactives ---
  AP --> OD;
  AP --> TD;
  TD --> SUM;
  SUM --> ES;

  %% --- Panel dependencies ---
  UD --> OV1;
  OD --> OV2;

  ES --> EVT1;
  TD --> EVT2;

  ES --> REG1;
  ES --> REG2;

  U3 --> FC1;
  U3 --> FC2;
  UD --> FC1;
  UD --> FC2;

  %% --- Event selection path ---
  EVT1 --> U2;
  U2 --> EVT2;

  %% =========================
  %% Styling (GitHub-safe)
  %% =========================
  classDef source fill:#eef,stroke:#333,stroke-width:1px;
  classDef reactive fill:#efe,stroke:#333,stroke-width:1px;
  classDef func fill:#ffe,stroke:#333,stroke-width:1px;
  classDef panel fill:#fef,stroke:#333,stroke-width:1px;
  classDef ui fill:#eee,stroke:#333,stroke-width:1px;

  class GH,US source;
  class AP,OD,TD,ES,UD,GID reactive;
  class GET,STD,NWIS,SUM func;
  class OV1,OV2,EVT1,EVT2,REG1,REG2,FC1,FC2 panel;
  class U1,U2,U3 ui;

Loading

Overview and Events Panel

graph TD;

  %% =========================
  %% Overview and Events Panel
  %% =========================

  subgraph Inputs[Inputs]
    GID[GageID]
    EVSEL[Selected Event GroupID]
  end;

  subgraph Sources[Sources]
    GH[GitHub Analysis CSV]
    US[USGS Daily Flow]
  end;

  subgraph Reactives[Reactives]
    AP[AnalysisPoints]
    TD[TrimmedDF]
    ES[EventSummary]
    UD[USGSDaily]
    EVDF[EventData filtered]
  end;

  subgraph Outputs[Outputs]
    HF[Historical flow plot]
    TAB[Events table]
    EP[Event inspection plots]
  end;

  %% Flow series
  GID --> US;
  US --> UD;
  UD --> HF;

  %% Event library
  GID --> GH;
  GH --> AP;
  AP --> TD;
  TD --> ES;
  ES --> TAB;

  %% Event inspection
  TAB --> EVSEL;
  EVSEL --> EVDF;
  TD --> EVDF;
  EVDF --> EP;

  classDef source fill:#eef,stroke:#333,stroke-width:1px;
  classDef reactive fill:#efe,stroke:#333,stroke-width:1px;
  classDef output fill:#fef,stroke:#333,stroke-width:1px;
  classDef input fill:#eee,stroke:#333,stroke-width:1px;

  class GH,US source;
  class AP,TD,ES,UD,EVDF reactive;
  class HF,TAB,EP output;
  class GID,EVSEL input;

Loading

Regression Panel

graph TD;

  %% =========================
  %% Regression Panel
  %% =========================

  subgraph Inputs[Inputs]
    GID[GageID]
    XSEL[X variable selection]
    YSEL[Y variable selection]
  end;

  subgraph Source[Source]
    GH[GitHub Analysis CSV]
  end;

  subgraph Reactives[Reactives]
    AP[AnalysisPoints]
    TD[TrimmedDF]
    ES[EventSummary]
    LM[Linear model]
  end;

  subgraph Outputs[Outputs]
    RP[Regression plot]
    RT[Regression text]
  end;

  GID --> GH;
  GH --> AP;
  AP --> TD;
  TD --> ES;

  XSEL --> LM;
  YSEL --> LM;
  ES --> LM;

  LM --> RP;
  LM --> RT;

  classDef source fill:#eef,stroke:#333,stroke-width:1px;
  classDef reactive fill:#efe,stroke:#333,stroke-width:1px;
  classDef output fill:#fef,stroke:#333,stroke-width:1px;
  classDef input fill:#eee,stroke:#333,stroke-width:1px;

  class GH source;
  class AP,TD,ES,LM reactive;
  class RP,RT output;
  class GID,XSEL,YSEL input;

Loading

Drought Forecast Panel

graph TD;

  %% =========================
  %% Drought Forecast Panel
  %% =========================

  subgraph Inputs[Inputs]
    GID[GageID]
    FSD[Forecast start date]
    FH[Forecast horizon]
    AG[AGWRC input]
  end;

  subgraph Sources[Sources]
    US[USGS Daily Flow]
    GH[GitHub Analysis CSV]
  end;

  subgraph Reactives[Reactives]
    UD[USGSDaily]
    Q0[Starting flow Q0]
    DD[Delta days]
    PROJ[Projection table]
    ES[EventSummary]
  end;

  subgraph Outputs[Outputs]
    FP[Forecast plot]
    FT[Forecast table]
  end;

  %% Anchor to observed flow
  GID --> US;
  US --> UD;
  FSD --> Q0;
  UD --> Q0;

  %% Projection scaffolding
  FH --> DD;
  AG --> PROJ;
  Q0 --> PROJ;
  DD --> PROJ;

  %% Outputs
  PROJ --> FP;
  PROJ --> FT;

  %% Event library (future enhancement)
  GID --> GH;
  GH --> ES;

  classDef source fill:#eef,stroke:#333,stroke-width:1px;
  classDef reactive fill:#efe,stroke:#333,stroke-width:1px;
  classDef output fill:#fef,stroke:#333,stroke-width:1px;
  classDef input fill:#eee,stroke:#333,stroke-width:1px;

  class GH,US source;
  class UD,Q0,DD,PROJ,ES reactive;
  class FP,FT output;
  class GID,FSD,FH,AG input;

Loading

Key updates (as of 2/10):

• Added a data source selector (Model vs Gage) that controls how both raw and analyzed data are loaded throughout the app.
• Implemented dual data ingestion logic:
  • Raw model data pulled directly from GitHub CSVs (daily flows) and used for historical flow overlays and forecast initialization.
  • Raw gage data pulled from USGS via dataRetrieval for equivalent historical context.
  • Analyzed event data (baseflow events, AGWRC, regression inputs) pulled from GitHub for both model and gage cases.
• Fixed a critical issue with model raw data date parsing: model CSVs contained empty thisdate fields, so dates are now robustly constructed from year/month/day when needed. This restores full plotting and forecasting functionality for model runs.
• Confirmed that all plots (historical flow, AGWRC vs flow regression, and drought forecast) now render correctly for both data sources with no runtime errors.

[willprokopik]

Updates for 11/18 Meeting

At the end of last week, I met with Connor to walk through some Shiny examples he shared. He explained his typical workflow when building Shiny apps, including:

• Using Shydrology: USGS gages surrounding Virginia, critical low flow calculations, low flows over time, and non‑exceedance probabilities.
• Structuring apps with the three‑file setup (global, ui, server) for clarity and modularity.
• Wrapping every input with an ns <- NS(id) when needed.
• Applying reactivity notation carefully.

Connor also reiterated that the current app already supports querying USGS gages, downloading gage data, and calculating statistics. The next step is to expand this functionality with a drought forecast module that can:

• Run the baseflow event ID script on a timeseries and summarize events in a DT::datatable().
• Provide a plotly scatterplot illustrating AGWRC by flow regression analysis.
• Optionally open a modal to inspect individual events in detail.
• Allow users to enter a single AGWRC or AGWRC vs. Flow matrix and project flows forward at 15, 30, 45, and 90 days.

To prepare, I built a practice prototype based on his examples. This lightweight app retrieves USGS gage data, lets the user select a date and AGWRC value, and then projects flows forward using a simple formula. Results are displayed in a DT table and a plotly chart. This exercise gave me some good practice with modular design, eventReactive, and integrating DT/plotly outputs.

End Goal (as discussed with Connor): A Shiny module that expects daily flow data, identifies and trims events, summarizes AGWR in a table, plots results, and allows the user to plug in flow‑dependent AGWR values to forecast drought outlooks in an adjustable way.

[willprokopik]

Notes from 11/18 Meeting

Important things to pay attention to moving forward (I will continue to add to this as needed):

• Historical data leading up to projection
• Axes on 0

[willprokopik]

App Prototype Update

This update reflects an unpolished prototype of the drought-forecast module. The core structure and functionality are in place, but several components still need refinement, cleanup, and design alignment with the final vision. The purpose of this version is mostly to validate the workflow, data connections, and establish a strong foundation to enhance.

Site Selection Panel (Sidebar)
What the user sees:

• A simple set of radio buttons:
  • Cootes Store
  • Mount Jackson
  • Strasburg
    What this controls:
• Determines which pair of datasets (historical + trimmed events) are loaded:
  • *_original_analysis_df.csv --> daily historical flow (Ben's most recent output)
  • *_trimmed_event_results.csv --> output from the baseflow-event trimming script (Ben's most recent output)
  • All downstream visualizations and tables update dynamically when the site changes.

“Overview & Events” Tab
This is the default landing tab when the user opens the app.
a. Historical Flow Plot

• UI element:
  • A Plotly line plot showing recent streamflow (last ~2 years).
• Data source:
  • Pulled from the site’s *_original_analysis_df.csv.
• Purpose:
  • Gives immediate context about recent hydrologic behavior before diving into events or forecasts.
  • y-axis anchored to 0 (rangemode = "tozero").

b. Event Summary Table

• UI element:
  • A sortable, paginated DT::datatable() listing all trimmed recession events.
• Columns include:
  • GroupID, start/end dates, duration (n_days) this needs to be fixed
  • summary flow metrics (median/min/max)
  • event-level AGWRC
  • event R² probably not needed
• Data source:
  • Computed from *_trimmed_event_results.csv.
• Notes:
  • Each row corresponds to one identified recession event.
  • User can click a row for deeper inspection.

c. “Inspect Selected Event” Modal

• UI element:
  • Clicking the “Inspect Selected Event” button opens a modal dialog with two Plotly charts:
• Flow vs. Date for that event
• AGWR vs. Date for that event
• Data source:
  • Same trimmed dataset; filtered by selected GroupID.
• Purpose:
  • Allows users to visually inspect the shape and behavior of individual recessions and AGWR trajectories.

“AGWRC vs Flow Regression” Tab
a. Scatterplot + Regression Line

• UI element:
  • Plotly scatterplot:
    • x = median flow for each event
    • y = event-level AGWRC
    • Includes a fitted regression line (lm(AGWRC ~ log(flow))).
• Data source:
  • Event summaries computed from trimmed data.
• Purpose:
  • Shows how AGWRC changes as a function of flow.
  • Useful for future implementation of flow-dependent AGWRC matrices.

b. Regression Summary

• UI element:
  • Printed model output showing coefficients, R², etc.
• Purpose:
  • Offers transparency into how AGWRC is modeled relative to flow for the selected gage.

“Drought Forecast” Tab
a. Forecast Controls

• UI elements:
  • dateInput: pick the start date for projection
  • numericInput: enter a single AGWRC value
• Behavior:
  • Start date automatically updates to the latest available date in the historical dataset whenever the site changes.

b. Forecast Plot (Observed + Projected)

• UI element:
  • Plotly line graph with two components:
    • Observed: last 90 days before the selected start date
    • Projected: flows at 15, 30, 45, 90 days into the future
    • using Q_future = Q_start * AGWRC^days
• Data source:
  • Observed flows --> original analysis dataset
  • Projections --> computed inside the module
• Purpose:
  • Provides a quick “what-if” drought outlook using a user-defined AGWRC.

c. Projection Table

• UI element:
  • DT table containing the projected flows for 15 / 30 / 45 / 90 days.

[rburghol]

@willprokopik This is an excellent start, and I am excited to start test driving. Comments/questions:

• I appreciate the fitted regression, this is great context, and while the overall trend may be a l. curious as to what the "Characteristic event flow, median cfs" is.
• Really appreciate the detailed outline. This is a smart approach. I suggest this gets moved to the body of the model -- unless @COBrogan feels differently.
• The flow-dependent AGWRC lookup is a very great idea!
• For the purpose of test-driving, as soon as you can get a link to the code, and brief instructons on how to use it, I would greatly appreciate it!

[willprokopik]

@rburghol responding to your comment:
The "Characteristic event flow, median cfs" in the fitted regression is just the median flow for the entire event. This value is also produced in the table on the opening screen that shows the values associated with each event:

There are a few things I have noted to adjust (such as fixing the n_days column, making the Drought Forecast a bit more intuitive with an event-based dropdown selection) that should not take too long. I'm going to be very busy the rest of today and tomorrow, but I should have these adjustments done by noon on Friday (12/5) and will push the code up for test driving. I'll include some additional instructional documentation to go along with it as well.

[COBrogan]

@willprokopik Sorry I missed the meeting, but this looks like a great start! Seems like you have a very solid foundation here with a lot of the features we've discussed! One question/comment I had is regarding the input data. You noted:

Determines which pair of datasets (historical + trimmed events) are loaded:
*_original_analysis_df.csv --> daily historical flow (Ben's most recent output)
*_trimmed_event_results.csv --> output from the baseflow-event trimming script (Ben's most recent output)

This is very valuable for testing and its great that you have that set-up. I do want to note that we will need this to be able to function with any USGS gage. So, at some point, we will need to integrate Ben's script directly into this such that it can take data directly from dataRetrieval and create the data.frames/lists the rest of your module expects. (You may have already covered this in the meeting so I apologize if I'm just reiterating a discussion point). After that, we should be able to run a test with integrating it in the bigger Shiny app and see what happens! If @rburghol you want to try this out sooner, we can spend five minutes with Will next Tuesday to get the test version up on our DEQ server.

[rburghol]

@COBrogan @willprokopik
Thanks for the quick review Connor! We spent our first 45 minutes reviewing the collab between Ilonah and Ben today, and then I got a call from DWR and had to jet -- so basically, Will's stuff didn't get much sunshine.

I agree with your assessment and also great point about the need to integrate the gage analysis functions in there -- I am itching to do the Potomac at Point of Rocks, so this is a crucial next step.

Maybe integration of the whole analysis workflow should be the priority? Then we can add bells and whistles? We will have to think of the best way to make that integration hum... and then in turn those design decisions will inform the work that needs to be done with @benhdye and @ilonah22 areas of focus.

Proposed Script function definition:

• Script bf_events.R takes USGS gage and optional date range (maybe other args later) and creates csv of data.
  • Use: Rscript bf_events.R gage_id [tsfile(default=FALSE)] [startdate(default=FALSE)] [enddate=(default=FALSE)] [pvalue(default=0.3)] [output_file(default=working/bf_events_{gage_id}.csv)] [savedb(default=FALSE)]
    • gage_id: usgs gage (insure leading zero) or watershed code (if supply a timeseries file)
    • tsfile: FALSE or path to a CSV with properly formatted flow column (we must decide on a name of this column, and be able to handle the fact that the USGS time series has a different column name)
    • startdate:
    • enddate:
    • pvalue: what minimum p-value for Mann-Kendall to insure NO trend
    • output_file: where to save the events
    • savedb: future, we may opt to store this data as a timeseries in the dh_timeseries table
• This app could call bf_events.R IF the user requests a gage that is not already done, or requests a re-do of the gage in question.

I welcome everyone's thoughts and then once we agree on the basic shape of the script (if that's what we agree is needed) I want us to make this function definition into it's own issue.

[rburghol]

@willprokopik

but I should have these adjustments done by noon on Friday (12/5) and will push the code up for test driving

This is awesome - thanks so much!!