Releases: myominnoo/wpgdata
Releases · myominnoo/wpgdata
wpgdata 0.3.0
Breaking changes
peg_get(),peg_query(), andpeg_all()have been removed and
consolidated into a singlepeg_data()function. Update existing code:peg_get("id")→peg_data("id")peg_query("id", filter = x > 1, select = "col")→peg_data("id", filter = x > 1, select = "col")peg_all("id")→peg_data("id")
New functions
peg_data()— replacespeg_get(),peg_query(), andpeg_all()with a
unified interface for fetching dataset rows. Supports server-side filtering,
column selection, row ordering, and offset pagination viatopandskip.
All pages are fetched in parallel usinghttr2::req_perform_parallel(),
delivering significant throughput improvements over the previous sequential
approach.
peg_catalogue()
- Catalogue pages and per-dataset metadata are now fetched in parallel using
httr2::req_perform_parallel(), with the count request and first page fired
simultaneously to save a full round trip. - Metadata is now sourced from the
/api/views/{id}.jsonendpoint per dataset,
replacing the Socrata Discovery API as the sole data source. - Added new columns:
row_count,col_count,view_count,group,
department,update_frequency,quality_rank,license,license_link,
tags,license_id,rows_updated_at,view_last_modified,
publication_date,index_updated_at. row_countis now sourced from cached column statistics (cachedContents),
making it fast without requiring a separate OData count request per dataset.- Removed the default
limitof 200 — all available datasets are fetched by
default. - Results are sorted by
rows_updated_atdescending. - HTTP errors and network failures for individual datasets are caught and
skipped with a warning rather than aborting the entire fetch.
peg_info()
- Added input validation for
dataset_id(rejectsNULL,NA, empty string,
non-character, and length > 1). - Added
httr2::req_timeout()andhttr2::req_retry()with exponential
backoff for transient 429, 500, and 503 responses. - Fixed double-read of the response body; the empty-body check is now handled
by a singletryCatcharoundresp_body_json().
peg_metadata()
- Added input validation for
dataset_idmatchingpeg_info(). - Added
httr2::req_timeout()andhttr2::req_retry()with exponential
backoff for transient 429, 500, and 503 responses. - Fixed double-read of the response body.
- Added
%||% NA_character_fallbacks forname,field_name, andtype—
previously these would silently insertNULLfor malformed column entries. - Replaced soft-deprecated
purrr::map_dfr()withpurrr::map()+
purrr::list_rbind().
wpgdata 0.2.0
New functions
peg_catalogue()— list all datasets available on the City of Winnipeg
Open Data Portal with name, ID, category, and last updated date.
Internal changes
- Added
get_catalogue_count(),fetch_catalogue_page(), and
parse_catalogue()internal helpers toutils.R. - Imported
rlang::.datapronoun to suppress R CMD check notes in
dplyr::mutate()calls.
wpgdata 0.1.0
Initial release
peg_get()— fetch the first page of a datasetpeg_query()— filter, select, sort and paginate datasetspeg_all()— download all rows with automatic paginationpeg_metadata()— retrieve column names and typespeg_info()— retrieve dataset-level information- Supports both R expressions and raw OData strings in
peg_query() - Automatic pagination detection and progress bar in
peg_all()