diff --git a/pages/catalog-api.md b/pages/catalog-api.md index ebf1500a8..57ac37d48 100644 --- a/pages/catalog-api.md +++ b/pages/catalog-api.md @@ -5,16 +5,112 @@ layout: page The Data.gov Catalog API provides access to metadata about datasets published by federal, state, local, and tribal governments. You can use it to search for datasets, filter by organization or topic, and retrieve detailed information about individual records. -**4/9/2026: Note on upcoming changes:** The base URL for the data.gov APIs will change when Data.gov routes its APIs through api.data.gov. This is expected to occur soon. If you're building a new integration, avoid hard-coding the base URL and check this page for updates before going to production. Email us with any questions at datagovhelp@gsa.gov +**Base URL:** `https://api.gsa.gov/technology/datagov/v4/` -**Base URL:** `https://catalog.data.gov` +Note: This API replaces the previous CKAN-based API. The prior endpoint remains available in a read-only state for existing integrations, but new development should use this API. -No API key is required. All endpoints are publicly accessible and return JSON. +**Authentication:** This API is accessed through [api.data.gov](https://api.data.gov), which manages API keys, rate limiting, and usage tracking. [Sign up for a free API key](https://api.data.gov/signup) and include it in the `X-Api-Key` header with every request. -Note: This API replaces the previous CKAN-based API. The prior endpoint remains available in a read-only state for existing integrations, but new development should use this API. +For initial exploration, you can use `DEMO_KEY` without signing up: + +``` +curl -H 'X-Api-Key: DEMO_KEY' 'https://api.gsa.gov/technology/datagov/v4/search?q=climate' +``` + +Replace `DEMO_KEY` with your personal key for production use or any automated querying. See [API Key and Rate Limit Errors](#api-key-and-rate-limit-errors) below. + +--- + +## Table of Contents + +1. [Authentication](#authentication) + +2. [Search Datasets](#search-datasets) + +3. [Get Keywords](#get-keywords) + +4. [Search Locations](#search-locations) + +5. [Get Location Geometry](#get-location-geometry) + +6. [Get Organizations](#get-organizations) + +7. [Get Harvest Record](#get-harvest-record) + +8. [Get Harvest Record Raw](#get-harvest-record-raw) + +9. [Get Harvest Record Transformed](#get-harvest-record-transformed) + +10. [Pagination](#pagination) + +11. [Response Codes and Errors](#response-codes-and-errors) --- + + +## Authentication + +All requests must include an API key issued through [api.data.gov](https://api.data.gov). + +**[Sign up for a free API key](https://api.data.gov/signup)** + +### Passing Your API Key + +Include your key in the `X-Api-Key` HTTP header: + +``` +curl -H 'X-Api-Key: YOUR_KEY_HERE' 'https://api.gsa.gov/technology/datagov/v4/search?q=climate' +``` + +### DEMO_KEY + +For initial exploration, use `DEMO_KEY` without signing up: + +``` +curl -H 'X-Api-Key: DEMO_KEY' 'https://api.gsa.gov/technology/datagov/v4/search?q=climate' +``` + +`DEMO_KEY` has much lower rate limits and is not suitable for production use or automated queries. + +### Rate Limits + + + + + + + + + + + + + + + + + + + + + + +
Key typeHourly limitDaily limit
Personal API key1,000 requests/hour--
DEMO_KEY30 requests/IP/hour50 requests/IP/day
+ +Every response includes these headers so you can track your current usage: + +``` +X-RateLimit-Limit: 1000 +X-RateLimit-Remaining: 998 +``` + +If you exceed your limit, you will receive an HTTP `429 Too Many Requests` response. The block lifts automatically after one hour. If you need higher limits, contact us at [datagovhelp@gsa.gov](mailto:datagovhelp@gsa.gov). + + +--- + + ## Getting Started Here is a complete working example that walks through finding NASA climate datasets step by step. @@ -25,7 +121,7 @@ To filter results by organization, you first need the organization's slug. Fetch **Request:** ``` -GET https://catalog.data.gov/api/organizations +GET https://api.gsa.gov/technology/datagov/v4/organizations ``` **Look for NASA in the response:** @@ -51,7 +147,7 @@ Use the `q` parameter for your keyword and `org_slug` to filter by organization. **Request:** ``` -GET https://catalog.data.gov/search?q=climate&org_slug=nasa&per_page=3 +GET https://api.gsa.gov/technology/datagov/v4/search?q=climate&org_slug=nasa&per_page=3 ``` **Response:** @@ -79,7 +175,7 @@ GET https://catalog.data.gov/search?q=climate&org_slug=nasa&per_page=3 } ``` -The response includes an `after` value, which means there are more results available. +The response includes an `after` value, indicating that more results are available. ### Step 3: Get the next page @@ -87,25 +183,11 @@ Pass the `after` value from the previous response to retrieve the next page. Kee **Request:** ``` -GET https://catalog.data.gov/search?q=climate&org_slug=nasa&per_page=3&after=WzY5LjM0NDY5NiwwLCJiYmRhZGNmYi00NDM1LTQzZWUtYjhlMy0yMzZiZjBlZDEwODIiXQ== +GET https://api.gsa.gov/technology/datagov/v4/search?q=climate&org_slug=nasa&per_page=3&after=WzY5LjM0NDY5NiwwLCJiYmRhZGNmYi00NDM1LTQzZWUtYjhlMy0yMzZiZjBlZDEwODIiXQ== ``` Continue repeating this step until the response no longer includes an `after` field, which means you have reached the last page. For more details see [Pagination](#pagination) below. ---- - -## Table of Contents - -1. [Search Datasets](#search-datasets) -2. [Get Keywords](#get-keywords) -3. [Search Locations](#search-locations) -4. [Get Location Geometry](#get-location-geometry) -5. [Get Organizations](#get-organizations) -6. [Get Harvest Record](#get-harvest-record) -7. [Get Harvest Record Raw](#get-harvest-record-raw) -8. [Get Harvest Record Transformed](#get-harvest-record-transformed) -9. [Pagination](#pagination) -10. [Response Codes and Errors](#response-codes-and-errors) --- @@ -205,12 +287,12 @@ Search the catalog for datasets using keywords, filters, and sorting options. ### Example Requests ``` -GET https://catalog.data.gov/search?q=water+quality -GET https://catalog.data.gov/search?q=climate&sort=popularity&per_page=25 -GET https://catalog.data.gov/search?org_slug=nasa&per_page=10 -GET https://catalog.data.gov/search?org_type=Federal+Government&spatial_filter=geospatial -GET https://catalog.data.gov/search?q=education&after=WzEwMC4wNjEzNiwwLCJiMWEzOTY3YzJhMTExZjE2NzgxN2IwMTI0YzUyYjBhYyJd -GET https://catalog.data.gov/search?spatial_geometry={"type":"Polygon","coordinates":[[[-109.05,37.0],[-102.05,37.0],[-102.05,41.0],[-109.05,41.0],[-109.05,37.0]]]}&spatial_within=true +GET https://api.gsa.gov/technology/datagov/v4/search?q=water+quality +GET https://api.gsa.gov/technology/datagov/v4/search?q=climate&sort=popularity&per_page=25 +GET https://api.gsa.gov/technology/datagov/v4/search?org_slug=nasa&per_page=10 +GET https://api.gsa.gov/technology/datagov/v4/search?org_type=Federal+Government&spatial_filter=geospatial +GET https://api.gsa.gov/technology/datagov/v4/search?q=education&after=WzEwMC4wNjEzNiwwLCJiMWEzOTY3YzJhMTExZjE2NzgxN2IwMTI0YzUyYjBhYyJd +GET https://api.gsa.gov/technology/datagov/v4/search?spatial_geometry={"type":"Polygon","coordinates":[[[-109.05,37.0],[-102.05,37.0],[-102.05,41.0],[-109.05,41.0],[-109.05,37.0]]]}&spatial_within=true ``` @@ -279,8 +361,8 @@ GET https://catalog.data.gov/search?spatial_geometry={"type":"Polygon","coordina "systemOfRecords": "https://www2.ed.gov/notices/pai/pai-18-13-01.pdf", "temporal": "2012/2012" }, - "harvest_record": "https://catalog.data.gov/harvest_record/0be6d0c0-8383-4966-acd1-38b0d7baea3c", - "harvest_record_raw": "https://catalog.data.gov/harvest_record/0be6d0c0-8383-4966-acd1-38b0d7baea3c/raw" + "harvest_record": "https://api.gsa.gov/technology/datagov/v4/harvest_record/0be6d0c0-8383-4966-acd1-38b0d7baea3c", + "harvest_record_raw": "https://api.gsa.gov/technology/datagov/v4/harvest_record/0be6d0c0-8383-4966-acd1-38b0d7baea3c/raw" } ] } @@ -642,8 +724,8 @@ Retrieve a list of the most commonly used keywords across all datasets, along wi ### Example Requests ``` -GET https://catalog.data.gov/api/keywords -GET https://catalog.data.gov/api/keywords?size=50&min_count=100 +GET https://api.gsa.gov/technology/datagov/v4/keywords +GET https://api.gsa.gov/technology/datagov/v4/keywords?size=50&min_count=100 ``` ### Response @@ -706,7 +788,7 @@ Search for location names to use with spatial filtering. This endpoint is design ### Example Request ``` -GET https://catalog.data.gov/api/locations/search?q=Colorado&size=5 +GET https://api.gsa.gov/technology/datagov/v4/locations/search?q=Colorado&size=5 ``` ### Response @@ -761,7 +843,7 @@ Retrieve the geographic boundary (GeoJSON geometry) for a specific location by i ### Example Request ``` -GET https://catalog.data.gov/api/location/a1b2c3d4-e5f6-7890-abcd-ef1234567890 +GET https://api.gsa.gov/technology/datagov/v4/location/a1b2c3d4-e5f6-7890-abcd-ef1234567890 ``` ### Response @@ -792,7 +874,7 @@ No query parameters. Returns all organizations. ### Example Request ``` -GET https://catalog.data.gov/api/organizations +GET https://api.gsa.gov/technology/datagov/v4/organizations ``` ### Response @@ -892,7 +974,7 @@ Retrieve metadata about a specific harvest record by its ID. Harvest records tra ### Example Request ``` -GET https://catalog.data.gov/harvest_record/d0e03fb2-f885-4b1d-8feb-2d8acc93f4f8 +GET https://api.gsa.gov/technology/datagov/v4/harvest_record/d0e03fb2-f885-4b1d-8feb-2d8acc93f4f8 ``` ### Response @@ -963,7 +1045,7 @@ Retrieve the original, unmodified source payload from a harvest record exactly a ### Example Request ``` -GET https://catalog.data.gov/harvest_record/d0e03fb2-f885-4b1d-8feb-2d8acc93f4f8/raw +GET https://api.gsa.gov/technology/datagov/v4/harvest_record/d0e03fb2-f885-4b1d-8feb-2d8acc93f4f8/raw ``` ### Response @@ -1011,7 +1093,7 @@ Retrieve the transformed DCAT-US payload for a harvest record. This is the versi ### Example Request ``` -GET https://catalog.data.gov/harvest_record/000c4ce7-90c6-405c-8ed7-3ae06c45005c/transformed +GET https://api.gsa.gov/technology/datagov/v4/harvest_record/000c4ce7-90c6-405c-8ed7-3ae06c45005c/transformed ``` ### Response @@ -1037,12 +1119,12 @@ The `/search` endpoint uses cursor-based pagination. This approach is more relia ``` # First request -GET https://catalog.data.gov/search?q=water&per_page=10 +GET https://api.gsa.gov/technology/datagov/v4/search?q=water&per_page=10 # Response includes: "after": "WzEwMC4wNjEzNiwwLCJiMWEz..." # Second request -GET https://catalog.data.gov/search?q=water&per_page=10&after=WzEwMC4wNjEzNiwwLCJiMWEz... +GET https://api.gsa.gov/technology/datagov/v4/search?q=water&per_page=10&after=WzEwMC4wNjEzNiwwLCJiMWEz... ``` Keep all other parameters the same across pages. Changing `q`, `sort`, or filter parameters while paginating will return inconsistent results. @@ -1098,3 +1180,51 @@ For validation errors (422), additional detail is provided: } } ``` +### API Key and Rate Limit Errors + +These errors are returned by api.data.gov before the request reaches the catalog. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Status CodeError CodeMeaning
403API_KEY_MISSINGNo API key was supplied. Include your key in the X-Api-Key header.
403API_KEY_INVALIDThe API key supplied is not valid. Double-check your key or sign up for a new one.
403API_KEY_UNVERIFIEDYour API key has not been verified yet. Check your email to complete signup.
429OVER_RATE_LIMITYou have exceeded your rate limit. Wait one hour for the block to lift automatically.
+ +Rate limit error responses use this JSON format: + +```json +{ + "error": { + "code": "OVER_RATE_LIMIT", + "message": "You have exceeded your rate limit. Try again later." + } +} +``` + +For more information see the [api.data.gov developer manual](https://api.data.gov/docs/developer-manual/). diff --git a/pages/data-gov-apis.md b/pages/data-gov-apis.md new file mode 100644 index 000000000..64a388f67 --- /dev/null +++ b/pages/data-gov-apis.md @@ -0,0 +1,100 @@ +--- +title: Data.gov APIs +layout: page +--- + +The Data.gov APIs provide programmatic access to federal open data. Use them to search for datasets, retrieve metadata, look up harvest sources, and check harvest job status, all in support of the open data principles established by the [OPEN Government Data Act](https://www.congress.gov/bill/115th-congress/house-bill/4174). + +**To get started, you will need a free API key.** [Register at open.gsa.gov](https://open.gsa.gov/api/datadotgov/) + +--- + +## What is api.data.gov? + +When you register for an API key and make requests to the Data.gov APIs, you are going through [api.data.gov](https://api.data.gov) (GSA's shared API gateway). It manages authentication, rate limiting, and usage tracking for Data.gov's APIs. + +api.data.gov is not a barrier to access, rather it is what makes access reliable. It ensures that federal open data is available consistently and at scale, in the spirit of the OPEN Government Data Act's mandate that government data be open and machine-readable by default. + +Your API key is free. It identifies your application, allows you to track your own usage, and gives you a higher rate limit than the anonymous `DEMO_KEY`. For more information about how api.data.gov works, visit [api.data.gov/docs/developer-manual/](https://api.data.gov/docs/developer-manual/). + +--- + +## API Key and Rate Limits + +All requests must include your API key in the `X-Api-Key` HTTP header: +``` +curl -H 'X-Api-Key: YOUR_KEY_HERE' 'https://api.gsa.gov/technology/datagov/v4/search?q=climate' +``` +For initial exploration, you can use `DEMO_KEY` without registering: +``` +curl -H 'X-Api-Key: DEMO_KEY' 'https://api.gsa.gov/technology/datagov/v4/search?q=climate' +``` + +`DEMO_KEY` has much lower rate limits and is not suitable for production use or automated queries. + + + + + + + + + + + + + + + + + + + + + + +
Key typeHourly limitDaily limit
Personal API key1,000 requests/hour--
DEMO_KEY30 requests/IP/hour50 requests/IP/day
+ +Every response includes `X-RateLimit-Limit` and `X-RateLimit-Remaining` headers so you can track your usage. If you exceed your limit, you will receive an HTTP `429 Too Many Requests` response. The block lifts automatically after one hour. For higher limits, contact us at datagovhelp@gsa.gov. + +--- + +## Available APIs + +### Catalog API + +The Catalog API provides access to metadata about datasets published by federal, state, local, and tribal governments. Use it to search for datasets, filter by organization or topic, and retrieve detailed information about individual records. + +**Base URL:** `https://api.gsa.gov/technology/datagov/v4/` + +**Example request:** + +``` +curl -H 'X-Api-Key: DEMO_KEY' 'https://api.gsa.gov/technology/datagov/v4/search?q=climate' +``` + +[View full Catalog API documentation](/catalog-api/) + + +--- + +### Harvester API + +The Harvester API provides access to Data.gov's harvest infrastructure which is the system that collects dataset metadata from federal agencies and other publishers. Use it to look up harvest sources, check job status, and investigate harvest errors. + +**Base URL:** `https://api.gsa.gov/technology/datagov_harvest/v2/` + +**Example request:** + +``` +curl -H 'X-Api-Key: DEMO_KEY' 'https://api.gsa.gov/technology/datagov_harvest/v2/harvest_sources/' +``` + +View full Harvester API documentation insert link + + +--- + +## Questions or Issues? + +Contact the Data.gov team at [datagovhelp@gsa.gov](mailto:datagovhelp@gsa.gov).