snowexsql-v1.1.0-rc1

Release Notes: v1.1.0-rc1

Release Date: TBD
Previous Release: v1.0.0

Overview

This release candidate adds comprehensive AWS Lambda support for serverless database access, enabling users to query SnowEx data without direct database connections or heavy geospatial dependencies.

New Features

Lambda Client (snowexsql.lambda_client)

• SnowExLambdaClient: New client class for accessing SnowEx database via AWS Lambda

  • Automatically discovers and exposes all measurement classes (PointMeasurements, LayerMeasurements)
  • Provides serverless access without requiring database credentials locally

• Measurement Class Accessors: Direct access to measurement classes via attributes

  client = SnowExLambdaClient()
  classes = client.get_measurement_classes() 
  PointMeasurements = classes['PointMeasurements']
  LayerMeasurements = classes['LayerMeasurements']
  df = client.PointMeasurements.from_filter(instrument='camera', limit=10)
  instruments = client.LayerMeasurements.all_instruments

• Dynamic Method Discovery: Automatically mirrors all API methods and properties without manual synchronization

  • Supports from_filter(), from_area(), from_unique_entries()
  • Supports all property accessors: all_instruments, all_campaigns, all_types, etc.

• Server-Side Spatial Queries: from_area() uses PostGIS for efficient database-side spatial filtering

  • Supports point + buffer and polygon geometries
  • Handles WKT string input or shapely geometry objects
  • Automatic CRS transformation

• Smart DataFrame Conversion:

  • Returns GeoDataFrame when geometry data is available (requires geopandas)
  • Falls back to pandas DataFrame when geopandas not installed
  • Handles multiple geometry formats (WKB hex, WKT, GeoJSON)

Lambda Handler (snowexsql.lambda_handler)

• Auto-Discovery System: Automatically discovers measurement classes based on naming conventions

  • Classes ending in 'Measurements' with a MODEL attribute are automatically exposed
  • No manual registration required for new measurement classes

• AWS Secrets Manager Integration: Secure credential management

  • Retrieves database credentials from AWS Secrets Manager
  • Writes temporary credentials file for API compatibility

• Action Routing: Handles multiple request types:

  • test_connection: Database connectivity verification
  • query: Raw SQL query execution
  • {ClassName}.{method}: Class-based API calls (e.g., PointMeasurements.from_filter)
  • Property access: {ClassName}.all_{property} (e.g., PointMeasurements.all_instruments)

• Response Standardization: Consistent JSON response format

  • Success: {action, data, count}
  • Error: {error, action}

API Enhancements (snowexsql.api)

PostGIS Spatial Queries

• from_area() Improvements: Now uses pure PostGIS for spatial filtering
  • Eliminates dependency on geoalchemy2.functions for spatial operations
  • Automatic SRID detection from database
  • Smart CRS transformation (transforms search geometry to match database SRID for index usage)
  • Supports WKT string input in addition to shapely geometries
  • Works in both local and Lambda environments

Verbose Mode Support

• New verbose Parameter: Added to from_filter() and from_area()

  • verbose=False (default): Returns minimal columns from primary table
  • verbose=True: Returns denormalized data with related table information
  • Includes explicit join management to avoid cartesian products

• Denormalized Columns (when verbose=True):

  • PointMeasurements: observation details, instrument specs, measurement types
  • LayerMeasurements: site details, instrument specs, measurement types, weather conditions

New Class Methods

• _build_select_clause(verbose): Customizable column selection per class
• _add_base_joins(): Minimal joins for non-verbose queries
• _add_verbose_joins(): Explicit joins for verbose mode

GeoDataFrame Compatibility

• query_to_geopandas(): Updated to handle both local and Lambda environments
  • Returns GeoDataFrame when geopandas available (local)
  • Returns pandas DataFrame with WKB/WKT geometry when geopandas not available (Lambda)
  • Client-side conversion to GeoDataFrame handled by lambda_client

Deployment Infrastructure

Docker Configuration (deployment/docker/)

• Lambda-Optimized Dockerfile: AWS Lambda Python 3.12 base image
• Minimal Dependencies: requirements-lambda.txt with lightweight packages
  • Excludes heavy dependencies (geopandas, rasterio, fiona)
  • Includes core requirements: sqlalchemy, psycopg2-binary, pandas, shapely, geoalchemy2

Deployment Scripts (deployment/scripts/)

• deploy.sh: Automated deployment pipeline

  • Builds Docker image
  • Pushes to AWS ECR
  • Updates Lambda function code
  • Optional function testing

• test_lambda.sh: Comprehensive testing script

  • Tests database connectivity
  • Verifies response format
  • Checks CloudWatch logs

• update_lambda_config.sh: Configuration management

  • Updates timeout (default: 90s)
  • Updates memory allocation (default: 1024MB)

AWS Policies (deployment/aws/)

• ecr_policy.json: ECR repository permissions for Lambda
• secrets_policy.json: Secrets Manager access policy

Breaking Changes

RasterMeasurements

• Lambda Limitation: RasterMeasurements not fully supported in Lambda environment
  • Raises ImportError with helpful message directing users to local API
  • Requires rasterio which is too heavy for Lambda deployment

Performance Improvements

Optimized Database Queries

• SRID Detection: Automatically detects database SRID to avoid unnecessary transformations
• Index-Aware Queries: Transforms search geometry (not database geometry) for optimal index usage
• EXISTS Clauses: Uses EXISTS instead of joins for large table counts (29GB+ tables)

Spatial Query Optimization

• PostGIS Server-Side: All spatial filtering happens in database
  • Significantly reduces data transfer
  • Leverages PostGIS spatial indexes (GIST)
  • Eliminates client-side geometry processing

Testing

New Test Suites

• tests/deployment/test_lambda_handler.py: Handler/server-side logic tests

  • Tests direct handler function calls with local credentials
  • Validates action routing, error handling, response formats
  • Includes PostGIS spatial query tests

• tests/deployment/test_lambda_client.py: Client/end-to-end tests

  • Tests deployed Lambda function via client
  • Validates full round-trip: client → Lambda → database → client
  • Tests GeoDataFrame conversion

Test Markers

• @pytest.mark.handler: Handler-only tests (no Lambda deployment required)
• @pytest.mark.integration: Full integration tests (requires deployed Lambda)

Requirements

Local Development

• Python 3.8+
• geopandas (optional, for GeoDataFrame support)
• shapely (required for spatial queries)

Lambda Deployment

• AWS CLI configured
• Docker installed
• ECR repository: snowexsql
• Lambda function: lambda-snowex-sql
• Secrets Manager secret with database credentials

Lambda Runtime

• Python 3.12
• Minimal dependencies (no geopandas, no rasterio)
• Secrets Manager access for credentials

Developer Notes

Adding New Measurement Classes

To make a new measurement class available via Lambda:

1. Class name MUST end with 'Measurements'

   class WeatherMeasurements(BaseDataset):

2. Class MUST have a MODEL attribute

   MODEL = WeatherData  # Required!

3. Class MUST inherit from BaseDataset

   class WeatherMeasurements(BaseDataset):

The Lambda handler will automatically discover and expose your class:

client.weather_measurements.from_filter()
client.weather_measurements.all_instruments

Validation Function

Use validate_measurement_class() to verify your class follows conventions:

from snowexsql.lambda_handler import validate_measurement_class

validate_measurement_class(WeatherMeasurements, 'WeatherMeasurements')

Documentation

New Documentation Files

• deployment/README.md: Deployment structure and quick start guide
• Inline documentation in all new modules
• Comprehensive docstrings following NumPy style

Updated Examples

All existing examples continue to work without modification. Lambda client provides alternative access method without changing local API behavior.

Migration Guide

For Existing Users

No changes required! All existing code continues to work:

from snowexsql.api import PointMeasurements
df = PointMeasurements.from_filter(instrument='camera', limit=10)

For New Lambda Users

# Instead of direct database connection
from snowexsql.lambda_client import SnowExLambdaClient
client = SnowExLambdaClient()
df = client.point_measurements.from_filter(instrument='camera', limit=10)

Known Limitations

1. Raster Operations: Not supported in Lambda (use local API)
2. Large Queries: Lambda has 15-minute timeout limit
3. Memory: Lambda configured with 1024MB (adjustable via update_lambda_config.sh)
4. Cold Starts: First request may take 5-10 seconds

Note: This is a release candidate (RC). Please test thoroughly before using in production environments.

snowexsql-v1.0.0

• Fully redesigned schema with faster performance
• Many dependency updates

snowexsql v0.6.0rc1

Release candidate 1 for 0.6.0

• New database structure with use of foreign keys for query speedup
• New naming conventions
• More distinct module-class relationships
• API changes to match new structure

snowexsql-v0.5.0

• Brought in by PR #98
• Improvements made to retrieving single value queries.
• Added in more support around rasters.
• Added in RasterMeasurements.all_descriptions to get unique descriptions
• Added in checking for whether a raster query would generate data from more than one unique dataset
• Added support for Geopandas > 1.0

snowexsql-v0.4.1

• Fix build

snowexsql-v0.4.0

• Implementation of API
• Drop outdated python

SnowEx Hackweek 2022 release

Release corresponding to the library version used in tutorials for the SnowEx Hackweek 2022

https://snowex-2022.hackweek.io/