Key Advantages:
- Separation of Storage & Compute Compute clusters (virtual warehouses) scale independently from data storage.
- Performance & Elasticity Warehouses auto-scale and auto-suspend → lower cost and high performance.
- Zero-copy Cloning Instant cloning of tables/databases without duplicating storage.
- Time Travel & Fail-safe Recover data from accidental deletes or changes.
- Multi-cluster Warehouses Avoid resource contention, but so far in my project experiance, 1 cluster is enough
- Native Features for Pipelines/Integration Streams, Tasks, Dynamic Tables, Snowflake connector for G4, Snowflake connector for MYSQL
- Security by Design Data masking, row access policies, encryption end-to-end.
This repository contains SQL and optional Python/Snowpark scripts implementing:
- Synthetic data generation and load (Bronze) using internal stage
- Bronze → Silver transformations using Streams + Tasks
- Gold layer implemented as a Dynamic Table with continuous aggregation
- Data quality checks using Data Metric Functions (DMFs)
- Dynamic Data Masking policy protecting PII
- Login to Snowflake (UI Worksheets or SnowSQL).
- Run
sql/00_SetupEnvironment.sql. - Run
sql/01_create_file_format_and_stage.sql. - Generate & load Bronze data:
sql/02_generate_and_load_bronze.sql.- you need to upload csv file to your internal stage (RAW_INTERNAL_STAGE) a. open your snowsight UI b. go to catalog => database explorer c. click on MEDALLION_DB => BRONZE_LAYER => STAGES => RAW_INTERNAL_STAGE d. on the right tab clik +Files ==> upload products.csv file (you can optionally create specific folder like the example i used PRODUCTS as folder name) e. click Upload and just wait till your file in your internal stage (RAW_INTERNAL_STAGE) successfully uploaded
- Create Silver transforms:
sql/03_silver_layer_transformation.sql. ---- - Create streams and tasks:
sql/04_create_streams_and_tasks.sql. - Create Gold dynamic table:
sql/05_gold_dynamic_table.sql - Configure DMFs:
sql/06_dmf.sql. - Configure masking:
sql/07_dynamic_data_masking.sql.