| Module | Topics Covered |
|---|---|
| Compute | Compute Infrastructure (Regions & Zones) Virtual Machines (Compute Engine) SSH Keys & OS Login Identity Platform (Customer Identity) Images & Instance Templates Shared Image Families RAID Configuration with Persistent Disks Startup Scripts & Metadata Load Balancing & Health Checks Managed Instance Group Autoscaling Cloud Monitoring & Logging Alerting Policies Managed Instance Groups (VM Scaling) App Engine / Cloud Run Monitoring |
| Networking | VPC Networking Basics External (Public) IP Addresses Network Interfaces (NICs) VPC Firewall Rules Network Tags & Service Accounts VPC Flow Logs Bastion Host / IAP TCP Forwarding VPC Routes & Cloud Router (BGP) |
| Storage | Cloud Storage (Object Storage) Persistent Disk & Hyperdisk (Block Storage) Pub/Sub & Firestore (Queue & Table Storage) Cloud Tasks Storage Buckets & IAM Filestore (Managed NFS) Partner Storage Services Disk Access Control Disk Encryption with CMEK Cloud Storage Browser & gsutil |
| Database (Firestore & NoSQL) | SQL vs NoSQL What is Firestore How to Run Firestore Benefits & Use Cases of Firestore Serverless Integration (Cloud Functions & Cloud Run) Firestore Integration with GCP Services Why Firestore for Serverless Managed Bigtable / Cassandra Drawbacks of Firestore |
| GCP Service Mapping | Cloud Data Integration Overview Importance of Data Integration in GCP Dataplex (Crawler & Metadata Discovery) Data Copy & Ingestion (Data Fusion & Dataflow) Creating Control Flow Creating Data Flow Scheduling Pipelines (Cloud Scheduler) Pipeline Monitoring (Cloud Monitoring & Logging) Integration with GCP Data Services |
| RDS (Cloud SQL) | Relational Database Concepts Relational Databases in GCP Setting Up Cloud SQL MySQL / PostgreSQL Interfaces DB Instances, Storage & Monitoring Event Notifications DB Access Control MySQL Features Creating Databases Connecting to Database Export & Import |
| Messaging in GCP | Messaging Concepts & Benefits Messaging Options (Pub/Sub, Cloud Tasks, Eventarc) Messaging Service Design Pub/Sub Basics Eventarc Streaming with Pub/Sub & Dataflow Real-Time Messaging Publishers & Subscribers Using Pub/Sub Benefits & Features Queue vs Pub/Sub Message Attributes & Filtering Raw Message Delivery System-to-System Messaging |
gcloud auth login
gcloud config set project mazenet-001
gcloud config set compute/region asia-south1
gcloud services enable pubsub.googleapis.com
gcloud services enable cloudfunctions.googleapis.com
gcloud services enable eventarc.googleapis.com
gcloud services enable run.googleapis.com
gcloud services enable bigquery.googleapis.com
cloud-sql-proxy.x64.exe mazenet-001:asia-south1:sainathdb --port 3307
mysql -h 127.0.0.1 -P 3307 -u sainathdemo -p
USE Managment_system;
gcloud pubsub topics create sainath-events
gcloud pubsub topics list
gcloud pubsub subscriptions create sainath-events-sub --topic=sainath-events
gsutil mb -l asia-south1 gs://mazenet-001-sainath-dataflow-temp
bq mk --dataset sainathevents
bq mk --dataset --location=asia-south1 sainathevents
bq ls
bq mk --table sainathevents.user_events event:STRING,user:STRING,ts:TIMESTAMP
bq ls sainathevents
gcloud functions deploy sainath --gen2
--region=asia-south1 --runtime=python310
--source=. --entry-point=on_user_create
--trigger-event-filters="type=google.cloud.firestore.document.v1.created" --trigger-event-filters="database=sainath"
--trigger-event-filters-path-pattern="document=users/{docId}"
or
gcloud functions deploy sainath --gen2 --region=asia-south1 --runtime=python310 --source=. --entry-point=on_user_create --trigger-event-filters="type=google.cloud.firestore.document.v1.created" --trigger-event-filters="database=sainath" --trigger-event-filters-path-pattern="document=users/{docId}"
gcloud dataflow jobs run sainath-pubsub-to-bq-job --gcs-location gs://dataflow-templates/latest/PubSub_to_BigQuery --region asia-south1 --staging-location gs://mazenet-001-sainath-dataflow-temp/temp --parameters inputTopic=projects/mazenet-001/topics/sainath-dataflow-topic,outputTableSpec=mazenet-001:sainath_dataflow_ds.events
gcloud dataflow jobs list --region=asia-south1
gcloud dataflow jobs describe use(jobId) --region=asia-south1
gcloud logging read "resource.type="dataflow_step" AND resource.labels.job_id="2025-12-28_20_12_32-10982917282909748665"" --limit=50 --project=mazenet-001
- Pub/Sub Topic and Subscription
- A Pub/Sub topic named 'sainathbigquery_events' was created.
- A subscription named 'sainathbigquery-events-sub' was created for the topic.
- The subscription is active and able to receive messages.
- Messages published to the topic are successfully delivered to the subscription.
- Publishing Messages to Pub/Sub
- User event messages are published in JSON format.
- Each message contains event type and user information.
- Example events include user login actions.
- Messages are successfully accepted by Pub/Sub.
- Cloud Function Deployment
- A Cloud Function named 'sainathfunctionapp' was deployed in the asia-south1 region.
- The function is configured with a Pub/Sub trigger.
- It runs automatically whenever a message is published to the topic.
- Message Processing in Cloud Function
- The Cloud Function decodes the Pub/Sub message.
- JSON data is extracted from the message payload.
- A timestamp is generated during processing.
- The processed data is prepared for storage in BigQuery.
- BigQuery Dataset and Table
- A BigQuery dataset named 'sainathevents' was created.
- A table named 'user_events' was created inside the dataset.
- Table schema includes event, user, and timestamp fields.
- Data Insertion into BigQuery
- The Cloud Function inserts processed records into the BigQuery table.
- Multiple records are stored successfully.
- Each record contains event name, user name, and timestamp.
- End-to-End Data Flow
- Events are published to Pub/Sub.
- Pub/Sub triggers the Cloud Function.
- The Cloud Function processes the data.
- BigQuery stores the final records for analysis.
Conclusion This implementation demonstrates a real-time event processing system using Google Cloud Pub/Sub, Cloud Functions, and BigQuery. The solution is scalable, reliable, and suitable for real-time data analytics.