SLT - Log Analytics Demo

A demo combining Amazon Kinesis Data Streams, Amazon Data Firehose and Amazon Managed Service for Apache Flink running a Zeppelin notebook.

Storm Library for Terraform

This repository is a member of the SLT | Storm Library for Terraform, a collection of Terraform modules for Amazon Web Services. The focus of these modules, maintained in separate GitHub™ repositories, is on building examples, demos and showcases on AWS. The audience of the library is learners and presenters alike - people that want to know or show how a certain service, pattern or solution looks like, or "feels".

Learn more

Installation

This demo can be built using GitHub Actions. In order to do so

• Install the Storm Library for Terraform
• Only if your account has AWS Lake Formation enabled:
  • Check the value of the DEPLOYMENT_ROLE environment variable as shown in the summary of the Apply workflow you executed in order to install the Storm Library for Terraform. It starts with slt-0 and ends with -deployment.
  • In AWS Lake Formation, open Administrative roles and tasks. In the section Database creators, click on Grant
  • In the opening Grant permissions panel, in the IAM users and roles input, select your deployment role which you fetched before
  • Tick both Create database radio buttons
  • Confirm by clicking Grant
• Deploy this member repository

Deployment of this member should take < 5 minutes on GitHub resources.

Architecture

Explore this demo

1. In your GitHub Apply workflow run, click on Summary. If you scroll down, you will see the apply / check summary.
2. Take a look at the ENVIRONMENT variables. Note the DEPLOYMENT_NAME environment variable. Its value should match the pattern slt-(some number)-log-analytics-demo-(your github owner).
3. From the same page, download the artifact ending on .zeppelin to your download folder and unzip it if necessary
4. In the AWS Console, on the Managed Apache Flink service page, click on Studio notebooks on the left-hand side menu
5. In the list of Studio notebooks in the center, click on the log-analytics-demo notebook that has been created
6. In the view that opens, click on the Run button. You will be asked to confirm. Do so by clicking Run once more.
7. A message will appear in your center view, informing you that the service is "Starting Studio notebook (your studio notebook)...". Wait until the startup process has completed. It may take a few minutes.
8. You will be notified in the AWS Console as soon as your notebook has successfully started. Then, click Open in Apache Zeppelin.
9. A new browser tab will open, welcoming you to Apache Zeppelin.
10. On that browser page, click Import note. A panel titled Import New Note will pop up. Click on the area saying Select JSON File/IPYNB File and choose your unzipped Zeppelin notebook from your download folder.
11. You will notice a new notebook link called Log Analytics Demo on the refreshed Zeppelin browser page. Click on the link.
12. The Log Analytics Demo notebook will open. Next to its title, find a small triangle (like a "play" button) that enables you to run all paragraphs one after the other. Click on that play button.
13. The paragraphs will run one after the other. Execution will take a minute or two.
14. Check the last paragraph. Don't worry if it's saying No Data available for a minute or so. A pie chart will show up. The pie chart will update every ten seconds, showing percentages of HTTP status codes sent from an instance running Faker.

Terraform Docs

Click to show

Requirements

Name	Version
terraform	>= 1
aws	>= 6

Providers

Name	Version
aws	>= 6
aws.no_tags	>= 6
awscc	n/a
cloudinit	n/a
external	n/a
local	n/a
null	n/a
time	n/a

Modules

Name	Source	Version
producer	git::https://github.com/stormreply/ssm-managed-instance.git	n/a

Resources

Name	Type
aws_cloudwatch_log_group.delivery_firehose	resource
aws_cloudwatch_log_stream.delivery_firehose	resource
aws_glue_catalog_database.hive_database	resource
aws_glue_catalog_database.zeppelin_database	resource
aws_iam_policy.delivery_firehose	resource
aws_iam_policy.zeppelin_notebook	resource
aws_iam_role.delivery_firehose	resource
aws_iam_role.zeppelin_notebook	resource
aws_iam_role_policy_attachment.delivery_firehose	resource
aws_iam_role_policy_attachment.zeppelin_notebook	resource
aws_kinesis_firehose_delivery_stream.delivery_firehose	resource
aws_kinesis_stream.ingestion_stream	resource
aws_lakeformation_permissions.hive_database	resource
aws_lakeformation_permissions.zeppelin_database	resource
aws_s3_bucket.bucket	resource
aws_s3_object.connector_jar	resource
aws_security_group.producer	resource
awscc_kinesisanalyticsv2_application.zeppelin_notebook	resource
local_file.zeppelin_notebook_json	resource
null_resource.connector_download	resource
time_sleep.iam_propagation	resource
aws_availability_zones.available	data source
aws_caller_identity.current	data source
aws_iam_policy.amazon_kinesis_firehose_full_access	data source
aws_iam_policy.amazon_kinesis_full_access	data source
aws_iam_policy.amazon_ssm_managed_instance_core	data source
aws_iam_policy.cloudwatch_full_access	data source
aws_iam_policy_document.delivery_firehose	data source
aws_iam_policy_document.delivery_firehose_assume_role	data source
aws_iam_policy_document.zeppelin_notebook	data source
aws_iam_policy_document.zeppelin_notebook_assume_role	data source
aws_iam_role.caller	data source
aws_region.current	data source
cloudinit_config.controller	data source
external_external.lakeformation_enabled	data source

Inputs

Name	Description	Type	Default	Required
_metadata	n/a	object({ actor = string # Github actor (deployer) of the deployment catalog_id = string # SLT catalog id of this module deployment = string # slt-<catalod_id>-- ref = string # Git reference of the deployment ref_name = string # Git ref_name (branch) of the deployment repo = string # GitHub short repository name (without owner) of the deployment repository = string # GitHub full repository name (including owner) of the deployment sha = string # Git (full-length, 40 char) commit SHA of the deployment short_name = string # slt-<catalog_id>- time = string # Timestamp of the deployment })	{ "actor": "", "catalog_id": "", "deployment": "", "ref": "", "ref_name": "", "repo": "", "repository": "", "sha": "", "short_name": "", "time": "" }	no

Outputs

Name	Description
_default_tags	n/a
_metadata	n/a
_name_tag	n/a
artifact	n/a

Credits

• Build a Log Analytics Solution on AWS
• Faker - Python package generating fake data