aws-emr

Spark 2.0 Python Machine Learning examples

Spark 2.0 Scala Machine Learning examples

⭐ CLI tool to launch Spark jobs on AWS EMR

Bits of code I use during live demos

The goal of this project is to offer an AWS EMR template using Spot Fleet and On-Demand Instances that you can use quickly. Just focus on writing pyspark code.

An AWS based solution using AWS CloudWatch and AWS Lambda based on Python to automatically terminate AWS EMR clusters that have been idle for a specified period of time.

Terraform module to create AWS EMR resources 🇺🇦

A batch processing data pipeline, using AWS resources (S3, EMR, Redshift, EC2, IAM), provisioned via Terraform, and orchestrated from locally hosted Airflow containers. The end product is a Superset dashboard and a Postgres database, hosted on an EC2 instance at this address (powered down):

EMR Cost Calculator

Cloud-based AI / ML workflow and data application development framework

Use aws-emr and aws-redshift to analyse dataset of adult census of USA

Run a Spark job within Amazon EMR

Analyzing Big Data with Amazon EMR

A collection of airflow sample workflows for data processing on aws

A Spark application, written in Python, to figure out strongly connected components with Bi-directional Label Propagation algorithm. This project implemented an 1.3GB Twitter network dataset on AWS EMR cluster.

AWS EMR Docker integration

Create Data Lake on AWS S3 to store dimensional tables after processing data using Spark on AWS EMR cluster

Data Engineering Project with Terraform, Spark, AWS, Docker, Airflow and other tools

A cookiecutter template for working with PySpark on AWS EMR

A large-scale data framework that will enable us to store and analyze financial market data and drive future predictions for investment.