Patrick O'Connor's
Website

A Website designed to showcase my skills.

About Me

  • I'm a cloud engineer and I set up this website to showcase my skills and Github repositories. To summarize this project, I have a website that tracks analytics via Snowplow and I use Databricks and dbt for the data modeling. Jenkins is used for CI/CD. Kubernetes is set up for the ML platform with Ray, Jupyther Hub and Mlflow. I tried to follow best practices to build this project, each page I try to explain the decisions I made. Everything is deployed via Terraform so its reproducible. When I'm not working on the project, I destroy the infrastructure to save money so the links will not work unless I spin everything back up.

Applications

  • ALB - Application Load Balancer is the gateway to all the applications
  • Jenkins - CI/CD Server responsible for testing and deploying all the infrastructure
  • Snoplow - Analytics tracker
  • Databricks - Lakehouse to store and transform data, also visualize it
  • DBT - Transformation layer to transform raw data into production-grade tables
  • EKS - Managed Kubernetes service
  • Istio - Service Mesh encapsulating the Kubernetes Cluster
  • Kiali - Dashboard for Service Mesh
  • Prometheus - Metrics collector
  • Grafana - Dashboard for metrics and Cloudwatch
  • ArgoCD - GitOps continuous delivery tool
  • Ray - Open source unified compute framework
  • Juptyer Hub - Interactive computing across multiple programming languages
  • Mlflow - Platform to streamline machine learning development

Best Practices used throughout this project

Least Privilege IAM roles and policies

Least Privilege Security Groups

Encrypt all AWS resources with custom KMS keys

Set up alarms for monitoring

Limit access to the public internet (private subnets, endpoints)

Encrypt all traffic with SSL

Continuous Integration ensures proper testing before deployment

Continuous Development and IaC ensures reproducibility

Skills

AWS

EC2, ECS, EKS, ECR, ALB, Lambda, Glue, RDS, SQS, SNS, Kinesis, Cloudwatch, IAM, KMS, Endpoints, Route53, Security Groups

Languages

Python, SQL, Linux

Open Source Frameworks

Terraform, Docker, Kubernetes - kubetctl, kustomize, helm, AWS CLI

Open Source Libraries

Snowplow, dbt, Nginx, Istio, Prometheus, Grafana, ArgoCD, Ray, Jupyter, Mlflow

CI/CD

Github Actions, AWS Codepipeline, Jenkins

Orchestration

Airflow, Prefect, AWS Eventbridge

Datawarehouse/Databases

Databricks, Snowflake, Postgres