A repository for exploring LLM Serving and MLOps practices on AWS Kubernetes (EKS) using Infrastructure as Code (AWS CDK).
| Project | Description | Blog Post |
|---|---|---|
| Serving LLMs on EKS | Creating EKS cluster and serving Llama-3.1-8B-Instruct (AWQ quantized) LLM |
LLM on EKS: Serving with vLLM |
| Scaling LLMs on EKS | Autoscaling Llama-3.1-8B-Instruct inference with Ray + KubeRay, GPU nodes provisioned on demand via Karpenter |
LLM on EKS: Scaling with Ray & Karpenter |
| Training Pipelines on EKS | LoRA fine-tuning and AWQ quantization of Qwen2.5-1.5B-Instruct as a repeatable Kubeflow Pipelines workflow, backed by Amazon RDS and S3 |
LLM on EKS: Training Pipelines with Kubeflow |