AI Engineer Roadmap 2026 (In-Depth)
A structured, English-only path from Python and math to production LLM systems, agents and MCP. Every resource is free to use.
Variables, data types, control flow, functions, lists, dicts, sets, comprehensions, file I/O, exceptions
Classes, inheritance, dataclasses, dunder methods, iterators, generators, decorators, context managers
- PLAYLISTCorey Schafer - Python OOP Tutorials: Working with Classes
- VIDRaymond Hettinger (PyCon US) - Transforming Code into Beautiful, Idiomatic Python
- VIDNed Batchelder (PyCon US) - Loop Like a Native: while, for, iterators, generators
- VIDCorey Schafer - Context Managers: Efficiently Managing Resources
Type annotations, generics, TypedDict, Protocols, Optional/Union, mypy and pyright type checking
async/await, event loop, tasks, asyncio.gather/TaskGroup, threads vs processes, the GIL, concurrent HTTP calls
Virtual environments, pip, uv, pyproject.toml, dependency locking, Python versions, ruff linting/formatting
Async Multi-API Fetcher CLI
beginnerBuild a typed command-line tool that concurrently calls several public REST APIs with asyncio + httpx, validates responses with dataclasses/Pydantic, and caches results to JSON
- Python
- asyncio
- httpx
- argparse
- +1
Installable Text-Utilities Package
intermediatePackage a small library (tokenizer, text cleaner, word-frequency counter) with pyproject.toml and uv, full type hints checked by mypy, ruff linting, and a CLI entry point
- Python
- uv
- pyproject.toml
- mypy
- +1
Rate-Limited Async Job Runner
advancedWrite an asyncio task runner that processes a queue of jobs (e.g. batch LLM/API calls) with concurrency limits, retries with exponential backoff, timeouts, and structured logging
- Python
- asyncio
- TaskGroup
- logging
- +1
Vectors, matrices, dot products, matrix multiplication, linear transformations, eigenvalues/eigenvectors, SVD
Derivatives, chain rule, partial derivatives, gradients, Jacobians, Hessians, matrix calculus
Loss functions, gradient descent, SGD, momentum, learning rates, convexity, backpropagation as the chain rule
Random variables, conditional probability, Bayes' theorem, common distributions, expectation, variance, likelihood
Descriptive statistics, sampling, confidence intervals, hypothesis testing, p-values, correlation, A/B testing
Matrix & Linear Algebra Toolkit from Scratch
beginnerImplement vector/matrix ops, Gaussian elimination, power-iteration eigenvalues and a PCA demo in pure Python, then verify results against NumPy
- Python
- NumPy
- Matplotlib
Gradient Descent Visualizer
intermediateImplement batch/mini-batch SGD, momentum and Adam on 2D loss surfaces (e.g. Rosenbrock) and fit linear/logistic regression by hand, animating optimization paths and comparing learning rates
- Python
- NumPy
- Matplotlib
- Jupyter
A/B Test & Bayesian Inference Simulator
advancedSimulate experiments, compute p-values, confidence intervals and bootstrap estimates, then compare with a Beta-Binomial Bayesian posterior; package findings in a notebook report
- Python
- NumPy
- SciPy
- pandas
- +1
ndarrays, dtypes, indexing/slicing, broadcasting, vectorization, linear algebra routines, random numbers
Series/DataFrames, reading CSV/Parquet, selection, filtering, groupby/aggregation, merges/joins, reshaping, lazy queries in Polars
Missing values, dtypes and parsing, duplicates, outliers, scaling, summary statistics, exploratory analysis workflow
SELECT, WHERE, JOINs, GROUP BY, aggregates, subqueries, window functions, schemas, indexes, querying from Python
Matplotlib figures/axes, line/bar/scatter/histogram plots, subplots, Seaborn statistical plots, choosing the right chart
Jupyter Notebook/JupyterLab, Google Colab, Kaggle notebooks, magics, kernels, notebook-to-module workflow
Exploratory Data Analysis Report
beginnerPick a public Kaggle dataset, clean it with pandas, answer 5-8 questions with groupby/merge analysis and Seaborn charts, and publish the notebook as a written report
- Python
- pandas
- Seaborn
- Jupyter
SQL + pandas Analytics Pipeline
intermediateLoad a multi-table dataset (e.g. an e-commerce or NYC taxi sample) into SQLite/PostgreSQL, write analytical SQL with joins and window functions, then pull results into pandas for visualization
- SQL
- SQLite
- PostgreSQL
- pandas
- +1
pandas vs Polars Benchmark on Large Data
advancedProcess a multi-GB Parquet dataset with both pandas and Polars lazy queries, compare memory and runtime, and produce a reproducible benchmark notebook with charts
- Python
- Polars
- pandas
- Parquet
- +1
Commits, staging, branches, merging, rebasing, resolving conflicts, remotes, pull requests, .gitignore
Shell navigation, files and permissions, pipes and redirection, environment variables, processes, SSH, shell scripting
HTTP methods and status codes, REST design, path/query params, request bodies, dependency injection, async endpoints, OpenAPI docs
BaseModel, field types and constraints, validators, serialization, settings management, JSON schema for structured LLM output
Images vs containers, Dockerfiles, layers and caching, volumes, networking, Docker Compose, containerizing a FastAPI app
Unit vs integration tests, assertions, fixtures, parametrization, mocking, testing APIs and data/ML code, coverage
Linting and formatting with ruff, pre-commit hooks, type checking in CI, GitHub Actions workflows, automated tests on every push
Compute, storage, networking, IAM, managed services, serverless, environment variables and secrets, deploying a container to the cloud
Dockerized ML Prediction API
beginnerWrap a scikit-learn model in a FastAPI service with Pydantic request/response schemas, pytest tests, a Dockerfile, and docker-compose for local runs
- FastAPI
- Pydantic
- Docker
- pytest
- +1
LLM Gateway Microservice with CI
intermediateBuild an async FastAPI service that proxies requests to an LLM API with structured Pydantic outputs, rate limiting and caching; add ruff, mypy and pytest in a GitHub Actions pipeline that builds and pushes a Docker image
- FastAPI
- httpx
- Pydantic
- GitHub Actions
- +2
Cloud-Deployed Data App
advancedDeploy a containerized FastAPI + PostgreSQL app to a cloud provider (e.g. AWS App Runner/ECS or Google Cloud Run) with secrets management, health checks, logging and an automated deploy on merge to main
- Docker
- PostgreSQL
- AWS
- GitHub Actions
- +2
Supervised vs unsupervised learning, train/validation/test splits, loss and gradient descent, overfitting, bias-variance tradeoff
Linear and polynomial regression, gradient descent, MSE/MAE, Ridge (L2), Lasso (L1), ElasticNet
Logistic regression, sigmoid and log loss, KNN, Naive Bayes, Support Vector Machines and kernels, decision thresholds
Decision trees, Gini/entropy, bagging, random forests, gradient boosting, XGBoost and LightGBM
K-Means, hierarchical clustering, DBSCAN, PCA, dimensionality reduction, t-SNE/UMAP for visualization
Missing values, categorical encoding, scaling/normalization, feature creation and selection, pipelines, data leakage
Confusion matrix, precision/recall/F1, ROC-AUC, regression metrics, k-fold cross-validation, hyperparameter tuning (grid/random search)
- VIDStatQuest - ROC and AUC, Clearly Explained
- VIDStatQuest - Machine Learning Fundamentals: Cross Validation
- INTGoogle MLCC - Classification (thresholds, confusion matrix, ROC/AUC)
- DOCscikit-learn - Cross-validation: evaluating estimator performance
- PPRSebastian Raschka - Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning
Estimator API (fit/predict/transform), Pipeline and ColumnTransformer, GridSearchCV, end-to-end modeling workflow
End-to-End House Price Regression Pipeline
beginnerBuild a scikit-learn Pipeline with a ColumnTransformer for imputation, encoding and scaling. Compare Linear, Ridge/Lasso and gradient boosting with k-fold cross-validation, then report RMSE with a residual analysis.
- Python
- pandas
- scikit-learn
- matplotlib
Customer Churn Prediction with XGBoost & SHAP
intermediateTrain an imbalanced binary classifier, tune it with randomized search, and pick a decision threshold from the precision-recall curve. Explain predictions with SHAP values.
- Python
- pandas
- XGBoost
- scikit-learn
- +1
Customer Segmentation with Clustering
intermediateEngineer RFM features from transaction data. Cluster with K-Means and DBSCAN, choose k with silhouette and elbow plots, visualize segments in 2D with PCA, and write up personas.
- Python
- pandas
- scikit-learn
- seaborn
Neurons and perceptrons, multilayer perceptrons, activation functions (ReLU, sigmoid, softmax), forward pass, universal approximation
Chain rule, computational graphs, reverse-mode autodiff, gradient descent, mini-batch SGD
- VIDAndrej Karpathy - The spelled-out intro to neural networks and backpropagation: building micrograd
- VID3Blue1Brown - Backpropagation calculus (Deep Learning Chapter 4)
- ARTStanford CS231n Notes - Backpropagation, Intuitions
- BOOKDive into Deep Learning - Forward Propagation, Backward Propagation, and Computational Graphs
MSE and cross-entropy, SGD, momentum, RMSProp, Adam/AdamW, learning-rate schedules and warmup
Overfitting vs underfitting, L2/weight decay, dropout, early stopping, data augmentation
Vanishing/exploding gradients, Xavier/He initialization, Batch Normalization, Layer Normalization, residual connections
Overfit-a-single-batch check, reading loss curves, learning-rate tuning, babysitting training, systematic hyperparameter search
Full free courses and textbooks covering DL end to end: lectures, theory, and exercises
Build micrograd: An Autograd Engine from Scratch
beginnerWrite a scalar autodiff engine with a Value class, topological-sort backward pass and a tiny MLP library. Train it on a 2D toy dataset and visualize the computation graph.
- Python
- graphviz
- matplotlib
NumPy MLP on MNIST (No Frameworks)
intermediateImplement forward/backward passes, softmax cross-entropy, mini-batch SGD and Adam in pure NumPy. Check gradients numerically and reach over 97% test accuracy.
- Python
- NumPy
- matplotlib
Training Dynamics Lab
advancedOn CIFAR-10, run controlled ablations: SGD vs Adam, He vs naive init, with/without BatchNorm and dropout, and LR schedules. Log loss curves and gradient statistics, then write up which choices mattered and why.
- PyTorch
- TensorBoard
- Weights & Biases
Tensor creation, indexing, broadcasting, dtypes and devices, requires_grad, backward(), computation graphs
nn.Module and nn.Sequential, layers and parameters, custom forward(), state_dict, saving and loading models
Dataset and DataLoader classes, custom datasets, transforms and augmentation, batching, shuffling, num_workers
Loss functions and optimizers in torch, zero_grad/backward/step, train vs eval mode, validation loop, checkpointing, organizing code (Lightning)
CUDA devices, moving models/data to GPU, mixed precision (AMP), torch.compile, DataLoader tuning, DistributedDataParallel basics
FashionMNIST Classifier with a Hand-Written Training Loop
beginnerBuild an MLP and a small CNN with nn.Module. Write the train/eval loops, track accuracy and loss per epoch, save/load the best checkpoint, and plot a confusion matrix.
- PyTorch
- torchvision
- matplotlib
Modular Image Classification Project on a Custom Dataset
intermediateTurn a notebook into a package: data_setup.py (custom Dataset + transforms), model.py, engine.py (train/test steps), train.py with CLI args. Log runs to TensorBoard and add early stopping.
- PyTorch
- torchvision
- TensorBoard
- argparse
Training Speed-Up Benchmark
advancedTake a ResNet training script and measure throughput and accuracy for baseline, AMP, torch.compile, tuned DataLoader and multi-GPU DDP via torchrun. Report time-to-accuracy for each.
- PyTorch
- CUDA
- torch.compile
- DDP
- +1
Convolution, kernels, padding, stride, pooling, feature maps, classic architectures (LeNet, AlexNet, VGG, ResNet)
- ARTStanford CS231n Notes - Convolutional Neural Networks
- PLAYLISTStanford CS231n - Convolutional Neural Networks for Visual Recognition (Spring 2017 lectures)
- INTPolo Club (Georgia Tech) - CNN Explainer
- BOOKDive into Deep Learning - Convolutional Neural Networks (chapters 7-8)
- PLAYLISTDeepLearningAI (Andrew Ng) - Convolutional Neural Networks (Course 4 of the DL Specialization)
Pretrained backbones (ResNet, EfficientNet, ViT), feature extraction vs fine-tuning, data augmentation, timm and torchvision models
Bounding boxes, IoU, anchor boxes, non-max suppression, mAP, two-stage (Faster R-CNN) vs one-stage (YOLO) detectors, semantic/instance segmentation
Recurrence and hidden state, backprop through time, vanishing gradients, LSTM/GRU gates, seq2seq, and why attention replaced them
- VIDStatQuest - Recurrent Neural Networks (RNNs), Clearly Explained
- VIDStatQuest - Long Short-Term Memory (LSTM), Clearly Explained
- ARTChristopher Olah - Understanding LSTM Networks
- ARTAndrej Karpathy - The Unreasonable Effectiveness of Recurrent Neural Networks
- PLAYLISTDeepLearningAI (Andrew Ng) - Sequence Models (Course 5 of the DL Specialization)
- DOCPyTorch - NLP From Scratch: Classifying Names with a Character-Level RNN
Transfer-Learning Image Classifier with a Web Demo
intermediateFine-tune a pretrained timm/torchvision backbone on a custom image dataset (e.g. plant disease or food classes) with augmentation. Compare feature extraction and full fine-tuning, then deploy a Gradio demo on Hugging Face Spaces.
- PyTorch
- timm
- torchvision
- Gradio
- +1
Custom Object Detector with YOLO
advancedLabel a small dataset (Roboflow/CVAT), fine-tune a YOLO model, and evaluate mAP@0.5 and mAP@0.5:0.95. Run real-time inference on video and export to ONNX.
- Ultralytics YOLO
- PyTorch
- OpenCV
- ONNX
Character-Level LSTM Text Generator
intermediateTrain a char-level RNN/LSTM in PyTorch on a text corpus. Experiment with sequence length, hidden size and temperature sampling, then compare against a tiny Transformer to see why attention won.
- PyTorch
- Python
- NumPy
Text preprocessing, bag-of-words/TF-IDF, word2vec, GloVe, embedding similarity, contextual embeddings
Seq2seq bottleneck, query/key/value, scaled dot-product attention, self vs cross attention, multi-head and causal attention
Encoder/decoder blocks, positional encoding, residual connections, layer norm, feed-forward layers, the original 'Attention Is All You Need' design
Encoder-only (BERT), decoder-only (GPT), encoder-decoder (T5), masked vs causal language modeling, transfer learning
Subword tokenization (BPE, WordPiece, Unigram/SentencePiece), vocabularies, special tokens, byte-level BPE, tokenizer quirks
Fine-tune BERT for Text Classification
beginnerFine-tune a pretrained encoder (BERT/DistilBERT/ModernBERT) on a sentiment or topic dataset with the Hugging Face Trainer, evaluate it, and push it to the Hub.
- Python
- Hugging Face Transformers
- Datasets
- PyTorch
Transformer from Scratch
intermediateImplement multi-head attention, positional encodings and encoder/decoder blocks in PyTorch, then train a small model on a toy translation or character-level task.
- Python
- PyTorch
- NumPy
Train Your Own BPE Tokenizer
intermediateImplement byte-level BPE from scratch (minbpe-style), train it on a corpus, and compare token counts and edge cases (code, emojis, other languages) against tiktoken and a Hugging Face tokenizer.
- Python
- tiktoken
- Hugging Face Tokenizers
Next-token prediction, the pretraining to post-training pipeline, base vs chat models, context windows, hallucinations
Next-token objective, web-scale datasets (FineWeb), filtering and deduplication, data mixtures, the training loop, distributed training basics
Kaplan vs Chinchilla scaling laws, compute-optimal training, parameter/FLOPs/memory math, emergent abilities
Implement the tokenizer, embeddings, causal self-attention, transformer blocks, training loop and text generation in PyTorch
- VIDAndrej Karpathy - Let's build GPT: from scratch, in code, spelled out
- REPOSebastian Raschka - LLMs-from-scratch (code for Build a Large Language Model From Scratch)
- VIDSebastian Raschka - Building LLMs from the Ground Up: A 3-hour Coding Workshop
- REPOAndrej Karpathy - nanochat: full-stack ChatGPT clone (tokenizer, pretraining, SFT, RL, inference)
Greedy and beam search, temperature, top-k, top-p (nucleus), repetition penalties, structured/constrained decoding
KV caching, prefill vs decode, memory and latency math, multi-query/grouped-query attention, FlashAttention
RoPE, RMSNorm, SwiGLU, grouped-query and latent attention, Mixture-of-Experts, how Llama/Qwen/DeepSeek/Gemma differ from GPT-2
Transformers pipelines and AutoModel, Datasets, Tokenizers, the Hub, model cards and licenses, choosing open-weight models (Llama, Qwen, Gemma, Mistral, DeepSeek)
Sampling Playground
beginnerBuild a small Gradio/Streamlit app that loads an open model and lets users compare greedy, beam, temperature, top-k and top-p outputs side by side, with token probabilities shown.
- Python
- Hugging Face Transformers
- Gradio
Pretrain a Mini GPT
intermediateFollowing nanoGPT or LLMs-from-scratch, pretrain a small GPT on TinyStories or Shakespeare, plot the loss curve, try scaling model size and data, and sample from checkpoints.
- Python
- PyTorch
- tiktoken
- Weights & Biases
Implement and Benchmark a KV Cache
advancedAdd a KV cache to your from-scratch GPT, then measure tokens/sec and memory with and without it. Extend the model to grouped-query attention and compare.
- Python
- PyTorch
Knowledge gaps vs behavior gaps, cost/latency trade-offs, when fine-tuning pays off, combining RAG with fine-tuned models
Instruction datasets, chat templates, SFT and loss masking, full fine-tuning vs PEFT, data quality over quantity
Low-rank adapters, rank/alpha/target modules, 4-bit QLoRA with bitsandbytes, merging adapters, other PEFT methods
- VIDUmar Jamil - LoRA explained visually + PyTorch code from scratch
- ARTSebastian Raschka - Finetuning LLMs with LoRA and QLoRA: Insights from Hundreds of Experiments
- ARTHugging Face - Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA
- ARTThinking Machines Lab - LoRA Without Regret
- DOCHugging Face - PEFT documentation
TRL trainers (SFT/DPO/GRPO), Unsloth for fast low-memory single-GPU training, Axolotl YAML configs for multi-GPU runs
Preference data collection, reward models (Bradley-Terry), PPO with KL penalty, the InstructGPT pipeline, reward hacking
Direct Preference Optimization, variants (IPO, KTO, ORPO, SimPO, APO), chosen/rejected preference datasets, TRL DPOTrainer
Chain-of-thought and test-time compute, RLVR, GRPO, rule-based reward functions, the DeepSeek-R1 recipe, distilling reasoning into small models
- ARTSebastian Raschka - Understanding Reasoning LLMs
- CRSHugging Face LLM Course - Open R1 for Students (reasoning, GRPO in TRL)
- PPRDeepSeek-AI - DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL (paper)
- CRSDeepLearning.AI - Reinforcement Fine-Tuning LLMs With GRPO (Predibase)· free to audit
LLM-generated instruction and preference data, quality filtering and deduplication, teacher-student distillation, model-collapse risks
- REPOMaxime Labonne - LLM Datasets (curated post-training datasets and tools)
- DOCArgilla - Distilabel documentation (synthetic data pipelines)
- PPRLiu et al. (Google DeepMind) - Best Practices and Lessons Learned on Synthetic Data (paper)
- ARTHugging Face - Cosmopedia: creating large-scale synthetic data for pretraining
Benchmarks (MMLU, IFEval, GSM8K), lm-evaluation-harness, LLM-as-a-judge, leaderboards, data contamination
Domain Assistant with QLoRA
intermediateBuild an instruction dataset for a niche domain (e.g., legal FAQs or your own docs), QLoRA fine-tune a 1-4B open model with Unsloth or TRL, compare it with prompting and RAG baselines, and publish the adapter to the Hub.
- Python
- Unsloth
- TRL
- PEFT
- +2
Align a Model with DPO
advancedGenerate a chosen/rejected preference dataset (synthetic or with distilabel), run DPO on top of an SFT checkpoint, and measure changes with lm-evaluation-harness and an LLM-as-judge.
- Python
- TRL
- distilabel
- lm-evaluation-harness
Mini Reasoning Model with GRPO
advancedTrain a small model (e.g., Qwen 0.5-1.5B) on GSM8K-style math with GRPO and rule-based verifiable rewards (answer correctness plus format), and track accuracy and reasoning length over training.
- Python
- TRL
- vLLM
- PyTorch
Number formats (FP32, FP16, BF16, INT8, 4-bit), absmax and zero-point quantization, post-training quantization vs QAT, memory math
Post-training quantization algorithms, activation-aware weight quantization, GGUF k-quants for llama.cpp, bitsandbytes, choosing a format for your hardware
- ARTMaxime Labonne - 4-bit LLM Quantization with GPTQ
- ARTMaxime Labonne - Quantize Llama models with GGUF and llama.cpp
- PPRLin et al. - AWQ: Activation-aware Weight Quantization (paper)
- DOCHugging Face Transformers - Quantization overview (compare bitsandbytes, GPTQ, AWQ, GGUF, ...)
- PPRFrantar et al. - GPTQ (paper)
Ollama, llama.cpp, LM Studio, GGUF models from the Hub, sizing VRAM/RAM, OpenAI-compatible local servers
Knowledge distillation (soft targets, teacher-student), structured/unstructured pruning, pruning plus distillation recipes (Minitron), sparsity
Vision encoders (ViT, CLIP/SigLIP), projecting image tokens into LLMs (LLaVA-style), VLM fine-tuning, OCR and document understanding
Audio features and spectrograms, speech recognition with Whisper, text-to-speech basics, fine-tuning ASR models
Forward and reverse diffusion, denoising U-Nets and diffusion transformers, latent diffusion (Stable Diffusion), text conditioning and guidance, the Diffusers library
Quantize & Serve a Model Locally
intermediateConvert a (fine-tuned) open model to GGUF at several quantization levels (Q4_K_M, Q5_K_M, Q8_0), run it with llama.cpp/Ollama, and benchmark size, tokens/sec and quality (perplexity or a small eval set).
- llama.cpp
- Ollama
- GGUF
- Python
Voice Notes Summarizer
beginnerTranscribe audio with Whisper (or faster-whisper), then summarize and extract action items with a local LLM running in Ollama, all offline.
- Whisper
- Ollama
- Python
- Gradio
Multimodal Document Assistant
advancedUse an open VLM (e.g., Qwen-VL or SmolVLM) to answer questions about scanned documents and charts, add voice queries via Whisper, and optionally fine-tune the VLM on a small domain dataset with TRL.
- Hugging Face Transformers
- TRL
- Whisper
- PyTorch
Text-to-Image Generator with Diffusers
intermediateBuild an image generation app with Stable Diffusion/FLUX via Diffusers, try schedulers and guidance scale, and train or load a LoRA for a custom style.
- Diffusers
- PyTorch
- PEFT
- Gradio
Messages/chat format, system vs user roles, model parameters (temperature, top-p, max tokens), SDK setup and multi-turn conversations across OpenAI, Anthropic and Gemini
Server-sent events, token-by-token rendering, stream event types, streaming with tool calls, and wiring streams into web UIs
JSON mode vs schema-constrained decoding, Pydantic/Zod schemas, validation and retries, and reliable data extraction
Defining tools with JSON Schema, the tool-call request/result loop, parallel calls, tool_choice, and handling tool errors
Running open-weight models locally with Ollama, hosted inference (Hugging Face Inference Providers, Groq, OpenRouter), OpenAI-compatible endpoints and provider-agnostic gateways like LiteLLM
Tokenization and token counting, input/output pricing, batch APIs for cheap offline jobs, rate limits, and latency optimization techniques
Multi-Provider Streaming Chat CLI
beginnerBuild a terminal chat app that streams responses and lets you switch between OpenAI, Claude, Gemini and a local Ollama model, logging tokens and cost per turn
- Python
- OpenAI SDK
- Anthropic SDK
- Ollama
- +1
Invoice & Receipt Data Extractor
intermediateExtract structured fields from PDFs and images into validated Pydantic models with automatic retries, then export to CSV or a database through an API
- Pydantic
- Instructor
- OpenAI API
- FastAPI
Tool-Calling Personal Assistant
intermediateCreate an assistant that calls weather, calendar and web-search tools (including parallel calls) and streams answers into a chat UI
- Claude API
- Function Calling
- Python
- Streamlit
Clear and specific instructions, role/system prompts, few-shot examples, XML/markdown structuring, output formatting and iterative refinement
Chain-of-thought, self-consistency, tree-of-thought, ReAct-style prompting and how to prompt reasoning models (o-series, extended thinking)
Breaking complex tasks into chained prompts, routing and classification steps, validating between calls, and production prompt templates
Curating everything in the context window (instructions, tools, retrieved data, history): write/select/compress/isolate strategies, context rot and failure modes, compaction and just-in-time retrieval
Cache-friendly prompt layout (static prefix first), cache breakpoints and TTLs, cost/latency savings, and tips for prompting over long documents
Prompt Technique Benchmark
beginnerCompare zero-shot, few-shot and chain-of-thought prompts on a small labeled dataset and report accuracy, latency and cost for each
- Python
- OpenAI API
- pandas
Long-Document Q&A with Prompt Caching
intermediateAnswer repeated questions over a long contract or book by caching the document prefix, and measure the cost and latency savings
- Claude API
- Prompt Caching
- Python
Context-Managed Chatbot
intermediateBuild a long-running chatbot that keeps a token budget using rolling summaries, compaction and a persistent memory file
- OpenAI Agents SDK
- tiktoken
- SQLite
Dense text embeddings, cosine similarity, choosing models with MTEB, open sentence-transformers vs API embeddings, dimensions and quantization
How vector databases work, approximate nearest neighbour indexes (HNSW, IVF, PQ), metadata filtering, and popular options: Chroma, Qdrant, FAISS, Pinecone, Weaviate
Storing embeddings alongside relational data, pgvector index types (HNSW, IVFFlat), distance operators, and managed options like Supabase
Combining keyword (BM25) and vector search, reciprocal rank fusion, cross-encoder rerankers and two-stage retrieval pipelines
Semantic Search Engine for Notes or Papers
beginnerEmbed a folder of notes or arXiv abstracts and build a natural-language search UI that shows similarity scores
- sentence-transformers
- Chroma
- Streamlit
Hybrid Search API on Postgres
intermediateCombine Postgres full-text search with pgvector similarity using reciprocal rank fusion plus a cross-encoder reranker, and benchmark recall against vector-only search
- PostgreSQL
- pgvector
- FastAPI
- sentence-transformers
Indexing, retrieval and generation pipeline, grounding answers with citations, and when to choose RAG vs long context vs fine-tuning
Converting PDFs, HTML and Office files into clean text/markdown, fixed vs recursive vs semantic chunking, chunk size/overlap and metadata
Query rewriting and multi-query, HyDE, contextual chunk headers, parent-document retrieval, reranking, Self-RAG and Corrective RAG
Entity and relationship extraction, community summaries, local vs global search, and when graph-based retrieval beats plain vector search
Retrieval as a tool, agents that decide when and what to retrieve, iterative multi-step search, and combining vector, keyword, SQL and web search
Retrieval metrics (recall@k, MRR, nDCG), faithfulness and answer relevance, LLM-as-judge, and building golden evaluation sets
PDF Q&A Chatbot with Citations
beginnerParse PDFs into markdown, chunk and embed them, and answer questions with inline citations linking back to the source page
- Docling
- Chroma
- OpenAI API
- Streamlit
Production RAG over Technical Documentation
advancedBuild a docs assistant with hybrid search, contextual retrieval, reranking and a Ragas evaluation suite that runs on every pipeline change
- Qdrant
- Cohere Rerank or cross-encoder
- Ragas
- FastAPI
Agentic Research RAG
advancedCreate an agent that chooses between a vector store, a SQL database and web search, iterates on its queries and writes a cited report
- LangGraph
- pgvector
- Tavily
- Claude API
What makes an agent, the agent loop, and workflows vs agents (prompt chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer)
Reason-act-observe loops, plan-and-execute agents, reflection and self-critique, and Andrew Ng's agentic design patterns
Implementing the tool-use loop without a framework, writing clear tool names/descriptions/schemas, error handling and token-efficient tool results
Short-term (thread/state) vs long-term memory, semantic/episodic/procedural memory, memory stores, and summarization strategies
Orchestrator-subagent and handoff patterns, supervisor vs swarm architectures, context sharing, and when multiple agents are worth the cost
Stateful agent graphs with nodes, edges and shared state, checkpointing/persistence, human-in-the-loop and streaming
OpenAI Agents SDK, Claude Agent SDK, PydanticAI, CrewAI and smolagents compared, and when to skip a framework and use the raw API
How coding agents (Claude Code, Codex, SWE-agent) work, agentic coding workflows, and computer/browser-use agents that operate GUIs
Long-running agent harnesses, guardrails and approval steps, tracing/observability, and evaluating agent trajectories
Web Research Assistant Agent
intermediateBuild a LangGraph agent that plans sub-questions, searches the web, reflects on gaps and writes a cited research brief, with a human approval step
- LangGraph
- Tavily
- OpenAI API
- LangSmith
Multi-Agent Content Team
advancedCreate an orchestrator that hands off to researcher, writer and editor agents, with shared memory and guardrails on the final output
- OpenAI Agents SDK
- CrewAI
- Python
Mini Coding Agent
advancedImplement a coding agent from scratch with read-file, edit-file and bash tools running in a Docker sandbox, and test it on small GitHub-issue style tasks
- Claude API
- Docker
- Python
- mini-swe-agent
Hosts, clients and servers, the core primitives (tools, resources, prompts), JSON-RPC messages, and stdio vs Streamable HTTP transports
Exposing tools, resources and prompts with the official Python/TypeScript SDKs or FastMCP, and testing servers with MCP Inspector
Writing an MCP client, connecting servers to hosts like Claude Desktop and IDEs, and plugging MCP tools into agent frameworks (OpenAI Agents SDK, Claude Agent SDK, LangChain)
Remote servers over Streamable HTTP, OAuth authorization, sampling and roots, prompt-injection and tool-poisoning risks, and code execution with MCP
Custom MCP Server for Your Data
intermediateWrap a SQLite/Postgres database or notes folder as an MCP server with tools, resources and prompts, test it in MCP Inspector, and use it from Claude Desktop or VS Code
- MCP Python SDK
- FastMCP
- SQLite
- MCP Inspector
MCP-Powered Agent
advancedBuild an agent that connects to multiple MCP servers (filesystem, GitHub, your custom server) and completes multi-step tasks across them
- OpenAI Agents SDK
- MCP
- LangChain MCP Adapters
- Claude API
Remote MCP Server with OAuth
advancedDeploy a Streamable HTTP MCP server behind OAuth, add rate limiting and input validation, and connect to it from a remote MCP client
- MCP TypeScript SDK
- OAuth 2.1
- Docker
- Cloud Run or Vercel
Why evals come first, logging and reading traces, error analysis to find failure modes, binary pass/fail criteria, and turning each failure into a regression test.
Golden and synthetic test sets, assertion-based unit tests, reference-based vs reference-free metrics (ROUGE, BERTScore, semantic similarity), and running evals automatically on every prompt or model change.
Direct scoring vs pairwise comparison, writing judge prompts and rubrics, position, verbosity and self-preference bias, and validating judges against human labels.
Retrieval metrics (hit rate, MRR, context precision/recall), faithfulness and answer relevance, tool-call and trajectory evaluation for agents, and offline vs online evaluation.
What public benchmarks (MMLU-Pro, GPQA, IFEval) and LMArena measure, data contamination and saturation, and why leaderboards don't replace task-specific evals when picking a model.
Running eval suites with promptfoo, Ragas, DeepEval and Inspect (also OpenAI Evals): comparing prompts and models side by side, custom scorers, and wiring evals into CI.
Prompt Regression Tester
beginnerWrite a promptfoo config that compares two prompts across two models on 30+ test cases with deterministic assertions (contains, JSON schema, regex) and one LLM-graded rubric, then publish the comparison matrix.
- promptfoo
- YAML
- OpenAI API
- Anthropic API
RAG Evaluation Suite in CI
intermediateBuild a golden dataset (hand-written plus synthetic) for a RAG chatbot, measure retrieval (hit rate, MRR) and answer faithfulness/relevance, and run the suite on every pull request so regressions fail the build.
- Python
- Ragas
- pytest
- GitHub Actions
Calibrated LLM Judge
advancedHand-label 200 real outputs pass/fail after error analysis, write a binary judge prompt, measure TPR/TNR against your labels on a held-out split, iterate until it agrees with humans, and use it to estimate production quality with confidence intervals.
- Python
- pandas
- Jupyter
- OpenAI API
Traces and spans for LLM calls, retrieval and tool steps; logging prompts, outputs, tokens, latency and cost; and OpenTelemetry-based instrumentation.
- DOCLangfuse - LLM Observability & Application Tracing (docs)· free (open source, self-hostable; cloud free tier)
- DOCArize - Phoenix Documentation· free (open source)
- ARTOpenTelemetry - An Introduction to Observability for LLM-based applications using OpenTelemetry
- CRSHugging Face Agents Course - Bonus Unit: AI Agent Observability & Evaluation
Prompt versioning and registries, datasets and experiments, comparing runs, and choosing between Langfuse, LangSmith, W&B Weave and MLflow.
Online evals on sampled traffic, explicit and implicit user feedback, dev-prod skew and drift, dashboards and alerts, and the data flywheel that feeds evals and fine-tuning.
Logging parameters, metrics and artifacts, comparing runs, versioning data and models, and promoting models through a registry (MLflow, W&B, DVC).
Testing code, data and models, continuous integration and delivery pipelines, automated retraining and deployment gates, and release strategies like canary and shadow deploys.
Instrument an LLM App with Langfuse
beginnerAdd tracing to an existing chatbot or RAG app, log token cost and latency per request, version the system prompt in Langfuse, and build a dashboard comparing two prompt versions.
- Python
- Langfuse
- OpenAI SDK
- Docker
Agent Observability with Online Evals
intermediateInstrument a tool-using agent with OpenTelemetry and Arize Phoenix, capture every tool call, run an LLM-as-judge evaluator on a daily sample of traces, and alert when the failure rate crosses a threshold.
- Arize Phoenix
- OpenTelemetry
- LangGraph
- Python
CI/CD Pipeline for Prompts and Models
advancedSet up GitHub Actions that run your eval suite on every PR, block merges on regressions, register passing model/prompt versions in MLflow, and deploy them automatically behind a canary.
- GitHub Actions
- MLflow
- promptfoo
- Docker
Prefill vs decode, why decoding is memory-bandwidth bound, KV cache size math, latency metrics (TTFT, TPOT) vs throughput, and GPU basics for choosing hardware.
PagedAttention and continuous batching, OpenAI-compatible inference servers, local inference with Ollama and llama.cpp, and how to pick an inference engine.
Continuous batching, speculative decoding, quantization for serving, FlashAttention, prefix caching, and serving many LoRA adapters on one GPU.
Prompt caching, exact and semantic response caching, streaming, model routing and fallbacks through an LLM gateway, and using the smallest model that passes your evals.
API vs self-hosted vs serverless GPUs, autoscaling and cold starts, batch vs real-time serving, deploying vLLM on serverless platforms, and containerizing model services.
Local LLM API with Ollama
beginnerServe a quantized open model locally with Ollama, wrap it in a FastAPI service with streaming responses and request logging, and package it with Docker Compose.
- Ollama
- FastAPI
- Docker
Inference Engine Benchmark
intermediateDeploy the same 7-8B open model with vLLM and SGLang on a cloud GPU, load-test TTFT, tokens/s and throughput at several concurrency levels, try prefix caching and speculative decoding, and publish a cost-per-1M-tokens report.
- vLLM
- SGLang
- Modal
- Python
- +1
Cost-Aware LLM Gateway
advancedBuild a gateway that adds semantic caching, provider fallbacks and difficulty-based routing between a cheap and a strong model, then use your eval suite to show cost saved vs quality lost.
- LiteLLM
- Redis
- RouteLLM
- Langfuse
Direct vs indirect prompt injection, jailbreak techniques, data exfiltration through tools, the lethal trifecta for agents, and why delimiters and "please ignore" instructions don't fix it.
The OWASP LLM Top 10 (prompt injection, sensitive information disclosure, supply chain, excessive agency, system prompt leakage, unbounded consumption...), threat modeling AI systems, and frameworks like MITRE ATLAS and NIST AI RMF.
Input and output validation, moderation APIs, safety classifiers like Llama Guard, programmable rails, running checks in parallel to limit latency, and failing safely.
Detecting and redacting PII before it reaches prompts, logs and traces, sensitive information disclosure risks, data retention, and privacy-safe observability.
Planning red-team exercises, manual and automated attacks, vulnerability scanners and attack libraries, and turning findings into regression tests.
- DOCMicrosoft Learn - Planning red teaming for large language models (LLMs) and their applications
- CRSDeepLearning.AI x Giskard - Red Teaming LLM Applications· free (free DeepLearning.AI account)
- REPOMicrosoft - PyRIT (Python Risk Identification Tool for GenAI)· free (open source)
- REPONVIDIA - garak: LLM vulnerability scanner· free (open source)
Sources of bias, fairness metrics and mitigation, human-AI interaction design, model/system documentation, and AI safety and governance basics.
Prompt Injection Attack & Defense Lab
intermediateBuild a small email or document assistant with tool access, attack it with direct and indirect (poisoned document) injections, add layered defenses (input classifier, privilege separation, output checks), and report attack success rate before and after.
- Python
- OpenAI API
- Llama Guard
- promptfoo
Guarded Support Bot with PII Redaction
beginnerAdd NeMo Guardrails topical rails, a moderation check, and Presidio-based PII redaction for both prompts and stored traces to a customer-support chatbot.
- NeMo Guardrails
- Microsoft Presidio
- FastAPI
Automated Red-Team Report
advancedRun garak, PyRIT and promptfoo red-team scans against your own LLM app, map every finding to the OWASP LLM Top 10, fix the issues, and keep the attacks as a CI regression suite.
- garak
- PyRIT
- promptfoo
- GitHub Actions
Reference architecture for GenAI apps: context construction and RAG, guardrails, model gateway and routing, caching, agents, evals and observability, defensive UX and feedback loops.
Framing business problems as ML problems, data and feature pipelines, training vs serving, batch vs online prediction, deployment strategies, and monitoring and iteration.
How companies ship LLM features in production: architectures, eval and monitoring setups, latency and cost trade-offs, and lessons learned from engineering blogs.
Picking a real problem, shipping an end-to-end app with retrieval, evals, monitoring and deployment, documenting results with metrics, and understanding the AI engineer role.
Case Study Teardown
beginnerPick three LLM case studies from the Evidently or ZenML databases, diagram each architecture, compare how they evaluate and monitor quality, and publish the takeaways as a blog post.
- Excalidraw
- Markdown
System Design Doc: AI Support Assistant
intermediateWrite a full design doc for an AI customer-support assistant: requirements, architecture diagram, retrieval and model choices, guardrails, eval plan, latency/cost estimates, failure modes and rollout plan, then review it against the OWASP LLM Top 10.
- Excalidraw
- Markdown
- Back-of-the-envelope cost math
End-to-End Production Capstone
advancedShip a RAG or agent app on your own dataset with automated ingestion, hybrid retrieval, an eval suite in CI, Langfuse monitoring with user feedback, guardrails, docker-compose and a cloud deployment, plus a README that reports your eval metrics.
- FastAPI
- pgvector
- Ragas
- Langfuse
- +2