Foundation Models Index

Pre-training paradigms, model families, and alignment of large language models.

Notes

• Foundation Model Overview — Architecture, model families, VRAM requirements, and selection criteria for LLMs.
• Tokenization — BPE, WordPiece, and SentencePiece tokenization algorithms and their engineering implications.
• Alignment and RLHF — SFT, RLHF/PPO, DPO, and Constitutional AI for aligning pre-trained models to HHH goals.
• Scaling Laws — Chinchilla compute-optimal training, power-law loss curves, emergent abilities, and inference-time compute scaling.

Navigation

Next → Evaluation →

Links

• AI Engineering