Tag: genai-interviews

All the articles with the tag "genai-interviews".

• Distributed Training at Scale: Parallelism, Memory, and the Optimizer Stack

  17 Jul, 2026

  How DDP, tensor/pipeline/sequence parallelism, ZeRO/FSDP, activation recomputation, mixed precision, and the optimizer stack combine to train models no single GPU could hold.

• Transformer Architecture & Key Design Decisions

  6 May, 2026

  A deep dive into the transformer architecture, why decoder-only models won, and the key design decisions — RoPE, GQA, Flash Attention, MoE — that define every modern LLM.