This document is a Universal Replication Protocol. You can copy-paste the template in Section 2 directly into any future AI session to instruct an agent (such as Antigravity) to perform the exact physical and mathematical analysis we developed here for any new model, precision format, or GPU configuration.
To make your future runs faster, here are the pre-computed physical parameters for the top enterprise and legacy/mid-tier GPUs (dense FP16/BF16 precision):
| GPU SKU | Form Factor | VRAM Capacity | HBM/GDDR Bandwidth | Peak FP16/BF16 (Dense) | Hardware Threshold ( |
|---|---|---|---|---|---|
| NVIDIA B200 | SXM5 | 192 GB HBM3e | 8.00 TB/s (8,000 GB/s) | 2,250 TFLOPS | 281.25 FLOP/Byte |
| NVIDIA H200 | SXM5 | 141 GB HBM3e | 4.80 TB/s (4,800 GB/s) | 989.5 TFLOPS | 206.15 FLOP/Byte |
| NVIDIA H100 | SXM5 | 80 GB HBM3 | 3.35 TB/s (3,350 GB/s) | 989.5 TFLOPS | 295.37 FLOP/Byte |
| NVIDIA A100 | SXM4 | 80 GB HBM2e | 2.039 TB/s (2,039 GB/s) | 312.0 TFLOPS | 153.02 FLOP/Byte |
| RTX 6000 Ada | PCIe | 48 GB GDDR6 (ECC) | 0.96 TB/s (960 GB/s) | 182.2 TFLOPS | 189.79 FLOP/Byte |
| NVIDIA L4 | PCIe | 24 GB GDDR6 | 0.30 TB/s (300 GB/s) | 121.0 TFLOPS | 403.33 FLOP/Byte |
Copy the text inside the block below and provide it to any advanced coding/reasoning AI to run the mathematical analysis on a new model.
You are an expert AI infrastructure architect and high-performance computing (HPC) engineer specializing in Large Language Model (LLM) serving physics.
I want you to perform a rigorous mathematical and hardware co-design analysis for a target model under different quantization formats across different GPU architectures.
Please perform this analysis step-by-step and output a highly detailed, beautifully formatted markdown report.
### INPUT SPECIFICATIONS
1. **Target Model Details**:
- Model Name: [e.g., DeepSeek-V3, LLaMA-3-70B, etc.]
- Total Parameters ($P_{\text{total}}$): [e.g., 671 Billion]
- Active Parameters per Token ($P_{\text{active}}$): [e.g., 37 Billion]
- Architecture Type: [e.g., Sparse MoE / Dense]
- Hidden Dimension ($d_{\text{model}}$): [e.g., 7168]
- Attention Layers ($N_{\text{layers}}$): [e.g., 61]
- Key-Value Heads ($N_{\text{heads\_kv}}$): [e.g., 8 (Grouped-Query Attention)]
- Head Dimension ($d_{\text{head}}$): [e.g., 128]
2. **Precision Formats to Analyze**:
- Format 1: Native FP16 (2 Bytes/param, FP16 KV Cache)
- Format 2: [e.g., Q5_K_XL (5.5 bits or 0.688 Bytes/param), FP8 KV Cache]
- Format 3: [e.g., Q4_K_XL (4.5 bits or 0.563 Bytes/param), FP8 KV Cache]
3. **Target GPU Architectures**:
- GPU SKU A: NVIDIA B200 SXM (192GB, 8.0 TB/s, 2250 TFLOPS FP16)
- GPU SKU B: NVIDIA H200 SXM (141GB, 4.8 TB/s, 989.5 TFLOPS FP16)
- GPU SKU C: [e.g., NVIDIA H100 SXM (80GB, 3.35 TB/s, 989.5 TFLOPS)]
- GPU SKU D (Optional Legacy/Mid-Tier): [e.g., NVIDIA A100 SXM (80GB, 2.039 TB/s, 312 TFLOPS) or RTX 6000 Ada (48GB, 960 GB/s, 182.2 TFLOPS) or L4 (24GB, 300 GB/s, 121 TFLOPS)]
4. **Operational Range**:
- Batch Size: B = 1 (Single User, Latency-Optimized serving)
- Prompt Context Lengths to Model: L = 1,000, 10,000, 50,000, and 100,000 tokens.
---
### STEPS OF THE MATHEMATICAL ANALYSIS
Please execute these steps with absolute mathematical rigor:
#### Step 1: Physical GPU Profiles & Arithmetic Intensity Thresholds
Calculate the Arithmetic Intensity Threshold ($\text{AI}_{\text{thresh}} = \text{Peak FLOPS} / \text{Memory Bandwidth}$) for each GPU SKU. Explain the physical meaning of this tipping point using the Factory-Warehouse metaphor (Workers vs Conveyor Belt).
#### Step 2: Weight Footprints & VRAM Allocation Limits
For each precision format, calculate:
1. Total model weight size in VRAM (GB).
2. Active parameters weight size loaded per token (GB).
3. The exact minimum number of physical GPUs required just to hold the static weights in VRAM for each GPU SKU. (Note: standard enterprise nodes scale in groups of 4 or 8 GPUs).
#### Step 3: KV Cache Fingerprint Calculations
Derive the exact KV Cache size per token in VRAM:
$$\text{KV Cache per Token (Bytes)} = 2 \times N_{\text{layers}} \times N_{\text{heads\_kv}} \times d_{\text{head}} \times \text{Bytes/param}$$
Generate a comparative memory footprint table showing the KV cache size (GB) and total static model memory footprint (Weights + KV) across your context lengths (1k, 10k, 50k, 100k) for each precision format.
#### Step 4: The Prefill Phase Math (Compute-Bound)
1. Derive the operational Arithmetic Intensity ($\text{AI}_{\text{prefill}}$) and show why it scales linearly with prompt length.
2. Solve for $L_{\text{crit}}$, the tipping point prompt length where the prefill phase transitions from memory-bound to 100% compute-bound:
$$L_{\text{crit}} \ge \frac{\text{Bytes/param} \times \text{AI}_{\text{thresh}}}{2}$$
3. Calculate the prefill execution time for a 100k prompt assuming a realistic 60% Model FLOPs Utilization (MFU) across your GPU clusters.
#### Step 5: The Decode Phase Math (Memory-Bound)
1. Derive the operational Arithmetic Intensity ($\text{AI}_{\text{decode}}$) including both the active weights and the growing historical KV cache read traffic:
$$\text{AI}_{\text{decode}} = \frac{2 \times P_{\text{active}}}{\text{Active Weights} + \text{KV Cache Size}}$$
2. Compare this against each GPU's $\text{AI}_{\text{thresh}}$ and calculate the exact percentage of compute power that is wasted (idle SMs) during single-user decode. Explain why we must accept this waste to maximize tokens/second.
#### Step 6: Physical Latency Slicing & EP Routing
In MoE architectures, the weights are distributed across the cluster via Expert Parallelism (EP). To generate 1 token:
- Dense layers (typically 25% of weights) are streamed parallelly using the aggregate VRAM bandwidth of the node.
- Active experts (typically 75% of weights) are streamed from their specific GPUs, bottlenecked by the physical single-GPU bandwidth.
- The distributed KV Cache is read over aggregate bandwidth.
Model the physical latency using this EP Routing formula:
$$\text{Latency per Token} = \frac{\text{Dense Weights}}{\text{Aggregate Bandwidth}} + \frac{\text{Active Expert Weights}}{\text{Single-GPU Bandwidth}} + \frac{\text{KV Cache Size}}{\text{Aggregate Bandwidth}}$$
#### Step 7: Final Performance Matrix
Compute the final latency per token (ms), Model FLOPs Utilization (MFU %), and the max physical generation speed (Tokens/Second) for B=1 across all context lengths (1k, 10k, 50k, 100k) for each precision format across each GPU SKU configuration.
#### Step 8: Infrastructure Co-Design & Software Optimizations
Propose concrete hardware and software recommendations to break through the memory-bound decode bottleneck:
1. Prefill-Decode Disaggregation (sizing the ratio of prefill nodes to decode pools and modeling the network transfer latency over 400 Gbps Ultra Ethernet).
2. Speculative Decoding (showing how draft-verification shifts the arithmetic intensity upward).
3. KV Cache Quantization (quantizing the KV cache to FP8 or FP4).
Make the formatting of your report pristine, using LaTeX formulas, alert blocks for critical tips, and Mermaid diagrams to represent the architecture and bottlenecks.