meta-llama/Llama-3.2-1B-Instruct

Overview

This recipe groups the Llama 3.2 1B Instruct family under the canonical Meta parent. The Xeon 6 validation workbook has PASS rows for the Red Hat AI W8A8 and AMead10 AWQ variants.

The W8A8 checkpoint is detected from compressed-tensors metadata. The AWQ variant uses the recipe's --quantization awq selector.

Prerequisites

• Hardware: 1x Xeon6/Xeon5 NUMA node
• vLLM >= 0.6.2

pip (Intel Xeon 6 CPUs)

For Intel and AMD x86 CPUs, follow the CPU pre-built wheels installation instructions.

Docker (Intel Xeon 6 CPUs)

docker pull vllm/vllm-openai-cpu:latest-x86_64

Intel Xeon 6

vllm serve meta-llama/Llama-3.2-1B-Instruct \
  --tensor-parallel-size 1

Docker (the image entrypoint is vllm serve):

docker run \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-cpu:latest-x86_64 meta-llama/Llama-3.2-1B-Instruct \
  --tensor-parallel-size 1

The example uses TP=1. Adjust TP/DP for the deployment topology; host-specific CPU binding is intentionally not hard-coded.

Runtime and Platform Tuning

The following validation settings are not model requirements and are intentionally not portable CPU recipe defaults:

• --max-num-batched-tokens and --max-num-seqs: scheduler batch/concurrency tuning for workload shape, latency targets, and platform capacity.
• --gpu-memory-utilization: platform memory-budget tuning.
• --no-enable-prefix-caching: workload/benchmark cache-policy tuning.
• VLLM_ENGINE_ITERATION_TIMEOUT_S: runtime operational timeout tuning.

Hardware-specific overrides may still use these settings when they are part of a validated platform configuration.

References

• Meta parent model
• Red Hat AI W8A8 checkpoint
• AMead10 AWQ checkpoint