Qwen3.8-27B on a single RTX 5090 - install notes

实战评估（2026-09-03）：上表为基准测试数据。在真实复杂代码任务（编码 agent 场景）上， drafter 命中率从基准 0.60–0.66 掉到 ~0.35，投机解码收益大幅缩水； 综合经济效率不如 4090-48G 方案（对比 self-deployed-qwen3.8-27b-4090-48GB.md 的 vLLM + MTP 路线）。

Working setup as of 2026-09-02, measured on this machine (RTX 5090 32 GB, driver 595.80, 16 CPU cores, Ubuntu, CUDA driver API 13.2). Results:

Metric Value
math / code throughput 218-233 tok/s (lossless, DFlash2)
prose throughput 157 tok/s (acceptance-bound)
throughput @ 262144 native ctx 202 tok/s
throughput @ 393216 ctx (YaRN 1.5x) 203 tok/s
VRAM @ 393216 31.4 / 32.6 GB
prefill @ 286k depth 934 tok/s
needle recall @ 276k depth (past native 262144) exact
draft acceptance 0.60-0.66

Stack: llama.cpp PR #27342 fork @ 5ecbe1ac1 (build b10498) + MMVQ->MMQ dispatch patch + DFlash2 block-diffusion drafter (incoai Q4_K_M) + unsloth UD-IQ4_XS main model.

0. Why CUDA 13.3 is mandatory

The ptxas shipped with CUDA 13.2 miscompiles the qwen3_5 gated-delta-net (hybrid linear attention) kernels for sm_120a. Symptom: the model loads fine but produces random garbage tokens (mixed CJK/symbols) at any temperature, with or without speculative decoding.

Verified on this machine:

Configuration Output
CUDA 13.2 build, GPU, PR fork garbage
CUDA 13.2 build, GPU, upstream master garbage
CUDA 13.2 build, same model, CPU only sane
CUDA 13.3 build, GPU sane

So: the GGUF is fine, the code is fine, the toolchain is the problem. Build with CUDA 13.3.

1. CUDA 13.3 toolchain (side-by-side, does not touch system CUDA)

conda create -y -n cuda133 -c "nvidia/label/cuda-13.3.73" -c conda-forge cuda-toolkit

# conda's toolkit lacks the driver stub needed at link time; link against the system stub
mkdir -p /usr/local/miniconda3/envs/cuda133/lib/stubs
ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/local/miniconda3/envs/cuda133/lib/stubs/libcuda.so

If there is no system CUDA at /usr/local/cuda, install any CUDA toolkit's lib64/stubs/libcuda.so - it is only used at link time; at runtime the real libcuda.so.1 from the driver is used.

2. Source: clone, patch, build

git clone https://github.com/ggml-org/llama.cpp.git /root/llama.cpp
cd /root/llama.cpp
git fetch origin refs/pull/27342/head:pr-27342
git checkout 5ecbe1ac          # "support DFlash2", the build behind published numbers

# MMVQ->MMQ patch (+11% on verify batches): runtime env GGML_CUDA_MMVQ_MAX_BATCH
git clone --depth 1 https://github.com/LukasParke/qwen38-27b-5090.git /root/qwen38-27b-5090
git apply /root/qwen38-27b-5090/patches/mmvq-max-batch-rebased.patch

# context-cap patch for >262k serving (see section 3) - apply BEFORE building

Build (about 7-10 min at -j16):

export PATH=/usr/local/miniconda3/envs/cuda133/bin:$PATH
export CUDACXX=/usr/local/miniconda3/envs/cuda133/bin/nvcc

cmake -B build-cuda133 \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_FA=ON \
  -DGGML_CUDA_GRAPHS=ON \
  -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DGGML_CUDA_F16=ON \
  -DCMAKE_CUDA_ARCHITECTURES=120 \
  -DCMAKE_CUDA_FLAGS="-O3" \
  -DCMAKE_CUDA_COMPILER=/usr/local/miniconda3/envs/cuda133/bin/nvcc \
  -DCUDAToolkit_ROOT=/usr/local/miniconda3/envs/cuda133 \
  -DCMAKE_SHARED_LINKER_FLAGS="-L/usr/local/miniconda3/envs/cuda133/lib/stubs -lcuda" \
  -DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/miniconda3/envs/cuda133/lib/stubs -lcuda" \
  -DCMAKE_BUILD_RPATH=/usr/local/miniconda3/envs/cuda133/lib \
  -DCMAKE_INSTALL_RPATH=/usr/local/miniconda3/envs/cuda133/lib

cmake --build build-cuda133 -j$(nproc)
./build-cuda133/bin/llama-server --version
# -> version: 0.1.2-dev (build 10498, commit 5ecbe1ac1)

Notes: - -DCMAKE_BUILD_TYPE=Release is required; --config Release at build time is a no-op with the Makefile generator. - npm ERR! during configure is the optional embedded web UI failing (old node); harmless. Silence it with -DLLAMA_BUILD_UI=OFF. - Link errors like undefined reference to cuMemMap mean the stub fix in step 1 was missed.

3. Context-cap patch (needed only for context > 262144)

The server caps the slot context at the model's training context (tools/server/server-context.cpp, around line 1202), which silently shrinks any extrapolated ctx back to 262144. Make the cap opt-out:

        int n_ctx_slot = llama_n_ctx_seq(ctx_tgt);
        // allow the server to serve contexts beyond n_ctx_train (rope extrapolation);
        // opt-in since output quality past the training horizon is not guaranteed
        static const bool allow_overtrain_ctx = getenv("LLAMA_SERVER_ALLOW_OVERTRAIN_CTX") != nullptr;
        if (n_ctx_slot > n_ctx_train && !allow_overtrain_ctx) {
            SRV_WRN("the slot context (%d) exceeds the training context of the model (%d) - capping\n", n_ctx_slot, n_ctx_train);
            n_ctx_slot = n_ctx_train;
        }

Also add #include <cstdlib> to the includes at the top of the file. Rebuild after the edit (incremental, fast). This is a local deployment patch, not upstream material - private forks are exempt from the project's contribution rules.

4. Models

# HF_HOME=/root/models
python3 - << 'EOF'
from huggingface_hub import snapshot_download
snapshot_download('unsloth/Qwen3.8-27B-GGUF', allow_patterns=['*IQ4*'],
                  local_dir='/root/models/Qwen3.8-27B-GGUF')
snapshot_download('incoai/Qwen3.8-27B-DFlash2-GGUF',
                  local_dir='/root/models/Qwen3.8-27B-DFlash2-GGUF')
EOF

Files used: - Main: Qwen3.8-27B-UD-IQ4_XS.gguf (14.2 GB; sha256 40fac4050e940397dbf13087afd50f4734a11805bf9d65ef8ddd7483470e6199) - Draft: Qwen3.8-27B-DFlash2-Q4_K_M.gguf (1.1 GB; Q8_0/BF16 variants exist but measured no better on 5090)

5. Server script

/root/start-dflash2.sh:

#!/usr/bin/env bash
set -euo pipefail

# GGML_CUDA_MMVQ_MAX_BATCH=1 routes spec-decode verify batches to GEMM (MMQ)
# instead of per-token MMVQ: +11%.
export GGML_CUDA_MMVQ_MAX_BATCH="${GGML_CUDA_MMVQ_MAX_BATCH:-1}"

MAIN_GGUF="${MAIN_GGUF:-/root/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-IQ4_XS.gguf}"
DRAFT_GGUF="${DRAFT_GGUF:-/root/models/Qwen3.8-27B-DFlash2-GGUF/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
CTX="${CTX:-262144}"
PORT="${PORT:-8080}"
CTK="${CTK:-q8_0}";  CTV="${CTV:-q8_0}"    # main KV
CTKD="${CTKD:-q4_0}"; CTVD="${CTVD:-q4_0}" # draft KV (drafter is verified by the
                                           # target model, so q4_0 is safe here)
# Context extrapolation beyond native 262144: e.g. ROPE_SCALE=1.5 CTX=393216
ROPE_ARGS=()
if [ "${ROPE_SCALE:-0}" != "0" ]; then
  ROPE_ARGS=(--rope-scaling yarn --rope-scale "$ROPE_SCALE" --yarn-orig-ctx 262144)
fi

exec /root/llama.cpp/build-cuda133/bin/llama-server \
  -m "$MAIN_GGUF" \
  --model-draft "$DRAFT_GGUF" \
  --alias qwen3.8-27b \
  --host 0.0.0.0 \
  --port "$PORT" \
  --ctx-size "$CTX" \
  --n-gpu-layers 999 \
  --flash-attn on \
  --cache-type-k "$CTK" \
  --cache-type-v "$CTV" \
  --cache-type-k-draft "$CTKD" \
  --cache-type-v-draft "$CTVD" \
  --parallel 1 \
  --ubatch-size "${UBATCH:-1024}" \
  --batch-size "${UBATCH:-1024}" \
  "${ROPE_ARGS[@]}" \
  --no-mmap \
  --jinja \
  --spec-type draft-dflash \
  --spec-draft-n-max 8 \
  --backend-sampling \
  --cache-reuse 256 \
  --reasoning-effort default

Profiles (all measured on this machine)

# Speed profile: 131k ctx, f16 KV  -> 218-233 tok/s
CTX=131072 CTK=f16 CTV=f16 CTKD=f16 CTVD=f16 bash start-dflash2.sh

# Max native profile: 262144 ctx, q8_0 KV  -> 202 tok/s, 27.2 GB  (script default)
bash start-dflash2.sh

# Extrapolated profile: 393216 ctx (YaRN 1.5x) -> 203 tok/s, 31.4 GB
LLAMA_SERVER_ALLOW_OVERTRAIN_CTX=1 ROPE_SCALE=1.5 CTX=393216 UBATCH=512 bash start-dflash2.sh

Notes: - 393216 with the default UBATCH=1024 OOMs on compute buffers; UBATCH=512 fits and still prefills at ~1300 tok/s shallow, ~930 tok/s at 286k depth. - --spec-draft-n-max 8 is clamped to 7 internally (block_size - 1); the warning is normal. - cache_reuse is not supported by this context warning: expected, feature is disabled. - Quality beyond 262144 is YaRN extrapolation territory; verified usable (needle recall at 276k) but expect some degradation vs native range.

6. Verification checklist

curl -s http://127.0.0.1:8080/health                                    # {"status":"ok"}
nvidia-smi --query-gpu=memory.used --format=csv,noheader                # ~25580/27232/31354 MiB

# sanity: must answer coherently (garbage = CUDA <13.3 build)
curl -s http://127.0.0.1:8080/completion -d '{"prompt":"The capital of France is","max_tokens":16,"temperature":0.0}' | jq -r .content

# throughput: OpenAI endpoint, check timings.predicted_per_second
# expect >200 tok/s on code/math, 130-160 on prose

Needle test past native horizon (262144): build ~1.85 MB of repetitive filler (~287k tokens at ~6.45 chars/token for repetitive text), insert The secret launch code for the project is BANANA-7391-QUASAR. at ~96% depth, ask for the code. Must answer BANANA-7391-QUASAR.

Watch draft acceptance in the server log: healthy is 0.4-0.65 with mean draft length 3.5-5.5. A value near 0 with garbage output means a broken build (see section 0).

7. Optional: opencode integration

Global config ~/.config/opencode/opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "llamacpp": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "llama.cpp (local RTX 5090)",
      "options": {
        "baseURL": "http://127.0.0.1:8080/v1",
        "apiKey": "sk-local"
      },
      "models": {
        "qwen3.8-27b": {
          "name": "Qwen3.8-27B DFlash2 (local)",
          "tool_call": true,
          "reasoning": true,
          "interleaved": { "field": "reasoning_content" },
          "attachment": false,
          "limit": { "context": 393216, "output": 32768 },
          "cost": { "input": 0, "output": 0 }
        }
      }
    }
  }
}

Restart opencode, then select llamacpp/qwen3.8-27b via /models. Add "model": "llamacpp/qwen3.8-27b" to the config to make it the default.

8. Troubleshooting

Symptom Cause / fix
Random garbage tokens at any sampling setting CUDA 13.2 ptxas miscompilation of qwen3_5 GDN kernels; rebuild with CUDA 13.3 (section 0-2)
undefined reference to cuMemMap/cuDeviceGet/... at link missing driver stub; redo the stub symlink + -lcuda linker flags
Server starts but ctx shrinks to 262144 despite -c 393216 server caps slot ctx at n_ctx_train; apply the section 3 patch and set LLAMA_SERVER_ALLOW_OVERTRAIN_CTX=1
OOM allocating compute buffers at 393k UBATCH=512
dflash requires ctx_other to be set error at startup transient memory-fitting probe; harmless if the server reaches "listening"
draft acceptance ~0 + garbage broken build (see first row) or tokenizer mismatch; verify model sha256
pkill -f llama-server kills your own shell use pkill -x llama-server
Web UI missing in llama-server npm ci failed (old node); harmless for API use

9. References