NInfer-3090

NInfer-3090 is a specialized C++20/CUDA inference engine for Qwen3.8-27B and Qwen3.6 on one 24 GB NVIDIA GeForce RTX 3090. Qwen3.8-27B is a first-class, tested target: the native SM86 runtime loads its official groupwise .ninfer artifact, serves OpenAI- and Anthropic-compatible APIs, and supports paged KV, compatible-prefix reuse, CUDA Graphs, MTP speculative decoding, reasoning-effort control, ReplaySSM state transactions, and concurrent cohorts through C8.

Community project, maintained on a best-effort basis. Issues and PRs are very welcome, but support and feature requests are not guaranteed.

On an RTX 3090, Qwen3.8-27B supports a measured 171K-token INT8 context with the standard 1 GiB safety headroom.

RotorQuant rk8v4 is temporarily unavailable. Upstream moved KV quantization out of the fused GQA attention kernels into a dedicated kv_cache_append Op whose contract defines K rotation as H256 and V as unrotated BF16. The rk8v4 int4-V representation has not been ported to that contract yet, so --kv-dtype rk8v4 is rejected at startup. The previously published 226K/248K RotorQuant context figures do not apply to this build. INT8 remains the default and recommended quantized profile.

This fork targets sm_86. Blackwell-only NVFP4/W4A4 and FP8 A8 tensor-core execution are unavailable. FP8 and NVFP4 weights are admitted through their A16 dequantizing routes, but the FP8 E4M3 KV-cache profile is not: its attention kernels have no SM86 implementation.

The goal is the make the utmost rippin Qwen inference stack for the 3000 series. Gladly taking PR's, all help much appreciated.

Release notes for this branch: v0.6.1.

The Dockerfile gives the shortest build path on Bazzite and other Linux distributions:

docker build --tag ninfer-3090:sm86 .

The Linux guide contains the GPU check, native Ubuntu build, model mount, server command, and Bash launchers. The project does not publish a prebuilt Linux archive or qualified Linux performance results yet.

1. Download and unzip the latest Windows release.
2. Double-click download-qwen38.bat to download the model. Interrupted downloads resume.
3. Double-click one launcher:

The API is then available at http://127.0.0.1:8080/v1. The Windows archive includes the required applications and DLLs.

Qwen3.8-27B is validated from one through eight simultaneous users. ReplaySSM cuts the memory cost of speculative decoding, allowing the faster MTP3 mode to remain enabled at C8. The table below is the new sustained test: every request generated 1,024 tokens with CUDA Graphs enabled.

The prompts were 29-34 input tokens and the server's maximum context window was 8,192 tokens per request. Each measured sequence therefore reached roughly 1,053-1,058 tokens including its generated output. This is a long-output/decode benchmark, not an 8K-prompt or long-prefill test. C1-C4 used an 8,192-token shared KV pool; C8 used 16,384 tokens so all eight requested outputs could be admitted simultaneously.

C1 is the responsive choice for a single user. C8 delivers 2.3x the total throughput when several requests are active. The C8 long-output test uses a 16K shared KV pool so all eight 1,024-token responses can be admitted together.

Prompt processing was tested separately with 4,362 fresh input tokens per request, an 8,192-token per-request context window, 512-token prefill chunks, INT8 KV, ReplaySSM/MTP3, CUDA Graphs, and prefix reuse disabled. Each request generated only 16 tokens so the run measures prefill rather than long decode.

Aggregate prefill is total fresh input tokens divided by the complete request-wave time, so it is the user-facing throughput number. NInfer currently processes one long prefill at a time; cohort batching accelerates decode, but does not multiply prompt ingestion. Consequently C1-C8 remain near 844-862 input tok/s while queued requests increase mean TTFT. Active-prefill speed excludes queue waiting and measures only the server's recorded prefill phase.

rk8v4 was an experimental, opt-in KV-cache mode for Qwen3.8-27B that applied a normalized transform to queries and keys, rotated values before four-bit storage, and reversed the value transform after attention.

It is not available in this build. Upstream now owns KV quantization in a dedicated kv_cache_append Op, whose published numerical contract rotates K with a normalized H256 transform and stores V from unrotated BF16 source values. rk8v4's H64 rotation and int4-V representation have no implementation against that contract yet, so --kv-dtype rk8v4 parses but is rejected during engine construction rather than silently degrading to INT8.

Porting RotorQuant onto the new Op is tracked as follow-up work. Until then the previously published 226,560-token and 247,872-token RotorQuant context measurements do not describe this build. Use --kv-dtype int8, which reaches 171,648 tokens at the 1 GiB headroom boundary.

The same Qwen3.8 artifact supports images. Start the server with --vision, MTP3, INT8 KV, and a 32K maximum context. The Windows archive includes run-qwen38-vision.bat for this profile.

A 1,920×1,080 image expanded to 2,074 prompt tokens and was read correctly. Measured TTFT was 3.29 seconds, decode reached 98.1 tok/s, MTP acceptance was 96.7%, and startup retained 2.16 GiB free VRAM. The artifact also declares multi-image and video support; this release test directly validated a single image.

Measured with the compact 20.84 GiB 35B-A3B artifact, a 4K shared INT8 group-64 paged KV pool, CUDA Graphs, MTP3, greedy decoding, and no competing GPU workload:

A longer 512-token-per-request check reached 286.8 tok/s at C1 and 399.1 aggregate tok/s at C2. These short-prompt measurements include request-level timing and are not directly comparable to v0.3.1's 1,500-token adaptive prompt-lookup benchmark.

Compatible-prefix reuse was validated end to end: a repeated 26-token prompt reused 24 tokens, reducing measured prefill from 371 ms to 10 ms.

The compact 35B artifact includes its vision encoder and accepts images through the same OpenAI- compatible API. Start the server with --vision and leave speculative decoding disabled. The Windows archive includes run-qwen36-35b-vision.bat for this profile.

The safe RTX 3090 profile is one request, 32K maximum context, INT8 KV, vision enabled, and MTP disabled. A current v0.6 test processed three 1,920×1,080 images correctly. Each image expanded to a 2,081-token prompt; engine TTFT was 3.76–4.07 seconds, decode was about 159 tok/s, and peak VRAM was 23,944 MiB. This leaves little room for another GPU workload.

MTP is intentionally off for this profile. At 32K, speculative recurrent state would exceed the 3090 memory budget; KV compression alone does not recover enough memory. Text-only 35B profiles can still use MTP3 as documented above.

• Native SM86 CLI and server applications for Linux and Windows.
• A prebuilt Windows archive with tested launchers.
• OpenAI Chat Completions, Responses, and Anthropic-compatible APIs.
• ReplaySSM and MTP3 for higher throughput without exceeding 24 GB VRAM.
• low, medium, and xhigh reasoning modes.
• Qwen3.8 image understanding with ReplaySSM and MTP3.
• Prefix reuse for faster repeated or shared prompts.
• Qwen3.6-35B image understanding with a guarded 32K profile.
• Windows one-user and eight-user launchers with safe tested defaults.

NInfer-3090 v0.5 and newer recognize both v1 and v2 container magic. The current 21.22 GiB v2 artifact contains additional DFlash weights and is not the artifact used for the published RTX 3090 concurrency results. The pinned compact v1 artifact keeps the measured model payload and omits DFlash, providing the known 24 GB memory profile.

Linux users build the applications from source or use the Docker image. Windows users can use the prebuilt archive, which includes the applications and required DLLs. Both platforms require an RTX 3090 or RTX 3090 Ti and a recent NVIDIA driver.

Download the official Qwen3.8 artifact as models/qwen3_8_27b.ninfer. Windows users can run download-qwen38.bat instead.

For Qwen3.6-35B-A3B, the smaller pinned container-v1 artifact is recommended on a 24 GB card. Releases v0.5 and newer also read the larger container-v2 file. An artifact magic is not NInfer version 1 message means the executable is outdated, not that the current model download is necessarily corrupt.

Developers can build from source on Windows or Linux. Windows uses Visual Studio 2022 and vcpkg. Linux uses GCC 13 with system packages or the pinned vcpkg manifest. Both builds require CUDA 12.8 or newer and CMake 3.28 or newer.

See the Windows build guide or the Linux build guide. Ordinary Windows release users do not need these tools.

Qwen3.8-27B supports distinct reasoning-effort modes. medium uses the model's normal thinking prompt. xhigh injects the checkpoint's extended deliberation instruction, asking it to validate assumptions and consider alternatives. This is a real prompt-template change, not a sampling alias.

OpenAI Chat Completions accepts a top-level reasoning_effort field:

{
  "model": "qwen3.8-27b",
  "messages": [{"role": "user", "content": "Solve this carefully..."}],
  "reasoning_effort": "xhigh",
  "max_tokens": 4096
}

OpenAI Responses uses "reasoning": {"effort": "xhigh"}. Anthropic Messages uses "output_config": {"effort": "xhigh"}. For the native CLI, pass --reasoning-effort low|medium|xhigh; use --no-thinking instead of an effort to disable reasoning. Chat Completions returns hidden reasoning separately as message.reasoning_content.

The server supports:

• OpenAI Chat Completions;
• OpenAI Responses Core with streaming and local continuation state;
• Anthropic Messages;
• compatible-prefix reuse;
• prompt-rendered function tools and parsed tool calls;
• bounded pending-request admission and JSONL request logs.

See HTTP serving and CLI usage.

The C number is the maximum number of requests NInfer can run together. C1 favors one interactive user; C8 can combine up to eight active requests into each GPU step for much higher total output.

Follow-up requests do not need to arrive at the same instant. When a running request finishes, the next waiting request can join at a safe generation boundary. Empty or finished lanes are skipped, so a C8 server also works normally with only one, two, or four active users.

This is deliberately more bounded than datacenter-style dynamic batching. The maximum number of users and GPU memory are chosen when the server starts. In return, memory use stays predictable on a 24 GB card and the server can reuse fast CUDA Graphs instead of rebuilding work continuously.

• One process owns one model on one RTX 3090.
• Concurrency is fixed at startup and limited to 1-8 by the API; compact 35B fits C1-C6 and Qwen3.8-27B fits C8/8K with MTP3 through ReplaySSM.
• The shared KV pool is fixed at startup and is not divided statically among request lanes.
• This is bounded small-scale batching, not preemptive large-scale continuous batching.
• No multi-GPU execution or CPU/GPU weight offload.
• Tool calls are returned to the client but are not executed by NInfer.
• NVFP4 A4, FP8 A8, and TMA kernels require Blackwell and are unavailable on SM86. FP8 and NVFP4 weights are admitted through their A16 dequantizing routes.
• The paged runtime exposes BF16 and INT8 group-64 KV; INT8 remains the quality-default path. Upstream's row-scaled FP8 E4M3 KV profile and rk8v4 both parse but are rejected on SM86.

The v0.6.0 Windows gate covered Qwen3.8 generation, materialization, request memory, admission, paged KV, prefix reuse, speculative rounds, and SM86 W8 Linear paths.

The v0.6.1 Linux source gate completed all 245 Docker compile and link steps with CUDA 13.1 on Ubuntu 24.04. Both Linux applications returned their --help output with GPU access enabled. A real-artifact Linux generation and Linux performance qualification remain open.

NInfer-3090 is derived from Neroued/ninfer. The upstream project targets RTX 5090/sm_120a; this fork carries the Windows and Linux SM86 compatibility layer, compact 35B artifact support, and RTX 3090-specific schedules and memory planning.

See CONTRIBUTORS.md for the complete, maintained credit list.

• airtonix added Linux and Docker build and release support in PR #1.
• ColeWheatley contributed SM86 runtime-count/GDN residency fixes, ECC diagnostics, and the GeForce-safe Docker fix in PR #7.
• justinlime added NixOS build support in PR #5.
• sry9681 contributed the device-wide GPU-memory startup fix in PR #6.
• iamwavecut contributed the swscale destination-alignment JPEG safety fix in PR #11.
• nasedkinpv contributed the tool-call parser crash fix in PR #12.
• wmehanna contributed in-place system-turn rendering for Claude Code prefix reuse in PR #13.

Please read the Pull Request Policy before opening an issue or pull request. It explains how to keep changes focused and how to document correctness, performance, VRAM, and compatibility evidence.

Apache License 2.0. See LICENSE.