bluegreencity-ki

LLM Setup and Configuration Guide

This guide covers setting up local LLM (Large Language Model) infrastructure on the workstation for AI-assisted development.

Primary solution: vLLM — a Docker container serving unsloth/Qwen3.8-27B-NVFP4 on port 8080. It replaces the earlier LM Studio, Ollama, and llama.cpp setups.

Estimated setup time: 30-45 minutes (vLLM is pre-configured in the repository)

Overview

This guide covers:

✅ vLLM setup and configuration (primary local LLM server)
✅ GitHub Copilot integration with the local vLLM model
✅ KiloCode extension as an alternative
✅ MCP Server setup for database access
✅ AnythingLLM and Open WebUI interfaces
⚠️ Legacy: LM Studio, Ollama, llama.cpp (superseded by vLLM — kept for reference)

Part 1: vLLM Setup (primary)

vLLM is the primary local LLM server. It runs in a Docker container and serves the unsloth/Qwen3.8-27B-NVFP4 model via an OpenAI-compatible API on port 8080. It replaces the earlier LM Studio, Ollama, and llama.cpp setups.

1.1 Why vLLM

High-throughput inference engine with continuous batching
Serves Qwen3.8-27B with MTP (Multi-Token Prediction) speculative decoding
262k-token native context (512k/1M via YaRN RoPE scaling / --hf-overrides possible, but currently not active — see 1.5/1.6)
OpenAI-compatible API (works with Copilot, KiloCode, AnythingLLM, Open WebUI)
Managed via Docker Compose in the repository (scripts/local-llm/)

1.2 Prerequisites

Docker Desktop for Windows with the NVIDIA runtime
NVIDIA GPU with 96 GB VRAM (RTX PRO 6000 Blackwell)
Hugging Face token for the model, stored in scripts/local-llm/.env.vllm

1.3 Starting the vLLM Server

.\scripts\local-llm\start_vllm_server.ps1

The script:

Refuses to start if llama-server is still running (port 8080 + VRAM conflict)
Removes any existing vllm-qwen38-27b container
Starts the container via Docker Compose (detached, using .env.vllm)
Waits for the health check

API endpoint: http://localhost:8080/v1

1.4 Stopping the vLLM Server

docker rm -f vllm-qwen38-27b

1.5 Configuration

The server is configured in scripts/local-llm/docker-compose.vllm.yml. Key parameters:

Current configuration (2026-09-15):

Parameter	Value	Purpose
`--max-model-len`	`auto`	Auto-fit to available KV memory (→ 262k native on 96 GB @ 0.9)
`--max-num-seqs`	`8`	Max parallel sequences (recipe value for `rtx_pro_6000`)
`--tensor-parallel-size`	`1`	Single GPU
`--gpu-memory-utilization`	`0.9`	VRAM budget for weights + KV cache + CUDA graphs
`--mm-encoder-tp-mode`	`data`	Data-parallel vision encoder (recipe default)
`--speculative-config`	`mtp`, 5 tokens	MTP speculative decoding (recipe-verified for `rtx_pro_6000` / `dgx_spark_gb10`)
`--enable-auto-tool-choice`	—	Tool-choice routing (active with the current `VLLM_USE_RUST_FRONTEND=0`)
`--reasoning-parser`	`qwen3`	Parse thinking blocks
`--tool-call-parser`	`qwen3_coder`	Tool calling (official recipe parser)
`--chat-template`	`/opt/chat_templates/qwen3.8-froggeric-v22.3.jinja`	froggeric v22.3 fixed chat template (JSON-string tool args, empty-think poisoning)
`--default-chat-template-kwargs`	`enable_thinking: true, preserve_thinking: true, reasoning_effort: medium`	Server-wide thinking level (medium = model default; override per request)
`--async-scheduling`	—	Overlap request scheduling with GPU execution (recipe-verified for this GPU)

V1 defaults (not in compose): enable_prefix_caching and enable_chunked_prefill are on by default in vLLM V1. kv_cache_dtype resolves to fp8 (torch.float8_e4m3fn) automatically for this mixed-precision NVFP4 checkpoint (confirmed in the startup log). max_num_batched_tokens defaults to 8192 (matches the recipe value for this GPU).

Environment (compose environment): VLLM_USE_RUST_FRONTEND=0 — the experimental Rust frontend is disabled (since 2026-09-16): while it was enabled it produced weird/mangled model responses, so it was switched off for output quality. While enabled it had improved decode throughput but also lacked --hf-overrides support (see 1.6 long-context note); with =0, --hf-overrides is supported again. VLLM_WSL2_ENABLE_PIN_MEMORY=1 avoids the WSL2 pinned-memory issue.

The model is multimodal (text + image + video, vision_config in the checkpoint); audio is not supported. Image tokens count against the context length (~2.3k tokens per 1920×1200 image).

Per-request thinking override: pass chat_template_kwargs (e.g. {"reasoning_effort":"low"} or {"enable_thinking":false}).

1.6 Performance Tuning

Current setup (measured 2026-09-15) — 262k native / 8 seqs / MTP=5 / async-scheduling / reasoning_effort: xhigh; warm server, short prompts, RTX PRO 6000 Blackwell (96 GB VRAM). Measured with the Rust frontend enabled; it was disabled again on 2026-09-16 (see findings)

Change	Result (short prompt, no thinking, ~300 tokens)
MTP=3, no async-scheduling (cold)	~109 tok/s
MTP=3 (warm)	~148–153 tok/s
MTP=5 (recipe-verified value for this GPU)	~200 tok/s
MTP=5 + `--async-scheduling` (current)	~230 tok/s

MTP acceptance at depth 5: ~90.7% overall (per-position 98/94/91/86/85). vLLM logs a warning that num_speculative_tokens > 1 “may result in lower acceptance rate” — here the recipe-verified 5 outperformed the conservative default with higher acceptance.
xhigh thinking (server default): ~155–175 tok/s on short prompts; thinking increases output length, so the tok/s is lower than the no-thinking number.
KV-cache usage under load: 0% (pool ~1.7M tokens), no preemptions — plenty of headroom for the single-user profile.

Earlier tuning (2026-08-19/21) — RTX PRO 6000 Blackwell (96 GB VRAM), 50k–225k-token prompts, 1–8 parallel requests:

Config	KV Pool	Generation Throughput	MTP Acceptance
1M / 16 seqs (original)	very small	0.2–0.5 tokens/s	0–75% (volatile, long 0% stretches)
512k / 8 seqs	1.89M tokens	125–161 tokens/s	41–58%
262k native / 8 seqs	1.82M tokens	~127 tokens/s	43–53%
1M / 6 seqs	1.92M tokens	85–120 tokens/s	65–66%
512k / 4 seqs (then)	1.89M tokens	30–76 tokens/s (long contexts)	60–95% (content-dependent)

Findings:

The original 1M/16 config collapsed the KV-cache pool, dropping throughput to 0.2–0.5 tokens/s under parallel load. The fix was max-num-seqs (16 → 4–6), not the context length.
The KV pool is ~1.9M tokens for all tested configs (vLLM does not reserve seqs × max_len); max-num-seqs mainly caps concurrency and CUDA-graph batch sizes.
Long-context decode is memory-bandwidth-bound: ~30 tokens/s at ~225k context is the physical limit on this GPU (100% util, ~300 W, no throttling).
MTP depth (updated 2026-09-15): the checkpoint’s MTP head has 3 layers (mtp_num_hidden_layers: 3). num_speculative_tokens: 5 per the official recipe’s rtx_pro_6000 / dgx_spark_gb10 overrides now replaces the earlier depth-1 default (which showed 75–97% acceptance): depth 5 with async-scheduling gives ~230 tok/s vs ~150 at depth 3, with ~91% acceptance. vLLM’s speculative-decoding warning (“may result in lower acceptance rate”) did not materialize. Acceptance is a property of the generated text, not the config — measure over varied prompts.
Checkpoint choice: unsloth (mixed-precision NVFP4) leaves ~2× the KV cache of the uniform-W4A4 Inferact build, which drafts slightly better. For long-context single-GPU serving, unsloth is the right choice (per the official vLLM recipe).
vLLM version: vllm/vllm-openai:latest (v0.29.0 as of 2026-09-15; 0.27.1 on 2026-08-20) is newer than the :nightly tag (0.26.1rc1.dev994 at the time) — use :latest.
Rust frontend (disabled 2026-09-16): VLLM_USE_RUST_FRONTEND=0. While it was enabled (=1, 2026-09-15) it served the API via a Rust router and improved decode throughput (together with --async-scheduling), but it was disabled again because it produced weird responses — output quality over raw throughput. Secondary gap while enabled: --hf-overrides not supported (tracked in vllm#56931, feature-parity: vllm#44280) — with =0 this is no longer a blocker for the context extension below.
Tokenizer bug (fixed upstream): the original unsloth checkpoint shipped a tokenizer.json with truncation: max_length 2048, silently truncating all prompts. Unsloth re-uploaded a fixed revision (truncation: null); the HF cache picked it up automatically. Verify with: grep -o 'truncation.: null' .../tokenizer.json.
Long context: the official recipe extends context via --max-model-len + --hf-overrides '{"text_config":{"max_position_embeddings": N}}' (nested under text_config). The model’s native limit is 262k (rope_type: default); 512k/1M are supported extensions (YaRN RoPE scaling). Currently NOT active: the server runs with --max-model-len auto → 262k native. To enable 512k/1M, add the --hf-overrides argument to docker-compose.vllm.yml (supported again with VLLM_USE_RUST_FRONTEND=0).

1.7 Monitoring

# Live engine metrics (throughput, KV-cache usage, MTP acceptance)
docker logs vllm-qwen38-27b 2>&1 | Select-Object -Last 10

# Prometheus metrics
curl -s http://localhost:8080/metrics

Key metrics to watch:

Avg generation throughput (tokens/s)
GPU KV cache usage (%)
Avg Draft acceptance rate (MTP)
Running / Waiting request counts

1.8 Logs

Startup and runtime logs are written to logs/vllm/vllm_start_YYYYMMDD_HHMMSS.log.

✓ Checkpoint: vLLM server is running at http://localhost:8080/v1

Legacy: LM Studio (superseded by vLLM)

The following LM Studio instructions are kept for reference only. vLLM (above) is the current primary local LLM server.

1.1 Installation

Download LM Studio:
- Visit https://lmstudio.ai/
- Download the installer for Windows
- Run the installer and follow the prompts
- Default installation location: C:\Users\<YourUsername>\AppData\Local\Programs\LM Studio
Launch LM Studio:
- Start LM Studio from the Start Menu or desktop shortcut
- Accept the terms of service on first launch

1.2 Downloading Models

LM Studio has a built-in model browser to download LLMs:

Open the Model Browser:
- Click on the 🔍 Search icon in the left sidebar
Browse Available Models:
- Explore the model browser to find models suitable for your hardware
- Consider your available VRAM when selecting a model
- See the “Recommended Models by Use Case” section (Part 8) for specific recommendations
Download a Model:
- Search for the model name (e.g., “DeepSeek-Coder”)
- Click the Download button next to your preferred quantization:
  - Q4_K_M - Good balance (recommended for most users)
  - Q5_K_M - Higher quality, more memory
  - Q8_0 - Highest quality, requires significant memory
- Wait for download to complete (models are 4-16 GB)

Note: Models are stored in: C:\Users\<YourUsername>\.cache\lm-studio\models

1.3 Loading and Configuring a Model

Load a Model:
- Click the 💬 Chat icon in the left sidebar
- At the top, click “Select a model to load”
- Choose your downloaded model from the list
- Wait for the model to load into memory (10-30 seconds)
Configure Model Parameters:

Click the ⚙️ Settings icon (top-right of chat window) to adjust:

Context Length:
- Default: 2048 tokens
- Recommended for code: 4096 - 8192 tokens
- Higher = more context but slower inference
- Check your model’s maximum context length
Temperature:
- Range: 0.0 (deterministic) to 2.0 (creative)
- For code generation: 0.1 - 0.3 (more precise)
- For creative writing: 0.7 - 1.0 (more varied)
- Default: 0.7
Top P (Nucleus Sampling):
- Range: 0.0 to 1.0
- Default: 0.95
- Lower = more focused, Higher = more diverse
Repeat Penalty:
- Range: 1.0 (no penalty) to 1.5 (high penalty)
- Default: 1.1
- Prevents the model from repeating itself
GPU Layers:
- Default: Auto (recommended)
- Manually set if you want to offload more/less to GPU
- More layers = faster but uses more VRAM

System Prompt (Optional):

Configure the model’s behavior with a system prompt:

You are an expert Python and geospatial data scientist. You help with 
code for analyzing urban tree drought risk using GeoPandas, PostGIS, 
and related libraries. Provide clear, well-documented code examples.

Click “Save Preset” to reuse this configuration.

1.4 Starting the API Server

LM Studio includes an OpenAI-compatible API server:

Open Server Tab:
- Click the 🔌 Local Server icon in the left sidebar
Configure Server:
- Port: Default is 1234 (do not change unless needed)
- CORS: Enable “Allow CORS requests” if accessing from web apps
- API Key: Optional - leave empty for local development
Select Model:
- Choose the model to serve from the dropdown
- Use the same model you loaded earlier
Start Server:
- Click the green “Start Server” button
- Server URL: http://127.0.0.1:1234/v1
- Keep LM Studio running while the server is active
Test the Server:

Open PowerShell and test with curl:
```
curl http://127.0.0.1:1234/v1/models
```
You should see a JSON response with your loaded model.

✓ Checkpoint: LM Studio server is running at http://127.0.0.1:1234/v1

1.5 Resources for LM Studio

Official Documentation:

LM Studio Docs: https://lmstudio.ai/docs
Model Browser: https://lmstudio.ai/docs/basics/model-browser
Local Server: https://lmstudio.ai/docs/basics/server

Community Resources:

LM Studio Discord: https://discord.gg/aPQfnNkxGC
Reddit r/LocalLLaMA: https://reddit.com/r/LocalLLaMA
Model recommendations: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard

Understanding Model Parameters:

Temperature explained: Lower values make output more deterministic and focused
Context length: Amount of text (in tokens) the model can process at once
Quantization (Q4, Q5, Q8): Compression method - higher = better quality but larger size
VRAM requirements: Generally model_size × 1.2 (e.g., 7B model needs ~8-10GB VRAM)

Part 2: GitHub Copilot Integration (15 minutes)

Connect GitHub Copilot in VSCode to use your local vLLM model instead of OpenAI.

Prerequisites:

Active GitHub Copilot subscription
vLLM server running (see Part 1.3)
VSCode Remote SSH connection to workstation (see Workstation_Remote_Development.md)

2.1 Configure GitHub Copilot Chat

Open VSCode Settings:
- Press Ctrl+, (or Cmd+, on macOS)
- Or: File → Preferences → Settings
Search for Copilot Settings:
- Type: github.copilot.chat.localhostChatModels

Add Local Model Configuration:

Click “Edit in settings.json” and add:

{
  "github.copilot.advanced": {
    "debug.overrideEngine": "local"
  },
  "github.copilot.chat.localhostChatModels": [
    {
      "model": "vllm",
      "endpoint": "http://127.0.0.1:8080/v1/chat/completions",
      "name": "vLLM Qwen3.8-27B (Local)"
    }
  ]
}

Important: Use /v1 endpoint:
- ✅ Correct: http://127.0.0.1:8080/v1/chat/completions
- ❌ Wrong: http://127.0.0.1:8080/chat/completions
- The /v1 is required for OpenAI API compatibility
Reload VSCode:
- Press Ctrl+Shift+P (or Cmd+Shift+P on macOS)
- Type: “Developer: Reload Window”
- Press Enter

2.2 Using Local LLM in Copilot

Open Copilot Chat:
- Click the Copilot icon in the Activity Bar (left sidebar)
- Or press Ctrl+Shift+I (Windows/Linux) or Cmd+Shift+I (macOS)
Select Local Model:
- At the top of the chat panel, click the model dropdown
- Select “vLLM Qwen3.8-27B (Local)”
Test the Integration:

Try a simple prompt:
```
Explain how to read a GeoPackage with GeoPandas
```
The response should come from your local vLLM model.

2.3 Limitations and Notes

Current Limitations:

Code completion (inline suggestions) still uses GitHub’s cloud models
Only chat functionality uses the local model
Requires active SSH session to workstation

Performance Tips:

Smaller models (7B) respond faster but may be less accurate
Use temperature 0.2-0.3 for code-related tasks
Close other GPU-intensive applications for better performance

Troubleshooting:

If connection fails, verify the vLLM container is running and healthy (docker ps)
Check the endpoint URL includes /v1 and uses port 8080
Ensure you’re connected via SSH to the workstation
Look at VSCode Output panel (View → Output → GitHub Copilot) for errors

2.4 llama.cpp Server Autostart on Windows Boot (legacy)

Note: llama.cpp is superseded by vLLM (Part 1). The llama.cpp autostart task is kept for reference only. The vLLM container is managed via start_vllm_server.ps1 and does not require a Windows Scheduled Task.

For the shared workstation, the repository also supports a local llama.cpp server that starts automatically during Windows boot, even before a user signs in.

Why this setup exists:

Windows updates can reboot the workstation automatically.
A startup task avoids manual re-login just to restart the local inference endpoint.
The task runs as SYSTEM in Session 0, so it is independent of interactive VS Code or RDP sessions.

Managed scripts in this repository:

scripts/local-llm/start_llama_server.ps1 starts llama-server.exe as a detached process.
scripts/local-llm/register_llama_server_startup.ps1 registers the Windows Scheduled Task.
scripts/local-llm/unregister_llama_server_startup.ps1 removes the Scheduled Task again.

Registration command:

pwsh -NoProfile -File scripts/local-llm/register_llama_server_startup.ps1

This creates the task BlueGreenCity-KI llama.cpp server with these properties:

Trigger: At system startup
User: SYSTEM
Run level: Highest
Action: powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts/start_llama_server.ps1

VS Code tasks:

Register llama.cpp startup task
Unregister llama.cpp startup task

Operational details:

Logs are written to logs/llama-cpp/llama_server_YYYYMMDD_HHMMSS.log
Hugging Face cache defaults to C:\ProgramData\llama-cpp\huggingface
Override the cache root with environment variable LLAMA_CPP_CACHE_ROOT if needed

Verify registration:

Get-ScheduledTask -TaskName 'BlueGreenCity-KI llama.cpp server'

Manual test run:

Start-ScheduledTask -TaskName 'BlueGreenCity-KI llama.cpp server'
Get-ScheduledTaskInfo -TaskName 'BlueGreenCity-KI llama.cpp server'

Verification target:

The process should appear as NT-AUTORITÄT\SYSTEM
The process should run in Session 0
The task should report LastTaskResult = 0

Removal command:

pwsh -NoProfile -File scripts/local-llm/unregister_llama_server_startup.ps1

✓ Checkpoint: GitHub Copilot Chat uses the local vLLM model

Part 3: KiloCode Extension (Alternative) (10 minutes)

KiloCode is an alternative AI coding assistant that works with local LLM servers.

3.1 Installation

Install KiloCode Extension:
- Open VSCode Extensions (Ctrl+Shift+X)
- Search for: “KiloCode”
- Click Install
Open KiloCode Settings:
- Press Ctrl+, and search for kilocode
- Or click the gear icon next to KiloCode in Extensions

3.2 Configuration

Set OpenAI-Compatible Endpoint:

In VSCode settings.json:

{
  "kilocode.apiEndpoint": "http://127.0.0.1:8080/v1",
  "kilocode.model": "unsloth/Qwen3.8-27B-NVFP4",
  "kilocode.apiKey": "not-needed"
}

Configure Model Parameters (Optional):

{
  "kilocode.temperature": 0.2,
  "kilocode.maxTokens": 2048
}

3.3 Usage

Open KiloCode Panel:
- Click the KiloCode icon in the Activity Bar
- Or use the command palette: Ctrl+Shift+P → “KiloCode: Open”
Ask Questions:
- Type your question or code request
- Press Enter to submit
- The local LLM will respond
Code Actions:
- Select code in editor
- Right-click → KiloCode → Choose action:
  - Explain Code
  - Refactor
  - Add Comments
  - Generate Tests

✓ Checkpoint: KiloCode is configured and working with local LLM

Part 4: MCP Server Setup (20-25 minutes)

Model Context Protocol (MCP) allows LLMs to access external tools like databases.

4.1 Understanding MCP

What is MCP?

Protocol developed by Anthropic for LLM-tool integration
Allows LLMs to query databases, run commands, access APIs
Servers provide specific capabilities (database, filesystem, etc.)

Architecture:

vLLM (LLM) ←→ MCP Client ←→ MCP Server ←→ Database/Tools

4.2 Installing Docker MCP Hub

Docker MCP Hub provides pre-configured MCP servers in containers.

Prerequisites:
- Docker Desktop for Windows installed
- Docker running (check system tray for Docker icon)
- Basic Docker knowledge

Install Docker MCP Hub:

Open PowerShell and run:

# Clone the Docker MCP Hub repository
git clone https://github.com/docker/mcp-hub.git
cd mcp-hub
   
# Start the hub
docker-compose up -d

Verify Installation:
```
docker ps
```
You should see MCP Hub containers running.

4.3 Setting Up Database MCP Server

Configure MCP to access the PostGIS database:

Configure Database Connection:

Create mcp-config.json in the MCP Hub directory:

{
  "mcpServers": {
    "postgres": {
      "command": "docker",
      "args": [
        "run",
        "--rm",
        "-e", "POSTGRES_CONNECTION_STRING=postgresql://postgres:your_password@host.docker.internal:5432/bgcki",
        "mcp/postgres"
      ]
    }
  }
}

Replace your_password with the actual database password from .env

Test Database Connection:

# From the mcp-hub directory
docker run --rm `
  -e "POSTGRES_CONNECTION_STRING=postgresql://postgres:your_password@host.docker.internal:5432/bgcki" `
  mcp/postgres test

Available Database Operations:
- Query tables
- Read schemas
- Execute SQL (read-only by default)
- List available tables and views

4.4 Connecting the LLM to MCP

Note: vLLM exposes an OpenAI-compatible API but does not include a native MCP client. Use an MCP-capable front end (Claude Desktop, Open WebUI, or AnythingLLM) to connect the LLM to MCP servers.

Option A: Use Claude Desktop with MCP (Recommended)

Claude Desktop has built-in MCP support
Download from: https://claude.ai/download
Configure in claude_desktop_config.json:

{
  "mcpServers": {
    "postgres": {
      "command": "docker",
      "args": [
        "run", "--rm",
        "-e", "POSTGRES_CONNECTION_STRING=postgresql://postgres:password@host.docker.internal:5432/bgcki",
        "mcp/postgres"
      ]
    }
  }
}

Option B: Use AnythingLLM with MCP Bridge

See Part 5 for AnythingLLM setup
Connect AnythingLLM to the vLLM server
Use MCP bridge plugins

✓ Checkpoint: MCP Server is running and accessible

Part 5: AnythingLLM and Open WebUI (15-20 minutes)

Web interfaces for interacting with local LLMs.

5.1 AnythingLLM Setup

AnythingLLM is a full-featured AI document chat and assistant application.

Download and Install:
- Visit: https://anythingllm.com/
- Download the Windows installer
- Install with default settings
First Launch Configuration:
- Launch AnythingLLM
- Choose “OpenAI Compatible” as the LLM provider
- Enter connection details:
  - Base URL: http://127.0.0.1:8080/v1
  - API Key: (leave empty)
  - Model: unsloth/Qwen3.8-27B-NVFP4
Create a Workspace:
- Click “New Workspace”
- Name: “BlueGreenCity Development”
- Configure:
  - LLM: Your vLLM model
  - Temperature: 0.7 (adjust as needed)
  - Context length: 4096
Add Documents (Optional):
- Click “Upload Documents”
- Add project documentation, code files, or PDFs
- AnythingLLM will create embeddings for semantic search
Connect to Database (via MCP):
- Settings → Integrations
- Add MCP connection (if supported)
- Or use SQL query plugin

5.2 Open WebUI Setup

Open WebUI (formerly Ollama WebUI) is a self-hosted ChatGPT-style interface.

Install with Docker:

docker run -d -p 3000:3000 `
  --name open-webui `
  -v open-webui:/app/backend/data `
  ghcr.io/open-webui/open-webui:main

Note: Open WebUI’s internal port is 8080, which conflicts with the vLLM host port. Map it to host port 3000 instead.

Access the Interface:
- Open browser: http://localhost:3000
- Create admin account on first launch
Configure vLLM Connection:
- Click Settings (gear icon)
- Connections → Add Connection
- Name: “vLLM”
- API Base URL: http://host.docker.internal:8080/v1
- API Key: (leave empty)
- Save
Select Model:
- In chat interface, click model dropdown
- Select unsloth/Qwen3.8-27B-NVFP4
- Start chatting!
Advanced Features:
- Documents: Upload and chat with documents
- Functions: Add custom tools and integrations
- Prompt Library: Save and share prompts
- Multi-user: Create accounts for team members

5.3 Comparison: AnythingLLM vs Open WebUI

Feature	AnythingLLM	Open WebUI
Installation	Desktop app	Docker container
Document Chat	✅ Excellent	✅ Good
UI/UX	Clean, modern	ChatGPT-like
Multi-model	✅ Yes	✅ Yes
Customization	High	Very High
Plugins	Growing	Extensive
Best for	Document Q&A	General chat

Recommendation: Use both!

AnythingLLM for document-based work and knowledge base
Open WebUI for quick chats and experimentation

✓ Checkpoint: Web UI is running and connected to the vLLM server

Part 6: Usage Workflows

Workflow 1: Code Assistance with VSCode

Start the vLLM server (.\scripts\local-llm\start_vllm_server.ps1)
Connect via SSH to workstation in VSCode
Use GitHub Copilot Chat with the local vLLM model
Ask for code help, explanations, or refactoring

Example Prompts:

“Explain this PostGIS query: [paste query]”
“How do I calculate tree crown area from diameter?”
“Refactor this function to use GeoPandas instead of Shapely”

Workflow 2: Document Q&A with AnythingLLM

Upload project documentation to AnythingLLM
Ask questions about the project
Get answers grounded in your actual docs

Example:

“How is drought risk calculated in this project?”
“What database schema is used for tree inventory?”
“Summarize the climate data import process”

Workflow 3: Database Queries with MCP

Connect Claude Desktop or Open WebUI to MCP
Ask questions about the database
LLM generates and executes SQL queries

Example:

“How many trees are in the inventory?”
“Show me the schema of the trees.inventory table”
“What’s the average tree age by species?”

Workflow 4: Research and Exploration

Use Open WebUI for general questions
Test different models for quality comparison
Save useful prompts to prompt library

Part 7: Best Practices

Model Selection

For Code:

DeepSeek-Coder: Best code quality
CodeLlama: Good balance
Small models (7B): Fast iteration

For Chat/Docs:

Llama-3.1: General purpose
Mistral: Fast responses
Larger models (13B+): Better comprehension

Performance Optimization

Hardware:

16GB+ VRAM recommended for 13B models
32GB+ system RAM for large context windows
NVMe SSD for faster model loading

Configuration:

Use Q4_K_M quantization for balance
Start with 4096 context, increase if needed
Lower temperature (0.2-0.3) for code
Enable GPU layers (all layers if possible)

Security Considerations

Local LLMs:

✅ Data stays on your machine
✅ No internet required for inference
✅ Full control over model and data

API Security:

🔒 The vLLM API listens on 0.0.0.0:8080 (Docker port mapping)
🔒 Don’t expose port 8080 to the internet
🔒 Use Tailscale VPN for remote access
🔒 Optional: Enable an API key for additional security

Database Access:

🔒 MCP servers should be read-only by default
🔒 Never expose database credentials in prompts
🔒 Use environment variables for connection strings

Troubleshooting Common Issues

Issue: vLLM API not responding

✅ Check the container is running and healthy: docker ps --filter name=vllm-qwen38-27b
✅ Check the logs: docker logs vllm-qwen38-27b 2>&1 | Select-Object -Last 30
✅ Test with: curl http://127.0.0.1:8080/v1/models
✅ Restart: .\scripts\local-llm\start_vllm_server.ps1

Issue: VSCode can’t connect to local model

✅ Verify SSH session is active
✅ Check endpoint includes /v1 and uses port 8080
✅ Restart VSCode window

Issue: Slow inference

✅ Check the generation throughput in the logs (target: 100+ tokens/s)
✅ Reduce the number of parallel requests (KV-cache pressure)
✅ Lower reasoning_effort per request (low or enable_thinking: false)
✅ Close other GPU-intensive applications

Issue: Out of memory / KV-cache exhaustion

✅ Reduce --max-num-seqs (biggest lever for KV-pool size)
✅ Lower --gpu-memory-utilization (currently 0.9) to shrink the weights+KV budget
✅ Check GPU KV cache usage in the logs (should stay below ~80%)

Part 8: Resources and Further Reading

Official Documentation

vLLM:

Website: https://vllm.ai/
Docs: https://docs.vllm.ai/
Recipes: https://recipes.vllm.ai/
GitHub: https://github.com/vllm-project/vllm

Qwen3.8-27B (Unsloth NVFP4):

Model card: https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4
vLLM recipe: https://recipes.vllm.ai/Qwen/Qwen3.8-27B

Model Context Protocol (MCP):

Specification: https://modelcontextprotocol.io/
GitHub: https://github.com/anthropics/mcp
Servers: https://github.com/anthropics/mcp-servers

AnythingLLM:

Website: https://anythingllm.com/
Docs: https://docs.anythingllm.com/
GitHub: https://github.com/Mintplex-Labs/anything-llm

Open WebUI:

Website: https://openwebui.com/
Docs: https://docs.openwebui.com/
GitHub: https://github.com/open-webui/open-webui

Community Resources

Forums and Communities:

r/LocalLLaMA: https://reddit.com/r/LocalLLaMA
r/LocalLLaMA Discord: https://discord.gg/localllama
Hugging Face Forums: https://discuss.huggingface.co/

Model Leaderboards:

Chatbot Arena: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard
Open LLM Leaderboard: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

Learning Resources:

“Prompt Engineering Guide”: https://www.promptingguide.ai/
“LLM Visualization”: https://bbycroft.net/llm
“Understanding LLMs”: https://www.understandingai.org/

Recommended Models by Use Case

Code Generation:

GPT-OSS-20B - High-quality open-source code generation
GPT-OSS-120B - Enterprise-grade code generation (requires significant VRAM)
DeepSeek-Coder-V2-Lite-Instruct (16B)
CodeLlama-13B-Instruct
Phind-CodeLlama-34B-v2

General Chat:

Llama-3.1-8B-Instruct
Mistral-7B-Instruct-v0.3
Mixtral-8x7B-Instruct

Document Q&A:

Llama-3.1-8B-Instruct
Mistral-7B-Instruct-v0.3
Zephyr-7B-Beta

Small/Fast (< 7B):

Phi-3-Mini-Instruct (3.8B)
TinyLlama-1.1B
Gemma-2B-Instruct

Experimental:

Apertus - Cutting-edge experimental model (may be unstable)

Appendix A: Quick Reference

vLLM

Server URL: http://127.0.0.1:8080/v1
Metrics: http://127.0.0.1:8080/metrics
Compose file: scripts/local-llm/docker-compose.vllm.yml
Start script: scripts/local-llm/start_vllm_server.ps1
Env file: scripts/local-llm/.env.vllm
Logs: logs/vllm/vllm_start_YYYYMMDD_HHMMSS.log
Model: unsloth/Qwen3.8-27B-NVFP4

VSCode Copilot Configuration

{
  "github.copilot.chat.localhostChatModels": [
    {
      "model": "vllm",
      "endpoint": "http://127.0.0.1:8080/v1/chat/completions",
      "name": "vLLM Qwen3.8-27B (Local)"
    }
  ]
}

Docker Commands

# Start vLLM
.\scripts\local-llm\start_vllm_server.ps1

# Stop vLLM
docker rm -f vllm-qwen38-27b

# View vLLM logs
docker logs -f vllm-qwen38-27b

# Start Open WebUI (host port 3000 to avoid vLLM's 8080)
docker run -d -p 3000:3000 --name open-webui ghcr.io/open-webui/open-webui:main

Useful Ports

vLLM API: 8080
Open WebUI: 3000
PostGIS: 5432
(legacy) LM Studio API: 1234
(legacy) Ollama API: 11434

Appendix B: Alternative Setup - Ollama (legacy)

Note: Ollama is superseded by vLLM (Part 1). This appendix is kept for reference only.

For a more lightweight, CLI-focused approach, consider Ollama:

Installation

# Windows (native)
winget install Ollama.Ollama

# Or download from: https://ollama.com/download

Basic Usage

# Run a model
ollama run codellama

# List models
ollama list

# Pull a model
ollama pull mistral

# Serve API
ollama serve  # Runs on port 11434

Connect to VSCode

{
  "github.copilot.chat.localhostChatModels": [
    {
      "model": "ollama",
      "endpoint": "http://127.0.0.1:11434/v1/chat/completions",
      "name": "Ollama (Local)"
    }
  ]
}

When to use Ollama:

Prefer CLI over GUI
Want automatic model management
Need scriptable LLM workflows
Docker-ready deployment

Support and Questions

For questions or issues:

Check the troubleshooting section above
Consult official documentation links
Ask in the project’s internal communication channel
Refer to Workstation_Remote_Development.md for SSH setup issues

Last updated: August 2026