The inference host
A 2017 Intel iMac Pro with a 10-core Xeon W, 128 GB RAM, and an AMD Radeon Pro Vega 64 with 16 GB VRAM, running macOS Sequoia.
The owner reports a locally tested stack using llama.cpp, Qwen3-Coder-30B-A3B-Instruct, an OpenAI-compatible API, and Open WebUI.
Component responsibilities
- Open WebUI: the chat interface, application accounts, and conversation history.
- llama.cpp: model loading and inference behind an OpenAI-compatible API.
- Qwen: the installed coding model.
- Cloudflare Access and Tunnel: the planned route for approved remote users.
Boundaries that matter
The public portfolio is deployed independently. It contains no inference credentials and makes no model requests.
The tunnel should expose the intended application only. When inference and the interface run on the same machine, the backend can remain bound to loopback. Account isolation, registration settings, roles, and logout must be verified against installed software versions.
Hardware constraints
CPU-only and Metal GPU inference have both been tested. The owner has not seen reliable performance improvement from GPU acceleration on the Vega 64. No numeric benchmark results are published.
Operational work still ahead
Record software versions, model quantization, configured context, startup behavior, backup and restore procedures, and repeatable capacity measurements. Publish sanitized findings rather than private operational configuration.
Read the project