Local AI in GitHub Copilot: plenty of RAM, still out of memory?
Running MAI Code 1.1 Flash, a 52 GB model, locally inside the GitHub Copilot desktop app failed even with four Tesla T4 GPUs and hundreds of GB of system RAM. The author had to install llama.cpp, point LLAMACPP_SERVER_PATH at llama-server, and cut the model's 262,145-token context down to 32,768 with a single parallel slot before it could answer.
Key points
- The GitHub Copilot desktop app needs llama.cpp installed and LLAMACPP_SERVER_PATH pointing at llama-server before any local model can start on Windows.
- Under Settings > Model providers > Windows ML, MAI Code 1.1 Flash was set to its full 262,145-token context, which could not fit alongside the weights and KV cache in 16 GB of VRAM per card.
- Setting parallel slots to 1 and context size to 32,768, or 16,384 if memory is still tight, let the model load and answer questions.
- Hundreds of gigabytes of system RAM do not translate into GPU memory, so the context window is the first thing to check on out-of-memory errors.
- The first answer took 95 seconds, which proves the setup works but says nothing about throughput.
A 52 GB model, four 16 GB Tesla T4 cards and hundreds of gigabytes of system RAM looked like an easy setup, but the GitHub Copilot desktop app still failed to load MAI Code 1.1 Flash. The app could not find llama-server, so it needed llama.cpp, an updated Microsoft Visual C++ x64 runtime and the LLAMACPP_SERVER_PATH variable set to the executable.
The real blocker was the context window: under Settings > Model providers > Windows ML the model used its full 262,145-token context, while weights, KV cache and compute buffers all have to fit in VRAM. Setting parallel slots to 1 and context size to 32,768 (16,384 if still tight) got it running. System RAM never becomes VRAM, and a 95-second first reply makes this a working setup, not a speed benchmark.
Real-world use
- A developer on a workstation with one or two 16 GB GPUs trying to run a large local code model inside the Copilot desktop app.
- A team that bought system RAM in bulk and expects it to make GPU-only model loading faster or possible.
- Troubleshooting a llama.cpp backend that the app cannot find after installation on Windows.
- Tuning context size and parallel slots so a local model still handles long files without blowing up VRAM.
Why it matters
Local model failures in Copilot are usually a configuration and VRAM-budget problem, not a hardware shortage, so a few settings save a costly GPU upgrade. Knowing that the context window drives memory use turns a dead end into a working local assistant in minutes.