A practical guide for engineering leads and developers who want large language models (LLMs) running on the developer's own laptop, on Linux, rather than on a shared server or a cloud GPU. It covers what fits in 16 GB and 24 GB of laptop GPU memory, the arithmetic behind that, the open models worth running in November 2026, setting up Ubuntu 26.04 with the NVIDIA stack, running models with Ollama, llama.cpp and vLLM, plugging them into coding tools, fine-tuning with LoRA, power and heat, security, and when a job should move to a server or the cloud.
This technote accompanies our blog post AI developer laptops: why every AI team should give developers their own GPU, which makes the business case. The example machine is the Juno Neptune 18", an 18-inch Linux laptop we sell. Model sizes are from the Ollama library and GPU figures from NVIDIA, checked on 18 November 2026. Speeds depend on the model, quantisation, context and power settings, so we explain how to measure rather than quote benchmark numbers.
1. Why put the GPU on the developer's desk
Most AI development is an inner loop: change a prompt, a retrieval step, a tool definition or a few lines of code, run it, read the output, repeat. That loop runs hundreds of times a day. When each run waits for a shared GPU, a cloud instance to start or an API with rate limits, developers batch changes and learn more slowly. A local model answers in seconds, works offline and costs nothing per call.
- Privacy. Source code, customer data and prompts never leave a company-managed device. That simplifies UK GDPR assessments and client contracts that forbid sending data to third-party APIs.
- Realism. Developers see the real limits of open models (context length, quantisation, memory) before production does.
- Parity. Linux, the NVIDIA driver, CUDA and containers are the same as on your GPU servers, so a prototype moves to a server without changes.
- Availability. No queue, no quota, no outage dependency during development.
A laptop does not replace servers. Training, large fine-tunes, long evaluation runs and production serving belong on a shared GPU server or a cluster (section 11).
2. The machine
The figures that matter for AI work, for the Neptune 18" as configured in our shop. GPU figures are NVIDIA's; the rest are the manufacturer's.
| Part | Specification | Why it matters |
|---|---|---|
| GPU (standard) | NVIDIA GeForce RTX 5080 Laptop GPU: 16 GB GDDR7, 256-bit, 7,680 CUDA cores, 1,334 AI TOPS | Holds 8–20B models and their KV cache on the GPU |
| GPU (on request) | NVIDIA GeForce RTX 5090 Laptop GPU: 24 GB GDDR7, 256-bit, 10,496 CUDA cores, 1,824 AI TOPS | Adds the 27–32B class and longer context |
| GPU power | 150 W + 25 W Dynamic Boost in this chassis (NVIDIA's ranges: 80–150 W for the RTX 5080, 95–150 W for the RTX 5090 Laptop GPU) | Sustained speed depends on power; run long jobs plugged in |
| CPU | Intel Core Ultra 9 290HX Plus, 24 cores (8P + 16E), up to 5.5 GHz; 55 W base, 160 W max | Tokenising, data loading and CPU offload of model layers |
| Memory | 32 to 192 GB DDR5 (2 modules at 5600 MHz up to 96 GB; 4 modules at 4000 MHz for 128 and 192 GB) | Room for models bigger than GPU memory (section 4) |
| Storage | 1 × PCIe Gen 5 + 3 × PCIe Gen 4 M.2 2280, up to 16 TB | Model files are 5–140 GB each; keep a library locally |
| Display | 18" 16:10, QHD+ 240 Hz (RTX 5080) or UHD+ 200 Hz (RTX 5090), 100% DCI-P3 | Room for code, terminal and notebook side by side |
| Ports | 2 × Thunderbolt 4 / USB-C, 2 × USB-A, HDMI and Mini DisplayPort (wired to the NVIDIA GPU), 2.5 Gb Ethernet, Wi-Fi 7 | Fast transfers of models and datasets |
| Power | 99 Wh battery, 330 W charger | GPU work is a plugged-in activity |
| Software | Kubuntu or Ubuntu 26.04 LTS pre-installed | The same OS family as most GPU servers |
3. What fits: GPU memory decides
When a model's weights, its KV cache and the runtime all fit in GPU memory, generation is fast. When they don't, part of the model runs from system memory over PCIe and from the CPU, and speed drops sharply. So the first question for any model is its size.
| Model (Ollama tag) | Download | RTX 5080 (16 GB) | RTX 5090 (24 GB) | Good for |
|---|---|---|---|---|
| qwen3:8b | 5.2 GB | On the GPU, long context | On the GPU | Fast general assistant, agents, RAG |
| qwen3:14b | 9.3 GB | On the GPU | On the GPU | Stronger reasoning at good speed |
| gpt-oss:20b (OpenAI, MoE) | 14 GB | On the GPU, short to medium context | On the GPU, longer context | Reasoning, tool use, general coding |
| gemma3:27b (Google) | 17 GB | Partly offloaded | On the GPU | Writing, multilingual, images as input |
| qwen3.8:27b | 18 GB | Partly offloaded | On the GPU | Coding, research and agent tasks, text and images |
| qwen3-coder:30b (MoE, about 3B active) | 19 GB | Experts offloaded, still usable | On the GPU | Code completion and coding agents |
| nemotron-3.5-lightning:30b (NVIDIA, MoE, 3B active) | 19–25 GB | Experts offloaded | Smaller quantisation on the GPU | Always-on agents, up to 1M context |
| qwen3:32b | 20 GB | Partly offloaded | On the GPU, short context | Dense 32B quality |
| llama3.3:70b (Meta) | 43 GB | GPU + system RAM (64 GB or more): slow, fine for checks | Comparing with a server-class model | |
| gpt-oss:120b (OpenAI, MoE, 5.1B active) | 65 GB | GPU + system RAM (96 GB or more): usable thanks to MoE | Near-frontier open reasoning on a laptop | |
| qwen3:235b (MoE) | 142 GB | GPU + 192 GB RAM: experimental | Trying a very large model before buying a server | |
Sizes are the default Ollama tags, mostly 4-bit quantisations. Licences differ (Apache 2.0, the Gemma terms, Meta's Llama licence, NVIDIA's open model licence and others): check each model card before using a model commercially.
4. The memory arithmetic
Three things share GPU memory: the weights, the KV cache and the runtime.
Weights
weights ≈ parameters × bits per weight ÷ 8. A 14B model is 28 GB at 16-bit, about 14–15 GB at 8-bit and about 8–9 GB at 4-bit (4-bit formats store a little extra per block, so real files are slightly larger than half a byte per parameter). Quality loss at a good 4-bit quantisation is small for most development work; 8-bit is close to the original. Blackwell GPUs, including the RTX 50 Series, also support FP4 and FP8 in hardware, which runtimes such as TensorRT-LLM and vLLM can use.
KV cache
For every token in the context, each layer stores a key and a value: KV bytes per token = 2 × layers × KV heads × head size × bytes per value. For Llama 3.1 8B that is 2 × 32 × 8 × 128 × 2 = 131,072 bytes (128 KiB) at 16-bit. It grows linearly with context length and with the number of requests served at once.
So an 8B model that is 5 GB on disk needs about 23 GB at its full 128K context. Three ways to fit:
- Use the context you need. Most coding and RAG work is fine at 16–32K. Runtimes reserve the cache for the context you set, so set it.
- Quantise the cache. An 8-bit KV cache halves it with little quality loss; 4-bit quarters it. In Ollama set
OLLAMA_FLASH_ATTENTION=1andOLLAMA_KV_CACHE_TYPE=q8_0; in llama.cpp use--cache-type-k q8_0 --cache-type-v q8_0. - Prefer models with grouped-query or other compact attention. Fewer KV heads mean a smaller cache; the model card lists them.
Runtime
Allow 1–2 GB for the CUDA context, activations and buffers, and remember the desktop uses some GPU memory too when displays run from the NVIDIA GPU. Check with nvidia-smi before loading a model.
Speed, and why system memory still matters
Generating each token reads every active weight once, so generation speed is limited mainly by memory bandwidth: GDDR7 on the GPU is several times faster than DDR5 in the laptop. That is why a model that spills into system memory slows down, and why mixture-of-experts (MoE) models are the exception: gpt-oss-120b has 117B parameters but only about 5.1B are active for each token, so keeping the shared layers on the GPU and the experts in system memory still gives usable speed. For that kind of work, more and faster system memory helps: two modules at 5600 MHz are faster than four at 4000 MHz, so choose 96 GB (2 × 48 GB) unless you need 128 or 192 GB for capacity.
Measure on your own work: ollama run <model> --verbose prints prompt-processing and generation rates in tokens per second, and llama-bench does the same for llama.cpp.
5. Setting up Ubuntu 26.04 for AI work
Machines from us arrive with the NVIDIA driver installed and checked. To check or rebuild one:
# Is the NVIDIA GPU visible, and which driver is in use?
nvidia-smi
# Ubuntu's recommended NVIDIA driver (signed for Secure Boot)
sudo ubuntu-drivers list
sudo ubuntu-drivers install
sudo reboot
# Laptops with two GPUs: run everything on the NVIDIA GPU, or only what asks for it
prime-select query
sudo prime-select nvidia # or: on-demand
With Secure Boot on, a driver built locally needs its signing key enrolled (MOK) at the next boot; Ubuntu's packaged drivers avoid that step. Keep the driver from Ubuntu's archive or NVIDIA's repository, not both.
Containers with the GPU
Containers keep CUDA versions per project and match what runs on servers. Install Docker (or Podman), then the NVIDIA Container Toolkit from NVIDIA's apt repository following NVIDIA's install guide, and wire it in:
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
# Test: the container should list the laptop GPU
docker run --rm --gpus all ubuntu nvidia-smi
Python
Use one virtual environment per project (uv or venv) and install PyTorch with the CUDA build its install page recommends for your driver. Check it with python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))".
6. Running models
Ollama: the quickest start
curl -fsSL https://ollama.com/install.sh | sh
ollama run gpt-oss:20b # downloads once, then runs locally
ollama ps # shows how much of the model is on the GPU vs the CPU
ollama ps is the first check: "100% GPU" is what you want. If it shows a CPU share, reduce the context, quantise the KV cache or pick a smaller model. Ollama listens on 127.0.0.1:11434 and serves an OpenAI-compatible API at /v1, so most tools can use it unchanged.
llama.cpp: full control
# All layers on the GPU, 32K context, 8-bit KV cache, OpenAI-compatible server on :8080
llama-server -m qwen3-14b-q4_k_m.gguf --n-gpu-layers 99 --ctx-size 32768 \
--cache-type-k q8_0 --cache-type-v q8_0 --host 127.0.0.1 --port 8080
# A MoE model bigger than GPU memory: keep shared layers on the GPU, put experts in system RAM
# (recent llama.cpp builds; raise the number until it fits)
llama-server -m gpt-oss-120b.gguf --n-gpu-layers 99 --n-cpu-moe 24 --ctx-size 16384
vLLM: closest to production serving
pip install vllm
vllm serve Qwen/Qwen3-8B --max-model-len 16384 --gpu-memory-utilization 0.85
# OpenAI-compatible API on http://127.0.0.1:8000/v1
vLLM pre-allocates GPU memory for its KV cache, so on a 16 GB laptop GPU keep models small and set --max-model-len. Its value is parity: the same server, flags and API as on your GPU servers.
LM Studio is a desktop app for the same job, with a model browser and a local API, useful for people who don't live in a terminal.
7. Coding assistants and agents on a local model
Most IDE assistants, coding agents and agent frameworks accept an OpenAI-compatible base URL. Point them at http://127.0.0.1:11434/v1 (Ollama), :8080/v1 (llama.cpp) or :8000/v1 (vLLM). Practical choices in November 2026:
- Completion while typing: a small, fast model (qwen3:8b or similar). Latency matters more than size.
- Chat and refactoring: gpt-oss:20b on 16 GB; qwen3-coder:30b or qwen3.8:27b on 24 GB.
- Agents with tools: models trained for tool calling (gpt-oss, Qwen3 family, Nemotron 3.5 Lightning), with enough context for the tool results, typically 32K or more.
Keep a hosted frontier model for the hardest problems if your policy allows it; the local model handles the everyday volume privately. Many teams route by task: local by default, hosted on request.
8. Fine-tuning on a laptop
Full fine-tuning of even a 7B model needs far more than 24 GB. LoRA and QLoRA train small adapter matrices on top of frozen weights (QLoRA keeps the base model in 4-bit), which brings fine-tuning onto a laptop:
- 16 GB (RTX 5080): QLoRA on 7–8B models at modest sequence lengths; up to about 14B with memory-saving tooling and short sequences.
- 24 GB (RTX 5090): QLoRA on 14B comfortably, and up to about 30B with short sequences.
- Beyond that: 70B fine-tunes and full fine-tunes go to a server (section 11).
Use Hugging Face PEFT and TRL, or Unsloth for lower memory use, with gradient checkpointing and a batch size of one or two plus gradient accumulation. These ranges depend on sequence length, rank and tooling: start small, watch nvidia-smi and grow. The laptop is ideal for checking data formats, hyperparameters and evaluation scripts on a small run before paying for a long one.
9. Power, heat and battery
- Plug in for GPU work. On battery the GPU runs at reduced power; the 330 W charger is what sustains 150 W+ GPU loads.
- Watch it:
nvidia-smi dmon -s pucmstreams power, utilisation, clocks and memory;nvidia-smi -q -d POWER,TEMPERATUREshows limits. - Long jobs: keep vents clear, use the performance power profile, and expect fans. For multi-hour jobs a desktop or server is the better tool.
10. Security and governance
- Encrypt the disk. Full-disk encryption (LUKS) protects code, data and model files if the laptop is lost; we set it up before dispatch when ordered.
- Keep model servers local. Ollama, llama.cpp and vLLM have no user authentication by default. Bind them to
127.0.0.1; never expose them on0.0.0.0on office or public Wi-Fi without a proxy that authenticates. - Know where models come from. Download from the publisher's official repositories, prefer safetensors and GGUF over formats that can run code when loaded, and keep a list of approved models and their licences.
- Manage the device. Updates, disk encryption and screen locks under your normal device management; data processed locally is still personal data under UK GDPR.
11. When to move work off the laptop
| Signal | Move to |
|---|---|
| The model needs more than 24 GB at full speed, or long context for many users | A workstation or server with RTX PRO 6000 (96 GB) or a DGX system |
| A job runs for hours, or blocks the developer's machine | A shared GPU server, scheduled |
| Several developers need the same model all day | One team inference endpoint instead of many copies |
| Multi-GPU training or production serving | Cluster or cloud, with UK data residency where needed |
Not ready to buy? Workstation Rentals rents AI laptops and GPU workstations by the week or month, and GPU Leasing explains leasing them.
12. Which configuration for which developer
| Role | GPU | System memory | Storage |
|---|---|---|---|
| Application developer using AI (RAG, agents, assistants) | RTX 5080, 16 GB | 64 GB | 2 TB |
| AI engineer (evaluation, fine-tuning, 27–32B models) | RTX 5090, 24 GB | 96 GB (2 × 48 GB, fastest) | 4 TB |
| Researcher trying very large MoE models locally | RTX 5090, 24 GB | 128–192 GB | 4 TB or more |
| Data scientist (notebooks, classic ML, small models) | RTX 5080, 16 GB | 64–96 GB | 2–4 TB |
13. Frequently asked questions
Can a laptop really run a 120B model? gpt-oss-120b, yes, with 96 GB or more of system memory, because only about 5.1B parameters are active per token. Dense 70B models also run, but slowly; use them to compare answers, not for interactive work.
16 GB or 24 GB? 16 GB covers the 8–20B class, which handles most development work. Choose 24 GB if you work with 27–32B models, long contexts or fine-tuning above 8B.
Why Linux rather than Windows? Servers, containers and most AI tooling are Linux-first. Developing on the same OS removes a class of "works on my machine" problems.
Does a local model replace hosted frontier models? Not for the hardest tasks. It takes the everyday volume privately and cheaply; keep a hosted model for what needs it, where policy allows.
What about Apple silicon? Macs with large unified memory run big models well with MLX, but without CUDA. If your servers are NVIDIA and Linux, an NVIDIA Linux laptop keeps the stack identical.
Sources
- NVIDIA, GeForce RTX 50 Series laptop specifications (CUDA cores, AI TOPS, memory, interface width, GPU power range), checked 18 November 2026.
- Ollama library pages for qwen3, gpt-oss, gemma3, qwen3.8, qwen3-coder, nemotron-3.5-lightning and llama3.3 (download sizes and descriptions), checked 18 November 2026.
- OpenAI, gpt-oss model card (117B total and about 5.1B active parameters for gpt-oss-120b).
- Meta, Llama 3.1 8B configuration (32 layers, 8 KV heads, head size 128), used for the KV cache example.
- Juno Computers, Neptune 18" specifications; Workstation shop listing (configuration, price and dispatch time).
- Commands follow the Ubuntu, NVIDIA Container Toolkit, Ollama, llama.cpp and vLLM documentation; flags change between releases, so check the current docs.
Acknowledgement of rights: NVIDIA, GeForce, RTX, CUDA, cuDNN, NCCL, TensorRT, DGX, Blackwell and Nemotron are trademarks of NVIDIA Corporation. Intel and Core Ultra are trademarks of Intel Corporation. Ubuntu and Kubuntu are trademarks of Canonical Ltd. Juno and Neptune are names of Juno Computers Ltd. Llama is a trademark of Meta Platforms, Inc.; Gemma of Google LLC; Qwen of Alibaba Cloud; gpt-oss and OpenAI of OpenAI. Other names are the property of their respective owners. This article is independent and not endorsed by any of these companies.