Writing
AI Infrastructure·18 April 2026
Running Language Models With Limited GPU Resources
Lessons from experimenting with quantisation, QLoRA and smaller language models on consumer hardware.
Useful AI systems should not assume unlimited compute.
Much of my work involves making speech and language models practical on constrained GPU hardware — the kind of environment many African engineering teams actually have.
Practical levers
Techniques that matter in practice:
- Quantisation — reduce memory footprint with controlled precision trade-offs
- QLoRA — efficient fine-tuning without full-parameter updates
- Smaller base models — choose capacity that fits the task
- Batching and caching — improve throughput for real workloads
- Local inference first — reduce dependency on unreliable connectivity
Rendering diagram…
Engineering, not magic
Resource-constrained AI is mostly systems engineering:
- know the latency budget
- measure VRAM carefully
- prefer predictable failure modes
- keep humans in the loop for clinical or high-stakes outputs
The goal is not the largest model. The goal is a reliable system that fits the environment where it must run.