Blog·
How to run LLMs locally on a Mac: Ollama, llama.cpp, MLX
How to run an LLM locally on a Mac: pick a model that fits unified memory, choose Ollama, llama.cpp, LM Studio or MLX, and call it from code. With a fit calculator.
Tags
4 posts tagged
How to run an LLM locally on a Mac: pick a model that fits unified memory, choose Ollama, llama.cpp, LM Studio or MLX, and call it from code. With a fit calculator.
How much VRAM an LLM needs: parameters times bytes per weight, plus the KV cache for your context. Worked numbers for 8B to 405B models, in a Python script.
Small language models are LLMs of about 1B to 14B parameters that run on a phone or laptop. How they are made, how much memory they need, and when to use one.
LoRA fine-tunes an LLM by training two small low-rank matrices per layer while the model stays frozen. QLoRA does it on a 4-bit base. How both work, in Python.