The Llama 3 article closing line summarises local LLM work as “run the model, turn the dials, and keep the transcript.” This workbook does exactly that, ten times. You will load a quantized model, extract completions, tune temperature and the other sampling dials, use chat roles and system prompts, teach the model with few-shot examples, force valid JSON, build conversation memory, and finish with a small assistant that combines all of it. Because everything runs locally, your data never leaves your machine and every token is free.
1. Setup: install, download, load
One-time setup: install llama-cpp-python, pull a quantized GGUF file from Hugging Face, and load it. GGUF is the compressed weight format that lets an 8-billion-parameter model fit consumer hardware; Q4_K_M is the quantization level with the best size-to-quality trade-off.
# once, in your terminal:# pip install llama-cpp-python huggingface_hubfrom huggingface_hub import hf_hub_downloadfrom llama_cpp import Llama# downloads ~5GB once, then loads from local cache forever aftermodel_path = hf_hub_download( repo_id="QuantFactory/Meta-Llama-3-8B-Instruct-GGUF", filename="Meta-Llama-3-8B-Instruct.Q4_K_M.gguf")llm = Llama(model_path=model_path, n_ctx=2048, verbose=False)print("Model loaded:", model_path)
n_ctx=2048 sets the context window, how much text the model can consider at once, and verbose=False silences the loader’s diagnostics. Every later example begins with these same lines; the download only happens the first time.
2. Your first completion
Call the model like a function: prompt in, dictionary out. The generated text lives at choices[0]["text"], a response shape deliberately modelled on the OpenAI API.
from llama_cpp import Llamallm = Llama(model_path=model_path, n_ctx=2048, verbose=False)response = llm( "The three most important habits for a data analyst are", max_tokens=80)print(response["choices"][0]["text"])print("---")print("Tokens used:", response["usage"]["total_tokens"])
Two things to notice. First, always set max_tokens, since forgetting it is the classic pitfall that produces truncated or runaway output. Second, the response carries a usage block, which is how you track how much of the context window a call consumed.
3. Temperature: the creativity dial
Temperature reshapes the probability distribution the model samples from. Low values make it pick the safest token almost every time; high values flatten the odds and invite variety. The same prompt at two settings shows the difference.
from llama_cpp import Llamallm = Llama(model_path=model_path, n_ctx=2048, verbose=False)prompt = "Write a one-sentence description of a coffee shop."factual = llm(prompt, max_tokens=50, temperature=0.1)creative = llm(prompt, max_tokens=50, temperature=1.0)print("temp 0.1:", factual["choices"][0]["text"].strip())print("temp 1.0:", creative["choices"][0]["text"].strip())
Run it twice more: the low-temperature line barely changes between runs, while the high-temperature one reinvents itself each time. The working ranges from the guide: 0.1 to 0.3 for factual tasks like extraction and support answers, 0.8 and up for creative work like marketing copy.
4. The other dials: top_k, top_p, max_tokens, stop
Four more parameters complete the sampling toolkit. top_k caps how many candidate tokens are considered, top_p keeps the smallest set of tokens whose probabilities sum to p, and stop halts generation the moment a given string appears.
from llama_cpp import Llamallm = Llama(model_path=model_path, n_ctx=2048, verbose=False)response = llm( "List planets: Mercury, Venus,", max_tokens=60, # hard cap on output length temperature=0.7, top_k=40, # only the 40 likeliest tokens are candidates top_p=0.9, # nucleus sampling: adaptive shortlist stop=["\n\n"] # stop at the first blank line)print(response["choices"][0]["text"])
top_p is the subtler of the pair: when the model is confident the shortlist is tiny, and when it is uncertain the list widens, which is why nucleus sampling is usually left on. stop is the practical star, since ending generation at a delimiter is how you keep answers from rambling past their natural end.
5. Chat completions: roles instead of raw text
Instruct models like this one are trained on conversations, so the chat interface fits them better than raw prompts. Messages carry roles, and the reply comes back at a different path: choices[0]["message"]["content"].
from llama_cpp import Llamallm = Llama(model_path=model_path, n_ctx=2048, verbose=False)response = llm.create_chat_completion( messages=[ {"role": "system", "content": "You are a concise assistant."}, {"role": "user", "content": "What does GGUF mean in one sentence?"} ], max_tokens=60)print(response["choices"][0]["message"]["content"])
Note the response-shape difference against Example 2: plain completions return choices[0]["text"], chat completions return choices[0]["message"]["content"]. Mixing those two paths up is one of the most common bugs in local LLM code, so it is worth committing both to memory now.
6. System prompts: steering behaviour
The system message is the backstage director whispering to the actor: instructions the model treats as its standing orders. The same question under two different system prompts produces two different assistants.
from llama_cpp import Llamallm = Llama(model_path=model_path, n_ctx=2048, verbose=False)question = {"role": "user", "content": "Should I use a spreadsheet or a database?"}for persona in [ "You are a blunt engineer. Answer in one short sentence.", "You are a patient teacher. Explain the trade-offs simply for a beginner."]: response = llm.create_chat_completion( messages=[{"role": "system", "content": persona}, question], max_tokens=120, temperature=0.4 ) print(f"--- {persona[:25]}...") print(response["choices"][0]["message"]["content"].strip(), "\n")
Same model, same question, two personalities. Tone, length, format rules, and guardrails all belong in the system prompt, and because you run the model locally, nobody else’s system prompt sits above yours.
7. Few-shot prompting: teaching by example
Showing the model two or three worked examples inside the prompt teaches it a task with no training at all. Here it learns a sentiment-labelling format and applies it to a new review.
from llama_cpp import Llamallm = Llama(model_path=model_path, n_ctx=2048, verbose=False)prompt = """Classify the sentiment as positive, negative, or neutral.Review: The delivery was fast and the keyboard feels great.Sentiment: positiveReview: Box arrived crushed and two weeks late.Sentiment: negativeReview: The parcel arrived on the day stated on the order.Sentiment: neutralReview: Fantastic screen, but the stand wobbles constantly.Sentiment:"""response = llm(prompt, max_tokens=5, temperature=0.1, stop=["\n"])print(response["choices"][0]["text"].strip())
The three examples establish the pattern, the labels, and the format, and the model completes the fourth case in kind. Few-shot plus low temperature plus a stop sequence is the standard recipe for cheap, reliable classification without any fine-tuning.
8. Structured output: forcing valid JSON
Asking nicely for JSON in the prompt fails often enough to break pipelines. response_format with a schema uses constrained decoding, meaning the model is only allowed to generate tokens that keep the JSON valid.
import jsonfrom llama_cpp import Llamallm = Llama(model_path=model_path, n_ctx=2048, verbose=False)response = llm.create_chat_completion( messages=[ {"role": "system", "content": "Extract the order details."}, {"role": "user", "content": "Hi, I'm Priya Shah, I'd like 3 of the " "steel water bottles sent to Manchester please."} ], response_format={ "type": "json_object", "schema": { "type": "object", "properties": { "customer": {"type": "string"}, "product": {"type": "string"}, "quantity": {"type": "integer"}, "city": {"type": "string"} }, "required": ["customer", "product", "quantity", "city"] } }, max_tokens=100, temperature=0.1)order = json.loads(response["choices"][0]["message"]["content"])print(order["customer"], "|", order["quantity"], "x", order["product"], "->", order["city"])
json.loads succeeds every time because the decoder was never permitted to emit invalid JSON, and the schema pins the exact field names and types your downstream code expects. This is the difference between a demo and a pipeline: prompt-only JSON requests are the pitfall, constrained decoding is the fix.
9. Memory: the ChatSession wrapper
The model is stateless: it remembers nothing between calls, and every call sees only what you send. Conversation memory is therefore your job, done by keeping the transcript and re-sending it every turn.
from llama_cpp import Llamallm = Llama(model_path=model_path, n_ctx=2048, verbose=False)class ChatSession: def __init__(self, llm, system_prompt): self.llm = llm self.messages = [{"role": "system", "content": system_prompt}] def ask(self, text): self.messages.append({"role": "user", "content": text}) response = self.llm.create_chat_completion( messages=self.messages, max_tokens=150, temperature=0.4) reply = response["choices"][0]["message"]["content"] self.messages.append({"role": "assistant", "content": reply}) return replychat = ChatSession(llm, "You are a helpful, brief assistant.")print(chat.ask("My name is Andrei and I run a data blog."))print(chat.ask("What did I say my name was?")) # it knows, because we kept the transcript
The second answer proves the memory works: the model recalls the name only because the full transcript, including its own earlier reply, travelled with the second call. Losing history, appending the user message but forgetting the assistant’s reply, is the pitfall this little class exists to prevent.
10. Putting it together: a local support assistant
The finale combines every dial and pattern: a system prompt for persona and guardrails, memory for multi-turn context, low temperature for reliability, and a JSON summary extracted at the end of the conversation.
import jsonfrom llama_cpp import Llamallm = Llama(model_path=model_path, n_ctx=2048, verbose=False)class SupportBot: def __init__(self, llm): self.llm = llm self.messages = [{"role": "system", "content": "You are a support agent for a keyboard shop. Be brief and " "practical. If a refund is requested, ask for the order number."}] def ask(self, text): self.messages.append({"role": "user", "content": text}) r = self.llm.create_chat_completion( messages=self.messages, max_tokens=120, temperature=0.2) reply = r["choices"][0]["message"]["content"] self.messages.append({"role": "assistant", "content": reply}) return reply def summarise_ticket(self): r = self.llm.create_chat_completion( messages=self.messages + [{"role": "user", "content": "Summarise this conversation as a ticket."}], response_format={"type": "json_object", "schema": { "type": "object", "properties": { "issue": {"type": "string"}, "refund_requested": {"type": "boolean"}, "next_action": {"type": "string"}}, "required": ["issue", "refund_requested", "next_action"]}}, max_tokens=120, temperature=0.1) return json.loads(r["choices"][0]["message"]["content"])bot = SupportBot(llm)print(bot.ask("My new keyboard has a stuck spacebar."))print(bot.ask("I'd rather just get a refund, order KB-2291."))print(bot.summarise_ticket())# {'issue': 'stuck spacebar on new keyboard', 'refund_requested': True, ...}
A persona with guardrails, a running transcript, conservative sampling, and a machine-readable ticket at the end, and all of it running on your own machine, with customer messages that never leave it. That is the local LLM proposition in one script: privacy from locality, reliability from the dials, and structure from constrained decoding.
Work through these ten and you have the full article in practice: GGUF loading, the response shapes, every sampling dial, chat roles and system prompts, few-shot patterns, guaranteed JSON, and self-managed memory. The guide’s summary really does cover it: run the model, turn the dials, and keep the transcript.
Hope this helps, Andrei.
[…] Working with Llama 3: 10 Code-Along Examples […]
[…] Working with Llama 3: 10 Code-Along Examples […]