On-device models
Why it matters An on-device model answers you without your words ever leaving your computer or phone. That turns "trust us with your data" into "we never had your data". It is how the CV Builder's AI polish and this course's tutor work, and you can check it for yourself.
The chef in your kitchen
A cloud AI is like a restaurant: you hand your order to a waiter and the cooking happens out of sight. An on-device model is like having the chef come to your kitchen. The chef has to travel to you once (that is the download), but after that every meal is cooked at home, and nobody else sees what you eat.
Model: a large file of numbers (called weights) plus code that uses them to predict the next words of a text. "Running a model" means doing a lot of maths with those numbers.
On-device model: a model whose weights are downloaded to your device and run there, using your own processor or graphics chip. Your prompt and the answer stay on the device.
How it works in a browser
Floati uses WebLLM, an open-source library that runs language models inside a web page. Three pieces make it work:
- WebGPU, a browser feature that lets a page use your graphics chip (GPU) for fast maths. Language models need a lot of maths, so this is what makes them usable in a tab.
- A small, compressed model. The course tutor and the CV Builder use a small instruction-following model of about half a billion parameters, compressed so it fits in a browser (checked October 2026).
- The browser's cache, which keeps the downloaded weights so the next visit starts much faster.
Parameters: the individual numbers inside a model. More parameters usually means a more capable model, and a bigger download.
Quantisation: storing each number with fewer bits (for example 4 bits instead of 16 or 32). Like saving a photo at lower quality: much smaller, a little less sharp.
What does travel over the network
Honesty matters here. Using an on-device model is not "no network at all". Two things are downloaded:
- The library code that runs the model.
- The model weights, which are hundreds of megabytes, downloaded once and then cached.
Both are downloads to you. What is never sent is your side: the questions you ask and the text you paste. The server that hosts the weights learns that someone downloaded a model, much like downloading any file, but it never sees your conversation.
The trade-offs
| Cloud model | On-device model | |
|---|---|---|
| Who sees your text | The provider | Only your device |
| Quality | Usually much stronger | Good at small, focused jobs |
| First use | Instant | A large one-time download |
| Needs | An internet connection | A browser with WebGPU and enough memory |
| Works offline | No | Yes, once downloaded |
WebGPU is available in current desktop versions of Chrome, Edge and Safari, and in Firefox on some systems (checked October 2026). On older or low-memory devices the model may not load at all, which is why Floati's CV Builder also has a simple rule-based suggester that needs no model. Be clear about the difference: that suggester runs on Floati's own server, so your CV text is sent there, though never to an AI company.
A common misconception
"Small models are useless." A half-billion parameter model will not write your novel, but it is good at narrow jobs: rewording a sentence, giving a hint, summarising a paragraph. For many private tasks a small local model is the right tool, and a giant cloud model is overkill that costs you your data.
Try it Open a lesson with the tutor and open your browser's DevTools (F12, or Cmd+Option+I on a Mac), then the Network tab. Start the tutor: you will see large downloads for the model. Once it has loaded, clear the list and ask the tutor a question. Watch the list: you may see a small page counter, but no request carries your question or your code.
Recap
- An on-device model runs on your own hardware, so your prompts and answers never leave the device.
- In a browser this uses WebGPU, a small quantised model and the cache; Floati's tutor and CV Builder use WebLLM this way.
- The code and weights are downloaded to you once; your text is never uploaded.
- The cost is lower quality, a big first download and device requirements.
This lesson is locked
Lessons open one at a time. Finish the previous lesson to unlock this one.