Trying out Ollama - Your 10 year old box would do too.
TLDR
- Yes, you CAN install an AI engine locally
- No, you DON'T need to spend thousands of dollars to get started!
- Agreed, that your ai engine wouldn't be snappy, it's still great to get started.
Server
You'd realise that any machine should get you going.
- I had recently bought a second-hand desktop box (Dell OptiPlex 3020) from FB Marketplace and repurposed it here.
- For specs, it was an Intel
i5-4590 CPU @ 3.30GHz
with 8GB of RAM and 250 GB of disk, nothing fancy. - It came with an AMD Radeon 8570 (2GB RAM) [4], and the Ollama install process recognized and optimized for the decade old GPU. Super-Nice!
- For completeness, the box cost me $70 AUD (~50 USD) in May 2024. In other words, even for a cash-strapped avid learner, there's a very low barrier to entry here.
Install
The install steps were pretty simple [1] but as you may know, the models themselves are huge.
For e.g. look at this [3]:
- mistral-7B - 4.1 GB
- gemma2-27B - 16 GB
- Code Llama - 4.8 GB
Given that, I'd recommend switching to a decent internet connection. If work allows, this may be a good time to go to work instead of WFH on this one. (Since I didn't have that luxury, my trusty but slow 60Mbps ADSL+ meant that I really worked up on my patience this weekend)
The thing that actually tripped me, was that Ollama threaded downloads really scream speed and it ended up clogging my test server (See my earlier blog post that goes into some details [2]).
Run with Nice
With system resources in short-supply, it made good sense, to ensure that once Ollama is installed, it is spun up with least priority.
On an Ubuntu server, I did this by modifying the ExecStart
config for Ollama's systemd
script.
ai@dell:~$ sudo service ollama status | grep etc
Loaded: loaded (/etc/systemd/system/ollama.service; enabled; preset: enabled)
ai@dell:~$ cat /etc/systemd/system/ollama.service | grep ExecStart
ExecStart=nice -n 19 /usr/local/bin/ollama serve
So when I do end up asking some fun questions, ollama is always playing "nice" :D
Enjoy ...
Reference:
Install + Quick Start: https://github.com/ollama/ollama/blob/main/README.md#quickstart
Model downloads made my server unresponsive: https://www.thatguyfromdelhi.com/2024/07/ollama-is-missing-rate-limits-on.html
Model sizes are in GBs: https://github.com/ollama/ollama/blob/main/README.md#model-library
Radeon 8570: https://www.techpowerup.com/gpu-specs/amd-radeon-hd-8570.b1325
No comments:
Post a Comment