Using Scikit-LLM with Open-Source LLMs

In this article, you will learn how to use locally hosted language models through Ollama to perform text classification tasks, all without spending a cent on API calls.

Topics we will cover include:

• How to install Ollama and pull open-source models like Llama 3, Mistral, and Gemma to run locally on your machine.
• How to configure the Scikit-LLM library to route requests to a local Ollama endpoint instead of a paid cloud API.
• How to build a zero-shot text classifier using a local large language model and scikit-LLM in a familiar scikit-learn-style workflow.

Using Scikit-LLM with Open-Source LLMs

Introduction

This article will teach you how to perform a language task like text classification by integrating locally hosted large language models (LLMs) of manageable size, like Mistral, Gemma, and Llama 3: all for free thanks to Ollama — a free repository for local LLMs — and the Scikit-LLM Python library.

Pre-requisite: Installing Ollama

It is recommended to use an IDE to run this tutorial, as we will need to interact with your locally installed version of Ollama from there. New to Ollama? Then I recommend you check this article out first. Nonetheless, here is a summary of what to do in the local command line terminal to download a local LLM after installing Ollama on your computer.

Once you see the model interaction window in the terminal, you can type “/bye” to keep it running in the background, waiting for API calls. Meanwhile, in a newly created project in your Python IDE, you will need to have the following libraries installed:

If you encounter a “Module not found” error when executing the Python code, try installing the above dependencies one by one.

Okay! Time to fill in our Python code file (name it as you wish!), step by step. First, of course, come the imports. One of them is the class ZeroShotGPTClassifier. Similar to classical scikit-learn, this is a dedicated class for training and using a model for zero-shot classification: concretely, an LLM from Ollama.

Next, we need to apply a couple of specific configurations to be able to communicate with Ollama.

After that, we create a small dataset and prepare it for classification. Since we are not going to evaluate the model’s classification performance in this tutorial — our main goal is to learn how to use Scikit-LLM locally with open-source models like those available through Ollama — we do not need a large number of data examples.

The dataset contains user reviews and their corresponding categories, e.g. types of customer inquiries or feedback. We also made a training/test split as usual with machine learning modeling.

In the next part of the code, we add the necessary instructions for initializing and running our classifier, which will be at its core a task-adapted running instance of one of our installed Ollama models, such as Llama 3:

To finish up, we print some outputs consisting of a couple of model inference results (classification predictions) on the two examples contained in the test set. This is a very small dataset, but the aim here is to show how we managed to link Scikit-LLM with a local, free Ollama model to elegantly use an LLM for a specific task at no cost!

The result (it may vary depending on your test examples):

Alternatively, you could run your Python script from your terminal. For example, if you named it local_classification.py, execute this command:

Either way, if you followed all the steps, you should have it working. Well done!

Wrapping Up

This article illustrated how to swap in free, locally run models served through Ollama, such as Llama, Mistral, or Gemma — all for free, and in a few easy steps — thanks to Python’s Scikit-LLM library, which enables the use of cutting-edge LLMs within a familiar classical machine learning workflow.