Scikit-LLM vs. Traditional Text Classifiers: When Should You Use an LLM?

In this article, you will learn how to benchmark three text classification approaches — from a classical TF-IDF pipeline to a zero-shot large language model — to understand when each is most appropriate.

Topics we will cover include:

How to implement and evaluate a classical TF-IDF and logistic regression text classification pipeline.
How to apply zero-shot classification using a transformer-based model (BART) and compare it against the classical baseline.
How to use scikit-LLM with a Groq-hosted large language model for production-ready zero-shot classification with minimal code changes.

Scikit-LLM vs. Traditional Text Classifiers: When Should You Use an LLM?

Introduction

In recent years, generative AI models like LLMs (large language models) have gradually taken over classical machine learning ones for addressing certain tasks, for instance, text classification. But the truth is: rather than having a one-beats-all solution, there are critical trade-offs developers need to face — should we stick with fast, battle-tested conventional models, invest in fine-tuning a transformer-based LLM, or perhaps leverage LLMs’ zero-shot reasoning potential?

In this article, we will implement a benchmarking between three distinct approaches for text classification:

TF-IDF and logistic regression (classic baseline).
Zero-shot classification with BART: a deep learning, transformer-based standard architecture.
Scikit-LLM with zero-shot classification: the most modern, prompt-based approach.

The tutorial below is kept entirely free for everyone to try, with no costs or API rate limits. To do so, we will use scikit-LLM alongside a model available from Groq. You will need to register at Groq and obtain an API key for evaluating the third solution below.

Implementing the Benchmarking

First, we install all the core libraries we will need.

!pip install scikit-learn transformers scikit-llm scikit-ollama pandas torch

!pip install scikit–learn transformers scikit–llm scikit–ollama pandas torch

For enabling reproducibility, we create a small, synthetic dataset containing customer support messages. The tickets are categorized into five classes. Once created, we store it in a DataFrame object and split it into training and test sets.

import pandas as pd from sklearn.model_selection import train_test_split data = { “text”: [ # Technical “My screen is completely black and won’t turn on.”, “The app keeps crashing every time I click save.”, “The Wi-Fi module is failing to connect to the router.”, “Data sync isn’t working across my devices.”, “My bluetooth headphones won’t pair with the app.”, “I keep getting an Error 404 on the login screen.”, “The database connection timed out during the export.”, “API rate limit exceeded even though I haven’t used it.”, “Profile images won’t load on the dashboard.”, “The software installation failed at 99%.”, # Billing “I was charged twice this month, please fix this.”, “How do I update my credit card information?”, “My invoice for last month is missing from the portal.”, “The VAT calculation on my receipt is wrong.”, “My transaction was declined but I have funds.”, “Can I change my billing cycle from monthly to annual?”, “Where can I find my official receipt?”, “My saved credit card expired and I need to swap it.”, “I was overcharged on my last statement.”, “Please remove my saved payment method.”, # Account “My account is locked and I forgot my password.”, “How do I change the email address on my profile?”, “Please delete my account and all associated data.”, “I want to update my profile picture.”, “How do I enable two-factor authentication (2FA)?”, “I didn’t receive the email verification link.”, “Can I merge two different accounts into one?”, “Is there a way to change my username?”, “I need to transfer account ownership to my manager.”, “I am locked out because I lost my 2FA phone.”, # Sales “Do you offer enterprise discounts for large teams?”, “Do you have an annual plan with a discount?”, “Can you compare the pro and basic tiers for me?”, “What is the pricing for a 50-user bulk license?”, “Is there a student discount available?”, “Can I schedule a demo with your sales team?”, “Do you sell and ship to customers in Europe?”, “How does your partner and reseller program work?”, “What are the usage limits on the free tier?”, “I need a custom quote for a government contract.”, # Refund “Can I get a refund for my last purchase? It was a mistake.”, “I want my money back for the subscription.”, “Accidental purchase, please reverse the charge.”, “I am not satisfied with the product, need a refund.”, “Cancel my subscription immediately and refund me.”, “I was charged after my free trial ended.”, “I need a prorated refund for the remaining months.”, “What is your official refund policy?”, “I was promised a refund last week but haven’t received it.”, “The item arrived broken, I want a full refund.” ], “label”: [ “Technical”] * 10 + [“Billing”] * 10 + [“Account”] * 10 + [“Sales”] * 10 + [“Refund”] * 10 } df = pd.DataFrame(data) # Stratified train-test splitting ensures all 5 categories are proportionally represented in both subsets when the dataset is small X_train, X_test, y_train, y_test = train_test_split( df[“text”], df[“label”], test_size=0.3, random_state=42, stratify=df[“label”] ) print(f”Training rows: {len(X_train)} | Testing rows: {len(X_test)}”)

import pandas as pd

from sklearn.model_selection import train_test_split

data = {

“text”: [

# Technical

“My screen is completely black and won’t turn on.”, “The app keeps crashing every time I click save.”,

“The Wi-Fi module is failing to connect to the router.”, “Data sync isn’t working across my devices.”,

“My bluetooth headphones won’t pair with the app.”, “I keep getting an Error 404 on the login screen.”,

“The database connection timed out during the export.”, “API rate limit exceeded even though I haven’t used it.”,

“Profile images won’t load on the dashboard.”, “The software installation failed at 99%.”,

# Billing

“I was charged twice this month, please fix this.”, “How do I update my credit card information?”,

“My invoice for last month is missing from the portal.”, “The VAT calculation on my receipt is wrong.”,

“My transaction was declined but I have funds.”, “Can I change my billing cycle from monthly to annual?”,

“Where can I find my official receipt?”, “My saved credit card expired and I need to swap it.”,

“I was overcharged on my last statement.”, “Please remove my saved payment method.”,

# Account

“My account is locked and I forgot my password.”, “How do I change the email address on my profile?”,

“Please delete my account and all associated data.”, “I want to update my profile picture.”,

“How do I enable two-factor authentication (2FA)?”, “I didn’t receive the email verification link.”,

“Can I merge two different accounts into one?”, “Is there a way to change my username?”,

“I need to transfer account ownership to my manager.”, “I am locked out because I lost my 2FA phone.”,

# Sales

“Do you offer enterprise discounts for large teams?”, “Do you have an annual plan with a discount?”,

“Can you compare the pro and basic tiers for me?”, “What is the pricing for a 50-user bulk license?”,

“Is there a student discount available?”, “Can I schedule a demo with your sales team?”,

“Do you sell and ship to customers in Europe?”, “How does your partner and reseller program work?”,

“What are the usage limits on the free tier?”, “I need a custom quote for a government contract.”,

# Refund

“Can I get a refund for my last purchase? It was a mistake.”, “I want my money back for the subscription.”,

“Accidental purchase, please reverse the charge.”, “I am not satisfied with the product, need a refund.”,

“Cancel my subscription immediately and refund me.”, “I was charged after my free trial ended.”,

“I need a prorated refund for the remaining months.”, “What is your official refund policy?”,

“I was promised a refund last week but haven’t received it.”, “The item arrived broken, I want a full refund.”

“label”: [

“Technical”] * 10 + [“Billing”] * 10 + [“Account”] * 10 + [“Sales”] * 10 + [“Refund”] * 10

}

df = pd.DataFrame(data)

# Stratified train-test splitting ensures all 5 categories are proportionally represented in both subsets when the dataset is small

X_train, X_test, y_train, y_test = train_test_split(

df[“text”], df[“label”], test_size=0.3, random_state=42, stratify=df[“label”]

)

print(f“Training rows: {len(X_train)} | Testing rows: {len(X_test)}”)

We first implement and evaluate the most classical approach: TF-IDF combined with a logistic regression classifier. The process is shown below:

import time from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.linear_model import LogisticRegression from sklearn.pipeline import make_pipeline from sklearn.metrics import classification_report start_time = time.time() # Creating and training the classical pipeline logreg_clf = make_pipeline(TfidfVectorizer(), LogisticRegression()) logreg_clf.fit(X_train, y_train) # Inference: predictions on the test examples y_pred_logreg = logreg_clf.predict(X_test) logreg_latency = time.time() – start_time # Latency is also measured to assess the model’s efficiency print(f”Logistic Regression Latency: {logreg_latency:.4f} seconds”) print(classification_report(y_test, y_pred_logreg, zero_division=0))

import time

from sklearn.feature_extraction.text import TfidfVectorizer

from sklearn.linear_model import LogisticRegression

from sklearn.pipeline import make_pipeline

from sklearn.metrics import classification_report

start_time = time.time()

# Creating and training the classical pipeline

logreg_clf = make_pipeline(TfidfVectorizer(), LogisticRegression())

logreg_clf.fit(X_train, y_train)

# Inference: predictions on the test examples

y_pred_logreg = logreg_clf.predict(X_test)

logreg_latency = time.time() – start_time

# Latency is also measured to assess the model’s efficiency

print(f“Logistic Regression Latency: {logreg_latency:.4f} seconds”)

print(classification_report(y_test, y_pred_logreg, zero_division=0))

Output:

Logistic Regression Latency: 0.0615 seconds precision recall f1-score support Account 0.25 0.33 0.29 3 Billing 1.00 1.00 1.00 3 Refund 0.67 0.67 0.67 3 Sales 0.25 0.33 0.29 3 Technical 1.00 0.33 0.50 3 accuracy 0.53 15 macro avg 0.63 0.53 0.55 15 weighted avg 0.63 0.53 0.55 15

Logistic Regression Latency: 0.0615 seconds

precision recall f1–score support

Account 0.25 0.33 0.29 3

Billing 1.00 1.00 1.00 3

Refund 0.67 0.67 0.67 3

Sales 0.25 0.33 0.29 3

Technical 1.00 0.33 0.50 3

accuracy 0.53 15

macro avg 0.63 0.53 0.55 15

weighted avg 0.63 0.53 0.55 15

The classifier shows a mixed behavior: it performs well on categories like Billing and, to some extent, Refund, but struggles with the rest. This is the fastest approach by far; however, its classification performance is limited by its inability to capture the complex linguistic nuances that more modern language models can effectively handle. Sticking to aggregated results, we get accuracies ranging between 0.53 and 0.55 overall.

Let’s see what our second approach — zero-shot classification with facebook/bart-large-mnli — has to offer:

from transformers import pipeline import time # Using a HuggingFace zero-shot classification pipeline as our transformer representative # We need to overload the default classifier to specify our own label set classifier = pipeline(“zero-shot-classification”, model=”facebook/bart-large-mnli”) candidate_labels = [“Technical”, “Billing”, “Account”, “Sales”, “Refund”] start_time = time.time() # Inference time! bert_preds = [] for text in X_test: result = classifier(text, candidate_labels) bert_preds.append(result[‘labels’][0]) # Get the highest scoring label bert_latency = time.time() – start_time print(f”Transformer Inference Latency: {bert_latency:.4f} seconds”) print(classification_report(y_test, bert_preds, zero_division=0))

from transformers import pipeline

import time

# Using a HuggingFace zero-shot classification pipeline as our transformer representative

# We need to overload the default classifier to specify our own label set

classifier = pipeline(“zero-shot-classification”, model=“facebook/bart-large-mnli”)

candidate_labels = [“Technical”, “Billing”, “Account”, “Sales”, “Refund”]

start_time = time.time()

# Inference time!

bert_preds = []

for text in X_test:

result = classifier(text, candidate_labels)

bert_preds.append(result[‘labels’][0]) # Get the highest scoring label

bert_latency = time.time() – start_time

print(f“Transformer Inference Latency: {bert_latency:.4f} seconds”)

print(classification_report(y_test, bert_preds, zero_division=0))

These are the results:

Transformer Inference Latency: 32.2503 seconds precision recall f1-score support Account 0.40 0.67 0.50 3 Billing 1.00 0.33 0.50 3 Refund 0.75 1.00 0.86 3 Sales 1.00 0.33 0.50 3 Technical 0.75 1.00 0.86 3 accuracy 0.67 15 macro avg 0.78 0.67 0.64 15 weighted avg 0.78 0.67 0.64 15

Transformer Inference Latency: 32.2503 seconds

precision recall f1–score support

Account 0.40 0.67 0.50 3

Billing 1.00 0.33 0.50 3

Refund 0.75 1.00 0.86 3

Sales 1.00 0.33 0.50 3

Technical 0.75 1.00 0.86 3

accuracy 0.67 15

macro avg 0.78 0.67 0.64 15

weighted avg 0.78 0.67 0.64 15

Much higher latency, and only a modest improvement in accuracy: 0.64–0.67 in broad terms.

Finally, the zero-shot LLM classifier with a scikit-LLM pipeline and a Groq model:

from skllm.config import SKLLMConfig from skllm.models.gpt.classification.zero_shot import ZeroShotGPTClassifier import getpass import time from sklearn.metrics import classification_report # 1. Securely asking for the key in a private input box: # GET YOURS AT https://console.groq.com/keys print(“Get your free Groq API key here: https://console.groq.com/keys”) api_key = getpass.getpass(“Paste your API Key here: “) # 2. Configuring Scikit-LLM SKLLMConfig.set_openai_key(api_key) SKLLMConfig.set_gpt_url(“https://api.groq.com/openai/v1/”) # 3. Initializing with the latest active model for zero-shot classification # ‘llama-3.3-70b-versatile’ is supported by Groq at the time of writing llm_clf = ZeroShotGPTClassifier(model=”custom_url::llama-3.3-70b-versatile”) start_time = time.time() # 4. Running the classification task llm_clf.fit(X_train, y_train) y_pred_llm = llm_clf.predict(X_test) llm_latency = time.time() – start_time print(f”\nScikit-LLM Latency: {llm_latency:.4f} seconds”) print(classification_report(y_test, y_pred_llm, zero_division=0))

from skllm.config import SKLLMConfig

from skllm.models.gpt.classification.zero_shot import ZeroShotGPTClassifier

import getpass

import time

from sklearn.metrics import classification_report

# 1. Securely asking for the key in a private input box:

# GET YOURS AT https://console.groq.com/keys

print(“Get your free Groq API key here: https://console.groq.com/keys”)

api_key = getpass.getpass(“Paste your API Key here: “)

# 2. Configuring Scikit-LLM

SKLLMConfig.set_openai_key(api_key)

SKLLMConfig.set_gpt_url(“https://api.groq.com/openai/v1/”)

# 3. Initializing with the latest active model for zero-shot classification

# ‘llama-3.3-70b-versatile’ is supported by Groq at the time of writing

llm_clf = ZeroShotGPTClassifier(model=“custom_url::llama-3.3-70b-versatile”)

start_time = time.time()

# 4. Running the classification task

llm_clf.fit(X_train, y_train)

y_pred_llm = llm_clf.predict(X_test)

llm_latency = time.time() – start_time

print(f“\nScikit-LLM Latency: {llm_latency:.4f} seconds”)

print(classification_report(y_test, y_pred_llm, zero_division=0))

Final results:

Scikit-LLM Latency: 2.5905 seconds precision recall f1-score support Account 0.67 0.67 0.67 3 Billing 1.00 0.67 0.80 3 Refund 1.00 1.00 1.00 3 Sales 1.00 1.00 1.00 3 Technical 0.75 1.00 0.86 3 accuracy 0.87 15 macro avg 0.88 0.87 0.86 15 weighted avg 0.88 0.87 0.86 15

Scikit–LLM Latency: 2.5905 seconds

precision recall f1–score support

Account 0.67 0.67 0.67 3

Billing 1.00 0.67 0.80 3

Refund 1.00 1.00 1.00 3

Sales 1.00 1.00 1.00 3

Technical 0.75 1.00 0.86 3

accuracy 0.87 15

macro avg 0.88 0.87 0.86 15

weighted avg 0.88 0.87 0.86 15

This is by far the best result in terms of classification accuracy (0.86–0.87). And surprisingly, it is also considerably faster than the BART-based zero-shot model. This is not all that surprising: the Groq-hosted model was trained on a massive, broad dataset. It does not need to learn what a given type of customer support ticket means — it already knows, unlike the zero-shot BART model used earlier.

So, we have a clear winner!

On a final note: this is where the value of scikit-LLM lies. It bridges the gap between classical and modern AI through a standardized, production-ready interface, using scikit-learn-like syntax throughout. With this in hand, you can swap between a classical logistic regressor and a modern Groq LLM with minimal effort.

Wrapping Up

This article benchmarked, on a toy dataset, scikit-LLM’s zero-shot classification against more classical approaches — logistic regression with TF-IDF, and a zero-shot transformer model (BART) sitting somewhere in between. As for the question posed in the title, when should you use an LLM for text classification? The choice of a small, toy dataset here was deliberate. When the amount of available data is limited and the task requires deep linguistic reasoning and contextual understanding, scikit-LLM is a compelling asset: it makes it possible to instantly deploy a model’s pre-trained world knowledge into a pipeline like ours, eliminating both the time and infrastructure costs of training a model of this magnitude from scratch.

Source link