Three levels of the Hugging Face API: pipelines, tokenizers, and models
The transformers library lets you work at whatever altitude you need — a one-line pipeline, or hands-on with the tokenizer and the model. Knowing the three levels tells you when to drop down.
Hugging Face's transformers library is deep, and that scares people off. It shouldn't: it's really just three levels of abstraction stacked on each other. Start at the top, and drop down a level only when you need the control. Here's the map.
Level 1: pipelines — the easy button
A pipeline wraps the whole process — tokenise, run the model, decode the output — behind a single call. For standard tasks (sentiment, summarisation, translation, generation) it's all you need.
from transformers import pipeline
clf = pipeline('sentiment-analysis')
clf('Hugging Face makes this easy')
# [{'label': 'POSITIVE', 'score': 0.9998}]Level 2: tokenizers — text to tokens
Drop a level and you handle the tokenizer yourself: turn text into the integer ids the model consumes, and decode ids back into text (see the tokens post for what tokens are). This is where you control padding, truncation, and batching.
Level 3: models — the network itself
At the bottom you load the model directly and run a forward pass, getting raw outputs (logits or hidden states) to post-process however you like. This is the level you work at for custom heads, fine-tuning, or squeezing out performance.
from transformers import AutoTokenizer, AutoModelForSequenceClassification
name = 'distilbert-base-uncased-finetuned-sst-2-english'
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name)
inputs = tok('Full control now', return_tensors='pt')
logits = model(**inputs).logits # raw scores you post-processWhich level should you use?
Use a pipeline for anything standard and quick — most of the time. Drop to the tokenizer and model when you need control the pipeline hides: custom batching, non-standard outputs, or fine-tuning. Same library, same models — you just choose how much of the machinery to hold yourself.
The trick with Hugging Face isn't learning it all at once. It's knowing which of the three levels the task in front of you actually needs.