Off the shelf Models categories

Archit Vora

Laying out the major categories of off-the-shelf models you’d typically pull from a shelf (HuggingFace, Cohere, OpenAI, etc.), grouped by what they’re for rather than architecture alone, since that’s usually how you’d reach for one.

  • Dense bi-encoders ( embedding models )
    • This is what people mean when they say “encoder”
    • BERT style or newer architecture 
    • Used for retrieval / ANN, clustering 
    • Example – E4, BGE
  • Cross encoders ( Rerankers )
    • Take query + document and output a relevance score 
    • So separate embedding, but much higher precision 
    • Example –
      • Cohere Reranker 
      • BGE reranker 
  • Instruction tuned decoder LLMs used as retrieval / ranking component
    • Qwen and friends 
    • They are not encoder
    • They are decoder, that gets purposed into IR tasks via prompting
      • listwise/pointwise LLM reranking, query rewriting/expansion, generating synthetic training pairs, or even being used to produce embeddings (LLM2Vec-style, or just taking last-hidden-state pooling).
    • The key difference from #1/#2 is that they weren’t trained with a contrastive/relevance objective — they’re repurposed, which has real implications for calibration and latency.
      • SFT hampers calibration, workaround – LR on top of logits
      • Latency because BERT is 100M params and qwen is 7B params
    • Yes, decoder can generate embedding too
      • They won’t be bi-directional but last token has seen everything else 
      • So in practice, “using an LLM as an encoder” often means: take the pretrained decoder weights, then adapt the architecture and objective toward something encoder-like, rather than using it purely as-is.
      • The last token has attended all previous one. Taking embedding of it most popular
        • Other options are like taking average of last n token
        • When we “instruct” model to do representation – it learns to auto regress in that way
  • Generative decoder LLMs ( general purpose )
    • base / pretrained model 
    • instruction tuned / chat models 
    • reasoning tuned variants
  • Classification / NLI models
    • NLI (natural language inference) models
    • Sentiment/toxicity/topic classifiers
    • NER (named entity recognition) / token classification
  • Multimodal models
    • CLIP style dual encoder
      • image<–> text retrieval 
      • Both on same embedding space 
    • Vision language generative models ( VLMs )
      • input – image + text 
      • output – text generation 
      • Different from CLIP
        • This is generative 
        • CLIP produces embedding 
    • Text to image / diffusion models
      • a genuinely different training paradigm (denoising) rather than transformer next-token or contrastive objectives.
  • Speech models
    • ASR : speech to text 
    • TTS
    • Speech embedding models

Leave a Reply