• Instruction tuning in general refers to SFT part of LLM
  • pertaining + SFT + RLHF
• General ideas of getting score from LLMs
  • SFT to produce one token
    • positive/negatives
    • Use logic probabilities
      • relevance_score = logit(“pos”) – logit(“neg”)
      • pass it through sigmoid
    • Use case – prod system, confidence intervals
  • Self generate score
    • Ask model to produce no between 0 to 1
    • Use case – quick analysis
  • Add a class-action head on some OSS model and retrain
• qwen embedding and qwen reranker model
  • You get embedding of (instruction + query )
  • Get embedding of label
  • For re-ranker label is just ( yes / no )
  • After ebedding you can do cosine similarity
    • Softmax on top if you need probabilities
• How instruction helps ?
  • When we “instruct” model to do representation – it learns to auto regress in that way
  • The last token has attended all previous one. Taking embedding of it most popular
    • Other options are average of last few

Rerank

• We use cross encoder
  • We can not cache document embedding here, because top k docs are passed along with query
• It is still point wise training
• We can do lamba-rank style training by weighting gradient by delta ndcg
  • qwen kept it simple and “LM” style
• Models like RankGPT, RankVicuna and RankZephyr put about 20 documents in the prompt and generate a permutation, sliding a window over longer lists. The documents attend to each other, so the model can compare them directly. The downsides are long contexts, position bias, no per-document score, and slow sequential decoding.

Code example

from sentence_transformers import CrossEncoder
model = CrossEncoder("Qwen/Qwen3-Reranker-0.6B")
query = "What is the capital of China?"
documents = [
    "The capital of China is Beijing.",
    "Gravity is a force that attracts two bodies towards each other. It gives weight to physical objects and is responsible for the movement of planets around the sun.",
]
pairs = [(query, doc) for doc in documents]
scores = model.predict(pairs)
print(scores)
# [ 7.625 -11.375]
rankings = model.rank(query, documents)
print(rankings)
# [{'corpus_id': 0, 'score': 7.625}, {'corpus_id': 1, 'score': -11.375}]

SFT training data

We need hard negatives as well

[
  {
    "instruction": "Given a query and a document, judge whether the document is relevant to the query. Answer 'yes' or 'no'.",
    "input": "Query: How do you pass multiple documents to a cross encoder?\nDocument: To pass multiple documents, you format them as a list of pairwise strings like [query, doc].",
    "output": "yes"
  },
  {
    "instruction": "Given a query and a document, judge whether the document is relevant to the query. Answer 'yes' or 'no'.",
    "input": "Query: How do you pass multiple documents to a cross encoder?\nDocument: The quick brown fox jumps over the lazy dog.",
    "output": "no"
  }
]
In agentic era -- 
[
  {
    "messages": [
      {"role": "system", "content": "You are a helpful assistant that judges document relevance."},
      {"role": "user", "content": "Query: What is the context length of Qwen3-Reranker?\nDocument: Qwen3-Reranker comes in various sizes and supports up to 32k tokens context length."},
      {"role": "assistant", "content": "yes"}
    ]
  }
]

Classification

• Instruction + query + 5 classes
• Get embedding for each of the class by passing ( class label + class description)
  • this can be cached
  • Description is an important part of this setup.
• For each query get embedding for ( instruction + query )
• Code example
  • Retrieval also creates similarity matrix ( via Vectors DB / ANN )
  • For classification we don’t need it because no of classes are limited

from sentence_transformers import SentenceTransformer
# Load Qwen3 via SentenceTransformers
model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")
# Your target sequences
reviews = [
    "The customer support was rude and unhelpful.",
    "Outstanding service, they resolved my issue within five minutes!"
]
# Standard template definitions for classes
classes = ["A negative complaint", "A positive compliment"]
# Encode inputs (Using the built-in 'query' configuration for task-guided inputs)
review_embeddings = model.encode(reviews, prompt_name="query")
class_embeddings = model.encode(classes)
# Evaluate cosine similarities
similarity_matrix = model.similarity(review_embeddings, class_embeddings)
class_labels = ["Negative", "Positive"]
for i, review in enumerate(reviews):
    best_match_idx = similarity_matrix[i].argmax().item()
    print(f"Text: '{review}' -> Sentiment: **{class_labels[best_match_idx]}**")

Semantic Text Clustering / Duplicate Detection

Training data

# Text A and Text B are treated symmetrically
text_a = "Instruct: Find semantically equivalent text or duplicate items with the same intent.\nQuery: How do I reset my account password?"
text_b = "Instruct: Find semantically equivalent text or duplicate items with the same intent.\nQuery: I forgot my login credentials and need a password reset link."

SFT training

• Contrastive loss with triplet
• Classification problem, pos if they are similar

Document Prefix

We are working with asymmetric embedding models here. Asymmetric because query can look completely different from docs. Apply document prefix when –

1. Arch mandates that – for example at training time prefix was used
2. Help model understand context
   1. When we prefix “Finance IRS information” model knows how to interpret number
   2. Some times when we chunk some information is taken away

Trade-offs

• Calibration lost due to SFT
• Much bigger model, hence higher serving cost and latency
  • BERT is 100 M, dues not support zero shot well
  • qwen is 7B
• Pros
  • Zeroshot performance
  • Flexibility

MTEB Benchmark dataset

• Before MTEB, embedding models were typically tested on a very narrow set of datasets—usually limited to Semantic Textual Similarity (STS)—making it hard to know how well a model would perform in real-world applications like search or classification. MTEB standardizes this evaluation so developers can choose the right model for their specific use case
• 8 categories
  • Retrieval: Finding relevant documents for a search query (main metric: nDCG@10).
  • Classification: Sorting texts into predefined categories.
  • Clustering: Grouping similar texts together.
  • Semantic Textual Similarity (STS): Measuring how close the meaning of two sentences is.
  • Reranking: Re-ordering a list of texts by relevance.
  • Bitext Mining: Finding matching sentences across different languages.
  • Pair Classification: Deciding if a pair of texts is similar/related.
  • Summarization: Checking how closely machine summaries match human
• Available in 100+ languages
• No model is universal winner
• Leaderboard – https://leaderboard.mteb.org