Batching should be performed at the latest possible stage in the processing chain, specifically for inferencing on GPUs. However, maintaining a certain response time Service Level Agreement (SLA) is essential. While it is not scheduled, we should batch whenever an opportunity arises.
Data pipelines in Hive are different from this context as they are scheduled to run daily.
Gatekeeping: Limiting the number of requests to 10 at a time.
Suppose the inference time of GPT is 50ms (99th percentile), we would guarantee a response time of 5 seconds once the request is accepted. If not accepted, we send an HTTP code 429, indicating “too many requests.” If excessive 429 responses are observed, we can consider spawning new machines.
Unsolved problems:
Loading input and pre/post-processing tasks consume CPU, while the expensive GPU remains idle during this time.
In a pub/sub model, the message injection rate should match the consumption rate.
They shared “What I Wish I Knew (WIWIK),” which primarily focuses on their experience with microservices and the potential downsides.
During a talk, the last question was about handling the decoupling of microservices. The answer was that they strive to do their best with engineering practices, but sometimes decoupling challenges still occur.
We need to understand this four terms from information theory:
Self Information
Entropy
KL Divergence
Cross Entropy
Self Information
Information conveyed by any event
Higher the probability lower the information conveyed
Entropy
Self information is of individual event, entropy is of distribution
Entropy is expected information of an event drown from that distribution
Distribution that are closer to uniform have highest entropy
Distribution that are nearly deterministic (where outcome is almost certain) have lower entropy
If log base is 2, it represents no of bites needed on average to encode symbols drawn from distribution of P. However this intuition is used prominently in communication theory than in machine learning.
KL Divergence
If we have two distribution P and Q of same random variable x, it tell how different this two distributions are
Extra amount of information (bits in base 2) needed to send a message containing symbols from P, while encoding was design for Q
KL divergence is always positive
It can be greater than 1
Bits required to encode information can be greater than 1
Example at [1]:
we have observed few events and we have the observed distribution
We want to represent is by some standard distribution say uniform or binomial.
Which one we should choose ?
The one for which KL divergence is minimum
This would same as to say the one for which extra information is minimum.
Binomial distribution has parameter p (probability of event = 1). Which value of this parameter we should choose ?
The one for which KL divergence is minimum.
It thus becomes an optimization problem
KL divergence is sometime termed as distance between two distribution
But it is not symmetric KL(P|Q) is not same as KL(Q|P)
Application :
One use case is in variational auto encoders (VAE) [2]
VAE generates sample data like GAN
For that we want output of encoder to be more generic before giving it to decoder
We measure this by measuring KL divergence of encoder output and uniform distribution
We add it in the final loss function
Cross Entropy
KL divergence measure extra information (bits) needed to encode P with symbols optimised for Q
Cross entropy measures total information needed to encode P with symbols optimised for Q
Formula for log-loss is exactly same (it is also called cross entropy loss)
Related
KS Test is used for goodness of fit
It is formulated in terms of hypothesis test and give p values
Based on empirical cumulative distribution function (empirical CDF)
Information retrieval (IR) deals with fetching relevant documents given search query. Purpose of this blog is to list evaluation metric that can be used to measure performance of this system.
Metrics for unranked retrieval
For all these definition is same as that for classification. However they have some distinct characteristics for IR system.
Precision and Recall
Example of a system where precision is important
Web search
Example of a system where recall is important
Individual searching their hard disk
Accuracy
For IR system data is genrally very skewed. 99 % of the document are in non relvant category. Hence accuracy does not make sense.
F Measure
We can have F1 score, F2 score etc depending upon how much weight we want to precision and recall in harmonic mean.
As ß > 1, we start giving more weight to recall.
Metrics for ranked retrieval
Precision recall curves
Historically during classification we plot ROC by changing the threshold of binary classification. In case of IR we plot it by changing no of documents retrieved.
11 point interpolated average precision
For the recall values of (0,0.1,0.2,…,0.9,1.0) find out the precision and average it.
Mean Reciprocal Rank (MRR)
Mean Reciprocal Rank answers one question: how far down the list did the user have to look before hitting something relevant?
For a single query, the reciprocal rank is 1 / (position of the first relevant result). MRR is just the average of that across all queries.
We have multiple queries in test set for offline eval
The average will get us a single number to work with
If a query has no relevant result in the top-k, its RR is 0 (not undefined) — otherwise you’d be silently dropping your worst queries.
Two things worth keeping in mind about its shape: it only looks at the first relevant item, so a list with one hit at position 2 scores identically whether the other nine results are all relevant or all garbage. And the drop-off is steep — 1 → 0.5 → 0.33 — so it’s dominated by whether you nailed position 1 or 2. That makes it a good fit for known-item lookup and QA, and a poor fit when recall across a set matters.
Mean average precision (MAP)
For a given query we calculate average precision. We take a mean of that for several queries.
Example:
There are 10 documents, document 1,2,5 are revalant.
MAP values typically varies a lot for different query in the same system. Say between 0.1 to 0.7
For different systems and same query MAP values does not vary that much. Hence for testing which system is better using MAP, large no of queries are needed.
Precision at k
MAP measure precision at various recall levels (until all the documents are retrieved). For application like web search what matters is result on first page or first three pages.
Disadvantages
Least stable and does not average well
Reason : total no of relevant document for a query has a strong influence on this metric.
R-precision
Different queries can have different number of relevant items ( call it R )
We set different k = R for each query
R is different for each query
R precision
Same as precision at k where k = no of relevant documents in a given query
It adjusts for the relevant document for a query (disadvantage of precision at k)
Empirically R-precision and MAP turn out be highly correlated.
NDCG
Normalized Discounted Cumulative Gain
Like precision at k it is evaluated at some values of k
It constitutes of cumulative gain at each position which is discount by position and is normalized
It is not max or sum normalized
It is normlized by ideal ndcg, which is calculated by sorting document based on relevance score. [3]
NDCG ranges between 0 to 1. For perfect ranking ideal value of NDCG is 1.
Example
positions : 1 2 3 4 5
eval score : 2 1 2 0 1
NDCG(q, k) = DCG(k)/Ideal DCG(k)
DCG(k) = sum (Gain(i) * Discount factor(i)) for i in (1,k)
Discount factor is generally taken as 1/log(1 + pos)
Gain is generally takes as (2^r - 1) where r is relevance score given by humans say 0 - not relevant ,1-near relevant , or 2-relevant.
Normalized Discounted Cumulative Gain (NDCG) becomes comparable across queries by dividing a query’s actual Discounted Cumulative Gain (DCG) by its Ideal DCG (IDCG), which scales every query’s maximum possible score to a uniform baseline of 1.0. [1, 2]
Why Raw DCG Fails Across Queries
Varying relevance: Different queries have different numbers of relevant items or varying maximum possible scores.
Absolute scale distortion: A broad query with many highly relevant documents naturally produces a much higher raw DCG than a niche query with few results.
Incomparable totals: Comparing raw DCG across two different queries is like comparing percentages to raw point totals.
Notes from Prof. Gilbert Strang’s Lecture on MIT OpenCourseWare: The Geometry of Linear Equations
In linear algebra, when faced with equations, we often try to visualize them using the row picture.
Row Picture: In 2-D, we can think of it as a line and aim to find its intersection.
Column Picture: We aim to find the weights of a linear combination of columns. In the image on the right, we see the addition of two vectors. We start from the origin and add them at the tail.
As we move to higher dimensions, the row picture becomes more challenging to visualize, while the column picture remains straightforward.
Furthermore, the column picture allows us to check if the combination of all columns fills the entire space. We can verify this through elimination.
Additionally, we can develop a habit of viewing matrix multiplication as a linear combination of columns.
In a simple regression model, the regression slope (β) represents the estimated change in the dependent variable (Y) corresponding to a one-unit increase in the independent variable (X). It quantifies the linear relationship between X and Y and indicates the direction and magnitude of the relationship.
The correlation coefficient (r) measures the strength and direction of the linear relationship between X and Y. It ranges from -1 to +1, with a value of 1 indicating a perfect positive linear relationship, -1 indicating a perfect negative linear relationship, and 0 indicating no linear relationship.
When the standard deviations of both X and Y are equal (SD(X) = SD(Y)), the regression slope (β) and the correlation coefficient (r) coincide.
The slope can be calculated as the correlation coefficient multiplied by the ratio of the standard deviations (β = r * SD(Y) / SD(X)). The correlation coefficient essentially represents the slope you would obtain from a regression of standardized variables (Y / SD(Y) on X / SD(X) or vice versa).
However, when the standard deviations of X and Y are not equal, the regression slope and the correlation coefficient provide distinct information:
The correlation coefficient is a bounded measure that can be interpreted independently of the scale of the variables. It indicates the strength of the linear relationship between X and Y, with values closer to ±1 indicating a stronger linear relationship. The regression slope, on its own, does not provide this information.
The regression slope represents the estimated change in the expected value of Y for a given unit increase in X. It provides information about the direction and magnitude of the relationship between X and Y in the original units of measurement. This information cannot be deduced from the correlation coefficient alone.
One more thing to add here is the relationship between correlation coefficient and co-variance. Formula is : r = Covariance (Y, X) / [ SD(Y) * SD(x) ]. We are normalising by SD of each variable. Also SD = sqrt ( variance ). We can also say that b = Covariance(X,Y) / VAR(X)
To explain softmax, Andrew Ng uses the terms “hard-max” and “soft-max.”
Softmax calculates the output probabilities of various classes using the formula: y_pred = exp(z_i) / sum_over_i ( exp(z_i) ).
Softmax outputs the probability distribution of the classes.
In hardmax, we assign one class as 1 and the others as 0.
Cross Entropy:
Cross entropy is a loss function commonly used in classification tasks.
The loss is calculated using the formula: Loss = - sum [y_actual * log(y_pred)].
For example, if the actual class is [1, 0, 0, 0, 0]:
y_pred_1 = [0.1, 0.5, 0.1, 0.1, 0.2]
y_pred_2 = [0.1, 0.6, 0.1, 0.1, 0.1]
The loss will be the same for y_pred_1 and y_pred_2.
This is a key feature of multiclass log loss: it rewards or penalizes the probabilities of correct classes only, and the value is independent of how the remaining probability is split between incorrect classes. [0]
Cross entropy is same as loss function of logistic regression, it is just that there are two classes.
When cluster labels are available we can use this matrix
It basically checks the similarity between two cluster assignments
Labels can also be seen as one type of cluster assignment
Score basically tells us how similar to cluster assignments are
This works by taking pair of points
Out of all pairs how many pairs are agreed in both clusters mechanism
Agree mean both
They are in same cluster in both mechanism
They are in different cluster in both mechanism
The Rand index has a value between 0 and 1, with 0 indicating that the two data clustering do not agree on any pair of points and 1 indicating that the data clustering are exactly the same
One drawback of Rand index is that it can given non zero value for random assignment of clusters. To mitigate that there is matrix called Adjusted Rand Index. [2]
It specifically does not work when no of clusters are high
Homogeneity, completeness and V-measure
To understand the difference, imagine you are sorting a basket of fruit containing Apple, Banana, and Orange into three boxes.
Here is how different sorting scenarios affect the scores:
Scenario 1: High Homogeneity, Low Completeness (The Splitter)
You put each piece of fruit into its own individual small box.
Result: Every box contains exactly one type of fruit (100% pure). However, your apples are scattered across multiple boxes.
Scores: Homogeneity = 1.0 | Completeness = Low
Scenario 2: Low Homogeneity, High Completeness (The Grouper)
You throw all the fruits—apples, bananas, and oranges—into one giant single box.
Result: Every single apple is in the exact same box, so no class is split up. However, the box is a messy mix of different fruits.
Scores: Homogeneity = Low | Completeness = 1.0
Scenario 3: Perfect Scores
You put all the apples in Box A, all the bananas in Box B, and all the oranges in Box C.
Scores: Homogeneity = 1.0 | Completeness = 1.0
Result: Every box is perfectly pure, and no class is split across multiple boxes.
Example
Final Evaluation Summary
Homogeneity (\(0.3112\)): Low score because Cluster 0 is heavily polluted with a mix of fruits.
Completeness (\(0.3836\)): Also quite low because the Banana class was completely split in half across both clusters.
When data is class-imbalanced there is a tendency to predict majority class. One way to tackle this would be apply more weight to minority classes in cost function. Another way is oversampling and under-smapling.
Over-sampling makes duplicate copies of minority classes
Under sampling randomly removes some samples from majority class
This should be used with caution
We need to check once that we still remain with enough sample for a given no of features
Practically we might want to over sample some classes and under-sample others.
Cross validation
Validation set should be taken out from original data[1]
We can do the sampling just before training only on training data
