Compare Tensor Precision: 3 Essential Methods for 2025

import torch

# Let's create two tensors that are nearly identical
a = torch.tensor([1.0, 2.0, 3.0])
b = torch.tensor([1.000001, 2.0, 3.0000009])

# Direct equality check will fail
print(f"torch.equal(a, b): {torch.equal(a, b)}")
# >> torch.equal(a, b): False

# But `allclose` sees they are functionally the same
print(f"torch.allclose(a, b): {torch.allclose(a, b)}")
# >> torch.allclose(a, b): True

# Now, let's introduce a more significant difference
c = torch.tensor([1.0, 2.1, 3.0])

# `allclose` will correctly identify the meaningful difference
print(f"torch.allclose(a, c): {torch.allclose(a, c)}")
# >> torch.allclose(a, c): False

Method 2: Beyond a Boolean: Visualizing the Difference

Sometimes, a simple `True` or `False` from `allclose` just isn't enough. What if it returns `False`? Your next question is, "Okay, but how different are they?" Is it a single outlier, or is there a systemic drift across the entire tensor? This is where element-wise difference analysis comes in.

Instead of boiling the comparison down to one boolean, you'll compute the difference between the tensors and then analyze the statistics of that difference. This gives you a rich, quantitative, and often visual understanding of the discrepancy.

How to Diagnose the Difference

1. Calculate the Error Tensor: Simply subtract one tensor from the other and take the absolute value: `error = torch.abs(tensor_a - tensor_b)`.
2. Get Descriptive Statistics: The most useful stats are the maximum, mean, and standard deviation of the error. `error.max()` tells you the worst-case difference, while `error.mean()` reveals the average discrepancy.
3. Visualize the Distribution: The real magic happens when you plot a histogram of the error tensor's values. This can instantly tell you if the errors are normally distributed around zero (as you'd expect from floating-point noise) or if there's a skew or large outliers indicating a more serious problem.

import torch
import matplotlib.pyplot as plt

# Simulate two slightly different model outputs
output_a = torch.randn(1, 3, 256, 256)
# Introduce tiny, random noise to simulate GPU vs. CPU differences
noise = torch.randn(1, 3, 256, 256) * 1e-6
output_b = output_a + noise

# `allclose` might pass or fail depending on the noise
# but let's investigate deeper.

abs_diff = torch.abs(output_a - output_b)

print(f"Maximum absolute difference: {abs_diff.max().item():.2e}")
print(f"Mean absolute difference:    {abs_diff.mean().item():.2e}")
print(f"Standard deviation:          {abs_diff.std().item():.2e}")

# Visualize the distribution of the differences
plt.figure(figsize=(10, 6))
plt.hist(abs_diff.numpy().flatten(), bins=50, log=True)
plt.title("Distribution of Element-wise Differences (Log Scale)")
plt.xlabel("Absolute Difference")
plt.ylabel("Frequency")
plt.grid(True)
plt.show()

When to use difference analysis: Use this method when `allclose` fails and you need to debug. It's essential for diagnosing issues in mixed-precision training, quantization-aware training, or when you see diverging behavior between different hardware platforms (e.g., NVIDIA A100 vs. H100, or CPU vs. GPU).

Method 3: Comparing Direction, Not Magnitude, with Cosine Similarity

Our first two methods are obsessed with values. But what if the overall pattern, or "direction," of the data is more important than the precise magnitudes? This is often the case with embeddings, attention maps, or feature vectors.

Imagine you have two word embeddings that are identical in pattern, but one is uniformly twice the magnitude of the other (e.g., `[0.1, 0.2, 0.3]` vs. `[0.2, 0.4, 0.6]`). `allclose` would fail spectacularly, and difference analysis would show a large, uniform error. Yet, for many downstream tasks, these vectors are functionally identical.

Enter Cosine Similarity. This method treats the tensors as high-dimensional vectors and measures the cosine of the angle between them. The result is a score between -1 and 1:

• 1: The vectors point in the exact same direction (perfect similarity).
• 0: The vectors are orthogonal (no similarity).
• -1: The vectors point in opposite directions.

It effectively ignores overall magnitude and focuses purely on the relational structure of the data.

import torch
import torch.nn.functional as F

# Two embedding vectors. `b` is a scaled version of `a`.
emb_a = torch.tensor([0.1, 0.8, -0.4])
emb_b = torch.tensor([0.2, 1.6, -0.8]) # emb_b = 2 * emb_a

# A third, unrelated embedding
emb_c = torch.tensor([0.5, -0.2, 0.9])

# `allclose` fails for a and b
print(f"allclose(a, b): {torch.allclose(emb_a, emb_b)}")
# >> allclose(a, b): False

# Cosine similarity shows they are perfectly aligned
# Note: The function expects batch dimension, so we unsqueeze
sim_ab = F.cosine_similarity(emb_a.unsqueeze(0), emb_b.unsqueeze(0))
print(f"Cosine Similarity (a, b): {sim_ab.item():.4f}")
# >> Cosine Similarity (a, b): 1.0000

# Similarity with the unrelated vector is low
sim_ac = F.cosine_similarity(emb_a.unsqueeze(0), emb_c.unsqueeze(0))
print(f"Cosine Similarity (a, c): {sim_ac.item():.4f}")
# >> Cosine Similarity (a, c): -0.1119

When to use cosine similarity: This is your go-to for comparing embeddings from language models, feature vectors from vision models, or any situation where the relative relationship between elements in a tensor is more important than their absolute values.

Choosing Your Method: A Quick Guide

To make it even clearer, here’s a quick breakdown of when to use each method.

Method	Best For	Answers the Question...	Primary Use Case
`allclose`	General purpose, value-based comparison	"Are these tensors close enough to be considered equal?"	Unit testing, checkpoint validation
Difference Analysis	Debugging and diagnostics	"How and where are these tensors different?"	Analyzing hardware/platform discrepancies
Cosine Similarity	Directional, pattern-based comparison	"Do these tensors represent the same concept, regardless of scale?"	Comparing embeddings and feature vectors

Conclusion: Compare with Confidence

The days of being stumped by a failed `==` comparison are over. The key is to realize that comparing tensors is not about seeking perfect equality, but about understanding the nature and significance of their differences.

Start with `allclose` for a quick and reliable verdict. If that fails, graduate to difference analysis to become a detective, hunting down the source and scale of the error. And when you care more about the forest than the trees—the pattern more than the value—reach for cosine similarity.

By internalizing these three essential methods, you'll stop letting floating-point gremlins derail your projects. You’ll be equipped to debug faster, build more robust tests, and compare tensors with the confidence and precision that modern AI development demands.