The Llama 3 70B model itself does not have a single fixed embedding size; its hidden dimension is 8192, but the embedding output size depends on how you use it (e.g., for text generation vs. sentence embeddings). If you are looking for a standard sentence-embedding vector from Llama 3 70B, the most common approach yields 8192 dimensions, though many libraries reduce this to 4096 or 1024 via pooling or projection layers.
Formula
The embedding size of a transformer model like Llama 3 70B is determined by its hidden dimension (often called d_model). For Llama 3 70B, the hidden dimension is 8192. When you run text through the model, each token is mapped to a vector of that size. If you use mean pooling over all token embeddings, the final sentence embedding is also 8192-dimensional. If you take only the last token’s hidden state (as in many causal LMs), it is also 8192. Some libraries add a projection layer to reduce the dimension, but the raw model output is 8192.
Worked Example
Using the defaults: n = 128 tokens, h = 8192, pooling = mean. The output dimension is simply h = 8192. The parameter count for the embedding matrix alone (without positional encodings) is n * h = 128 * 8192 = 1,048,576. That is the size of the embedding matrix for a sequence of 128 tokens, not the model’s total parameters.
Background: Llama 3 Architecture
Llama 3 is a family of large language models released by Meta in 2024. The 70B version has 70.6 billion parameters. The architecture is a decoder-only transformer with grouped-query attention (GQA). The hidden dimension is 8192, which is the size of the embedding vector for each token. This is larger than Llama 2 70B (which used 8192 as well) but smaller than some other models like GPT-3 (12288). The embedding size directly affects memory usage and computational cost.
How Embedding Size Is Used
When you use Llama 3 70B for text generation, the embedding size is internal. But when you want to use it for semantic search, clustering, or classification, you need to extract a fixed-size vector for a whole sentence or document. Common methods:
- Mean pooling: average all token embeddings (size 8192).
- CLS token: take the first token’s hidden state (size 8192).
- Last token: take the final token’s hidden state (size 8192).
- Projection: some libraries (e.g., sentence-transformers) add a linear layer to reduce to 1024 or 4096.
For Llama 3 specifically, the official Meta release did not include a sentence-embedding head, so the raw hidden states are not optimized for similarity tasks. You may need fine-tuning.
Choosing the Right Embedding Size
For most applications, a larger embedding size (8192) captures more information but requires more storage and compute. If you are building a retrieval system over millions of documents, 8192-dim vectors can be heavy. Consider:
- Storage: 8192 floats = 32 KB per vector (float32) or 8 KB (float16).
- Speed: cosine similarity on 8192 dims is slower than on 1024 dims.
- Quality: for many tasks, 1024 or 4096 is sufficient; larger isn’t always better.
If you use a projection layer, you can train it to compress to a smaller size while retaining performance.
Common Problems and Misconceptions
One common mistake is confusing the embedding size with the vocabulary size. Llama 3 has a vocabulary of 128,256 tokens, but each token embedding is 8192 dimensions. Another issue: some people assume that the output of the last hidden layer is a sentence embedding, but it is actually a sequence of token embeddings. You must pool them. Also, the hidden dimension is not the same as the context length (which is 8192 tokens for Llama 3).
Comparisons with Other Models
For context: Llama 2 70B also uses 8192 hidden size. GPT-3 (175B) uses 12288. BERT-base uses 768. The trend is that larger models have larger embedding sizes. However, embedding size is not the only factor; the quality of embeddings depends on training data and objectives. For semantic search, dedicated models like Sentence-BERT (384 or 768 dims) often outperform raw LLM embeddings despite being smaller.
Cost and Memory Considerations
Running Llama 3 70B requires significant GPU memory. The model has 70B parameters, and with 16-bit precision, that’s about 140 GB. The embedding matrix alone (vocab 128k * 8192) is about 1 GB in 16-bit. When generating embeddings for a large corpus, you need to store the resulting vectors. For 1 million documents, 8192-dim float16 vectors would take 16 GB. That’s manageable but not trivial. Using a smaller embedding size (e.g., 1024) would reduce that to 2 GB.
Alternatives to Llama 3 70B for Embeddings
If you need high-quality sentence embeddings without the massive resource footprint, consider:
- OpenAI’s text-embedding-3-small (1536 dims) or large (3072 dims) – but that’s a paid API.
- Sentence-transformers models like all-MiniLM-L6-v2 (384 dims) – free and fast.
- E5-large (1024 dims) – good for retrieval.
- BGE models (1024 dims) – strong performance.
These are trained specifically for embeddings and often outperform raw LLM hidden states on similarity tasks.
Frequently Asked Questions
What is the exact embedding size of Llama 3 70B?
The hidden dimension is 8192. So each token’s embedding vector has 8192 numbers. If you apply mean pooling, the sentence embedding is also 8192-dimensional.
Can I use Llama 3 70B as a sentence embedding model?
Yes, but not out-of-the-box. You need to extract token embeddings and pool them. The result may not be as good as dedicated embedding models unless you fine-tune on a contrastive loss.
How many dimensions does the embedding output have after a projection layer?
That depends on the projection layer. Some libraries project to 1024 or 4096. There is no standard; you can choose any size.
Is a larger embedding size always better?
No. Larger embeddings can capture more nuance but require more memory and compute. For many tasks, smaller embeddings work just as well and are faster to compare.
What is the difference between embedding size and context length?
Embedding size is the number of dimensions per token. Context length is the maximum number of tokens the model can process at once. Llama 3 70B has a context length of 8192 tokens, and each token has an 8192-dim embedding.