Kent Wynn

Embeddings & Vector Search

Practical Lessons in Embeddings & Vector Search for Production Systems

Building robust AI systems requires more than models—here’s how to design, optimize, and debug embeddings and vector search in real-world scenarios.

By Kent Wynn·
EmbeddingsVector SearchAi EngineeringProductionData StructuresMl Models

When building AI systems, embeddings and vector search are essential, but getting them right in production is tricky. I’ve spent years shipping large-scale AI applications where these components are central—whether it’s recommendation systems, search engines, or semantic similarity pipelines. The difference between a working prototype and a production-ready system often hinges on decisions around model selection, data preprocessing, and system architecture. In this post, I’ll share concrete lessons from real projects, including tradeoffs between accuracy and performance, failure modes to avoid, and a checklist for implementation.

Choosing the Right Model and Vector Store

The first critical decision is selecting the right embedding model and vector store. For example, when building a product search feature for an e-commerce platform, we evaluated several options:

  • Model choice: We opted for a pre-trained transformer model (like BERT or Sentence-BERT) instead of training a custom one. While training a model on domain-specific data can improve accuracy, the time and compute costs were prohibitive. Instead, we fine-tuned a public model on a small labeled dataset of product descriptions, which gave us a 15% lift in relevance compared to using a generic model.
  • Vector store: We tested Pinecone, FAISS, and Milvus. FAISS was faster for nearest-neighbor lookups, but Pinecone’s managed service reduced operational overhead. For our use case, we prioritized ease of scaling and maintenance, so we chose Pinecone.

A common pitfall is assuming that a “better” model always means a better system. In practice, the right model depends on your data size, latency requirements, and team expertise. For example, if you’re dealing with 100k+ vectors, FAISS is often more performant than Pinecone, but it requires more engineering effort to manage.

Code Example: Preprocessing Text for Embedding

from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np

def preprocess_text(texts):
    vectorizer = TfidfVectorizer()
    vectors = vectorizer.fit_transform(texts)
    return vectors.toarray()

This example shows how text normalization (e.g., removing stop words, lowercasing) is critical for consistency. If you skip preprocessing, models will treat "Apple" and "apple" as different terms, leading to lower accuracy.

Tradeoffs Between Accuracy and Performance

In production systems, you’ll often face a tradeoff between accuracy and performance. For instance, a high-accuracy model might require more memory or slower inference, while a lightweight model could sacrifice precision. Here’s how to approach this:

  • Latency vs. recall: If your system requires sub-100ms queries (e.g., real-time search), prioritize lightweight models like FastText or even simpler algorithms like cosine similarity. For batch processing or offline tasks, you can afford higher latency.
  • Precision vs. recall: In a recommendation system, a model that returns 10 relevant items (high precision) is better than one that returns 100 items (high recall) if the user only clicks a few. However, if your goal is to surface diverse results, you might need a lower precision threshold.

A practical example: When building a document search feature for a legal firm, we initially used a high-accuracy model that took 200ms per query. Users complained about slow response times, so we switched to a faster model with slightly lower accuracy, which reduced query latency to 50ms. We also added a secondary layer of filtering using metadata (e.g., document type) to maintain relevance.

Handling Data Ingestion and Indexing

Data ingestion and indexing are often overlooked but critical steps. Poorly structured data can lead to subpar results, even with the best model. Here’s what to watch for:

  1. Batch vs. stream: If you’re ingesting data in batches (e.g., daily updates), you can reindex the entire dataset. For streaming data (e.g., user activity logs), you’ll need incremental updates. FAISS supports partial updates, but Pinecone requires full reindexing, which can be expensive.
  2. Duplicates: Duplicate vectors can skew results. For example, if the same document is indexed multiple times, the model might prioritize it over relevant but unique documents. Use deduplication logic (e.g., hashing) to avoid this.
  3. Dimensionality: High-dimensional vectors (e.g., 768 dimensions) are computationally expensive. If you’re using a model like BERT, consider reducing dimensions with techniques like PCA or autoencoders.

Code Example: Deduplication Using Hashing

import hashlib

def hash_text(text):
    return hashlib.sha256(text.encode()).hexdigest()

This ensures that identical documents are treated as a single vector, preventing redundancy in your index.

Failure Modes and Debugging

Even with the best setup, embeddings and vector search can fail in unexpected ways. Here are common failure modes and how to debug them:

  • Data drift: Over time, your training data may become outdated. For example, if you’re using a model trained on 2020 product descriptions, it may not rank newer products correctly. Monitor performance metrics (e.g., mAP, precision@k) and retrain the model periodically.
  • Query ambiguity: If users search for vague terms like "best phone," the model may return irrelevant results. Add a secondary layer of filtering (e.g., category-based relevance) or use a hybrid search approach combining embeddings and keyword matching.
  • Index corruption: If your vector store becomes corrupted (e.g., due to a failed write), you’ll need a backup or a way to rebuild the index. Always test index recovery procedures in staging environments.

Conclusion

Embeddings and vector search are powerful tools, but their success depends on careful engineering decisions. Prioritize models and vector stores that align with your latency, accuracy, and scalability needs. Always validate your implementation with real-world data, and don’t overlook the importance of preprocessing, deduplication, and monitoring. If you’re building an AI system, treat embeddings as a critical component—investing time in this area will save you from costly rework later.

When in doubt, start small: prototype with a lightweight model and a simple vector store, then iterate. The goal isn’t to chase perfection but to build a system that works reliably in production.

← Back to Blog