Master vector embeddings and semantic search to build intelligent search engines, recommendation systems, and similarity-based features. Understand the mathematical foundation powering modern AI applications.
Vector embeddings are the mathematical foundation of modern AI - they transform text, images, and other data into numerical vectors that capture semantic meaning. Unlike traditional keyword search, embeddings enable AI to understand context, find similar content, and power intelligent features like semantic search and recommendations.
Think of embeddings as coordinates in a multi-dimensional space where similar concepts are close together. Words like "king" and "queen" have embeddings that are near each other, while "king" and "bicycle" are far apart. This mathematical representation unlocks powerful capabilities.
In this advanced lesson, you'll learn how embeddings work, implement semantic search, build recommendation engines, and use vector databases. These techniques power features like "similar products," "related articles," and intelligent search in modern applications.
Vector embeddings convert text into numerical representations that capture semantic meaning - the foundation for semantic search, recommendations, and AI understanding.
What Are Embeddings?
An embedding is a numerical vector (array of numbers) that represents the meaning of text:
Example Visualization (Simplified to 3D):
1Text: "king" → [0.5, 0.8, 0.1]
2Text: "queen" → [0.4, 0.9, 0.2]
3Text: "man" → [0.6, 0.3, 0.1]
4Text: "woman" → [0.5, 0.4, 0.2]
5Text: "apple" → [-0.3, 0.2, 0.7]
6Text: "orange" → [-0.2, 0.3, 0.8]
7
8Note: Real embeddings have 1536+ dimensions!Why Embeddings Matter:
1. Semantic Understanding:
2. Enable New Capabilities:
3. Efficient at Scale:
How Embeddings Are Created:
Using OpenAI Embeddings API:
1import OpenAI from 'openai';
2
3const openai = new OpenAI({
4 apiKey: process.env.OPENAI_API_KEY
5});
6
7async function getEmbedding(text) {
8 const response = await openai.embeddings.create({
9 model: "text-embedding-3-small", // Fast, cost-effective
10 input: text,
11 encoding_format: "float"
12 });
13
14 return response.data[0].embedding; // Array of 1536 numbers
15}
16
17// Usage
18const embedding = await getEmbedding("Machine learning is fascinating");
19console.log(embedding);
20// [0.0023, -0.0142, 0.0089, ... 1536 numbers total]
21console.log(embedding.length); // 1536Available Embedding Models:
text-embedding-3-small:
text-embedding-3-large:
text-embedding-ada-002 (Legacy):
Batch Embedding Creation:
1async function getEmbeddings(texts) {
2 const response = await openai.embeddings.create({
3 model: "text-embedding-3-small",
4 input: texts // Array of strings (max 2048 per request)
5 });
6
7 // Returns embeddings in same order as input
8 return response.data.map(item => item.embedding);
9}
10
11// Usage
12const texts = [
13 "Artificial intelligence",
14 "Machine learning",
15 "Deep learning",
16 "Neural networks"
17];
18
19const embeddings = await getEmbeddings(texts);
20console.log(embeddings.length); // 4
21console.log(embeddings[0].length); // 1536Embedding Properties:
1. Dimensionality:
2. Similarity Measurement:
3. Consistency:
Cost Optimization:
1// Calculate tokens for cost estimation
2function estimateTokens(text) {
3 // Rough estimate: 1 token ≈ 4 characters
4 return Math.ceil(text.length / 4);
5}
6
7// Calculate cost
8function calculateEmbeddingCost(texts, model = "text-embedding-3-small") {
9 const totalTokens = texts.reduce((sum, text) =>
10 sum + estimateTokens(text), 0
11 );
12
13 const costPerMillion = model === "text-embedding-3-large"
14 ? 0.13
15 : 0.02;
16
17 return (totalTokens / 1_000_000) * costPerMillion;
18}
19
20// Usage
21const documents = ["doc1...", "doc2...", "doc3..."];
22const cost = calculateEmbeddingCost(documents);
23console.log(`Estimated cost: $${cost.toFixed(4)}`);Embedding Best Practices:
1. Text Preprocessing:
1function prepareTextForEmbedding(text) {
2 return text
3 .trim()
4 .replace(/\s+/g, ' ') // Normalize whitespace
5 .toLowerCase() // Optional: normalize case
6 .substring(0, 8000); // Limit length (safety)
7}2. Chunking Long Documents:
1function chunkDocument(text, maxTokens = 500) {
2 const sentences = text.match(/[^.!?]+[.!?]+/g) || [text];
3 const chunks = [];
4 let currentChunk = "";
5
6 for (const sentence of sentences) {
7 const tokens = estimateTokens(currentChunk + sentence);
8
9 if (tokens > maxTokens && currentChunk) {
10 chunks.push(currentChunk.trim());
11 currentChunk = sentence;
12 } else {
13 currentChunk += sentence;
14 }
15 }
16
17 if (currentChunk) {
18 chunks.push(currentChunk.trim());
19 }
20
21 return chunks;
22}3. Caching Embeddings:
1// Cache embeddings to avoid recomputation
2const embeddingCache = new Map();
3
4async function getEmbeddingCached(text) {
5 const cacheKey = text.trim().toLowerCase();
6
7 if (embeddingCache.has(cacheKey)) {
8 return embeddingCache.get(cacheKey);
9 }
10
11 const embedding = await getEmbedding(text);
12 embeddingCache.set(cacheKey, embedding);
13
14 return embedding;
15}Common Pitfalls:
❌ Don't: Embed entire documents without chunking
✅ Do: Split long documents into semantic chunks
❌ Don't: Recompute embeddings for same text
✅ Do: Cache and store embeddings
❌ Don't: Ignore token limits (8191 tokens max)
✅ Do: Validate and truncate input text
❌ Don't: Use embeddings for exact text matching
✅ Do: Use for semantic similarity and context
Cosine similarity measures how similar two embedding vectors are - the fundamental calculation for semantic search and recommendations.
What is Cosine Similarity?
Cosine similarity measures the angle between two vectors:
Why Cosine Similarity?
Unlike Euclidean distance, cosine similarity:
Implementation:
1// Calculate cosine similarity between two vectors
2function cosineSimilarity(vecA, vecB) {
3 // Dot product: sum of element-wise multiplication
4 const dotProduct = vecA.reduce((sum, a, i) =>
5 sum + a * vecB[i], 0
6 );
7
8 // Magnitudes: square root of sum of squares
9 const magnitudeA = Math.sqrt(
10 vecA.reduce((sum, a) => sum + a * a, 0)
11 );
12 const magnitudeB = Math.sqrt(
13 vecB.reduce((sum, b) => sum + b * b, 0)
14 );
15
16 // Cosine similarity = dot product / (mag_a * mag_b)
17 return dotProduct / (magnitudeA * magnitudeB);
18}
19
20// Usage
21const embedding1 = await getEmbedding("machine learning");
22const embedding2 = await getEmbedding("artificial intelligence");
23const embedding3 = await getEmbedding("banana recipe");
24
25console.log(cosineSimilarity(embedding1, embedding2)); // ~0.85 (similar)
26console.log(cosineSimilarity(embedding1, embedding3)); // ~0.45 (different)Optimized Version:
1// Optimized for many comparisons
2class EmbeddingComparator {
3 constructor(embedding) {
4 this.embedding = embedding;
5 this.magnitude = Math.sqrt(
6 embedding.reduce((sum, val) => sum + val * val, 0)
7 );
8 }
9
10 similarity(otherEmbedding) {
11 const dotProduct = this.embedding.reduce((sum, val, i) =>
12 sum + val * otherEmbedding[i], 0
13 );
14
15 const otherMagnitude = Math.sqrt(
16 otherEmbedding.reduce((sum, val) => sum + val * val, 0)
17 );
18
19 return dotProduct / (this.magnitude * otherMagnitude);
20 }
21}
22
23// Usage
24const queryComparator = new EmbeddingComparator(queryEmbedding);
25const similarities = documents.map(doc => ({
26 doc,
27 similarity: queryComparator.similarity(doc.embedding)
28}));Finding Similar Items:
1async function findSimilar(query, documents, topK = 5) {
2 // Get query embedding
3 const queryEmbedding = await getEmbedding(query);
4
5 // Calculate similarities
6 const similarities = documents.map(doc => ({
7 ...doc,
8 similarity: cosineSimilarity(queryEmbedding, doc.embedding)
9 }));
10
11 // Sort by similarity (highest first)
12 return similarities
13 .sort((a, b) => b.similarity - a.similarity)
14 .slice(0, topK);
15}
16
17// Usage
18const documents = [
19 { id: 1, text: "ML tutorial", embedding: [...] },
20 { id: 2, text: "Cooking recipes", embedding: [...] },
21 { id: 3, text: "AI overview", embedding: [...] }
22];
23
24const results = await findSimilar(
25 "learning artificial intelligence",
26 documents,
27 3
28);
29
30results.forEach(result => {
31 console.log(`${result.text}: ${result.similarity.toFixed(3)}`);
32});
33// AI overview: 0.892
34// ML tutorial: 0.876
35// Cooking recipes: 0.421Similarity Thresholds:
1function categorizeSimilarity(similarity) {
2 if (similarity >= 0.95) return "Nearly Identical";
3 if (similarity >= 0.90) return "Very Similar";
4 if (similarity >= 0.80) return "Similar";
5 if (similarity >= 0.70) return "Related";
6 if (similarity >= 0.60) return "Somewhat Related";
7 return "Different";
8}
9
10// Usage with filtering
11const results = await findSimilar(query, documents, 10);
12const relevant = results.filter(r => r.similarity >= 0.7);Alternative Distance Metrics:
Dot Product:
1function dotProduct(vecA, vecB) {
2 return vecA.reduce((sum, a, i) => sum + a * vecB[i], 0);
3}
4
5// Faster but requires normalized vectors
6// OpenAI embeddings are already normalizedEuclidean Distance:
1function euclideanDistance(vecA, vecB) {
2 const squaredDiffs = vecA.map((a, i) =>
3 Math.pow(a - vecB[i], 2)
4 );
5 return Math.sqrt(
6 squaredDiffs.reduce((sum, diff) => sum + diff, 0)
7 );
8}
9
10// Lower = more similar (opposite of cosine)
11// Less common for embeddingsBatch Similarity Calculations:
1// Calculate similarity matrix for all pairs
2function similarityMatrix(embeddings) {
3 const n = embeddings.length;
4 const matrix = Array(n).fill(null).map(() => Array(n));
5
6 for (let i = 0; i < n; i++) {
7 for (let j = i; j < n; j++) {
8 const sim = cosineSimilarity(embeddings[i], embeddings[j]);
9 matrix[i][j] = sim;
10 matrix[j][i] = sim; // Symmetric
11 }
12 }
13
14 return matrix;
15}Performance Optimization:
1// Use Float32Array for better performance
2function toFloat32Array(embedding) {
3 return new Float32Array(embedding);
4}
5
6// Optimized cosine similarity
7function cosineSimilarityFast(vecA, vecB) {
8 let dotProduct = 0;
9 let magA = 0;
10 let magB = 0;
11
12 for (let i = 0; i < vecA.length; i++) {
13 dotProduct += vecA[i] * vecB[i];
14 magA += vecA[i] * vecA[i];
15 magB += vecB[i] * vecB[i];
16 }
17
18 return dotProduct / (Math.sqrt(magA) * Math.sqrt(magB));
19}Best Practices:
1. Set Appropriate Thresholds:
2. Normalize When Needed:
3. Handle Edge Cases:
4. Optimize for Scale:
Semantic search understands meaning and context, not just keywords - enabling intelligent search that finds relevant content even with different wording.
Semantic Search vs Keyword Search:
Keyword Search (Traditional):
Semantic Search (AI-Powered):
Basic Semantic Search Implementation:
1class SemanticSearch {
2 constructor() {
3 this.documents = [];
4 this.embeddings = [];
5 }
6
7 // Index documents with embeddings
8 async indexDocuments(documents) {
9 console.log(`Indexing ${documents.length} documents...`);
10
11 // Get embeddings for all documents
12 const texts = documents.map(doc => doc.text);
13 this.embeddings = await getEmbeddings(texts);
14 this.documents = documents;
15
16 console.log("Indexing complete!");
17 }
18
19 // Search by query
20 async search(query, topK = 10, minSimilarity = 0.0) {
21 // Get query embedding
22 const queryEmbedding = await getEmbedding(query);
23
24 // Calculate similarities
25 const results = this.documents.map((doc, i) => ({
26 ...doc,
27 similarity: cosineSimilarity(
28 queryEmbedding,
29 this.embeddings[i]
30 )
31 }));
32
33 // Filter and sort
34 return results
35 .filter(r => r.similarity >= minSimilarity)
36 .sort((a, b) => b.similarity - a.similarity)
37 .slice(0, topK);
38 }
39
40 // Find similar documents to a given document
41 async findSimilar(documentId, topK = 5) {
42 const docIndex = this.documents.findIndex(d => d.id === documentId);
43 if (docIndex === -1) return [];
44
45 const docEmbedding = this.embeddings[docIndex];
46
47 const results = this.documents
48 .map((doc, i) => ({
49 ...doc,
50 similarity: cosineSimilarity(docEmbedding, this.embeddings[i])
51 }))
52 .filter(r => r.id !== documentId); // Exclude self
53
54 return results
55 .sort((a, b) => b.similarity - a.similarity)
56 .slice(0, topK);
57 }
58}
59
60// Usage
61const search = new SemanticSearch();
62
63// Index your documents
64await search.indexDocuments([
65 { id: 1, text: "Machine learning basics for beginners", category: "AI" },
66 { id: 2, text: "Introduction to artificial intelligence", category: "AI" },
67 { id: 3, text: "Baking chocolate chip cookies", category: "Cooking" },
68 { id: 4, text: "Neural networks explained simply", category: "AI" },
69 { id: 5, text: "Best pasta recipes", category: "Cooking" }
70]);
71
72// Search
73const results = await search.search("learning AI", 3);
74results.forEach(r => {
75 console.log(`[${r.similarity.toFixed(3)}] ${r.text}`);
76});
77// [0.912] Machine learning basics for beginners
78// [0.897] Introduction to artificial intelligence
79// [0.884] Neural networks explained simply
80
81// Find similar
82const similar = await search.findSimilar(1, 3);Production-Ready Semantic Search:
1class ProductionSemanticSearch {
2 constructor(options = {}) {
3 this.documents = [];
4 this.embeddings = [];
5 this.metadata = new Map();
6 this.options = {
7 chunkSize: 500,
8 minSimilarity: 0.6,
9 ...options
10 };
11 }
12
13 // Advanced indexing with metadata
14 async indexDocuments(documents) {
15 const chunks = [];
16
17 for (const doc of documents) {
18 // Split long documents
19 const docChunks = this.chunkDocument(doc.text);
20
21 for (let i = 0; i < docChunks.length; i++) {
22 const chunkId = `${doc.id}_${i}`;
23 chunks.push({
24 id: chunkId,
25 text: docChunks[i],
26 originalId: doc.id,
27 chunkIndex: i,
28 totalChunks: docChunks.length
29 });
30
31 // Store metadata
32 this.metadata.set(chunkId, {
33 ...doc,
34 text: docChunks[i],
35 chunkIndex: i
36 });
37 }
38 }
39
40 // Get embeddings
41 const texts = chunks.map(c => c.text);
42 this.embeddings = await this.getEmbeddingsBatch(texts);
43 this.documents = chunks;
44
45 return {
46 totalDocuments: documents.length,
47 totalChunks: chunks.length
48 };
49 }
50
51 // Batch embedding with rate limiting
52 async getEmbeddingsBatch(texts, batchSize = 100) {
53 const embeddings = [];
54
55 for (let i = 0; i < texts.length; i += batchSize) {
56 const batch = texts.slice(i, i + batchSize);
57 const batchEmbeddings = await getEmbeddings(batch);
58 embeddings.push(...batchEmbeddings);
59
60 // Rate limiting
61 if (i + batchSize < texts.length) {
62 await new Promise(resolve => setTimeout(resolve, 100));
63 }
64 }
65
66 return embeddings;
67 }
68
69 chunkDocument(text) {
70 const sentences = text.match(/[^.!?]+[.!?]+/g) || [text];
71 const chunks = [];
72 let current = "";
73
74 for (const sentence of sentences) {
75 if (current.length + sentence.length > this.options.chunkSize) {
76 if (current) chunks.push(current.trim());
77 current = sentence;
78 } else {
79 current += sentence;
80 }
81 }
82
83 if (current) chunks.push(current.trim());
84 return chunks;
85 }
86
87 // Advanced search with filters
88 async search(query, options = {}) {
89 const {
90 topK = 10,
91 minSimilarity = this.options.minSimilarity,
92 filters = {},
93 rerank = true
94 } = options;
95
96 const queryEmbedding = await getEmbedding(query);
97
98 // Calculate similarities
99 let results = this.documents.map((doc, i) => {
100 const metadata = this.metadata.get(doc.id);
101 return {
102 ...metadata,
103 similarity: cosineSimilarity(queryEmbedding, this.embeddings[i])
104 };
105 });
106
107 // Apply filters
108 if (Object.keys(filters).length > 0) {
109 results = results.filter(r => {
110 return Object.entries(filters).every(([key, value]) =>
111 r[key] === value
112 );
113 });
114 }
115
116 // Filter by similarity
117 results = results.filter(r => r.similarity >= minSimilarity);
118
119 // Group chunks from same document
120 if (rerank) {
121 results = this.rerankByDocument(results);
122 }
123
124 // Sort and limit
125 return results
126 .sort((a, b) => b.similarity - a.similarity)
127 .slice(0, topK);
128 }
129
130 // Rerank: combine chunk scores per document
131 rerankByDocument(results) {
132 const docScores = new Map();
133
134 for (const result of results) {
135 const docId = result.originalId || result.id;
136
137 if (!docScores.has(docId)) {
138 docScores.set(docId, {
139 ...result,
140 maxSimilarity: result.similarity,
141 avgSimilarity: result.similarity,
142 chunkCount: 1
143 });
144 } else {
145 const doc = docScores.get(docId);
146 doc.maxSimilarity = Math.max(doc.maxSimilarity, result.similarity);
147 doc.avgSimilarity = (
148 (doc.avgSimilarity * doc.chunkCount + result.similarity) /
149 (doc.chunkCount + 1)
150 );
151 doc.chunkCount++;
152 }
153 }
154
155 return Array.from(docScores.values()).map(doc => ({
156 ...doc,
157 similarity: doc.maxSimilarity * 0.7 + doc.avgSimilarity * 0.3
158 }));
159 }
160}
161
162// Usage
163const search = new ProductionSemanticSearch({
164 chunkSize: 500,
165 minSimilarity: 0.7
166});
167
168await search.indexDocuments(documents);
169
170const results = await search.search("machine learning tutorial", {
171 topK: 5,
172 filters: { category: "AI" },
173 rerank: true
174});Hybrid Search (Semantic + Keyword):
1class HybridSearch extends ProductionSemanticSearch {
2 // Combine semantic and keyword search
3 async hybridSearch(query, options = {}) {
4 const { semanticWeight = 0.7, keywordWeight = 0.3 } = options;
5
6 // Semantic search
7 const semanticResults = await this.search(query, {
8 ...options,
9 topK: options.topK * 2 // Get more for reranking
10 });
11
12 // Keyword search
13 const keywordResults = this.keywordSearch(query, options.topK * 2);
14
15 // Combine scores
16 const combined = new Map();
17
18 for (const result of semanticResults) {
19 combined.set(result.id, {
20 ...result,
21 score: result.similarity * semanticWeight
22 });
23 }
24
25 for (const result of keywordResults) {
26 if (combined.has(result.id)) {
27 const existing = combined.get(result.id);
28 existing.score += result.score * keywordWeight;
29 } else {
30 combined.set(result.id, {
31 ...result,
32 score: result.score * keywordWeight
33 });
34 }
35 }
36
37 return Array.from(combined.values())
38 .sort((a, b) => b.score - a.score)
39 .slice(0, options.topK);
40 }
41
42 keywordSearch(query, topK) {
43 const terms = query.toLowerCase().split(/\s+/);
44
45 return this.documents.map(doc => {
46 const text = doc.text.toLowerCase();
47 const score = terms.reduce((sum, term) => {
48 const matches = (text.match(new RegExp(term, 'g')) || []).length;
49 return sum + matches;
50 }, 0) / terms.length;
51
52 return { ...doc, score };
53 })
54 .filter(r => r.score > 0)
55 .sort((a, b) => b.score - a.score)
56 .slice(0, topK);
57 }
58}Best Practices:
1. Chunk Long Documents:
2. Use Metadata Filtering:
3. Implement Caching:
4. Monitor and Improve:
Vector databases are specialized databases optimized for storing and searching embeddings at scale - essential for production semantic search and recommendation systems.
Why Vector Databases?
Problems with In-Memory Search:
Vector Database Benefits:
Popular Vector Databases:
Pinecone (Managed Service):
Weaviate (Open Source):
Qdrant (Open Source):
Milvus (Open Source):
Chroma (Embedded):
Pinecone Implementation:
1import { Pinecone } from '@pinecone-database/pinecone';
2
3// Initialize Pinecone
4const pinecone = new Pinecone({
5 apiKey: process.env.PINECONE_API_KEY
6});
7
8// Create index (do once)
9async function createIndex(indexName) {
10 await pinecone.createIndex({
11 name: indexName,
12 dimension: 1536, // OpenAI embedding size
13 metric: 'cosine',
14 spec: {
15 serverless: {
16 cloud: 'aws',
17 region: 'us-east-1'
18 }
19 }
20 });
21}
22
23// Get index
24const index = pinecone.index('semantic-search');
25
26// Upsert vectors (insert/update)
27async function indexDocuments(documents) {
28 const vectors = [];
29
30 for (const doc of documents) {
31 const embedding = await getEmbedding(doc.text);
32
33 vectors.push({
34 id: doc.id.toString(),
35 values: embedding,
36 metadata: {
37 text: doc.text,
38 category: doc.category,
39 timestamp: Date.now()
40 }
41 });
42 }
43
44 // Upsert in batches
45 const batchSize = 100;
46 for (let i = 0; i < vectors.length; i += batchSize) {
47 const batch = vectors.slice(i, i + batchSize);
48 await index.upsert(batch);
49 }
50
51 console.log(`Indexed ${vectors.length} documents`);
52}
53
54// Search
55async function search(query, topK = 10, filter = {}) {
56 const queryEmbedding = await getEmbedding(query);
57
58 const results = await index.query({
59 vector: queryEmbedding,
60 topK,
61 includeMetadata: true,
62 filter: Object.keys(filter).length > 0 ? filter : undefined
63 });
64
65 return results.matches.map(match => ({
66 id: match.id,
67 score: match.score,
68 ...match.metadata
69 }));
70}
71
72// Usage
73const results = await search("machine learning tutorial", 5, {
74 category: { $eq: "AI" }
75});
76
77results.forEach(r => {
78 console.log(`[${r.score.toFixed(3)}] ${r.text}`);
79});Qdrant Implementation:
1import { QdrantClient } from '@qdrant/js-client-rest';
2
3const client = new QdrantClient({
4 url: 'http://localhost:6333'
5});
6
7// Create collection
8async function createCollection(collectionName) {
9 await client.createCollection(collectionName, {
10 vectors: {
11 size: 1536,
12 distance: 'Cosine'
13 }
14 });
15}
16
17// Index documents
18async function indexDocuments(collectionName, documents) {
19 const points = [];
20
21 for (const doc of documents) {
22 const embedding = await getEmbedding(doc.text);
23
24 points.push({
25 id: doc.id,
26 vector: embedding,
27 payload: {
28 text: doc.text,
29 category: doc.category,
30 timestamp: Date.now()
31 }
32 });
33 }
34
35 await client.upsert(collectionName, {
36 wait: true,
37 points
38 });
39}
40
41// Search with filtering
42async function search(collectionName, query, limit = 10) {
43 const queryEmbedding = await getEmbedding(query);
44
45 const results = await client.search(collectionName, {
46 vector: queryEmbedding,
47 limit,
48 filter: {
49 must: [
50 {
51 key: 'category',
52 match: { value: 'AI' }
53 }
54 ]
55 }
56 });
57
58 return results.map(r => ({
59 id: r.id,
60 score: r.score,
61 ...r.payload
62 }));
63}Chroma (Embedded Database):
1import { ChromaClient } from 'chromadb';
2
3const client = new ChromaClient();
4
5// Create collection
6const collection = await client.createCollection({
7 name: "semantic_search",
8 metadata: { "hnsw:space": "cosine" }
9});
10
11// Add documents
12async function addDocuments(documents) {
13 const embeddings = await getEmbeddings(
14 documents.map(d => d.text)
15 );
16
17 await collection.add({
18 ids: documents.map(d => d.id.toString()),
19 embeddings,
20 metadatas: documents.map(d => ({
21 text: d.text,
22 category: d.category
23 }))
24 });
25}
26
27// Query
28async function search(query, topK = 10) {
29 const queryEmbedding = await getEmbedding(query);
30
31 const results = await collection.query({
32 queryEmbeddings: [queryEmbedding],
33 nResults: topK
34 });
35
36 return results.ids[0].map((id, i) => ({
37 id,
38 distance: results.distances[0][i],
39 ...results.metadatas[0][i]
40 }));
41}Production Architecture:
1User Query
2 ↓
3API Server
4 ↓
5Generate Query Embedding (OpenAI)
6 ↓
7Vector Database Search (Pinecone/Qdrant)
8 ↓
9Apply Filters & Reranking
10 ↓
11Format Results
12 ↓
13Return to UserComplete Production Example:
1class ProductionSemanticSearchService {
2 constructor(vectorDb, embeddingService) {
3 this.vectorDb = vectorDb;
4 this.embeddingService = embeddingService;
5 }
6
7 async indexDocument(document) {
8 // Chunk if needed
9 const chunks = this.chunkDocument(document.text);
10
11 for (let i = 0; i < chunks.length; i++) {
12 const embedding = await this.embeddingService.getEmbedding(chunks[i]);
13
14 await this.vectorDb.upsert({
15 id: `${document.id}_${i}`,
16 vector: embedding,
17 metadata: {
18 documentId: document.id,
19 text: chunks[i],
20 chunkIndex: i,
21 totalChunks: chunks.length,
22 ...document.metadata
23 }
24 });
25 }
26 }
27
28 async search(query, options = {}) {
29 const {
30 topK = 10,
31 filters = {},
32 rerank = true
33 } = options;
34
35 // Get query embedding
36 const queryEmbedding = await this.embeddingService.getEmbedding(query);
37
38 // Search vector database
39 const results = await this.vectorDb.search({
40 vector: queryEmbedding,
41 topK: topK * 2, // Get more for reranking
42 filters
43 });
44
45 // Rerank if needed
46 let finalResults = results;
47 if (rerank) {
48 finalResults = this.rerankResults(results);
49 }
50
51 return finalResults.slice(0, topK);
52 }
53
54 async findSimilar(documentId, topK = 5) {
55 // Get document embedding
56 const doc = await this.vectorDb.fetch(documentId);
57
58 // Search with document's embedding
59 return await this.vectorDb.search({
60 vector: doc.vector,
61 topK: topK + 1, // +1 to exclude self
62 filters: { documentId: { $ne: documentId } }
63 });
64 }
65
66 chunkDocument(text, chunkSize = 500) {
67 // Implementation from previous examples
68 }
69
70 rerankResults(results) {
71 // Group by document and combine scores
72 const docGroups = new Map();
73
74 for (const result of results) {
75 const docId = result.metadata.documentId;
76
77 if (!docGroups.has(docId)) {
78 docGroups.set(docId, []);
79 }
80
81 docGroups.get(docId).push(result);
82 }
83
84 // Calculate combined score per document
85 return Array.from(docGroups.values()).map(chunks => {
86 const maxScore = Math.max(...chunks.map(c => c.score));
87 const avgScore = chunks.reduce((sum, c) => sum + c.score, 0) / chunks.length;
88
89 return {
90 ...chunks[0], // Use first chunk's metadata
91 score: maxScore * 0.7 + avgScore * 0.3,
92 matchedChunks: chunks.length
93 };
94 });
95 }
96}Best Practices:
1. Choose Right Database:
2. Optimize Indexing:
3. Design for Scale:
4. Handle Failures:
A production-ready semantic search system with document indexing, intelligent search, similarity detection, and recommendation features. Demonstrates embeddings, cosine similarity, and vector search in action.
1<!DOCTYPE html>
2<html lang="en">
3<head>
4 <meta charset="UTF-8">
5 <meta name="viewport" content="width=device-width, initial-scale=1.0">
6 <title>Semantic Search Engine</title>
7</head>
8<body>
9 <div class="container">
10 <header class="hero">
11 <h1>🔍 Semantic Search Engine</h1>
12 <p class="subtitle">Intelligent search powered by vector embeddings and AI</p>
13 </header>
14
15 <!-- Search Interface -->
16 <section class="card">
17 <h2>🎯 Semantic Search</h2>
18 <p>Search by meaning, not just keywords - finds relevant content even with different wording</p>
19
20 <div class="search-box">
21 <input
22 type="text"
23 id="search-input"
24 placeholder="Search for anything... (e.g., 'learning artificial intelligence')"
25 class="search-input"
26 >
27 <button class="btn btn-primary" onclick="performSearch()">
28 🔍 Search
29 </button>
30 </div>
31
32 <div class="search-options">
33 <div class="option-group">
34 <label>Results:</label>
35 <select id="result-count">
36 <option value="5">Top 5</option>
37 <option value="10" selected>Top 10</option>
38 <option value="20">Top 20</option>
39 </select>
40 </div>
41
42 <div class="option-group">
43 <label>Minimum Similarity:</label>
44 <select id="min-similarity">
45 <option value="0.5">50% (Loose)</option>
46 <option value="0.6">60%</option>
47 <option value="0.7" selected>70% (Balanced)</option>
48 <option value="0.8">80% (Strict)</option>
49 <option value="0.9">90% (Very Strict)</option>
50 </select>
51 </div>
52
53 <div class="option-group">
54 <label>Category Filter:</label>
55 <select id="category-filter">
56 <option value="">All Categories</option>
57 <option value="AI">AI & Machine Learning</option>
58 <option value="Web Dev">Web Development</option>
59 <option value="Data Science">Data Science</option>
60 <option value="DevOps">DevOps</option>
61 </select>
62 </div>
63 </div>
64
65 <div id="search-results" class="results-container"></div>
66 </section>
67
68 <!-- Document Similarity -->
69 <section class="card">
70 <h2>🔗 Find Similar Content</h2>
71 <p>Discover content related to any document using embedding similarity</p>
72
73 <div class="form-group">
74 <label>Select a document:</label>
75 <select id="similarity-doc" class="full-width">
76 <option value="">Choose a document...</option>
77 <option value="1">Machine Learning Basics</option>
78 <option value="2">Introduction to Neural Networks</option>
79 <option value="3">Web Development with React</option>
80 <option value="4">Data Analysis with Python</option>
81 <option value="5">Docker and Containerization</option>
82 </select>
83 </div>
84
85 <button class="btn btn-primary" onclick="findSimilarDocs()">
86 Find Similar Documents
87 </button>
88
89 <div id="similarity-results" class="results-container hidden"></div>
90 </section>
91
92 <!-- Sample Documents Database -->
93 <section class="card">
94 <h2>📚 Document Database (Demo)</h2>
95 <p>Sample documents with embeddings for semantic search</p>
96
97 <div class="stats-row">
98 <div class="stat-box">
99 <div class="stat-number">25</div>
100 <div class="stat-label">Documents Indexed</div>
101 </div>
102 <div class="stat-box">
103 <div class="stat-number">1536</div>
104 <div class="stat-label">Embedding Dimensions</div>
105 </div>
106 <div class="stat-box">
107 <div class="stat-number">0.85</div>
108 <div class="stat-label">Avg Similarity Score</div>
109 </div>
110 </div>
111
112 <div class="documents-preview">
113 <h3>Sample Documents:</h3>
114 <div class="document-list">
115 <div class="doc-item">
116 <span class="doc-icon">📄</span>
117 <div class="doc-info">
118 <div class="doc-title">Machine Learning Fundamentals</div>
119 <div class="doc-category">AI & Machine Learning</div>
120 </div>
121 </div>
122 <div class="doc-item">
123 <span class="doc-icon">📄</span>
124 <div class="doc-info">
125 <div class="doc-title">Building Neural Networks from Scratch</div>
126 <div class="doc-category">AI & Machine Learning</div>
127 </div>
128 </div>
129 <div class="doc-item">
130 <span class="doc-icon">📄</span>
131 <div class="doc-info">
132 <div class="doc-title">React Hooks and Modern Patterns</div>
133 <div class="doc-category">Web Development</div>
134 </div>
135 </div>
136 <div class="doc-item">
137 <span class="doc-icon">📄</span>
138 <div class="doc-info">
139 <div class="doc-title">Data Visualization with D3.js</div>
140 <div class="doc-category">Data Science</div>
141 </div>
142 </div>
143 <div class="doc-item">
144 <span class="doc-icon">📄</span>
145 <div class="doc-info">
146 <div class="doc-title">Kubernetes Deployment Strategies</div>
147 <div class="doc-category">DevOps</div>
148 </div>
149 </div>
150 </div>
151 </div>
152 </section>
153
154 <!-- How It Works -->
155 <section class="card">
156 <h2>⚙️ How Semantic Search Works</h2>
157
158 <div class="process-steps">
159 <div class="step">
160 <div class="step-number">1</div>
161 <div class="step-content">
162 <h3>Create Embeddings</h3>
163 <p>Convert documents and queries into 1536-dimensional vectors</p>
164 </div>
165 </div>
166
167 <div class="step-arrow">↓</div>
168
169 <div class="step">
170 <div class="step-number">2</div>
171 <div class="step-content">
172 <h3>Calculate Similarity</h3>
173 <p>Use cosine similarity to measure how related vectors are</p>
174 </div>
175 </div>
176
177 <div class="step-arrow">↓</div>
178
179 <div class="step">
180 <div class="step-number">3</div>
181 <div class="step-content">
182 <h3>Rank Results</h3>
183 <p>Sort by similarity score and return most relevant results</p>
184 </div>
185 </div>
186 </div>
187 </section>
188
189 <!-- Comparison: Keyword vs Semantic -->
190 <section class="card">
191 <h2>🆚 Keyword Search vs Semantic Search</h2>
192
193 <div class="comparison-grid">
194 <div class="comparison-card keyword">
195 <h3>🔤 Keyword Search</h3>
196 <ul class="comparison-list">
197 <li class="negative">Exact word matching only</li>
198 <li class="negative">Misses synonyms</li>
199 <li class="negative">No context understanding</li>
200 <li class="negative">Query: "car" ≠ "automobile"</li>
201 </ul>
202 <div class="example">
203 <strong>Example:</strong>
204 <div>Query: "machine learning"</div>
205 <div>❌ Won't find: "AI algorithms", "neural networks"</div>
206 </div>
207 </div>
208
209 <div class="comparison-card semantic">
210 <h3>🧠 Semantic Search</h3>
211 <ul class="comparison-list">
212 <li class="positive">Understands meaning</li>
213 <li class="positive">Finds synonyms</li>
214 <li class="positive">Context-aware</li>
215 <li class="positive">Query: "car" = "automobile"</li>
216 </ul>
217 <div class="example">
218 <strong>Example:</strong>
219 <div>Query: "machine learning"</div>
220 <div>✅ Also finds: "AI algorithms", "neural networks"</div>
221 </div>
222 </div>
223 </div>
224 </section>
225
226 <!-- Example Queries -->
227 <section class="card">
228 <h2>💡 Try These Example Queries</h2>
229 <div class="examples-grid">
230 <button class="example-btn" onclick="searchExample('learning artificial intelligence')">
231 "learning artificial intelligence"
232 </button>
233 <button class="example-btn" onclick="searchExample('building web applications')">
234 "building web applications"
235 </button>
236 <button class="example-btn" onclick="searchExample('analyzing data')">
237 "analyzing data"
238 </button>
239 <button class="example-btn" onclick="searchExample('deploying applications')">
240 "deploying applications"
241 </button>
242 <button class="example-btn" onclick="searchExample('neural network architecture')">
243 "neural network architecture"
244 </button>
245 <button class="example-btn" onclick="searchExample('frontend frameworks')">
246 "frontend frameworks"
247 </button>
248 </div>
249 </section>
250
251 <!-- Implementation Code -->
252 <section class="card">
253 <h2>💻 Implementation Example</h2>
254
255 <div class="code-tabs">
256 <button class="code-tab active" onclick="showCode('embeddings')">Create Embeddings</button>
257 <button class="code-tab" onclick="showCode('similarity')">Calculate Similarity</button>
258 <button class="code-tab" onclick="showCode('search')">Semantic Search</button>
259 </div>
260
261 <div id="code-embeddings" class="code-panel">
262 <pre class="code-block">
263// Create embeddings for documents
264async function getEmbedding(text) {
265 const response = await openai.embeddings.create({
266 model: "text-embedding-3-small",
267 input: text,
268 encoding_format: "float"
269 });
270
271 return response.data[0].embedding;
272}
273
274// Batch processing
275async function getEmbeddings(texts) {
276 const response = await openai.embeddings.create({
277 model: "text-embedding-3-small",
278 input: texts // Array of strings
279 });
280
281 return response.data.map(item => item.embedding);
282}
283
284// Usage
285const embedding = await getEmbedding("Machine learning");
286console.log(embedding.length); // 1536 dimensions
287 </pre>
288 </div>
289
290 <div id="code-similarity" class="code-panel hidden">
291 <pre class="code-block">
292// Calculate cosine similarity
293function cosineSimilarity(vecA, vecB) {
294 let dotProduct = 0;
295 let magA = 0;
296 let magB = 0;
297
298 for (let i = 0; i < vecA.length; i++) {
299 dotProduct += vecA[i] * vecB[i];
300 magA += vecA[i] * vecA[i];
301 magB += vecB[i] * vecB[i];
302 }
303
304 return dotProduct / (Math.sqrt(magA) * Math.sqrt(magB));
305}
306
307// Usage
308const similarity = cosineSimilarity(embedding1, embedding2);
309console.log(similarity); // 0.0 to 1.0
310// 0.9+ = very similar
311// 0.7-0.9 = related
312// < 0.7 = different
313 </pre>
314 </div>
315
316 <div id="code-search" class="code-panel hidden">
317 <pre class="code-block">
318// Semantic search implementation
319async function semanticSearch(query, documents, topK = 10) {
320 // Get query embedding
321 const queryEmbedding = await getEmbedding(query);
322
323 // Calculate similarities
324 const results = documents.map(doc => ({
325 ...doc,
326 similarity: cosineSimilarity(
327 queryEmbedding,
328 doc.embedding
329 )
330 }));
331
332 // Sort by similarity and return top K
333 return results
334 .sort((a, b) => b.similarity - a.similarity)
335 .slice(0, topK);
336}
337
338// Usage
339const results = await semanticSearch(
340 "machine learning tutorial",
341 documents,
342 5
343);
344 </pre>
345 </div>
346 </section>
347
348 <!-- Best Practices -->
349 <section class="card tips-section">
350 <h2>🎯 Best Practices</h2>
351 <div class="tips-grid">
352 <div class="tip-card">
353 <h3>📊 Use Right Model</h3>
354 <p>text-embedding-3-small for most cases, 3-large for high precision needs</p>
355 </div>
356 <div class="tip-card">
357 <h3>✂️ Chunk Documents</h3>
358 <p>Split long documents into 300-500 token chunks for better results</p>
359 </div>
360 <div class="tip-card">
361 <h3>💾 Cache Embeddings</h3>
362 <p>Store embeddings in database - don't recompute for same text</p>
363 </div>
364 <div class="tip-card">
365 <h3>🎯 Set Thresholds</h3>
366 <p>Use similarity thresholds (0.7+) to filter irrelevant results</p>
367 </div>
368 <div class="tip-card">
369 <h3>🔄 Batch Process</h3>
370 <p>Create embeddings in batches (up to 2048 texts) for efficiency</p>
371 </div>
372 <div class="tip-card">
373 <h3>🗄️ Use Vector DB</h3>
374 <p>Use Pinecone/Qdrant for >10K documents - much faster than linear search</p>
375 </div>
376 </div>
377 </section>
378
379 <!-- Use Cases -->
380 <section class="card">
381 <h2>🎯 Real-World Use Cases</h2>
382 <div class="use-cases-grid">
383 <div class="use-case-card">
384 <div class="use-case-icon">🔍</div>
385 <h3>Intelligent Search</h3>
386 <p>Search documentation, articles, products by meaning</p>
387 </div>
388 <div class="use-case-card">
389 <div class="use-case-icon">⭐</div>
390 <h3>Recommendations</h3>
391 <p>Suggest similar products, articles, videos</p>
392 </div>
393 <div class="use-case-card">
394 <div class="use-case-icon">🔗</div>
395 <h3>Duplicate Detection</h3>
396 <p>Find duplicate or near-duplicate content</p>
397 </div>
398 <div class="use-case-card">
399 <div class="use-case-icon">🏷️</div>
400 <h3>Content Clustering</h3>
401 <p>Automatically group related content</p>
402 </div>
403 <div class="use-case-card">
404 <div class="use-case-icon">❓</div>
405 <h3>Q&A Systems</h3>
406 <p>Find relevant answers to questions</p>
407 </div>
408 <div class="use-case-card">
409 <div class="use-case-icon">🎨</div>
410 <h3>Multi-Modal Search</h3>
411 <p>Search images, audio, video by description</p>
412 </div>
413 </div>
414 </section>
415 </div>
416</body>
417</html>This semantic search engine demonstrates the power of embeddings and vector similarity:
Core Concepts Demonstrated:
1. Vector Embeddings:
2. Cosine Similarity:
3. Semantic Search:
4. Similarity Detection:
Features:
Search Capabilities:
Document Management:
Results Display:
Educational Components:
Production Patterns:
Optimization:
Scalability:
Quality:
This demo shows how embeddings power modern search and recommendation systems!
Excellent work mastering embeddings and semantic search! You now understand vector representations, similarity calculations, and how to build intelligent search systems. In our next lesson, "Computer Vision with AI APIs," you'll learn to analyze images, extract text with OCR, detect objects, and generate image descriptions using AI vision capabilities. Get ready to give your applications the ability to see and understand images!
Your progress is saved in this browser, on this device only.