Claude Prompt to Design a Vector Database Schema for RAG

You are a vector database architect. Design an optimized schema and indexing strategy for my embedding-based application.

**Application Context:**
- Use case: RAG system for internal knowledge base — search across documentation, Slack messages, Confluence pages, and code comments
- Data types: Technical documentation (Markdown), Slack messages (short text), code files (Python/JS), PDF reports
- Scale: 2M chunks currently, growing 100K/month, average chunk size 500 tokens
- Query patterns: Natural language questions, code similarity search, filtered by team/project, date-range constrained
- Latency requirement: Under 200ms for top-10 results including metadata filtering
- Vector database: Pinecone (considering Qdrant or Weaviate as alternatives)

**Design These Components:**

**1. Embedding Model Selection:**

| Content Type | Recommended Model | Dimensions | Why | Cost |
|-------------|------------------|------------|-----|------|
| [Type 1] | [Model] | [Dims] | [Reason] | [Cost] |
| [Type 2] | [Model] | [Dims] | [Reason] | [Cost] |

Considerations:
- Embedding dimensions vs. retrieval quality trade-off
- Multilingual support needs
- Code vs. natural language embedding models
- Need to handle code snippets mixed with natural language descriptions effectively

**2. Chunking Strategy:**
For Technical documentation (Markdown), Slack messages (short text), code files (Python/JS), PDF reports, recommend chunking approach:

| Content Type | Chunk Method | Chunk Size | Overlap | Why |
|-------------|-------------|-----------|---------|-----|
| Long documents | [Method] | [Size] | [Overlap] | [Reason] |
| Code files | [Method] | [Size] | [Overlap] | [Reason] |
| Short content | [Method] | [Size] | [Overlap] | [Reason] |

Chunking methods to evaluate:
- **Fixed-size:** Simple, predictable — good baseline
- **Sentence-based:** Natural boundaries — better retrieval
- **Semantic:** Split on topic changes — best quality, highest cost
- **Recursive:** Try larger chunks first, split only if too big
- **Parent-child:** Store small chunks for retrieval, return larger context

For each method, show: input example → chunked output example

**3. Schema Design:**
Design the collection/index schema for Pinecone (considering Qdrant or Weaviate as alternatives):

```
Collection: [Name]
├── Vector field: embedding ([Dims] dimensions, [Distance Metric])
├── Metadata fields:
│   ├── source_id (filterable) — reference to source document
│   ├── content_type (filterable) — Technical documentation (Markdown), Slack messages (short text), code files (Python/JS), PDF reports classification
│   ├── created_at (filterable) — for freshness ranking
│   ├── [Custom Field 1] (filterable)
│   ├── [Custom Field 2] (filterable)
│   └── chunk_text (stored, not indexed) — original text
├── Index configuration:
│   ├── Type: [HNSW/IVF/PQ] with parameters
│   └── Distance metric: [cosine/euclidean/dot_product]
```

**4. Index Optimization:**

| Index Type | Build Time | Query Latency | Memory | Recall@10 | Best For |
|-----------|-----------|---------------|--------|-----------|---------|
| HNSW | Slow | Fast | High | 95%+ | [use case] |
| IVF-PQ | Fast | Medium | Low | 85-90% | [use case] |
| Flat | None | Slow | Highest | 100% | [use case] |

Recommended for YOUR scale (2M chunks currently, growing 100K/month, average chunk size 500 tokens):
- Index type: [Recommendation] with these parameters: [Params]
- Build process: [batch indexing strategy]
- Reindexing strategy: [when and how to rebuild]

**5. Query Optimization:**
For each Natural language questions, code similarity search, filtered by team/project, date-range constrained:

```python
# Pattern: [Description]
# Optimization: [what makes this fast]
[query code with filters, top_k, and score threshold]
```

Advanced techniques:
- **Hybrid search:** Combine vector similarity with keyword (BM25) search
- **Metadata pre-filtering** vs. **post-filtering** (which is faster for your data)
- **Reranking:** Use a cross-encoder to rerank top-N results for higher precision
- **Multi-vector queries:** Query with multiple embeddings for better recall
- **Contextual retrieval — prepend document context to each chunk before embedding for better retrieval**

**6. Operational Concerns:**
- Backup and disaster recovery for vector data
- Embedding versioning: what happens when you change the embedding model?
- Index compaction and maintenance schedule
- Cost projection at 2M chunks currently, growing 100K/month, average chunk size 500 tokens for Pinecone (considering Qdrant or Weaviate as alternatives)
- Monitoring: query latency, recall quality, index health