Do You Actually Need a Vector Database?

Someone on the support team searches the company archive for "return policy". Nothing comes back. The document is sitting right there, except it is titled "conditions for sending goods back". Same meaning, different words, and the search box has no idea.

This scene plays out in almost every mid-sized company, and the usual workaround is that people stop searching and start asking each other. Institutional knowledge migrates out of the search box and into hallway conversations.

A vector database exists to fix exactly this. But before you go shopping for one, the more useful question is whether you need a separate system at all, because for most teams the honest answer is no. This piece walks through what these systems do, where the threshold sits, what they cost, and what actually determines search quality. For the wider picture, our guide to AI infrastructure decisions maps how this fits alongside everything else.

What is a vector database, and how is it different from normal search?

Conventional search matches character strings: type a word, and it finds documents containing that word. Vector search matches meaning instead. Text is first converted into a list of numbers called an embedding, and passages with similar meaning end up close together in that numeric space. A vector database stores those lists and answers one question quickly: which records sit nearest to this one in meaning?

So "return policy" and "conditions for sending goods back" land near each other, and the search finds what the user meant rather than what they typed.

There is an important counterweight, though. For exact matches, conventional search is still better. When someone searches for a product code, an error number, or the precise wording of a contract clause, semantic similarity actively works against you; you want a literal match. This is why mature systems run both together, which we come back to below.

We already run PostgreSQL. Do we need a separate database?

Almost certainly not. The pgvector extension lets PostgreSQL store and search embeddings inside the database you already operate. Your document, its metadata, and its vector live in the same row and can be queried together. You are not adding a system; you are adding an extension to one that already exists.

The argument for this is stronger than price. A separate vector store creates a permanent consistency problem between two systems. If one write succeeds and the other fails, you are left with either a document that has no vector or an orphaned vector whose document was deleted. Fixing that means writing a reconciliation job, and that job needs maintenance of its own. With pgvector, the document and its vector are written as a single record and the problem never arises.

The real cost of a dedicated vector database does not appear in the subscription line. It appears in the standing obligation to keep two systems in sync, and that obligation never shows up on a monthly invoice.

How many documents before a dedicated system pays off?

The honest answer is that there is no single threshold, and published sources disagree with each other. Thinking in three bands works better in practice. Up to roughly one million vectors, do not even open the discussion; pgvector handles it comfortably. Between one and ten million you are in genuinely ambiguous territory. Above ten million, start seriously evaluating a dedicated system.

Inside that ambiguous band, vector count is not what decides it. Two other factors do. The first is concurrent query volume, because a few hundred searches a day and dozens per second are entirely different problems. The second is how heavily you filter on metadata: once queries routinely look like "search only this department's approved documents from the last two years", purpose-built engines start pulling ahead.

When scale does force the move, Qdrant and Pinecone are the common destinations. Qdrant is a practical choice if you want to run it yourself; Pinecone takes the operational burden off entirely as a managed service. Systems built for billion-vector workloads exist, but they are not weight you can carry without a dedicated infrastructure team.

What does it cost?

Separating the line items matters here, because the one companies assume is expensive usually turns out to be the cheapest.

Generating embeddings is nearly free. Widely used small embedding models are priced in cents per million tokens. Picture an archive of ten thousand pages: that is somewhere in the region of ten to fifteen million tokens. Converting the entire thing costs well under a dollar. Processing every document your company has accumulated over years is a one-time expense smaller than lunch.

The money goes to storage and queries. On managed services, a million vectors starts in the tens of dollars a month and fifty million can reach the thousands. Self-hosted, the calculation runs on memory: budget a few gigabytes of RAM per hundred thousand to million vectors. A deployment holding a few million vectors runs comfortably on a server in the $30 to $100 a month range.

Treat all of these as orders of magnitude rather than quotes. Most managed vendors price on usage, and predicting a monthly bill in advance is genuinely difficult. Run your own query volume estimate through the provider's calculator before committing. And factor in something people forget: when a search box starts working well, people use it more. The query volume you budget for in month one may have doubled by month six, and on usage-based pricing that lands directly on the invoice.

Why is vector search alone not enough?

Because a meaningful share of users are not searching for meaning at all. They are searching for an exact string. In one production evaluation, pure vector search returned the right result in the top ten 78% of the time, pure keyword search managed 65%, and combining the two reached 91%. The gap exists because each method fails on a different kind of question.

Those figures come from a single source and are not an academic benchmark, so hold them loosely. The direction, though, matches what we see in the field: a substantial portion of support queries contain an error code, a product identifier, or a specific clause reference, and pure vector search is weakest on exactly those.

Combining the two means merging two result lists. The rookie mistake is to add the scores together and average them. The two methods produce scores on incompatible scales, so that merge produces noise. The correct approach merges the rankings rather than the scores, and the standard technique for it is a simple rank-fusion method known as RRF.

Chunking: the invisible decision that determines your search quality

Long documents do not go into the database whole; they are split into pieces first. This splitting is called chunking, and it influences search quality more than your choice of database does.

The reason is that a vector represents the average meaning of the text behind it. Store a forty-page policy document as a single chunk and its vector sits slightly far from every specific question, close to none of them. Split it too finely and context disappears, so the paragraph you retrieve makes no sense on its own.

Consistency matters as well. If half your archive is in two-hundred-word chunks and half is in five-thousand-word documents, the keyword side has to handle two very different length distributions at once and accuracy suffers. Keeping chunk sizes consistent is the cheapest quality improvement available, and it requires buying nothing.

Common questions

How is this different from a normal database?

A normal database answers "return the records equal to this value". A vector database answers "return the records closest in meaning to this one". The first produces exact matches, the second produces a ranked list by similarity. Both can hold the same record; they differ in the kind of question they serve.

Does document count tell me when to upgrade?

Not reliably, because documents get split before storage. A hundred-page document can become hundreds of vectors. As a rough guide, a ten-thousand-page archive produces a few hundred thousand vectors, which is an easy load for pgvector. Revisit the question when you reach the millions.

Does our data leave the building?

That depends on embedding generation rather than the database. Even if you run the vector store on your own hardware, sending text to an external service to convert it into numbers means the data has left. For documents that must stay entirely in-house, you need to run the embedding model in-house too.

We already have a search engine. Should we replace it?

No, and keeping it is the better move. Your existing keyword search is already half of a hybrid setup. The work is to add a semantic layer beside it and merge the two result lists, which is cheaper than a replacement and avoids regressions on the queries your current search handles well. Teams that switch off keyword search in favour of pure vector search tend to reverse the decision within weeks, once users searching for product codes start complaining.

How is this different from uploading files to an assistant?

For a handful of files there is no difference, and uploading is more convenient. The difference emerges as the archive grows: you cannot re-upload tens of thousands of documents on every question. A vector database processes that archive once and retrieves only the relevant fragments each time.

Four mistakes we see in the field

• Treating the database choice as the whole job. Quality is determined by chunking, hybrid retrieval, and reranking when needed. The vendor name is a secondary decision next to those three.
• Building the index on day one. On small tables, unindexed search already returns in milliseconds. Adding an HNSW index when queries slow down beats guessing at parameters upfront.
• Adding reranking too early. Models that reorder results do improve precision, but they add a few hundred milliseconds per query and cost substantially more per call. Get hybrid search working, measure, then decide.
• Never building an evaluation set. Without a short list of fifty real questions paired with the documents that correctly answer them, you cannot tell whether any change helped. That list takes an afternoon to assemble and stays useful for months.

What should you do?

• Count your scale before anything else. How many documents, how many pages, how many searches a day? Under a million vectors, close the tooling debate and start with pgvector.
• Build on the database you already run. If PostgreSQL is in production, try the extension before paying the operational price of a second system.
• Plan for hybrid retrieval from the start. Most projects that launch with pure vector search add a keyword layer back within a few months. Building it in initially costs less.
• Spend real time on chunking. How you split your documents matters more than which database stores them.
• Know where the data goes. If you send text to an external service to generate embeddings, it has crossed a boundary. For sensitive documents that becomes a legal question rather than a technical one.

A vector database is a genuine step forward for corporate search, moving it from matching words to matching intent. It is also a building block rather than a finished solution, and put in the right place it changes something small but valuable: knowledge stops living in the hallway and starts being findable. If you want to see what gets built on top of this layer, our piece on how RAG works and what it costs picks up where this one ends.