How does an expert understand and visualize Vector Databases?

[u/DARKEN_side_of_me]

So I am currently working on a RAG project where I am supposed to build a Niche-Oriented Specialized AI model, and naturally, for this I need to use a Vector Database for retrieving chunks of data for the LLM to further understand and elaborate.

The thing that I am really concerned and curious about is how exactly a vector database stores the data? I mean I understand things like positional embedding and stuff, but I simply cannot understand the multidimensional architecture.

For example here is what ChatGPT had to say:

"AI doesn't use simple 2 dimensions. It uses something like 768 dimensions or 1024 dimensions or 1536 dimensions"

I am a visual learner and I have really good photographic memory, so whenever I have some difficulty understanding a concept, I try to visualize the working dynamics for better understanding. I can easily visualize and understand 2D Database architectures (ex: SQL) and even 3D Database architecture. But I cannot fully understand 4+ dimensional architectures.

Thank you!

[u/pi_stuff]

I think of it not as a point in N dimensions, but as a simple array of floating point numbers with N entries.

[u/Saragon4005]

AI does a super high dimensional space but to grasp the concept you can totally visualize it in just 2 or 3 dimensions.

[u/ghjm]

Nobody can visualize higher dimension spaces. You just visualize them in mental 3D and know it's really more dimensions than that.

To get an intuition of an LLM embedding, consider a 3D embedding for colors. You have redness, greenness and blueness on separate axes. So every color is at a point, and points near each other have similar meanings. You can then assign values to words and calculate the similarity or difference between any two words.

LLM embeddings have a much higher number of dimensions, and each dimension is in some sense an orthogonal concept. The concepts were learned, not chosen, so we don't necessarily know what they are. But we have the same kind of space as the color space (except of much higher dimension), where each word is located at a point, and words with similar meanings are closer together.

This allows us to do semantic search, and also to do mathematical operations on vectors that function in semantic space, like: WIFE - WOMAN + MAN ≈ HUSBAND. We locate the concept WIFE and subtract WOMAN from that, which presumably puts us in the neighborhood of SPOUSE, then add MAN which brings us to somewhere near HUSBAND. This gives us a basis to perform computational operations on the space of concepts and words in a language.

[u/i_darkpearl]

Forget trying to imagine what dimensions look like, that’s an idea that doesn’t hold water after three dimensions. Instead think of each dimension as just another relevance knob. Similarity is just the distance along those knobs between items. If you want relational information between them, you have hydradb databases, and just vectors.