-
KDB.AI Cosine similarity
When using negative values in vectors, the cosine similarity is calculated incorrectly. The distance is more than 1 or less than -1 more than often if the vector contains negative values. The vector dimension size we use is 768.
-
4 Replies
-
* editing after speaking to KDB.AI team *
Hi ,
Thanks for your message. Can you confirm you’re using ‘metric’: ‘CS’ in your definition to assign cosine similarity?
And can you give us some more information to we can try recreate this?
- index config
- server version
- client version
- descriptive stats on inserted data data (dimension, outliers, presence of constant vectors)
- descriptive stats on query data data (dimension, outliers, presence of constant vectors)
- are there any outlier values in the data
Would also like to know- if indexes look off or is it just distances
- how far off 1/-1 are distances
Feel free to email me lkerr@kx.com, or send me a DM in the community if you’d prefer to.
Thanks,
Laura
-
the dimension size for the above one is 768
-
Schema config: “
'columns': [{ 'name': 'stock_name', 'pytype': 'str' }, { 'name': 'Date', 'pytype': 'str' }, {'name': 'embeddings', 'vectorIndex': {'dims': len(tables[table_name][stock_names[0]]['embeddings'].iloc[0]), 'metric': 'CS', 'type': 'flat'} }I’m using kdb ai cloud version, and the outliers are negative values below -30e-3. No major outliers as this is an output of distilbert. The distances look off, manually calculating it, it seems like its not scaled (dot product and cosine similarity give me similar distances). The distance is as high as 60 -
Hi ,
Thanks for reaching out about this topic!
After investigation from the KDB.AI team, I can confirm they were able to replicate your issue, and this will be addressed in an upcoming release.
Keep an eye on KDB.AI Cloud Release Notes to see when the next release is.
Thanks again for reaching out!
Laura
Log in to reply.