## Description of changes
The `IncludeEnum` enum is not exported, cause lint errors when using
`.get` or `.query`, as follows:
```js
const result = await collection.query({
queryTexts: [query],
// THIS LINE WILL PRODUCE LINT ERROR as it needs IncludeEnum.Distances etc.
include: ['distances', 'documents', 'metadatas'],
nResults: 2,
})
```
## Test plan
Nil
## Documentation Changes
Nil
Refs: #1105
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- JS Client support for $in and $nin
## Test plan
*How are these changes tested?*
- [x] Tests pass locally `yarn test` for js
## Documentation Changes
TBD
Refs: #1104
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Expanding Cohere version also to support 6.x This plays nice with the
rest of the ecosystem
## Test plan
*How are these changes tested?*
- [x] `yarn test` for js
## Documentation Changes
N/A
Refs: #1104
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Removed transformers and web-ai peer dependencies.
## Test plan
*How are these changes tested?*
- [ ] Manual testing - `mkdir testproject && cd testproject && npm init
-y && npm link chromadb && npm add langchain`
## Documentation Changes
N/A
Refs: #1083
## Description of changes
*Summarize the changes made by this PR.*
- New functionality
- JS Client now supports Authorization, and X-Chroma-Token auths
supported
- Tests and integration tests updated
## Test plan
*How are these changes tested?*
- [x] Tests pass locally `yarn test` for js
## Documentation Changes
TBD
This proposes an initial CLI
The CLI is installed when you installed `pip install chromadb`.
You then call the CLI with
`chroma run --path <persist_dir> --port <port>` where path and port are
optional.
This also adds `chroma help` and `chroma docs` as convenience links -
but I'm open to removing those.
To make this easy - I added `typer` (by the author of FastAPI). I'm not
sure this is the tool that we want to commit to for a fuller featured
CLI, but given the extremely minimal footprint of this - I don't think
it's a one way door.
<img width="1477" alt="Screenshot 2023-08-23 at 4 59 54 PM"
src="https://github.com/chroma-core/chroma/assets/891664/30374228-d303-41e1-8e9e-188b7f8532d4">
***
#### TODO
- [x] test in fresh env - i think i need to add `typer` as a req
- [ ] consider expanding the test to make sure the service is actually
running
- [x] hide the test option from the typer UI
- [x] linking to a getting started guide could be interesting at the top
of the logs
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Adds release automation for JS client.
## Test plan
They will have to be tested after merging this PR
## Documentation Changes
Update the js client Develop guide for how to use this automation.
Cherry-picked from #1029
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Added support for `$in` and `$nin` metadata filters
> Note: See CIP in `docs/` or example notebook for more info
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python
## Documentation Changes
TBD
---------
Co-authored-by: Hammad Bashir <HammadB@users.noreply.github.com>
resolves#953
## Description of changes
- Bug fixes
- implement a workaround in
clients/js/src/embeddings/WebAIEmbeddingFunction.ts to resolve#953
## Test plan
At present, we lack a testing setup specifically tailored for a browser
environment. Implementing this would be a necessary step to implement
tests for changes in this branch.
I've tested it locally using this change:
https://github.com/jeffchuber/nextjs-chroma/pull/1/files
- http://localhost:3000/api/hello (node version) works fine
- http://localhost:3000 (browser version) works fine
Refs: #1038
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Added support for both v3 and v4 of the OpenAI node library
## Test plan
`yarn test`
## Documentation Changes
TBD
Refs: #1026
## Description of changes
*Summarize the changes made by this PR.*
- New functionality
- Auth abstractions
- Basic auth supported
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `yarn test` for js (Regression)
- [x] Add new tests for auth
## Documentation Changes
Yes docs need to be updated to provide information how to configure auth
with the JS client.
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- #799 introduced a max version for pydantic but this change was not
propagated to the thin client. This makes the same change in the thin
client.
- New functionality
- ...
## Test plan
Existing tests for client
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js
## Documentation Changes
None
## Description of changes
Currently not all bundlers will chose the correct CJS/ESM build when
importing the js client because we don't define them using
[conditional-exports](https://nodejs.org/api/packages.html#conditional-exports).
This change adds the required configuration in the package.json to fix
that.
- Improvements & Bug fixes
- support conditional exports in js client
## Test plan
I have tested this manually. We have a Next.js setup which throws an
error with v1.5.6, we make use of that to detect which build (cjs/esm)
nextjs tries to load. In both cases we expect an error with the path
from which nextjs loads the module.
### Reproduce the issue
- checkout
2df1a21686
- run yarn && yarn dev
- open http://localhost:3000/
- you should see the following error in the browser
```
Failed to compile
./node_modules/chromadb/dist/main/embeddings/WebAIEmbeddingFunction.js:157:0
Module not found: Can't resolve '@visheratin/web-ai'
https://nextjs.org/docs/messages/module-not-found
Import trace for requested module:
./node_modules/chromadb/dist/main/index.js
./src/app/page.js
This error occurred during the build process and can only be dismissed by fixing the error.
```
Nextjs loads the cjs build from `dist/main`.
### Confirm the change resolves the issue
- checkout
2df1a21686
- remove current chromadb package (1.5.6) using `yarn remove chromadb`
- install chromadb client with the [change in this
pr](450159ec23)
using package linking or using verdaccio
- open http://localhost:3000/
- you should see the following error in the browser
```
Failed to compile
./node_modules/chromadb/dist/module/embeddings/WebAIEmbeddingFunction.js:131:0
Module not found: Can't resolve '@visheratin/web-ai'
https://nextjs.org/docs/messages/module-not-found
Import trace for requested module:
./node_modules/chromadb/dist/module/index.js
./src/app/page.js
This error occurred during the build process and can only be dismissed by fixing the error.
```
Nextjs now loads the esm build from `dist/module`
## Documentation Changes
I added a reference to
https://nodejs.org/api/packages.html#conditional-exports in the commit
message
## Description of changes
After the recent changes in the Web AI structure, the embedder function
became invalid because of different imports and because the browser
version of the library is only ESM, as mentioned
[here](https://github.com/chroma-core/chroma/issues/824#issuecomment-1649761279).
To fix the issue, I updated the import paths and changed the `require`
statements to `import` for the browser version of the function.
**Important note:**
When using the embedded in the browser, you should import
`chromadb/dist/module` because the [browser
version](https://www.npmjs.com/package/@visheratin/web-ai) of Web AI is
ESM-only.
For Node.js, you can use regular `chromadb` import because [Node.js
version](https://www.npmjs.com/package/@visheratin/web-ai-node) of the
library is CJS.
Additionally, because Web AI since recently supports multimodal
embeddings, I added building multimodal embeddings using CLIP-base.
## Test plan
I tested the changes locally for both browser and Node.js. The built
`chromadb` package if someone wants to test it in their projects, is
available
[here](https://drive.google.com/file/d/1cNLsHGd1VmiFiamvsEaMVGA7ng56QQKG/view?usp=sharing).
## Documentation Changes
There likely will be needed changes regarding the multimodal
functionality.
This cleans up the JS API
- remove `increment_index`
- removes `createIndex`
- removes `rawSql`
It also bumps the version to 1.5.6 (not yet released) and fixes a type
that changed.
This is technically a breaking change since we remove the endpoint for
`raw_sql`. `raw_sql` was an undocumented internal tool so we should
discuss how we want to handle this.
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- update js client usage to align with the type definication
## Test plan
*How are these changes tested?*
- put the usage example into a typescript file, test against the current
version of chromadb client package
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
This update only change the usage example, there maybe some places in
the documentations also need update for similar reason but it is not
directly related to this pull request.
## Description of changes
Base PR to release sqlite refactor, which spans many stacked PRs.
Remaining
- [x] Merge this to main
- [x] Layered Persistent Index #761
- [x] Remove old impls (In #781 )
- [x] Remove persist() API (In #787)
- [x] Add telemetry to SegmentAPI, it was not included. (#788)
- [x] New clients #805
- [x] locking and soak tests for thread-safety
- [x] Migration tool
- [x] Fix#739
- [x] Fix metadata None vs empty
- [x] Fix persist directory (addressed in #761)
- [x] Leave files open in #761 (merge stacked PR)
Post Release
- [ ] Un xfail cross version tests once we cut the release
- [x] Documentation updates for new silent ADD failure.
- [x] Update all documentation for new API instantiation
- [x] Update all documentation for settings changes
- [ ] Update terraform deployment
- [ ] Update cloudformation deployment
---------
Co-authored-by: Luke VanderHart <luke@vanderhart.net>
Co-authored-by: Jeffrey Huber <jeff@trychroma.com>
Co-authored-by: Anton Troynikov <atroyn@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sebastian Sosa <37946988+CakeCrusher@users.noreply.github.com>
Co-authored-by: Russell Pollari <russell@sharpestminds.com>
Co-authored-by: russell-pollari <pollarir@mgail.com>
Related issue #684
## Description of changes
- Improvements & Bug fixes
- npm client is now cross compatible for Node versions <18.
- New functionality
- I tested 4 versions: 18.16.0 PASS, 16.18.1 PASS, 14.21.3 PASS, 10.24.1
FAIL
## Test plan
The same tests in
[root/client/js/test](https://github.com/chroma-core/chroma/tree/main/clients/js/test)
must successfully pass under different Node versions for this to work
(they did pass).
## Documentation Changes
Perhaps add here https://docs.trychroma.com/api-reference the supported
version range.
## Notes
The core of this improvement is incredibly simple, it boils down to
adding
```ts
import 'isomorphic-fetch';
```
to
[runtime.ts](https://github.com/chroma-core/chroma/blob/main/clients/js/src/generated/runtime.ts).
Due to the fact that `openapi-generator-plus` is being used I included
some commands to the
[genapi.sh](https://github.com/chroma-core/chroma/blob/main/clients/js/genapi.sh)
to ensure `isomorphic-fetch` will be imported correctly whenever the api
schema is regenerated.
Unfortunately, the openapi-generator has not been run for a while
leading me to believe there is discontinuity between the typescript
package and server openapi typings. As a result running
[genapi.sh](https://github.com/chroma-core/chroma/blob/main/clients/js/genapi.sh)
(running openapi-generator) will lead to clashes in typings. I have
created the following issue #751 reguarding this
This PR has a few fixes
- fix the JS examples to `1.5.1` and add documentation for CORS
- add a test for `queryTexts` with a stubbed out embedding function
## Description of changes
New API implementation backed by the segment-based architecture. Should
be extensible to a full distributed architecture.
---------
Co-authored-by: Jeffrey Huber <jeff@trychroma.com>
Co-authored-by: hammadb <hammad@trychroma.com>
Co-authored-by: Anton Troynikov <atroyn@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Hammad Bashir <HammadB@users.noreply.github.com>
connect #725
## Description of changes
Add a new `fetchOptions` parameter on the JS ChromaClient creation that
allows you to pass in custom `fetch` options.
Example:
```
const chroma = new ChromaClient({
path: "http://localhost:8000",
fetchOptions: {
headers: {
'X-Api-Token': "sk-live-Hunt3r2", // Works like regular node-fetch headers!
}
}
});
```
## Test plan
The existing test suite was run and validated as passing.
## Documentation Changes
Typescript would allow the developer to know that these options are
available. I do not think it currently requires updates to the Chroma
Documentation - especially since this change is only impacting the JS
client.
---------
Co-authored-by: Jeffrey Huber <jeff@trychroma.com>
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Fixes cohere embeddings not using the api key.
## Test plan
* I tested that cohere embeddings returned a result
* Without this change the result is undefined
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
no
## Description of changes
*Summarize the changes made by this PR.*
- New functionality
- Add support for
[Transformers.js](https://github.com/xenova/transformers.js). This
allows users to run Hugging Face transformers directly from their JS
applications. For more information, see
https://huggingface.co/docs/transformers.js.
## Test plan
*How are these changes tested?*
I followed the [JS "Getting started"
guide](https://docs.trychroma.com/getting-started?lang=js) to ensure it
works as intended. Here's the script, and its output:
```js
const {ChromaClient, TransformersEmbeddingFunction} = require('chromadb');
const client = new ChromaClient();
// Create the embedder. In this case, I just use the defaults, but you can change the model,
// quantization, revision, or add a progress callback, if desired.
const embedder = new TransformersEmbeddingFunction({ /* Configuration goes here */ });
const main = async () => {
// Empties and completely resets the database.
await client.reset()
// Create the collection
const collection = await client.createCollection({name: "my_collection", embeddingFunction: embedder})
// Add some data to the collection
await collection.add({
ids: ["id1", "id2", "id3"],
metadatas: [{"source": "my_source"}, {"source": "my_source"}, {"source": "my_source"}],
documents: ["I love walking my dog", "This is another document", "This is a legal document"],
})
// Query the collection
const results = await collection.query({
nResults: 2,
queryTexts: ["This is a query document"]
})
console.log(results)
// {
// ids: [ [ 'id2', 'id3' ] ],
// embeddings: null,
// documents: [ [ 'This is another document', 'This is a legal document' ] ],
// metadatas: [ [ [Object], [Object] ] ],
// distances: [ [ 1.0109775066375732, 1.0756263732910156 ] ]
// }
}
main();
```
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
I just added some type annotations for the constructor.
## Description of changes
It turns out `isinstance(bool, int)` returns True. We need to test for
`bool` separately.
Addresses the first part of
https://github.com/chroma-core/chroma/issues/671
## Test plan
A new test, `test_boolean_metadata`, marked xfail.
## Documentation Changes
None. This now matches the documentation.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
## Description of changes
- Improvements & Bug fixes
- Fix incorrect types for `embeddings` and `distances` in
`QueryResponse`
- closes: #660
## Test plan
Tests ran and passed
## Documentation Changes
None
Co-authored-by: russell-pollari <pollarir@mgail.com>
## Description of changes
Added a new JavaScript embedding function to generate embeddings for
text and images using the [Web AI](https://github.com/visheratin/web-ai)
library. The users will now be able to generate the embeddings in both
NodeJS and (IMO more importantly) in the browser. This, along with the
existing JS library design, enables the decoupling of embeddings
generation from storage (see example demo below).
## Test plan
I created a demo to check how the function works in web apps -
https://web-ai-chroma.vercel.app/
The embeddings are generated on the client side and sent to the server
to add to the collection or query for similar items.
## Documentation Changes
Likely this change will require changes in the documentation.
## Description of changes
When fetching a collection that does not exist, the JS client would fail
silently, returning a collection with `undefined` for all its
attributes.
- Improvements & Bug fixes
- `ChromaClient.getCollection` will throw an error if there is an error
in the API response
- closes: #634
## Test plan
Passed tests with `$ yarn test`
## Documentation Changes
None
---------
Co-authored-by: russell-pollari <pollarir@mgail.com>
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Typing cleanup
- Changes embedding function defaults to work through the default params
as opposed to via None so that None now means - NO embedding function as
opposed to use the default. This is technically a breaking change.
- New functionality
- Adds a thin client that restricts what you can create to the REST api
client and builds it separately so it can be published to its own pypi
package. The thin client restricts the default embedding function to be
None always - forcing manual specification of the embedding function
while using the thin client.
- The thin client is built with its own pyproject.toml with a limited
set of dependencies and a is_thin_client.py file that acts as a compile
flag. The build script stages the toml, places the two files in the
right place, performs the build and then tears down the changes.
Addresses #289
## Test plan
The existing tests should cover this configuration. We can add CI for
the thin-client in the future.
## Documentation Changes
We will add a section on the docs that explains the thin client and its
limitations.
## Description of changes
Please let me know if anything needs to be changed or done differently!
We can certainly continue to make efforts to make our JS client more
structured and maintainable.
*Summary:*
- Improvements & Bug fixes
- Broke `client/js/src/index.ts` down and put them in separate files
- Created `IEmbeddedFunction` interface to improve the maintainability
and extensibility of the client code, making it easier to add new
embedding function types in the future
- Replaced `CallableFunction` with `IEmbeddingFunction`
- Closes#536
- New functionality
- None
## Test plan
Refactored tests
**Note:** the Collection tests have been failing on main when I run
using `yarn test`, did we know about this?
## Documentation Changes
Not sure, couldn't find anything?
In this PR
- docstrings!
- types!
- named params!
This is a breaking change to the JS Client - it switches from positional
to named arguments originally brought up by @transitive-bullshit in
https://github.com/chroma-core/chroma/issues/254
Example:
```
// before
const collection = await chroma.createCollection("test");
//after
const collection = await chroma.createCollection({ name: "test" });
```
This is especially nice when you are only using a few params.
This PR does not bump the `npm` version number - we will cut a new
release on Monday.
We will need to land an update to our docs at that time as well. TBD
Adds precommit hooks based on #433 to our repository. Only one file here is new - the configuration for the hooks, everything else is linting/formatting fixes. We do not run the typechecker globally since that would be quite lengthy to clean up - instead we will have to incrementally clean up type check issues as we go.
