## Description of changes
*Summarize the changes made by this PR.*
- New functionality
- Adds a basic pulsar producer, consumer and associated tests. As well
as a docker compose for the distributed version of chroma.
## Test plan
We added bin/cluster-test.sh, which starts pulsar and allows
test_producer_consumer to run the pulsar fixture.
## Documentation Changes
None required.
## Description of changes
*Summarize the changes made by this PR.*
- Added a workflow_dispatch to manually trigger test workflows
- will be good for development experience
---------
Signed-off-by: sunilkumardash9 <sunilkumardash9@gmail.com>
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Added bandit scanning for all pushes to repo
## Test plan
*How are these changes tested?*
Manual testing of the workflow
## Documentation Changes
N/A - unless we want to start a separate security section in the main
docs repo.
---------
Co-authored-by: Hammad Bashir <HammadB@users.noreply.github.com>
This proposes an initial CLI
The CLI is installed when you installed `pip install chromadb`.
You then call the CLI with
`chroma run --path <persist_dir> --port <port>` where path and port are
optional.
This also adds `chroma help` and `chroma docs` as convenience links -
but I'm open to removing those.
To make this easy - I added `typer` (by the author of FastAPI). I'm not
sure this is the tool that we want to commit to for a fuller featured
CLI, but given the extremely minimal footprint of this - I don't think
it's a one way door.
<img width="1477" alt="Screenshot 2023-08-23 at 4 59 54 PM"
src="https://github.com/chroma-core/chroma/assets/891664/30374228-d303-41e1-8e9e-188b7f8532d4">
***
#### TODO
- [x] test in fresh env - i think i need to add `typer` as a req
- [ ] consider expanding the test to make sure the service is actually
running
- [x] hide the test option from the typer UI
- [x] linking to a getting started guide could be interesting at the top
of the logs
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Pushes images to dockerhub
## Test plan
*How are these changes tested?*
Will have to be tested on main as part of CI/CD
- [x] Tests pass locally with `pytest` for python, `yarn test` for js
## Documentation Changes
None required.
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Add npm run db:run to run docker-compose up
- New functionality
- ...
## Test plan
Will be tested manually. This is an iteration on #1095 .
- [x] Tests pass locally with `pytest` for python, `yarn test` for js
## Documentation Changes
None
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Prevent python release from running on js tag pushes
## Test plan
Manually on main via triggering
## Documentation Changes
None
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Adds release automation for JS client.
## Test plan
They will have to be tested after merging this PR
## Documentation Changes
Update the js client Develop guide for how to use this automation.
Currently our ghrc (and future dockerhub) builds will not run on Mac.
This should build `linux/amd64` and `linux/arm64` in serial.
In the future, we can add ` linux/amd64/v2, linux/amd64/v3` and
parallelize the builds.
https://docs.docker.com/build/ci/github-actions/multi-platform/
Plan
1. let tests pass
2. merge into `main` and see if `latest` release branch works
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Adds an LRU cache for segment file handle management to make sure we
respect OS file handle limits
- New functionality
- ...
## Test plan
Added a stress test which creates many collections and forces them to
open index files by writing to them
## Documentation Changes
We should add a troubleshooting section about filesystem limits so that
users can set max limits more appropriately - similar to
https://www.mongodb.com/docs/manual/reference/ulimit/
## Description of changes
Addresses #887
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Adds windows to github test runners
- New functionality
- ...
## Test plan
These are tests.
## Documentation Changes
None required.
## Description of changes
Base PR to release sqlite refactor, which spans many stacked PRs.
Remaining
- [x] Merge this to main
- [x] Layered Persistent Index #761
- [x] Remove old impls (In #781 )
- [x] Remove persist() API (In #787)
- [x] Add telemetry to SegmentAPI, it was not included. (#788)
- [x] New clients #805
- [x] locking and soak tests for thread-safety
- [x] Migration tool
- [x] Fix#739
- [x] Fix metadata None vs empty
- [x] Fix persist directory (addressed in #761)
- [x] Leave files open in #761 (merge stacked PR)
Post Release
- [ ] Un xfail cross version tests once we cut the release
- [x] Documentation updates for new silent ADD failure.
- [x] Update all documentation for new API instantiation
- [x] Update all documentation for settings changes
- [ ] Update terraform deployment
- [ ] Update cloudformation deployment
---------
Co-authored-by: Luke VanderHart <luke@vanderhart.net>
Co-authored-by: Jeffrey Huber <jeff@trychroma.com>
Co-authored-by: Anton Troynikov <atroyn@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sebastian Sosa <37946988+CakeCrusher@users.noreply.github.com>
Co-authored-by: Russell Pollari <russell@sharpestminds.com>
Co-authored-by: russell-pollari <pollarir@mgail.com>
## Description of changes
New API implementation backed by the segment-based architecture. Should
be extensible to a full distributed architecture.
---------
Co-authored-by: Jeffrey Huber <jeff@trychroma.com>
Co-authored-by: hammadb <hammad@trychroma.com>
Co-authored-by: Anton Troynikov <atroyn@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Hammad Bashir <HammadB@users.noreply.github.com>
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Typing cleanup
- Changes embedding function defaults to work through the default params
as opposed to via None so that None now means - NO embedding function as
opposed to use the default. This is technically a breaking change.
- New functionality
- Adds a thin client that restricts what you can create to the REST api
client and builds it separately so it can be published to its own pypi
package. The thin client restricts the default embedding function to be
None always - forcing manual specification of the embedding function
while using the thin client.
- The thin client is built with its own pyproject.toml with a limited
set of dependencies and a is_thin_client.py file that acts as a compile
flag. The build script stages the toml, places the two files in the
right place, performs the build and then tears down the changes.
Addresses #289
## Test plan
The existing tests should cover this configuration. We can add CI for
the thin-client in the future.
## Documentation Changes
We will add a section on the docs that explains the thin client and its
limitations.
Adds precommit hooks based on #433 to our repository. Only one file here is new - the configuration for the hooks, everything else is linting/formatting fixes. We do not run the typechecker globally since that would be quite lengthy to clean up - instead we will have to incrementally clean up type check issues as we go.
