* fix: don't remove first block * fix: make etching_terms nullable * fix: fix panic if empty pkscript * chore: change testnet starting block * feat: more logs * fix: extract tapscript bug * feat: more logs * fix: switch pk to block height * chore: remove redundant log * fix: repo * fix: not found error * fix: golangci-lint * feat: add etching tx hash to rune entries * feat: stop main if indexer failed * fix: check balance after populating current balance * fix: sql ambiguous column * feat: add tx hash and out index in tx output * fix: actually use transactions to write db * fix: create rune entry states only during flushes * fix: mint cap reached off by one * fix: debug log unsafe * feat: prevent processing of txs before activation height * feat: add rune number to rune entry * feat: include new rune entries in event hash and flushing * refactor(config): separate init and get config func Co-authored-by: Gaze <dev@gaze.network> * feat: remove annoying log Co-authored-by: Gaze <dev@gaze.network> * feat: mod tidy Co-authored-by: Gaze <dev@gaze.network> * refactor: move main to root Co-authored-by: Gaze <dev@gaze.network> * feat(cli): create cli commands Co-authored-by: Gaze <dev@gaze.network> * refactor: move main logic to command Co-authored-by: Gaze <dev@gaze.network> * doc: remove unused desc Co-authored-by: Gaze <dev@gaze.network> * refactor: test structure in runestone_test.go * fix: edict flaws were ignored * feat: more tests * refactor(cli): add local flag Co-authored-by: Gaze <dev@gaze.network> * feat: set symbol limit to utf8.MaxRune * refactor(cli): flags for each module Co-authored-by: Gaze <dev@gaze.network> * feat(cli): support db selection Co-authored-by: Gaze <dev@gaze.network> * fix: remove temp code Co-authored-by: Gaze <dev@gaze.network> * fix: get data from cache in processor first, then dg * feat(cli): add version command Co-authored-by: Gaze <dev@gaze.network> * doc(cli): add refactor plan Co-authored-by: Gaze <dev@gaze.network> * refactor(cli): rename files Co-authored-by: Gaze <dev@gaze.network> * feat: add main.go Co-authored-by: Gaze <dev@gaze.network> * feat: more tests * feat: add overflow err * feat: finish runestone tests * refactor(cli): separate protocol config and cli flag Co-authored-by: Gaze <dev@gaze.network> * chore(btc): update example config Co-authored-by: Gaze <dev@gaze.network> * feat(btc): add get block header to datasource interface Co-authored-by: Gaze <dev@gaze.network> * feat(btc): reorg handling Co-authored-by: Gaze <dev@gaze.network> * fix: interface Co-authored-by: Gaze <dev@gaze.network> * fix: rename postgres config key * fix: migrated runes indexer integration to new cli * fix: commit every block * feat(btc): add revert data query Co-authored-by: Gaze <dev@gaze.network> * feat(btc): add revert data to processor Co-authored-by: Gaze <dev@gaze.network> * feat: implement public errors * fix: use errs in api * refactor: move api and usecase outside of internal * feat: add custom opcode check for datapush * fix: break if input utxo is not P2TR * fix: zero len destination case * fix: get the rest of transaction data in GetTransaction * refactor: create subscription utils tools Co-authored-by: Gaze <dev@gaze.network> * feat(btc): add btc_database from datasource Co-authored-by: Gaze <dev@gaze.network> * doc(btc): add note Co-authored-by: Gaze <dev@gaze.network> * wip(btc): imple prepare range func Co-authored-by: Gaze <dev@gaze.network> * feat(btc): add pg queries for datasource Co-authored-by: Gaze <dev@gaze.network> * feat(btc): update queries Co-authored-by: Gaze <dev@gaze.network> * feat(btc): implement repo for get blocks Co-authored-by: Gaze <dev@gaze.network> * feat(btc): update dg Co-authored-by: Gaze <dev@gaze.network> * fix(btc): return nil if errors Co-authored-by: Gaze <dev@gaze.network> * feat(btc): update fetch async for db datasource Co-authored-by: Gaze <dev@gaze.network> * feat(btc): add get block header from db for reorg handling Co-authored-by: Gaze <dev@gaze.network> * feat(btc): add todo notes Co-authored-by: Gaze <dev@gaze.network> * feat: implement get tx by hash * fix: rename func * fix: rename func * fix: rename func * fix: fix get transaction by hash * feat: integrate bitcoin client db to main * fix: reduce chunk size * fix: stop main if bitcoin indexer failed * fix: stop main if runes indexer failed * fix: move stop() inside goroutine * chore: add log * fix: duplicate rune entry number * feat(btc): add witness utils Co-authored-by: Gaze <dev@gaze.network> * feat(btc): witness datamodel parsing Co-authored-by: Gaze <dev@gaze.network> * fix(btc): invalid table name Co-authored-by: Gaze <dev@gaze.network> * fix(btc): remove uniqte index for hash Co-authored-by: Gaze <dev@gaze.network> * doc: add todo Co-authored-by: Gaze <dev@gaze.network> * feat(logger): remove error verbose Co-authored-by: Gaze <dev@gaze.network> * feat: support postgresql db Co-authored-by: Gaze <dev@gaze.network> * feat(btc): add err notfound Co-authored-by: Gaze <dev@gaze.network> * fix: invalid pgx version Co-authored-by: Gaze <dev@gaze.network> * fix: invalid indexer flow Co-authored-by: Gaze <dev@gaze.network> * feat: refactor runes api * feat: implement http server * fix: mount runes api * fix: error handler * fix: first empty state error Co-authored-by: Gaze <dev@gaze.network> * fix: off by one confirmation * ci: ignore RollBack error * fix: change WithPublicMessage to be prefix * feat: bump cstream version Co-authored-by: Gaze <dev@gaze.network> * feat(btc): nullable pkscript Co-authored-by: Gaze <dev@gaze.network> * feat(btc): change rollback style Co-authored-by: Gaze <dev@gaze.network> * refactor: move runes out of internal * feat: rename id field to runeId in rune transaction * feat(btc): update index Co-authored-by: Gaze <dev@gaze.network> * feat(btc): add default current block Co-authored-by: Gaze <dev@gaze.network> * doc: add note Co-authored-by: Gaze <dev@gaze.network> * fix(btc): use int64 to store sequence Co-authored-by: Gaze <dev@gaze.network> * fix(btc): upgrade data type for numbers Co-authored-by: Gaze <dev@gaze.network> * feat(btc): upgrade data type for idx Co-authored-by: Gaze <dev@gaze.network> * feat(btc): get indexed block impl Co-authored-by: Gaze <dev@gaze.network> * feat(btc): add common.ZeroHash Co-authored-by: Gaze <dev@gaze.network> * feat: add chainparam * feat: implement get transactions * fix: wrong condition for non-OP_RETURN output * feat(btc): add verify indexer states Co-authored-by: Gaze <dev@gaze.network> * refactor: sorting code Co-authored-by: Gaze <dev@gaze.network> * feat: fix interface * feat(btc): update chuunk size Co-authored-by: Gaze <dev@gaze.network> * feat: add rune_etched column in rune transaction * fix: missing field in create * feat: add runeEtched in get transactions * feat: implement get token info * feat: add holders count in token info * feat: implement get holders * fix: return a new repository when beginning a new tx * fix: rename type * feat: add pkscript to outpoint balance * feat: implement get utxos by address api * fix: spend outpoint bug * feat: implement get balances by address batch * feat: sort balances result by amount * ci: create Dockerfile Co-authored-by: Gaze <dev@gaze.network> * ci: add arg run Co-authored-by: Gaze <dev@gaze.network> * perf: add automaxprocs Co-authored-by: Gaze <dev@gaze.network> * chore: add performance logging Co-authored-by: Gaze <dev@gaze.network> * chore: add performance logger for debyug Co-authored-by: Gaze <dev@gaze.network> * fix: empty etched at * fix: revert data sequentially * fix: remove unused funcs * fix: main.go * feat: add flag --api-only to run cmd * fix: create index * fix: don't add zero mint to unallocated * fix: ignore zero burn amount * feat(reorg): add reorg detail Co-authored-by: Gaze <dev@gaze.network> * fix: wrong index type * feat: implement reporting client to report runes blocks * feat: implement report node * feat(runes): add latest block api Co-authored-by: Gaze <dev@gaze.network> * feat(btc): use logger warn Co-authored-by: Gaze <dev@gaze.network> * fix(btc): txout aren't revert if it's have to revert spent Co-authored-by: Gaze <dev@gaze.network> * fix: annoying error when unsubscribe fetcher Co-authored-by: Gaze <dev@gaze.network> * refactor(btc): readable code Co-authored-by: Gaze <dev@gaze.network> * fix(indexer): fix subscription closed before process when success fetch Co-authored-by: Gaze <dev@gaze.network> * fix: remove module enum * fix: increase max reorg limit * feat: add starting height for runes mainnet * fix(btc): fix `with` modified same row twice Co-authored-by: Gaze <dev@gaze.network> * fix(runes): handling latest block not found Co-authored-by: Gaze <dev@gaze.network> * feat: add decimals in get transactions * fix: wrong condition * feat: add more index * feat: implement get transactions by pkscript * feat: allow query by rune id too * feat: more comments * perf(btc): bitcoin indexer performance optimization (#4) * feat(btc): not null to witness Co-authored-by: Gaze <dev@gaze.network> * perf(btc): add batch insert txin Co-authored-by: Gaze <dev@gaze.network> * perf(btc): batch insert txout Co-authored-by: Gaze <dev@gaze.network> * perf(btc): batch insert transaction Co-authored-by: Gaze <dev@gaze.network> * feat(btc): remove old queries Co-authored-by: Gaze <dev@gaze.network> * fix(btc): typo Co-authored-by: Gaze <dev@gaze.network> * perf(btc): batch insert blocks (#5) Co-authored-by: Gaze <gazenw@users.noreply.github.com> --------- Co-authored-by: Gaze <gazenw@users.noreply.github.com> * feat(btc): Duplicate coinbase transaction handling (#7) * feat(btc): tx_hash can duplicated in block v1 Co-authored-by: Gaze <dev@gaze.network> * feat(btc): duplicate tx will use same txin/txout from previous tx Co-authored-by: Gaze <dev@gaze.network> * feat(btc): prevent revert block v1 data if you really want to revert the data before the block version 2, you should reset the database and reindex the data instead. Co-authored-by: Gaze <dev@gaze.network> * doc(btc): update list duplicate tx hash Co-authored-by: Gaze <dev@gaze.network> * doc(btc): update docs Co-authored-by: Gaze <dev@gaze.network> * fix(btc): use last v1 block instead Co-authored-by: Gaze <dev@gaze.network> --------- Co-authored-by: Gaze <gazenw@users.noreply.github.com> * feat: add ping handler * fix: type Co-authored-by: Gaze <dev@gaze.network> * doc: add refactor note Co-authored-by: Gaze <dev@gaze.network> * ci: add golang linter and test runner gh action * ci: use go-test-action@v0 * ci: annotate test result * ci: update running flag * fix: try to fix malformed import path * feat: add mock test * ci: remove annotation ci * ci: add annotate test result * chore: remove unused * feat: try testify * feat: remove test * ci: add go test on macos, windows and go latest version * ci: test building * feat: remove mock code * ci: add sqlc diff checker action (#10) * feat: Graceful shutdown (#8) * feat: add shutdown function for indexer Co-authored-by: Gaze <dev@gaze.network> * feat: add force shutdown Co-authored-by: Gaze <dev@gaze.network> * revert Co-authored-by: Gaze <dev@gaze.network> * feat(btc): remove unused Co-authored-by: Gaze <dev@gaze.network> * style: go fmt Co-authored-by: Gaze <dev@gaze.network> * feat: separate context for worker and application * feat: increase force shutdown timeout Co-authored-by: Gaze <dev@gaze.network> * feat(btc): update logging Co-authored-by: Gaze <dev@gaze.network> * feat(btc): update shutdown function Co-authored-by: Gaze <dev@gaze.network> * feat: remove wg for shutdown Co-authored-by: Gaze <dev@gaze.network> * feat: refactor shutdown flow Co-authored-by: Gaze <dev@gaze.network> * feat: update shutdown flow Co-authored-by: Gaze <dev@gaze.network> * feat: update maming Co-authored-by: Gaze <dev@gaze.network> * feat: update force shutdown logic Co-authored-by: Gaze <dev@gaze.network> --------- Co-authored-by: Gaze <gazenw@users.noreply.github.com> * feat: check reporting config name * fix: use db config in bitcoin module for runes datasource * Add migrate commands (#2) * feat: add migrate up * feat: add down migration * fix: example * feat: change description * fix: hardcode migration source directory * Update README.md for public release. (#11) * feat: initial draft for README.md * fix: remove some sections * feat: add block reporting to first description * fix: reduce redundancy * feat: update README.md * Update README.md * feat: update README.md * fix: update config.yaml in README * fix: remove redundant words * fix: change default datasource * fix: config.yaml comments * feat: update README.md * refactor(logger): format logging (#12) * feat(logger): format main logger * feat(logger): use duration ms for gcp output * refactor(logger): bitcoin node logger * refactor(logger): indexer logger * refactor(logger): fix cmd logger * refactor(logger): logger in config pacakge * refactor(logger): set pgx error log level debug * refactor(logger): btcclient datasource * refactor: processor name * refactor(logger): runese logger * refactor(logger): update logger * fix(runes): wrong btc db datasource * refactor(logger): remove unnecessary debug log * refactor: update logger in indexer * fix(logger): deadlock in load() * fix: remove unused --------- Co-authored-by: Gaze <gazenw@users.noreply.github.com> * feat(btc): remove unused func * fix: fix golangci-lint error * fix(pg): update logger level * doc: update config example * feat: go mod tidy * doc: update readme * fix: panic cause didn't handle error * doc: update example config * doc: update example config in readme * feat(logger): only log error stacktrace when debug mode is on * feat(reporting): handling invalid config error * feat(pg): handling invalid config error * fix: panic in get_token_info --------- Co-authored-by: Gaze <gazenw@users.noreply.github.com> Co-authored-by: Planxnx <thanee@cleverse.com> Co-authored-by: Thanee Charattrakool <37617738+Planxnx@users.noreply.github.com>
6.9 KiB
Gaze Indexer
Gaze Indexer is an open-source and modular indexing client for Bitcoin meta-protocols. It has support for Bitcoin and Runes out of the box, with Unified Consistent APIs across fungible token protocols.
Gaze Indexer is built with modularity in mind, allowing users to run all modules in one monolithic instance with a single command, or as a distributed cluster of micro-services.
Gaze Indexer serves as a foundation for building ANY meta-protocol indexers, with efficient data fetching, reorg detection, and database migration tool. This allows developers to focus on what truly matters: Meta-protocol indexing logic. New meta-protocols can be easily added by implementing new modules.
Gaze Indexer also comes with a block reporting system for verifying data integrity of indexers. Visit the Gaze Network dashboard to see the status of other indexers.
Modules
1. Bitcoin
The Bitcoin Indexer, the heart of every meta-protocol, is responsible for indexing Bitcoin transactions, blocks, and UTXOs. It requires a Bitcoin Core RPC as source of Bitcoin transactions, and stores the indexed data in database to be used by other modules.
2. Runes
The Runes Indexer is our first meta-protocol indexer. It indexes Runes states, transactions, runestones, and balances using Bitcoin transactions. It comes with a set of APIs for querying historical Runes data. See our API Reference for full details.
Installation
Prerequisites
1. Hardware Requirements
Each module requires different hardware requirements.
| Module | CPU | RAM |
|---|---|---|
| Bitcoin | 0.25 cores | 256 MB |
| Runes | 0.5 cores | 1 GB |
2. Prepare Bitcoin Core RPC server.
Gaze Indexer needs to fetch transaction data from a Bitcoin Core RPC, either self-hosted or using managed providers like QuickNode. To self host a Bitcoin Core, see https://bitcoin.org/en/full-node.
3. Prepare database.
Gaze Indexer has first-class support for PostgreSQL. If you wish to use other databases, you can implement your own database repository that satisfies each module's Data Gateway interface. Here is our minimum database disk space requirement for each module.
| Module | Database Storage |
|---|---|
| Bitcoin | 240 GB |
| Runes | 150 GB |
4. Prepare config.yaml file.
# config.yaml
logger:
output: TEXT # Output format for logs. current supported formats: "TEXT" | "JSON" | "GCP"
debug: false
# Network to run the indexer on. Current supported networks: "mainnet" | "testnet"
network: mainnet
# Bitcoin Core RPC configuration options.
bitcoin_node:
host: "" # [Required] Host of Bitcoin Core RPC (without https://)
user: "" # Username to authenticate with Bitcoin Core RPC
pass: "" # Password to authenticate with Bitcoin Core RPC
disable_tls: false # Set to true to disable tls
# Block reporting configuration options. See Block Reporting section for more details.
reporting:
disabled: false # Set to true to disable block reporting to Gaze Network. Default is false.
base_url: "https://indexer.api.gaze.network" # Defaults to "https://indexer.api.gaze.network" if left empty
name: "" # [Required if not disabled] Name of this indexer to show on the Gaze Network dashboard
website_url: "" # Public website URL to show on the dashboard. Can be left empty.
indexer_api_url: "" # Public url to access this indexer's API. Can be left empty if you want to keep your indexer private.
# HTTP server configuration options.
http_server:
port: 8080 # Port to run the HTTP server on for modules with HTTP API handlers.
# Meta-protocol modules configuration options.
modules:
# Configuration options for Bitcoin module. Can be removed if not used.
bitcoin:
database: "postgres" # Database to store bitcoin data. current supported databases: "postgres"
postgres:
host: "localhost"
port: 5432
user: "postgres"
password: "password"
db_name: "postgres"
# url: "postgres://postgres:password@localhost:5432/postgres?sslmode=prefer" # [Optional] This will override other database credentials above.
# Configuration options for Runes module. Can be removed if not used.
runes:
database: "postgres" # Database to store Runes data. current supported databases: "postgres"
datasource: "database" # Data source to be used for Bitcoin data. current supported data sources: "bitcoin-node" | "database". If "database" is used, it will use the database config in bitcoin module as datasource.
api_handlers: # API handlers to enable. current supported handlers: "http"
- http
postgres:
host: "localhost"
port: 5432
user: "postgres"
password: "password"
db_name: "postgres"
# url: "postgres://postgres:password@localhost:5432/postgres?sslmode=prefer" # [Optional] This will override other database credentials above.
Install with Docker (recommended)
We will be using docker-compose for our installation guide. Make sure the docker-compose.yaml file is in the same directory as the config.yaml file.
# docker-compose.yaml
services:
gaze-indexer:
image: ghcr.io/gaze-network/gaze-indexer:v1.0.0
container_name: gaze-indexer
restart: unless-stopped
ports:
- 8080:8080 # Expose HTTP server port to host
volumes:
- "./config.yaml:/app/config.yaml" # mount config.yaml file to the container as "/app/config.yaml"
command: ["/app/main", "run", "--bitcoin", "--runes"] # Put module flags after "run" commands to select which modules to run.
Install from source
- Install
goversion 1.22 or higher. See Go installation guide here. - Clone this repository.
git clone https://github.com/gaze-network/gaze-indexer.git
cd gaze-indexer
- Build the main binary.
# Get dependencies
go mod download
# Build the main binary
go build -o gaze main.go
- Run database migrations with the
migratecommand and module flags.
./gaze migrate up --bitcoin --runes --database postgres://postgres:password@localhost:5432/postgres
- Start the indexer with the
runcommand and module flags.
./gaze run --bitcoin --runes
If config.yaml is not located at ./app/config.yaml, use the --config flag to specify the path to the config.yaml file.
./gaze run --bitcoin --runes --config /path/to/config.yaml