gazenw fcdecd4046 feat: v0.1.0 release (#13)
* fix: don't remove first block

* fix: make etching_terms nullable

* fix: fix panic if empty pkscript

* chore: change testnet starting block

* feat: more logs

* fix: extract tapscript bug

* feat: more logs

* fix: switch pk to block height

* chore: remove redundant log

* fix: repo

* fix: not found error

* fix: golangci-lint

* feat: add etching tx hash to rune entries

* feat: stop main if indexer failed

* fix: check balance after populating current balance

* fix: sql ambiguous column

* feat: add tx hash and out index in tx output

* fix: actually use transactions to write db

* fix: create rune entry states only during flushes

* fix: mint cap reached off by one

* fix: debug log unsafe

* feat: prevent processing of txs before activation height

* feat: add rune number to rune entry

* feat: include new rune entries in event hash and flushing

* refactor(config): separate init and get config func

Co-authored-by: Gaze <dev@gaze.network>

* feat: remove annoying log

Co-authored-by: Gaze <dev@gaze.network>

* feat: mod tidy

Co-authored-by: Gaze <dev@gaze.network>

* refactor: move main to root

Co-authored-by: Gaze <dev@gaze.network>

* feat(cli): create cli commands

Co-authored-by: Gaze <dev@gaze.network>

* refactor: move main logic to command

Co-authored-by: Gaze <dev@gaze.network>

* doc: remove unused desc

Co-authored-by: Gaze <dev@gaze.network>

* refactor: test structure in runestone_test.go

* fix: edict flaws were ignored

* feat: more tests

* refactor(cli): add local flag

Co-authored-by: Gaze <dev@gaze.network>

* feat: set symbol limit to utf8.MaxRune

* refactor(cli): flags for each module

Co-authored-by: Gaze <dev@gaze.network>

* feat(cli): support db selection

Co-authored-by: Gaze <dev@gaze.network>

* fix: remove temp code

Co-authored-by: Gaze <dev@gaze.network>

* fix: get data from cache in processor first, then dg

* feat(cli): add version command

Co-authored-by: Gaze <dev@gaze.network>

* doc(cli): add refactor plan

Co-authored-by: Gaze <dev@gaze.network>

* refactor(cli): rename files

Co-authored-by: Gaze <dev@gaze.network>

* feat: add main.go

Co-authored-by: Gaze <dev@gaze.network>

* feat: more tests

* feat: add overflow err

* feat: finish runestone tests

* refactor(cli): separate protocol config and cli flag

Co-authored-by: Gaze <dev@gaze.network>

* chore(btc): update example config

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): add get block header to datasource interface

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): reorg handling

Co-authored-by: Gaze <dev@gaze.network>

* fix: interface

Co-authored-by: Gaze <dev@gaze.network>

* fix: rename postgres config key

* fix: migrated runes indexer integration to new cli

* fix: commit every block

* feat(btc): add revert data query

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): add revert data to processor

Co-authored-by: Gaze <dev@gaze.network>

* feat: implement public errors

* fix: use errs in api

* refactor: move api and usecase outside of internal

* feat: add custom opcode check for datapush

* fix: break if input utxo is not P2TR

* fix: zero len destination case

* fix: get the rest of transaction data in GetTransaction

* refactor: create subscription utils tools

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): add btc_database from datasource

Co-authored-by: Gaze <dev@gaze.network>

* doc(btc): add note

Co-authored-by: Gaze <dev@gaze.network>

* wip(btc): imple prepare range func

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): add pg queries for datasource

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): update queries

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): implement repo for get blocks

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): update dg

Co-authored-by: Gaze <dev@gaze.network>

* fix(btc): return nil if errors

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): update fetch async for db datasource

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): add get block header from db for reorg handling

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): add todo notes

Co-authored-by: Gaze <dev@gaze.network>

* feat: implement get tx by hash

* fix: rename func

* fix: rename func

* fix: rename func

* fix: fix get transaction by hash

* feat: integrate bitcoin client db to main

* fix: reduce chunk size

* fix: stop main if bitcoin indexer failed

* fix: stop main if runes indexer failed

* fix: move stop() inside goroutine

* chore: add log

* fix: duplicate rune entry number

* feat(btc): add witness utils

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): witness datamodel parsing

Co-authored-by: Gaze <dev@gaze.network>

* fix(btc): invalid table name

Co-authored-by: Gaze <dev@gaze.network>

* fix(btc): remove uniqte index for hash

Co-authored-by: Gaze <dev@gaze.network>

* doc: add todo

Co-authored-by: Gaze <dev@gaze.network>

* feat(logger): remove error verbose

Co-authored-by: Gaze <dev@gaze.network>

* feat: support postgresql db

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): add err notfound

Co-authored-by: Gaze <dev@gaze.network>

* fix: invalid pgx version

Co-authored-by: Gaze <dev@gaze.network>

* fix: invalid indexer flow

Co-authored-by: Gaze <dev@gaze.network>

* feat: refactor runes api

* feat: implement http server

* fix: mount runes api

* fix: error handler

* fix: first empty state error

Co-authored-by: Gaze <dev@gaze.network>

* fix: off by one confirmation

* ci: ignore RollBack error

* fix: change WithPublicMessage to be prefix

* feat: bump cstream version

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): nullable pkscript

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): change rollback style

Co-authored-by: Gaze <dev@gaze.network>

* refactor: move runes out of internal

* feat: rename id field to runeId in rune transaction

* feat(btc): update index

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): add default current block

Co-authored-by: Gaze <dev@gaze.network>

* doc: add note

Co-authored-by: Gaze <dev@gaze.network>

* fix(btc): use int64 to store sequence

Co-authored-by: Gaze <dev@gaze.network>

* fix(btc): upgrade data type for numbers

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc):  upgrade data type for idx

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): get indexed block impl

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): add common.ZeroHash

Co-authored-by: Gaze <dev@gaze.network>

* feat: add chainparam

* feat: implement get transactions

* fix: wrong condition for non-OP_RETURN output

* feat(btc): add verify indexer states

Co-authored-by: Gaze <dev@gaze.network>

* refactor: sorting code

Co-authored-by: Gaze <dev@gaze.network>

* feat: fix interface

* feat(btc): update chuunk size

Co-authored-by: Gaze <dev@gaze.network>

* feat: add rune_etched column in rune transaction

* fix: missing field in create

* feat: add runeEtched in get transactions

* feat: implement get token info

* feat: add holders count in token info

* feat: implement get holders

* fix: return a new repository when beginning a new tx

* fix: rename type

* feat: add pkscript to outpoint balance

* feat: implement get utxos by address api

* fix: spend outpoint bug

* feat: implement get balances by address batch

* feat: sort balances result by amount

* ci: create Dockerfile

Co-authored-by: Gaze <dev@gaze.network>

* ci: add arg run

Co-authored-by: Gaze <dev@gaze.network>

* perf: add automaxprocs

Co-authored-by: Gaze <dev@gaze.network>

* chore: add performance logging

Co-authored-by: Gaze <dev@gaze.network>

* chore: add performance logger for debyug

Co-authored-by: Gaze <dev@gaze.network>

* fix: empty etched at

* fix: revert data sequentially

* fix: remove unused funcs

* fix: main.go

* feat: add flag --api-only to run cmd

* fix: create index

* fix: don't add zero mint to unallocated

* fix: ignore zero burn amount

* feat(reorg): add reorg detail

Co-authored-by: Gaze <dev@gaze.network>

* fix: wrong index type

* feat: implement reporting client to report runes blocks

* feat: implement report node

* feat(runes): add latest block api

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): use logger warn

Co-authored-by: Gaze <dev@gaze.network>

* fix(btc): txout aren't revert if it's have to revert spent

Co-authored-by: Gaze <dev@gaze.network>

* fix: annoying error when unsubscribe fetcher

Co-authored-by: Gaze <dev@gaze.network>

* refactor(btc): readable code

Co-authored-by: Gaze <dev@gaze.network>

* fix(indexer): fix subscription closed before process when success fetch

Co-authored-by: Gaze <dev@gaze.network>

* fix: remove module enum

* fix: increase max reorg limit

* feat: add starting height for runes mainnet

* fix(btc): fix `with` modified same row twice

Co-authored-by: Gaze <dev@gaze.network>

* fix(runes): handling latest block not found

Co-authored-by: Gaze <dev@gaze.network>

* feat: add decimals in get transactions

* fix: wrong condition

* feat: add more index

* feat: implement get transactions by pkscript

* feat: allow query by rune id too

* feat: more comments

* perf(btc): bitcoin indexer performance optimization (#4)

* feat(btc): not null to witness

Co-authored-by: Gaze <dev@gaze.network>

* perf(btc): add batch insert txin

Co-authored-by: Gaze <dev@gaze.network>

* perf(btc): batch insert txout

Co-authored-by: Gaze <dev@gaze.network>

* perf(btc): batch insert transaction

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): remove old queries

Co-authored-by: Gaze <dev@gaze.network>

* fix(btc): typo

Co-authored-by: Gaze <dev@gaze.network>

* perf(btc): batch insert blocks (#5)

Co-authored-by: Gaze <gazenw@users.noreply.github.com>

---------

Co-authored-by: Gaze <gazenw@users.noreply.github.com>

* feat(btc): Duplicate coinbase transaction handling (#7)

* feat(btc): tx_hash can duplicated in block v1

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): duplicate tx  will use same txin/txout from previous tx

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): prevent revert block v1 data

if you really want to revert the data before the block version 2, you should reset the database and reindex the data instead.

Co-authored-by: Gaze <dev@gaze.network>

* doc(btc): update list duplicate tx hash

Co-authored-by: Gaze <dev@gaze.network>

* doc(btc): update docs

Co-authored-by: Gaze <dev@gaze.network>

* fix(btc): use last v1 block instead

Co-authored-by: Gaze <dev@gaze.network>

---------

Co-authored-by: Gaze <gazenw@users.noreply.github.com>

* feat: add ping handler

* fix: type

Co-authored-by: Gaze <dev@gaze.network>

* doc: add refactor note

Co-authored-by: Gaze <dev@gaze.network>

* ci: add golang linter and test runner gh action

* ci: use go-test-action@v0

* ci: annotate test result

* ci: update running flag

* fix: try to fix malformed import path

* feat: add mock test

* ci: remove annotation ci

* ci: add annotate test result

* chore: remove unused

* feat: try testify

* feat: remove test

* ci: add go test on macos, windows and go latest version

* ci: test building

* feat: remove mock code

* ci: add sqlc diff checker action (#10)

* feat: Graceful shutdown (#8)

* feat: add shutdown function for indexer

Co-authored-by: Gaze <dev@gaze.network>

* feat: add force shutdown

Co-authored-by: Gaze <dev@gaze.network>

* revert

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): remove unused

Co-authored-by: Gaze <dev@gaze.network>

* style: go fmt

Co-authored-by: Gaze <dev@gaze.network>

* feat: separate context for worker and application

* feat: increase force shutdown timeout

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): update logging

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): update shutdown function

Co-authored-by: Gaze <dev@gaze.network>

* feat: remove wg for shutdown

Co-authored-by: Gaze <dev@gaze.network>

* feat: refactor shutdown flow

Co-authored-by: Gaze <dev@gaze.network>

* feat: update shutdown flow

Co-authored-by: Gaze <dev@gaze.network>

* feat: update maming

Co-authored-by: Gaze <dev@gaze.network>

* feat: update force shutdown logic

Co-authored-by: Gaze <dev@gaze.network>

---------

Co-authored-by: Gaze <gazenw@users.noreply.github.com>

* feat: check reporting config name

* fix: use db config in bitcoin module for runes datasource

* Add migrate commands (#2)

* feat: add migrate up

* feat: add down migration

* fix: example

* feat: change description

* fix: hardcode migration source directory

* Update README.md for public release. (#11)

* feat: initial draft for README.md

* fix: remove some sections

* feat: add block reporting to first description

* fix: reduce redundancy

* feat: update README.md

* Update README.md

* feat: update README.md

* fix: update config.yaml in README

* fix: remove redundant words

* fix: change default datasource

* fix: config.yaml comments

* feat: update README.md

* refactor(logger): format logging (#12)

* feat(logger): format main logger

* feat(logger): use duration ms for gcp output

* refactor(logger): bitcoin node logger

* refactor(logger): indexer logger

* refactor(logger): fix cmd logger

* refactor(logger): logger in config pacakge

* refactor(logger): set pgx error log level debug

* refactor(logger): btcclient datasource

* refactor: processor name

* refactor(logger): runese logger

* refactor(logger): update logger

* fix(runes): wrong btc db datasource

* refactor(logger): remove unnecessary debug log

* refactor: update logger in indexer

* fix(logger): deadlock in load()

* fix: remove unused

---------

Co-authored-by: Gaze <gazenw@users.noreply.github.com>

* feat(btc): remove unused func

* fix: fix golangci-lint error

* fix(pg): update logger level

* doc: update config example

* feat: go mod tidy

* doc: update readme

* fix: panic cause didn't handle error

* doc: update example config

* doc: update example config in readme

* feat(logger): only log error stacktrace when debug mode is on

* feat(reporting): handling invalid config error

* feat(pg): handling invalid config error

* fix: panic in get_token_info

---------

Co-authored-by: Gaze <gazenw@users.noreply.github.com>
Co-authored-by: Planxnx <thanee@cleverse.com>
Co-authored-by: Thanee Charattrakool <37617738+Planxnx@users.noreply.github.com>
2024-04-29 15:16:10 +07:00
2024-04-29 15:16:10 +07:00
2024-04-29 15:16:10 +07:00
2024-04-29 15:16:10 +07:00
2024-04-29 15:16:10 +07:00
2024-04-29 15:16:10 +07:00
2024-04-29 15:16:10 +07:00
2024-04-29 15:16:10 +07:00
2024-04-29 15:16:10 +07:00
2024-04-29 15:16:10 +07:00
2024-03-29 01:09:47 +07:00
2024-04-29 15:16:10 +07:00
2024-04-29 15:16:10 +07:00
2024-04-29 15:16:10 +07:00
2024-04-29 15:16:10 +07:00
2024-04-29 15:16:10 +07:00
2024-04-29 15:16:10 +07:00
2024-04-29 15:16:10 +07:00
2024-04-29 15:16:10 +07:00

Gaze Indexer

Gaze Indexer is an open-source and modular indexing client for Bitcoin meta-protocols. It has support for Bitcoin and Runes out of the box, with Unified Consistent APIs across fungible token protocols.

Gaze Indexer is built with modularity in mind, allowing users to run all modules in one monolithic instance with a single command, or as a distributed cluster of micro-services.

Gaze Indexer serves as a foundation for building ANY meta-protocol indexers, with efficient data fetching, reorg detection, and database migration tool. This allows developers to focus on what truly matters: Meta-protocol indexing logic. New meta-protocols can be easily added by implementing new modules.

Gaze Indexer also comes with a block reporting system for verifying data integrity of indexers. Visit the Gaze Network dashboard to see the status of other indexers.

Modules

1. Bitcoin

The Bitcoin Indexer, the heart of every meta-protocol, is responsible for indexing Bitcoin transactions, blocks, and UTXOs. It requires a Bitcoin Core RPC as source of Bitcoin transactions, and stores the indexed data in database to be used by other modules.

2. Runes

The Runes Indexer is our first meta-protocol indexer. It indexes Runes states, transactions, runestones, and balances using Bitcoin transactions. It comes with a set of APIs for querying historical Runes data. See our API Reference for full details.

Installation

Prerequisites

1. Hardware Requirements

Each module requires different hardware requirements.

Module CPU RAM
Bitcoin 0.25 cores 256 MB
Runes 0.5 cores 1 GB

2. Prepare Bitcoin Core RPC server.

Gaze Indexer needs to fetch transaction data from a Bitcoin Core RPC, either self-hosted or using managed providers like QuickNode. To self host a Bitcoin Core, see https://bitcoin.org/en/full-node.

3. Prepare database.

Gaze Indexer has first-class support for PostgreSQL. If you wish to use other databases, you can implement your own database repository that satisfies each module's Data Gateway interface. Here is our minimum database disk space requirement for each module.

Module Database Storage
Bitcoin 240 GB
Runes 150 GB

4. Prepare config.yaml file.

# config.yaml
logger:
  output: TEXT # Output format for logs. current supported formats: "TEXT" | "JSON" | "GCP"
  debug: false

# Network to run the indexer on. Current supported networks: "mainnet" | "testnet"
network: mainnet

# Bitcoin Core RPC configuration options.
bitcoin_node:
  host: "" # [Required] Host of Bitcoin Core RPC (without https://)
  user: "" # Username to authenticate with Bitcoin Core RPC
  pass: "" # Password to authenticate with Bitcoin Core RPC
  disable_tls: false # Set to true to disable tls

# Block reporting configuration options. See Block Reporting section for more details.
reporting:
  disabled: false # Set to true to disable block reporting to Gaze Network. Default is false.
  base_url: "https://indexer.api.gaze.network" # Defaults to "https://indexer.api.gaze.network" if left empty
  name: "" # [Required if not disabled] Name of this indexer to show on the Gaze Network dashboard
  website_url: "" # Public website URL to show on the dashboard. Can be left empty.
  indexer_api_url: "" # Public url to access this indexer's API. Can be left empty if you want to keep your indexer private.

# HTTP server configuration options.
http_server:
  port: 8080 # Port to run the HTTP server on for modules with HTTP API handlers.

# Meta-protocol modules configuration options.
modules:
  # Configuration options for Bitcoin module. Can be removed if not used.
  bitcoin:
    database: "postgres" # Database to store bitcoin data. current supported databases: "postgres"
    postgres:
      host: "localhost"
      port: 5432
      user: "postgres"
      password: "password"
      db_name: "postgres"
      # url: "postgres://postgres:password@localhost:5432/postgres?sslmode=prefer" # [Optional] This will override other database credentials above.

  # Configuration options for Runes module. Can be removed if not used.
  runes:
    database: "postgres" # Database to store Runes data. current supported databases: "postgres"
    datasource: "database" # Data source to be used for Bitcoin data. current supported data sources: "bitcoin-node" | "database". If "database" is used, it will use the database config in bitcoin module as datasource.
    api_handlers: # API handlers to enable. current supported handlers: "http"
      - http
    postgres:
      host: "localhost"
      port: 5432
      user: "postgres"
      password: "password"
      db_name: "postgres"
      # url: "postgres://postgres:password@localhost:5432/postgres?sslmode=prefer" # [Optional] This will override other database credentials above.

We will be using docker-compose for our installation guide. Make sure the docker-compose.yaml file is in the same directory as the config.yaml file.

# docker-compose.yaml
services:
  gaze-indexer:
    image: ghcr.io/gaze-network/gaze-indexer:v1.0.0
    container_name: gaze-indexer
    restart: unless-stopped
    ports:
      - 8080:8080 # Expose HTTP server port to host
    volumes:
      - "./config.yaml:/app/config.yaml" # mount config.yaml file to the container as "/app/config.yaml"
    command: ["/app/main", "run", "--bitcoin", "--runes"] # Put module flags after "run" commands to select which modules to run.

Install from source

  1. Install go version 1.22 or higher. See Go installation guide here.
  2. Clone this repository.
git clone https://github.com/gaze-network/gaze-indexer.git
cd gaze-indexer
  1. Build the main binary.
# Get dependencies
go mod download

# Build the main binary
go build -o gaze main.go
  1. Run database migrations with the migrate command and module flags.
./gaze migrate up --bitcoin --runes --database postgres://postgres:password@localhost:5432/postgres
  1. Start the indexer with the run command and module flags.
./gaze run --bitcoin --runes

If config.yaml is not located at ./app/config.yaml, use the --config flag to specify the path to the config.yaml file.

./gaze run --bitcoin --runes --config /path/to/config.yaml
Description
No description provided
Readme GPL-3.0 513 KiB
Languages
Go 98.1%
PLpgSQL 1.8%