35 Commits

Author SHA1 Message Date
gazenw
c5c9a7bdeb feat: add get Runes info batch api (#73)
* fix: make existing handlers use new total holders usecase

* fix: error msg

* feat: add get token info batch

* feat: add includeHoldersCount in get tokens api

* refactor: extract response mapping

* fix: rename new field and add holdersCount to extend

* fix: query params array

* fix: error msg

* fix: struct tags

* fix: remove error

* feat: add default value to additional fields
2024-10-31 14:14:58 +07:00
Gaze
cffe378beb feat: add etching tx hash for runes info api 2024-10-25 16:22:24 +07:00
gazenw
9739f61067 feat: implement batch insert using multirow inserts (#70)
* feat: add new batch inserts

* fix: migration

* fix: add casting to unnest with patch

* fix: add UTC() to timestamp mappers

* chore: unused imports

* chore: remove unnecessary comments
2024-10-17 14:34:04 +07:00
gazenw
8883c24c77 fix: only call VerifyStates if not api only (#68) 2024-10-15 17:58:51 +07:00
gazenw
96afdfd255 fix: order get ongoing tokens by mint transactions (#66) 2024-10-11 14:23:31 +07:00
gazenw
c49e39be97 feat: optimize flush speed for Runes (#65)
* feat: switch to batchexec for faster inserts

* feat: add time taken log

* feat: add process time log

* feat: add event logs
2024-10-11 14:21:26 +07:00
gazenw
90ed7bc350 fix: update sql (#63) 2024-10-07 21:17:16 +07:00
gazenw
f1d4651042 feat(runes): add Get Transaction by hash api (#39)
* feat: implement pagination on get balance, get holders

* feat: paginate get transactions

* fix: remove debug

* feat: implement pagination in get utxos

* feat: sort response in get holders

* feat: cap batch query

* feat: add default limits to all endpoints

* chore: rename endpoint funcs

* fix: parse rune name spacers

* feat(runes): get tx by hash api

* fix: error

* refactor: use map to collect rune ids

---------

Co-authored-by: Gaze <gazenw@users.noreply.github.com>
2024-10-06 23:50:13 +07:00
gazenw
32c3c5c1d4 fix: correctly insert etchedAt in rune entry (#60) 2024-10-06 20:55:38 +07:00
Gaze
2a572e6d1e fix: migration 2024-10-06 20:29:25 +07:00
gazenw
e1f8eaa3e1 fix: unescape query id (#58) 2024-10-06 19:59:06 +07:00
gazenw
107836ae39 feat(runes): add Get Tokens API (#38)
* feat: implement pagination on get balance, get holders

* feat: paginate get transactions

* fix: remove debug

* feat: implement pagination in get utxos

* feat: sort response in get holders

* feat: cap batch query

* feat: add default limits to all endpoints

* chore: rename endpoint funcs

* fix: parse rune name spacers

* feat(runes): add get token list api

* fix(runes): use distinct to get token list

* feat: remove unused code

* fix: count holders distinct pkscript

* feat: implement additional scopes

* chore: comments

* feat: implement search

* refactor: switch to use paginationRequest

* refactor: rename get token list to get tokens

* fix: count total holders by rune ids

* fix: rename file

* fix: rename minting to ongoing

* fix: get ongoing check rune is mintable

* chore: disable gosec g115

* fix: pr

---------

Co-authored-by: Gaze <gazenw@users.noreply.github.com>
2024-10-06 19:30:57 +07:00
gazenw
754fd1e997 fix: only check for chain reorg if current block has hash (#56)
* fix: only check for chain reorg if current block has hash

* fix: remove starting block hash

* fix: don't use starting block hash
2024-10-05 01:35:04 +07:00
gazenw
f9c6ef8dfd fix: add different genesis runes config for each network (#54)
* fix: add different genesis runes config for each network

* fix: use slogx.Stringer

* refactor: remove unused value
2024-10-04 13:22:53 +07:00
gazenw
b1d9f4f574 feat: add fractal support for runes (#52)
* feat: add fractal support for runes

* chore: remove common.HalvingInterval

* fix: update starting block height

* refactor: move network-genesis-rune definition to constants

* fix: use logger.Panic() instead of panic()

* fix: golangci-lint

* fix: missing return
2024-10-03 18:25:13 +07:00
gazenw
8b690c4f7f fix: add decimals field in runes get holders (#49) 2024-08-30 16:17:57 +07:00
Gaze
0131de6717 feat: add network support for fractal 2024-08-28 23:34:54 +07:00
gazenw
fa810b0aed feat: add get utxo by tx hash and output idx for Runes (#42)
* feat: add handler

* feat: add get transaction

* feat: add get utxos output

* refactor: function parameter

* feat: add check utxo not found

* feat: add sats to get utxo output api

* feat: add utxo sats entity

* feat: add get utxos output batch

* feat: handle error

* fix: context

* fix: sqlc queries

* fix: remove unused code

* fix: comment

* fix: check utxo not found error

* refactor: add some space

* fix: comment

* fix: use public field
2024-08-13 17:20:46 +07:00
waiemwor
dca63a49fe Modify Nodesale API to allow query all nodes from a deployment. (#43)
* feat: allow query all nodes from a deployment.

* fix : function GetNodesByDeployment name.
2024-08-13 16:35:56 +07:00
waiemwor
db5dc75c41 Feature/nodesale (#40)
* feat: recover nodesale module.

* fix: refactored.

* fix: fix table type.

* fix: add entity

* fix: bug UTC time.

* ci: try to tidy before testing

* ci: touch result file

* ci: use echo to create new file

* fix: try to skip test in ci

* fix: remove os.Exit

* fix: handle error

* feat: add todo note

* fix: Cannot run nodesale test because qtx is not initiated.

* fix: 50% chance public key compare incorrectly.

* fix: more consistent SQL

* fix: sanity refactor.

* fix: remove unused code.

* fix: move last_block_default to config file.

* fix: minor mistakes.

* fix:

* fix: refactor

* fix: refactor

* fix: delegate tx hash not record into db.

* refactor: prepare for moving integration tests.

* refactor: convert to unit tests.

* fix: change to using input values since output values deducted fee.

* feat: add extra unit test.

* fix: wrong timestamp format.

* fix: handle block timeout = 0

---------

Co-authored-by: Gaze <gazenw@users.noreply.github.com>
2024-08-05 11:33:20 +07:00
gazenw
7dcbd082ee feat: add Runes API pagination (#36)
* feat: implement pagination on get balance, get holders

* feat: paginate get transactions

* fix: remove debug

* feat: implement pagination in get utxos

* feat: sort response in get holders

* feat: cap batch query

* feat: add default limits to all endpoints

* chore: rename endpoint funcs

* fix: parse rune name spacers

* chore: use compare.Cmp

* feat: handle not found errors on all usecase
2024-07-23 15:46:45 +07:00
gazenw
880f4b2e6a fix: handle case where input rune id is not found (#37) 2024-07-15 18:32:28 +07:00
Gaze
694baef0aa chore: golangci-lint 2024-07-09 15:48:09 +07:00
gazenw
47119c3220 feat: remove unnecessary verbose query (#33) 2024-07-09 15:44:14 +07:00
gazenw
b24f27ec9a fix: incorrect condition for finding output destinations (#31) 2024-07-08 14:32:58 +07:00
gazenw
7cb717af11 feat(runes): get txs by block range (#28)
* feat(runes): get txs by block range

* feat(runes): validate block range

* perf(runes): limit 10k txs

---------

Co-authored-by: Gaze <gazenw@users.noreply.github.com>
2024-06-30 18:45:23 +07:00
Nut Pinyo
d84e30ed11 fix: implement Shutdown() for processors (#22) 2024-05-31 01:13:12 +07:00
Nut Pinyo
62ecd7ea49 fix: runes tag parsing (#19) 2024-05-16 13:54:13 +07:00
gazenw
66ea2766a0 refactor: Separated modules initiator (#17)
* refactor: separated modules initiator

* fix: able to run with empty modules

* doc: update cmd desc

* refactor: sorting code flow

* fix: invalid apionly flow

* refactor: remove unnecessary flags

* feat: add default value for runes config

* refactor: use config instead bind flag to opts struct

* chore: remove unused comment

* refactor(runes): invoke only when in case

* feat: add http server default port

* feat: add logger context

* doc: update readme

---------

Co-authored-by: Gaze <gazenw@users.noreply.github.com>
2024-05-15 16:14:29 +07:00
gazenw
575c144428 fix: invalid pgx version (#18)
Co-authored-by: Planxnx <planxnx@users.noreply.github.com>
2024-05-15 03:19:10 +07:00
gazenw
c75b62bdf9 Remove bitcoin indexer (#16)
* doc: update README.md

* fix: remove bitcoin module

* fix: remove more config
2024-05-14 19:29:43 +07:00
gazenw
c9a5c6d217 refactor: Generic Indexer (#14)
* refactor(indexer): init generic indexer

* refactor(btc): updatet datasource

* refactor: remove old indexers pkg

* doc: update comment

---------

Co-authored-by: Gaze <gazenw@users.noreply.github.com>
Co-authored-by: Planxnx <thanee@cleverse.com>
2024-05-08 16:15:33 +07:00
Gaze
c6057d9511 perf(btc): remove unnecessary db index 2024-05-07 21:29:12 +07:00
gazenw
fcdecd4046 feat: v0.1.0 release (#13)
* fix: don't remove first block

* fix: make etching_terms nullable

* fix: fix panic if empty pkscript

* chore: change testnet starting block

* feat: more logs

* fix: extract tapscript bug

* feat: more logs

* fix: switch pk to block height

* chore: remove redundant log

* fix: repo

* fix: not found error

* fix: golangci-lint

* feat: add etching tx hash to rune entries

* feat: stop main if indexer failed

* fix: check balance after populating current balance

* fix: sql ambiguous column

* feat: add tx hash and out index in tx output

* fix: actually use transactions to write db

* fix: create rune entry states only during flushes

* fix: mint cap reached off by one

* fix: debug log unsafe

* feat: prevent processing of txs before activation height

* feat: add rune number to rune entry

* feat: include new rune entries in event hash and flushing

* refactor(config): separate init and get config func

Co-authored-by: Gaze <dev@gaze.network>

* feat: remove annoying log

Co-authored-by: Gaze <dev@gaze.network>

* feat: mod tidy

Co-authored-by: Gaze <dev@gaze.network>

* refactor: move main to root

Co-authored-by: Gaze <dev@gaze.network>

* feat(cli): create cli commands

Co-authored-by: Gaze <dev@gaze.network>

* refactor: move main logic to command

Co-authored-by: Gaze <dev@gaze.network>

* doc: remove unused desc

Co-authored-by: Gaze <dev@gaze.network>

* refactor: test structure in runestone_test.go

* fix: edict flaws were ignored

* feat: more tests

* refactor(cli): add local flag

Co-authored-by: Gaze <dev@gaze.network>

* feat: set symbol limit to utf8.MaxRune

* refactor(cli): flags for each module

Co-authored-by: Gaze <dev@gaze.network>

* feat(cli): support db selection

Co-authored-by: Gaze <dev@gaze.network>

* fix: remove temp code

Co-authored-by: Gaze <dev@gaze.network>

* fix: get data from cache in processor first, then dg

* feat(cli): add version command

Co-authored-by: Gaze <dev@gaze.network>

* doc(cli): add refactor plan

Co-authored-by: Gaze <dev@gaze.network>

* refactor(cli): rename files

Co-authored-by: Gaze <dev@gaze.network>

* feat: add main.go

Co-authored-by: Gaze <dev@gaze.network>

* feat: more tests

* feat: add overflow err

* feat: finish runestone tests

* refactor(cli): separate protocol config and cli flag

Co-authored-by: Gaze <dev@gaze.network>

* chore(btc): update example config

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): add get block header to datasource interface

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): reorg handling

Co-authored-by: Gaze <dev@gaze.network>

* fix: interface

Co-authored-by: Gaze <dev@gaze.network>

* fix: rename postgres config key

* fix: migrated runes indexer integration to new cli

* fix: commit every block

* feat(btc): add revert data query

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): add revert data to processor

Co-authored-by: Gaze <dev@gaze.network>

* feat: implement public errors

* fix: use errs in api

* refactor: move api and usecase outside of internal

* feat: add custom opcode check for datapush

* fix: break if input utxo is not P2TR

* fix: zero len destination case

* fix: get the rest of transaction data in GetTransaction

* refactor: create subscription utils tools

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): add btc_database from datasource

Co-authored-by: Gaze <dev@gaze.network>

* doc(btc): add note

Co-authored-by: Gaze <dev@gaze.network>

* wip(btc): imple prepare range func

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): add pg queries for datasource

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): update queries

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): implement repo for get blocks

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): update dg

Co-authored-by: Gaze <dev@gaze.network>

* fix(btc): return nil if errors

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): update fetch async for db datasource

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): add get block header from db for reorg handling

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): add todo notes

Co-authored-by: Gaze <dev@gaze.network>

* feat: implement get tx by hash

* fix: rename func

* fix: rename func

* fix: rename func

* fix: fix get transaction by hash

* feat: integrate bitcoin client db to main

* fix: reduce chunk size

* fix: stop main if bitcoin indexer failed

* fix: stop main if runes indexer failed

* fix: move stop() inside goroutine

* chore: add log

* fix: duplicate rune entry number

* feat(btc): add witness utils

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): witness datamodel parsing

Co-authored-by: Gaze <dev@gaze.network>

* fix(btc): invalid table name

Co-authored-by: Gaze <dev@gaze.network>

* fix(btc): remove uniqte index for hash

Co-authored-by: Gaze <dev@gaze.network>

* doc: add todo

Co-authored-by: Gaze <dev@gaze.network>

* feat(logger): remove error verbose

Co-authored-by: Gaze <dev@gaze.network>

* feat: support postgresql db

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): add err notfound

Co-authored-by: Gaze <dev@gaze.network>

* fix: invalid pgx version

Co-authored-by: Gaze <dev@gaze.network>

* fix: invalid indexer flow

Co-authored-by: Gaze <dev@gaze.network>

* feat: refactor runes api

* feat: implement http server

* fix: mount runes api

* fix: error handler

* fix: first empty state error

Co-authored-by: Gaze <dev@gaze.network>

* fix: off by one confirmation

* ci: ignore RollBack error

* fix: change WithPublicMessage to be prefix

* feat: bump cstream version

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): nullable pkscript

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): change rollback style

Co-authored-by: Gaze <dev@gaze.network>

* refactor: move runes out of internal

* feat: rename id field to runeId in rune transaction

* feat(btc): update index

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): add default current block

Co-authored-by: Gaze <dev@gaze.network>

* doc: add note

Co-authored-by: Gaze <dev@gaze.network>

* fix(btc): use int64 to store sequence

Co-authored-by: Gaze <dev@gaze.network>

* fix(btc): upgrade data type for numbers

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc):  upgrade data type for idx

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): get indexed block impl

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): add common.ZeroHash

Co-authored-by: Gaze <dev@gaze.network>

* feat: add chainparam

* feat: implement get transactions

* fix: wrong condition for non-OP_RETURN output

* feat(btc): add verify indexer states

Co-authored-by: Gaze <dev@gaze.network>

* refactor: sorting code

Co-authored-by: Gaze <dev@gaze.network>

* feat: fix interface

* feat(btc): update chuunk size

Co-authored-by: Gaze <dev@gaze.network>

* feat: add rune_etched column in rune transaction

* fix: missing field in create

* feat: add runeEtched in get transactions

* feat: implement get token info

* feat: add holders count in token info

* feat: implement get holders

* fix: return a new repository when beginning a new tx

* fix: rename type

* feat: add pkscript to outpoint balance

* feat: implement get utxos by address api

* fix: spend outpoint bug

* feat: implement get balances by address batch

* feat: sort balances result by amount

* ci: create Dockerfile

Co-authored-by: Gaze <dev@gaze.network>

* ci: add arg run

Co-authored-by: Gaze <dev@gaze.network>

* perf: add automaxprocs

Co-authored-by: Gaze <dev@gaze.network>

* chore: add performance logging

Co-authored-by: Gaze <dev@gaze.network>

* chore: add performance logger for debyug

Co-authored-by: Gaze <dev@gaze.network>

* fix: empty etched at

* fix: revert data sequentially

* fix: remove unused funcs

* fix: main.go

* feat: add flag --api-only to run cmd

* fix: create index

* fix: don't add zero mint to unallocated

* fix: ignore zero burn amount

* feat(reorg): add reorg detail

Co-authored-by: Gaze <dev@gaze.network>

* fix: wrong index type

* feat: implement reporting client to report runes blocks

* feat: implement report node

* feat(runes): add latest block api

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): use logger warn

Co-authored-by: Gaze <dev@gaze.network>

* fix(btc): txout aren't revert if it's have to revert spent

Co-authored-by: Gaze <dev@gaze.network>

* fix: annoying error when unsubscribe fetcher

Co-authored-by: Gaze <dev@gaze.network>

* refactor(btc): readable code

Co-authored-by: Gaze <dev@gaze.network>

* fix(indexer): fix subscription closed before process when success fetch

Co-authored-by: Gaze <dev@gaze.network>

* fix: remove module enum

* fix: increase max reorg limit

* feat: add starting height for runes mainnet

* fix(btc): fix `with` modified same row twice

Co-authored-by: Gaze <dev@gaze.network>

* fix(runes): handling latest block not found

Co-authored-by: Gaze <dev@gaze.network>

* feat: add decimals in get transactions

* fix: wrong condition

* feat: add more index

* feat: implement get transactions by pkscript

* feat: allow query by rune id too

* feat: more comments

* perf(btc): bitcoin indexer performance optimization (#4)

* feat(btc): not null to witness

Co-authored-by: Gaze <dev@gaze.network>

* perf(btc): add batch insert txin

Co-authored-by: Gaze <dev@gaze.network>

* perf(btc): batch insert txout

Co-authored-by: Gaze <dev@gaze.network>

* perf(btc): batch insert transaction

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): remove old queries

Co-authored-by: Gaze <dev@gaze.network>

* fix(btc): typo

Co-authored-by: Gaze <dev@gaze.network>

* perf(btc): batch insert blocks (#5)

Co-authored-by: Gaze <gazenw@users.noreply.github.com>

---------

Co-authored-by: Gaze <gazenw@users.noreply.github.com>

* feat(btc): Duplicate coinbase transaction handling (#7)

* feat(btc): tx_hash can duplicated in block v1

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): duplicate tx  will use same txin/txout from previous tx

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): prevent revert block v1 data

if you really want to revert the data before the block version 2, you should reset the database and reindex the data instead.

Co-authored-by: Gaze <dev@gaze.network>

* doc(btc): update list duplicate tx hash

Co-authored-by: Gaze <dev@gaze.network>

* doc(btc): update docs

Co-authored-by: Gaze <dev@gaze.network>

* fix(btc): use last v1 block instead

Co-authored-by: Gaze <dev@gaze.network>

---------

Co-authored-by: Gaze <gazenw@users.noreply.github.com>

* feat: add ping handler

* fix: type

Co-authored-by: Gaze <dev@gaze.network>

* doc: add refactor note

Co-authored-by: Gaze <dev@gaze.network>

* ci: add golang linter and test runner gh action

* ci: use go-test-action@v0

* ci: annotate test result

* ci: update running flag

* fix: try to fix malformed import path

* feat: add mock test

* ci: remove annotation ci

* ci: add annotate test result

* chore: remove unused

* feat: try testify

* feat: remove test

* ci: add go test on macos, windows and go latest version

* ci: test building

* feat: remove mock code

* ci: add sqlc diff checker action (#10)

* feat: Graceful shutdown (#8)

* feat: add shutdown function for indexer

Co-authored-by: Gaze <dev@gaze.network>

* feat: add force shutdown

Co-authored-by: Gaze <dev@gaze.network>

* revert

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): remove unused

Co-authored-by: Gaze <dev@gaze.network>

* style: go fmt

Co-authored-by: Gaze <dev@gaze.network>

* feat: separate context for worker and application

* feat: increase force shutdown timeout

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): update logging

Co-authored-by: Gaze <dev@gaze.network>

* feat(btc): update shutdown function

Co-authored-by: Gaze <dev@gaze.network>

* feat: remove wg for shutdown

Co-authored-by: Gaze <dev@gaze.network>

* feat: refactor shutdown flow

Co-authored-by: Gaze <dev@gaze.network>

* feat: update shutdown flow

Co-authored-by: Gaze <dev@gaze.network>

* feat: update maming

Co-authored-by: Gaze <dev@gaze.network>

* feat: update force shutdown logic

Co-authored-by: Gaze <dev@gaze.network>

---------

Co-authored-by: Gaze <gazenw@users.noreply.github.com>

* feat: check reporting config name

* fix: use db config in bitcoin module for runes datasource

* Add migrate commands (#2)

* feat: add migrate up

* feat: add down migration

* fix: example

* feat: change description

* fix: hardcode migration source directory

* Update README.md for public release. (#11)

* feat: initial draft for README.md

* fix: remove some sections

* feat: add block reporting to first description

* fix: reduce redundancy

* feat: update README.md

* Update README.md

* feat: update README.md

* fix: update config.yaml in README

* fix: remove redundant words

* fix: change default datasource

* fix: config.yaml comments

* feat: update README.md

* refactor(logger): format logging (#12)

* feat(logger): format main logger

* feat(logger): use duration ms for gcp output

* refactor(logger): bitcoin node logger

* refactor(logger): indexer logger

* refactor(logger): fix cmd logger

* refactor(logger): logger in config pacakge

* refactor(logger): set pgx error log level debug

* refactor(logger): btcclient datasource

* refactor: processor name

* refactor(logger): runese logger

* refactor(logger): update logger

* fix(runes): wrong btc db datasource

* refactor(logger): remove unnecessary debug log

* refactor: update logger in indexer

* fix(logger): deadlock in load()

* fix: remove unused

---------

Co-authored-by: Gaze <gazenw@users.noreply.github.com>

* feat(btc): remove unused func

* fix: fix golangci-lint error

* fix(pg): update logger level

* doc: update config example

* feat: go mod tidy

* doc: update readme

* fix: panic cause didn't handle error

* doc: update example config

* doc: update example config in readme

* feat(logger): only log error stacktrace when debug mode is on

* feat(reporting): handling invalid config error

* feat(pg): handling invalid config error

* fix: panic in get_token_info

---------

Co-authored-by: Gaze <gazenw@users.noreply.github.com>
Co-authored-by: Planxnx <thanee@cleverse.com>
Co-authored-by: Thanee Charattrakool <37617738+Planxnx@users.noreply.github.com>
2024-04-29 15:16:10 +07:00
Gaze
50062f80e3 feat: draft architecture 2024-04-01 17:21:02 +07:00