WIP (do not merge!): refactor pretty much everything (#78)

* refactor handlers

* refactor creation of stds and aggs tablesg

* stds table update trigger

* group crud func into controller module

* consolidate stds and aggs into stats

* key bug fix;

* bug fix func name

* move filtered pool to outlier section]

* func for welford step update

* bug fix imports

* field bug fix

* smol changes

* rename func

* exclusion of projects, pools, lb, ub

* add confirm msg to scripts

* bug fix

* move apy filter to controller

* modify pools query

* update stats boostrap

* add ssm path

* bug fix

* handler args

* welford bug fix

* ignore test

* include test case
This commit is contained in:
slasher125
2022-07-16 02:51:32 +02:00
committed by GitHub
parent 70f3427dc3
commit 9a9ee7d400
34 changed files with 616 additions and 1020 deletions

View File

@@ -0,0 +1,22 @@
import pandas as pd
def trim():
# this script reduces hourly data to daily data (latest value per pool based on timestamp)
# note: i didn't manage to run this directly on db server, hence the script
timestamp = "2022_07_15"
df = pd.read_csv(f"pools_{timestamp}.csv")
df["timestamp"] = pd.to_datetime(df["timestamp"])
df = df.sort_values(["pool", "timestamp"]).reset_index(drop=True)
(
df.groupby(["pool", pd.Grouper(key="timestamp", freq="1D")])
.last()
.reset_index()
.drop(columns=["timestamp"])
.to_json(f"pools_{timestamp}_daily.json", orient="records")
)
if __name__ == "__main__":
trim()