+
+ | Currency |
+ Address |
+ QR |
+
+
+ | ₿ BTC |
+ bc1qwn7ea6s8wqx66hl5rr2supk4kv7qtcxnlqcqfk |
+  |
+
+
+ | Ξ ETH |
+ 0x7cdB1861AC1B4385521a6e16dF198e7bc43fDE5f |
+  |
+
+
+ | ɱ XMR |
+ 463fMSWyDrk9DVQ8QCiAir8TQd4h3aRAiDGA8CKKjknGaip7cnHGmS7bQmxSiS2aYtE9tT31Zf7dSbK1wyVARNgA9pkzVxX |
+  |
+
+
+ | ◈ BNB |
+ 0x7cdB1861AC1B4385521a6e16dF198e7bc43fDE5f |
+  |
+
+
+
+## 📦 Installation
+
+### PyPI (Recommended)
+
+```bash
+pip install hyperdrive -U
```
-pythom -m pip install hyperdrive -U
-```
-
-## Examples
-Most secrets must be passed as environment variables. Future updates will allow secrets to be passed directly into class object (see example on order execution).
-
-
+## 🚀 Examples
+
+Most secrets must be passed as environment variables. Future updates will allow secrets to be passed directly into class objects (see example on order execution).
### 1. Storing data
@@ -94,7 +98,7 @@ Environment Variables:
- `AWS_DEFAULT_REGION`
- `S3_BUCKET`
-```
+```python
from hyperdrive import DataSource
from DataSource import Polygon, MarketData
@@ -131,7 +135,7 @@ Much of this code is still closed-source, but you can take a look at the [`Histo
We use [_vectorbt_](https://vectorbt.dev/) to backtest strategies.
-```
+```python
from hyperdrive import History, DataSource, Constants as C
from History import Historian
from DataSource import MarketData
@@ -222,7 +226,7 @@ Environment Variables:
- `BINANCE`
-```
+```python
from pprint import pprint
from hyperdrive import Exchange
from Exchange import Binance
@@ -260,28 +264,107 @@ Output:
'type': 'MARKET'}
```
-## Use
+## 🏗️ Architecture
-Use the scripts provided in the [`scripts/`](https://github.com/suchak1/hyperdrive/tree/master/scripts) directory as a reference since they are actually used in production daily.
+### Project Structure
-Available data collection functions:
+```
+hyperdrive/
+├── hyperdrive/ # Core package
+│ ├── DataSource.py # Market data providers (Polygon, Robinhood)
+│ ├── Exchange.py # Trading integrations (Binance, Alpaca, Kraken)
+│ ├── History.py # Backtesting with vectorbt
+│ ├── Precognition.py # ML predictions with AutoGluon
+│ ├── Storage.py # S3 cloud storage
+│ ├── Broker.py # Order execution
+│ ├── Constants.py # Shared constants
+│ └── _version.py # Package version
+├── scripts/ # Automation & data update scripts
+├── tests/ # Unit and integration tests
+├── .github/workflows/ # CI/CD pipelines
+│ ├── pr.yml # PR checks + TestPyPI
+│ ├── release.yml # Release + PyPI
+│ ├── test.yml # Reusable test workflow
+│ └── *.yml # Data collection jobs
+└── pyproject.toml # Project configuration
+```
-- [x] [](https://github.com/suchak1/hyperdrive/actions?query=workflow%3ASymbols) (from Robinhood)
-- [x] [](https://github.com/suchak1/hyperdrive/actions?query=workflow%3AOHLC) (from Polygon)
-- [x] [](https://github.com/suchak1/hyperdrive/actions?query=workflow%3AIntraday) (from Polygon)
-- [x] [](https://github.com/suchak1/hyperdrive/actions?query=workflow%3ADividends) (from Polygon)
-- [x] [](https://github.com/suchak1/hyperdrive/actions?query=workflow%3ASplits) (from Polygon)
-
-- [x] [](https://github.com/suchak1/hyperdrive/actions?query=workflow%3AUnemployment) (from the Bureau of Labor Statistics)
+### End-to-End Flow
+
+```mermaid
+flowchart LR
+ subgraph Data["📊 Data Sources"]
+ Polygon[Polygon API]
+ RH[Robinhood]
+ BLS[BLS API]
+ end
+
+ subgraph Core["🔧 hyperdrive"]
+ DS[DataSource]
+ S3[(S3 Storage)]
+ Hist[History]
+ ML[Precognition]
+ end
+
+ subgraph Trade["💹 Exchanges"]
+ Binance[Binance]
+ Alpaca[Alpaca]
+ Kraken[Kraken]
+ end
+
+ subgraph CI["⚙️ GitHub Actions"]
+ Scheduled[Scheduled Jobs]
+ PR[pr.yml]
+ Release[release.yml]
+ end
+
+ Polygon --> DS
+ RH --> DS
+ BLS --> DS
+ DS --> S3
+ S3 --> Hist
+ Hist --> ML
+ ML --> Binance
+ ML --> Alpaca
+ ML --> Kraken
+ Scheduled --> DS
+ PR --> |TestPyPI| S3
+ Release --> |PyPI| S3
+```
----
+## 🧪 Testing
-
-
-
-
-
+Run the test suite:
+```bash
+make test
```
+Run with coverage reporting:
+
+```bash
+make cov
+```
+
+Run smoke tests:
+
+```bash
+make smoke
```
+
+## 📊 Data Collection
+
+Use the scripts provided in the [`scripts/`](https://github.com/suchak1/hyperdrive/tree/master/scripts) directory as a reference since they are actually used in production daily.
+
+Available data collection workflows:
+
+- [](https://github.com/suchak1/hyperdrive/actions?query=workflow%3ASymbols) (from Robinhood)
+- [](https://github.com/suchak1/hyperdrive/actions?query=workflow%3AOHLC) (from Polygon)
+- [](https://github.com/suchak1/hyperdrive/actions?query=workflow%3AIntraday) (from Polygon)
+- [](https://github.com/suchak1/hyperdrive/actions?query=workflow%3ADividends) (from Polygon)
+- [](https://github.com/suchak1/hyperdrive/actions?query=workflow%3ASplits) (from Polygon)
+- [](https://github.com/suchak1/hyperdrive/actions?query=workflow%3AUnemployment) (from Bureau of Labor Statistics)
+
+## 📄 License
+
+MIT License — see [LICENSE](LICENSE) for details.
diff --git a/assets/qr_bnb.png b/assets/qr_bnb.png
new file mode 100644
index 00000000..eda4aae7
Binary files /dev/null and b/assets/qr_bnb.png differ
diff --git a/assets/qr_btc.png b/assets/qr_btc.png
new file mode 100644
index 00000000..4bcb9d64
Binary files /dev/null and b/assets/qr_btc.png differ
diff --git a/assets/qr_eth.png b/assets/qr_eth.png
new file mode 100644
index 00000000..eda4aae7
Binary files /dev/null and b/assets/qr_eth.png differ
diff --git a/assets/qr_xmr.png b/assets/qr_xmr.png
new file mode 100644
index 00000000..f37978ec
Binary files /dev/null and b/assets/qr_xmr.png differ
diff --git a/encrypted/generate_preview.py.encrypted b/encrypted/generate_preview.py.encrypted
new file mode 100644
index 00000000..9fe8a3a9
Binary files /dev/null and b/encrypted/generate_preview.py.encrypted differ
diff --git a/hyperdrive/Algotrader.py b/hyperdrive/Algotrader.py
index a524a657..7abd480f 100755
--- a/hyperdrive/Algotrader.py
+++ b/hyperdrive/Algotrader.py
@@ -1,7 +1,18 @@
+"""HyperDrive main algorithmic trading class."""
+
+
class HyperDrive:
- # HyperDrive should have attrs Broker, Backtester, Strategy
+ """Main algorithmic trading orchestrator.
+
+ HyperDrive coordinates trading operations by integrating broker connections,
+ backtesting capabilities, and trading strategies.
+
+ Attributes:
+ None currently - placeholder for future implementation.
+ """
- def __init__(self):
+ def __init__(self) -> None:
+ """Initialize HyperDrive instance."""
pass
diff --git a/hyperdrive/Broker.py b/hyperdrive/Broker.py
index b2467940..29f694df 100755
--- a/hyperdrive/Broker.py
+++ b/hyperdrive/Broker.py
@@ -1,21 +1,50 @@
+"""Robinhood brokerage integration for portfolio management."""
+
import os
+from typing import Any
+
+import pandas as pd
import pyotp
import robin_stocks.robinhood as rh
-from dotenv import load_dotenv, find_dotenv
-import pandas as pd
-from Constants import PathFinder
-import Constants as C
-from FileOps import FileReader, FileWriter
+from dotenv import find_dotenv, load_dotenv
+
+from . import Constants as C
+from .Constants import PathFinder
+from .FileOps import FileReader, FileWriter
class Robinhood:
- def __init__(self, usr=None, pwd=None, mfa=None):
+ """Robinhood brokerage client for portfolio operations.
+
+ Handles authentication and provides methods for retrieving
+ historical data, holdings, and portfolio symbols.
+
+ Attributes:
+ api: The robin_stocks Robinhood API module.
+ writer: FileWriter instance for saving data.
+ reader: FileReader instance for loading data.
+ finder: PathFinder instance for path resolution.
+ """
+
+ def __init__(
+ self,
+ usr: str | None = None,
+ pwd: str | None = None,
+ mfa: str | None = None,
+ ) -> None:
+ """Initialize and authenticate with Robinhood.
+
+ Args:
+ usr: Robinhood username. Defaults to RH_USERNAME env var.
+ pwd: Robinhood password. Defaults to RH_PASSWORD env var.
+ mfa: MFA code. Defaults to generated code from RH_2FA env var.
+ """
# Authentication
- load_dotenv(find_dotenv('config.env'))
+ load_dotenv(find_dotenv("config.env"))
- username = usr or os.environ['RH_USERNAME']
- password = pwd or os.environ['RH_PASSWORD']
- mfa_code = mfa or pyotp.TOTP(os.environ['RH_2FA']).now()
+ username = usr or os.environ["RH_USERNAME"]
+ password = pwd or os.environ["RH_PASSWORD"]
+ mfa_code = mfa or pyotp.TOTP(os.environ["RH_2FA"]).now()
rh.login(username, password, mfa_code=mfa_code)
self.api = rh
@@ -23,54 +52,124 @@ def __init__(self, usr=None, pwd=None, mfa=None):
self.reader = FileReader()
self.finder = PathFinder()
- def flatten(self, xxs):
- # flattens 2d list into 1d list
+ def flatten(self, xxs: list[list[Any]]) -> list[Any]:
+ """Flatten a 2D list into a 1D list.
+
+ Args:
+ xxs: A list of lists to flatten.
+
+ Returns:
+ A single flattened list.
+ """
return [x for xs in xxs for x in xs]
- def get_hists(self, symbols, span='year', interval='day', save=False):
- # given a list of symbols,
- # return a DataFrame with historical data
- hists = [self.api.get_stock_historicals(
- symbol, interval, span) for symbol in symbols]
+ def get_hists(
+ self,
+ symbols: list[str],
+ span: str = "year",
+ interval: str = "day",
+ save: bool = False,
+ ) -> pd.DataFrame:
+ """Get historical price data for multiple symbols.
+
+ Args:
+ symbols: List of stock symbols.
+ span: Time span for historical data.
+ interval: Data interval (e.g., 'day', 'hour').
+ save: Whether to save the data to CSV.
+
+ Returns:
+ DataFrame with historical OHLC data.
+ """
+ hists = [
+ self.api.get_stock_historicals(symbol, interval, span) for symbol in symbols
+ ]
clean = [hist for hist in hists if hist != [None]]
df = pd.DataFrame.from_records(self.flatten(clean))
# look into diff b/w tz_localize and tz_convert w param 'US/Eastern'
# ideally store utc time
- df['begins_at'] = pd.to_datetime(df['begins_at']).apply(
- lambda x: x.tz_localize(None))
+ df["begins_at"] = pd.to_datetime(df["begins_at"]).apply(
+ lambda x: x.tz_localize(None)
+ )
# df = df.sort_values('begins_at')
if save:
- self.writer.save_csv('data/data.csv', df)
+ self.writer.save_csv("data/data.csv", df)
return df
- def get_names(self, symbols):
- # given a list of stock symbols
- # return a list of company names
+ def get_names(self, symbols: list[str]) -> list[str]:
+ """Get company names for a list of stock symbols.
+
+ Args:
+ symbols: List of stock symbols.
+
+ Returns:
+ List of company names.
+ """
names = []
for symbol in symbols:
- if hasattr(self, 'holdings') and symbol in self.holdings:
- names.append(self.holdings[symbol]['name'])
+ if hasattr(self, "holdings") and symbol in self.holdings:
+ names.append(self.holdings[symbol]["name"])
else:
names.append(self.api.get_name_by_symbol(symbol))
return names
- def save_symbols(self):
- # save all the portfolio symbols in a table
+ def save_symbols(self) -> None:
+ """Save portfolio symbols and names to CSV."""
symbols = self.get_symbols()
names = self.get_names(symbols)
- df = pd.DataFrame({
- C.SYMBOL: symbols,
- C.NAME: names
- })
+ df = pd.DataFrame({C.SYMBOL: symbols, C.NAME: names})
self.writer.save_csv(self.finder.get_symbols_path(), df)
- def get_holdings(self):
- if not hasattr(self, 'holdings'):
+ def get_holdings(self) -> dict[str, Any]:
+ """Get current portfolio holdings.
+
+ Returns:
+ Dictionary of holdings keyed by symbol.
+ """
+ if not hasattr(self, "holdings"):
self.holdings = self.api.build_holdings()
return self.holdings
- def get_symbols(self):
- if not hasattr(self, 'holdings'):
+ def get_symbols(self) -> list[str]:
+ """Get list of symbols in portfolio.
+
+ Returns:
+ List of stock symbols in holdings.
+ """
+ if not hasattr(self, "holdings"):
self.get_holdings()
- return [symbol for symbol in self.holdings]
+ return list(self.holdings)
+
+ def get_dividends(self) -> list[dict[str, Any]]:
+ """Get dividends for all stocks in portfolio.
+
+ Returns:
+ List of dividends.
+ """
+ if not hasattr(self, "dividends"):
+ self.dividends = self.api.get_dividends()
+ return self.dividends
+
+ def get_options(self) -> list[dict[str, Any]]:
+ """Get options for all stocks in portfolio.
+
+ Returns:
+ List of options.
+ """
+ if not hasattr(self, "options"):
+ self.options = self.api.get_all_option_orders()
+ return self.options
+
+ def get_events(self, symbol: str) -> list[dict[str, Any]]:
+ """Get events for all stocks in portfolio.
+
+ Args:
+ symbol: Stock symbol to get events for.
+
+ Returns:
+ List of events.
+ """
+ if not hasattr(self, "events"):
+ self.events = self.api.get_events(symbol)
+ return self.events
diff --git a/hyperdrive/Calculus.py b/hyperdrive/Calculus.py
index bcb81574..ef9debb2 100755
--- a/hyperdrive/Calculus.py
+++ b/hyperdrive/Calculus.py
@@ -1,62 +1,151 @@
+"""Mathematical and geometric calculation utilities."""
import math
+from collections.abc import Sequence
+from itertools import permutations
+from typing import Any
+
import numpy as np
import pandas as pd
-from numpy.linalg import norm
from icosphere import icosphere
-from itertools import permutations
+from numpy.linalg import norm
from scipy.signal import savgol_filter
+# Type alias for 3D point coordinates (can be tuple, list, or array-like)
+Point3D = Sequence[float] | np.ndarray
+
class Calculator:
- def avg(self, xs):
+ """Mathematical utility class for various calculations.
+
+ Provides methods for statistics, derivatives, geometry,
+ and 3D shape operations.
+ """
+
+ def avg(self, xs: list[float] | np.ndarray) -> float:
+ """Calculate the mean of a collection of numbers.
+
+ Args:
+ xs: Collection of numbers.
+
+ Returns:
+ The arithmetic mean.
+ """
return np.mean(xs)
- def find_centroid(self, points, method='mean'):
+ def find_centroid(self, points: np.ndarray, method: str = "mean") -> list[float]:
+ """Find the centroid of a set of points.
+
+ Args:
+ points: Array of points with shape (n_points, n_dimensions).
+ method: 'mean' for average, 'extrema' for midpoint of bounds.
+
+ Returns:
+ List of coordinates for the centroid.
+ """
components = points.T
- if method != 'mean':
- components = [
- [
- min(component),
- max(component)
- ]
- for component in components
- ]
+ if method != "mean":
+ components = [[min(component), max(component)] for component in components]
return [self.avg(component) for component in components]
- def delta(self, series, window=1):
+ def delta(self, series: pd.Series, window: int = 1) -> pd.Series:
+ """Calculate percentage change over a rolling window.
+
+ Args:
+ series: Input time series.
+ window: Lag period for comparison.
+
+ Returns:
+ Series of percentage changes.
+ """
return series / series.shift(window) - 1
- def roll(self, series, window):
+ def roll(self, series: pd.Series, window: int) -> pd.Series:
+ """Calculate rolling mean of a series.
+
+ Args:
+ series: Input time series.
+ window: Rolling window size.
+
+ Returns:
+ Rolling mean series.
+ """
return series.rolling(window).mean()
- def smooth(self, series, window, order):
+ def smooth(self, series: pd.Series, window: int, order: int) -> np.ndarray:
+ """Smooth a series using Savitzky-Golay filter.
+
+ Args:
+ series: Input time series.
+ window: Filter window size (will be adjusted to odd number).
+ order: Polynomial order for fitting.
+
+ Returns:
+ Smoothed array.
+ """
return savgol_filter(
- series,
- window + 1 if window % 2 == 0 else window + 2,
- order
+ series, window + 1 if window % 2 == 0 else window + 2, order
)
- def get_difference(self, old: set, new: set) -> tuple[set, set]:
+ def get_difference(self, old: set[Any], new: set[Any]) -> tuple[set[Any], set[Any]]:
+ """Find elements removed and added between two sets.
+
+ Args:
+ old: Original set.
+ new: New set.
+
+ Returns:
+ Tuple of (removed elements, added elements).
+ """
minus = old.difference(new)
plus = new.difference(old)
return minus, plus
- def derive(self, y, x=np.array([0, 1])):
+ def derive(
+ self, y: np.ndarray | pd.Series, x: np.ndarray | pd.Series | None = None
+ ) -> np.ndarray:
+ """Compute the numerical derivative of y with respect to x.
+
+ Args:
+ y: Dependent variable array.
+ x: Independent variable array. Defaults to [0, 1] for unit spacing.
+
+ Returns:
+ Array of derivative values.
+ """
+ if x is None:
+ x = np.array([0, 1])
if isinstance(x, pd.Series):
x = x.to_numpy()
- x = x.astype('float64')
- x_delta = (x[1] - x[0])
+ x = x.astype("float64")
+ x_delta = x[1] - x[0]
return np.gradient(y, x_delta)
- def cv(self, x, ddof=0):
+ def cv(self, x: pd.Series | np.ndarray, ddof: int = 0) -> float | np.ndarray:
+ """Calculate coefficient of variation.
+
+ Args:
+ x: Input data (Series or 2D array).
+ ddof: Delta degrees of freedom for std calculation.
+
+ Returns:
+ Coefficient of variation (std/mean).
+ """
if isinstance(x, pd.Series):
axis = 0
else:
axis = 1
return x.std(axis=axis, ddof=ddof) / x.mean(axis=axis)
- def fib(self, n):
+ def fib(self, n: int) -> list[int]:
+ """Generate Fibonacci sequence up to n terms.
+
+ Args:
+ n: Number of terms to generate.
+
+ Returns:
+ List of Fibonacci numbers.
+ """
if n <= 1:
return [0]
elif n == 2:
@@ -65,46 +154,104 @@ def fib(self, n):
lst = self.fib(n - 1)
return self.fib(n - 1) + [lst[-1] + lst[-2]]
- def find_plane(self, pt1, pt2, pt3):
- pt1, pt2, pt3 = [
- np.array(pt) for pt in [pt1, pt2, pt3]
- ]
- u = pt2 - pt1
- v = pt3 - pt1
+ def find_plane(
+ self,
+ pt1: Point3D,
+ pt2: Point3D,
+ pt3: Point3D,
+ ) -> tuple[float, float, float, float]:
+ """Find plane equation coefficients from three points.
- point = np.array(pt1)
+ Args:
+ pt1: First point on the plane.
+ pt2: Second point on the plane.
+ pt3: Third point on the plane.
+
+ Returns:
+ Tuple (a, b, c, d) for plane equation ax + by + cz + d = 0.
+ """
+ pt1_arr, pt2_arr, pt3_arr = [np.array(pt) for pt in [pt1, pt2, pt3]]
+ u = pt2_arr - pt1_arr
+ v = pt3_arr - pt1_arr
+
+ point = np.array(pt1_arr)
normal = np.cross(u, v)
a, b, c = normal
d = -point.dot(normal)
return a, b, c, d
- def eval_plane(self, pt, coeffs):
- x, y, z = pt
+ def eval_plane(
+ self, pt: Point3D, coeffs: tuple[float, float, float, float]
+ ) -> float:
+ """Evaluate a point against a plane equation.
+
+ Args:
+ pt: Point coordinates (x, y, z).
+ coeffs: Plane coefficients (a, b, c, d).
+
+ Returns:
+ Value of ax + by + cz + d (0 if on plane).
+ """
+ x, y, z = pt[0], pt[1], pt[2]
a, b, c, d = coeffs
return a * x + b * y + c * z + d
- def find_shortest_dist(self, points):
+ def find_shortest_dist(
+ self, points: Sequence[np.ndarray | tuple[float, ...]]
+ ) -> float:
+ """Find the shortest distance from first point to any other.
+
+ Args:
+ points: List of point coordinates.
+
+ Returns:
+ Minimum distance from points[0] to any other point.
+ """
point1 = points[0]
dists = [math.dist(point1, point) for point in points[1:]]
return min(dists)
- def same_plane_side(self, pt1, pt2, plane):
+ def same_plane_side(
+ self,
+ pt1: Point3D,
+ pt2: Point3D,
+ plane: tuple[float, float, float, float],
+ ) -> bool:
+ """Check if two points are on the same side of a plane.
+
+ Args:
+ pt1: First point coordinates.
+ pt2: Second point coordinates.
+ plane: Plane coefficients (a, b, c, d).
+
+ Returns:
+ True if both points are on the same side.
+ """
pt1_side = self.eval_plane(pt1, plane)
pt2_side = self.eval_plane(pt2, plane)
- plane_side = (
- pt1_side == abs(pt1_side)) == (
- pt2_side == abs(pt2_side)
- )
+ plane_side = (pt1_side == abs(pt1_side)) == (pt2_side == abs(pt2_side))
return plane_side
- def get_plane_pts(self, points):
- points = [tuple(point) for point in points]
- shortest_dist = self.find_shortest_dist(points)
- plane_sets = set()
- for i, pt1 in enumerate(points):
- for j, pt2 in enumerate(points):
- for k, pt3 in enumerate(points):
+ def get_plane_pts(
+ self, points: Sequence[np.ndarray | tuple[float, ...]]
+ ) -> list[tuple[tuple[float, ...], tuple[float, ...], tuple[float, ...]]]:
+ """Find sets of three equidistant points forming plane faces.
+
+ Args:
+ points: List of point coordinates.
+
+ Returns:
+ List of point triplets that define planar faces.
+ """
+ tuple_points: list[tuple[float, ...]] = [tuple(point) for point in points]
+ shortest_dist = self.find_shortest_dist(list(points))
+ plane_sets: set[
+ tuple[tuple[float, ...], tuple[float, ...], tuple[float, ...]]
+ ] = set()
+ for i, pt1 in enumerate(tuple_points):
+ for j, pt2 in enumerate(tuple_points):
+ for k, pt3 in enumerate(tuple_points):
# further optimizations by making sure k >= j and j >= i
if i == j or j == k or i == k or k < j or j < i:
continue
@@ -121,46 +268,88 @@ def get_plane_pts(self, points):
if is_planar_set:
plane_set = (pt1, pt2, pt3)
perms = permutations(plane_set)
- if any([perm in plane_sets for perm in perms]):
+ if any(perm in plane_sets for perm in perms):
continue
plane_sets.add(plane_set)
return list(plane_sets)
- def check_pt_in_shape(self, point, vertices):
- # only works with triangular faces
+ def check_pt_in_shape(
+ self, point: tuple[float, float, float], vertices: np.ndarray
+ ) -> bool:
+ """Check if a point is inside a shape defined by vertices.
+
+ Only works with triangular faces.
+
+ Args:
+ point: Point to test.
+ vertices: Array of shape vertices.
+
+ Returns:
+ True if point is inside the shape.
+ """
centroid = self.find_centroid(vertices)
- plane_pts = self.get_plane_pts(vertices)
+ vertices_list: list[np.ndarray] = list(vertices)
+ plane_pts = self.get_plane_pts(vertices_list)
planes = [self.find_plane(*pts) for pts in plane_pts]
for plane in planes:
if not self.same_plane_side(centroid, point, plane):
return False
return True
- def generate_icosphere(self, radius, center, refinement):
+ def generate_icosphere(
+ self, radius: float, center: Point3D, refinement: int
+ ) -> tuple[np.ndarray, np.ndarray]:
+ """Generate an icosphere mesh.
+
+ Args:
+ radius: Radius of the sphere.
+ center: Center point coordinates.
+ refinement: Subdivision level for mesh refinement.
+
+ Returns:
+ Tuple of (vertices, faces) arrays.
+ """
vertices, faces = icosphere(refinement)
length = norm(vertices, axis=1).reshape((-1, 1))
vertices = vertices / length * radius + center
return vertices, faces
- def generate_octahedron(self, radius, center):
- vertices = np.array([
- (0, 0, -1),
- (0, +1, 0),
- (0, 0, +1),
- (0, -1, 0),
- (-1, 0, 0),
- (+1, 0, 0)
- ]).astype(float)
+ def generate_octahedron(self, radius: float, center: Point3D) -> np.ndarray:
+ """Generate octahedron vertices.
+
+ Args:
+ radius: Distance from center to vertices.
+ center: Center point coordinates.
+
+ Returns:
+ Array of 6 vertex coordinates.
+ """
+ vertices = np.array(
+ [(0, 0, -1), (0, +1, 0), (0, 0, +1), (0, -1, 0), (-1, 0, 0), (+1, 0, 0)]
+ ).astype(float)
vertices *= radius
vertices += center
return vertices
- # 4 more!
- def get_3D_circle(self, center, pt1, pt2, refinement=360):
- # method 1: https://math.stackexchange.com/a/2375120
- # method 2: https://math.stackexchange.com/a/73242 - CHOSEN
+ def get_3D_circle(
+ self,
+ center: np.ndarray,
+ pt1: np.ndarray,
+ pt2: np.ndarray,
+ refinement: int = 360,
+ ) -> np.ndarray:
+ """Generate points on a 3D circle in an arbitrary plane.
+ Args:
+ center: Center of the circle.
+ pt1: First point on the circle.
+ pt2: Second point to define the plane.
+ refinement: Number of points on the circle.
+
+ Returns:
+ Array of circle coordinates with shape (3, refinement).
+ """
plane = self.find_plane(center, pt1, pt2)
normal = np.array(plane[0:3])
unit_normal = normal / norm(normal)
@@ -175,16 +364,14 @@ def get_3D_circle(self, center, pt1, pt2, refinement=360):
q1 /= norm(q1)
q2 = center + np.cross(q1, unit_normal)
- def convert_to_xyz(theta, idx):
+ def convert_to_xyz(theta: float, idx: int) -> float:
return (
- center[idx] +
- radius * math.cos(theta) * q1[idx] +
- radius * math.sin(theta) * q2[idx]
+ center[idx]
+ + radius * math.cos(theta) * q1[idx]
+ + radius * math.sin(theta) * q2[idx]
)
- circle = np.array([
- [convert_to_xyz(theta, idx)
- for theta in angles]
- for idx in [0, 1, 2]
- ])
+ circle = np.array(
+ [[convert_to_xyz(theta, idx) for theta in angles] for idx in [0, 1, 2]]
+ )
return circle
diff --git a/hyperdrive/Constants.py b/hyperdrive/Constants.py
index 9eeb737d..77ec74d5 100755
--- a/hyperdrive/Constants.py
+++ b/hyperdrive/Constants.py
@@ -1,111 +1,125 @@
+"""Constants, configuration values, and path utilities for hyperdrive."""
+
import os
from pathlib import Path
-from dotenv import load_dotenv, find_dotenv
-from pytz import timezone
+
import vectorbt as vbt
+from dotenv import find_dotenv, load_dotenv
+from pytz import timezone
+
+load_dotenv(find_dotenv("config.env"))
-load_dotenv(find_dotenv('config.env'))
+def get_env_int(var_name: str, default: int | None = None) -> int | None:
+ """Get an environment variable as an integer.
-def get_env_int(var_name, default=None):
+ Args:
+ var_name: Name of the environment variable.
+ default: Default value if variable is not set or not numeric.
+
+ Returns:
+ The integer value or default.
+ """
return (
int(os.environ[var_name])
- if os.environ.get(var_name)
- and os.environ[var_name].isnumeric()
+ if os.environ.get(var_name) and os.environ[var_name].isnumeric()
else default
)
-def get_env_bool(var_name):
- return bool(os.environ.get(var_name)
- and os.environ[var_name].lower() == 'true')
+def get_env_bool(var_name: str) -> bool:
+ """Get an environment variable as a boolean.
+
+ Args:
+ var_name: Name of the environment variable.
+
+ Returns:
+ True if the variable is set to 'true' (case-insensitive).
+ """
+ return bool(os.environ.get(var_name) and os.environ[var_name].lower() == "true")
# Environment
-DEV = get_env_bool('DEV')
-CI = get_env_bool('CI')
-TEST = get_env_bool('TEST')
+DEV = get_env_bool("DEV")
+CI = get_env_bool("CI")
+TEST = get_env_bool("TEST")
# File Paths
# data
-DATA_DIR = 'data'
-API_DIR = 'api'
-DEV_DIR = 'dev'
-DIV_DIR = 'dividends'
-SPLT_DIR = 'splits'
-OHLC_DIR = 'ohlc'
-SENT_DIR = 'sentiment'
-INTRA_DIR = 'intraday'
-IDX_DIR = 'indices'
+DATA_DIR = "data"
+API_DIR = "api"
+DEV_DIR = "dev"
+DIV_DIR = "dividends"
+SPLT_DIR = "splits"
+OHLC_DIR = "ohlc"
+SENT_DIR = "sentiment"
+INTRA_DIR = "intraday"
+IDX_DIR = "indices"
# providers
-POLY_DIR = 'polygon'
-ALPACA_DIR = 'alpaca'
+POLY_DIR = "polygon"
+ALPACA_DIR = "alpaca"
# models
-MODELS_DIR = 'models'
+MODELS_DIR = "models"
-folders = {
- 'polygon': POLY_DIR,
- 'alpaca': ALPACA_DIR
-}
+folders = {"polygon": POLY_DIR, "alpaca": ALPACA_DIR}
# Column Names
# Symbols / Generic
-SYMBOL = 'Symbol'
-NAME = 'Name'
+SYMBOL = "Symbol"
+NAME = "Name"
# Dividends
-DIV = 'Div'
-EX = 'Ex' # Ex Dividend Date
-DEC = 'Dec' # Declaration Date
-PAY = 'Pay' # Payment Date
+DIV = "Div"
+EX = "Ex" # Ex Dividend Date
+DEC = "Dec" # Declaration Date
+PAY = "Pay" # Payment Date
# Splits
-RATIO = 'Ratio'
+RATIO = "Ratio"
# OHLCV
# DATE = 'Date'
-TIME = 'Time'
-OPEN = 'Open'
-HIGH = 'High'
-LOW = 'Low'
-CLOSE = 'Close'
-VOL = 'Vol'
-AVG = 'Avg'
-TRADES = 'Trades'
+TIME = "Time"
+OPEN = "Open"
+HIGH = "High"
+LOW = "Low"
+CLOSE = "Close"
+VOL = "Vol"
+AVG = "Avg"
+TRADES = "Trades"
# Time
-TZ = timezone('US/Eastern')
-UTC = timezone('UTC')
-DATE_FMT = '%Y-%m-%d'
-TIME_FMT = '%H:%M'
-PRECISE_TIME_FMT = '%H:%M:%S'
+TZ = timezone("US/Eastern")
+UTC = timezone("UTC")
+DATE_FMT = "%Y-%m-%d"
+TIME_FMT = "%H:%M"
+PRECISE_TIME_FMT = "%H:%M:%S"
# Sentiment
-POS = 'Pos'
-NEG = 'Neg'
-DELTA = 'Delta'
+POS = "Pos"
+NEG = "Neg"
+DELTA = "Delta"
# Unemployment
-UN_RATE = 'UnRate'
+UN_RATE = "UnRate"
# S2F
-HALVING = 'Halving'
+HALVING = "Halving"
# Difficulty Ribbon
-MAs = ['MA9', 'MA14', 'MA25', 'MA40', 'MA60', 'MA90', 'MA128', 'MA200']
+MAs = ["MA9", "MA14", "MA25", "MA40", "MA60", "MA90", "MA128", "MA200"]
# SOPR
-SOPR = 'SOPR'
+SOPR = "SOPR"
# Oracle
-SIG = 'Sig'
-BUY = 'BUY'
-SELL = 'SELL'
+SIG = "Sig"
+BUY = "BUY"
+SELL = "SELL"
# API
-BAL = 'Bal'
-NAME = 'Name'
-ABS_TOL = vbt.utils.math_.abs_tol
+BAL = "Bal"
+ABS_TOL = vbt.utils.math_.abs_tol # type: ignore[attr-defined]
# Model
MAX_MODEL_AGE_DAYS = 90 # 3 months
@@ -113,16 +127,32 @@ def get_env_bool(var_name):
# Misc
POLY_CRYPTO_SYMBOLS = [
- 'X%3ABTCUSD', 'X%3AETHUSD',
- 'X%3ALTCUSD', 'X%3AXMRUSD', 'X%3AIOTUSD'
+ "X%3ABTCUSD",
+ "X%3AETHUSD",
+ "X%3ALTCUSD",
+ "X%3AXMRUSD",
+ "X%3AIOTUSD",
]
-ALPC_CRYPTO_SYMBOLS = ['BTC/USD', 'ETH/USD', 'LTC/USD']
+ALPC_CRYPTO_SYMBOLS = ["BTC/USD", "ETH/USD", "LTC/USD"]
+
+# Mapping from Alpaca crypto symbols to Polygon symbols (for S3-safe paths)
+ALPC_TO_POLY_CRYPTO = {
+ "BTC/USD": "X%3ABTCUSD",
+ "ETH/USD": "X%3AETHUSD",
+ "LTC/USD": "X%3ALTCUSD",
+}
SENTIMENT_SYMBOLS_IGNORE = {
- 'SPYD', 'VWDRY', 'BPMP',
- 'FOX', 'YYY', 'SDIV',
- 'DIV', 'SHECY', 'PALL'
+ "SPYD",
+ "VWDRY",
+ "BPMP",
+ "FOX",
+ "YYY",
+ "SDIV",
+ "DIV",
+ "SHECY",
+ "PALL",
}
DEFAULT_RETRIES = 2
@@ -131,146 +161,204 @@ def get_env_bool(var_name):
POLY_MAX_LIMIT = 1000
POLY_MAX_AGGS_LIMIT = 50000
FEW = 3
-FEW_DAYS = str(FEW) + 'd'
+FEW_DAYS = str(FEW) + "d"
SCRIPT_FAILURE_THRESHOLD = 0.95
ALPACA_FREE_DELAY = 0.5
# Exchanges
-BINANCE = 'BINANCE'
-KRAKEN = 'KRAKEN'
-PREF_EXCHANGE = os.environ.get(
- 'PREF_EXCHANGE') and os.environ['PREF_EXCHANGE'].upper()
+BINANCE = "BINANCE"
+KRAKEN = "KRAKEN"
+PREF_EXCHANGE = os.environ.get("PREF_EXCHANGE") and os.environ["PREF_EXCHANGE"].upper()
# fee is 0.1%
BINANCE_FEE = 0.001
-KRAKEN_SYMBOLS = {'BTC': 'XXBT', 'USD': 'ZUSD'}
+KRAKEN_SYMBOLS = {"BTC": "XXBT", "USD": "ZUSD"}
BINANCE_TEST_SPEND = 0.01
KRAKEN_TEST_SPEND = 0.005
class PathFinder:
- def make_path(self, path):
+ """Utility class for constructing file paths.
+
+ Provides methods for generating paths to various data files
+ including symbols, dividends, OHLC data, and more.
+ """
+
+ def make_path(self, path: str) -> None:
+ """Create parent directories for a path if they don't exist.
+
+ Args:
+ path: File path whose parent directories should be created.
+ """
Path(path).parent.mkdir(parents=True, exist_ok=True)
- def get_symbols_path(self):
- # return the path for the symbols reference csv
- return os.path.join(
- DATA_DIR,
- 'symbols.csv'
- )
+ def get_symbols_path(self) -> str:
+ """Get the path for the symbols reference CSV.
- def get_dividends_path(self, symbol, provider=POLY_DIR):
- # given a symbol
- # return the path to its csv
- return os.path.join(
- DATA_DIR,
- DIV_DIR,
- folders[provider],
- f'{symbol.upper()}.csv'
- )
+ Returns:
+ Path to the symbols CSV file.
+ """
+ return os.path.join(DATA_DIR, "symbols.csv")
- def get_splits_path(self, symbol, provider=POLY_DIR):
- # given a symbol
- # return the path to its stock splits
- return os.path.join(
- DATA_DIR,
- SPLT_DIR,
- folders[provider],
- f'{symbol.upper()}.csv'
- )
+ def get_dividends_path(self, symbol: str, provider: str = POLY_DIR) -> str:
+ """Get the path to a symbol's dividends CSV.
- def get_ohlc_path(self, symbol, provider=POLY_DIR):
- # given a symbol
- # return the path to its ohlc data
- return os.path.join(
- DATA_DIR,
- OHLC_DIR,
- folders[provider],
- f'{symbol.upper()}.csv'
- )
+ Args:
+ symbol: Stock symbol.
+ provider: Data provider directory name.
- def get_intraday_path(self, symbol, date, provider=POLY_DIR):
- # given a symbol,
- # return the path to its intraday ohlc data
+ Returns:
+ Path to the dividends CSV file.
+ """
return os.path.join(
- DATA_DIR,
- INTRA_DIR,
- folders[provider],
- symbol.upper(),
- f'{date}.csv'
+ DATA_DIR, DIV_DIR, folders[provider], f"{symbol.upper()}.csv"
)
- def get_unemployment_path(self):
- return os.path.join(
- DATA_DIR,
- 'unemployment.csv'
- )
+ def get_splits_path(self, symbol: str, provider: str = POLY_DIR) -> str:
+ """Get the path to a symbol's stock splits CSV.
- def get_s2f_path(self):
- return os.path.join(
- DATA_DIR,
- 's2f.csv'
- )
+ Args:
+ symbol: Stock symbol.
+ provider: Data provider directory name.
- def get_diff_ribbon_path(self):
+ Returns:
+ Path to the splits CSV file.
+ """
return os.path.join(
- DATA_DIR,
- 'diff_ribbon.csv'
+ DATA_DIR, SPLT_DIR, folders[provider], f"{symbol.upper()}.csv"
)
- def get_sopr_path(self):
- return os.path.join(
- DATA_DIR,
- 'sopr.csv'
- )
+ def get_ohlc_path(self, symbol: str, provider: str = POLY_DIR) -> str:
+ """Get the path to a symbol's OHLC data CSV.
- def get_signals_path(self):
- return os.path.join(
- MODELS_DIR,
- 'latest',
- 'signals.csv'
- )
+ Args:
+ symbol: Stock symbol.
+ provider: Data provider directory name.
- def get_orders_path(self):
+ Returns:
+ Path to the OHLC CSV file.
+ """
return os.path.join(
- MODELS_DIR,
- 'latest',
- 'orders.csv'
+ DATA_DIR, OHLC_DIR, folders[provider], f"{symbol.upper()}.csv"
)
- def get_new_orders_path(self, provider):
+ def get_intraday_path(
+ self, symbol: str, date: str, provider: str = POLY_DIR
+ ) -> str:
+ """Get the path to a symbol's intraday OHLC data CSV.
+
+ Args:
+ symbol: Stock symbol.
+ date: Date string for the intraday data.
+ provider: Data provider directory name.
+
+ Returns:
+ Path to the intraday CSV file.
+ """
return os.path.join(
- DATA_DIR,
- 'orders',
- f'{provider}.csv'
+ DATA_DIR, INTRA_DIR, folders[provider], symbol.upper(), f"{date}.csv"
)
- def get_api_path(self, endpoint):
+ def get_unemployment_path(self) -> str:
+ """Get the path to the unemployment data CSV.
+
+ Returns:
+ Path to the unemployment CSV file.
+ """
+ return os.path.join(DATA_DIR, "unemployment.csv")
+
+ def get_s2f_path(self) -> str:
+ """Get the path to the stock-to-flow data CSV.
+
+ Returns:
+ Path to the S2F CSV file.
+ """
+ return os.path.join(DATA_DIR, "s2f.csv")
+
+ def get_diff_ribbon_path(self) -> str:
+ """Get the path to the difficulty ribbon data CSV.
+
+ Returns:
+ Path to the difficulty ribbon CSV file.
+ """
+ return os.path.join(DATA_DIR, "diff_ribbon.csv")
+
+ def get_sopr_path(self) -> str:
+ """Get the path to the SOPR data CSV.
+
+ Returns:
+ Path to the SOPR CSV file.
+ """
+ return os.path.join(DATA_DIR, "sopr.csv")
+
+ def get_signals_path(self) -> str:
+ """Get the path to the trading signals CSV.
+
+ Returns:
+ Path to the signals CSV file.
+ """
+ return os.path.join(MODELS_DIR, "latest", "signals.csv")
+
+ def get_orders_path(self) -> str:
+ """Get the path to the orders CSV.
+
+ Returns:
+ Path to the orders CSV file.
+ """
+ return os.path.join(MODELS_DIR, "latest", "orders.csv")
+
+ def get_new_orders_path(self, provider: str) -> str:
+ """Get the path to new orders CSV for a provider.
+
+ Args:
+ provider: Data provider name.
+
+ Returns:
+ Path to the new orders CSV file.
+ """
+ return os.path.join(DATA_DIR, "orders", f"{provider}.csv")
+
+ def get_api_path(self, endpoint: str) -> str:
+ """Get the path to an API response JSON file.
+
+ Args:
+ endpoint: API endpoint name.
+
+ Returns:
+ Path to the API JSON file.
+ """
return os.path.join(
DATA_DIR,
API_DIR,
- f'{endpoint}.json',
+ f"{endpoint}.json",
)
- def get_ndx_path(self):
- return os.path.join(
- DATA_DIR,
- IDX_DIR,
- 'ndx.csv'
- )
+ def get_ndx_path(self) -> str:
+ """Get the path to the NASDAQ index data CSV.
+
+ Returns:
+ Path to the NDX CSV file.
+ """
+ return os.path.join(DATA_DIR, IDX_DIR, "ndx.csv")
+
+ def get_all_paths(self, path: str, truncate: bool = False) -> list[str]:
+ """Get all file paths under a directory.
+
+ Args:
+ path: Root directory to search.
+ truncate: If True, remove the root path prefix from results.
- def get_all_paths(self, path, truncate=False):
- # given a path, get all sub paths
+ Returns:
+ List of file paths, excluding cache and temp files.
+ """
paths = []
for root, _, files in os.walk(path):
for file in files:
- curr_path = os.path.join(root, file)[
- len(path) + 1 if truncate else 0:]
- to_skip = ['__pycache__/', '.pytest',
- '.git/', '.ipynb', '.env']
+ curr_path = os.path.join(root, file)[len(path) + 1 if truncate else 0 :]
+ to_skip = ["__pycache__/", ".pytest", ".git/", ".ipynb", ".env"]
keep = [skip not in curr_path for skip in to_skip]
# remove caches but keep workflows
- if all(keep) or '.github' in curr_path:
+ if all(keep) or ".github" in curr_path:
paths.append(curr_path)
return paths
diff --git a/hyperdrive/Crypt.py b/hyperdrive/Crypt.py
index 4f558aba..f0d46676 100644
--- a/hyperdrive/Crypt.py
+++ b/hyperdrive/Crypt.py
@@ -1,30 +1,34 @@
+"""Cryptographic utilities for symmetric encryption using AES-256-GCM."""
+
import os
-from typing import Union
-from cryptography.hazmat.primitives.kdf.scrypt import Scrypt
-from cryptography.hazmat.primitives.ciphers.aead import AESGCM
+from cryptography.hazmat.primitives.ciphers.aead import AESGCM
+from cryptography.hazmat.primitives.kdf.scrypt import Scrypt
-FlexibleBytes = Union[str, bytes]
+FlexibleBytes = str | bytes
class Cryptographer:
- """
- A class for symmetric encryption using AES-256 in GCM mode.
+ """Symmetric encryption using AES-256 in GCM mode.
The key is derived from a user-provided password and salt using Scrypt.
- This approach is considered post-quantum
- resistant for symmetric encryption.
+ This approach is considered post-quantum resistant for symmetric encryption.
Derives a 256-bit (32-byte) key from the password and salt.
- Args:
- password (FlexibleBytes):
- The password to use for key derivation.
- salt (FlexibleBytes):
- A random salt, which should be stored and reused for decryption.
+ Attributes:
+ key: The derived encryption key.
+ aesgcm: The AES-GCM cipher instance.
+ nonce_size: Size of the nonce in bytes (12).
"""
- def __init__(self, password: FlexibleBytes, salt: FlexibleBytes):
+ def __init__(self, password: FlexibleBytes, salt: FlexibleBytes) -> None:
+ """Initialize the Cryptographer with password and salt.
+
+ Args:
+ password: The password to use for key derivation.
+ salt: A random salt, which should be stored and reused for decryption.
+ """
password = self.convert_to_bytes(password)
salt = self.convert_to_bytes(salt)
kdf = Scrypt(
@@ -42,29 +46,26 @@ def __init__(self, password: FlexibleBytes, salt: FlexibleBytes):
self.nonce_size = 12
def convert_to_bytes(self, value: FlexibleBytes) -> bytes:
- """
- Convert a string to bytes, if necessary.
+ """Convert a string to bytes, if necessary.
Args:
- value (FlexibleBytes): The value to convert.
+ value: The value to convert.
Returns:
- bytes: The converted value.
+ The converted value as bytes.
"""
if isinstance(value, str):
- return value.encode('UTF-8')
+ return value.encode("UTF-8")
return value
def encrypt(self, plaintext: FlexibleBytes) -> bytes:
- """
- Encrypts and authenticates plaintext using AES-256-GCM.
+ """Encrypt and authenticate plaintext using AES-256-GCM.
Args:
- plaintext (FlexibleBytes): The data to encrypt.
+ plaintext: The data to encrypt.
Returns:
- bytes: A self-contained ciphertext blob in the format:
- nonce + encrypted_data_and_tag.
+ A self-contained ciphertext blob in the format: nonce + encrypted_data_and_tag.
"""
plaintext = self.convert_to_bytes(plaintext)
# Generate a random nonce. It must be unique for each encryption.
@@ -75,24 +76,22 @@ def encrypt(self, plaintext: FlexibleBytes) -> bytes:
return nonce + ciphertext
def decrypt(self, ciphertext: bytes) -> FlexibleBytes:
- """
- Decrypts and verifies a ciphertext blob.
+ """Decrypt and verify a ciphertext blob.
Args:
- ciphertext (bytes): The combined nonce and ciphertext.
+ ciphertext: The combined nonce and ciphertext.
Returns:
- FlexibleBytes: The original plaintext
- if decryption and authentication are successful.
+ The original plaintext if decryption and authentication are successful.
"""
# Extract the nonce
- nonce = ciphertext[:self.nonce_size]
+ nonce = ciphertext[: self.nonce_size]
# Extract the actual ciphertext (without the nonce)
- ciphertext = ciphertext[self.nonce_size:]
+ ciphertext = ciphertext[self.nonce_size :]
# Decrypt the data. The tag is verified automatically.
plaintext = self.aesgcm.decrypt(nonce, ciphertext, None)
try:
- plaintext = plaintext.decode('UTF-8')
+ plaintext = plaintext.decode()
except UnicodeDecodeError:
pass
return plaintext
diff --git a/hyperdrive/DataSource.py b/hyperdrive/DataSource.py
index 463c805e..8c51d49d 100755
--- a/hyperdrive/DataSource.py
+++ b/hyperdrive/DataSource.py
@@ -1,48 +1,85 @@
-import os
+"""Data source classes for fetching market data from various providers.
+
+This module contains classes for retrieving financial data including stock prices,
+dividends, splits, and crypto indicators from providers like Polygon, Alpaca,
+BLS, and Glassnode.
+"""
+
import json
-import requests
+import os
+from collections.abc import Callable, Generator, Iterable, Iterator
+from datetime import datetime
+from io import StringIO
+from random import random
from time import sleep, time
-from bs4 import BeautifulSoup
+from typing import Any
+
import pandas as pd
-from random import random
-from polygon import RESTClient, exceptions
-from dotenv import load_dotenv, find_dotenv
-from FileOps import FileReader, FileWriter
-from Calculus import Calculator
-from TimeMachine import TimeTraveller
-from Constants import PathFinder
-import Constants as C
-from datetime import datetime
+import requests
+from bs4 import BeautifulSoup
+from dotenv import find_dotenv, load_dotenv
+from polygon import RESTClient
from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions
-from selenium.webdriver.support.ui import WebDriverWait
-from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
+from selenium.webdriver.support import expected_conditions as EC
+from selenium.webdriver.support.ui import WebDriverWait
+
+from . import Constants as C
+from .Calculus import Calculator
+from .Constants import PathFinder
+from .FileOps import FileReader, FileWriter
+from .TimeMachine import TimeTraveller
class MarketData:
- def __init__(self):
- load_dotenv(find_dotenv('config.env'))
+ """Base class for market data providers.
+
+ Provides common functionality for fetching and storing market data including
+ OHLC prices, dividends, splits, and other financial indicators.
+ """
+
+ def __init__(self) -> None:
+ """Initialize the MarketData instance with default components."""
+ load_dotenv(find_dotenv("config.env"))
self.writer = FileWriter()
self.reader = FileReader()
self.finder = PathFinder()
self.traveller = TimeTraveller()
self.calculator = Calculator()
- self.provider = 'polygon'
+ self.provider = "polygon"
+
+ def get_indexer(self, s1: set[str], s2: Iterable[str]) -> list[str]:
+ """Get intersection of two sets as a list.
+
+ Args:
+ s1: First set of strings.
+ s2: Second iterable of strings.
- def get_indexer(self, s1, s2):
+ Returns:
+ List of strings that appear in both s1 and s2.
+ """
return list(s1.intersection(s2))
- def try_again(self, func, **kwargs):
- retries = (kwargs['retries']
- if 'retries' in kwargs
- else C.DEFAULT_RETRIES)
- delay = (kwargs['delay']
- if 'delay' in kwargs
- else C.DEFAULT_DELAY)
- func_args = {k: v for k, v in kwargs.items() if k not in {
- 'retries', 'delay'}}
+ def try_again(self, func: Callable[..., Any], **kwargs: Any) -> Any:
+ """Retry a function with exponential backoff on failure.
+
+ Args:
+ func: The function to retry.
+ **kwargs: Arguments passed to the function. Special keys:
+ - retries: Number of retry attempts (default: C.DEFAULT_RETRIES)
+ - delay: Seconds to wait between retries (default: C.DEFAULT_DELAY)
+
+ Returns:
+ The result of the function call.
+
+ Raises:
+ Exception: If all retries are exhausted.
+ """
+ retries = kwargs.get("retries", C.DEFAULT_RETRIES)
+ delay = kwargs.get("delay", C.DEFAULT_DELAY)
+ func_args = {k: v for k, v in kwargs.items() if k not in {"retries", "delay"}}
for retry in range(retries):
try:
return func(**func_args)
@@ -52,366 +89,617 @@ def try_again(self, func, **kwargs):
else:
sleep(delay)
- def get_symbols(self):
- # get cached list of symbols
+ def get_symbols(self) -> list[str]:
+ """Get cached list of tradable symbols.
+
+ Returns:
+ List of stock/crypto symbols.
+ """
symbols_path = self.finder.get_symbols_path()
return list(self.reader.load_csv(symbols_path)[C.SYMBOL])
- def get_dividends(self, symbol, timeframe='max'):
- # given a symbol, return a cached dataframe
- df = self.reader.load_csv(
- self.finder.get_dividends_path(symbol, self.provider))
+ def get_dividends(
+ self, symbol: str = "", timeframe: str = "max", **kwargs: Any
+ ) -> pd.DataFrame:
+ """Get cached dividend data for a symbol.
+
+ Args:
+ symbol: The stock symbol.
+ timeframe: Time range for data (default: "max").
+ **kwargs: Additional arguments for subclass implementations.
+
+ Returns:
+ DataFrame with dividend history.
+ """
+ df = self.reader.load_csv(self.finder.get_dividends_path(symbol, self.provider))
filtered = self.reader.data_in_timeframe(df, C.EX, timeframe)
return filtered
- def standardize(self, df, full_mapping,
- filename, columns, default):
+ def standardize(
+ self,
+ df: pd.DataFrame,
+ full_mapping: dict[str, str],
+ filename: str,
+ columns: list[str],
+ default: float | int,
+ ) -> pd.DataFrame:
+ """Standardize a DataFrame to a common column format.
+
+ Args:
+ df: Input DataFrame to standardize.
+ full_mapping: Column name mapping (old -> new).
+ filename: Path to save/load cached data.
+ columns: List of column names [time_col, *value_cols].
+ default: Default value for missing data.
+
+ Returns:
+ Standardized DataFrame.
+ """
mapping = {k: v for k, v in full_mapping.items() if k in df}
df = df[list(mapping)].rename(columns=mapping)
time_col, val_cols = columns[0], columns[1:]
if time_col in df and set(val_cols).issubset(df.columns):
- df = self.reader.update_df(
- filename, df, time_col).sort_values(by=[time_col])
- # since time col is pd.datetime,
- # consider converting to YYYY-MM-DD str format
+ df = self.reader.update_df(filename, df, time_col).sort_values(
+ by=[time_col]
+ )
for val_col in val_cols:
df[val_col] = df[val_col].apply(
- lambda val: float(val) if val else default)
+ lambda val: float(val) if val else default
+ )
return df
- def standardize_dividends(self, symbol, df):
+ def standardize_dividends(self, symbol: str, df: pd.DataFrame) -> pd.DataFrame:
+ """Standardize dividend data to common format.
+
+ Args:
+ symbol: The stock symbol.
+ df: Raw dividend DataFrame.
+
+ Returns:
+ Standardized DataFrame with columns [EX, PAY, DEC, DIV].
+ """
full_mapping = dict(
zip(
- ['exDate', 'paymentDate', 'declaredDate', 'amount'],
- [C.EX, C.PAY, C.DEC, C.DIV]
+ ["exDate", "paymentDate", "declaredDate", "amount"],
+ [C.EX, C.PAY, C.DEC, C.DIV],
+ strict=True,
)
)
filename = self.finder.get_dividends_path(symbol, self.provider)
- return self.standardize(
- df,
- full_mapping,
- filename,
- [C.EX, C.DIV],
- 0
- )
+ return self.standardize(df, full_mapping, filename, [C.EX, C.DIV], 0)
- def save_dividends(self, **kwargs):
- # given a symbol, save its dividend history
- symbol = kwargs['symbol']
+ def save_dividends(self, **kwargs: Any) -> str | None:
+ """Save dividend history for a symbol.
+
+ Args:
+ **kwargs: Must include 'symbol'. Other args passed to get_dividends.
+
+ Returns:
+ Path to saved file, or None if save failed.
+ """
+ symbol = kwargs["symbol"]
filename = self.finder.get_dividends_path(symbol, self.provider)
if os.path.exists(filename):
os.remove(filename)
df = self.reader.update_df(
- filename, self.get_dividends(**kwargs), C.EX, C.DATE_FMT)
+ filename, self.get_dividends(**kwargs), C.EX, C.DATE_FMT
+ )
self.writer.update_csv(filename, df)
if os.path.exists(filename):
return filename
-
- def get_splits(self, symbol, timeframe='max'):
- # given a symbol, return a cached dataframe
- df = self.reader.load_csv(
- self.finder.get_splits_path(symbol, self.provider))
+ return None
+
+ def get_splits(
+ self, symbol: str = "", timeframe: str = "max", **kwargs: Any
+ ) -> pd.DataFrame:
+ """Get cached split data for a symbol.
+
+ Args:
+ symbol: The stock symbol.
+ timeframe: Time range for data (default: "max").
+ **kwargs: Additional arguments for subclass implementations.
+
+ Returns:
+ DataFrame with split history.
+ """
+ df = self.reader.load_csv(self.finder.get_splits_path(symbol, self.provider))
filtered = self.reader.data_in_timeframe(df, C.EX, timeframe)
return filtered
- def standardize_splits(self, symbol, df):
+ def standardize_splits(self, symbol: str, df: pd.DataFrame) -> pd.DataFrame:
+ """Standardize split data to common format.
+
+ Args:
+ symbol: The stock symbol.
+ df: Raw split DataFrame.
+
+ Returns:
+ Standardized DataFrame with columns [EX, PAY, DEC, RATIO].
+ """
full_mapping = dict(
zip(
- ['exDate', 'paymentDate', 'declaredDate', 'ratio'],
- [C.EX, C.PAY, C.DEC, C.RATIO]
+ ["exDate", "paymentDate", "declaredDate", "ratio"],
+ [C.EX, C.PAY, C.DEC, C.RATIO],
+ strict=True,
)
)
filename = self.finder.get_splits_path(symbol, self.provider)
- return self.standardize(
- df,
- full_mapping,
- filename,
- [C.EX, C.RATIO],
- 1
- )
+ return self.standardize(df, full_mapping, filename, [C.EX, C.RATIO], 1)
+
+ def save_splits(self, **kwargs: Any) -> str | None:
+ """Save split history for a symbol.
- def save_splits(self, **kwargs):
- # given a symbol, save its splits history
- symbol = kwargs['symbol']
+ Args:
+ **kwargs: Must include 'symbol'. Other args passed to get_splits.
+
+ Returns:
+ Path to saved file, or None if save failed.
+ """
+ symbol = kwargs["symbol"]
filename = self.finder.get_splits_path(symbol, self.provider)
if os.path.exists(filename):
os.remove(filename)
df = self.reader.update_df(
- filename, self.get_splits(**kwargs), C.EX, C.DATE_FMT)
+ filename, self.get_splits(**kwargs), C.EX, C.DATE_FMT
+ )
self.writer.update_csv(filename, df)
if os.path.exists(filename):
return filename
+ return None
- def standardize_ohlc(self, symbol, df, filename=None):
+ def standardize_ohlc(
+ self, symbol: str, df: pd.DataFrame, filename: str | None = None
+ ) -> pd.DataFrame:
+ """Standardize OHLC data to common format.
+
+ Args:
+ symbol: The stock/crypto symbol.
+ df: Raw OHLC DataFrame.
+ filename: Optional custom filename for caching.
+
+ Returns:
+ Standardized DataFrame with OHLC columns.
+ """
full_mapping = dict(
zip(
- ['date', 'open', 'high', 'low', 'close',
- 'volume', 'average', 'trades'],
- [C.TIME, C.OPEN, C.HIGH, C.LOW, C.CLOSE,
- C.VOL, C.AVG, C.TRADES]
+ ["date", "open", "high", "low", "close", "volume", "average", "trades"],
+ [C.TIME, C.OPEN, C.HIGH, C.LOW, C.CLOSE, C.VOL, C.AVG, C.TRADES],
+ strict=True,
)
)
filename = filename or self.finder.get_ohlc_path(symbol, self.provider)
df = self.standardize(
- df,
- full_mapping,
- filename,
- [C.TIME, C.OPEN, C.HIGH, C.LOW, C.CLOSE],
- 0
+ df, full_mapping, filename, [C.TIME, C.OPEN, C.HIGH, C.LOW, C.CLOSE], 0
)
for col in [C.VOL, C.TRADES]:
if col in df:
- df[col] = df[col].apply(
- lambda val: 0 if pd.isnull(val) else int(val))
+ df[col] = df[col].apply(lambda val: 0 if pd.isnull(val) else int(val))
return df
- def get_ohlc(self, symbol, timeframe='max'):
- df = self.reader.load_csv(
- self.finder.get_ohlc_path(symbol, self.provider))
+ def get_ohlc(
+ self, symbol: str = "", timeframe: str = "max", **kwargs: Any
+ ) -> pd.DataFrame:
+ """Get cached OHLC data for a symbol.
+
+ Args:
+ symbol: The stock/crypto symbol.
+ timeframe: Time range for data (default: "max").
+ **kwargs: Additional arguments for subclass implementations.
+
+ Returns:
+ DataFrame with OHLC price history.
+ """
+ df = self.reader.load_csv(self.finder.get_ohlc_path(symbol, self.provider))
filtered = self.reader.data_in_timeframe(df, C.TIME, timeframe)
return filtered
- def save_ohlc(self, **kwargs):
- symbol = kwargs['symbol']
+ def save_ohlc(self, **kwargs: Any) -> str | None:
+ """Save OHLC data for a symbol.
+
+ Args:
+ **kwargs: Must include 'symbol'. Other args passed to get_ohlc.
+
+ Returns:
+ Path to saved file, or None if save failed.
+ """
+ symbol = kwargs["symbol"]
filename = self.finder.get_ohlc_path(symbol, self.provider)
if os.path.exists(filename):
os.remove(filename)
df = self.reader.update_df(
- filename, self.get_ohlc(**kwargs), C.TIME, C.DATE_FMT)
+ filename, self.get_ohlc(**kwargs), C.TIME, C.DATE_FMT
+ )
self.writer.update_csv(filename, df)
if os.path.exists(filename):
return filename
-
- def get_intraday(self, symbol, min=1, timeframe='max', extra_hrs=False):
- # implement way to transform 1 min dataset to 5 min data
- # or 30 or 60 should be flexible soln
- # implement way to only get market hours
- # given a symbol, return a cached dataframe
+ return None
+
+ def get_intraday(
+ self,
+ symbol: str = "",
+ min: int = 1, # noqa: A002
+ timeframe: str = "max",
+ extra_hrs: bool = False,
+ **kwargs: Any,
+ ) -> Generator[pd.DataFrame, None, None] | None:
+ """Get cached intraday data for a symbol.
+
+ Args:
+ symbol: The stock/crypto symbol.
+ min: Minute interval for data (default: 1).
+ timeframe: Time range for data (default: "max").
+ extra_hrs: Include extended hours data (default: False).
+ **kwargs: Additional arguments for subclass implementations.
+
+ Yields:
+ DataFrames with intraday OHLC data for each date.
+ """
dates = self.traveller.dates_in_range(timeframe)
for date in dates:
+ date_str = date if isinstance(date, str) else date.strftime("%Y-%m-%d")
df = self.reader.load_csv(
- self.finder.get_intraday_path(symbol, date, self.provider))
+ self.finder.get_intraday_path(symbol, date_str, self.provider)
+ )
yield self.reader.data_in_timeframe(df, C.TIME, timeframe)
- def save_intraday(self, **kwargs):
- symbol = kwargs['symbol']
+ def save_intraday(self, **kwargs: Any) -> list[str]:
+ """Save intraday data for a symbol.
+
+ Args:
+ **kwargs: Must include 'symbol'. Other args passed to get_intraday.
+
+ Returns:
+ List of paths to saved files.
+ """
+ symbol = kwargs["symbol"]
dfs = self.get_intraday(**kwargs)
filenames = []
+ if dfs is None:
+ return filenames
+
for df in dfs:
date = df[C.TIME].iloc[0].strftime(C.DATE_FMT)
- filename = self.finder.get_intraday_path(
- symbol, date, self.provider)
+ filename = self.finder.get_intraday_path(symbol, date, self.provider)
if os.path.exists(filename):
os.remove(filename)
- save_fmt = f'{C.DATE_FMT} {C.TIME_FMT}'
- df = self.reader.update_df(
- filename, df, C.TIME, save_fmt)
+ save_fmt = f"{C.DATE_FMT} {C.TIME_FMT}"
+ df = self.reader.update_df(filename, df, C.TIME, save_fmt)
self.writer.update_csv(filename, df)
if os.path.exists(filename):
filenames.append(filename)
return filenames
- def get_unemployment_rate(self, timeframe='max'):
- # given a timeframe, return a cached dataframe
- df = self.reader.load_csv(
- self.finder.get_unemployment_path())
+ def get_unemployment_rate(
+ self, timeframe: str = "max", **kwargs: Any
+ ) -> pd.DataFrame:
+ """Get cached unemployment rate data.
+
+ Args:
+ timeframe: Time range for data (default: "max").
+ **kwargs: Additional arguments for subclass implementations.
+
+ Returns:
+ DataFrame with unemployment rate history.
+ """
+ df = self.reader.load_csv(self.finder.get_unemployment_path())
filtered = self.reader.data_in_timeframe(df, C.TIME, timeframe)
return filtered
- def standardize_unemployment(self, df):
+ def standardize_unemployment(self, df: pd.DataFrame) -> pd.DataFrame:
+ """Standardize unemployment data to common format.
+
+ Args:
+ df: Raw unemployment DataFrame.
+
+ Returns:
+ Standardized DataFrame with [TIME, UN_RATE] columns.
+ """
full_mapping = dict(
zip(
- ['time', 'value'],
- [C.TIME, C.UN_RATE]
+ ["time", "value"],
+ [C.TIME, C.UN_RATE],
+ strict=True,
)
)
filename = self.finder.get_unemployment_path()
- return self.standardize(
- df,
- full_mapping,
- filename,
- [C.TIME, C.UN_RATE],
- 0
- )
+ return self.standardize(df, full_mapping, filename, [C.TIME, C.UN_RATE], 0)
- def save_unemployment_rate(self, **kwargs):
- # given a symbol, save its dividend history
+ def save_unemployment_rate(self, **kwargs: Any) -> str | None:
+ """Save unemployment rate data.
+
+ Args:
+ **kwargs: Arguments passed to get_unemployment_rate.
+
+ Returns:
+ Path to saved file, or None if save failed.
+ """
filename = self.finder.get_unemployment_path()
if os.path.exists(filename):
os.remove(filename)
df = self.reader.update_df(
- filename, self.get_unemployment_rate(**kwargs), C.TIME, '%Y-%m')
+ filename, self.get_unemployment_rate(**kwargs), C.TIME, "%Y-%m"
+ )
self.writer.update_csv(filename, df)
if os.path.exists(filename):
return filename
+ return None
+
+ def standardize_s2f_ratio(self, df: pd.DataFrame) -> pd.DataFrame:
+ """Standardize stock-to-flow ratio data.
+
+ Args:
+ df: Raw S2F DataFrame.
- def standardize_s2f_ratio(self, df):
+ Returns:
+ Standardized DataFrame with [TIME, HALVING, RATIO] columns.
+ """
full_mapping = dict(
zip(
- ['t', 'o.daysTillHalving', 'o.ratio'],
- [C.TIME, C.HALVING, C.RATIO]
+ ["t", "o.daysTillHalving", "o.ratio"],
+ [C.TIME, C.HALVING, C.RATIO],
+ strict=True,
)
)
filename = self.finder.get_s2f_path()
df = self.standardize(
- df,
- full_mapping,
- filename,
- [C.TIME, C.HALVING, C.RATIO],
- 0
+ df, full_mapping, filename, [C.TIME, C.HALVING, C.RATIO], 0
)
return df[self.get_indexer({C.TIME, C.HALVING, C.RATIO}, df.columns)]
- def get_s2f_ratio(self, timeframe='max'):
- # given a symbol, return a cached dataframe
- df = self.reader.load_csv(
- self.finder.get_s2f_path())
+ def get_s2f_ratio(self, timeframe: str = "max", **kwargs: Any) -> pd.DataFrame:
+ """Get cached stock-to-flow ratio data.
+
+ Args:
+ timeframe: Time range for data (default: "max").
+ **kwargs: Additional arguments for subclass implementations.
+
+ Returns:
+ DataFrame with S2F ratio history.
+ """
+ df = self.reader.load_csv(self.finder.get_s2f_path())
filtered = self.reader.data_in_timeframe(df, C.TIME, timeframe)[
- [C.TIME, C.HALVING, C.RATIO]]
+ [C.TIME, C.HALVING, C.RATIO]
+ ]
return filtered
- def save_s2f_ratio(self, **kwargs):
- # # given a symbol, save its s2f data
+ def save_s2f_ratio(self, **kwargs: Any) -> str | None:
+ """Save stock-to-flow ratio data.
+
+ Args:
+ **kwargs: Arguments passed to get_s2f_ratio.
+
+ Returns:
+ Path to saved file, or None if save failed.
+ """
filename = self.finder.get_s2f_path()
if os.path.exists(filename):
os.remove(filename)
df = self.reader.update_df(
- filename, self.get_s2f_ratio(**kwargs), C.TIME, C.DATE_FMT)
+ filename, self.get_s2f_ratio(**kwargs), C.TIME, C.DATE_FMT
+ )
self.writer.update_csv(filename, df)
if os.path.exists(filename):
return filename
+ return None
+
+ def standardize_diff_ribbon(self, df: pd.DataFrame) -> pd.DataFrame:
+ """Standardize difficulty ribbon data.
- def standardize_diff_ribbon(self, df):
+ Args:
+ df: Raw difficulty ribbon DataFrame.
+
+ Returns:
+ Standardized DataFrame with moving average columns.
+ """
full_mapping = dict(
zip(
- ['t', 'o.ma9', 'o.ma14', 'o.ma25', 'o.ma40',
- 'o.ma60', 'o.ma90', 'o.ma128', 'o.ma200'],
- [C.TIME] + C.MAs
+ [
+ "t",
+ "o.ma9",
+ "o.ma14",
+ "o.ma25",
+ "o.ma40",
+ "o.ma60",
+ "o.ma90",
+ "o.ma128",
+ "o.ma200",
+ ],
+ [C.TIME] + C.MAs,
+ strict=True,
)
)
filename = self.finder.get_diff_ribbon_path()
- df = self.standardize(
- df,
- full_mapping,
- filename,
- [C.TIME] + C.MAs,
- 0
- )
+ df = self.standardize(df, full_mapping, filename, [C.TIME] + C.MAs, 0)
return df[self.get_indexer(set([C.TIME] + C.MAs), df.columns)]
- def get_diff_ribbon(self, timeframe='max'):
- # given a symbol, return a cached dataframe
- df = self.reader.load_csv(
- self.finder.get_diff_ribbon_path())
+ def get_diff_ribbon(self, timeframe: str = "max", **kwargs: Any) -> pd.DataFrame:
+ """Get cached difficulty ribbon data.
+
+ Args:
+ timeframe: Time range for data (default: "max").
+ **kwargs: Additional arguments for subclass implementations.
+
+ Returns:
+ DataFrame with difficulty ribbon moving averages.
+ """
+ df = self.reader.load_csv(self.finder.get_diff_ribbon_path())
filtered = self.reader.data_in_timeframe(df, C.TIME, timeframe)[
- [C.TIME] + C.MAs]
+ [C.TIME] + C.MAs
+ ]
return filtered
- def save_diff_ribbon(self, **kwargs):
- # # given a symbol, save its s2f data
+ def save_diff_ribbon(self, **kwargs: Any) -> str | None:
+ """Save difficulty ribbon data.
+
+ Args:
+ **kwargs: Arguments passed to get_diff_ribbon.
+
+ Returns:
+ Path to saved file, or None if save failed.
+ """
filename = self.finder.get_diff_ribbon_path()
if os.path.exists(filename):
os.remove(filename)
df = self.reader.update_df(
- filename, self.get_diff_ribbon(**kwargs), C.TIME, C.DATE_FMT)
+ filename, self.get_diff_ribbon(**kwargs), C.TIME, C.DATE_FMT
+ )
self.writer.update_csv(filename, df)
if os.path.exists(filename):
return filename
+ return None
+
+ def standardize_sopr(self, df: pd.DataFrame) -> pd.DataFrame:
+ """Standardize SOPR (Spent Output Profit Ratio) data.
+
+ Args:
+ df: Raw SOPR DataFrame.
- def standardize_sopr(self, df):
+ Returns:
+ Standardized DataFrame with [TIME, SOPR] columns.
+ """
full_mapping = dict(
zip(
- ['t', 'v'],
- [C.TIME, C.SOPR]
+ ["t", "v"],
+ [C.TIME, C.SOPR],
+ strict=True,
)
)
filename = self.finder.get_diff_ribbon_path()
- df = self.standardize(
- df,
- full_mapping,
- filename,
- [C.TIME, C.SOPR],
- 1
- )
+ df = self.standardize(df, full_mapping, filename, [C.TIME, C.SOPR], 1)
return df[self.get_indexer({C.TIME, C.SOPR}, df.columns)]
- def get_sopr(self, timeframe='max'):
- # given a symbol, return a cached dataframe
- df = self.reader.load_csv(
- self.finder.get_sopr_path())
+ def get_sopr(self, timeframe: str = "max", **kwargs: Any) -> pd.DataFrame:
+ """Get cached SOPR data.
+
+ Args:
+ timeframe: Time range for data (default: "max").
+ **kwargs: Additional arguments for subclass implementations.
+
+ Returns:
+ DataFrame with SOPR history.
+ """
+ df = self.reader.load_csv(self.finder.get_sopr_path())
filtered = self.reader.data_in_timeframe(df, C.TIME, timeframe)[
- [C.TIME, C.SOPR]]
+ [C.TIME, C.SOPR]
+ ]
return filtered
- def save_sopr(self, **kwargs):
- # # given a symbol, save its s2f data
+ def save_sopr(self, **kwargs: Any) -> str | None:
+ """Save SOPR data.
+
+ Args:
+ **kwargs: Arguments passed to get_sopr.
+
+ Returns:
+ Path to saved file, or None if save failed.
+ """
filename = self.finder.get_sopr_path()
if os.path.exists(filename):
os.remove(filename)
df = self.reader.update_df(
- filename, self.get_sopr(**kwargs), C.TIME, C.DATE_FMT)
+ filename, self.get_sopr(**kwargs), C.TIME, C.DATE_FMT
+ )
self.writer.update_csv(filename, df)
if os.path.exists(filename):
return filename
- # def handle_request(self, url, err_msg):
+ return None
+
+ def standardize_ndx(self, df: pd.DataFrame) -> pd.DataFrame:
+ """Standardize NDX (Nasdaq-100) constituency data.
- def standardize_ndx(self, df):
+ Args:
+ df: Raw NDX DataFrame.
+
+ Returns:
+ Standardized DataFrame with current NDX constituents.
+ """
if df.empty:
- df = pd.DataFrame(columns=[C.TIME, C.SYMBOL, C.DELTA])
- df = df.sort_values(
- by=[C.TIME, C.SYMBOL]
- ).drop_duplicates(C.SYMBOL, keep='last')
- df = df[df[C.DELTA] == '+'].reset_index(drop=True)
+ df = pd.DataFrame(columns=pd.Index([C.TIME, C.SYMBOL, C.DELTA]))
+ df = df.sort_values(by=[C.TIME, C.SYMBOL]).drop_duplicates(
+ C.SYMBOL, keep="last"
+ )
+ df = df[df[C.DELTA] == "+"].reset_index(drop=True)
return df
- def get_saved_ndx(self):
+ def get_saved_ndx(self) -> pd.DataFrame:
+ """Get cached NDX constituency data.
+
+ Returns:
+ DataFrame with NDX constituency history.
+ """
df = self.reader.load_csv(self.finder.get_ndx_path())
return df
- def get_ndx(self, date=datetime.now()):
- date = self.traveller.convert_date(date)
+ def get_ndx(self, date: datetime | None = None) -> pd.DataFrame:
+ """Get NDX constituents as of a specific date.
+
+ Args:
+ date: Date to get constituents for (default: now).
+
+ Returns:
+ DataFrame with NDX constituents.
+ """
+ if date is None:
+ date = datetime.now()
+ date_str = self.traveller.convert_date(date)
df = self.get_saved_ndx()
- return self.standardize_ndx(
- df[df[C.TIME] <= date] if C.TIME in df else df)
-
- def get_latest_ndx(self, **kwargs):
- def _get_latest_ndx():
- url = "https://en.wikipedia.org/wiki/Nasdaq-100#Components"
- # alternatives:
- # https://www.nasdaq.com/solutions/nasdaq-100/companies
- # https://www.cnbc.com/nasdaq-100/
- res = requests.get(url)
- soup = BeautifulSoup(res.text, 'html.parser')
+ return self.standardize_ndx(df[df[C.TIME] <= date_str] if C.TIME in df else df)
+
+ def get_latest_ndx(self, **kwargs: Any) -> pd.DataFrame:
+ """Fetch latest NDX constituents from Wikipedia.
+
+ Args:
+ **kwargs: Arguments for retry logic (retries, delay).
+
+ Returns:
+ DataFrame with current NDX constituents.
+ """
+
+ def _get_latest_ndx() -> pd.DataFrame:
+ url = "https://en.wikipedia.org/wiki/Nasdaq-100"
+ headers = {"User-Agent": '"Google Chrome";"Chromium"'}
+ res = requests.get(url, headers=headers)
+ soup = BeautifulSoup(res.text, "html.parser")
html = soup.select("table#constituents")[0]
- df = pd.read_html(str(html))[0]
- symbols = df['Ticker']
+ df = pd.read_html(StringIO(str(html)))[0]
+ symbols = df["Ticker"]
today = datetime.today().strftime(C.DATE_FMT)
- df = pd.DataFrame({
- C.TIME: len(symbols) * [today],
- C.SYMBOL: symbols,
- C.DELTA: len(symbols) * ['+']
- })
- return df
+ return pd.DataFrame(
+ {
+ C.TIME: len(symbols) * [today],
+ C.SYMBOL: symbols,
+ C.DELTA: len(symbols) * ["+"],
+ }
+ )
+
return self.try_again(func=_get_latest_ndx, **kwargs)
- def save_ndx(self, **kwargs):
+ def save_ndx(self, **kwargs: Any) -> str | None:
+ """Save NDX constituency data with changes.
+
+ Args:
+ **kwargs: Arguments for get_latest_ndx.
+
+ Returns:
+ Path to saved file, or None if save failed.
+ """
filename = self.finder.get_ndx_path()
if os.path.exists(filename):
@@ -423,23 +711,35 @@ def save_ndx(self, **kwargs):
today = datetime.now().strftime(C.DATE_FMT)
minus, plus = self.calculator.get_difference(before, after)
union = list(minus.union(plus))
- to_append = pd.DataFrame({
- C.TIME: [today] * len(union),
- C.SYMBOL: union,
- C.DELTA: ['+' if u in plus else '-' for u in union]
- })
- df = pd.concat([saved, to_append], ignore_index=True).sort_values(
- by=[C.TIME, C.SYMBOL]).reset_index(drop=True)
+ to_append = pd.DataFrame(
+ {
+ C.TIME: [today] * len(union),
+ C.SYMBOL: union,
+ C.DELTA: ["+" if u in plus else "-" for u in union],
+ }
+ )
+ df = (
+ pd.concat([saved, to_append], ignore_index=True)
+ .sort_values(by=[C.TIME, C.SYMBOL])
+ .reset_index(drop=True)
+ )
self.writer.update_csv(filename, df)
if os.path.exists(filename):
return filename
+ return None
- def log_api_call_time(self):
+ def log_api_call_time(self) -> None:
+ """Record the timestamp of the last API call."""
self.last_api_call_time = time()
- def obey_free_limit(self, free_delay):
- if self.free and hasattr(self, 'last_api_call_time'):
+ def obey_free_limit(self, free_delay: float) -> None:
+ """Wait if needed to respect free tier rate limits.
+
+ Args:
+ free_delay: Minimum seconds between API calls.
+ """
+ if hasattr(self, "free") and self.free and hasattr(self, "last_api_call_time"):
time_since_last_call = time() - self.last_api_call_time
delay = free_delay - time_since_last_call
if delay > 0:
@@ -447,124 +747,183 @@ def obey_free_limit(self, free_delay):
class Indices(MarketData):
- def __init__(self):
+ """Market data provider for index constituency data."""
+
+ def __init__(self) -> None:
+ """Initialize the Indices provider."""
super().__init__()
- def get_ndx(self, date=datetime.now()):
+ def get_ndx(self, date: datetime | None = None) -> pd.DataFrame:
+ """Get NDX constituents combining cached and live data.
+
+ Args:
+ date: Date to get constituents for (default: now).
+
+ Returns:
+ DataFrame with NDX constituents.
+ """
+ if date is None:
+ date = datetime.now()
old = super().get_ndx(date)
- date = self.traveller.convert_date(date)
+ date_str = self.traveller.convert_date(date)
new = self.get_latest_ndx()
- new = new[new[C.TIME] <= date]
+ new = new[new[C.TIME] <= date_str]
df = pd.concat([old, new])
return self.standardize_ndx(df)
class AlpacaData(MarketData):
+ """Market data provider for Alpaca API."""
+
def __init__(
- self,
- token=os.environ.get('ALPACA'),
- secret=os.environ.get('ALPACA_SECRET'),
- free=True,
- paper=False
- ):
+ self,
+ token: str | None = None,
+ secret: str | None = None,
+ free: bool = True,
+ paper: bool = False,
+ ) -> None:
+ """Initialize the Alpaca data provider.
+
+ Args:
+ token: API key (default: from ALPACA env var).
+ secret: API secret (default: from ALPACA_SECRET env var).
+ free: Use free tier rate limits (default: True).
+ paper: Use paper trading credentials (default: False).
+
+ Raises:
+ Exception: If credentials are missing.
+ """
super().__init__()
- self.base = 'https://data.alpaca.markets'
- self.token = os.environ.get(
- 'ALPACA_PAPER') if paper or C.TEST else token
- self.secret = os.environ.get(
- 'ALPACA_PAPER_SECRET') if paper or C.TEST else secret
+ if token is None:
+ token = os.environ.get("ALPACA")
+ if secret is None:
+ secret = os.environ.get("ALPACA_SECRET")
+ self.base = "https://data.alpaca.markets"
+ self.token = os.environ.get("ALPACA_PAPER") if paper or C.TEST else token
+ self.secret = (
+ os.environ.get("ALPACA_PAPER_SECRET") if paper or C.TEST else secret
+ )
if not (self.token and self.secret):
- raise Exception('missing Alpaca credentials')
- self.provider = 'alpaca'
+ raise Exception("missing Alpaca credentials")
+ self.provider = "alpaca"
self.free = free
- # def get_dividends(self, **kwargs):
- # pass
- # def get_splits(self, **kwargs):
- # pass
+ def get_ohlc(
+ self, symbol: str = "", timeframe: str = "max", **kwargs: Any
+ ) -> pd.DataFrame:
+ """Fetch OHLC data from Alpaca API.
+
+ Args:
+ symbol: The stock/crypto symbol.
+ timeframe: Time range for data (default: "max").
+ **kwargs: Additional arguments.
+
+ Returns:
+ DataFrame with OHLC price history.
+
+ Raises:
+ Exception: If API request fails.
+ """
- def get_ohlc(self, **kwargs):
- def _get_ohlc(symbol, timeframe='max'):
+ def _get_ohlc(symbol: str, timeframe: str = "max") -> pd.DataFrame:
is_crypto = symbol in C.ALPC_CRYPTO_SYMBOLS
- version = 'v1beta3' if is_crypto else 'v2'
+ version = "v1beta3" if is_crypto else "v2"
page_token = None
start, _ = self.traveller.convert_dates(timeframe)
parts = [
self.base,
version,
- 'crypto/us' if is_crypto else 'stocks',
- 'bars',
+ "crypto/us" if is_crypto else "stocks",
+ "bars",
]
- url = '/'.join(parts)
+ url = "/".join(parts)
pre_params = {
- 'symbols': symbol,
- 'timeframe': '1D',
- 'start': start,
- # end should be > 15 min before current UTC time in this format
- # 2025-01-01T00:00:00Z
- 'limit': 10000,
- } | ({} if is_crypto else {'adjustment': 'all'})
+ "symbols": symbol,
+ "timeframe": "1D",
+ "start": start,
+ "limit": 10000,
+ } | ({} if is_crypto else {"adjustment": "all"})
headers = {
- 'APCA-API-KEY-ID': self.token,
- 'APCA-API-SECRET-KEY': self.secret
+ "APCA-API-KEY-ID": self.token,
+ "APCA-API-SECRET-KEY": self.secret,
}
- results = []
+ results: list[dict[str, Any]] = []
while True:
self.obey_free_limit(C.ALPACA_FREE_DELAY)
try:
- post_params = {
- 'page_token': page_token} if page_token else {}
+ post_params = {"page_token": page_token} if page_token else {}
params = pre_params | post_params
response = requests.get(url, params, headers=headers)
if not response.ok:
raise Exception(
- 'Invalid response from Alpaca for OHLC',
+ "Invalid response from Alpaca for OHLC",
response.status_code,
- response.text
+ response.text,
)
data = response.json()
- if data.get('bars') and data['bars'].get(symbol):
- results += data['bars'][symbol]
+ if data.get("bars") and data["bars"].get(symbol):
+ results += data["bars"][symbol]
finally:
self.log_api_call_time()
- if data.get('next_page_token'):
- page_token = data['next_page_token']
+ if data.get("next_page_token"):
+ page_token = data["next_page_token"]
else:
break
df = pd.DataFrame(results)
columns = {
- 't': 'date',
- 'o': 'open',
- 'h': 'high',
- 'l': 'low',
- 'c': 'close',
- 'v': 'volume',
- 'vw': 'average',
- 'n': 'trades'
+ "t": "date",
+ "o": "open",
+ "h": "high",
+ "l": "low",
+ "c": "close",
+ "v": "volume",
+ "vw": "average",
+ "n": "trades",
}
df = df.rename(columns=columns)
- df['date'] = pd.to_datetime(df['date']).dt.tz_convert(
- C.TZ).dt.tz_localize(None)
+ df["date"] = (
+ pd.to_datetime(df["date"]).dt.tz_convert(C.TZ).dt.tz_localize(None)
+ )
df = self.standardize_ohlc(symbol, df)
return self.reader.data_in_timeframe(df, C.TIME, timeframe)
- return self.try_again(func=_get_ohlc, **kwargs)
-
- # def get_intraday(self, **kwargs):
- # pass
- # def get_news(self, **kwargs):
- # pass
+ return self.try_again(
+ func=_get_ohlc, symbol=symbol, timeframe=timeframe, **kwargs
+ )
class Polygon(MarketData):
- def __init__(self, token=os.environ.get('POLYGON'), free=True):
+ """Market data provider for Polygon.io API."""
+
+ def __init__(self, token: str | None = None, free: bool = True) -> None:
+ """Initialize the Polygon data provider.
+
+ Args:
+ token: API key (default: from POLYGON env var).
+ free: Use free tier rate limits (default: True).
+ """
super().__init__()
+ if token is None:
+ token = os.environ.get("POLYGON")
self.client = RESTClient(token)
- self.provider = 'polygon'
+ self.provider = "polygon"
self.free = free
- def paginate(self, gen, apply):
- results = []
+ def paginate(
+ self,
+ gen: Generator[Any, None, None] | Iterator[Any],
+ apply: Callable[[Any], dict[str, Any]],
+ ) -> list[dict[str, Any]]:
+ """Paginate through API results with rate limiting.
+
+ Args:
+ gen: Generator or iterator yielding API response items.
+ apply: Function to transform each item.
+
+ Returns:
+ List of transformed results.
+ """
+ results: list[dict[str, Any]] = []
for idx, item in enumerate(gen):
if idx % C.POLY_MAX_LIMIT == 0:
self.log_api_call_time()
@@ -573,8 +932,21 @@ def paginate(self, gen, apply):
results.append(apply(item))
return results
- def get_dividends(self, **kwargs):
- def _get_dividends(symbol, timeframe='max'):
+ def get_dividends(
+ self, symbol: str = "", timeframe: str = "max", **kwargs: Any
+ ) -> pd.DataFrame:
+ """Fetch dividend data from Polygon API.
+
+ Args:
+ symbol: The stock symbol.
+ timeframe: Time range for data (default: "max").
+ **kwargs: Additional arguments.
+
+ Returns:
+ DataFrame with dividend history.
+ """
+
+ def _get_dividends(symbol: str, timeframe: str = "max") -> pd.DataFrame:
self.obey_free_limit(C.POLY_FREE_DELAY)
try:
start, _ = self.traveller.convert_dates(timeframe)
@@ -582,26 +954,42 @@ def _get_dividends(symbol, timeframe='max'):
self.client.list_dividends(
symbol,
ex_dividend_date_gte=start,
- order='desc',
- sort='ex_dividend_date',
- limit=C.POLY_MAX_LIMIT
+ order="desc",
+ sort="ex_dividend_date",
+ limit=C.POLY_MAX_LIMIT,
),
lambda div: {
- 'exDate': div.ex_dividend_date,
- 'paymentDate': div.pay_date,
- 'declaredDate': div.declaration_date,
- 'amount': div.cash_amount
- }
+ "exDate": div.ex_dividend_date,
+ "paymentDate": div.pay_date,
+ "declaredDate": div.declaration_date,
+ "amount": div.cash_amount,
+ },
)
finally:
self.log_api_call_time()
raw = pd.DataFrame(response)
df = self.standardize_dividends(symbol, raw)
return self.reader.data_in_timeframe(df, C.EX, timeframe)
- return self.try_again(func=_get_dividends, **kwargs)
- def get_splits(self, **kwargs):
- def _get_splits(symbol, timeframe='max'):
+ return self.try_again(
+ func=_get_dividends, symbol=symbol, timeframe=timeframe, **kwargs
+ )
+
+ def get_splits(
+ self, symbol: str = "", timeframe: str = "max", **kwargs: Any
+ ) -> pd.DataFrame:
+ """Fetch split data from Polygon API.
+
+ Args:
+ symbol: The stock symbol.
+ timeframe: Time range for data (default: "max").
+ **kwargs: Additional arguments.
+
+ Returns:
+ DataFrame with split history.
+ """
+
+ def _get_splits(symbol: str, timeframe: str = "max") -> pd.DataFrame:
self.obey_free_limit(C.POLY_FREE_DELAY)
try:
start, _ = self.traveller.convert_dates(timeframe)
@@ -609,239 +997,343 @@ def _get_splits(symbol, timeframe='max'):
self.client.list_splits(
symbol,
execution_date_gte=start,
- order='desc',
- sort='execution_date',
- limit=C.POLY_MAX_LIMIT
+ order="desc",
+ sort="execution_date",
+ limit=C.POLY_MAX_LIMIT,
),
lambda split: {
- 'exDate': split.execution_date,
- 'ratio': split.split_from / split.split_to
- }
+ "exDate": split.execution_date,
+ "ratio": split.split_from / split.split_to,
+ },
)
finally:
self.log_api_call_time()
raw = pd.DataFrame(response)
df = self.standardize_splits(symbol, raw)
return self.reader.data_in_timeframe(df, C.EX, timeframe)
- return self.try_again(func=_get_splits, **kwargs)
- def get_ohlc(self, **kwargs):
- def _get_ohlc(symbol, timeframe='max'):
- is_crypto = symbol.find('X%3A') == 0
- formatted_start, formatted_end = self.traveller.convert_dates(
- timeframe)
+ return self.try_again(
+ func=_get_splits, symbol=symbol, timeframe=timeframe, **kwargs
+ )
+
+ def get_ohlc(
+ self, symbol: str = "", timeframe: str = "max", **kwargs: Any
+ ) -> pd.DataFrame:
+ """Fetch OHLC data from Polygon API.
+
+ Args:
+ symbol: The stock/crypto symbol.
+ timeframe: Time range for data (default: "max").
+ **kwargs: Additional arguments.
+
+ Returns:
+ DataFrame with OHLC price history.
+ """
+
+ def _get_ohlc(symbol: str, timeframe: str = "max") -> pd.DataFrame:
+ is_crypto = symbol.find("X%3A") == 0
+ formatted_start, formatted_end = self.traveller.convert_dates(timeframe)
self.obey_free_limit(C.POLY_FREE_DELAY)
try:
response = self.client.get_aggs(
- symbol, 1, 'day',
+ symbol,
+ 1,
+ "day",
from_=formatted_start,
to=formatted_end,
adjusted=True,
- limit=C.POLY_MAX_AGGS_LIMIT
+ limit=C.POLY_MAX_AGGS_LIMIT,
)
finally:
self.log_api_call_time()
raw = [vars(item) for item in response]
- columns = {'timestamp': 'date',
- 'vwap': 'average', 'transactions': 'trades'}
+ columns = {"timestamp": "date", "vwap": "average", "transactions": "trades"}
df = pd.DataFrame(raw).rename(columns=columns)
if is_crypto:
- df['date'] = pd.to_datetime(
- df['date'], unit='ms')
+ df["date"] = pd.to_datetime(df["date"], unit="ms")
else:
- df['date'] = pd.to_datetime(
- df['date'], unit='ms').dt.tz_localize(
- 'UTC').dt.tz_convert(
- C.TZ).dt.tz_localize(None)
+ df["date"] = (
+ pd.to_datetime(df["date"], unit="ms")
+ .dt.tz_localize("UTC")
+ .dt.tz_convert(C.TZ)
+ .dt.tz_localize(None)
+ )
df = self.standardize_ohlc(symbol, df)
return self.reader.data_in_timeframe(df, C.TIME, timeframe)
- return self.try_again(func=_get_ohlc, **kwargs)
+ return self.try_again(
+ func=_get_ohlc, symbol=symbol, timeframe=timeframe, **kwargs
+ )
- def get_intraday(self, **kwargs):
- def _get_intraday(symbol, min=1, timeframe='max', extra_hrs=True):
- # pass min directly into stock_aggs function as multiplier
- is_crypto = symbol.find('X%3A') == 0
+ def get_intraday(
+ self,
+ symbol: str = "",
+ min: int = 1, # noqa: A002
+ timeframe: str = "max",
+ extra_hrs: bool = False,
+ **kwargs: Any,
+ ) -> Generator[pd.DataFrame, None, None] | None:
+ """Fetch intraday data from Polygon API.
+
+ Args:
+ symbol: The stock/crypto symbol.
+ min: Minute interval for data (default: 1).
+ timeframe: Time range for data (default: "max").
+ extra_hrs: Include extended hours data (default: False).
+ **kwargs: Additional arguments.
+ **kwargs: Must include 'symbol'. Optional 'min', 'timeframe', 'extra_hrs'.
+
+ Returns:
+ Generator yielding DataFrames with intraday OHLC data.
+ """
+
+ def _get_intraday(
+ symbol: str,
+ min: int = 1, # noqa: A002
+ timeframe: str = "max",
+ extra_hrs: bool = True,
+ ) -> Generator[pd.DataFrame, None, None]:
+ is_crypto = symbol.find("X%3A") == 0
dates = self.traveller.dates_in_range(timeframe)
if dates == []:
- raise Exception(f'No dates in timeframe: {timeframe}.')
+ raise Exception(f"No dates in timeframe: {timeframe}.")
for _, date in enumerate(dates):
self.obey_free_limit(C.POLY_FREE_DELAY)
try:
response = self.client.get_aggs(
- symbol, min, 'minute', from_=date, to=date,
- adjusted=True, limit=C.POLY_MAX_AGGS_LIMIT
+ symbol,
+ min,
+ "minute",
+ from_=date,
+ to=date,
+ adjusted=True,
+ limit=C.POLY_MAX_AGGS_LIMIT,
)
- except exceptions.NoResultsError:
- # This is to prevent breaking the loop over weekends
+ except (
+ Exception
+ ): # NoResultsError may not be available in all polygon versions
continue
finally:
self.log_api_call_time()
raw = [vars(item) for item in response]
- columns = {'timestamp': 'date',
- 'vwap': 'average', 'transactions': 'trades'}
+ columns = {
+ "timestamp": "date",
+ "vwap": "average",
+ "transactions": "trades",
+ }
df = pd.DataFrame(raw).rename(columns=columns)
if is_crypto:
- df['date'] = pd.to_datetime(
- df['date'], unit='ms')
+ df["date"] = pd.to_datetime(df["date"], unit="ms")
else:
- df['date'] = pd.to_datetime(
- df['date'], unit='ms').dt.tz_localize(
- 'UTC').dt.tz_convert(
- C.TZ).dt.tz_localize(None)
- filename = self.finder.get_intraday_path(
- symbol, date, self.provider)
+ df["date"] = (
+ pd.to_datetime(df["date"], unit="ms")
+ .dt.tz_localize("UTC")
+ .dt.tz_convert(C.TZ)
+ .dt.tz_localize(None)
+ )
+ filename = self.finder.get_intraday_path(symbol, date, self.provider)
df = self.standardize_ohlc(symbol, df, filename)
df = df[df[C.TIME].dt.strftime(C.DATE_FMT) == date]
yield df
- return self.try_again(func=_get_intraday, **kwargs)
-
-
-# newShares = oldShares / ratio
+ return self.try_again(
+ func=_get_intraday,
+ symbol=symbol,
+ min=min,
+ timeframe=timeframe,
+ extra_hrs=extra_hrs,
+ **kwargs,
+ )
class LaborStats(MarketData):
- def __init__(self):
- super().__init__()
- self.base = 'https://api.bls.gov'
- self.version = 'v2'
- self.token = os.environ.get('BLS')
- self.provider = 'bls'
+ """Market data provider for Bureau of Labor Statistics API."""
- def get_unemployment_rate(self, **kwargs):
- def _get_unemployment_rate(timeframe):
- start, end = self.traveller.convert_dates(timeframe, '%Y')
-
- parts = [
- self.base,
- 'publicAPI',
- self.version,
- 'timeseries',
- 'data'
- ]
- url = '/'.join(parts)
- params = {'registrationkey': self.token,
- 'startyear': start, 'endyear': end,
- 'seriesid': 'LNS14000000'}
+ def __init__(self) -> None:
+ """Initialize the BLS data provider."""
+ super().__init__()
+ self.base = "https://api.bls.gov"
+ self.version = "v2"
+ self.token = os.environ.get("BLS")
+ self.provider = "bls"
+
+ def get_unemployment_rate(
+ self, timeframe: str = "max", **kwargs: Any
+ ) -> pd.DataFrame:
+ """Fetch unemployment rate from BLS API.
+
+ Args:
+ timeframe: Time range for data (default: "max").
+ **kwargs: Additional arguments.
+
+ Returns:
+ DataFrame with unemployment rate history.
+
+ Raises:
+ Exception: If API request fails.
+ """
+
+ def _get_unemployment_rate(timeframe: str) -> pd.DataFrame:
+ start, end = self.traveller.convert_dates(timeframe, "%Y")
+
+ parts = [self.base, "publicAPI", self.version, "timeseries", "data"]
+ url = "/".join(parts)
+ params = {
+ "registrationkey": self.token,
+ "startyear": start,
+ "endyear": end,
+ "seriesid": "LNS14000000",
+ }
response = requests.post(url, data=params)
- if (
- response.ok and
- response.json()['status'] == 'REQUEST_SUCCEEDED'
- ):
-
+ if response.ok and response.json()["status"] == "REQUEST_SUCCEEDED":
payload = response.json()
- if payload['status'] == 'REQUEST_SUCCEEDED':
- data = payload['Results']['series'][0]['data']
+ if payload["status"] == "REQUEST_SUCCEEDED":
+ data = payload["Results"]["series"][0]["data"]
else:
raise Exception(
- f'''
- Invalid response from BLS because {data["message"][0]}
- '''
+ f"""
+ Invalid response from BLS because {payload.get("message", ["Unknown error"])[0]}
+ """
)
else:
raise Exception(
- 'Invalid response from BLS for unemployment rate',
- response.status_code, response.json()
+ "Invalid response from BLS for unemployment rate",
+ response.status_code,
+ response.json(),
)
df = pd.DataFrame(data)
- df['time'] = df['year'] + '-' + \
- df['period'].str.slice(start=1)
+ df["time"] = df["year"] + "-" + df["period"].str.slice(start=1)
df = self.standardize_unemployment(df)
return self.reader.data_in_timeframe(df, C.TIME, timeframe)
- return self.try_again(func=_get_unemployment_rate, **kwargs)
+ return self.try_again(
+ func=_get_unemployment_rate, timeframe=timeframe, **kwargs
+ )
class Glassnode(MarketData):
- def __init__(self, use_cookies=False):
+ """Market data provider for Glassnode crypto analytics API."""
+
+ def __init__(self, use_cookies: bool = False) -> None:
+ """Initialize the Glassnode data provider.
+
+ Args:
+ use_cookies: Use browser-based authentication (default: False).
+ """
super().__init__()
- self.base = 'https://api.glassnode.com'
- self.version = 'v1'
- self.token = os.environ.get('GLASSNODE')
- self.provider = 'glassnode'
+ self.base = "https://api.glassnode.com"
+ self.version = "v1"
+ self.token = os.environ.get("GLASSNODE")
+ self.provider = "glassnode"
self.use_cookies = use_cookies
if self.use_cookies:
self.use_auth()
- def use_auth(self):
+ def use_auth(self) -> None:
+ """Authenticate via browser to get session cookies."""
options = ChromeOptions()
- options.add_argument('--headless')
- options.add_argument('--no-sandbox')
- options.add_argument('--disable-dev-shm-usage')
- options.set_capability('goog:loggingPrefs', {"performance": "ALL"})
+ options.add_argument("--headless")
+ options.add_argument("--no-sandbox")
+ options.add_argument("--disable-dev-shm-usage")
+ options.set_capability("goog:loggingPrefs", {"performance": "ALL"})
driver = webdriver.Chrome(options=options)
- driver.get('https://studio.glassnode.com/auth/login')
+ driver.get("https://studio.glassnode.com/auth/login")
delay = 10
- def get_element(id):
+ def get_element(id: str) -> Any: # noqa: A002
return WebDriverWait(driver, delay).until(
- EC.presence_of_element_located((By.ID, id)))
+ EC.presence_of_element_located((By.ID, id))
+ )
- email = get_element('email')
- email.send_keys(os.environ['RH_USERNAME'])
- password = get_element('current-password')
- password.send_keys(os.environ['GLASSNODE_PASS'])
+ email = get_element("email")
+ email.send_keys(os.environ["RH_USERNAME"])
+ password = get_element("current-password")
+ password.send_keys(os.environ["GLASSNODE_PASS"])
password.send_keys(Keys.ENTER)
sleep(15)
- driver.get('https://studio.glassnode.com/metrics')
+ driver.get("https://studio.glassnode.com/metrics")
sleep(5)
url = (
"https://api.glassnode.com/v1/metrics/market/price_usd_close"
"?a=BTC&i=24h&referer=charts"
)
driver.get(url)
- sleep(5) # wait for the requests to take place
+ sleep(5)
- # extract requests from logs
raw_logs = driver.get_log("performance")
- logs = [json.loads(raw_log["message"])["message"]
- for raw_log in raw_logs]
+ logs = [json.loads(raw_log["message"])["message"] for raw_log in raw_logs]
- def log_filter(log_):
+ def log_filter(log_: dict[str, Any]) -> bool:
return (
- log_["method"] == "Network.requestWillBeSent" and
- log_['params']['request']['url'] == url and
- log_['params']['request']['method'] == 'GET'
+ log_["method"] == "Network.requestWillBeSent"
+ and log_["params"]["request"]["url"] == url
+ and log_["params"]["request"]["method"] == "GET"
)
- self.headers = [log['params']['request']['headers']
- for log in filter(log_filter, logs)][-1]
- self.cookies = {cookie['name']: cookie['value']
- for cookie in driver.get_cookies()}
-
- def make_request(self, url):
- params = {'a': 'BTC', 'c': 'native', 'i': '24h', 'referer': 'charts'}
+ self.headers = [
+ log["params"]["request"]["headers"] for log in filter(log_filter, logs)
+ ][-1]
+ self.cookies = {
+ cookie["name"]: cookie["value"] for cookie in driver.get_cookies()
+ }
+
+ def make_request(self, url: str) -> requests.Response:
+ """Make authenticated request to Glassnode API.
+
+ Args:
+ url: API endpoint URL.
+
+ Returns:
+ Response object from the API.
+ """
+ params: dict[str, str] = {
+ "a": "BTC",
+ "c": "native",
+ "i": "24h",
+ "referer": "charts",
+ }
if self.use_cookies:
headers = self.headers
cookies = self.cookies
else:
- params['api_key'] = self.token
+ params["api_key"] = self.token or ""
headers = {}
cookies = {}
- response = requests.get(
- url, params=params, headers=headers, cookies=cookies)
+ response = requests.get(url, params=params, headers=headers, cookies=cookies)
sleep(random() * 5)
return response
- def get_s2f_ratio(self, **kwargs):
- def _get_s2f_ratio(timeframe):
+ def get_s2f_ratio(self, timeframe: str = "max", **kwargs: Any) -> pd.DataFrame:
+ """Fetch stock-to-flow ratio from Glassnode API.
+
+ Args:
+ timeframe: Time range for data (default: "max").
+ **kwargs: Additional arguments.
+
+ Returns:
+ DataFrame with S2F ratio history.
+
+ Raises:
+ Exception: If API request fails.
+ """
+
+ def _get_s2f_ratio(timeframe: str) -> pd.DataFrame:
parts = [
self.base,
self.version,
- 'metrics',
- 'indicators',
- 'stock_to_flow_ratio'
+ "metrics",
+ "indicators",
+ "stock_to_flow_ratio",
]
- url = '/'.join(parts)
+ url = "/".join(parts)
empty = pd.DataFrame()
response = self.make_request(url)
@@ -849,28 +1341,42 @@ def _get_s2f_ratio(timeframe):
data = response.json()
else:
raise Exception(
- 'Invalid response from Glassnode for S2F Ratio', response)
+ "Invalid response from Glassnode for S2F Ratio", response
+ )
if data == []:
return empty
df = pd.json_normalize(data)
- df['t'] = pd.to_datetime(df['t'], unit='s')
+ df["t"] = pd.to_datetime(df["t"], unit="s")
df = self.standardize_s2f_ratio(df)
return self.reader.data_in_timeframe(df, C.TIME, timeframe)
- return self.try_again(func=_get_s2f_ratio, **kwargs)
+ return self.try_again(func=_get_s2f_ratio, timeframe=timeframe, **kwargs)
+
+ def get_diff_ribbon(self, timeframe: str = "max", **kwargs: Any) -> pd.DataFrame:
+ """Fetch difficulty ribbon from Glassnode API.
+
+ Args:
+ timeframe: Time range for data (default: "max").
+ **kwargs: Additional arguments.
+
+ Returns:
+ DataFrame with difficulty ribbon moving averages.
- def get_diff_ribbon(self, **kwargs):
- def _get_diff_ribbon(timeframe):
+ Raises:
+ Exception: If API request fails.
+ """
+
+ def _get_diff_ribbon(timeframe: str) -> pd.DataFrame:
parts = [
self.base,
self.version,
- 'metrics',
- 'indicators',
- 'difficulty_ribbon'
+ "metrics",
+ "indicators",
+ "difficulty_ribbon",
]
- url = '/'.join(parts)
+ url = "/".join(parts)
empty = pd.DataFrame()
response = self.make_request(url)
@@ -878,45 +1384,50 @@ def _get_diff_ribbon(timeframe):
data = response.json()
else:
raise Exception(
- 'Invalid response from Glassnode for Difficulty Ribbon',
- response
+ "Invalid response from Glassnode for Difficulty Ribbon", response
)
if data == []:
return empty
df = pd.json_normalize(data)
- df['t'] = pd.to_datetime(df['t'], unit='s')
+ df["t"] = pd.to_datetime(df["t"], unit="s")
df = self.standardize_diff_ribbon(df)
return self.reader.data_in_timeframe(df, C.TIME, timeframe)
- return self.try_again(func=_get_diff_ribbon, **kwargs)
+ return self.try_again(func=_get_diff_ribbon, timeframe=timeframe, **kwargs)
- def get_sopr(self, **kwargs):
- def _get_sopr(timeframe):
- parts = [
- self.base,
- self.version,
- 'metrics',
- 'indicators',
- 'sopr'
- ]
- url = '/'.join(parts)
+ def get_sopr(self, timeframe: str = "max", **kwargs: Any) -> pd.DataFrame:
+ """Fetch SOPR from Glassnode API.
+
+ Args:
+ timeframe: Time range for data (default: "max").
+ **kwargs: Additional arguments.
+
+ Returns:
+ DataFrame with SOPR history.
+
+ Raises:
+ Exception: If API request fails.
+ """
+
+ def _get_sopr(timeframe: str) -> pd.DataFrame:
+ parts = [self.base, self.version, "metrics", "indicators", "sopr"]
+ url = "/".join(parts)
empty = pd.DataFrame()
response = self.make_request(url)
if response.ok:
data = response.json()
else:
- raise Exception(
- 'Invalid response from Glassnode for SOPR', response)
+ raise Exception("Invalid response from Glassnode for SOPR", response)
if data == []:
return empty
df = pd.json_normalize(data)
- df['t'] = pd.to_datetime(df['t'], unit='s')
+ df["t"] = pd.to_datetime(df["t"], unit="s")
df = self.standardize_sopr(df)
return self.reader.data_in_timeframe(df, C.TIME, timeframe)
- return self.try_again(func=_get_sopr, **kwargs)
+ return self.try_again(func=_get_sopr, timeframe=timeframe, **kwargs)
diff --git a/hyperdrive/Exchange.py b/hyperdrive/Exchange.py
index bc275a25..ef094c95 100755
--- a/hyperdrive/Exchange.py
+++ b/hyperdrive/Exchange.py
@@ -1,210 +1,377 @@
-import os
-import time
-import hmac
+"""Cryptocurrency exchange integrations for trading operations."""
+
import base64
import hashlib
-import requests
+import hmac
+import os
+import time
import urllib.parse
+from collections.abc import Callable, Iterable
from time import sleep
+from typing import Any
+
+import requests
from binance import Client
-from typing import Iterable, Union, Optional
from binance.helpers import round_step_size
-from dotenv import load_dotenv, find_dotenv
-import Constants as C
-load_dotenv(find_dotenv('config.env'))
+from dotenv import find_dotenv, load_dotenv
+
+from . import Constants as C
+
+load_dotenv(find_dotenv("config.env"))
class CEX:
+ """Base class for centralized exchange clients.
+
+ Provides common functionality shared by exchange implementations.
+ """
+
def create_pair(self, base: str, quote: str) -> str:
- return f'{base}{quote}'
+ """Create a trading pair symbol from base and quote currencies.
+
+ Args:
+ base: Base currency symbol (e.g., 'BTC').
+ quote: Quote currency symbol (e.g., 'USD').
+
+ Returns:
+ Combined trading pair (e.g., 'BTCUSD').
+ """
+ return f"{base}{quote}"
class AlpacaEx(CEX):
+ """Alpaca exchange client for stock and crypto trading.
+
+ Attributes:
+ base: Base API URL.
+ version: API version string.
+ token: API key.
+ secret: API secret.
+ """
+
def __init__(
- self,
- token: Optional[str] = os.environ.get('ALPACA'),
- secret: Optional[str] = os.environ.get('ALPACA_SECRET'),
- paper: bool = False
+ self,
+ token: str | None = os.environ.get("ALPACA"),
+ secret: str | None = os.environ.get("ALPACA_SECRET"),
+ paper: bool = False,
) -> None:
+ """Initialize the Alpaca client.
+
+ Args:
+ token: API key. Defaults to ALPACA env var.
+ secret: API secret. Defaults to ALPACA_SECRET env var.
+ paper: Use paper trading account if True.
+
+ Raises:
+ Exception: If credentials are missing.
+ """
super().__init__()
- self.base = (f'https://{"paper-" if paper or C.TEST else ""}'
- 'api.alpaca.markets')
- self.version = 'v2'
- self.token = os.environ.get(
- 'ALPACA_PAPER') if paper or C.TEST else token
- self.secret = os.environ.get(
- 'ALPACA_PAPER_SECRET') if paper or C.TEST else secret
+ self.base = f"https://{'paper-' if paper or C.TEST else ''}api.alpaca.markets"
+ self.version = "v2"
+ self.token = os.environ.get("ALPACA_PAPER") if paper or C.TEST else token
+ self.secret = (
+ os.environ.get("ALPACA_PAPER_SECRET") if paper or C.TEST else secret
+ )
if not (self.token and self.secret):
- raise Exception('missing Alpaca credentials')
+ raise Exception("missing Alpaca credentials")
def fill_orders(
- self,
- symbols: Iterable[str],
- func: callable,
- **kwargs: dict[str, any]
- ) -> list[dict[str, any]]:
- pending_orders = set()
- completed_orders = []
+ self,
+ symbols: Iterable[str],
+ func: Callable[..., dict[str, Any]],
+ **kwargs: Any,
+ ) -> list[dict[str, Any]]:
+ """Execute orders for multiple symbols and wait for fills.
+
+ Args:
+ symbols: Symbols to trade.
+ func: Order function to call for each symbol.
+ **kwargs: Additional arguments for the order function.
+
+ Returns:
+ List of completed order responses.
+ """
+ pending_orders: set[str] = set()
+ completed_orders: list[dict[str, Any]] = []
for symbol in symbols:
order = func(symbol, **kwargs)
- if order['status'] == 'filled':
+ if order["status"] == "filled":
completed_orders.append(order)
else:
- pending_orders.add(order['id'])
+ pending_orders.add(order["id"])
while pending_orders:
for id in list(pending_orders):
order = self.get_order(id)
- if order['status'] == 'filled':
+ if order["status"] == "filled":
completed_orders.append(order)
pending_orders.discard(id)
sleep(1)
return completed_orders
def make_request(
- self,
- method: str,
- route: str,
- payload: Optional[dict[str, any]] = {}
- ) -> any:
+ self,
+ method: str,
+ route: str,
+ payload: dict[str, Any] | None = None,
+ ) -> Any:
+ """Make an authenticated API request.
+
+ Args:
+ method: HTTP method (GET, POST, DELETE, etc.).
+ route: API endpoint route.
+ payload: Request body for POST requests.
+
+ Returns:
+ JSON response data.
+
+ Raises:
+ RuntimeError: If the request fails.
+ """
+ if payload is None:
+ payload = {}
parts = [self.base, self.version, route]
- url = '/'.join(parts)
+ url = "/".join(parts)
headers = {
"accept": "application/json",
"APCA-API-KEY-ID": self.token,
- "APCA-API-SECRET-KEY": self.secret
+ "APCA-API-SECRET-KEY": self.secret,
}
response = requests.request(method, url, json=payload, headers=headers)
if response.ok:
return response.json()
else:
- raise Exception(response.text)
+ raise RuntimeError(response.text)
+
+ def get_positions(self) -> Any:
+ """Get all open positions.
+
+ Returns:
+ List of position data.
+ """
+ return self.make_request("GET", "positions")
+
+ def close_position(self, symbol: str) -> Any:
+ """Close a position for a symbol.
+
+ Args:
+ symbol: Trading symbol to close.
+
+ Returns:
+ Close order response.
+ """
+ return self.make_request("DELETE", f"positions/{symbol}")
+
+ def get_order(self, id: str) -> Any:
+ """Get order details by ID.
+
+ Args:
+ id: Order ID.
- def get_positions(self) -> any:
- return self.make_request('GET', 'positions')
+ Returns:
+ Order data.
+ """
+ return self.make_request("GET", f"orders/{id}")
- def close_position(self, symbol: str) -> any:
- return self.make_request('DELETE', f'positions/{symbol}')
+ def get_account(self) -> Any:
+ """Get account information.
- def get_order(self, id: str) -> any:
- return self.make_request('GET', f'orders/{id}')
+ Returns:
+ Account data including balances.
+ """
+ return self.make_request("GET", "account")
- def get_account(self) -> any:
- return self.make_request('GET', 'account')
+ def create_order(self, symbol: str, side: str, notional: int | float | str) -> Any:
+ """Create a market order by notional value.
- def create_order(
- self,
- symbol: str,
- side: str,
- notional: Union[int, float, str]
- ) -> any:
+ Args:
+ symbol: Trading symbol.
+ side: 'buy' or 'sell'.
+ notional: Dollar amount to trade.
+
+ Returns:
+ Order response.
+ """
payload = {
- 'symbol': symbol,
- 'side': side.lower(),
- 'type': 'market',
- 'notional': str(notional),
- 'time_in_force': (
- 'gtc' if symbol in C.ALPC_CRYPTO_SYMBOLS else 'day'
- )
+ "symbol": symbol,
+ "side": side.lower(),
+ "type": "market",
+ "notional": str(notional),
+ "time_in_force": ("gtc" if symbol in C.ALPC_CRYPTO_SYMBOLS else "day"),
}
- return self.make_request('POST', 'orders', payload)
+ return self.make_request("POST", "orders", payload)
class Kraken(CEX):
- def __init__(self, key=None, secret=None, test=False):
+ """Kraken exchange client for cryptocurrency trading.
+
+ Attributes:
+ key: API key.
+ secret: API secret.
+ test: Enable test mode.
+ api_url: Base API URL.
+ version: API version.
+ """
+
+ def __init__(
+ self,
+ key: str | None = None,
+ secret: str | None = None,
+ test: bool = False,
+ ) -> None:
+ """Initialize the Kraken client.
+
+ Args:
+ key: API key. Defaults to KRAKEN_KEY env var.
+ secret: API secret. Defaults to KRAKEN_SECRET env var.
+ test: Enable test/validation mode.
+ """
super().__init__()
self.key = key
self.secret = secret
self.test = test
if not key:
- self.key = os.environ['KRAKEN_KEY']
+ self.key = os.environ["KRAKEN_KEY"]
if not secret:
- self.secret = os.environ['KRAKEN_SECRET']
- self.api_url = 'https://api.kraken.com'
- self.version = '0'
+ self.secret = os.environ["KRAKEN_SECRET"]
+ self.api_url = "https://api.kraken.com"
+ self.version = "0"
+
+ def get_signature(self, urlpath: str, data: dict[str, Any]) -> str:
+ """Generate API signature for authenticated requests.
- def get_signature(self, urlpath, data):
+ Args:
+ urlpath: API endpoint path.
+ data: Request data including nonce.
+
+ Returns:
+ Base64-encoded signature string.
+ """
postdata = urllib.parse.urlencode(data)
- encoded = (str(data['nonce']) + postdata).encode()
+ encoded = (str(data["nonce"]) + postdata).encode()
message = urlpath.encode() + hashlib.sha256(encoded).digest()
- mac = hmac.new(base64.b64decode(self.secret), message, hashlib.sha512)
+ mac = hmac.new(base64.b64decode(self.secret or ""), message, hashlib.sha512)
sigdigest = base64.b64encode(mac.digest())
return sigdigest.decode()
- def make_auth_req(self, uri_path, data={}):
- data['nonce'] = self.gen_nonce()
- headers = {}
- headers['API-Key'] = self.key
- # get_kraken_signature() as defined in the 'Authentication' section
- headers['API-Sign'] = self.get_signature(uri_path, data)
- response = requests.post(
- (self.api_url + uri_path),
- headers=headers,
- data=data
- )
+ def make_auth_req(self, uri_path: str, data: dict[str, Any] | None = None) -> Any:
+ """Make an authenticated API request.
+
+ Args:
+ uri_path: API endpoint path.
+ data: Request data.
+
+ Returns:
+ API response result.
+ """
+ if data is None:
+ data = {}
+ data["nonce"] = self.gen_nonce()
+ headers: dict[str, str] = {}
+ headers["API-Key"] = self.key or ""
+ headers["API-Sign"] = self.get_signature(uri_path, data)
+ response = requests.post((self.api_url + uri_path), headers=headers, data=data)
return self.handle_response(response)
- def gen_nonce(self):
+ def gen_nonce(self) -> str:
+ """Generate a unique nonce for API requests.
+
+ Returns:
+ Millisecond timestamp as string.
+ """
return str(int(1000 * time.time()))
- def get_balance(self):
- access = 'private'
- endpoint = 'Balance'
+ def get_balance(self) -> dict[str, float]:
+ """Get account balances for all assets.
+
+ Returns:
+ Dictionary mapping asset symbols to balances.
+ """
+ access = "private"
+ endpoint = "Balance"
parts = [
- '',
+ "",
self.version,
access,
endpoint,
]
- url = '/'.join(parts)
+ url = "/".join(parts)
response = self.make_auth_req(url)
for asset in response:
response[asset] = float(response[asset])
return response
- def get_asset_pair(self, pair):
- access = 'public'
- endpoint = 'AssetPairs'
+ def get_asset_pair(self, pair: str) -> dict[str, Any]:
+ """Get trading pair information.
+
+ Args:
+ pair: Trading pair symbol.
+
+ Returns:
+ Pair configuration and limits.
+ """
+ access = "public"
+ endpoint = "AssetPairs"
parts = [
self.api_url,
self.version,
access,
endpoint,
]
- url = '/'.join(parts)
- params = {
- 'pair': pair
- }
+ url = "/".join(parts)
+ params = {"pair": pair}
response = requests.get(url, params=params)
result = self.handle_response(response)[pair]
return result
- def order(self, base, quote, side, spend_ratio=1, test=False):
+ def order(
+ self,
+ base: str,
+ quote: str,
+ side: str,
+ spend_ratio: float = 1,
+ test: bool = False,
+ ) -> Any:
+ """Place a market order.
+
+ Args:
+ base: Base currency.
+ quote: Quote currency.
+ side: 'buy' or 'sell'.
+ spend_ratio: Fraction of balance to use.
+ test: Validate order without executing.
+
+ Returns:
+ Order response.
+
+ Raises:
+ Exception: If side is not BUY or SELL.
+ """
pair = self.create_pair(base, quote)
pair_info = self.get_asset_pair(pair)
fee = self.get_fee(pair) / 100
spend_ratio = spend_ratio - fee
side = side.lower()
- access = 'private'
- endpoint = 'AddOrder'
+ access = "private"
+ endpoint = "AddOrder"
parts = [
- '',
+ "",
self.version,
access,
endpoint,
]
- url = '/'.join(parts)
+ url = "/".join(parts)
- oflags = ['nompp']
+ oflags = ["nompp"]
balance_label = base
- precision_label = 'lot_decimals'
+ precision_label = "lot_decimals"
if side.upper() == C.BUY:
- oflags.append('viqc')
+ oflags.append("viqc")
balance_label = quote
- precision_label = 'cost_decimals'
+ precision_label = "cost_decimals"
elif side.upper() != C.SELL:
- raise Exception('Need to specify BUY or SELL side for order')
+ raise Exception("Need to specify BUY or SELL side for order")
balance = self.get_balance()[balance_label]
amount = spend_ratio * balance
@@ -212,189 +379,288 @@ def order(self, base, quote, side, spend_ratio=1, test=False):
volume = "{:0.0{}f}".format(amount, precision)
data = {
- 'ordertype': 'market',
- 'type': side.lower(),
- 'pair': pair,
- 'oflags': ','.join(oflags),
- 'volume': volume,
- 'validate': test or self.test
+ "ordertype": "market",
+ "type": side.lower(),
+ "pair": pair,
+ "oflags": ",".join(oflags),
+ "volume": volume,
+ "validate": test or self.test,
}
response = self.make_auth_req(url, data)
return response
- def handle_response(self, response):
+ def handle_response(self, response: requests.Response) -> Any:
+ """Handle API response and check for errors.
+
+ Args:
+ response: HTTP response object.
+
+ Returns:
+ Result data from response.
+
+ Raises:
+ Exception: If response contains errors.
+ """
response = response.json()
- error = response['error']
+ error = response["error"]
if error:
raise Exception(error)
- return response['result']
+ return response["result"]
+
+ def get_order(self, order_id: str) -> dict[str, Any]:
+ """Get order details by ID.
- def get_order(self, order_id):
- access = 'private'
- endpoint = 'QueryOrders'
+ Args:
+ order_id: Transaction ID of the order.
+
+ Returns:
+ Order data with order_id field added.
+ """
+ access = "private"
+ endpoint = "QueryOrders"
parts = [
- '',
+ "",
self.version,
access,
endpoint,
]
- url = '/'.join(parts)
- data = {
- 'txid': order_id,
- 'trades': True
- }
+ url = "/".join(parts)
+ data = {"txid": order_id, "trades": True}
response = self.make_auth_req(url, data)
order = response[order_id]
- order['order_id'] = order_id
+ order["order_id"] = order_id
return order
- def get_trades(self, trade_ids):
- access = 'private'
- endpoint = 'QueryTrades'
+ def get_trades(self, trade_ids: list[str]) -> list[dict[str, Any]]:
+ """Get trade details for multiple trade IDs.
+
+ Args:
+ trade_ids: List of trade transaction IDs.
+
+ Returns:
+ List of trade data with trade_id field added.
+ """
+ access = "private"
+ endpoint = "QueryTrades"
parts = [
- '',
+ "",
self.version,
access,
endpoint,
]
- url = '/'.join(parts)
- data = {
- 'txid': ','.join(trade_ids),
- 'trades': True
- }
+ url = "/".join(parts)
+ data = {"txid": ",".join(trade_ids), "trades": True}
response = self.make_auth_req(url, data)
trades = [
- {
- **response[trade_id],
- **{'trade_id': trade_id}
- }
- for trade_id in trade_ids
+ {**response[trade_id], **{"trade_id": trade_id}} for trade_id in trade_ids
]
return trades
- def standardize_order(self, order, trades):
- std = {}
- std['symbol'] = order['descr']['pair']
- std['orderId'] = order['order_id']
- std['transactTime'] = int(
- (order['closetm'] + order['opentm']) / 2 * 1000)
- std['price'] = round(float(order['price']), 10)
- side = order['descr']['type'].upper()
- origQty = float(order['vol'])
+ def standardize_order(
+ self, order: dict[str, Any], trades: list[dict[str, Any]]
+ ) -> dict[str, Any]:
+ """Convert Kraken order format to standardized format.
+
+ Args:
+ order: Kraken order data.
+ trades: List of associated trades.
+
+ Returns:
+ Standardized order data compatible with other exchanges.
+ """
+ std: dict[str, Any] = {}
+ std["symbol"] = order["descr"]["pair"]
+ std["orderId"] = order["order_id"]
+ std["transactTime"] = int((order["closetm"] + order["opentm"]) / 2 * 1000)
+ std["price"] = round(float(order["price"]), 10)
+ side = order["descr"]["type"].upper()
+ origQty = float(order["vol"])
if side == C.BUY:
- # other test would be [if 'viqc' in order['oflags'].split(','):]
- origQty = round(origQty / std['price'], 10)
- std['origQty'] = origQty
- std['executedQty'] = float(order['vol_exec'])
- std['cummulativeQuoteQty'] = round(
- std['price'] * std['executedQty'], 10)
- std['status'] = order['status'].upper()
- std['type'] = order['descr']['ordertype'].upper()
- std['side'] = side
-
- def standardize_trade(trade):
- std_trade = {}
- std_trade['price'] = str(round(float(trade['price']), 10))
- std_trade['qty'] = trade['vol']
- std_trade['commission'] = trade['fee']
- std_trade['tradeId'] = trade['trade_id']
+ origQty = round(origQty / std["price"], 10)
+ std["origQty"] = origQty
+ std["executedQty"] = float(order["vol_exec"])
+ std["cummulativeQuoteQty"] = round(std["price"] * std["executedQty"], 10)
+ std["status"] = order["status"].upper()
+ std["type"] = order["descr"]["ordertype"].upper()
+ std["side"] = side
+
+ def standardize_trade(trade: dict[str, Any]) -> dict[str, Any]:
+ std_trade: dict[str, Any] = {}
+ std_trade["price"] = str(round(float(trade["price"]), 10))
+ std_trade["qty"] = trade["vol"]
+ std_trade["commission"] = trade["fee"]
+ std_trade["tradeId"] = trade["trade_id"]
return std_trade
+
fills = [standardize_trade(trade) for trade in trades]
- std['fills'] = fills
+ std["fills"] = fills
return std
- def get_test_side(self, base, quote):
- pair = f'{base}{quote}'
+ def get_test_side(self, base: str, quote: str) -> str:
+ """Determine optimal trade side for testing.
+
+ Args:
+ base: Base currency.
+ quote: Quote currency.
+
+ Returns:
+ 'buy' or 'sell' based on current balances.
+ """
+ pair = f"{base}{quote}"
balances = self.get_balance()
base_bal = balances[base]
quote_bal = balances[quote]
price = self.get_price(pair)
base_val = base_bal * price
- side = 'buy' if quote_bal > base_val else 'sell'
+ side = "buy" if quote_bal > base_val else "sell"
return side
- def get_fee(self, pair):
- access = 'private'
- endpoint = 'TradeVolume'
+ def get_fee(self, pair: str) -> float:
+ """Get trading fee for a pair.
+
+ Args:
+ pair: Trading pair symbol.
+
+ Returns:
+ Fee as a percentage (e.g., 0.26 for 0.26%).
+ """
+ access = "private"
+ endpoint = "TradeVolume"
parts = [
- '',
+ "",
self.version,
access,
endpoint,
]
- url = '/'.join(parts)
- data = {
- "pair": pair
- }
+ url = "/".join(parts)
+ data = {"pair": pair}
response = self.make_auth_req(url, data)
- fee = float(response['fees'][pair]['fee'])
+ fee = float(response["fees"][pair]["fee"])
return fee
- def get_ticker(self, pair=None):
- access = 'public'
- endpoint = 'Ticker'
+ def get_ticker(self, pair: str | None = None) -> dict[str, Any]:
+ """Get ticker data for a trading pair.
+
+ Args:
+ pair: Trading pair symbol. None for all pairs.
+
+ Returns:
+ Ticker data including price and volume.
+ """
+ access = "public"
+ endpoint = "Ticker"
parts = [
- '',
+ "",
self.version,
access,
endpoint,
]
- url = '/'.join(parts)
- data = {
- "pair": pair
- } if pair else {}
+ url = "/".join(parts)
+ data = {"pair": pair} if pair else {}
response = self.make_auth_req(url, data)
return response
- def get_price(self, pair):
+ def get_price(self, pair: str) -> float:
+ """Get current price for a trading pair.
+
+ Args:
+ pair: Trading pair symbol.
+
+ Returns:
+ Current price as float.
+ """
ticker = self.get_ticker(pair)
- price = float(ticker[pair]['c'][0])
+ price = float(ticker[pair]["c"][0])
return price
class Binance(CEX):
- def __init__(self, key=None, secret=None, testnet=False):
+ """Binance exchange client for cryptocurrency trading.
+
+ Attributes:
+ key: API key.
+ secret: API secret.
+ client: Binance API client instance.
+ """
+
+ def __init__(
+ self,
+ key: str | None = None,
+ secret: str | None = None,
+ testnet: bool = False,
+ ) -> None:
+ """Initialize the Binance client.
+
+ Args:
+ key: API key. Defaults to BINANCE_KEY env var.
+ secret: API secret. Defaults to BINANCE_SECRET env var.
+ testnet: Use testnet if True.
+ """
super().__init__()
self.key = key
self.secret = secret
if not key:
if testnet:
- self.key = os.environ['BINANCE_TESTNET_KEY']
+ self.key = os.environ["BINANCE_TESTNET_KEY"]
else:
- self.key = os.environ['BINANCE_KEY']
+ self.key = os.environ["BINANCE_KEY"]
if not secret:
if testnet:
- self.secret = os.environ['BINANCE_TESTNET_SECRET']
+ self.secret = os.environ["BINANCE_TESTNET_SECRET"]
else:
- self.secret = os.environ['BINANCE_SECRET']
- self.client = Client(self.key, self.secret, testnet=testnet, tld='us')
-
- def order(self, base, quote, side, spend_ratio=1, test=False):
+ self.secret = os.environ["BINANCE_SECRET"]
+ self.client = Client(self.key, self.secret, testnet=testnet, tld="us")
+
+ def order(
+ self,
+ base: str,
+ quote: str,
+ side: str,
+ spend_ratio: float = 1,
+ test: bool = False,
+ ) -> dict[str, Any]:
+ """Place a market order.
+
+ Args:
+ base: Base currency.
+ quote: Quote currency.
+ side: 'BUY' or 'SELL'.
+ spend_ratio: Fraction of balance to use.
+ test: Validate order without executing.
+
+ Returns:
+ Order response data.
+
+ Raises:
+ Exception: If side is not BUY or SELL.
+ """
# fee is 0.1%, so max spend_ratio is 99.9%
spend_ratio = spend_ratio - C.BINANCE_FEE
pair = self.create_pair(base, quote)
side = side.upper()
order_type = self.client.ORDER_TYPE_MARKET
- params = {'symbol': pair, 'type': order_type}
+ params: dict[str, Any] = {"symbol": pair, "type": order_type}
symbol_info = self.client.get_symbol_info(pair)
+ if symbol_info is None:
+ raise Exception(f"Symbol info not found for {pair}")
if side == C.SELL:
side = self.client.SIDE_SELL
balance_label = base
- quantity_label = 'quantity'
- filters = symbol_info['filters']
+ quantity_label = "quantity"
+ filters = symbol_info["filters"]
for filter in filters:
- if filter['filterType'] == 'LOT_SIZE':
- step_size = float(filter['stepSize'])
+ if filter["filterType"] == "LOT_SIZE":
+ step_size = float(filter["stepSize"])
elif side == C.BUY:
side = self.client.SIDE_BUY
balance_label = quote
- quantity_label = 'quoteOrderQty'
- precision = int(symbol_info['quoteAssetPrecision'])
+ quantity_label = "quoteOrderQty"
+ precision = int(symbol_info["quoteAssetPrecision"])
else:
- raise Exception('Need to specify BUY or SELL side for order')
+ raise Exception("Need to specify BUY or SELL side for order")
- balance = float(self.client.get_asset_balance(balance_label)['free'])
+ balance = float(self.client.get_asset_balance(balance_label)["free"])
amount = spend_ratio * balance
if side == C.BUY:
@@ -404,16 +670,8 @@ def order(self, base, quote, side, spend_ratio=1, test=False):
params[quantity_label] = quantity
- params['side'] = side
- fx = (
- self.client.create_test_order
- if test else self.client.create_order
- )
+ params["side"] = side
+ fx = self.client.create_test_order if test else self.client.create_order
order = fx(**params)
return order
-
-# write script that gets most recent data at 9pm est
-# predicts using model
-# writes that back to predict.csv
-# write successful orders to binance.csv
diff --git a/hyperdrive/FileOps.py b/hyperdrive/FileOps.py
index 273f6046..0c4d43d1 100755
--- a/hyperdrive/FileOps.py
+++ b/hyperdrive/FileOps.py
@@ -1,22 +1,45 @@
-import os
+"""File read and write operations for CSV, JSON, and pickle files."""
+
import json
-import time
+import os
import pickle
+import time
from datetime import datetime
+from typing import Any
+
import pandas as pd
-from Storage import Store
-from Constants import TZ
-from TimeMachine import TimeTraveller
-# consider combining fileoperations into one class
+import polars as pl
+
+from .Constants import TZ
+from .Storage import Store
+from .TimeMachine import TimeTraveller
class FileReader:
- # file read operations
- def __init__(self):
+ """File reading operations with S3 synchronization.
+
+ Handles reading CSV, JSON, and pickle files with automatic
+ download from S3 when local files are stale.
+
+ Attributes:
+ store: Store instance for S3 operations.
+ traveller: TimeTraveller instance for date calculations.
+ """
+
+ def __init__(self) -> None:
+ """Initialize the FileReader with storage and time utilities."""
self.store = Store()
self.traveller = TimeTraveller()
- def should_be_updated(self, filename):
+ def should_be_updated(self, filename: str) -> bool:
+ """Check if a file should be updated from S3.
+
+ Args:
+ filename: Path to the file to check.
+
+ Returns:
+ True if file doesn't exist or is older than one day.
+ """
one_day = 60 * 60 * 24
now = datetime.fromtimestamp(time.time())
file_exists = os.path.exists(filename)
@@ -27,35 +50,91 @@ def should_be_updated(self, filename):
last_modified = delta.total_seconds()
return not file_exists or last_modified > one_day
- def load_json(self, filename):
- # loads json file as dictionary data
+ def load_json(self, filename: str) -> dict[str, Any]:
+ """Load a JSON file as a dictionary.
+
+ Args:
+ filename: Path to the JSON file.
+
+ Returns:
+ Dictionary containing the JSON data.
+ """
if self.should_be_updated(filename):
self.store.download_file(filename)
- with open(filename, 'r') as file:
+ with open(filename) as file:
return json.load(file)
- def load_csv(self, filename):
- # loads csv file as Dataframe
+ def load_csv(self, filename: str) -> pd.DataFrame:
+ """Load a CSV file as a DataFrame.
+
+ Uses polars internally for faster I/O, returns pandas
+ for backward compatibility.
+
+ Args:
+ filename: Path to the CSV file.
+
+ Returns:
+ DataFrame containing the CSV data.
+
+ Raises:
+ pd.errors.EmptyDataError: If the CSV file is empty.
+ FileNotFoundError: If the file doesn't exist.
+ """
try:
if self.should_be_updated(filename):
self.store.download_file(filename)
- df = pd.read_csv(filename).round(10)
- except pd.errors.EmptyDataError:
- print(f'{filename} is an empty csv file.')
- raise
+ # Use polars for fast CSV reading, convert to pandas for compatibility
+ df_pl = pl.read_csv(filename)
+ df = df_pl.to_pandas()
+ # Round numeric columns to avoid floating point precision issues
+ numeric_cols = df.select_dtypes(include=["float64", "float32"]).columns
+ df[numeric_cols] = df[numeric_cols].round(10)
+ except pl.exceptions.NoDataError:
+ print(f"{filename} is an empty csv file.")
+ raise pd.errors.EmptyDataError(
+ f"{filename} is an empty csv file."
+ ) from None
except FileNotFoundError:
- print(f'{filename} does not exist locally.')
+ print(f"{filename} does not exist locally.")
+ raise
+ except pd.errors.EmptyDataError:
raise
- except Exception:
+ except BaseException:
df = pd.DataFrame()
return df
- def check_update(self, filename, df):
- # given a csv filename and dataframe
- # return whether the csv needs to be updated
+ def check_update(self, filename: str, df: pd.DataFrame) -> bool:
+ """Check if a CSV file needs to be updated with new data.
+
+ Args:
+ filename: Path to the CSV file.
+ df: New DataFrame to compare against.
+
+ Returns:
+ True if the new DataFrame has at least as many rows as the file.
+ """
return len(df) >= len(self.load_csv(filename))
- def update_df(self, filename, new, column, save_fmt=None):
+ def update_df(
+ self,
+ filename: str,
+ new: pd.DataFrame,
+ column: str,
+ save_fmt: str | None = None,
+ ) -> pd.DataFrame:
+ """Merge new data with existing CSV data.
+
+ Uses polars internally for faster concat/dedup operations.
+
+ Args:
+ filename: Path to the CSV file.
+ new: New DataFrame to merge.
+ column: Column name to use for deduplication.
+ save_fmt: Optional date format for saving.
+
+ Returns:
+ Merged DataFrame with new entries taking precedence.
+ """
old = self.load_csv(filename)
if not old.empty:
old[column] = pd.to_datetime(old[column])
@@ -67,78 +146,162 @@ def update_df(self, filename, new, column, save_fmt=None):
new[column] = pd.to_datetime(new[column]).dt.strftime(save_fmt)
return new
- def check_file_exists(self, filename):
+ def check_file_exists(self, filename: str) -> bool:
+ """Check if a file exists both locally and in S3.
+
+ Args:
+ filename: Path to the file.
+
+ Returns:
+ True if the file exists in both locations.
+ """
return os.path.exists(filename) and self.store.key_exists(filename)
- def data_in_timeframe(self, df, col, timeframe='max'): # noqa , tolerance='0d'):
+ def data_in_timeframe(
+ self, df: pd.DataFrame, col: str, timeframe: str = "max"
+ ) -> pd.DataFrame:
+ """Filter DataFrame to data within a timeframe.
+
+ Uses polars internally for faster filtering.
+
+ Args:
+ df: DataFrame to filter.
+ col: Date column name.
+ timeframe: Timeframe string (e.g., '1y', '30d', 'max').
+
+ Returns:
+ Filtered DataFrame with dates within the timeframe.
+ """
if col not in df:
return df
delta = self.traveller.convert_delta(timeframe)
- # tol = self.traveller.convert_delta(tolerance)
df[col] = pd.to_datetime(df[col]).dt.tz_localize(TZ)
today = datetime.now(TZ)
- filtered = df[df[col].apply(
- lambda date: date.strftime('%Y-%m-%d')) >= pd.to_datetime(
- today - delta).strftime('%Y-%m-%d')].copy(deep=True)
- # if filtered.empty:
- # filtered = df[df[col] > pd.to_datetime(today - (delta + tol))]
+ filtered = df[
+ df[col].apply(lambda date: date.strftime("%Y-%m-%d"))
+ >= pd.to_datetime(today - delta).strftime("%Y-%m-%d")
+ ].copy(deep=True)
filtered[col] = filtered[col].dt.tz_localize(None)
return filtered
- def load_pickle(self, filename):
+ def load_pickle(self, filename: str) -> Any:
+ """Load a pickled object from file.
+
+ Args:
+ filename: Path to the pickle file.
+
+ Returns:
+ The unpickled object.
+ """
if self.should_be_updated(filename):
self.store.download_file(filename)
- with open(filename, 'rb') as file:
+ with open(filename, "rb") as file:
return pickle.load(file)
class FileWriter:
- # file write operations
- def __init__(self):
+ """File writing operations with S3 synchronization.
+
+ Handles writing CSV, JSON, and pickle files with automatic
+ upload to S3.
+
+ Attributes:
+ store: Store instance for S3 operations.
+ """
+
+ def __init__(self) -> None:
+ """Initialize the FileWriter with storage utilities."""
self.store = Store()
- def save_json(self, filename, data):
- # saves data as json file with provided filename
+ def save_json(self, filename: str, data: dict[str, Any] | list[Any]) -> bool:
+ """Save data as a JSON file and upload to S3.
+
+ Args:
+ filename: Path to save the JSON file.
+ data: Dictionary or list data to save.
+
+ Returns:
+ True on success.
+ """
self.store.finder.make_path(filename)
- with open(filename, 'w') as file:
+ with open(filename, "w") as file:
json.dump(data, file, indent=4)
self.store.upload_file(filename)
return True
- def save_csv(self, filename, data):
- # saves df as csv file with provided filename
- if data.empty:
- return False
- else:
+ def save_csv(self, filename: str, data: pd.DataFrame | pl.DataFrame) -> bool:
+ """Save a DataFrame as a CSV file and upload to S3.
+
+ Accepts both pandas and polars DataFrames.
+
+ Args:
+ filename: Path to save the CSV file.
+ data: DataFrame to save (pandas or polars).
+
+ Returns:
+ True on success, False if DataFrame is empty.
+ """
+ # Handle polars DataFrame
+ if isinstance(data, pl.DataFrame):
+ if data.is_empty():
+ return False
self.store.finder.make_path(filename)
- with open(filename, 'w') as f:
- data.to_csv(f, index=False)
+ data.write_csv(filename)
self.store.upload_file(filename)
return True
- def update_csv(self, filename, df):
- # update csv if needed
+ # Handle pandas DataFrame
+ if data.empty:
+ return False
+ self.store.finder.make_path(filename)
+ # Convert to polars for faster CSV writing
+ df_pl = pl.from_pandas(data)
+ df_pl.write_csv(filename)
+ self.store.upload_file(filename)
+ return True
+
+ def update_csv(self, filename: str, df: pd.DataFrame) -> None:
+ """Update a CSV file if the new data has more rows.
+
+ Args:
+ filename: Path to the CSV file.
+ df: New DataFrame to save.
+ """
if FileReader().check_update(filename, df):
self.save_csv(filename, df)
- def remove_files(self, filenames):
- [os.remove(file) for file in filenames]
+ def remove_files(self, filenames: list[str]) -> None:
+ """Remove files locally and from S3.
+
+ Args:
+ filenames: List of file paths to remove.
+ """
+ for file in filenames:
+ os.remove(file)
self.store.delete_objects(filenames)
- def rename_file(self, old_name, new_name):
+ def rename_file(self, old_name: str, new_name: str) -> None:
+ """Rename a file locally and in S3.
+
+ Args:
+ old_name: Current file path.
+ new_name: New file path.
+ """
os.rename(old_name, new_name)
self.store.rename_key(old_name, new_name)
- def save_pickle(self, filename, data):
+ def save_pickle(self, filename: str, data: Any) -> bool:
+ """Save an object as a pickle file and upload to S3.
+
+ Args:
+ filename: Path to save the pickle file.
+ data: Object to pickle and save.
+
+ Returns:
+ True on success.
+ """
self.store.finder.make_path(filename)
- with open(filename, 'wb') as file:
+ with open(filename, "wb") as file:
pickle.dump(data, file)
self.store.upload_file(filename)
return True
-
-
-# add function that takes in a Constants directory, old to new column mapping
-# and renames the cols using df.rename(columns=mapping) for all csvs in the dir
-
-
-# FileReader().get_all_paths('.')
diff --git a/hyperdrive/History.py b/hyperdrive/History.py
index c35fd811..c5cff44b 100755
--- a/hyperdrive/History.py
+++ b/hyperdrive/History.py
@@ -1,75 +1,123 @@
+"""Backtesting and machine learning utilities for trading strategies."""
+
+from collections.abc import Callable
+from typing import Any, cast
+
import numpy as np
import pandas as pd
import vectorbt as vbt
+from imblearn.over_sampling import SMOTE
from scipy.signal import argrelextrema
-from sklearn.model_selection import train_test_split
from sklearn.decomposition import PCA
-from sklearn.preprocessing import StandardScaler
-from sklearn.neural_network import MLPClassifier
-from sklearn.neighbors import KNeighborsClassifier
-from sklearn.svm import SVC
+from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
+from sklearn.ensemble import AdaBoostClassifier, RandomForestClassifier
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF
-from sklearn.tree import DecisionTreeClassifier
-from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
-from sklearn.naive_bayes import GaussianNB
-from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
from sklearn.metrics import classification_report
-from imblearn.over_sampling import SMOTE
-from Calculus import Calculator
-import Constants as C
+from sklearn.model_selection import train_test_split
+from sklearn.naive_bayes import GaussianNB
+from sklearn.neighbors import KNeighborsClassifier
+from sklearn.neural_network import MLPClassifier
+from sklearn.preprocessing import StandardScaler
+from sklearn.svm import SVC
+from sklearn.tree import DecisionTreeClassifier
+
+from . import Constants as C
+from .Calculus import Calculator
class Historian:
- def __init__(self):
+ """Backtesting and ML utility for trading strategy development.
+
+ Provides methods for portfolio creation, signal generation,
+ feature preprocessing, and classifier evaluation.
+
+ Attributes:
+ calc: Calculator instance for mathematical operations.
+ """
+
+ def __init__(self) -> None:
+ """Initialize the Historian with a Calculator instance."""
self.calc = Calculator()
- # add fx to perform calculations on columns
- # takes calc.fx, df, and column names as args, fx args
+ def from_holding(self, close: pd.Series, init_cash: float = 1000) -> vbt.Portfolio:
+ """Create a portfolio based on buy-and-hold strategy.
- def from_holding(self, close, init_cash=1000):
- # returns a portfolio based on buy and hold strategy
- portfolio = vbt.Portfolio.from_holding(
- close, init_cash=init_cash, freq='D')
+ Args:
+ close: Series of closing prices.
+ init_cash: Initial cash amount.
+
+ Returns:
+ A vectorbt Portfolio object.
+ """
+ portfolio = vbt.Portfolio.from_holding(close, init_cash=init_cash, freq="D")
return portfolio
- def from_signals(self, close, signals, init_cash=1000, fee=0):
- # returns a portfolio based on signals
+ def from_signals(
+ self,
+ close: pd.Series,
+ signals: pd.Series,
+ init_cash: float = 1000,
+ fee: float = 0,
+ ) -> vbt.Portfolio:
+ """Create a portfolio based on trading signals.
+
+ Args:
+ close: Series of closing prices.
+ signals: Boolean series indicating buy signals.
+ init_cash: Initial cash amount.
+ fee: Trading fee as a decimal.
+
+ Returns:
+ A vectorbt Portfolio object.
+ """
portfolio = vbt.Portfolio.from_signals(
- close, signals, ~signals, init_cash=init_cash, freq='D', fees=fee
+ close, signals, ~signals, init_cash=init_cash, freq="D", fees=fee
)
return portfolio
def optimize_portfolio(
- self,
- close: pd.DataFrame,
- indicator: callable,
- top_n: int,
- period: str,
- init_cash: float,
- **kwargs: dict[str, any]
+ self,
+ close: pd.DataFrame,
+ indicator: Callable[..., pd.Series],
+ top_n: int,
+ period: str,
+ init_cash: float,
+ **kwargs: Any,
) -> vbt.Portfolio:
- if C.CLOSE in close.columns:
- close = close.set_index(C.CLOSE)
+ """Optimize a portfolio by rotating into top-ranked assets.
+
+ Args:
+ close: DataFrame of closing prices with symbols as columns.
+ indicator: Function to compute ranking indicator.
+ top_n: Number of top assets to hold.
+ period: Rebalancing period attribute (e.g., 'month', 'week').
+ init_cash: Initial cash amount.
+ **kwargs: Additional arguments passed to indicator function.
+
+ Returns:
+ A vectorbt Portfolio object.
+ """
+ if C.TIME in close.columns:
+ close = close.set_index(C.TIME)
signals = close.apply(indicator, **kwargs)
close = close.dropna()
- positions = pd.DataFrame(
- 0, index=close.index, columns=close.columns)
- holdings = {"cash": init_cash}
+ positions = pd.DataFrame(0, index=close.index, columns=close.columns)
+ holdings: dict[str, float] = {"cash": init_cash}
prev_period = None
- prev_symbols = set()
+ prev_symbols: set[str] = set()
for day in close.index:
curr_period = getattr(day, period)
# if is first of the period
if prev_period != curr_period:
# Rank symbols by indicator and select top_n
+ # filter signals by include/exclude param for each period
top_symbols = set(signals.loc[day].nlargest(top_n).index)
- minus, plus = self.calc.get_difference(
- prev_symbols, top_symbols)
+ minus, plus = self.calc.get_difference(prev_symbols, top_symbols)
# Sell old positions for the top symbols
for symbol in minus:
size = holdings[symbol]
- positions.loc[day, symbol] = - size
+ positions.loc[day, symbol] = -size
holdings["cash"] += close.loc[day][symbol] * size
del holdings[symbol]
# Buy new positions for the top symbols
@@ -88,37 +136,59 @@ def optimize_portfolio(
# Convert to orders format
portfolio = vbt.Portfolio.from_orders(
- close=close,
- size=positions,
- freq='D',
- init_cash=0,
- group_by=True
+ close=close, size=positions, freq="D", init_cash=0, group_by=True
)
return portfolio
def from_orders(
- self,
- close: pd.DataFrame,
- size: pd.DataFrame,
- fee: float = 0
+ self, close: pd.DataFrame, size: pd.DataFrame, fee: float = 0
) -> vbt.Portfolio:
+ """Create a portfolio from order sizes.
+
+ Args:
+ close: DataFrame of closing prices.
+ size: DataFrame of order sizes.
+ fee: Trading fee as a decimal.
+
+ Returns:
+ A vectorbt Portfolio object.
+ """
portfolio = vbt.Portfolio.from_orders(
- close, size, freq='D', fees=fee, init_cash=0, group_by=True
+ close, size, freq="D", fees=fee, init_cash=0, group_by=True
)
return portfolio
- def fill(self, arr, method='ffill', type='bool'):
- # forward fills or nearest fills an array
+ def fill(
+ self, arr: np.ndarray, method: str = "ffill", type: str = "bool"
+ ) -> np.ndarray:
+ """Fill missing values in an array.
+
+ Args:
+ arr: Array with missing values.
+ method: Fill method ('ffill' for forward fill).
+ type: Output dtype.
+
+ Returns:
+ Filled array.
+ """
df = pd.DataFrame(arr)
s = df.iloc[:, 0]
- if method == 'ffill':
- s = s.fillna(method='ffill')
+ if method == "ffill":
+ s = s.fillna(method="ffill")
s = pd.to_numeric(s)
- s = s.interpolate(method='nearest').astype(type)
+ s = s.interpolate(method="nearest").astype(type)
out = s.to_numpy().flatten()
return out
- def unfill(self, xs):
+ def unfill(self, xs: list[Any]) -> list[Any]:
+ """Remove consecutive duplicates from a list.
+
+ Args:
+ xs: Input list with potential duplicates.
+
+ Returns:
+ List with consecutive duplicates replaced by None.
+ """
if not len(xs):
return xs
curr = xs[0]
@@ -131,23 +201,40 @@ def unfill(self, xs):
new.append(None)
return new
- def get_optimal_signals(self, close, n=10, method='ffill'):
- # finds the optimal signals for an array of prices
- close = np.array(close)
- mins = argrelextrema(close, np.less_equal,
- order=n)[0]
- maxs = argrelextrema(close, np.greater_equal,
- order=n)[0]
+ def get_optimal_signals(
+ self, close: np.ndarray | pd.Series, n: int = 10, method: str = "ffill"
+ ) -> np.ndarray:
+ """Find optimal buy/sell signals based on local extrema.
- signals = np.empty_like(close, dtype='object')
+ Args:
+ close: Array of closing prices.
+ n: Order parameter for extrema detection.
+ method: Fill method for missing signals.
+
+ Returns:
+ Boolean array of buy signals.
+ """
+ close_arr = np.array(close)
+ mins = argrelextrema(close_arr, np.less_equal, order=n)[0]
+ maxs = argrelextrema(close_arr, np.greater_equal, order=n)[0]
+
+ signals = np.empty_like(close_arr, dtype="object")
signals[:] = np.nan
signals[mins] = True
signals[maxs] = False
return self.fill(signals, method=method)
- def generate_random(self, close, num=10**4):
- # generate random strategies
+ def generate_random(self, close: pd.Series, num: int = 10**4) -> list[pd.Series]:
+ """Generate random trading strategies and return top performers.
+
+ Args:
+ close: Series of closing prices.
+ num: Number of random strategies to generate.
+
+ Returns:
+ List of signal series for top-performing strategies.
+ """
good_signals = []
portfolios = []
sortinos = []
@@ -157,12 +244,14 @@ def generate_random(self, close, num=10**4):
prob = self.get_optimal_signals(close).mean()
for _ in range(num_strats):
- signals = pd.DataFrame.vbt.signals.generate_random(
- (len(close), 1), prob=prob)[0]
+ signals = pd.DataFrame.vbt.signals.generate_random( # type: ignore[attr-defined]
+ (len(close), 1), prob=prob
+ )[0]
portfolio = vbt.Portfolio.from_signals(
- close, signals, ~signals, init_cash=1000, freq='D')
- sortinos.append(portfolio.sortino_ratio())
- calmars.append(portfolio.calmar_ratio())
+ close, signals, ~signals, init_cash=1000, freq="D"
+ )
+ sortinos.append(portfolio.sortino_ratio()) # type: ignore[attr-defined]
+ calmars.append(portfolio.calmar_ratio()) # type: ignore[attr-defined]
good_signals.append(signals)
portfolios.append(portfolio)
@@ -170,51 +259,124 @@ def generate_random(self, close, num=10**4):
top_c = set(np.argpartition(calmars, -top_n)[-top_n:])
top_idxs = top_s.intersection(top_c)
- portfolios = [portfolio for idx, portfolio in enumerate(
- portfolios) if idx in top_idxs]
- good_signals = [signal for idx, signal in enumerate(
- good_signals) if idx in top_idxs]
+ portfolios = [
+ portfolio for idx, portfolio in enumerate(portfolios) if idx in top_idxs
+ ]
+ good_signals = [
+ signal for idx, signal in enumerate(good_signals) if idx in top_idxs
+ ]
return good_signals
- def oversample(self, X_train, y_train):
+ def oversample(
+ self, X_train: np.ndarray, y_train: np.ndarray
+ ) -> tuple[np.ndarray, np.ndarray]:
+ """Oversample minority class using SMOTE.
+
+ Args:
+ X_train: Training features.
+ y_train: Training labels.
+
+ Returns:
+ Tuple of resampled features and labels.
+ """
sm = SMOTE()
X_res, y_res = sm.fit_resample(X_train, y_train)
return X_res, y_res
- def preprocess(self, X, y, num_pca=2):
+ def preprocess(
+ self, X: np.ndarray, y: np.ndarray, num_pca: int = 2
+ ) -> tuple[
+ np.ndarray,
+ np.ndarray,
+ np.ndarray,
+ np.ndarray,
+ np.ndarray,
+ np.ndarray,
+ StandardScaler,
+ PCA | None,
+ StandardScaler,
+ PCA | None,
+ ]:
+ """Preprocess data with train/test split, scaling, and PCA.
+
+ Args:
+ X: Feature matrix.
+ y: Target labels.
+ num_pca: Number of PCA components (0 to skip PCA).
+
+ Returns:
+ Tuple of processed data and fitted transformers.
+ """
df = pd.DataFrame(X)
- df['y'] = y
+ df["y"] = y
df = df.dropna()
- y = df['y'].to_numpy()
- X = df.drop('y', axis=1).to_numpy()
- X_train, X_test, y_train, y_test = \
- train_test_split(X, y, test_size=.2)
+ y = df["y"].to_numpy()
+ X = df.drop("y", axis=1).to_numpy()
+ X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
X_train, y_train = self.oversample(X_train, y_train)
- X_train, X_test, scaler = self.standardize(X_train, X_test)
+ X_train, X_test, scaler = cast(
+ tuple[np.ndarray, np.ndarray, StandardScaler],
+ self.standardize(X_train, X_test),
+ )
if num_pca:
- X_train, X_test, pca = self.pca(X_train, num_pca, X_test)
+ X_train, X_test, pca = cast(
+ tuple[np.ndarray, np.ndarray, PCA],
+ self.pca(X_train, num_pca, X_test),
+ )
else:
pca = None
- X, full_scaler = self.standardize(X)
+ X, full_scaler = cast(
+ tuple[np.ndarray, StandardScaler],
+ self.standardize(X),
+ )
if num_pca:
- X, full_pca = self.pca(X, num_pca)
+ X, full_pca = cast(tuple[np.ndarray, PCA], self.pca(X, num_pca))
else:
full_pca = None
return (
- X_train, X_test, y_train, y_test,
- X, y, scaler, pca, full_scaler, full_pca
+ X_train,
+ X_test,
+ y_train,
+ y_test,
+ X,
+ y,
+ scaler,
+ pca,
+ full_scaler,
+ full_pca,
)
- def undersample(self, X, y, n=2):
- # undersample, split train / test data, and standardize
+ def undersample(
+ self, X: np.ndarray, y: np.ndarray, n: int = 2
+ ) -> tuple[
+ np.ndarray,
+ np.ndarray,
+ np.ndarray,
+ np.ndarray,
+ np.ndarray,
+ np.ndarray,
+ StandardScaler,
+ PCA,
+ StandardScaler,
+ PCA,
+ ]:
+ """Undersample majority class and preprocess data.
+
+ Args:
+ X: Feature matrix.
+ y: Target labels.
+ n: Number of PCA components.
+
+ Returns:
+ Tuple of processed data and fitted transformers.
+ """
df = pd.DataFrame(X)
- df['y'] = y
+ df["y"] = y
df = df.dropna()
- y = df['y'].to_numpy()
- X = df.drop('y', axis=1).to_numpy()
- X_train, X_test, y_train, y_test = \
- train_test_split(X, y, test_size=.2)
+ y = df["y"].to_numpy()
+ X = df.drop("y", axis=1).to_numpy()
+ X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
train_true = 0
train_false = 0
@@ -234,17 +396,45 @@ def undersample(self, X, y, n=2):
X_train = np.array(X_train_new)
y_train = np.array(y_train_new)
- X_train, X_test, scaler = self.standardize(X_train, X_test)
- X_train, X_test, pca = self.pca(X_train, n, X_test)
- X, full_scaler = self.standardize(X)
- X, full_pca = self.pca(X, n)
+ X_train, X_test, scaler = cast(
+ tuple[np.ndarray, np.ndarray, StandardScaler],
+ self.standardize(X_train, X_test),
+ )
+ X_train, X_test, pca = cast(
+ tuple[np.ndarray, np.ndarray, PCA],
+ self.pca(X_train, n, X_test),
+ )
+ X, full_scaler = cast(tuple[np.ndarray, StandardScaler], self.standardize(X))
+ X, full_pca = cast(tuple[np.ndarray, PCA], self.pca(X, n))
return (
- X_train, X_test, y_train, y_test,
- X, y, scaler, pca, full_scaler, full_pca
+ X_train,
+ X_test,
+ y_train,
+ y_test,
+ X,
+ y,
+ scaler,
+ pca,
+ full_scaler,
+ full_pca,
)
- def standardize(self, X_train, X_test=None):
+ def standardize(
+ self, X_train: np.ndarray, X_test: np.ndarray | None = None
+ ) -> (
+ tuple[np.ndarray, StandardScaler]
+ | tuple[np.ndarray, np.ndarray, StandardScaler]
+ ):
+ """Standardize features using StandardScaler.
+
+ Args:
+ X_train: Training features to fit and transform.
+ X_test: Optional test features to transform.
+
+ Returns:
+ Transformed features and fitted scaler.
+ """
scaler = StandardScaler().fit(X_train)
X_train = scaler.transform(X_train)
if isinstance(X_test, np.ndarray):
@@ -252,7 +442,19 @@ def standardize(self, X_train, X_test=None):
return X_train, X_test, scaler
return X_train, scaler
- def pca(self, X_train, n, X_test=None):
+ def pca(
+ self, X_train: np.ndarray, n: int, X_test: np.ndarray | None = None
+ ) -> tuple[np.ndarray, PCA] | tuple[np.ndarray, np.ndarray, PCA]:
+ """Apply PCA dimensionality reduction.
+
+ Args:
+ X_train: Training features to fit and transform.
+ n: Number of components.
+ X_test: Optional test features to transform.
+
+ Returns:
+ Transformed features and fitted PCA object.
+ """
num_features = X_train.shape[1]
n = n if n <= num_features else num_features
pca = PCA(n_components=n).fit(X_train)
@@ -260,41 +462,72 @@ def pca(self, X_train, n, X_test=None):
if isinstance(X_test, np.ndarray):
X_test = pca.transform(X_test)
var = pca.explained_variance_ratio_.sum() * 100
- print(f'Explained variance (X_train): {round(var, 2)}%')
+ print(f"Explained variance (X_train): {round(var, 2)}%")
return X_train, X_test, pca
var = pca.explained_variance_ratio_.sum() * 100
- print(f'Explained variance (X): {round(var, 2)}%')
+ print(f"Explained variance (X): {round(var, 2)}%")
return X_train, pca
- def run_classifiers(self, X_train, X_test, y_train, y_test):
+ def run_classifiers(
+ self,
+ X_train: np.ndarray,
+ X_test: np.ndarray,
+ y_train: np.ndarray,
+ y_test: np.ndarray,
+ ) -> list[tuple[str, dict[str, Any]]]:
+ """Evaluate multiple classifiers on the data.
+
+ Args:
+ X_train: Training features.
+ X_test: Test features.
+ y_train: Training labels.
+ y_test: Test labels.
+
+ Returns:
+ Sorted list of (name, results) tuples for non-overfitting classifiers.
+ """
names = [
- "Nearest Neighbors", "Linear SVM", "RBF SVM", "Gaussian Process",
- "Decision Tree", "Random Forest", "Neural Net", "AdaBoost",
- "Naive Bayes", "QDA"]
+ "Nearest Neighbors",
+ "Linear SVM",
+ "RBF SVM",
+ "Gaussian Process",
+ "Decision Tree",
+ "Random Forest",
+ "Neural Net",
+ "AdaBoost",
+ "Naive Bayes",
+ "QDA",
+ ]
classifiers = [
KNeighborsClassifier(3),
SVC(kernel="linear", C=0.025),
SVC(gamma=2, C=1),
GaussianProcessClassifier(1.0 * RBF(1.0)),
DecisionTreeClassifier(max_depth=5),
- RandomForestClassifier(
- max_depth=5, n_estimators=10, max_features=1),
+ RandomForestClassifier(max_depth=5, n_estimators=10, max_features=1),
MLPClassifier(alpha=1, max_iter=1000),
AdaBoostClassifier(),
GaussianNB(),
- QuadraticDiscriminantAnalysis()]
+ QuadraticDiscriminantAnalysis(),
+ ]
- clfs = {}
+ clfs: dict[str, dict[str, Any]] = {}
- for name, clf in zip(names, classifiers):
+ for name, clf in zip(names, classifiers, strict=True):
clf.fit(X_train, y_train)
score = clf.score(X_test, y_test)
report = classification_report(
- y_test, clf.predict(X_test), output_dict=True)
+ y_test, clf.predict(X_test), output_dict=True
+ )
ratio = clf.score(X_train, y_train) / score
if ratio < 1.15:
- clfs[name] = {'score': score, 'report': report,
- 'ratio': ratio, 'clf': clf}
- clfs = sorted(clfs.items(), reverse=True,
- key=lambda clf: clf[1]['score'])
- return clfs
+ clfs[name] = {
+ "score": score,
+ "report": report,
+ "ratio": ratio,
+ "clf": clf,
+ }
+ clfs_sorted = sorted(
+ clfs.items(), reverse=True, key=lambda clf: clf[1]["score"]
+ )
+ return clfs_sorted
diff --git a/hyperdrive/Precognition.py b/hyperdrive/Precognition.py
index c6ec2835..f3a38a07 100644
--- a/hyperdrive/Precognition.py
+++ b/hyperdrive/Precognition.py
@@ -1,79 +1,163 @@
+"""Machine learning prediction utilities using AutoGluon."""
+
+from typing import Any
+
import numpy as np
-from sklearn.decomposition import PCA
-from autogluon.tabular import TabularDataset, TabularPredictor
import pandas as pd
-from FileOps import FileReader, FileWriter
-from Calculus import Calculator
-import Constants as C
+from autogluon.tabular import TabularDataset, TabularPredictor
+from sklearn.decomposition import PCA
+
+from . import Constants as C
+from .Calculus import Calculator
+from .FileOps import FileReader, FileWriter
class Oracle:
- def __init__(self):
+ """Machine learning prediction and visualization engine.
+
+ Provides methods for loading models, making predictions,
+ and visualizing decision boundaries using PCA.
+
+ Attributes:
+ reader: FileReader instance for loading files.
+ writer: FileWriter instance for saving files.
+ calc: Calculator instance for mathematical operations.
+ """
+
+ def __init__(self) -> None:
+ """Initialize the Oracle with file handlers and calculator."""
self.reader = FileReader()
self.writer = FileWriter()
self.calc = Calculator()
- def get_filename(self, name, ext='pkl'):
- return f'models/latest/{name}.{ext}'
+ def get_filename(self, name: str, ext: str = "pkl") -> str:
+ """Generate a model file path.
+
+ Args:
+ name: Base name of the model file.
+ ext: File extension.
- def load_metadata(self):
- filename = self.get_filename('metadata', 'json')
+ Returns:
+ Full path to the model file.
+ """
+ return f"models/latest/{name}.{ext}"
+
+ def load_metadata(self) -> dict[str, Any]:
+ """Load model metadata from JSON.
+
+ Returns:
+ Dictionary containing model metadata.
+ """
+ filename = self.get_filename("metadata", "json")
return self.reader.load_json(filename)
- def load_model_pickle(self, name):
+ def load_model_pickle(self, name: str) -> Any:
+ """Load a pickled model.
+
+ Args:
+ name: Name of the model file (without extension).
+
+ Returns:
+ The unpickled model object.
+ """
filename = self.get_filename(name)
return self.reader.load_pickle(filename)
- def save_model_pickle(self, name, data):
+ def save_model_pickle(self, name: str, data: Any) -> bool | None:
+ """Save a model as pickle.
+
+ Args:
+ name: Name of the model file (without extension).
+ data: Model object to save.
+
+ Returns:
+ True if saved successfully, None otherwise.
+ """
filename = self.get_filename(name)
return self.writer.save_pickle(filename, data)
- def predict(self, data):
- model_path = 'models/latest/autogluon'
+ def predict(self, data: pd.DataFrame | TabularDataset) -> pd.Series | np.ndarray:
+ """Make predictions using the AutoGluon model.
+
+ Args:
+ data: Input data for prediction.
+
+ Returns:
+ Series or array of predictions.
+ """
+ model_path = "models/latest/autogluon"
self.reader.store.download_dir(model_path)
model = TabularPredictor.load(model_path)
- if (
- isinstance(model, TabularPredictor) and not
- isinstance(data, TabularDataset)
- ):
- data = TabularDataset(data)
+ # AutoGluon predict accepts DataFrame, use the DataFrame directly
+ if isinstance(data, TabularDataset):
+ return model.predict(pd.DataFrame(data))
return model.predict(data)
- def visualize(self, X, y, dimensions, refinement, increase_percent=0):
- # # recommended in the range [4, 100]
+ def visualize(
+ self,
+ X: np.ndarray,
+ y: np.ndarray,
+ dimensions: int,
+ refinement: int,
+ increase_percent: float = 0,
+ ) -> tuple[
+ list[dict[str, list[Any]]], list[float], float, list[np.ndarray], np.ndarray
+ ]:
+ """Visualize decision boundaries using PCA reduction.
+
+ Args:
+ X: Feature matrix.
+ y: Target labels.
+ dimensions: Number of PCA dimensions (recommended 2-3).
+ refinement: Grid resolution (recommended 4-100).
+ increase_percent: Percentage to expand the visualization bounds.
+
+ Returns:
+ Tuple containing:
+ - actual: List of dicts with buy/sell coordinates per component
+ - centroid: Center point of the data
+ - radius: Radius of the visualization space
+ - flattened: Grid coordinates for prediction
+ - preds: Predicted labels for grid points
+ """
+ # recommended in the range [4, 100]
num_points = refinement
reducer = PCA(n_components=dimensions)
X_transformed = reducer.fit_transform(X)
components = X_transformed.T
all_coords = np.concatenate(X_transformed)
# or method='mean'
- centroid = self.calc.find_centroid(X_transformed, method='extrema')
+ centroid = self.calc.find_centroid(X_transformed, method="extrema")
super_min = min(all_coords)
super_min -= abs(super_min) * increase_percent
super_max = max(all_coords)
super_max += abs(super_max) * increase_percent
radius = (super_max - super_min) / 2
- lins = [np.linspace(
- component - radius if dimensions > 2 else min(components[idx]),
- component + radius if dimensions > 2 else max(components[idx]),
- num_points
- ) for idx, component in enumerate(centroid)]
+ lins = [
+ np.linspace(
+ component - radius if dimensions > 2 else min(components[idx]),
+ component + radius if dimensions > 2 else max(components[idx]),
+ num_points,
+ )
+ for idx, component in enumerate(centroid)
+ ]
unflattened = np.meshgrid(*lins)
flattened = [arr.flatten() for arr in unflattened]
reduced = np.array(flattened).T
unreduced = reducer.inverse_transform(reduced)
metadata = self.load_metadata()
- features = metadata['features']
- data = pd.DataFrame(unreduced, columns=features[:unreduced.shape[1]])
- preds = self.predict(data).astype(int).to_numpy()
+ features = metadata["features"]
+ data = pd.DataFrame(unreduced, columns=features[: unreduced.shape[1]])
+ predictions = self.predict(data)
+ if isinstance(predictions, pd.Series):
+ preds = predictions.astype(int).to_numpy()
+ else:
+ preds = predictions.astype(int)
actual = [
{
- C.BUY: [
- datum for idx, datum in enumerate(component) if y[idx]
- ],
- C.SELL: [
- datum for idx, datum in enumerate(component) if not y[idx]
- ]
- } for component in components
+ C.BUY: [datum for idx, datum in enumerate(component) if y[idx]],
+ C.SELL: [datum for idx, datum in enumerate(component) if not y[idx]],
+ }
+ for component in components
]
return actual, centroid, radius, flattened, preds
diff --git a/hyperdrive/Storage.py b/hyperdrive/Storage.py
index 14abb78a..4eb2ea72 100755
--- a/hyperdrive/Storage.py
+++ b/hyperdrive/Storage.py
@@ -1,52 +1,113 @@
+"""AWS S3 storage utilities for file operations."""
+
import os
-from datetime import datetime, timedelta
+from datetime import UTC, datetime, timedelta
+from multiprocessing import Pool
+from typing import Any
+
import boto3
from botocore.exceptions import ClientError
-from dotenv import load_dotenv, find_dotenv
-from multiprocessing import Pool
-from Constants import PathFinder
-import Constants as C
-# from typing import Optional
+from dotenv import find_dotenv, load_dotenv
+
+from . import Constants as C
+from .Constants import PathFinder
class Store:
- def __init__(self):
- load_dotenv(find_dotenv('config.env'))
+ """AWS S3 storage client for file operations.
+
+ Provides methods for uploading, downloading, copying, and deleting
+ files and directories in S3 buckets.
+
+ Attributes:
+ bucket_name: Name of the S3 bucket to use.
+ finder: PathFinder instance for path operations.
+ """
+
+ def __init__(self) -> None:
+ """Initialize the Store with bucket configuration from environment."""
+ load_dotenv(find_dotenv("config.env"))
self.bucket_name = self.get_bucket_name()
self.finder = PathFinder()
- def get_bucket_name(self):
- return os.environ.get(
- 'S3_BUCKET') if not C.DEV else os.environ.get('S3_DEV_BUCKET')
+ def get_bucket_name(self) -> str:
+ """Get the S3 bucket name based on environment.
+
+ Returns:
+ The production or dev bucket name depending on C.DEV setting.
+ """
+ bucket = (
+ os.environ.get("S3_BUCKET")
+ if not C.DEV
+ else os.environ.get("S3_DEV_BUCKET")
+ )
+ return bucket or ""
+
+ def get_bucket(self) -> Any:
+ """Get the S3 bucket resource.
- def get_bucket(self):
- s3 = boto3.resource('s3')
+ Returns:
+ A boto3 S3 Bucket resource object.
+ """
+ s3 = boto3.resource("s3")
bucket = s3.Bucket(self.bucket_name)
return bucket
- def upload_file(self, path):
+ def upload_file(self, path: str) -> None:
+ """Upload a local file to S3.
+
+ Args:
+ path: Local file path to upload.
+ """
bucket = self.get_bucket()
- key = path.replace('\\', '/')
+ key = path.replace("\\", "/")
bucket.upload_file(path, key)
- def upload_dir(self, **kwargs):
+ def upload_dir(self, **kwargs: Any) -> None:
+ """Upload all files in a directory to S3.
+
+ Args:
+ **kwargs: Arguments passed to PathFinder.get_all_paths().
+ """
paths = self.finder.get_all_paths(**kwargs)
with Pool() as p:
p.map(self.upload_file, paths)
def delete_objects(self, keys: list[str]) -> None:
+ """Delete multiple objects from S3.
+
+ Args:
+ keys: List of S3 keys to delete.
+ """
if keys:
- objects = [{'Key': key.replace('\\', '/')} for key in keys]
+ objects = [{"Key": key.replace("\\", "/")} for key in keys]
bucket = self.get_bucket()
- bucket.delete_objects(Delete={'Objects': objects})
+ bucket.delete_objects(Delete={"Objects": objects})
- def get_keys(self, filter: str = '') -> list[str]:
+ def get_keys(self, filter: str = "") -> list[str]:
+ """List all keys in the bucket matching a prefix.
+
+ Args:
+ filter: Prefix to filter keys by.
+
+ Returns:
+ List of matching S3 keys.
+ """
bucket = self.get_bucket()
keys = [obj.key for obj in bucket.objects.filter(Prefix=filter)]
return keys
- def key_exists(self, key: str, download=False) -> bool:
- key = key.replace('\\', '/')
+ def key_exists(self, key: str, download: bool = False) -> bool:
+ """Check if a key exists in S3.
+
+ Args:
+ key: S3 key to check.
+ download: If True, download the file to verify existence.
+
+ Returns:
+ True if the key exists, False otherwise.
+ """
+ key = key.replace("\\", "/")
try:
if download:
self.download_file(key)
@@ -59,47 +120,84 @@ def key_exists(self, key: str, download=False) -> bool:
return True
def download_file(self, key: str) -> None:
+ """Download a file from S3.
+
+ Args:
+ key: S3 key to download.
+
+ Raises:
+ ClientError: If the key does not exist in S3.
+ """
try:
self.finder.make_path(key)
- with open(key, 'wb') as file:
+ with open(key, "wb") as file:
bucket = self.get_bucket()
- s3_key = key.replace('\\', '/')
+ s3_key = key.replace("\\", "/")
bucket.download_fileobj(s3_key, file)
except ClientError as e:
- print(f'{key} does not exist in S3.')
+ print(f"{key} does not exist in S3.")
os.remove(key)
raise e
def download_dir(self, path: str) -> None:
+ """Download all files in an S3 directory.
+
+ Args:
+ path: S3 prefix path to download.
+ """
keys = self.get_keys(path)
with Pool() as p:
- p.starmap(self.key_exists, zip(keys, [True] * len(keys)))
+ p.starmap(self.key_exists, zip(keys, [True] * len(keys), strict=True))
def copy_object(self, src: str, dst: str) -> None:
- src = src.replace('\\', '/')
- dst = dst.replace('\\', '/')
+ """Copy an object within S3.
+
+ Args:
+ src: Source S3 key.
+ dst: Destination S3 key.
+ """
+ src = src.replace("\\", "/")
+ dst = dst.replace("\\", "/")
bucket = self.get_bucket()
- copy_source = {
- 'Bucket': self.bucket_name,
- 'Key': src
- }
+ copy_source = {"Bucket": self.bucket_name, "Key": src}
bucket.copy(copy_source, dst)
def rename_key(self, old_key: str, new_key: str) -> None:
- old_key = old_key.replace('\\', '/')
- new_key = new_key.replace('\\', '/')
+ """Rename an S3 object by copying and deleting.
+
+ Args:
+ old_key: Current S3 key.
+ new_key: New S3 key.
+ """
+ old_key = old_key.replace("\\", "/")
+ new_key = new_key.replace("\\", "/")
self.copy_object(old_key, new_key)
self.delete_objects([old_key])
def last_modified(self, key: str) -> datetime:
- key = key.replace('\\', '/')
+ """Get the last modified time of an S3 object.
+
+ Args:
+ key: S3 key to check.
+
+ Returns:
+ The last modified datetime (timezone-aware UTC).
+ """
+ key = key.replace("\\", "/")
bucket = self.get_bucket()
obj = bucket.Object(key)
- then = obj.last_modified.replace(tzinfo=None)
- return then
+ return obj.last_modified # S3 returns timezone-aware UTC datetime
def modified_delta(self, key: str) -> timedelta:
- key = key.replace('\\', '/')
+ """Get time since an S3 object was last modified.
+
+ Args:
+ key: S3 key to check.
+
+ Returns:
+ Time elapsed since last modification.
+ """
+ key = key.replace("\\", "/")
then = self.last_modified(key)
- now = datetime.utcnow()
+ now = datetime.now(UTC)
return now - then
diff --git a/hyperdrive/TimeMachine.py b/hyperdrive/TimeMachine.py
index 29771eaa..25bc2237 100755
--- a/hyperdrive/TimeMachine.py
+++ b/hyperdrive/TimeMachine.py
@@ -1,22 +1,23 @@
-from time import sleep
-from typing import Union
+"""Time manipulation utilities for date calculations and scheduling."""
+
from datetime import datetime, timedelta, tzinfo
-from Constants import TZ, UTC, DATE_FMT, TIME_FMT, PRECISE_TIME_FMT
+from datetime import time as dt_time
+from time import sleep
+
+from .Constants import DATE_FMT, PRECISE_TIME_FMT, TIME_FMT, TZ, UTC
-FlexibleDate = Union[datetime, str]
+FlexibleDate = datetime | str
class TimeTraveller:
- """
- A class to handle time-related operations, such as calculating deltas,
- converting timeframes, and managing sleep intervals.
+ """Time manipulation utility for date calculations and scheduling.
+
+ Provides methods for calculating date differences, converting timeframes,
+ generating date ranges, and managing sleep intervals based on schedules.
"""
def get_delta(
- self,
- d1: FlexibleDate,
- d2: FlexibleDate = datetime.now(),
- format: str = DATE_FMT
+ self, d1: FlexibleDate, d2: FlexibleDate | None = None, format: str = DATE_FMT
) -> timedelta:
"""
Calculate the difference between two dates.
@@ -32,30 +33,29 @@ def get_delta(
Returns:
timedelta: The absolute difference between the two dates.
"""
- if isinstance(d1, str):
- d1 = datetime.strptime(d1, format)
- if isinstance(d2, str):
- d2 = datetime.strptime(d2, format)
+ d2_resolved: FlexibleDate = d2 if d2 is not None else datetime.now()
+ d1_dt = datetime.strptime(d1, format) if isinstance(d1, str) else d1
+ d2_dt = (
+ datetime.strptime(d2_resolved, format)
+ if isinstance(d2_resolved, str)
+ else d2_resolved
+ )
- return abs(d2 - d1)
+ return abs(d2_dt - d1_dt)
def convert_timeframe(self, d1: FlexibleDate, d2: FlexibleDate) -> str:
- """
- Convert two datetime objects
- to a string representation of the timeframe.
+ """Convert two datetime objects to a string representation of the timeframe.
Args:
- d1 (FlexibleDate):
- The first date, can be a datetime object or a string.
- d2 (FlexibleDate):
- The second date, can be a datetime object or a string.
+ d1: The first date, can be a datetime object or a string.
+ d2: The second date, can be a datetime object or a string.
Returns:
- str: A string representation of the timeframe in days.
+ A string representation of the timeframe in days.
"""
delta = self.get_delta(d1, d2)
days = delta.days
- return f'{days}d'
+ return f"{days}d"
def convert_delta(self, timeframe: str) -> timedelta:
"""
@@ -72,11 +72,11 @@ def convert_delta(self, timeframe: str) -> timedelta:
Raises:
ValueError: If the timeframe string is not in a supported format.
"""
- if timeframe == 'max':
+ if timeframe == "max":
return timedelta(days=36500)
- periods = {'y': 365, 'm': 30, 'w': 7, 'd': 1}
- period = 'y'
+ periods = {"y": 365, "m": 30, "w": 7, "d": 1}
+ period = "y"
idx = -1
for curr_period in periods:
@@ -86,8 +86,8 @@ def convert_delta(self, timeframe: str) -> timedelta:
break
if idx == -1:
- supported = ', '.join(list(periods))
- error_msg = f'Only certain suffixes ({supported}) are supported.'
+ supported = ", ".join(list(periods))
+ error_msg = f"Only certain suffixes ({supported}) are supported."
raise ValueError(error_msg)
num = int(timeframe[:idx])
@@ -97,9 +97,7 @@ def convert_delta(self, timeframe: str) -> timedelta:
return delta
def convert_dates(
- self,
- timeframe: str,
- format: str = DATE_FMT
+ self, timeframe: str, format: str = DATE_FMT
) -> tuple[FlexibleDate, FlexibleDate]:
"""
Convert a timeframe to a start and end date.
@@ -116,18 +114,20 @@ def convert_dates(
A tuple containing the start and end dates as strings.
"""
# if timeframe='max': timeframe = '25y'
- end = datetime.now(TZ) - self.convert_delta('1d')
- delta = self.convert_delta(timeframe) - self.convert_delta('1d')
+ end = datetime.now(TZ) - self.convert_delta("1d")
+ delta = self.convert_delta(timeframe) - self.convert_delta("1d")
start = end - delta
if format:
start = start.strftime(format)
end = end.strftime(format)
return start, end
- def dates_in_range(self, timeframe: str, format: str = DATE_FMT
- ) -> list[FlexibleDate]:
+ def dates_in_range(
+ self, timeframe: str, format: str = DATE_FMT
+ ) -> list[FlexibleDate]:
"""
Get a list of dates in the specified timeframe.
+
Args:
timeframe (str):
A string representing the timeframe,
@@ -138,44 +138,48 @@ def dates_in_range(self, timeframe: str, format: str = DATE_FMT
Returns:
list[FlexibleDate]: A list of dates in the specified timeframe.
"""
- start, end = self.convert_dates(timeframe, None)
- dates = [start + timedelta(days=x)
- for x in range(0, (end - start).days + 1)]
+ start, end = self.convert_dates(timeframe, "")
+ # When format is empty string, convert_dates returns datetime objects
+ if isinstance(start, str):
+ start = datetime.strptime(start, DATE_FMT)
+ if isinstance(end, str):
+ end = datetime.strptime(end, DATE_FMT)
+ dates_dt = [start + timedelta(days=x) for x in range(0, (end - start).days + 1)]
if format:
- dates = [date.strftime(format) for date in dates]
- return dates
+ return [date.strftime(format) for date in dates_dt]
+ return dates_dt
- def get_time(self, time: str) -> datetime.time:
+ def get_time(self, time_str: str) -> dt_time:
"""
Converts time string to a time object.
Args:
- time (str):
+ time_str (str):
A string representing the time, e.g., '14:30', '14:30:00'.
Returns:
- datetime.time: A time object representing the specified time.
+ dt_time: A time object representing the specified time.
"""
return datetime.strptime(
- time, TIME_FMT if len(time.split(':')) == 2 else PRECISE_TIME_FMT
+ time_str, TIME_FMT if len(time_str.split(":")) == 2 else PRECISE_TIME_FMT
).time()
- def combine_date_time(self, date: str, time: str) -> datetime:
+ def combine_date_time(self, date: str, time_str: str) -> datetime:
"""
Combines date and time into a datetime object.
Args:
date (str):
A string representing the date, e.g., '2025-01-01'.
- time (str):
+ time_str (str):
A string representing the time, e.g., '14:30', '14:30:00'.
Returns:
datetime: A datetime object combining the specified date and time.
"""
- date = datetime.strptime(date, DATE_FMT)
- time = self.get_time(time)
- return date.combine(date, time)
+ date_dt = datetime.strptime(date, DATE_FMT)
+ time_obj = self.get_time(time_str)
+ return datetime.combine(date_dt, time_obj)
def get_diff(self, t1: datetime, t2: datetime) -> float:
"""
diff --git a/hyperdrive/Transformer.py b/hyperdrive/Transformer.py
index 64d45be6..1e317ea6 100644
--- a/hyperdrive/Transformer.py
+++ b/hyperdrive/Transformer.py
@@ -1,30 +1,43 @@
+"""JSON encoders for numpy data types."""
+
import json
+from typing import Any
+
import numpy as np
class NumpyEncoder(json.JSONEncoder):
- """ Custom encoder for numpy data types """
+ """Custom JSON encoder for numpy data types.
+
+ Converts numpy types to native Python types for JSON serialization.
+ Handles integers, floats, complex numbers, arrays, booleans, and void types.
+ """
+
+ def default(self, o: Any) -> Any:
+ """Convert numpy types to JSON-serializable Python types.
- def default(self, obj):
- if isinstance(obj, (np.int_, np.intc, np.intp, np.int8,
- np.int16, np.int32, np.int64, np.uint8,
- np.uint16, np.uint32, np.uint64)):
+ Args:
+ o: Object to encode. If a numpy type, converts to Python equivalent.
- return int(obj)
+ Returns:
+ JSON-serializable Python object.
+ """
+ if isinstance(o, np.integer):
+ return int(o)
- elif isinstance(obj, (np.float_, np.float16, np.float32, np.float64)):
- return float(obj)
+ elif isinstance(o, np.floating):
+ return float(o)
- elif isinstance(obj, (np.complex_, np.complex64, np.complex128)):
- return {'real': float(obj.real), 'imag': float(obj.imag)}
+ elif isinstance(o, np.complexfloating):
+ return {"real": float(o.real), "imag": float(o.imag)}
- elif isinstance(obj, (np.ndarray,)):
- return obj.tolist()
+ elif isinstance(o, (np.ndarray,)):
+ return o.tolist()
- elif isinstance(obj, (np.bool_)):
- return bool(obj)
+ elif isinstance(o, (np.bool_)):
+ return bool(o)
- elif isinstance(obj, (np.void)):
+ elif isinstance(o, (np.void)):
return None
- return json.JSONEncoder.default(self, obj)
+ return json.JSONEncoder.default(self, o)
diff --git a/hyperdrive/Utils.py b/hyperdrive/Utils.py
index ad91a956..18ade9d2 100644
--- a/hyperdrive/Utils.py
+++ b/hyperdrive/Utils.py
@@ -1,31 +1,71 @@
+"""Utility classes for object manipulation and dev environment configuration."""
+
import os
-import Constants as C # noqa autopep8
+from typing import Any, TypeVar
+
+from . import Constants as C
+
+T = TypeVar("T")
class SwissArmyKnife:
+ """Multi-purpose utility class for object manipulation.
+
+ Provides methods for recursively replacing object attributes
+ and configuring objects for development environments.
+ """
+
+ def replace_attr(self, obj: T, find_key: str, replace_val: Any) -> T:
+ """Recursively replace an attribute value in an object.
- def replace_attr(self, obj, find_key, replace_val):
+ Searches for the specified attribute on the object and its nested
+ attributes, replacing all occurrences with the new value.
+
+ Args:
+ obj: The object to modify.
+ find_key: The attribute name to search for.
+ replace_val: The new value to set.
+
+ Returns:
+ The modified object with replaced attribute values.
+ """
try:
getattr(obj, find_key)
setattr(obj, find_key, replace_val)
return obj
except AttributeError:
- attrs = [attr for attr in dir(obj) if not (
- attr.startswith('__') and attr.endswith('__'))]
+ attrs = [
+ attr
+ for attr in dir(obj)
+ if not (attr.startswith("__") and attr.endswith("__"))
+ ]
for key in attrs:
try:
- setattr(obj, key, self.replace_attr(
- getattr(obj, key), find_key, replace_val))
+ setattr(
+ obj,
+ key,
+ self.replace_attr(getattr(obj, key), find_key, replace_val),
+ )
except AttributeError:
# This happens for read-only attributes
# like capitalize attr for a str obj
pass
return obj
- def use_dev(self, obj):
+ def use_dev(self, obj: T) -> T:
+ """Configure an object to use the development S3 bucket.
+
+ Replaces the bucket_name attribute with the dev bucket when not in CI.
+
+ Args:
+ obj: The object to configure for development.
+
+ Returns:
+ The object configured with dev bucket name (or unchanged in CI).
+ """
# or simply make DevStore class that has s3 dev bucket name
dev_obj = obj
if not C.CI:
- dev_bucket = os.environ['S3_DEV_BUCKET']
- dev_obj = self.replace_attr(obj, 'bucket_name', dev_bucket)
+ dev_bucket = os.environ["S3_DEV_BUCKET"]
+ dev_obj = self.replace_attr(obj, "bucket_name", dev_bucket)
return dev_obj
diff --git a/hyperdrive/Workflow.py b/hyperdrive/Workflow.py
index a50590ed..092623fc 100755
--- a/hyperdrive/Workflow.py
+++ b/hyperdrive/Workflow.py
@@ -1,49 +1,87 @@
+"""Workflow scheduling utilities for GitHub Actions coordination."""
+
import re
-from datetime import datetime, timedelta
-from DataSource import MarketData
-from Constants import POLY_FREE_DELAY, FEW, POLY_CRYPTO_SYMBOLS
+from datetime import UTC, datetime, timedelta
+
+from .Constants import FEW, POLY_CRYPTO_SYMBOLS, POLY_FREE_DELAY
+from .DataSource import MarketData
class Flow:
- def get_workflow_start_time(self, workflow_name):
- with open(f'.github/workflows/{workflow_name}.yml') as file:
+ """GitHub Actions workflow scheduling coordinator.
+
+ Provides methods to determine workflow timing and check if
+ workflows are currently running based on cron schedules.
+ """
+
+ def get_workflow_start_time(self, workflow_name: str) -> datetime:
+ """Parse the scheduled start time from a workflow file.
+
+ Args:
+ workflow_name: Name of the workflow file (without .yml extension).
+
+ Returns:
+ The datetime when the workflow is scheduled to start.
+
+ Raises:
+ AttributeError: If the workflow doesn't have a scheduled cron job.
+ """
+ with open(f".github/workflows/{workflow_name}.yml") as file:
workflow_content = file.read()
line_pattern = '- cron: "(.*)"'
- try:
- cron_line = re.search(line_pattern, workflow_content).group(1)
- except AttributeError:
+ match = re.search(line_pattern, workflow_content)
+ if match is None:
raise AttributeError(
- f"{workflow_name}.yml doesn't have a scheduled cron job")
+ f"{workflow_name}.yml doesn't have a scheduled cron job"
+ )
+ cron_line = match.group(1)
- now = datetime.utcnow()
+ now = datetime.now(UTC)
default_times = [now.minute, now.hour, now.day, now.month]
- times = [default_times[idx] if time ==
- '*' else int(time) for idx, time in enumerate(
- cron_line.split(' ')[:-1])]
+ times = [
+ default_times[idx] if time == "*" else int(time)
+ for idx, time in enumerate(cron_line.split(" ")[:-1])
+ ]
minute, hour, day, month = times
- return datetime(now.year, month, day, hour, minute)
+ # Return timezone-aware datetime in UTC for consistent comparisons
+ return datetime(now.year, month, day, hour, minute, tzinfo=UTC)
+
+ def is_workflow_running(self, workflow_name: str, buffer_min: int = 30) -> bool:
+ """Check if a workflow is currently running.
+
+ Estimates workflow duration based on the number of symbols to process
+ and checks if current time falls within the execution window.
- def is_workflow_running(self, workflow_name, buffer_min=30):
+ Args:
+ workflow_name: Name of the workflow to check.
+ buffer_min: Buffer time in minutes around the workflow window.
+
+ Returns:
+ True if the workflow is likely running, False otherwise.
+ """
md = MarketData()
start_time = self.get_workflow_start_time(workflow_name)
num_stock = len(md.get_symbols())
num_crypto = len(POLY_CRYPTO_SYMBOLS)
duration = timedelta(seconds=POLY_FREE_DELAY)
- now = datetime.utcnow()
+ now = datetime.now(UTC)
- if workflow_name in {'ohlc', 'intraday'}:
+ if workflow_name in {"ohlc", "intraday"}:
duration *= (num_stock + num_crypto) * FEW
- elif workflow_name in {'dividends', 'splits'}:
+ elif workflow_name in {"dividends", "splits"}:
duration *= num_stock
else:
return False
buffer = timedelta(minutes=buffer_min)
- return (now < start_time + duration + buffer and
- now > start_time - buffer)
+ return now < start_time + duration + buffer and now > start_time - buffer
+
+ def is_any_workflow_running(self) -> bool:
+ """Check if any data collection workflow is currently running.
- def is_any_workflow_running(self):
- workflows = ['ohlc', 'intraday', 'dividends', 'splits']
- return any(
- [self.is_workflow_running(workflow) for workflow in workflows])
+ Returns:
+ True if any workflow (ohlc, intraday, dividends, splits) is running.
+ """
+ workflows = ["ohlc", "intraday", "dividends", "splits"]
+ return any(self.is_workflow_running(workflow) for workflow in workflows)
diff --git a/hyperdrive/__init__.py b/hyperdrive/__init__.py
index a3f7c39b..f9952a7e 100755
--- a/hyperdrive/__init__.py
+++ b/hyperdrive/__init__.py
@@ -1,5 +1,6 @@
-import os
-import sys
+# hyperdrive package
+"""Algorithmic trading platform."""
-dir_path = os.path.dirname(os.path.realpath(__file__))
-sys.path.append(dir_path)
+from hyperdrive._version import __version__
+
+__all__ = ["__version__"]
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 00000000..720ba518
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,124 @@
+[project]
+name = "hyperdrive"
+dynamic = ["version"]
+description = "An algorithmic trading platform"
+readme = "README.md"
+requires-python = "==3.11.*"
+license = "MIT"
+authors = [{ name = "Krish Suchak" }]
+dependencies = [
+ "python-dotenv>=1.1.0",
+ "pandas>=2.2.3",
+ "robin-stocks>=3.4.0",
+ "boto3>=1.38.13",
+ "polygon-api-client>=1.11.0",
+ "pytz>=2025.2",
+ "vectorbt>=0.28.0",
+ "scipy>=1.15.3",
+ "scikit-learn>=1.6.1",
+ "cryptography>=44.0.3",
+ "python-binance>=1.0.28",
+ "imbalanced-learn>=0.13.0",
+ "icosphere>=0.1.3",
+ "numpy>=1.26.4",
+ "polars>=1.0.0",
+ "pyarrow>=14.0.0",
+ "beautifulsoup4>=4.13.4",
+ "lxml>=5.4.0",
+ "autogluon>=1.3.0",
+ "selenium>=4.27.1",
+]
+
+[dependency-groups]
+dev = [
+ "pytest",
+ "pytest-cov",
+ "pytest-xdist",
+ "ordered-set",
+ "ruff",
+ "moto",
+ "pytest-mock",
+ "responses",
+ "freezegun",
+ "qrcode[pil]",
+ "PyGithub",
+ "gspread",
+ "ty",
+]
+
+[project.urls]
+Homepage = "https://github.com/suchak1/hyperdrive"
+"Bug Reports" = "https://github.com/suchak1/hyperdrive/issues"
+Source = "https://github.com/suchak1/hyperdrive"
+
+[build-system]
+requires = ["hatchling", "hatch-vcs"]
+build-backend = "hatchling.build"
+
+[tool.hatch.version]
+source = "vcs"
+
+[tool.hatch.build.hooks.vcs]
+version-file = "hyperdrive/_version.py"
+
+[tool.ruff]
+target-version = "py311"
+line-length = 88
+
+[tool.ruff.lint]
+select = [
+ "E", # pycodestyle errors
+ "W", # pycodestyle warnings
+ "F", # pyflakes
+ "I", # isort
+ "B", # flake8-bugbear
+ "C4", # flake8-comprehensions
+ "UP", # pyupgrade
+ "ANN", # flake8-annotations (type hints)
+ "D", # pydocstyle (docstrings)
+]
+ignore = [
+ "E501", # line too long (handled by formatter)
+ "ANN401", # dynamically typed expressions (Any)
+ "D100", # missing docstring in public module
+ "D104", # missing docstring in public package
+ "D203", # 1 blank line required before class docstring (conflicts with D211)
+ "D212", # multi-line docstring summary should start at first line (conflicts with D213)
+ "D213", # multi-line docstring summary should start at second line (conflicts with D212)
+]
+
+[tool.ruff.lint.pydocstyle]
+convention = "google"
+
+
+[tool.ruff.format]
+quote-style = "double"
+indent-style = "space"
+
+[tool.ty]
+[tool.ty.environment]
+python-version = "3.11"
+
+[[tool.ty.overrides]]
+include = ["tests/**"]
+
+[tool.ty.overrides.rules]
+invalid-assignment = "warn"
+
+[tool.pytest.ini_options]
+addopts = "-n auto -m 'not integration'"
+testpaths = ["tests"]
+markers = [
+ "integration: marks tests as integration tests (require real API calls)",
+]
+
+[tool.coverage.run]
+source = ["hyperdrive"]
+omit = ["tests/*", ".venv/*", "build/*", "scripts/*"]
+
+[tool.coverage.report]
+fail_under = 95
+show_missing = true
+
+[tool.coverage.term]
+skip_covered = false
diff --git a/requirements.txt b/requirements.txt
deleted file mode 100755
index 7a470c74..00000000
--- a/requirements.txt
+++ /dev/null
@@ -1,19 +0,0 @@
-python-dotenv == 1.1.0
-pandas == 2.2.3
-robin-stocks == 3.4.0
-boto3 == 1.38.13
-polygon-api-client == 1.11.0
-pytz == 2025.2
-vectorbt == 0.27.3
-scipy == 1.15.3
-scikit-learn == 1.6.1
-cryptography == 44.0.3
-ta == 0.11.0
-python-binance == 1.0.28
-imbalanced-learn == 0.13.0
-icosphere == 0.1.3
-numpy == 1.26.4
-beautifulsoup4 == 4.13.4
-lxml == 5.4.0
-autogluon == 1.3.0
-selenium
diff --git a/scripts/delete_data_dir.py b/scripts/delete_data_dir.py
index 5099ecbf..deb7f6e4 100755
--- a/scripts/delete_data_dir.py
+++ b/scripts/delete_data_dir.py
@@ -1,18 +1,29 @@
-import sys
-sys.path.append('hyperdrive')
-from Storage import Store # noqa autopep8
+"""Delete data directory contents from S3 storage."""
+
+from typing import Any
+
+from hyperdrive.Storage import Store
store = Store()
bucket = store.get_bucket()
-def chunks(lst, size):
+def chunks(lst: list[Any], size: int) -> list[list[Any]]:
+ """Split a list into chunks of specified size.
+
+ Args:
+ lst: List to split.
+ size: Maximum size of each chunk.
+
+ Returns:
+ List of sublists, each with at most 'size' elements.
+ """
size = max(1, size)
- return [lst[i:i+size] for i in range(0, len(lst), size)]
+ return [lst[i : i + size] for i in range(0, len(lst), size)]
-keys = [obj.key for obj in bucket.objects.filter(Prefix='data\\')]
+keys = [obj.key for obj in bucket.objects.filter(Prefix="data/")]
keys = chunks(keys, 500)
for key in keys:
store.delete_objects(key)
diff --git a/scripts/delete_dir.py b/scripts/delete_dir.py
index 89f99f03..48c64857 100755
--- a/scripts/delete_dir.py
+++ b/scripts/delete_dir.py
@@ -1,22 +1,35 @@
-import sys
-sys.path.append('hyperdrive')
-from Storage import Store # noqa autopep8
+"""Delete specific symbol directories from S3 storage."""
+
+from typing import Any
+
+from hyperdrive.Storage import Store
store = Store()
bucket = store.get_bucket()
-symbols = ['IAC', 'OTIS', 'VTRS']
+symbols = ["IAC", "OTIS", "VTRS"]
+
+
+def chunks(lst: list[Any], size: int) -> list[list[Any]]:
+ """Split a list into chunks of specified size.
+ Args:
+ lst: List to split.
+ size: Maximum size of each chunk.
-def chunks(lst, size):
+ Returns:
+ List of sublists, each with at most 'size' elements.
+ """
size = max(1, size)
- return [lst[i:i+size] for i in range(0, len(lst), size)]
+ return [lst[i : i + size] for i in range(0, len(lst), size)]
for symbol in symbols:
- keys = [obj.key for obj in bucket.objects.filter(
- Prefix=f'data/intraday/polygon/{symbol}\\')]
+ keys = [
+ obj.key
+ for obj in bucket.objects.filter(Prefix=f"data/intraday/polygon/{symbol}/")
+ ]
keys = chunks(keys, 500)
for key in keys:
store.delete_objects(key)
diff --git a/scripts/execute_order.py b/scripts/execute_order.py
index 1fd2dbb6..fae629c5 100755
--- a/scripts/execute_order.py
+++ b/scripts/execute_order.py
@@ -1,17 +1,17 @@
-import sys
-import pandas as pd
from datetime import datetime, timedelta
-sys.path.append('hyperdrive')
-from DataSource import MarketData # noqa
-from Exchange import Binance, Kraken # noqa autopep8
-import Constants as C # noqa
+
+import pandas as pd
+
+from hyperdrive import Constants as C
+from hyperdrive.DataSource import MarketData
+from hyperdrive.Exchange import Binance, Kraken
test = C.TEST or C.DEV
bn = Binance(testnet=test)
kr = Kraken(test=test)
md = MarketData()
-md.provider = 'polygon'
+md.provider = "polygon"
signals_path = md.finder.get_signals_path()
orders_path = md.finder.get_orders_path()
@@ -26,34 +26,31 @@
if should_order:
side = C.BUY if signal else C.SELL
if C.PREF_EXCHANGE == C.BINANCE:
- base = 'BTC'
- quote = 'USDT' if test else 'USD'
+ base = "BTC"
+ quote = "USDT" if test else "USD"
spend_ratio = C.BINANCE_TEST_SPEND if test else 1
order = bn.order(base, quote, side, spend_ratio, test)
- order['exchange'] = C.BINANCE
+ order["exchange"] = C.BINANCE
else:
- base = 'XXBT'
- quote = 'ZUSD'
+ base = "XXBT"
+ quote = "ZUSD"
spend_ratio = C.KRAKEN_TEST_SPEND if test else 1
if test:
side = kr.get_test_side(base, quote)
order = kr.order(base, quote, side, spend_ratio, test)
if not test:
- order_id = order['txid'][0]
+ order_id = order["txid"][0]
order = kr.get_order(order_id)
- trades = kr.get_trades(order['trades'])
+ trades = kr.get_trades(order["trades"])
order = kr.standardize_order(order, trades)
- order['exchange'] = C.KRAKEN
+ order["exchange"] = C.KRAKEN
order_df = pd.json_normalize(order)
# to keep track of multiple orders (from multiple exchanges),
# order_df = pd.concat(bin_order_df, kr_order_df)
- yesterday = (
- datetime.utcnow().date() -
- timedelta(days=1)
- ).strftime(C.DATE_FMT)
+ yesterday = (datetime.utcnow().date() - timedelta(days=1)).strftime(C.DATE_FMT)
order_df[C.TIME] = [yesterday for _ in range(len(order_df))]
orders = md.reader.update_df(orders_path, order_df, C.TIME, C.DATE_FMT)
diff --git a/scripts/login.py b/scripts/login.py
index 3c823501..bbb7a480 100755
--- a/scripts/login.py
+++ b/scripts/login.py
@@ -1,5 +1,3 @@
-import sys
-sys.path.append('hyperdrive')
-from Algotrader import HyperDrive # noqa autopep8
+from hyperdrive.Algotrader import HyperDrive
drive = HyperDrive()
diff --git a/scripts/qr.py b/scripts/qr.py
new file mode 100644
index 00000000..de44129c
--- /dev/null
+++ b/scripts/qr.py
@@ -0,0 +1,109 @@
+#!/usr/bin/env python3
+"""Generate QR codes for donation addresses from README.md."""
+
+import hashlib
+import re
+import sys
+from io import BytesIO
+from pathlib import Path
+
+import qrcode
+
+
+def parse_addresses(readme: Path) -> dict[str, str]:
+ """Parse donation addresses from README.md.
+
+ Looks for backtick-wrapped addresses in the donation table.
+ Returns dict mapping currency code to address.
+ """
+ content = readme.read_text()
+
+ # Match HTML table rows with currency and code-wrapped address
+ # Pattern: 
| **_hyperdrive_**: an algorithmic trading library |
| -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
-
-
-   [](https://pepy.tech/project/hyperdrive) 
+[](https://github.com/suchak1/hyperdrive/actions/workflows/release.yml)
+[](https://github.com/suchak1/hyperdrive/actions/workflows/pr.yml)
+[](https://pypi.org/project/hyperdrive/)
+[](https://www.python.org/downloads/)
+[](https://opensource.org/licenses/MIT)
+[](https://pepy.tech/project/hyperdrive)
-
-
-
-
-
-
-**_hyperdrive_** is an algorithmic trading library that powers quant research firm [
**Algotrade.io**](https://algotrade.io).
+**_hyperdrive_** is an algorithmic trading library that powers quant research firm [