commit 29038833368fcbfea001cda698a665a39e6193aa Author: A. F. Dudley Date: Tue Jul 15 21:16:36 2025 -0400 first commit diff --git a/README.md b/README.md new file mode 100644 index 0000000..781020d --- /dev/null +++ b/README.md @@ -0,0 +1 @@ +The trenches are brutal. diff --git a/docs/solana-free-websocket-providers.md b/docs/solana-free-websocket-providers.md new file mode 100644 index 0000000..d61e6f3 --- /dev/null +++ b/docs/solana-free-websocket-providers.md @@ -0,0 +1,99 @@ +# Free Solana WebSocket Providers + +## 1. Alchemy +- **Endpoint**: `wss://solana-mainnet.g.alchemy.com/v2/YOUR_API_KEY` +- **Free Tier Limits**: + - 300M Compute Units (CUs) monthly (~12M transactions) + - 10 concurrent requests per WebSocket connection on free tier + - 20,000 WebSocket connections per API Key limit + - 1,000 parallel WebSocket subscriptions per connection +- **Authentication**: API key required (free signup) +- **Pros**: + - Large monthly CU allocation + - Professional infrastructure + - 99.9% uptime SLA + - Additional features (Smart Wallets, Webhooks) +- **Cons**: + - Severe concurrent request limit on free tier (only 10) + - Some users report issues with Solana WebSocket implementation + - Requires registration + +## 2. PublicNode +- **Endpoint**: `wss://solana-rpc.publicnode.com` +- **Limits**: None specified (completely free) +- **Authentication**: None required +- **Pros**: + - No registration needed + - No API keys + - Privacy-focused + - Simple to get started +- **Cons**: + - No SLA or uptime guarantees + - May have undocumented rate limits + - No support + +## 2. Helius +- **Endpoint**: `wss://mainnet.helius-rpc.com/?api-key=YOUR_API_KEY` +- **Free Tier Limits**: + - 1M credits per month + - 10 requests per second + - Standard WebSockets included +- **Authentication**: API key required (free signup) +- **Pros**: + - Reliable infrastructure + - Community support + - Additional APIs available (NFT, Token APIs) + - Clear rate limits +- **Cons**: + - Limited credits (1M/month) + - Enhanced WebSockets only on paid tiers + - Requires registration + +## 4. QuickNode +- **Endpoint**: `wss://YOUR_ENDPOINT_NAME.quiknode.pro/YOUR_TOKEN/` +- **Free Tier Limits**: + - 10M credits per month + - WebSocket responses cost 50 credits each + - ~200,000 WebSocket responses per month +- **Authentication**: Endpoint URL with token +- **Billing Model**: Pay per response, not per subscription +- **Pros**: + - Generous credit allocation + - Professional infrastructure + - Good documentation + - Multiple solutions (WebSockets, Streams, gRPC) +- **Cons**: + - Credit-based billing for responses + - Advanced features require paid plans + - Must track response usage + +## 4. Solana Public RPC +- **Endpoint**: `wss://api.mainnet-beta.solana.com` +- **Limits**: Subject to change without notice +- **Authentication**: None +- **Pros**: + - Official Solana endpoint + - No registration required + - Completely free +- **Cons**: + - NOT for production use + - Aggressive rate limiting + - Can block IPs without warning + - No SLA or reliability guarantees + - Poor performance under load + +## Recommendations + +1. **For Development/Testing**: Use PublicNode or Solana Public RPC +2. **For Small Projects**: Helius (1M credits) or PublicNode +3. **For Medium Projects**: Alchemy (300M CUs but with concurrent limit) or QuickNode (10M credits) +4. **For High-Volume HTTP with occasional WebSocket**: Alchemy (best CU allocation) +5. **For WebSocket-Heavy Applications**: Avoid Alchemy free tier due to 10 concurrent request limit + +## Important Notes + +- WebSocket subscriptions are stateful - you cannot easily switch providers mid-subscription +- Each provider may have slight differences in response format or timing +- Always implement reconnection logic for production applications +- Consider implementing client-side failover rather than load balancing for WebSockets +- Monitor your usage carefully, especially on QuickNode where responses consume credits \ No newline at end of file diff --git a/docs/solana-proxy-implementation-plan.md b/docs/solana-proxy-implementation-plan.md new file mode 100644 index 0000000..be340cb --- /dev/null +++ b/docs/solana-proxy-implementation-plan.md @@ -0,0 +1,280 @@ +# Solana RPC Proxy Implementation Plan + +## Project Overview +A Python-based reverse proxy for Solana RPC endpoints that provides unified access to multiple free providers with automatic failover, response caching, and detailed error tracking. + +## Architecture Components + +### 1. Provider Module (`providers.py`) +**Purpose**: Abstract provider differences and handle authentication + +``` +Provider class: +- name: str +- http_url: str +- ws_url: str +- transform_request(request) -> request +- transform_response(response) -> response +- is_available() -> bool +- mark_failed() -> None +- backoff_until: datetime +``` + +**Provider Implementations**: +- `AlchemyProvider` - API key in URL path +- `PublicNodeProvider` - No auth +- `HeliusProvider` - API key in query param +- `QuickNodeProvider` - Token in URL path +- `SolanaPublicProvider` - No auth + +### 2. Cache Module (`cache.py`) +**Purpose**: Disk-based caching for HTTP and WebSocket responses + +``` +Cache class: +- get(method: str, params: dict) -> Optional[response] +- set(method: str, params: dict, response: dict) -> None +- size_check() -> None # Enforce 100GB limit +- clear_oldest() -> None # LRU eviction +``` + +**Implementation Notes**: +- Use `diskcache` library for simplicity +- Key format: `f"{method}:{json.dumps(params, sort_keys=True)}"` +- Store both HTTP responses and WebSocket messages +- Implement 100GB limit with LRU eviction + +### 3. Error Logger Module (`errors.py`) +**Purpose**: SQLite-based error logging with UUID tracking + +``` +ErrorLogger class: +- log_error(provider: str, request: dict, error: Exception) -> str (UUID) +- get_error(error_id: str) -> Optional[dict] +- setup_db() -> None +``` + +**Database Schema**: +```sql +CREATE TABLE errors ( + id TEXT PRIMARY KEY, -- UUID + timestamp DATETIME, + provider TEXT, + request_method TEXT, + request_params TEXT, -- JSON + error_type TEXT, + error_message TEXT, + error_traceback TEXT +); +``` + +### 4. Response Normalizer Module (`normalizer.py`) +**Purpose**: Handle minor provider response differences + +``` +normalize_response(provider: str, response: dict) -> dict +normalize_error(error: Exception, error_id: str) -> dict +``` + +**Normalization Tasks**: +- Ensure consistent field names +- Handle null vs missing fields +- Standardize error formats +- Add proxy metadata (cached, provider used) + +### 5. Request Router Module (`router.py`) +**Purpose**: Core failover and routing logic + +``` +Router class: +- providers: List[Provider] +- cache: Cache +- error_logger: ErrorLogger +- +- route_request(method: str, params: dict) -> response +- get_available_provider() -> Optional[Provider] +- mark_provider_failed(provider: Provider) -> None +``` + +**Failover Algorithm**: +1. Check cache first +2. Get next available provider (round-robin) +3. Try request +4. On success: cache and return +5. On failure: log error, mark provider failed, try next +6. All failed: return error with ID + +### 6. HTTP Proxy Module (`http_proxy.py`) +**Purpose**: aiohttp server for HTTP JSON-RPC + +``` +- setup_routes(app: aiohttp.Application) +- handle_rpc_request(request: aiohttp.Request) -> aiohttp.Response +``` + +### 7. WebSocket Proxy Module (`ws_proxy.py`) +**Purpose**: WebSocket subscription handling + +``` +- handle_ws_connection(request: aiohttp.Request) -> aiohttp.WebSocketResponse +- proxy_ws_messages(client_ws, provider_ws, cache, provider_name) +``` + +**WebSocket Complexity**: +- Maintain subscription ID mapping +- Cache subscription responses +- Handle reconnection on provider failure + +### 8. Main Application (`main.py`) +**Purpose**: Wire everything together + +``` +- load_config() -> dict # From .env +- setup_providers(config) -> List[Provider] +- create_app() -> aiohttp.Application +- main() -> None +``` + +## Configuration (.env) + +```env +# Provider endpoints and auth +ALCHEMY_API_KEY=your_key_here +HELIUS_API_KEY=your_key_here +QUICKNODE_ENDPOINT=your_endpoint.quiknode.pro +QUICKNODE_TOKEN=your_token_here + +# Proxy settings +PROXY_PORT=8545 +CACHE_SIZE_GB=100 +BACKOFF_MINUTES=30 + +# Logging +LOG_LEVEL=INFO +ERROR_DB_PATH=./errors.db +``` + +## Rate Limits Documentation + +```python +# providers.py comments + +# Alchemy (Source: https://docs.alchemy.com/reference/throughput) +# Free tier: 330 CUPs (Compute Units per Second) +# WebSocket: 10 concurrent requests per connection + +# PublicNode (Source: https://publicnode.com) +# No published rate limits - "completely free" + +# Helius (Source: https://docs.helius.dev/pricing) +# Free tier: 10 requests/second +# 1M credits per month + +# QuickNode (Source: https://www.quicknode.com/pricing) +# Free tier: 10M credits/month +# WebSocket: 50 credits per response + +# Solana Public (Source: https://solana.com/docs/cluster/rpc-endpoints) +# Rate limits subject to change without notice +# Not intended for production use +``` + +## Testing Strategy (`test_e2e.py`) + +Happy-path end-to-end tests only: + +1. **Test HTTP Proxy**: + - Start proxy + - Make getBalance request + - Verify response format + - Verify cache hit on second request + +2. **Test WebSocket Proxy**: + - Connect WebSocket + - Subscribe to account + - Verify subscription response + - Verify cached messages + +3. **Test Failover**: + - Mock provider failure + - Verify failover to next provider + - Verify error logged with ID + +4. **Test All Providers**: + - Iterate through each provider + - Verify basic request works + - Verify auth handled correctly + +## Implementation Notes + +### Functional Programming Style +- Use pure functions where possible +- Avoid class state mutations +- Use immutable data structures +- Compose small functions + +### KISS Principles +- No complex health checking (just try request) +- No credit tracking (let providers handle) +- Simple round-robin selection +- Basic LRU cache eviction + +### DRY Principles +- Single Provider base class +- Reuse request/response transformation +- Common error handling flow +- Shared cache logic for HTTP/WS + +## Deployment Considerations + +1. **Cache Storage**: Need ~100GB disk space +2. **Memory Usage**: Keep minimal, use disk cache +3. **Concurrent Clients**: Basic round-robin if multiple connect +4. **Monitoring**: Log all errors, provide error IDs +5. **Security**: Keep API keys in .env, never log them + +## Future Enhancements (Out of Scope) + +- Credit/quota tracking +- Advanced health checking +- Response time optimization +- Geographic routing +- Analytics dashboard +- Webhook error notifications + +## File Structure + +``` +solana-proxy/ +├── .env # Configuration +├── .env.example # Template +├── requirements.txt # Dependencies +├── main.py # Entry point +├── providers.py # Provider abstractions +├── cache.py # Caching logic +├── errors.py # Error logging +├── normalizer.py # Response normalization +├── router.py # Request routing +├── http_proxy.py # HTTP handler +├── ws_proxy.py # WebSocket handler +└── test_e2e.py # End-to-end tests +``` + +## Dependencies + +``` +aiohttp==3.9.0 +python-dotenv==1.0.0 +diskcache==5.6.0 +aiohttp-cors==0.7.0 +``` + +## Success Criteria + +1. Single endpoint proxies to 5 providers +2. Automatic failover works +3. Responses are cached (up to 100GB) +4. Errors logged with retrievable IDs +5. Both HTTP and WebSocket work +6. Response format is unified +7. Happy-path tests pass \ No newline at end of file diff --git a/docs/solana-rpc-signup-guide.md b/docs/solana-rpc-signup-guide.md new file mode 100644 index 0000000..5bcd48e --- /dev/null +++ b/docs/solana-rpc-signup-guide.md @@ -0,0 +1,108 @@ +# Solana Free RPC Provider Signup Guide + +## Quick Comparison Table + +| Provider | Signup Required | Credit Card | Email Verification | Time to Start | +|----------|----------------|-------------|-------------------|---------------| +| **PublicNode** | ❌ No | ❌ No | ❌ No | Instant | +| **Solana Public** | ❌ No | ❌ No | ❌ No | Instant | +| **Helius** | ✅ Yes | ❌ No | ✅ Yes | < 10 seconds | +| **Alchemy** | ✅ Yes | ❌ No | ✅ Yes | ~1 minute | +| **QuickNode** | ✅ Yes | ❌ No | ✅ Yes | ~2-3 minutes | + +## Detailed Signup Requirements + +### 1. **PublicNode** 🚀 +- **Signup Required**: NO +- **Process**: None - just use the endpoints +- **HTTP Endpoint**: `https://solana-rpc.publicnode.com` +- **WebSocket Endpoint**: `wss://solana-rpc.publicnode.com` +- **Key Features**: + - Completely anonymous + - No API keys needed + - No tracking or data collection + - "Fastest, free-est, and privacy first" + +### 2. **Solana Public RPC** ⚠️ +- **Signup Required**: NO +- **Process**: None - direct access +- **HTTP Endpoint**: `https://api.mainnet-beta.solana.com` +- **WebSocket Endpoint**: `wss://api.mainnet-beta.solana.com` +- **Warning**: Not for production use, heavily rate-limited + +### 3. **Helius** ⭐ +- **Signup Required**: YES (but very quick) +- **Process**: + 1. Go to dashboard.helius.dev + 2. Click "Start for Free" + 3. Enter email and password + 4. Verify email + 5. API key auto-generated +- **What You Get**: + - 1M credits/month free + - 10 requests/second + - Access to enhanced APIs + - Dashboard with usage metrics + +### 4. **Alchemy** ✅ +- **Signup Required**: YES +- **Process**: + 1. Create account with email/password + 2. Verify email + 3. Access dashboard + 4. Create Solana app + 5. Get endpoint URLs +- **What You Get**: + - 300M Compute Units/month (~12M transactions) + - Professional dashboard + - Additional features (Smart Wallets, Webhooks) + - 99.9% uptime SLA + +### 5. **QuickNode** 📊 +- **Signup Required**: YES (most involved) +- **Process**: + 1. Sign up at quicknode.com + 2. Verify email ("Check your email to verify") + 3. Click "Create an endpoint" + 4. Select Solana blockchain + 5. Choose network (mainnet) + 6. Optional: Add marketplace add-ons + 7. Select Free plan + 8. Get HTTP and WSS URLs +- **What You Get**: + - 10M credits/month + - Professional infrastructure + - Marketplace add-ons available + - Detailed analytics + +## Recommendations by Use Case + +### For Immediate Testing/Prototyping +1. **PublicNode** - No signup, start immediately +2. **Solana Public** - No signup but unreliable + +### For Development Projects +1. **Helius** - Quick signup, good free tier +2. **Alchemy** - Best free tier allocation +3. **QuickNode** - Most features but complex setup + +### For Production (Paid Tiers Recommended) +- All providers offer paid upgrades +- Avoid Solana Public RPC entirely +- Consider multiple providers for redundancy + +## Security Notes + +- **Helius**: Keep API key secure, don't commit to GitHub +- **Alchemy**: API key in URL, protect in environment variables +- **QuickNode**: Token in URL path, requires careful handling +- **PublicNode**: No auth = no security concerns for credentials + +## Getting Started Tips + +1. **Fastest Start**: Use PublicNode - no signup needed +2. **Best Free Tier**: Alchemy (300M CUs) but with WebSocket limitations +3. **Most Balanced**: Helius - quick signup, decent limits +4. **Most Professional**: QuickNode - complex but feature-rich + +Remember: Free tiers are great for development but consider paid plans for production applications to ensure reliability and support. \ No newline at end of file