first commit

2025-07-15 21:16:36 -04:00 · 2025-07-15 21:16:36 -04:00 · 2903883336
commit 2903883336
4 changed files with 488 additions and 0 deletions
--- a/README.md
+++ b/README.md
@ -0,0 +1 @@
+The trenches are brutal.
--- a/docs/solana-free-websocket-providers.md
+++ b/docs/solana-free-websocket-providers.md
@ -0,0 +1,99 @@
+# Free Solana WebSocket Providers
+
+## 1. Alchemy
+- **Endpoint**: `wss://solana-mainnet.g.alchemy.com/v2/YOUR_API_KEY`
+- **Free Tier Limits**:
+  - 300M Compute Units (CUs) monthly (~12M transactions)
+  - 10 concurrent requests per WebSocket connection on free tier
+  - 20,000 WebSocket connections per API Key limit
+  - 1,000 parallel WebSocket subscriptions per connection
+- **Authentication**: API key required (free signup)
+- **Pros**:
+  - Large monthly CU allocation
+  - Professional infrastructure
+  - 99.9% uptime SLA
+  - Additional features (Smart Wallets, Webhooks)
+- **Cons**:
+  - Severe concurrent request limit on free tier (only 10)
+  - Some users report issues with Solana WebSocket implementation
+  - Requires registration
+
+## 2. PublicNode
+- **Endpoint**: `wss://solana-rpc.publicnode.com`
+- **Limits**: None specified (completely free)
+- **Authentication**: None required
+- **Pros**: 
+  - No registration needed
+  - No API keys
+  - Privacy-focused
+  - Simple to get started
+- **Cons**: 
+  - No SLA or uptime guarantees
+  - May have undocumented rate limits
+  - No support
+
+## 2. Helius
+- **Endpoint**: `wss://mainnet.helius-rpc.com/?api-key=YOUR_API_KEY`
+- **Free Tier Limits**:
+  - 1M credits per month
+  - 10 requests per second
+  - Standard WebSockets included
+- **Authentication**: API key required (free signup)
+- **Pros**:
+  - Reliable infrastructure
+  - Community support
+  - Additional APIs available (NFT, Token APIs)
+  - Clear rate limits
+- **Cons**:
+  - Limited credits (1M/month)
+  - Enhanced WebSockets only on paid tiers
+  - Requires registration
+
+## 4. QuickNode
+- **Endpoint**: `wss://YOUR_ENDPOINT_NAME.quiknode.pro/YOUR_TOKEN/`
+- **Free Tier Limits**:
+  - 10M credits per month
+  - WebSocket responses cost 50 credits each
+  - ~200,000 WebSocket responses per month
+- **Authentication**: Endpoint URL with token
+- **Billing Model**: Pay per response, not per subscription
+- **Pros**:
+  - Generous credit allocation
+  - Professional infrastructure
+  - Good documentation
+  - Multiple solutions (WebSockets, Streams, gRPC)
+- **Cons**:
+  - Credit-based billing for responses
+  - Advanced features require paid plans
+  - Must track response usage
+
+## 4. Solana Public RPC
+- **Endpoint**: `wss://api.mainnet-beta.solana.com`
+- **Limits**: Subject to change without notice
+- **Authentication**: None
+- **Pros**:
+  - Official Solana endpoint
+  - No registration required
+  - Completely free
+- **Cons**:
+  - NOT for production use
+  - Aggressive rate limiting
+  - Can block IPs without warning
+  - No SLA or reliability guarantees
+  - Poor performance under load
+
+## Recommendations
+
+1. **For Development/Testing**: Use PublicNode or Solana Public RPC
+2. **For Small Projects**: Helius (1M credits) or PublicNode
+3. **For Medium Projects**: Alchemy (300M CUs but with concurrent limit) or QuickNode (10M credits)
+4. **For High-Volume HTTP with occasional WebSocket**: Alchemy (best CU allocation)
+5. **For WebSocket-Heavy Applications**: Avoid Alchemy free tier due to 10 concurrent request limit
+
+## Important Notes
+
+- WebSocket subscriptions are stateful - you cannot easily switch providers mid-subscription
+- Each provider may have slight differences in response format or timing
+- Always implement reconnection logic for production applications
+- Consider implementing client-side failover rather than load balancing for WebSockets
+- Monitor your usage carefully, especially on QuickNode where responses consume credits
--- a/docs/solana-proxy-implementation-plan.md
+++ b/docs/solana-proxy-implementation-plan.md
@ -0,0 +1,280 @@
+# Solana RPC Proxy Implementation Plan
+
+## Project Overview
+A Python-based reverse proxy for Solana RPC endpoints that provides unified access to multiple free providers with automatic failover, response caching, and detailed error tracking.
+
+## Architecture Components
+
+### 1. Provider Module (`providers.py`)
+**Purpose**: Abstract provider differences and handle authentication
+
+```
+Provider class:
+- name: str
+- http_url: str  
+- ws_url: str
+- transform_request(request) -> request
+- transform_response(response) -> response
+- is_available() -> bool
+- mark_failed() -> None
+- backoff_until: datetime
+```
+
+**Provider Implementations**:
+- `AlchemyProvider` - API key in URL path
+- `PublicNodeProvider` - No auth
+- `HeliusProvider` - API key in query param
+- `QuickNodeProvider` - Token in URL path
+- `SolanaPublicProvider` - No auth
+
+### 2. Cache Module (`cache.py`)
+**Purpose**: Disk-based caching for HTTP and WebSocket responses
+
+```
+Cache class:
+- get(method: str, params: dict) -> Optional[response]
+- set(method: str, params: dict, response: dict) -> None
+- size_check() -> None  # Enforce 100GB limit
+- clear_oldest() -> None  # LRU eviction
+```
+
+**Implementation Notes**:
+- Use `diskcache` library for simplicity
+- Key format: `f"{method}:{json.dumps(params, sort_keys=True)}"` 
+- Store both HTTP responses and WebSocket messages
+- Implement 100GB limit with LRU eviction
+
+### 3. Error Logger Module (`errors.py`)
+**Purpose**: SQLite-based error logging with UUID tracking
+
+```
+ErrorLogger class:
+- log_error(provider: str, request: dict, error: Exception) -> str (UUID)
+- get_error(error_id: str) -> Optional[dict]
+- setup_db() -> None
+```
+
+**Database Schema**:
+```sql
+CREATE TABLE errors (
+    id TEXT PRIMARY KEY,  -- UUID
+    timestamp DATETIME,
+    provider TEXT,
+    request_method TEXT,
+    request_params TEXT,  -- JSON
+    error_type TEXT,
+    error_message TEXT,
+    error_traceback TEXT
+);
+```
+
+### 4. Response Normalizer Module (`normalizer.py`)
+**Purpose**: Handle minor provider response differences
+
+```
+normalize_response(provider: str, response: dict) -> dict
+normalize_error(error: Exception, error_id: str) -> dict
+```
+
+**Normalization Tasks**:
+- Ensure consistent field names
+- Handle null vs missing fields
+- Standardize error formats
+- Add proxy metadata (cached, provider used)
+
+### 5. Request Router Module (`router.py`)
+**Purpose**: Core failover and routing logic
+
+```
+Router class:
+- providers: List[Provider]
+- cache: Cache
+- error_logger: ErrorLogger
+- 
+- route_request(method: str, params: dict) -> response
+- get_available_provider() -> Optional[Provider]
+- mark_provider_failed(provider: Provider) -> None
+```
+
+**Failover Algorithm**:
+1. Check cache first
+2. Get next available provider (round-robin)
+3. Try request
+4. On success: cache and return
+5. On failure: log error, mark provider failed, try next
+6. All failed: return error with ID
+
+### 6. HTTP Proxy Module (`http_proxy.py`)
+**Purpose**: aiohttp server for HTTP JSON-RPC
+
+```
+- setup_routes(app: aiohttp.Application)
+- handle_rpc_request(request: aiohttp.Request) -> aiohttp.Response
+```
+
+### 7. WebSocket Proxy Module (`ws_proxy.py`)
+**Purpose**: WebSocket subscription handling
+
+```
+- handle_ws_connection(request: aiohttp.Request) -> aiohttp.WebSocketResponse
+- proxy_ws_messages(client_ws, provider_ws, cache, provider_name)
+```
+
+**WebSocket Complexity**:
+- Maintain subscription ID mapping
+- Cache subscription responses
+- Handle reconnection on provider failure
+
+### 8. Main Application (`main.py`)
+**Purpose**: Wire everything together
+
+```
+- load_config() -> dict  # From .env
+- setup_providers(config) -> List[Provider]
+- create_app() -> aiohttp.Application
+- main() -> None
+```
+
+## Configuration (.env)
+
+```env
+# Provider endpoints and auth
+ALCHEMY_API_KEY=your_key_here
+HELIUS_API_KEY=your_key_here
+QUICKNODE_ENDPOINT=your_endpoint.quiknode.pro
+QUICKNODE_TOKEN=your_token_here
+
+# Proxy settings
+PROXY_PORT=8545
+CACHE_SIZE_GB=100
+BACKOFF_MINUTES=30
+
+# Logging
+LOG_LEVEL=INFO
+ERROR_DB_PATH=./errors.db
+```
+
+## Rate Limits Documentation
+
+```python
+# providers.py comments
+
+# Alchemy (Source: https://docs.alchemy.com/reference/throughput)
+# Free tier: 330 CUPs (Compute Units per Second)
+# WebSocket: 10 concurrent requests per connection
+
+# PublicNode (Source: https://publicnode.com)
+# No published rate limits - "completely free"
+
+# Helius (Source: https://docs.helius.dev/pricing)
+# Free tier: 10 requests/second
+# 1M credits per month
+
+# QuickNode (Source: https://www.quicknode.com/pricing)
+# Free tier: 10M credits/month
+# WebSocket: 50 credits per response
+
+# Solana Public (Source: https://solana.com/docs/cluster/rpc-endpoints)
+# Rate limits subject to change without notice
+# Not intended for production use
+```
+
+## Testing Strategy (`test_e2e.py`)
+
+Happy-path end-to-end tests only:
+
+1. **Test HTTP Proxy**:
+   - Start proxy
+   - Make getBalance request
+   - Verify response format
+   - Verify cache hit on second request
+
+2. **Test WebSocket Proxy**:
+   - Connect WebSocket
+   - Subscribe to account
+   - Verify subscription response
+   - Verify cached messages
+
+3. **Test Failover**:
+   - Mock provider failure
+   - Verify failover to next provider
+   - Verify error logged with ID
+
+4. **Test All Providers**:
+   - Iterate through each provider
+   - Verify basic request works
+   - Verify auth handled correctly
+
+## Implementation Notes
+
+### Functional Programming Style
+- Use pure functions where possible
+- Avoid class state mutations
+- Use immutable data structures
+- Compose small functions
+
+### KISS Principles
+- No complex health checking (just try request)
+- No credit tracking (let providers handle)
+- Simple round-robin selection
+- Basic LRU cache eviction
+
+### DRY Principles
+- Single Provider base class
+- Reuse request/response transformation
+- Common error handling flow
+- Shared cache logic for HTTP/WS
+
+## Deployment Considerations
+
+1. **Cache Storage**: Need ~100GB disk space
+2. **Memory Usage**: Keep minimal, use disk cache
+3. **Concurrent Clients**: Basic round-robin if multiple connect
+4. **Monitoring**: Log all errors, provide error IDs
+5. **Security**: Keep API keys in .env, never log them
+
+## Future Enhancements (Out of Scope)
+
+- Credit/quota tracking
+- Advanced health checking
+- Response time optimization
+- Geographic routing
+- Analytics dashboard
+- Webhook error notifications
+
+## File Structure
+
+```
+solana-proxy/
+├── .env                 # Configuration
+├── .env.example         # Template
+├── requirements.txt     # Dependencies
+├── main.py             # Entry point
+├── providers.py        # Provider abstractions
+├── cache.py            # Caching logic
+├── errors.py           # Error logging
+├── normalizer.py       # Response normalization
+├── router.py           # Request routing
+├── http_proxy.py       # HTTP handler
+├── ws_proxy.py         # WebSocket handler
+└── test_e2e.py         # End-to-end tests
+```
+
+## Dependencies
+
+```
+aiohttp==3.9.0
+python-dotenv==1.0.0
+diskcache==5.6.0
+aiohttp-cors==0.7.0
+```
+
+## Success Criteria
+
+1. Single endpoint proxies to 5 providers
+2. Automatic failover works
+3. Responses are cached (up to 100GB)
+4. Errors logged with retrievable IDs
+5. Both HTTP and WebSocket work
+6. Response format is unified
+7. Happy-path tests pass
--- a/docs/solana-rpc-signup-guide.md
+++ b/docs/solana-rpc-signup-guide.md
@ -0,0 +1,108 @@
+# Solana Free RPC Provider Signup Guide
+
+## Quick Comparison Table
+
+| Provider | Signup Required | Credit Card | Email Verification | Time to Start |
+|----------|----------------|-------------|-------------------|---------------|
+| **PublicNode** | ❌ No | ❌ No | ❌ No | Instant |
+| **Solana Public** | ❌ No | ❌ No | ❌ No | Instant |
+| **Helius** | ✅ Yes | ❌ No | ✅ Yes | < 10 seconds |
+| **Alchemy** | ✅ Yes | ❌ No | ✅ Yes | ~1 minute |
+| **QuickNode** | ✅ Yes | ❌ No | ✅ Yes | ~2-3 minutes |
+
+## Detailed Signup Requirements
+
+### 1. **PublicNode** 🚀
+- **Signup Required**: NO
+- **Process**: None - just use the endpoints
+- **HTTP Endpoint**: `https://solana-rpc.publicnode.com`
+- **WebSocket Endpoint**: `wss://solana-rpc.publicnode.com`
+- **Key Features**:
+  - Completely anonymous
+  - No API keys needed
+  - No tracking or data collection
+  - "Fastest, free-est, and privacy first"
+
+### 2. **Solana Public RPC** ⚠️
+- **Signup Required**: NO
+- **Process**: None - direct access
+- **HTTP Endpoint**: `https://api.mainnet-beta.solana.com`
+- **WebSocket Endpoint**: `wss://api.mainnet-beta.solana.com`
+- **Warning**: Not for production use, heavily rate-limited
+
+### 3. **Helius** ⭐
+- **Signup Required**: YES (but very quick)
+- **Process**:
+  1. Go to dashboard.helius.dev
+  2. Click "Start for Free"
+  3. Enter email and password
+  4. Verify email
+  5. API key auto-generated
+- **What You Get**:
+  - 1M credits/month free
+  - 10 requests/second
+  - Access to enhanced APIs
+  - Dashboard with usage metrics
+
+### 4. **Alchemy** ✅
+- **Signup Required**: YES
+- **Process**:
+  1. Create account with email/password
+  2. Verify email
+  3. Access dashboard
+  4. Create Solana app
+  5. Get endpoint URLs
+- **What You Get**:
+  - 300M Compute Units/month (~12M transactions)
+  - Professional dashboard
+  - Additional features (Smart Wallets, Webhooks)
+  - 99.9% uptime SLA
+
+### 5. **QuickNode** 📊
+- **Signup Required**: YES (most involved)
+- **Process**:
+  1. Sign up at quicknode.com
+  2. Verify email ("Check your email to verify")
+  3. Click "Create an endpoint"
+  4. Select Solana blockchain
+  5. Choose network (mainnet)
+  6. Optional: Add marketplace add-ons
+  7. Select Free plan
+  8. Get HTTP and WSS URLs
+- **What You Get**:
+  - 10M credits/month
+  - Professional infrastructure
+  - Marketplace add-ons available
+  - Detailed analytics
+
+## Recommendations by Use Case
+
+### For Immediate Testing/Prototyping
+1. **PublicNode** - No signup, start immediately
+2. **Solana Public** - No signup but unreliable
+
+### For Development Projects
+1. **Helius** - Quick signup, good free tier
+2. **Alchemy** - Best free tier allocation
+3. **QuickNode** - Most features but complex setup
+
+### For Production (Paid Tiers Recommended)
+- All providers offer paid upgrades
+- Avoid Solana Public RPC entirely
+- Consider multiple providers for redundancy
+
+## Security Notes
+
+- **Helius**: Keep API key secure, don't commit to GitHub
+- **Alchemy**: API key in URL, protect in environment variables
+- **QuickNode**: Token in URL path, requires careful handling
+- **PublicNode**: No auth = no security concerns for credentials
+
+## Getting Started Tips
+
+1. **Fastest Start**: Use PublicNode - no signup needed
+2. **Best Free Tier**: Alchemy (300M CUs) but with WebSocket limitations
+3. **Most Balanced**: Helius - quick signup, decent limits
+4. **Most Professional**: QuickNode - complex but feature-rich
+
+Remember: Free tiers are great for development but consider paid plans for production applications to ensure reliability and support.