Technical Design
This section details the key technical design decisions, patterns, and implementation approaches used in Khedra.
Code Organization
Khedra follows a modular code organization pattern to promote maintainability and separation of concerns.
Directory Structure
khedra/
├── app/ // Application core
│ ├── khedra.go // Main application definition
│ └── commands/ // CLI command implementations
├── cmd/ // Command line entry points
│ └── khedra/ // Main CLI command
├── pkg/ // Core packages
│ ├── config/ // Configuration management
│ ├── services/ // Service implementations
│ │ ├── api/ // API service
│ │ ├── control/ // Control service
│ │ ├── ipfs/ // IPFS service
│ │ ├── monitor/ // Monitor service
│ │ └── scraper/ // Scraper service
│ ├── index/ // Unchained Index implementation
│ ├── cache/ // Caching logic
│ ├── chains/ // Chain-specific code
│ ├── rpc/ // RPC client implementations
│ ├── wizard/ // Configuration wizard
│ └── utils/ // Shared utilities
└── main.go // Application entry point
Package Design Principles
- Clear Responsibilities: Each package has a single, well-defined responsibility
- Minimal Dependencies: Packages depend only on what they need
- Interface-Based Design: Dependencies defined as interfaces, not concrete types
- Internal Encapsulation: Implementation details hidden behind public interfaces
- Context-Based Operations: Functions accept context for cancellation and timeout
Service Architecture
Khedra implements a service-oriented architecture within a single application.
Service Interface
Each service implements a common interface:
type Service interface {
// Initialize the service
Init(ctx context.Context) error
// Start the service
Start(ctx context.Context) error
// Stop the service
Stop(ctx context.Context) error
// Return the service name
Name() string
// Return the service status
Status() ServiceStatus
// Return service-specific metrics
Metrics() map[string]interface{}
}
Service Lifecycle
- Registration: Services register with the application core
- Initialization: Services initialize resources and validate configuration
- Starting: Services begin operations in coordinated sequence
- Running: Services perform their core functions
- Stopping: Services gracefully terminate when requested
- Cleanup: Services release resources during application shutdown
Service Coordination
Services coordinate through several mechanisms:
- Direct References: Services can hold references to other services when needed
- Event Bus: Publish-subscribe pattern for decoupled communication
- Shared State: Limited shared state for cross-service information
- Context Propagation: Request context flows through service operations
Data Storage Design
Khedra employs a hybrid storage approach for different data types.
Directory Layout
~/.khedra/
├── config.yaml // Main configuration file
├── data/ // Main data directory
│ ├── mainnet/ // Chain-specific data
│ │ ├── cache/ // Binary caches
│ │ │ ├── blocks/ // Cached blocks
│ │ │ ├── traces/ // Cached traces
│ │ │ └── receipts/ // Cached receipts
│ │ ├── index/ // Unchained Index chunks
│ │ └── monitors/ // Address monitor data
│ └── [other-chains]/ // Other chain data
└── logs/ // Application logs
Storage Formats
- Index Data: Custom binary format optimized for size and query speed
- Cache Data: Compressed binary representation of blockchain data
- Monitor Data: Structured JSON for flexibility and human readability
- Configuration: YAML for readability and easy editing
- Logs: Structured JSON for machine processing and analysis
Storage Persistence Strategy
- Atomic Writes: Prevent corruption during unexpected shutdowns
- Version Headers: Include format version for backward compatibility
- Checksums: Verify data integrity through hash validation
- Backup Points: Periodic snapshots for recovery
- Incremental Updates: Minimize disk writes for frequently changed data
Error Handling and Resilience
Khedra implements robust error handling to ensure reliability in various failure scenarios.
Error Categories
- Transient Errors: Temporary failures that can be retried (network issues, rate limiting)
- Persistent Errors: Failures requiring intervention (misconfiguration, permission issues)
- Fatal Errors: Unrecoverable errors requiring application restart
- Validation Errors: Issues with user input or configuration
- Resource Errors: Problems with system resources (disk space, memory)
Resilience Patterns
- Retry with Backoff: Exponential backoff for transient failures
- Circuit Breakers: Prevent cascading failures when services are unhealthy
- Graceful Degradation: Reduce functionality rather than failing completely
- Health Checks: Proactive monitoring of dependent services
- Recovery Points: Maintain state that allows resuming after failures
Error Reporting
- Structured Logging: Detailed error information in structured format
- Context Preservation: Include context when errors cross boundaries
- Error Wrapping: Maintain error chains without losing information
- User-Friendly Messages: Translate technical errors to actionable information
- Error Metrics: Track error rates and patterns for analysis
Concurrency Model
Khedra leverages Go's concurrency primitives for efficient parallel processing.
Concurrency Patterns
- Worker Pools: Process batches of blocks concurrently with controlled parallelism
- Fan-Out/Fan-In: Distribute work to multiple goroutines and collect results
- Pipelines: Connect processing stages with channels for streaming data
- Context Propagation: Pass cancellation signals through processing chains
- Rate Limiting: Control resource usage and external API calls
Resource Management
- Connection Pooling: Reuse network connections to blockchain nodes
- Goroutine Limiting: Prevent excessive goroutine creation
- Memory Budgeting: Control memory usage during large operations
- I/O Throttling: Balance disk operations to prevent saturation
- Adaptive Concurrency: Adjust parallelism based on system capabilities
Synchronization Techniques
- Mutexes: Protect shared data structures from concurrent access
- Read/Write Locks: Optimize for read-heavy access patterns
- Atomic Operations: Use atomic primitives for simple counters and flags
- Channels: Communicate between goroutines and implement synchronization
- WaitGroups: Coordinate completion of multiple goroutines
Configuration Wizard
The configuration wizard provides an interactive interface for setting up Khedra.
Wizard Architecture
- Screen-Based Flow: Organized as a sequence of screens
- Question Framework: Standardized interface for user input
- Validation Layer: Real-time validation of user inputs
- Navigation System: Forward/backward movement between screens
- Help Integration: Contextual help for each configuration option
User Interface Design
- Text-Based UI: Terminal-friendly interface with box drawing
- Color Coding: Visual cues for different types of information
- Navigation Bar: Consistent display of available commands
- Progress Indication: Show position in the configuration process
- Direct Editing: Option to edit configuration files directly
Implementation Approach
The wizard uses a structured approach to manage screens and user interaction:
type Screen struct {
Title string
Subtitle string
Body string
Instructions string
Replacements []Replacement
Questions []Questioner
Style Style
Current int
Wizard *Wizard
NavigationBar *NavigationBar
}
type Wizard struct {
Config *config.Config
Screens []Screen
Current int
History []int
// Additional fields for wizard state
}
This design allows for a flexible, extensible configuration process that can adapt to different user needs and configuration scenarios.
Testing Strategy
Khedra employs a comprehensive testing strategy to ensure reliability and correctness.
Testing Levels
- Unit Tests: Verify individual functions and components
- Integration Tests: Test interaction between components
- Service Tests: Validate complete service behavior
- End-to-End Tests: Test full application workflows
- Performance Tests: Benchmark critical operations
Test Implementation
- Mock Objects: Simulate external dependencies
- Test Fixtures: Standard data sets for reproducible tests
- Property-Based Testing: Generate test cases to find edge cases
- Parallel Testing: Run tests concurrently for faster feedback
- Coverage Analysis: Track code coverage to identify untested areas
These technical design choices provide Khedra with a solid foundation for reliable, maintainable, and efficient operation across a variety of deployment scenarios and use cases.