Technical Design

This section details the key technical design decisions, patterns, and implementation approaches used in Khedra.

Code Organization

Khedra follows a modular code organization pattern to promote maintainability and separation of concerns.

Directory Structure

khedra/
├── app/                 // Application core
│   ├── khedra.go        // Main application definition
│   └── commands/        // CLI command implementations
├── cmd/                 // Command line entry points
│   └── khedra/          // Main CLI command
├── pkg/                 // Core packages
│   ├── config/          // Configuration management
│   ├── services/        // Service implementations
│   │   ├── api/         // API service
│   │   ├── control/     // Control service
│   │   ├── ipfs/        // IPFS service
│   │   ├── monitor/     // Monitor service
│   │   └── scraper/     // Scraper service
│   ├── index/           // Unchained Index implementation
│   ├── cache/           // Caching logic
│   ├── chains/          // Chain-specific code
│   ├── rpc/             // RPC client implementations
│   ├── wizard/          // Configuration wizard
│   └── utils/           // Shared utilities
└── main.go              // Application entry point

Package Design Principles

  1. Clear Responsibilities: Each package has a single, well-defined responsibility
  2. Minimal Dependencies: Packages depend only on what they need
  3. Interface-Based Design: Dependencies defined as interfaces, not concrete types
  4. Internal Encapsulation: Implementation details hidden behind public interfaces
  5. Context-Based Operations: Functions accept context for cancellation and timeout

Service Architecture

Khedra implements a service-oriented architecture within a single application.

Service Interface

Each service implements a common interface:

type Service interface {
    // Initialize the service
    Init(ctx context.Context) error
    
    // Start the service
    Start(ctx context.Context) error
    
    // Stop the service
    Stop(ctx context.Context) error
    
    // Return the service name
    Name() string
    
    // Return the service status
    Status() ServiceStatus
    
    // Return service-specific metrics
    Metrics() map[string]interface{}
}

Service Lifecycle

  1. Registration: Services register with the application core
  2. Initialization: Services initialize resources and validate configuration
  3. Starting: Services begin operations in coordinated sequence
  4. Running: Services perform their core functions
  5. Stopping: Services gracefully terminate when requested
  6. Cleanup: Services release resources during application shutdown

Service Coordination

Services coordinate through several mechanisms:

  1. Direct References: Services can hold references to other services when needed
  2. Event Bus: Publish-subscribe pattern for decoupled communication
  3. Shared State: Limited shared state for cross-service information
  4. Context Propagation: Request context flows through service operations

Data Storage Design

Khedra employs a hybrid storage approach for different data types.

Directory Layout

~/.khedra/
├── config.yaml           // Main configuration file
├── data/                 // Main data directory
│   ├── mainnet/          // Chain-specific data
│   │   ├── cache/        // Binary caches
│   │   │   ├── blocks/   // Cached blocks
│   │   │   ├── traces/   // Cached traces
│   │   │   └── receipts/ // Cached receipts
│   │   ├── index/        // Unchained Index chunks
│   │   └── monitors/     // Address monitor data
│   └── [other-chains]/   // Other chain data
└── logs/                 // Application logs

Storage Formats

  1. Index Data: Custom binary format optimized for size and query speed
  2. Cache Data: Compressed binary representation of blockchain data
  3. Monitor Data: Structured JSON for flexibility and human readability
  4. Configuration: YAML for readability and easy editing
  5. Logs: Structured JSON for machine processing and analysis

Storage Persistence Strategy

  1. Atomic Writes: Prevent corruption during unexpected shutdowns
  2. Version Headers: Include format version for backward compatibility
  3. Checksums: Verify data integrity through hash validation
  4. Backup Points: Periodic snapshots for recovery
  5. Incremental Updates: Minimize disk writes for frequently changed data

Error Handling and Resilience

Khedra implements robust error handling to ensure reliability in various failure scenarios.

Error Categories

  1. Transient Errors: Temporary failures that can be retried (network issues, rate limiting)
  2. Persistent Errors: Failures requiring intervention (misconfiguration, permission issues)
  3. Fatal Errors: Unrecoverable errors requiring application restart
  4. Validation Errors: Issues with user input or configuration
  5. Resource Errors: Problems with system resources (disk space, memory)

Resilience Patterns

  1. Retry with Backoff: Exponential backoff for transient failures
  2. Circuit Breakers: Prevent cascading failures when services are unhealthy
  3. Graceful Degradation: Reduce functionality rather than failing completely
  4. Health Checks: Proactive monitoring of dependent services
  5. Recovery Points: Maintain state that allows resuming after failures

Error Reporting

  1. Structured Logging: Detailed error information in structured format
  2. Context Preservation: Include context when errors cross boundaries
  3. Error Wrapping: Maintain error chains without losing information
  4. User-Friendly Messages: Translate technical errors to actionable information
  5. Error Metrics: Track error rates and patterns for analysis

Concurrency Model

Khedra leverages Go's concurrency primitives for efficient parallel processing.

Concurrency Patterns

  1. Worker Pools: Process batches of blocks concurrently with controlled parallelism
  2. Fan-Out/Fan-In: Distribute work to multiple goroutines and collect results
  3. Pipelines: Connect processing stages with channels for streaming data
  4. Context Propagation: Pass cancellation signals through processing chains
  5. Rate Limiting: Control resource usage and external API calls

Resource Management

  1. Connection Pooling: Reuse network connections to blockchain nodes
  2. Goroutine Limiting: Prevent excessive goroutine creation
  3. Memory Budgeting: Control memory usage during large operations
  4. I/O Throttling: Balance disk operations to prevent saturation
  5. Adaptive Concurrency: Adjust parallelism based on system capabilities

Synchronization Techniques

  1. Mutexes: Protect shared data structures from concurrent access
  2. Read/Write Locks: Optimize for read-heavy access patterns
  3. Atomic Operations: Use atomic primitives for simple counters and flags
  4. Channels: Communicate between goroutines and implement synchronization
  5. WaitGroups: Coordinate completion of multiple goroutines

Configuration Wizard

The configuration wizard provides an interactive interface for setting up Khedra.

Wizard Architecture

  1. Screen-Based Flow: Organized as a sequence of screens
  2. Question Framework: Standardized interface for user input
  3. Validation Layer: Real-time validation of user inputs
  4. Navigation System: Forward/backward movement between screens
  5. Help Integration: Contextual help for each configuration option

User Interface Design

  1. Text-Based UI: Terminal-friendly interface with box drawing
  2. Color Coding: Visual cues for different types of information
  3. Navigation Bar: Consistent display of available commands
  4. Progress Indication: Show position in the configuration process
  5. Direct Editing: Option to edit configuration files directly

Implementation Approach

The wizard uses a structured approach to manage screens and user interaction:

type Screen struct {
    Title         string
    Subtitle      string
    Body          string
    Instructions  string
    Replacements  []Replacement
    Questions     []Questioner
    Style         Style
    Current       int
    Wizard        *Wizard
    NavigationBar *NavigationBar
}

type Wizard struct {
    Config    *config.Config
    Screens   []Screen
    Current   int
    History   []int
    // Additional fields for wizard state
}

This design allows for a flexible, extensible configuration process that can adapt to different user needs and configuration scenarios.

Testing Strategy

Khedra employs a comprehensive testing strategy to ensure reliability and correctness.

Testing Levels

  1. Unit Tests: Verify individual functions and components
  2. Integration Tests: Test interaction between components
  3. Service Tests: Validate complete service behavior
  4. End-to-End Tests: Test full application workflows
  5. Performance Tests: Benchmark critical operations

Test Implementation

  1. Mock Objects: Simulate external dependencies
  2. Test Fixtures: Standard data sets for reproducible tests
  3. Property-Based Testing: Generate test cases to find edge cases
  4. Parallel Testing: Run tests concurrently for faster feedback
  5. Coverage Analysis: Track code coverage to identify untested areas

These technical design choices provide Khedra with a solid foundation for reliable, maintainable, and efficient operation across a variety of deployment scenarios and use cases.