The Khedra Book

Khedra (prononced kɛd-ɾɑ) is an all-in-one "long-running" tool for indexing and sharing the Unchained Index and monitoring individual addresses on EVM-compatible blockchains.

The tool creates and shares the Unchained Index which is a permissionless index of "address appearances," including appearances in event logs, execution traces, incoming transactions, modifications to smart contract state, staking or block rewards, prefund allocations and many other locations.

This detailed indexing allows for near-perfect monitoring and notifications of address activity, which leads to many benefits. The benefits include native and ERC-20 account balance histories, address auditing and accounting, and even custom indexing. It works for any address on any chain (as long as you have access to the chain's RPC).

Enjoy!

Please help us improve this software by providing any feedback or suggestions. Contact information and links to our socials are available at our website.

About the Name

The name khedra (prononced kɛd-ɾɑ) is inspired by the Persian word خدمت (khedmat), meaning "service."

In ancient Persian culture, service was considered a noble pursuit, emphasizing dedication, reliability, and humility in action. Drawing from this tradition, the name khedra embodies the essence of a system designed to serve--efficiently, continuously, and with purpose.

Simliar to its counterpart, chifra (derived from the Persian word for "cipher"), the name khedra symbolizes a long-running, dependable processes that tirelessly "serves" the needs of its users.

More technically, khedra is a collection of go routines that:

• creates and publishes the Unchained Index,
• monitors a user-provided customized list of addresses automating caching, notifications, and other ETL processes,
• provides a RESTful API exposing chifra's many data access commands,
• allows for starting, stopping, pausing, and resuming these individual services.

By choosing the name khedra, we honor a legacy of service while committing to building tools that are as resilient, adaptive, and reliable as the meaning behind its name.

User Manual

Overview of Khedra

Khedra is a blockchain indexing and monitoring application designed to provide users with an efficient way to interact with and manage transactional histories for EVM-compatible blockchains. It supports functionalities such as transaction monitoring, address indexing, publishing and pinning the indexes to IPFS and a smart contract, and a RESTful API for accessing data.

Purpose of this Document

This "User's Manual" is designed to help users get started with Khedra, understand its features, and operate the application effectively for both basic and advanced use cases. For a more technical treatment of the software, refer to the Technical Specification.

Intended Audience

This manual is intended for:

• End-users looking to index and monitor blockchain data.
• Developers integrating blockchain data into their applications.
• System administrators managing blockchain-related infrastructure.

Introduction

What is Khedra?

Khedra (pronounced kɛd-ɾɑ) is an all-in-one blockchain indexing and monitoring solution for EVM-compatible blockchains. It provides a comprehensive suite of tools to index, monitor, serve, and share blockchain data in a local-first, privacy-preserving manner.

At its core, Khedra creates and maintains the Unchained Index - a permissionless index of address appearances across blockchain data, including transactions, event logs, execution traces, and more. This detailed indexing enables powerful monitoring capabilities for any address on any supported chain.

Key Features

1. Comprehensive Indexing

Khedra indexes address appearances from multiple sources:

• Transactions (senders and recipients)
• Event logs (topics and data fields)
• Execution traces (internal calls)
• Smart contract state changes
• Block rewards and staking activities
• Genesis allocations

The resulting index allows for lightning-fast lookups of any address's complete on-chain history.

2. Multi-Chain Support

While Ethereum mainnet is required, Khedra works with any EVM-compatible blockchain, including:

• Test networks (Sepolia, etc.)
• Layer 2 solutions (Optimism, Arbitrum)
• Alternative EVMs (Gnosis Chain, etc.)

Each chain requires only a valid RPC endpoint to begin indexing.

3. Modular Service Architecture

Khedra operates through five interconnected services:

• Control Service: Central management API
• Scraper Service: Builds and maintains the Unchained Index
• Monitor Service: Tracks specific addresses of interest
• API Service: Provides data access via REST endpoints
• IPFS Service: Enables distributed sharing of index data

These services can be enabled or disabled independently to suit your needs.

4. Privacy-Preserving Design

Unlike traditional blockchain explorers that track user behavior, Khedra:

• Runs entirely on your local machine
• Never sends queries to third-party servers
• Doesn't track or log your address lookups
• Gives you complete control over your data

5. Distributed Index Sharing

The Unchained Index can be optionally shared and downloaded via IPFS, creating a collaborative network where:

• Users can contribute to building different parts of the index
• New users can download existing index portions instead of rebuilding
• The index becomes more resilient through distribution

Use Cases

Khedra excels in numerous blockchain data scenarios:

• Account History: Track complete transaction and interaction history for any address
• Balance Tracking: Monitor native and ERC-20 token balances over time
• Smart Contract Monitoring: Watch for specific events or interactions with contracts
• Auditing and Accounting: Export complete financial histories for tax or business purposes
• Custom Indexing: Build specialized indices for specific protocols or applications
• Data Analysis: Extract patterns and insights from comprehensive on-chain data

Getting Started

The following chapters will guide you through:

1. Installing and configuring Khedra
2. Understanding the core concepts and architecture
3. Using the various components and services
4. Advanced operations and customization
5. Maintenance and troubleshooting

Whether you're a developer, researcher, trader, or blockchain enthusiast, Khedra provides the tools you need to extract maximum value from blockchain data while maintaining your privacy and autonomy.

Implementation Details

The core features of Khedra described in this introduction are implemented in the following Go files:

• Main Application Structure: The primary application entry point is defined in main.go which initializes the KhedraApp struct defined in app/app.go.

• Service Architecture Implementation:

  • The service framework is initialized in app/action_daemon.go
  • Service definitions and validation are in pkg/types/service.go
  • Service initialization happens in the daemonAction function

• Configuration System:

  • Configuration loading and validation: app/config.go
  • Environment variable processing: pkg/types/apply_env.go

• Multi-Chain Support:

  • Chain configuration and validation: app/action_init_chains.go
  • RPC connection validation: pkg/validate/try_connect.go

Getting Started

Quick Start

Get Khedra running in 3 simple steps:

1. Initialize Khedra

Run the configuration wizard to set up your blockchain connections and services:

khedra init

This interactive wizard will guide you through:

• Setting up blockchain RPC connections (Ethereum, Polygon, etc.)
• Configuring which services to enable (scraper, monitor, API, IPFS)
• Setting up logging and data storage paths

2. Start Khedra

Start all configured services:

khedra daemon

This starts the daemon with all enabled services. The Control Service runs automatically and manages other services.

3. Control Services via API

Once running, manage services through the REST API:

# Check service status
curl http://localhost:8080/api/v1/services

# Start/stop individual services
curl -X POST http://localhost:8080/api/v1/services/scraper/start
curl -X POST http://localhost:8080/api/v1/services/monitor/stop

# Get system status
curl http://localhost:8080/api/v1/status

That's it! Your Khedra instance is now indexing blockchain data and ready for queries.

Detailed Configuration

Khedra runs primarily from a configuration file called config.yaml. This file lives at ~/.khedra/config.yaml by default. If the file is not found, Khedra creates a default configuration in this location.

The config file allows you to specify key parameters for running khedra, including which chains to index/monitor, which services to enable, how detailed to log the processes, and where and how to publish (that is, share) the results.

You may use environment variables to override specific options. This document outlines the configuration file structure, validation rules, default values, and environment variable usage.

Installation

1. Download, build, and test khedra:

   git clone https://github.com/TrueBlocks/trueblocks-khedra.git
   cd trueblocks-khedra
   go build -o khedra main.go
   ./khedra version
   

   You should get something similar to khedra v4.0.0-release.

2. You may edit the config file with:

   ./khedra config edit
   

   Modify the file according to your requirements (see below).

   The minimal configuration needed is to provide a valid RPC to Ethereum mainnet. (All configurations require access to Ethereum mainnet.)

   You may configure as many other EVM-compatible chains (each with its own RPC) as you like.

3. Use the Wizard:

   You may also use the khedra wizard to create a configuration file. The wizard will prompt you for the required information and generate a config.yaml file.

   ./khedra init
   

4. Location of the configuration file:

   By default, the config file resides at ~/.khedra/config.yaml. (The folder and the file will be created if it does not exist).

   You may, however, place a config.yaml file in the current working folder (the folder from which you run khedra). If found locally, this configuration file will dominate. This allows for running multiple instances of the software concurrently.

Advanced Configuration Examples

Production Deployment Configuration

For production environments with high availability and performance requirements:

general:
  indexPath: "/var/lib/khedra/index"     # Fast SSD storage
  cachePath: "/var/lib/khedra/cache"     # Local SSD cache
  dataDir: "/var/lib/khedra"             # Dedicated data directory

chains:
  mainnet:
    rpcs:
      - "https://eth-mainnet.alchemyapi.io/v2/YOUR_PREMIUM_KEY"
      - "https://mainnet.infura.io/v3/YOUR_PROJECT_ID"
      - "https://rpc.ankr.com/eth"              # Fallback
      - "https://ethereum.publicnode.com"       # Additional fallback
    enabled: true

  polygon:
    rpcs:
      - "https://polygon-mainnet.g.alchemy.com/v2/YOUR_KEY"
      - "https://polygon-rpc.com"
    enabled: true

  arbitrum:
    rpcs:
      - "https://arb-mainnet.g.alchemy.com/v2/YOUR_KEY"
      - "https://arb1.arbitrum.io/rpc"
    enabled: true

services:
  scraper:
    enabled: true
    sleep: 5                     # Aggressive indexing
    batchSize: 2000             # Large batches for efficiency

  monitor:
    enabled: true
    sleep: 5                    # Fast monitoring
    batchSize: 500

  api:
    enabled: true
    port: 8080

  ipfs:
    enabled: true
    port: 8083

logging:
  folder: "/var/log/khedra"     # System log directory
  filename: "khedra.log"
  toFile: true                  # Always log to file in production
  level: "info"                 # Balanced logging
  maxSize: 100                  # Larger log files
  maxBackups: 10               # More backup files
  maxAge: 90                   # Longer retention
  compress: true               # Compress old logs

Multi-Chain Development Environment

For developers working with multiple blockchain networks:

general:
  indexPath: "~/.khedra/dev/index"
  cachePath: "~/.khedra/dev/cache"

chains:
  mainnet:
    rpcs:
      - "https://eth-mainnet.alchemyapi.io/v2/YOUR_DEV_KEY"
    enabled: true

  sepolia:
    rpcs:
      - "https://eth-sepolia.g.alchemy.com/v2/YOUR_KEY"
      - "https://sepolia.infura.io/v3/YOUR_PROJECT_ID"
    enabled: true

  polygon:
    rpcs:
      - "https://polygon-mumbai.g.alchemy.com/v2/YOUR_KEY"
    enabled: true

  optimism:
    rpcs:
      - "https://opt-goerli.g.alchemy.com/v2/YOUR_KEY"
    enabled: true

  arbitrum:
    rpcs:
      - "https://arb-goerli.g.alchemy.com/v2/YOUR_KEY"
    enabled: true

  base:
    rpcs:
      - "https://base-goerli.g.alchemy.com/v2/YOUR_KEY"
    enabled: true

services:
  scraper:
    enabled: true
    sleep: 15                   # Moderate speed for development
    batchSize: 500

  monitor:
    enabled: true               # Enable for testing monitoring features
    sleep: 30
    batchSize: 100

  api:
    enabled: true
    port: 8080

  ipfs:
    enabled: false              # Disable to reduce resource usage

logging:
  folder: "~/.khedra/dev/logs"
  filename: "khedra-dev.log"
  toFile: true
  level: "debug"               # Verbose logging for development
  maxSize: 10
  maxBackups: 5
  maxAge: 7                    # Shorter retention for dev
  compress: false              # No compression for easier reading

High-Availability Load-Balanced Setup

Configuration for running multiple Khedra instances behind a load balancer:

# Instance 1: Primary indexing node
general:
  indexPath: "/shared/khedra/index"      # Shared storage
  cachePath: "/local/khedra/cache1"      # Local cache per instance

chains:
  mainnet:
    rpcs:
      - "https://eth-mainnet-primary.alchemyapi.io/v2/KEY1"
      - "https://eth-mainnet-backup.infura.io/v3/PROJECT1"
    enabled: true

services:
  scraper:
    enabled: true              # Primary indexer
    sleep: 5
    batchSize: 2000

  monitor:
    enabled: false             # Disabled on indexing nodes

  api:
    enabled: false             # Dedicated API nodes

  ipfs:
    enabled: true              # IPFS on indexing nodes
    port: 8083

logging:
  folder: "/var/log/khedra"
  filename: "khedra-indexer-1.log"
  toFile: true
  level: "info"
---

# Instance 2: API-only node
general:
  indexPath: "/shared/khedra/index"      # Same shared storage
  cachePath: "/local/khedra/cache2"      # Different local cache

chains:
  mainnet:
    rpcs:
      - "https://eth-mainnet-api.alchemyapi.io/v2/KEY2"
    enabled: true

services:
  scraper:
    enabled: false             # No indexing on API nodes

  monitor:
    enabled: true              # Monitoring on API nodes
    sleep: 10
    batchSize: 200

  api:
    enabled: true              # Primary function
    port: 8080

  ipfs:
    enabled: false             # Not needed on API nodes

logging:
  folder: "/var/log/khedra"
  filename: "khedra-api-2.log"
  toFile: true
  level: "warn"               # Less verbose for API nodes

Resource-Constrained Environment

Configuration for running Khedra on limited hardware (VPS, Raspberry Pi, etc.):

general:
  indexPath: "~/.khedra/index"
  cachePath: "~/.khedra/cache"

chains:
  mainnet:
    rpcs:
      - "https://ethereum.publicnode.com"    # Free RPC
      - "https://rpc.ankr.com/eth"           # Backup free RPC
    enabled: true

  # Only enable additional chains if needed
  sepolia:
    rpcs:
      - "https://ethereum-sepolia.publicnode.com"
    enabled: false               # Disabled to save resources

services:
  scraper:
    enabled: true
    sleep: 60                   # Very conservative indexing
    batchSize: 50               # Small batches

  monitor:
    enabled: false              # Disable to save resources
    sleep: 300
    batchSize: 10

  api:
    enabled: true
    port: 8080

  ipfs:
    enabled: false              # Disable to save bandwidth/storage

logging:
  folder: "~/.khedra/logs"
  filename: "khedra.log"
  toFile: false                 # Console only to save disk space
  level: "warn"                # Minimal logging
  maxSize: 5                   # Small log files
  maxBackups: 2                # Minimal retention
  maxAge: 7
  compress: true

Security-Focused Configuration

Configuration with enhanced security for sensitive environments:

general:
  indexPath: "/encrypted/khedra/index"   # Encrypted storage
  cachePath: "/encrypted/khedra/cache"

chains:
  mainnet:
    rpcs:
      - "https://your-private-node.internal:8545"  # Private RPC node
    enabled: true

services:
  scraper:
    enabled: true
    sleep: 10
    batchSize: 1000

  monitor:
    enabled: true
    sleep: 15
    batchSize: 100

  api:
    enabled: true
    port: 8080                  # Consider using reverse proxy with TLS

  ipfs:
    enabled: false              # Disable external data sharing

logging:
  folder: "/secure/logs/khedra"
  filename: "khedra.log"
  toFile: true
  level: "info"
  maxSize: 50
  maxBackups: 20              # Extended retention for audit
  maxAge: 365                 # Long retention for compliance
  compress: true

# Environment variables for sensitive data:
# TB_KHEDRA_CHAINS_MAINNET_RPCS="https://user:pass@private-node:8545"
# TB_KHEDRA_API_AUTH_TOKEN="your-secure-api-token"

Testing and CI/CD Configuration

Configuration optimized for automated testing environments:

general:
  indexPath: "./test-data/index"
  cachePath: "./test-data/cache"

chains:
  sepolia:                      # Use testnet for testing
    rpcs:
      - "https://ethereum-sepolia.publicnode.com"
    enabled: true

  mainnet:
    rpcs:
      - "https://ethereum.publicnode.com"
    enabled: false              # Disabled for testing

services:
  scraper:
    enabled: true
    sleep: 30                   # Conservative for CI resources
    batchSize: 100

  monitor:
    enabled: true               # Test monitoring functionality
    sleep: 60
    batchSize: 50

  api:
    enabled: true
    port: 8080

  ipfs:
    enabled: false              # Not needed for testing

logging:
  folder: "./test-logs"
  filename: "khedra-test.log"
  toFile: true
  level: "debug"               # Verbose for troubleshooting tests
  maxSize: 10
  maxBackups: 3
  maxAge: 1                    # Short retention for CI
  compress: false              # Easier to read in CI logs

Understanding Khedra

Core Concepts

The Unchained Index

The foundation of Khedra is the Unchained Index - a specialized data structure that maps blockchain addresses to their appearances in blockchain data. Think of it as a reverse index: while a blockchain explorer lets you look up a transaction and see which addresses were involved, the Unchained Index lets you look up an address and see all transactions where it appears.

The index captures appearances from multiple sources:

• External Transactions: Direct sends and receives
• Internal Transactions: Contract-to-contract calls (from traces)
• Event Logs: Events emitted by smart contracts
• State Changes: Modifications to contract storage
• Special Appearances: Block rewards, validators, etc.

What makes this particularly powerful is that the index includes trace-derived appearances - meaning it captures internal contract interactions that normal blockchain explorers miss.

Address Appearances

An "appearance" in Khedra means any instance where an address is referenced in blockchain data. Each appearance record contains:

• The address that appeared
• The block number where it appeared
• The transaction index within that block
• Additional metadata about the appearance type

These compact records allow Khedra to quickly answer the fundamental question: "Where does this address appear in the blockchain?"

Local-First Architecture

Khedra operates as a "local-first" application, meaning:

1. All data processing happens on your local machine
2. Your queries never leave your computer
3. You maintain complete ownership of your data
4. The application continues to work without internet access

This approach maximizes privacy and resilience while minimizing dependency on external services.

Distributed Collaboration

While Khedra is local-first, it also embraces distributed collaboration through IPFS integration:

• The Unchained Index can be shared and downloaded in chunks
• Users can contribute to different parts of the index
• New users can bootstrap quickly by downloading existing index portions
• The system becomes more resilient as more people participate

This creates a hybrid model that preserves privacy while enabling community benefits.

System Architecture

Service Components

Khedra is organized into five core services:

1. Control Service

   • Central Management Hub: Provides unified control interface for all Khedra services
   • Service Operations: Start, stop, restart, pause, and resume any service via API
   • Health Monitoring: Real-time status monitoring and health checks for all services
   • Configuration Management: Runtime configuration updates and validation
   • External Integration: Enables external tools to manage Khedra programmatically
   • Always Enabled: The Control Service is always active and cannot be disabled

2. Scraper Service

   • Processes blockchain data to build the Unchained Index
   • Extracts address appearances from blocks, transactions, and traces
   • Works in configurable batches with adjustable sleep intervals

3. Monitor Service

   • Tracks specific addresses of interest
   • Provides notifications for address activities
   • Maintains focused indices for monitored addresses

4. API Service

   • Exposes data through REST endpoints (defined here: API Docs)
   • Provides query interfaces for the index and monitors
   • Enables integration with other tools and services

5. IPFS Service

   • Facilitates distributed sharing of index data
   • Handles publishing and retrieving chunks via IPFS
   • Enables collaborative index building

Data Flow and Service Interactions

Here's how data flows through the Khedra system and how services interact:

graph TD
    A[RPC Endpoints] --> B[Scraper Service]
    B --> C[Unchained Index]
    B --> D[IPFS Service]
    C --> E[API Service]
    C --> F[Monitor Service]
    G[Control Service] --> B
    G --> F
    G --> E
    G --> D
    E --> H[External Applications]
    F --> I[Address Notifications]
    D --> J[IPFS Network]

    style G fill:#ff9999
    style B fill:#99ccff
    style F fill:#99ffcc
    style E fill:#ffcc99
    style D fill:#cc99ff

Detailed Service Interactions

1. Control Service (Central Hub)

   • Manages lifecycle of all other services
   • Coordinates startup/shutdown sequences
   • Monitors health and performance of all services
   • Provides management API accessible to external tools

2. Data Processing Flow:

   • Scraper retrieves blockchain data from RPC endpoints
   • Address appearances are extracted and stored in the Unchained Index
   • Monitor service checks new blocks against watched address lists
   • API service provides query access to indexed data
   • IPFS service optionally shares index chunks with the network

3. Service Dependencies:

   • Monitor depends on Scraper for real-time block data
   • API depends on Unchained Index created by Scraper
   • IPFS can operate independently but enhances Scraper functionality
   • All services depend on Control service for management operations

Independent vs Coordinated Services

Services that can run independently:

• API Service: Can serve existing index data without other services
• IPFS Service: Can share/retrieve data independently of indexing
• Control Service: Always runs and manages others

Services that work better together:

• Scraper + Monitor: Monitor gets real-time data from Scraper
• Scraper + IPFS: IPFS can automatically share new index chunks
• Scraper + API: API serves fresh data as Scraper creates it

Service Startup Order

Services start in this coordinated sequence:

1. Control Service (first - manages others)
2. IPFS Service (if enabled - provides infrastructure)
3. Scraper Service (if enabled - begins indexing)
4. API Service (if enabled - serves data)
5. Monitor Service (last - monitors real-time data)

This order ensures dependencies are available when each service starts.

Using Khedra

This chapter covers the practical aspects of working with Khedra once it's installed and configured.

Understanding Khedra's Command Structure

Khedra provides a streamlined set of commands designed to index, monitor, serve, and share blockchain data:

NAME:
   khedra - A tool to index, monitor, serve, and share blockchain data

USAGE:
   khedra [global options] command [command options]

VERSION:
   v5.1.0

COMMANDS:
   init     Initializes Khedra
   daemon   Runs Khedras services
   config   Manages Khedra configuration
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version

Getting Started with Khedra

Initializing Khedra

Before using Khedra, you need to initialize it. This sets up the necessary data structures and configurations:

khedra init

During initialization, Khedra will:

• Set up its directory structure
• Configure initial settings
• Prepare the system for indexing blockchain data

Managing Configuration

To view or modify Khedra's configuration:

khedra config [show | edit]

The configuration command allows you to:

• View current settings
• Update connection parameters
• Adjust service behaviors
• Configure chain connections

Running Khedra's Services

To start Khedra's daemon services:

khedra daemon

This command:

• Starts the indexing service
• Enables the API server if configured
• Processes monitored addresses
• Handles data serving capabilities

You can use various options with the daemon command to customize its behavior. For detailed options:

khedra daemon --help

Common Workflows

Basic Setup

1. Install Khedra using the installation instructions

2. Initialize the system:

   khedra init
   

3. Configure as needed:

   khedra config edit
   

4. Start the daemon services:

   khedra daemon
   

Checking System Status

You can view the current status of Khedra by examining the daemon process:

curl http://localhost:8338/status | jq

• Note: The port for the above command defaults to one of 8338, 8337, 8336 or 8335 in that order whichever one is first available. If none of those ports is available, the daemon will not start.

Accessing the Data API

If so configured, when the daemon is running, it provides API endpoints for accessing blockchain data. The default configuration typically serves on:

curl http://localhost:8080/status

See the API documentation for more details on available endpoints and their usage.

Getting Help

Each command provides detailed help information. To access help for any command:

khedra [command] --help

For general help:

khedra --help

Version Information

To check which version of Khedra you're running:

khedra --version

Advanced Usage

For more detailed information about advanced operations and configurations, please refer to the documentation for each specific command:

khedra init --help
khedra daemon --help
khedra config --help

The next chapter covers advanced operations for users who want to maximize Khedra's capabilities.

Implementation Details

The command structure and functionality described in this section are implemented in these Go files:

Core Command Structure

• CLI Framework: app/cli.go - Defines the top-level command structure using the urfave/cli package

Command Implementations

• Init Command: app/action_init.go - Handles initialization of Khedra

  • Welcome Screen: app/action_init_welcome.go
  • General Settings: app/action_init_general.go
  • Services Config: app/action_init_services.go
  • Chain Config: app/action_init_chains.go
  • Summary Screen: app/action_init_summary.go

• Daemon Command: app/action_daemon.go - Implements the daemon service that runs the various Khedra services

• Config Commands:

  • app/action_config_show.go - Displays current configuration
  • app/action_config_edit.go - Opens configuration in editor

• Version Command: app/action_version.go - Shows version information

Helper Functions

• Command Line Arguments: app/args.go - Processes command line arguments
• Help System: app/help_system.go - Provides help text for commands

Maintenance and Troubleshooting

This chapter covers routine maintenance tasks and solutions to common issues you might encounter when using Khedra.

Routine Maintenance

Regular Updates

To keep Khedra running smoothly, periodically check for and install updates:

# Check current version
khedra version

# Update to the latest version
go get -u github.com/TrueBlocks/trueblocks-khedra/v5

# Rebuild and install
cd <path_for_khedra_github_repo>
git pull --recurse-submodules
go build -o bin/khedra main.go
./bin/khedra version

Log Rotation

Khedra automatically rotates logs based on your configuration, but you should periodically check log usage:

# Check log directory size
du -sh ~/.khedra/logs

# List log files
ls -la ~/.khedra/logs

If logs are consuming too much space, adjust your logging configuration:

logging:
  maxSize: 10      # Maximum size in MB before rotation
  maxBackups: 5    # Number of rotated files to keep
  maxAge: 30       # Days to keep rotated logs
  compress: true   # Compress rotated logs

Index Verification

Periodically verify the integrity of your Unchained Index:

chifra chunks index --check --chain <chain_name>

This checks for any gaps or inconsistencies in the index and reports issues.

Cache Management

You may check on the cache size and prune old caches (by hand) to free up space:

# Check cache size
chifra status --verbose

Troubleshooting

Common Issues and Solutions

Service Won't Start

Symptoms: A service fails to start or immediately stops.

Solutions:

1. Check the logs for error messages:

   tail -n 100 ~/.khedra/logs/khedra.log
   

2. Verify the service's port isn't in use by another application:

   lsof -i :<port_number>
   

3. Ensure the RPC endpoints are accessible:

   chifra status
   

4. Try starting with verbose logging:

   TB_KHEDRA_LOGGING_LEVEL=debug TB_KHEDRA_LOGGING_TOFILE=true khedra start
   

Service-Specific Troubleshooting

Scraper Service Issues

Symptoms: Scraper service fails to start, stops unexpectedly, or indexes slowly.

Common Issues and Solutions:

1. RPC Connection Failures:

   # Test RPC connectivity
   curl -X POST -H "Content-Type: application/json" \
     --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
     http://your-rpc-endpoint
   
   # Check RPC provider limits
   grep -i "rate limit\|too many requests" ~/.khedra/logs/khedra.log
   

2. Batch Size Optimization:

   # For fast RPC endpoints
   services:
     scraper:
       batchSize: 2000
       sleep: 5
   
   # For slower/limited RPC endpoints  
   services:
     scraper:
       batchSize: 100
       sleep: 30
   

3. Memory Issues:

   # Monitor scraper memory usage
   ps -o pid,vsz,rss,comm -p $(pgrep -f "scraper")
   
   # Reduce batch size if memory usage is high
   

4. Scraper-Specific Log Analysis:

   # Filter scraper logs
   grep "scraper" ~/.khedra/logs/khedra.log | tail -50
   
   # Look for specific errors
   grep -E "error|failed|timeout" ~/.khedra/logs/khedra.log | grep scraper
   

Monitor Service Issues

Symptoms: Monitor service doesn't detect address activity or sends duplicate notifications.

Common Issues and Solutions:

1. No Monitored Addresses:

   # Check if addresses are properly configured
   chifra list --monitors
   
   # Add addresses to monitor
   chifra monitors --addrs 0x742d35Cc6634C0532925a3b844Bc454e4438f44e
   

2. Monitor Service Dependencies:

   # Ensure scraper is running for real-time monitoring
   curl http://localhost:8080/api/v1/services/scraper
   
   # Check if index is up to date
   chifra status --index
   

3. Monitor Configuration Issues:

   services:
     monitor:
       enabled: true
       sleep: 12        # Check every 12 seconds
       batchSize: 100   # Process 100 addresses at once
   

4. Monitor-Specific Logs:

   # Filter monitor logs
   grep "monitor" ~/.khedra/logs/khedra.log | tail -50
   
   # Check for address activity detection
   grep -i "activity\|appearance" ~/.khedra/logs/khedra.log
   

API Service Issues

Symptoms: API service returns errors, timeouts, or incorrect data.

Common Issues and Solutions:

1. Port Conflicts:

   # Check if API port is available
   lsof -i :8080
   
   # Change API port if needed
   export TB_KHEDRA_SERVICES_API_PORT=8081
   

2. API Performance Issues:

   # Test API response time
   time curl http://localhost:8080/status
   
   # Check for slow queries
   grep -E "slow|timeout" ~/.khedra/logs/khedra.log | grep api
   

3. API Authentication Issues:

   # Verify API is accessible
   curl -v http://localhost:8080/api/v1/services
   
   # Check for auth-related errors
   grep -i "auth\|unauthorized" ~/.khedra/logs/khedra.log
   

4. Data Consistency Issues:

   # Compare API data with direct index queries
   chifra list 0x742d35Cc6634C0532925a3b844Bc454e4438f44e
   curl http://localhost:8080/api/v1/list/0x742d35Cc6634C0532925a3b844Bc454e4438f44e
   

IPFS Service Issues

Symptoms: IPFS service fails to start, can't connect to network, or sharing fails.

Common Issues and Solutions:

1. IPFS Daemon Issues:

   # Check IPFS daemon status
   ps aux | grep ipfs
   
   # Restart IPFS if needed
   curl -X POST http://localhost:8080/api/v1/services/ipfs/restart
   

2. IPFS Port Conflicts:

   # Check IPFS ports
   lsof -i :5001  # IPFS API port
   lsof -i :4001  # IPFS swarm port
   
   # Configure different IPFS port
   export TB_KHEDRA_SERVICES_IPFS_PORT=5002
   

3. IPFS Network Connectivity:

   # Test IPFS connectivity
   curl http://localhost:5001/api/v0/id
   
   # Check peer connections
   curl http://localhost:5001/api/v0/swarm/peers
   

4. Index Sharing Issues:

   # Check IPFS pinning status
   curl http://localhost:5001/api/v0/pin/ls
   
   # Verify index chunk uploads
   grep -i "ipfs\|pin" ~/.khedra/logs/khedra.log
   

Control Service Issues

Symptoms: Cannot manage other services via API or CLI commands fail.

Common Issues and Solutions:

1. Control Service Availability:

   # Verify control service is running
   curl http://localhost:8080/api/v1/services
   
   # Check control service logs
   grep "control" ~/.khedra/logs/khedra.log
   

2. Service Management Failures:

   # Test individual service control
   curl -X POST http://localhost:8080/api/v1/services/scraper/status
   
   # Check for permission issues
   grep -i "permission\|access denied" ~/.khedra/logs/khedra.log
   

3. Configuration Issues:

   # Verify control service configuration
   khedra config show | grep -A5 -B5 control
   
   # Test configuration validation
   khedra config validate
   

Log Analysis

Khedra's logs are your best resource for troubleshooting. Here's how to use them effectively:

# View recent log entries
tail -f ~/.khedra/logs/khedra.log

# Search for error messages
grep -i error ~/.khedra/logs/khedra.log

# Find logs related to a specific service
grep "scraper" ~/.khedra/logs/khedra.log

# Find logs related to a specific address
grep "0x742d35Cc6634C0532925a3b844Bc454e4438f44e" ~/.khedra/logs/khedra.log

Getting Help

If you encounter issues you can't resolve:

1. Check the Khedra GitHub repository for known issues
2. Search the discussions forum for similar problems
3. Submit a detailed issue report including:
   • Khedra version (khedra version)
   • Relevant log extracts
   • Steps to reproduce the problem
   • Your configuration (with sensitive data redacted)

Regular maintenance and prompt troubleshooting will keep your Khedra installation running smoothly and efficiently.

Implementation Details

The maintenance and troubleshooting procedures described in this document are implemented in several key files:

Service Management

• Service Lifecycle Management: app/action_daemon.go - Contains the core service management code that starts, stops, and monitors services
• Service Health Checks: Service status monitoring is implemented in the daemon action function

RPC Connection Management

• RPC Endpoint Testing: pkg/validate/try_connect.go - Contains the TestRpcEndpoint function used to verify endpoints are functioning correctly
• RPC Validation: app/has_valid_rpc.go - Implements validation logic for RPC endpoints

Logging System

• Log Configuration: Defined in the Logging struct in pkg/types/general.go which handles log rotation and management
• Logger Implementation: Custom logger in pkg/types/custom_logger.go that provides structured logging capabilities

Error Recovery

The troubleshooting techniques described are supported by robust error handling throughout the codebase, especially in:

• Service error handling: Found in the daemon action function
• Validation error reporting: Implemented in the validation framework
• Index management functions: For identifying and fixing gaps in the index

Wizard Screen Documentation

Introduction

Khedra's configuration wizard provides a streamlined, interactive way to set up your installation. Rather than manually editing the config.yaml file, the wizard walks you through each configuration section with clear explanations and validation.

User Interface Features

The wizard provides several helpful features:

• Keyboard Navigation: Use arrow keys and shortcuts to navigate
• Contextual Help: Press 'h' on any screen for detailed documentation
• Editor Integration: Press 'e' to directly edit configuration files
• Validation: Input is checked for correctness before proceeding
• Visual Cues: Consistent layout with clear indicators for navigation options

Using the Wizard

Start the Wizard with:

khedra init

Implementation Details

The configuration wizard described in this document is implemented through a package of Go files in the pkg/wizard directory:

Core Wizard Framework

• Main Wizard Structure: pkg/wizard/wizard.go - Defines the Wizard struct and methods for managing the wizard state, navigation between screens, and execution flow

• Screen Component: pkg/wizard/screen.go - Implements the Screen struct representing individual wizard pages with questions and display logic

• Question Framework: pkg/wizard/question.go - Provides the Question struct and interface for gathering and validating user input

• User Interface:

  • pkg/wizard/display.go - Handles rendering screens and questions
  • pkg/wizard/style.go - Controls visual styling of the wizard interface
  • pkg/boxes/boxes.go - Implements the ASCII box drawing for wizard screens

• Navigation:

  • pkg/wizard/navigation.go - Implements the navigation bar and controls
  • pkg/wizard/shortcuts.go - Handles keyboard shortcuts

Wizard Screen Implementations

The specific wizard screens visible in the user interface are implemented in these files:

• Welcome Screen: app/action_init_welcome.go
• General Settings: app/action_init_general.go
• Services Config: app/action_init_services.go
• Chain Config: app/action_init_chains.go
• Summary Screen: app/action_init_summary.go

Integration with Configuration System

The wizard integrates with the configuration system through:

• Configuration Loading: In the ReloaderFn function passed to the wizard
• Configuration Validation: Through the validation functions for each input field
• Configuration Saving: In the final step of the wizard workflow

The wizard framework uses a screen-based approach with forward/backward navigation, real-time validation, and contextual help, exactly as described in this document.

Welcome Screen

Function

┌──────────────────────────────────────────────────────────────────────────────┐
│ ╔═══════════════════════════════════════════════════════╗                    │
│ ║                     KHEDRA WIZARD                     ║                    │
│ ║                                                       ║                    │
│ ║   Index, monitor, serve, and share blockchain data.   ║                    │
│ ╚═══════════════════════════════════════════════════════╝                    │
│                                                                              │
│ Welcome to Khedra, the world's only local-first indexer/monitor for          │
│ EVM blockchains. This wizard will help you configure Khedra. There are       │
│ three groups of settings: General, Services, and Chains.                     │
│                                                                              │
│ Type "q" or "quit" to quit, "b" or "back" to return to a previous screen,    │
│ "h" or "help" to get more information, or "e" to edit the file directly.     │
│                                                                              │
│ Press enter to continue.                                                     │
│                                                                              │
│ Keyboard: [h] Help [q] Quit [b] Back [enter] Continue                        │
└──────────────────────────────────────────────────────────────────────────────┘

Purpose

• Introduces the wizard to the user
• Orients the user to the configuration process
• Provides clear navigation instructions

Navigation Options

• Enter: Proceed to the next screen
• h/help: Open browser with documentation
• q/quit: Exit the wizard
• b/back: Return to previous screen
• e/edit: Edit configuration file directly

You may directly edit the configuration from any screen by typing e or edit. This will open the configuration file in the user's preferred text editor (defined by the EDITOR environment variable).

The welcome screen serves as the entry point to the configuration process, designed to be approachable while providing clear direction on how to proceed.

General Configuration Screen

┌──────────────────────────────────────────────────────────────────────────────┐
│ General Settings                                                             │
├──────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│ The General group of options controls where Khedra stores the Unchained      │
│ Index and its caches. It also helps you choose a download strategy for       │
│ the index and helps you set up Khedra's logging options.                     │
│                                                                              │
│ Choose your folders carefully. The index and logs can get quite large        │
│ depending on the configuration. As always, type "help" to get more           │
│ information.                                                                 │
│                                                                              │
│ You may use $HOME or ~/ in your paths to refer to your home directory.       │
│                                                                              │
│ Press enter to continue.                                                     │
│                                                                              │
│ Keyboard: [h] Help [q] Quit [b] Back [enter] Continue                        │
└──────────────────────────────────────────────────────────────────────────────┘

Purpose

• Allows users to configure high-level application settings
• Sets up crucial file paths for data storage
• Configures logging behavior

Key Features

• Define the main data folder location with path expansion support
• Configure index download and update strategies
• Set up logging preferences for troubleshooting
• Options for path expansion (supporting $HOME and ~/ notation)
• Disk space requirement warnings
• Input validation for directory existence and write permissions

Configuration Options

The General Settings screen presents these key configuration options:

1. Data Folder: Where Khedra stores all index and cache data

   • Default: ~/.khedra/data
   • Must be a writable location with sufficient disk space

2. Index Download Strategy:

   • IPFS-first: Prioritize downloading from the distributed network
   • Local-first: Prioritize building the index locally
   • Hybrid: Balance between downloading and local building

Services Configuration Screen

┌──────────────────────────────────────────────────────────────────────────────┐
│ Services Settings                                                            │
├──────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│ Khedra provides five services. The first, "control," exposes endpoints to    │
│ control the other four: "scrape", "monitor", "api", and "ipfs".              │
│                                                                              │
│ You may disable/enable any combination of services, but at least one must    │
│ be enabled.                                                                  │
│                                                                              │
│ The next few screens will allow you to configure each service.               │
│                                                                              │
│                                                                              │
│                                                                              │
│ Press enter to continue.                                                     │
│                                                                              │
│ Keyboard: [h] Help [q] Quit [b] Back [enter] Continue                        │
└──────────────────────────────────────────────────────────────────────────────┘

Purpose

• Enables users to select and configure Khedra's core services
• Explains the relationship between the services
• Ensures at least one service is enabled for proper functionality

Available Services

Khedra offers five core services that work together:

1. Control Service: Management endpoints for the other services

   • Always enabled by default
   • Provides a central API for managing other services

2. Scraper Service: Builds and maintains the Unchained Index

   • Processes blocks to extract address appearances
   • Configurable batch size and sleep interval

3. Monitor Service: Tracks specific addresses of interest

   • Provides notifications for address activities
   • Configurable batch size and sleep interval

4. API Service: REST API for data access

   • Configurable port number
   • Provides endpoints for querying the index and monitors

5. IPFS Service: Distributed data sharing

   • Enables sharing and downloading index data
   • Configurable port number

Configuration Parameters

For each service, you can configure:

• Enabled/Disabled: Toggle the service on or off
• Port numbers: For services that expose network endpoints
• Batch size: Number of blocks processed in one batch (for scraper/monitor)
• Sleep interval: Time to wait between batches (for scraper/monitor)
• Resource limits: Memory and CPU constraints

Chain Settings Screen

┌──────────────────────────────────────────────────────────────────────────────┐
│ Chain Settings                                                               │
├──────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│ Khedra works with the Ethereum mainnet chain and any EVM-compatible          │
│ blockchain. Each chain requires at least one RPC endpoint URL and a          │
│ chain name.                                                                  │
│                                                                              │
│ Ethereum mainnet must be configured even if other chains are enabled.        │
│ The format of an RPC endpoint is protocol://host:port. For example:          │
│ http://localhost:8545 or https://mainnet.infura.io/v3/YOUR-PROJECT-ID.       │
│                                                                              │
│ The next few screens will help you configure your chains.                    │
│                                                                              │
│ Press enter to continue.                                                     │
│                                                                              │
│ Keyboard: [h] Help [q] Quit [b] Back [enter] Continue                        │
└──────────────────────────────────────────────────────────────────────────────┘

Purpose

• Configures blockchain connections for indexing and monitoring
• Ensures proper RPC endpoint setup for each chain
• Explains the requirement for Ethereum mainnet

Key Features

• Multiple chain support with standardized naming
• RPC endpoint configuration and validation
• Clear explanation of requirements and format

Chain Configuration

The chains configuration screen guides you through setting up:

1. Ethereum Mainnet (Required)

   • At least one valid RPC endpoint
   • Used for core functionality and the Unchained Index

2. Additional EVM Chains (Optional)

   • Sepolia, Gnosis, Optimism, and other EVM-compatible chains
   • Each requires at least one RPC endpoint
   • Enable/disable option for each chain

RPC Endpoint Requirements

For each chain, you must provide:

• A valid RPC URL in the format protocol://host:port
• Proper authentication details if required (e.g., Infura project ID)
• Endpoints with sufficient capabilities for indexing (archive nodes recommended)

THIS TEXT NEEDS TO BE REVIEWED.

Validation Checks

The wizard performs these validations on each RPC endpoint:

• URL format validation
• Connection test to verify the endpoint is reachable
• Chain ID verification to ensure the endpoint matches the selected chain
• API method support check for required JSON-RPC methods THIS TEXT NEEDS TO BE REVIEWED.

Implementation

The chain configuration uses the Screen struct with specialized validation for RPC endpoints. The wizard prioritizes setting up Ethereum mainnet first, then offers options to configure additional chains as needed.

For each chain, the wizard walks through enabling the chain, configuring RPC endpoints, and validating the connection before proceeding to the next chain.

Summary Screen

┌──────────────────────────────────────────────────────────────────────────────┐
│ Summary                                                                      │
├──────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│ Question: Would you like to edit the config by hand?                         │
│ Current:  no                                                                 │
│                                                                              │
│                                                                              │
│                                                                              │
│                                                                              │
│                                                                              │
│                                                                              │
│                                                                              │
│                                                                              │
│ Press enter to finish the wizard. ("b"=back, "h"=help)                       │
│                                                                              │
│ Keyboard: [h] Help [q] Quit [b] Back [e] Edit [enter] Finish                 │
└──────────────────────────────────────────────────────────────────────────────┘

Purpose

• Provides a review of all configured settings
• Offers a final chance to make adjustments before saving
• Summarizes the configuration in a clear, readable format

Configuration Summary Display

The summary screen presents the configuration organized by section:

1. General Settings

   • Data folder location
   • Download strategy
   • Logging configuration

2. Services Configuration

   • Enabled/disabled status for each service
   • Port numbers and key parameters
   • Resource allocations

3. Chain Settings

   • Configured blockchains
   • RPC endpoints
   • Chain-specific settings

Final Options

From the summary screen, you can:

1. Finish: Accept the configuration and write it to the config file
2. Edit: Open the configuration in a text editor for manual changes
3. Back: Return to previous screens to make adjustments
4. Help: Access documentation about configuration options
5. Quit: Exit without saving changes

When the user chooses to finish, the wizard writes the configuration to ~/.khedra/config.yaml by default, or to an alternative location if specified during the process.

If the user chooses to edit the file directly, the wizard will invoke the system's default editor (or the editor specified in the EDITOR environment variable) and then reload the configuration after editing.

Appendices

Glossary of Terms

• EVM: Ethereum Virtual Machine, the runtime environment for smart contracts in Ethereum and similar blockchains.
• RPC: Remote Procedure Call, a protocol allowing the application to communicate with blockchain nodes.
• Indexing: The process of organizing blockchain data for fast and efficient retrieval.
• IPFS: InterPlanetary File System, a decentralized storage system for sharing and retrieving data.

Frequently Asked Questions (FAQ)

1. What chains are supported by Khedra?

Khedra supports Ethereum mainnet and other EVM-compatible chains such as Sepolia and Gnosis. Additional chains can be added by configuring the TB_NODE_CHAINS environment variable.

2. Do I need an RPC endpoint for every chain?

Yes, each chain you want to index or interact with requires a valid RPC endpoint specified in the .env file.

3. Can I run Khedra without IPFS?

Yes, IPFS integration is optional and can be enabled or disabled using the --ipfs command-line option.

References and Further Reading

• TrueBlocks GitHub Repository
• TrueBlocks Official Website
• Ethereum Developer Documentation
• IPFS Documentation

Index

• Address Monitoring:

  • Documentation: Chapter 4, Section "Monitoring Addresses"
  • Implementation: app/action_daemon.go (Monitor service initialization and MonitorsOptions struct)

• API Access:

  • Documentation: Chapter 4, Section "Accessing the REST API"
  • Implementation: app/action_daemon.go (API service initialization)

• Blockchain Indexing:

  • Documentation: Chapter 4, Section "Indexing Blockchains"
  • Implementation: app/action_daemon.go (Scraper service initialization)

• Chains Configuration:

  • Documentation: Chapter 3, Section "Terminology and Concepts"
  • Implementation:
    • app/action_init_chains.go (Chain wizard implementation)
    • pkg/types/chain.go (Chain struct definition and validation)

• Configuration Management:

  • Documentation: Chapter 4, Section "Managing Configurations"
  • Implementation:
    • app/config.go (Configuration loading and initialization)
    • app/action_config_show.go (Show config command)
    • app/action_config_edit.go (Edit config command)

• Glossary: Chapter 7, Section "Glossary of Terms"

• IPFS Integration:

  • Documentation: Chapter 5, Section "Integrating with IPFS"
  • Implementation:
    • app/action_daemon.go (IPFS service initialization)
    • pkg/types/service.go (IPFS service definition)

• Logging and Debugging:

  • Documentation: Chapter 6, Section "Log Files and Debugging"
  • Implementation:
    • app/action_init_logging.go (Logging configuration)
    • pkg/types/general.go (Logging struct definition)

• RPC Endpoints:

  • Documentation: Chapter 2, Section "Initial Configuration"
  • Implementation:
    • pkg/validate/try_connect.go (RPC connection validation)
    • app/has_valid_rpc.go (RPC validation logic)

• Service Configuration:

  • Documentation: Chapter 2, Section "Configuration File Format"
  • Implementation:
    • app/action_init_services.go (Services wizard implementation)
    • pkg/types/service.go (Service struct definition and validation)

• Troubleshooting:

  • Documentation: Chapter 6, Section "Troubleshooting"
  • Implementation: Error handling throughout the codebase, especially in:
    • app/action_daemon.go (Service error handling)
    • app/config.go (Configuration error handling)

• Wizard Interface:

  • Documentation: Chapter 6, Section "Installation Wizard"
  • Implementation:
    • pkg/wizard/ directory (Wizard framework)
    • app/action_init.go (Wizard initialization)

Technical Specification

Purpose of this Document

This document defines the technical architecture, design, and functionalities of Khedra, enabling developers and engineers to understand its internal workings and design principles. For a less technical overview of the application, refer to the User Manual.

Intended Audience

This specification is for:

• Developers working on Khedra or integrating it into applications.
• System architects designing systems that use Khedra.
• Technical professionals looking for a detailed understanding of the system.

Scope and Objectives

The specification covers:

• High-level architecture.
• Core functionalities such as blockchain indexing, REST API, and address monitoring.
• Design principles, including scalability, error handling, and integration with IPFS.
• Supported chains, RPC requirements, and testing methodologies.

Introduction

Purpose of This Document

This technical specification document provides a comprehensive overview of Khedra's architecture, implementation details, and technical design decisions. It serves as a reference for developers, system architects, and technical stakeholders who need to understand the system's inner workings, extend its functionality, or integrate with it.

System Overview

Khedra is a sophisticated blockchain indexing and monitoring solution designed with a local-first architecture. It creates and maintains the Unchained Index - a permissionless index of address appearances across blockchain data - enabling powerful monitoring capabilities for any address on any supported EVM-compatible chain.

Core Technical Components

1. Indexing Engine: Processes blockchain data to extract and store address appearances
2. Service Framework: Manages the lifecycle of modular services (scraper, monitor, API, IPFS, control)
3. Data Storage Layer: Organizes and persists index data and caches
4. Configuration System: Manages user preferences and system settings
5. API Layer: Provides programmatic access to indexed data

Key Design Principles

Khedra's technical design adheres to several foundational principles:

1. Local-First Processing: All data processing happens on the user's machine, maximizing privacy
2. Chain Agnosticism: Support for any EVM-compatible blockchain with minimal configuration
3. Modularity: Clean separation of concerns between services for flexibility and maintainability
4. Resource Efficiency: Careful management of system resources, especially during indexing
5. Resilience: Robust error handling and recovery mechanisms
6. Extensibility: Well-defined interfaces to enable future enhancements

Technology Stack

Khedra is built on a modern technology stack:

• Go: The primary implementation language, chosen for its performance, concurrency model, and cross-platform support
• IPFS: For distributed sharing of index data
• RESTful API: For service integration and data access
• YAML: For configuration management
• Structured Logging: For operational monitoring and debugging

Target Audience

This technical specification is intended for:

• Developers: Contributing to Khedra or building on top of it
• System Administrators: Deploying and maintaining Khedra instances
• Technical Architects: Evaluating Khedra for integration with other systems
• Advanced Users: Seeking a deeper understanding of how Khedra works

Document Structure

The remaining sections of this specification are organized as follows:

• System Architecture: The high-level structure and components
• Core Functionalities: Detailed explanations of key features
• Technical Design: Implementation details and design patterns
• Supported Chains: Technical requirements and integration details
• Command-Line Interface: API and usage patterns
• Performance and Scalability: Benchmarks and optimization strategies
• Integration Points: APIs and interfaces for external systems
• Testing and Validation: Approaches to quality assurance
• Appendices: Technical reference materials

This specification aims to provide a comprehensive understanding of Khedra's technical aspects while serving as a reference for implementation, maintenance, and extension of the system.

System Architecture

Architectural Overview

Khedra employs a modular, service-oriented architecture designed for flexibility, resilience, and extensibility. The system is structured around a central application core that coordinates multiple specialized services, each with distinct responsibilities.

High-Level Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                      Khedra Application                          │
├─────────┬─────────┬─────────┬─────────┬─────────────────────────┤
│ Control │ Scraper │ Monitor │   API   │         IPFS            │
│ Service │ Service │ Service │ Service │        Service          │
├─────────┴─────────┴─────────┴─────────┴─────────────────────────┤
│                      Configuration Manager                       │
├─────────────────────────────────────────────────────────────────┤
│                          Data Layer                              │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────────┐ │
│  │ Unchained│  │  Binary  │  │ Monitor  │  │ Chain-Specific   │ │
│  │   Index  │  │  Caches  │  │   Data   │  │     Data         │ │
│  └──────────┘  └──────────┘  └──────────┘  └──────────────────┘ │
├─────────────────────────────────────────────────────────────────┤
│                      Blockchain Connectors                       │
└─────────────────────────────────────────────────────────────────┘
             ▲                    ▲                     ▲
             │                    │                     │
 ┌───────────┴──────────┐ ┌──────┴───────┐  ┌──────────┴──────────┐
 │  Ethereum Mainnet    │ │   Testnets   │  │   Other EVM Chains  │
 └──────────────────────┘ └──────────────┘  └─────────────────────┘

Core Components

1. Khedra Application

The main application container that initializes, configures, and manages the lifecycle of all services. It provides:

• Service registration and coordination
• Application startup and shutdown sequences
• Signal handling for graceful termination
• Global state management
• Cross-service communication

Implementation: app/khedra.go

2. Service Framework

Khedra implements five primary services:

2.1 Control Service

• Central Management: Provides unified management interface for all other services
• Service Lifecycle: Handles start, stop, restart, pause, and resume operations
• Health Monitoring: Continuously monitors service status and performance
• Runtime Configuration: Enables dynamic configuration updates without restart
• API Gateway: Serves as primary management interface for external tools
• Dependency Coordination: Manages service startup order and dependencies
• Error Recovery: Implements automatic restart policies for failed services
• Metrics Aggregation: Collects and exposes system-wide performance metrics

The Control Service runs on the same process as other services but provides external API access for management operations. It coordinates all service operations and serves as the single point of control for the Khedra system.

Implementation: pkg/services/control/service.go

2.2 Scraper Service

• Processes blockchain data to build the Unchained Index
• Extracts address appearances from transactions, logs, and traces
• Manages indexing state and progress tracking
• Handles retry logic for failed operations
• Implements batch processing with configurable parameters

Implementation: pkg/services/scraper/service.go

2.3 Monitor Service

• Tracks specified addresses for on-chain activity
• Maintains focused indices for monitored addresses
• Processes real-time blocks for quick notifications
• Supports flexible notification configurations
• Manages monitor definitions and states

Implementation: pkg/services/monitor/service.go

2.4 API Service

• Exposes RESTful endpoints for data access
• Implements query interfaces for the index and monitors
• Handles authentication and rate limiting
• Provides structured data responses in multiple formats
• Includes Swagger documentation for API endpoints

Implementation: pkg/services/api/service.go

2.5 IPFS Service

• Manages distributed sharing of index data
• Handles chunking of index data for efficient distribution
• Implements publishing and retrieval mechanisms
• Provides peer discovery and connection management
• Integrates with the IPFS network protocol

Implementation: pkg/services/ipfs/service.go

3. Configuration Manager

A centralized system for managing application settings, including:

• Configuration file parsing and validation
• Environment variable integration
• Runtime configuration updates
• Defaults management
• Chain-specific configuration handling

Implementation: pkg/config/config.go

4. Data Layer

The persistent storage infrastructure for Khedra:

4.1 Unchained Index

• Core data structure mapping addresses to appearances
• Optimized for fast lookups and efficient storage
• Implements chunking for distributed sharing
• Includes versioning for format compatibility

Implementation: pkg/index/index.go

4.2 Binary Caches

• Stores raw blockchain data for efficient reprocessing
• Implements cache invalidation and management
• Optimizes storage space usage with compression
• Supports pruning and maintenance operations

Implementation: pkg/cache/cache.go

4.3 Monitor Data

• Stores monitor definitions and state
• Tracks monitored address appearances
• Maintains notification history
• Implements efficient storage for frequent updates

Implementation: pkg/monitor/data.go

4.4 Chain-Specific Data

• Segregates data by blockchain
• Stores chain metadata and state
• Manages chain-specific configurations
• Handles chain reorganizations

Implementation: pkg/chains/data.go

5. Blockchain Connectors

The interface layer between Khedra and blockchain nodes:

• RPC client implementations
• Connection pooling and management
• Request rate limiting and backoff strategies
• Error handling and resilience patterns
• Chain-specific adaptations

Implementation: pkg/rpc/client.go

Communication Patterns

Khedra employs several communication patterns between components:

1. Service-to-Service Communication: Structured message passing between services using channels
2. RPC Communication: JSON-RPC communication with blockchain nodes
3. REST API: HTTP-based communication for external interfaces
4. File-Based Storage: Persistent data storage using structured files

Deployment Architecture

Khedra supports multiple deployment models:

1. Standalone Application: Single-process deployment for individual users
2. Docker Container: Containerized deployment for managed environments
3. Distributed Deployment: Multiple instances sharing index data via IPFS

Security Architecture

Security considerations in Khedra's architecture include:

1. Local-First Processing: Minimizes exposure of query data
2. API Authentication: Optional key-based authentication for API access
3. Configuration Protection: Secure handling of RPC credentials
4. Update Verification: Integrity checks for application updates
5. Resource Isolation: Service-level resource constraints

The modular design of Khedra allows for individual components to be extended, replaced, or enhanced without affecting the entire system, providing a solid foundation for future development and integration.

Core Functionalities

This section details Khedra's primary technical functionalities, explaining how each core feature is implemented and the technical approaches used.

Control Service

Service Management Interface

The Control Service provides centralized management capabilities for all other Khedra services through a RESTful API interface.

Technical Implementation

The Control Service implements these core management functions:

1. Service Lifecycle Management: Start, stop, restart, and pause individual services
2. Health Monitoring: Real-time status monitoring of all services
3. Configuration Management: Runtime configuration updates and validation
4. Metrics Collection: Aggregation of service performance metrics

// Simplified Control Service interface
type ControlService struct {
    serviceManager *ServiceManager
    httpServer     *http.Server
    logger         *slog.Logger
}

type ServiceStatus struct {
    Name        string
    State       ServiceState
    LastStarted time.Time
    Uptime      time.Duration
    Metrics     map[string]interface{}
}

type ServiceState int
const (
    StateStopped ServiceState = iota
    StateStarting
    StateRunning
    StatePausing
    StatePaused
    StateStopping
)

Management Endpoints

The Control Service exposes these API endpoints:

1. Service Status: GET /api/v1/services - List all services and their status
2. Individual Service Status: GET /api/v1/services/{serviceName} - Get specific service status
3. Start Service: POST /api/v1/services/{serviceName}/start - Start a specific service
4. Stop Service: POST /api/v1/services/{serviceName}/stop - Stop a specific service
5. Restart Service: POST /api/v1/services/{serviceName}/restart - Restart a specific service
6. Pause Service: POST /api/v1/services/{serviceName}/pause - Pause a specific service
7. Resume Service: POST /api/v1/services/{serviceName}/resume - Resume a paused service
8. System Status: GET /api/v1/status - Overall system health and status

Service Coordination

The Control Service coordinates service operations through:

• Dependency Management: Ensures services start in proper order
• Graceful Shutdown: Coordinates orderly service termination
• Error Recovery: Automatic restart of failed services (if configured)
• Resource Monitoring: Tracks system resource usage across services

Blockchain Indexing

The Unchained Index

The Unchained Index is the foundational data structure of Khedra, providing a reverse-lookup capability from addresses to their appearances in blockchain data.

Technical Implementation

The index is implemented as a specialized data structure with these key characteristics:

1. Bloom Filter Front-End: A probabilistic data structure that quickly determines if an address might appear in a block
2. Address-to-Appearance Mapping: Maps each address to a list of its appearances
3. Chunked Storage: Divides the index into manageable chunks (typically 1,000,000 blocks per chunk)
4. Versioned Format: Includes version metadata to handle format evolution

// Simplified representation of the index structure
type UnchainedIndex struct {
    Version string
    Chunks  map[uint64]*IndexChunk  // Key is chunk ID
}

type IndexChunk struct {
    BloomFilter   *BloomFilter
    Appearances   map[string][]Appearance  // Key is hex address
    StartBlock    uint64
    EndBlock      uint64
    LastUpdated   time.Time
}

type Appearance struct {
    BlockNumber    uint64
    TransactionIndex uint16
    AppearanceType  uint8
    LogIndex        uint16
}

Indexing Process

1. Block Retrieval: Fetch blocks from the RPC endpoint in configurable batches
2. Appearance Extraction: Process each block to extract address appearances from:
   • Transaction senders and recipients
   • Log topics and indexed parameters
   • Trace calls and results
   • State changes
3. Deduplication: Remove duplicate appearances within the same transaction
4. Storage: Update the appropriate index chunk with the new appearances
5. Bloom Filter Update: Update the bloom filter for quick future lookups

Performance Optimizations

• Parallel Processing: Multiple blocks processed concurrently
• Bloom Filters: Fast negative lookups to avoid unnecessary disk access
• Binary Encoding: Compact storage format for index data
• Caching: Frequently accessed index portions kept in memory

Address Monitoring

Monitor Implementation

The monitoring system tracks specific addresses for on-chain activity and provides notifications when activity is detected.

Technical Implementation

Monitors are implemented using these components:

1. Monitor Registry: Central store of all monitored addresses
2. Address Index: Fast lookup structure for monitored addresses
3. Activity Tracker: Records and timestamps address activity
4. Notification Manager: Handles alert distribution based on configuration

// Simplified monitor implementation
type Monitor struct {
    Address       string
    Description   string
    CreatedAt     time.Time
    LastActivity  time.Time
    Config        MonitorConfig
    ActivityLog   []Activity
}

type MonitorConfig struct {
    NotificationChannels []string
    Filters              *ActivityFilter
    Thresholds           map[string]interface{}
}

type Activity struct {
    BlockNumber      uint64
    TransactionHash  string
    Timestamp        time.Time
    ActivityType     string
    Details          map[string]interface{}
}

Monitoring Process

1. Registration: Add addresses to the monitor registry
2. Block Processing: As new blocks are processed, check for monitored addresses
3. Activity Detection: When a monitored address appears, record the activity
4. Notification: Based on configuration, send notifications via configured channels
5. State Update: Update the monitor's state with the new activity

Optimization Approaches

• Focused Index: Maintain a separate index for just monitored addresses
• Early Detection: Check monitored addresses early in the processing pipeline
• Configurable Sensitivity: Allow users to set thresholds for notifications
• Batched Notifications: Group notifications to prevent excessive alerts

API Service

RESTful Interface

The API service provides HTTP endpoints for querying indexed data and managing Khedra's operations.

Technical Implementation

The API is implemented using these components:

1. HTTP Server: Handles incoming requests and routing
2. Route Handlers: Process specific endpoint requests
3. Authentication Middleware: Optional API key verification
4. Response Formatter: Structures data in requested format (JSON, CSV, etc.)
5. Documentation: Auto-generated Swagger documentation

// Simplified API route implementation
type APIRoute struct {
    Path        string
    Method      string
    Handler     http.HandlerFunc
    Description string
    Params      []Parameter
    Responses   map[int]Response
}

// API server initialization
func NewAPIServer(config Config) *APIServer {
    server := &APIServer{
        router: mux.NewRouter(),
        port:   config.Port,
        auth:   config.Auth,
    }
    server.registerRoutes()
    return server
}

API Endpoints

The API provides endpoints in several categories:

1. Status Endpoints: System and service status information
2. Index Endpoints: Query the Unchained Index for address appearances
3. Monitor Endpoints: Manage and query address monitors
4. Chain Endpoints: Blockchain information and operations
5. Admin Endpoints: Configuration and management operations

Performance Considerations

• Connection Pooling: Reuse connections for efficiency
• Response Caching: Cache frequent queries with appropriate invalidation
• Pagination: Limit response sizes for large result sets
• Query Optimization: Efficient translation of API queries to index lookups
• Rate Limiting: Prevent resource exhaustion from excessive requests

IPFS Integration

Distributed Index Sharing

The IPFS integration enables sharing and retrieving index chunks through the distributed IPFS network.

Technical Implementation

The IPFS functionality is implemented with these components:

1. IPFS Node: Either embedded or external IPFS node connection
2. Chunk Manager: Handles breaking the index into shareable chunks
3. Publishing Logic: Manages uploading chunks to IPFS
4. Discovery Service: Finds and retrieves chunks from the network
5. Validation: Verifies the integrity of downloaded chunks

// Simplified IPFS service implementation
type IPFSService struct {
    node        *ipfs.CoreAPI
    chunkManager *ChunkManager
    config      IPFSConfig
}

type ChunkManager struct {
    chunkSize      uint64
    validationFunc func([]byte) bool
    storage        *Storage
}

Distribution Process

1. Chunking: Divide the index into manageable chunks with metadata
2. Publishing: Add chunks to IPFS and record their content identifiers (CIDs)
3. Announcement: Share availability information through the network
4. Discovery: Find chunks needed by querying the IPFS network
5. Retrieval: Download needed chunks from peers
6. Validation: Verify chunk integrity before integration

Optimization Strategies

• Incremental Updates: Share only changed or new chunks
• Prioritized Retrieval: Download most useful chunks first
• Peer Selection: Connect to reliable peers for better performance
• Background Syncing: Retrieve chunks in the background without blocking
• Compressed Storage: Minimize bandwidth and storage requirements

Configuration Management

Flexible Configuration System

Khedra's configuration system provides multiple ways to configure the application, with clear precedence rules.

Technical Implementation

The configuration system is implemented with these components:

1. YAML Parser: Reads the configuration file format
2. Environment Variable Processor: Overrides from environment variables
3. Validation Engine: Ensures configuration values are valid
4. Defaults Manager: Provides sensible defaults where needed
5. Runtime Updater: Handles configuration changes during operation

// Simplified configuration structure
type Config struct {
    General  GeneralConfig
    Chains   map[string]ChainConfig
    Services map[string]ServiceConfig
    Logging  LoggingConfig
}

// Configuration loading process
func LoadConfig(path string) (*Config, error) {
    config := DefaultConfig()
    
    // Load from file if exists
    if fileExists(path) {
        if err := loadFromFile(path, config); err != nil {
            return nil, err
        }
    }
    
    // Override with environment variables
    applyEnvironmentOverrides(config)
    
    // Validate the final configuration
    if err := validateConfig(config); err != nil {
        return nil, err
    }
    
    return config, nil
}

Configuration Sources

The system processes configuration from these sources, in order of precedence:

1. Environment Variables: Highest precedence, override all other sources
2. Configuration File: User-provided settings in YAML format
3. Default Values: Built-in defaults for unspecified settings

Validation Rules

The configuration system enforces these kinds of validation:

1. Type Validation: Ensures values have the correct data type
2. Range Validation: Numeric values within acceptable ranges
3. Format Validation: Strings matching required patterns (e.g., URLs)
4. Dependency Validation: Related settings are consistent
5. Resource Validation: Settings are compatible with available resources

These core functionalities form the technical foundation of Khedra, enabling its primary capabilities while providing the flexibility and performance required for blockchain data processing.

Technical Design

This section details the key technical design decisions, patterns, and implementation approaches used in Khedra.

Code Organization

Khedra follows a modular code organization pattern to promote maintainability and separation of concerns.

Directory Structure

khedra/
├── app/                 // Application core
│   ├── khedra.go        // Main application definition
│   └── commands/        // CLI command implementations
├── cmd/                 // Command line entry points
│   └── khedra/          // Main CLI command
├── pkg/                 // Core packages
│   ├── config/          // Configuration management
│   ├── services/        // Service implementations
│   │   ├── api/         // API service
│   │   ├── control/     // Control service
│   │   ├── ipfs/        // IPFS service
│   │   ├── monitor/     // Monitor service
│   │   └── scraper/     // Scraper service
│   ├── index/           // Unchained Index implementation
│   ├── cache/           // Caching logic
│   ├── chains/          // Chain-specific code
│   ├── rpc/             // RPC client implementations
│   ├── wizard/          // Configuration wizard
│   └── utils/           // Shared utilities
└── main.go              // Application entry point

Package Design Principles

1. Clear Responsibilities: Each package has a single, well-defined responsibility
2. Minimal Dependencies: Packages depend only on what they need
3. Interface-Based Design: Dependencies defined as interfaces, not concrete types
4. Internal Encapsulation: Implementation details hidden behind public interfaces
5. Context-Based Operations: Functions accept context for cancellation and timeout

Service Architecture

Khedra implements a service-oriented architecture within a single application.

Service Interface

Each service implements a common interface:

type Service interface {
    // Initialize the service
    Init(ctx context.Context) error
    
    // Start the service
    Start(ctx context.Context) error
    
    // Stop the service
    Stop(ctx context.Context) error
    
    // Return the service name
    Name() string
    
    // Return the service status
    Status() ServiceStatus
    
    // Return service-specific metrics
    Metrics() map[string]interface{}
}

Service Lifecycle

1. Registration: Services register with the application core
2. Initialization: Services initialize resources and validate configuration
3. Starting: Services begin operations in coordinated sequence
4. Running: Services perform their core functions
5. Stopping: Services gracefully terminate when requested
6. Cleanup: Services release resources during application shutdown

Service Coordination

Services coordinate through several mechanisms:

1. Direct References: Services can hold references to other services when needed
2. Event Bus: Publish-subscribe pattern for decoupled communication
3. Shared State: Limited shared state for cross-service information
4. Context Propagation: Request context flows through service operations

Data Storage Design

Khedra employs a hybrid storage approach for different data types.

Directory Layout

~/.khedra/
├── config.yaml           // Main configuration file
├── data/                 // Main data directory
│   ├── mainnet/          // Chain-specific data
│   │   ├── cache/        // Binary caches
│   │   │   ├── blocks/   // Cached blocks
│   │   │   ├── traces/   // Cached traces
│   │   │   └── receipts/ // Cached receipts
│   │   ├── index/        // Unchained Index chunks
│   │   └── monitors/     // Address monitor data
│   └── [other-chains]/   // Other chain data
└── logs/                 // Application logs

Storage Formats

1. Index Data: Custom binary format optimized for size and query speed
2. Cache Data: Compressed binary representation of blockchain data
3. Monitor Data: Structured JSON for flexibility and human readability
4. Configuration: YAML for readability and easy editing
5. Logs: Structured JSON for machine processing and analysis

Storage Persistence Strategy

1. Atomic Writes: Prevent corruption during unexpected shutdowns
2. Version Headers: Include format version for backward compatibility
3. Checksums: Verify data integrity through hash validation
4. Backup Points: Periodic snapshots for recovery
5. Incremental Updates: Minimize disk writes for frequently changed data

Error Handling and Resilience

Khedra implements robust error handling to ensure reliability in various failure scenarios.

Error Categories

1. Transient Errors: Temporary failures that can be retried (network issues, rate limiting)
2. Persistent Errors: Failures requiring intervention (misconfiguration, permission issues)
3. Fatal Errors: Unrecoverable errors requiring application restart
4. Validation Errors: Issues with user input or configuration
5. Resource Errors: Problems with system resources (disk space, memory)

Resilience Patterns

1. Retry with Backoff: Exponential backoff for transient failures
2. Circuit Breakers: Prevent cascading failures when services are unhealthy
3. Graceful Degradation: Reduce functionality rather than failing completely
4. Health Checks: Proactive monitoring of dependent services
5. Recovery Points: Maintain state that allows resuming after failures

Error Reporting

1. Structured Logging: Detailed error information in structured format
2. Context Preservation: Include context when errors cross boundaries
3. Error Wrapping: Maintain error chains without losing information
4. User-Friendly Messages: Translate technical errors to actionable information
5. Error Metrics: Track error rates and patterns for analysis

Concurrency Model

Khedra leverages Go's concurrency primitives for efficient parallel processing.

Concurrency Patterns

1. Worker Pools: Process batches of blocks concurrently with controlled parallelism
2. Fan-Out/Fan-In: Distribute work to multiple goroutines and collect results
3. Pipelines: Connect processing stages with channels for streaming data
4. Context Propagation: Pass cancellation signals through processing chains
5. Rate Limiting: Control resource usage and external API calls

Resource Management

1. Connection Pooling: Reuse network connections to blockchain nodes
2. Goroutine Limiting: Prevent excessive goroutine creation
3. Memory Budgeting: Control memory usage during large operations
4. I/O Throttling: Balance disk operations to prevent saturation
5. Adaptive Concurrency: Adjust parallelism based on system capabilities

Synchronization Techniques

1. Mutexes: Protect shared data structures from concurrent access
2. Read/Write Locks: Optimize for read-heavy access patterns
3. Atomic Operations: Use atomic primitives for simple counters and flags
4. Channels: Communicate between goroutines and implement synchronization
5. WaitGroups: Coordinate completion of multiple goroutines

Configuration Wizard

The configuration wizard provides an interactive interface for setting up Khedra.

Wizard Architecture

1. Screen-Based Flow: Organized as a sequence of screens
2. Question Framework: Standardized interface for user input
3. Validation Layer: Real-time validation of user inputs
4. Navigation System: Forward/backward movement between screens
5. Help Integration: Contextual help for each configuration option

User Interface Design

1. Text-Based UI: Terminal-friendly interface with box drawing
2. Color Coding: Visual cues for different types of information
3. Navigation Bar: Consistent display of available commands
4. Progress Indication: Show position in the configuration process
5. Direct Editing: Option to edit configuration files directly

Implementation Approach

The wizard uses a structured approach to manage screens and user interaction:

type Screen struct {
    Title         string
    Subtitle      string
    Body          string
    Instructions  string
    Replacements  []Replacement
    Questions     []Questioner
    Style         Style
    Current       int
    Wizard        *Wizard
    NavigationBar *NavigationBar
}

type Wizard struct {
    Config    *config.Config
    Screens   []Screen
    Current   int
    History   []int
    // Additional fields for wizard state
}

This design allows for a flexible, extensible configuration process that can adapt to different user needs and configuration scenarios.

Testing Strategy

Khedra employs a comprehensive testing strategy to ensure reliability and correctness.

Testing Levels

1. Unit Tests: Verify individual functions and components
2. Integration Tests: Test interaction between components
3. Service Tests: Validate complete service behavior
4. End-to-End Tests: Test full application workflows
5. Performance Tests: Benchmark critical operations

Test Implementation

1. Mock Objects: Simulate external dependencies
2. Test Fixtures: Standard data sets for reproducible tests
3. Property-Based Testing: Generate test cases to find edge cases
4. Parallel Testing: Run tests concurrently for faster feedback
5. Coverage Analysis: Track code coverage to identify untested areas

These technical design choices provide Khedra with a solid foundation for reliable, maintainable, and efficient operation across a variety of deployment scenarios and use cases.

Supported Chains

This section details the blockchain networks supported by Khedra, the technical requirements for each, and the implementation approaches for multi-chain support.

Chain Support Architecture

Khedra implements a flexible architecture for supporting multiple EVM-compatible blockchains simultaneously.

Chain Abstraction Layer

At the core of Khedra's multi-chain support is a chain abstraction layer that:

1. Normalizes differences between chain implementations
2. Provides a uniform interface for blockchain interactions
3. Manages chain-specific configurations and behaviors
4. Isolates chain-specific code from the core application logic

// Simplified Chain interface
type Chain interface {
    // Return the chain name
    Name() string
    
    // Return the chain ID
    ChainID() uint64
    
    // Get RPC client for this chain
    Client() rpc.Client
    
    // Get path to chain-specific data directory
    DataDir() string
    
    // Check if this chain requires special handling for a feature
    SupportsFeature(feature string) bool
    
    // Get chain-specific configuration
    Config() ChainConfig
}

Core Chain Requirements

For a blockchain to be fully supported by Khedra, it must meet these technical requirements:

RPC Support

The chain must provide an Ethereum-compatible JSON-RPC API with these essential methods:

1. Basic Methods:

   • eth_blockNumber: Get the latest block number
   • eth_getBlockByNumber: Retrieve block data
   • eth_getTransactionReceipt: Get transaction receipts with logs
   • eth_chainId: Return the chain identifier

2. Trace Support:

   • Either debug_traceTransaction or trace_transaction: Retrieve execution traces
   • Alternatively: trace_block or debug_traceBlockByNumber: Get all traces in a block

Data Structures

The chain must use compatible data structures:

1. Addresses: 20-byte Ethereum-compatible addresses
2. Transactions: Compatible transaction format with standard fields
3. Logs: EVM-compatible event logs
4. Traces: Call traces in a format compatible with Khedra's processors

Consensus and Finality

The chain should have:

1. Deterministic Finality: Clear rules for when blocks are considered final
2. Manageable Reorgs: Limited blockchain reorganizations
3. Block Time Consistency: Relatively consistent block production times

Ethereum Mainnet

Ethereum mainnet is the primary supported chain and is required even when indexing other chains.

Special Considerations

1. Block Range: Support for full historical range from genesis
2. Archive Node: Full archive node required for historical traces
3. Trace Support: Must support either Geth or Parity trace methods
4. Size Considerations: Largest data volume among supported chains

Implementation Details

// Ethereum mainnet-specific configuration
type EthereumMainnetChain struct {
    BaseChain
    traceMethod string  // "geth" or "parity" style traces
}

func (c *EthereumMainnetChain) ProcessTraces(traces []interface{}) []Appearance {
    // Mainnet-specific trace processing logic
    // ...
}

EVM-Compatible Chains

Khedra supports a variety of EVM-compatible chains with minimal configuration.

Officially Supported Chains

These chains are officially supported with tested implementations:

1. Ethereum Testnets:

   • Sepolia
   • Goerli (legacy support)

2. Layer 2 Networks:

   • Optimism
   • Arbitrum
   • Polygon

3. EVM Sidechains:

   • Gnosis Chain (formerly xDai)
   • Avalanche C-Chain
   • Binance Smart Chain

Chain Configuration

Each chain is configured with these parameters:

chains:
  mainnet:  # Chain identifier
    rpcs:   # List of RPC endpoints
      - "https://ethereum-rpc.example.com"
    enabled: true  # Whether the chain is active
    trace_support: "geth"  # Trace API style
    # Chain-specific overrides
    scraper:
      batch_size: 500

Chain-Specific Adaptations

Some chains require special handling:

1. Optimism/Arbitrum: Modified trace processing for rollup architecture
2. Polygon: Adjusted finality assumptions for PoS consensus
3. BSC/Avalanche: Faster block times requiring different batch sizing

Chain Detection and Validation

Khedra implements robust chain detection and validation:

Auto-Detection

When connecting to an RPC endpoint:

1. Query eth_chainId to determine the actual chain
2. Verify against the configured chain identifier
3. Detect trace method support through API probing
4. Identify chain-specific capabilities

Connection Management

For each configured chain:

1. Connection Pool: Maintain multiple connections for parallel operations
2. Failover Support: Try alternative endpoints when primary fails
3. Health Monitoring: Track endpoint reliability and performance
4. Rate Limiting: Respect provider-specific rate limits

Data Isolation

Khedra maintains strict data isolation between chains:

1. Chain-Specific Directories: Separate storage locations for each chain
2. Independent Indices: Each chain has its own Unchained Index
3. Configuration Isolation: Chain-specific settings don't affect other chains
4. Parallel Processing: Chains can be processed concurrently

Adding New Chain Support

For adding support for a new EVM-compatible chain:

1. Configuration: Add the chain definition to config.yaml
2. Custom Handling: Implement any chain-specific processors if needed
3. Testing: Verify compatibility with the new chain
4. Documentation: Update supported chains documentation

Example: Adding a New Chain

// Register a new chain type
func RegisterNewChain() {
    registry.RegisterChain("new-chain", func(config ChainConfig) (Chain, error) {
        return &NewChain{
            BaseChain: NewBaseChain(config),
            // Chain-specific initialization
        }, nil
    })
}

// Implement chain-specific behavior
type NewChain struct {
    BaseChain
    // Chain-specific fields
}

func (c *NewChain) SupportsFeature(feature string) bool {
    // Chain-specific feature support
    switch feature {
    case "trace":
        return true
    case "state_diff":
        return false
    default:
        return c.BaseChain.SupportsFeature(feature)
    }
}

Khedra's flexible chain support architecture allows it to adapt to the evolving ecosystem of EVM-compatible blockchains while maintaining consistent indexing and monitoring capabilities across all supported networks.

Command-Line Interface

Khedra provides a simple, focused command-line interface (CLI) for managing the system. The CLI is designed around a core workflow of initialization, daemon management, and service control via REST API.

CLI Architecture

The CLI is built using the urfave/cli library, providing a lightweight and user-friendly interface focused on essential operations.

Design Principles

1. Simplicity: Minimal command set focused on core functionality
2. Clarity: Each command has a clear, single purpose
3. Automation-Friendly: Services are controlled via REST API for scriptability
4. Self-Documenting: Built-in help for all commands

Implementation Structure

func initCli(k *KhedraApp) *cli.App {
    app := &cli.App{
        Name:     "khedra",
        Usage:    "A tool to index, monitor, serve, and share blockchain data",
        Version:  version.Version,
        Commands: []*cli.Command{
            // Core commands only - no complex service management via CLI
        },
    }
    return app
}

Command Structure

Khedra's CLI is organized into logical command groups.

Root Command

The base khedra command serves as the entry point and provides global options:

khedra [global options] command [command options] [arguments...]

Global options include:

• --config: Specify an alternate configuration file path
• --format: Control output format (text, json, csv)
• --verbose: Enable verbose output
• --quiet: Suppress non-error output
• --chain: Specify the target blockchain (defaults to "mainnet")

Service Management Commands

Commands for controlling Khedra's services:

Service Lifecycle Commands

• khedra daemon: Start Khedra in daemon mode with all configured services

  # Start with default configuration
  khedra daemon
  
  # Start with specific log level
  TB_KHEDRA_LOGGING_LEVEL=debug khedra daemon
  
  # Start with custom configuration file
  khedra daemon --config=/path/to/config.yaml
  

• khedra start [service...]: Start specific services (if supported)

  # Start all services
  khedra start
  
  # Start specific services
  khedra start scraper api
  
  # Start in foreground mode
  khedra start --foreground
  

• khedra stop [service...]: Stop specific services (if supported)

  # Stop all services
  khedra stop
  
  # Stop specific services  
  khedra stop scraper monitor
  
  # Force stop (ungraceful shutdown)
  khedra stop --force
  

• khedra restart [service...]: Restart specific services (if supported)

  # Restart all services
  khedra restart
  
  # Restart specific services
  khedra restart api ipfs
  
  # Restart with configuration reload
  khedra restart --reload-config
  

Service Status Commands

• khedra status: Show status of all services

  # Basic status
  khedra status
  
  # Detailed status with metrics
  khedra status --verbose
  
  # Status for specific services
  khedra status scraper monitor
  
  # JSON output for scripting
  khedra status --format=json
  

Service Control via API

All service management operations are also available via REST API:

# Check service status
curl http://localhost:8080/api/v1/services

# Start a service
curl -X POST http://localhost:8080/api/v1/services/scraper/start

# Stop a service  
curl -X POST http://localhost:8080/api/v1/services/monitor/stop

# Get detailed service metrics
curl http://localhost:8080/api/v1/services/api?verbose=true

Index Management Commands

Commands for managing the Unchained Index:

Index Status and Information

• khedra index status [chain]: Show index status for specified chain

  # Status for default chain
  khedra index status
  
  # Status for specific chain
  khedra index status --chain=mainnet
  
  # Show gaps in the index
  khedra index status --show-gaps
  
  # Show detailed analytics
  khedra index status --analytics
  
  # Export status to file
  khedra index status --output=/path/to/status.json
  

Index Maintenance Commands

• khedra index rebuild [options]: Rebuild portions of the index

  # Rebuild specific block range
  khedra index rebuild --start=18000000 --end=18001000
  
  # Rebuild from specific block to latest
  khedra index rebuild --from=18000000
  
  # Rebuild with specific batch size
  khedra index rebuild --batch-size=1000 --start=18000000 --end=18001000
  
  # Rebuild for specific chain
  khedra index rebuild --chain=sepolia --start=4000000 --end=4001000
  

• khedra index verify [options]: Verify index integrity

  # Verify entire index
  khedra index verify
  
  # Verify specific block range
  khedra index verify --start=18000000 --end=18001000
  
  # Verify and attempt to repair issues
  khedra index verify --repair
  
  # Verify for specific chain
  khedra index verify --chain=mainnet
  

• khedra index optimize: Optimize index storage and performance

  # Optimize entire index
  khedra index optimize
  
  # Optimize specific chunks
  khedra index optimize --chunks=1800,1801,1802
  
  # Optimize and compress
  khedra index optimize --compress
  

Monitor Commands

Commands for managing address monitors:

Monitor Management

• khedra monitor add ADDRESS [ADDRESS...]: Add addresses to monitor

  # Add single address
  khedra monitor add 0x742d35Cc6634C0532925a3b844Bc454e4438f44e
  
  # Add multiple addresses
  khedra monitor add 0x742d35Cc... 0x1234567... 0xabcdef...
  
  # Add with custom name
  khedra monitor add 0x742d35Cc... --name="Vitalik Buterin"
  
  # Add with notification settings
  khedra monitor add 0x742d35Cc... --notifications=webhook,email
  
  # Add from file
  khedra monitor add --file=/path/to/addresses.txt
  

• khedra monitor remove ADDRESS [ADDRESS...]: Remove monitored addresses

  # Remove specific addresses
  khedra monitor remove 0x742d35Cc6634C0532925a3b844Bc454e4438f44e
  
  # Remove without confirmation prompt
  khedra monitor remove 0x742d35Cc... --force
  
  # Remove all monitors (with confirmation)
  khedra monitor remove --all
  
  # Remove monitors matching pattern
  khedra monitor remove --pattern="test_*"
  

Monitor Information

• khedra monitor list: List all monitored addresses

  # List all monitors
  khedra monitor list
  
  # List with detailed information
  khedra monitor list --details
  
  # List only active monitors
  khedra monitor list --active-only
  
  # Export to CSV
  khedra monitor list --format=csv --output=monitors.csv
  
  # Filter by pattern
  khedra monitor list --pattern="*buterin*"
  

• khedra monitor activity ADDRESS: Show activity for monitored address

  # Show recent activity
  khedra monitor activity 0x742d35Cc6634C0532925a3b844Bc454e4438f44e
  
  # Show activity in block range
  khedra monitor activity 0x742d35Cc... --from=18000000 --to=18001000
  
  # Limit number of results
  khedra monitor activity 0x742d35Cc... --limit=100
  
  # Show activity in specific time range
  khedra monitor activity 0x742d35Cc... --since="2023-01-01" --until="2023-12-31"
  
  # Export activity to file
  khedra monitor activity 0x742d35Cc... --output=/path/to/activity.json
  

Monitor Configuration

• khedra monitor config ADDRESS: Configure monitor settings

  # Update monitor name
  khedra monitor config 0x742d35Cc... --name="Updated Name"
  
  # Configure notifications
  khedra monitor config 0x742d35Cc... --notifications=webhook --webhook-url=https://...
  
  # Set monitoring thresholds
  khedra monitor config 0x742d35Cc... --min-value=1.0 --currency=ETH
  
  # Disable/enable monitor
  khedra monitor config 0x742d35Cc... --enabled=false
  

Chain Management Commands

Commands for managing blockchain connections:

Chain Configuration

• khedra chains list: List configured blockchain networks

  # List all configured chains
  khedra chains list
  
  # List only enabled chains
  khedra chains list --enabled-only
  
  # Show detailed chain information
  khedra chains list --details
  
  # Export chain configuration
  khedra chains list --format=json --output=chains.json
  

• khedra chains add NAME URL: Add new blockchain network

  # Add new chain with single RPC
  khedra chains add polygon https://polygon-rpc.com
  
  # Add chain and enable immediately
  khedra chains add arbitrum https://arb1.arbitrum.io/rpc --enable
  
  # Add chain with multiple RPC endpoints
  khedra chains add optimism https://mainnet.optimism.io,https://opt-mainnet.g.alchemy.com/v2/KEY
  
  # Add chain with custom configuration
  khedra chains add base https://mainnet.base.org --chain-id=8453 --symbol=ETH
  

• khedra chains remove NAME: Remove blockchain network

  # Remove specific chain
  khedra chains remove polygon
  
  # Remove without confirmation
  khedra chains remove arbitrum --force
  
  # Remove and clean up data
  khedra chains remove optimism --cleanup-data
  

Chain Testing and Validation

• khedra chains test NAME: Test connection to blockchain network

  # Test specific chain connectivity
  khedra chains test mainnet
  
  # Test with verbose output
  khedra chains test polygon --verbose
  
  # Test all RPC endpoints for a chain
  khedra chains test mainnet --test-all-rpcs
  
  # Test and benchmark performance
  khedra chains test mainnet --benchmark
  

• khedra chains validate: Validate all chain configurations

  # Validate all chains
  khedra chains validate
  
  # Validate specific chain
  khedra chains validate --chain=mainnet
  
  # Validate and show detailed results
  khedra chains validate --verbose
  
  # Validate and attempt to fix issues
  khedra chains validate --auto-fix
  

Configuration Commands

Commands for managing Khedra's configuration:

Configuration Display and Editing

• khedra config show: Display current configuration

  # Show complete configuration
  khedra config show
  
  # Show specific section
  khedra config show --section=services
  khedra config show --section=chains
  khedra config show --section=logging
  
  # Hide sensitive information
  khedra config show --redact
  
  # Export configuration
  khedra config show --format=yaml --output=config.yaml
  khedra config show --format=json --output=config.json
  

• khedra config edit: Open configuration in editor

  # Edit with default editor
  khedra config edit
  
  # Edit with specific editor
  khedra config edit --editor=vim
  khedra config edit --editor=code
  
  # Edit specific section
  khedra config edit --section=services
  
  # Edit and validate before saving
  khedra config edit --validate
  

Configuration Management

• khedra config wizard: Run interactive configuration wizard

  # Run full configuration wizard
  khedra config wizard
  
  # Run simplified wizard
  khedra config wizard --simple
  
  # Run wizard for specific section
  khedra config wizard --section=services
  khedra config wizard --section=chains
  
  # Run wizard and save to custom location
  khedra config wizard --output=/path/to/config.yaml
  

• khedra config validate: Validate configuration

  # Validate current configuration
  khedra config validate
  
  # Validate specific file
  khedra config validate --file=/path/to/config.yaml
  
  # Validate and show detailed errors
  khedra config validate --verbose
  
  # Validate against specific schema version
  khedra config validate --schema-version=v1.0
  

• khedra config reset: Reset configuration to defaults

  # Reset entire configuration (with confirmation)
  khedra config reset
  
  # Reset specific section
  khedra config reset --section=services
  
  # Reset without confirmation prompt
  khedra config reset --force
  
  # Reset and backup current config
  khedra config reset --backup=/path/to/backup.yaml
  

Utility Commands

Commands for various utility operations:

Version and Information

• khedra version: Display version information

  # Show version
  khedra version
  
  # Show detailed build information
  khedra version --build-info
  
  # Show version in JSON format
  khedra version --json
  
  # Check for updates
  khedra version --check-updates
  

Initialization and Setup

• khedra init: Initialize Khedra configuration and data directories

  # Initialize with wizard
  khedra init
  
  # Initialize with minimal setup
  khedra init --minimal
  
  # Initialize in specific directory
  khedra init --data-dir=/custom/path
  
  # Initialize with custom configuration
  khedra init --config=/path/to/config.yaml
  
  # Reinitialize existing setup
  khedra init --force
  

Health and Diagnostics

• khedra health: Check overall system health

  # Basic health check
  khedra health
  
  # Detailed health report
  khedra health --detailed
  
  # Health check with remediation suggestions
  khedra health --suggest-fixes
  
  # Export health report
  khedra health --output=/path/to/health-report.json
  

• khedra doctor: Run comprehensive system diagnostics

  # Run all diagnostic checks
  khedra doctor
  
  # Run specific diagnostic categories
  khedra doctor --checks=network,storage,performance
  
  # Run diagnostics and attempt auto-fixes
  khedra doctor --auto-fix
  
  # Run diagnostics for specific chain
  khedra doctor --chain=mainnet
  

Log and Debug Commands

• khedra logs: Display and manage log files

  # Show recent logs
  khedra logs
  
  # Follow logs in real-time
  khedra logs --follow
  
  # Show logs for specific service
  khedra logs --service=scraper
  khedra logs --service=monitor
  
  # Filter logs by level
  khedra logs --level=error
  khedra logs --level=debug
  
  # Show logs in specific time range
  khedra logs --since="2023-01-01" --until="2023-12-31"
  
  # Export logs to file
  khedra logs --output=/path/to/logs.txt
  

Data Management

• khedra cleanup: Clean up temporary files and optimize storage

  # Clean temporary files
  khedra cleanup
  
  # Clean and optimize index
  khedra cleanup --optimize-index
  
  # Clean old log files
  khedra cleanup --logs --older-than=30d
  
  # Dry run to see what would be cleaned
  khedra cleanup --dry-run
  
  # Clean specific data types
  khedra cleanup --cache --temp --logs
  

Performance and Scalability

This section details Khedra's performance characteristics, optimization strategies, and scalability considerations.

Performance Tuning Guide

Optimal Configuration for Different Use Cases

Light Usage (Personal Development/Testing)

services:
  scraper:
    enabled: true
    sleep: 30        # Longer sleep for less aggressive indexing
    batchSize: 100   # Smaller batches to reduce memory usage
  monitor:
    enabled: false   # Disable if not needed
    sleep: 60
    batchSize: 50
  api:
    enabled: true
    port: 8080       # Standard configuration
  ipfs:
    enabled: false   # Disable to reduce resource usage

Expected Performance: 5-10 blocks/sec indexing, minimal resource usage

Standard Usage (Regular Development/Analysis)

services:
  scraper:
    enabled: true
    sleep: 12        # Balanced sleep interval
    batchSize: 500   # Default batch size
  monitor:
    enabled: true    # Enable for address tracking
    sleep: 12
    batchSize: 100
  api:
    enabled: true
    port: 8080
  ipfs:
    enabled: true    # Enable for collaboration
    port: 8083

Expected Performance: 15-25 blocks/sec indexing, moderate resource usage

High-Performance Usage (Production/Heavy Analysis)

services:
  scraper:
    enabled: true
    sleep: 5         # Aggressive indexing
    batchSize: 2000  # Large batches for efficiency
  monitor:
    enabled: true
    sleep: 5         # Fast monitoring
    batchSize: 500
  api:
    enabled: true
    port: 8080
  ipfs:
    enabled: true
    port: 8083

Expected Performance: 25-40 blocks/sec indexing, high resource usage

Batch Size Optimization Guidelines

Factors Affecting Optimal Batch Size

1. RPC Endpoint Performance:

   • Fast/unlimited RPC: 1000-5000 blocks
   • Standard RPC: 500-1000 blocks
   • Rate-limited RPC: 50-200 blocks

2. Available Memory:

   • 8GB+ RAM: 1000-2000 blocks
   • 4-8GB RAM: 500-1000 blocks
   • <4GB RAM: 100-500 blocks

3. Network Latency:

   • Local RPC node: 2000-5000 blocks
   • Same region: 1000-2000 blocks
   • Remote/high latency: 100-500 blocks

Batch Size Tuning Process

1. Start with defaults (500 blocks)
2. Monitor performance metrics
3. Adjust based on bottlenecks:
   • If RPC timeouts: decrease batch size
   • If memory issues: decrease batch size
   • If CPU idle time: increase batch size
   • If slow overall progress: increase batch size

Sleep Interval Recommendations

Based on System Resources

High-End Systems (8+ cores, 16GB+ RAM):

• Scraper: 5-10 seconds
• Monitor: 5-10 seconds

Mid-Range Systems (4-8 cores, 8-16GB RAM):

• Scraper: 10-15 seconds
• Monitor: 10-15 seconds

Resource-Constrained Systems (<4 cores, <8GB RAM):

• Scraper: 20-30 seconds
• Monitor: 30-60 seconds

Based on RPC Provider

Unlimited/Premium RPC:

• Scraper: 5-10 seconds
• Monitor: 5-10 seconds

Standard RPC with rate limits:

• Scraper: 15-30 seconds
• Monitor: 30-60 seconds

Free/heavily limited RPC:

• Scraper: 60-120 seconds
• Monitor: 120-300 seconds

RPC Endpoint Optimization

Choosing RPC Providers

Recommended for High Performance:

1. Local RPC node (best performance)
2. Premium providers (Alchemy, Infura Pro)
3. Archive nodes with trace support

Configuration for Multiple RPC Endpoints:

chains:
  mainnet:
    rpcs:
      - "https://eth-mainnet.alchemyapi.io/v2/YOUR_KEY"
      - "https://mainnet.infura.io/v3/YOUR_KEY"  
      - "https://rpc.ankr.com/eth"
    enabled: true

RPC Performance Testing

# Test RPC response time
time curl -X POST -H "Content-Type: application/json" \
  --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
  https://your-rpc-endpoint

# Test batch request performance  
time curl -X POST -H "Content-Type: application/json" \
  --data '[{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x1000000",false],"id":1}]' \
  https://your-rpc-endpoint

System Resource Monitoring

Key Metrics to Monitor

1. CPU Usage:

   top -p $(pgrep khedra)
   

2. Memory Usage:

   ps -o pid,vsz,rss,comm -p $(pgrep khedra)
   

3. Disk I/O:

   iotop -p $(pgrep khedra)
   

4. Network Usage:

   nethogs -p $(pgrep khedra)
   

Performance Thresholds

CPU Usage:

• <50%: Can increase batch size or decrease sleep
• 50-80%: Optimal range

• 80%: Decrease batch size or increase sleep

Memory Usage:

• <2GB: Can increase batch size
• 2-4GB: Monitor for memory leaks

• 4GB: Decrease batch size

Disk I/O:

• High read: Index queries are efficient
• High write: Indexing in progress (normal)
• Very high write: May need to reduce batch size

Scaling Considerations

Horizontal Scaling Strategies

1. Chain Separation:

   • Run separate Khedra instances per blockchain
   • Distribute chains across multiple servers
   • Use load balancer for API access

2. Service Separation:

   • Run API service on separate instances
   • Dedicated IPFS nodes for data sharing
   • Centralized monitoring service

3. Geographic Distribution:

   • Deploy close to RPC providers
   • Regional API instances for lower latency
   • IPFS network for global data sharing

Vertical Scaling Guidelines

Memory Scaling:

• 8GB: Single chain, moderate usage
• 16GB: Multiple chains or heavy usage
• 32GB+: High-performance production usage

CPU Scaling:

• 4 cores: Basic usage
• 8 cores: Standard production
• 16+ cores: High-performance or multiple chains

Storage Scaling:

• SSD required for optimal performance
• 100GB per chain per year (estimate)
• Consider compression and archival strategies

Service Metrics and Monitoring

Available Performance Metrics

Each Khedra service exposes performance metrics that can be accessed through the Control Service API. These metrics provide insight into service health, performance, and resource utilization.

Control Service Metrics

Service Status Metrics:

• uptime: Service runtime duration since last start
• state: Current service state (running, paused, stopped, etc.)
• last_started: Timestamp of last service start
• restart_count: Number of times service has been restarted
• health_score: Overall service health indicator (0-100)

System Resource Metrics:

• memory_usage_bytes: Current memory consumption
• cpu_usage_percent: Current CPU utilization
• goroutines_count: Number of active goroutines
• gc_cycles: Garbage collection statistics

Scraper Service Metrics

Indexing Performance:

• blocks_processed_total: Total number of blocks indexed
• blocks_per_second: Current indexing throughput
• batch_size_current: Current batch size setting
• batch_processing_time_ms: Average time per batch
• index_chunks_created: Number of index chunks generated
• appearances_extracted_total: Total address appearances found

RPC Performance:

• rpc_requests_total: Total RPC requests made
• rpc_requests_failed: Number of failed RPC requests
• rpc_response_time_ms: Average RPC response time
• rpc_rate_limit_hits: Number of rate limit encounters
• rpc_endpoint_health: Status of each configured RPC endpoint

Processing State:

• current_block_number: Latest block being processed
• target_block_number: Target block (chain tip)
• blocks_behind: Number of blocks behind chain tip
• indexing_progress_percent: Overall indexing completion percentage

Monitor Service Metrics

Monitoring Performance:

• addresses_monitored: Number of addresses being tracked
• monitoring_checks_total: Total monitoring checks performed
• activity_detected_total: Number of activities detected
• notifications_sent_total: Number of notifications dispatched
• false_positives: Number of false positive detections

Detection Metrics:

• detection_latency_ms: Time from block to activity detection
• monitoring_batch_size: Current batch size for monitoring
• monitoring_frequency_seconds: Current monitoring interval

API Service Metrics

Request Performance:

• api_requests_total: Total API requests served
• api_requests_per_second: Current request throughput
• api_response_time_ms: Average response time
• api_errors_total: Number of API errors
• api_cache_hits: Number of cache hits
• api_cache_misses: Number of cache misses

Endpoint Metrics:

• status_endpoint_calls: Calls to status endpoints
• index_endpoint_calls: Calls to index query endpoints
• monitor_endpoint_calls: Calls to monitor endpoints
• admin_endpoint_calls: Calls to admin endpoints

IPFS Service Metrics

Network Performance:

• ipfs_peers_connected: Number of connected IPFS peers
• ipfs_data_uploaded_bytes: Total data uploaded to IPFS
• ipfs_data_downloaded_bytes: Total data downloaded from IPFS
• ipfs_pin_operations: Number of pin operations performed
• ipfs_chunks_shared: Number of index chunks shared

Synchronization Metrics:

• ipfs_sync_operations: Number of sync operations
• ipfs_sync_latency_ms: Average sync operation time
• ipfs_failed_retrievals: Number of failed chunk retrievals

Accessing Service Metrics

REST API Access

Metrics are available through the Control Service API:

# Get metrics for all services
curl http://localhost:8080/api/v1/metrics

# Get metrics for specific service
curl http://localhost:8080/api/v1/services/scraper/metrics

# Get detailed metrics with verbose output
curl http://localhost:8080/api/v1/services/scraper?verbose=true&include=metrics

# Get metrics in different formats
curl http://localhost:8080/api/v1/metrics?format=json
curl http://localhost:8080/api/v1/metrics?format=prometheus

CLI Access

# Show basic service status with key metrics
khedra status --metrics

# Show detailed metrics for all services
khedra metrics

# Show metrics for specific service
khedra metrics --service=scraper

# Export metrics to file
khedra metrics --output=/path/to/metrics.json

# Watch metrics in real-time
khedra metrics --watch --interval=5s

Programmatic Access

// Example: Getting service metrics programmatically
import "github.com/TrueBlocks/trueblocks-khedra/v5/pkg/client"

client := client.NewKhedraClient("http://localhost:8080")
metrics, err := client.GetServiceMetrics("scraper")
if err != nil {
    log.Fatal(err)
}

fmt.Printf("Blocks per second: %f\n", metrics["blocks_per_second"])
fmt.Printf("Memory usage: %d bytes\n", metrics["memory_usage_bytes"])

Interpreting Metrics

Performance Health Indicators

Scraper Service Health:

• Healthy: blocks_per_second > 10, rpc_response_time_ms < 500, memory_usage_bytes stable
• Warning: blocks_per_second < 5, rpc_response_time_ms > 1000, blocks_behind > 1000
• Critical: blocks_per_second < 1, rpc_requests_failed > 10%, memory_usage_bytes increasing rapidly

Monitor Service Health:

• Healthy: detection_latency_ms < 30000, false_positives < 5%, all monitored addresses active
• Warning: detection_latency_ms > 60000, false_positives > 10%, some addresses not responding
• Critical: detection_latency_ms > 300000, false_positives > 25%, monitoring completely behind

API Service Health:

• Healthy: api_response_time_ms < 100, api_errors_total < 1%, api_cache_hits > 80%
• Warning: api_response_time_ms > 500, api_errors_total > 5%, api_cache_hits < 60%
• Critical: api_response_time_ms > 2000, api_errors_total > 15%, service unresponsive

Resource Utilization Thresholds

Memory Usage:

• Normal: < 2GB per service
• High: 2-4GB per service (monitor for leaks)
• Critical: > 4GB per service (immediate attention required)

CPU Usage:

• Normal: < 50% average
• High: 50-80% average (acceptable under load)
• Critical: > 80% sustained (performance degradation likely)

Metrics-Based Troubleshooting

High Resource Usage

High Memory Usage:

# Check memory metrics
curl http://localhost:8080/api/v1/services/scraper/metrics | jq '.memory_usage_bytes'

# If memory usage is high:
# 1. Reduce batch size
# 2. Increase sleep interval
# 3. Check for memory leaks in logs
# 4. Restart service if memory continues growing

High CPU Usage:

# Check CPU metrics and goroutine count
curl http://localhost:8080/api/v1/metrics | jq '.cpu_usage_percent, .goroutines_count'

# If CPU usage is high:
# 1. Reduce batch size
# 2. Increase sleep interval
# 3. Check for infinite loops in logs
# 4. Verify RPC endpoint performance

Performance Degradation

Slow Indexing:

# Check indexing performance
curl http://localhost:8080/api/v1/services/scraper/metrics | jq '.blocks_per_second, .rpc_response_time_ms'

# Troubleshooting steps:
# 1. Check RPC response times
# 2. Verify network connectivity
# 3. Adjust batch size based on performance
# 4. Check for rate limiting

API Response Delays:

# Check API performance
curl http://localhost:8080/api/v1/services/api/metrics | jq '.api_response_time_ms, .api_cache_hits'

# Troubleshooting steps:
# 1. Check cache hit ratio
# 2. Verify index integrity
# 3. Monitor concurrent request load
# 4. Check for slow database queries

Service Failures

RPC Connection Issues:

# Check RPC health metrics
curl http://localhost:8080/api/v1/services/scraper/metrics | jq '.rpc_requests_failed, .rpc_rate_limit_hits'

# Troubleshooting steps:
# 1. Test RPC endpoints directly
# 2. Increase sleep intervals if rate limited
# 3. Switch to backup RPC endpoints
# 4. Check network connectivity

Alerting and Monitoring Setup

Prometheus Integration

Khedra can export metrics in Prometheus format for integration with monitoring systems:

# prometheus.yml configuration
scrape_configs:
  - job_name: 'khedra'
    static_configs:
      - targets: ['localhost:8080']
    metrics_path: '/api/v1/metrics'
    params:
      format: ['prometheus']
    scrape_interval: 30s

Grafana Dashboard

Key metrics to monitor in Grafana:

Performance Dashboard:

• Blocks per second (Scraper)
• API response times
• Memory and CPU usage
• RPC response times

Health Dashboard:

• Service uptime
• Error rates
• Detection latency
• System resource utilization

Alerting Rules

Example alerting rules for common issues:

# Slow indexing alert
- alert: SlowIndexing
  expr: khedra_blocks_per_second < 5
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Khedra indexing is slow"
    description: "Indexing rate is {{ $value }} blocks/sec, below threshold"

# High memory usage alert
- alert: HighMemoryUsage
  expr: khedra_memory_usage_bytes > 4000000000
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "High memory usage detected"
    description: "Memory usage is {{ $value }} bytes, above 4GB threshold"

# API response time alert
- alert: SlowAPI
  expr: khedra_api_response_time_ms > 1000
  for: 3m
  labels:
    severity: warning
  annotations:
    summary: "API responses are slow"
    description: "Average response time is {{ $value }}ms"

Custom Monitoring Scripts

#!/bin/bash
# Simple monitoring script
METRICS_URL="http://localhost:8080/api/v1/metrics"

# Check blocks per second
BPS=$(curl -s $METRICS_URL | jq -r '.scraper.blocks_per_second // 0')
if (( $(echo "$BPS < 5" | bc -l) )); then
    echo "WARNING: Slow indexing detected: $BPS blocks/sec"
fi

# Check memory usage
MEMORY=$(curl -s $METRICS_URL | jq -r '.scraper.memory_usage_bytes // 0')
if (( MEMORY > 4000000000 )); then
    echo "CRITICAL: High memory usage: $((MEMORY/1024/1024))MB"
fi

# Check API health
API_TIME=$(curl -s $METRICS_URL | jq -r '.api.api_response_time_ms // 0')
if (( $(echo "$API_TIME > 1000" | bc -l) )); then
    echo "WARNING: Slow API responses: ${API_TIME}ms"
fi

Best Practices for Metrics Monitoring

Regular Monitoring

1. Establish Baselines: Monitor metrics during normal operation to establish performance baselines
2. Set Appropriate Thresholds: Configure alerts based on your specific environment and requirements
3. Monitor Trends: Look for gradual degradation over time, not just immediate issues
4. Correlate Metrics: Use multiple metrics together to diagnose issues accurately

Performance Optimization

1. Use Metrics for Tuning: Adjust batch sizes and sleep intervals based on actual performance metrics
2. Monitor Resource Efficiency: Track resource usage to optimize system utilization
3. Identify Bottlenecks: Use metrics to identify which component is limiting performance
4. Validate Changes: Use metrics to verify that configuration changes improve performance

Operational Excellence

1. Automate Monitoring: Set up automated alerts for critical metrics
2. Create Dashboards: Visualize key metrics for easier monitoring
3. Document Thresholds: Maintain documentation of what constitutes healthy vs. problematic metrics
4. Regular Reviews: Periodically review and adjust monitoring thresholds based on operational experience

Integration Points

Integration with External APIs

Khedra exposes data through a REST API, making it compatible with external applications. Example use cases:

• Fetching transaction details for a given address.
• Retrieving block information for analysis.

Interfacing with IPFS

Data indexed by Khedra can be pinned to IPFS for decentralized storage:

./khedra --ipfs on

Customizing for Specific Use Cases

Users can tailor the configuration by:

• Adjusting .env variables to include specific chains and RPC endpoints.
• Writing custom scripts to query the REST API and process the data.

Testing and Validation

Unit Testing

Unit tests cover:

• Blockchain indexing logic.
• Configuration parsing and validation.
• REST API endpoint functionality.

Run tests with:

go test ./...

Integration Testing

Integration tests ensure all components work together as expected. Tests include:

• RPC connectivity validation.
• Multi-chain indexing workflows.

Testing Guidelines for Developers

1. Use mock RPC endpoints for testing without consuming live resources.
2. Validate .env configuration in test environments before deployment.
3. Automate tests with CI/CD pipelines to ensure reliability.

Appendices

Glossary of Technical Terms

• EVM: Ethereum Virtual Machine, the runtime environment for smart contracts.
• RPC: Remote Procedure Call, a protocol for interacting with blockchain nodes.
• IPFS: InterPlanetary File System, a decentralized storage solution.

References and Resources

• TrueBlocks GitHub Repository
• TrueBlocks Official Website
• Ethereum Developer Documentation
• IPFS Documentation

Index

• Address Monitoring: Section 3, Core Functionalities
• API Access: Section 3, Core Functionalities
• Architecture Overview: Section 2, System Architecture
• Blockchain Indexing: Section 3, Core Functionalities
• Configuration Files: Section 4, Technical Design
• Data Flow: Section 4, Technical Design
• Error Handling: Section 4, Technical Design
• Integration Points: Section 8, Integration Points
• IPFS Integration: Section 3, Core Functionalities; Section 8, Integration Points
• Logging: Section 4, Technical Design
• Performance Benchmarks: Section 7, Performance and Scalability
• REST API: Section 3, Core Functionalities; Section 8, Integration Points
• RPC Requirements: Section 5, Supported Chains and RPCs
• Scalability Strategies: Section 7, Performance and Scalability
• System Components: Section 2, System Architecture
• Testing Guidelines: Section 9, Testing and Validation