Maintenance and Troubleshooting
This chapter covers routine maintenance tasks and solutions to common issues you might encounter when using Khedra.
Routine Maintenance
Regular Updates
To keep Khedra running smoothly, periodically check for and install updates:
# Check current version
khedra version
# Update to the latest version
go get -u github.com/TrueBlocks/trueblocks-khedra/v5
# Rebuild and install
cd <path_for_khedra_github_repo>
git pull --recurse-submodules
go build -o bin/khedra main.go
./bin/khedra version
Log Rotation
Khedra automatically rotates logs based on your configuration, but you should periodically check log usage:
# Check log directory size
du -sh ~/.khedra/logs
# List log files
ls -la ~/.khedra/logs
If logs are consuming too much space, adjust your logging configuration:
logging:
maxSize: 10 # Maximum size in MB before rotation
maxBackups: 5 # Number of rotated files to keep
maxAge: 30 # Days to keep rotated logs
compress: true # Compress rotated logs
Index Verification
Periodically verify the integrity of your Unchained Index:
chifra chunks index --check --chain <chain_name>
This checks for any gaps or inconsistencies in the index and reports issues.
Cache Management
You may check on the cache size and prune old caches (by hand) to free up space:
# Check cache size
chifra status --verbose
Troubleshooting
Common Issues and Solutions
Service Won't Start
Symptoms: A service fails to start or immediately stops.
Solutions:
-
Check the logs for error messages:
tail -n 100 ~/.khedra/logs/khedra.log
-
Verify the service's port isn't in use by another application:
lsof -i :<port_number>
-
Ensure the RPC endpoints are accessible:
chifra status
-
Try starting with verbose logging:
TB_KHEDRA_LOGGING_LEVEL=debug TB_KHEDRA_LOGGING_TOFILE=true khedra start
Service-Specific Troubleshooting
Scraper Service Issues
Symptoms: Scraper service fails to start, stops unexpectedly, or indexes slowly.
Common Issues and Solutions:
-
RPC Connection Failures:
# Test RPC connectivity curl -X POST -H "Content-Type: application/json" \ --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \ http://your-rpc-endpoint # Check RPC provider limits grep -i "rate limit\|too many requests" ~/.khedra/logs/khedra.log
-
Batch Size Optimization:
# For fast RPC endpoints services: scraper: batchSize: 2000 sleep: 5 # For slower/limited RPC endpoints services: scraper: batchSize: 100 sleep: 30
-
Memory Issues:
# Monitor scraper memory usage ps -o pid,vsz,rss,comm -p $(pgrep -f "scraper") # Reduce batch size if memory usage is high
-
Scraper-Specific Log Analysis:
# Filter scraper logs grep "scraper" ~/.khedra/logs/khedra.log | tail -50 # Look for specific errors grep -E "error|failed|timeout" ~/.khedra/logs/khedra.log | grep scraper
Monitor Service Issues
Symptoms: Monitor service doesn't detect address activity or sends duplicate notifications.
Common Issues and Solutions:
-
No Monitored Addresses:
# Check if addresses are properly configured chifra list --monitors # Add addresses to monitor chifra monitors --addrs 0x742d35Cc6634C0532925a3b844Bc454e4438f44e
-
Monitor Service Dependencies:
# Ensure scraper is running for real-time monitoring curl http://localhost:8080/api/v1/services/scraper # Check if index is up to date chifra status --index
-
Monitor Configuration Issues:
services: monitor: enabled: true sleep: 12 # Check every 12 seconds batchSize: 100 # Process 100 addresses at once
-
Monitor-Specific Logs:
# Filter monitor logs grep "monitor" ~/.khedra/logs/khedra.log | tail -50 # Check for address activity detection grep -i "activity\|appearance" ~/.khedra/logs/khedra.log
API Service Issues
Symptoms: API service returns errors, timeouts, or incorrect data.
Common Issues and Solutions:
-
Port Conflicts:
# Check if API port is available lsof -i :8080 # Change API port if needed export TB_KHEDRA_SERVICES_API_PORT=8081
-
API Performance Issues:
# Test API response time time curl http://localhost:8080/status # Check for slow queries grep -E "slow|timeout" ~/.khedra/logs/khedra.log | grep api
-
API Authentication Issues:
# Verify API is accessible curl -v http://localhost:8080/api/v1/services # Check for auth-related errors grep -i "auth\|unauthorized" ~/.khedra/logs/khedra.log
-
Data Consistency Issues:
# Compare API data with direct index queries chifra list 0x742d35Cc6634C0532925a3b844Bc454e4438f44e curl http://localhost:8080/api/v1/list/0x742d35Cc6634C0532925a3b844Bc454e4438f44e
IPFS Service Issues
Symptoms: IPFS service fails to start, can't connect to network, or sharing fails.
Common Issues and Solutions:
-
IPFS Daemon Issues:
# Check IPFS daemon status ps aux | grep ipfs # Restart IPFS if needed curl -X POST http://localhost:8080/api/v1/services/ipfs/restart
-
IPFS Port Conflicts:
# Check IPFS ports lsof -i :5001 # IPFS API port lsof -i :4001 # IPFS swarm port # Configure different IPFS port export TB_KHEDRA_SERVICES_IPFS_PORT=5002
-
IPFS Network Connectivity:
# Test IPFS connectivity curl http://localhost:5001/api/v0/id # Check peer connections curl http://localhost:5001/api/v0/swarm/peers
-
Index Sharing Issues:
# Check IPFS pinning status curl http://localhost:5001/api/v0/pin/ls # Verify index chunk uploads grep -i "ipfs\|pin" ~/.khedra/logs/khedra.log
Control Service Issues
Symptoms: Cannot manage other services via API or CLI commands fail.
Common Issues and Solutions:
-
Control Service Availability:
# Verify control service is running curl http://localhost:8080/api/v1/services # Check control service logs grep "control" ~/.khedra/logs/khedra.log
-
Service Management Failures:
# Test individual service control curl -X POST http://localhost:8080/api/v1/services/scraper/status # Check for permission issues grep -i "permission\|access denied" ~/.khedra/logs/khedra.log
-
Configuration Issues:
# Verify control service configuration khedra config show | grep -A5 -B5 control # Test configuration validation khedra config validate
Log Analysis
Khedra's logs are your best resource for troubleshooting. Here's how to use them effectively:
# View recent log entries
tail -f ~/.khedra/logs/khedra.log
# Search for error messages
grep -i error ~/.khedra/logs/khedra.log
# Find logs related to a specific service
grep "scraper" ~/.khedra/logs/khedra.log
# Find logs related to a specific address
grep "0x742d35Cc6634C0532925a3b844Bc454e4438f44e" ~/.khedra/logs/khedra.log
Getting Help
If you encounter issues you can't resolve:
- Check the Khedra GitHub repository for known issues
- Search the discussions forum for similar problems
- Submit a detailed issue report including:
- Khedra version (
khedra version
) - Relevant log extracts
- Steps to reproduce the problem
- Your configuration (with sensitive data redacted)
- Khedra version (
Regular maintenance and prompt troubleshooting will keep your Khedra installation running smoothly and efficiently.
Implementation Details
The maintenance and troubleshooting procedures described in this document are implemented in several key files:
Service Management
- Service Lifecycle Management:
app/action_daemon.go
- Contains the core service management code that starts, stops, and monitors services - Service Health Checks: Service status monitoring is implemented in the daemon action function
RPC Connection Management
- RPC Endpoint Testing:
pkg/validate/try_connect.go
- Contains theTestRpcEndpoint
function used to verify endpoints are functioning correctly - RPC Validation:
app/has_valid_rpc.go
- Implements validation logic for RPC endpoints
Logging System
- Log Configuration: Defined in the
Logging
struct inpkg/types/general.go
which handles log rotation and management - Logger Implementation: Custom logger in
pkg/types/custom_logger.go
that provides structured logging capabilities
Error Recovery
The troubleshooting techniques described are supported by robust error handling throughout the codebase, especially in:
- Service error handling: Found in the daemon action function
- Validation error reporting: Implemented in the validation framework
- Index management functions: For identifying and fixing gaps in the index