Skip to main content

Testing Your Agent

Overview​

This guide covers integration testing for SMOCS agents, focusing on verifying agent behavior through logs, and Kafka UI.

Testing approach: Since agents run in containers and interact via Kafka, testing focuses on:

  • Verifying correct startup and initialization
  • Monitoring data flow through logs
  • Validating database storage
  • Confirming Kafka message publishing

Prerequisites​

  • Agent built and configured
  • Docker Compose profiles set
  • .env file configured
  • Kafka UI enabled (optional but helpful)

Step 1: Basic Startup Test​

Start Agent​

# Enable Kafka UI for testing
COMPOSE_PROFILES=gymnasium,threshold1,ui

docker compose up --build threshold-agent1 gymnasium-kafka-controller kafka-ui

Verify Container Running​

docker compose ps

# Should show threshold-agent1 as "running" or "healthy"

Check Startup Logs​

docker compose logs threshold-agent1 | head -50

Look for success indicators:

✓ Agent initialized
✓ Data Ingest Thread initialized
✓ ML Training Thread initialized
✓ ML Inference Thread initialized
✓ Connected to DB: agentdb
✓ Subscribed to topics: ['gymnasium-output']

Look for errors:

✗ Failed to connect to Kafka
✗ Database connection error
✗ Configuration error
✗ Module not found

Common Startup Issues​

"Module not found" error:

# Check PYTHONPATH
docker exec threshold-agent1 echo $PYTHONPATH
# Should show: /app

# Verify smocs package exists
docker exec threshold-agent1 ls /app/smocs

Database connection failed:

# Check MySQL is running
docker exec threshold-agent1 mysqladmin ping -u root -p

# Check database exists
docker exec threshold-agent1 mysql -u root -p -e "SHOW DATABASES;"

Kafka connection failed:

# Verify Kafka is healthy
docker compose ps kafka-broker

# Test connectivity
docker exec threshold-agent1 nc -zv kafka-broker 9092

Step 2: Test Data Ingestion​

Monitor Ingestion Logs​

docker compose logs -f threshold-agent1 | grep "DataIngestThread"

Expected output:

DataIngestThread: Extracted 3 channels for processing
DataIngestThread: Stored sensor data: 3 channels
DataIngestThread: Extracted 3 channels for processing
DataIngestThread: Stored sensor data: 3 channels

Step 3: Test Training Thread​

Monitor Training Logs​

docker compose logs -f threshold-agent1 | grep "MLTrainingThread"

Before training:

MLTrainingThread: Database contains 150 samples (need 500)
MLTrainingThread: Not enough samples for training

During training:

MLTrainingThread: Found 600 samples
MLTrainingThread: Retrieved 600 samples for threshold calculation
MLTrainingThread: Calculated thresholds: {'state_0': 0.82, 'state_1': 0.79}
MLTrainingThread: Model evaluation complete

Verify Model Files Created​

# List model files
docker exec threshold-agent1 ls -lh /app/models

# View threshold file
docker exec threshold-agent1 cat /app/models/current_thresholds.json

Expected content:

{
  "thresholds": {
    "state_0": 0.82,
    "state_1": 0.79,
    "state_2": 5.95
  },
  "metrics": {...},
  "timestamp": 1704067200.0
}

Test Training Frequency​

Monitor training cycles:

docker compose logs threshold-agent1 | grep "Training complete"

# Count training events
docker compose logs threshold-agent1 | grep "Training complete" | wc -l

Verify Alerts Published​

Check Kafka UI:

  1. Open http://localhost:8080
  2. Navigate to "Topics"
  3. Select "threshold-alerts"
  4. View messages

Expected message format:

{
  "timestamp": 1704067200.123,
  "channels": {
    "agent_id": "threshold-agent1",
    "has_violation": true,
    "violation_count": 1,
    "state_0_value": 0.95,
    "state_0_threshold": 0.82,
    "state_0_exceeded_by": 0.13
  }
}

Test Inference Without Training​

Scenario: Inference thread starts before training completes

Expected behavior: Uses static thresholds from config

Verify:

# Check for static threshold usage
docker compose logs threshold-agent1 | grep "using static thresholds"

Step 5: Test End-to-End Flow​

Full System Test​

Setup:

  1. Clean database: docker compose down -v
  2. Start system: docker compose up --build
  3. Wait 5 minutes

Performance Check​

Message processing rate:

# Count messages in 30 seconds
docker compose logs --since 30s threshold-agent1 | grep "Stored sensor data" | wc -l

Expected: Should match Gymnasium output rate

Check for lag:

# Look for timeout warnings
docker compose logs threshold-agent1 | grep -i timeout

Step 6: Test Error Handling​

Invalid Message Format​

Test without channels:

# Publish invalid message to Kafka
docker exec kafka-broker kafka-console-producer.sh \
  --topic gymnasium-output \
  --bootstrap-server localhost:9092 << EOF
{"timestamp": 1234567890}
EOF

Expected logs:

DataIngestThread: No channels in message
# Should continue processing, not crash

Database Unavailable​

Test recovery:

# Stop MySQL inside container
docker exec threshold-agent1 service mysql stop

# Wait for error logs
docker compose logs -f threshold-agent1

# Restart MySQL
docker exec threshold-agent1 service mysql start

# Verify recovery
docker compose logs -f threshold-agent1 | grep "Connected to DB"

Kafka Unavailable​

Test behavior:

# Stop Kafka
docker compose stop kafka-broker

# Check agent logs
docker compose logs threshold-agent1

# Restart Kafka
docker compose start kafka-broker

Expected: Agent should reconnect automatically

Step 7: Kafka UI Debugging​

View Topics​

Access: http://localhost:8080

Navigate: Topics → Select topic

Inspect Messages​

Input messages (gymnasium-output):

  • Click "Messages" tab
  • Set "Live mode" to ON
  • Observe incoming messages
  • Verify format matches expected

Output messages (threshold-alerts):

  • Check message count
  • Verify only published on violations
  • Inspect message content

Monitor Consumer Groups​

Navigate: Consumers → Select group

Check metrics:

  • Lag: Should be near 0
  • Offset: Should be increasing
  • Members: Should show 1 consumer

Troubleshooting Guide​

Agent Not Processing Messages​

Check:

  1. Kafka topic exists: Kafka UI
  2. Agent subscribed: Look for "Subscribed to topics" in logs
  3. Messages available: Check Kafka UI message count
  4. Consumer group active: Kafka UI → Consumers

Data Not Storing to Database​

Check:

  1. Database accessible: mysqladmin ping
  2. Tables exist: SHOW TABLES;
  3. Permissions correct: SHOW GRANTS;
  4. Logs for SQL errors: grep "DB Error"

Training Not Occurring​

Check:

  1. Sufficient data: Query agent_inferences count
  2. Thread running: grep "MLTrainingThread" logs
  3. Configuration correct: Verify min_training_samples
  4. No errors in training logic: grep "Error.*training"

Inference Not Working​

Check:

  1. Model/thresholds loaded: ls /app/models
  2. Input format correct: Log sample message
  3. Output topic created: Kafka UI
  4. Thread running: grep "MLInferenceThread" logs

Clean state between tests:

docker compose down -v  # Remove all data
docker compose up --build  # Fresh start