Skip to main content

Using RL Control Agents

What RL Control Agents Do​

RL control agents learn optimal control policies through interaction with environments. They use the algorithms from JLab's SOCT (SciOptControlToolkit) to train deep neural networks that map states to actions.

Key capability: Continuous learning from streaming data without manual intervention.

Configuration​

Basic Configuration​

rl_control_agent1:
  # Environment
  environment: "Pendulum-v1"
  
  # Algorithm
  soct_agent_type: "KerasTD3-v0"
  soct_agent_config_path: "keras_td3.cfg"
  
  # Buffer
  buffer_type: "ER-v0"
  buffer_size: 1000000
  
  # Threads
  enabled_threads: ['ingest', 'training', 'inference']
  
  # Topics
  kafka_topics:
    input_sarsa: "gymnasium-sarsa"
    input_state: "gymnasium-state"
    output_action: "gymnasium-action"
  
  # Pipeline sync
  data_ingest:
    use_pipeline_sync: true
    pipeline_timeout_sec: 100.0
  
  training:
    check_interval_ms: 10
    use_pipeline_sync: true
    pipeline_timeout_sec: 100.0
  
  inference:
    train_mode: true
    use_pipeline_sync: true
    pipeline_timeout_sec: 100.0

Key Parameters​

soct_agent_type: Algorithm choice

  • "KerasTD3-v0": Twin Delayed DDPG (default)
  • "KerasSAC-v0": Soft Actor-Critic

buffer_size: Experience replay capacity

  • Typical: 1,000,000
  • Trade-off: Memory vs training diversity

train_mode: Exploration vs exploitation

  • true: Add exploration noise (training)
  • false: Deterministic actions (testing)

use_pipeline_sync: Thread coordination

  • true: Strict ordering (inference → ingest → train)
  • false: Independent operation (not recommended currently)

Starting an RL Agent​

# .env
COMPOSE_PROFILES=gymnasium,rl1

docker compose up rl-control-agent1

Verify Startup​

Check logs:

docker compose logs rl-control-agent1 | head -50

Look for:

Agent initialized
Created SOCT agent: KerasTD3-v0
Buffer capacity: 1000000
Warmup size: 2500
Data Ingest Thread initialized
ML Training Thread initialized
ML Inference Thread initialized

Pipeline Synchronization​

Three-Way Handshake​

The agent enforces strict ordering:

1. INFERENCE: Generate action Aₜ for state Sₜ
              Signal: inference_done_event
              
2. INGEST:    Wait for inference
              Store SARSA(Sₜ₋₁, Aₜ₋₁, Rₜ₋₁, Sₜ, done)
              Signal: ingestion_done_event
              
3. TRAINING:  Wait for ingestion
              Train on new experience
              Signal: training_done_event
              
4. INFERENCE: Wait for training (back to step 1)

Purpose: Ensure consistent state-action-reward sequences

When to Disable Sync​

Set use_pipeline_sync: false if:

  • Training is too slow (GPU-bound)
  • Want asynchronous operation
  • Debugging thread issues

Caution: May cause instability in training!

Monitoring Training​

InfluxDB​

If youre instance of SMOCS has InfluxDB running, statistics from the environment will be published to help understand the runtime.

Since SOCT is built around utilizing Tensorboard for logging, we provide those logs to the user by mounting them in the orchestration folder and give the user access to them as they need.

TensorBoard​

Start TensorBoard:

tensorboard --logdir ./tb-logs

Access: http://localhost:6006

Key metrics:

  • episode_reward: Cumulative reward per episode
  • actor_loss: Policy network loss
  • critic_loss: Value network loss
  • buffer_size: Number of experiences stored