Research Note CRL July 4th

LLama error

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [04:00<12:00, 240.23s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [05:46<05:22, 161.27s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [07:13<02:07, 127.33s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [07:56<00:00, 94.20s/it] 
Loading checkpoint shards: 100%|██████████| 4/4 [07:56<00:00, 119.14s/it]
wandb: Currently logged in as: seonglae (texonom). Use `wandb login --relogin` to force relogin
wandb: Using wandb-core as the SDK backend.  Please refer to https://wandb.me/wandb-core for more information.
wandb: Tracking run with wandb version 0.19.4
wandb: Run data is saved locally in /cs/student/projects2/aisd/2024/seongcho/control-ai/wandb/run-20250722_015756-0bl1xzna
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run llama8_simpleqa_0_ppo_1e-05_0722_015756_1.0
wandb: ⭐️ View project at https://wandb.ai/texonom/control_rl
wandb: 🚀 View run at https://wandb.ai/texonom/control_rl/runs/0bl1xzna

Training Steps:   0%|          | 0/2001 [00:00<?, ?it/s]/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:629: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
  warnings.warn(
/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:634: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
  warnings.warn(

Training Steps:   0%|          | 1/2001 [00:20<11:21:17, 20.44s/it]
Training Steps:   0%|          | 1/2001 [00:20<11:35:03, 20.85s/it]
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 414, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 765, in train
    train_metrics = self.perform_training_step(batch)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 240, in perform_training_step
    layer_metrics = self.backward(batch_rewards, eos_position)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 213, in backward
    policy_loss, critic_loss, policy_gn, critic_gn = self.ppo_trainers[layer].train_step(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/ppo.py", line 218, in train_step
    sample_dist: torch.distributions.Normal = torch.distributions.Normal(sample_mean, sample_sigma)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/distributions/normal.py", line 59, in __init__
    super().__init__(batch_shape, validate_args=validate_args)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/distributions/distribution.py", line 71, in __init__
    raise ValueError(
ValueError: Expected parameter loc (Tensor of shape (1, 32768)) of distribution Normal(loc: torch.Size([1, 32768]), scale: torch.Size([1, 32768])) to satisfy the constraint Real(), but found invalid values:
tensor([[nan, nan, nan,  ..., nan, nan, nan]], device='cuda:0',
       dtype=torch.bfloat16, grad_fn=<TanhBackward0>)
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 414, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 765, in train
    train_metrics = self.perform_training_step(batch)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 240, in perform_training_step
    layer_metrics = self.backward(batch_rewards, eos_position)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 213, in backward
    policy_loss, critic_loss, policy_gn, critic_gn = self.ppo_trainers[layer].train_step(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/ppo.py", line 218, in train_step
    sample_dist: torch.distributions.Normal = torch.distributions.Normal(sample_mean, sample_sigma)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/distributions/normal.py", line 59, in __init__
    super().__init__(batch_shape, validate_args=validate_args)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/distributions/distribution.py", line 71, in __init__
    raise ValueError(
ValueError: Expected parameter loc (Tensor of shape (1, 32768)) of distribution Normal(loc: torch.Size([1, 32768]), scale: torch.Size([1, 32768])) to satisfy the constraint Real(), but found invalid values:
tensor([[nan, nan, nan,  ..., nan, nan, nan]], device='cuda:0',
       dtype=torch.bfloat16, grad_fn=<TanhBackward0>)

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [02:07<06:23, 127.89s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [04:09<04:08, 124.45s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [06:00<01:58, 118.12s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [06:43<00:00, 88.61s/it] 
Loading checkpoint shards: 100%|██████████| 4/4 [06:43<00:00, 100.98s/it]
wandb: Currently logged in as: seonglae (texonom). Use `wandb login --relogin` to force relogin
wandb: Using wandb-core as the SDK backend.  Please refer to https://wandb.me/wandb-core for more information.
wandb: Tracking run with wandb version 0.19.4
wandb: Run data is saved locally in /cs/student/projects2/aisd/2024/seongcho/control-ai/wandb/run-20250722_015800-il6u3606
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run llama8_wmdp_0_ppo_1e-05_0722_015759_1.0_select
wandb: ⭐️ View project at https://wandb.ai/texonom/control_rl
wandb: 🚀 View run at https://wandb.ai/texonom/control_rl/runs/il6u3606

Training Steps:   0%|          | 0/2001 [00:00<?, ?it/s]/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:629: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
  warnings.warn(
/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:634: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
  warnings.warn(

Training Steps:   0%|          | 0/2001 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 414, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 765, in train
    train_metrics = self.perform_training_step(batch)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 240, in perform_training_step
    layer_metrics = self.backward(batch_rewards, eos_position)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 213, in backward
    policy_loss, critic_loss, policy_gn, critic_gn = self.ppo_trainers[layer].train_step(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/ppo.py", line 239, in train_step
    total_loss.backward()
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/_tensor.py", line 581, in backward
    torch.autograd.backward(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward
    _engine_run_backward(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/autograd/graph.py", line 825, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [CUDABFloat16Type [2, 4096]] is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 414, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 765, in train
    train_metrics = self.perform_training_step(batch)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 240, in perform_training_step
    layer_metrics = self.backward(batch_rewards, eos_position)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 213, in backward
    policy_loss, critic_loss, policy_gn, critic_gn = self.ppo_trainers[layer].train_step(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/ppo.py", line 239, in train_step
    total_loss.backward()
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/_tensor.py", line 581, in backward
    torch.autograd.backward(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward
    _engine_run_backward(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/autograd/graph.py", line 825, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [CUDABFloat16Type [2, 4096]] is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [01:22<04:06, 82.07s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [03:37<03:47, 113.75s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [06:48<02:28, 148.99s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [07:18<00:00, 101.70s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [07:18<00:00, 109.53s/it]
wandb: Currently logged in as: seonglae (texonom). Use `wandb login --relogin` to force relogin
wandb: Using wandb-core as the SDK backend.  Please refer to https://wandb.me/wandb-core for more information.
wandb: Tracking run with wandb version 0.19.4
wandb: Run data is saved locally in /cs/student/projects2/aisd/2024/seongcho/control-ai/wandb/run-20250722_015459-1rmzv795
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run llama8_mmlu_0_ppo_1e-05_0722_015459_1.0_select
wandb: ⭐️ View project at https://wandb.ai/texonom/control_rl
wandb: 🚀 View run at https://wandb.ai/texonom/control_rl/runs/1rmzv795

Training Steps:   0%|          | 0/2001 [00:00<?, ?it/s]/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:629: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
  warnings.warn(
/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:634: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
  warnings.warn(

Training Steps:   0%|          | 0/2001 [00:02<?, ?it/s]
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 414, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 765, in train
    train_metrics = self.perform_training_step(batch)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 240, in perform_training_step
    layer_metrics = self.backward(batch_rewards, eos_position)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 213, in backward
    policy_loss, critic_loss, policy_gn, critic_gn = self.ppo_trainers[layer].train_step(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/ppo.py", line 239, in train_step
    total_loss.backward()
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/_tensor.py", line 581, in backward
    torch.autograd.backward(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward
    _engine_run_backward(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/autograd/graph.py", line 825, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [CUDABFloat16Type [2, 4096]] is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 414, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 765, in train
    train_metrics = self.perform_training_step(batch)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 240, in perform_training_step
    layer_metrics = self.backward(batch_rewards, eos_position)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 213, in backward
    policy_loss, critic_loss, policy_gn, critic_gn = self.ppo_trainers[layer].train_step(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/ppo.py", line 239, in train_step
    total_loss.backward()
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/_tensor.py", line 581, in backward
    torch.autograd.backward(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward
    _engine_run_backward(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/autograd/graph.py", line 825, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [CUDABFloat16Type [2, 4096]] is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Loading checkpoint shards: 100%|██████████| 4/4 [05:09<00:00, 68.56s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [05:09<00:00, 77.33s/it]
Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
Found the latest cached dataset configuration 'all' at /cs/student/msc/aisd/2024/seongcho/.cache/huggingface/datasets/cais___mmlu/all/0.0.0/c30699e8356da336a370243923dbaf21066bb9fe (last modified on Tue Apr 22 01:07:44 2025).
wandb: Currently logged in as: seonglae (texonom). Use `wandb login --relogin` to force relogin
wandb: Using wandb-core as the SDK backend.  Please refer to https://wandb.me/wandb-core for more information.
wandb: Tracking run with wandb version 0.19.4
wandb: Run data is saved locally in /cs/student/projects2/aisd/2024/seongcho/steer-rl/wandb/run-20250718_114418-t5tnvab8
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run llama8_mmlu_0_ppo_1e-05_0718_114418_1.0
wandb: ⭐️ View project at https://wandb.ai/texonom/control_rl
wandb: 🚀 View run at https://wandb.ai/texonom/control_rl/runs/t5tnvab8

Training Steps:   0%|          | 0/2001 [00:00<?, ?it/s]/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:629: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
  warnings.warn(
/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:634: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
  warnings.warn(

Training Steps:   0%|          | 1/2001 [00:30<17:08:04, 30.84s/it]
Training Steps:   0%|          | 1/2001 [00:31<17:18:45, 31.16s/it]
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/steer-rl/experiment.py", line 331, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/steer-rl/experiment.py", line 195, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/steer-rl/experiment.py", line 95, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/steer-rl/train.py", line 500, in train
    train_metrics = self.perform_training_step(batch)
  File "/cs/student/projects2/aisd/2024/seongcho/steer-rl/train.py", line 228, in perform_training_step
    policy_loss, critic_loss, policy_gn, critic_gn = ppo_trainers[layer].train_step(
  File "/cs/student/projects2/aisd/2024/seongcho/steer-rl/control_rl/ppo.py", line 185, in train_step
    sample_dist: torch.distributions.Normal = torch.distributions.Normal(sample_mean, sample_sigma)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/distributions/normal.py", line 59, in __init__
    super().__init__(batch_shape, validate_args=validate_args)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/distributions/distribution.py", line 71, in __init__
    raise ValueError(
ValueError: Expected parameter loc (Tensor of shape (1, 32768)) of distribution Normal(loc: torch.Size([1, 32768]), scale: torch.Size([1, 32768])) to satisfy the constraint Real(), but found invalid values:
tensor([[nan, nan, nan,  ..., nan, nan, nan]], device='cuda:0',
       dtype=torch.bfloat16, grad_fn=<TanhBackward0>)
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/steer-rl/experiment.py", line 331, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/steer-rl/experiment.py", line 195, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/steer-rl/experiment.py", line 95, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/steer-rl/train.py", line 500, in train
    train_metrics = self.perform_training_step(batch)
  File "/cs/student/projects2/aisd/2024/seongcho/steer-rl/train.py", line 228, in perform_training_step
    policy_loss, critic_loss, policy_gn, critic_gn = ppo_trainers[layer].train_step(
  File "/cs/student/projects2/aisd/2024/seongcho/steer-rl/control_rl/ppo.py", line 185, in train_step
    sample_dist: torch.distributions.Normal = torch.distributions.Normal(sample_mean, sample_sigma)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/distributions/normal.py", line 59, in __init__
    super().__init__(batch_shape, validate_args=validate_args)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/distributions/distribution.py", line 71, in __init__
    raise ValueError(
ValueError: Expected parameter loc (Tensor of shape (1, 32768)) of distribution Normal(loc: torch.Size([1, 32768]), scale: torch.Size([1, 32768])) to satisfy the constraint Real(), but found invalid values:
tensor([[nan, nan, nan,  ..., nan, nan, nan]], device='cuda:0',
       dtype=torch.bfloat16, grad_fn=<TanhBackward0>)

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [01:31<04:33, 91.18s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [02:49<02:46, 83.45s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [05:32<01:59, 119.94s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [06:40<00:00, 99.36s/it] 
Loading checkpoint shards: 100%|██████████| 4/4 [06:40<00:00, 100.10s/it]

Filter:   0%|          | 0/58492 [00:00<?, ? examples/s]
Filter:   2%|▏         | 1000/58492 [00:00<00:23, 2403.84 examples/s]
Filter:  15%|█▌        | 9000/58492 [00:00<00:02, 21510.15 examples/s]
Filter:  29%|██▉       | 17000/58492 [00:00<00:01, 35414.69 examples/s]
Filter:  43%|████▎     | 25000/58492 [00:00<00:00, 45531.37 examples/s]
Filter:  56%|█████▋    | 33000/58492 [00:00<00:00, 53279.67 examples/s]
Filter:  70%|███████   | 41000/58492 [00:00<00:00, 59113.30 examples/s]
Filter:  84%|████████▍ | 49000/58492 [00:01<00:00, 63006.92 examples/s]
Filter: 100%|██████████| 58492/58492 [00:01<00:00, 40897.14 examples/s]
Filter: 100%|██████████| 58492/58492 [00:01<00:00, 37378.56 examples/s]
wandb: Currently logged in as: seonglae (texonom). Use `wandb login --relogin` to force relogin
wandb: Using wandb-core as the SDK backend.  Please refer to https://wandb.me/wandb-core for more information.
wandb: Tracking run with wandb version 0.19.4
wandb: Run data is saved locally in /cs/student/projects2/aisd/2024/seongcho/control-ai/wandb/run-20250722_015511-t9jjyvmm
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run llama8_bbq_0_ppo_1e-05_0722_015511_-5.0_select
wandb: ⭐️ View project at https://wandb.ai/texonom/control_rl
wandb: 🚀 View run at https://wandb.ai/texonom/control_rl/runs/t9jjyvmm

Training Steps:   0%|          | 0/2001 [00:00<?, ?it/s]/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:629: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
  warnings.warn(
/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:634: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
  warnings.warn(

Training Steps:   0%|          | 0/2001 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 414, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 765, in train
    train_metrics = self.perform_training_step(batch)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 240, in perform_training_step
    layer_metrics = self.backward(batch_rewards, eos_position)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 213, in backward
    policy_loss, critic_loss, policy_gn, critic_gn = self.ppo_trainers[layer].train_step(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/ppo.py", line 239, in train_step
    total_loss.backward()
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/_tensor.py", line 581, in backward
    torch.autograd.backward(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward
    _engine_run_backward(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/autograd/graph.py", line 825, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [CUDABFloat16Type [2, 4096]] is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 414, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 765, in train
    train_metrics = self.perform_training_step(batch)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 240, in perform_training_step
    layer_metrics = self.backward(batch_rewards, eos_position)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 213, in backward
    policy_loss, critic_loss, policy_gn, critic_gn = self.ppo_trainers[layer].train_step(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/ppo.py", line 239, in train_step
    total_loss.backward()
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/_tensor.py", line 581, in backward
    torch.autograd.backward(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward
    _engine_run_backward(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/autograd/graph.py", line 825, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [CUDABFloat16Type [2, 4096]] is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Gemma

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  1.58it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  2.93it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  2.60it/s]
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 414, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 723, in train
    train_loader, val_loader, _ = load_dataloaders(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/utils.py", line 296, in load_dataloaders
    val_loader = dataset_config[task].dataloader(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/dataset.py", line 153, in __init__
    super().__init__(dataset, split=split, limit=limit)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/dataset.py", line 19, in __init__
    self.data = self.data.select(range(limit))
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 567, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/datasets/fingerprint.py", line 482, in wrapper
    out = func(dataset, *args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3949, in select
    return self._select_contiguous(start, length, new_fingerprint=new_fingerprint)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 567, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/datasets/fingerprint.py", line 482, in wrapper
    out = func(dataset, *args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 4010, in _select_contiguous
    _check_valid_indices_value(start + length - 1, len(self))
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 659, in _check_valid_indices_value
    raise IndexError(f"Index {index} out of range for dataset of size {size}.")
IndexError: Index 149 out of range for dataset of size 14.

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  1.14it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:01<00:00,  2.19it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:01<00:00,  1.92it/s]
wandb: Currently logged in as: seonglae (texonom). Use `wandb login --relogin` to force relogin
wandb: Using wandb-core as the SDK backend.  Please refer to https://wandb.me/wandb-core for more information.
wandb: Tracking run with wandb version 0.19.4
wandb: Run data is saved locally in /cs/student/projects2/aisd/2024/seongcho/control-ai/wandb/run-20250722_014346-6ucmt25q
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run gemma2b_xstest_0_ppo_1e-05_0722_014345_10.0
wandb: ⭐️ View project at https://wandb.ai/texonom/control_rl
wandb: 🚀 View run at https://wandb.ai/texonom/control_rl/runs/6ucmt25q

Training Steps:   0%|          | 0/501 [00:00<?, ?it/s]Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.

Training Steps:   0%|          | 1/501 [00:04<34:27,  4.13s/it]
Training Steps:   0%|          | 2/501 [00:05<21:43,  2.61s/it]
Training Steps:   1%|          | 3/501 [00:07<17:25,  2.10s/it]
Training Steps:   1%|          | 4/501 [00:08<15:22,  1.86s/it]
Training Steps:   1%|          | 5/501 [00:10<14:15,  1.73s/it]
Training Steps:   1%|          | 6/501 [00:11<13:34,  1.65s/it]
Training Steps:   1%|▏         | 7/501 [00:13<13:09,  1.60s/it]
Training Steps:   2%|▏         | 8/501 [00:14<12:50,  1.56s/it]
Training Steps:   2%|▏         | 9/501 [00:16<12:39,  1.54s/it]
Training Steps:   2%|▏         | 10/501 [00:17<12:26,  1.52s/it]
Training Steps:   2%|▏         | 11/501 [00:19<12:18,  1.51s/it]
Training Steps:   2%|▏         | 12/501 [00:20<12:14,  1.50s/it]
Training Steps:   3%|▎         | 13/501 [00:22<12:10,  1.50s/it]
Training Steps:   3%|▎         | 14/501 [00:23<12:06,  1.49s/it]
Training Steps:   3%|▎         | 15/501 [00:25<12:05,  1.49s/it]
Training Steps:   3%|▎         | 16/501 [00:26<12:58,  1.61s/it]
Training Steps:   3%|▎         | 17/501 [00:28<12:38,  1.57s/it]
Training Steps:   4%|▎         | 18/501 [00:29<12:25,  1.54s/it]
Training Steps:   4%|▍         | 19/501 [00:31<12:15,  1.53s/it]
Training Steps:   4%|▍         | 20/501 [00:32<12:06,  1.51s/it]
Training Steps:   4%|▍         | 21/501 [00:34<12:00,  1.50s/it]
Training Steps:   4%|▍         | 22/501 [00:35<11:56,  1.50s/it]
Training Steps:   5%|▍         | 23/501 [00:37<11:52,  1.49s/it]
Training Steps:   5%|▍         | 24/501 [00:38<11:49,  1.49s/it]
Training Steps:   5%|▍         | 25/501 [00:40<11:49,  1.49s/it]
Training Steps:   5%|▌         | 26/501 [00:41<11:44,  1.48s/it]
Training Steps:   5%|▌         | 27/501 [00:43<11:42,  1.48s/it]
Training Steps:   6%|▌         | 28/501 [00:44<11:41,  1.48s/it]
Training Steps:   6%|▌         | 29/501 [00:46<11:41,  1.49s/it]
Training Steps:   6%|▌         | 30/501 [00:47<11:37,  1.48s/it]
Training Steps:   6%|▌         | 31/501 [00:49<11:35,  1.48s/it]
Training Steps:   6%|▋         | 32/501 [00:50<11:29,  1.47s/it]
Training Steps:   7%|▋         | 33/501 [00:52<11:29,  1.47s/it]
Training Steps:   7%|▋         | 34/501 [00:53<11:27,  1.47s/it]
Training Steps:   7%|▋         | 35/501 [00:54<11:29,  1.48s/it]
Training Steps:   7%|▋         | 36/501 [00:56<11:28,  1.48s/it]
Training Steps:   7%|▋         | 37/501 [00:57<11:26,  1.48s/it]
Training Steps:   8%|▊         | 38/501 [00:59<11:26,  1.48s/it]
Training Steps:   8%|▊         | 39/501 [01:00<11:26,  1.49s/it]
Training Steps:   8%|▊         | 40/501 [01:02<11:24,  1.48s/it]
Training Steps:   8%|▊         | 41/501 [01:03<11:22,  1.48s/it]
Training Steps:   8%|▊         | 42/501 [01:05<11:24,  1.49s/it]
Training Steps:   9%|▊         | 43/501 [01:06<11:22,  1.49s/it]
Training Steps:   9%|▉         | 44/501 [01:08<11:19,  1.49s/it]
Training Steps:   9%|▉         | 45/501 [01:09<11:18,  1.49s/it]
Training Steps:   9%|▉         | 46/501 [01:11<11:15,  1.49s/it]
Training Steps:   9%|▉         | 47/501 [01:12<11:13,  1.48s/it]
Training Steps:  10%|▉         | 48/501 [01:14<11:09,  1.48s/it]
Training Steps:  10%|▉         | 49/501 [01:15<11:09,  1.48s/it]
Training Steps:  10%|▉         | 50/501 [01:17<11:07,  1.48s/it]
Training Steps:  10%|█         | 51/501 [01:18<11:05,  1.48s/it]
Training Steps:  10%|█         | 52/501 [01:20<11:05,  1.48s/it]
Training Steps:  11%|█         | 53/501 [01:21<11:02,  1.48s/it]
Training Steps:  11%|█         | 54/501 [01:23<11:00,  1.48s/it]
Training Steps:  11%|█         | 55/501 [01:24<11:00,  1.48s/it]
Training Steps:  11%|█         | 56/501 [01:26<10:59,  1.48s/it]
Training Steps:  11%|█▏        | 57/501 [01:27<10:57,  1.48s/it]
Training Steps:  12%|█▏        | 58/501 [01:29<10:52,  1.47s/it]
Training Steps:  12%|█▏        | 59/501 [01:30<10:52,  1.48s/it]
Training Steps:  12%|█▏        | 60/501 [01:32<10:52,  1.48s/it]
Training Steps:  12%|█▏        | 61/501 [01:33<10:49,  1.48s/it]
Training Steps:  12%|█▏        | 62/501 [01:34<10:46,  1.47s/it]
Training Steps:  13%|█▎        | 63/501 [01:36<10:44,  1.47s/it]
Training Steps:  13%|█▎        | 64/501 [01:37<10:37,  1.46s/it]
Training Steps:  13%|█▎        | 65/501 [01:39<10:38,  1.46s/it]
Training Steps:  13%|█▎        | 66/501 [01:40<10:37,  1.47s/it]
Training Steps:  13%|█▎        | 67/501 [01:42<10:37,  1.47s/it]
Training Steps:  14%|█▎        | 68/501 [01:43<10:36,  1.47s/it]
Training Steps:  14%|█▍        | 69/501 [01:45<10:35,  1.47s/it]
Training Steps:  14%|█▍        | 70/501 [01:46<10:35,  1.47s/it]
Training Steps:  14%|█▍        | 71/501 [01:48<10:33,  1.47s/it]
Training Steps:  14%|█▍        | 72/501 [01:49<10:31,  1.47s/it]
Training Steps:  15%|█▍        | 73/501 [01:51<10:30,  1.47s/it]
Training Steps:  15%|█▍        | 74/501 [01:52<10:24,  1.46s/it]
Training Steps:  15%|█▍        | 75/501 [01:54<10:24,  1.47s/it]
Training Steps:  15%|█▌        | 76/501 [01:55<10:23,  1.47s/it]
Training Steps:  15%|█▌        | 77/501 [01:56<10:22,  1.47s/it]
Training Steps:  16%|█▌        | 78/501 [01:58<10:25,  1.48s/it]
Training Steps:  16%|█▌        | 79/501 [01:59<10:23,  1.48s/it]
Training Steps:  16%|█▌        | 80/501 [02:01<10:16,  1.46s/it]
Training Steps:  16%|█▌        | 81/501 [02:02<10:16,  1.47s/it]
Training Steps:  16%|█▋        | 82/501 [02:04<10:16,  1.47s/it]
Training Steps:  17%|█▋        | 83/501 [02:05<10:15,  1.47s/it]
Training Steps:  17%|█▋        | 84/501 [02:07<10:13,  1.47s/it]
Training Steps:  17%|█▋        | 85/501 [02:08<10:11,  1.47s/it]
Training Steps:  17%|█▋        | 86/501 [02:10<10:11,  1.47s/it]
Training Steps:  17%|█▋        | 87/501 [02:11<10:09,  1.47s/it]
Training Steps:  18%|█▊        | 88/501 [02:13<10:07,  1.47s/it]
Training Steps:  18%|█▊        | 89/501 [02:14<10:07,  1.47s/it]
Training Steps:  18%|█▊        | 90/501 [02:16<10:03,  1.47s/it]
Training Steps:  18%|█▊        | 91/501 [02:17<10:03,  1.47s/it]
Training Steps:  18%|█▊        | 92/501 [02:19<10:01,  1.47s/it]
Training Steps:  19%|█▊        | 93/501 [02:20<10:01,  1.47s/it]
Training Steps:  19%|█▉        | 94/501 [02:22<09:59,  1.47s/it]
Training Steps:  19%|█▉        | 95/501 [02:23<09:57,  1.47s/it]
Training Steps:  19%|█▉        | 96/501 [02:24<09:51,  1.46s/it]
Training Steps:  19%|█▉        | 97/501 [02:26<09:51,  1.46s/it]
Training Steps:  20%|█▉        | 98/501 [02:27<09:51,  1.47s/it]
Training Steps:  20%|█▉        | 99/501 [02:29<09:53,  1.48s/it]
Training Steps:  20%|█▉        | 100/501 [02:30<09:54,  1.48s/it]
Training Steps:  20%|██        | 101/501 [02:32<09:52,  1.48s/it]
Training Steps:  20%|██        | 102/501 [02:33<09:51,  1.48s/it]
Training Steps:  21%|██        | 103/501 [02:35<09:50,  1.48s/it]
Training Steps:  21%|██        | 104/501 [02:36<09:48,  1.48s/it]
Training Steps:  21%|██        | 105/501 [02:38<09:46,  1.48s/it]
Training Steps:  21%|██        | 106/501 [02:39<09:43,  1.48s/it]
Training Steps:  21%|██▏       | 107/501 [02:41<09:41,  1.48s/it]
Training Steps:  22%|██▏       | 108/501 [02:42<09:40,  1.48s/it]
Training Steps:  22%|██▏       | 109/501 [02:44<09:39,  1.48s/it]
Training Steps:  22%|██▏       | 110/501 [02:45<09:37,  1.48s/it]
Training Steps:  22%|██▏       | 111/501 [02:47<09:37,  1.48s/it]
Training Steps:  22%|██▏       | 112/501 [02:48<09:30,  1.47s/it]
Training Steps:  23%|██▎       | 113/501 [02:50<09:30,  1.47s/it]
Training Steps:  23%|██▎       | 114/501 [02:51<09:29,  1.47s/it]
Training Steps:  23%|██▎       | 115/501 [02:53<09:28,  1.47s/it]
Training Steps:  23%|██▎       | 116/501 [02:54<09:27,  1.47s/it]
Training Steps:  23%|██▎       | 116/501 [02:55<09:42,  1.51s/it]
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 414, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 773, in train
    val_metrics = self.validation_step(step, layers, train_metrics, avg_train_accuracy)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 302, in validation_step
    val_metrics = self.perform_validation_step()
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 259, in perform_validation_step
    input_ids, _, generated_ids, correct_answers = self.generate_steered(val_batch)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 127, in generate_steered
    generated_ids = self.llm.generate(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/utils.py", line 2223, in generate
    result = self._sample(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/utils.py", line 3214, in _sample
    outputs = model_forward(**model_inputs, return_dict=True)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
    return func(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/models/gemma2/modeling_gemma2.py", line 887, in forward
    outputs = self.model(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/models/gemma2/modeling_gemma2.py", line 667, in forward
    layer_outputs = decoder_layer(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/models/gemma2/modeling_gemma2.py", line 321, in forward
    hidden_states, self_attn_weights = self.self_attn(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/models/gemma2/modeling_gemma2.py", line 231, in forward
    key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/cache_utils.py", line 1748, in update
    return update_fn(
KeyboardInterrupt
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 414, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 773, in train
    val_metrics = self.validation_step(step, layers, train_metrics, avg_train_accuracy)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 302, in validation_step
    val_metrics = self.perform_validation_step()
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 259, in perform_validation_step
    input_ids, _, generated_ids, correct_answers = self.generate_steered(val_batch)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 127, in generate_steered
    generated_ids = self.llm.generate(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/utils.py", line 2223, in generate
    result = self._sample(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/utils.py", line 3214, in _sample
    outputs = model_forward(**model_inputs, return_dict=True)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
    return func(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/models/gemma2/modeling_gemma2.py", line 887, in forward
    outputs = self.model(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/models/gemma2/modeling_gemma2.py", line 667, in forward
    layer_outputs = decoder_layer(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/models/gemma2/modeling_gemma2.py", line 321, in forward
    hidden_states, self_attn_weights = self.self_attn(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/models/gemma2/modeling_gemma2.py", line 231, in forward
    key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/cache_utils.py", line 1748, in update
    return update_fn(
KeyboardInterrupt
Exception ignored in atexit callback: <function _start_and_connect_service.<locals>.teardown_atexit at 0x7f9837c7b640>
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/wandb/sdk/lib/service_connection.py", line 93, in teardown_atexit
    conn.teardown(hooks.exit_code)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/wandb/sdk/lib/service_connection.py", line 210, in teardown
    self._client.send(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 212, in send
    self.send_server_request(server_req)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 154, in send_server_request
    self._send_message(msg)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 151, in _send_message
    self._sendall_with_error_handle(header + data)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
    sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:53<00:53, 53.42s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [01:00<00:00, 25.94s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [01:00<00:00, 30.06s/it]
wandb: Currently logged in as: seonglae (texonom). Use `wandb login --relogin` to force relogin
wandb: Using wandb-core as the SDK backend.  Please refer to https://wandb.me/wandb-core for more information.
wandb: - Waiting for wandb.init()...
wandb: \ Waiting for wandb.init()...
wandb: Tracking run with wandb version 0.19.4
wandb: Run data is saved locally in /cs/student/projects2/aisd/2024/seongcho/control-ai/wandb/run-20250722_020950-vk8cx4hj
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run gemma2b_harmbench_0_ppo_1e-05_0722_020950_10.0
wandb: ⭐️ View project at https://wandb.ai/texonom/control_rl
wandb: 🚀 View run at https://wandb.ai/texonom/control_rl/runs/vk8cx4hj

Training Steps:   0%|          | 0/14 [00:00<?, ?it/s]Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.

Training Steps:   7%|▋         | 1/14 [00:54<11:54, 54.99s/it]
Training Steps:  14%|█▍        | 2/14 [01:06<05:51, 29.25s/it]
Training Steps:  21%|██▏       | 3/14 [01:14<03:34, 19.51s/it]
Training Steps:  29%|██▊       | 4/14 [01:23<02:34, 15.47s/it]
Training Steps:  29%|██▊       | 4/14 [01:27<03:39, 21.97s/it]
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 414, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 765, in train
    train_metrics = self.perform_training_step(batch)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 237, in perform_training_step
    input_ids, attention_mask, generated_ids, correct_answers = self.generate_steered(batch)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 127, in generate_steered
    generated_ids = self.llm.generate(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/utils.py", line 2223, in generate
    result = self._sample(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/utils.py", line 3200, in _sample
    while self._has_unfinished_sequences(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/utils.py", line 2401, in _has_unfinished_sequences
    elif this_peer_finished:
KeyboardInterrupt
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 414, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 765, in train
    train_metrics = self.perform_training_step(batch)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 237, in perform_training_step
    input_ids, attention_mask, generated_ids, correct_answers = self.generate_steered(batch)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 127, in generate_steered
    generated_ids = self.llm.generate(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/utils.py", line 2223, in generate
    result = self._sample(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/utils.py", line 3200, in _sample
    while self._has_unfinished_sequences(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/transformers/generation/utils.py", line 2401, in _has_unfinished_sequences
    elif this_peer_finished:
KeyboardInterrupt
Exception ignored in atexit callback: <function _start_and_connect_service.<locals>.teardown_atexit at 0x7f6121af91b0>
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/wandb/sdk/lib/service_connection.py", line 93, in teardown_atexit
    conn.teardown(hooks.exit_code)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/wandb/sdk/lib/service_connection.py", line 210, in teardown
    self._client.send(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 212, in send
    self.send_server_request(server_req)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 154, in send_server_request
    self._send_message(msg)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 151, in _send_message
    self._sendall_with_error_handle(header + data)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/wandb/sdk/lib/sock_client.py", line 130, in _sendall_with_error_handle
    sent = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:21<00:21, 21.81s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:22<00:00,  9.11s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:22<00:00, 11.02s/it]
wandb: Currently logged in as: seonglae (texonom). Use `wandb login --relogin` to force relogin
wandb: Using wandb-core as the SDK backend.  Please refer to https://wandb.me/wandb-core for more information.
wandb: Tracking run with wandb version 0.19.4
wandb: Run data is saved locally in /cs/student/projects2/aisd/2024/seongcho/control-ai/wandb/run-20250722_021226-fr89nb1r
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run gemma2b_harmbench_0_ppo_1e-05_0722_021226_10.0
wandb: ⭐️ View project at https://wandb.ai/texonom/control_rl
wandb: 🚀 View run at https://wandb.ai/texonom/control_rl/runs/fr89nb1r

Training Steps:   0%|          | 0/14 [00:00<?, ?it/s]Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.

Training Steps:   7%|▋         | 1/14 [00:27<05:59, 27.63s/it]
Training Steps:  14%|█▍        | 2/14 [00:50<04:57, 24.83s/it]
Training Steps:  21%|██▏       | 3/14 [01:08<03:58, 21.66s/it]
Training Steps:  29%|██▊       | 4/14 [01:28<03:29, 20.96s/it]
Training Steps:  36%|███▌      | 5/14 [01:43<02:49, 18.81s/it]
Training Steps:  43%|████▎     | 6/14 [02:06<02:41, 20.22s/it]
Training Steps:  50%|█████     | 7/14 [02:25<02:18, 19.86s/it]
Training Steps:  57%|█████▋    | 8/14 [02:40<01:49, 18.30s/it]
Training Steps:  64%|██████▍   | 9/14 [03:01<01:36, 19.21s/it]
Training Steps:  71%|███████▏  | 10/14 [03:20<01:16, 19.01s/it]
Training Steps:  79%|███████▊  | 11/14 [03:40<00:58, 19.36s/it]
Training Steps:  86%|████████▌ | 12/14 [03:57<00:37, 18.75s/it]
Training Steps:  93%|█████████▎| 13/14 [04:17<00:19, 19.24s/it]
Training Steps: 100%|██████████| 14/14 [04:35<00:00, 18.57s/it]
Training Steps: 100%|██████████| 14/14 [04:35<00:00, 19.64s/it]
/cs/student/projects2/aisd/2024/seongcho/control-ai/eval.py:480: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  ckpt = TrainResult.model_validate(torch.load(checkpoint))

Evaluating:   0%|          | 0/400 [00:00<?, ?it/s]
Evaluating:   2%|▏         | 8/400 [00:11<09:13,  1.41s/it]
Evaluating:   4%|▍         | 16/400 [00:17<06:41,  1.05s/it]
Evaluating:   6%|▌         | 24/400 [00:24<05:55,  1.06it/s]
Evaluating:   8%|▊         | 32/400 [00:33<06:21,  1.04s/it]
Evaluating:  10%|█         | 40/400 [00:43<06:34,  1.10s/it]
Evaluating:  12%|█▏        | 48/400 [00:51<06:23,  1.09s/it]
Evaluating:  14%|█▍        | 56/400 [01:01<06:24,  1.12s/it]
Evaluating:  16%|█▌        | 64/400 [01:11<06:31,  1.17s/it]
Evaluating:  18%|█▊        | 72/400 [01:20<06:21,  1.16s/it]
Evaluating:  20%|██        | 80/400 [01:33<07:01,  1.32s/it]
Evaluating:  22%|██▏       | 88/400 [01:42<06:30,  1.25s/it]
Evaluating:  24%|██▍       | 96/400 [01:53<06:33,  1.29s/it]
Evaluating:  26%|██▌       | 104/400 [02:04<06:21,  1.29s/it]
Evaluating:  28%|██▊       | 112/400 [02:13<05:59,  1.25s/it]
Evaluating:  30%|███       | 120/400 [02:21<05:29,  1.18s/it]
Evaluating:  32%|███▏      | 128/400 [02:27<04:48,  1.06s/it]
Evaluating:  34%|███▍      | 136/400 [02:37<04:53,  1.11s/it]
Evaluating:  36%|███▌      | 144/400 [02:45<04:36,  1.08s/it]
Evaluating:  38%|███▊      | 152/400 [02:55<04:43,  1.14s/it]
Evaluating:  40%|████      | 160/400 [03:05<04:37,  1.15s/it]
Evaluating:  42%|████▏     | 168/400 [03:13<04:16,  1.11s/it]
Evaluating:  44%|████▍     | 176/400 [03:21<04:05,  1.10s/it]
Evaluating:  46%|████▌     | 184/400 [03:28<03:37,  1.01s/it]
Evaluating:  48%|████▊     | 192/400 [03:37<03:39,  1.05s/it]
Evaluating:  50%|█████     | 200/400 [03:46<03:31,  1.06s/it]
Evaluating:  52%|█████▏    | 208/400 [03:58<03:47,  1.19s/it]
Evaluating:  54%|█████▍    | 216/400 [04:06<03:27,  1.13s/it]
Evaluating:  56%|█████▌    | 224/400 [04:15<03:22,  1.15s/it]
Evaluating:  58%|█████▊    | 232/400 [04:23<03:02,  1.08s/it]
Evaluating:  60%|██████    | 240/400 [04:32<02:59,  1.12s/it]
Evaluating:  62%|██████▏   | 248/400 [04:42<02:52,  1.13s/it]
Evaluating:  64%|██████▍   | 256/400 [04:50<02:41,  1.12s/it]
Evaluating:  66%|██████▌   | 264/400 [05:00<02:38,  1.16s/it]
Evaluating:  68%|██████▊   | 272/400 [05:07<02:15,  1.06s/it]
Evaluating:  70%|███████   | 280/400 [05:18<02:16,  1.14s/it]
Evaluating:  72%|███████▏  | 288/400 [05:27<02:09,  1.16s/it]
Evaluating:  74%|███████▍  | 296/400 [05:40<02:16,  1.31s/it]
Evaluating:  76%|███████▌  | 304/400 [05:50<02:02,  1.28s/it]
Evaluating:  78%|███████▊  | 312/400 [06:00<01:49,  1.25s/it]
Evaluating:  80%|████████  | 320/400 [06:11<01:45,  1.31s/it]
Evaluating:  82%|████████▏ | 328/400 [06:22<01:34,  1.32s/it]
Evaluating:  84%|████████▍ | 336/400 [06:32<01:22,  1.29s/it]
Evaluating:  86%|████████▌ | 344/400 [06:43<01:15,  1.34s/it]
Evaluating:  88%|████████▊ | 352/400 [06:51<00:58,  1.23s/it]
Evaluating:  90%|█████████ | 360/400 [07:00<00:48,  1.21s/it]
Evaluating:  92%|█████████▏| 368/400 [07:10<00:38,  1.20s/it]
Evaluating:  94%|█████████▍| 376/400 [07:19<00:28,  1.17s/it]
Evaluating:  96%|█████████▌| 384/400 [07:30<00:19,  1.24s/it]
Evaluating:  98%|█████████▊| 392/400 [07:39<00:09,  1.20s/it]
Evaluating: 100%|██████████| 400/400 [07:49<00:00,  1.22s/it]
Evaluating: 100%|██████████| 400/400 [07:49<00:00,  1.17s/it]
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 414, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 784, in train
    final_stats = self.collect_stats(cfg, prev_best_ckpt, ckpt_dir)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 633, in collect_stats
    steered_stats = self.perform_analysis(cfg, steered_stats, prev_best_ckpt)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 435, in perform_analysis
    self.answer_analysis(steered_file, baseline_csv, output_dir, task=cfg.task)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 360, in answer_analysis
    accuracy(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/analyze.py", line 264, in accuracy
    nonsteered_rewards = [calculate_reward(row["predicted"], row["ground_truth"], task)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/analyze.py", line 264, in <listcomp>
    nonsteered_rewards = [calculate_reward(row["predicted"], row["ground_truth"], task)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/config.py", line 96, in calculate_reward
    return reward_func(pred, gold, tokenizer)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/config.py", line 68, in got_rejected
    text = pred.lower()
AttributeError: 'float' object has no attribute 'lower'
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 414, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 784, in train
    final_stats = self.collect_stats(cfg, prev_best_ckpt, ckpt_dir)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 633, in collect_stats
    steered_stats = self.perform_analysis(cfg, steered_stats, prev_best_ckpt)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 435, in perform_analysis
    self.answer_analysis(steered_file, baseline_csv, output_dir, task=cfg.task)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 360, in answer_analysis
    accuracy(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/analyze.py", line 264, in accuracy
    nonsteered_rewards = [calculate_reward(row["predicted"], row["ground_truth"], task)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/analyze.py", line 264, in <listcomp>
    nonsteered_rewards = [calculate_reward(row["predicted"], row["ground_truth"], task)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/config.py", line 96, in calculate_reward
    return reward_func(pred, gold, tokenizer)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/config.py", line 68, in got_rejected
    text = pred.lower()
AttributeError: 'float' object has no attribute 'lower'

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  1.35it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  2.44it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  2.18it/s]
wandb: Currently logged in as: seonglae (texonom). Use `wandb login --relogin` to force relogin
wandb: Using wandb-core as the SDK backend.  Please refer to https://wandb.me/wandb-core for more information.
wandb: Tracking run with wandb version 0.19.4
wandb: Run data is saved locally in /cs/student/projects2/aisd/2024/seongcho/control-ai/wandb/run-20250723_015910-qguaucnz
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run gemma2b_harmbench_0_ppo_1e-05_0723_015909_10.0
wandb: ⭐️ View project at https://wandb.ai/texonom/control_rl
wandb: 🚀 View run at https://wandb.ai/texonom/control_rl/runs/qguaucnz

Training Steps:   0%|          | 0/14 [00:00<?, ?it/s]Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.

Training Steps:   7%|▋         | 1/14 [00:04<01:02,  4.82s/it]
Training Steps:  14%|█▍        | 2/14 [00:07<00:39,  3.32s/it]
Training Steps:  21%|██▏       | 3/14 [00:08<00:28,  2.62s/it]
Training Steps:  29%|██▊       | 4/14 [00:10<00:23,  2.37s/it]
Training Steps:  36%|███▌      | 5/14 [00:12<00:18,  2.10s/it]
Training Steps:  43%|████▎     | 6/14 [00:14<00:17,  2.15s/it]
Training Steps:  50%|█████     | 7/14 [00:16<00:14,  2.07s/it]
Training Steps:  57%|█████▋    | 8/14 [00:18<00:11,  1.92s/it]
Training Steps:  64%|██████▍   | 9/14 [00:20<00:09,  1.97s/it]
Training Steps:  71%|███████▏  | 10/14 [00:22<00:07,  1.94s/it]
Training Steps:  79%|███████▊  | 11/14 [00:24<00:05,  1.97s/it]
Training Steps:  86%|████████▌ | 12/14 [00:25<00:03,  1.90s/it]
Training Steps:  93%|█████████▎| 13/14 [00:28<00:01,  1.94s/it]
Training Steps: 100%|██████████| 14/14 [00:29<00:00,  1.87s/it]
Training Steps: 100%|██████████| 14/14 [00:29<00:00,  2.12s/it]
/cs/student/projects2/aisd/2024/seongcho/control-ai/eval.py:480: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  ckpt = TrainResult.model_validate(torch.load(checkpoint))

Evaluating:   0%|          | 0/400 [00:00<?, ?it/s]
Evaluating:   2%|▏         | 8/400 [00:01<00:52,  7.52it/s]
Evaluating:   4%|▍         | 16/400 [00:01<00:40,  9.56it/s]
Evaluating:   6%|▌         | 24/400 [00:02<00:36, 10.33it/s]
Evaluating:   8%|▊         | 32/400 [00:03<00:38,  9.61it/s]
Evaluating:  10%|█         | 40/400 [00:04<00:39,  9.19it/s]
Evaluating:  12%|█▏        | 48/400 [00:05<00:38,  9.25it/s]
Evaluating:  14%|█▍        | 56/400 [00:06<00:37,  9.06it/s]
Evaluating:  16%|█▌        | 64/400 [00:07<00:38,  8.82it/s]
Evaluating:  18%|█▊        | 72/400 [00:07<00:37,  8.81it/s]
Evaluating:  20%|██        | 80/400 [00:09<00:40,  7.89it/s]
Evaluating:  22%|██▏       | 88/400 [00:10<00:38,  8.15it/s]
Evaluating:  24%|██▍       | 96/400 [00:11<00:38,  7.93it/s]
Evaluating:  26%|██▌       | 104/400 [00:12<00:37,  7.94it/s]
Evaluating:  28%|██▊       | 112/400 [00:13<00:35,  8.12it/s]
Evaluating:  30%|███       | 120/400 [00:13<00:32,  8.51it/s]
Evaluating:  32%|███▏      | 128/400 [00:14<00:29,  9.18it/s]
Evaluating:  34%|███▍      | 136/400 [00:15<00:29,  8.83it/s]
Evaluating:  36%|███▌      | 144/400 [00:16<00:28,  9.01it/s]
Evaluating:  38%|███▊      | 152/400 [00:17<00:28,  8.65it/s]
Evaluating:  40%|████      | 160/400 [00:18<00:27,  8.60it/s]
Evaluating:  42%|████▏     | 168/400 [00:19<00:26,  8.87it/s]
Evaluating:  44%|████▍     | 176/400 [00:20<00:25,  8.94it/s]
Evaluating:  46%|████▌     | 184/400 [00:20<00:22,  9.53it/s]
Evaluating:  48%|████▊     | 192/400 [00:21<00:22,  9.21it/s]
Evaluating:  50%|█████     | 200/400 [00:22<00:21,  9.20it/s]
Evaluating:  52%|█████▏    | 208/400 [00:23<00:22,  8.41it/s]
Evaluating:  54%|█████▍    | 216/400 [00:24<00:21,  8.73it/s]
Evaluating:  56%|█████▌    | 224/400 [00:25<00:20,  8.58it/s]
Evaluating:  58%|█████▊    | 232/400 [00:26<00:18,  8.96it/s]
Evaluating:  60%|██████    | 240/400 [00:27<00:18,  8.72it/s]
Evaluating:  62%|██████▏   | 248/400 [00:28<00:17,  8.66it/s]
Evaluating:  64%|██████▍   | 256/400 [00:29<00:16,  8.72it/s]
Evaluating:  66%|██████▌   | 264/400 [00:30<00:16,  8.48it/s]
Evaluating:  68%|██████▊   | 272/400 [00:30<00:14,  9.09it/s]
Evaluating:  70%|███████   | 280/400 [00:32<00:13,  8.66it/s]
Evaluating:  72%|███████▏  | 288/400 [00:32<00:13,  8.54it/s]
Evaluating:  74%|███████▍  | 296/400 [00:34<00:13,  7.69it/s]
Evaluating:  76%|███████▌  | 304/400 [00:35<00:12,  7.84it/s]
Evaluating:  78%|███████▊  | 312/400 [00:36<00:10,  8.03it/s]
Evaluating:  80%|████████  | 320/400 [00:37<00:10,  7.77it/s]
Evaluating:  82%|████████▏ | 328/400 [00:38<00:09,  7.78it/s]
Evaluating:  84%|████████▍ | 336/400 [00:39<00:08,  7.85it/s]
Evaluating:  86%|████████▌ | 344/400 [00:40<00:07,  7.66it/s]
Evaluating:  88%|████████▊ | 352/400 [00:41<00:05,  8.19it/s]
Evaluating:  90%|█████████ | 360/400 [00:42<00:04,  8.28it/s]
Evaluating:  92%|█████████▏| 368/400 [00:43<00:03,  8.35it/s]
Evaluating:  94%|█████████▍| 376/400 [00:44<00:02,  8.49it/s]
Evaluating:  96%|█████████▌| 384/400 [00:45<00:01,  8.15it/s]
Evaluating:  98%|█████████▊| 392/400 [00:45<00:00,  8.31it/s]
Evaluating: 100%|██████████| 400/400 [00:47<00:00,  8.20it/s]
Evaluating: 100%|██████████| 400/400 [00:47<00:00,  8.51it/s]
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 414, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 784, in train
    final_stats = self.collect_stats(cfg, prev_best_ckpt, ckpt_dir)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 633, in collect_stats
    steered_stats = self.perform_analysis(cfg, steered_stats, prev_best_ckpt)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 435, in perform_analysis
    self.answer_analysis(steered_file, baseline_csv, output_dir, task=cfg.task)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 360, in answer_analysis
    accuracy(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/analyze.py", line 264, in accuracy
    nonsteered_rewards = [calculate_reward(row["predicted"], row["ground_truth"], task)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/analyze.py", line 264, in <listcomp>
    nonsteered_rewards = [calculate_reward(row["predicted"], row["ground_truth"], task)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/config.py", line 96, in calculate_reward
    return reward_func(pred, gold, tokenizer)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/config.py", line 68, in got_rejected
    text = pred.lower()
AttributeError: 'float' object has no attribute 'lower'
Traceback (most recent call last):
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 414, in <module>
    fire.Fire(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
    run_experiments(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
    stats = train_controller.train(**train_params.model_dump())
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 784, in train
    final_stats = self.collect_stats(cfg, prev_best_ckpt, ckpt_dir)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 633, in collect_stats
    steered_stats = self.perform_analysis(cfg, steered_stats, prev_best_ckpt)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 435, in perform_analysis
    self.answer_analysis(steered_file, baseline_csv, output_dir, task=cfg.task)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 360, in answer_analysis
    accuracy(
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/analyze.py", line 264, in accuracy
    nonsteered_rewards = [calculate_reward(row["predicted"], row["ground_truth"], task)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/analyze.py", line 264, in <listcomp>
    nonsteered_rewards = [calculate_reward(row["predicted"], row["ground_truth"], task)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/config.py", line 96, in calculate_reward
    return reward_func(pred, gold, tokenizer)
  File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/config.py", line 68, in got_rejected
    text = pred.lower()
AttributeError: 'float' object has no attribute 'lower'

New errors

ting:  80%|████████  | 320/400 [00:52<00:14,  5.61it/s]
Evaluating:  82%|████████▏ | 328/400 [00:54<00:12,  5.61it/s]
Evaluating:  84%|████████▍ | 336/400 [00:55<00:11,  5.68it/s]
Evaluating:  86%|████████▌ | 344/400 [00:57<00:10,  5.56it/s]
Evaluating:  88%|████████▊ | 352/400 [00:58<00:08,  5.88it/s]
Evaluating:  90%|█████████ | 360/400 [00:59<00:06,  5.92it/s]
Evaluating:  92%|█████████▏| 368/400 [01:01<00:05,  5.96it/s]
Evaluating:  94%|█████████▍| 376/400 [01:02<00:03,  6.08it/s]
Evaluating:  96%|█████████▌| 384/400 [01:03<00:02,  5.88it/s]
Evaluating:  98%|█████████▊| 392/400 [01:05<00:01,  5.98it/s]
Evaluating: 100%|██████████| 400/400 [01:06<00:00,  5.93it/s]
Evaluating: 100%|██████████| 400/400 [01:06<00:00,  6.02it/s]
0|layers-gemma-harmbench  | Final harmbench Accuracy with Steering: 32.50%
0|layers-gemma-harmbench  | Results saved to ./checkpoints/gemma2b_harmbench_0_ppo_1e-05_0724_015857_-200.0/harmbench_0_steered.json
0|layers-gemma-harmbench  | Stats saved to ./checkpoints/gemma2b_harmbench_0_ppo_1e-05_0724_015857_-200.0/harmbench_eval.json
0|layers-gemma-harmbench  | Starting analysis...
0|layers-gemma-harmbench  | Getting baselines took: 0.00s
0|layers-gemma-harmbench  | Overall Accuracy:
0|layers-gemma-harmbench  | Steered Model: 32.50% (130.0/400)
0|layers-gemma-harmbench  | Baseline Model: 34.25% (137.0/400)
0|layers-gemma-harmbench  | /cs/student/projects2/aisd/2024/seongcho/control-ai/analyze.py:300: UserWarning: Tight layout not applied. The bottom and top margins cannot be made large enough to accommodate all Axes decorations.
0|layers-gemma-harmbench  |   plt.tight_layout()
0|layers-gemma-harmbench  | Baseline answer analysis took: 2.15s
0|layers-gemma-harmbench  | Analyzing layer 0...
0|layers-gemma-harmbench  | Critic Analysis Results:
0|layers-gemma-harmbench  | Total samples: 400
0|layers-gemma-harmbench  | Correct (reward > 0): 130
0|layers-gemma-harmbench  | Incorrect (reward = 0): 270
0|layers-gemma-harmbench  | Corrected (steered reward > baseline reward): 16
0|layers-gemma-harmbench  | Misguided (steered reward < baseline reward): 23
0|layers-gemma-harmbench  | /cs/student/projects2/aisd/2024/seongcho/control-ai/analyze.py:111: FutureWarning: 
0|layers-gemma-harmbench  | Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.
0|layers-gemma-harmbench  |   return original_barplot(*args, **kwargs)
0|layers-gemma-harmbench  | Feature analysis saved to ./checkpoints/gemma2b_harmbench_0_ppo_1e-05_0724_015857_-200.0/feature_analysis_0.json
0|layers-gemma-harmbench  |   Layer 0 naive analysis took: 31.89s
0|layers-gemma-harmbench  | Layer 0 total analysis took: 31.89s
0|layers-gemma-harmbench  | Building result dictionaries took: 0.00s
0|layers-gemma-harmbench  | Total analysis completed in: 34.04s
0|layers-gemma-harmbench  | Every outputs are saved to the folder ./checkpoints/gemma2b_harmbench_0_ppo_1e-05_0724_015857_-200.0
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:01<00:01,  1.87s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:02<00:00,  1.06s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:02<00:00,  1.19s/it]
Training Steps:   0%|          | 0/14 [00:00<?, ?it/s]
0|layers-gemma-harmbench  | Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.
Training Steps:   0%|          | 0/14 [00:02<?, ?it/s]
0|layers-gemma-harmbench  | Traceback (most recent call last):
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 415, in <module>
0|layers-gemma-harmbench  |     fire.Fire(
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
0|layers-gemma-harmbench  |     component_trace = _Fire(component, args, parsed_flag_args, context, name)
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
0|layers-gemma-harmbench  |     component, remaining_args = _CallAndUpdateTrace(
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
0|layers-gemma-harmbench  |     component = fn(*varargs, **kwargs)
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
0|layers-gemma-harmbench  |     run_experiments(
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
0|layers-gemma-harmbench  |     stats = train_controller.train(**train_params.model_dump())
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 768, in train
0|layers-gemma-harmbench  |     train_metrics = self.perform_training_step(batch)
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 244, in perform_training_step
0|layers-gemma-harmbench  |     layer_metrics = self.backward(batch_rewards, eos_position)
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 214, in backward
0|layers-gemma-harmbench  |     train_result = self.ppo_trainers[layer].train_step(
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/ppo.py", line 223, in train_step
0|layers-gemma-harmbench  |     sample_ratio: Tensor = torch.exp(sample_new_log_probs - sample_log_probs)
0|layers-gemma-harmbench  | RuntimeError: The size of tensor a (32) must match the size of tensor b (16384) at non-singleton dimension 1
0|layers-gemma-harmbench  | Traceback (most recent call last):
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 415, in <module>
0|layers-gemma-harmbench  |     fire.Fire(
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
0|layers-gemma-harmbench  |     component_trace = _Fire(component, args, parsed_flag_args, context, name)
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
0|layers-gemma-harmbench  |     component, remaining_args = _CallAndUpdateTrace(
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/miniconda3/envs/sae/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
0|layers-gemma-harmbench  |     component = fn(*varargs, **kwargs)
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 277, in layers
0|layers-gemma-harmbench  |     run_experiments(
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/control-ai/experiment.py", line 108, in run_experiments
0|layers-gemma-harmbench  |     stats = train_controller.train(**train_params.model_dump())
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 768, in train
0|layers-gemma-harmbench  |     train_metrics = self.perform_training_step(batch)
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 244, in perform_training_step
0|layers-gemma-harmbench  |     layer_metrics = self.backward(batch_rewards, eos_position)
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/control-ai/train.py", line 214, in backward
0|layers-gemma-harmbench  |     train_result = self.ppo_trainers[layer].train_step(
0|layers-gemma-harmbench  |   File "/cs/student/projects2/aisd/2024/seongcho/control-ai/control_rl/ppo.py", line 223, in train_step
0|layers-gemma-harmbench  |     sample_ratio: Tensor = torch.exp(sample_new_log_probs - sample_log_probs)
0|layers-gemma-harmbench  | RuntimeError: The size of tensor a (32) must match the size of tensor b (16384) at non-singleton dimension 1
0|layers-gemma-harmbench  | wandb: 
0|layers-gemma-harmbench  | wandb: 🚀 View run gemma2b_harmbench_0_ppo_1e-05_0724_015857_-200.0 at: https://wandb.ai/texonom/control_rl/runs/jcencapl
0|layers-gemma-harmbench  | wandb: Find logs at: wandb/run-20250724_01