YSU RL Final

Offline RL, model-based RL, reward learning, skill discovery, hierarchical RL, and HW3-5

과제 pdf랑 코드 꼭 보기

Offline Learning

• Off-policy,
  On-policy

• Offline dataset is given, Train policy on offline data; Offline RL reuses previously collected data

• Data collected from some unknown policy , called behavior policy or mixture of policies

• Offline RL finds the optimal path by combining segments, while BC only mimics the actions without optimizing.

• Offline RL need to handle unseen actions in a safe way, while doing better than data

• Can we simply use
  Off-policy algorithms?
  Q overestimation matters more because of static data. (small dataset size → larger over-estimation)

• Where to use offline RL? Pre-training for robots… We do offline RL to improve the policy beyond the behavior policy with policy improvement vs. distribution shift.

TD3+BC (Explicitly constrain to the data)

CQL (Implicitly constrain to data by penalizing Q-values)

Filtered Behavioral Cloning

AWR (Advantage-Weighted Regression)

Implicit Q Learning

• Prediction is larger than target → small loss → prediction stays large

• prediction is smaller than target → larger loss → prediction becomes larger

• two hyperparameters and compared to
  CQL has 1 which means hard to tune.

• Once converged, it extracts using
  AWR (works very well in practice)

• Decoupling actor & critic training -> computationally fast

Model based RL

Model-based optimization (gradient based & sampling based)

• CEM Iteration sampling based

• Augmenting data with model-simulated roll-outs. Sampling based optimization doesn’t scale to high-dimensions in terms of both horizon and action space.

Model-based Planning (gradient based & sampling based)

• Open-loop system
  Closed-loop system. More planning → better performance

• Brute-force tree search planning needs exponential complexity.

• Alpha Go
• approximates tree search with sampling (Estimate a value (expected return) with several simulations of entire games starting from current state)
• Tree Policy: Selection → Expansion
• Default Policy: Run a simulation using quick policy
• Update action values Q and visit counts of all traversed edges



• Efficient MCTS only search plausible states with reduced breadth (actions) and depth (result)
• New tree policy to reduce breadth with policy network



The Q encourages while the N encourages
Alpha zero do not MCTS rollout with only simulations. No human data, no domain knowledge only self-play. In other words, it does not go to the end for single policy consistency.

Model learning

Model-Predictive Control

TD-MPC

Dreamer

• Dyna (Online Q-learning with a model) style model-based RL

Reward model

Goal classifier

• Behavior Cloning: action | state

• GAIL: state action pair

• GAIfO: state only without expert action demo

• Maximum Entropy Objective

Controllability-aware skill discovery

Skill chaining

• T-STAR

• Transition Policy

Skill dynamics model (learn skill prior - task policy)

• SPiRL

• SkiMo

Homework

Homework 3 DQN

        if np.random.random() < epsilon:
          action = torch.tensor([np.random.randint(self.num_actions)])
        else:
          with torch.no_grad():
            q_values = self.critic(observation)
            action = q_values.argmax(dim=1)

        # Compute target values
        with torch.no_grad():
            # HINT: first, compute q(s',a') values for all actions with the target network
            # (we will later find the maximum values among all possible actions)
            next_qa_values = self.target_critic(next_obs)
            if self.use_double_q:
                # HINT: find the best action by evaluting q(s',a') with the current critic
                next_action = self.critic(next_obs).argmax(dim=1, keepdim=True)
            else:
                # HINT: find the next action to choose (hint: argmax) among `next_qa_values`
                next_action = next_qa_values.argmax(dim=1, keepdim=True)
            # HINT: find the next q values among `next_qa_values` by specifying with `next_action`
            next_q_values = next_qa_values.gather(1, next_action).squeeze()
            target_values = reward + self.discount * (~done) * next_q_values
        qa_values = self.critic(obs)
        q_values = qa_values.gather(1, action.unsqueeze(1)).squeeze()
        assert q_values.shape == target_values.shape
        # HINT: compute loss with q_values and target_values
        loss = self.critic_loss(q_values, target_values.detach())
        self.critic_optimizer.zero_grad()
        loss.backward()
        self.critic_optimizer.step()
        self.lr_scheduler.step()

1. compute loss

2. optimizer zero grad

3. loss backward

4. step optimizer

5. step scheduler

Homework 4

        if self.target_critic_backup_type == "doubleq":
            indices = torch.arange(num_critic_networks).to(ptu.device)
            double_q_values = torch.where(indices.unsqueeze(1) % 2 == 0, next_qs[1], next_qs[0])
            return double_q_values
        elif self.target_critic_backup_type == "min":
            min_qs, _ = torch.min(next_qs, dim=0, keepdim=True)
            return min_qs.expand_as(next_qs)

    def entropy(self, action_distribution: torch.distributions.Distribution):
        action = action_distribution.rsample()
        entropy = -action_distribution.log_prob(action)

        assert entropy.shape == action.shape[:-1]
        return entropy

next_action_entropy = self.entropy(next_action_distribution)
next_qs = next_qs.clone() + self.temperature * next_action_entropy.unsqueeze(0)

            # TODO(student): Compute the target Q-value
            target_values: torch.Tensor = reward + self.discount * ~done * next_qs
            assert target_values.shape == (
                self.num_critic_networks,
                batch_size
            )

        # TODO(student): Predict Q-values using `self.critic`
        q_values = self.critic(obs, action)
        assert q_values.shape == (self.num_critic_networks, batch_size), q_values.shape
        loss: torch.Tensor = self.critic_loss(q_values, target_values)
        self.critic_optimizer.zero_grad()
        loss.backward()
        self.critic_optimizer.step()

            obs_expanded = obs.unsqueeze(0).expand(self.num_actor_samples, *obs.shape).reshape(-1, *self.observation_shape)
            action_expanded = action.unsqueeze(0).reshape(-1, self.action_dim)
            q_values = self.critic(obs_expanded, action_expanded).view(self.num_critic_networks, self.num_actor_samples, batch_size)
            assert q_values.shape == (
                self.num_critic_networks,
                self.num_actor_samples,
                batch_size,
            ), q_values.shape
            
            # Our best guess of the Q-values is the mean of the ensemble
            q_values = torch.mean(q_values, axis=0)

        # Do REINFORCE (without baseline)
        # TODO(student): Calculate log-probs
        log_probs = action_distribution.log_prob(action)
        assert log_probs.shape == q_values.shape

        # TODO(student): Compute policy gradient using log-probs and Q-values
        loss = -(log_probs * q_values).mean()

    def actor_loss_reparametrize(self, obs: torch.Tensor):
        batch_size = obs.shape[0]

        # Sample from the actor
        action_distribution: torch.distributions.Distribution = self.actor(obs)

        # TODO(student): Sample actions
        # Note: Think about whether to use .rsample() or .sample() here...
        action = action_distribution.rsample()

        # TODO(student): Compute Q-values for the sampled state-action pair
        q_values = self.critic(obs, action).mean(0)

        # TODO(student): Compute the actor loss using Q-values
        loss = -q_values.mean()

Homework 5 CQL

    def dqn_loss(self, ob_no, ac_na, next_ob_no, reward_n, terminal_n):
        qa_t_values = self.q_net(ob_no)
        q_t_values = torch.gather(qa_t_values, 1, ac_na.unsqueeze(1)).squeeze(1)
        qa_tp1_values = self.q_net_target(next_ob_no)

        next_actions = self.q_net(next_ob_no).argmax(dim=1)
        q_tp1 = torch.gather(qa_tp1_values, 1, next_actions.unsqueeze(1)).squeeze(1)

        target = reward_n + self.gamma * q_tp1 * (1 - terminal_n)
        target = target.detach()
        loss = self.loss(q_t_values, target)

        return loss, qa_t_values, q_t_values

        # TODO: CQL Implementation (Equation 1)
        # Hint #1: Obtain dqn_loss, qa_t_values, q_t_values using self.dqn_loss (the first summation in Equation 1)
        # Hint #2: Compute q_t_logsumexp in cql regularizer using torch.logsumexp
        # Hint #3: Compute cql_loss using q_t_logsumexp and q_t_values (the second summation in Equation 1)
        # Hint #4: Finally, compute loss using dqn_loss and cql_loss
        dqn_loss, qa_t_values, q_t_values = self.dqn_loss(ob_no, ac_na, next_ob_no, reward_n, terminal_n)
        q_t_logsumexp = torch.logsumexp(qa_t_values, dim=1)
        cql_loss =  self.cql_alpha * (q_t_logsumexp - q_t_values).mean()
        loss = dqn_loss + cql_loss

• q_t_values 실제로 취한 행동 D

• qa_t_values 모든 가능한 행동 a (logsumexp)

• 실제 한 행동 overestimate 하는데 loss에서 빼주는 이유는 반대이기 때문

시험

• loss 랑 엔트로피 등

• skill learning 사용할 때 문제점 gail

• IQL ex~ 적용위치

• CQL ood query term

• skill dynamics한계와 chaining 2 일때와 3 일때