Unlike traditional reinforcement learning (RL), that focuses on a single goal (maximizing a cumulative reward), constrained RL (CRL) additionally seeks to satisfy requirements described by a lower bound on a set of cumulative rewards. Though this appears to be a minor distinction, CRL turns out to be both more expressive and more complex than unconstrained RL. In fact, despite strong duality results, traditional primal-dual methods often fail to find optimal solutions that satisfy the requirements. To address this issue, a state augmentation procedure (A-CRL) has been proposed based on training a policy that maximizes the Lagrangian. Though A-CRL guarantees near-optimality and near-feasibility, maximizing the Lagrangian is equivalent to solving a parametrized class of unconstrained RL problems. In this work, we propose different methods to tackle this problem based on sampling and worst-case optimization techniques. Using a challenging monitoring problem, we showcase the effectiveness of these methods compared to the traditional approach of giving equal weight to all RL problems induced by the Lagrangian. These techniques may also be of interest in the context of multitask RL problem.