Recurrent neural networks (RNNs) exhibit outstanding performance when handling time-series inputs due to their ability to model latent state dynamics. This makes RNNs particularly suitable for system identification in nonlinear dynamic systems. Although neural networks possess great power and a wide range of applications, their lack of rigorous safety guarantees, such as stability and robustness, renders them uncommon in safety-critical fields like medical devices and autonomous driving. By recasting RNNs as a category of nonlinear dynamical systems and examining them through the perspective of system theory, it becomes possible to analyze their stability. Recent studies have established a sufficient condition concerning network weights that ensure Input-to-State Stability (ISS). However, the prevailing approach for enforcing this condition becomes computationally prohibitive when applied to large-scale networks and datasets due to the necessity of solving a complex, constrained nonlinear optimization problem. Furthermore, the existing techniques struggle to train the constraints to a certain desired value with such substantial scale of data and networks. To enable the training of large ISS-compatible Long Short Term Memory (LSTM) in a computationally feasible manner, this work proposes some training methods and loss function that ensure input-to-state stability for LSTMs by training the ISS constraints to a desired value while maintaining the computational benefits of backpropagation-based training. Specifically, we suggest an adaptive PID-based dynamic regularization weight approach, which ensures a good prediction performance of LSTMs while converging the ISS constraint to the specified value.