This predoctoral presentation will mainly focus on understanding and exploring the properties of recurrent neural networks (RNNs) which are proved to have strong capacities for modeling highly nonlinear dynamics in sequential data. I will introduce two of my finished works on RNNs: In the first work, I will talk about measuring the architectural complexity of the general RNN structure, which is strongly related to RNN's connecting topology. In the second work, I will talk about a general structural design which is applicable for any popular recurrent structures (vanilla RNN, LSTM, GRU, etc.). This structural design is extremely easy to implement and can bring consistent performance improvement on a wide range of sequential tasks.