In this talk, we study an exploration version of continuous time expected utility maximization problem with reinforcement learning. It is shown that the optimal feedback policy is Gaussian. We then prove a policy improvement theorem. An implementable reinforcement learning algorithm is designed. Numerical examples are provided for illustrations.