Optimal control designed with reinforcement learning can be sensitive to model mismatch. We demonstrate that designing such controllers in a virtual simulation environment with an inaccurate model is not suitable for deployment in a physical setup. Controllers designed using an accurate model are robust against disturbance and small mismatch between the physical setup and the mathematical model derived from first principles; while a poor model results in a controller that performs well in simulation but terrible in physical experiments. Sensitivity analysis was used to justify these discrepancies. We are particularly interested with a controller that is non-smooth and piecewise-affine with respect to the state variables.