To show Xcode build times, open a termina window and execute the following command:

defaults write com.apple.dt.Xcode ShowBuildOperationDuration YES

You don't even need to restart Xcode!

Thank you Sean Allen for your helpful tip!