how to start every new command line in vi-command mode for readline bash

The title says it all. I have set -o vi set for readline, but whenever I hit enter and execute a command, the new line will start in insert mode and I’ll have to manually exit. Is there are an option or workaround to always start each new line in vi command mode?