The --output specifier, although it is syntactically an option and can appear at any point among the options (it doesn't have to be first), is not optional. The specified filename is created under umask 077 if nonexistent; if it already exists and is non-empty, a warning message about that is sent to standard error, and the output is appended to the file.
The --quiet option suppresses both the rsasigkey narrative and the existing-file warning message.
The --bits option specifies the number of bits in the key. If not specified, rsasigkey will be invoked with the --vary option, so as to generate a key with a variant number of bits (within certain limits; see ipsec_rsasigkey(8)).
The --hostname option is passed through to rsasigkey to tell it what host name to label the output with (via its --hostname option).
The output format is that of rsasigkey, with bracketing added to complete the ipsec.secrets format. In the usual case, where ipsec.secrets contains only the host's own private key, the output of newhostkey is sufficient as a complete ipsec.secrets file.
A higher-level tool which could handle the clerical details of changing to a new key would be helpful.
The requirement for --output is a blemish, but private keys are extremely sensitive information and unusual precautions seem justified.