VM slow down when video recording enabled

[gim]

To reproduce bug you can try create VM with one core and start it with and without video recording. With video recording enabled you must see some slow down of working VM.

This happens because of slow working vpx_codec_encode call under lock inside RecordingStream::Process:

https://www.virtualbox.org/browser/vbox/trunk/src/VBox/Main/src-client/RecordingStream.cpp#L373

int RecordingStream::Process(RecordingBlockMap &mapBlocksCommon)
{
    LogFlowFuncEnter();

    lock();

    ...
                    rc2 = writeVideoVPX(msTimestamp, pVideoFrame);
    ...

    unlock();

    LogFlowFuncLeaveRC(rc);
    return rc;
}

And writeVideoVPX calls vpx_codec_encode: https://www.virtualbox.org/browser/vbox/trunk/src/VBox/Main/src-client/RecordingStream.cpp#L1161

vpx_codec_err_t rcv = vpx_codec_encode(&pCodec->VPX.Ctx,
                                           &pCodec->VPX.RawImage,
                                           pts                          /* Timestamp */,
                                           this->Video.uDelayMs         /* How long to show this frame */,
                                           0                            /* Flags */,
                                           pCodec->VPX.uEncoderDeadline /* Quality setting */);

In fact vpx_codec_encode working slow beacuse of last parameter "deadline" by default is zero (which means infinite deadline for encoding)

Interesting part is that in 5.x virtualbox branch there was a code which fill the default value of uEncoderDeadline but this code not present in 6.0/6.1 branches...

/**
 * Invalidates the video recording configuration.
 */
void Display::i_videoRecInvalidate(void)
{
    AssertPtr(mParent);
    ComPtr<IMachine> pMachine = mParent->i_machine();
    Assert(pMachine.isNotNull());

    mVideoRecCfg.enmDst = VIDEORECDEST_FILE; /** @todo Make this configurable once we have more variations. */

    /*
     * Cache parameters from API.
     */
    com::SafeArray<BOOL> aScreens;
    HRESULT hrc = pMachine->COMGETTER(VideoCaptureScreens)(ComSafeArrayAsOutParam(aScreens));
    AssertComRCReturnVoid(hrc);

    mVideoRecCfg.aScreens.resize(aScreens.size());
    for (size_t i = 0; i < aScreens.size(); ++i)
        mVideoRecCfg.aScreens[i] = aScreens[i];

    hrc = pMachine->COMGETTER(VideoCaptureWidth)((ULONG *)&mVideoRecCfg.Video.uWidth);
    AssertComRCReturnVoid(hrc);
    hrc = pMachine->COMGETTER(VideoCaptureHeight)((ULONG *)&mVideoRecCfg.Video.uHeight);
    AssertComRCReturnVoid(hrc);
    hrc = pMachine->COMGETTER(VideoCaptureRate)((ULONG *)&mVideoRecCfg.Video.uRate);
    AssertComRCReturnVoid(hrc);
    hrc = pMachine->COMGETTER(VideoCaptureFPS)((ULONG *)&mVideoRecCfg.Video.uFPS);
    AssertComRCReturnVoid(hrc);
    hrc = pMachine->COMGETTER(VideoCaptureFile)(mVideoRecCfg.File.strName.asOutParam());
    AssertComRCReturnVoid(hrc);
    hrc = pMachine->COMGETTER(VideoCaptureMaxFileSize)((ULONG *)&mVideoRecCfg.File.uMaxSizeMB);
    AssertComRCReturnVoid(hrc);
    hrc = pMachine->COMGETTER(VideoCaptureMaxTime)((ULONG *)&mVideoRecCfg.uMaxTimeS);
    AssertComRCReturnVoid(hrc);
    BSTR bstrOptions;
    hrc = pMachine->COMGETTER(VideoCaptureOptions)(&bstrOptions);
    AssertComRCReturnVoid(hrc);

    /*
     * Set sensible defaults.
     */
    mVideoRecCfg.Video.fEnabled = true; /* Enabled by default. */

    if (!mVideoRecCfg.Video.uFPS) /* Prevent division by zero. */
        mVideoRecCfg.Video.uFPS = 15;

#ifdef VBOX_WITH_LIBVPX
    mVideoRecCfg.Video.Codec.VPX.uEncoderDeadline = 1000000 / mVideoRecCfg.Video.uFPS;
#endif

#ifdef VBOX_WITH_AUDIO_VIDEOREC
    mVideoRecCfg.Audio.fEnabled  = false; /* Disabled by default, unless set otherwise below. */
    /* By default we use 48kHz, 16-bit, stereo for the audio track. */
    mVideoRecCfg.Audio.uHz       = 48000;
    mVideoRecCfg.Audio.cBits     = 16;
    mVideoRecCfg.Audio.cChannels = 2;
#endif

Here is:

mVideoRecCfg.Video.Codec.VPX.uEncoderDeadline = 1000000 / mVideoRecCfg.Video.uFPS;

[gim]

Actually this problem exists more than one year probably: ​https://forums.virtualbox.org/viewtopic.php?f=1&t=93307