Changeset 82256 in vbox for trunk/include
- Timestamp:
- Nov 27, 2019 11:45:34 PM (5 years ago)
- File:
-
- 1 edited
-
trunk/include/VBox/vmm/pdmaudioifs.h (modified) (2 diffs)
Legend:
- Unmodified
- Added
- Removed
-
trunk/include/VBox/vmm/pdmaudioifs.h
r82255 r82256 24 24 */ 25 25 26 /** 27 * == Audio architecture overview 28 * 29 * The audio architecture mainly consists of two PDM interfaces, PDMAUDIOCONNECTOR 30 * and PDMIHOSTAUDIO. 31 * 32 * The PDMAUDIOCONNECTOR interface is responsible of connecting a device emulation, such 33 * as SB16, AC'97 and HDA to one or multiple audio backend(s). Its API abstracts audio 34 * stream handling and I/O functions, device enumeration and so on. 35 * 36 * The PDMIHOSTAUDIO interface must be implemented by all audio backends to provide an 37 * abstract and common way of accessing needed functions, such as transferring output audio 38 * data for playing audio or recording input from the host. 39 * 40 * A device emulation can have one or more LUNs attached to it, whereas these LUNs in turn 41 * then all have their own PDMIAUDIOCONNECTOR, making it possible to connect multiple backends 42 * to a certain device emulation stream (multiplexing). 43 * 44 * An audio backend's job is to record and/or play audio data (depending on its capabilities). 45 * It highly depends on the host it's running on and needs very specific (host-OS-dependent) code. 46 * The backend itself only has very limited ways of accessing and/or communicating with the 47 * PDMIAUDIOCONNECTOR interface via callbacks, but never directly with the device emulation or 48 * other parts of the audio sub system. 49 * 50 * 51 * == Mixing 52 * 53 * The AUDIOMIXER API is optionally available to create and manage virtual audio mixers. 54 * Such an audio mixer in turn then can be used by the device emulation code to manage all 55 * the multiplexing to/from the connected LUN audio streams. 56 * 57 * Currently only input and output stream are supported. Duplex stream are not supported yet. 58 * 59 * This also is handy if certain LUN audio streams should be added or removed during runtime. 60 * 61 * To create a group of either input or output streams the AUDMIXSINK API can be used. 62 * 63 * For example: The device emulation has one hardware output stream (HW0), and that output 64 * stream shall be available to all connected LUN backends. For that to happen, 65 * an AUDMIXSINK sink has to be created and attached to the device's AUDIOMIXER object. 66 * 67 * As every LUN has its own AUDMIXSTREAM object, adding all those objects to the 68 * just created audio mixer sink will do the job. 26 /** @page pg_pdm_audio PDM Audio 27 * 28 * @section sec_pdm_audio_overview Audio architecture overview 29 * 30 * The audio architecture mainly consists of two PDM interfaces, 31 * PDMAUDIOCONNECTOR and PDMIHOSTAUDIO. 32 * 33 * The PDMAUDIOCONNECTOR interface is responsible of connecting a device 34 * emulation, such as SB16, AC'97 and HDA to one or multiple audio backend(s). 35 * Its API abstracts audio stream handling and I/O functions, device enumeration 36 * and so on. 37 * 38 * The PDMIHOSTAUDIO interface must be implemented by all audio backends to 39 * provide an abstract and common way of accessing needed functions, such as 40 * transferring output audio data for playing audio or recording input from the 41 * host. 42 * 43 * A device emulation can have one or more LUNs attached to it, whereas these 44 * LUNs in turn then all have their own PDMIAUDIOCONNECTOR, making it possible 45 * to connect multiple backends to a certain device emulation stream 46 * (multiplexing). 47 * 48 * An audio backend's job is to record and/or play audio data (depending on its 49 * capabilities). It highly depends on the host it's running on and needs very 50 * specific (host-OS-dependent) code. The backend itself only has very limited 51 * ways of accessing and/or communicating with the PDMIAUDIOCONNECTOR interface 52 * via callbacks, but never directly with the device emulation or other parts of 53 * the audio sub system. 54 * 55 * 56 * @section sec_pdm_audio_mixing Mixing 57 * 58 * The AUDIOMIXER API is optionally available to create and manage virtual audio 59 * mixers. Such an audio mixer in turn then can be used by the device emulation 60 * code to manage all the multiplexing to/from the connected LUN audio streams. 61 * 62 * Currently only input and output stream are supported. Duplex stream are not 63 * supported yet. 64 * 65 * This also is handy if certain LUN audio streams should be added or removed 66 * during runtime. 67 * 68 * To create a group of either input or output streams the AUDMIXSINK API can be 69 * used. 70 * 71 * For example: The device emulation has one hardware output stream (HW0), and 72 * that output stream shall be available to all connected LUN backends. For that 73 * to happen, an AUDMIXSINK sink has to be created and attached to the device's 74 * AUDIOMIXER object. 75 * 76 * As every LUN has its own AUDMIXSTREAM object, adding all those 77 * objects to the just created audio mixer sink will do the job. 69 78 * 70 79 * @note The AUDIOMIXER API is purely optional and is not used by all currently … … 72 81 * 73 82 * 74 * == Data processing 75 * 76 * Audio input / output data gets handed-off to/from the device emulation in an unmodified 77 * - that is, raw - way. The actual audio frame / sample conversion is done via the PDMAUDIOMIXBUF API. 78 * 79 * This concentrates the audio data processing in one place and makes it easier to test / benchmark 80 * such code. 81 * 82 * A PDMAUDIOFRAME is the internal representation of a single audio frame, which consists of a single left 83 * and right audio sample in time. Only mono (1) and stereo (2) channel(s) currently are supported. 84 * 85 * 86 * == Timing 87 * 88 * Handling audio data in a virtual environment is hard, as the human perception is very sensitive 89 * to the slightest cracks and stutters in the audible data. This can happen if the VM's timing is 90 * lagging behind or not within the expected time frame. 91 * 92 * The two main components which unfortunately contradict each other is a) the audio device emulation 93 * and b) the audio backend(s) on the host. Those need to be served in a timely manner to function correctly. 94 * To make e.g. the device emulation rely on the pace the host backend(s) set - or vice versa - will not work, 95 * as the guest's audio system / drivers then will not be able to compensate this accordingly. 96 * 97 * So each component, the device emulation, the audio connector(s) and the backend(s) must do its thing 98 * *when* it needs to do it, independently of the others. For that we use various (small) ring buffers to 99 * (hopefully) serve all components with the amount of data *when* they need it. 100 * 101 * Additionally, the device emulation can run with a different audio frame size, while the backends(s) may 102 * require a different frame size (16 bit stereo -> 8 bit mono, for example). 103 * 104 * The device emulation can give the audio connector(s) a scheduling hint (optional), e.g. in which interval 105 * it expects any data processing. 106 * 107 * A data transfer for playing audio data from the guest on the host looks like this: 108 * (RB = Ring Buffer, MB = Mixing Buffer) 109 * 110 * (A) Device DMA -> (B) Device RB -> (C) Audio Connector Guest MB -> (D) Audio Connector Host MB -> \ 111 * (E) Backend RB (optional, up to the backend) > (F) Backend audio framework 112 * 113 * For capturing audio data the above chain is similar, just in a different direction, of course. 114 * 115 * The audio connector hereby plays a key role when it comes to (pre-) buffering data to minimize any audio stutters 116 * and/or cracks. The following values, which also can be tweaked via CFGM / extra-data are available: 117 * 118 * - The pre-buffering time (in ms): Audio data which needs to be buffered before any playback (or capturing) can happen. 119 * - The actual buffer size (in ms): How big the mixing buffer (for C and D) will be. 120 * - The period size (in ms): How big a chunk of audio (often called period or fragment) for F must be to get handled correctly. 121 * 122 * The above values can be set on a per-driver level, whereas input and output streams for a driver also can be handled 123 * set independently. The verbose audio (release) log will tell about the (final) state of each audio stream. 124 * 125 * 126 * == Diagram 127 * 128 * +-------------------------+ 129 * +-------------------------+ +-------------------------+ +-------------------+ 130 * |PDMAUDIOSTREAM | |PDMAUDIOCONNECTOR | + ++|LUN | 131 * |-------------------------| |-------------------------| | |||-------------------| 132 * |PDMAUDIOMIXBUF |+------>|PDMAUDIOSTREAM Host |+---|-|||PDMIAUDIOCONNECTOR | 133 * |PDMAUDIOSTREAMCFG |+------>|PDMAUDIOSTREAM Guest | | |||AUDMIXSTREAM | 134 * | | |Device capabilities | | ||| | 135 * | | |Device configuration | | ||| | 136 * | | | | | ||| | 137 * | | +|PDMIHOSTAUDIO | | ||| | 138 * | | ||+-----------------------+| | ||+-------------------+ 139 * +-------------------------+ |||Backend storage space || | || 140 * ||+-----------------------+| | || 141 * |+-------------------------+ | || 142 * | | || 143 * +---------------------+ | | || 144 * |PDMIHOSTAUDIO | | | || 145 * |+--------------+ | | +-------------------+ | || +-------------+ 146 * ||DirectSound | | | |AUDMIXSINK | | || |AUDIOMIXER | 147 * |+--------------+ | | |-------------------| | || |-------------| 148 * | | | |AUDMIXSTREAM0 |+---|-||----->|AUDMIXSINK0 | 149 * |+--------------+ | | |AUDMIXSTREAM1 |+---|-||----->|AUDMIXSINK1 | 150 * ||PulseAudio | | | |AUDMIXSTREAMn |+---|-||----->|AUDMIXSINKn | 151 * |+--------------+ |+----------+ +-------------------+ | || +-------------+ 152 * | | | || 153 * |+--------------+ | | || 154 * ||Core Audio | | | || 155 * |+--------------+ | | || 156 * | | | || 157 * | | | ||+----------------------------------+ 158 * | | | |||Device (SB16 / AC'97 / HDA) | 159 * | | | |||----------------------------------| 160 * +---------------------+ | |||AUDIOMIXER (Optional) | 161 * | |||AUDMIXSINK0 (Optional) | 162 * | |||AUDMIXSINK1 (Optional) | 163 * | |||AUDMIXSINKn (Optional) | 164 * | ||| | 165 * | |+|LUN0 | 166 * | ++|LUN1 | 167 * +--+|LUNn | 168 * | | 169 * | | 170 * | | 171 * +----------------------------------+ 83 * @section sec_pdm_audio_data_processing Data processing 84 * 85 * Audio input / output data gets handed off to/from the device emulation in an 86 * unmodified (raw) way. The actual audio frame / sample conversion is done via 87 * the PDMAUDIOMIXBUF API. 88 * 89 * This concentrates the audio data processing in one place and makes it easier 90 * to test / benchmark such code. 91 * 92 * A PDMAUDIOFRAME is the internal representation of a single audio frame, which 93 * consists of a single left and right audio sample in time. Only mono (1) and 94 * stereo (2) channel(s) currently are supported. 95 * 96 * 97 * @section sec_pdm_audio_timing Timing 98 * 99 * Handling audio data in a virtual environment is hard, as the human perception 100 * is very sensitive to the slightest cracks and stutters in the audible data. 101 * This can happen if the VM's timing is lagging behind or not within the 102 * expected time frame. 103 * 104 * The two main components which unfortunately contradict each other is a) the 105 * audio device emulation and b) the audio backend(s) on the host. Those need to 106 * be served in a timely manner to function correctly. To make e.g. the device 107 * emulation rely on the pace the host backend(s) set - or vice versa - will not 108 * work, as the guest's audio system / drivers then will not be able to 109 * compensate this accordingly. 110 * 111 * So each component, the device emulation, the audio connector(s) and the 112 * backend(s) must do its thing *when* it needs to do it, independently of the 113 * others. For that we use various (small) ring buffers to (hopefully) serve all 114 * components with the amount of data *when* they need it. 115 * 116 * Additionally, the device emulation can run with a different audio frame size, 117 * while the backends(s) may require a different frame size (16 bit stereo 118 * -> 8 bit mono, for example). 119 * 120 * The device emulation can give the audio connector(s) a scheduling hint 121 * (optional), e.g. in which interval it expects any data processing. 122 * 123 * A data transfer for playing audio data from the guest on the host looks like 124 * this: (RB = Ring Buffer, MB = Mixing Buffer) 125 * 126 * (A) Device DMA -> (B) Device RB -> (C) Audio Connector Guest MB -> (D) Audio 127 * Connector Host MB -> (E) Backend RB (optional, up to the backend) -> (F) 128 * Backend audio framework. 129 * 130 * When capturing audio data the chain is similar to the above one, just in a 131 * different direction, of course. 132 * 133 * The audio connector hereby plays a key role when it comes to (pre-)buffering 134 * data to minimize any audio stutters and/or cracks. The following values, 135 * which also can be tweaked via CFGM / extra-data are available: 136 * 137 * - The pre-buffering time (in ms): Audio data which needs to be buffered 138 * before any playback (or capturing) can happen. 139 * - The actual buffer size (in ms): How big the mixing buffer (for C and D) 140 * will be. 141 * - The period size (in ms): How big a chunk of audio (often called period or 142 * fragment) for F must be to get handled correctly. 143 * 144 * The above values can be set on a per-driver level, whereas input and output 145 * streams for a driver also can be handled set independently. The verbose audio 146 * (release) log will tell about the (final) state of each audio stream. 147 * 148 * 149 * @section sec_pdm_audio_diagram Diagram 150 * 151 * @todo r=bird: Not quite able to make sense of this, esp. the 152 * AUDMIXSINK/AUDIOMIXER bits crossing the LUN connections. 153 * 154 * @verbatim 155 +----------------------------------+ 156 |Device (SB16 / AC'97 / HDA) | 157 |----------------------------------| 158 |AUDIOMIXER (Optional) | 159 |AUDMIXSINK0 (Optional) | 160 |AUDMIXSINK1 (Optional) | 161 |AUDMIXSINKn (Optional) | 162 | | 163 | L L L | 164 | U U U | 165 | N N N | 166 | 0 1 n | 167 +-----+----+----+------------------+ 168 | | | 169 | | | 170 +--------------+ | | | +-------------+ 171 |AUDMIXSINK | | | | |AUDIOMIXER | 172 |--------------| | | | |-------------| 173 |AUDMIXSTREAM0 |+-|----|----|-->|AUDMIXSINK0 | 174 |AUDMIXSTREAM1 |+-|----|----|-->|AUDMIXSINK1 | 175 |AUDMIXSTREAMn |+-|----|----|-->|AUDMIXSINKn | 176 +--------------+ | | | +-------------+ 177 | | | 178 | | | 179 +----+----+----+----+ 180 |LUN | 181 |-------------------| 182 |PDMIAUDIOCONNECTOR | 183 |AUDMIXSTREAM | 184 | +------+ 185 | | | 186 | | | 187 | | | 188 +-------------------+ | 189 | 190 +-------------------------+ | 191 +-------------------------+ +----+--------------------+ 192 |PDMAUDIOSTREAM | |PDMIAUDIOCONNECTOR | 193 |-------------------------| |-------------------------| 194 |PDMAUDIOMIXBUF |+------>|PDMAUDIOSTREAM Host | 195 |PDMAUDIOSTREAMCFG |+------>|PDMAUDIOSTREAM Guest | 196 | | |Device capabilities | 197 | | |Device configuration | 198 | | | | 199 | | +--+|PDMIHOSTAUDIO | 200 | | | |+-----------------------+| 201 +-------------------------+ | ||Backend storage space || 202 | |+-----------------------+| 203 | +-------------------------+ 204 | 205 +---------------------+ | 206 |PDMIHOSTAUDIO | | 207 |+--------------+ | | 208 ||DirectSound | | | 209 |+--------------+ | | 210 | | | 211 |+--------------+ | | 212 ||PulseAudio | | | 213 |+--------------+ |+-------+ 214 | | 215 |+--------------+ | 216 ||Core Audio | | 217 |+--------------+ | 218 | | 219 | | 220 | | 221 | | 222 +---------------------+ 223 @endverbatim 172 224 */ 173 225
Note:
See TracChangeset
for help on using the changeset viewer.
