Low latency для android

Прогресс налицо: за последние 4 года задержка звука на Android снизилась в три раза

Компания Google сообщила об изменениях в экосистеме Android для разработчиков, которые работают с аудио. В частности, за последние четыре года были предприняты ряд действий, направленных на уменьшение задержки звука в реальном времени. Соответственно, это привело к значительным улучшениям работы аудиоприложений на популярных Android-устройствах.

Средняя задержка звука на наиболее распространённых смартфонах на сегодняшний день составляет 40 мс, что вполне соответствует тому уровню, который необходим для аудиоприложений. Если в 2017 году разница между самой высокой и самой низкой задержками звука на Android составляла колоссальные 222 мс, то уже в 2021 году эта разница уменьшилась в 8 раз и сейчас составляет 28 мс. Впечатляющий результат, особенно учитывая, что это касается не только флагманских устройств.

Под задержкой подразумевается процесс приёма-передачи звука, который состоит из трёх этапов: аудиовход, обработка звука и аудиовыход. Многие аудиоприложения в своей работе полагаются на такие действия, как касания экрана, а не на входящий сигнал (принцип tap-to-tone). Поэтому они чувствительны к задержке — времени, которое требуется от нажатия на экран до воспроизведения звука.

Несмотря на значительное сокращение этого параметра в экосистеме Android, результат ещё далёк от идеала. Для профессиональных аудиоприложений требуется задержка 20 мс, а в долгосрочной перспективе — 10 мс. Разработчики зачастую воздерживаются от создания таких приложения на Android, но, возможно, пришло время пересмотреть своё отношение к этой платформе.

Источник

What Developers Can and Cannot Do to Lower Android Audio Latency: The Android Audio Low Latency Primer to Android’s 10ms Problem.

To best understand the origins (and fixes) of Android audio latency, it is best to segregate contributions to total round-trip Android audio latency into two parts:

1. Audio latency in Android user space. In userland, developers have many methods to lower latency. They are described below.
2. Audio latency stemming from Android system space. In system space, developers cannot affect latency, but OEMs and device manufacturers can. Options to lower audio latency originating in Android system space are also described below.

Latency is the lag or delay between between when an audio signal is sent, processed and received. Audio latency is considered professional-grade or ‘real-time’ when round-trip latency is 10 ms or less.

Audio latency in Android User Space

This includes the Android app itself, application frameworks, and 3rd party and native API and libraries such as OpenSL ES. Developers can should remember low latency best practices in Android app programming logic such as:

1. Do not do any audio processing in Java. Keep all audio processing in C++.
2. Use the JNI (Java Native Interface) for successful interoperability between native (C++) code and Java. (Prepare yourself, this is not necessarily fun).
3. Always make your code sample-rate and buffer size agnostic. Your app should work with any sample-rate or buffer size. Do not expect buffer sizes to be a power of 2, they can be any multiples of 8.
4. Do not depend on audio features offered by Android because Android audio feature implementation/function is erratic, unpredictable and may be device dependent.
5. The CPU governor reduces CPU frequency and may cause audio drop-outs. We once advised developers to use ‘fake touch events’ to keep the CPU governor from activating. However, this only works on Nexus devices. SuperpoweredCPU is a universal solution for Android CPU Frequency Scaling and is now included in Superpowered.
6. Keep in mind that there are multiple, different audio signal paths and each has its own specific latency.
7. Using Bluetooth audio will add minimum 100ms to latency.
8. Wifi will also add latency. Highly variable, depends on implementation and needs to be measured on site. Can range from 40 ms to several seconds.
9. For audio processing, do not use OpenSL ES ‘wrappers’ or OpenSL ES itself. Use self-contained android low latency audio API in static libraries like Superpowered Audio SDK.
10. Use OpenSL ES for audio I/O only.

How do I know any Android device’s round-trip audio latency?

Android audio latency is not device-specific or tied to specific hardware.

The round-trip latency on a HTC Nexus 9 LTE running Android 6.0.1 (MMB29R 2482564) is 14ms, while the same device running Android 5.1.1 (LMY47X 1849464) clocks in at 42 ms round-trip latency. The same device may exhibit different latencies depending on Android version and build as well as carrier specific modifications in the HAL (hardware abstraction layer), audio driver, and/or audio hardware settings.

To help developers determine device latency, Superpowered maintains a free, open-source iOS and Android audio latency test app as well as continuously updated public database of latencies and native buffer sizes for Android devices.

All things being equal, the lower the buffer size, the lower the latency. However, all things are rarely equal in Android.

The latency test app also records the presence of hardware feature flags:

• android.hardware.audio.low_latency indicates a continuous output latency of 45 ms or less.
• android.hardware.audio.pro indicates a continuous round-trip latency of 20 ms or less.

How does the Superpowered iOS and Android audio latency test app work?

Our latency test app measures the total round-trip latency of every component of the entire audio chain.

NOTE: the Superpowered audio latency test app for Android and iOS does not use Superpowered, and in fact, does not do any audio processing whatsoever. It measures the theoretical minimum round-trip latency of any device by measuring audio in and audio out. The latency data-table displays the best (lowest) measured value, if the dispersion is within 2x. We display a latency range if dispersion is outside of 2x.

For full transparency, the latency test app’s source is open and available on GitHub. We use the same code on all Android and iOS devices, and it works as follows:

How one measurement step works:

1. Listen and measure the average loudness of the environment for 1 second.
2. Create a threshold value 24 decibels higher than the average loudness.
3. Begin playing a 1000 Hz sine wave and start counting the samples elapsed.
4. Stop counting and playing if the input’s loudness is higher than the threshold, as the output wave is coming back (probably).
5. Divide the the elapsed samples with the sample rate to get the round-trip audio latency value in seconds.
6. We expect the threshold exceeded within 1 second. If it did not, then fail with error. Usually happens when the environment is too noisy (loud).

How the measurement process works:

1. Perform 10 measurement steps.
2. Repeat every step until it returns without an error.
3. Store the results in an array of 10 floats.
4. After each step, check the minimum and maximum values.
5. If the maximum is higher than the minimum’s double, stop the measurement process with an error. It indicates an unknown error, perhaps an unwanted noise happened. Double jitter (dispersion) is too high, an audio system can not be so bad.

The latency test app does not distinguish between input and output latency. In theory, input latency and output latency are symmetrical, however devices may have an uneven split, due to different period sizes and period counts. Furthermore, many devices feature built-in mic processing to prevent echo, suppress noise and the Larsen effect (feedback).

Lastly, the latency test app only tests for one of the four main audio signal paths. The four main possible audio signal path options, that can and do vary in latency on the same device, are:

Audio Path	Example
Inbuilt speaker and inbuilt microphone.	Your phone in your hand.
Audio output plugged into headphone jack and inbuilt microphone.	Your phone with your favorite headphones plugged in. For example, a karaoke app.
Audio input and output plugged into the headphone jack.	Your phone with either a headset (headphone + microphone) plugged in or with a special audio loopback dongle for latency measurement.
USB or HDMI audio.	A USB sound card connected via the OTG adapter.

A casual user might think each audio path returns a similar result, however, this is not the case. Below we tested a few common devices to demonstrate the differences in latencies between audio paths.

Audio Path	HTC Nexus 9 Round-Trip Audio Latency (Android 6.0)
Inbuilt speaker and inbuilt microphone.	26 ms
Audio output plugged into headphone jack and inbuilt microphone.	14 ms
Audio input and output plugged into the headphone jack. (Audio Loopback Dongle)	14 ms
USB or HDMI audio.	N/A

What about Android USB audio and Android MIDI audio latency?

This article is for readers interested in Android USB audio and Android MIDI.

How should developers work with OpenSL ES?

OpenSL ES is the standard audio API for Android apps. Android mandates that all audio in and audio out pass through OpenSL ES. However, it does not mandate that apps use OpenSL ES to process audio in the app. This allows developers to use 3rd party C++ audio libraries such as Superpowered for low latency audio processing, which radically reduces latency as compared to audio features offered by OpenSL ES. OpenSL ES feature availability also varies wildly between the different devices.

If I use Superpowered to process audio and then send it back it out through OpenSL ES, is Superpowered a wrapper?

No. Wrappers are an abstraction layer above and around the audio features of an operating system and built on top of the existing audio environment. Wrappers attempt to make it easier to interact and program for audio APIs but have no impact on the fundamental technological capabilities such as latency, audio quality or power consumption.

For example, FooAudioWrapper might allow you to interface with OpenSL ES in less complex ways but cannot deliver better actual audio performance because it still processes audio in OpenSL ES.

With Superpowered, a true audio ‘engine’, every Superpowered feature is implemented via the Superpowered library. Therefore, you not only get better performance, but since it is cross-platform, you get the same features, processing speed and audio quality regardless you run it on iOS or Android.

Audio latency in Android System (Kernel) Space

[To be more precise, Android has a system space while Linux has a kernel. The audio driver is a Linux system component, while the HAL is an Android system space component.]

For the purposes of understanding audio latency reduction, Android system space is where all the system drivers reside. These cannot be modified directly by app developers.

Of most interest in the Android audio path and its contributions to latency in system space are (in no particular order) the ALSA audio driver, the HAL (hardware abstraction layer) and AudioFlinger inside the Android Media Server.

What are the fundamental contributors to Android audio latency in system space?

There are many misconceptions as to the root cause of latency in Android system space. They are typically thought to be hardware related, Linux audio related or ALSA driver related.

Our research suggests that the root cause lies in how the ALSA and AudioFlinger communicate. Specifically, AudioFlinger ‘pushes’ audio buffers to the ALSA. ‘Pushing’ causes scheduling problems, which cause audio quality problems. All pro audio, low latency systems use the pull mechanism as does the Superpowered Media Server for Android.

If Android system space latency is non-trivial, what should developers do?

While developers cannot affect Android audio latency directly, they can lobby OEMs and device manufacturers to license the Superpowered Low Latency Media Server for Android. The Superpowered Media Server is injected into system space and radically improves audio latency performance.

How does Superpowered offer low latency audio performance?

The heart of Superpowered technology involves patent-pending innovation in algorithm selection and optimization for audio processing on low-power mobile devices. Superpowered technology use fewer CPU clock cycles which means not only is processing faster, but because it uses fewer clock cycles, it uses less power as well — allowing a mobile device to run other non-audio related tasks concurrently.

It’s like taking a VW Bug, adding technology to it, which would make it accelerate like a Porsche and get the fuel efficiency of a Prius at the same time.

The physics of automobiles won’t actually allow you do that — but the physics of digital signal processing do.

More questions you’d like to see answered here?

Superpowered products and services include:

Professional audio on Android just got Superpowered.

Источник

Android Audio’s 10 Millisecond Problem: The Android Audio Path Latency Explainer

[For practical tips on dealing with latency, Android developers please also see Android Audio Low Latency Primer.]

[UPDATED AS OF JUNE 2016: Please see Android’s 10 ms Problem? SOLVED.]

Many mobile apps that are critically dependent on low latency audio functionality such as some games, synthesizers, DAWs (Digital Audio Workstations), interactive audio apps and virtual instrument apps, and the coming wave of virtual reality apps, all of which thrive on Apple’s platform (App Store + iOS devices) — and generate big revenues for App Store and iOS developers are largely non-existent on Android.

Android Audio’s 10 Millisecond Problem, a little understood yet extremely difficult technical challenge with enormous ramifications, prevents these sorts of revenue producing apps from performing in an acceptable manner and even being published (!) on Android at this point in time.

Startups and developers are unwilling to port and publish otherwise successful iOS apps (with

10 ms audio latency needs) on Android for fear of degraded audio performance resulting in negative word-of-mouth and a hit to their professional reputation and brand.

Consumers lose because have a strong desire to buy such apps on Android, as shown by revenue data on iOS, and currently, are unable to do so. One can appreciate the scale of this problem/opportunity when one takes into account the so-called ‘next billion’ consumers who will be ‘mobile-only’.

We want to solve this. This explainer provides an easily understood overview of the Android 10 Millisecond Problem with actual latency data from the Google Nexus 9.

Contents

How Android’s 10 Millisecond Problem and Android Audio Path Latency Impacts App Developers and Android OEMs

Even though music apps make up only 3% of all downloads in the iOS App Store, the Music app category is the 3rd highest revenue generating app category after Games and Social Networking. Which suggests that music apps monetize disproportionately well on platforms that offer low latency performance such as the App Store/iOS devices.

On Android, it is a different story. In the Google Play store, the Music category is not even a top five revenue producing app category.

The overwhelming majority of Android devices suffer from too high audio latency, preventing developers from building apps that would satisfy consumer demand on Android.

As such, Google and Android app developers are leaving billions of dollars on the table for Apple and iOS developers because of Android’s 10 Millisecond Problem.

For the purposes of this explainer, roundtrip audio latency is simply the difference in time between when an audio input is introduced into a mobile device, undergo some sort of needed processing, and exits the same device. As any musician will tell you, we as humans are most comfortable with latencies of

10 milliseconds. Anything significantly higher tends to disturb us.

Most Android apps have more than 100 ms of audio output latency, and more than 200 ms of round-trip (audio input to audio output) latency. To give you a quick example from the Oscar winning film Whiplash, it’s like the drummer is dragging by a half beat behind the band!

Some specific examples on how audio related applications suffer from roundtrip audio latency greater than

• Music instruments apps, audio effect apps: musicians cannot play together on stage, as the performer using an Android device will be half beat behind the others. It’s not even usable for practicing.
• DJs can not perform beat-matching, as their pre-listening signal in their headphones is far behind the master signal playing for the audience. Applying effects like a loop roll or echo is very hard too.
• Games: sound effects, such as explosions or gun sounds lag behind by a few frames. Game audio is then “detached” from visuals, making for poor user-experience, preventing immersive gaming experiences.
• VOIP apps, such as Skype: if both users are using a high latency Android phone, the overall audio latency is higher than the network latency. In other words, it takes more time for audio to “flow” through Android, than data packets to be transferred between continents.
• Virtual reality (VR): when the viewer turns his head, the audio “follows” too late, destroying the 3D audio experience. Check the Paul McCartney Google Cardboard app for an example. Google is on the verge of leaving billions in revenue in VR opportunities for Apple.

In order to educate and inform tech industry leaders, app developers, technologists, product managers, executives, journalists, entrepreneurs, musicians, gamers and investors about the scope and ramifications of Android’s 10 Millisecond Problem, one whose existence that no one benefits from, we at Superpowered have developed the explainer you are reading right now to provide an easily digestible overview of the entire Android audio chain and potential bottlenecks.

Our goal is that we rally and to unite around this challenge of 10 ms roundtrip audio latency on Android, and moreover, transform it into an opportunity that fosters innovation, better user-experiences and benefits Google Play customers, Android developers, Android OEMs and the entire Android ecosystem.

Notes about Audio Latency

Digital audio latency measurement has two useful measurement units:

• Millisecond (ms): the 1/1000 of one second. Most latency is referred to with this measurement unit and happens in this time scale.
• Sample (or frame): represents one discrete digital point (a number) in the audio stream. Sampling is how software converts a continuous signal like a sound wave to a sequence of samples. Samples are independent of the number of audio channels. For a one channel signal, one sample means one number. One sample for a two channel signal means two numbers and so on.

We calculate the audio signal flow’s overall latency using the very best case scenario:

• Audio in Android’s native layer (Android NDK) is set up using Google’s low latency recommendations. Unfortunately, most Android applications do not follow Google’s low latency recommendations.
• On-device Android is configured appropriately and can make use of the “Fast Mixer” path for both audio input and output. Apart from the most recent Nexus models, most other manufacturers do not configure Android to support Fast Mixer, hence round-trip latency is significantly higher on those devices. Please see Superpowered’s Mobile Audio Census and Latency Test App for latency measurement data on many popular Android devices.

Android 5.0 Lollipop Audio Path Latency Explanation in Plain English

There may be several different analog components, such as a pre-amplifier for the built-in microphone. These analog components can be considered as «zero latency» in this case, because their true latency is typically magnitudes below 1 ms.

The audio chip measures the incoming audio stream in predefined intervals and converts every measurement to a number. This predefined interval is called the sampling rate, measured in Hz. Our Mobile Audio Census and Latency Test App shows that the 48000 Hz is the native sample rate for most audio chips on Android and iOS devices, meaning that the audio stream is sampled 48000 times in every second.

Because ADC implementations often contain an oversampling filter inside, a rule of thumb is to attribute 1 ms latency for the ADC step.

Now that the audio stream has been digitized, from this point forward the audio stream is now digital audio. Digital audio almost never travels one-by-one, but rather, in chunks, called «buffers» or «periods».

The audio chip has several tasks. It handles ADC and DAC, switches between or mixes several inputs and outputs, applies volume, etc. It also «groups» the discrete digital audio samples into buffers and handles the transfer of these buffers to the operating system.

The audio chip is connected to the CPU with a bus, such as USB, PCI, Firewire, etc. Every bus has its own transfer latency depending on its internal buffer sizes and buffer counts. The latency here ranges from 1 ms (audio chip on an internal system bus) to 6 ms (USB sound card with conservative USB bus settings) typically.

The audio driver receives the incoming audio into a ring buffer in «bus buffer size» steps using the audio chip’s native sample rate, 48000 Hz in most cases.

This ring buffer plays an essential part in smoothing bus transfer jitter («roughness»), and «connects» the bus transfer buffer size to the operating system audio stack’s buffer size. Consuming data from the ring buffer happens in the operating system audio stack’s buffer size, so it naturally adds some latency.

Android runs «on top» of Linux, and most Android devices use the most popular Linux audio driver system, ALSA (Advanced Linux Sound Architecture). ALSA handles the ring buffer like this:

• Audio is consumed from the ring buffer in «period size» steps.
• The ring buffer’s size is a multiple of the «period size».

• Period size = 480 samples.
• Period count = 2.
• The ring buffer’s size is 480×2 = 960 samples.
• Audio input is received into one period (480 samples), while the audio stack reads/processes the other period (480 samples).
• Latency = 1 period, 480 samples. It equals to 10 ms at 48000 Hz.

ring buffer (960 samples)
period (480 samples)	period (480 samples)

A common period count is 2, but some systems may go higher..

Latency: one or more periods

The HAL acts as a middleman between the Android media server and the Linux audio driver. HAL implementations are provided by the mobile device’s manufacturer upon «porting» Android onto the device.

Implementations are open, vendors are free to create any kind of HAL code. Communication with the media server happens using predefined structures. The media server loads the HAL and asks to create input or output streams with optional preferred parameters such as the sample rate, buffer size or audio effects.

Note: The HAL may or may not perform according to the parameters and the media server must «adapt» to the HAL.

The typical HAL implementation is tinyALSA, which is used to communicate with the ALSA audio driver. Some vendors put closed source code here to implement audio features they feel important.

After analyzing the code of a number of open source HAL implementations in the Android source repository, we found a few quirks adding significant amount of latency and CPU load unnecessarily to the audio path due strange configurations and poor coding.

A good HAL implementation should not add any latency.

Latency: 0 or more samples

The Android media server consists of two services:

• The AudioPolicy service handles audio session and permission handling, such as enabling access to the microphone or interrupts on calls. It’s very similar to iOS’ audio session handling.
• The AudioFlinger service handles the digital audio streams.

Audio Flinger creates a RecordThread, which acts as a middleman between an application and the audio driver. Its basic job is:

• Obtaining the next input audio buffer from the driver’s ring buffer using the Android HAL.
• Resampling the buffer if the application requests different sample rate than the native sample rate.
• Performing additional buffering if the application requests different buffer size than the native period size.

Audio Flinger has a «fast mixer» path, if Android is configured that way. If a user application is using native (Android NDK) code and sets up an audio buffer queue with the native hardware sample rate and period size, no resampling, additional buffering or mixing («MixerThread») will happen in this step.

The RecordThread works with a «push» method, without any strict synchronization to the audio driver. It tries to make an «educated guess» when to wake up and run, but the «push» method is way more sensitive to dropouts. Low latency systems always use the «pull» method, where the audio driver «dictates» audio i/o through the entire audio chain. It’s clear that when Android OS was initially conceived, designed and developed, low latency audio was not a priority.

Latency: 1 period (best case scenario)

Shared memory in Android’s main inter-process communication system is used to transfer the audio buffers between Audio Flinger and the user application. It’s the heart of Android, used everywhere internally in Android.

We are in the user application’s process now. AudioRecord implements the application side of the audio input. This is a client library feature accessible via OpenSL ES for example.

AudioRecord runs a thread to periodically acquire a new buffer from Audio Flinger, with the «push» philosophy described at Audio Flinger. It doesn’t add latency to the audio path if the developer sets it to work with only one buffer.

Latency: 0+ samples

Finally, the audio input reaches its destination, the user application.

Because the input and output threads are not the same, a user application must implement a ring buffer between the threads. Its size is 2 periods minimum (1 for audio input and 1 for audio output), but poorly written applications often use brute force and use more periods to solve CPU bottlenecks.

From this point, we are we start traveling back out with some audio output.

Latency: more than 1 period, near 2 typically (best case scenario)

AudioTrack Implements the user application’s side of the audio output. This is a client library feature accessible via OpenSL ES for example. It runs a thread to periodically send the next audio buffer to Audio Flinger. After Android 4.4.4, AudioTrack doesn’t add latency to the audio path as it can be set up to use one buffer only.

Latency: 0+ samples

Same as for audio input.

Creates a PlaybackThread, which works as the inverse of the RecordThread described at audio input.

Latency: 1 period (best case scenario)

Same as for audio input.

Latency: 0 or more samples

Audio output in the audio driver works identically to the audio input and uses a ring buffer too.

Latency: one or more periods

Similar to the audio input’s bus transfer, the latency here ranges from 1 ms to 6 ms typically.

The inverse of ADC, digital audio is «converted» back to analog in this point. For the same reasons at ADC, a rule of thumb is to assume 1 ms of latency for DAC.

The DAC’s output signal is analog audio, but it needs additional components to drive connected devices, such as headphones. Similar to the analog audio input, the analog components can be considered to be «zero latency».

Android Audio Path Latency Animation

Want to embed this image? Copy the following code:

Android Audio Path Latency Case study: Google Nexus 9

To date, the Google Nexus 9 performs best in Android round-trip audio latency measurement tests.

The best result is 35 ms using a USB sound card or a special audio dongle directly connecting the headphone connector’s mic input and output, to disable the built-in microphone array’s noise canceling/feedback destroying feature which adds about

13 ms of additional latency.

So, using the same model as above, let’s decompose the 35 ms best-case round-trip audio latency of Google Nexus 9:

How the 35 ms round-trip latency of Google Nexus 9 comes up?

Component	Samples	Ms
ADC	1
Bus	1
ALSA audio driver	256	5.3
Audio Flinger	256	5.3
User Application’s Ring Buffer	512	10.6
Audio Flinger	256	5.3
ALSA audio driver	256	5.3
Bus	1
DAC	1
Result:		35.8

About Superpowered

Our mission is to extend the makers’ creative and productive capabilities – allowing them to create and make things real – profoundly shaping them, the builders, to build things that weren’t possible without Superpowered audio technology.

To that end, we are building technology, traversing the audio stack, that will solve Android’s 10 Millisecond Problem. In the meantime, the Superpowered Audio SDK for Android and iOS is:

• Cross-platform: Developers can use and re-use the same code on Android, iOS and OSX.
• Super fast: it has the highest performance audio DSP on mobile devices, providing desktop-grade processing and pro audio quality. The reduced CPU load improves battery life and smoothness of all applications.
• Works great for both «push» and «pull» audio stacks, and even offline processing.
• 0 latency: Superpowered features and processing don’t add any latency. User applications built with the Superpowered Audio SDK easily run with the lowest audio latency possible on any iOS or Android device.

We’d love to hear from you. Please email us with your suggestions, comments and questions. Hello@Superpowered.com

Thanks for reading.

Gabor (@szantog) and Patrick (@Pv), founders of Superpowered

PS Please join the great conversations about Android’s 10 ms problem at Reddit and Hacker News.

Источник