ExoPlayer Hacks: Part 2 – Improved Buffering

This is 2nd post in ExoPlayer Hacks series, if you are wondering what was there in Part1, you can catch it up here.

What about Buffering?

OK there is nothing wrong with way Buffering is handled in Exo, but when I went through DefaultLoadControl component it seems that buffering policy is constrained. I will explain how part in detail later. When asked why, one of the Exo developers responded as below

“There are arguments that mobile carriers prefer this kind of traffic pattern over their networks (i.e. bursts rather than drip-feeding). Which is an important consideration given ExoPlayer is used by some very popular services. It may also be more battery efficient.”

though default policy seem to work in general, it might not be sufficient for all scenarios.

In my case I have to increase the buffer to 2-3 minutes on our TV platform with only 2 renditions available which means rendition switch wont happen too often.

First, let’s understand current Buffer policy

Default Buffer Policy

/**
 * The default minimum duration of media that the player will attempt to ensure is buffered at all
 * times, in milliseconds.
 */
public static final int DEFAULT_MIN_BUFFER_MS = 15000;
/**
 * The default maximum duration of media that the player will attempt to buffer, in milliseconds.
 */
public static final int DEFAULT_MAX_BUFFER_MS = 30000;
/**
 * The default duration of media that must be buffered for playback to start or resume following a
 * user action such as a seek, in milliseconds.
 */
public static final int DEFAULT_BUFFER_FOR_PLAYBACK_MS = 2500;
/**
 * The default duration of media that must be buffered for playback to resume after a rebuffer,
 * in milliseconds. A rebuffer is defined to be caused by buffer depletion rather than a user
 * action.
 */
public static final int DEFAULT_BUFFER_FOR_PLAYBACK_AFTER_REBUFFER_MS  = 5000;

Out of above 4 constants, 2 of them control the load timing and the other two control when the playback starts by having sufficient buffering. Please refer to the figure below.

The maxBufferUs & minBufferUs are about the load timing but bufferForPlaybackAfterRebufferUs & bufferForPlaybackUs are about when the playback starts by having sufficient buffering.

b8cfffe2-f02f-11e6-8781-3e3ba43aa81d

Refer to shouldContinueLoading(), a typical loading pattern includes 3 stages.

@Override
public boolean shouldContinueLoading(long bufferedDurationUs) {
  int bufferTimeState = getBufferTimeState(bufferedDurationUs);
  boolean targetBufferSizeReached = allocator.getTotalBytesAllocated() >= targetBufferSize;
  boolean wasBuffering = isBuffering;
  isBuffering = bufferTimeState == BELOW_LOW_WATERMARK
      || (bufferTimeState == BETWEEN_WATERMARKS && isBuffering && !targetBufferSizeReached);
  if (priorityTaskManager != null && isBuffering != wasBuffering) {
    if (isBuffering) {
      priorityTaskManager.add(C.PRIORITY_PLAYBACK);
    } else {
      priorityTaskManager.remove(C.PRIORITY_PLAYBACK);
    }
  }
  return isBuffering;
}

At the first stage we continuing loading until maxBufferUs is reached as below.

812da840-f030-11e6-9aee-f0f224144a37

Since then isBuffering = false and bufferTimeState == BETWEEN_WATERMARKS; so shouldContinueLoading() returns false as below.

d758d3a2-f030-11e6-9b14-ff4f6c7857df

Finally, when bufferTimeState == BELOW_LOW_WATERMARK again we recover the download, as the figure below.

ffc588ee-f030-11e6-9bd9-fffb11aa52b3

As per the flow described, you could see within the period of [t1, t3] actually we DO NOT load anymore. Therefore you may meet the condition, for example to have merely ~ 15 seconds buffering.

Besides, there is a limit upon buffering size. In short, typically the buffering data will be from 15 to 30 second if your connection speed is good enough.

How to enlarge the buffer?

Remove the upper limit by applying ‘Drip – Feeding’ method.

isBuffering = bufferTimeState == BELOW_LOW_WATERMARK
    || (bufferTimeState == BETWEEN_WATERMARKS
        /*
         * commented below line to achieve drip-feeding method for better caching. once you are below maxBufferUs, do fetch immediately.
         * Added by Sri
         */
        /* && isBuffering */
        && !targetBufferSizeReached);

Also enlarge maxBufferUs & minBufferUs

/**
 * To increase buffer time and size.
 */
public static int VIDEO_BUFFER_SCALE_UP_FACTOR = 4;
....
minBufferUs = VIDEO_BUFFER_SCALE_UP_FACTOR * minBufferMs * 1000L;
maxBufferUs = VIDEO_BUFFER_SCALE_UP_FACTOR * maxBufferMs * 1000L;
...

checkout the CustomLoadControl for code reference.

You can add below log in shouldContinueLoading() to verify the improvement before and after.

Log.e("CustomLoadControl","current buff Dur: "+bufferedDurationUs+",max buff:" + maxBufferUs +" shouldContinueLoading: "+isBuffering);

Cool. yeah!!

Hope you enjoyed both of these articles. If you are ExoPlayer lover, do try these hacks (github demo) and leave me your feedback.

ExoPlayer Hacks: Part 1 – Stats For Nerds

Google’s ExoPlayer is best in class media player for Android and it supports many features which are not currently supported by stock MediaPlayer.

Best part about Exo is it’s  OpenSource-ness 🙂

In this series of blog posts I’m gonna walk through a couple of useful hacks/customisations on some of Exo components:

  • Stats For Nerds – Part 1
  • Improved Buffering – Part 2

Find the demo on github which covers both features.

Stats for Nerds:

charts

With Adaptive streaming techniques(HLS, Smooth, DASH), playback is heavily relied on connection speed, meaning resolution changes to adapt to fluctuations in bandwidth which means data consumption varies with time.

As a geek I’ve wondered what’s my n/w consumption, conn speed & sometimes buffer length when I stream a video online. Youtube on Chrome answered these concerns with ‘Stats For Nerd’ mode which depicts all those stats using simple dynamic charts and I found it intriguing. So I decided to do something similar on ExoPlayer and turned out it’s easier than I thought.

Which Stats?

  • Connection Speed:
  • Network Activity
  • Buffer Health
  • Dropped Frames
  • Video & Screen Resolutions

Code Please

  • Connection speed & n/w activity can be obtained by passing BandwidthMeter.EventListener while creating DefaultBandwidthMeter object.
bandwidthMeter = new DefaultBandwidthMeter(mUiUpdateHandler, new BandwidthMeter.EventListener() {
    @Override
    public void onBandwidthSample(int elapsedMs, long bytes, long bitrate) {
        bitrateEstimate = bitrate;
        bytesDownloaded = bytes;
    }
});
  • ExoPlayer exposes all available Tracks and their corresponding Formats and setting a Video debug listener will give update you whenever input video format changes. Format is a container of all meta data related to a Video Rendition (width, height, bitrate etc.)
  • Dropped frames can be obtained by Video debug listener too.
player.setVideoDebugListener(new VideoRendererEventListener() {
  
    ....
    @Override
    public void onVideoInputFormatChanged(Format format) {
        currentVideoFormat = format;
    }

    @Override
    public void onDroppedFrames(int count, long elapsedMs) {
        droppedFrames += count;
    }
    ......
}

 Buffer health & LoadControl

LoadControl is an Exo Component interface to control buffering of Media and there is a DefaultLoadControl in SDK which takes of when to start buffering and for how long.

We can create our version of DefaultLoadControl with an event listener & a handler to notify player on a buffer data. Check CustomLoadControl class for more info.

player = ExoPlayerFactory.newSimpleInstance(renderersFactory, trackSelector, new CustomLoadControl(new CustomLoadControl.EventListener() {
    @Override
    public void onBufferedDurationSample(long bufferedDurationUs) {
        bufferedDurationMs = bufferedDurationUs;
    }
}, mUiUpdateHandler));
player.addListener(this);

OK. How to draw those awesome charts?

MPAndroidChart is a second to none charting library in Android and it covers all our charting requirements with no exceptions.

Create a Handler to repetitively call to itself every 500ms to update speed & buffer data and N/W activity has to be updated atleast with 1100ms so that updates wont overlap.

private void startPlayerStats() {
    mUiUpdateHandler.removeMessages(MSG_UPDATE_STATS);
    mUiUpdateHandler.removeMessages(MSG_UPDATE_STATS_NW_ONLY);
    depictPlayerStats();
    depictPlayerNWStats();
}

protected void depictPlayerStats() {
    if (!canShowStats())
        return;
    String buffer = DemoUtil.getFormattedDouble((bufferedDurationMs / Math.pow(10, 6)), 1);
    String brEstimate = DemoUtil.getFormattedDouble((bitrateEstimate / Math.pow(10, 3)), 1);
    updateStatChart(player_stats_health_chart, Float.parseFloat(buffer), ColorTemplate.getHoloBlue(), "Buffer Health: " + buffer + " s");
    updateStatChart(player_stats_speed_chart, Float.parseFloat(brEstimate), Color.LTGRAY, "Conn Speed: " + DemoUtil.humanReadableByteCount(
            bitrateEstimate, true, true) + "ps");
    player_stats_size.setText("Screen Dimensions: " + simpleExoPlayerView.getWidth() + " x " + simpleExoPlayerView.getHeight());
    player_stats_res.setText("Video Resolution: " + (null != currentVideoFormat ? (currentVideoFormat.width + " x " + currentVideoFormat.height) : "NA"));
    player_stats_dropframes.setText("Dropped Frames: " + droppedFrames);
    mUiUpdateHandler.sendEmptyMessageDelayed(MSG_UPDATE_STATS, 500);
}

protected void depictPlayerNWStats() {
    if (!canShowStats())
        return;
    updateStatChart(player_stats_nw_chart, (float) (bytesDownloaded / Math.pow(10, 3)), Color.CYAN, "Network Activity: " + DemoUtil.humanReadableByteCount(
            bytesDownloaded, true));
    bytesDownloaded = 0;
    mUiUpdateHandler.sendEmptyMessageDelayed(MSG_UPDATE_STATS_NW_ONLY, 1100);
}

 

Follow my next blog in series for Part 2 – Improved Buffering.