Android paging 3 refresh

Android Paging 3 library with page and limit parameters

Even though still in the Alpha version, many developers welcomed the new Paging 3 library from Android Jetpack considering the previous attempt was poorly designed and required a lot of coding.

Also, one of the main cons was the lack of support for combined network and local queries, which is actually a very common case in our apps.

Luckily, Paging 3 came to the rescue and as it looks like it performs better and it’s actually not THAT complicated to implement. However, there’s always a but!

Oh RemoteMediator, the great source combinator

The new library introduces the RemoteMediator class. It helps you combine local and remote queries to provide consistent data flow to the user, regardless if the network is available or not.

Basically, everything you fetch from a remote source should be saved to your local database so users can quickly access it later at app startup until the data is refreshed from the remote source. Of course, this will also mean that your app supports offline mode, so users can load everything that was previously in the database without an actual network connection.

Paging using item keys seems defaultish

The idea of how the RemoteMediator should work and combine local and remote data is by using item keys (or IDs), not page number parameters.

For example, you have 5 items in your local database, the last one has an id = abcd.
In this type of API pagination, to query the next page of items, the API requires a parameter afterId, which means it would return a limit of items after the specified id.

This query will return the next 5 items, after the item with id = abcd.

This is actually demoed in Google’s Architecture Components repository with an example of Reddit and it works quite nicely, but…

You need to create an extra model and database table to pair up item IDs with remote keys and then write DAO for it.

It makes sense for this type of API, but what about if your API is using simple page & limit parameters?

Paging using page & limit parameters

To my disappointment, it looks like there is no easier way to paginate using just page number and limit.

We tried to use AtomicInteger to increment page numbers between loads, but that was inconsistent. Also, the RemoteMediator reacts to how the user scrolls the list, so if you go back up, it will request the previous page, so you need to have smarter logic than just a plain integer variable.

Ultimately, we used the same solution as for pagination with item keys, only that our keys were not actually item IDs but the numbers of the next and previous page.

Models and database setup

Let’s create a model for the items we want to paginate. For example, let’s say we have a news feed, we can name the item NewsItem. This is your basic model that will correspond to your API specification, but it’s important to note the ID of the object.

For RemoteMediator to work, we will create another class that will actually hold the information about the previous and next page number for each item.

In the case where you have paging with item keys (afterId, beforeId), the prevKey and nextKey would be the ID’s of the previous and next NewsItem.

But here, to achieve pagination using page number, prevKey and nextKey will represent the previous and next page number of the current item.

If you’re loading in the batch of 20 items and the current page is 4, all these 20 items on page 4 will have a prevKey=3 and nextKey=5.

This will help RemoteMediator later to know which page to load next.

Create DAOs for your new models

Of course, to access the list of your items and show them to the user, you will need a DAO that will enable you to observe or to fetch them.

For your NewsItem, three DAO functions will be enough to achieve proper pagination.

Function insertNewsList() will enable you to save new pages of data, observeNewsPaginated() will create a PagingSource directly from the Room database and connect it with your PagingDataAdapter instance.

When refreshing data, you will use deleteNewsItems() to delete local copies and freshly save new batches fetched from the server.As for NewsRemoteKeyDao, you will also need three DAO functions.

We will need to save all of the NewsRemoteKeys objects after fetching a fresh batch of items from the server and assign them their proper prevKey and nextKey values.

The remoteKeysByNewsId() function will help the RemoteMediator to get a specific NewsRemoteKeys object so it knows which page to load next.

Finally, same as for NewsItem DAO, you will need to clear all of the NewsRemoteKeys when the user performs a refresh.

How to RemoteMediator

Here’s the hardest part. We have to load data from a remote source, save it to the database and somehow link the two into a one pageable flow.

Let’s break it down into several parts.
First, to create your RemoteMediator instance, you will create a new class which extends RemoteMediator by specifying some generic types.

By extending from RemoteMediator I’m specifying that the pagination key will be integer (which will represent page number), and that the items that will be paginated are eof type NewsItem.

Notice that I’m passing a reference to the database instance and my API interface. We will need this in a minute.

Now, all the magic is created in the load() function which should return some MediatorResult.

MediatorResult can be Success or Error, depending on if we successfully fetched and saved all the items and their corresponding RemoteKeys items or not.

Which page to load

First, inside the load() method, you need to calculate which page should be loaded.

If the loadType is REFRESH, then we need to check on which scroll position we are currently, which item it is and find its corresponding NewsRemoteKeys object so we can see which page is a current page.

Using the function getRemoteKeyClosestToCurrentPosition() we can get the NewsRemoteKeys which points to the current item at this scroll position, so the value of (nextKey – 1) will actually be current page that we need to refresh.

If there is no corresponding NewsRemoteKeys object, that means that we don’t have anything in the database yet and we are starting from the beginning (initialPage).

If the loadType is APPEND, then we are gonna look for the last item in the list and see what it’s NewsRemoteKeys data specifies as the next page.

Using the function getRemoteKeyForLastItem() we can get the last NewsRemoteKeys and by so, we can get the appropriate next page number from its nextKey value.

If there is no such object, that means that we reached the end of the pagination and in this case we will return MediatorResult.Success with endOfPaginationReached = true .

Prepending the data is similar to the APPEND, it just goes in a different direction. For example, if your initialPage is 10 (let’s say you’re starting from the middle of your items list), then you can paginate towards the first page.

The function getRemoteKeyForFirstItem() would be similar to getRemotKeyForLastItem(), except you take the state.firstItemOrNull() value, instead of state.lastItemOrNull() value.

Here in this case, where we don’t have pagination in two directions, we just return for PREPEND type result Success with endOfPaginationReached = true in this direction.

Load remote data and update database

Now when we know which page to load, we can simply create a new network request and fetch the data we need.

Here you can see that we are using pageSize value from the pagination config object and the page number we previously calculated.

To see if we reached the end, for now we just compare if the number of items in the result is lower than the requested number, which would mean there are no more items on the server side. But this logic can be custom and you can do it your own way (if you have metadata in your API response which you can use for this, do it).

Important thing here is to have the same sort locally (in your DAO) as it is on the remote source, otherwise the RemoteKeys will not be ordered properly and your next & previous page numbers might be wrong.

Now, finally let’s save the data into the database so it can be rendered to the user.

At this point, it’s important to do these operations in a transaction so both items and their RemoteKeys objects are saved (or cancelled) together.
First, you need to check if the loadType was REFRESH. If it was, then delete the items and their RemoteKeys so we store only the fresh new ones received from the server.

Here we also create new corresponding NewsRemoteKeys for each item, assigning their corresponding nextKey and prevKey values, which are actually our page numbers.

Finally we can store all of this into the database and voila, this is your RemoteMediator.

You can find the entire NewsPageKeyedRemoteMediator implementation here.

Show me the data

Now we can finally make use of RemoteMediator to connect our data with PagingDataAdapter and magically achieve pagination.

If you’re using a Repository Pattern, then I suggest you add this block of code into your repository class, but if not, you can add it directly into your ViewModel (given that you have access to database and API interface).

Create a new Pager object and pass some configuration parameters, like pagination batch size.

Also, if you’re experiencing glitches or jumps in position when scrolling & loading new pages, you can enable placeholders so it keeps the list steady.

For the remoteMediator parameter, pass the newly created NewsPageKeyedRemoteMediator which will do all the pagination magic, fetch the remote data and update the local source.

Finally, the pagingSourceFactory lambda will be a direct link to your database table.So as soon as RemoteMediator saves new data, this observer will be updated and the adapter will update as well. We will pass the observeNewsPaginated() function we previously created in our NewsDao as an observable source of data.

Finally, this Pager object can provide a Flow object, or you can adapt it as a LiveData object.
This is what you observe in your fragment / activity.

Now, for each new update in the database, a newsPagingAdapter will be updated using the proprietary submitData(pagingData) method.

Pager provides an PagingData object which contains information not only about the items in the list but also it contains metadata about scroll position and direction, so it can notify RemoteMediator if additional items need to be loaded.

We will not get into details on how to create a PagingDataAdapter as it is similar to a regular RecyclerView adapter, it just enforces you to create a DiffUtil which it will use to compare changes in the list of items.

Scroll, Forrest, Scroll

There, now you’ve got yourself a pagination which combines remote and local data to provide your users seamless paging experience.

All the data that was stored in the database can later be accessed even without network and users can manually refresh the list if and when they want.

We would expect to have a simpler way of setting up pagination with page numbers and batch size, but after a couple of tries and combinations, the only working solution was using RemoteKeys pattern to store next and previous page numbers.

Maybe Paging 4 library will make it even simpler 😀

I hope you found this article helpful. Let us know if you have any questions about this, if you already tried a similar solution and how did you solve the issue.

We’re curious to hear about your experiences!

_____
We’re available for partnerships and open for new projects.
If you have an idea you’d like to discuss, share it with our team!

Subscribe to our newsletter

Subscribe now and get our top news once a month.

100% development, design & strategy.

You’re now a part of our club!

We got your e-mail address and you’ll get our next newsletter!

Источник

Paging3 — Doing Recyclerview Pagination the Right Way

Jetpack Paging library release version 3.0.0-alpha03 is full of new features and is Kotlin first with a pinch of Coroutines & Flow 😊.

Intro:

Paging3 is one of the new Jetpack libraries for managing and loading a large chunk of the dataset from various data sources efficiently. It allows us to load pages of data from the network or local database with ease and saves our development time. Paging3 is designed to follow the Android app architecture and coordinates with other Jetpack components. It is Kotlin first and works with some of the new async threading solutions like Coroutines and Flow and of course, it has the support for RxJava & LiveData users as well.

How it’s different:

• It caches the paged data in-memory for better usage of the system resources which not only gives fast response but also helps in data loading without hiccups.
• It handles the network request duplication very elegantly hence saving the user’s bandwidth and system resources.
• A much flexible Recyclerview Adapter which requests & loads the data gracefully whenever the user reaches near the end of the page, yes now adapter is controlling when and what to load with a one-time setup.
• It is Kotlin first means the whole library is written in Kotlin and works very well with other offerings of the Kotlin like Coroutines and Flow. Also, it supports the much-used libraries like RxJava and LiveData as well.
• It has inbuilt support for error handling, retry and refresh use cases.

Paging3 & Application Architecture:

When I say Paging3 works and integrates well with our application architecture is means that it sits in all the basic layers of our application like Repository -> ViewModel -> UI and makes it very easy to understand and include in our existing flow. Have a look below to understand it visually.

PagingSource: It is a generic abstract class that is responsible for loading the paging data from the network. To implement PagingSource we need to define the Page Key type in our case it will be of type Int and the response data type from API in our case it will be DoggoImageModel.

RemoteMediator: It is responsible for loading the paging data from the network and local DB. This is a good way to implement paging since in this case, our local DB is the main source of data for the paging adapter. This method is much more reliable and less error-prone.

Pager: This API consumes whatever the RemoteMediator or PagingSource returns as a data source to it and returns a reactive stream of PagingData. It can be returned as a Flow, Observable, LiveData as shown in the above diagram.

PagingData: This is the final return type and something that PagingDataAdapter understands and has the original data type inside it. It acts as a paging data container.

PagingConfig: This is our paging configuration class here you can define how the PagingSource should be constructed means you can define how much data should be there on each page and many more options are there to customize our PagingSource.

PagingDataAdapter: This is the primary UI component that is responsible for presenting the data in the RecyclerView. It consumes the PagingData as the input type and listens to its internal loading events. It loads data after fine graining using DiffUtil on a background thread, so expect no hiccups while adding new items on the UI thread.

So what’s the plan?

We will be creating a small Doggo android application to exploit 😄 the new Paging3 library different use cases and see how different implementations of data source works with newly introduced Coroutines and Flow as well as RxJava and LiveData. We will try loading data from local Room DB as well as hot network calls. I know we have covered a lot of heavy definitions and terms but as we go ahead and implement them it will start to make sense and looks easy once we implement them so bear with me a bit longer 🙇.

Before we start:

We need to add this compulsory dependency of Paging3 available in google maven.

I have added those above along with few other dependencies like Retrofit, Coil, Navigation, Lifecycle LiveData, ViewModel have a look here at the full dependencies snapshot. Ok now dependencies are in place let’s start with the implementation of the Paging3

Network as a Data Source 🌐:

In our 🐶 application, we will be fetching the list of all the cute dogs from the remote network using the TheDogApi API service. It seems my love for dogs is now showing into my medium articles as well 🤷‍♂️, you can create an application for cats 😻 if you want. Let’s define our API service endpoint as we do use a retrofit.

Please note the important thing here, the page & limit which is important for our Pagination /Endless scrolling /Lazy Loading. The param page is here to keep track of the current page for which we will be requesting new data similarly the limit defines how much data we need per page. These keys might be different for your API so add them accordingly to the definition.

We have completed the first part of our API call now let’s see how Paging3 helps us automate this process of pagination. PagingSource is the way to go here since we are going to use remote network API as the data source. so let’s create a class DoggoImagePagingSource and implement PagingSource like below. Here I have passed the DoggoApiService which we have created earlier so that our PagingSource can call our doggo API and get results. Apart from this while inheriting from PagingSource we need to define the type of paging our API supports in our case it is simple Int based number paging and the return type of the API response that is DoggoImageModel. Now we have implemented the PagingSource let’s dig and get familiar with the load function.

— params: Keeps the basic information related to the current page for which API needs to be called and the page size.

— LoadResult: It’s a Kotlin Sealed class and we can return LoadResult.Error(exception) in case of exception or some error and in case of success we will return LoadResult.Page from load() function. If for some reason data is not available and might have reached the end of the list then pass null as the value for prevKey or nextKey to indicate the end of the list condition .

Inside the load() function we call the doggo API service to get the data from the network passing the current page and page loadSize. The current page can be retrieved from params.key which is null on start so we will assign a default value in that case. Click to see full DoggoImagePagingSource class implementation.

We have just completed our first phase according to the app architecture diagram defined above. Now we need to configure and return PagingData with the help of Pager class. To do it let’s create a DoggoImagesRepository class. Inside this class let’s define a function that returns the reactive stream of PagingData using Pager class.

now to construct Pager we need PagingConfig first. This allows you to change various configurations related to the paging like page size, enable placeholders, etc. Here we are just returning a very basic PagingConfig instance.

Now the second thing that Pager needs is the PagingSource which we have created earlier i.e DoggoImagePagingSource. Let’s assemble these two to return a reactive PagingData like below. Click to see full DoggoImagesRepository.kt

Now, this almost completes the second phase we just need to call this from ViewModel class where we can modify the returned PagingData or perform some collection actions like mapping or filtering if needed and many more Flow related operations can be done which is a quite interesting addition. If you are working with RxJava then just call .observable or .liveData on Pager() if you are working with LiveData.

Let’s create RemoteViewModel class and expose the data from the repository to the UI components by adding the following methods in our ViewModel. See full RemoteViewModel here.

for showing the mapping operation on Flow we have here tweaked the data type from Flow

> and mapped it to Flow

> you can always return whatever you want.

Note: If you’re doing any operations on the Flow , like map or filter , make sure you call cachedIn after you execute these operations to ensure you don’t need to trigger them again.

Let’s move to our final stage of implementing paging with the network as the data source. We are going to club the data returned from RemoteViewModel functions to our customized PagingDataAdapter in the fragment. We have created a basic RecyclerviewAdapter extending the PagingDataAdapter have a look here 👇. The only difference you might have noticed here is that in this new adapter we are passing implementation of the DiffUtil to the PagingDataAdapter constructor, I guess the rest is quite understandable and is regular Recyclerview adapter implementation.

let’s see how we can pass the data to this adapter from UI. Just call this function from your activity or fragment life cycle methods like onCreate() or onViewCreated() to collect the created flow in the view model. You can subscribe to the Rxjava Observables or Observe the LiveData if you are returning the same from view model class functions. See here the full implementation of our RemoteFragment class.

That’s it if you run this setup now it will produce the following output 👏.

I hope some of you are still with me to see this 😄. Now, this completes our basic Paging3 implementation. Next, we will see how to add a progress loader to this adapter for automatically handling the error cases and will add a try button to fetch again.

PagingDataAdapter with loading states:

This enables our adapter to have the additional capability of handling the error cases automatically and loads the right state accordingly.

To leverage this feature Paging3 comes with additional LoadStateAdapter just create an adapter and extend to it. This is again no different than our regular Recyclerview adapter the only difference is that it gives LoadState as the data type then our regular data model. This is quite helpful in knowing in which state is the paging adapter right now. Here is our implementation of the LoadStateAdapter.

we can use this LoadState sealed class to react accordingly. It returns three states Loading, Error, NotLoading. We are hiding the retry button in case the adapter returns LoadState as Loading and showing a progress bar. Now let’s see how we can use this newly created adapter with our existing created RemoteDoggoImageAdapter.

noting much here, we have just created an instance passing a higher-order function in the constructor which calls adapter.retry() function on RemoteDoggoImageAdapter and second thing we did is we called withLoadStateFooter(loaderStateAdapter) function on the RemoteDoggoImageAdapter passing our newly created loaderStateAdapter. Let’s run to see this in action.

Room as a data source:

We can use local DB for pagination as well and it’s good to provide offline support as well. If you do offline content support and don’t know how to leverage local DB for your paging use case then this feature is made for you 😎. In this case, the local Db will be the only source of data and whenever needed it will get new data from the network. So it handles both the cases where we need to fetch the new data from the network and save it to the local DB and UI will observe this to reflect new changes.

To support this type of paging first we need to create some local DB. Here we are using Room as our local DB. Let’s create some entities quickly to save our API response model as well as paging related info Entity.
We have converted our existing data class DoggoImageModel to a Room Entity.

Our model is quite simple since we are just taking a URL to show images from API. Next, we need some Dao as well for this newly created Entity. So do something like below.

Next, create an Entity to store the paging information for offline paging which Paging3 needs while making paging assumptions.

Dao for RemoteKeys

Now we are ready with our Entities and Dao’s let’s assemble them in one place and create an AppDatabase class to provide the room database object for use.

This is the same thing we do while creating regular Room DB nothing special but here comes the complicated part of this article 😬. Let me now introduce you guys to RemoteMediator this class is responsible for getting the results from the Db for pagination and whenever needed it gets the fresh data from the network as well and saves it to the local DB. This manages both the network and local DB and coordinates with both of them to perform the pagination.

We gonna create a class DoggoMediator to implement the RemoteMediator.

here we are passing DoggoApiService and AppDatabase for performing network and DB related operations. RemoteMediator is quite the same as PagingSource where we defined the page type as Int and passed DoggoImageModel as the data model the same thing we need to do here as well. Let’s dig and understand the load() function here:

• MediatorResult: As we can see it’s a return type of this function and it can be MediatorResult.Success for success case and MediatorResult.Error for error cases.
• LoadType: This tells us where we need to append the data in the page. It can be of the following types.
  — LoadType.APPEND: Means we need to load the new data at the end of the page.
  — LoadType.PREPEND: Means we need to load the data at the beginning of the previously loaded data.
  — LoadType.REFRESH: Means this is the first time we are loading data for pagination.
• PagingState: It holds the information related to the recently accessed index in the list and some information related to the pages which have been loaded earlier. This gives information about the paging configuration which we add while returning Pager.

We need to do the following to complete the load() function for the mediator.

1. Find out what page we need to load from the network, based on the LoadType .
2. Trigger the network request.
3. Once the network request completes, if the received list of repositories is not empty, then do the following:

• We compute the RemoteKeys for every DoggoImageModel .
• If this a new query ( loadType = REFRESH ) then we clear the database.
• Save the RemoteKeys and DoggoImageModel in the database.
• Return MediatorResult.Success(endOfPaginationReached = false) .
• If the list of DoggoImageModel was empty then we return MediatorResult.Success(endOfPaginationReached = true) . If we get an error requesting data we return MediatorResult.Error .

Let’s distribute the work into functions for more clarity on what is going on.

Create a function called getFirstRemoteKey() which returns RemoteKeys for the loadType= LoadType.PREPEND. It basically gets the first page from PagingState and queries the database with the id of the DoggoImageModel.

now let’s create function getLastRemoteKey() for the loadType=LoadType. APPEND and return the RemoteKeys as below. This queries the last page from PagingState and queries the database for RemoteKeys.

create last function getClosestRemoteKey() for the loadType=LoadType. REFRESH in case of first time data loading or we have called PagingDataAdapter.refresh() from UI.

The point of reference for loading our data is the state.anchorPosition . If this is the first load, then the anchorPosition is null . When PagingDataAdapter.refresh() is called, the anchorPosition is the first visible position in the displayed list. So the above function calls state.closestItemToPosition() to get the closest DoggoImageModel.

Let’s club these functions into one function and return a result based on LoadType.

The first point for loading page type is now done now fill the rest of the load() function as per given points which is easy to follow. Here is a full implementation of the load() function after fill up.

This completes the DoggoMediator. Now let’s see how we can call this mediator from the repository to get the reactive PagingData. Add the following function in the DoggoImagesRepository class.

again you can return Observable and LiveData which is the same as we have done for PagingSource implementation. Calling this from ViewModel is the same nothing changed except the function name.

If we tie this up with our UI PagingDataAdapter then it will produce some output like below.

There are more cool features in the Paging3 Jetpack library and is under active development. Here is the full repository for this article feel free to fork or pull to see the implementation in details.

Источник