We’re going to build an app that can handle filtering and sorting large amounts of data with AgGrid.
Sending all of your data to the browser is a quick way to get started with AgGrid, but what if you have a million rows? That’s when server-side filtering and sorting come to the rescue.
This guide includes:
using AgGrid in infinite scroll mode so that the server is responsible for paginating, filtering, and sorting
implementing filtering, sorting, and pagination for AgGrid on top of a Django ListView class
creating a Django management command to populate test data
But first, here’s a quick peek at what we’re building today:
Previewing the project
Want to see a live version of the app? You can create a copy of this project now and try it out.
Create a fork of the AgGridDemo project on Circumeo.
Setting up the Django app
Install packages and create the Django application.
django-admin startproject bigdata .
python3 manage.py startapp core
Add core to the INSTALLED_APPS list.
INSTALLED_APPS = [
“core“,
…
]
Adding the templates
Create a directory named templates within the core app.
Create a file named index.html within the templates directory.
<html>
<head>
<script src=“https://unpkg.com/ag-grid/dist/ag-grid.min.noStyle.js”></script>
<link rel=“stylesheet” href=“https://unpkg.com/ag-grid/dist/styles/ag-grid.css”>
<link rel=“stylesheet” href=“https://unpkg.com/ag-grid/dist/styles/ag-theme-balham.css”>
<style>
#container {
max-width: 1250px;
width: 100%;
height: 100%;
margin: 0px auto;
margin-top: 30px;
}
</style>
</head>
<body>
<div id=“container”>
<div id=“data-grid” style=“height: 600px; width: 100%” class=“ag-theme-balham”></div>
</div>
<script type=“text/javascript” charset=“utf-8”>
const numberFilterParams = {
filterParams: {
filterOptions: [“equals“, “notEqual“, “lessThan“, “greaterThan“],
suppressAndOrCondition: true,
},
};
var columnDefs = [
{ headerName: “Name“, field: “name“, filter: “agTextColumnFilter“ },
{
headerName: “Description“,
field: “description“,
filter: “agTextColumnFilter“,
},
{ headerName: “Category“, field: “category“, filter: “agTextColumnFilter“ },
{
headerName: “Price“,
field: “price“,
filter: “agNumberColumnFilter“,
…numberFilterParams,
},
{
headerName: “Stock Quantity“,
field: “stock_quantity“,
filter: “agNumberColumnFilter“,
…numberFilterParams,
},
{
headerName: “Rating“,
field: “rating“,
filter: “agNumberColumnFilter“,
…numberFilterParams,
},
];
var gridOptions = {
columnDefs: columnDefs,
defaultColDef: {
filterParams: {
suppressAndOrCondition: true,
},
},
enableServerSideSorting: true,
enableServerSideFilter: true,
rowModelType: “infinite“,
cacheBlockSize: 100,
maxBlocksInCache: 10,
};
var dataSource = {
getRows: function (params) {
var filtering = encodeURIComponent(JSON.stringify(params.filterModel));
var sorting = encodeURIComponent(JSON.stringify(params.sortModel));
var startRow = params.startRow;
var endRow = params.endRow;
var url = `/products?startRow=${startRow}&endRow=${endRow}&filter=${filtering}&sort=${sorting}`;
fetch(url)
.then((response) => response.json())
.then((data) => {
params.successCallback(data.rows, data.totalRows);
})
.catch((err) => {
params.failCallback();
});
},
};
var gridDiv = document.querySelector(“#data-grid“);
new agGrid.Grid(gridDiv, gridOptions);
gridOptions.api.setDatasource(dataSource);
gridOptions.api.sizeColumnsToFit();
</script>
</body>
</html>
Adding the views
Remove the existing views.py file.
Create a new directory named views within the core folder.
Create a new file named index.py within the views directory.
Copy and paste the following into index.py within the views directory.
def index_view(request):
return render(request, “core/index.html“)
Create another file named products.py within the same directory.
Copy and paste the following into the products.py file.
from django.views.generic.list import ListView
from django.http import JsonResponse
from django.db.models import Q
from core.models import Product
class ProductListView(ListView):
model = Product
def get_queryset(self):
“””
Convert AgGrid filter and sort objects into a Django query.
An example filter:
{“category“: {“type“: “contains“, “filter“: “electronics“}}
“””
queryset = super().get_queryset()
filter_params = self.request.GET.get(“filter“, None)
if filter_params:
filters = json.loads(filter_params)
q_objects = Q()
for key, filter_info in filters.items():
filter_type = filter_info.get(“type“)
filter_value = filter_info.get(“filter“)
if filter_type == “contains“:
lookup = f“{key}__icontains“
q_objects &= Q(**{lookup: filter_value})
elif filter_type == “equals“:
lookup = f“{key}__exact“
q_objects &= Q(**{lookup: filter_value})
elif filter_type == “notEqual“:
lookup = f“{key}__exact“
q_objects &= ~Q(**{lookup: filter_value})
elif filter_type == “greaterThan“:
lookup = f“{key}__gt“
q_objects &= Q(**{lookup: filter_value})
elif filter_type == “lessThan“:
lookup = f“{key}__lt“
q_objects &= Q(**{lookup: filter_value})
queryset = queryset.filter(q_objects)
sort_params = self.request.GET.get(“sort“, None)
if sort_params:
sort_objects = json.loads(sort_params)
sort_fields = []
for sort_object in sort_objects:
col_id = sort_object[“colId“]
sort_order = sort_object[“sort“]
if sort_order == “asc“:
sort_fields.append(col_id)
elif sort_order == “desc“:
sort_fields.append(f“–{col_id}“)
if sort_fields:
queryset = queryset.order_by(*sort_fields)
return queryset
def get(self, request, *args, **kwargs):
start_row = int(request.GET.get(“startRow“, 0))
end_row = int(request.GET.get(“endRow“, 100))
queryset = self.get_queryset()
total_rows = queryset.count()
queryset = queryset[start_row:end_row]
products = list(
queryset.values(
“name“, “description“, “category“, “price“, “stock_quantity“, “rating“
)
)
return JsonResponse({“rows“: products, “totalRows“: total_rows})
Updating URLs
Create urls.py in the core directory.
from core.views.index import index_view
from core.views.products import ProductListView
urlpatterns = [
path(“”, index_view, name=“index“),
path(“products“, ProductListView.as_view(), name=“products“),
]
Update the existing urls.py within the project bigdata directory.
from django.urls import include, path
urlpatterns = [
path(“admin/“, admin.site.urls),
path(“”, include(“core.urls“)),
]
Adding the database models
Overwrite the existing models.py with the following:
class Product(models.Model):
name = models.CharField(max_length=255)
description = models.TextField(blank=True)
category = models.CharField(max_length=255)
price = models.DecimalField(max_digits=10, decimal_places=2)
stock_quantity = models.IntegerField()
rating = models.FloatField(blank=True, null=True)
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
Adding a Django Management Command for Test Data
Create the directory structure management/commands within the core folder.
Open a file named populate_products.py within the new directory and enter the following.
import random
from django.core.management.base import BaseCommand
from core.models import Product
class Command(BaseCommand):
def handle(self, *args, **kwargs):
faker = Faker()
categories = [
‘Electronics‘,
‘Books‘,
‘Clothing‘,
‘Home & Garden‘,
‘Toys & Games‘,
‘Sports & Outdoors‘,
‘Health & Beauty‘,
‘Automotive‘,
‘Groceries‘,
‘Pet Supplies‘
]
for _ in range(1000):
Product.objects.create(
name=faker.text(max_nb_chars=20).capitalize(),
description=faker.text(),
category=random.choice(categories),
price=round(random.uniform(5.0, 500.0), 2),
stock_quantity=random.randint(0, 100),
rating=round(random.uniform(1.0, 5.0), 1)
)
self.stdout.write(self.style.SUCCESS(‘Successfully populated the database with products.‘))
Open a shell session in order to run the Django management command.
Up and Running with Django and AgGrid
You’re now ready to handle millions of rows without crashing browser tabs! Add database indexes where needed, and you can scale this approach pretty far.