Big Data with DukeScript

At the JCrete conference last week we had some DukeScript hacking sessions, and we were was asked for best practices when you have large data sets to display in a grid. One solution is paging. If you want to load all the data at once, here’s another solution.

Let’s assume we have a model like this:

@Model(className = "BigData", targetId = "", properties = {
    @Property(name = "values", type = Row.class, array = true)
})
final class DataModel {

    @Model(className = "Row", properties = {
        @Property(name = "firstName", type = String.class)
    })
    public static class RowVMD {
    }
}

The first thing you can do in order to improve performance is to check if your table data needs to be mutable. Very often it’s an easy win to add a simple attribute to your tabular data:

@Model(className = "BigData", targetId = "", properties = {
    @Property(name = "values", type = Row.class, array = true)
})
final class DataModel {

    @Model(className = "Row", properties = {
        @Property(name = "firstName", type = String.class, mutable=false)
    })
    public static class RowVMD {
    }
}

This “mutable=false” will create a plain value instead of an Observable and thus save memory and construction time. It makes sense even if you want to edit the data in the table. (You would then create also a mutable version of “Row”, copy the data of the selected element to this “EditableRow”, edit it, and replace the old Row in the BigData ViewModel with an updated new one).

But the main problem with displaying large datasets is a slow UI. If you want to, for example, display a table with 100.000 rows, your DOM will get really big and this will kill performance.

One way to deal with that is to “virtualize” your table. That means the table will behave as if it actually has 100.000 rows, while it only displays the 20 rows that are currently visible. The best way to model that in DukeScript is to register a custom component like this:

ko.components.register('big-table', {
    viewModel: function (params) {
        var self = this;
        self.originalValues = params.value;
        self.firstIndex = ko.observable(0);
        self.numVisible = ko.observable(0);
        this.visibleRows = ko.computed(function () {
            var visibleValues = self.originalValues().slice(self.firstIndex(), self.firstIndex() + self.numVisible() + 1);
            return visibleValues;
        });
    },
    template: {element: 'big-table-template'}
});

A component consists of a ViewModel and a template. Our ViewModel has only four properties, the original values of our BigData object, the index of the first visible row, and the number of visible rows. The fourth is a computed property derived from the other three. It contains the values of the rows that are currently visible.

The template is loaded from an element in the page. This way we can modify the layout when we add more properties, or if we e.g. want to switch from a table to a list. Here’s a simple example:

<script id="big-table-template">
    <div
        data-bind = 'scroll: $data'
        style='height: 100px; overflow-y: scroll;
        overflow-x: auto;
        position: relative;
        background: grey;'>
        <div class = 'scroll-dummy' style = 'height: 110px;' > < /div>
            <div class = 'big-table-table' data-bind = 'foreach: visibleRows' style = 'position: absolute; top: 0; left: 0; background: red;' >
                <div class = 'big-table-row' style = 'height: 50px;' > <span data-bind = 'text: firstName' ></span></div >
            </div>
            </script>

There’s a div that wraps everything. We can set a fixed height on it. In our case it’s 100px. It contains another div that acts as a “scroll-dummy”. The only purpose of this is to make the scrollbar behave as if we have lots of elements in our table. We’ll resize it later to match the height of all rows. On top of it there’s an element that wraps the rows. It can be a list, a table, or in this case, a simple div. It uses the foreach binding to display the visible rows.

Now we need to make this dynamic. You can use a custom binding for this. It calculates how many elements are visible and listens to scroll events in order to update the firstIndex property of our components ViewModel. And it moves our “big-table-table” to the view port of our virtualized control.

ko.bindingHandlers.scroll = {
    init: function (element, valueAccessor, allBindings, viewModel, bindingContext) {
        var self = this;
        self.element = element;
        self.viewModel = viewModel;
        var scrollDummy = self.element.getElementsByClassName('scroll-dummy')[0];
        self.table = self.element.getElementsByClassName('big-table-table')[0];
        var maxPos = self.element.offsetHeight;
        var els = self.element.getElementsByClassName('big-table-row');
        var rowHeight = els[0].offsetHeight;
        self.viewModel.numVisible(maxPos / rowHeight);
        scrollDummy.style.height = rowHeight * self.viewModel.originalValues().length + "px";
        self.firstIndex = viewModel.firstIndex();
        self.viewModel.firstIndex.subscribe(function (newValue) {
            var scrollTop = self.element.scrollTop;
            if (self.firstIndex > newValue) {
                scrollTop -= rowHeight;
            }
            self.table.style.top = (scrollTop < 0 ? 0 : scrollTop) + "px";
            self.firstIndex = newValue;
        });
        self.element.addEventListener('scroll', function (e) {
            var scrollTop = self.element.scrollTop;
            self.viewModel.firstIndex(Math.floor(scrollTop / rowHeight));
        });
    }
};

As a result, you will only have DOM nodes for the nodes that are currently visible and a nice and responsive UI.

In order to make this work, you will need to register the custom binding and custom component. I’ve put them together in a file called “bigtable.js” and register them like this:

@JavaScriptResource("bigtable.js")
public final class BigTableBinding {
    private BigTableBinding() {
    }
    
    @JavaScriptBody(args = {}, body = "")
    public static native void init();
    
}

Now you only need to make sure you call the init method before you bind the data:

    static void onPageLoad() throws Exception {
        ui = new BigData();
        Models.toRaw(ui);
        BigTableBinding.init();
        for (int i = 0; i < 1000; i++) {
            ui.getValues().add(new Row("index " + i));
        }
        ui.applyBindings();
    }

Enjoy coding DukeScript!