We are publishing a translation of the article, which describes in detail the long-term work of the team to create and maintain a large data portal in JavaScript.
In 2019, an article was written about Maintaining large JavaScript applications. In continuation of this material, we would like to share a client project that my team has been supporting since 2014.
Data portal for the Organization for Economic Co-operation and Development (OECD)
Portal homepage
The Organization for Economic Co-operation and Development is an intergovernmental body that collects data and publishes research on behalf of its member states. The organization's portal contains information from various fields: economics, ecology, education, etc.
The OECD data portal is the main repository of statistical data. It helps researchers, journalists, and policymakers find important information and quickly visualize it using diagrams. The OECD portal also integrates a library with OECD iLibrary publications , and an OECD.Stat resource where all data is stored.
The OECD is funded by member states, that is, taxpayers like you and me. One of the project's requirements is to use cost-effective and reliable technologies, as maintaining the code is important for a long time.
The data portal is a collaboration between OECD staff and external developers and designers. The initial design and prototype was created by Moritz Stefaner and the Raureif team . And 9elements developed the front-end part and still maintains it.
Complex JavaScript codebase
The hardest part of the frontend on this portal is the JavaScript charting engine. It contains ten main types of charts with numerous configuration options. Using powerful interfaces, users can query the database and create diagrams for embedding or sharing.
We started work on the data portal in 2014. Since then, it has not been rewritten much, only new features have been added, small improvements and refactorings of the code. In December 2020, we added several new features for the OECD Economic Outlook report , including four more chart types. We also reorganized the codebase significantly this time around.
In this article, I'll show you how we manage to maintain the code for so long and improve it step by step. I will also reveal what did not work out.
Boring mainstream technology
The project started in 2014 and that's when we chose plain HTML, XSLT templates, Sass for table styles, and CoffeeScript as a language that compiles with JavaScript. We chose jQuery, D3, D3.chart and Backbone as JavaScript libraries.
In 2014, these technologies were the safest and most interoperable, of all, choosing CoffeeScript was a risky gamble. Thanks to CoffeeScript, we have been able to become more productive and have written reliable code. But we knew that this new technology could be difficult.
Since 2015 the 9elements team started using React for most JavaScript web applications. We thought about using React for a new version of the charting engine, but everyone couldn't find the right moment. As a result, it turned out that sticking to the original technology stack was the right decision.
The JavaScript stack just described may seem outdated, but in fact, the codebase has stood the test of time. One reason: the technologies we have chosen are still relevant.
The destructive influence of time
Many JavaScript libraries have come and gone, but jQuery is still the most popular. It is reliable, well supported, and widespread. According to Web Almanac 2020 , jQuery is used by 83% of all websites. (In comparison, React was only found at 4%.)
Of course, jQuery has lost its leadership position in tackling complex DOM problems. As mentioned, right now we would choose React or Preact to create such a data portal.
Second library, D3remains the standard for data visualization in the browser. It has existed since 2010 and is still the leader. Although a couple of major releases have significantly changed the structure and API, D3 is still an outstanding piece of engineering.
The Backbone library is not that popular, but it has its advantages. For example, it's relatively simple: you can read the source code in one morning and redo the main parts in a day. Plus, Backbone is still supported. It is fully functional, which is especially important.
From a technological point of view, only CoffeeScript is not a relevant technology in the current realities. This language was developed due to obvious flaws in ECMAScript 5. Later, many ideas from CoffeeScript were incorporated into the ECMAScript 6 (2015) and ECMAScript 7 (2016) standards. From now on, we have no reason to use it.
We chose CoffeeScript in 2014 because of its "It's just JavaScript" philosophy. Unlike other languages compiled with JavaScript, CoffeeScript is a straightforward abstraction. CoffeeScript compiles to pure JavaScript without surprises.
Most companies today have migrated their codebases from CoffeeScript to modern JavaScript. And we did the same.
From CoffeeScript to TypeScript
With this decaffeinating tool , we converted the CoffeeScript code to ECMAScript 6 (2015). We wanted to continue to support the same browsers, so we are now using the Babel compiler to create backward-compatible ECMAScript 5.
Overall, the transition went smoothly, but we didn't want to stop there.
In new projects, 9elements developers use TypeScript. In my opinion, TypeScript is the best thing that has happened in the JavaScript world in the last couple of years. As I mentioned in my previous article, TypeScript makes you think about types and teaches you to name them correctly.
For our data portal, we were going to take advantage of TypeScript without converting the codebase to fully typed TypeScript.
TypeScript is a superset of JavaScript. The compiler understands .js files well. Therefore, we gradually added type annotations using technology from 20 years ago - JSDOC . In addition to this, several types (typings) were written in the .ts files to reference them in the JSDOC annotations.
Thus, the development experience in Visual Studio Code has grown significantly without much effort. While there is no strict type checking here, editing the code is just as convenient as in a regular TypeScript project.
By combining rather boring but robust technology with the latest TypeScript compiler, we were able to add new features and refactor code safely and easily.
Documentation and code comments
On the surface, coding is a conversation between you and the computer: you tell the computer what it needs to do.
But, in reality, coding is a conversation between you and the reader of the code. It is well known that code is written once and read over and over again. First of all, you write code for your future self.
We have added many comments to the portal codebase, and almost all of them have managed to prove their worth over the past six years. Obviously, the code should be structured in such a way as to help readers understand it. But I don't believe in "self-describing" or "self-documenting" code.
Prior to moving to JSDOC, we made easy-to-read type annotations, documented function parameters and return values. We have also documented basic data types and complex nested object structures.
These comments were really helpful six years later. We translated them into machine-readable JSDOC and type declarations for the TypeScript compiler.
Things break - always have a test suite handy
The project has only a few automated unit tests and more than 50 (!) Test pages that demonstrate all portal pages, components, data query interfaces, chart types and settings. They test both live, staging, and mock data.
These test pages do the same thing as automated tests: if we fix a bug, we first add the script to the corresponding test page. If we are developing a new feature, then at the same time we create a complete test page.
Test page
Before release, we manually check all test pages and compare them to the last one both visually and functionally. This is time consuming, but allows you to find regressions quickly.
I don't think an automated test suite would be more efficient. It is nearly impossible to automatically test interactive data visualizations in a browser. Visual regression testing is a valuable tool, but in our case it can give too many false errors.
Backward and forward compatibility
In 2014, our portal was supposed to work with Internet Explorer 9. Now Internet Explorer is not so important, especially when creating a dynamic engine for plotting in the browser.
However, we decided to keep compatibility with older browsers. The data portal is an international platform where users from all over the world visit. Not everyone has the latest computers and new browsers.
Portal in Internet Explorer 9
We were able to maintain a basic level of browser support by using boring standard technologies. Of course, there are also a few modern web features, but we only activate them if the browser supports it.This is where the Progressive Enhancement approach helps us. (progressive improvement). We also use Babel and polyfills to make modern JavaScript functions work in older browsers.
Your abstractions might bite
Over the years, we have not been limited by the technology stack. Rather, their own abstractions stood in the way.
We split the user interface into visual parts (views) and created a base class similar to Backbone.View. (Today, all large JavaScript libraries use the term "component" instead of "view" for parts of the user interface.) We used Backbone.Model to store data and state. It worked great, but we decided to stick with our own best practices.
The idea behind splitting the Backbone model-view is that this model is the only source of truth. The DOM should just reflect the model data. All changes must also come from the model. Modern frameworks such as React, Vue, and Angular follow the convention that the UI is a "state function," meaning the UI is definitely state derived.
We violated this principle and sometimes made the DOM the source of truth. This led to confusion with the code that viewed the model as an authoritative source.
Object oriented charts
For charts, we took a different approach: we created chart classes that are different from those described above.
The D3 itself is functional. A chart is usually created and updated with a render function that calls other functions. These diagrams are introductory to this great feature. More state is contained in specific objects.
This makes D3 expressive and flexible. But the D3 code is hard to read because there are little conventions (conventions that are not documented) for dealing with chart structures.
Irene Ros and Mike Pennisi, developers at Bocoup, invented d3.chart, a small library on top of D3 that represents class-based OOP. Its main goal was to structure and reuse charting code. These diagrams are made up of layers, each of which renders and updates a specific part of the DOM using D3. In addition, other diagrams can be attached to charts.
The general rule of OOP is "Composition over inheritance." Unfortunately, we chose a strange combination of composition and inheritance for the diagram's behavior.
We needed to use functions or simple classes instead of complex class hierarchies. People are still wrapping D3 in class-based OOP, but no class-based solution has been able to outperform the functional structure of D3.
Let's sum up
Since we developed the front-end portion of the data portal in 2014, there have been many cool approaches to building JavaScript-based web interfaces.
Now UI components are declarative, you can do without rendering HTML templates and updating the DOM manually. You just update the state and the framework updates the DOM. This unidirectional data flow eliminates a whole class of bugs.
The technologies we chose in 2014 either stood the test of time or made it easy to switch to a different stack. You might think that we were lucky, but we deliberately made a choice in favor of durable technologies.
At 9elements we always try to use modern technologies, including evaluating experimental front-end technologies that may not be relevant in 3-4 years. Unfortunately, many open source JavaScript projects can be technically progressive, but unstable.
For each project, we are looking for the optimal balance between sustainable technologies with minimal risks and an innovative stack, which helps us create a quality product.