Parsing: the merit of strictly typed JSON

The programming language of the web is JavaScript (JS), but its lack of static typing is the top pain point raised from the 2023 State of JS survey. We rely on TypeScript (TS) to get a strong structural type analysis of our code, as mentioned in our article on standardisation. Static typing helps prevent runtime errors on our users’ machines. However, to ensure that no TypeError ever appears, we need to avoid the loose `any` type, which is an escape hatch designed to opt-out of TS’ static analysis.

JavaScript Object Notation (JSON) is widely used as a data exchange format for REST APIs. Its syntax is compatible with JS, with only a subset of the language’s primitives: objects, arrays, strings, numbers and nulls. Simple objects can be turned into a string with JSON.stringify and turned back into an object with JSON.parse, making it an adequate choice for many uses. We use it for powering our comments, new live blog updates, football match scores, weather, most read and other features of the website and Apps.

How things can go wrong

The TS compiler is able to analyse code based on the types description objects and functions. For code that you write, this will be done via annotations alongside your code, either using its own syntax or JSDocs. For the browser APIs, Microsoft provides the DOM declaration files which are aligned with versions of the ECMAScript specification. For example, calls to `JSON.parse` will return `any` in ES5. As mentioned above, this means these objects are opted out of static analysis. This means that we can no longer rely on the compiler for catching errors that could occur on users’ devices before we publish our code.

const object = JSON.parse(`{ "key": "value", "year": 1821, }`) // TypeScript is unable to catch that accessing // these keys will throw a TypeError: // can't access property "shape", "wrong" is undefined object.wrong.shape; 

Things can get even more confusing if you assign a type to the parsed object, as the TS compiler will use that value in the future, turning the ‘any’ into a specific shape. Such errors will be hard to debug, because they will only occur when your code is accessing the undefined properties, rather than when you receive a JSON response from the REST API. As errors will interrupt execution all the way up to the nearest try…catch statement, they can break large parts of the interactions for your users. These errors can occur because you’ve inadvertently declared the wrong shape, but they could also start appearing if the API changes its schema. If you have chosen TS for its ability to prevent runtime errors, these kinds of surprises are exactly what you were trying to avoid in the first place.

Check everything is as expected

“For some developers, it may be more informative to see a production implementation, so we implemented an example of this pattern in dotcom-rendering#11835”

This problem has been encountered by many developers, and the simplest way to force TS to warn of any possible issues is to explicitly assign the ‘unknown’ type to the object returned from JSON.parse. There is a proposal for this to become the default in TS and a custom declaration library to ensure that your code will gracefully handle any unexpected object shape.

Thanks to TS’s control flow analysis, you could check that each of the properties are as you expect them, using the ‘typeof’ operator recursively, and only accessing valid properties in nested conditional blocks. However, this can easily lead to repetitive boilerplate for objects with complex shapes.

async function getTemperature(): Promise<number> { const response = await fetch('https://api.nextgen.guardianapps.co.uk/weather.json') const data: unknown = await response.json(); // we must check that the data has a specific shape if (typeof data === 'object' && data != null && 'weather' in data) { if (typeof data.weather === 'object' && data.weather != null && 'temperature' in data.weather) { if (typeof data.weather.temperature === 'object' && data.weather.temperature != null && 'metric' in data.weather.temperature) { if (typeof data.weather.temperature.metric === 'number') { return data.weather.temperature.metric; } } } } throw TypeError('No valid temperature'); }

Thankfully, this problem has been encountered by many developers and there are a multitude of parsing libraries that bring more ergonomic APIs for these operations. In order to integrate with the TS compiler, they generally expose custom schemas that describe the expected shape of the data, which the unknown objects can then be validated against. Unlike TS, their work is done at runtime, on the user’s device, so the size of the library and its performance must be taken into consideration. We therefore picked Valibot, which has a similar API to the popular Zod library with a typically much smaller footprint. Comparing the declarative schema below with the imperative checks above shows a stark improvement in readability, and this distinction increases with the complexity of the data model.

import { parse, object, number } from 'valibot'; const schema = object({ weather: object({ temperature: object({ metric: number(), }), }), }); async function getTemperature(): Promise<number> { const response = await fetch('https://api.nextgen.guardianapps.co.uk/weather.json') const data = parse(schema, await response.json()); // the TS compiler is happy accessing this object // and the parsing library ensures that it actually is return data.weather.temperature.metric; } 

We’ve used this approach for our comments, which has enabled us to make changes with confidence, as we are now guaranteed that the data model will actually have the shape the TS compiler expects. This is especially helpful as the team implementing the web interface is distinct from the team providing the JSON API, which could evolve its data model over time. Now, a breaking change will be caught at the parsing step and dealt with there.

Failing gracefully

When the data does not match the schema, the ‘parse’ function will throw an error. While this ensures that no further errors will be thrown, it still stops code execution and may break other interactions on the website. In order to prevent exceptions and handle this case, we can treat errors as values by using the `safeParse` helpers, which return a tagged union that allows code to check for success or failure. To prevent relying too much on a specific library’s implementation, we wrap this in our custom ‘Result’ type, so this pattern becomes more familiar to developers as it can be used outside of the specific cases of parsing that could fail.

When making a network request to a REST API, there are several other types of errors that could occur: dropped or timed out network requests, a backend server error or an invalid JSON string. By using a consistent pattern for dealing with failures, we can display the most adequate messaging to our users. For example, we could offer to retry the operation, or give a plain language explanation of why the error occurred.