Acorel
Free demo

Getting the information you need: processing JSON with jq

Cees Paris, 22 March 2023

In this blog I will give a short demonstration of jq, a JSON processing tool that can help you find the information you are looking for. 

JSON 

JSON is a standardised format that uses human-readable text in order to store and transmit data-objects. It is known (and appreciated) for being relatively easy to read compared to other data formats like XML or CSV, and for being a common data format in electronic data interchange, such as communication from servers to web applications.

Sometimes, however, we might get more than we asked for. Especially when we are troubleshooting or investigating it can be insightful to only look at a subset of the JSON object that is received, or to quickly check whether a particular field or value is included in the JSON. For these situations, jq is an extremely useful tool. jq is a command-line JSON-processor which offers a quick and easy way to single out the information that is relevant to you from any JSON file. 

Use Case 

I would like to illustrate the functionality of jq with a use case I encountered at a client of Acorel. Because of the complexity of this company’s products, the JSON file that represents a single product and all its possible configurations can exceed 30.000 lines. Because of a bug that required some investigation, I was interested only in the name of those characteristics of the product where the fields ‘readOnly’, ‘visible’, ‘consistent’ and ‘complete’ were true and ‘required’ was false. Without jq, this is what a small portion of the ‘characteristics’ field of this gargantuan JSON object looks like in my terminal. 

json fragment zonder jq
Fragment of the raw JSON output

Finding all characteristics that fit to the key values to be investigated manually is virtually impossible. But with jq installed and a little bit of knowledge of its syntax, it becomes a breeze. 

The Basics of jq 

jq works as follows. It takes in as input any JSON file and allows the user to transform this data using jq commands. The simplest program in jq consists of a single dot, which takes any JSON input and returns it, save for some prettification, unchanged.

Here’s the program:

 
cat DEDSER119.json | jq '.' 

Let’s briefly reflect on how this program is structured to gain a better understanding of jq. We use the ‘cat’ command to output the file and the pipe symbol to run the jq command using the JSON file as input. As we shall see shortly, it is also possible to chain multiple jq commands using the same pipe symbol, where the output of the previous command is used as input for the next. Once the output is exactly structured as we want it to be, we can finally return the output as text to the terminal window or store it in a new file.

prettified JSON
Prettified JSON (fragment of output from above program)

You can access the value stored at a specific key in the JSON-file by typing a dot followed by the name of the key. For example, if we have a JSON object that contains a field that has a key named ‘foo’, we can return the value of this field by typing ‘.foo’ in jq. In the use case used as an example here, I had to do this twice: I was looking for the ‘characteristics’ field, which was encapsulated in the ‘rootItem’ field. 

Here’s the program:

 
cat DEDSER119.json | jq '.rootItem.characteristics'

Let’s see how far we are in achieving our goal. We have successfully narrowed down our output to consist only of an array of characteristics. But now we need to further filter these characteristics based on the specific keys and the specific values that were mentioned at the start of this blog. With the current output this is not possible, because the output is an array of characteristics, which does not in itself has keys and values, but only contains elements that contain these keys and values. Thankfully, we can ‘unpack’ the array using the array iterator of jq, by typing ‘[]’ after ‘.characteristics’. This results in an output that consists of separate JSON-objects, each representing a single characteristic. By using the ‘select’ command we can filter these JSON-objects based on whether or not they contain a particular key with a particular value. By chaining these ‘select’ statements using pipes, we get the exact output we were looking for. 

Here’s the program:

cat DEDSER119.json | jq '.rootItem.characteristics[]
| select(.required==false) | select(.readOnly==true) 
| select(.visible==true) | select(.consistent==true)
| select(.complete==true) | .id’
end result
End result (complete output of above program)

Conclusion 

And there we have it! We have gone from printing out a 30.000 line JSON object to the terminal, to only printing the names of the seven characteristics we were interested in. This example only scratches the surface of what is possible with jq, and there are many more helpful commands it contains, but hopefully some light has been shed on how jq can help us to get the information we need. You can download jq here.

Receive our weekly blog by email?
Subscribe here: