An Introduction to JQ

by


18 minute read

    Up thus far:

   
Adam Gordon Bell

Background: Fingers, Head, and Google

At any time after I attain a stopping level in my work, I spend a bash alias called gwip1 to construct a ‘work in development’ commit. It occurs with out wakeful pondering on my share. The same ability my fingers know the vim key bindings, they know gwip.

Varied actions, I perceive how they work, but I in actuality must take into chronicle them at any time when. They are in my head, not my fingers.2

On the different hand, some things never stick in my head, nor my fingers, and I in actuality must google them at any time when. jq is one among these.

I perceive it’s a extremely effective instrument, but I always find yourself attend at Google after which copying and pasting a resolution from someplace. So I resolve my say of affairs but never learn the instrument.

It’s time to fix that. On this article, I’m going to recede over the basics building blocks of jq in sufficient depth that you just are going to be in a score 22 situation to know how jq works. Keep in mind that, you serene also can each now and then must head to google to salvage a operate name or test your syntax, but as a minimal you’ll luxuriate in a firm grounding in the basics.

What Is JQ

jq is a lightweight, suppose-line JSON processor. I set up it with brew (brew set up jq), but it completely’s a single transportable executable, so it’s simple to set up on Linux, Home windows, or macOS. To spend it, you construct various filters, and it applies those filters to a JSON myth.

The finest filter is the identification filter which returns all its input (.):

$ echo '{"key1":{"key2":"value1"}}' | jq '.'
{
  "key1":  {
    "key2":  "value1"
  }
}

This filter is to hand for correct reasonably-printing a JSON myth.3 I’m going to ignore the reasonably-printing and leap just into the usage of jq to transform JSON paperwork.

The usage of JQ to Determine Elements

I’m going to make spend of jq to filter the records returned by the GitHub repository API. The records I score attend by default looks to be like relish this:

$ curl https://api.github.com/repos/stedolan/jq
{
  "identification":  5101141,
  "node_id":  "MDEwOlJlcG9zaXRvcnk1MTAxMTQx",
  "name":  "jq",
  "full_name":  "stedolan/jq",
  "deepest":  wrong,
  "owner":  {
    "login":  "stedolan",
    "identification":  79765
  },
  "html_url":  "https://github.com/stedolan/jq",
  "description":  "Shriek-line JSON processor",
  "stargazers_count":  19967,
  "watchers_count":  19967,
  "language":  "C",
  "license":  {
    "key":  "other",
    "name":  "Varied",
    "spdx_id":  "NOASSERTION",
    "url":  null,
    "node_id":  "MDc6TGljZW5zZTA="
  }
}

jq lets us take care of the JSON myth as an object and steal facets internal of it.

Here is how I filter the JSON myth to score the price of the name key:

$ curl https://api.github.com/repos/stedolan/jq | jq ' .name' 
"jq"

Within the same vogue, for deciding on the price of the owner key:

$ curl https://api.github.com/repos/stedolan/jq | jq ' .owner' 
{
    "login":  "stedolan",
    "identification":  79765
}

You would possibly well maybe possibly possibly drill in as far as you will want to luxuriate in relish this:

$ curl https://api.github.com/repos/stedolan/jq | jq ' .owner.login' 
"stedolan"

What I Learned: Object Identifier-Index

jq lets you score facets in a JSON myth relish it’s a JavaScript object. Factual originate with . ( for the whole myth) and drill down to the price you will want to luxuriate in. It ends up attempting one thing relish this:

jq '.key.subkey.subsubkey'

The usage of JQ to Determine Arrays

If you curl the GitHub Factors API, you would possibly possibly also score attend an array of points:

$ curl https://api.github.com/repos/stedolan/jq/points?per_page=5    
[
  {
    "id": 966024429,
    "number": 2341,
    "title": "Question about license.",
    "body": "I would like to create a [winget](https://github.com/microsoft/winget-cli) kit for jq. 🙏🏻"
  },
  {
  
    "identification":  962477084,
    "quantity":  2340,
    "title":  "visibility of wiki pages",
    "physique":  "The visibility of wiki pages to search engines like google is frequently tiny; as an instance, the search end result for "jq Cookbook" looks to be like relish this:"
  },
  {
   
    "identification":  955350543,
    "quantity":  2337,
    "title":  "Launch 1.6 would not luxuriate in pre-autoreconf'ed configure script",
    "physique":  "If you've a usage seek info from, please ask us on either Stack Overflow (https://stackoverflow.com/questions/tagged/jq) or in the #jq channel (http://irc.lc/freenode/%23jq/) on Freenode (https://webchat.freenode.win/)."
  },
  {
    "identification":  954792209,
    "quantity":  2336,
    "title":  "Fix typo",
    "physique":  ""
  },
  {
    "identification":  940627936,
    "quantity":  2334,
    "title":  "Compile error messages make not provide column finest line quantity",
    "physique":  "Compile errors in filter expressions make not embody the column quantity the establish the parser roughly or precisely locates the error. Most filter expressions are one-liners (are just a few traces even supported?), so the drag wager that the error is on line 1 will not be priceless."
  }
]

To score a particular accept as true with the array, give jq an index:

$ curl https://api.github.com/repos/stedolan/jq/points?per_page=5 | jq '.[4]' 
 {
    "identification":  940627936,
    "quantity":  2334,
    "title":  "Compile error messages make not provide column finest line quantity",
    "physique":  "Compile errors in filter expressions make not embody the column quantity the establish the parser roughly or precisely locates the error. Most filter expressions are one-liners (are just a few traces even supported?), so the drag wager that the error is on line 1 will not be priceless."
  }

Facet Expose: Array Indexing in jq

Array indexing has some priceless consolation syntax.

You would possibly well maybe possibly possibly steal ranges:

$ echo "[1,2,3,4,5]" | jq '.[2:4]'

You would possibly well maybe possibly possibly steal one sided ranges:

$ echo "[1,2,3,4,5]" | jq '.[2:]'

Also, you would possibly possibly possibly spend negatives to score from the end:

$ echo "[1,2,3,4,5]" | jq '.[-2:]'

You would possibly well maybe possibly possibly spend the array index with the object index:

$ curl https://api.github.com/repos/stedolan/jq/points?per_page=5 | jq '.[4].title' 
"Compile error messages make not provide column finest line quantity"

And also you would possibly possibly possibly spend [] to score the whole facets in the array. As an illustration, right here is how I would score the titles of the points returned by my API seek info from:

$ curl https://api.github.com/repos/stedolan/jq/points?per_page=5 | jq '.[].title' 
"Ask about license."
"visibility of wiki pages"
"Launch 1.6 would not luxuriate in pre-autoreconf'ed configure script"
"Fix typo"
"Compile error messages make not provide column finest line quantity"

What I Learned: Array-Index

jq lets you score the whole array [], a particular factor [3], or ranges [2:5] and blend these with the object index if compulsory.

It ends up attempting one thing relish this:

Facet Expose: Making an attempt down Quotes From JQ Output

The -r option in jq offers you raw strings once you would possibly like that.

$ echo '["1","2","3"]' | jq -r '.[]'
1
2
3

The -j option (for join) can mix together your output.

$ echo '["1","2","3"]' | jq -j '.[]'
123

Putting Elements in an Array the usage of jq

Whilst you originate the usage of the array index to score facets, you’ve a original say of affairs. The records returned won’t be a loyal JSON myth. Within the instance above, the realm titles luxuriate in been original line delimited:

"Ask about license."
"visibility of wiki pages"
"Launch 1.6 would not luxuriate in pre-autoreconf'ed configure script"
"Fix typo"
"Compile error messages make not provide column finest line quantity"
...

Genuinely, everytime you ask jq to attain an unwrapped sequence of facets, it prints them each on a original line. You would possibly well maybe possibly possibly seek for this by explicitly asking jq to ignore its input and as a replacement return two numbers:

$ echo '""' | jq '1,2' 
1
2

You would possibly well maybe possibly possibly resolve this the same ability you would possibly flip the text 1,2 into an array in JavaScript: By wrapping it in an array constructor [ ... ].

Within the same vogue, to place a generated sequence of results into a JSON array, you wrap it in an array constructor [ ... ].

My GitHub arena title filter (.[].title) then turns into [ .[].title ] relish this:

$ curl https://api.github.com/repos/stedolan/jq/points?per_page=5 | 
  jq '[ .[].title ] ' 
[
  "Question about license.",
  "visibility of wiki pages",
  "Release 1.6 does not have pre-autoreconf'ed configure script",
  "Fix typo",
  "Compile error messages don't provide column only line number"
]

Now I in actuality luxuriate in a loyal JSON myth.

What I Learned: Array Constructors

If your jq seek info from returns bigger than one factor, they shall be returned newline delimited.

$ echo '[{"a":"b"},{"a":"c"}]' | jq -r '.[].a'
  "b"
  "c"

To flip these values into a JSON array, what you discontinue is linked to establishing an array in JavaScript: You wrap the values in an array constructor ([...]).

It ends up attempting one thing relish this:

$ echo '[{"a":"b"},{"a":"c"}]' | jq -r '[ .[].a ]'

The usage of jq to Determine More than one Fields

The GitHub points API has various facts I don’t care about. I want to score just a few fields from the returned JSON myth and recede away the the rest on the attend of.

The best seemingly ability to discontinue this is the usage of , to specify just a few filters:

$ curl https://api.github.com/repos/stedolan/jq/points?per_page=2 |  
  jq ' .[].title, .[].quantity'
"Ask about license."
"visibility of wiki pages"
2341
2340

Nonetheless this is returning the results of 1 different after the different. To alternate the ordering, I’m capable of factor out the array selector:

$ curl https://api.github.com/repos/stedolan/jq/points?per_page=2 | 
  jq '.[] |  .title, .quantity'
"Ask about license."
2341
"visibility of wiki pages"
2340

This refactoring uses a pipe (|), which I’ll discuss quickly, and runs my object selectors (.title and .quantity) on each array factor.

If you wrap the seek info from in the array constructor you score this:

$ curl https://api.github.com/repos/stedolan/jq/points?per_page=2 | 
  jq '[ .[] |  .title, .number ]'
[
  "Question about license.",
  2341,
  "visibility of wiki pages",
  2340
]

Nonetheless this serene isn’t the JSON myth I desire. To score these values into a just JSON object, I desire an object constructor { ... }.

Putting Elements Into an Object The usage of jq

Let’s seek for at some simple examples earlier than exhibiting how my GitHub seek info from can spend an object constructor.

A Puny Example

I in actuality luxuriate in an array that contains my name (["Adam","Gordon","Bell"]), and I want to flip it into a JSON object relish this:

{
  "first_name": "Adam",
  "last_name": "Bell"
}

I’m capable of steal the facets I desire the usage of array indexing relish this:

$ echo '["Adam","Gordon","Bell"]' | jq -r '.[0], .[2]'
Adam
Bell

To wrap those values into the shape I desire, I’m capable of replace the values with the array indexes that return them:

{
  "first_name": .[0],
  "last_name": .[2]
}

Or on a single line relish this:

$ echo '["Adam","Gordon","Bell"]' | jq -r '{ "first_name":.[0], "last_name": .[2]}'
{
  "first_name":  "Adam",
  "last_name":  "Bell"
}

This syntax is the same syntax for establishing an object in a JSON myth. The finest incompatibility is you would possibly possibly possibly spend the object and array queries you’ve built up because the values.

Support to GitHub

Returning to my GitHub API say of affairs, to wrap the quantity and the title up into an array I spend the object constructor relish this:

$ curl https://api.github.com/repos/stedolan/jq/points?per_page=2 | 
  jq '[ .[] | { title: .title, number: .number} ]'
[
  {
    "title": "Question about license.",
    "number": 2341
  },
  {
    "title": "visibility of wiki pages",
    "number": 2340
  }
]

What I Learned: Object Constructors

To position the facets you’ve selected attend into a JSON myth, you would possibly possibly possibly wrap them in an object constructor { ... }.

If you luxuriate in been building up a JSON object out of several selectors, it would possibly well maybe possibly find yourself attempting one thing relish this:

jq '{ "key1": <>, "key2": <> }'

Which is the same syntax for an object in a JSON myth, as opposed to with jq you would possibly possibly possibly spend filters as values.4

Sorting and Counting With JQ

The next say of affairs I in actuality luxuriate in is that I want to summarize some this JSON records. Each arena returned by GitHub has a series of labels:

$ curl https://api.github.com/repos/stedolan/jq/points/2289 |    
  jq ' { title: .title, quantity: .quantity, labels: .labels} '
  {
    "title":  "Bump jinja2 from 2.10 to 2.11.3 in /scientific doctors",
    "quantity":  2289,
    "labels":  [
      "feature request",
      "dependencies"
    ]
  }

jq Built-in Positive aspects

If I desire those labels in alphabetical narrate I’m capable of spend the built in form operate. It works relish this:

$  echo '["3","2","1"]' | jq 'form'
["1", "2", "3"]

Here’s linked to how I would form an array in JavaScript:

const l = ["3","2","1"];
l.form();

Varied built-ins that contemplate JavaScript functionality are on hand, relish dimension, reverse, and tostring and they’ll all be ragged in a same ability:

$  echo '["3","2","1"]' | jq 'reverse'
["1", "2", "3"]
$  echo '["3","2","1"]' | jq 'dimension'
3

If I’m capable of mix these built-ins with the selectors I’ve built as much as this level, I’ll luxuriate in solved my label sorting say of affairs. So I’ll gift that subsequent.

What I Learned: jq built-ins

jq has many built-in capabilities. There are potentially too many to place in mind but the built-ins are inclined to contemplate JavaScript capabilities, so give those a attempt earlier than heading to jq manual , and also you would possibly possibly also score lucky.5

Pipes and Filters

Before I’m capable of spend form to form the labels from my GitHub API seek info from, I must gift how pipes and filters work in jq.

jq is a filter in the UNIX suppose line sense. You pipe (|) a JSON myth to it, and it filters it and outputs it to celebrated out. I also can with out issues spend this option to chain together jq invocations relish this:

echo '{"title":"JQ Determine"}' | jq '.title' | jq 'dimension'
9

Here’s a wordy, even supposing simple, ability to discover the size of a string in a JSON myth. You would possibly well maybe possibly possibly spend this same thought to mix diverse jq built-in capabilities with the parts I’ve proven to this level. Nonetheless there is an more straightforward ability, even supposing. You would possibly well maybe possibly possibly spend pipes internal of jq and conceptually they work correct relish shell pipes:

echo '{"title":"JQ Determine"}' | jq '.title | dimension'
9

Here are some more examples:

  • .title | dimension will return the size of the title
  • .quantity | tostring will return the realm quantity as a string
  • .[] | .key will return the values of key key in the array (this is same to this .[].key)

Which ability that sorting my labels array is understated. I’m capable of correct alternate .labels to .labels | form:

$ curl https://api.github.com/repos/stedolan/jq/points/2289 | 
  jq ' { title: .title, quantity: .quantity, labels: .labels | form } '    
  {
    "title":  "Bump jinja2 from 2.10 to 2.11.3 in /scientific doctors",
    "quantity":  2289,
    "labels":  [
      "dependencies",
      "feature request"
    ]
  }

And once you will want to luxuriate in correct a label count that’s simple as effectively:

$ curl https://api.github.com/repos/stedolan/jq/points/2289 | 
  jq ' { title: .title, quantity: .quantity, labels: .labels | dimension } '    
  {
    "title":  "Bump jinja2 from 2.10 to 2.11.3 in /scientific doctors",
    "quantity":  2289,
    "labels":  2
  }

What I Learned: Pipes and Filters

All the pieces in jq is a filter that you just would possibly possibly possibly mix with pipes (|). This mimics the behavior of a UNIX shell.

You would possibly well maybe possibly possibly spend the pipes and the jq built-ins to develop refined transformations from simple operations.

It ends up attempting one thing relish this:

jq ' .key1.subkey2[] | form ' # sorting
jq ' .key2.subkey | dimension' # dimension of string or array
jq ' .key3 | floor | tostring | dimension' # etc

Maps and Selects The usage of JQ

The points list I became once has many low-quality points in it.6 Let’s convey I want to grab the whole items which shall be labeled. This would possibly well let me skip the whole drive-by fix-my-say of affairs points.

Sadly, it’s not attainable to discontinue this with the GitHub API unless you specify the whole seemingly labels in your seek info from. On the different hand, I’m capable of with out issues discontinue this seek info from on the suppose line by filtering our results with jq. On the different hand, to discontinue so, I’m going to need a couple more jq capabilities.

My seek info from to this level looks to be like relish this:

  jq '[ .[] | { title: .title, number: .number, labels: .labels | length } ]'

The most important thing I’m capable of discontinue is simplify it the usage of scheme.

  jq 'scheme({ title: .title, quantity: .quantity, labels: .labels | dimension }) 

scheme(...) let’s you unwrap an array, observe a filter after which rewrap the results attend into an array. You would possibly well maybe possibly possibly bring to mind it as a shorthand for [ .[] | ... ] and it comes up reasonably a chunk in my expertise, so it’s charge it committing to reminiscence.

I’m capable of mix that with a steal assertion that looks relish this:

steal is a built-in operate that takes a boolean expression and finest returns facets that match. It’s linked to the WHERE clause in a SQL assertion or array filter in JavaScript.

Like scheme, I salvage steal comes up reasonably a chunk, so while you would possibly possibly also must attain attend to this article or google it the first few times you would possibly prefer it, with luck, this would possibly well maybe originate to follow your reminiscence after that.

Putting this all together looks to be like relish this:

curl https://api.github.com/repos/stedolan/jq/points?per_page=100 | 
   jq 'scheme({ title: .title, quantity: .quantity, labels: .labels | dimension }) | 
   scheme(steal(.labels > 0))'
[
  {
    "title": "Bump lxml from 4.3.1 to 4.6.3 in /docs",
    "number": 2295,
    "labels": 1
  },
  {
    "title": "Bump pyyaml from 3.13 to 5.4 in /docs",
    "number": 2291,
    "labels": 1
  },
  {
    "title": "Bump jinja2 from 2.10 to 2.11.3 in /docs",
    "number": 2289,
    "labels": 1
  },
  {
    "title": "Debugging help through showing pipeline intermediates. ",
    "number": 2206,
    "labels": 1
  }
]

This uses three object indexes, two maps, two pipes, a dimension operate, and a steal predicate. Nonetheless once you’ve followed along, this must serene all construct sense. It’s all correct composing together filters except you score the final end result you would possibly like.

Now lets discuss simple strategies to place this records into educate.

In Review

What I Learned

Here is what I’ve learned to this level:

jq lets you score facets by beginning with a . and gaining access to keys and arrays relish it’s a JavaScript Object (which is it is). This feature uses the Object and Array index jq creates of a JSON myth and seek for relish this:

jq '.key[0].subkey[2:3].subsubkey'

jq programs can hang object constructors { ... } and array constructors [ ... ]. You divulge these whenever you happen to want to must wrap attend up one thing you’ve pulled out of a JSON myth the usage of the above indexes:

jq '[ { key1: .key1, key2: .key2 }  ]'

jq contains built-in capabilities (dimension,form,steal,scheme) and pipes (|), and also you would possibly possibly possibly build these together correct equivalent to you would possibly possibly possibly mix pipes and filters on the suppose line:

  jq 'scheme({ narrate-of-magitude: .items | dimension | tostring | dimension }) 

Next Steps for Mastering jq

Reading about (or writing about) a instrument will not be sufficient to master it. Motion is compulsory. Here is my assignment for cementing this records:

1. Full the jq Tutorial

jq-tutorial will not be an instructional the least bit, but a series of around 20 interactive workouts that test you records of jq. I’ve chanced on it extraordinarily priceless.

2. Are attempting The usage of Your Memory First

At any time after I must extract records or transform a JSON myth, I strive to discontinue it first with out attempting one thing up. If reminiscence fails me, most frequently jqterm, which has auto-completion, is priceless. Normally, I serene must seek for one thing up, but science has proven that repeated retrieval yields retention. So over time, my retention must serene reinforce.

3. Train It

If you don’t spend a instrument, you would possibly possibly also not ever master it. So after I in actuality luxuriate in a assignment that shall be solved the usage of jq, then it is what I spend. Now not decrease than for the next tiny while, despite the truth that there is an more straightforward ability to discontinue it. Whether or not it’s exploring a REST API or docker seek for results, JSON is in every single place, so alternatives abound.

4. Be taught More

Lastly, to deepen my records, I’m studying about recursive descent, declaring variables, and defining capabilities and evolved parts chanced on in the manual. Keep in mind that, these items hardly ever attain up, but after writing all this, I’m curved on this instrument.

Doing all this isn’t compulsory, but once you educate me in a majority of these steps, I feel the usage of jq will change into 2nd nature.

Conclusion

To this level I’ve finest lined the basics of jq. The jq seek info from language is a plump programming language and also you would possibly possibly possibly discontinue hundreds intriguing things with it. You substitute from JSON to CSV. You would possibly well maybe possibly possibly provide an explanation for you luxuriate in capabilities and even salvage primes with jq:

# Denoting the input by $n, which is assumed to be a obvious integer,
# eratosthenes/0 produces an array of primes decrease than or equal to $n: 
def eratosthenes: 
  (. + 1) as $n
  | (($n|sqrt) / 2) as $s
  | [null, null, range(2; $n)]
  | decrease (2, 1 + (2 * fluctuate(1; $s))) as $i (.; erase($i))
  | scheme(steal(.));

On the different hand, simple stuff – relish deciding on facets, filtering by key, or price – is frequently all you would possibly like.

I am hoping this helps construct jq more approachable and that you just not must recede to google at any time once you will want to must seek info from a JSON myth7.

Whilst you’re right here:

Earthly is a syntax for defining your develop. It works along with your existing develop machine. Derive repeatable and comprehensible builds recently.

Feedback?

I’d want to listen to your suggestions, or once you’ve any tips of your luxuriate in. Let me know on twitter:

jq is huge for reasonably-printing JSON but it completely does a lot more.

I never in actual fact mastered it and it became once a say of affairs at any time after I attempted to make spend of it.

So I took some time to learn the basics. Here is what it’s vital to know: 🧵

— Adam Gordon Bell 🤓 (@adamgordonbell) August 24, 2021