قالب وردپرس درنا توس
Home / Tips and Tricks / How to parse JSON files on the Linux command line with jq

How to parse JSON files on the Linux command line with jq



  A terminal prompt on a Linux computer.
Fatmawati Achmad Zaenuri / Shutterstock

JSON is one of the most popular formats for transmitting text-based data around the web. It is everywhere, and you will probably encounter it. We show how to handle it from the Linux command line with the command jq .

JSON and jq

JSON stands for JavaScript Object Notation. It is a schema that allows data to be encoded into plain text files in a self-describing way. There are no comments in a JSON file ̵

1; the content should be self explanatory. Each data value has a text string called "name" or "key." This tells you what the data value is. Together they are called names: value pairs or key: value pairs. A colon (: ) separates a key from its value.

An "object" is a collection of key: value pairs. In a JSON file, an object starts with an open curly bar ( {) and ends with a closing bar (} ). JSON also supports "matrices", which are organized lists of values. An array begins with an opening bracket ( [) and ends with a closing (] ).

Of course, from these simple definitions, arbitrary complexity can arise. For example, objects can be encapsulated in objects. Objects can contain matrices and matrices can also contain objects. Everyone can have open levels of nesting.

But in practice, if the design of JSON data is interconnected, the design of the data layout should probably use a new retest. Of course, if you do not generate JSON data, just try to use it, you have nothing to say in its layout. Unfortunately, in these cases you just have to deal with it.

Most programming languages ​​have libraries or modules that allow them to analyze JSON data. Unfortunately, the Bash shell has no such functionality.

Necessity is the mother of invention, but the jq tool was born! With and we can easily analyze JSON in the Bash shell. And it doesn't matter if you have to work with well-designed, elegant JSON, or things nightmares are made of.

How to install jq

We had to install jq on all the Linux distributions that we used to investigate this article.

To install jq on Ubuntu, type this command:

  sudo apt-get install jq 

  The

To install jq on Fedora , enter this command:

  sudo dnf install jq 

  The

To install jq on Manjaro, enter this command:

  sudo pacman -Sy jq 

  The

How to make JSON readable

JSON does not care about white space and layout does not affect it. As long as it follows the rules of the JSON grammar, systems that process JSON can read and understand it. Because of this, JSON is often transmitted as a simple, long string without regard to layout. This saves some space because tabs, spaces and newline characters do not need to be included in JSON. Of course, the downside to all this is when a person tries to read it.

Let's pull a short JSON object from the NASA Web site that tells the position of the International Space Station. We use curl which can download files to retrieve the JSON object for us.

We don't care about any of the status messages curl usually generates, so we type the following, with the option -s (silent):

  curl -s http: // api.open-notify.org/iss-now.json Thu 1919909027: The "curl -s http://api.open-notify.org/iss-now.json" command in a terminal window. "width =" 646 " height = "97" src = "/ pagespeed_static / 1.JiBnMqyl6S.gif" onload = "pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" onerror = "this.onerror = null; pagespeed.lazyLoadImages.loadIfVisible; > 

Now, with some effort, you can read this. You have to select the data values, but it is not easy or convenient. Let's repeat this, but this time we move it through or .

jq uses filters to analyze JSON, and the simplest of these filters is a period (. ), which means "printing the entire object." By default, jq the print is printed fairly.

We put it all together and write the following:

  curl -s http://api.open-notify.org/iss-now.json | jq. 

  The

It's much better! Now we can see exactly what is happening.

The whole item is wrapped in curly braces. It contains two keys: name pair: message and timestamp . It also contains an object named iss_position that contains two keys: value pairs: longitude and latitude .

We'll try it again. This time, we will type the following and redirect the output to a file named "iss.json":

  curl -s http://api.open-notify.org/iss-now.json | jq. > iss.json 
  cat iss.json 

  The commands iss.json "and" cat iss.json "in a terminal window." width = "646" height = "262" src = "/ sidespeed_static / 1.JiBnMqyl6S.gif" onload = "sidespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" onerror = "this.onerror = null; sidespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" />

This gives us a well-planned copy of the JSON object on our hard drive.

RELATED: [19659041] How to use curl to download files from Linux command line

Access data values ​​

As we saw above, can extract data values ​​moving from JSON. It can also work with JSON stored in a file. We will work with local files so that the command line is not concerned with curl commands. This should make it a little easier to follow.

The easiest way to extract data from a JSON file is to provide a key name to get its data value. Enter a period and the key name with no spaces between them. This creates a filter from the key name. We also need to tell jq which JSON file to use.

We write the following to retrieve the message :

  jq .message iss.json 

jq prints the text of the message in the terminal window.

If you have a key name that contains spaces or punctuation, you must wrap its filter with quotation marks. It is common to use only characters, numbers and underscores so that the JSON key names are not problematic.

First, we write the following to retrieve the value timestamp :

  jq .timestamp utf .json 

  The

The timestamp value is retrieved and printed in the terminal window.

But how can we access the values ​​in the object iss_position ? We can use the JSON point notation. We include the item iss_position in the "path" to the key value. To do this, the name of the object as the key is inside the previous name of the key itself.

We write the following, including latitude key name (Note that there are no spaces between ". Iss_position" and ".latitude"):

  jq .iss_position.latitude iss.json 

  The

To extract multiple values, you must do the following:

  • List the key names on the command line.
  • Separate them with commas ().
  • Enclose them with quotation marks ( " ) or apostrophes ( ").

With that in mind, we write the following:

  jq" .iss_position.latitude, .timestamp "iss.json 

  The

The two values ​​are printed to the terminal window. [19659011] Working with Arrays

Let's take another JSON object from NASA.

This time we use a list of astronauts currently in space:

  curl -s http: // api. Open notif y.org/astros.jsonebrit19659070??The "curl -s http: //api.ope n-notify.org/astros.json "command in a terminal window." width = "646" height = "147" src = "/ pagespeed_static / 1.JiBnMqyl6S.gif" onload = "pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" onerror = "this.onerror = null; pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" /> 

Okay, it worked, so let's do it again.

We write the following to pass it through jq and redirect it to a file named "astro.json":

  curl -s http://api.open-notify.org/ astros.json | jq. > astro.json 

 astros.json "the command in a terminal window." width = "646" height = "77" src = "/ sidespeed_static / 1.JiBnMqyl6S.gif" onload = "sidespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" onerror = "this.onerror = null; sidespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" />

Now let's write the following to check our file:

  less astro.json 

  The

As shown below, we now see the list of astronauts in space, as well as their spacecraft.

 Output from

This JSON object contains a group called people . We know it is a matrix because of the opening bracket ( [) (highlighted in the screenshot above). It is a series of objects each containing two keys: value pairs: names and crafts .

As we did before, we can use the JSON point notation to access the values. We must also include the parentheses ([]) in the matrix name.

With all this in mind, we write the following:

  jq ".people [] .name" astro.json [19659085] The "jq" .people [] .name "astros.json" command in a terminal window. "width =" 646 "height =" 197 "src =" / pagespeed_static / 1.JiBnMqyl6S.gif "onload =" pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this); "onerror =" this.onerror = null; pagespeed. lazyLoadImages.loadIfVisibleAndMaybeBeacon (this); "/> 

This time all the name values ​​print to the terminal window. What we asked jq to do was print the name value for each object in the array. Pretty neat, huh? [19659006] We can retrieve the name of a single object if we place its position in the array in the parentheses ([]) on the command line.The array uses zero offset indexing, which means the object in the first position in the array is zero.

to access the last object in the array you can use -1; To get the second last object in the array you can d u use -2, etc.

Sometimes the JSON object provides the number of elements in the matrix, which is the case here. Together with the matrix it contains a key: name pairs named number with a value of six.

The following number of objects is in this matrix:

  jq ".people [1]. Name" astro.json 
  jq ".people [3] .name" astro.json 
  jq ". people [-1] .name "astro.json 
  jq" .people [-2] .name "astro.json 

  The

You can also provide a start and end item within the matrix. This is called "slicing", and it can be a bit confusing. Remember that the matrix uses a zero offset.

To retrieve the objects from index position two, up to (but not including) the object at index position four, we type the following command:

  jq ".people [2:4]" astro.json 

  The [19659006] This prints the objects at array index two (the third object in the array) and three (the fourth object in the array). It ends processing at array index four, which is the fifth object in the array.

The way to better understand this is to experiment on the command line. You will soon see how it works.

How to use pipes with filters

You can direct output from one filter to another and you do not need to learn a new symbol. Same as the Linux command line, jq uses the vertical field ( | ) to represent a pipe.

We tell and to lead the pipe people enter the .name filter, which should list the names of astronauts in the terminal window.

We write the following:

  jq ".people [] | .name" astro.json 

  The

RELATED: How To Use Pipes On Linux

Creating arrays and changing results

We can use jq to create new objects, such as arrays. In this example, we extract three values ​​and create a new matrix containing these values. Note that the opening ( [) and the closing brackets (] ) are also the first and last characters of the filter string.

We write the following:

  jq "[.iss-position.latitude, iss_position.longitude, .timestamp]" iss.json 

  The

The output is wrapped in parentheses and separated by a comma, making it a properly formed matrix.

Numeric values ​​can also be manipulated when retrieved. Let's drag the timestamp from the ISS position file and then extract it again and change the value that has been returned.

To do so, we write the following:

  jq ".timestamp" iss. json 
  jq ".timestamp - 1570000000" iss.json 

  The

This is useful if you need to add or remove a default offset from a set of values.

Let's write the following to remind us what the file iss.json contains:

  jq. iss.json 

Let's say we want to get rid of the message key: value-pairs. It has nothing to do with the position of the International Space Station. Only one flag indicates that the site was successfully retrieved. If that is the excess of the requirements, we can avoid it. (You can also just ignore it.)

We can use jq & # 39; delete function, part () to remove a key: value-pair . To delete the message key: value pairs, we write this command:

  jq "del (.message)" iss.json 

  The

Note that it does not actually delete it from "iss .json" - file; it just removes it from the command. If you need to create a new file without the message key: the value pair in it, run the command and then redirect the output to a new file.

More Complicated JSON Objects

Let's get some more NASA data, this time we will use a JSON object that contains information about meteor impact locations from around the world, a larger file with a much more complicated JSON structure than the ones we have previously processed.

First, we write the following to redirect it to a file named "strikes.json":

  curl - s https://data.nasa.gov/resource/y77d-th95.json | jq .> strikejson 

 strikejson "command in a terminal window." width = "646 "height =" 77 "src =" / sidespeed_static / 1.JiBnMqyl6S.gif "onload =" sidespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this); "onerror =" this.onerror = null; sidespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this); "/>

To see how JSON looks, we write the following:

  minor strikes.json 

As shown below, the file starts with an opening bracket ( [), so the whole object is a matrix. The objects in the matrix are collections of keys: value pairs and there is a capsular object called geolocation . geolocation object contains additional key: value pairs and a matrix called coordinates .

 Exit from

Let's get the meteor's name from the object at index position 995 through the end of the matrix.

We write the following to pass JSON through three filters:

  jq ". [995:] |. [] | .name" strikejson 

  The

The filters work as follows :

  • . [995:]: This tells and to process objects from array index 995 to the end of the matrix. No number after colon (: ) is what says or to continue to the end of the matrix.
  • . []: This array iterator prompts and to process each object in the matrix.
  • .name : This filter extracts the name value.

With a small change, we can extract the last 10 objects from the matrix. A "-10" instructs or to begin processing object 10 back from the end of the matrix.

We write the following:

  jq ". [-10:] |. [-2] |. Name" strikes.json 

  The

Just as we have in previous examples, we can write the following for to select a single object:

  jq ". [650] .name" strikes.json [19659158] The "jq". [650] .name "strikes.json" command in a terminal window. "width =" 646 "height =" 97 "src =" / pagespeed_static / 1.JiBnMqyl6S.gif "onload =" pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this); "onerror =" this.onerror = null; pagespeed.lazyLoadImages.loadIfVisible /> 

We can also apply slicing to strings. To do so, we write the following to request the first four characters of the object's name on array index 234:

  jq ". [234] .name [0:4] "strikes.json 

  The

We can also see a specific object in its entirety. To do this, we write the following and include an array index without any key: value filter :

  jq ". [234] "strjker.json 

  The

If you just want to see the values, you can do the same without the key names.

For our example, we write this command:

  jq". [234][] "strikes.json 

To retrieve multiple values ​​from each object, we separate them with commas in the following command:

  jq ". [450:455] |. [] | .name, .mass" strikes. json 

  The

To retrieve nested values, you must identify the objects that form the "path" to them.

For example, to refer to the coordinates values, we must include the all-inclusive matrix, geolocation nested object, and the nested coordinates shown below.

 The path to array in a nested JSON object highlighted in a terminal window.

To see the coordinates values ​​for the object in index position 121 in the matrix, we type the following command:

  jq ". [121]. Geolocation. Coordinates []" strikes .json 

  The function

Length Function

jq The function provides different measurement values ​​depending on what it has been applied, such as:

  • Strings : The length of the string in bytes.
  • Object : Number of keys: value pairs in the object.
  • Arrays : The number of array elements in the array.

The following command returns the length of the name in 10 of the objects in the JSON array, starting at index position 100:

  jq ". [100:110] |. [] .name | length" strikes .json 

  The

To see how many key: value pairs are in the first object in the matrix, we write this command:

  jq ". [0] | length" strikes.json [19659184] The "jq ".[0] | length" strikes.json" command in a terminal window." width="646" height="97" src="/pagespeed_static/1.JiBnMqyl6S.gif" onload="pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);" onerror="this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);"/> 

Keys Function

You can use the key function to find out the JSON you are working with. It can tell you what the names of the keys are and how many objects there are in a matrix.

To find the keys in the people object in the file "astro.json" we write this command:

  jq ". People. [0] | keys" astro.json 

  The

To see how many elements are in the group we write this command:

  jq ".people | keys" astro.json 

  The

This shows that There are six, zero-offset array elements, numbered zero to five.

The has () Function

You can use the has () function to interrogate JSON and see if an object has a certain key name. Note that the key name must be wrapped in quotation marks. ]), as follows:

  jq & # 39 ;. [] | has ("name type") "strikes.json 

  The

Each object in the matrix is ​​checked, as shown below. [19659006]   Output from" jq ". [] | has (" nametype ") & # 39; strikes.json "command in a terminal window.

To check a specific object, include its index position in the matrix filter, as follows:

  jq & # 39; . [678] | its ("name type") & # 39; strikes.json 

Don't go near JSON Without It

jq tool is the perfect example of the professional, powerful, fast software that makes life in the Linux world such a pleasure.

This was just a brief introduction to the usual features of this command - there is much more to it. Be sure to check out the comprehensive jq manual if you want to dig deeper.




Source link