As part of Side Project Summer, an initiative set up by the team behind Consonance to promote digital literacy in the publishing industry, I attempted to build a static site that interacts with the Consonance API. I had recently learned the basics of what an API is, and how to make API calls using curl and later the Ruby gem httparty, so I thought it would be useful to put these into practical application – and so, side project!

I wasn’t initially sure whether I would be able to do this using the static site generator I’m familiar with, Jekyll – I had never written a custom plugin (or much Ruby at all for that matter) – but was spurred on by the example of Snowbooks, built using a similar workflow, which its author, Emma Barnes, kindly shared with me. After scratching my head over the code for quite some time, I thought I’d have a go anyway and hope it’d start to make some sense along the way. Normally I find myself getting bogged down in aesthetics (i.e. CSS) before scoping out ideas properly, but for once I thought I’d focus on the plugin and the data, and forget about what the site was going to look like.

The plugin is a Ruby script that calls on the Consonance Product API to retrieve book data from my work’s (University of London Press) database that can then be displayed on the site.

Here is an example of an API request to the Consonance Product API:

NB The example is from a previous version – the Product API is now at version 3.

curl -g -H "Authorization: Token token=b0c15687c2c3483b9efca116db0917b9" 
"https://demo.consonance.app/api/v2/products.json"
=> {
      "current_page": 1,
      "per_page": 50,
      "total_pages": 10,
      "url": "https://demo.consonance.app/api/v2/products.json",
      "products": [
         {...},{...},{...}
      ]
   }

The objects in the products array contain the book metadata that I need to display on my site, but as the information in the header tells us, only 50 of a total of 500 (50 x 10 total_pages) products are in the array at any given time. So we need to find a way to iterate over the pages so that we can retrieve all the product data available.

I created a Ruby file in the _plugins folder (so the program would run when the site is built by Jekyll) and translated the above curl request using httparty and our account’s token (starred out for security purposes).

require 'rubygems'
require 'httparty'
require 'json'

url = 'https://web.consonance.app/api/v3/products.json'
query = {
}
headers = {
  Authorization: 'Token token=**********************'
}
response = HTTParty.get(url, query: query, headers: headers)

Though not sure how exactly, I knew I would need to use the current and total pages later, so I assigned them to variables:

current_page = JSON.parse(response.body)['current_page']
=> 1
total_pages = JSON.parse(response.body)['total_pages']
=> 21

NB The JSON.parse method is used to parse the unparsed response before accessing the values.

Though I didn’t know what needed to happen yet, I knew whatever it was it needed to happen inside a block wrapped by these variables, probably in a while loop that runs as long as the current_page is less than the total_pages – something along the lines of:

while current_page <= total_pages do
   ...
end

Now to work out what to put inside it. I presumably needed a similar call to the one that I used to get the page variables in the first place, but each time the code block ran again in the loop, it needed to make a slightly different call, i.e. the current_page number needed to be incremented by 1 with each new iteration:

while current_page <= total_pages do
   ...
   current_page += 1
end

So, back to the documentation, which tells me that a page query parameter can be passed into the API endpoint, e.g. https://demo.consonance.app/api/v2/products.json?q[page]=1. Whereas my first call to the API had no parameters, this call specifies what page number to return results for by passing in the current_page number variable using Ruby’s string interpolation method:

while current_page <= total_pages do

  url = 'https://web.consonance.app/api/v3/products.json'
  # pass in current_page as query parameter
  query = {
    'page' => "#{current_page}",
  }
  headers = {
    Authorization: "Token token=************************"
  }
  request = HTTParty.get(url, query: query, headers: headers)

  # increment current_page variable 
  current_page += 1

end

At the moment, though, this doesn’t do anything except make the call, store the response as the request variable, then overwrite it each time the code reruns. The information from each response needs to be stored all together so that it can be used later. A reasonable way to do this seemed to be storing it in the same form it came in, i.e. an array, only without the per page limit of 50 that the original response had. The array needs to be initialised outside of the while loop otherwise, as before, the data will be overwritten. Then I can push the data from each call to it to be stored for later:

# initialise empty array
product_array = []

while current_page <= total_pages do

  url = 'https://web.consonance.app/api/v3/products.json'
  # pass in current_page as query parameter
  query = {
    'page' => "#{current_page}",
  }
  headers = {
    Authorization: "Token token=************************"
  }
  request = HTTParty.get(url, query: query, headers: headers)

  # push each product from each request to the array
  JSON.parse(request.body)['products'].each do |product|
    product_array.push(product) if product["pub_date"]
  end

  # increment current_page variable 
  current_page += 1

end

NB The if product["pub_date"] condition was put in later for error handling in order to filter out any items without a publication date.

To check that everything is working so far we can check how many products are in the product_array

puts product_array.length
=> 1,029

which, if need be, I can check against the information given in the Consonance dashboard:

Consonance products overview
Consonance products overview

Success! It tallies.

It might be useful at this point to have a retrievable store of the response to check, so now might be a good time to write it to a file:

File.open("./_data/all_products.json", "w") { |file| 
  # The product_array is being written back to JSON 
  # as the JSON.parse method converted it to a hash
  file.puts(product_array.to_json)
}

NB I’m nesting the file in the _data folder as this is where it will be retrieved by Jekyll later. See the Jekyll config for more details.

Now I can take a look at the JSON file that’s been populated to see if it contains the key/value pairs of all the good stuff that I’m looking for:

JSON editor view in VS Code
JSON editor view in VS Code

Looks promising! At the moment though each product contains a lot of information I won’t necessarily need to display on my site pages. We can try to customise it by using the .map array method:

product_array.map do |product|
  {
    "status" => product["publishing_status"],
    "id" => product["id"],
    "work_id" => product["work_id"],
    "format" => product["in_house_edition"],
    "isbn" => product["isbn"],
    "title" => product["full_title"],
    "author" => product["authorship"],
    "pub_date" => product["pub_date"],
    "prices" => product["prices"].map do |price|
      {
        "currency" => price["currency_code"],
        "amount" => price["price_amount"]
      }
    end.flatten
  }
end.flatten

The first part of each key/value pair assigns a name of our choice, e.g. "status", and then assigns it a value from the original array using it’s original key name, e.g. "publishing_status". What this gives is a simplified version of the data with only some select values from the API’s raw data:

JSON editor view in VS Code with mapped data
JSON editor view in VS Code (mapped data)

Now that we have a data file, we can start accessing its data as variables by using Jekyll’s templating language, Liquid, e.g:

{% for product in site.data.products %}
  <h1>{{ product.title }}</h1>
  <h2>{{ product.author }}</h2>
  <p>{{ product.pub_date }}</p>
{% endfor %}

Needless to say there’s an awful lot more that can be done to clean up this process and refactor the code so it looks more like a well-structured Ruby file with proper Classes and methods, but unfortunately that’s a bit beyond me at the moment. Another thing would also to be make better use of the query paramaters available for the Product API so we can retrieve the data we’re after from the off. All that aside, we have at the very least now got a serviceable data file which we can start passing in to our Jekyll template pages using site.data variables.

Stay tuned for Side Project Summer 2020.

Screenshots

Forthcoming publications block
Forthcoming publications block
Example publication page 1
Example publication page 1
Example publication page 2
Example publication page 2
Written by Jamie Bowman
Last updated 4th May 2020