The Magic of APIs

Back to Listing

18 October 2018


The internet is full of data… and full of data about data (metadata). It is presented to us in many ways. There are tables and lists and graphs (Oh My!), all just a mouse-click away (or maybe two, or seven, or maybe you need pages to get through it). However, there are many reasons you may want your data in a different format. Perhaps you need a new way to gather/present data for accessibility reasons, or perhaps you simply need to be able to save it offline. Maybe you want to send a memo to your boss (or professor) showing how many people have cloned your GitHub repo (or, in the case of my demo, we’ll demonstrate how many people have starred it). Maybe your organization requires a listing of trouble tickets that requires some field manipulation that the GUI itself cannot perform. Or maybe you want to leave your job as a software developer and become a social media specialist, so you’d like to quantify your twitter popularity in a spreadsheet… So, let’s learn about APIs!

“In computer programming, an application programming interface (API) is a set of subroutine definitions, communication protocols, and tools for building software“ (from Wikipedia.)

This blog post works with examples from my GitHub repo to show you how to use the GitHub API via command line including Ruby code to get information on the repo itself. It’s slightly recursive and maybe fun! Let’s get started!

Here is a link to the GitHub API docs - you may want to refer to these as you read, or check them out later: https://developer.github.com/v3/. Here is a link to the GitHub repo we’ll be using, in GitHub’s web page view: https://github.com/judyj/Magic-of-APIs

You can see the attributes of my repo in JSON by clicking here - or here’s a glance at the code:

{
  "id": 147799901,
  "node_id": "MDEwOlJlcG9zaXRvcnkxNDc3OTk5MDE=",
  "name": "Magic-of-APIs",
  "full_name": "judyj/Magic-of-APIs",
  "private": false,
  "owner": {
    "login": "judyj",
    "id": 1534760,
    "node_id": "MDQ6VXNlcjE1MzQ3NjA=",
    "avatar_url": "https://avatars3.githubusercontent.com/u/1534760?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/judyj",
    "html_url": "https://github.com/judyj",
    "followers_url": "https://api.github.com/users/judyj/followers",
    "following_url": "https://api.github.com/users/judyj/following{/other_user}",
    "gists_url": "https://api.github.com/users/judyj/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/judyj/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/judyj/subscriptions",
    "organizations_url": "https://api.github.com/users/judyj/orgs",
    "repos_url": "https://api.github.com/users/judyj/repos",
    "events_url": "https://api.github.com/users/judyj/events{/privacy}",
    "received_events_url": "https://api.github.com/users/judyj/received_events",
    "type": "User",
    "site_admin": false
  },
  "html_url": "https://github.com/judyj/Magic-of-APIs",
  "description": "Demo repo for my upcoming blog",
  "fork": false,
  "url": "https://api.github.com/repos/judyj/Magic-of-APIs",
  "forks_url": "https://api.github.com/repos/judyj/Magic-of-APIs/forks",
  "keys_url": "https://api.github.com/repos/judyj/Magic-of-APIs/keys{/key_id}",
  "collaborators_url": "https://api.github.com/repos/judyj/Magic-of-APIs/collaborators{/collaborator}",
  "teams_url": "https://api.github.com/repos/judyj/Magic-of-APIs/teams",
  "hooks_url": "https://api.github.com/repos/judyj/Magic-of-APIs/hooks",
  "issue_events_url": "https://api.github.com/repos/judyj/Magic-of-APIs/issues/events{/number}",
  "events_url": "https://api.github.com/repos/judyj/Magic-of-APIs/events",
  "assignees_url": "https://api.github.com/repos/judyj/Magic-of-APIs/assignees{/user}",
  "branches_url": "https://api.github.com/repos/judyj/Magic-of-APIs/branches{/branch}",
  "tags_url": "https://api.github.com/repos/judyj/Magic-of-APIs/tags",
  "blobs_url": "https://api.github.com/repos/judyj/Magic-of-APIs/git/blobs{/sha}",
  "git_tags_url": "https://api.github.com/repos/judyj/Magic-of-APIs/git/tags{/sha}",
  "git_refs_url": "https://api.github.com/repos/judyj/Magic-of-APIs/git/refs{/sha}",
  "trees_url": "https://api.github.com/repos/judyj/Magic-of-APIs/git/trees{/sha}",
  "statuses_url": "https://api.github.com/repos/judyj/Magic-of-APIs/statuses/{sha}",
  "languages_url": "https://api.github.com/repos/judyj/Magic-of-APIs/languages",
  "stargazers_url": "https://api.github.com/repos/judyj/Magic-of-APIs/stargazers",
  "contributors_url": "https://api.github.com/repos/judyj/Magic-of-APIs/contributors",
  "subscribers_url": "https://api.github.com/repos/judyj/Magic-of-APIs/subscribers",
  "subscription_url": "https://api.github.com/repos/judyj/Magic-of-APIs/subscription",
  "commits_url": "https://api.github.com/repos/judyj/Magic-of-APIs/commits{/sha}",
  "git_commits_url": "https://api.github.com/repos/judyj/Magic-of-APIs/git/commits{/sha}",
  "comments_url": "https://api.github.com/repos/judyj/Magic-of-APIs/comments{/number}",
  "issue_comment_url": "https://api.github.com/repos/judyj/Magic-of-APIs/issues/comments{/number}",
  "contents_url": "https://api.github.com/repos/judyj/Magic-of-APIs/contents/{+path}",
  "compare_url": "https://api.github.com/repos/judyj/Magic-of-APIs/compare/{base}...{head}",
  "merges_url": "https://api.github.com/repos/judyj/Magic-of-APIs/merges",
  "archive_url": "https://api.github.com/repos/judyj/Magic-of-APIs/{archive_format}{/ref}",
  "downloads_url": "https://api.github.com/repos/judyj/Magic-of-APIs/downloads",
  "issues_url": "https://api.github.com/repos/judyj/Magic-of-APIs/issues{/number}",
  "pulls_url": "https://api.github.com/repos/judyj/Magic-of-APIs/pulls{/number}",
  "milestones_url": "https://api.github.com/repos/judyj/Magic-of-APIs/milestones{/number}",
  "notifications_url": "https://api.github.com/repos/judyj/Magic-of-APIs/notifications{?since,all,participating}",
  "labels_url": "https://api.github.com/repos/judyj/Magic-of-APIs/labels{/name}",
  "releases_url": "https://api.github.com/repos/judyj/Magic-of-APIs/releases{/id}",
  "deployments_url": "https://api.github.com/repos/judyj/Magic-of-APIs/deployments",
  "created_at": "2018-09-07T09:10:22Z",
  "updated_at": "2018-10-02T00:34:14Z",
  "pushed_at": "2018-09-12T00:50:06Z",
  "git_url": "git://github.com/judyj/Magic-of-APIs.git",
  "ssh_url": "git@github.com:judyj/Magic-of-APIs.git",
  "clone_url": "https://github.com/judyj/Magic-of-APIs.git",
  "svn_url": "https://github.com/judyj/Magic-of-APIs",
  "homepage": null,
  "size": 10,
  "stargazers_count": 2,
  "watchers_count": 2,
  "language": "Ruby",
  "has_issues": true,
  "has_projects": true,
  "has_downloads": true,
  "has_wiki": true,
  "has_pages": false,
  "forks_count": 0,
  "mirror_url": null,
  "archived": false,
  "open_issues_count": 2,
  "license": null,
  "forks": 0,
  "open_issues": 2,
  "watchers": 2,
  "default_branch": "master",
  "network_count": 0,
  "subscribers_count": 1
}

What’s JSON? Here’s another quick wiki definition for you:

“In computing, JavaScript Object Notation or JSON is an open-standard file format that uses human-readable text to transmit data objects consisting of attribute–value pairs and array data types (or any other serializable value). It is a very common data format used for asynchronous browser–server communication.”

(There is a cool viewer that may help you understand the JSON data here: https://codebeautify.org/jsonviewer - copy and paste your page of gobbledygook and you can see your data in views you may find easier to understand…)

Now, from a developer’s perspective, we can use the curl function here to show how data can be sent to you or to a file.

$ curl https://api.github.com/repos/judyj/Magic-of-APIs > my_stats_file

You can use the grep function to find the data you are looking for, for example

$ curl https://api.github.com/repos/judyj/Magic-of-APIs | grep languages
% Total    % Received % Xferd  Average Speed   Time    Time     Time Current                                 Dload  Upload   Total   Spent    Left  Speed
100  5216  100  5216    0    0   3070      0  0:00:01  0:00:01 --:--:--  3068
"languages_url": "https://api.github.com/repos/judyj/Magic-of-APIs/languages",
"language": "Ruby",

…will give you the language of the repo (plus some downloading stats, which we will ignore from now on)

Or if we look for name here, we’ll find several values that contain the string ‘name’

$curl https://api.github.com/repos/judyj/Magic-of-APIs | grep name
  "name": "Magic-of-APIs",
  "full_name": "judyj/Magic-of-APIs",
  "labels_url": "https://api.github.com/repos/judyj/Magic-of-APIs/labels{/name}",

…but we still can figure out that the repo’s name is Magic-of-APIs

Another cool tool available on some distributions is jq. You may need to

$ sudo yum install jq

(note alternatives to yum, depending on your distro, may be apt-get, dnf, zypper etc) jq will put your JSON in a readable format. The simplest way to use it is with the -r flag and a . before the field you are looking for - this requires a little more knowledge of what your fields are, as well as some manipulation of arrays and complicated data structures, but for something simple, it’s an effective way to extract the value with no formatting. There are also many more customization options.

$ curl https://api.github.com/repos/judyj/Magic-of-APIs | jq -r .name
Magic-of-APIs

…will give you the name of the repo.

Moving on, let’s say we want to see if we have any issues, so we can do a curl and a grep

$ curl https://api.github.com/repos/judyj/Magic-of-APIs | grep issues
  "Issue_events_url":     "https://api.github.com/repos/judyj/Magic-of-APIs/issues/events{/number}",
  "issue_comment_url": "https://api.github.com/repos/judyj/Magic-of-APIs/issues/comments{/number}",
  "issues_url": "https://api.github.com/repos/judyj/Magic-of-APIs/issues{/number}",
  "has_issues": true,
  "open_issues_count": 2,
  "open_issues": 2,

…we learn not only that we have two issues, but it tells us where to learn more about them. Notice that our output contained other URLs – let’s try to get more information on this one,

"Issues_url": "https://api.github.com/repos/judyj/Magic-of-APIs/issues{/number}",

So let’s peel the layers: https://api.github.com/repos/judyj/Magic-of-APIs/issues shows you all the issues.

We can grep to see their titles:

$ curl https://api.github.com/repos/judyj/Magic-of-APIs/issues | grep title
"title": "try to do the same in python code",
"title": "Create a blog centered around how we use the api with this GitHub",

Then, https://api.github.com/repos/judyj/Magic-of-APIs/issues/2 will get you just the second issue - and what is it?

$ curl https://api.github.com/repos/judyj/Magic-of-APIs/issues/2 | grep title
"title": "Write some sample ruby code to use the api and create a csv",

Later I’ll be discussing this Ruby code - allowing you to save this type of data in a CSV file.

Here is a hands-on way to show you how we can use the GitHub API from the browser to see data on a repository:

  1. Open a page with this repo: https://github.com/judyj/Magic-of-APIs Screenshot of the 'Magic of APIs' GitHub repo
  2. Open another browser with https://api.github.com/repos/judyj/Magic-of-APIs/stargazers - note the list of people who have starred this Screenshot of the list of people who starred the 'Magic of APIs' repo
  3. or run:
$ curl -s https://api.github.com/repos/judyj/Magic-of-APIs/stargazers | grep login
    "login": "Fabercloudware",
    "login": "judyj",
  1. Now, If you have a GitHub account, go back to the repo and click the “star” button on the top where it says “Watch/Star/Fork” (if you do not have a GitHub account, you should create one, but not for me, for your resume!). Notice how on the page, the “star” button changes to “unstar” and the number beside it increases.
  2. Now click back on the https://api.github.com/repos/judyj/Magic-of-APIs/stargazers link and hit refresh. You’ll see your own GitHub info added - so we not only know that there was a new star, but we know who starred it.
  3. Or run the curl again:
$ curl -s https://api.github.com/repos/judyj/Magic-of-APIs/stargazers | grep login
    "login": "Fabercloudware",
    "login": "judyj",
    "login": "DearReader",
  1. (You can unstar the repo now, my feelings will not be hurt…)

In the following example, I will show how easy it is to use a script to extract, display, or save the data you are interested in.

My sample code will be in Ruby. I start by installing the two gems I need to get the data properly:

$ sudo yum install ruby-devel  # allows you to install the gems
$ sudo yum install gcc         # you may need this for the gems
$ sudo yum install gcc-c++     # this too
$ gem install rest-client      # allows it to use the rest calls
$ gem install json             # allows it to decode the json

You can see we installed the REST client - what does that mean? REST stands for Representational state transfer - here’s what Wikipedia says:

“Web Services that conform to the REST architectural style, or RESTful web services, provide interoperability between computer systems on the Internet. REST-compliant web services allow the requesting systems to access and manipulate textual representations of web resources by using a uniform and predefined set of stateless operations.” In my own words, I’d say that we use it as an interface to a web page in a standardized format (JSON) to get results that we can easily parse into what we need.

My sample code is in this repo: https://github.com/judyj/Magic-of-APIs/blob/master/github_api_sample.rb

Or you can get it by cloning

$ git clone https://github.com/judyj/Magic-of-APIs.git
$ cd Magic-of-APIs/

Run it by typing

$ ruby github_api_sample.rb

Here’s how it works:

If I make this rest call:

command = "https://api.github.com/repos/#{username}/#{reponame}"

It translates to https://api.github.com/repos/judyj/Magic-of-APIs - we saw this page earlier. Then I parse out the data I need:

call_result = JSON.parse(RestClient.get(command))
repo_name = call_result['name']

That gives you the attribute called ‘name’ that we parsed out from the JSON results of the API call, which we already know is “Magic-of-APIs” …and that’s it, we got the repo’s name - pretty simple.

If we are looking at an array, it gets a little more complicated - let’s list my issues:

command = "https://api.github.com/repos/#{username}/#{reponame}/issues"
issues_result = JSON.parse(RestClient.get(command))
  num_issues = issues_result.length  # how many in the array?
  issue_count = 0
  if num_issues > 0
    issues_result.each do |issues|    # loop through them!
      issue_count += 1
      issue = issues['title']
      puts "issue #{issue_count}-#{issue}"
    end
  end

This gets the array of issues, parses them out, and displays the titles one by one. Now I can take that text and display it, but I can also write it to a text file or even a comma-separated spreadsheet (because I’ll want to fix all those issues right away…)

This code gives you some of the building blocks to use GitHub’s API, and you can expand these tools to any API that is out there.

Another cool thing about using APIs is that you can mix and match tools. Let’s say you have Jira tickets that are related to GitHub repositories -you can use the API to create a Jira ticket from each issue in GitHub!

So there you have it… you see a pretty GUI and think, “looks nice, but I want DATA!” Just use the API to customize the data to your taste.

Personal anecdote: While in the editing phase of this blog post, I ran my Ruby script that (as described in the first paragraph) extracts and manipulates data to create a custom report for my customer. A friend on another team asked me to “do that magic you do” to create her report (“it helps me A LOT!”) - another API success story.

Here are a couple more cool APIs to practice on:

(User note: the GitHub API has a rate limit which means if you call it too many times in a short period, you’ll get cut off. This may affect you if you play here too much. Once you have created your account, however, you can get it authenticated and use that authentication for calls to increase that limit. There is an example of this commented out in my sample code.)

Click here to learn more about how Onyx Point, Inc's professional services and development teams can help you.

Judy has been a software engineer for over thirty-five years, and has been at Onyx Point since 2015. She has also functioned as a System Engineer, Project Manager, ScrumMaster, and a record store clerk. She was lucky enough to have started programming in the 19XXes when her Dad brought home a PDP-8 - she eventually progressed from paper-tape and punched cards to more modern computing systems. When not at work, Judy can be found baking yummy treats for family, friends, and coworkers; attending hockey games and rock concerts; or trying to finish a good book. Judy also loves to volunteer, especially in events that promote diversity in technology. Proof of her dedication to this cause is the fact that both of her awesome daughters are engineers.

At Onyx Point, our engineers focus on Security, System Administration, Automation, Dataflow, and DevOps consulting for government and commercial clients. We offer professional services for Puppet, RedHat, SIMP, NiFi, GitLab, and the other solutions in place that keep your systems running securely and efficiently. We offer Open Source Software support and Engineering and Consulting services through GSA IT Schedule 70. As Open Source contributors and advocates, we encourage the use of FOSS products in Government as part of an overarching IT Efficiencies plan to reduce ongoing IT expenditures attributed to software licensing. Our support and contributions to Open Source, are just one of our many guiding principles

  • Customer First.
  • Security in All We Do.
  • Pursue Innovation with Integrity.
  • Communicate Openly and Respectfully.
  • Offer Your Talents, and Appreciate the Talents of Others

api, ruby, git, github, programming, automation, json, rest

Share this story

We work with these Technologies + Partners

puppet
gitlab
simp
beaker
redhat
`
AFCEA
GitHub
FOSSFeb