class: center, middle, inverse, title-slide .title[ # Github API and status ] .subtitle[ ## Using <code>httr</code>, <code>jsonlite</code> and <code>kable</code> ] .author[ ### Christine Stawitz <br>NOAA Fisheries, OST ] .date[ ### 2022-07-18 ] --- layout: true .footnote[U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service] --- # Introduction to REST APIs * [More info](https://restfulapi.net/) * Conforms to the REST protocol and application which fulfils this is called RESTful * GET, POST, PUT, DELETE * We will focus on GET which returns data in the form of a JSON nested list. <img src="data:image/png;base64,#assets/git.png" width="100" /> --- ## What is your experience with the REST API? Poll: PollEv.com/christinestawitznoaafederal561 .hand-blue[ <iframe src="https://pollev-embeds.com/multiple_choice_polls/4t0K0voWl7KZQPsjrE2zV/respond" width="800px" height="600px"></iframe> ] --- # Github * Projects on Github return API data about repositories and/or organizations * Inbuilt tools in `R` to access, format, and present this data * There is also the `{gh}` package which is more minimal but may meet your needs: https://gh.r-lib.org/ --- class: middle # Putting it all together <!-- --> --- ## How it works - getting the data (REST API using the `{httr}` package) - filtering and grouping the data (JSON output using `{jsonlite} ` package) - presenting the data (`{kable}` and `{kableExtra}` packages) --- ## Getting the data More info on Github REST API: https://docs.github.com/en/rest/overview/resources-in-the-rest-api To get the data, you will first need to allow access to your API via Github. Open your profile -> Settings -> Developer Settings Under "Github Apps" make sure your profile has access <!-- --> --- ## Creating a token Click "Oauth apps" and follow these instructions to create a new Oauth app. https://docs.github.com/en/developers/apps/building-oauth-apps/creating-an-oauth-app You can use http://localhost:1410 as the port. While filling out the form, you will see two hexadecimal codes - one is your Secret and the other is the Key. COPY THESE DOWN. You will not be able to see them again. --- # Using `{httr}` once you get the code https://nmfs-fish-tools.github.io/projectstats/ This is an example from Hadley Wickham's `{httr}` documentation so the key and secret are published on the web. ```r remotes::install_github("nmfs-fish-tools/projectstats") ``` ``` ## Skipping install of 'projectstats' from a github remote, the SHA1 (c3facbf4) has not changed since last install. ## Use `force = TRUE` to force installation ``` ```r require(projectstats) httr_table <- get_issues(appname_ = "github", key_ = "56b637a5baffac62cad9", secret_ = "8e107541ae1791259e9987d544ca568633da2ebf", repo_name = "r-lib/httr") ``` --- # Privacy You should not publish your key & secret to Github or within shared code! Keep the key & secret stored locally. You can hide it in your operating system's credential store using the `{keyring}` package if needed. ```r require(keyring) keyring::key_set_with_value("httr_secret", "8e107541ae1791259e9987d544ca568633da2ebf", username = "chris") keyring::key_set_with_value("httr_key", "56b637a5baffac62cad9", username = "chris") secret <- keyring::key_get("httr_secret", "chris") key <- keyring::key_get("httr_key", "chris") ``` --- # Pulling relevant information out of the JSON The JSON file you get back from Github is large and duplicative, so I wrote the `create_issue_table` function to extract a number of key column headers you might be interested in: `number`, `title`, `labels`, `milestone`, `pull_request`, and `assignee`. Many of these are returned as nested lists with auxiliary information, e.g. the color of the label in addition to its name. Future work on the package could include customizing how the table is returned. --- # Creating html tables with `{kable}` and `{kableExtra}` `{kable}` creates basic HTML tables and `{kableExtra}` allows you to format them in certain ways. --- # Issue table ```r create_issue_table(httr_table) ``` <table> <thead> <tr> <th style="text-align:left;"> number </th> <th style="text-align:left;"> title </th> <th style="text-align:left;"> milestone </th> <th style="text-align:left;"> assignee </th> <th style="text-align:left;"> labels </th> <th style="text-align:left;"> status </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> 719 </td> <td style="text-align:left;"> Error in curl fetch memory port 443 timed out </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 719 </td> </tr> <tr> <td style="text-align:left;"> 718 </td> <td style="text-align:left;"> httr POST is failing timeout </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 718 </td> </tr> <tr> <td style="text-align:left;"> 717 </td> <td style="text-align:left;"> installation error if User file path contains an apostrophe </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 717 </td> </tr> <tr> <td style="text-align:left;"> 714 </td> <td style="text-align:left;"> Timeouts greater than 10 s are ignored </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 714 </td> </tr> <tr> <td style="text-align:left;"> 713 </td> <td style="text-align:left;"> Better documentation for modify_url() </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 713 </td> </tr> <tr> <td style="text-align:left;"> 711 </td> <td style="text-align:left;"> `httr:::body_config()` removes empty arrays from body </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 711 </td> </tr> <tr> <td style="text-align:left;"> 710 </td> <td style="text-align:left;"> PATCH is broken </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 710 </td> </tr> <tr> <td style="text-align:left;"> 709 </td> <td style="text-align:left;"> HTTR authenticate gives 401 with valid password containing £ </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 709 </td> </tr> <tr> <td style="text-align:left;"> 708 </td> <td style="text-align:left;"> "Error in if (is_http) { : argument is of length zero" in shinytest </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 708 </td> </tr> <tr> <td style="text-align:left;"> 704 </td> <td style="text-align:left;"> R/httr and R/curl much slower than download.file </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 704 </td> </tr> <tr> <td style="text-align:left;"> 703 </td> <td style="text-align:left;"> Kerberos authentication with flags </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 703 </td> </tr> <tr> <td style="text-align:left;"> 700 </td> <td style="text-align:left;"> Move `master` branch to `main` </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 700 </td> </tr> <tr> <td style="text-align:left;"> 695 </td> <td style="text-align:left;"> Control programs based on httr to print the raw requests </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 695 </td> </tr> <tr> <td style="text-align:left;"> 694 </td> <td style="text-align:left;"> httr doesn't get the 301 error ? </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 694 </td> </tr> <tr> <td style="text-align:left;"> 693 </td> <td style="text-align:left;"> Error with non-unicode character in response from POST </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 693 </td> </tr> <tr> <td style="text-align:left;"> 692 </td> <td style="text-align:left;"> RapidAPI - Status 401 </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 692 </td> </tr> <tr> <td style="text-align:left;"> 691 </td> <td style="text-align:left;"> Error in curl::curl_fetch_memory(url, handle = handle) : schannel: next InitializeSecurityContext failed: SEC_E_UNTRUSTED_ROOT (0x80090325) - The certificate chain was issued by an authority that is not trusted. </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 691 </td> </tr> <tr> <td style="text-align:left;"> 690 </td> <td style="text-align:left;"> httr::oauth2.0_token$refresh() returns "argument is of length zero" </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 690 </td> </tr> <tr> <td style="text-align:left;"> 689 </td> <td style="text-align:left;"> httr::parse_url cannot handle a missing protocol </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 689 </td> </tr> <tr> <td style="text-align:left;"> 688 </td> <td style="text-align:left;"> OAuth2.0 in shiny app </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 688 </td> </tr> <tr> <td style="text-align:left;"> 685 </td> <td style="text-align:left;"> encourage type/length stability in best practice vignette </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 685 </td> </tr> <tr> <td style="text-align:left;"> 684 </td> <td style="text-align:left;"> parse_text checking encoding more than necessary </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 684 </td> </tr> <tr> <td style="text-align:left;"> 683 </td> <td style="text-align:left;"> progress bar immediately shows 100% </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 683 </td> </tr> <tr> <td style="text-align:left;"> 680 </td> <td style="text-align:left;"> Feature Request: Expose bytes publicly </td> <td style="text-align:left;"> NA </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> </td> <td style="text-align:left;"> 680 </td> </tr> </tbody> </table> --- # Pull request table Pull requests are stored in Github as issues, so they get returned in the JSON list and will need to be filtered in or out depending on the table. `create_pr_table` will create a similar HTML table for pull requests. ```r create_pr_table(httr_table) ``` <table> <thead> <tr> <th style="text-align:right;"> number </th> <th style="text-align:left;"> title </th> <th style="text-align:right;"> assignee </th> <th style="text-align:right;"> status </th> </tr> </thead> <tbody> <tr> <td style="text-align:right;"> 706 </td> <td style="text-align:left;"> Add an escape control in `set_cookies` </td> <td style="text-align:right;"> NA </td> <td style="text-align:right;"> 706 </td> </tr> <tr> <td style="text-align:right;"> 702 </td> <td style="text-align:left;"> Fix argument value in return value docs </td> <td style="text-align:right;"> NA </td> <td style="text-align:right;"> 702 </td> </tr> <tr> <td style="text-align:right;"> 701 </td> <td style="text-align:left;"> Fixed typo in content_type() documentation </td> <td style="text-align:right;"> NA </td> <td style="text-align:right;"> 701 </td> </tr> <tr> <td style="text-align:right;"> 698 </td> <td style="text-align:left;"> Make it clear httr uses different default than jsonlite </td> <td style="text-align:right;"> NA </td> <td style="text-align:right;"> 698 </td> </tr> <tr> <td style="text-align:right;"> 696 </td> <td style="text-align:left;"> Tweak `content()` documentation </td> <td style="text-align:right;"> NA </td> <td style="text-align:right;"> 696 </td> </tr> <tr> <td style="text-align:right;"> 687 </td> <td style="text-align:left;"> Add oauth error for deleted client </td> <td style="text-align:right;"> NA </td> <td style="text-align:right;"> 687 </td> </tr> </tbody> </table> --- # Emailing :( There is a package called `{blastula}` that enables sending emails from R but I have unfortunately not yet worked out if it is possible to authenticate. Our RStudio Connect instance can also send email but I need to format our listserv to do it. In other words, stay tuned! --- # Resources * * *