Tidying and manipulating data using the tidyverse

<!---- SLIDES SAVED TO PDF USING:

decktape remark "slides/intro-to-data-analysis-R.html" slides/intro-to-data-analysis.pdf

using node.js

---->

# Tidying and manipulating data using the tidyverse

**Dr. Calum Webb**<br>
Sheffield Methods Institute, the University of Sheffield<br>
[c.j.webb@sheffield.ac.uk](mailto:c.j.webb@sheffield.ac.uk)

<div>
<style type="text/css">.xaringan-extra-logo {
width: 180px;
height: 128px;
z-index: 0;
background-image: url(header/smi-logo-white.png);
background-size: contain;
background-repeat: no-repeat;
position: absolute;
top:1em;right:2em;
}
</style>
<script>(function () {
  let tries = 0
  function addLogo () {
    if (typeof slideshow === 'undefined') {
      tries += 1
      if (tries < 10) {
        setTimeout(addLogo, 100)
      }
    } else {
      document.querySelectorAll('.remark-slide-content:not(.inverse):not(.hide_logo)')
        .forEach(function (slide) {
          const logo = document.createElement('div')
          logo.classList = 'xaringan-extra-logo'
          logo.href = null
          slide.appendChild(logo)
        })
    }
  }
  document.addEventListener('DOMContentLoaded', addLogo)
})()</script>
</div>

---

## This training course is designed to be hands-on. We'll be spending most of our time working through real applications of the data tidying tools that make up the tidyverse.

---

# Data tidying requires the use of multiple tools

* The idea of today is to introduce you to all the tools that will allow you to tidy any untidy dataset.
* You won't be able to use all of them perfectly right from the start.
* But if you invest time into learning to use them, you can become very proficient in data tidying.

]

]

---

# Why spend time learning how to tidy data?

* Tidying data and preparing it for analysis or visualisation is often the most time consuming part of any quantitative research project.
* Tidying data is not often reproducable *unless it has been tidied programatically*.

]

]

---

## Introduction: What is tidy data?

---

background-color: white

.footnote[
.right[Illustrations from the [Openscapes](https://www.openscapes.org/) blog [Tidy Data for reproducibility, efficiency, and collaboration](https://www.openscapes.org/blog/2020/10/12/tidy-data/) by Julia Lowndes and Allison Horst] 
]

---

background-color: white

---

background-color: white

---

background-color: white

---

## 1. Reading data from different data sources and select/rename columns.

* Reading data from Stata, SPSS, and Excel files
* Filtering variables using select()
* Generalised select()

---

background-color: white

<br>

]

---

background-color: white

<br>

]

---

## 2. Creating new variables using mutate()

* Creating new variables that are transformations of existing variables
* Recoding categorical variables using case_when()
* Extracting numbers with parse_number()
* Performing repeated/generalised transformations

---

background-color: white

<br>

]

---

background-color: white

<br>

]

---

background-color: white

<br>

]

---

## 3. Aggregating data to higher levels with group_by()

* Creating new, aggregated datasets using group_by() and summarise()
* Adding group-level variables for multilevel models using group_by() and mutate()

---

background-color: white

]

.footnote[
.right[[Animation by Andrew Heiss](https://www.andrewheiss.com/blog/2024/04/04/group_by-summarize-ungroup-animations/)]

]

---

## 4. Pivoting data between wide and long formats

* Converting wide datasets suitable for Latent Growth Structural Equation Modelling to long datasets suitable for multilevel modelling.
* ... and the reverse.

---

background-color: white

.footnote[
.right[[Animation by Garrick Aden-Buie](https://www.garrickadenbuie.com/project/tidyexplain/)]

]

---

background-color: white

<br>

.footnote[
.right[[Animation by Garrick Aden-Buie](https://www.garrickadenbuie.com/project/tidyexplain/)]

]

---

## 5. Working with strings and a little bit of regex

* How to remove certain characters or strings from character type variables (especially footnotes).
* Extracting subsets of characters from longer strings.
* Splitting variables into multiple columns based on a character within a string.

---

background-color: white

<br>

]

---

background-color: white

<br>

]

---

## 6. Joining relational datasets

* Joining datasets based on a shared key
* Joining datasets together based on a combination of variables that form a key
* Joining higher level data to lower level data
* Checking for missing observations with anti_join()

Key: A value, usually a string, that uniquely identifies each observation across multiple related (relational) datasets

---

background-color: white

<br>

.footnote[
.right[[Animation by Garrick Aden-Buie](https://www.garrickadenbuie.com/project/tidyexplain/)]

]

---

background-color: white

<br>

]

---

## 7. Working with dates

* How to fix how R interprets dates when they aren't in YYYY-MM-DD format.

---

background-color: white

<br>

]

---

## 8. Filtering rows of observations

* How to filter data based on values in character/factor type variables
* How to filter data based on numeric type variables
* How to filter data based on dates

---

background-color: white

<br>

]

---

# You now have all the tools — all that's left is the practice.

* In this course I've tried to give you, in as short a time as possible, information and practical examples of how to use all of the tools I've picked up over more than 10 years of using R.

]

]

---

# You now have all the tools — all that's left is the practice.

* If you keep practicing tidying untidy datasets, using these tools will eventually become effortless. Untidy data becomes a puzzle to solve. But when you're just starting out, the puzzles will be frustrating.

]

]

---

background-color: white

<br>

]