Chapter Summaries

Data Summaries

This chapter covers the foundational steps of importing data into R and computing summary statistics using the tidyverse. The below summary was created in collaboration with Claude.

Key Concepts:

• CSV Files: Comma-separated value files are a common format for storing tabular data. Each row is a line of text, and columns are separated by commas.
• Data Frames and Tibbles: The primary data structures for storing tabular data in R. Tibbles are the tidyverse version of data frames, with slightly improved printing and behavior.
• The $ Operator: Used to access a single column from a data frame by name, e.g., data$column_name. Returns a vector.
• Pipes (%>%): A way to chain functions together, passing the output of one function as the input to the next. Read as “and then.” For example, data %>% summary() is equivalent to summary(data).

Key Functions for Importing Data:

• read_csv():
  • Reads a CSV file into a tibble.
  • Syntax: read_csv(file)
    • file: Path to the CSV file (relative to your R project folder, or an absolute path).
  • Example: dat <- read_csv("data/screentime.csv")

Key Functions for Summarizing Data:

• summary():
  • Provides a quick overview of a data frame, including min, max, mean, and quartiles for numeric columns, and counts for categorical columns.
  • Syntax: summary(data)
• count():
  • Counts the number of observations for each unique value of one or more variables.
  • Syntax: count(data, variable)
  • Example: data %>% count(gender) returns a table with each gender and its frequency.
• summarise() (or summarize()):
  • Computes summary statistics, collapsing a data frame into a single row (or one row per group, if used with group_by()).
  • Syntax: summarise(data, new_name = function(column))
  • Common summary functions: mean(), sd(), min(), max(), median(), n(), sum()
  • Example: data %>% summarise(avg_age = mean(age), n = n())
• group_by():
  • Groups a data frame by one or more variables, so that subsequent operations (like summarise()) are applied within each group.
  • Syntax: group_by(data, variable)
  • Example: data %>% group_by(gender) %>% summarise(avg_age = mean(age))
  • Use ungroup() to remove grouping when it is no longer needed.

Key Functions for Inspecting Data:

• glimpse(): Provides a transposed overview of a data frame, showing column names, types, and the first few values.
• str(): Displays the internal structure of an R object, including data types and dimensions.
• head() and tail(): Show the first or last few rows of a data frame.
• nrow() and ncol(): Return the number of rows or columns.
• names(): Returns the column names of a data frame.

Practical Applications:

• Loading data from external files into R for analysis.
• Getting a quick overview of dataset contents and structure.
• Computing descriptive statistics (means, counts, standard deviations) overall or by group.

Key Takeaways:

• read_csv() is the standard way to import CSV data in the tidyverse.
• The $ operator and pipes (%>%) are essential tools for accessing and manipulating data.
• count() and summarise() are the workhorses for computing data summaries.
• group_by() combined with summarise() is a powerful pattern for computing statistics broken down by category.

Data Wrangling

This chapter covers the core dplyr functions for transforming and manipulating data within a data frame. The below summary was created in collaboration with Claude.

Key Concepts:

• Data Wrangling: The process of transforming raw data into a more useful format for analysis. This includes selecting, filtering, reordering, and creating new variables.
• The Pipe (%>%): Central to data wrangling workflows. Pipes allow you to chain multiple operations together in a readable, step-by-step fashion, rather than nesting functions inside one another.
• Missing Values (NA): R’s representation of missing or undefined data. Many functions require special handling of NA values, e.g., mean(x, na.rm = TRUE).

Key Functions from dplyr:

• select():
  • Chooses which columns to keep (or remove) from a data frame.
  • Syntax: select(data, column1, column2, ...)
  • Use - to exclude columns: select(data, -column_to_remove)
  • Helper functions: starts_with(), ends_with(), contains(), num_range()
  • Example: data %>% select(id, age, gender) keeps only those three columns.
• filter():
  • Keeps only the rows that meet a specified condition.
  • Syntax: filter(data, condition)
  • Common operators: ==, !=, >, <, >=, <=, & (and), | (or), ! (not)
  • Example: data %>% filter(age > 18 & gender == "female")
  • Use %in% to match against multiple values: filter(data, region %in% c("North", "South"))
• arrange():
  • Reorders rows based on the values of one or more columns.
  • Syntax: arrange(data, column)
  • Use desc() for descending order: arrange(data, desc(age))
  • Example: data %>% arrange(age) sorts from youngest to oldest.
• mutate():
  • Creates new columns or modifies existing ones.
  • Syntax: mutate(data, new_column = expression)
  • The expression can use existing columns, arithmetic operators, and functions.
  • Example: data %>% mutate(age_months = age * 12)
  • Multiple new columns can be created in a single call: mutate(data, col1 = expr1, col2 = expr2)
• summarise() and group_by():
  • Covered in the Data Summaries chapter. In the context of data wrangling, these functions are often combined with the above to create complete data processing pipelines.
  • Example: data %>% filter(!is.na(score)) %>% group_by(condition) %>% summarise(m = mean(score))

Handling Missing Values:

• is.na(): Tests whether values are NA. Returns TRUE or FALSE.
• na.rm = TRUE: An argument available in many summary functions (e.g., mean(), sum(), sd()) that tells R to ignore NA values when computing the result.
• drop_na(): Removes rows with NA values (from tidyr).
• Filtering out NA values: data %>% filter(!is.na(column)) keeps only rows where the specified column is not missing.

Putting It All Together:

A typical data wrangling pipeline chains multiple operations:

data %>%
  select(id, condition, score) %>%
  filter(!is.na(score)) %>%
  mutate(score_centered = score - mean(score)) %>%
  group_by(condition) %>%
  summarise(mean_score = mean(score_centered))

Key Takeaways:

• The six core dplyr verbs (select, filter, arrange, mutate, summarise, group_by) cover the vast majority of data manipulation tasks.
• Pipes (%>%) make it possible to build readable, step-by-step data processing workflows.
• Handling missing values is an essential part of data wrangling; always check for and address NA values before computing statistics.
• These functions can be combined in flexible ways to answer a wide range of data analysis questions.

Data Tidying

This chapter focuses on the principles of tidy data and how to reshape data using tidyr. The below summary was created in collaboration with Claude.

Key Concepts:

• Tidy Data: A standard way of structuring datasets that makes them easier to manipulate and analyze.
• Principles of Tidy Data:
  • Each variable forms a column.
  • Each observation forms a row.
  • Each type of observational unit forms a table.
• Reshaping Data: Transforming data from a wide format (multiple variables in columns) to a long format (multiple observations in rows), or vice versa.

Key Functions from tidyr:

• pivot_longer():

  • Converts data from wide to long format.
  • Syntax: pivot_longer(data, cols, names_to, values_to)
    • cols: Columns to pivot.
    • names_to: Name of the new column that will contain the pivoted column names.
    • values_to: Name of the new column that will contain the pivoted values.
  • Example: Transforming columns representing measurements at different time points into rows.

• pivot_wider():

  • Converts data from long to wide format.
  • Syntax: pivot_wider(data, names_from, values_from)
    • names_from: Column whose values will become column names.
    • values_from: Column whose values will populate the new columns.
  • Example: Transforming rows representing different categories into columns.

• separate():

  • Splits a single column into multiple columns.
  • Syntax: separate(data, col, into, sep)
    • col: Column to separate.
    • into: Vector of new column names.
    • sep: Separator character or regular expression.
  • Example: Splitting a date column into year, month, and day columns.

• drop_na():

  • Removes rows with missing values (NA).
  • Syntax: drop_na(data, ...)
  • ...: Optional columns to check for NAs. If empty, checks all columns.

Practical Applications:

• Preparing data for analysis and visualization.
• Standardizing data formats for consistency.
• Reshaping data to meet the requirements of specific functions or packages.

Key Takeaways:

• Tidy data is essential for efficient data analysis.
• tidyr provides powerful tools for reshaping data.
• Understanding pivot_longer() and pivot_wider() is crucial for transforming data between wide and long formats.
• drop_na() is useful for cleaning data.

Additional Concepts Related to Processing fMRI Data:

• NIfTI file: A common file format for storing MRI data.
• Anatomical image: An image that maps out the anatomical structures of the brain due to its sensitivity to differences between tissue types, e.g., gray and white matter. Typically only one anatomical image is collected, with high spatial resolution to visualize brain structures.
• Functional image: An image that measures the BOLD (blood oxygen level dependent) signal, which is used as a proxy for brain activity. Typically many functional images are collected to capture how the BOLD signal changes over time, e.g., while a participant completes a cognitive task. Functional images are typically lower in spatial resolution than the anatomical image.
• Voxel: The spatial unit of measurement for MRI, i.e., a volumetric (3-D) pixel. The smaller the voxel, the higher the resolution and the clearer the image.
• Time-point: Related to the temporal unit of measurement for functional MRI. We collect many functional images, one per time-point.
• Time-series: The ordered collection of time-points, reflecting the time-course of the BOLD signal.

Data Relations

The chapter focuses on combining data from multiple tables (data frames) using dplyr’s join functions. The below summary was created in collaboration with Claude.

Key Concepts:

• Relational Data: Data is often stored in multiple tables that are related to each other.
• Key Variables: Columns that are common between tables and used to link them.
• Joining: The process of combining data from multiple tables based on key variables.

Types of Joins:

• inner_join():
  • Returns only rows where the key variable(s) match in both tables.
  • Syntax: inner_join(x, y, by = "common_column")
  • Example: Combining customer orders with customer details, only for customers who have placed orders.
• left_join():
  • Returns all rows from the left table (x) and matching rows from the right table (y).
  • If a row in x has no match in y, NA values are added for y’s columns.
  • Syntax: left_join(x, y, by = "common_column")
  • Example: Adding information about departments to a list of employees, even if some departments have no employees.
• right_join():
  • Returns all rows from the right table (y) and matching rows from the left table (x).
  • If a row in y has no match in x, NA values are added for x’s columns.
  • Syntax: right_join(x, y, by = "common_column")
• full_join():
  • Returns all rows from both tables.
  • NA values are added where there are no matches.
  • Syntax: full_join(x, y, by = "common_column")
  • Example: Combining customer data from two different databases, including all customers from both.

Specifying Key Variables:

• by Argument:
  • Specifies the key variable(s).
  • If the key variables have the same name in both tables, use by = "column_name".
  • If the key variables have different names, use by = c("x_column" = "y_column").

Handling Duplicate Column Names:

• If non-key columns have the same name in both tables, dplyr will add suffixes (.x and .y) to distinguish them. You can change the suffixes by using the argument suffix = c(".x", ".y").

Row and Column Binding:

• bind_rows():
  • Combines data frames vertically (adds rows).
  • Syntax: bind_rows(df1, df2, ...)
  • Requirement: Data frames must have the same columns (or columns that can be matched by name).
  • Example: Merging daily sales data from multiple files.
  • If columns do not match, then NA values will be placed into the combined dataframe.
• bind_cols():
  • Combines data frames horizontally (adds columns).
  • Syntax: bind_cols(df1, df2, ...)
  • Requirement: Data frames must have the same number of rows.
  • Caveat: You will often want to use a join function instead of bind_cols.

Set Operations:

• intersect():
  • Returns the rows that are present in both data frames.
  • Syntax: intersect(df1, df2)
  • Example: Finding common customers between two lists.
• union():
  • Returns all unique rows from both data frames. This is similar to bind_rows except it removes duplicates.
  • Syntax: union(df1, df2)
  • Example: Combining customer lists from different sources, removing duplicates.
• setdiff():
  • Returns the rows that are present in the first data frame but not in the second.
  • Syntax: setdiff(df1, df2)
  • Example: Finding all customers that are in the first customer list, but not the second.

Key Takeaways:

• Joins are essential for combining related data from multiple tables.
• Joins work with only two tables at a time. To combine multiple tables, use a pipe to connect multiple calls to a join function.
• The by argument is crucial for specifying the key variables used to link tables.