More data wrangling and visualization

Reshape and combine tables with summarise, joins, and pivots; visualize distributions and layered raw-plus-summary plots.

Slides

Make slides full screen

Exercise 1: Reshaping and combining tables

Mode setup. The room is split into Group A and Group B. For Exercise 1, Group A works agentic — Claude Code reads your notebook, writes code, and runs it. Group B works manual — you write each step yourself; you may ask a chat LLM conceptual questions, but it does not act in your environment. Exercise 2 flips the modes.

Open notebooks/04-wrangling.qmd in your Codespace. The wrapper question for both exercises is: how does vaginal pH change across the study, and does it differ between arms or by participant characteristics?

Part 1 — Predict (no code)

For each pipeline below, write — in plain words, before running anything — how many rows the result has and what one row represents.

  1. table_02 |> group_by(pid) |> summarise(mean_ph = mean(ph))
  2. table_02 |> group_by(arm, time_point) |> summarise(mean_ph = mean(ph))
  3. per_person |> left_join(table_01, by = "pid"), where per_person is the 44-row result from prediction 1.
  4. arm_timepoint |> pivot_wider(names_from = time_point, values_from = mean_ph), where arm_timepoint is the 6-row result from prediction 2.

We’ll briefly compare answers across the room before moving to code.

Part 2 — Build the pipeline

Build one growing pipeline on table_02, one verb per step.

  1. Pipe table_02 into head() — remind yourself of the starting shape (132 rows).
  2. group_by(pid) |> summarise(mean_ph = mean(ph), mean_nugent = mean(nugent_score)) — collapse visits per participant.
  3. Add |> left_join(table_01, by = "pid") — attach demographics.
  4. Separately, table_02 |> group_by(arm, time_point) |> summarise(mean_ph = mean(ph)) — collapse to arm-timepoint.
  5. Add |> pivot_wider(names_from = time_point, values_from = mean_ph) — compare time points side-by-side.
  6. Separately, table_02 |> pivot_longer(c(ph, nugent_score, crp_blood), names_to = "measurement", values_to = "value") — long-format setup that Exercise 2 will use.

Part 3 — Reach (optional)

Use semi_join to filter the per-participant summary to smokers only:

smokers <- table_01 |> filter(smoker == "smoker")
per_person |> semi_join(smokers, by = "pid")

Then group_by(arm) |> summarise(...) on the result to compare per-arm pH means among smokers only.

Discussion

Once the room is back together:

  • Which step’s result surprised you most? Was your prediction right?
  • Where did your mode help most? Where did it slow you down?

Exercise 2: Visualizing distributions and summaries

Mode flip. If you were agentic in Exercise 1, you are manual now; if you were manual, you are agentic. Same definitions as before — manual means you may ask a chat LLM conceptual questions but it does not touch your notebook.

Open notebooks/04-visualization.qmd in your Codespace.

Part 1 — Read three plots (no code)

The notebook shows three pre-built plots of ph by arm: a histogram faceted by arm, a boxplot, and a quasirandom. For each plot, answer in plain words:

  1. What is the mark?
  2. What property of the mark comes from what column?
  3. How many rows of data made each mark?

The three plots illustrate the spectrum — many rows per bar (histogram), many rows per box (boxplot), one row per dot (quasirandom). Same underlying data, three different answers to question 3.

Then sketch (paper or text) a plot that would answer does pH differ between arms? using a layered approach — raw observations underneath a summary mark.

Part 2 — Build the plot

Build the viz pieces, growing toward a layered final figure.

  1. ggplot(table_02, aes(ph)) + geom_histogram(), then add + facet_wrap(~ arm).
  2. ggplot(table_02, aes(arm, ph)) + geom_boxplot().
  3. Swap geom_boxplot() for geom_quasirandom().
  4. Swap for geom_violin().
  5. Compute ph_summary <- table_02 |> group_by(arm) |> summarise(mean_ph = mean(ph), sd_ph = sd(ph)). Then plot ggplot(ph_summary, aes(arm, mean_ph)) + geom_pointrange(aes(ymin = mean_ph - sd_ph, ymax = mean_ph + sd_ph)).
  6. Layered: combine geom_quasirandom(data = table_02, ...) with geom_pointrange(data = ph_summary, ...) in one plot.
  7. From Exercise 1’s pivot_longer step, facet by measurement: long_table |> ggplot(aes(arm, value)) + geom_boxplot() + facet_wrap(~ measurement, scales = "free_y").

Part 3 — Reach (optional)

Build the polished final figure: the layered plot from step 6, faceted by time_point (so we see the layered comparison at each visit), with time_point panels in chronological order (baseline, week_1, week_7), axis labels, and a clean theme.

Optional add-on: density overlay on the histogram from step 1: geom_histogram(aes(y = after_stat(density))) + geom_density().

Discussion

Once the room is back together:

  • What was easy in your mode for the plotting work? What was hard?
  • Of the two exercises you’ve now done, one each way — which mode felt better for which kind of task?
  • Where did the assistant misread what you wanted? How did you catch it?