Shedding light on a black box

Chris Rowan posted earlier today about a heart-stopping experience – the sudden fear that there was something wrong with his data. (He was wrong, thank goodness, but he has some interesting things to say about science’s self-correcting mechanisms... or not.)

Me... I teach. I mean, I do research as well, though for the past fourteen years it has all been collaborative with undergraduates (and therefore slow, and with many backtracks and sidetracks as student interests diverge from mine). But really, my job is to teach undergraduates to become scientists. And as a result, I’ve been worrying a lot about how to get students to think about the data they collect: What does it mean? Is it reliable – were there any problems with the way it was collected? Does it fit with their expectations?

Four years ago – actually, the month after my son was born – we (that is, a colleague and I) received an NSF grant to buy a new ICP-OES. An “inductively coupled argon plasma optical emission spectrometer.” It can measure the concentration of a lot of different elements dissolved in water (and can measure rock chemistries if you can dissolve the rock in a strong acid), which means it can be used for a variety of environmental geology and igneous petrology projects. It was our department’s first analytical instrument (well, aside from an X-ray diffractometer that was already broken when a research university donated it to us), even though the department required an undergraduate research thesis from all the majors. We noticed that students didn’t seem to think very carefully about their data when they sent samples out to a lab and received some numbers in return – they weren’t aware that each technique has error associated with it, and they treated their data as if they were infallible.

But now we’ve got an instrument in-house. My colleague is working on getting it ready to analyze major and trace elements in igneous rocks. A few students have used it for water quality data. I’ve used it to collect data for intro class projects... but it’s still a black box to our students. And it’s a bad idea to treat it that way.

So I’ve been developing a research exercise for our Earth Systems Science course. We’ve had a group research project in the past, but it hasn’t worked all that well. The students had to come up with their own idea for a project, and design their data collection, and implement their project... on their own. Perhaps a good idea for graduate students. Not so good for students who haven’t even thought about stream flow, or what minerals are found in which rocks, or how weather systems form. The projects are frustrating for the students, and it isn’t really clear that the students are learning much. So I want to ditch the old project model.

My new idea is to have all the lab sections monitor a small local river. Each lab section will be responsible for collecting data on one of eight reaches along the river, and groups within the lab will measure discharge (amount of water going down the river), sediment load (probably as turbidity, but I would like them to also think about bed load), and water chemistry (two different groups, one responsible for nitrates and nitrites and phosphates and chloride and pH and TDS, and another responsible for cations, measured with the ICP). We would monitor the same reaches every year, fall and spring, and students could compare their data with previous years, other seasons, and other reaches. I’ve got a thesis student collecting baseline data right now – and he’s also making a GIS database that includes current land use (this could change through time), the location of irrigation outflows (this isn’t an entirely natural river – human influences are significant), and surface discharges of formation water from natural gas drilling.

So, all well and good. I’m excited about having the students be part of an organized study, and about having them compare data without having to collect a thesis-worth. But that still doesn’t necessarily solve the problem of getting the students to think about the quality of the data.

So I’m doing an experiment this semester. I’m making the students do some prep work before we actually collect the data. Last week each group had to give me a short background write-up – what data were they collecting, and how would it be measured, and what factors would control whether it would be high or low. This week they turned in another short write-up: what did they expect to find? Some of them found historical discharge measurements. Some checked out River Watch data from the local area. I’m hoping that the pH group will tell me that 7 is neutral, and that surface waters are usually slightly more acidic than that, and that the ICP group will tell me that Ca, Mg, Na, and K are likely to be a lot higher than Pb. (Well, I sure hope we don’t find measurable Pb.)

My hope is that, by making the students think about the units and quantities that they will be measuring, that they will notice if the numbers are unreasonably high, or unreasonably low... and check to see that the correct switch is flipped, or that the sample probe is actually in the water, or that they used the same units to measure the river’s width and depth.

And I’m hoping that the length of the assignment will allow the students to think more than they normally do in a three-hour lab.

We’ll see. The intro students probably won’t spend time fretting over whether one element is interfering with another in the ICP analysis – they won’t become analytical chemists while doing an intro geology project. But maybe the numbers will mean more to them.