Science, we have a systemic problem

I read with growing horror this account of the research practices of the Wansink lab. They do research in nutrition, or maybe some combination of economics, psychology, and dietary practices — it’s described as “research about how people perceive, consume, and think about food”, and it’s not stuff I’d ever be interested in reading (although that does not imply that it has no value). The PI, Brian Wansink, wrote up a summary of his process on a blog, though, and honestly, my jaw just dropped reading this.

A PhD student from a Turkish university called to interview to be a visiting scholar for 6 months. Her dissertation was on a topic that was only indirectly related to our Lab’s mission, but she really wanted to come and we had the room, so I said “Yes.”

When she arrived, I gave her a data set of a self-funded, failed study which had null results (it was a one month study in an all-you-can-eat Italian restaurant buffet where we had charged some people ½ as much as others). I said, “This cost us a lot of time and our own money to collect. There’s got to be something here we can salvage because it’s a cool (rich & unique) data set.” I had three ideas for potential Plan B, C, & D directions (since Plan A had failed). I told her what the analyses should be and what the tables should look like. I then asked her if she wanted to do them.

He described it as a failed study with null results. There’s nothing wrong with that; it happens. What I would think would be appropriate next would be to step back, redesign the experiment to correct flaws (if you thought it had some; if it didn’t, you simply have a negative result and that’s what you ought to report), and repeat the experiment (again, if you thought there was something to your hypothesis).

That’s not what he did.

He gave his student the same old data from the same set of observations and asked her to rework the analyses to get a statistically significant result of some sort. This is deplorable. It is unacceptable. It means this visiting student was not doing something I would call research — she was assigned the job of p-hacking.

Further, what’s just as shocking is that Wansink sees so little wrong with this behavior that he would publicly write about it.

He’s not done.

Every day she came back with puzzling new results, and every day we would scratch our heads, ask “Why,” and come up with another way to reanalyze the data with yet another set of plausible hypotheses. Eventually we started discovering solutions that held up regardless of how we pressure-tested them.

Note: no new experiments. This is all just churning over the same failed experiment, the same failed data set. Back in the day, I learned that you design an experiment to test a specific hypothesis, and that you don’t get to use the data to test different hypotheses until you get a result that you like. But what do I know, I’m old.

Still not done.

I outlined the first paper, and she wrote it up, and every day for a month I told her how to rewrite it and she did. This happened with a second paper, and then a third paper (which was one that was based on her own discovery while digging through the data).

Out of this one failed (I repeat, fucking failed) data set, they ground out FOUR papers. Four. Within a few months. Good god, I’ve been doing everything wrong.

You might be wondering what these papers were that he milked out of this failed data set. Here are the titles:

Lower Buffet Prices Lead to Less Taste Satisfaction
Peak-end pizza: prices delay evaluations of quality
Low prices and high regret: how pricing influences regret at all-you-can-eat buffets
Eating Heavily: Men Eat More in the Company of Women

I am trying hard not to be judgmental, and failing. These sound like superficial, pointless crap churned out to appease a capitalistic marketing machine, with virtually no value and making no contribution to human knowledge. But I guess it’s good enough to get you a leadership position in a prestigious lab at Cornell.

It’s also a huge problem that this kind of strategy works. It’s not just Wansink — it’s a science establishment that allows and even encourages this kind of garbage production.

I hear there’s a replication crisis in the sciences. I have no idea how that could be.

The third and fourth week of ecological developmental biology

morningonthefarside

I’d intended to make these reflections on the progress of my new course in ecological development a weekly feature on the blog, and then I failed to post an update last week. Bad professor, very bad. My excuse, though, is that I’m on a job search committee, we had three interviews in the last week, and they’ve all been kicking my butt and leaving me exhausted at the end of the day. That’s a duty that’s also a lot of work for us academics: there’s the gay social whirl all of a sudden, the scrutiny we have to give to each candidate, and sitting through job talks. The stress can be enormous, too — not for the candidates, although I’m sure they’re feeling a little anxiety, but for us. In the dream search, you bring in two candidates who suck and third that is gloriously qualified and a joy to spend time with, because then the decision-making is easy. In this case, we got three marvelous candidates and I want to hire them all, and we have to pick one. Just one. We’re going to make that painful decision on Saturday, so while all the work is done, the agonizing has only just begun.

And meanwhile, classes go on!

Last week was assessment time. I’m also teaching our genetics course, and they got an exam…an exam they did pretty well on, with an average of 79%. It could be that this cohort of students is just generally brilliant (but all of our students are brilliant), or it could be that some changes I’ve made in this class have been effective. I’ve been concentrating on laying a solid foundation: we’ve gone over basic Mendelian genetics, something I remind them over and over that they should have already been thoroughly exposed to and so this should just be review, and I also remind them over and over that later it’s going to get much harder and that we’re going to spend almost the entire semester talking about exceptions to this simplistic Mendelian stuff, and if they don’t fully grok the basics they’re going to be so screwed. We’ve also been working on a probability and statistics toolbox that they’ll be using repeatedly throughout the term.

I may have scared them into studying hard. Not only did they get a higher average score than past years, but the range tightened up considerably. I’m trying to build a strong foundation here, because as Al Franken explained to the nation in the DeVos hearings, we care more about growth than an arbitrary standard of proficiency. Give ’em the basics so the weaker students have something to build on rather than floundering and falling apart on the first day, and keep nudging them upwards at every step in the class.

My ecological development course also took a turn. The first two weeks, you may recall, consisted of the traditional Old Bearded Guy standing at the front of the room Old-Bearded-Guysplaining developmental biology to them — again, trying to put everyone on a firm footing in the fundamentals. The next step is to coax them into student-splain stuff to me. This has been harder than it should be, because this is an 8-fucking-am course, and I’m not my perkiest, and the students aren’t either. Next time I teach an interactive course, I must insist that it be offered sometime in the mid-day. Either that or demand IV bags from the ceiling filled with caffeinated beverages and start the morning going to each desk and jabbing everyone into alertness with a needle in a vein.

Instead of intravenous drugs, though, my approach to jump-starting their brains and making them comfortable speaking was to force them to do presentations last Tuesday. Short presentations; I gave them copies of Langman’s Medical Embryology, used a deck of cards to randomly assign each of them a week of human development, and had them give five-minute summaries of what was happening then: they had a few questions to guide them, like show what the embryo looked like, say something about critical events in their week, and discuss clinical correlates. It was straightforward and didn’t require intense thought, so it was simply a way to get them all to say a bit in class, as well as introducing a topic that we’ll return to in, for instance, a later discussion of teratogenesis.

Last Thursday, they had to talk again (I am such a cruel tyrant). They’d been assigned to read Lewontin’s Triple Helix, and this day was dedicated to a critical assessment of the text. I gave them a set of questions about the book, and then sat back and let them tell me the answers.

That actually went fairly well, I think. It still takes some time for them to warm up and get a conversation going, but they’re a smart bunch and we got some good discussion. It went well enough that we didn’t finish, so we extended the review to this past Tuesday. We identified a central theme of the book as construction: organisms are assemble themselves in an environmental context, and they are continually modifying their environment. These cycles of self-referential feedback mean that you simply cannot define an organism from nothing but its genome. They’re getting it!

This morning, I twisted the game around on them a little more. We’re digging into Gilbert’s Ecological Developmental Biology text with chapter 1, on normal plasticity, and this time I gave them the assignment ahead of time to write down three questions that chapter inspired in them. We spent most of our time bouncing questions and answers back and forth, which is always fun. I ended the session by listing some of the questions that got some vigorous responses, and putting them on the board. They were:

  • Temperature-dependent sex determination in reptiles: are there reptile intersexes? How often? We also got a suggestion that we should look more into behavioral sex determination in fish.

  • Inheritance of behavior differences: what causes differences in aggression in dog breeds? Is it genetically determined, how much and what genes are involved? (I asked where they fell on the continuum of biases about pit bulls, whether they where inherently vicious and needed to be put down, vs. a maligned breed that has a bad reputation because they are abused. I was surprised: 100% of the class came down in the not-intrinsically-evil camp. Dang liberals!)

  • Sneaker and dominant males: How do these differences within a sex in a single species arise? We discussed rhinocerous beetles and cephalopods.

  • Gravity. How dependent is development on this pervasive influence of gravity? We talked about some clear examples, like how the chicken body axis is dependent on rotation, and that led to speculation about human development and plasticity in microgravity. What happens to bodies in space? Can human fetuses grow normally in space?

  • Epigenetics…there were some good questions about that, but I deferred them all, telling them that we’re going to spend a whole week on epigenetics, so let’s take it off the table temporarily.

That was a good start. Then I divvied up the students — they volunteered for what subject most interested them — and sent them off to the library with an assignment, to find papers to address their question, and come back next Tuesday prepared to explain what they learned to the whole class.

Brains full. We stopped there. I’m looking forward to learning what they find next week.

Science is done by and for people

I am putting this here because I want to refer to it later, and it’s on Twitter, not the most convenient medium for archiving a lengthy story, and because it’s important: Michael Eisen explains the importance of the human element in science.

[Read more…]

Mary’s Monday Metazoan: What? Females aren’t beautiful for me?

bluecrabfemale

That female crab doesn’t make herself gorgeous for the males of her species, it seems — sometimes a lady just has to look good.

Contrary to expectation, the model shows that winning the romantic interest of picky males is not enough to explain how desirable feminine features become widespread — even when better-looking females are more likely to land a good catch.

The results of their mathematical approach support other research suggesting that female beauty doesn’t evolve just to win mates.

Instead, traits such as the dance fly’s frilly legs or the blue crab’s red-tipped claws may help their bearers compete for other resources, such as social status or protection from predators. The results are consistent with an idea called the “social selection” hypothesis, first proposed three decades ago by theoretical biologist Mary Jane West-Eberhard of the Smithsonian Tropical Research Institute.

Impossible. They might consider other resources than access to my magnificent manliness to be valuable? Heresy.

It’s Flyday

I don’t have any classes today, so I catch up on my custodial work. That means I’m going to spend a few hours scrubbing fly bottles: bottles caked with medium the consistency of slimy oatmeal, full of maggots, with dead flies scattered around like raisins.

May not have an appetite tonight.

Ah, the glamor of the scientific life…

Not enough hours in the day

farsidequestions

Yikes. I have been buried in work — we’ve just begun a week of interviews for job candidates, and I’m on the search committee, so I’ve been tied up all last night, all day today, and this evening. And what time hasn’t been occupied in service work has been involved in preparing for tomorrow’s ecological development class.

The big project tomorrow is a critical analysis of Richard Lewontin’s The Triple Helix. It’s a short book, but it’s packed to the gills with concepts they may not have encountered before…and most importantly, concepts they may not have questioned before. So I had to put together a framework for discussion. I’ll let you read it, too, although it’s not going to be very useful unless you’ve read the book as well.

The book is only 3 meaty chapters long with a concluding summary. I’m trying to provoke some arguments with these questions.

I. Gene and Organism

Lewontin complains about metaphors: what’s wrong with the DNA as blueprint metaphor?

We have a bias in our language. The word “development” implies an “unrolling” of a program. Is that a good explanation of the process?

We talked about preformation vs epigenesis on the first day. I told you that preformation is an untenable explanation, but Lewontin argues that preformation has won. How?

He explains that there is a deep difference between transformational vs variational change. Explain.

Brenner, p10: he claimed that with the complete sequence of DNA, he could compute the organism. What’s wrong with that statement?

Similarly, Gilbert, p11: with the genomic sequence, we will know what it means to be human. Do you believe it.

He gives several examples of complicating “transformations”:
p19: Explain phototropism, geotropism
p21: What are norms of reaction

Contrast fig 1.8 (p 29), Jensen’s IQ model, with 1.6 (p25), Drosophila viability as a function of temperature. What’s the obvious flaw with Jensen’s hypotheses?

Leads up to fig 1.10 — what are all these different theoretical patterns? Can you explain what each one means?

II. Organism and Environment

Adaptation and fitness…what are they? What’s wrong with the idea of an organism “fitting” to an environment?

p44: “Adaptive explanations have both a forward and a backward form”. Explain what he means.

What’s the problem with these modes of explanation? See discussion of Orians & Pearson results for an explanation.

p47: “the organism is the object of evolutionary forces”. Is this reasonable?

Lewontin says the concept of construction best captures the process of evolution. Explain.

He objects to the search for life on Mars for what reasons? (not that he thinks we shouldn’t look, but that he thinks the methods are wrong)

p54: “If one wants to know what the environment of an organism is, one must ask the organism.” How did he arrive at that conclusion?

p57-58: Explain Van Valen’s “Red Queen hypothesis”. Why is it somehow different from what Lewontin proposes?

p68: “Save the Environment!” But “the environment” does not exist to be saved. Is Lewontin a (shock, horror) an anti-environmentalist? What is the point he is making in his conclusion?

III. Parts and Wholes, Causes and Effects

A critique of the analytical, reductionist examination of the organism as a machine. This is generally how we teach biology; Lewontin argues that much of it is invalid. How would you alternatively expect biology to be taught?

p74: What are the current failures of that analytic approach? (with the understanding that that approach might still succeed, with enough time and data).

p77: What is the problem of the development of the human chin? What is the “error of arbitrary aggregation”?

p81: “Only a quasi-religious commitment to the belief that everything in the world has a purpose would lead us to provide a functional explanation for fingerprint ridges or eyebrows or patches of hair on men’s chests.” Does finding a functional explanation for any of those things invalidate the criticism? Why or why not?

Explain Tables 3.1 and 3.2. What do they tell us about the relationship between fitness and genetics when more than one gene is involved?

p90: When you see the variation in ceratopsian horns, how do you personally try to explain it? What is your default explanation?

p96: What are the “acute problems” in genetics? How much of it is a genuine problem with the scientific approach vs. attempts to shoehorn our explanations into simplistic causal models?

Lewontin gets political in the last page of this section, blaming environmental problems on “an anarchic scheme of production that was developed by industrial capitalism and adopted by industrial socialism”. What do you think?

IV. Directions in the study of biology

Lewontin admits that he’s been negative and strongly critical in the earlier parts of the text, so he has a brief epilogue in which he tries to advocate for some positive directions we can take. What are some ideas you might have?

It’s entirely possible we’ll only get two or three questions in, if we get argumentative, and that’s OK!