My Track Record

Paper by @DynamicEcology: “Pubcreds: fixing the peer review process by ‘privatizing’ the reviewer commons.” http://www.esajournals.org/doi/pdf/10.1890/0012-9623-91.3.325
1:01 PM

Does doing reviews help you get promoted? Not really, Rivara says, but quality reviews help you get onto editorial boards, which DOES count.
12:58 PM

Rivara notes the problem of PIs who don’t do their share of reviews, but stops short of recommending PubCreds solution by @DynamicEcology.
12:55 PM

Rivara: “People who write the best book chapters aren’t my age. People my age just recycle what they’ve already written.”
12:52 PM

Rivara closes with a quote from Ziman (1968): “The referee is the lynchpin about which the whole business of science is pivoted.”
12:44 PM

@NIHforHealth Early Career Reviewer program trains young scientists in peer review. http://public.csr.nih.gov/ReviewerResources/BecomeAReviewer/Pages/Overview-of-ECR-program.aspx
12:43 PM

NIH grant review criteria: overall impact, significance, investigator(s), innovation, approach, environment.
12:40 PM

Discrepancies among reviewers were studied/reported in a 1993 Nature paper by E. Ernst, T. Saradeth, & K.L. Resch: http://www.ncbi.nlm.nih.gov/pubmed/8497307
12:39 PM

Reviewing results sections: are figures and tables used effectively to tell the story? Discussion: are conclusions justified by the data?
12:35 PM

Brevity of articles is important, says Fred Rivara. The 1953 Watson & Crick Nature paper on DNA structure was 1 page!
12:32 PM

Why be a reviewer? Makes you a better scientist; helps develop your rep; leads to other national positions; a duty of scientific community.
12:30 PM

How do journals select reviewers? Author recommendations, journal database, editors’ contacts, reference lists, paid statistical reviewers.
12:27 PM

Ethics of reviewing: treat work confidentially; respect the intellectual property; tell editor if you had help with a review; disclose COIs.
12:24 PM

Journals have their own copy editors, so there’s no need for reviewers to correct spelling, etc.
12:21 PM

What’s NOT a reviewer’s role? Decide on acceptance/rejection; copyedit; be mean; nitpick; tell authors how they should have done the study.
12:20 PM

More on peer review as seen by Richard Smith (former editor of BMJ): http://blogs.bmj.com/bmj/2011/11/11/richard-smith-a-woeful-tale-of-the-uselessness-of-peer-review/
12:17 PM

Richard Smith’s problems of peer review: a lottery; a black box; “ineffective”; slow; expensive?; biased; easily abused; can’t detect fraud.
12:14 PM

Peer review started around the 1940s. Before that a small group of editors at each journal did all the reviews themselves.
12:12 PM

Most journals rate the quality of the reviews they get back from reviewers: excellent, very good, good, fair, poor, don’t use.
12:10 PM

1st slide: “How journals work” — from submission to acceptance or rejection. His journal rejects ~50% of submissions without review!
12:08 PM

Sorry — Fred RIVARA, not Rivera. He’s editor in chief of the Archives of Pediatrics and Adolescent Medicine (JAMA Pediatrics).
12:05 PM

Today’s UW Biomedical Research Integrity talk: “The what, why and how of peer reviewing” by Fred Rivera, Depts. of Pediatrics & Epidemiology
12:02 PM

—@trappedinlab

Janice DeCosmo: Tell students in advance how to interact with you. Is it OK to interrupt frequently w/ questions? Or make lists for later?
10:14 AM

Jennifer Harris: Asking “What draws you to THIS project, in particular?” will help you gauge students’ level of interest and motivation.
10:09 AM

How to gauge interviewees’ technical skills? Ask for a reference from a professor or TA from a lab class. Also consider a trial period.
10:07 AM

Janice DeCosmo: Look for evidence of students’ initiative on resumes and at interviews.
10:01 AM

Janice DeCosmo: Disclose your limitations as a mentor (tendency to micromanage, or to have high expectations, or whatever) up front!
9:41 AM

Janice DeCosmo: Remember that mentoring is a relationship. Both mentor and mentee have expectations and needs.
9:37 AM

The 4 men at the seminar are all along one side of the square around which we are arranged. The 11 women are on the other 3 sides.
9:24 AM

My question: how can we best predict which undergrads will succeed in research?
9:20 AM

Question from seminar attendee: how do you get money for students without using your own grant?
9:16 AM

After 7 years of mentoring undergrads, it’s time to find out what I’ve been doing wrong. 🙂
9:05 AM

Today’s seminar: “Mentoring the Undergraduate Research Experience” with Janice DeCosmo and Jennifer Harris of @UW.
9:04 AM

12:56pm
@smfullerton on the video’s strength: a realistic depiction of the many chances we have to speak up or stay silent, and the consequences.

12:48pm
OK, this “Reverend Lee said that my daughter is a hero” part IS a bit hoaky.

12:44pm
The villain of “The Lab,” who falsified data, is a postdoc named Greg Anderson. Same name as Barry Bonds’ sketchy trainer!

12:35pm
“… The little punks have always counted because, in the long run, the character of a country is the sum total of its little punks.”

12:34pm
Kim’s cousin, quoting Gary Cooper: “I know a lot of you are saying, ‘What can I do, I’m just a little punk. I don’t count.’ …”

12:22pm
Quote from grad student’s mom: “Why didn’t you go into law like your brother? He never has [ethical] problems like this!”

12:19pm
Video interlude: postdoc creates Johnny Cash / Wu Tang Clan mashup. Hilarious!

12:18pm
(I am texting my poll responses via http://PollEv.com , not via my phone.)

12:16pm
1st vote: Should Kim just sign the proofs, or tell Greg she needs to read them? “No, slapping him is not an option.”

12:11pm
American Idol-style polling on the seminar will begin soon!

12:09pm
When the Office of Research Integrity first came out w/ the video “The Lab,” @smfullerton thought, “Oh, how hoaky.” She likes it better now!

12:05pm
Research misconduct: “The big ‘duh’ of the BRI series,” says @smfullerton … Don’t lie, cheat, or steal! Yet not always black & white.

12:02pm
Seminar presenter/moderator: @smfullerton. Website for “The Lab” (cool interactive video): http://www.ori.dhhs.gov/thelab

11:57am
Today’s UW Biomedical Research Integrity seminar: “Avoiding research misconduct: a virtual experience interactive learning simulation.”

I’m continuing my test of whether live-tweeting from seminars improves my attentiveness. Below are today’s tweets (in reverse chronological order, as usual).

* * * * * * *

5:04pm
Slide title: “Integrating phylogeny with GIS [geographic information systems].” Again, I’m lost, but this sounds sexy as hell.

4:54pm
Hepatitus C genome: only ~9600 bases long (not base pairs — it’s RNA!). About the same length as HIV.

4:40pm
“Occult” HCV infection: patient seems cured, then (years later) presents an infection related to the original one (not a new infection).

4:31pm
“Bayesian Skyline” plot. Don’t understand it, but love the sound of it.

4:28pm
Rebecca Gray’s talk is reminding me how bad I am at understanding phylogenetic trees. @phylogenomics would be ashamed.

4:17pm
Where data are lacking, HCV is often assumed to be like HIV, a fellow RNA virus.

4:16pm
Problems with studying HCV: lack of animal model, difficulty with in vitro culture, long asymptomatic period.

4:14pm
Global burden of hepatitis C virus (HCV): 180 million people are infected! That’s ~3% of the world!

4:05pm
URL for Rebecca Gray: http://evolve.zoo.ox.ac.uk/evolve/Rebecca_Gray.html

4:05pm
Today’s UW seminar: “Infectious disease evolutionary dynamics: the strange case of hepatitis C” by Rebecca Gray, Ph.D.

[my tweets from a seminar I went to today, presented in reverse chronological order]

1:36pm
The middle-author problem: if you get little credit for your role in a study, where’s the incentive to check & vouch for the whole project?

12:58pm
Larry Kessler: As part of their training, grad students can re-run publicly available code and see whether they get the same results.

12:57pm
One Duke prof falsified a line of his CV, saying he got a Rhodes Scholarship … to Australia. Not really Rhodes Scholar material.

12:53pm
Kessler: Also, letters of concern to journals should not just be forwarded back to the authors of the article under suspicion.

12:52pm
Witten: How can 2 reviewers fairly evaluate a multidisciplinary 30-person paper? No way to do that, plus it isn’t their job.

12:51pm
Question from audience: role of journal editors and peer review in Duke case?

12:50pm
Kessler: Duke scientists wanted fame, promotions, grants … but weren’t aiming for royalties on diagnostic test. Wasn’t about $$ per se.

12:48pm
@smfullerton asks about conflict of interest, the elephant in the room. Duke scientists had vested interests in clinical trials?

12:46pm
Funders support discovery of exciting new approaches but are reluctant to support validation, so validation naturally gets neglected.

12:44pm
FDA needs to develop a guide on how to bring an omics test to the clinic. It’s been “putzing around” for a decade, Kessler says.

12:43pm
Institutions don’t always support multi-discipline work to ensure proper credit and accountability. What about middle authors?

12:38pm
There is a “bright line” between test discovery/validation on one side and evaluation for clinical utility/use on the other.

12:37pm
Kessler says we think we know how to do discovery and test validation, but lots of mistakes get made. Don’t validate on already-used data!

12:33pm
IOM report explains how to “correctly” translate omics diagnostics from bench to beside, and lists responsibilities of various parties.

12:31pm
Institute of Medicine report was released in March. Larry Kessler (who served on the IoM committee with Witten) will tell us about it now.

12:30pm
Aftermath of Duke debacle: dozens of papers retracted, careers of 162 coauthors jeopardized; public faith undermined.

12:28pm
The Duke problem was not just an academic one. The Duke papers, with their mistakes, were being used to guide clinical trials!

12:25pm
Keith Baggerly & Kevin Coombes of MD Anderson reported mistakes in publicly available Duke data (Annals of Applied Statistics 2009).

12:22pm
Concerns about Duke work were initially brushed off as “squabbles among statisticians.”

12:20pm
2006: Duke people started publishing high-profile papers on using omics to predict cancer outcomes. Others had trouble replicating results.

12:18pm
Challenges of omics data: (6) multidisciplinary analyses require cooperation and trust.

12:17pm
Challenges of omics data: (5) it’s hard to apply intuition to models with thousands of genes. Hard to tell what the data *should* look like.

12:16pm
Challenges of omics data: (4) expense of experiments and limited samples mean that results aren’t always validated.

12:14pm
Challenges of omics data: (3) complicated experiments and analysis increase the likelihood of errors.

12:14pm
Challenges of omics data: (2) Batch effects may cause results in one lab not to generalize to other labs.

12:13pm
Challenges of omics data: (1) many variables (e.g., genes) vs. number of observations (e.g., patients) leads to high risk of overfitting.

12:10pm
Recent @nytimes series “Genetic Gamble” highlights promise and difficulty of using genomics to guide cancer treatment.

12:08pm
Daniela Witten’s definition of omics: characterization of global sets of biological molecules (DNA, RNA, protein, metabolites, etc.).

12:04pm
Today’s topic: recommendations of Institute of Medicine panel on responsible use of omics data in clinical research.

12:03pm
Website for UW Biomedical Research Integrity program: http://depts.washington.edu/uwbri/

12:00pm
BRI seminar will feature Larry Kessler and Daniela Witten, with moderation by @smfullerton.

11:58am
Now attending the UW Biomedical Research Integrity seminar “Responsible Research in the Era of Omics: Past, Present, & Future.”