How peer review and publication can make a good protocol even better.

A guiding principle of STAR Protocols is that we make researchers' lives easier by publishing robust and usable protocols. We leverage the strength of peer review to help authors improve their protocol. This Backstory details the transformation of a bench protocol to a published protocol, highlighting the improvements to the article through the drafting, review, and revision stages. This underscores the value of the peer review process in general and the collaborative peer review philosophy at STAR Protocols specifically. For complete details, please refer to Chhoy et al. (2021).

Above image: The published protocol being used for reference in the Mercurio lab.

Chhoy describes drafting the protocol as a valuable learning experience: the exercise helped to transfer the lab’s knowledge internally, as he became proficient in a technique that his colleague Caitlin Brown had originally developed, and he better appreciated how other research groups went about conducting similar experiments.

Clarity and usefulness to the research community were important factors to Chhoy in drafting the protocol: if someone else picked up the protocol, he wondered, where would they most likely become confused?

He appreciated the chance to learn more deeply about the field from the reviewers and that the reviewers took the time to explain their comments rather than simply criticizing the sections of the manuscript that they believed were inaccurate.

He believes that the review process served a broader purpose in his own development as a scientist, illustrating how different researchers communicate in different ways and improving his capacity to receive and respond to feedback.

Chhoy is also optimistic that the visual appeal of the typeset and published protocol will encourage its widespread adoption over the heavily annotated notebook pages to which experimentalists so often refer.

Main Text

It started as a paragraph in a STAR Methods section

The STAR Protocols editorial team recruits submissions by looking at what our colleagues are publishing across Cell Press. We then invite authors to convert one of the methods from their research article into a detailed, step-by-step protocol. Arthur Mercurio and his colleagues at the University of Massachusetts Medical School published an article in Developmental Cell in 2019 about iron export via extracellular vesicles (EVs) (Brown et al., 2019). As part of this work, the authors developed a protocol for EV separation, which was described briefly in the STAR Methods section of their paper. When we invited the Mercurio lab to contribute a protocol, they saw the value in sharing the full details of their approach.

From bench protocol to STAR Protocols submission

We reached out to the first author of the protocol, Peter Chhoy, to learn more about his process for drafting the submitted protocol. Chhoy said that writing the protocol was a good learning experience for him, as he had contributed to the lab’s previous publication but had never performed the separation experiments himself. To begin, he compared the lab’s bench protocol with the paper’s STAR Methods section; he also performed a literature search on other methods for separating EVs. Next, Chhoy repeated the protocol himself and drafted a STAR Protocols manuscript according to the author template. He made some changes based on other versions of the protocol and his own experiences and, finally, incorporated some feedback from his coauthors before submission.

At this stage, the authors had already made a few relatively straightforward changes that significantly improved the utility of the protocol, such as expanding abbreviations, separating each action into its own step, and adding a short introduction. This version of the protocol also incorporated a few critical steps as suggested by the STAR Protocols author instructions. Beyond these additions, Chhoy describes drafting the protocol as a valuable learning experience: the exercise helped to transfer the lab’s knowledge internally, as he became proficient in a technique that his colleague Caitlin Brown had originally developed, and he better appreciated how other research groups went about conducting similar experiments.

According to Chhoy, the process of drafting the manuscript was “very reasonable” with requirements that were “doable and easy to understand.” While it was sometimes difficult to phrase the critical steps of the procedure, familiarity with Cell Press’s STAR Methods standards from his work on the Developmental Cell article and the STAR Protocols article template made his writing process easier. Clarity and usefulness to the research community were important factors to Chhoy in drafting the protocol: if someone else picked up the protocol, he wondered, where would they most likely become confused?

Improving the protocol through peer review and revision

The most readily apparent change to the manuscript during peer review and subsequent revision was to the title. Initially submitted as “Protocol for isolation of exosomes from in vitro cell culture models,” the title was changed to “Protocol for the separation of extracellular vesicles by ultracentrifugation from in vitro cell culture models” after some important feedback from the reviewers. First, multiple reviewers commented that the preferred generic term for the sorts of particles discussed here is “extracellular vesicle” and not “exosome”; they cited a 2018 position paper that gives guidelines for nomenclature (Théry et al., 2018). Second, one reviewer believed that “complete isolation” of EVs is not a realistic goal and that “separation” should be used instead. Finally, the same reviewer suggested mentioning the EV separation technique in the title because multiple protocols might exist for separating similar types of EVs.

Chhoy views this as an important learning experience. While he was familiar with this position paper and cited it in the initial draft of the protocol, the peer review process helped him to understand the standards in the field at a level he hadn’t considered before. He appreciated the chance to learn more deeply about the field from the reviewers and that the reviewers took the time to explain their comments rather than simply criticizing sections of the manuscript that they believed were inaccurate.

In the revised manuscript, you can see the reviewers’ anonymized comments (marked “Author” and shared with their permission) and the responses by the authors of the protocol (marked “Chhoy”). In-line reviewer commenting is a central tenet of the STAR Protocols peer review process. We adopted this approach to make the process more streamlined for authors, reviewers, and editors. Of course, it has the added benefit of highlighting the collaboration between the reviewers and the authors to improve the clarity and the usability of the protocol.

Some of the reviewer comments are technical and address important factors for repeatability, like centrifuge rotor settings. Others are more tutorial, like a request for hands-on tips to avoid disturbing a pellet after centrifugation. Still others relate to the adaptability of the protocol to other laboratories. These comments were made in the spirit of making the protocol more useful for other researchers and helping to ensure that the protocol was clear and comprehensive enough to be repeated in a different lab.

Chhoy praised the reviewer comments as thorough and described the review process as a collaboration between the reviewers and authors. This experience reflects the ethos of the STAR Protocols team: reviewing a protocol is fundamentally about helping authors make their protocol more useful. Chhoy admits that his primary expertise is not in EVs—the Mercurio lab mostly focuses on the molecular and cell biology of tumors—and he enjoyed the opportunity to hear from some authorities in this area. He believes that the review process served a broader purpose in his own development as a scientist, illustrating how different researchers communicate in different ways and improving his capacity to receive and respond to feedback.

Reflections on the published protocol

Chhoy acknowledges that he is not the first person to publish a step-by-step guide to separating EVs, but if his protocol makes someone’s experiments easier as they go about the task, he will consider it a success. That might be a simple tip like the best way to look at scarce samples following the separation process or a full-fledged reintroduction to the technique for someone who has been away from the lab for a long time. Chhoy is also optimistic that the visual appeal of the typeset and published protocol will encourage its widespread adoption over the heavily annotated notebook pages to which experimentalists so often refer.

Because the protocol (Chhoy et al., 2021) was published so recently, Chhoy hasn’t yet heard any of these success stories from fellow researchers. But the protocol has already made its way into at least one laboratory setting, taking up residence right next to Caitlin’s Lucky Red Bucket on a Mercurio lab bench. We hope that it, and the many other protocols we have published, will fulfill our mission to provide resources that make researchers’ lives just a little bit easier.

Excerpt from the STAR Methods section of the authors’ previous publication describing the separation of EVs, reproduced from (Brown et al., 2019).

The authors' original lab protocol for separating EVs.

Exosome isolation section from the authors’ initial submission to STAR Protocols.

EV isolation section showing the reviewers’ comments and the authors’ responses to them.

EV isolation section in the published protocol.