Governance

Review Guidance for consensus reports submitted to Medical Physics

1.    Scope and Purpose

Medical Physics welcomes submission of high-impact consensus reports that address clinical and scientific practice on topics of interest to our readership.  While these manuscripts are not scientific articles in the traditional sense, their publication in Medical Physics is mutually beneficial to the Journal, the report authors, and the sponsoring organization.  They are among the most highly cited of our published articles.  Publication in the Journal provides the authors with citable peer-reviewed publication and disseminates the report to a larger international audience.  While the vast majority of these reports are American Association of Physicists in Medicine (AAPM) Reports, Medical Physics does occasionally publish consensus reports from other organizations.  Recent examples include reports from the GEANT4 Collaboration, NRG Oncology, IAEA, and the NIH.  The scope of these review guidelines includes all consensus reports whether sponsored by AAPM or other organizations.

For the purposes of this review guidance document, a consensus report is a report on a specific topic produced by a committee of experts convened by the sponsoring organization (most often but not necessarily the AAPM) and published in the name of this organization.  AAPM reports (and those of most sponsoring organizations) undergo rigorous internal review before submission to the Journal.

2.    Overview and Guiding Principles of the Review Process

All consensus reports sponsored by the AAPM or other organizations must undergo peer review and editorial decision.  Medical Physics will not review manuscripts from the AAPM or other organizations under the condition that they be published verbatim.   Only if the report achieves an acceptable level of quality will it be accepted by the editor for publication.  Sponsoring organizations should be aware that submission of an approved report does not guarantee acceptance.  As a condition of acceptance, our referees and editors may require significant revisions: sponsoring organizations and report authors are expected to make a good-faith effort to respond to our critiques.  While Medical Physics defers to the sponsoring organization regarding report charge, content, and prescriptive recommendations, reports that do not articulate a clear charge; fail to scientifically support their conclusions, or are poorly written will likely require extensive revisions or may even be rejected.

Consensus reports are published electronically only, and have a maximum length of 30 printed pages.  For longer reports, typically a shorter executive summary is normally submitted to the Journal, which is electronically linked to the full online report normally posted on the sponsoring organization’s website.  In rare circumstances, the editor-in-chief may approve a longer published report.  However, in this circumstance, the sponsoring organization must agree to pay the publisher’s excess page charges for every page over 30 (currently $200 for every published page over 30). 

The review process outlined here is that negotiated between various Medical Physics editors and the AAPM Science Council and Therapy Physics and Imaging Physics Committees for reviewing AAPM reports (usually but not necessarily Task Group reports).  These procedures should be adapted as needed by the editor(s) for other types of AAPM Reports and documents from other sponsoring organizations.  For sponsoring organizations other than AAPM, the report authors should contact the editor-in-chief before submission, who will determine whether the proposed report falls within the scope of the Journal and negotiate an appropriate adaptation of the AAPM report review procedures.

1)    Once a report is approved by Science Council (SC), the corresponding author (usually the AAPM Task Group chair and first author) may submit the report to Medical Physics.  With concurrence of the editor-in-chief and SC chair, mature and well-written reports may undergo Medical Physics peer review concurrently with later stages of parent committee and/or council review.  The term “Parent Committee” refers to the committee structure reporting directly to the SC from which the report in question originated. The current SC Committees are BRM (Big Data, Radiomics, and Machine Learning), IPC (Imaging Physics), TPC (Therapy Physics), RSRCH (Research), and (TAC) Technology Assessment.

2)    Four referees and one associate editor (AE) should be appointed.  The parent committee chair appoints the AE (typically the lead internal reviewer) and two of the 4 referees (typically internal reviewers who are familiar with the report).  We use the term “reviewer” to mean an individual, who is not an author of the report, has no other conflict of interest with the report authors or sponsoring organization, and who has performed “internal” peer review on behalf of the AAPM before final approval of the report.  “Referee” refers a peer reviewer who evaluates a manuscript on behalf of Medical Physics. The editor should contact the parent committee chair to identify the AAPM-appointed referees and AE and to get the routing form documenting the review history, including the names of all internal reviewers.  The editor selects two independent referees who are experts in the field but who have not been previously been involved in reviewing the report.  These individuals need not be AAPM members.  For both AAPM and independent reviewers, it is almost always better to contact prospective referees/AEs to confirm their willingness to accept these assignments before formally inviting them through EJP.  Once the AE (appointed by the parent committee chair) accepts the editor’s formal EJP invitation to manage the AAPM Report, the editor then passes the four previously identified referees to the AE, who then formally invites them to evaluate the report through EJP.

Review criteria for AAPM and other consensus reports differ significantly from those of original scientific manuscripts.  They are more similar to, but not identical, to review criteria for Review Articles.  Below are some guiding principles.

A.    The identities of the Review Team members (referees and AE) should remain anonymous from the report authors.  While the lead and other internal reviewers may be known to the authors, the editor and AE must not reveal their identities to the authors.  If absolutely essential, anonymity may be breached only with consent of the parent committee chair, the review team member in question, and the editor.

B.    AAPM Science Council classifies AAPM reports into the following categories

1.   New or Emerging Technology. 

2.   Established Area.

3.   Specific Clinical Problem.

4.   Educational or Informational. 

5.   Consensus Document. 

Thus, reports have varying levels of prescriptive content and recommendations and very different goals depending upon the category.  Category 1 reports may offer interim guidance to early adopters or may identify broad questions that need to be further researched before consensus recommendations or answers can be given.  Categories 2, 3, and 5 are more likely to offer specific recommendations or even standard-of-practice “requirements” that affect the daily practice of physicists.

C.    Review Team members need to bear in mind that AAPM report charges and associated recommendations represent the considered positions of the AAPM on the issues in question.  These reports have undergone significant internal debate and review, culminating in recommendations that represent the consensus of a panel of experts.  Normally, Medical Physics peer reviewers should defer to the AAPM’s judgment regarding the value and relevance of a given report charge and the ethical principles underlying its prescriptive recommendations, e.g., balance between cost and benefit of a recommended course of action. 

Referee or AE recommendations to make major changes to the report charge or core recommendations (or other report deliverables) should be made cautiously and only when a compelling case for their necessity can be made.  Major changes to the core content of the report may invalidate the parent committee and council approvals, thereby triggering additional internal review and approval cycles before the revision can be submitted to the Journal.  It is not appropriate for peer-review team members to substitute their personal opinions for the consensus developed by the sponsoring organization.

In general, Medical Physics peer review should be focused on the following questions:

·       Is the charge clearly articulated?

·       How well does the report execute its charge?

·       Does available scientific knowledge support the report conclusions or other deliverables?

·       As written, does the report succeed in providing useful guidance to its target audience with respect to its stated purpose and charge?

D.    Notwithstanding the qualifications of Item C., Medical Physics peer reviewers and editors are under no obligation to recommend publication of what they consider to be a flawed or poorly written report that fails to articulate a clear or coherent charge or fails to adequately fulfill its charge.  The final decision whether to publish an advisory report is solely at the discretion of the editor-in-chief with guidance from the editor of record.

E.    In contrast to Research Articles and Technical Notes, Medical Physics does not require a particular structure or organization for Consensus Reports, which may adhere to the structure prescribed by the sponsoring organization.  Abstracts need not be structured.  As the attached AAPM report guidelines indicate, there is typically a table of contents for long reports, an Introduction section that identifies the problem, articulates the charge, and justifies the charge.  This is followed by one or more literature review or methodology sections that reviews the material or data that supports the report product, which can be anything from a set of recommendations to a benchmark dataset.  Typically, there are one or more summary sections which distill the findings or recommendations of the report.

3.    Detailed Review Guidelines and Evaluation Template

The following paragraphs offer more detailed guidance on evaluation rubrics for different components of any consensus report.  Because the structure and charges of consensus reports are so variable, it is not possible to impose a universal structure of headings and content features that submitted reports must adhere to.  To provide report authors with maximum flexibility, the editors have chosen to outline review criteria for various functional components that all consensus reports possess in one form or another.  The functional components are “Introduction”, “Approach”, “Deliverables”, “Discussion,” and “Conclusions.”  We again emphasize these functional components need not define the structure of the report in terms of headings.

a)    Report Title (required component)

The title must identify the type and sponsoring organization in addition to describing the content of the report.

b)    Abstract (required component)

As with any scientific manuscript, the abstract must be a short (<300 words) standalone summary of the report including its provenance, charge, approach, and deliverables.

c)    Charge (functional component)

Typically presented in an Introduction section, the charge is a succinct summary of the report deliverables and its overall purpose.  For example, AAPM “Established Area” reports may seek to develop protocols or other prescriptive recommendations to standardize the conduct of medical physics activity in a specified domain.  On the other hand, a “Specific Clinical Problem” report might be charged with developing a benchmark dataset for validating a class of dose-calculation engines or a test methodology for measuring performance of an imaging system.  In addition to clearly articulating the charge, the report must identify the sponsoring organization and the entity within the organization that approved the report.  In addition, the charge component should include a review of the literature or considerations justifying the need and importance of executing the charge. 

Appropriate review criteria include:

·       Is the charge clearly and unambiguously described?

·       Is the charge adequately justified by literature review, analysis of risk, or assessment of potential gains in cost-effectiveness, safety, accuracy or other relevant endpoints?

·       Does the report address duplication and conflict with existent consensus reports?

As noted above, minor revisions to the charge in the interest of clarity are acceptable.  However, major changes may invalidate the sponsoring organization’s approval and should therefore be recommended only if necessary.  If there are major issues with core content of the report, these should be brought to the editor’s attention for evaluation and potential communication with the sponsoring organization.

d)    Approach (functional component)

The approach is the means or methodology by which the deliverables specified in the charge are to be realized.  In the case of a report developing quality management (QM) guidance for a specified modality, the approach might be limited to reviewing the literature to identify the range of practices and reviewing incident report databases to identify risks, and then convening a panel of experts to harmonize these practices into a single set of consensus guidelines. More likely, the approach might involve performing failure modes and effects analysis (FMEA) or fault tree analysis (FTA) to a generic medical process to justify QM recommendations based on risk.  When the deliverable is benchmark dataset, the approach will consist of more standard measurement or computational methodologies.

Appropriate review criteria include:

·       Is the approach clearly and completely described, striking an acceptable balance between a self-contained description and offloading details to cited literature?

·       Does the approach provide a technically and scientifically sound method for realizing the report deliverables?

·       Do the authors adequately address weaknesses in the methodology?

·       Are deviations from accepted approaches adequately justified?

e)    Deliverables (functional component)

The Deliverables functional component is like the “results” section of a scientific paper.  It describes the results realized by executing the approach.  For a prescriptive QM report, this might be a set of proposed QA and QC tests, preferably with recommended tolerance limits, derived from FMEA and FTA of a generic medical process of a specified type.  It could be a protocol and set of tables for performing absolute calibration measurements.  Note that deliverables can include products other than text and illustration, e.g., computer codes and datasets which are linked to the main manuscript.

Appropriate review criteria include:

·       Are the deliverables clearly described and illustrated?  Does the report make appropriate use of figures and tables to present its deliverables?

·       Do the deliverables comply with report charge?  Do the deliverables address the overall purpose of the report?

·       Are the deliverables based upon a technically sound and complete execution of the approach?

·       If the deliverables are recommendations that impact scientific or clinical practice, does the report justify the costs incurred relative to anticipated benefits?  Have competing approaches been considered?

f)      Discussion (functional component)

As with any scientific paper, Discussion should address strengths and weaknesses of the proposed approach and deliverables; similarities and advantages over competing consensus report findings; implementation issues; and need for future developments.

Appropriate review criteria include:

·       For AAPM “established area” reports, do the recommendations address conflicts with past AAPM report guidance or government regulations?

·       Are potential nuances and difficulties of report implementation discussed? For example, does the report address how to adapt a generic FMEA to clinic-specific workflows?

·       Are the needs for future consensus report development or other appropriate actions identified, e.g., by AAPM, vendors, or US NRC?

·       Are weaknesses and limitations identified?  E.g., for a benchmark dataset report, is a careful uncertainty analysis provided?

g)    Conclusion (functional component)

This component should include a succinct summary of the accomplishments of the report along with a summary statement of significance and impact.

Appropriate review criteria include:

·       For long or complex prescriptive recommendations, is a condensed summary of the important “should” and “shall” recommendations included?

·       Does the Conclusion provide a useful synthesis of the report findings rather than just repeating the Deliverables component?