The process of peer review is the cornerstone of the research evaluation process in the physical sciences, life sciences and engineering.
All CCST reports are peer reviewed using guidelines and processes established by CCST to assure the highest scientific and technical standards. Guidelines are similar to those of the National Academy of Sciences, adapted to be appropriate for California. Reviewers are not asked to approve the report or to replace the scientific judgment of the committee with their own, but rather asked to indicate whether:
- the report is clear and concise,
- its arguments and conclusions appear to rest on adequate data, properly represented,
- uncertainties in the data are recognized,
- policy matters are handled objectively,
- the report reveals or suggests bias, and
- the report seems to be complete, fair, and responsive to the committee's charge.
This process ensures the credibility and authority of every CCST report by subjecting it to critical review by a body of peers highly knowledgeable in the subject matter. Adherence to the review process protects against the report taking a narrow or parochial view of a problem, or failing to consider fully or properly document data or information pertinent to the issue under review. The process is particularly aggressive in differentiating opinions and judgment findings of fact well grounded in research.
CCST Report Review Process
The timeline for the review process depends on the scope and size of the report and takes approximately one to three months.
Phase 1 - Identify Appropriate Expert Reviewers
- The number of reviewers depends on size and scope of the report
CCST will select independent scientific expert reviewers for review of products and reports based upon expert recommendations.
- Review Oversight - CCST selects one or more individuals who will oversee the review process. Persons serving in this capacity are termed Review Monitors. If more than one Review Monitor is appointed, the report under review can be divided into sections to distribute review load.
- CCST evaluates the product to determine the range of expertise needed. CCST generates an evaluation request in order to compile internal recommendations for expert reviewers.
- Peer Reviewers - CCST sends the expert review request to the CCST Council, Board and other recognized experts. The experts may suggest him/herself. CCST compiles the final recommendation list, and further vets it through members of CCST's Oversight Committee and makes a final determination for reviewers using the criteria below:
- Expertise: The reviewer should have demonstrated knowledge, experience, and skills in an area relevant to the product or report.
- Objectivity: The reviewer should be able to provide an objective, open minded, and thoughtful review, in the best interest of a thorough and objective review of the product. In addition, the reviewer should be comfortable sharing his or her knowledge and perspectives and openly identifying his or her knowledge gaps.
- Conflict of Interest: The reviewer should not have any financial or other interest that conflicts or that could impair his or her objectivity or create an unfair competitive advantage (see section on conflict of interest below).
- CCST contacts and engages expert reviewers. The names and affiliations of the expert reviewers who accept the request are made public only when the report is released.
Phase 2 - The Evaluation Process
- CCST prepares detailed review instructions and questions based on the report and the needs of the client. CCST provides these instructions and a copy of the report to the reviewers.
- Reviewers are asked to prepare written responses according to the instructions. CCST receives reviewer comments and compiles and organizes reviewer comments into a summary document.
Phase 3 - Comments Summary and Response to Review
- CCST will present all the written comments received to the project team.
- The project team will address each of the comments and amend the report as deemed appropriate.
- The revised report will be reviewed by the Review Monitor to determine if the comments have been adequately addressed.
- Finally, CCST prepares and provides to the client, the final report.
Conflicts of Interest
A reviewer cannot review a product if any of the parties below have a specific financial interest in the outcome of the review:
- The reviewer, the reviewer's spouse, minor child, or business partner;
- The organization where the reviewer is employed, has an arrangement for future employment or is negotiating for employment; or
- The organization where the reviewer is an officer, director, trustee, or partner.
Furthermore, a potential reviewer is asked to disclose other potential conflicts of interest, including having a personal relationship with an author, such as a close relative, current or former collaborator, or former thesis student/advisor. Potential conflicts will be considered and may exclude a potential reviewers' participation.
A potential reviewer is instructed to disclose if the report involves an institution or other entity with which the potential reviewer has an association. Conflicts of association include:
- A reviewer's recent former employer;
- An organization in which the reviewer is an active participant;
- An institution at which the reviewer is currently enrolled as a student, or at which he/she serves as a visiting committee member; or
- An entity with which the reviewer had, has or seeks some other business or financial relationship (including receipt of an honorarium).
Conflicts of association, such as those above, may preclude a reviewer's participation.