Initial UFIRST Proposal Reports are now available via Enterprise Reporting. The reports are available through the myUFL portal. Anyone with the appropriate security roles has access via Enterprise Reporting>Access Reporting>Sponsored Research Information>UFIRST.

The first report is the “Submitted Proposals Activity Report” providing a listing of each proposal by submitting department or by PI. It is available in two formats, optimized differently for easy printing to a PDF or exporting to Excel.  More standardized reports will be released as they become available later this summer and fall.

Raw UFIRST data is also available in the “UFIRST Data” folder, and you are free to export to Excel and build reports tailored to your local needs. (For ease, view in Excel 2007 format and Save As a new file on your personal computer).

Raw UFIRST data is also available via Query Studio, and you are free to build queries and reports as needed. Access to Query Studio does require training and an additional security role.

A Data Dictionary is also available for assistance in understanding the data elements included in the raw data http://hr.ufl.edu/learn-grow/just-in-time-training/myufl-toolkits/grants/.

Guides for Enterprise Reporting Basics are available http://hr.ufl.edu/learn-grow/just-in-time-training/myufl-toolkits/enterprise-reporting-basics/ as is general information about Enterprise Reporting http://www.it.ufl.edu/myufl/reporting/.

If you are having issues accessing the reports, please contact the UF Computing Help Desk (392-HELP). For questions about the data or future UFIRST reporting suggestions, please contact Lisa Stroud in the UF Office of Research.

Excerpted from Institute of Education Sciences Procedures for Peer Review of Grant Applications

The Institute of Education Sciences (IES) Standards and Review Office is responsible for the IES scientific peer review process. Highly qualified individuals with an in-depth knowledge of the subject review and evaluate all applications for grants and cooperative agreements exceeding $100,000.

Standards and Review Office staff work with a contractor that handles the logistics and manages the electronic system for grant application submission and review. The review process includes the following steps:

1. Pre-submission procedures
2. Application processing procedures
3. Selection and appointment of scientific review panels
4. Pre-review panel meeting process
5. Panel meeting process
6. Post-review panel meeting process

1. Pre-Submission Procedures

Letters of Intent

IES encourages potential applicants to send a letter of intent (LOI). The LOI is optional, is not binding, and does not enter into the review of subsequent applications but allows IES staff to estimate the potential workload to plan the review.

2. Application Processing Procedures

Screening Procedures

Upon submission, applications are screened for compliance with the application rules, responsiveness to the Request for Applications, and identification of resubmissions.

• Compliance. IES staff screen each application for compliance with the application rules (e.g., page length and formatting requirements, completion of all parts of the application).

• Responsiveness. IES staff screen each application for responsiveness to the Request for Applications (e.g., submitted to wrong competition/goal or non-responsive). For applications submitted to the wrong competition or goal, IES contacts the applicant to discuss the appropriate placement of the application. Non-responsive applications are those that do not meet the basic requirements. IES takes a conservative approach to responsiveness screening, and relatively few applications are judged to be non-responsive.

• Resubmission. IES staff screen each application for text in the abstract, narrative, or Appendix A indicating that the applicant meant for the application to be a resubmission but did not check the appropriate resubmission box on the application form. If the applicant confirms that the application is a resubmission, reviews from the previous year are sent along with the application to reviewers.

Assignment of Applications to Panel

When assigning applications to panels, IES considers any conflict of interest, the match between the overall expertise of reviewers and methodological approach proposed, and the number and type of reviewers.

• Initial Conflict of Interest Determination. Prior to assignment of applications, IES staff review each application to identify possible conflicts of interest by comparing the individuals associated with each application against the individuals on each panel and the applicant institutions with the organizational affiliations of each panel member.

• Match of Expertise. Applications are assigned to reviewers according to the match between the expertise of each reviewer and the content and methodological approach proposed in each application, with consideration given to the balance of applications across reviewers.

• Number and Type of Reviewers. Each application is assigned to at least two reviewers. Applications to conduct randomized trials submitted to Goal 3 (efficacy evaluations) or Goal 4 (effectiveness evaluations) have three reviewers, one of which is an experienced methodologist in the implementation and evaluation of randomized trials. Applications to conduct studies implementing single-subject methodologies are assigned at least one reviewer who is experienced in the implementation and analysis of single-subject studies.

3. Selection and Appointment of Scientific Review Panels

Selection of Panel Members

Scientific review panels are composed of a panel chair, a scientific review administrator, and panel members. The deputy director for science appoints the panel chair from among the members of the panel. IES identifies highly qualified potential reviewers who have the scientific expertise to evaluate the proposals on the criteria listed in the Request for Applications. Potential reviewers are identified primarily on the basis of quality of the research they have conducted and published in scientific peer-reviewed journals and the degree to which they are an in-depth expert in the research methods and subject matter that are relevant to the applications dealt with by any given panel. IES uses three types of reviewers on its panels:

• Principal Panel Member. Principal panel members serve 1 to 3 year terms on a review panel or until the specific research grant competition assigned to the panel is discontinued. Only standing panels (see below) have principal panel members.

• Rotating Panel Member. Rotating panel members are appointed to a particular panel (standing or single session) for one review session and serve as full members of the panel, receiving a full assignment of applications to review, attending the panel review meeting, and scoring all applications considered by the panel.

• Ad Hoc Panel Member. Ad hoc panel members are appointed to a particular panel (standing or single session) for one review session and are typically assigned two to four applications to review. Ad hoc panel members participate in the full panel review meeting on those applications for which they served as a primary reviewer. They do not participate in the discussion and scoring of any application for which they were not a primary reviewer. Their participation is typically via teleconference. Ad hoc reviewers are recruited when the number of applications received is greater than what a panel can efficiently handle or when particular expertise is needed for a specific application.

Types of Panels

IES has two types of scientific review panels:

• Standing Panels. For standing panels, members may be appointed for multiple, consecutive review sessions. Standing panels may be composed of principal, rotating, and ad hoc members. At least 50%–100% of the members of a full panel are principal panel members. A typical full panel is limited to 15 to 20 members, in order to facilitate discussion of the applications. The remaining members of the full panel are rotating panel members or ad hoc panel members. Generally, no more than three or four ad hoc panel members are used to supplement a panel.

• Single Session Panels. Single session panel members are appointed for one review meeting and may be composed of rotating and ad hoc members. A typical full panel is limited to 15 to 20 members, in order to facilitate discussion of the applications. Generally, no more than three or four ad hoc panel members are used to supplement a panel.

Recruitment of Panel Members

The identification of potential panel members is a continuous process involving an annual call for nominations from the IES board, consideration of IES report external reviewers, solicitation of suggestions from IES and center experts, literature review, and networking with known experts in relevant fields.

Conflicts of Interest

IES is concerned with three types of conflicts of interest:

• Personal Financial Interest. IES considers reviewers to be in conflict if they can benefit financially from the outcome of the review. An individual may not serve on a panel if the individual is part of an application that is being considered by the panel. Also, an individual may not serve on a panel if he or she has a financial interest in a for-profit organization that has an application being considered by the panel or if his or her spouse, partner, child, household member, or other relative has a financial interest in the outcome of the review.

• Personal or Professional Relationship. Individuals may not serve as a primary reviewer for any application involving individuals with whom they have a close personal or professional relationship. Individuals may not participate in the review of applications from individuals with whom they have professional differences that could reasonably be viewed as affecting the objectivity of their review.

• Professional Relationship with an Applicant’s Institution. Individuals may not serve as a primary reviewer for any application submitted by other individuals from the institution or organization that employs them.

4. Pre-review Panel Meeting Process

Contact with Panel Members

After individuals have agreed to participate on a panel, their names are given to the logistics contractor so that all necessary forms are completed (e.g., Conflict of Interest, Consulting Agreement) prior to the release of applications to the panel and travel arrangements can be made in a timely fashion.

Instructions to Panel Members

There are two primary means for providing guidance to panel members prior to the actual panel review session:

• Pre-Meeting Orientation Teleconference. Approximately 6 to 7 weeks before the panel review session, a pre-meeting orientation teleconference is conducted with IES staff, the logistics contractor, and the panel members. The purpose is to provide reviewers with an overview of the peer review process, identify key points in the relevant Requests for Applications, familiarize the reviewers with the scoring system, emphasize the need for looking over the applications as soon as they receive them to identify any conflicts of interest with assigned applications, and stress the importance of completing reviews by the due date.

• Peer Review Handbook. Each panel member receives a Peer Review Handbook providing detailed information about the responsibilities of the panel members and the panel chair.

Release of Applications to Reviewers

About 6 to 7 weeks prior to the panel meeting, each reviewer receives hard copies of the applications that have been assigned to the reviewer. At this time, each reviewer is also given access to those applications that have been assigned to him or her through the electronic peer review system. In addition, if any of the assigned applications are resubmissions, the reviewer will receive the previous year’s reviews for that application.

Review Criteria

All IES regular research competitions have a common goal structure that categorizes the types of research projects funded and the requirements for projects submitted under each goal. These requirements and the review criteria are described in each Request for Applications. Reviewers are expected to assess the significance, research plan, personnel, and resources of an application to judge whether the proposed research will have a substantial impact on the pursuit of that goal.

• Significance. Does the applicant make a compelling case for the potential contribution of the project to the solution of an education problem? For development projects, does the applicant present a strong rationale, including a theoretical foundation and prior empirical evidence, justifying the development of the selected intervention or assessment? For cases in which the applicant proposes to evaluate an intervention, does the applicant present a strong rationale justifying the need to evaluate the selected intervention (e.g., does prior evidence suggest that the intervention is likely to substantially improve student learning and achievement)?

• Research Plan. The primary methodological issue is whether the research plan is appropriate for answering the research questions or testing the proposed hypotheses. Reviewers are asked to consider whether the applicant presents (a) clear hypotheses or research questions; (b) clear descriptions of and strong rationales for the sample, the measures (including information on the reliability and validity of measures), data collection procedures, and research design; and (c) a detailed and well-justified data analysis plan.

• Personnel. Does the description of the personnel make it apparent that the principal investigator, project director, and other key personnel possess the training and experience and will commit sufficient time to competently implement the proposed research?

• Resources. Does the applicant have the facilities, equipment, supplies, and other resources required to support the proposed activities? Do the commitments of each partner show support for the implementation and success of the project?

Review of Applications by the Primary Reviewers

Typically, each panel member is assigned to be a primary reviewer for about eight applications. The primary reviewers prepare written critiques of the applications to which they have been assigned, addressing the strengths and weaknesses of each application. Reviews are submitted electronically through the electronic peer review system.

Scoring of Applications

Primary reviewers assign a score for each review criterion (significance, research plan, personnel, and resources) and an overall quality score for each application. Reviewers are asked to score each application against an ideal application, as opposed to evaluating applications against the other applications submitted to a competition.

• Criteria Scores. Each criterion is rated on a 7-point scale from 1 (poor) to 7 (excellent), where intermediate values are treated as equal steps along the scale.

• Overall Scores. Each reviewer rates the overall quality of the application on a scale from 5.0 (poor) to 1.0 (outstanding). The overall score is not an average of the criterion scores or a mathematical derivative of the criterion scores. Reviewers are given the adjectival equivalents listed below to help anchor their scores:

Overall Score Range	Adjectival Equivalent
1.0 to 1.5	Outstanding
1.6 to 2.0	Excellent
2.1 to 2.5	Very Good
2.6 to 3.0	Good
3.1 to 4.0	Fair
4.1 to 5.0	Poor

 

Triage of Applications

To facilitate the discussion by the full panel of the most competitive applications, IES generally employs a system of triaging so that only the most competitive applications are discussed and scored by the full panel at the panel review meeting. Determination of the most competitive applications is based on the preliminary rank order of the average overall scores of the applications.

• Average Overall Score Report. Based on the overall scores assigned by primary reviewers, an average overall score for each application is calculated and used to generate a preliminary rank order of applications. Reviewers must submit their scores and critiques of the applications 10 business days prior to the panel meeting so that reviewers have sufficient time to become familiar with any of the applications for which they were not a primary reviewer.

• Triage Cut Point. In general, the triage cut point is set so that the full panel discusses about 25 applications. Triage cut points may vary across competitions when a panel reviews applications for more than one competition.

• Panel Nominations of Applications to be Considered by the Full Panel. At the beginning of each panel meeting, panel members are given the opportunity to nominate for consideration by the full panel any application that they reviewed and judged to be of equal merit to the applications being considered by the full panel on the basis of their average overall score. This process is described in the section on the process of the panel meeting.

• Discrepant Scores. Primary reviewers may reach different judgments (i.e., scores) about the merit of a particular application. IES staff members calculate the difference in overall scores between reviewers for each application and then the average of the differences between scores across applications. For applications receiving only two primary reviews, applications that have widely discrepant scores are identified and a third review by a panel member is obtained. All three reviewers’ scores are then used to calculate the average overall scores of such applications.

Applications to be Discussed by the Full Panel    

After the triage has been determined, panel members are notified of the applications to be discussed and scored by the full panel. Notification is done as soon as possible to allow panel members sufficient time to become familiar with the applications that will be discussed by the full panel.

5. Panel Meeting Process

Review of Procedures

The evening before a panel review session, the deputy director for science meets with the panel chairpersons to go over the panel meeting procedures and discuss any issues that have arisen during the application review period.

Plenary Session

Typically the panel sessions begin with a short plenary session during which members of all of the panels meet together. The deputy director for science provides a brief overview of the competitions for the panel members. At the end of the plenary session, panels are dismissed to convene their separate review meetings.

Panel Orientation     

Each panel review session begins with a brief introduction and orientation. Panel members and all observers introduce themselves. The order of procedures for the panel and materials in the panel notebook are reviewed.

Nomination of Triaged Applications for Discussion by the Full Panel

The panel chair begins by asking panel members if anyone wishes to nominate for consideration by the full panel any applications that they reviewed that did not survive triage but in their opinion were of equal scientific and technical merit to applications that are being considered by the full panel. If a panel member chooses to nominate an application, he or she is given 2 minutes to present the case for including the application. A vote of the full panel occurs and a simple majority vote determines whether the application is added to the panel’s agenda.

Order of Review

IES staff determine the order of review before the panel meeting. Typically, applications are first sorted by competition goal, and then, within each goal, applications are sorted by competition. Within each goal and competition category, applications are randomly ordered to the extent possible (e.g., consideration is generally given to arranging the order so that the same reviewer is not a primary reviewer on consecutive applications).

• Primary Reviewers. In the order of review, the primary reviewers are assigned to be reviewer 1, reviewer 2, or reviewer 3. In the panel meeting, reviewer 1 presents a short overview of the application in addition to his or her critique of the application. Reviewers 2 and 3 are asked not to repeat what reviewer 1 has said, but to provide additional comments and to note where they might disagree with reviewer 1.

• Note Taker. For each application, a panel member other than one of the primary reviewers for an application is assigned the task of being the note taker who provides a summary of the panel discussion.

Review of Applications

The review of applications begins when the chair identifies the application to be reviewed and asks anyone who has a conflict of interest to leave the room.

Scores of Primary Reviewers

The scores of the primary reviewers are presented to the panel.

Presentation of Application and Critiques by Primary Reviewers

Reviewer 1 presents a brief description of the application and his or her critique of the application. Reviewer 2 then provides any additional comments about the application. If there is a third reviewer, reviewer 3 also gives his or her critique. During these presentations the primary reviewers are invited to express any strong personal points of view or personal perspectives, positive or negative, with respect to the application.

Panel Discussion     

After the presentations of the primary reviewers, the entire panel discusses the merits of the application. Any panelist intending to comment is invited to express any strong points of view or personal perspectives. The chair and the scientific review administrator are responsible for ensuring that the discussion focuses on the scientific and technical merit of the application. The chair is responsible for ensuring that the application is reviewed according to the criteria described in the Request for Applications.

Comments on Budget

After the discussion of the scientific and technical merit of the application, the primary reviewers may offer comments on the appropriateness of the budget.

Summary of Panel Discussion by Note Taker

The note taker reads his or her summary of the panel discussion. The chair is responsible for ensuring that the summary reflects the issues that were raised by the panel. After hearing the summary, the chair asks panel members if they concur with the summary as presented or if any additions or corrections are necessary.

• Minority report. On occasion, two or more panel members disagree with the majority view of the application as presented by the note taker. In such instances, these reviewers may submit a comment reflecting their view of the application.

Revisit Scores of Primary Reviewers        

After the panel discussion, the primary reviewers are given the opportunity to change any of their original scores.

• Revision of Reviews. If a primary reviewer substantially changes his or her scores based on the panel discussion, the reviewer is asked to make any necessary revisions to his or her written review so that the scores are aligned with the review.

Confidential Scoring by All Panel Members

After the primary reviewers have indicated any changes in their scores, all panel members, acting independently and individually, score the application. A scoring sheet is provided in the review panel notebook with applications listed in order of review. At the end of each day, the scoring sheets are collected.

Funding Enthusiasm Scores          

After all of the applications have been discussed and scored, reviewers are asked to give each application a funding recommendation: (a) highly recommend funding, (b) recommend funding, or (c) do not recommend funding.

6. Post-Review Panel Meeting Process

Summary Reports

The Standards and Review Office provides two basic reports to the program officers after each panel meeting:

• Overall Score Report. The Final Scientific Merit Report includes average criteria and average overall scores for each application. The averages are based on the scores provided by the full panel, excluding any members who were in conflict.

• Funding Enthusiasm Report. The Funding Enthusiasm Ratings Report is the average of the funding enthusiasm ratings provided by the full panel, excluding any members who were in conflict.

Feedback to Applicants

All applicants receive copies of the primary reviewers’ critiques. For applications receiving full panel discussion, a summary of those comments about the application is also sent. These materials are sent to the principal investigator and to the applicant institution’s authorized representative.

Procedures for Responding to Applicant Inquiries

Applicants who have questions regarding the review of their application are directed first to the cognizant program officer. If an applicant questions whether his or her proposal received appropriate treatment and is not satisfied with the program officer’s feedback, the inquiry is directed to the deputy director for science. The deputy director for science reviews the process under which the proposal was handled to determine if there was anything irregular or inappropriate, which is the only basis on which the results of a grant competition review can be put aside. A re-review of an application is the only remedy available for an application that the deputy director for science judges to have been mishandled. A re-review consists of a review of the same application, not a revised version, during the next scheduled panel review meeting for recurring competitions. The re-review will occur without panel access to the reviews from the contested review session. In instances where competitions occur infrequently, the application will be re-reviewed during the fiscal year when the next competition to which the application was submitted occurs.