In January 2012, IOS implemented a new procedure for assessing research proposals. Instead of receiving full proposals twice per year, the division now has a single annual date for receipt of short preliminary proposals (pre-proposals). Authors of the most meritorious pre-proposals are then invited to submit full proposals, and awards are made from this pool. This change was made in response to a steady increase in the proposal workload both within the Division and in the scientific community. For example, in 2011, program officers made 14,000 requests for ad hoc reviews. This increased workload ultimately jeopardized our ability to perform quality merit review of the proposals submitted. The goals of the new submission procedure were to reduce the burden on the principal investigator (PI) and reviewer communities, as well as to maintain high-quality merit review.

Currently, we have completed one full cycle from submission of pre-proposals to final awards (FY 2013). The final decisions on the second cycle of full proposals (FY14) and the review of pre-proposals for the third cycle (FY15) are ongoing. To date, we have heard a number of concerns regarding the outcomes of this transition. Therefore, we performed a preliminary analysis of the proposal submission and success rates for the full first cycle compared to the four years prior, which operated under a semi-annual cycle of full proposals. Through surveys given to panelists, we have also been assessing the impact of the pre-proposal process on the quality of proposals, on the review process, and on the quality of merit review. We will discuss some of our major findings in a series of blog posts.

Submissions per year and overall funding rate

One immediate concern raised at the time of the transition was the loss of additional submission opportunities due to the annual cycle. So far, there has indeed been a small decrease in the number of PIs submitting more than one proposal per year. However, as the table below indicates, a majority of PIs only submit a single proposal per year, even under the previous submission system.

(Number of submissions indicates submissions as Lead PI to IOS core programs, including CAREER proposals.)

A second concern expressed by the PIs was the relatively long lead-time from initial submission of a project in the form of a pre-proposal to its eventual award, given the annual submission cycle. While we understand that the time to award may be a challenge of the new system, we anticipate that PIs will adapt accordingly in the coming years. Again, we would like to note that the transition to the pre-proposal system was made in response to increased workloads negatively impacting the quality of merit review. This has become an issue throughout the National Science Foundation, and many divisions and directorates are responding in similar ways (see analysis conducted by BIO/DEB here).

We have evaluated the overall success rates of the first year of the pre-proposal/full proposal system. In the first full cycle, nearly 30% of pre-proposals were invited for a full submission and 175 proposals (representing nearly 150 projects) were ultimately awarded. This equates to an overall success rate of 8.2%. Note that the number and size of awards is not affected by the change to a pre-proposal process. This change simply altered the manner in which the most meritorious grants were identified. The pre-proposal process had no effect on the money available to IOS or on the budgets (either requested or awarded) in proposals.

The pre-proposal/full proposal system achieved the goals of

1. reducing the workload imposed on the scientific community for ad hoc reviews,
2. lessening the number of proposal pages written by PIs, and
3. simplifying the documents required from Sponsored Research Offices since pre-proposals do not have to specify a budget.

A more detailed look at groups of concern, as well as the responses from the community on the reviewing workload and merit review quality will be discussed in subsequent posts in this series.