In January 2012, IOS implemented a new procedure for assessing research proposals. Instead of receiving full proposals twice per year, the division now has a single annual date for receipt of short preliminary proposals (pre-proposals). Authors of the most meritorious pre-proposals are then invited to submit full proposals, and awards are made from this pool. This change was made in response to a steady increase in the proposal workload both within the Division and in the scientific community. For example, in 2011, program officers made 14,000 requests for ad hoc reviews. This increased workload ultimately jeopardized our ability to perform quality merit review of the proposals submitted. The goals of the new submission procedure were to reduce the burden on the principal investigator (PI) and reviewer communities, as well as to maintain high-quality merit review.
Currently, we have completed one full cycle from submission of pre-proposals to final awards (FY 2013). The final decisions on the second cycle of full proposals (FY14) and the review of pre-proposals for the third cycle (FY15) are ongoing. To date, we have heard a number of concerns regarding the outcomes of this transition. Therefore, we performed a preliminary analysis of the proposal submission and success rates for the full first cycle compared to the four years prior, which operated under a semi-annual cycle of full proposals. Through surveys given to panelists, we have also been assessing the impact of the pre-proposal process on the quality of proposals, on the review process, and on the quality of merit review. We will discuss some of our major findings in a series of blog posts.
Groups of concern
We have evaluated the potential impacts of this new process on various groups of concern. One issue raised by the community was the potential adverse effects on beginning investigators, presumably due to reduced chances for funding prior to tenure review, as well as the limited ability to demonstrate feasibility with pilot data in pre-proposals. However, we have found an increase in the fraction of submitted pre-proposals and no change in the percentage of funded awards to beginning investigators in the entire portfolio. Similarly, the overall fraction of awards to female investigators (30%), under-represented minorities (6%) and primarily undergraduate institutions (~6%) remained unchanged in the new system.
We’d like to highlight that beginning investigators are able to submit proposals to our core programs more than once per year. They may submit up to 2 pre-proposals in January to IOS. Then, as pre-tenure investigators they may submit a CAREER proposal in July, even if they were not invited to submit a full proposal after review of their pre-proposal(s). In this way, beginning PIs are able to submit proposals twice per year, and can use the pre-proposal system to get feedback on ideas for their CAREER proposals. They may also submit to any other solicitations including those in other divisions such as DEB or MCB.