Difference between revisions of "Main Page"

From mn.ifi.proposalfailures
Jump to: navigation, search
(Welcome to the "learning from NFR proposal failures" Wiki !)
(12 intermediate revisions by 2 users not shown)
Line 2: Line 2:
 
'''What's the point of this?'''
 
'''What's the point of this?'''
  
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid making unnecessary mistakes in future proposals. Please feel free to add statements to this list!  ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.
+
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid unnecessary mistakes in future proposals. Please feel free to add statements to this list!  ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.
  
A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].
+
'''A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].'''
 +
 
 +
<br>
 +
 
 +
'''ABOUT ACCESS RIGHTS:'''
 +
 
 +
'''READ:''' this Wiki is ''open for read access to the whole world'' - limiting read and/or edit access to IFI or UiO was not an option (it once worked in the past, but the code that made this possible is now broken, and fixing it is not a current priority, I was told). I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, well, I'll probably have to delete this Wiki, or look for other options (I'm out of ideas, though).
 +
 
 +
'''WRITE:''' click around, visit [[Guidelines|'''this''']] page, try to log in, and you'll see that you cannot edit. Only then will I, as wiki administrator, see your user name in a list, and can give you edit rights. So, after failing to edit, send [http://www.welzl.at me] an email (usernames match UiO email addresses) and I'll give you the rights. No, [https://www.uio.no/tjenester/it/utdanning/wiki/hjelp/admin/brukerrettigheter.html there is no more convenient way] to do this.
  
NOTE: this Wiki is open for read and edit access for all members of IFI. I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict such internal publication of these reviewer feedback statements (actually, I found no statement about any restrictions at all). If I did miss such a statement, please tell me! Then, access to this Wiki could be limited more, or it could even be deleted altogether.
 
 
== The list ==
 
== The list ==
  
Line 26: Line 33:
 
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.
 
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.
 
**The scope of the project is very broad. Probably, each of these topics is a project on its own.
 
**The scope of the project is very broad. Probably, each of these topics is a project on its own.
 +
**[…] is a popular topic. Originality should be highlighted.
 +
**The actual benefits are presented as a vision, yet they are not substantiated.
 +
**The actual need or benefits are not well identified.
 +
**… some applications are only vaguely described, like […] use case …
 
*'''Problems with methods:'''
 
*'''Problems with methods:'''
 
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.
 
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.
Line 35: Line 46:
 
**How the project will differentiate from other approaches is not described in detail.
 
**How the project will differentiate from other approaches is not described in detail.
 
**No adequate comparison to existing approaches.
 
**No adequate comparison to existing approaches.
 +
*'''Other matters:'''
 +
**Ethics, safety and gender issues are not explicitly detailed in the proposal.
  
 
==== Implementation (management) ====
 
==== Implementation (management) ====
Line 40: Line 53:
 
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.
 
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.
 
**It is, however, not clear who will advise the researcher to be hired during Phase 1.
 
**It is, however, not clear who will advise the researcher to be hired during Phase 1.
*'''International cooperation:'''
+
*'''International or industry cooperation:'''
 
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.
 
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.
 
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.
 
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.
 
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.
 
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.
 +
**Engagement with stakeholders from the […] industry other than [major company] could have been stronger.
 
*'''Other management issues:'''
 
*'''Other management issues:'''
 
**The number of requested PhD students might be at the upper edge for this work.
 
**The number of requested PhD students might be at the upper edge for this work.
 
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.
 
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.
 +
**… task dependencies, timing and deliverables are not detailed.
 
**The management structure is not sufficiently described.
 
**The management structure is not sufficiently described.
 +
**Project management and governance mechanisms have not been sufficiently detailed.
 
**The proposal is rather vague about the division of research tasks and responsibilities between team members.
 
**The proposal is rather vague about the division of research tasks and responsibilities between team members.
 
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.
 
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.
 +
**The risk management is adequately described although lacking more general risks like failing to recruit the right staff.
 
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.
 
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.
  
Line 55: Line 72:
 
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.
 
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.
 
*The technological impact is described, which is however not the case for the impact on / importance for society.
 
*The technological impact is described, which is however not the case for the impact on / importance for society.
 +
*The impact of the proposed research is described in the form of specific challenges but fails to map those challenges to specific impact for industry or the society at large.
 
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.
 
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.
 
*There is a lack of KPIs for dissemination activities.
 
*There is a lack of KPIs for dissemination activities.
Line 60: Line 78:
 
*But there is a lack of information about contacts to standardisation channels.
 
*But there is a lack of information about contacts to standardisation channels.
 
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.
 
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.
*Ethics, safety and gender issues are not explicitly detailed in the proposal.
 
 
*The whole impact description appears to be detailed but quite generic.
 
*The whole impact description appears to be detailed but quite generic.
 +
*The proposed system will require systemic change across different parts of the value chain before they can become effective.
  
 
== Final words ==
 
== Final words ==
Line 68: Line 86:
 
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?'''  I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal.  
 
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?'''  I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal.  
 
* '''Why not make this page more general, instead of focusing on NFR proposals only?'''  Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.
 
* '''Why not make this page more general, instead of focusing on NFR proposals only?'''  Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.
* '''Why this page, are you just a grumpy failure man?'''  No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.
+
* '''Why this page, are you just a grumpy failure man?'''  No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing (a lot) and succeeding (a little), in two national funding bodies and the EC - but no, not ''ALL'' the comments above are from my own proposals! I just felt that this is a constructive (and easy) thing to do.
* '''Who are you anyway?'''  [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page - I hope it will develop a life of its own!
+
* '''Who are you anyway?'''  [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page (and, sigh, also the dude that needs to individually give everyone access rights... I wish there were an easier way) - in the long run, I hope that it will develop a life of its own!
 
----
 
----
 
General [[Wikiinfo|Wiki info]]
 
General [[Wikiinfo|Wiki info]]

Revision as of 11:02, 26 January 2021

Welcome to the "learning from NFR proposal failures" Wiki !

What's the point of this?

This is a catalogue of statements from NFR proposal rejection letters that are generic, i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do not include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.

A subjective interpretation of this feedback, to derive guidelines for proposals, is here.


ABOUT ACCESS RIGHTS:

READ: this Wiki is open for read access to the whole world - limiting read and/or edit access to IFI or UiO was not an option (it once worked in the past, but the code that made this possible is now broken, and fixing it is not a current priority, I was told). I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, well, I'll probably have to delete this Wiki, or look for other options (I'm out of ideas, though).

WRITE: click around, visit this page, try to log in, and you'll see that you cannot edit. Only then will I, as wiki administrator, see your user name in a list, and can give you edit rights. So, after failing to edit, send me an email (usernames match UiO email addresses) and I'll give you the rights. No, there is no more convenient way to do this.

The list

Generic reviewer statements from NFR proposal rejections

Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!

All plain text, per bullet, is directly copy+pasted from a proposal review document.

Excellence (the research idea itself, methods, ..)
  • Unclear, not enough detail, not ground-breaking enough:
    • The proposal also mentions analytic evaluations, but does not become specific there.
    • In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.
    • The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).
    • The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.
    • The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.
    • On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.
    • Some more details on how some of the questions are going to be tackled might have been useful.
    • Limited novelty due to the relatively small size of the community doing research around [..].
    • A lot of repetitions in the text because the research questions are somehow embedded in the objectives.
    • The scope of the project is very broad. Probably, each of these topics is a project on its own.
    • […] is a popular topic. Originality should be highlighted.
    • The actual benefits are presented as a vision, yet they are not substantiated.
    • The actual need or benefits are not well identified.
    • … some applications are only vaguely described, like […] use case …
  • Problems with methods:
    • The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.
    • Research methods are not well defined.
    • In some of the research activities related to the topics mentioned, quantitative research methods are missing.
    • Details on experimental evaluation are not sufficient.
  • Related work:
    • The proposal does not consider alternative Future Internet approaches.
    • How the project will differentiate from other approaches is not described in detail.
    • No adequate comparison to existing approaches.
  • Other matters:
    • Ethics, safety and gender issues are not explicitly detailed in the proposal.

Implementation (management)

  • PhD student supervision:
    • The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.
    • It is, however, not clear who will advise the researcher to be hired during Phase 1.
  • International or industry cooperation:
    • This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.
    • It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.
    • Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.
    • Engagement with stakeholders from the […] industry other than [major company] could have been stronger.
  • Other management issues:
    • The number of requested PhD students might be at the upper edge for this work.
    • No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.
    • … task dependencies, timing and deliverables are not detailed.
    • The management structure is not sufficiently described.
    • Project management and governance mechanisms have not been sufficiently detailed.
    • The proposal is rather vague about the division of research tasks and responsibilities between team members.
    • The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.
    • The risk management is adequately described although lacking more general risks like failing to recruit the right staff.
    • Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.
Impact (dissemination, exploitation, ..)
  • Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.
  • The technological impact is described, which is however not the case for the impact on / importance for society.
  • The impact of the proposed research is described in the form of specific challenges but fails to map those challenges to specific impact for industry or the society at large.
  • Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.
  • There is a lack of KPIs for dissemination activities.
  • Also, the potential exploitation of the research results is not addressed.
  • But there is a lack of information about contacts to standardisation channels.
  • Better explaining how this project will generate impact on Internet standards would strengthen the proposal.
  • The whole impact description appears to be detailed but quite generic.
  • The proposed system will require systemic change across different parts of the value chain before they can become effective.

Final words

Here's some kind of FAQ... not that anyone really asked a question, let alone frequently  :-) ...but the style seems to fit:

  • Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail? I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal.
  • Why not make this page more general, instead of focusing on NFR proposals only? Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.
  • Why this page, are you just a grumpy failure man? No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing (a lot) and succeeding (a little), in two national funding bodies and the EC - but no, not ALL the comments above are from my own proposals! I just felt that this is a constructive (and easy) thing to do.
  • Who are you anyway? Michael. But I'm only the dude who started this page (and, sigh, also the dude that needs to individually give everyone access rights... I wish there were an easier way) - in the long run, I hope that it will develop a life of its own!

General Wiki info