https://wiki.uio.no/mn/ifi/proposalfailures/api.php?action=feedcontributions&user=Michawe%40uio.no&feedformat=atommn.ifi.proposalfailures - User contributions [en]2024-03-29T11:40:29ZUser contributionsMediaWiki 1.27.4https://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Guidelines&diff=57Guidelines2021-06-30T11:56:01Z<p>Michawe@uio.no: /* Clarity, being ground-breaking, ..: */</p>
<hr />
<div>'''These guidelines are derived as ''subjective'' interpretations of the reviewer statements on [[Main Page|this page]].'''<br />
==Excellence (the research idea itself, methods, ..)==<br />
<br />
=== '''Clarity, being ground-breaking, ..:''' ===<br />
*Project outputs: be specific about analytic evaluations.<br />
*Describe concrete project results (e.g., simulation model, prototype etc.) in sufficient detail. Benefits shown as a vision are not concrete enough.<br />
*Be clear about goals. E.g., when describing a % improvement over the state of the art, clarify which metrics will be used to quantify the improvement.<br />
*Be ambitious. E.g., 20% better than the SoA is not enough. Explain the ground-breaking nature of the research and its generalisability.<br />
*Avoid looking incremental - careful with showing initial results.<br />
*Be clear about the research plan: how will the research questions be tackled?<br />
*Avoid looking too narrow: there should be a reasonably large research community interested in your work.<br />
*Avoid looking too broad: research objectives should not seem like they could each be a project in its own right.<br />
*Avoid redundancy in the text - e.g., don't embed the research questions in the objectives.<br />
*Think twice about which field to address (try to aim for new fields, if possible).<br />
*Make sure to motivate and explain all choices well.<br />
<br />
=== '''Methods:''' ===<br />
*If the focus is on experiments, 1) provide sufficient details on experimental evaluation, and 2) consider adding complementary analytic methods to also attain more theory-oriented scientific achievements.<br />
*Make sure that quantitative research methods are described for all the research activities related to the topics mentioned.<br />
<br />
=== '''Related work:''' ===<br />
*Provide a comparison with alternative approaches; clarify how the project will differentiate from them. Highlight originality.<br />
<br />
=== '''Other matters:''' ===<br />
* Explicitly address ethics, safety and gender issues.<br />
<br />
==Implementation (management)==<br />
<br />
=== '''PhD student supervision:''' ===<br />
*State who will advise the Ph.D. students.<br />
*Give enough details regarding the supervision of PhD students, especially when the project manager's own research has previously had a different focus.<br />
<br />
=== '''International cooperation:''' ===<br />
*When cooperating with international partners, explain how the international consortium will be managed.<br />
*When cooperating with industry, engage with other stakeholders from the same industry beyond only one major company.<br />
*Clarify how collaboration will work in the project (a plan of exchange visits, and stating the intention to submit joint publications is not enough).<br />
*If external collaborators primarily have the expertise on a topic, this is a risk for the project. Avoid this, or state it as a risk and provide a mitigation plan.<br />
<br />
=== '''Other management issues:''' ===<br />
*Consider including (a) work package(s) for integration, result evaluation, project management and dissemination.<br />
*Describe the management structure in sufficient detail.<br />
*Describe task dependencies, timing and deliverables.<br />
*Clarify the division of research tasks and responsibilities between team members.<br />
*Describe WP results not only in terms of of publications but in terms of concrete achievements and developments.<br />
*Avoid looking too generic with the risk management plan (still, include general risks like failing to recruit the right staff, and delay in hiring people); mitigation actions must be detailed and clear.<br />
*A "waterfall arrangement" work package structure may look too simple. Clarify the timeline and dependencies.<br />
*Search the proposal for "he", "his" and "him" before submitting to say she/he instead, wherever appropriate.<br />
<br />
==Impact (dissemination, exploitation, ..)==<br />
*If dissemination activities only describe publications in journals and international conferences, consider adding something more. Maybe what the EC calls "communication", i.e. talking to the public, via other media?<br />
*In addition to the technological impact, describe the impact on / importance for society at large and/or industry.<br />
*To convince reviewers that there will be a significant impact on the research community, avoid too many self-cites in the reference list.<br />
*Provide KPIs for dissemination activities. Plan for enough papers!<br />
*When working across multiple fields, ensure that the publication plans evenly spreads across all of them.<br />
*Address the potential exploitation of the research results.<br />
*In case of doing standardisation, provide information about contacts to standardisation channels, and explain in detail how an impact in standardisation will be achieved.<br />
*Explicitly address ethics, safety and gender issues.<br />
*Explicitly address societal impact.<br />
*Avoid making the impact description too generic looking. It must be specific to the project.<br />
*Avoid road-blocks like requiring a systemic change across different parts of the value chain before the proposed system can become effective.</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=56Main Page2021-06-30T11:55:20Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.<br />
<br />
'''A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].'''<br />
<br />
<br><br />
<br />
'''ABOUT ACCESS RIGHTS:'''<br />
<br />
'''READ:''' this Wiki is ''open for read access to the whole world'' - limiting read and/or edit access to IFI or UiO was not an option (it once worked in the past, but the code that made this possible is now broken, and fixing it is not a current priority, I was told). I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, well, I'll probably have to delete this Wiki, or look for other options (I'm out of ideas, though).<br />
<br />
'''WRITE:''' click around, visit [[Guidelines|'''this''']] page, try to log in, and you'll see that you cannot edit. Only then will I, as wiki administrator, see your user name in a list, and can give you edit rights. So, after failing to edit, send [http://www.welzl.at me] an email (usernames match UiO email addresses) and I'll give you the rights. No, [https://www.uio.no/tjenester/it/utdanning/wiki/hjelp/admin/brukerrettigheter.html there is no more convenient way] to do this.<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
**[…] is a popular topic. Originality should be highlighted.<br />
**The actual benefits are presented as a vision, yet they are not substantiated.<br />
**The actual need or benefits are not well identified.<br />
**… some applications are only vaguely described, like […] use case …<br />
**On the downside, this is an area of the literature that is already very crowded. This creates the risk that the project may lead to incremental advancements of the state of the art.<br />
**Some realization points of the [..] framework are not very clear. Often the choices are taken without a motivation and explanation, even though the proposer demonstrated a very good knowledge of his specific field.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
*'''Other matters:'''<br />
**Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International or industry cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
**Engagement with stakeholders from the […] industry other than [major company] could have been stronger.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**… task dependencies, timing and deliverables are not detailed.<br />
**Work packages are well planned, but results are not appropriately described. Results are mainly publications but not the concrete achievements and developments of the work.<br />
**The management structure is not sufficiently described.<br />
**Project management and governance mechanisms have not been sufficiently detailed.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**The risk management is adequately described although lacking more general risks like failing to recruit the right staff.<br />
**One risk that is not discussed is the delay in hiring the full-time researcher: in this case the PM will be the only senior member at UiO and there is the risk that the PI may not have sufficient time to allocate to the project.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
**Regarding this sentence: "IFI has well-established procedures to ensure that a Ph.D. student can pass _his_ work successfully." Does the Ph.D. program at IFI support only male students?<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*The impact of the proposed research is described in the form of specific challenges but fails to map those challenges to specific impact for industry or the society at large.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
*The proposed system will require systemic change across different parts of the value chain before they can become effective.<br />
*The planned publication output is not very ambitious: only one conference paper per year, albeit in very competitive venues.<br />
*Societal impact is not analyzed in detail.<br />
*In terms of publications the work can have great potential in the conferences and journals. However, those are rather focusing on [main field], but less on [two other fields that this project interacts with].<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing (a lot) and succeeding (a little), in two national funding bodies and the EC - but no, not ''ALL'' the comments above are from my own proposals! I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page (and, sigh, also the dude that needs to individually give everyone access rights... I wish there were an easier way) - in the long run, I hope that it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=55Main Page2021-06-30T11:53:32Z<p>Michawe@uio.no: /* Implementation (management) */</p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.<br />
<br />
'''A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].'''<br />
<br />
<br><br />
<br />
'''ABOUT ACCESS RIGHTS:'''<br />
<br />
'''READ:''' this Wiki is ''open for read access to the whole world'' - limiting read and/or edit access to IFI or UiO was not an option (it once worked in the past, but the code that made this possible is now broken, and fixing it is not a current priority, I was told). I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, well, I'll probably have to delete this Wiki, or look for other options (I'm out of ideas, though).<br />
<br />
'''WRITE:''' click around, visit [[Guidelines|'''this''']] page, try to log in, and you'll see that you cannot edit. Only then will I, as wiki administrator, see your user name in a list, and can give you edit rights. So, after failing to edit, send [http://www.welzl.at me] an email (usernames match UiO email addresses) and I'll give you the rights. No, [https://www.uio.no/tjenester/it/utdanning/wiki/hjelp/admin/brukerrettigheter.html there is no more convenient way] to do this.<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
**[…] is a popular topic. Originality should be highlighted.<br />
**The actual benefits are presented as a vision, yet they are not substantiated.<br />
**The actual need or benefits are not well identified.<br />
**… some applications are only vaguely described, like […] use case …<br />
**On the downside, this is an area of the literature that is already very crowded. This creates the risk that the project may lead to incremental advancements of the state of the art.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
*'''Other matters:'''<br />
**Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International or industry cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
**Engagement with stakeholders from the […] industry other than [major company] could have been stronger.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**… task dependencies, timing and deliverables are not detailed.<br />
**Work packages are well planned, but results are not appropriately described. Results are mainly publications but not the concrete achievements and developments of the work.<br />
**The management structure is not sufficiently described.<br />
**Project management and governance mechanisms have not been sufficiently detailed.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**The risk management is adequately described although lacking more general risks like failing to recruit the right staff.<br />
**One risk that is not discussed is the delay in hiring the full-time researcher: in this case the PM will be the only senior member at UiO and there is the risk that the PI may not have sufficient time to allocate to the project.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
**Regarding this sentence: "IFI has well-established procedures to ensure that a Ph.D. student can pass _his_ work successfully." Does the Ph.D. program at IFI support only male students?<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*The impact of the proposed research is described in the form of specific challenges but fails to map those challenges to specific impact for industry or the society at large.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
*The proposed system will require systemic change across different parts of the value chain before they can become effective.<br />
*The planned publication output is not very ambitious: only one conference paper per year, albeit in very competitive venues.<br />
*Societal impact is not analyzed in detail.<br />
*In terms of publications the work can have great potential in the conferences and journals. However, those are rather focusing on [main field], but less on [two other fields that this project interacts with].<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing (a lot) and succeeding (a little), in two national funding bodies and the EC - but no, not ''ALL'' the comments above are from my own proposals! I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page (and, sigh, also the dude that needs to individually give everyone access rights... I wish there were an easier way) - in the long run, I hope that it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Guidelines&diff=54Guidelines2021-06-30T11:52:47Z<p>Michawe@uio.no: </p>
<hr />
<div>'''These guidelines are derived as ''subjective'' interpretations of the reviewer statements on [[Main Page|this page]].'''<br />
==Excellence (the research idea itself, methods, ..)==<br />
<br />
=== '''Clarity, being ground-breaking, ..:''' ===<br />
*Project outputs: be specific about analytic evaluations.<br />
*Describe concrete project results (e.g., simulation model, prototype etc.) in sufficient detail. Benefits shown as a vision are not concrete enough.<br />
*Be clear about goals. E.g., when describing a % improvement over the state of the art, clarify which metrics will be used to quantify the improvement.<br />
*Be ambitious. E.g., 20% better than the SoA is not enough. Explain the ground-breaking nature of the research and its generalisability.<br />
*Avoid looking incremental - careful with showing initial results.<br />
*Be clear about the research plan: how will the research questions be tackled?<br />
*Avoid looking too narrow: there should be a reasonably large research community interested in your work.<br />
*Avoid looking too broad: research objectives should not seem like they could each be a project in its own right.<br />
*Avoid redundancy in the text - e.g., don't embed the research questions in the objectives.<br />
*Think twice about which field to address (try to aim for new fields, if possible).<br />
<br />
=== '''Methods:''' ===<br />
*If the focus is on experiments, 1) provide sufficient details on experimental evaluation, and 2) consider adding complementary analytic methods to also attain more theory-oriented scientific achievements.<br />
*Make sure that quantitative research methods are described for all the research activities related to the topics mentioned.<br />
<br />
=== '''Related work:''' ===<br />
*Provide a comparison with alternative approaches; clarify how the project will differentiate from them. Highlight originality.<br />
<br />
=== '''Other matters:''' ===<br />
* Explicitly address ethics, safety and gender issues.<br />
<br />
==Implementation (management)==<br />
<br />
=== '''PhD student supervision:''' ===<br />
*State who will advise the Ph.D. students.<br />
*Give enough details regarding the supervision of PhD students, especially when the project manager's own research has previously had a different focus.<br />
<br />
=== '''International cooperation:''' ===<br />
*When cooperating with international partners, explain how the international consortium will be managed.<br />
*When cooperating with industry, engage with other stakeholders from the same industry beyond only one major company.<br />
*Clarify how collaboration will work in the project (a plan of exchange visits, and stating the intention to submit joint publications is not enough).<br />
*If external collaborators primarily have the expertise on a topic, this is a risk for the project. Avoid this, or state it as a risk and provide a mitigation plan.<br />
<br />
=== '''Other management issues:''' ===<br />
*Consider including (a) work package(s) for integration, result evaluation, project management and dissemination.<br />
*Describe the management structure in sufficient detail.<br />
*Describe task dependencies, timing and deliverables.<br />
*Clarify the division of research tasks and responsibilities between team members.<br />
*Describe WP results not only in terms of of publications but in terms of concrete achievements and developments.<br />
*Avoid looking too generic with the risk management plan (still, include general risks like failing to recruit the right staff, and delay in hiring people); mitigation actions must be detailed and clear.<br />
*A "waterfall arrangement" work package structure may look too simple. Clarify the timeline and dependencies.<br />
*Search the proposal for "he", "his" and "him" before submitting to say she/he instead, wherever appropriate.<br />
<br />
==Impact (dissemination, exploitation, ..)==<br />
*If dissemination activities only describe publications in journals and international conferences, consider adding something more. Maybe what the EC calls "communication", i.e. talking to the public, via other media?<br />
*In addition to the technological impact, describe the impact on / importance for society at large and/or industry.<br />
*To convince reviewers that there will be a significant impact on the research community, avoid too many self-cites in the reference list.<br />
*Provide KPIs for dissemination activities. Plan for enough papers!<br />
*When working across multiple fields, ensure that the publication plans evenly spreads across all of them.<br />
*Address the potential exploitation of the research results.<br />
*In case of doing standardisation, provide information about contacts to standardisation channels, and explain in detail how an impact in standardisation will be achieved.<br />
*Explicitly address ethics, safety and gender issues.<br />
*Explicitly address societal impact.<br />
*Avoid making the impact description too generic looking. It must be specific to the project.<br />
*Avoid road-blocks like requiring a systemic change across different parts of the value chain before the proposed system can become effective.</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=53Main Page2021-06-30T11:49:52Z<p>Michawe@uio.no: /* Impact (dissemination, exploitation, ..) */</p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.<br />
<br />
'''A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].'''<br />
<br />
<br><br />
<br />
'''ABOUT ACCESS RIGHTS:'''<br />
<br />
'''READ:''' this Wiki is ''open for read access to the whole world'' - limiting read and/or edit access to IFI or UiO was not an option (it once worked in the past, but the code that made this possible is now broken, and fixing it is not a current priority, I was told). I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, well, I'll probably have to delete this Wiki, or look for other options (I'm out of ideas, though).<br />
<br />
'''WRITE:''' click around, visit [[Guidelines|'''this''']] page, try to log in, and you'll see that you cannot edit. Only then will I, as wiki administrator, see your user name in a list, and can give you edit rights. So, after failing to edit, send [http://www.welzl.at me] an email (usernames match UiO email addresses) and I'll give you the rights. No, [https://www.uio.no/tjenester/it/utdanning/wiki/hjelp/admin/brukerrettigheter.html there is no more convenient way] to do this.<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
**[…] is a popular topic. Originality should be highlighted.<br />
**The actual benefits are presented as a vision, yet they are not substantiated.<br />
**The actual need or benefits are not well identified.<br />
**… some applications are only vaguely described, like […] use case …<br />
**On the downside, this is an area of the literature that is already very crowded. This creates the risk that the project may lead to incremental advancements of the state of the art.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
*'''Other matters:'''<br />
**Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International or industry cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
**Engagement with stakeholders from the […] industry other than [major company] could have been stronger.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**… task dependencies, timing and deliverables are not detailed.<br />
**The management structure is not sufficiently described.<br />
**Project management and governance mechanisms have not been sufficiently detailed.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**The risk management is adequately described although lacking more general risks like failing to recruit the right staff.<br />
**One risk that is not discussed is the delay in hiring the full-time researcher: in this case the PM will be the only senior member at UiO and there is the risk that the PI may not have sufficient time to allocate to the project.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
**Regarding this sentence: "IFI has well-established procedures to ensure that a Ph.D. student can pass _his_ work successfully." Does the Ph.D. program at IFI support only male students?<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*The impact of the proposed research is described in the form of specific challenges but fails to map those challenges to specific impact for industry or the society at large.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
*The proposed system will require systemic change across different parts of the value chain before they can become effective.<br />
*The planned publication output is not very ambitious: only one conference paper per year, albeit in very competitive venues.<br />
*Societal impact is not analyzed in detail.<br />
*In terms of publications the work can have great potential in the conferences and journals. However, those are rather focusing on [main field], but less on [two other fields that this project interacts with].<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing (a lot) and succeeding (a little), in two national funding bodies and the EC - but no, not ''ALL'' the comments above are from my own proposals! I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page (and, sigh, also the dude that needs to individually give everyone access rights... I wish there were an easier way) - in the long run, I hope that it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Guidelines&diff=52Guidelines2021-06-30T11:46:25Z<p>Michawe@uio.no: </p>
<hr />
<div>'''These guidelines are derived as ''subjective'' interpretations of the reviewer statements on [[Main Page|this page]].'''<br />
==Excellence (the research idea itself, methods, ..)==<br />
<br />
=== '''Clarity, being ground-breaking, ..:''' ===<br />
*Project outputs: be specific about analytic evaluations.<br />
*Describe concrete project results (e.g., simulation model, prototype etc.) in sufficient detail. Benefits shown as a vision are not concrete enough.<br />
*Be clear about goals. E.g., when describing a % improvement over the state of the art, clarify which metrics will be used to quantify the improvement.<br />
*Be ambitious. E.g., 20% better than the SoA is not enough. Explain the ground-breaking nature of the research and its generalisability.<br />
*Avoid looking incremental - careful with showing initial results.<br />
*Be clear about the research plan: how will the research questions be tackled?<br />
*Avoid looking too narrow: there should be a reasonably large research community interested in your work.<br />
*Avoid looking too broad: research objectives should not seem like they could each be a project in its own right.<br />
*Avoid redundancy in the text - e.g., don't embed the research questions in the objectives.<br />
*Think twice about which field to address (try to aim for new fields, if possible).<br />
<br />
=== '''Methods:''' ===<br />
*If the focus is on experiments, 1) provide sufficient details on experimental evaluation, and 2) consider adding complementary analytic methods to also attain more theory-oriented scientific achievements.<br />
*Make sure that quantitative research methods are described for all the research activities related to the topics mentioned.<br />
<br />
=== '''Related work:''' ===<br />
*Provide a comparison with alternative approaches; clarify how the project will differentiate from them. Highlight originality.<br />
<br />
=== '''Other matters:''' ===<br />
* Explicitly address ethics, safety and gender issues.<br />
<br />
==Implementation (management)==<br />
<br />
=== '''PhD student supervision:''' ===<br />
*State who will advise the Ph.D. students.<br />
*Give enough details regarding the supervision of PhD students, especially when the project manager's own research has previously had a different focus.<br />
<br />
=== '''International cooperation:''' ===<br />
*When cooperating with international partners, explain how the international consortium will be managed.<br />
*When cooperating with industry, engage with other stakeholders from the same industry beyond only one major company.<br />
*Clarify how collaboration will work in the project (a plan of exchange visits, and stating the intention to submit joint publications is not enough).<br />
*If external collaborators primarily have the expertise on a topic, this is a risk for the project. Avoid this, or state it as a risk and provide a mitigation plan.<br />
<br />
=== '''Other management issues:''' ===<br />
*Consider including (a) work package(s) for integration, result evaluation, project management and dissemination.<br />
*Describe the management structure in sufficient detail.<br />
*Describe task dependencies, timing and deliverables.<br />
*Clarify the division of research tasks and responsibilities between team members.<br />
*Avoid looking too generic with the risk management plan (still, include general risks like failing to recruit the right staff, and delay in hiring people); mitigation actions must be detailed and clear.<br />
*A "waterfall arrangement" work package structure may look too simple. Clarify the timeline and dependencies.<br />
*Search the proposal for "he", "his" and "him" before submitting to say she/he instead, wherever appropriate.<br />
<br />
==Impact (dissemination, exploitation, ..)==<br />
*If dissemination activities only describe publications in journals and international conferences, consider adding something more. Maybe what the EC calls "communication", i.e. talking to the public, via other media?<br />
*In addition to the technological impact, describe the impact on / importance for society at large and/or industry.<br />
*To convince reviewers that there will be a significant impact on the research community, avoid too many self-cites in the reference list.<br />
*Provide KPIs for dissemination activities. Plan for enough papers!<br />
*Address the potential exploitation of the research results.<br />
*In case of doing standardisation, provide information about contacts to standardisation channels, and explain in detail how an impact in standardisation will be achieved.<br />
*Explicitly address ethics, safety and gender issues.<br />
*Avoid making the impact description too generic looking. It must be specific to the project.<br />
*Avoid road-blocks like requiring a systemic change across different parts of the value chain before the proposed system can become effective.</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=51Main Page2021-06-30T11:44:01Z<p>Michawe@uio.no: /* Excellence (the research idea itself, methods, ..) */</p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.<br />
<br />
'''A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].'''<br />
<br />
<br><br />
<br />
'''ABOUT ACCESS RIGHTS:'''<br />
<br />
'''READ:''' this Wiki is ''open for read access to the whole world'' - limiting read and/or edit access to IFI or UiO was not an option (it once worked in the past, but the code that made this possible is now broken, and fixing it is not a current priority, I was told). I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, well, I'll probably have to delete this Wiki, or look for other options (I'm out of ideas, though).<br />
<br />
'''WRITE:''' click around, visit [[Guidelines|'''this''']] page, try to log in, and you'll see that you cannot edit. Only then will I, as wiki administrator, see your user name in a list, and can give you edit rights. So, after failing to edit, send [http://www.welzl.at me] an email (usernames match UiO email addresses) and I'll give you the rights. No, [https://www.uio.no/tjenester/it/utdanning/wiki/hjelp/admin/brukerrettigheter.html there is no more convenient way] to do this.<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
**[…] is a popular topic. Originality should be highlighted.<br />
**The actual benefits are presented as a vision, yet they are not substantiated.<br />
**The actual need or benefits are not well identified.<br />
**… some applications are only vaguely described, like […] use case …<br />
**On the downside, this is an area of the literature that is already very crowded. This creates the risk that the project may lead to incremental advancements of the state of the art.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
*'''Other matters:'''<br />
**Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International or industry cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
**Engagement with stakeholders from the […] industry other than [major company] could have been stronger.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**… task dependencies, timing and deliverables are not detailed.<br />
**The management structure is not sufficiently described.<br />
**Project management and governance mechanisms have not been sufficiently detailed.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**The risk management is adequately described although lacking more general risks like failing to recruit the right staff.<br />
**One risk that is not discussed is the delay in hiring the full-time researcher: in this case the PM will be the only senior member at UiO and there is the risk that the PI may not have sufficient time to allocate to the project.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
**Regarding this sentence: "IFI has well-established procedures to ensure that a Ph.D. student can pass _his_ work successfully." Does the Ph.D. program at IFI support only male students?<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*The impact of the proposed research is described in the form of specific challenges but fails to map those challenges to specific impact for industry or the society at large.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
*The proposed system will require systemic change across different parts of the value chain before they can become effective.<br />
*The planned publication output is not very ambitious: only one conference paper per year, albeit in very competitive venues.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing (a lot) and succeeding (a little), in two national funding bodies and the EC - but no, not ''ALL'' the comments above are from my own proposals! I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page (and, sigh, also the dude that needs to individually give everyone access rights... I wish there were an easier way) - in the long run, I hope that it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Guidelines&diff=50Guidelines2021-01-26T10:04:05Z<p>Michawe@uio.no: </p>
<hr />
<div>'''These guidelines are derived as ''subjective'' interpretations of the reviewer statements on [[Main Page|this page]].'''<br />
==Excellence (the research idea itself, methods, ..)==<br />
<br />
=== '''Clarity, being ground-breaking, ..:''' ===<br />
*Project outputs: be specific about analytic evaluations.<br />
*Describe concrete project results (e.g., simulation model, prototype etc.) in sufficient detail. Benefits shown as a vision are not concrete enough.<br />
*Be clear about goals. E.g., when describing a % improvement over the state of the art, clarify which metrics will be used to quantify the improvement.<br />
*Be ambitious. E.g., 20% better than the SoA is not enough. Explain the ground-breaking nature of the research and its generalisability.<br />
*Avoid looking incremental - careful with showing initial results.<br />
*Be clear about the research plan: how will the research questions be tackled?<br />
*Avoid looking too narrow: there should be a reasonably large research community interested in your work.<br />
*Avoid looking too broad: research objectives should not seem like they could each be a project in its own right.<br />
*Avoid redundancy in the text - e.g., don't embed the research questions in the objectives.<br />
<br />
=== '''Methods:''' ===<br />
*If the focus is on experiments, 1) provide sufficient details on experimental evaluation, and 2) consider adding complementary analytic methods to also attain more theory-oriented scientific achievements.<br />
*Make sure that quantitative research methods are described for all the research activities related to the topics mentioned.<br />
<br />
=== '''Related work:''' ===<br />
*Provide a comparison with alternative approaches; clarify how the project will differentiate from them. Highlight originality.<br />
<br />
=== '''Other matters:''' ===<br />
* Explicitly address ethics, safety and gender issues.<br />
<br />
==Implementation (management)==<br />
<br />
=== '''PhD student supervision:''' ===<br />
*State who will advise the Ph.D. students.<br />
*Give enough details regarding the supervision of PhD students, especially when the project manager's own research has previously had a different focus.<br />
<br />
=== '''International cooperation:''' ===<br />
*When cooperating with international partners, explain how the international consortium will be managed.<br />
*When cooperating with industry, engage with other stakeholders from the same industry beyond only one major company.<br />
*Clarify how collaboration will work in the project (a plan of exchange visits, and stating the intention to submit joint publications is not enough).<br />
*If external collaborators primarily have the expertise on a topic, this is a risk for the project. Avoid this, or state it as a risk and provide a mitigation plan.<br />
<br />
=== '''Other management issues:''' ===<br />
*Consider including (a) work package(s) for integration, result evaluation, project management and dissemination.<br />
*Describe the management structure in sufficient detail.<br />
*Describe task dependencies, timing and deliverables.<br />
*Clarify the division of research tasks and responsibilities between team members.<br />
*Avoid looking too generic with the risk management plan (still, include general risks like failing to recruit the right staff); mitigation actions must be detailed and clear.<br />
*A "waterfall arrangement" work package structure may look too simple. Clarify the timeline and dependencies.<br />
<br />
==Impact (dissemination, exploitation, ..)==<br />
*If dissemination activities only describe publications in journals and international conferences, consider adding something more. Maybe what the EC calls "communication", i.e. talking to the public, via other media?<br />
*In addition to the technological impact, describe the impact on / importance for society at large and/or industry.<br />
*To convince reviewers that there will be a significant impact on the research community, avoid too many self-cites in the reference list.<br />
*Provide KPIs for dissemination activities.<br />
*Address the potential exploitation of the research results.<br />
*In case of doing standardisation, provide information about contacts to standardisation channels, and explain in detail how an impact in standardisation will be achieved.<br />
*Explicitly address ethics, safety and gender issues.<br />
*Avoid making the impact description too generic looking. It must be specific to the project.<br />
*Avoid road-blocks like requiring a systemic change across different parts of the value chain before the proposed system can become effective.</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=49Main Page2021-01-26T10:02:12Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.<br />
<br />
'''A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].'''<br />
<br />
<br><br />
<br />
'''ABOUT ACCESS RIGHTS:'''<br />
<br />
'''READ:''' this Wiki is ''open for read access to the whole world'' - limiting read and/or edit access to IFI or UiO was not an option (it once worked in the past, but the code that made this possible is now broken, and fixing it is not a current priority, I was told). I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, well, I'll probably have to delete this Wiki, or look for other options (I'm out of ideas, though).<br />
<br />
'''WRITE:''' click around, visit [[Guidelines|'''this''']] page, try to log in, and you'll see that you cannot edit. Only then will I, as wiki administrator, see your user name in a list, and can give you edit rights. So, after failing to edit, send [http://www.welzl.at me] an email (usernames match UiO email addresses) and I'll give you the rights. No, [https://www.uio.no/tjenester/it/utdanning/wiki/hjelp/admin/brukerrettigheter.html there is no more convenient way] to do this.<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
**[…] is a popular topic. Originality should be highlighted.<br />
**The actual benefits are presented as a vision, yet they are not substantiated.<br />
**The actual need or benefits are not well identified.<br />
**… some applications are only vaguely described, like […] use case …<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
*'''Other matters:'''<br />
**Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International or industry cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
**Engagement with stakeholders from the […] industry other than [major company] could have been stronger.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**… task dependencies, timing and deliverables are not detailed.<br />
**The management structure is not sufficiently described.<br />
**Project management and governance mechanisms have not been sufficiently detailed.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**The risk management is adequately described although lacking more general risks like failing to recruit the right staff.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*The impact of the proposed research is described in the form of specific challenges but fails to map those challenges to specific impact for industry or the society at large.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
*The proposed system will require systemic change across different parts of the value chain before they can become effective.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing (a lot) and succeeding (a little), in two national funding bodies and the EC - but no, not ''ALL'' the comments above are from my own proposals! I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page (and, sigh, also the dude that needs to individually give everyone access rights... I wish there were an easier way) - in the long run, I hope that it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Guidelines&diff=48Guidelines2021-01-26T10:00:35Z<p>Michawe@uio.no: </p>
<hr />
<div>'''These guidelines are derived as ''subjective'' interpretations of the reviewer statements on [[Main Page|this page]].'''<br />
==Excellence (the research idea itself, methods, ..)==<br />
<br />
=== '''Clarity, being ground-breaking, ..:''' ===<br />
*Project outputs: be specific about analytic evaluations.<br />
*Describe concrete project results (e.g., simulation model, prototype etc.) in sufficient detail. Benefits shown as a vision are not concrete enough.<br />
*Be clear about goals. E.g., when describing a % improvement over the state of the art, clarify which metrics will be used to quantify the improvement.<br />
*Be ambitious. E.g., 20% better than the SoA is not enough. Explain the ground-breaking nature of the research and its generalisability.<br />
*Avoid looking incremental - careful with showing initial results.<br />
*Be clear about the research plan: how will the research questions be tackled?<br />
*Avoid looking too narrow: there should be a reasonably large research community interested in your work.<br />
*Avoid looking too broad: research objectives should not seem like they could each be a project in its own right.<br />
*Avoid redundancy in the text - e.g., don't embed the research questions in the objectives.<br />
<br />
=== '''Methods:''' ===<br />
*If the focus is on experiments, 1) provide sufficient details on experimental evaluation, and 2) consider adding complementary analytic methods to also attain more theory-oriented scientific achievements.<br />
*Make sure that quantitative research methods are described for all the research activities related to the topics mentioned.<br />
<br />
=== '''Related work:''' ===<br />
*Make sure to provide a comparison with alternative approaches; clarify how the project will differentiate from them. Highlight originality.<br />
<br />
=== '''Other matters:''' ===<br />
* Explicitly address ethics, safety and gender issues.<br />
<br />
==Implementation (management)==<br />
<br />
=== '''PhD student supervision:''' ===<br />
*State who will advise the Ph.D. students.<br />
*Give enough details regarding the supervision of PhD students, especially when the project manager's own research has previously had a different focus.<br />
<br />
=== '''International cooperation:''' ===<br />
*When cooperating with international partners, explain how the international consortium will be managed.<br />
*When cooperating with industry, engage with other stakeholders from the same industry beyond only one major company.<br />
*Clarify how collaboration will work in the project (a plan of exchange visits, and stating the intention to submit joint publications is not enough).<br />
*If external collaborators primarily have the expertise on a topic, this is a risk for the project. Avoid this, or state it as a risk and provide a mitigation plan.<br />
<br />
=== '''Other management issues:''' ===<br />
*Consider including (a) work package(s) for integration, result evaluation, project management and dissemination.<br />
*Describe the management structure in sufficient detail.<br />
*Describe task dependencies, timing and deliverables.<br />
*Clarify the division of research tasks and responsibilities between team members.<br />
*Avoid looking too generic with the risk management plan (still, include general risks like failing to recruit the right staff); mitigation actions must be detailed and clear.<br />
*A "waterfall arrangement" work package structure may look too simple. Clarify the timeline and dependencies.<br />
<br />
==Impact (dissemination, exploitation, ..)==<br />
*If dissemination activities only describe publications in journals and international conferences, consider adding something more. Maybe what the EC calls "communication", i.e. talking to the public, via other media?<br />
*In addition to the technological impact, describe the impact on / importance for society at large and/or industry.<br />
*To convince reviewers that there will be a significant impact on the research community, avoid too many self-cites in the reference list.<br />
*Provide KPIs for dissemination activities.<br />
*Address the potential exploitation of the research results.<br />
*In case of doing standardisation, provide information about contacts to standardisation channels, and explain in detail how an impact in standardisation will be achieved.<br />
*Explicitly address ethics, safety and gender issues.<br />
*Avoid making the impact description too generic looking. It must be specific to the project.<br />
*Avoid road-blocks like requiring a systemic change across different parts of the value chain before the proposed system can become effective.</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Guidelines&diff=47Guidelines2021-01-26T09:58:59Z<p>Michawe@uio.no: </p>
<hr />
<div>'''These guidelines are derived as ''subjective'' interpretations of the reviewer statements on [[Main Page|this page]].'''<br />
==Excellence (the research idea itself, methods, ..)==<br />
<br />
=== '''Clarity, being ground-breaking, ..:''' ===<br />
**Project outputs: be specific about analytic evaluations.<br />
**Describe concrete project results (e.g., simulation model, prototype etc.) in sufficient detail. Benefits shown as a vision are not concrete enough.<br />
**Be clear about goals. E.g., when describing a % improvement over the state of the art, clarify which metrics will be used to quantify the improvement.<br />
**Be ambitious. E.g., 20% better than the SoA is not enough. Explain the ground-breaking nature of the research and its generalisability.<br />
**Avoid looking incremental - careful with showing initial results.<br />
**Be clear about the research plan: how will the research questions be tackled?<br />
**Avoid looking too narrow: there should be a reasonably large research community interested in your work.<br />
**Avoid looking too broad: research objectives should not seem like they could each be a project in its own right.<br />
**Avoid redundancy in the text - e.g., don't embed the research questions in the objectives.<br />
<br />
=== '''Methods:''' ===<br />
**If the focus is on experiments, 1) provide sufficient details on experimental evaluation, and 2) consider adding complementary analytic methods to also attain more theory-oriented scientific achievements.<br />
**Make sure that quantitative research methods are described for all the research activities related to the topics mentioned.<br />
<br />
=== '''Related work:''' ===<br />
**Make sure to provide a comparison with alternative approaches; clarify how the project will differentiate from them. Highlight originality.<br />
*<br />
<br />
=== '''Other matters:''' ===<br />
* Explicitly address ethics, safety and gender issues.<br />
<br />
*<br />
==Implementation (management)==<br />
<br />
=== '''PhD student supervision:''' ===<br />
**State who will advise the Ph.D. students.<br />
**Give enough details regarding the supervision of PhD students, especially when the project manager's own research has previously had a different focus.<br />
<br />
=== '''International cooperation:''' ===<br />
**When cooperating with international partners, explain how the international consortium will be managed.<br />
**When cooperating with industry, engage with other stakeholders from the same industry beyond only one major company.<br />
**Clarify how collaboration will work in the project (a plan of exchange visits, and stating the intention to submit joint publications is not enough).<br />
**If external collaborators primarily have the expertise on a topic, this is a risk for the project. Avoid this, or state it as a risk and provide a mitigation plan.<br />
<br />
=== '''Other management issues:''' ===<br />
**Consider including (a) work package(s) for integration, result evaluation, project management and dissemination.<br />
**Describe the management structure in sufficient detail.<br />
**Describe task dependencies, timing and deliverables.<br />
**Clarify the division of research tasks and responsibilities between team members.<br />
**Avoid looking too generic with the risk management plan (still, include general risks like failing to recruit the right staff); mitigation actions must be detailed and clear.<br />
**A "waterfall arrangement" work package structure may look too simple. Clarify the timeline and dependencies.<br />
==Impact (dissemination, exploitation, ..)==<br />
*If dissemination activities only describe publications in journals and international conferences, consider adding something more. Maybe what the EC calls "communication", i.e. talking to the public, via other media?<br />
*In addition to the technological impact, describe the impact on / importance for society at large and/or industry.<br />
*To convince reviewers that there will be a significant impact on the research community, avoid too many self-cites in the reference list.<br />
*Provide KPIs for dissemination activities.<br />
*Address the potential exploitation of the research results.<br />
*In case of doing standardisation, provide information about contacts to standardisation channels, and explain in detail how an impact in standardisation will be achieved.<br />
*Explicitly address ethics, safety and gender issues.<br />
*Avoid making the impact description too generic looking. It must be specific to the project.<br />
*Avoid road-blocks like requiring a systemic change across different parts of the value chain before the proposed system can become effective.</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=46Main Page2021-01-20T08:16:10Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.<br />
<br />
'''A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].'''<br />
<br />
<br><br />
<br />
'''ABOUT ACCESS RIGHTS:'''<br />
<br />
'''READ:''' this Wiki is ''open for read access to the whole world'' - limiting read and/or edit access to IFI or UiO was not an option (it once worked in the past, but the code that made this possible is now broken, and fixing it is not a current priority, I was told). I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, well, I'll probably have to delete this Wiki, or look for other options (I'm out of ideas, though).<br />
<br />
'''WRITE:''' click around, visit [[Guidelines|'''this''']] page, try to log in, and you'll see that you cannot edit. Only then will I, as wiki administrator, see your user name in a list, and can give you edit rights. So, after failing to edit, send [http://www.welzl.at me] an email (usernames match UiO email addresses) and I'll give you the rights. No, [https://www.uio.no/tjenester/it/utdanning/wiki/hjelp/admin/brukerrettigheter.html there is no more convenient way] to do this.<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
**[…] is a popular topic. Originality should be highlighted.<br />
**The actual benefits are presented as a vision, yet they are not substantiated.<br />
**The actual need or benefits are not well identified.<br />
**… some applications are only vaguely described, like […] use case …<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International or industry cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
**Engagement with stakeholders from the […] industry other than [major company] could have been stronger.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**… task dependencies, timing and deliverables are not detailed.<br />
**The management structure is not sufficiently described.<br />
**Project management and governance mechanisms have not been sufficiently detailed.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**The risk management is adequately described although lacking more general risks like failing to recruit the right staff.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*The impact of the proposed research is described in the form of specific challenges but fails to map those challenges to specific impact for industry or the society at large.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
*The proposed system will require systemic change across different parts of the value chain before they can become effective.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing (a lot) and succeeding (a little), in two national funding bodies and the EC - but no, not ''ALL'' the comments above are from my own proposals! I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page (and, sigh, also the dude that needs to individually give everyone access rights... I wish there were an easier way) - in the long run, I hope that it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Guidelines&diff=45Guidelines2021-01-14T09:49:27Z<p>Michawe@uio.no: /* Clarity, being ground-breaking, ..: */</p>
<hr />
<div>'''These guidelines are derived as ''subjective'' interpretations of the reviewer statements on [[Main Page|this page]].'''<br />
==Excellence (the research idea itself, methods, ..)==<br />
<br />
=== '''Clarity, being ground-breaking, ..:''' ===<br />
**Project outputs: be specific about analytic evaluations.<br />
**Describe concrete project results (e.g., simulation model, prototype etc.) in sufficient detail. Benefits shown as a vision are not concrete enough.<br />
**Be clear about goals. E.g., when describing a % improvement over the state of the art, clarify which metrics will be used to quantify the improvement.<br />
**Be ambitious. E.g., 20% better than the SoA is not enough. Explain the ground-breaking nature of the research and its generalisability.<br />
**Avoid looking incremental - careful with showing initial results.<br />
**Be clear about the research plan: how will the research questions be tackled?<br />
**Avoid looking too narrow: there should be a reasonably large research community interested in your work.<br />
**Avoid looking too broad: research objectives should not seem like they could each be a project in its own right.<br />
**Avoid redundancy in the text - e.g., don't embed the research questions in the objectives.<br />
<br />
=== '''Methods:''' ===<br />
**If the focus is on experiments, 1) provide sufficient details on experimental evaluation, and 2) consider adding complementary analytic methods to also attain more theory-oriented scientific achievements.<br />
**Make sure that quantitative research methods are described for all the research activities related to the topics mentioned.<br />
<br />
=== '''Related work:''' ===<br />
**Make sure to provide a comparison with alternative approaches; clarify how the project will differentiate from them. Highlight originality.<br />
==Implementation (management)==<br />
<br />
=== '''PhD student supervision:''' ===<br />
**State who will advise the Ph.D. students.<br />
**Give enough details regarding the supervision of PhD students, especially when the project manager's own research has previously had a different focus.<br />
<br />
=== '''International cooperation:''' ===<br />
**When cooperating with international partners, explain how the international consortium will be managed.<br />
**When cooperating with industry, engage with other stakeholders from the same industry beyond only one major company.<br />
**Clarify how collaboration will work in the project (a plan of exchange visits, and stating the intention to submit joint publications is not enough).<br />
**If external collaborators primarily have the expertise on a topic, this is a risk for the project. Avoid this, or state it as a risk and provide a mitigation plan.<br />
<br />
=== '''Other management issues:''' ===<br />
**Consider including (a) work package(s) for integration, result evaluation, project management and dissemination.<br />
**Describe the management structure in sufficient detail.<br />
**Describe task dependencies, timing and deliverables.<br />
**Clarify the division of research tasks and responsibilities between team members.<br />
**Avoid looking too generic with the risk management plan (still, include general risks like failing to recruit the right staff); mitigation actions must be detailed and clear.<br />
**A "waterfall arrangement" work package structure may look too simple. Clarify the timeline and dependencies.<br />
==Impact (dissemination, exploitation, ..)==<br />
*If dissemination activities only describe publications in journals and international conferences, consider adding something more. Maybe what the EC calls "communication", i.e. talking to the public, via other media?<br />
*In addition to the technological impact, describe the impact on / importance for society at large and/or industry.<br />
*To convince reviewers that there will be a significant impact on the research community, avoid too many self-cites in the reference list.<br />
*Provide KPIs for dissemination activities.<br />
*Address the potential exploitation of the research results.<br />
*In case of doing standardisation, provide information about contacts to standardisation channels, and explain in detail how an impact in standardisation will be achieved.<br />
*Explicitly address ethics, safety and gender issues.<br />
*Avoid making the impact description too generic looking. It must be specific to the project.<br />
*Avoid road-blocks like requiring a systemic change across different parts of the value chain before the proposed system can become effective.</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=44Main Page2021-01-14T09:38:04Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.<br />
<br />
'''A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].'''<br />
<br />
<br><br />
<br />
'''ABOUT ACCESS RIGHTS:'''<br />
<br />
'''READ:''' this Wiki is ''open for read access to the whole world'' - limiting read and/or edit access to IFI or UiO was not an option (it once worked in the past, but the code that made this possible is now broken, and fixing it is not a current priority, I was told). I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, well, I'll probably have to delete this Wiki, or look for other options (I'm out of ideas, though).<br />
<br />
'''WRITE:''' click around, visit [[Guidelines|'''this''']] page, try to log in, and you'll see that you cannot edit. Only then will I, as wiki administrator, see your user name in a list, and can give you edit rights. So, after failing to edit, send [http://www.welzl.at me] an email (usernames match UiO email addresses) and I'll give you the rights. No, [https://www.uio.no/tjenester/it/utdanning/wiki/hjelp/admin/brukerrettigheter.html there is no more convenient way] to do this.<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
**[…] is a popular topic. Originality should be highlighted.<br />
**The actual benefits are presented as a vision, yet they are not substantiated.<br />
**The actual need or benefits are not well identified.<br />
**… some applications are only vaguely described, like […] use case …<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International or industry cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
**Engagement with stakeholders from the […] industry other than [major company] could have been stronger.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**… task dependencies, timing and deliverables are not detailed.<br />
**The management structure is not sufficiently described.<br />
**Project management and governance mechanisms have not been sufficiently detailed.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**The risk management is adequately described although lacking more general risks like failing to recruit the right staff.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*The impact of the proposed research is described in the form of specific challenges but fails to map those challenges to specific impact for industry or the society at large.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
*The proposed system will require systemic change across different parts of the value chain before they can become effective.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing (a lot) and succeeding (a little), in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page (and, sigh, also the dude that needs to individually give everyone access rights... I wish there were an easier way) - in the long run, I hope that it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Guidelines&diff=43Guidelines2021-01-14T08:30:09Z<p>Michawe@uio.no: </p>
<hr />
<div>'''These guidelines are derived as ''subjective'' interpretations of the reviewer statements on [[Main Page|this page]].'''<br />
==Excellence (the research idea itself, methods, ..)==<br />
<br />
=== '''Clarity, being ground-breaking, ..:''' ===<br />
**Project outputs: be specific about analytic evaluations.<br />
**Describe concrete project results (e.g., simulation model, prototype etc.) in sufficient detail.<br />
**Be clear about goals. E.g., when describing a % improvement over the state of the art, clarify which metrics will be used to quantify the improvement.<br />
**Be ambitious. E.g., 20% better than the SoA is not enough. Explain the ground-breaking nature of the research and its generalisability.<br />
**Avoid looking incremental - careful with showing initial results.<br />
**Be clear about the research plan: how will the research questions be tackled?<br />
**Avoid looking too narrow: there should be a reasonably large research community interested in your work.<br />
**Avoid looking too broad: research objectives should not seem like they could each be a project in its own right.<br />
**Avoid redundancy in the text - e.g., don't embed the research questions in the objectives.<br />
<br />
=== '''Methods:''' ===<br />
**If the focus is on experiments, 1) provide sufficient details on experimental evaluation, and 2) consider adding complementary analytic methods to also attain more theory-oriented scientific achievements.<br />
**Make sure that quantitative research methods are described for all the research activities related to the topics mentioned.<br />
<br />
=== '''Related work:''' ===<br />
**Make sure to provide a comparison with alternative approaches; clarify how the project will differentiate from them.<br />
==Implementation (management)==<br />
<br />
=== '''PhD student supervision:''' ===<br />
**State who will advise the Ph.D. students.<br />
**Give enough details regarding the supervision of PhD students, especially when the project manager's own research has previously had a different focus.<br />
<br />
=== '''International cooperation:''' ===<br />
**When cooperating with international partners, explain how the international consortium will be managed.<br />
**Clarify how collaboration will work in the project (a plan of exchange visits, and stating the intention to submit joint publications is not enough).<br />
**If external collaborators primarily have the expertise on a topic, this is a risk for the project. Avoid this, or state it as a risk and provide a mitigation plan.<br />
<br />
=== '''Other management issues:''' ===<br />
**Consider including (a) work package(s) for integration, result evaluation, project management and dissemination.<br />
**Describe the management structure in sufficient detail.<br />
**Clarify the division of research tasks and responsibilities between team members.<br />
**Avoid looking too generic with the risk management plan; mitigation actions must be detailed and clear.<br />
**A "waterfall arrangement" work package structure may look too simple. Clarify the timeline and dependencies.<br />
==Impact (dissemination, exploitation, ..)==<br />
*If dissemination activities only describe publications in journals and international conferences, consider adding something more. Maybe what the EC calls "communication", i.e. talking to the public, via other media?<br />
*In addition to the technological impact, describe the impact on / importance for society.<br />
*To convince reviewers that there will be a significant impact on the research community, avoid too many self-cites in the reference list.<br />
*Provide KPIs for dissemination activities.<br />
*Address the potential exploitation of the research results.<br />
*In case of doing standardisation, provide information about contacts to standardisation channels, and explain in detail how an impact in standardisation will be achieved.<br />
*Explicitly address ethics, safety and gender issues.<br />
*Avoid making the impact description too generic looking. It must be specific to the project.</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=42Main Page2021-01-14T08:25:06Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.<br />
<br />
'''A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].'''<br />
<br />
<br><br />
<br />
'''ABOUT ACCESS RIGHTS:'''<br />
<br />
'''READ:''' this Wiki is ''open for read access to the whole world'' - limiting read and/or edit access to IFI or UiO was not an option (it once worked in the past, but the code that made this possible is now broken, and fixing it is not a current priority, I was told). I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, well, I'll probably have to delete this Wiki, or look for other options (I'm out of ideas, though).<br />
<br />
'''WRITE:''' click around, visit [[Guidelines|'''this''']] page, try to log in, and you'll see that you cannot edit. Only then will I, as wiki administrator, see your user name in a list, and can give you edit rights. So, after failing to edit, send [http://www.welzl.at me] an email (usernames match UiO email addresses) and I'll give you the rights. No, [https://www.uio.no/tjenester/it/utdanning/wiki/hjelp/admin/brukerrettigheter.html there is no more convenient way] to do this.<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**The management structure is not sufficiently described.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing (a lot) and succeeding (a little), in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page (and, sigh, also the dude that needs to individually give everyone access rights... I wish there were an easier way) - in the long run, I hope that it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=41Main Page2021-01-13T13:10:29Z<p>Michawe@uio.no: /* Final words */</p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.<br />
<br />
'''A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].'''<br />
<br />
<br><br />
<br />
'''ABOUT ACCESS RIGHTS:'''<br />
<br />
'''READ:''' this Wiki is ''open for read access to the whole world'' - limiting read and/or edit access to IFI or UiO was not an option (it once worked in the past, but the code that made this possible is now broken, and fixing it is not a current priority, I was told). I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, well, I'll probably have to delete this Wiki, or look for other options (I'm out of ideas, though).<br />
<br />
'''WRITE:''' click around, try to log in, and you'll see that you cannot edit. Only then will I, as wiki administrator, see your user name in a list, and can give you edit rights. So, after failing to edit, send [http://www.welzl.at me] an email (usernames match UiO email addresses) and I'll give you the rights. No, [https://www.uio.no/tjenester/it/utdanning/wiki/hjelp/admin/brukerrettigheter.html there is no more convenient way] to do this.<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**The management structure is not sufficiently described.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing (a lot) and succeeding (a little), in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page (and, sigh, also the dude that needs to individually give everyone access rights... I wish there were an easier way) - in the long run, I hope that it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Guidelines&diff=40Guidelines2021-01-13T13:08:18Z<p>Michawe@uio.no: /* Methods: */</p>
<hr />
<div>'''These guidelines are derived as ''subjective'' interpretations of the reviewer statements on [[Main Page|this page]].'''<br />
==Excellence (the research idea itself, methods, ..)==<br />
<br />
=== '''Clarity, being ground-breaking, ..:''' ===<br />
**Project outputs: be specific about analytic evaluations.<br />
**Describe concrete project results (e.g., simulation model, prototype etc.) in sufficient detail.<br />
**Be clear about goals. E.g., when describing a % improvement over the state of the art, clarify which metrics will be used to quantify the improvement.<br />
**Be ambitious. E.g., 20% better than the SoA is not enough. Explain the ground-breaking nature of the research and its generalisability.<br />
**Avoid looking incremental - careful with showing initial results.<br />
**Be clear about the research plan: how will the research questions be tackled?<br />
**Avoid looking too narrow: there should be a reasonably large research community interested in your work.<br />
**Avoid looking too broad: research objectives should not seem like they could each be a project in its own right.<br />
**Avoid redundancy in the text - e.g., don't embed the research questions in the objectives.<br />
<br />
=== '''Methods:''' ===<br />
**If the focus is on experiments, 1) provide sufficient details on experimental evaluation, and 2) consider adding complementary analytic methods to also attain more theory-oriented scientific achievements.<br />
**Make sure that quantitative research methods are described for all the research activities related to the topics mentioned.<br />
<br />
=== '''Related work:''' ===<br />
**Make sure to provide a comparison with alternative approaches; clarify how the project will differentiate from them.<br />
==Implementation (management)==<br />
<br />
=== '''PhD student supervision:''' ===<br />
**State who will advise the Ph.D. students.<br />
**Give enough details regarding the supervision of PhD students, especially when the project manager's own research has previously had a different focus.<br />
<br />
=== '''International cooperation:''' ===<br />
**When cooperating with international partners, explain how the international consortium will be managed.<br />
**Clarify how collaboration will work in the project (a plan of exchange visits, and stating the intention to submit joint publications is not enough).<br />
**If external collaborators primarily have the expertise on a topic, this is a risk for the project. Avoid this, or state it as a risk and provide a mitigation plan.<br />
<br />
=== '''Other management issues:''' ===<br />
**Consider including (a) work package(s) for integration, result evaluation, project management and dissemination.<br />
**Describe the management structure in sufficient detail.<br />
**Clarify the division of research tasks and responsibilities between team members.<br />
**Avoid looking too generic with the risk management plan; mitigation actions must be detailed and clear.<br />
**A "waterfall arrangement" work package structure may look too simple. Clarify the timeline and dependencies.<br />
==Impact (dissemination, exploitation, ..)==<br />
*If dissemination activities only describe publications in journals and international conferences, consider adding something more. Maybe what the EC calls "communication", i.e. talking to the public, via other media?<br />
*In addition to the technological impact, describe the impact on / importance for society.<br />
*To convince reviewers that there will be a significant impact on the research community, provide avoid too many self-cites in the reference list.<br />
*Provide KPIs for dissemination activities.<br />
*Address the potential exploitation of the research results.<br />
*In case of doing standardisation, provide information about contacts to standardisation channels, and explain in detail how an impact in standardisation will be achieved.<br />
*Explicitly address ethics, safety and gender issues.<br />
*Avoid making the impact description too generic looking. It must be specific to the project.</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=39Main Page2021-01-13T13:04:38Z<p>Michawe@uio.no: /* Welcome to the "learning from NFR proposal failures" Wiki ! */</p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.<br />
<br />
'''A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].'''<br />
<br />
<br><br />
<br />
'''ABOUT ACCESS RIGHTS:'''<br />
<br />
'''READ:''' this Wiki is ''open for read access to the whole world'' - limiting read and/or edit access to IFI or UiO was not an option (it once worked in the past, but the code that made this possible is now broken, and fixing it is not a current priority, I was told). I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, well, I'll probably have to delete this Wiki, or look for other options (I'm out of ideas, though).<br />
<br />
'''WRITE:''' click around, try to log in, and you'll see that you cannot edit. Only then will I, as wiki administrator, see your user name in a list, and can give you edit rights. So, after failing to edit, send [http://www.welzl.at me] an email (usernames match UiO email addresses) and I'll give you the rights. No, [https://www.uio.no/tjenester/it/utdanning/wiki/hjelp/admin/brukerrettigheter.html there is no more convenient way] to do this.<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**The management structure is not sufficiently described.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page (and, sigh, also the dude that needs to individually give everyone access rights... I wish there were an easier way) - in the long run, I hope that it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=38Main Page2021-01-13T13:03:55Z<p>Michawe@uio.no: /* Welcome to the "learning from NFR proposal failures" Wiki ! */</p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.<br />
<br />
'''A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].'''<br />
<br />
<br><br />
<br />
'''ABOUT ACCESS RIGHTS:'''<br />
<br />
'''READ:''' this Wiki is ''open for read access to the whole world'' - limiting read and/or edit access to IFI or UiO was not an option (it once worked in the past, but the code is now broken, I was told). I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, well, I'll probably have to delete this Wiki, or look for other options (I'm out of ideas, though).<br />
<br />
'''WRITE:''' click around, try to log in, and you'll see that you cannot edit. Only then will I, as wiki administrator, see your user name in a list, and can give you edit rights. So, after failing to edit, send [http://www.welzl.at me] an email (usernames match UiO email addresses) and I'll give you the rights. No, [https://www.uio.no/tjenester/it/utdanning/wiki/hjelp/admin/brukerrettigheter.html there is no more convenient way] to do this.<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**The management structure is not sufficiently described.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page (and, sigh, also the dude that needs to individually give everyone access rights... I wish there were an easier way) - in the long run, I hope that it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=37Main Page2021-01-13T13:03:19Z<p>Michawe@uio.no: /* Welcome to the "learning from NFR proposal failures" Wiki ! */</p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.<br />
<br />
'''A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].'''<br />
<br />
<br><br />
<br />
'''ABOUT ACCESS RIGHTS:'''<br />
<br />
'''READ:''' this Wiki is ''open for read access to the whole world'' - limiting read and/or edit access to IFI or UiO was not an option (it worked in the past, but the code is now broken, I was told). I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, well, I'll probably have to delete this Wiki, or look for other options (I'm out of ideas, though).<br />
<br />
'''WRITE:''' click around, try to log in, and you'll see that you cannot edit. Only then will I, as wiki administrator, see your user name in a list, and can give you edit rights. So, after failing to edit, send [http://www.welzl.at me] an email (usernames match UiO email addresses) and I'll give you the rights. No, [https://www.uio.no/tjenester/it/utdanning/wiki/hjelp/admin/brukerrettigheter.html there is no more convenient way] to do this.<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**The management structure is not sufficiently described.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page (and, sigh, also the dude that needs to individually give everyone access rights... I wish there were an easier way) - in the long run, I hope that it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=36Main Page2021-01-13T12:57:56Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid making unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.<br />
<br />
'''A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].'''<br />
<br />
<br><br />
<br />
'''ABOUT ACCESS RIGHTS:'''<br />
<br />
'''READ:''' this Wiki is ''open for read access to the whole world'' - limiting read and/or edit access to IFI or UiO was not an option (it worked in the past, but the code is now broken, I was told). I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, well, I'll probably have to delete this Wiki, or look for other options (I'm out of ideas, though).<br />
<br />
'''WRITE:''' click around, try to log in, and you'll see that you cannot edit. Only then will I, as wiki administrator, see your user name in a list, and can give you edit rights. So, after failing to edit, send [http://www.welzl.at me] an email (usernames match UiO email addresses) and I'll give you the rights. No, [https://www.uio.no/tjenester/it/utdanning/wiki/hjelp/admin/brukerrettigheter.html there is no more convenient way] to do this.<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**The management structure is not sufficiently described.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page (and, sigh, also the dude that needs to individually give everyone access rights... I wish there were an easier way) - in the long run, I hope that it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=35Main Page2021-01-13T12:56:19Z<p>Michawe@uio.no: /* Welcome to the "learning from NFR proposal failures" Wiki ! */</p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid making unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.<br />
<br />
'''A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].'''<br />
<br />
<br><br />
<br />
'''ABOUT ACCESS RIGHTS:'''<br />
<br />
'''READ:''' this Wiki is ''open for read access to the whole world'' - limiting read and/or edit access to IFI or UiO was not an option (it worked in the past, but the code is now broken, I was told). I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, well, I'll probably have to delete this Wiki, or look for other options (I'm out of ideas, though).<br />
<br />
'''WRITE:''' click around, try to log in, and you'll see that you cannot edit. Only then will I, as wiki administrator, see your user name in a list, and can give you edit rights. So, after failing to edit, send [http://www.welzl.at me] an email (usernames match UiO email addresses) and I'll give you the rights. No, [https://www.uio.no/tjenester/it/utdanning/wiki/hjelp/admin/brukerrettigheter.html there is no more convenient way] to do this.<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**The management structure is not sufficiently described.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page - I hope it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=34Main Page2021-01-13T12:54:36Z<p>Michawe@uio.no: /* Welcome to the "learning from NFR proposal failures" Wiki ! */</p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid making unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.<br />
<br />
'''A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].'''<br />
<br />
'''ABOUT ACCESS RIGHTS:'''<br />
<br />
'''READ:''' this Wiki is ''open for read access to the whole world'' - limiting read and/or edit access to IFI or UiO was not an option (it worked in the past, but the code is now broken, I was told). I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, well, I'll probably have to delete this Wiki, or look for other options (I'm out of ideas, though).<br />
<br />
'''WRITE:''' click around, try to log in, and you'll see that you cannot edit. Only then will I, as wiki administrator, see your user name in a list, and can give you edit rights. So, after failing to edit, send [http://www.welzl.at me] an email (usernames match UiO email addresses) and I'll give you the rights. No, [https://www.uio.no/tjenester/it/utdanning/wiki/hjelp/admin/brukerrettigheter.html there is no more convenient way] to do this. <br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**The management structure is not sufficiently described.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page - I hope it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Guidelines&diff=33Guidelines2021-01-13T12:53:56Z<p>Michawe@uio.no: </p>
<hr />
<div>'''These guidelines are derived as ''subjective'' interpretations of the reviewer statements on [[Main Page|this page]].'''<br />
==Excellence (the research idea itself, methods, ..)==<br />
<br />
=== '''Clarity, being ground-breaking, ..:''' ===<br />
**Project outputs: be specific about analytic evaluations.<br />
**Describe concrete project results (e.g., simulation model, prototype etc.) in sufficient detail.<br />
**Be clear about goals. E.g., when describing a % improvement over the state of the art, clarify which metrics will be used to quantify the improvement.<br />
**Be ambitious. E.g., 20% better than the SoA is not enough. Explain the ground-breaking nature of the research and its generalisability.<br />
**Avoid looking incremental - careful with showing initial results.<br />
**Be clear about the research plan: how will the research questions be tackled?<br />
**Avoid looking too narrow: there should be a reasonably large research community interested in your work.<br />
**Avoid looking too broad: research objectives should not seem like they could each be a project in its own right.<br />
**Avoid redundancy in the text - e.g., don't embed the research questions in the objectives.<br />
<br />
=== '''Methods:''' ===<br />
**Define methods well, be clear.<br />
**If the focus is on experiments, 1) provide sufficient details on experimental evaluation, and 2) consider adding complementary analytic methods to also attain more theory-oriented scientific achievements.<br />
**Make sure that quantitative research methods are described for all the research activities related to the topics mentioned.<br />
<br />
=== '''Related work:''' ===<br />
**Make sure to provide a comparison with alternative approaches; clarify how the project will differentiate from them.<br />
==Implementation (management)==<br />
<br />
=== '''PhD student supervision:''' ===<br />
**State who will advise the Ph.D. students.<br />
**Give enough details regarding the supervision of PhD students, especially when the project manager's own research has previously had a different focus.<br />
<br />
=== '''International cooperation:''' ===<br />
**When cooperating with international partners, explain how the international consortium will be managed.<br />
**Clarify how collaboration will work in the project (a plan of exchange visits, and stating the intention to submit joint publications is not enough).<br />
**If external collaborators primarily have the expertise on a topic, this is a risk for the project. Avoid this, or state it as a risk and provide a mitigation plan.<br />
<br />
=== '''Other management issues:''' ===<br />
**Consider including (a) work package(s) for integration, result evaluation, project management and dissemination.<br />
**Describe the management structure in sufficient detail.<br />
**Clarify the division of research tasks and responsibilities between team members.<br />
**Avoid looking too generic with the risk management plan; mitigation actions must be detailed and clear.<br />
**A "waterfall arrangement" work package structure may look too simple. Clarify the timeline and dependencies.<br />
==Impact (dissemination, exploitation, ..)==<br />
*If dissemination activities only describe publications in journals and international conferences, consider adding something more. Maybe what the EC calls "communication", i.e. talking to the public, via other media?<br />
*In addition to the technological impact, describe the impact on / importance for society.<br />
*To convince reviewers that there will be a significant impact on the research community, provide avoid too many self-cites in the reference list.<br />
*Provide KPIs for dissemination activities.<br />
*Address the potential exploitation of the research results.<br />
*In case of doing standardisation, provide information about contacts to standardisation channels, and explain in detail how an impact in standardisation will be achieved.<br />
*Explicitly address ethics, safety and gender issues.<br />
*Avoid making the impact description too generic looking. It must be specific to the project.</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=32Main Page2021-01-13T12:53:04Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid making unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.<br />
<br />
A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].<br />
<br />
'''NOTE ABOUT ACCESS RIGHTS:'''<br />
<br />
'''READ:''' this Wiki is ''open for read access to the whole world'' - limiting read and/or edit access to IFI or UiO was not an option (it worked in the past, but the code is now broken, I was told). I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, well, I'll probably have to delete this Wiki, or look for other options (I'm out of ideas, though).<br />
<br />
'''WRITE:''' click around, try to log in, and you'll see that you cannot edit. Only then will I, as wiki administrator, see your user name in a list, and can give you edit rights. So, after failing to edit, send [http://www.welzl.at me] an email (usernames match UiO email addresses) and I'll give you the rights. No, [https://www.uio.no/tjenester/it/utdanning/wiki/hjelp/admin/brukerrettigheter.html there is no more convenient way] to do this. <br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**The management structure is not sufficiently described.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page - I hope it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Guidelines&diff=30Guidelines2021-01-07T10:12:16Z<p>Michawe@uio.no: /* Unclear, not enough detail, not ground-breaking enough: */</p>
<hr />
<div>These guidelines are derived as ''subjective'' interpretations of the reviewer statements on [[Main Page|this page]].<br />
==Excellence (the research idea itself, methods, ..)==<br />
<br />
=== '''Clarity, being ground-breaking, ..:''' ===<br />
**Project outputs: be specific about analytic evaluations.<br />
**Describe concrete project results (e.g., simulation model, prototype etc.) in sufficient detail.<br />
**Be clear about goals. E.g., when describing a % improvement over the state of the art, clarify which metrics will be used to quantify the improvement.<br />
**Be ambitious. E.g., 20% better than the SoA is not enough. Explain the ground-breaking nature of the research and its generalisability.<br />
**Avoid looking incremental - careful with showing initial results.<br />
**Be clear about the research plan: how will the research questions be tackled?<br />
**Avoid looking too narrow: there should be a reasonably large research community interested in your work.<br />
**Avoid looking too broad: research objectives should not seem like they could each be a project in its own right.<br />
**Avoid redundancy in the text - e.g., don't embed the research questions in the objectives.<br />
<br />
=== '''Methods:''' ===<br />
**Define methods well, be clear.<br />
**If the focus is on experiments, 1) provide sufficient details on experimental evaluation, and 2) consider adding complementary analytic methods to also attain more theory-oriented scientific achievements.<br />
**Make sure that quantitative research methods are described for all the research activities related to the topics mentioned.<br />
<br />
=== '''Related work:''' ===<br />
**Make sure to provide a comparison with alternative approaches; clarify how the project will differentiate from them.<br />
==Implementation (management)==<br />
<br />
=== '''PhD student supervision:''' ===<br />
**State who will advise the Ph.D. students.<br />
**Give enough details regarding the supervision of PhD students, especially when the project manager's own research has previously had a different focus.<br />
<br />
=== '''International cooperation:''' ===<br />
**When cooperating with international partners, explain how the international consortium will be managed.<br />
**Clarify how collaboration will work in the project (a plan of exchange visits, and stating the intention to submit joint publications is not enough).<br />
**If external collaborators primarily have the expertise on a topic, this is a risk for the project. Avoid this, or state it as a risk and provide a mitigation plan.<br />
<br />
=== '''Other management issues:''' ===<br />
**Consider including (a) work package(s) for integration, result evaluation, project management and dissemination.<br />
**Describe the management structure in sufficient detail.<br />
**Clarify the division of research tasks and responsibilities between team members.<br />
**Avoid looking too generic with the risk management plan; mitigation actions must be detailed and clear.<br />
**A "waterfall arrangement" work package structure may look too simple. Clarify the timeline and dependencies.<br />
==Impact (dissemination, exploitation, ..)==<br />
*If dissemination activities only describe publications in journals and international conferences, consider adding something more. Maybe what the EC calls "communication", i.e. talking to the public, via other media?<br />
*In addition to the technological impact, describe the impact on / importance for society.<br />
*To convince reviewers that there will be a significant impact on the research community, provide avoid too many self-cites in the reference list.<br />
*Provide KPIs for dissemination activities.<br />
*Address the potential exploitation of the research results.<br />
*In case of doing standardisation, provide information about contacts to standardisation channels, and explain in detail how an impact in standardisation will be achieved.<br />
*Explicitly address ethics, safety and gender issues.<br />
*Avoid making the impact description too generic looking. It must be specific to the project.</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Guidelines&diff=29Guidelines2021-01-07T09:46:28Z<p>Michawe@uio.no: /* blabla */</p>
<hr />
<div>These guidelines are derived as ''subjective'' interpretations of the reviewer statements on [[Main Page|this page]].<br />
==Excellence (the research idea itself, methods, ..)==<br />
<br />
=== '''Unclear, not enough detail, not ground-breaking enough:''' ===<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
<br />
=== '''Problems with methods:''' ===<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
<br />
=== '''Related work:''' ===<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
==Implementation (management)==<br />
<br />
=== '''PhD student supervision:''' ===<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
<br />
=== '''International cooperation:''' ===<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
<br />
=== '''Other management issues:''' ===<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**The management structure is not sufficiently described.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
==Impact (dissemination, exploitation, ..)==<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Guidelines&diff=28Guidelines2021-01-07T09:45:12Z<p>Michawe@uio.no: Created page with "===blabla=== ''blabla'' blabla ==Excellence (the research idea itself, methods, ..)== === '''Unclear, not enough detail, not ground-breaking enough:''' === **The proposal al..."</p>
<hr />
<div>===blabla===<br />
''blabla''<br />
<br />
blabla<br />
==Excellence (the research idea itself, methods, ..)==<br />
<br />
=== '''Unclear, not enough detail, not ground-breaking enough:''' ===<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
<br />
=== '''Problems with methods:''' ===<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
<br />
=== '''Related work:''' ===<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
==Implementation (management)==<br />
<br />
=== '''PhD student supervision:''' ===<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
<br />
=== '''International cooperation:''' ===<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
<br />
=== '''Other management issues:''' ===<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**The management structure is not sufficiently described.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
==Impact (dissemination, exploitation, ..)==<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=27Main Page2021-01-07T09:42:13Z<p>Michawe@uio.no: /* Welcome to the "learning from NFR proposal failures" Wiki ! */</p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid making unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.<br />
<br />
A subjective interpretation of this feedback, to derive guidelines for proposals, is [[Guidelines|here]].<br />
<br />
NOTE: this Wiki is open for read and edit access for all members of IFI. I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict such internal publication of these reviewer feedback statements (actually, I found no statement about any restrictions at all). If I did miss such a statement, please tell me! Then, access to this Wiki could be limited more, or it could even be deleted altogether.<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**The management structure is not sufficiently described.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page - I hope it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=26Main Page2021-01-06T08:01:36Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid making unnecessary mistakes in future proposals. Please feel free to add statements to this list! ...but please do ''not'' include your own interpretation of the feedback here - this page is meant to be strictly used for literal copies of reviewer statements only.<br />
<br />
NOTE: this Wiki is open for read and edit access for all members of IFI. I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict such internal publication of these reviewer feedback statements (actually, I found no statement about any restrictions at all). If I did miss such a statement, please tell me! Then, access to this Wiki could be limited more, or it could even be deleted altogether.<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**The management structure is not sufficiently described.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Research papers and other types of project proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next fitting call may only be around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page - I hope it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=25Main Page2021-01-06T07:55:53Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid making unnecessary mistakes in future proposals. Please feel free to add statements to this list!<br />
<br />
This page is open for read and edit access for all members of IFI.<br />
<br />
I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict such internal publication of these reviewer feedback statements (actually, I found no statement about any restrictions at all). If I did miss such a statement, please tell me! Then, access to this Wiki could be limited more, or it could even be deleted altogether.<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**The management structure is not sufficiently described.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page - I hope it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=24Main Page2021-01-04T13:57:55Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid making unnecessary mistakes in future proposals. Please feel free to add statements to this list!<br />
<br />
NOTE WELL: This page is currently completely open; keep this in mind when adding statements!<br />
<br />
I could not find a statement on the NFR webpage or in the NFR feedback letter that would restrict publication of these reviewer feedback statements. If I did miss such a statement, please tell me! Then, access to this Wiki could be limited, or it could even be deleted altogether.<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**The management structure is not sufficiently described.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page - I hope it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=23Main Page2021-01-04T12:29:07Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid making unnecessary mistakes in future proposals. Please feel free to add statements to this list!<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [..].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [..], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**The management structure is not sufficiently described.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page - I hope it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=22Main Page2021-01-04T12:27:31Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid making unnecessary mistakes in future proposals. Please feel free to add statements to this list!<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
'''TO BE REMOVED BEFORE DISSEMINATING WIDER:'''<br />
<br />
'''Proposal names after the statements below!!'''<br />
<br />
'''This is currently based on 9 NFR proposals.'''<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there.<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail.<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement).<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability.<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking.<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality.<br />
**Some more details on how some of the questions are going to be tackled might have been useful.<br />
**Limited novelty due to the relatively small size of the community doing research around [the RINA architecture].<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives.<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own.<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects.<br />
**Research methods are not well defined.<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing.<br />
**Details on experimental evaluation are not sufficient.<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches.<br />
**How the project will differentiate from other approaches is not described in detail.<br />
**No adequate comparison to existing approaches.<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [irrelevant], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal.<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1.<br />
*'''International cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed.<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications.<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic.<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work.<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation.<br />
**The management structure is not sufficiently described.<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members.<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined.<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified.<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences.<br />
*The technological impact is described, which is however not the case for the impact on / importance for society.<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites.<br />
*There is a lack of KPIs for dissemination activities.<br />
*Also, the potential exploitation of the research results is not addressed.<br />
*But there is a lack of information about contacts to standardisation channels.<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal.<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal.<br />
*The whole impact description appears to be detailed but quite generic.<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page - I hope it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=21Main Page2021-01-04T12:02:43Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid making unnecessary mistakes in future proposals. Please feel free to add statements to this list!<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
'''TO BE REMOVED BEFORE DISSEMINATING WIDER:'''<br />
<br />
'''Proposal names after the statements below!!'''<br />
<br />
'''This is currently based on 7 NFR proposals. RAMIFY, OCARINA, CATRINA, IC-RAIN, DiReQT, MOROCCO, MOROCCOv2 (NAMES TO BE REMOVED)'''<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there. [OCARINA]<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail. [CATRINA]<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement). [RAMIFY]<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability. [DiReQT]<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking. [RAMIFY]<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality. [RAMIFY]<br />
**Some more details on how some of the questions are going to be tackled might have been useful. [DiReQT]<br />
**Limited novelty due to the relatively small size of the community doing research around [the RINA architecture]. [CATINA]<br />
**A lot of repetitions in the text because the research questions are somehow embedded in the objectives. [CATRINA]<br />
**The scope of the project is very broad. Probably, each of these topics is a project on its own. [CATINA]<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects. [IC-RAIN]<br />
**Research methods are not well defined. [CATRINA]<br />
**In some of the research activities related to the topics mentioned, quantitative research methods are missing. [CATINA]<br />
**Details on experimental evaluation are not sufficient. [CATINA]<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches. [OCARINA]<br />
**How the project will differentiate from other approaches is not described in detail. [IC-RAIN]<br />
**No adequate comparison to existing approaches. [CATRINA]<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [irrelevant], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal. [MOROCCOv2]<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1. [IC-RAIN]<br />
*'''International cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed. [MOROCCOv2]<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications. [IC-RAIN]<br />
**Most expertise [on security] lays in the external collaborators – this is a risk as the project investigations focus on this topic. [CATINA]<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work. [OCARINA]<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation. [MOROCCOv2]<br />
**The management structure is not sufficiently described. [MOROCCOv2]<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members. [MOROCCOv2]<br />
**The risk management plan appears to be quite generic and the mitigation actions do not appear to be so refined. [CARTINA]<br />
**Work package structure highly simplified in a waterfall arrangement. The timeline and dependencies are not well identified. [CATINA]<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences. [CATRINA]<br />
*The technological impact is described, which is however not the case for the impact on / importance for society. [IC-RAIN]<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites. [IC-RAIN]<br />
*There is a lack of KPIs for dissemination activities. [IC-RAIN]<br />
*Also, the potential exploitation of the research results is not addressed. [IC-RAIN]<br />
*But there is a lack of information about contacts to standardisation channels. [IC-RAIN]<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal. [DiReQT]<br />
*Ethics, safety and gender issues are not explicitly detailed in the proposal. [CATINA]<br />
*The whole impact description appears to be detailed but quite generic. [CATINA]<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page - I hope it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=20Main Page2020-12-16T12:06:56Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid making unnecessary mistakes in future proposals. Please feel free to add statements to this list!<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
'''TO BE REMOVED BEFORE DISSEMINATING WIDER:'''<br />
<br />
'''Proposal names after the statements below!!'''<br />
<br />
'''This is currently based on 7 NFR proposals. RAMIFY, OCARINA, CATRINA, IC-RAIN, DiReQT, MOROCCO, MOROCCOv2 (NAMES TO BE REMOVED)'''<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there. [OCARINA]<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail. [CATRINA]<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement). [RAMIFY]<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability. [DiReQT]<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking. [RAMIFY]<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality. [RAMIFY]<br />
**Some more details on how some of the questions are going to be tackled might have been useful. [DiReQT]<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects. [IC-RAIN]<br />
**Research methods are not well defined. [CATRINA]<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches. [OCARINA]<br />
**How the project will differentiate from other approaches is not described in detail. [IC-RAIN]<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [irrelevant], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal. [MOROCCOv2]<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1. [IC-RAIN]<br />
*'''International cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed. [MOROCCOv2]<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications. [IC-RAIN]<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work. [OCARINA]<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation. [MOROCCOv2]<br />
**The management structure is not sufficiently described. [MOROCCOv2]<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members. [MOROCCOv2]<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences. [CATRINA]<br />
*The technological impact is described, which is however not the case for the impact on / importance for society. [IC-RAIN]<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites. [IC-RAIN]<br />
*There is a lack of KPIs for dissemination activities. [IC-RAIN]<br />
*Also, the potential exploitation of the research results is not addressed. [IC-RAIN]<br />
*But there is a lack of information about contacts to standardisation channels. [IC-RAIN]<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal. [DiReQT]<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael]. But I'm only the dude who ''started'' this page - I hope it will develop a life of its own!<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=19Main Page2020-12-16T11:56:55Z<p>Michawe@uio.no: /* Final words */</p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid making unnecessary mistakes in future proposals. Please feel free to add statements to this list!<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
All plain text, per bullet, is directly copy+pasted from a proposal review document.<br />
<br />
'''TO BE REMOVED BEFORE DISSEMINATING WIDER:'''<br />
<br />
'''Proposal names after the statements below!!'''<br />
<br />
'''This is currently based on 7 NFR proposals. RAMIFY, OCARINA, CATRINA, IC-RAIN, DiReQT, MOROCCO, MOROCCOv2 (NAMES TO BE REMOVED)'''<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*'''Unclear, not enough detail, not ground-breaking enough:'''<br />
**The proposal also mentions analytic evaluations, but does not become specific there. [OCARINA]<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail. [CATRINA]<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement). [RAMIFY]<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability. [DiReQT]<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking. [RAMIFY]<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality. [RAMIFY]<br />
**Some more details on how some of the questions are going to be tackled might have been useful. [DiReQT]<br />
*'''Problems with methods:'''<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects. [IC-RAIN]<br />
**Research methods are not well defined. [CATRINA]<br />
*'''Related work:'''<br />
**The proposal does not consider alternative Future Internet approaches. [OCARINA]<br />
**How the project will differentiate from other approaches is not described in detail. [IC-RAIN]<br />
<br />
==== Implementation (management) ====<br />
*'''PhD student supervision:'''<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [irrelevant], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal. [MOROCCOv2]<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1. [IC-RAIN]<br />
*'''International cooperation:'''<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed. [MOROCCOv2]<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications. [IC-RAIN]<br />
*'''Other management issues:'''<br />
**The number of requested PhD students might be at the upper edge for this work. [OCARINA]<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation. [MOROCCOv2]<br />
**The management structure is not sufficiently described. [MOROCCOv2]<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members. [MOROCCOv2]<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences. [CATRINA]<br />
*The technological impact is described, which is however not the case for the impact on / importance for society. [IC-RAIN]<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites. [IC-RAIN]<br />
*There is a lack of KPIs for dissemination activities. [IC-RAIN]<br />
*Also, the potential exploitation of the research results is not addressed. [IC-RAIN]<br />
*But there is a lack of information about contacts to standardisation channels. [IC-RAIN]<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal. [DiReQT]<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea. Reviewers who don't like an idea will find negative points in a proposal, but... why give them negative points to begin with? Only top score proposals tend to get funded - every minimal reason to take a point away can kill your proposal. <br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: e.g., there is much more focus on the management side of things, proposals dedicate much more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share. I have experience both in failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael].<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=18Main Page2020-12-16T11:02:52Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid making unnecessary mistakes in future proposals. Please feel free to add statements to this list!<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
'''TO BE REMOVED BEFORE DISSEMINATING WIDER:'''<br />
<br />
'''Proposal names after the statements!!'''<br />
<br />
'''This is currently based on the following proposals: RAMIFY, OCARINA, CATRINA, IC-RAIN, DiReQT, MOROCCO, MOROCCOv2 (will be replaced with just a number)'''<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement). [RAMIFY]<br />
*The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking. [RAMIFY]<br />
*The proposal does not consider alternative Future Internet approaches. [OCARINA]<br />
*The proposal also mentions analytic evaluations, but does not become specific there. [OCARINA]<br />
*Research methods are not well defined. [CATRINA]<br />
*In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail. [CATRINA]<br />
*The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects. [IC-RAIN]<br />
*How the project will differentiate from other approaches is not described in detail. [IC-RAIN]<br />
*On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality. [RAMIFY]<br />
*Some more details on how some of the questions are going to be tackled might have been useful. [DiReQT]<br />
*The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability. [DiReQT]<br />
<br />
==== Implementation (management) ====<br />
*The number of requested PhD students might be at the upper edge for this work. [OCARINA]<br />
*This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed. [MOROCCOv2]<br />
*The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [irrelevant], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal. [MOROCCOv2]<br />
<br />
*No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation. [MOROCCOv2]<br />
*The management structure is not sufficiently described. [MOROCCOv2]<br />
<br />
*The proposal is rather vague about the division of research tasks and responsibilities between team members. [MOROCCOv2]<br />
*It is, however, not clear who will advise the researcher to be hired during Phase 1. [IC-RAIN]<br />
*It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications. [IC-RAIN]<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences. [CATRINA]<br />
*The technological impact is described, which is however not the case for the impact on / importance for society. [IC-RAIN]<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites. [IC-RAIN]<br />
*There is a lack of KPIs for dissemination activities. [IC-RAIN]<br />
*But there is a lack of information about contacts to standardisation channels. [IC-RAIN]<br />
*Also, the potential exploitation of the research results is not addressed. [IC-RAIN]<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal. [DiReQT]<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that it's mainly about the idea quality. Reviewers who don't like an idea will find negative points in a proposal and vice versa, but... why give them negative points to begin with? ... <br />
* and then they use minor details are usually just that – minor. Yet, trust me, I've had proposals that received top scores across the board except for one or two of the details above - and bam, no money. Certainly, a successful proposal is about having a good idea and a good plan to work on it, convincingly described! The list above is only a small element of the whole thing - but it's also easy to make this catalogue, and easy (and probably useful) to apply.<br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: there is so much more focus on the management side of things, proposals dedicate more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share, but I've also had success. So this is based on experience, both failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael].<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=17Main Page2020-12-16T10:59:29Z<p>Michawe@uio.no: /* The list */</p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid making unnecessary mistakes in future proposals. Please feel free to add statements to this list!<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
'''TO BE REMOVED BEFORE DISSEMINATING WIDER:'''<br />
<br />
'''Proposal names after the statements!!'''<br />
<br />
'''This is currently based on the following proposals: RAMIFY, OCARINA, CATRINA, IC-RAIN, DiReQT, MOROCCO, MOROCCOv2 (will be replaced with just a number)'''<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement). [RAMIFY]<br />
*The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking. [RAMIFY]<br />
*The proposal does not consider alternative Future Internet approaches. [OCARINA]<br />
*The proposal also mentions analytic evaluations, but does not become specific there. [OCARINA]<br />
*Research methods are not well defined. [CATRINA]<br />
*In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail. [CATRINA]<br />
*The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects. [IC-RAIN]<br />
*How the project will differentiate from other approaches is not described in detail. [IC-RAIN]<br />
*On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality. [RAMIFY]<br />
*Some more details on how some of the questions are going to be tackled might have been useful. [DiReQT]<br />
*The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability. [DiReQT]<br />
<br />
==== Implementation (management) ====<br />
*The number of requested PhD students might be at the upper edge for this work. [OCARINA]<br />
*This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed. [MOROCCOv2]<br />
*The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [irrelevant], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal. [MOROCCOv2]<br />
<br />
*No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation. [MOROCCOv2]<br />
*The management structure is not sufficiently described. [MOROCCOv2]<br />
<br />
*The proposal is rather vague about the division of research tasks and responsibilities between team members. [MOROCCOv2]<br />
*It is, however, not clear who will advise the researcher to be hired during Phase 1. [IC-RAIN]<br />
*It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications. [IC-RAIN]<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences. [CATRINA]<br />
*The technological impact is described, which is however not the case for the impact on / importance for society. [IC-RAIN]<br />
*Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites. [IC-RAIN]<br />
*There is a lack of KPIs for dissemination activities. [IC-RAIN]<br />
*But there is a lack of information about contacts to standardisation channels. [IC-RAIN]<br />
*Also, the potential exploitation of the research results is not addressed. [IC-RAIN]<br />
*Better explaining how this project will generate impact on Internet standards would strengthen the proposal. [DiReQT]<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that reviewers who don't like an idea will find negative points in a proposal and vice versa. So, minor details are usually just that – minor. Yet, trust me, I've had proposals that received top scores across the board except for one or two of the details above - and bam, no money. Certainly, a successful proposal is about having a good idea and a good plan to work on it, convincingly described! The list above is only a small element of the whole thing - but it's also easy to make this catalogue, and easy (and probably useful) to apply.<br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: there is so much more focus on the management side of things, proposals dedicate more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share, but I've also had success. So this is based on experience, both failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael].<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=16Main Page2020-12-16T10:43:25Z<p>Michawe@uio.no: /* Generic reviewer statements from NFR proposal rejections */</p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid making unnecessary mistakes in future proposals. Please feel free to add statements to this list!<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
''Note: the headlines below are just my effort to categorize the comments; this is not direct advice to write text addressing them into the respective section of a proposal!''<br />
<br />
===== Excellence (the research idea itself, methods, ..) =====<br />
*The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement). [RAMIFY]<br />
*The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking. [RAMIFY]<br />
<br />
==== Implementation (management) ====<br />
*The number of requested PhD students might be at the upper edge for this work. [OCARINA]<br />
*This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed. [MOROCCOv2]<br />
*The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [irrelevant], but less so in [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal. [MOROCCOv2]<br />
<br />
*No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation. [MOROCCOv2]<br />
*The management structure is not sufficiently described. [MOROCCOv2]<br />
<br />
*The proposal is rather vague about the division of research tasks and responsibilities between team members. [MOROCCOv2]<br />
<br />
===== Impact (dissemination, exploitation, ..) =====<br />
*Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences. [CATRINA]<br />
**The proposal does not consider alternative Future Internet approaches. [OCARINA]<br />
**The proposal also mentions analytic evaluations, but does not become specific there. [OCARINA]<br />
**Research methods are not well defined. [CATRINA]<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail. [CATRINA]<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects. [IC-RAIN]<br />
**How the project will differentiate from other approaches is not described in detail. [IC-RAIN]<br />
**The technological impact is described, which is however not the case for the impact on / importance for society. [IC-RAIN]<br />
**Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites. [IC-RAIN]<br />
**There is a lack of KPIs for dissemination activities. [IC-RAIN]<br />
**But there is a lack of information about contacts to standardisation channels. [IC-RAIN]<br />
**Also, the potential exploitation of the research results is not addressed. [IC-RAIN]<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1. [IC-RAIN]<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications. [IC-RAIN]<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality. [RAMIFY]<br />
**<br />
**Some more details on how some of the questions are going to be tackled might have been useful. [DiReQT]<br />
**Better explaining how this project will generate impact on Internet standards would strengthen the proposal. [DiReQT]<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability. [DiReQT]<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that reviewers who don't like an idea will find negative points in a proposal and vice versa. So, minor details are usually just that – minor. Yet, trust me, I've had proposals that received top scores across the board except for one or two of the details above - and bam, no money. Certainly, a successful proposal is about having a good idea and a good plan to work on it, convincingly described! The list above is only a small element of the whole thing - but it's also easy to make this catalogue, and easy (and probably useful) to apply.<br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: there is so much more focus on the management side of things, proposals dedicate more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share, but I've also had success. So this is based on experience, both failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael].<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=15Main Page2020-12-16T10:31:45Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. The idea is that this can become a checklist to avoid making unnecessary mistakes in future proposals. Please feel free to add statements to this list!<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
[proposal names will be removed when the first version of this list is complete!]<br />
**The proposal does not consider alternative Future Internet approaches. [OCARINA]<br />
**The proposal also mentions analytic evaluations, but does not become specific there. [OCARINA]<br />
**The number of requested PhD students might be at the upper edge for this work. [OCARINA]<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed. [MOROCCOv2]<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [irrelevant], but less so in [proposal theme]. The support from the external members will be significant at least in the areas of [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal. [MOROCCOv2]<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation. [MOROCCOv2]<br />
**The management structure is not sufficiently described. [MOROCCOv2]<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members. [MOROCCOv2]<br />
**Research methods are not well defined. [CATRINA]<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail. [CATRINA]<br />
**Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences. [CATRINA]<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects. [IC-RAIN]<br />
**How the project will differentiate from other approaches is not described in detail. [IC-RAIN]<br />
**The technological impact is described, which is however not the case for the impact on / importance for society. [IC-RAIN]<br />
**Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites. [IC-RAIN]<br />
**There is a lack of KPIs for dissemination activities. [IC-RAIN]<br />
**But there is a lack of information about contacts to standardisation channels. [IC-RAIN]<br />
**Also, the potential exploitation of the research results is not addressed. [IC-RAIN]<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1. [IC-RAIN]<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications. [IC-RAIN]<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality. [RAMIFY]<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement). [RAMIFY]<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking. [RAMIFY]<br />
**Some more details on how some of the questions are going to be tackled might have been useful [DiReQT].<br />
**Better explaining how this project will generate impact on Internet standards would strengthen the proposal. [DiReQT]<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability. [DiReQT]<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that reviewers who don't like an idea will find negative points in a proposal and vice versa. So, minor details are usually just that – minor. Yet, trust me, I've had proposals that received top scores across the board except for one or two of the details above - and bam, no money. Certainly, a successful proposal is about having a good idea and a good plan to work on it, convincingly described! The list above is only a small element of the whole thing - but it's also easy to make this catalogue, and easy (and probably useful) to apply.<br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: there is so much more focus on the management side of things, proposals dedicate more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share, but I've also had success. So this is based on experience, both failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael].<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=14Main Page2020-12-16T10:28:20Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
There's nothing more annoying than getting a review back from Forskningsrådet (NFR) that's generally favourable in text, perhaps also with high scores, only minor details being criticized, and then nevertheless not getting any funding. I know that, I've had this a few times.<br />
<br />
Hence, a checklist of general mistakes to avoid is probably useful.<br />
<br />
This is a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. This should help future proposal writing efforts. Please feel free to add statements to this list!<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
[proposal names will be removed when the first version of this list is complete!]<br />
**The proposal does not consider alternative Future Internet approaches. [OCARINA]<br />
**The proposal also mentions analytic evaluations, but does not become specific there. [OCARINA]<br />
**The number of requested PhD students might be at the upper edge for this work. [OCARINA]<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed. [MOROCCOv2]<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [irrelevant], but less so in [proposal theme]. The support from the external members will be significant at least in the areas of [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal. [MOROCCOv2]<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation. [MOROCCOv2]<br />
**The management structure is not sufficiently described. [MOROCCOv2]<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members. [MOROCCOv2]<br />
**Research methods are not well defined. [CATRINA]<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail. [CATRINA]<br />
**Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences. [CATRINA]<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects. [IC-RAIN]<br />
**How the project will differentiate from other approaches is not described in detail. [IC-RAIN]<br />
**The technological impact is described, which is however not the case for the impact on / importance for society. [IC-RAIN]<br />
**Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites. [IC-RAIN]<br />
**There is a lack of KPIs for dissemination activities. [IC-RAIN]<br />
**But there is a lack of information about contacts to standardisation channels. [IC-RAIN]<br />
**Also, the potential exploitation of the research results is not addressed. [IC-RAIN]<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1. [IC-RAIN]<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications. [IC-RAIN]<br />
**On the one hand, the existence of a solid set of initial results reduces the risk that the project fails in achieving its planned goals. On the other hand, though, it makes the proposal somewhat incremental, reducing its creativity and originality. [RAMIFY]<br />
**The quantitative goals defined in the proposal seem realistic, but somewhat vaguely specified not very ambitious (20% improvement over state of the art does not seem a ground-breaking result, nor it is clear what metrics will be used to quantify the improvement). [RAMIFY]<br />
**The proposed research directions are clearly innovative, but the goals seem somewhat incremental rather than truly ground-breaking. [RAMIFY]<br />
**Some more details on how some of the questions are going to be tackled might have been useful [DiReQT].<br />
**Better explaining how this project will generate impact on Internet standards would strengthen the proposal. [DiReQT]<br />
**The proposal can be strengthened though by better explaining its groundbreaking nature and generalisability. [DiReQT]<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that reviewers who don't like an idea will find negative points in a proposal and vice versa. So, minor details are usually just that – minor. Yet, trust me, I've had proposals that received top scores across the board except for one or two of the details above - and bam, no money. Certainly, a successful proposal is about having a good idea and a good plan to work on it, convincingly described! The list above is only a small element of the whole thing - but it's also easy to make this catalogue, and easy (and probably useful) to apply.<br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: there is so much more focus on the management side of things, proposals dedicate more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share, but I've also had success. So this is based on experience, both failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael].<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=13Main Page2020-10-02T13:30:36Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
There's nothing more annoying than getting a review back from Forskningsrådet (NFR) that's generally favourable, with high scores, and only minor details being criticized, and then nevertheless not getting any funding. I know that, I've had this a few times. I suspect that NFR tends to use quite few reviewers per proposal, which means that luck plays a greater role, as a minor detail giving a minus point somewhere ''can'' make all the difference.<br />
<br />
Hence, a checklist of general mistakes to avoid is probably useful.<br />
<br />
Here comes a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. Please feel free to statements to this list!<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
[proposal names will be removed when the first version of this list is complete!]<br />
**The proposal does not consider alternative Future Internet approaches. [OCARINA]<br />
**The proposal also mentions analytic evaluations, but does not become specific there. [OCARINA]<br />
**The number of requested PhD students might be at the upper edge for this work. [OCARINA]<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed. [MOROCCOv2]<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [irrelevant], but less so in [proposal theme]. The support from the external members will be significant at least in the areas of [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal. [MOROCCOv2]<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation. [MOROCCOv2]<br />
**The management structure is not sufficiently described. [MOROCCOv2]<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members. [MOROCCOv2]<br />
**Research methods are not well defined. [CATRINA]<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail. [CATRINA]<br />
**Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences. [CATRINA]<br />
**The scientific method limits itself to the experimental space, complementary analytic alternatives are not considered. Scientific achievements are expected more on experimental developments rather than on theoretical aspects. [IC-RAIN]<br />
**How the project will differentiate from other approaches is not described in detail. [IC-RAIN]<br />
**The technological impact is described, which is however not the case for the impact on / importance for society. [IC-RAIN]<br />
**Outstanding impact on research community is not fully convincing in view of the reference list provided, since most of them are of the research team without a significant number of non self-cites. [IC-RAIN]<br />
**there is a lack of KPIs for dissemination activities. [IC-RAIN]<br />
**But there is a lack of information about contacts to standardisation channels. [IC-RAIN]<br />
**Also, the potential exploitation of the research results is not addressed. [IC-RAIN]<br />
**It is, however, not clear who will advise the researcher to be hired during Phase 1. [IC-RAIN]<br />
**It is not clear how they will collaborate in the project, but there is a plan of exchange visits and the intention to submit joint publications. [IC-RAIN]<br />
**<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that reviewers who don't like an idea will find negative points in a proposal and vice versa. So, minor details are usually just that – minor. Yet, trust me, I've had proposals that received top scores across the board except for one or two of the details above - and bam, no money. Certainly, a successful proposal is about having a good idea and a good plan to work on it, convincingly described! The list above is only a small element of the whole thing - but it's also easy to make this catalogue, and easy (and probably useful) to apply.<br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: there is so much more focus on the management side of things, proposals dedicate more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share, but I've also had success. So this is based on experience, both failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael].<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=12Main Page2020-10-02T12:53:15Z<p>Michawe@uio.no: /* Generic reviewer statements from NFR proposal rejections */</p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
There's nothing more annoying than getting a review back from Forskningsrådet (NFR) that's generally favourable, with high scores, and only minor details being criticized, and then nevertheless not getting any funding. I know that, I've had this a few times. I suspect that NFR tends to use quite few reviewers per proposal, which means that luck plays a greater role, as a minor detail giving a minus point somewhere ''can'' make all the difference.<br />
<br />
Hence, a checklist of general mistakes to avoid is probably useful.<br />
<br />
Here comes a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. Please feel free to statements to this list!<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
[proposal names will be removed when the first version of this list is complete!]<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed. [MOROCCOv2]<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [irrelevant], but less so in [proposal theme]. The support from the external members will be significant at least in the areas of [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal. [MOROCCOv2]<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation. [MOROCCOv2]<br />
**The management structure is not sufficiently described. [MOROCCOv2]<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members. [MOROCCOv2]<br />
**Research methods are not well defined. [CATRINA]<br />
**In a few cases, the concrete results, e.g., simulation model, prototype etc. are not given in sufficient detail. [CATRINA]<br />
**Dissemination is rather standard, i.e. dissemination activities are mainly based on publications in journals and international conferences. [CATRINA]<br />
**<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that reviewers who don't like an idea will find negative points in a proposal and vice versa. So, minor details are usually just that – minor. Yet, trust me, I've had proposals that received top scores across the board except for one or two of the details above - and bam, no money. Certainly, a successful proposal is about having a good idea and a good plan to work on it, convincingly described! The list above is only a small element of the whole thing - but it's also easy to make this catalogue, and easy (and probably useful) to apply.<br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: there is so much more focus on the management side of things, proposals dedicate more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share, but I've also had success. So this is based on experience, both failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael].<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=11Main Page2020-10-02T12:50:56Z<p>Michawe@uio.no: /* Generic reviewer statements from NFR proposal rejections */</p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
There's nothing more annoying than getting a review back from Forskningsrådet (NFR) that's generally favourable, with high scores, and only minor details being criticized, and then nevertheless not getting any funding. I know that, I've had this a few times. I suspect that NFR tends to use quite few reviewers per proposal, which means that luck plays a greater role, as a minor detail giving a minus point somewhere ''can'' make all the difference.<br />
<br />
Hence, a checklist of general mistakes to avoid is probably useful.<br />
<br />
Here comes a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. Please feel free to statements to this list!<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
[proposal names will be removed when the first version of this list is complete!]<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed. [MOROCCOv2]<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [irrelevant], but less so in [proposal theme]. The support from the external members will be significant at least in the areas of [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal. [MOROCCOv2]<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation. [MOROCCOv2]<br />
**The management structure is not sufficiently described. [MOROCCOv2]<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members. [MOROCCOv2]<br />
**Research methods are not well defined. [CATRINA]<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that reviewers who don't like an idea will find negative points in a proposal and vice versa. So, minor details are usually just that – minor. Yet, trust me, I've had proposals that received top scores across the board except for one or two of the details above - and bam, no money. Certainly, a successful proposal is about having a good idea and a good plan to work on it, convincingly described! The list above is only a small element of the whole thing - but it's also easy to make this catalogue, and easy (and probably useful) to apply.<br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: there is so much more focus on the management side of things, proposals dedicate more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share, but I've also had success. So this is based on experience, both failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael].<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=10Main Page2020-09-28T13:57:18Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
There's nothing more annoying than getting a review back from Forskningsrådet (NFR) that's generally favourable, with high scores, and only minor details being criticized, and then nevertheless not getting any funding. I know that, I've had this a few times. I suspect that NFR tends to use quite few reviewers per proposal, which means that luck plays a greater role, as a minor detail giving a minus point somewhere ''can'' make all the difference.<br />
<br />
Hence, a checklist of general mistakes to avoid is probably useful.<br />
<br />
Here comes a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. Please feel free to statements to this list!<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
[proposal names will be removed when the first version of this list is complete!]<br />
**This project is an international cooperation (..). However, the proposal does not provide much detail on how this international consortium will be managed. [MOROCCOv2]<br />
**The project manager main research so far has been in the area of [different from proposal theme]. He has a strong experience in [irrelevant], but less so in [proposal theme]. The support from the external members will be significant at least in the areas of [proposal theme]. More details regarding the supervision of PhD students should have been given in the proposal. [MOROCCOv2]<br />
**No work package is dedicated to the integration of WP1 and WP2, the evaluation of the results, the project management and the dissemination, which can constitute a risk for the implementation. [MOROCCOv2]<br />
**The management structure is not sufficiently described. [MOROCCOv2]<br />
**The proposal is rather vague about the division of research tasks and responsibilities between team members. [MOROCCOv2]<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that reviewers who don't like an idea will find negative points in a proposal and vice versa. So, minor details are usually just that – minor. Yet, trust me, I've had proposals that received top scores across the board except for one or two of the details above - and bam, no money. Certainly, a successful proposal is about having a good idea and a good plan to work on it, convincingly described! The list above is only a small element of the whole thing - but it's also easy to make this catalogue, and easy (and probably useful) to apply.<br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: there is so much more focus on the management side of things, proposals dedicate more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share, but I've also had success. So this is based on experience, both failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael].<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=9Main Page2020-09-28T12:21:28Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
There's nothing more annoying than getting a review back from Forskningsrådet (NFR) that's generally favourable, with high scores, and only minor details being criticized, and then nevertheless not getting any funding. I know that, I've had this a few times. I suspect that NFR tends to use quite few reviewers per proposal, which means that luck plays a greater role, as a minor detail giving a minus point somewhere ''can'' make all the difference.<br />
<br />
Hence, a checklist of general mistakes to avoid is probably useful.<br />
<br />
Here comes a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. Please feel free to statements to this list!<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
* TODO<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that reviewers who don't like an idea will find negative points in a proposal and vice versa. So, minor details are usually just that – minor. Yet, trust me, I've had proposals that received top scores across the board except for one or two of the details above - and bam, no money. Certainly, a successful proposal is about having a good idea and a good plan to work on it, convincingly described! The list above is only a small element of the whole thing - but it's also easy to make this catalogue, and easy (and probably useful) to apply.<br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: there is so much more focus on the management side of things, proposals dedicate more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share, but I've also had success. So this is based on experience, both failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael].<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=8Main Page2020-09-28T12:19:48Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
There's nothing more annoying than getting a review back from Forskningsrådet (NFR) that's generally favourable, with high scores, and only minor details being criticized, and then nevertheless not getting any funding. I know that, I've had this a few times. I suspect that NFR tends to use few reviewers per proposal, and so this becomes more of a lottery, where a minor detail giving a minus point somewhere can easily make all the difference.<br />
<br />
Hence, a checklist of general mistakes to avoid is probably useful.<br />
<br />
So, here comes a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. Please feel free to statements to this list!<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
* TODO<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of a minor detail?''' I do believe that reviewers who don't like an idea will find negative points in a proposal and vice versa. So, minor details are usually just that – minor. Yet, trust me, I've had proposals that received top scores across the board except for one or two of the details above - and bam, no money. Certainly, a successful proposal is about having a good idea and a good plan to work on it, convincingly described! The list above is only a small element of the whole thing - but it's also easy to make this catalogue, and easy (and probably useful) to apply.<br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: there is so much more focus on the management side of things, proposals dedicate more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share, but I've also had success. So this is based on experience, both failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael].<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.nohttps://wiki.uio.no/mn/ifi/proposalfailures/index.php?title=Main_Page&diff=7Main Page2020-09-28T12:02:29Z<p>Michawe@uio.no: </p>
<hr />
<div>==Welcome to the "''learning from NFR proposal failures''" Wiki ! ==<br />
'''What's the point of this?'''<br />
<br />
There's nothing more annoying than getting a review back from Forskningsrådet (NFR) that's generally favourable, with high scores, and only minor details being criticized, and then nevertheless not getting any funding. I know that, I've had this a few times. I suspect that NFR tends to use few reviewers per proposal, and so this becomes more of a lottery, where a minor detail giving a minus point somewhere can easily make all the difference.<br />
<br />
Hence, a checklist of general mistakes to avoid is probably useful.<br />
<br />
So, here comes a catalogue of statements from NFR proposal rejection letters that are ''generic'', i.e. not about the specific idea being proposed. Please feel free to statements to this list!<br />
<br />
== The list ==<br />
<br />
=== Generic reviewer statements from NFR proposal rejections ===<br />
* asdfsdaf<br />
<br />
== Final words ==<br />
''Here's some kind of FAQ... not that anyone really asked a question, let alone frequently :-) ...but the style seems to fit:''<br />
<br />
* '''Isn't this useless, isn't it just about the quality of the idea? Will good proposals really fail only because of some such minor detail?''' I do believe that reviewers who don't like an idea will find negative points in a proposal and vice versa. Yet, trust me, I've had proposals that were excellent across the board but except for one or two of the details above - and bam, no money, because proposals get ranked by score and there wasn't much funding available for so many proposals. Certainly, a successful proposal begins with a good idea and a good plan to work on it, convincingly described! The list above is certainly only a small element of the whole thing - but it's also easy to make this catalogue, and easy (and probably useful) to apply.<br />
* '''Why not make this page more general, instead of focusing on NFR proposals only?''' Papers and other proposals are different. In case of papers, good venues tend to give constructive feedback, and the next opportunity to use this feedback is right ahead - different from NFR, where there may not be another suitable call at all, or the next call is around a year later. European proposals are a different ballgame too: there is so much more focus on the management side of things, proposals dedicate more space to this kind of text - so, it doesn't seem very useful to me to make (yet another) catalogue of all the general things to consider for EC proposals.<br />
* '''Why this page, are you just a grumpy failure man?''' No no, not grumpy. Failure, sure, I've had my share, but I've also had success. So this is based on experience, both failing and succeeding, in two national funding bodies and the EC. I just felt that this is a constructive (and easy) thing to do.<br />
* '''Who are you anyway?''' [http://www.welzl.at Michael].<br />
----<br />
General [[Wikiinfo|Wiki info]]</div>Michawe@uio.no