THE NATIONAL PLAN FOR RESEARCH, DEVELOPMENT AND

INNOVATION 2015-2020, PNIII

Experimental Demonstrative Project

2021 Call for proposals

Guidelines for Experts (Evaluators)

www.uefiscdi.gov.ro
Content :

1. General Information of the call ............................................................................................3

2. Estimated results ..................................................................................................................4
3. Project proposals ..................................................................................................................4
4. Governance of the call .........................................................................................................4
5. Conflict of Interest ................................................................................................................5
6. Contractual agreement ........................................................................................................6
7. Evaluation process................................................................................................................6
7.1. General principles of evaluation (for individual/consensus/panel step) .........................7
7.2 Task of expert evaluators ...................................................................................................8
7.3 Proposal evaluation stages ................................................................................................8
8. Transparency ......................................................................................................................12
9. TRL Definition .....................................................................................................................12
10. References: .......................................................................................................................15

2
Ministry of Research, Innovation and Digitalization and its executive agency, UEFISCDI, welcome you
as a scientific evaluator for the Experimental Demonstrative Project Call 2021 (PED - 2021). This
document specifies in detail the evaluation process, its inputs and outputs, and defines the
responsibilities of the participants in the process.

1. General Information of the call

The major goal of this call is to support the development and testing of demonstrative models
(functional/experimental) for new or significantly improved products, technologies, methods,
systems or services in the areas of smart specialization or public priority.
 Areas of smart specialization:
 Bioeconomy:
o Agro-food
o Bioenergy – biogas, biomass, biofuel
o Biotechnology
 Information and communication technology, space and security:
o Information and communication technology
o Space
o Security
 Energy, environment and climate changes:
o Energy
o Environment and climate changes
o Smart city
 Eco-nano-technology and advanced materials:
o Eco-nano-technology
o Decontamination technologies
o Advanced materials.
 Public health priority areas:
 Health (including science of medicament):
 Heritage and cultural identity:
o Heritage
o Cultural identity.

 The maximum funding granted for a project, with duration between 18 - 24 months, is
600.000 lei (about 122 K EUR).
 The budget allocated to this call for proposals, for the entire implementation period, is
100.000.000 lei (about 20.36 Million EUR).
 The estimated funding rate is 8%.

The selection of the project proposals for funding is based strictly on their merits, assessed through
peer review evaluation performed by experts in the field, with excellence as the sole criterion.

3
2. Estimated results

By financing this type of projects, the following aspects are pursued:

- using of knowledge generated by basic research for developing a higher level of technological
maturity (demonstrator, laboratory-validated technology);
- increasing the research organizations capacity to generate laboratory-validated solutions for new
or significantly improved products/technologies/services and to provide them to the enterprises.

3. Project proposals

Project proposals should address one of the following situations (please see Chapter 9 of this
document for the definitions related to “Technology Readiness Level – TRL”):
- start from a concept of technology/product (TRL 2) and focus on laboratory experimental model
(TRL 3);
- start from a laboratory experimental model (TRL 3) and focus on laboratory validated technology
(TRL 4);
- start from a concept of technology/product (TRL 2) and focus on laboratory validated technology
(TRL 4).

The project proposals were submitted by a research organization1, project coordinator, or in

partnership with other research organizations or enterprises.

4. Governance of the call

Advisory Board for Research-Development and Innovation (CCCDI) is an advisory body of Ministry of
Research, Innovation and Digitization (MCID). The CCCDI consists of representatives of national RDI
system (academia, national research institutes, Romanian Academy, private sector), having a
scientific profile internationally recognized. The CCCDI is the scientific coordinator of the call.

The Executive Agency for Higher Education, Research, Development and Innovation Funding,
UEFISCDI, is a research funding agency. It organizes competitions and subsequently monitors the
implementation of research projects accepted for funding. UEFISCDI manages approximately 22% of
the public funds allocated to research, development and innovation.

The expert evaluators are internationally recognized independent experts who meet the selection
criteria according to the call document. They are responsible for the scientific evaluation of the
submitted proposals according to the evaluation criteria.
The experts perform the work in a personal capacity and must not represent any organization.
During the evaluation process an expert could receive the task of Rapporteur (may act as both
evaluator and Rapporteur for a number of allocated projects).
Each expert evaluator involved in the evaluation process will receive a user name and an individual
access password via e-mail which can authenticate/Log in on the on-line evaluation platform,
www.uefiscdi-direct.ro. A guideline for using evaluation platform PED 2021 is available on section
“Useful Docs”.

1
According to research organization definition from State aid rules
4
5. Conflict of Interest
Experts must declare that they can carry out the evaluation of a proposal with total confidentiality,
impartiality and competence. They must not find themselves in situations where their impartiality
might be questioned, or that could raise suspicion on their recommendations being affected by
elements that lie outside the scope of the evaluation.
A disqualifying conflict of interest exists if an evaluator:
- was involved in the preparation of the proposal;
- is involved as principal investigator or as member of research team in the current call (PED
2021);
- stands to benefit directly should the proposal be accepted;
- has a close family relationship with principal investigator;
- is a director, trustee or partner of an applicant organizations (Project Coordinator or Project
Partners);
- is employed by one of the applicant organizations (Project Coordinator or Project Partners) in
a proposal;
- is in any other situation that compromises his or her ability to evaluate the proposal
impartially.
A potential conflict of interest may be considered, even in cases not covered by above disqualifying
conflicts, if an evaluator:
- was employed by the organization that has applied a proposal within the previous 3 years;
- is involved in a research collaboration or contract of principal investigator, or had been so in
the previous 3 years;
- is in any other situation that could cast doubt on his or her ability to evaluate the proposal
impartially, or that could reasonably appear to do so in the eyes of an external third party.
Before starting the assessment of proposals allocated to them, the evaluators will need to login at
the Evaluation platform, www.uefiscdi-direct.ro. Upon login, the evaluators get access to the
Principal Investigator information and the proposal’s summary of the respective allocated proposals.
This will allow the experts to check their potential conflict of interest and directly announce the
UEFISCDI staff if such conflicts are identified. Full access to the proposal is given after the expert has
confirmed that there is no conflict of interest.
The expert evaluators must notify UEFISCDI, via email/in writing, at any moment during the
evaluation process, if they become aware that either one of these conditions is not satisfied or that
they are in conflict of interest. When a potential conflict of interest is reported by an expert or
brought to the attention of UEFISCDI by other means, UEFISCDI will analyse the circumstances and
decide on a case-by-case basis whether the conflict is real. In the latter case, the expert will be
excluded from the evaluation of the respective proposal.
Once you have confirmed that you do not have a conflict of interest, you will be given full access to
each confirmed proposal.

5
6. Contractual agreement
The relationship between UEFISCDI and the expert evaluators is defined by a contractual agreement
written and signed by both parties. By signing this agreement, the expert evaluators accept the
conditions regarding the evaluation tasks, the confidentiality, the conflict of interest, and the use of
personal data by UEFISCDI, according to the provisions of the Regulation (EU) 2016/679 (GDPR) and
to the Law 190/2018 on the protection of natural persons with regard to the processing of personal
data and on the free movement of such data. More details are available at:
https://uefiscdi.ro/protectia-datelor-cu-caracter-personal. UEFISCDI cannot allocate proposals to an
expert who has not been officially appointed (i.e., the expert has signed the contractual agreement
and, in doing so, has agreed to the terms laid down in it, including, in particular, confidentiality and
conflict of interest aspects).

The evaluators will be remunerated for their activities after the finalization of the evaluation process.
The following expenses will be covered by the call:
- 55 € (49 € net amount) evaluation fee per one individual evaluation;
- 55 € (49 € net amount) additional fee per Consensus Report made by Rapporteur;
- 275 € (247 € net amount) fee per day for participation in panel meeting.
The expert must sign the “Contractual Agreement” with all the appendixes and upload them to the
evaluation platform (dedicated section).
Omission to upload the „Contractual Agreement” and its appendixes, in due time, to the dedicated
section of the online evaluation platform, will lead to delay of payment for the activity as expert
evaluator/Rapporteur!

7. Evaluation process
The evaluation process is described in the Call document available here:
https://uefiscdi.gov.ro/resource-868644-pi_ped_en_2021.pdf.

The evaluation process consists of following steps:

 Step 1 - The eligibility check is made by UEFISCDI staff. If, during or after completion of the
evaluation phase, a non-compliance with any of eligibility criteria is found, the project
proposal will be declared ineligible and will be excluded from the competition.

 Step 2 - The individual evaluation

Each project proposal declared eligible is evaluated, from the quality point of view,
independently, online, by three expert evaluators. Each expert prepares an Individual
Evaluation Report (IER).

 Step 3 – Rebuttal
The applicants receive the invitation to submit a rebuttal on the comments expressed by
experts in individual evaluation step. This step is not mandatory and the absence of rebuttal
must not affect the evaluation of the project.

 Step 4 - Reaching consensus - The Consensus Report (CR) is elaborated by the rapporteur
based on the IER, on discussions between the all 3 experts and the rebuttal (if any).

6
  Step 5 - Evaluation Panel Meeting – The proposals without consensus are discussed in the
panel meeting (one for each scientific domain).

7.1. General principles of evaluation (for individual/consensus/panel step)

 The experts must assess the proposals by themselves – do not delegate this task to anybody
else.
 The expert should evaluate the proposal as it is, not as it could be or as the expert would like
it to be, and without giving any recommendations or suggestions.
 Each criterion must contain the Strengths/Weaknesses of the respective criterion.
 Each argument should be placed under the specific criterion, with great care of not mixing
criteria (e.g., comments about the activities and deliverables will not be placed under “Project
objectives and scope” of the art” or comments about the state of the art should not be placed
under “Project implementation”).
 All comments should be clear statements, based on facts presented in the proposal and not
on opinions of the experts (e.g. comments as “I think that”, “My impression is”, “It seems
that”, “The applicant should”, “It may be better”, etc. must be avoided).
 All facts that are considered relevant to the current proposal should be considered, regardless
of the section of the proposal where these are to be found.
 Any comment referring to inexistent or irrelevant criteria for the present competition is
considered a procedural mistake which may lead to a successful redress and justify a re-
evaluation of the proposal. Never penalize a proposal based on information that the applicant
was not expected to provide.
 The comments should assess the quality of the described criterion under evaluation and not
summarise it or suggest improvements.
 A weakness should be addressed only once so that double penalization does not occur.
 The IER must be carefully checked for conflicting statements, especially when these may pop
up under different criteria.
 The expert must treat all proposals equally and evaluate them impartially on their merits.
Comments that hint, indicate or refer to names, numbers, gender, institutions, nationality,
and age are strictly forbidden.
 Scores must reflect the overall assessment of a criterion and the experts must use the full
range of scores to appropriately highlight the quality of the proposal;
 The panel members will analyse with objectivity the proposals (with no consensus reached)
and will carefully prepare the final report which will be sent to the Principal Investigator. The
report should be based on facts and solely on proposal content.
 The experts should keep in mind that they are evaluating a project proposal and not a
research paper. The comments should assess the quality of the described criterion under
evaluation and not summarise it or suggest improvements.

A personal touch: Please, evaluate the proposal as you would like it to be evaluated if it were
yours: be objective, dispassionate, unbiased, fair and polite.

7
7.2 Task of expert evaluators
The expert evaluators are required to:
- read the “Call document” and the “Guide for Experts”;
- inform UEFISCDI about a disqualifying or a potential conflict of interest;
- read and objectively evaluate the assigned project proposals;
- meet all deadlines of the evaluation process;
- fill in and submit the evaluation sheet for each assigned project proposal, providing
comprehensive comments that evaluate the proposal in a critical way, addressing all the
evaluation criteria for each point, avoiding summarizing and advising on improvements,
clearly highlighting the strengths and weaknesses of the proposal as it was submitted by the
applicant and not its potential;
- read the rebuttal (if any);
- actively participating in consensus discussions regarding all assigned project proposals, by
using the "forum" type interface available on the online evaluation platform; comments at
the consensus stage are compulsory; express the agreement or disagreement (vote) for the
consensus report;
- elaborate the consensus report (Rapporteurs only) based on the individual evaluations, the
rebuttal (if any) and the discussions with the other evaluators and ultimately agree the scores
for each evaluation criterion that fully reflects the agreed comments in the final evaluation
report;
- not disclose the proposals assigned to third parties;
- communicate with the assigned Technical Officer from UEFISCDI for any issue that might
appear at any moment during the evaluation process;
- take part at the panel meeting (if case).

7.3 Proposal evaluation stages

According to the Call Document, the Funding Application uses Times New Roman font type, 11 font
size, 1 line spacing and 2 cm margins. Any modification to these parameters (excepting the tables,
figures or their captions) is forbidden. The pages exceeding the established limits (section B2 – max.
15 pages and section C – max. 2 pages) will not be taken into consideration in the evaluation
process.

7.3.1. Individual Evaluation

Each eligible proposal is evaluated independently, online, by 3 experts, using the platform
www.uefiscdi-direct.ro.
Each of the three evaluators prepares an Individual Evaluation Report (IER), awarding individual
scores for each criterion according to the evaluation criteria. Given scores for each criterion are
necessarily to be justified by comments. The comments must be accurate, complete and consistent,
highlighting strengths and weaknesses.
 Information found in pages that exceed the maximum limit for each section will be
disregarded!

8
When all IERs have been submitted for a proposal, the experts will have access to each other’s scores
and comments and may adjust their own scores/remarks during this phase. This is required
especially when there are conflicting statements, which should be solved before the rebuttal.

7.3.2. Rebuttal
The IERs are sent to the principal investigators, with the invitation to submit a rebuttal on the
comments expressed by the experts within 3 working days. The rebuttal written in English is limited
to 6.000 characters (including spaces) and consists in counterarguments strictly regarding the
criticism formulated by evaluators in IERs. The principal investigator should not include in the
rebuttal any supplementary information. New facts or information that appear in the rebuttal but
have not been described in the proposal shall be disregarded. The experts should carefully read the
responses of the applicant and assess whether, after checking with the proposal, they maintain their
opinion.
The answer of the principal investigator (rebuttal) is not compulsory, and its absence will not affect
the next stage of the evaluation process.

7.3.3. The Consensus Report (CR)

After rebuttals are received from the applicants, the evaluation process enters the consensus phase,
under the coordination of a Rapporteur, selected among the three initial experts. The task of the
Rapporteur is to coordinate the consensus discussions through a “forum” type interface available on
the online evaluation platform, taking into consideration the applicant’s rebuttal (if any). The
Rapporteur will identify agreements as well as divergences in the IERs, will exchange views with the
other experts and will propose comments acceptable to all the experts involved. Where the views
are very different, the Rapporteur will engage the other two experts in a focused discussion on the
major disagreements and will seek to clarify any difference of opinion and contradictions. All experts
have to actively participate in the discussion and clearly express their opinion in a reasonable time
Afterwards, the Rapporteur will write the Consensus Report.
The other two experts will then express their opinion on the CR (comments and scores) by voting
“agree” or “disagree” on the same platform.
If the CR receives “agree” in unanimity, then the consensus has been reached and Consensus Report
becomes the Final evaluation report.
If one of the evaluators votes “disagree” or she/he does not express her/his vote on the CR, for that
specific project, the consensus has not been reached.
If the consensus is not reached, the expert(s) who has (have) voted “disagree” will be asked to
detail(s) the reasons for disagreeing with the CR for any particular criterion.

7.3.4 Evaluation Panel Meeting

Proposals for which no consensus has been reached will be analysed/discussed in the panel
meetings. At the competition level, 6 panels will be established, each panel covering a domain of
smart specialization / public priority.

9
The domain panel is constituted of rapporteurs and evaluators. The size of each panel will be
correlated with the number of projects without consensus reached. Depending on the size of the
panel, more plenary sessions can be organized.
Each panel member, prior to the panel meeting, will have access to the project proposals, the IERs,
CRs and the rebuttals (if any) for all projects allocated to the panel.
Within the panel meetings, every project proposal which has at least a “disagree” vote or lacks a vote
to CR is presented and analysed in the panel. For each project, the panel establishes, by consensus,
the final score and writes the Final evaluation report.
The panel meetings are coordinated by a chair /co-chair and they will moderate the panel discussion,
without interfering in decision making.
The final score will be decided by the panel and will take into account the scores and comments from
previous evaluation stages and also the discussions from the panel. The major changes of scores will
be motivated for each modified criterion.
After the discussions, for each proposal, a member of panel will be appointed by chair/co-chair and
will prepare the Final Report, in accordance with those established during the panel meeting.
The Final Report may contain parts from the IER or the CR, agreed by the panel.

7.3.6 Evaluation Form

Make your judgment against the official evaluation criteria, as stated in the Evaluation Form, and
nothing else. For all the projects accepted for evaluation, the evaluator has to be sure that the whole
rating scale is used for each criterion.
The comments must be in accordance with the scores and be accurate, complete and consistent.

Organizing comments on each criterion:

Comments should take the form of a statement reflecting the overall quality of the proposal, in the
light of the above mentioned criteria. In particular, the following guidelines should be followed:
- make sure that the TRL is clearly argued at the beginning of the project and the TRL reached
after the project implementation is well determined on TRL scale;
- make sure that each argument is put under the right criterion and comments are confined
only to the criterion concerned;
- make sure that you reach all the aspects raised by the questions listed under each criterion;
- give comments and scores for all evaluation criteria; the scores must match the comments;
- do not apply a penalty twice to a proposal for the same weak point; a basic underlying fault in
a proposal could impact more than one criterion, make clear that these are different and
distinct problems;
- never apply a penalty to a proposal based on information that the applicant was not expected
to provide;
- avoid references to the applicant age, nationality, gender, or personal matters;
- avoid any direct comparison with any other proposals under current call;
- critical comments should be constructive and not offensive.

10
Structure of evaluation form:
I. Project objectives and scope (30%)
The criterion relates to Project Application Form – Part B.2.1
Evaluate to what extent:
• The project scope is clearly presented, describing explicitly the demonstration model
(product, technology, method, system or service) to be developed and tested/ validated?
• Are the results innovative and relevant in relation to the national and international state of
the art?
• Are the project objectives correlated with the outcome of the project? Is the project feasible?
• Is the TRL level clearly argued at beginning of the project, and is the level reached after
project implementation well determined on the TRL scale?

II. Presentation of the technology / product concept or the existing lab product (40%)
The criterion relates to Project Application Form – Part B.2.2
Evaluate to what extent:
• Are the preliminary results significant on the date of submitting the proposal: publications,
patents and research projects that led to basic concept of the project?
• Is the expertise level of experienced researchers and postdoctoral researchers nominated in
the project team good? Is the expertise well-correlated with their contribution in the project?

III. Project implementation (30%)

The criterion relates to Project Application Form – Part B.2.3
Evaluate to what extent:
 Are the proposed activities and deliverables well structured?
 Is the budget and timetable of the project well justified (resources / time / results)?
 Is the research infrastructure useful/innovative (used in order to implement the project)?
 Is each team member’s role and team project partner well described (if applicable)?
 Is the impact and dissemination of project results well described? Are intellectual property
rights well shared?
 Are the risks associated with project implementation activities identified and the ways to
treating them well described?

NOTES:
1. Each Criterion will be scored from 0 to 5. Scores with a resolution of decimal place may be
awarded.
2. The final score will be calculated as a sum of the marks for each criterion multiplied by the
appropriate percentage and multiplied by 20 (final score is between 0 and 100).
Final grade = (s.1*30/100 + s.2*40/100 + s.3*30/100)*20, where s.i is the score for criterion i.
3. Experts should make sure that their comments on each criterion are:
 Concrete i.e., they are explicitly referring to the information in the proposal
 Complete i.e., they address all the facets specified by the criterion
 Consistent i.e., they match the score, according to the scoring table and are not contradictory
 Inoffensive i.e., they do not contain discriminatory, offensive statements or adjectives
11
  Explanatory i.e., it is clear what makes a comment a weakness or a strength. Examples of
poor comments “The methodology is described” (is it enough? insufficient? excellent? new?
obsolete?), “The novelty is not good” (why? what is missing?) “The team is not appropriate”
(what competency is missing?), etc.
4. The project proposals under the score of 80 points will not be considered for funding.

7.3.7 Assessment against the Evaluation Criteria

The Evaluators/Rapporteurs propose a score only after written comments (accurate, concrete,
complete (i.e., address all questions) and consistent with the semantics of each score, namely:

0 The proposal does not address this criterion, thus it cannot be assessed
0 Insufficient due to missing or incomplete information
1 Addressing criterion is done improperly, or there are serious weaknesses
1 Poor
2 The proposal broadly addresses the criterion, but there are significant
2 Fair weaknesses
3 The proposal addresses the criterion well, although improvements would
3 GOOD be necessary.
4 The proposal addresses the criterion very well, although certain
4 VERY GOOD improvements are still possible.
5 The proposal successfully addresses all relevant aspects of the criterion.
5 EXCELLENT Any shortcomings are minor appeared.

The scores must reflect the strengths and weaknesses and they must be in line with the comments.

8. Transparency
The list of the expert evaluators who participated in the evaluation process will be published on the
UEFISCDI’s website, after the end of the competition. The list will not identify the expert evaluators
assigned for each project proposal.

9. TRL Definition
All the TRL definitions are presented below, but the project proposals must be in one of the
following situations:
- start from a concept of technology / product (TRL 2) and focus on laboratory experimental
model (TRL 3);
- start from a laboratory experimental model (TRL 3) and focus on laboratory validated technology
(TRL 4);
- start from a concept of technology / product (TRL 2) and focus on laboratory validated
technology (TRL 4).

12
TRL Definition & Description2:
Technology TRL Definition Description
Readiness
Level
TRL 1 Initial scientific research begins. Basic principles are observed. Focus is on fundamental
Examples include studies on understanding of a material or process.
basic material properties.
Principles are qualitatively
postulated and observed.
TRL 2 Technology concept and/or Once basic principles are observed, practical
application formulated. Initial applications can be identified. Applications are
practical applications are speculative, and there may be no proof or detailed
identified. Potential of material analysis to support the assumptions. Examples are still
or process to satisfy a limited to analytic studies. Supporting information
technology need is confirmed. includes publications or other references that outline
the application being considered and that provide
analysis to support the concept. The step up from TRL 1
to TRL 2 moves the ideas from basic to applied
research. Most of the work is analytical or paper studies
with the emphasis on understanding the science better.
Experimental work is designed to corroborate the basic
scientific observations made during TRL 1 work.
TRL 3 Analytical and experimental Analytical studies and laboratory-scale studies are
critical function and/or designed to physically validate the predictions of
characteristic proof of concept. separate elements of the technology. Supporting
Applied research continues and information includes results of laboratory tests
early stage development begins. performed to measure parameters of interest and
Includes studies and initial comparison to analytical predictions for critical
laboratory measurements to components. At TRL 3 experimental work is intended to
validate analytical predictions of verify that the concept works as expected. Components
separate elements of the of the technology are validated, but there is no strong
technology. attempt to integrate the components into a complete
system. Modeling and simulation may be used to
complement physical experiments.
TRL 4 Laboratory Testing/Validation of The basic technological components are integrated to
Alpha Prototype establish that the pieces will work together. This is
Component/Process. Design, relatively "low fidelity" compared with the eventual
development and lab testing of system. Supporting information includes the results of
technological components are the integrated experiments and estimates of how the
performed. Results provide experimental components and experimental test results
evidence that applicable differ from the expected system performance goals. TRL

2
H2020 – Work Programme 2018-2020, General Annexes – pg. 27
http://ec.europa.eu/research/participants/data/ref/h2020/other/wp/2018-2020/annexes/h2020-wp1820-annex-ga_en.pdf
Technology Readiness Assessment Guide - Department of Energy, U.S, pg. 9-10/34
https://www.directives.doe.gov/directives-documents/400-series/0413.3-EGuide-04/@@images/file;
Technology Readiness Levels definitions and descriptions
https://www.dst.defence.gov.au/sites/default/files/basic_pages/documents/TRL%20Explanations_1.pdf
13
Technology TRL Definition Description
Readiness
Level
component/process 4-6 represent the bridge from scientific research to
performance targets may be engineering, from development to demonstration. TRL
attainable based on projected 4 is the first step in determining whether the individual
or modeled systems. components will work together as a system. The goal of
TRL 4 should be the narrowing of possible options in the
complete system.
TRL 5 Laboratory Testing of The basic technological components are integrated so
Integrated/Semi-Integrated that the system configuration is similar to (matches) the
System. Component and/or final application in almost all respects. Supporting
process validation in relevant information includes results from the laboratory scale
environment- (Beta prototype testing, analysis of the differences between the
component level). laboratory and eventual operating
system/environment, and analysis of what the
experimental results mean for the eventual operating
system/environment. The major difference between
TRL 4 and 5 is the increase in the fidelity of the system
and environment to the actual application. The system
tested is almost prototypical. Scientific risk should be
retired at the end of TRL 5. Results presented should be
statistically relevant.
TRL 6 Prototype System Verified. Engineering-scale models or prototypes are tested in a
System/process prototype relevant environment. This represents a major step up
demonstration in an operational in a technology’s demonstrated readiness. Examples
environment- (Beta prototype include fabrication of the device on an engineering pilot
system level). line. Supporting information includes results from the
engineering scale testing and analysis of the differences
between the engineering scale, prototypical
system/environment, and analysis of what the
experimental results mean for the eventual operating
system/environment. TRL 6 begins true engineering
development of the technology as an operational
system. The major difference between TRL 5 and 6 is
the step up from laboratory scale to engineering scale
and the determination of scaling factors that will enable
design of the final system. The engineering pilot scale
demonstration should be capable of performing all the
functions that will be required of a full manufacturing
system. The operating environment for the testing
should closely represent the actual operating
environment. Refinement of the cost model is expected
at this stage based on new learning from the pilot line.
The goal while in TRL 6 is to reduce engineering risk.
Results presented should be statistically relevant.

14
 Technology TRL Definition Description
Readiness
Level
TRL 7 Integrated Pilot System This represents a major step up from TRL 6, requiring
Demonstrated. System/process demonstration of an actual system prototype in a
prototype demonstration in an relevant environment. Final design is virtually complete.
operational environment- The goal of this stage is to retire engineering and
(integrated pilot system level). manufacturing risk. To credibly achieve this goal and
exit TRL 7, scale is required as many significant
engineering and manufacturing issues can surface
during the transition between TRL 6 and 7.
TRL 8 System Incorporated in The technology has been proven to work in its final
Commercial Design. Actual form and under expected conditions. In almost all
system/process completed and cases, this TRL represents the end of true system
qualified through test and development. Examples include full scale volume
demonstration- (Pre- manufacturing of commercial end product. True
commercial demonstration). manufacturing costs will be determined and deltas to
models will need to be highlighted and plans developed
to address them. Product performance delta to plan
needs to be highlighted and plans to close the gap will
need to be developed.
TRL 9 System Proven and Ready for The technology is in its final form and operated under
Full Commercial Deployment. the full range of operating conditions. Examples include
Actual system proven through steady state 24/7 manufacturing meeting cost, yield,
successful operations in and output targets. Emphasis shifts toward statistical
operating environment, and process control.
ready for full commercial
deployment.

10. References:

In creating this guide, we adopted several guidelines and principles from the following sources:
1. H2020-MSCA, EJD Manual for experts, 2019
2. ESF - European Peer Review Guide. Integrating Policies and Practices into Coherent
Procedures, 2011: http://www.esf.org/activities/mo-fora/publications.html
3. ERC - ERC Grant Schemes Guide for Peer Reviewers, 2011:
http://erc.europa.eu/sites/default/files/document/file/GuideForERCPeerReviewers_2012%20
20092011.pdf
4. http://ec.europa.eu/research/pdf/workshop_igb/rtd_evaluation_process.pdf

NOTE: Remember that nobody knows everything, and other peoples’ point of view is as valid as
your own. And there is no shame in learning and changing your mind!

15