Assessment and final results

How do we assess the results?

Independent evaluation

It is not straightforward to share information between companies due to confidentiality. The EU has strict competition laws, for instance OEMs cannot share any cost information. Compliance, i.e. acting conform to the EU competition regulations, makes it a significant challenge to develop new technology and pre-standards together, certainly if these are designed to bring down costs. In addition, it is very important to have an independent evaluation of the demonstrator developments in the ECOCHAMPS project.

In ECOCHAMPS this challenge is met by including a separate horizontal work package, to make sure that the evaluation of the demonstrator vehicles (compared to the project targets) is independent. In this work package, a key role was given to “Golden Engineers”, independent technical experts from Ricardo or the Joint Research Centre (JRC) under the work package lead of Fraunhofer IVI. A Golden Engineer is actually a team of specialists, who were involved in the target setting, tracking and evaluation of the project. The Ricardo engineers were responsible for the two passenger car demonstrators, whereas the JRC engineers evaluated the commercial vehicle demonstrators. Efficiency, emissions, mass and volume figures are made available by the OEMs for inspection by the Golden Engineers, and performance can be assessed by driving the vehicles. The Golden Engineers can see the prototype powertrain costs at a macro-cluster level, and how many vehicles need to be produced to have a successful business case.

The Golden Engineers are not influenced by the commercial relationship between the OEM and supplier. Furthermore, this multi-layer approach ensured confidentiality and compliance within the project. For example, no cost information was shared among the OEMs and suppliers; only the Golden Engineers had direct access to this information. Therefore, the Golden Engineers were able assess all information in more detail than was summarised in the reports. Figure 1 shows how the information goes from very detailed to finally very basic in the report. More details on the work of the Golden Engineers is given in the Transport Research Arena (TRA) publication^[1].

Figure 1 Level of detail during the steps of the assessment.

Setting the targets

When inviting research proposals in 2013, the EU set targets for the to-be-developed hybrid vehicles in the GV-04-2017 call with respect to the best-in-class hybrid vehicle in the market in 2013 or the conventional vehicle on which the demonstrators will be based. The ECOCHAMPS project ambitions were geared to the impact expected by the EU. During the project, the ambitions were refined into three sets of targets:

• End User Requirements (EURs), that represent the consumers’ desires at a mostly qualitative level,
• Key Performance Indicators (KPIs), that are quantified technical targets derived from consumers’ wishes,
• Technical Targets, which are related to the impacts expected by the EC, and the project ambitions.

Since ECOCHAMPS is a large and complex project, each demonstrator has its own target values.

In the case of Light Duty (passenger car) vehicles, the end users are customers that buy and use the cars, but this could also include fleet management companies. In the case of commercial vehicles, the end users are customers who buy the vehicles for commercial use.

Reference vehicles were necessary to baseline the performance of the to-be-developed demonstrator vehicles.  The reference vehicles were carefully selected, to be as close to the call definition as possible whilst enabling a sensible comparison. Sometimes, no best-in-class hybrid vehicle was available in the market in 2013, e.g. for the long haul truck. The Golden Engineers also considered the peculiarities of the different powertrain concepts, when for example comparing a plug-in-hybrid to a conventional hybrid vehicle. The detailed argumentation for the reference vehicle selection is included in D1.2. A summary of the vehicles is provided in Table 1 below.

Table 1 Reference vehicle selection.

End User Requirements

As described above, the End User Requirements (EURs) are mostly qualitative customer’s desires given in general descriptions, since end users are normally no technical experts. Examples of those requirements are given in Table 3‑2. Some requirements, such as reliability, are included to make sure that those topics are considered in the evaluation of the demonstrators. Nevertheless, they are not investigated in detail in this project, because this is not feasible at a demonstrator vehicle level, since this is not as mature as a serial production vehicle.

Table 2 Examples of EURs for the different demonstrator vehicles

Key Performance Indicators

KPIs are defined to quantify EURs, as needed. They provide numbers as targets for the evaluation. Whilst some EURs only needed to be quantified with a specific value to become a KPI, some EURs needed to be split in separate KPIs. E.g. “available capacity” was defined with a value at which the capacity is considered to be sufficient. The “performance” EUR was split into, e.g., maximum speed, acceleration, charging time (where applicable), gradeability and pure electric range (where applicable).

Technical Targets

As mentioned, the starting points for the Technical Targets were the impacts that the EU expected, such as a better powertrain efficiency or a maximum weight increase (hybrid vehicles are typically heavier than their conventional counterpart). When writing the ECOCHAMPS proposal, taking into account the 2013 conventional vehicles and the 2013 Best-in-Class Full Hybrid vehicles (BIC), it became clear that the EU ambitions were well beyond general development trends. The conclusion was that achieving all expected impacts in each of the demonstrators was unachievable. Therefore, each demonstrator focuses on a different sub-set of improvements. Simulations were used during this stage to define realistic targets for the demonstrator vehicles. The resulting Technical Target values are compiled in Table 3.

Evaluating the end result

The Golden Engineers compiled all data (the targets, their definition and the evaluation method) in an assessment matrix to evaluate the demonstrator vehicles. This matrix served as a guideline through the evaluation process and formed the basis for the public assessment report, D1.5.

The testing of the demonstrator vehicles was performed by the OEMs in the presence of the Golden Engineers. The actual testing was slightly different for each demonstrator vehicle. The individual sections below describe this in more detail. For example, the fuel efficiency tests were handled in different ways. They all were carried out by the OEMs themselves. However, the Golden Engineers witnessed part of the tests and assessed the testing method based on data provided.

In general, the project targets were achieved. The table below summarises the results of the assessment of the Technical Targets and End User Requirements. For all details, please see the assessment report in the public deliverable, D1.5: ECOCHAMPS-D1.5-Evaluation Report-PU-FINAL-2018.07.18.

And the Final report.

Overview of the assessment results for the Technical Targets

Overview of the assessment results for the End User Requirements.

^[1] Nitzsche, G., et. al., ECOCHAMPS – Project Targets, their Tracking and the Evaluation of the Demonstrator Vehicles, 7th Transport Research Arena 2018, Vienna, Austria