Evaluation Matrix RDE-LDV Test Procedure

Evaluation matrix RDE-LDV test procedures

Note: JRC has consolidated the evaluation matrix based on available knowledge with the aim of providing an unbiased and objective evaluation of two candidate procedures. We ask stakeholders to carefully check whether both the definitions and the consolidated evaluation matrix sufficiently represent their views and insights.

Background

Objective - test procedure:The procedure shall assess the real-driving emissions of light-duty vehicles under normal European operating conditions, therebyensuring the proper functioning of emissions control technologies.

Objective - evaluation matrix:The evaluation matrix shall: (i) provide rationale for assessing the technical feasibility of test procedures and (ii) help identifying one test procedure that reproduces the real-driving emissions of light-duty vehicles most reliably at a given cost.

Definitions

On-road emissions testing

with PEMS: A procedures that employs Portable Emissions Measurement Systems (PEMS) to test vehicle emissions on the road under the range of operating conditions as they occur during normal vehicle use.

Orographic conditions:The local and regional elevation characteristics of a location, typically characterized by the absolute altitude and the altitude gradient of surrounding plains, valleys, hills, or mountains.

Random cycle: A laboratory procedure that employs a random driving cycle to test vehicle emissions under a potentially wide range of driving and ambient conditions.

Repeatability:The degree of agreement between multiple emission measurements conducted on one specific vehicle under the same boundary conditions by one test facility. High repeatability leads to comparable results, i.e., results which vary negligibly from each other given the objective of the test procedure.

Reproducibility:Reproducibility can be understood as the degree of agreement between multiple emissions measurements conducted on one specific vehicle under the wide range of permissible boundary conditions by various test facilities. In the context of the RDE-LDV procedure, reproducibility may be understood as pass-fail reproducibilityof emission tests, i.e., the likelihood that a pass-fail conclusion for a specific vehicle based on one single emissions test remains valid if the same vehicle is tested under any otherpermissible boundary condition by any other test facility. Example: A manufacturer has tested Vehicle A once on a random cycle in the laboratory/on the road. The results indicate that the emissions of Vehicle A are below a predefined non-to-exceed limit. High reproducibility would now mean a high likelihood that this conclusion can be maintained for any other emissions test within the permissible boundary conditions of the test procedure?

RobustnessRobustness is defined here as the capability of an emissions test procedure to determine whether the emissions control system is able to keep the emissions below a certain not-to-exceed limit regardless of the operating conditions within the boundaries of normal vehicle use.

Table 1: Overview evaluation matrix

Random cycle / On-road emissions testing with PEMS
Effectiveness
Preventing detection andby-passing of emissions test by the engine control unit / Medium to very low / Very high
Preventing a too narrow calibration of vehicle technologies / Potentially medium / Potentially high
Reproducibility and repeatability1 / High / Medium
Accuracy of the analytical equipment / Very high / Very high
Influence of measurement procedure/equipment on results / Medium / Medium
Use of the method for development purposes / Medium / Medium
Equal treatment of OEMs from different countries regarding, e.g., speed limits, orographic location / Potentially medium-high / Medium
Coverage
Coverage of the range of driving pattern, and if necessary allows targeting specific driving conditions (e.g., urban driving) / Potentially high / Potentially very high
Coverage of road profiles (slopes, etc.) / Potentially low / Potentially high
Coverage of ambient conditions (temperature, altitude, humidity, background pollutants, intake air, etc.) / Potentially medium / Potentially medium
Coverage of regulated pollutants / Very high / High
Applicability
Applicability to a wide range of LDVs (e.g., categories, powertrain technologies) / High / High
Neutrality towards engine and after-treatment technologies / High / High
Possibility to execute tests during the whole year / Very high / Potentially high
Capacity of test facilities / Uncertain / Uncertain
Costs and benefits
Availability of know-how and dissemination thereof / High / Medium-high
Initial cost of implementing the procedure2 / Potentially medium / Potentially medium
Running costs/resources of executing vehicle tests2 / Potentially medium / Potentially medium
Degree to which methods for emissions characterization are available / Medium / Medium
Principle practicability / Potentially low-high / Potentially medium-high
International attractiveness (contribution to the harmonization of emission tests and standards) / Medium / Potentially high
Safety of measurements (risk of CO exposure, drive on public roads with Helium and Hydrogen bottles) / Very high / Medium
Certainty of the certification schedule / Potentially very high / Potentially high
Necessary lead time for the test facility / Potentially low-high / Potentially medium
Legal issues
Legal certainty (pass/fail decision based on a non-reproducible test procedure) / High / High
Others

1These criteria are critical for vehicle manufacturers, who require confidence in the results of emission tests and the possibility to retrieve useful information for the design of vehicles and after-treatment systems. Low repeatability may allow covering a wider range of operating conditions and may thus increase the robustness of the test procedure.

2 The initial and running costs for manufacturers may depend on the definition of boundary conditions and furthermore on the possibility to test and approve vehicle families.

Table 2: Consolidated evaluation matrix

Random cycle / On-road emissions testing with PEMS
Effectiveness
Preventing detection andby-passing of emission tests by the engine control unit / Medium to very low: cycle detection depending on the degree of randomization of boundary conditions and the characteristics of the short trip data base; cycle detectionpossible, i.e., based on non-rotating back wheels; even with an all-wheel chassis dynamometer, activation of a test mode may be needed to be able to test vehicles (due to ACC, acceleration sensors etc.). / Very high: cycle detection and calibration virtually impossible as there is nothing like a 'cycle' in on-road driving;ECU data logging not necessary for PEMS tests; however, if ECU data loggers are used, the ECU may be able to detect PEMS emission tests
Preventing a too narrow calibration of vehicle technologies / Potentially medium: may cover a wide range of engine map and ambient conditions depending on the permitted boundary conditions; potentially limited effectiveness with regards to effects like driving curves, uphill-downhill, micro-transient driving, weather conditions; driving pattern may not be sufficiently severe for high-power vehicles; shorter test durations than on-road tests may limit robustness (this issue is critical for SCR urea dosing) / Potentially high:same as random cycle; in addition: long test durations allow a more robust assessment of vehicle emissions; theoretically, all driving and ambient conditions can be covered, practically there are limitations due to local climate, orographic conditions, and the enforced speed limits (i.e., 110 km/h in Sweden and 112 km/h in the UK); can better ensure sufficient severity for high-power vehicles; good coverage of micro-transient driving
Reproducibility and repeatability1 / High:very high repeatability of individual testsafter training; medium reproducibility of the same cycle results in different test facilities, as evidenced from previous Round Robin exercises; pass-fail reproducibility strongly depends on the definition of boundary conditions; pass-fail data evaluation based on integrated bag measurements is robust; remaining statistical uncertainty could be absorbed by the definition of pass-fail limits / Low:low repeatability of individual PEMS tests (assuming the accuracy standards of laboratory testing); very low reproducibility of on-road tests in different test facilities; pass-fail reproducibility strongly depends on the definition of boundary conditions; given the higher variability of parameters during on-road driving, pass-fail reproducibility can be expected to be lower than during laboratory testing – this may be to the advantage of the robustness of the test procedure;robust pass-fail data evaluation is feasible but has to be developed; experience from heavy-duty vehicles available; statistical uncertainty could be absorbed by the definition of pass-fail limits
Accuracy of the analytical equipment / Very high: identical to the current laboratory equipment used for type approval; limited variability in accuracy due to relatively constant ambient conditions and the absence of vibrations; limited inter-laboratory variability measurement accuracy / Very high: accuracy confirmed in the testing of heavy-duty engines; instrumentation performance identical with laboratory equipment under laboratory conditions; accuracy requirements laid down in regulations (e.g., UN ECE R49); larger uncertainty margins can be expected during variable on-vehicle conditions (due to variability in temperature, vibrations, altitude which can be absorbed by the definition of not-to-exceed limits;
Influence of measurement procedure/equipment on results / Medium: negligible with respect to the accuracy of the analytical equipment; engine cooling is usually different from that on the road; control systems may go into safety or fall-back mode, potentially influencing the emissions behavior of the vehicle / Medium: PEMS introduces additional weight (10 to 15% depending on the vehicle) and almost no changes of the vehicle’s aerodynamics and stability.
Use of the method for development purposes / Medium: both procedures valuable to develop robust emission control systems; emission values can be related to engine load and driving conditions; for development purposes, a certain target value setting will be necessary that is impossible for a random cycle (true when the vehicle will be calibrated for certain test conditions) / Medium: both procedures valuable to develop robust emission control systems; emission values can be related to engine load and driving conditions; however, for development purposes, a certain target value setting will be necessary that is impossible for PEMS (true when the vehicle will be calibrated for a certain test conditions)
Equal treatment of OEMs from different countries regarding, e.g., speed limits, orographic location / Potentially medium-high: generally unproblematic; depending on the definition of boundary conditions, availability of test facilities could be a limiting factor especially for technical services of member states; procedure based on the not-to-exceed principle - every OEM is likely to be tested under any condition falling within the agreed boundary conditions / Medium: strongly depending on the definition of boundary conditions; covering a wide range of ambient, driving, and orographic conditions might not be achievable through test at manufacturers’ sites all year around; alternatively, this criterion may be uncritical asall manufacturers perform winter, summer, and high-altitude tests; emissions verification with PEMS can even help solving technical issues
Coverage
Coverage of the range of driving pattern, and if necessary allows targeting specific driving conditions (e.g., urban driving) / Potentially high: strongly depending on the definition of boundary conditions; available short trips likely cover a wide range of driving conditions (in particular urban driving) but are limited to the WLTC data base; technical restrictions of test bench need to be taken into account; limited ability to reproduce the dynamic transients encountered in real-world conditions, especially for high-power vehicles; limitations of test benches (e.g., test of 4 wheel vehicles); see also “Preventing a too narrow calibration of vehicle technologies” / Potentially very high: depending on the definition of boundary conditions; theoretically, any driving pattern (including micro-transient driving behavior and urban driving) could be covered; practically, traffic conditions and enforced rules limit on-road testing; see also “Preventing a too narrow calibration of vehicle technologies”
Coverage of road profiles (slopes, etc.) / Potentially low: Uphill or downhill driving can theoretically be simulated by load variation under additional high costs; the current database would need to be extended by slope information; generally no possibility to vary road loads during emissions testing / Potentially high: a large range of road profiles can be covered; however, the geographical location of test facilities may limit the actual coverage of on-road tests (see equal treatment of manufacturers)
Coverage of ambient conditions (temp, altitude, humidity, background pollutants, intake air, etc.) / Potentially medium: covering the wide range of ambient conditions is to some extent possible but practically unfeasible due to high investment and running costs; limited capacities for vehicle conditioning and testing especially for national technical services / Potentially medium: the variability of several ambient conditions will likely be higher than in the laboratory; the full range of ambient conditions could potentially be covered; however actual ranges are restricted in practice by geographical location of test facility and annual seasons
Coverage of regulated pollutants / Very high: complete coverage of regulated gaseous and particulate pollutants, even if additional pollutants are included for future testing / High: coverage of regulated gaseous pollutants; PM/PN can currently no be measured, PM measurement equipment currently in the phase of pilot tests, PN measurements with PEMS may be available only by the end of the decade; omitting HC measurements in the first step of the RDE-LDV procedure can decrease costs and weight of test equipment and simplify the application of PEMS
Applicability
Applicability to a wide range of LDVs (e.g., categories, powertrain technologies) / High: applicable to most light-duty vehicles; comparatively short test durations may limit the effectiveness of tests for hybrid-electric vehicles; 4-wheel drives are difficult to cover; test capacities at type approval services potentially limited / High: applicable to most light-duty vehicles; tail-pipe attachments and available flow meter sizes could be problematic in some cases; unresolved issues on the testing of customers’ cars (for in-service conformity)
Neutrality towards engine and after-treatment technologies / High: generally high neutrality; however concerns over the ability to capture the degradation in SCR efficiency during low-speed/urban driving as well as during high engine loads / High: generally high neutrality; however, currently unsuitable to capture particle emissions
Possibility to execute tests during the whole year / Very high: similar to current laboratory testing / Potentially high: depending on the definition of boundary conditions; potentially limited by annual seasons or associated with additional costs
Capacity of test facilities / Uncertain: test facilities available; lower capacity than for standard laboratory testing; strongly depending on boundary conditions (e.g., test durations, temperature ranges, variable load); soak and test facilities would need to be upgraded significantly to accommodate a wide range of temperatures, air pressures, and 4-wheel drives, etc.;member states may not have access to fully climatized test facilities with all-wheel chassis dyno’s / Uncertain: test capacities not yet available and depending on the acquisition of PEMS equipment; after acquisition, probably better than for the random cycle;
Costs and benefits
Availability of know-how and dissemination thereof / High:Know-how is widely available for every emissions laboratory; drivers’ training may be required; implementation of random cycles potentially unproblematic;drivability of severe random cycles need to be further investigated / Medium-high: PEMS measurement know-how known for heavy-duty vehicles; most OEMs are already dealing with PEMS measurement devices but knowledge dissemination is still needed for light-duty vehicles; PEMS application likely to be unproblematic after an initial learning phase
Initial cost of implementing the procedure2 / Potentially medium: strongly depending on the definition of boundary conditions; if additional laboratories need to be constructed for covering (e.g., four wheel vehicles, a wide range of ambient conditions, variable load roads) cost might be substantial (costs of new test facilities >1 to 2 million Euro); family testing assumed; / Potentially medium: additional equipment will have to be procured at limited costs (0.2-0.3 million Euro per PEMS unit); depending on the definition of boundary conditions, additional test sites may need to be organized/established; cost for training/recruitingof personnel; depending on the range of pollutants tested potentially higher costs than the random cycle for manufacturers and technical services of member states
Running costs/resources of executing vehicle tests2 / Potentially medium: strongly depending on the boundary conditions; randomization of ambient and driving conditions may lead to high running costs (environmental chassis dynamometer, variable resistance chassis dynamometer, soaking facilities; see also cost statement of ACEA in Appendix B); high severity of cycles might require repetitive testing / Potentially medium: strongly depending on the definition of boundary conditions; potentially higher than for the random cycle in case of a narrow definition of boundary conditions; need to install/remove PEMS from vehicles
Degree to which methods for emissions characterization are available / Medium: integrated bag measurements available; suitable averaging methods need to be developed; more data analyses are necessary / Medium: suitable averaging methods need to be developed; more data analyses are necessary; experience from heavy-duty vehicles is available
Principle practicability / Potentially low-high: depending on the definition of system boundaries; see also, e.g., coverage of ambient conditions, safety, legal issues;regulations on family building critical / Potentially medium-high: depending on the definition of system boundaries; see also, e.g., coverage of ambient conditions, safety, legal issues; issues of training, dissemination of know-how (e.g., calibration of PEMS; installation/removal of PEMS from vehicles is necessary; regulations on family building critical
International attractiveness (contribution to the harmonization of emission tests and standards) / Medium:shows potential for the international harmonization of emissions legislation; however, short trips only based on the European data base of the WLTP is not appropriate for internationally harmonizing standards such as R83. / Potentially high:shows potential for the international harmonization of emissions legislation; PEMS already introduced with Euro VI legislation and used in other regions of the world; enables real-world emissions testing and could bring simplicity and effectiveness in emissions standards for a wide range of fuel and powertrain technologies; however, method and equipment is relatively new; Japan does not authorize the use of PEMS; OEMs partially disagree with this evaluation and suggest that PEMS is not appropriate for internationally harmonized emissions testing
Safety of measurements / Very high: identical to standard emissions testing in the laboratory / Medium: several issues may need to be discussed before the implementation of PEMS at large-scale such as proper fixture of PEMS equipment inside/outside vehicles, crash safety, transport of hazardous materials (FID hydrogen-helium mixture), health and safety issues for drivers and pedestrians; risk of CO exposure; general compliance with national road safety standards
Certainty of the certification schedule / Potentially very high: minor uncertainty may result depending on the amount of tests, test durations, and the possibility for family building / High: strongly depending on the definition of boundary conditions; annual seasons and weather conditions may be critical
Necessary lead time for the test facility / Potentially low-high: strongly depending on the definition of boundary conditions (e.g., temperature range) and the possibilities for family building / Potentially medium: PEMS installation in vehicles and calibration requires 2 hours or less for trained personnel; before testing, procurement of PEMS critical for OEMs
Legal issues
Legal certainty (pass/fail decision) / High: depending on the definition of boundary conditions, the randomness of the test procedure will lead to a certain statistical uncertainty that could be absorbed in the definition of a compliance factor; see also repeatability and reproducibility / High: depending on the definition of boundary conditions, the randomness of the test procedure will lead to a certain statistical uncertainty that could be absorbed in the definition of a compliance factor; see also repeatability and reproducibility
Others