MOD-033-2 — Steady-State and Dynamic System Model Validation
To establish consistent validation requirements to facilitate the collection of accurate data and building of planning models to analyze the reliability of the interconnected transmission system.
See Implementation Plan.
Requirements and Measures
R1. Each Planning Coordinator shall implement a documented data validation process that includes the following attributes: [Violation Risk Factor: Medium] [Time Horizon: Long-term Planning]
1.1. Comparison of the performance of the Planning Coordinator’s portion of the existing system in a planning power flow model to actual system behavior, represented by a state estimator case or other Real-time data sources, at least once every 24 calendar months through simulation;
1.2. Comparison of the performance of the Planning Coordinator’s portion of the existing system in a planning dynamic model to actual system response, through simulation of a dynamic local event, at least once every 24 calendar months (use a dynamic local event that occurs within 24 calendar months of the last dynamic local event used in comparison, and complete each comparison within 24 calendar months of the dynamic local event). If no dynamic local event occurs within the 24 calendar months, use the next dynamic local event that occurs;
1.3. Guidelines the Planning Coordinator will use to determine unacceptable differences in performance under Part 1.1 or 1.2; and
1.4. Guidelines to resolve the unacceptable differences in performance identified under Part 1.3.
M1. Each Planning Coordinator shall provide evidence that it has a documented validation process according to Requirement R1 as well as evidence that demonstrates the implementation of the required components of the process.
R2. Each Reliability Coordinator and Transmission Operator shall provide actual system behavior data (or a written response that it does not have the requested data) to any Planning Coordinator performing validation under Requirement R1 within 30 calendar days of a written request, such as, but not limited to, state estimator case or other Real-time data (including disturbance data recordings) necessary for actual system response validation. [Violation Risk Factor: Lower] [Time Horizon: Long-term Planning]
M2. Each Reliability Coordinator and Transmission Operator shall provide evidence, such as email notices or postal receipts showing recipient and date that it has distributed the requested data or written response that it does not have the data, to any Planning Coordinator performing validation under Requirement R1 within 30 days of a written request in accordance with Requirement R2; or a statement by the Reliability Coordinator or Transmission Operator that it has not received notification regarding data necessary for validation by any Planning Coordinator.
Compliance Monitoring Process
1.1. Compliance Enforcement Authority
“Compliance Enforcement Authority” means NERC or the Regional Entity in their respective roles of monitoring and enforcing compliance with the NERC Reliability Standards.
1.2. Evidence Retention
The following evidence retention periods identify the period of time an entity is required to retain specific evidence to demonstrate compliance. For instances where the evidence retention period specified below is shorter than the time since the last audit, the Compliance Enforcement Authority may ask an entity to provide other evidence to show that it was compliant for the full time period since the last audit.
The applicable entity shall keep data or evidence to show compliance with Requirements R1 through R2, and Measures M1 through M2, since the last audit, unless directed by its Compliance Enforcement Authority to retain specific evidence for a longer period of time as part of an investigation.
If an applicable entity is found non-compliant, it shall keep information related to the non-compliance until mitigation is complete and approved, or for the time specified above, whichever is longer.
The Compliance Enforcement Authority shall keep the last audit records and all requested and submitted subsequent audit records.
1.3. Compliance Monitoring and Assessment Processes:
Refer to Section 3.0 of Appendix 4C of the NERC Rules of Procedure for a list of compliance monitoring and assessment processes.
1.4. Additional Compliance Information
Guidelines and Technical Basis
The requirement focuses on the results-based outcome of developing a process for and performing a validation, but does not prescribe a specific method or procedure for the validation outside of the attributes specified in the requirement. For further information on suggested validation procedures, see “Procedures for Validation of Powerflow and Dynamics Cases” produced by the NERC Model Working Group.
The specific process is left to the judgment of the Planning Coordinator, but the Planning Coordinator is required to develop and include in its process guidelines for evaluating discrepancies between actual system behavior or response and expected system performance for determining whether the discrepancies are unacceptable.
For the validation in part 1.1, the state estimator case or other Real-time data should be taken as close to system peak as possible. However, other snapshots of the system could be used if deemed to be more appropriate by the Planning Coordinator. While the requirement specifies “once every 24 calendar months,” entities are encouraged to perform the comparison on a more frequent basis.
In performing the comparison required in part 1.1, the Planning Coordinator may consider, among other criteria:
The validation in part 1.1 would include consideration of the load distribution and load power factors (as applicable) used in the power flow models. The validation may be made using metered load data if state estimator cases are not available. The comparison of system load distribution and load power factors shall be made on an aggregate company or power flow zone level at a minimum but may also be made on a bus by bus, load pocket (e.g., within a Balancing Authority), or smaller area basis as deemed appropriate by the Planning Coordinator.
The scope of dynamics model validation is intended to be limited, for purposes of part 1.2, to the Planning Coordinator’s planning area, and the intended emphasis under the requirement is on local events or local phenomena, not the whole Interconnection.
The validation required in part 1.2 may include simulations that are to be compared with actual system data and may include comparisons of:
- Voltage oscillations at major buses
- System frequency (for events with frequency excursions)
- Real and reactive power oscillations on generating units and major inter-area ties
Determining when a dynamic local event might occur may be unpredictable, and because of the analytic complexities involved in simulation, the time parameters in part 1.2 specify that the comparison period of “at least once every 24 calendar months” is intended to both provide for at least 24 months between dynamic local events used in the comparisons and that comparisons must be completed within 24 months of the date of the dynamic local event used. This clarification ensures that PCs will not face a timing scenario that makes it impossible to comply. If the time referred to the completion time of the comparison, it would be possible for an event to occur in month 23 since the last comparison, leaving only one month to complete the comparison. With the 30 day timeframe in Requirement R2 for TOPs or RCs to provide actual system behavior data (if necessary in the comparison), it would potentially be impossible to complete the comparison within the 24 month timeframe.
In contrast, the requirement language clarifies that the time frame between dynamic local events used in the comparisons should be within 24 months of each other (or, as specified at the end of part 1.2, in the event more than 24 months passes before the next dynamic local event, the comparison should use the next dynamic local event that occurs). Each comparison must be completed within 24 months of the dynamic local event used. In this manner, the potential problem with a “month 23” dynamic local event described above is resolved. For example, if a PC uses for comparison a dynamic local event occurring on day 1 of month 1, the PC has 24 calendar months from that dynamic local event’s occurrence to complete the comparison. If the next dynamic event the PC chooses for comparison occurs in month 23, the PC has 24 months from that dynamic local event’s occurrence to complete the comparison.
Part 1.3 requires the PC to include guidelines in its documented validation process for determining when discrepancies in the comparison of simulation results with actual system results are unacceptable. The PC may develop the guidelines required by parts 1.3 and 1.4 itself, reference other established guidelines, or both. For the power flow comparison, as an example, this could include a guideline the Planning Coordinator will use that flows on 500 kV lines should be within 10% or 100 MW, whichever is larger. It could be different percentages or MW amounts for different voltage levels. Or, as another example, the guideline for voltage comparisons could be that it must be within 1%. But the guidelines the PC includes within its documented validation process should be meaningful for the Planning Coordinator’s system. Guidelines for the dynamic event comparison may be less precise. Regardless, the comparison should indicate that the conclusions drawn from the two results should be consistent. For example, the guideline could state that the simulation result will be plotted on the same graph as the actual system response. Then the two plots could be given a visual inspection to see if they look similar or not. Or a guideline could be defined such that the rise time of the transient response in the simulation should be within 20% of the rise time of the actual system response. As for the power flow guidelines, the dynamic comparison criteria should be meaningful for the Planning Coordinator’s system.
The guidelines the PC includes in its documented validation process to resolve differences in Part 1.4 could include direct coordination with the data owner, and, if necessary, through the provisions of MOD-032-1, Requirement R3 (i.e., the validation performed under this requirement could identify technical concerns with the data). In other words, while this standard is focused on validation, results of the validation may identify data provided under the modeling data standard that needs to be corrected. If a model with estimated data or a generic model is used for a generator, and the model response does not match the actual response, then the estimated data should be corrected or a more detailed model should be requested from the data provider.
While the validation is focused on the Planning Coordinator’s planning area, the model for the validation should be one that contains a wider area of the Interconnection than the Planning Coordinator’s area. If the simulations can be made to match the actual system responses by reasonable changes to the data in the Planning Coordinator’s area, then the Planning Coordinator should make those changes in coordination with the data provider. However, for some disturbances, the data in the Planning Coordinator’s area may not be what is causing the simulations to not match actual responses. These situations should be reported to the Electric Reliability Organization (ERO). The guidelines the Planning Coordinator includes under Part 1.4 could cover these situations.
During development of this standard, text boxes were embedded within the standard to explain the rationale for various parts of the standard. Upon BOT approval, the text from the rationale text boxes was moved to this section.
Rationale for R1:
In FERC Order No. 693, paragraph 1210, the Commission directed inclusion of “a requirement that the models be validated against actual system responses.” Furthermore, the Commission directs in paragraph 1211, “that actual system events be simulated and if the model output is not within the accuracy required, the model shall be modified to achieve the necessary accuracy.” Paragraph 1220 similarly directs validation against actual system responses relative to dynamics system models. In FERC Order 890, paragraph 290, the Commission states that “the models should be updated and benchmarked to actual events.” Requirement R1 addresses these directives.
Requirement R1 requires the Planning Coordinator to implement a documented data validation process to validate data in the Planning Coordinator’s portion of the existing system in the steady-state and dynamic models to compare performance against expected behavior or response, which is consistent with the Commission directives. The validation of the full Interconnection-wide cases is left up to the Electric Reliability Organization (ERO) or its designees, and is not addressed by this standard. The following items were chosen for the validation requirement:
A. Comparison of performance of the existing system in a planning power flow model to actual system behavior; and
B. Comparison of the performance of the existing system in a planning dynamics model to actual system response.
Implementation of these validations will result in more accurate power flow and dynamic models. This, in turn, should result in better correlation between system flows and voltages seen in power flow studies and the actual values seen by system operators during outage conditions. Similar improvements should be expected for dynamics studies, such that the results will more closely match the actual responses of the power system to disturbances.
Validation of model data is a good utility practice, but it does not easily lend itself to Reliability Standards requirement language. Furthermore, it is challenging to determine specifications for thresholds of disturbances that should be validated and how they are determined. Therefore, this requirement focuses on the Planning Coordinator performing validation pursuant to its process, which must include the attributes listed in parts 1.1 through 1.4, without specifying the details of “how” it must validate, which is necessarily dependent upon facts and circumstances. Other validations are best left to guidance rather than standard requirements.
Rationale for R2:
The Planning Coordinator will need actual system behavior data in order to perform the validations required in R1. The Reliability Coordinator or Transmission Operator may have this data. Requirement R2 requires the Reliability Coordinator and Transmission Operator to supply actual system data, if it has the data, to any requesting Planning Coordinator for purposes of model validation under Requirement R1.
This could also include information the Reliability Coordinator or Transmission Operator has at a field site. For example, if a PMU or DFR is at a generator site and it is recording the disturbance, the Reliability Coordinator or Transmission Operator would typically have that data.