IEC 61850 Network Architectures

IEC 61850 Network Architectures
July, 2010 Maciej Goraj maciejgoraj@ruggedcom.com
Agenda
1. 2. 3. 4. Requirements for substation communications network Types of protocols and traffic patterns in IEC 61850 standard Typical network architectures Problem of Multicast and Physical vs. Logical separation of Process Bus and Station Bus
Copyright RuggedCom Inc
Requirements for Substation Hardened Networking Equipment
Substation Environment
Generation Plant HV/MV Substation Wind Farm
EMI & Environmental Phenomena Typical of Substation Environments Electric and Magnetic Fields Electrostatic Discharge Conducted High Frequency Electrical Transients High Energy Power Surges Ground Potential Rise during ground faults Climactic Variation: Temperature & Humidity Seismic / Vibration Pollution: Dust, Metallic Particles, Corrosive Chemical Particles, Condensation, Solar Radiation, Salt, Bird Guano, etc.
4
EMI Phenomenon
Continuous
Phenomena Radiated RFI Induced RFI Power freq. Magnetic Field Slow Voltage Variations Harmonics, Interharmonics Ripple on d.c. power supply Power Frequency Voltage
Transient Phenomena (High Occurrence)

Electrostatic Discharge Voltage Dips Lightning Ground Potential Rise (GPR) HV Switching by Isolators Reactive Load Switching
Transient Phenomena (Low Occurrence)

Power Frequency Variation Power System Faults Short Duration Power Freq. Magnetic Fields
DevicesinsubstationsmustdealwithacombinationofEMI phenomenawhicharebothcontinuousandtransient.
Requirements for IEDs According to IEC 61850-3

Must operate properly under the influence of a variety of EMI phenomena commonly found in the substation IEC 61850-3 specifies a variety of type withstands tests designed to simulate EMI phenomena such as:
Inductive load switching Lightening strikes Electrostatic discharges from human contact Radio frequency interference due to personnel using portable radio handsets Ground potential rise resulting from high current fault conditions within the substation
Ethernetswitches,routers,deviceservers,mediaconverters shallmeetEMIrequirementstothesameextentasIEDs
Requirements for IEDs According to IEEE 1613

Standard for Environmental and Testing Requirements for Communications Networking Devices in Electric Power Substations It goes one step further by defining Class 2 operation which requires that, during the application of the type tests, the switch must experiment:
No communications errors No communications delays No communication interruptions
RuggedizedEthernetswitchshallbeseenasyetanotherIED
Fiber Optics Overview

Future proof Theoretically infinite bandwidth Up to 100 km distance possible Immune to EMI Supported by all current IEDs Lightweight Costs continue to drop Multi-mode for short distances Single-mode for long distances
Common Fiber Optic Connectors

ST Stick and Twist and SC Stick and Click historically popular LC becoming prevalent especially for Gigabit because small form factor (SFF) allows greater port density GBIC are pluggable SC transceivers using SC connectors SFP are Small Form Factor Pluggable ST SC
LC
MTRJ
Types of Protocols and Traffic Patterns in IEC 61850 Standard
10
Number of Devices in Electrical Substations

Typically 2040IEDs per substation Large substations mayhave 80120IEDs Power Plants,Oil&Gasinstallation 150 500IEDs Large installations with LVIEDs 10001500IEDs Large Wind Farms canhave +200IEDs LargeSolargenerationsitescanhave600 1500IEDs
11
IEC 61850 Ed. I Profiles and Protocols Stack
Will be moved to an Annex in Edition II of IEC 61850

12
IEC 61850 Ed. II Profiles and Protocols Stack

Sampled Values (Multicast) Generic Object Oriented Substation Event Core ACSI Services
Time Sync
SV
(Type 4)
GOOSE
(Type 1, 1A)
TimeSync (SNTP)
(Type 6)
MMS Protocol Suite

(Type 2, 3, 5)
UDP/IP
TCP/IP T-Profile
SMV
GOOSE HSR (O)
IP (O)
802.1Q
802.1Q
802.1Q (O)
802.1Q (O)
ISO/IEC 8802-3 ISO/IEC 8802-3 Ethertype
13
Types of traffic
Client-server MMS services:
Polling Reporting (Unsolicited and/or periodic) Asynchronous and unsolicited Less often synchronous (for heartbeat and for analogue values) Synchronous unsolicited transmission
GOOSE
Sampled Values (Process Bus)
IEC61850networkisacombinationofRawEthernet, MMS/TCP,SNTP,IEEE1588,TFTP,FTP,RSTP,SNMP,and otherEthernetbasedprotocols

14
Non-IP traffic in IEC 61850 - GOOSE

Encapsulated directly in Ethernet layer High priority, critical, asynchronous and unsolicited Less often synchronous (for heartbeat and for analogue values) MAC Multicast, uses VLAN for priority and traffic segregation Frame size approx. 92 250 bytes Periodic heartbeat messages of 1-60 seconds interval if no events occur 99% of time just the heartbeat message In case of event an avalanche can occur as many IEDs detect state changes Typically used for fast transmission of digital events Less often for transmission of analogue data, e.g. sent every 250ms
15
Retransmission Scheme in GOOSE

GOOSE is connectionless No confirmation from receivers Retransmission to increase the probablity of sucessful reception A burst of 5-6 messages sent in case of event (critical information) Example of implementation:
1st message: on event 2nd message: 4ms after event 3rd message: 16ms after event 4th message: 80ms after event 5th message: 500ms after event
Event occurs, GOOSE with incremented stNum sent immediately Heartbeat GOOSEs
Time
Burst of GOOSEs sent on event occurence

16
GOOSE and Network Performance

GOOSE messages shall be priority tagged
Configuration needed in IEDs and in Ethernet switches GOOSE frames with the priority tag in VLAN field configured are placed in the front of the store and forward queue Frames already being sent are not interrupted
Delay of frames introduced by network is almost zero Worst case of total network delay is
100 s at 100MBps links speeds 10 s at 1Gbps
17
GOOSE and Network Performance

IEC 61850-5 Type 1A Trip total transfer time defined at 4ms Transfer time = Application to Application and includes:
GOOSE encoding at sender + network delays + GOOSE decoding at receiver
It is difficult to measure as defined in IEC 61850-5 Because the timestamp is added in IED after the internal function execution time (one scan period) Typical measured GOOSE total transfer time including function execution time in IED is in the range of 6-12ms
18
Non-IP traffic in IEC 61850 Sampled Values

Encapsulated directly in Ethernet layer High priority, critical, synchronous and unsolicited MAC Multicast, uses VLAN for priority and traffic segregation Currently dedicated wiring (IRIG-B or 1PPS) used for time synch of devices, future will be IEEE 1588 A Merging Unit (sensor) sends 80 or 256 samples/power cycle. At 50Hz it is 4000 and 12800 samples per second respectively. A sample is a set of 8 analog values, 4 voltages + 4 currents @80 samples 4000 packets/sec A single Merging Unit uses approx. 4.4 5.2Mbit/s of bandwidth at 80 Smp The bandwidth used depends of sampling rate and if Data Set is according to IEC61850-9-2LE implementation or other Data Set 1Gbit Ethernet highly recommended for Process Bus in switched Ethernet
19
IP based traffic in IEC 61850

Client-Server services
MMS protocol over TCP, port 102 measurements, events, status indications 100-500ms delay accepted Traffic generated by a single IED rarely exceeds 10kbps Reports save bandwidth. Digitals via Buffered, Analogs via Unbuffred.
Time synchronization
SNTP or IEEE 1588 For redundancy mutiple time masters used
File transfer
MMS over TCP, FTP, TFTP, other protocols e.g. Modbus/TCP Typically Oscillography, sequence of events, data logs. Ocassionally configuration, settings, firmware upgrades, etc. File size typically 4 200 kbytes,
20
Typical Network Architectures
21
Star Topology
Not protected against single point of failure Simplicity
HMI
Gateway
Protection and Control IEDs

22
Redundant Star Topology

Blue LAN A Red LAN B
The entire network is duplicated Configuration and application complexity, cost issues Each device has 2 IP addresses, 2 application instances PRP will be the alternative HMI Gateway

23
Single Ring Topology

Dashed Redundant Connections Blue Electrical 100Mpbs Red Fiber Optic 100Mbps
IEDs can be dual homed and connected via redundant links Redundancy with RSTP PRP or HSR will be the alternative HMI Gateway
Fiber Optic Ethernet Ring 100/1000 Mbps

24
Multiple Rings Topology

Limited number of switches in each ring Minimize recovery time Division criteria by voltage levels or by several bays Redundancy with RSTP HMI Gateway
Primary Ring
Secondary Ring 1
Secondary Ring n

25
Ring of IEDs
Dashed Lines Redundant LAN Connections
IEDs with Embedded Switch functionality Multiple rings may be needed Redundancy with RSTP HSR will be the alternative Gateway
HMI
Fiber Optic Ethernet Ring 100 Mbps

26
Problem of Multicast and Physical vs. Logical separation of Process Bus and Station Bus
27
Problem of Multicast
Multicast is one-to-many communication scheme Multicast MAC traffic is by default propagated through the whole LAN Consumes link bandwidth and increases latency at switches Introduces significant overhead at receiving IEDs if multicast addresses not allocated properly
28
Impact of Multicast
Red MU (Merging Unit) multicasts Sampled Values to small group of IEDs It is dictated by the protection application In a large substation there can be dozens of IEDs sending multicast GOOSE and dozens of Merging Units sending multicast Sampled Values
NTP P IED IED
Primary Ring
Secondary Rings
P C MU
P C
P C MU
P C MU
P C
MU
29
Impact of Multicast
All nodes get the traffic red area Repeat for every IED/MU in network Critical messages delayed or maybe dropped Steady state traffic load can exceed 100Mbps for many MUs Excessive MU traffic can cause IEDs and PCs can mis-operate or crash
NTP P IED IED
Primary Ring
Secondary Rings
P C MU
P C
P C MU
P C MU
P C
MU
Multicastmustbefiltered
30
Multicast Addresses and Traffic Management

Efficient layer 2 multicast application
Proper allocation of multicast addresses Filtering of multicast traffic
Allocation of multicast addresses

improves processing times at receiving devices by discarding unwanted multicast traffic at hardware level required for multicast filtering
Multicast filtering
saves bandwidth and decreases latency at network switches by limiting the traffic only to restricted areas of the network Multicast filtering solves the primary problem of filtering unwanted GOOSE and SV traffic Use VLAN or MAC address filtering ? Static or dynamic filtering methods ?
31
Where we are today ?

In todays substations often no multicast management used at all Lack of knowledge at integrators and utilities Many users just tend to minimize configuration efforts and rely on default settings Until now the dominant method for restricting multicast traffic was the use of VLANs Static configuration: manual process for all IEDs and all network devices
32
Example of Misconfiguration
Case Study
50 IEDs in the same network all sending GOOSE No multicast filtering used Wrong! All IEDs send multicast with the same destination multicast MAC address Wrong! In case of event there is an avalanche of GOOSEs in the network and approx 20ms additional processing delay observed at the receiver Improper functioning!
Implementation internals of an IED

Network controller at IEDs has hash table that maps all possible multicast MACs to a small group of addresses Hash table permits discard unwanted multicast MACs at hardware level If all IEDs send with the same multicast destination MAC then at receiving IED these are mapped to the same hash and need to be discarded by software In some IED implementations decoding of GOOSE message takes up to 1.5ms Software decoding of 20 unwanted GOOSE messages can take up to 30ms!
33
VLAN (IEEE 802.1Q)

Virtual LAN: an independent Ethernet network that shares cabling infrastructure with other networks Each VLAN has a separate broadcast domain VLANs permit:
Priority tagging Logical separation of the network into various domains
6 bytes 6 bytes 2 bytes Variable
Dest.
Src.
Length / Type
Data
Standard Frame
Variable
6 bytes
6 bytes
2 bytes 2 bytes
2 bytes
Dest.
Src.
TPID
TCI
Length / Type
Data
3 bits
1 bit
12 bits
Priority CFI
VID
Tagged Frame
34
Use of VLANs
VLAN is suitable mechanism for isolation of unrelated traffic, eg. surveillance video from SCADA traffic VLANs configuration can be:
Static Dynamic (GVRP)
Today static configuration is a manual process Static configuration can be semi-automatic with future enhanced configuration tools Can use MAC address filtering instead of VLAN VLANs for priority tagging in order to increasing performance
35
Traffic Segregation with VLANs
Traffic separated with VLANs:

Substation LAN management SCADA/Engineering Access GOOSE Messages Process Bus (Sampled Values) Synchrophasors Protection A vs. Protection B
Differenttrafficflowsinasubstationnetworkmerit segregatingintoseparateVLANs
36
GMRP/MMRP for Dynamic Multicast Filtering

Generic Multicast Registration Protocol Publisher / subscriber model like IGMP Multicast filtered by default must subscribe to get it Adapts dynamically to any network topology and accommodates any application of 9-2 or GOOSE edge only pruning results in no traffic delay after topology change Allows process and station bus to co-exist on same physical network
P
Primary Ring
Secondary Rings
Switches prune the traffic automatically. Either optimally or edge ony
P C MU
P C
SV consumer sends a subscribe message to network periodically
MU
SV producer simply multicasts no change
FirstIEC61850110kVsubstationwithIEEE1588v2and dynamicGMRPmulticastfilteringcommissionedin2010
37
Problem of Time Synchronization and Data Sharing

Process Bus requires that Sampled Values coming from different sensors and received by an IED have to be synchronized Synchronization islands are possible, each island spans a protection zone Problem of Line Differential protection with one line end using Process Bus and the other line end using conventional wiring
38
Process Bus and Station Bus Separation

Process Bus and Station Bus are logically different Multicast traffic from Merging Units flooding the network A single Merging Unit consumes approx 5Mbps of bandwidth The problem of busbar protection based on Process Bus
In a topology with 60 feeders a process bus based busbar protection would have multicast traffic of > 400Mbps!
39
Physically or Logically Separate Networks?

Physically separate LANs are more costly network switches are duplicated Physically separate LANs are perceived as more secure Logically separate LANs are more flexible as Merging Units can be accessed from SCADA (remote maintenance, management, etc.) Logically separate LANs require network engineering or more sophisticated dynamic methods (GMRP, GVRP, etc.) Station Bus could also be connected to Process Bus via router
40
Questions?
41

IEC 61850 Network Architectures

Cargado por

Información del documento

Derechos de autor

Formatos disponibles

Compartir este documento

Compartir o incrustar documentos

Opciones para compartir

¿Le pareció útil este documento?

¿Este contenido es inapropiado?

Copyright:

Formatos disponibles

IEC 61850 Network Architectures

Cargado por

Copyright:

Formatos disponibles

IEC 61850 Network Architectures

July, 2010 Maciej Goraj maciejgoraj@ruggedcom.com

Copyright RuggedCom Inc

Requirements for Substation Hardened Networking Equipment

Copyright RuggedCom Inc

Transient Phenomena (High Occurrence)

Transient Phenomena (Low Occurrence)

Requirements for IEDs According to IEC 61850-3

Requirements for IEDs According to IEEE 1613

Fiber Optics Overview

Copyright RuggedCom Inc

Common Fiber Optic Connectors

Copyright RuggedCom Inc

Types of Protocols and Traffic Patterns in IEC 61850 Standard

Number of Devices in Electrical Substations

Copyright RuggedCom Inc

IEC 61850 Ed. I Profiles and Protocols Stack

Will be moved to an Annex in Edition II of IEC 61850

IEC 61850 Ed. II Profiles and Protocols Stack

MMS Protocol Suite

GOOSE HSR (O)

ISO/IEC 8802-3 ISO/IEC 8802-3 Ethertype

Copyright RuggedCom Inc

Sampled Values (Process Bus)

IEC61850networkisacombinationofRawEthernet, MMS/TCP,SNTP,IEEE1588,TFTP,FTP,RSTP,SNMP,and otherEthernetbasedprotocols

Non-IP traffic in IEC 61850 - GOOSE

Copyright RuggedCom Inc

Retransmission Scheme in GOOSE

Burst of GOOSEs sent on event occurence

GOOSE and Network Performance

Copyright RuggedCom Inc

GOOSE and Network Performance

Copyright RuggedCom Inc

Non-IP traffic in IEC 61850 Sampled Values

IP based traffic in IEC 61850

Copyright RuggedCom Inc

Typical Network Architectures

Protection and Control IEDs

Redundant Star Topology

Protection and Control IEDs

Single Ring Topology

Fiber Optic Ethernet Ring 100/1000 Mbps

Protection and Control IEDs

Multiple Rings Topology

Protection and Control IEDs

Fiber Optic Ethernet Ring 100 Mbps

Protection and Control IEDs

Copyright RuggedCom Inc

NTP P IED IED

Copyright RuggedCom Inc

Multicast Addresses and Traffic Management

Allocation of multicast addresses

Copyright RuggedCom Inc

Where we are today ?

Copyright RuggedCom Inc

Implementation internals of an IED

Copyright RuggedCom Inc

VLAN (IEEE 802.1Q)

Copyright RuggedCom Inc

Copyright RuggedCom Inc

Traffic Segregation with VLANs

Traffic separated with VLANs:

GMRP/MMRP for Dynamic Multicast Filtering

Switches prune the traffic automatically. Either optimally or edge ony