CODESYSControlV3 MultiCore
CODESYSControlV3 MultiCore
CODESYSControlV3 MultiCore
Version 1.0
Content
1 Introduction 4
2 MultiCore aspects 4
2.1 Performance 4
2.1.1 Task execution on different CPU cores 4
2.1.2 Avoid global synchronizations 4
2.1.3 Decouple tasks 4
2.1.4 Task binding 4
2.2 Data Consistency 5
2.2.1 32bit systems 5
2.2.2 64bit systems 5
3 Licensing 5
4 Operating system and processor specific aspects 6
4.1 General 6
4.2 OS specific data consistency 6
4.3 Windows 7
4.4 Linux 7
4.5 VxWorks 7
4.6 Windows CE 7
4.6.1 Windows CE 7 7
4.6.2 Windows Embedded Compact 2013 (“CE 8”) 7
4.6.3 Measuring the Core Load on Windows CE 7 and 8 7
5 Coding Guidlines 8
5.1 Lock free programming 8
5.1.1 SysCpuTestAndSetBit 8
5.1.2 SysCpuAtomicAdd - SysCpuAtomicAdd64 8
5.1.3 SysCpuReadValueAtomic/SysCpuWriteValueAtomic 8
5.1.4 SysCpuAtomicReadInt64/SysCpuAtomicWriteInt64 - SysCpuAtomicReadReal64/SysCpuAtomicWriteReal64 8
5.1.5 SysCpuCompareAndSwap 8
5.2 Data access synchronization 8
5.2.1 SysMutex / SysSem 8
5.2.2 SysSemCount 8
5.2.3 SysReadWriteLock 9
5.2.4 Interrupt handlers 9
5.2.4.1 VxWorks 9
5.3 Asynchronous Operations 9
5.3.1 SysMsgQ 9
5.3.2 CmpAsyncMgr 9
6 Diagnostics 9
6.1 CPU and PLC load system traces 9
6.2 IEC-Tasks system trace 10
6.3 PlcShell commands 10
7 IEC programming aspects 12
7.1 IEC-task core binding 12
7.2 IEC data consistency 12
7.2.1 BIT datatypes 12
7.2.2 32bit datatypes 12
7.2.3 64bit datatypes (integer and real) 12
7.2.4 Strings 12
7.2.5 UserDefined datatypes 12
7.3 IEC data task cycle consistency 13
7.3.1 Memory reordering problems 13
8 Features not released with V3.5.13.0 on MultiCore targets 13
3S-Smart Software Solutions GmbH · Memminger Str. 151 · 87439 Kempten · Germany
Tel.: +49-831-54031-0 · Fax: +49-831-54031-50 · info@codesys.com · www.codesys.com Page 2 of 14
9 Glossary 13
3S-Smart Software Solutions GmbH · Memminger Str. 151 · 87439 Kempten · Germany
Tel.: +49-831-54031-0 · Fax: +49-831-54031-50 · info@codesys.com · www.codesys.com Page 3 of 14
CODESYS Inspiring Automation Solutions MultiCore Support
1 Introduction
Controller manufacturers are equipping more an more new hardware with multiple processor cores, known as multi-core CPUs. The subject
of this document is the multi-core support for controllers with the CODESYS runtime system.
For multicore targets we assume an SMP system. AMP systems are not assumed here.
These are the main goals for a software running on a multicore system:
1. Improve Performance
2. Keep Data Consistency
2 MultiCore aspects
2.1 Performance
To improve the performance on multicore systems, there are lot of concepts. Following we describe some main principles.
The way how tasks are executed on several cores must be recognized here too. This is described in the chapter Task binding.
1. Message queues
2. Asynchronous manager
Here the Operating system decides always on which core a task is running at a time. This decision depends typically on the load on
the different CPU cores (load balancing), but can be based on different additional characteristics. And so the CPU core can be
changed during the runtime of a task or from task cycle to task cycle.
If deterministic tasks (realtime tasks) are running as freefloating, the execution time can vary on which CPU core the task is currently
running. So it is more predictably, if the task runs only on one CPU core. This is described following.
Here a task can be fix bound to a CPU core and so the task only is running in this dedicated CPU core. This could improve the
predictability of the execution time of a task, but but this is possibly not true on all targets! But you take away the possibility of the
operating system to do a load balancing and so it mus be measured, if a fix binding of a task has a really benefit.
To manage the CPU core configuration of all tasks, typically we assign a task to a so called “task group”. The CPU core binding of a
3S-Smart Software Solutions GmbH · Memminger Str. 151 · 87439 Kempten · Germany
Tel.: +49-831-54031-0 · Fax: +49-831-54031-50 · info@codesys.com · www.codesys.com Page 4 of 14
CODESYS Inspiring Automation Solutions MultiCore Support
complete task group can be configured or you can bind a single task to one or several cores.
In the runtime system, we assign realtime tasks (scheduler tick and IEC-tasks) fix to one core and all other tasks are running free floating
(unbound).
“System”:
“IEC-Tasks”:
Default taskgroup for all IEC-tasks. They are bound by default to one core (core=1)!
“Communication”:
For all communication tasks we create this task group by default. All tasks in this task group are executed be default free floating.
“Visualization”:
For all visualization tasks in the runtime system. All tasks in this task group are executed be default free floating.
A generic interface in the runtime system can be used to read/write 2, 4 and 8 byte data values atomic, if your code should run on a 32bit
and a 64bit target:
The data consistency in the IEC-program is described in the chapter IEC data consistency.
If “core” or “multi-core” is not mentioned explicitly in the following subchapters, then this is a general issue for multitasking on UP systems
too.
Access to 8 byte integer and real datatypes (RTS_UI64/RTS_I64 and RTS_LREAL64) are not atomic! So we recommend using always the
atomic functions SysCpuReadValueAtomic/SysCpuWriteValueAtomic or for a single 64bit access the following functions in the runtime
system to achive data consistency:
On which platforms these atomic functions are implemented is described in chapter Operating system and processor specific aspects.
3 Licensing
3S-Smart Software Solutions GmbH · Memminger Str. 151 · 87439 Kempten · Germany
Tel.: +49-831-54031-0 · Fax: +49-831-54031-50 · info@codesys.com · www.codesys.com Page 5 of 14
CODESYS Inspiring Automation Solutions MultiCore Support
The number of CPU cores that are supported for the IEC Tasks are licensed via a device family license in the 3S.dat file!
If the number of licensed cores are less than the physical available CPU cores, the following aspects must be known:
1. You cannot use all CPU cores for IEC task groups
3. You have to specify the really used cores in the following setting:
[SysCpuMultiCore]
IecCoreSet={0..MAXCOREID}
Example for a 4 CPU core target but only 2 CPU cores are licensed:
[SysCpuMultiCore]
IecCoreSet={0,2}
3S-Smart Software Solutions GmbH · Memminger Str. 151 · 87439 Kempten · Germany
Tel.: +49-831-54031-0 · Fax: +49-831-54031-50 · info@codesys.com · www.codesys.com Page 6 of 14
CODESYS Inspiring Automation Solutions MultiCore Support
4.3 Windows
Windows means here the product CODESYSControlWin, _not_ CODESYSControlRTE! CODESYSControlWin can run on PCs with
multiple CPU cores, with or without hyperthreading.
x86/x64: all access to 64bit values are consistent and are handled atomic.
4.4 Linux
On a 32bit Linux, we have the following restrictions on handling 64bit datatypes:
x86/x64/ARM/Cortex: all access to 64bit values are consistent and are handled atomic. gcc >= 4.4 is necessary.
In the IEC application you have to use the AtomicReadXXX or AtomicWriteXXX to handle 64bit values atomic (see chapter IEC data
consistency)!
4.5 VxWorks
On 32bit VxWorks we have the following restrictions on handling 64bit datatypes:
x86/x64: all access to 64bit value are consistent and are handled atomic.
In the IEC application you have to use the AtomicReadXXX or AtomicWriteXXX to handle 64bit values atomic (see chapter IEC data
consistency)!
4.6 Windows CE
4.6.1 Windows CE 7
MultiCore support is available under Windows CE 7 ARM and x86. But Windows CE 7 uses the ARM32 instruction set without support of
atomic instructions for 64 bit access! So on ARM platforms, atomic handling of 64bit datatypes for data consistency is not supported with an
atomic operator!
3S-Smart Software Solutions GmbH · Memminger Str. 151 · 87439 Kempten · Germany
Tel.: +49-831-54031-0 · Fax: +49-831-54031-50 · info@codesys.com · www.codesys.com Page 7 of 14
CODESYS Inspiring Automation Solutions MultiCore Support
5 Coding Guidlines
5.1 Lock free programming
The main goal for multicore application should be a lock free programming to improve performance! For lock free programming, you can
use so called atomic functions of the runtime system. The following atomic functions are available.
5.1.1 SysCpuTestAndSetBit
With this function, you can set a bit in a 4 byte integer atomically. Atomic means here, that if the same 4 byte integer is used by several tasks
and if the tasks modifies single bits, the bit remains consistent.
5.1.3 SysCpuReadValueAtomic/SysCpuWriteValueAtomic
With this functions, a 2, 4 and 8 byte value can be read/written atomic on a 32bit and 64bit platform!
5.1.4 SysCpuAtomicReadInt64/SysCpuAtomicWriteInt64 -
SysCpuAtomicReadReal64/SysCpuAtomicWriteReal64
With this functions, a 8 byte integer or real value can be read/written atomic on a 32bit platform. This is because accessing a 8 byte
datatype needs mostly 2 machine instructions which are not atomic.
So with these functions the data consistency of a 8 byte datatype can be achieved. For details see chapter Data Consistency.
5.1.5 SysCpuCompareAndSwap
Use this function to atomically compare and exchange a 4 / 8 byte value in one atomic operation. This function will only change the value if
the value matches a defined compare value. If not the value will not be exchanged. Instead the function will return an error.
NOTE:
Be aware that every synchronization operation will decrease your overall performance especially on MultiCore targets! So it is
recommended to prefer atomic operations (see Lock free programming) instead of synchronization operations!
For this we have several synchronization functions in the runtime system, which achieves different synchronization aspects and can be used
for different usages.
NOTE:
A mutex can be entered several times in one task! But you must be aware, that you have to leave as much as times you have entered
the mutex!
5.2.2 SysSemCount
The initial value of a counting semaphore can be specified at creating the semaphore. You can enter only so many times at the same time,
until this initial counter is reached! You to leave the counting semaphore as much as times you have entered it!
NOTE:
A counting semaphore should not be entered several times in one task, if the initial value is 1!
3S-Smart Software Solutions GmbH · Memminger Str. 151 · 87439 Kempten · Germany
Tel.: +49-831-54031-0 · Fax: +49-831-54031-50 · info@codesys.com · www.codesys.com Page 8 of 14
CODESYS Inspiring Automation Solutions MultiCore Support
5.2.3 SysReadWriteLock
Instead of using mutexes or semaphores you should prefer a ReadWriteLock (multiple reader / single writer). A readwrite lock can be used,
if there are several readers which uses the data structure, but only a view writers.
NOTE:
Several readers can enter the lock at the same time. A writer can enter only exclusively the lock and no reader can enter at the same
time. A task cannot enter a readlock several times or after entering a write lock! Vice versa a task cannot enter a writelock several
times or after entering a readlock!
NOTE:
In general: SysIntDisableAll/SysIntEnableAll use interrupt locking machanisms of the operating system, if implemented at all, and
therefore cannot be used here, because they work only on UP systems or on the current CPU core on SMP systems.
5.2.4.1 VxWorks
To support also a locking machanism for SMP systems, SysIntDisableAll/SysIntEnableAll use the ISR callable spinlock functions of the
VxWorks system library “spinLockLib” (spinLockIsrTake/spinLockIsrGive). For more information refer to the according Wind River manual.
5.3.1 SysMsgQ
Message queues can be used to delegate an operation to another task, which operates on the same message queue.
5.3.2 CmpAsyncMgr
The asynchronous manager can be used to delegate an operation to be executed typically in the background (in the context of the
CH_COMM_CYCLE hook, a separate task or an event).
6 Diagnostics
To measure multicore performance, the runtime system has several builtin diagnostic features.
For this you have to add a trace object under your device in CODESYS and with the command upload you can upload and start the traces
with the name “CpuCoreLoad” or “PlcLoad”.
3S-Smart Software Solutions GmbH · Memminger Str. 151 · 87439 Kempten · Germany
Tel.: +49-831-54031-0 · Fax: +49-831-54031-50 · info@codesys.com · www.codesys.com Page 9 of 14
CODESYS Inspiring Automation Solutions MultiCore Support
[CmpIecTask]
EnableTaskTrace=1
You can upload and start this trace to see the timing (start/end) of each IEC-task and its corresponding cycle.
cpuload - Get the current CPU load listed for each CPU core.
3S-Smart Software Solutions GmbH · Memminger Str. 151 · 87439 Kempten · Germany
Tel.: +49-831-54031-0 · Fax: +49-831-54031-50 · info@codesys.com · www.codesys.com Page 10 of 14
CODESYS Inspiring Automation Solutions MultiCore Support
Example:
CoreID: 0 1 2 3 4 5 6 7
----------------------------------------------------------------------------
CPU Core load: 9% 1% 16% 0% 2% 10% 0% 12%
plcload - Get the current PLC load listed for each CPU core.
Example:
CoreID: 0 1 2 3 4 5 6 7
----------------------------------------------------------------------------
PLC Core load: 4% 3% 3% 3% 5% 4% 4% 3%
Example:
getmulticoreinfo - Display some general information about the multicore architecture and configuration.
Example:
MultiCore Info:
-- MultiCore support: [activated], all cores are used
-- Number of cores: 8
3S-Smart Software Solutions GmbH · Memminger Str. 151 · 87439 Kempten · Germany
Tel.: +49-831-54031-0 · Fax: +49-831-54031-50 · info@codesys.com · www.codesys.com Page 11 of 14
CODESYS Inspiring Automation Solutions MultiCore Support
For IEC tasks you can manage taskgroups in the taskconfiguration and you can assign every task to such a task group. By default there is
one group names “IEC-Tasks”.
For an IEC taskgroup you can specify one of the following core bindings:
NOTE:
To implement an atomic incrementer/decrementer we recommend to use the SysCpuAtomicAdd() library function from
SysCpuHandling.library!
7.2.4 Strings
There is a library under construction which is called “Shared Data Area Library”. This can be used to handle a string consistent. Every
reader gets a copy of the latest value. Writers are using a new buffer to write consistent and after the value is completely written, the value is
published for the readers.
SysSem (SysSem.library) and SysReadWriteLock (SysReadWriteLock.library) are available as libraries and can be used to synchronize
3S-Smart Software Solutions GmbH · Memminger Str. 151 · 87439 Kempten · Germany
Tel.: +49-831-54031-0 · Fax: +49-831-54031-50 · info@codesys.com · www.codesys.com Page 12 of 14
CODESYS Inspiring Automation Solutions MultiCore Support
Perphaps the “Shared Data Area Library” can be used for this usage (see chapter Strings).
So the IEC application must be aware about this changed behaviour on MultiCore targets!
In some cases it can happen that the operations to the same memory location are seen in a different order on one core than on another
core. This can cause problems where the data is used to synchronize data between threads on different cores.
To handle this issue there is a new operator __MemoryBarrier available in IEC code. This operation takes care that laod/store instructions
where the order matters are seen in the same order on all CPU cores (cache synchronization).
Typically the use of the operator is highly platfrom dependend. If the atomic operations of SysCpuHandling are used, the runtimesystem
takes care about the order on all cores. For details see documentation of target-setting codegeneratormemory-barrier.
2. SL Runtimes:
CODESYS Control RTE is not released for MultiCore yet. CODESYS Control RaspberryPi and EmPC are not released for MultiCore
yet.
Multicore is not supported for robotics and CNC with SoftMotion 4.3. Therefore, the planning task (for robotics) and the path task (for
CNC) have to run on the same processor core as the bus cycle task.
9 Glossary
SMP
Symmetrical Multi Processor system: System with several identical CPU cores/processors
AMP
Asymmetrical Multi Processor system: System with several different CPU cores/procesors
UP
Uni Processor system: System with one CPU core/procesor (e.g. single core CPU)
3S-Smart Software Solutions GmbH · Memminger Str. 151 · 87439 Kempten · Germany
Tel.: +49-831-54031-0 · Fax: +49-831-54031-50 · info@codesys.com · www.codesys.com Page 13 of 14
CODESYS Inspiring Automation Solutions MultiCore Support
Version History
Version Description Author Date
0.1 Creation AH 20.03.2017
0.2 Rework and update AH 12.04.2017
0.3 Rework and update AH 27.11.2017
0.4 Beta Release v3.5.12.0 AH 30.11.2017
0.5 Rework for V3.5.13.0 ASch 28.06.2018
0.6 Review AH 28.06.2018
1.0 Release AH 02.07.2018
3S-Smart Software Solutions GmbH · Memminger Str. 151 · 87439 Kempten · Germany
Tel.: +49-831-54031-0 · Fax: +49-831-54031-50 · info@codesys.com · www.codesys.com Page 14 of 14