API for these operations to be used in multiple OSes or “bare-metal”

August 2012 

TM 

Freescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, 

ColdFire+, C-Ware, the Energy Efficient Solutions logo, Kinetis, mobileGT, PowerQUICC, 

Processor Expert, QorIQ, Qorivva, StarCore, Symphony and VortiQa are trademarks of 

Freescale Semiconductor, Inc., Reg. U.S. Pat. & Tm. Off. Airfast, BeeKit, BeeStack, 

CoreNet, Flexis, MagniV, MXC, Platform in a Package, QorIQ Qonverge, QUICC Engine, 

Ready Play, SafeAssure, the SafeAssure logo, SMARTMOS, TurboLink, Vybrid and 

Xtrinsic are trademarks of Freescale Semiconductor, Inc. All other product or service 

names are the property of their respective owners. © 2012 Freescale Semiconductor, Inc. 

.

• What is Multicore 2.0 

• Optimizations 

• Virtualization 

• Usability 

• Distribution 

• Software Environments 

• A Multicore Decision Tree 

TM 

2 

Freescale, the Freescale logo, AltiVec, C-5, CodeTEST, CodeWarrior, ColdFire, ColdFire+, C-Ware, the Energy Efficient Solutions logo, Kinetis, 

mobileGT, PowerQUICC, Processor Expert, QorIQ, Qorivva, StarCore, Symphony and VortiQa are trademarks of Freescale Semiconductor, Inc., 

Reg. U.S. Pat. & Tm. Off. Airfast, BeeKit, BeeStack, CoreNet, Flexis, MagniV, MXC, Platform in a Package, QorIQ Qonverge, QUICC Engine, 

Ready Play, SafeAssure, the SafeAssure logo, SMARTMOS, TurboLink, Vybrid and Xtrinsic are trademarks of Freescale Semiconductor, Inc. All 

other product or service names are the property of their respective owners. © 2012 Freescale Semiconductor, Inc.

• Optimization 

− Layer 2 

� UIO and USDPAA 

� Core affinity 

� ASF/Bypass 

� Hugetlbfs 

− Layer 3 

� TCP termination 

− Power 

• Virtualization 

− KVM 

− Containers 

− Topaz 

TM 

• Usability 

− DPAA 

� Reference SW 

� SEC enablement 

� DPAA 

programmability 

� New User Guides 

− Distribution options 

� Git trees 

� Virtual environments 

� Yocto 

3 

• SW environments 

− Linux 

− Bare Metal 

− Foundational 

Libraries 






Core Affinity 

TM 

Bypass 

4 

User Space IO - UIO 

Hugetlbfs 






User space apps get 

direct zero-copy 

access to all DPAA 

HW 

Benefits: 

• Rich and flexible environment– use Linux standard services rather 

than inventing new ones (C++, 36-bit, 8 th core, debug, etc). 

• Standard– Linux is ubiquitous and supported from multiple sources. 

• Provides high-performance run-to-completion but other use-cases 

also– a use case super-set. 

TM 

Buffer 

Manager 

others 

User Space Application 

Queue 

Manager 

Security Engine 

(crypto) 

5 

Pattern Match 

Engine 

Frame 

Manager 

Ethernet 






• Low-level, high-efficiency programming for subset of system SW 

that represents bulk of workload and is worth a lot of optimization. 

• Fast-path offload of a router stack (like a SW ASIC) 

• Layer 2 and up in a cell base station 

• Software fed by bump-in-wire net stack (like IPSEC), often with specialcase 

assumptions allowed because network is controlled. 

Linux Bare Metal Style SW 

core core core core 

TM 

Linux 

core 

core core core core 

6 

Bare metal software 

migrates from running 

alongside Linux on its own 

cores to running within a 

Linux user space process 

Bare Metal Style SW 

core 

Not really intended for legacy user space applications, but there is 

interest in this. 






Portals mapped 

directly into 

application’s virtual 

address space. 

enqueue, 

dequeue, 

and more 

TM 

Linux User Space 

Application 

QMan and BMap Access 

API Library 

QMan SW 

Portal 

QMan 

Queue 

Manager 

7 

BMan SW 

Portal 

BMan 

Buffer 

Manager 

No system call or 

kernel context switch 

to access a portal. 

acquire, 

release, 

and more 






Other Other 

Processes Processes 

Net Stack 

Eth Driver 

thread 

QPortal 

BPortal 

C0 



kernel 


TM 

C1 

Other Driver 

BQMan Drivers 

USDPAA Application Can Use 1 to 8 Cores (P4080) 

Each thread has a dedicated portal and is affine to a core, 

1 thread per core 




C2 

USDPAA Memcached 

thread thread thread thread thread thread thread 



C3 

8 



C4 

BMan / QMan 

FMan Ethernet 

Ports 



C5 



C6 



C7 

isolated cores 

7 cores are isolated but 1 can 

run an USDPAA thread as 

well as other processes. 

Core 0 has portal for kernel use and standard Linux networking 






TM 

Linux Network Processing 

ASF: Data Path Acceleration in Software 

9 






• When a process uses memory, the CPU marks the RAM as 

used by that process. 

• For efficiency, the CPU allocates RAM by chunks of 4K bytes 

(it's the default value on many platforms) called pages (can be 

swapped to disk, etc) 

• Since the process address space is virtual, the CPU and the 

operating system have to remember which page belong to 

which process, and where it is stored 

• Obviously, the more pages you have, the more time it takes to 

find where the memory is mapped 

− e.g. when a process uses 1GB of memory, that's 262144 entries to 

look up (1GB / 4K). 

− If one Page Table Entry consume 8 bytes, that's 2MB (262144 * 8) 

to look-up. 

TM 

10 






• The default page size in Linux is 4KB 

• The amount of memory mappable in TLB0 with 4KB pages is small, which leads 

to TLB0 misses. Major contributors are; 

− TLB0 is shared between multiple processes 

− TLB0 is used for mapping both instructions and data 

− A single MMU-hogging process in the system can have performance impact on 

other processes running in the system due to TLB0 thrashing 

− A single processes may use significantly more than 2MB 

• TLB miss can impact the latency by reloading TLB entries 

• TLB1: for e500 processor families, and supports a number of large page sizes 

Processor # of TLB0 

entries 

TM 

Amount mappable by 

TLB0 

# of TLB1 entries Page size for TLB1 

E500V1 256 1MB 16 4KB, 16KB, 64KB, 256KB, 1MB, 4MB, 

16MB, 64MB, 256MB 

E500V2 512 2MB 16 Same as e599V1, plus 1GB, 4GB 

E500mc/e5500 512 2MB 64 Same as E500v2 

11 






• Hugetlbfs allows the Linux kernel to use large pages mapped 

by TLB1 for user processes 

− The kernel itself does not use “hugetlb”; it uses a few large TLB1 

entries to map some amount of memory using large mappings 

• Hugepages can be allocated/used a number of ways: 

− Specify SHM_HUGETLB flag to shmget() 

− Link application with hugetlbfs to enable shmget() to use 

hugepages 

− Link application with libhugetlbfs to cause malloc() to use 

hugepages 

− Explicitly link application program for hugepage text, data, and/or 

BSS 

− Specify MAP_HUGE to anonymous mmap() 

− mmap() a file on the hugetlb mount point 

� *Note – some of these methods can be combined 

TM 

12 






• BGP/BFD performance benchmark: 

− BFD utilize USDPAA technology, which has utilized hugetlb technology, and 

shows very competible result by comparing competitors on P4080. P50xx 

will be much better. 

1G Ethernet port 4,629 sessions/3.3ms (1G port line rate) 

10G Ethernet port 10,200 sessions/3.3ms 

− BGP result is under preparation, we are trying to utilize hugetlb for 

optimization on P5020: 

� Here is the prelimenary result: 

• BGP data comparison (Note: T1: Learning 300,000 routing tables, T2: 300,000 routing tables 

convergence time ) 

TM 

T1 (Seconds) T2 (Seconds) 

MPC8572ds 19 24 

P5020ds 11 13 

P5020ds Vs.MPC8572ds 142% 137.5% 

13 






• Benefits 

− User processes which access a lot of memory have fewer page faults 

− Other processes can benefit because their 4k pages aren’t getting 

evicted by memory hog processes 

− Performance gains can be *significant* for applications with a large 

memory footprint 

• Limitations 

− Number of concurrent huge pages in TLB1 is limited 

− Hugepage adds minor latency to critical kernel paths even for 4k pages 

− Hugepages should only be used when needed, as they are more costly 

for the kernel to manage 

TM 

14 






TCP receive 

hardware offload 

FAST-TCP-ACK 

L4 SKB-recycling 

Soft-TSO 

Receive file 

support 

Cache optimizations 

(L2SRAM,data prefetch) 

Adaptive load-balance 

Non-DPAA DPAA 

Done 

Implementing 

TM 

Planning 

Non-DPAA SDK 

Non-DPAA SDK 

non-DPAA SDK 

Non-DPAA SDK 

non-DPAA SDK 

2010 2011 2012 2013 

15 

Unified SDK1.2 

Unified SDK1.2 

Unified SDK 

Unified SDK 

Unified SDK 

Unified SDK 

Unified SDK 






MPC83xx 

e500v2 

e500mc/e5500 

e6500 

dynamic power 

management 

MPC83xx suspend 

e500v2 sleep 

e500v2 deep sleep 

e500v2 idle 

Already available 

TM 

e500v2 JOG 

eTSEC wakeup on network 

SDK 

1.2 

Compatibility of legacy drivers 

e500mc sleep 

Synology on-demand NAS server product 

e500mc frequency scaling 

PH30 and PCL10 states 

Cascade Power 

Cascade 

Management 

PM demo 

demo 

2012 2013 

16 

SDK 

1.3 

TMU 

T1040 Deep Sleep 

Compatibility of DPAA drivers 

SDK 

1.4 

Proposal 

Planning 

Execution 

Finished 






Partitioning Virtualization 

• Hardware consolidation 

• Partitioned/dedicated 

resources, minimal 

sharing. 

• Dedicated CPUs, I/O 

devices 

OS OS 

TM 

17 

OS 

• N virtual machines 

• Resource sharing, 

oversubscription 

• Virtual I/O 

• Highly virtualized 

environment 

• Live migration 

OS 

OS OS 

CPU CPU CPU CPU 






Utilization 

Consolidation 

Dynamic Resource Management 

TM 

18 

Security/Sandboxing 

Fail over 






• KVM is a Linux ® kernel 

driver 

• User space tool, 

QEMU, is used in 

conjunction with KVM 

• Solution is open 

source 

• Number of virtual 

machines is only 

limited by available 

resources (CPU 

cycles, memory) 

TM 

• A lightweight framework 

for partitioning an SoC 

• Gives you the best of 

both worlds– bare metal 

performance with 

enforced partitioning and 

fully architected 

approach to meeting 

AMP requirements 

• Solves many of the 

headaches of running 

multiple unsupervised 

OSes 

• Threads appear as 

cores to OS 

19 

• 

• 

• 





other product or service names are the property of their respective owners. © 2012 Freescale Semiconductor, Inc. 

C 

P 

U 

COS 

App 

COS 


Multicore Hardware 

COS 

LXC LXC LXC 

CP 

U 

USDPAA USDPAA 

CP 

U 

C 

P 

U

TM 

20 






1. Design of user-space (US) DPAA apps via reference software 

− 8 sample apps (benchmarks of code, doc, perf) for IPFwd, IPSec, crypto, PME, sRIO, RMAN 

− Optimization of open-source libraries for USDPAA (SNORT-DAQ, OpenSSL) 

− Additional set of customer-specific demos/benchmarks delivered by L3 team 

2. New tools to offer control of DPAA 

− QorIQ Configuration Suite 

− Scenario Analysis Tool, including 70 pre-defined scenarios 

− SPID (Services for Profiling, Instrumentation, and Debugging) 

3. Improved accessibility and usability of documentation 

− Better guidance through documentation maps 

− Redesigned documentation for USDPAA examples and tools 

− Application notes and design primers derived from customer optimization experience 

4. Improved out-of-the-box experience and reduced installation effort 

− virtual environment (Virtual Box) that bundles pre-installed SDK, CW, DPAA tools, docs 

− “a-la-carte” delivery method for open-source software from git.freescale.com 

TM 

21 






1. SEC Enablement 

− New Descriptor development tools 

− More robust and complete API’s and Descriptor Libraries 

2. Extension of reference software for User Space DPAA 

− Functional updates to existing USDPAA reference code 

− Optimized implementation of OSS: SNORT-DAQ, OpenSSL 

3. Extension of DPAA based analysis tools 

− QCS support for USDPAA and more processors 

− Evolution of SPID and Packet tracing tools for DPAA 

4. Additions and Improvements of documentation delivery 

− Conceptual (why), procedural (how), functional (what) 

− New app notes and DPAA porting guidelines 

5. Diversify means of distributing products 

− Continue with Virtual Environment 

− Launch “a-la-carte” delivery via git.freescale.com (copy of internal repos) 

− Embedded distributions 

TM 

22 






• Implemented run-time assembly language for DECO’s programming 

− Similar to standard assemblers 

− Source is available and easy to understand 

− Error checking 

− Intuitive syntax 

− Simplifying understanding and writing code 

TM 

23 






• Scenarios Tool 

− Simple one click access to common performance analysis 

scenarios for QorIQ – including DPAA Debug IP 

− Removed need from customer to know how to configure 

DPAA debug IP to collect performance data from common 

scenarios. 

• Register Analyzer 

− Alpha version delivered 

− Provides feedback on incorrect configuration of DPAA blocks 

(FM, QM, BM and SEC) 

• SPID (Major Updates for 2012) 

− Target Side Library to access debug IP inside 

(instrumentation) a customer application. 

− Sample code to show how to collect trace and performance 

data for basic cases. 

− Extensive documentation. 

− Supports Linux and Bare Metal Applications 

• SPID Utilities (New for 2012) 

− Applications focused on specific use-cases to improve 

usability of debug IP in FSL SDK Linux applications. 

− Packaged for release in SDK 1.2 

− First release 3Q 2012 

TM 

• Application Notes 

24 

− Performance Analysis 

− LTTng Trace 

− Application Debugging 

• qoriq-dbg (Major Updates for 2012) 

− Easy access to QorIQ debug IP from user scripts or 

applications via standard file system operations. 

− Present tools in a manner that is familiar to Linux developers 

• DPAA Tools. (New for 2012) 

− Packet Tracing tool – Allows a customer to trace a packet as it 

moves through system. 

− Focus for 2012 

− First version scheduled for 4Q 2012. 

− This tooling makes use of the on-board trace facility of each 

block. 

− DPAA Trace supporting ALU USDPAA application Alpha being 

delivered in CW 10.1.2 (may 2012) and Production versions 

(Expert mode) scheduled for CW 10.2 (June 2012) 

− LTTng/LTTngX 

− Linux Community standard Kernel Tracing Framework 

− LTTngX – Freescale’s unique extensions to LTTng 

Framework to support access to Freescale QorIQ Debug IP 






TM 

Modify classification 

For example: 

-LPM 

-Add support for 

proprietary shim 

headers 

1 

2 

Perform unique Header Manipulation/ 

Add Statistics 

25 

Example of a typical DPAA Down-Link 

Data Flows 

Processing 

All the processing elements are 

standard general Availability building 

blocks provided by FSL to all 

customers. 

• Addition new “processing elements” to the flow 

• Code should obey the programming model 

defined by FSL 

• Non-intrusive extension of existing code 

• No flow changes. Transparent from the 

processing flow perspective. FMD assisted 

configuration 






Host CPU (e500) 

Configures the 

Parameters (once) 

Each App will be created by 

chaining up existing and new 

building blocks by the tool 

(FMC) creating the desired 

flow. 

TM 

Initialize 

Descriptors 

BMI or other 

module 

Custom 

App 2 


App 1 

Example: 

ARP 

Scheme 1 

Scheme 2 

Scheme 3 

Scheme 4 

Scheme n 

Key Gen 

26 

Initialize 

Descriptor 


App n 

(Netflow 

) 

API Call 



Each function has 

a well defined 

definition obeying 

rules 

FMAN HW BLOCK 

HW/SW PARSER 

Packet In 

Optional 

Extension Optional 

Extension Optional 

Extension 

RISC LIBRARY 

Optional 

Extension 

Existing function 




Internal 

Context 






Linux is an open-source integration of many components from 

many sources, most of which are architecture-independent and 

don’t originate with Freescale. 

Customers (and FSL 

internally) cannot use 

Linux without a complete 

kit. 

• Many distros exist. 

• Some customers create 

their own. 

• Major FSL SW must be 

usable with arbitrary 

distros. 

• But FSL also must use 

and ship one. 

• FSL choice: Yocto 

TM 

Kernel 

FSL-specific 

Support 

GNU Tools & Libs 

FSL-specific 


Bootloader 

FSL-specific 


Major FSL component 

27 

Package & Update 

Manager 

Window System 

(if needed) 

Proprietary value-add 

(Often Tools in 

Embedded Linux) 

10’s of Thousands 

of User Space 

Packages 

Build 

Environment 

Pass-through from other source 

BOM (License 

management) 

Docs 

User Space 

Management 

Scripts 

FSL User Space 

Packages 






27

Model Approach When to Use Attributes 

Native on 

Eval Board 

Yocto 

(Complete) 

Provide evaluation boards with 

complete native GNU Tool 

environments right on the board. 

This embedded distribution helps 

customer create entire Linux 

system. Package both as ISO 

image and also in virtual 

machine. 

A la Carte Simplify customer access to just 

the major Freescale-created 

Linux components. Perfect for 

integration into Linux 

distributions from other sources, 

home-brew or 3 rd party. Supports 

TM fast delivery of patches. 

28 

Desire zero “getting 

started” effort to 

building and running 

FSL and standard 

OSS 

Need a tool to 

generate a 

complete Linux 

environment 

including tailored file 

system. 

Desire to integrate 

Freescale Linux 

components into a 

Linux development 

environment that 

the customer 

Easy to use. 

Complex, 

but very 

flexible and 

powerful. 

Simple when 

the 

customer is 

also the 

integrator. 




Ready Play, already SafeAssure, the SafeAssure logo, has. 

SMARTMOS, TurboLink, Vybrid and Xtrinsic are trademarks of Freescale Semiconductor, Inc. All 


C 

P 

U 

C 

P 

U 

VM 

VM 


VM 

App App App 

OS OS OS 

Embedded Hypervisor 

C 

P 

U 

App 

CP 

U 

C 

P 

U 

App App 


CP 

U 


Linux SMP 

Partitioning / 

supervised AMP, 

failover 

C 

P 

U 

TM 

C 

P 

U 

App App 

C 

P 

U 

C 

P 

U 


CP 

U 

CP 

U 

BB 


Unsupervised AMP 

App 

CP 

U 

VM 

App 

OS 


CP 

U 


29 

C 

P 

U 

VM 

BB 

CPU, I/O 

virtualization 

App 

OS 

KVM KVM 

Topaz KVM: Linux-based Hypervisor 

C 

P 

U 

Consolidation, high 

performance 

user space DPA 

engines 

Linux OS 

Convergence 

Leverage both 

KVM & LXC 

OS 

Virtualization 






C 

P 

U 

C 

P 

U 

COS 

App 

App 

COS 



COS 

LXC LXC LXC 

CP 

U 

CP 

U 

USDPAA USDPAA 



USDPAA on Linux SMP 

USDPAA USDPAA 

CP 

U 

CP 

U 

Isolated 

Containers, 

resource control & 

monitoring 

Linux Containers 

C 

P 

U 

C 

P 

U

• Purpose of foundation libraries (flibs): 

• Library functions that perform 

operations on Freescale hardware IP 

blocks, like DPAA peripherals. 

• Single OS-decoupled (not abstracted) 

API for these operations to be used in 

multiple OSes or “bare-metal” 

applications. 

• Helper functions for writers of OSspecific 

“native” drivers or applications 

that directly access hardware. 

• Focus on simplicity and avoidance of 

OS-coupling. 

• Flibs are low-level software 

TM 

30 

OS-Specific Driver 

FLIB FLIB 






• Customers desire more commonality in FSL software, regardless of 

the operating system used. 

• Customers want greater simplicity– even with complex hardware. 

TM 

31 






OS-Specific I/O systems or 

network stacks 

“Native” Driver 

--- OS-defined APIs 

Hardware 

TM 

FLIB for IP 

Block 

Thin adaptation layer, e.g. 

register accessors. 

FLIBs present OS-independent API for 

hardware functionality with focus on 

configuring HW. 

FLIBs are not complete drivers. They 

leave functionality to the native driver 

when the functionality is deeply OScoupled, 

for example: 

32 

• Registering ISRs. 

• Interacting with scheduler 

• Mapping memory. 

• Accessing native driver APIs. 

• Etc. 

Think of FLIBs as an implementation style for native drivers that emphasizes 

a standard API for HW configuration and reuse across multiple OSes. 






Hardware 

FLIB FLIB 

Linux Driver 

(from FSL) 

FLIB FLIB 

TM 

Analyze hardware– derive from it the essential 

low-level functions 

Derive what FLIBs do from this thought 

process– but don’t let them do “OS things”. Be 

OS-decoupled, not OS-independent. 

Use FLIBs to create OS-Specific drivers for two 

FSL-supported OSes. 

SDOS Driver 

(from FSL) 

FLIB FLIB 

Offer FLIBs to customers and 3 rd 

parties for other OSes. 

33 

Other OS Driver 

(not from FSL) 

FLIB FLIB 






• Standard Linux drivers. 

• Ecosystem acceptance of those drivers. 

• SDOS as a “bare metal-type” OS environment– with 

reasonable drivers. 

• FLIBs themselves as driver building-blocks, not 

drivers. 

• Greater Simplicity 

• Less OS integration conflict 

• Less architectural imposition 

• Less assumption of global stateful management 

(aiding system decomposition) 

TM 

34 






http://www.eetimes.com/design/embedded/4372692/A-decision-tree-approach-to-picking-the-right-embedded-multicore-software-architecture 

TM 

35 






TM 

36 






TM 

37 






TM 

38 






TM 

39 






Performance 

Tooling 

Book ‘E’ Debug 

Core 

Debug 

JTAG/Aurora 

Trace Tooling 

CodeWarrior 

Workbench 

Eclipse 

Workbench 

GNU tools 

(gcc, gdb) 

CodeWarrior / 3 rd 

Party Debug / Trace 

Probe 

Debugger 

Real Time Debug 

Watchpoint 

Cross 

Trigger 

Perf 

Monitor 

3 rd party Eclipse Tooling 

CoreNet 

Trace 

Aurora 

3 rd party Tooling 

TM 

Netcomm 

Software 

(NCSW) 

Drivers 

Debug / 

Perf agents 

VortiQa 

Applications 

QorIQ 

Network 

Services 

Linux operating 

environment 

Net 

drivers 


control 

ASF control 

Host linux (optional) 

Power 

Architecture TM 

Core (e500v2) 

Development Host Runtime environment(s) 

40 

Fastpath 

(ASF) 

Market-specific solutions 

Power 


Core (e500dp) 

DPAA 

configuration and 

services 

Networking services & 

utilities 

QorIQ 

Drivers 

Networking stack 

Power ISA 2.06+ 

32/64bit core 

Power 


Core (e500mc) 

DPAA 

drivers 

Customer 

Application 

QorIQ SDK QorIQ DPAA SDK 

Power ISA 2.06+ 

32/64bit core 

User Space DPAA 

Execution 

environment 

Drivers 

Linux User-space 

Linux Kernel 

KVM/Qemu 

Power 


Core (e5500,e6500) 






Linux Platform

TM 

Freescale on Kaixin 

Tag yourself in photos 

and upload your own! 

Session materials will be posted @ www.freescale.com/FTF 

41 

Weibo? 

Please use hashtag 

#FTF2012#

API for these operations to be used in multiple OSes or “bare-metal”

Create successful ePaper yourself

Delete template?

Save as template?