Tracing Merge System 'Integrated Trace' - Linux Foundation Events

Tracing Merge System ‘Integrated Trace’for a Virtualization EnvironmentYoshihiro YunomaeLinux Technology CenterYokohama Research Lab.Hitachi, Ltd.Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved1

Agenda1. What is “Integrated trace”?2. How to merge trace data- methods- restrictions3. Case study4. SummaryCopyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved2

Agenda1. What is “Integrated trace”?2. How to merge trace data- methods- restrictions3. Case study4. SummaryCopyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved3

What is the Integrated Trace ?• Trace merging system for a virtualization env.– Merge a number of tracing data• Multiple guests• A host– Using ftrace, which is an in-kernel tracer• If user application uses trace_marker I/F, the tracing data are also integrated.– Merge tracing data in a chronological order• Who uses this system?– Cloud vender / System vender– Platform engineer / developerCopyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved4

Why do we need to merge tracing?• Difficult to analyze performance problems on a guest– A delay of I/O request on a guest– Guests and host share H/W devices.• CPU, memory, HDD, NIC, etc…– We need to analyze tracing data of multiple guests and ahost even if the problem occurred on a guest.Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved5

Examplewhich needs to merge trace data• A guest occupies one CPU core.• A real-time application on the guest sometimesbecomes delay.– Memory is enough.– The application does not issue any I/O requests.– Other applications are not scheduled on that time.• We found that this problem was caused by CPUstealing on a host by applying the integrated trace.– I’ll show you this cause later.Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved6

Our goal imageguest1trace dataof guest1guest2trace dataof guest2merged trace dataftraceftracehostKVMftracetrace dataof hostmergeHow do we merge trace data?Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved7

Agenda1. What is “Integrated trace”?2. How to merge trace data- methods- restrictions3. Case study4. SummaryCopyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved8

Merge tracing data• Merge tracing data in a chronological order• The kind of timestamp of ftrace(trace_clock)Timestamplocalglobalcounteruptimeperfx86-tscSummaryUse sched_clock() (not synchronized for all cores)Use sched_clock_cpu() (synchronozied for all cores)Increment ordering numberUse jiffiesUse local_clock() (same clock that perf uses)Use TSC• x86-tsc is useful for merging trace data in chronological order by 2 following reasons:Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved9

Two reasons for using x86-tsc1. Guests can execute native rdtsc instruction.- Not a software clock like sched_clock()2. TSC of recent x86 CPUs(assuming Intel arch.) isconstantly incremented.- Recent x86 CPUs have the “Invariant TSC” feature.We can check the feature in /proc/cpuinfo.# cat /proc/cpuinfo…flags: fpu vme de pse tsc msr …. constant_tsc … nonstop_tsc …- No worry about pace of the timestampWe need to consider about just one matter for using TSC.Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved10

Read TSC value on a guest• TSC values on a guest do not equal to the values on ahost. Why?• This is because TSC value on a guest is always thehost TSC plus the guest’s TSC offset.– User needs to get TSC offset for merging trace data.TSC_guest = TSC_host + TSC_offsetguestrdtschostCPUTSC_hostTSC_host + TSC_offsetCopyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved11

TSC offset• TSC offset is stored in VMCS.• The value is basically an sign inversion value of hostTSC when guest booted.• The value can be changed when guest executeswrite_tsc.– Any kernel codes don’t execute write_tsc.• User programs can execute write_tsc using wrmsr instruction.# wrmsr 0x010 – If TSC is unstable, TSC offset will be changed, but we don’tthink about this problem here to make it monotonic.Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved12

Getting TSC offset• When a kernel changes TSC offset, it executesvmcs_write64(TSC_OFFSET, offset value)• Not exported in anywhere• Only the host kernel can read TSC offset from VMCS– Any user tools cannot get TSC offset.We need to output a TSC offset.Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved13

How do we output a TSC offset?1. Add a TSC offset parameter of kvm_entry/exit events- Most reliable method- But overkill in common case(because almost TSC offset is same value)2. Use printk()- my first try(RFC patch)“I chose printk() to output TSC offset value, but I think this is not thebest method.”=> Reject (not beautiful)Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved14

How do we output a TSC offset?1. Add a TSC offset parameter of kvm_entry/exit events- Most reliable method- But overkill in common case(because almost TSC offset is same value)2. Use printk()- my first try(RFC patch)“I chose printk() to output TSC offset value, but I think this is not thebest method.”=> Reject (not beautiful)3. Define a tracepointCopyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved15

Introduce multiple buffers• From the latest kernel(3.10-rc), we can use multiplering-buffers for some purposes.• If we make a ring-buffer for outputting a TSC offsetevent, the event will not be overwritten by otherevents.Ring-bufferA eventB eventC eventD eventE event…After that…Ring-buffera eventb eventc eventd evente event…Tracing data in amain ring-bufferwill be overwritten.Tracing data in asub-buffer for TSCoffset will not beoverwritten.TSC offset eventTSC offset eventCopyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved17

TSC offset tracepoint• TSC offset tracepoint is defined before changing TSCoffset.– vmx_write_tsc_offset(), vmx_adjust_tsc_offset()trace_kvm_write_tsc_offset(); /* tracepoint */vmcs_write64(); /* changing TSC offset */• Output a previous offset and a changed offset– These two values are used when we merge data.• Discuss later[snip] kvm_write_tsc_offset: previous 0 next 1234567890initial valuechanged valueCopyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved18

Merge tool• After collecting trace data of guests and a host, wemerge those data using TSC offset values.• We have a number of scenarios for merging those.– Multiple guests– Multiple cpus/vcpus• For the sake of ease, we assume that …– Only one CPU core– Only one guest– Assign one CPU for the guestCopyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved19

Merging procedure1. Check if the kvm_write_tsc_offset events arerecorded except for boot or not.- a vcpu thread writes TSC offset when a guest boots.- prev_offset = 0 or prev_offset = next_offset are boot events.1-1. If not recorded, it means that a guest does notexecute write_tsc on a guest.1-2. If recorded, it means that a guest execute write_tscon a guest.Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved20

Basic merging procedure• tsc_offset changed only once(when a guest boots).timestampevents1005 A event(host)1020 write_tsc_offset prev 0 next 10101030 kvm_entry1100 kvm_exit1105 kvm_entry1200 kvm_exittimestampevents80 A event(guest)110 B event(guest)Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved21

Basic merging procedure• tsc_offset changed only once(when a guest boots).timestampevents1005 A event(host)1020 write_tsc_offset prev 0 next -10101030 kvm_entry1100 kvm_exit1105 kvm_entry1200 kvm_exit1. Check write_tsc_offset eventTimestamp, prev, nextWe find out tsc_offset changed onlyonce, so we use only next value.timestampevents80 A event(guest)110 B event(guest)Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved22

Basic merging procedure• tsc_offset changed only once(when a guest boots).timestampevents1005 A event(host)1020 write_tsc_offset prev 0 next -10101030 kvm_entry1100 kvm_exit1105 kvm_entry1200 kvm_exittimestampevents80 A event(guest)110 B event(guest)2. Conver guest TSC into host TSC1090 A event(guest)1120 B event(guest)Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved23

Basic merging procedure• tsc_offset changed only once(when a guest boots).timestampevents3. Merge tracing data1005 A event(host)1020 write_tsc_offset prev 0 next -10101030 kvm_entry1100 kvm_exit1105 kvm_entry1200 kvm_exittimestampevents1090 A event(guest)1120 B event(guest)1005 A event(host)1020 write_tsc_offset prev 0 next -10101030 kvm_entry1090 A event(guest)1100 kvm_exit1105 kvm_entry1120 B event(guest)1200 kvm_exitCopyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved24

ResultTSCeventsCopyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved25

Merging procedure1. Check if the kvm_write_tsc_offset events arerecorded except for boot or not.- a vcpu thread writes TSC offset when a guest boots.- prev_offset = 0 or prev_offset = next_offset are boot events.1-1. If not recorded, it means that a guest does notexecute write_tsc on a guest.1-2. If recorded, it means that a guest execute write_tscon a guest.Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved26

Execute write_tsc on a guest(1)• A guest executes write_tsc, then tsc_offset is chaged.– Not changed a host TSC.• We can consider about 2 situations:– Guest TSC monotonically proceeds.– Guest TSC goes backward.timestampeventstimestampeventsJumping80 A event(guest)110 B event(guest)Jumping80 A event(guest)110 B event(guest)500 C event(guest)505 D event(guest)35 C event(guest)40 D event(guest)• A merge tool will change TSC offset adding for eachtrace event of guest.Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved27

Execute write_tsc on a guest(2)• write_tsc or wrmsr tracepoints are not available.– We don’t know when write_tsc was executed.• Monotonic proceeding events do not always mean thata guest executes write_tsc for forward.– A merge tool uses a TSC offset event.timestampevents80 A event(guest)110 B event(guest)will_write_tsc = current_TSC – 50;120 C event(guest)A guest executed write_tsc for backward.For some time, this next event was not hit.Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved28

When was write_tsc executed?• TSC offset events output timestamp(TSC value) andprevious TSC offset.• When TSC offset is changed, guest does not operateon the VCPU.– A merge tool can apply previous TSC offset or next TSCoffset for each trace event of a guest.Tg = TSC_host + prev_offsetkvm_exitwrite_tsc_offsetkvm_entryExecute write_tscEvent AEvent BEvent CTgCopyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reservedPrevious offsetNext offset29

When was write_tsc executed?• TSC offset events output timestamp(TSC value) andprevious TSC offset.• When TSC offset is changed, guest does not operateon the VCPU.– A merge tool can apply previous TSC offset or next TSCoffset for each trace event of a guest.Tg = TSC_host + prev_offsetkvm_exitwrite_tsc_offsetkvm_entryExecute write_tscIs this a general method?Event AEvent BEvent CCopyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reservedTgPrevious offsetNext offset30

General or not (1)• To search the point where the guest executedtsc_write, we find ‘n’ which satisfies the conditionTg = TSC_host + prev_offsetTn < Tg < Tn+1 or Tn+1 < Tn < Tgmonotonic proccedgo backward• If write_tsc is executed only once on a guest, themethod is useful.timestampeventsPrevious offset(=boot)timestampeventsTnTn+180 A event(guest)110 B event(guest)500 C event(guest)505 D event(guest)TgTnTn+180 A event(guest)110 B event(guest)35 C event(guest)40 D event(guest)Next offsetCopyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved31

General or not (2)• If all timestamp of a guest monotonically proceed, themethod is useable even if tsc_offset was changedmany times.Changed tsc_offset event910 write_tsc_offset prev 0 next -9001100 write_tsc_offset prev -900 next -7001250 write_tsc_offset prev -700 next -600Tg = 200Tg = 550timestampevents80 A event(guest)110 B event(guest)500 C event(guest)505 D event(guest)600 E event(guest)610 F event(guest)900700600Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved32

General or not (3)• If timestamp of a guest go backward many times, themethod is not useful because points executedwrite_tsc are not identified.– We put the restrict which trace data must have monotonictimestamp for this method or write_tsc can be executed onlyonce.Changed tsc_offset event910 write_tsc_offset prev 0 next -9001100 write_tsc_offset prev -900 next -10001250 write_tsc_offset prev -1100 next -1150timestampevents80 A event(guest)110 B event(guest)35 C event(guest)40 D event(guest)30 E event(guest)45 F event(guest)Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved33

Agenda1. What is “Integrated trace”?2. How to merge trace data- methods- restrictions3. Case study4. SummaryCopyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved34

Go back to a first example• A guest occupies one CPU core.• A real-time application on the guest sometimesbecomes delay.– Memory is enough.– The application does not issue any I/O requests.– Other applications are not scheduled on that time.• Let’s analyze this problem in detail.Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved35

1. Using cyclictest• We use cyclictest, which is a test program for a realtimetendency, for this problem.# ./cyclictest –p 99 –m -n –i 2000 –qp: priority, m: use mlockall, n: use nanosleep, i: sleep interval(us), q: queit1. Get current time on a real-time cyclictest processA2. Nanosleep on A3. Context switch to another processB4. After finishing sleep time, B wakes A up5. Conext switch to Ananosleep()sched_switchwake cyclictest upnanosleep_exitcyclictestA another proces BAsleep intervalCopyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved36

2. Define an wakeup-latency• We define an wakeup-latency as follows:wakeup_latency = nanosleep_exit - nanosleep_enter – sleep_intervalnanosleep_entersched_switchnanosleep_exitcyclictestanother proces cyclictestsleep intervalwakeup_latency• Sometimes, wakeup-latency becomes big.normally ~300us(sometimes) ~4ms (more than 10 times!!)Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved37

3. Apply the integrated trace• Host events:– kvm_exit, kvm_entry, kvm_inj_virq, sched_switch(mainbuffer)– kvm_write_tsc_offset(another buffer)• Guest events:– sched_switch, sched_wakeup, sys_enter_clock_nanosleep,sys_exit_clock_nanosleep• Merge trace data of the guest and the host– When guest’s wakeup latency became delay, what washappened?Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved38

4. Analysis• Use timedoctor, which is a visualizing tool of tracing– We set a nanosleep time to 2ms.– If nanosleep time exceeds 3ms, the color will be yellow.• normally bluenanosleep_enter nanosleep_exitnanosleep_enternanosleep_exit~2.5ms~5.6mskvm_exitkvm_entryCheck timestampCopyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved39

5. Check trace data~3.7ms!!TSC=780018178690448We switched the flush process to another CPUwhich the performance is not needed.Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved40

Summary and Future work- Merging trace data of guests and a host is important foranalyzing any problems on guests.- We implemented write_tsc_offset tracepoint for mergingtrace data.- write_tsc_offset event should be enabled on a sub-buffer.- We implemented a merging tool but there are somerestrictions.- Stable TSC- Not backward TSC- Synchronized TSC- Update trace-cmd- Enable the write_tsc_offset event for the integrated traceCopyright (c) 2012 Hitachi LTD., Yokohama Research Laboratory. All right reserved41

Patch Setpatch set of outputting tsc_offset has been submittedfor LKML on November 13.http://lkml.org/lkml/2012/11/13/746Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved42

Legal statements• Linux is a registered trademark of Linus Torvalds.• UNIX is a registered trademark of The Open Group.• All other trademarks and copyrights are the property oftheir respective owners.Copyright (c) 2013 Hitachi LTD., Yokohama Research Laboratory. All right reserved44