1. technology and computing
2. hardware
3. computer components
4. chips and processors

2011
IBM Power Systems Technical University
October 10-14 | Fontainebleau Miami Beach | Miami, FL
IBM
Part 1a: Updated Concepts and Tactics -How to Monitor and Analyze the VMM and
Storage I/O Statistics of a Power/AIX LPAR
Earl Jew (earlj@us.ibm.com)
310-251-2907 cell
Senior IT Management Consultant - IBM Power Systems and IBM Systems Storage
IBM Lab Services and Training - US Power Systems (group/dept)
400 North Brand Blvd., c/o IBM 8th floor, Glendale, CA 91203 [Select IBM content added: May 10th, 2013]
© Copyright IBM Corporation 2012
Materials may not be reproduced in whole or in part without the prior written permission of IBM.
5.3
Part I: Updated Concepts and Tactics -- How to Monitor and Analyze
the VMM and Storage I/O Statistics of a Power/AIX LPAR
ABSTRACT
This presentation updates AIX/VMM (Virtual Memory Management) and LVM/JFS2
storage IO performance concepts and tactics for the day-to-day Power/AIX system
administrator. It explains the meaning of the numbers offered by AIX commands
(vmstat, iostat, mpstat, sar, etc.) to monitor and analyze the AIX VMM and storage
IO performance and capacity of a given Power7/AIX LPAR.
These tactics are further illustrated in Part II: Updated Real-world Case Histories -How to Monitor and Analyze the VMM and Storage I/O Statistics of a Power/AIX
LPAR.
© Copyright IBM Corporation 2012
2
Part II: Updated Real-world Case Histories -- How to Monitor and Analyze
the VMM and Storage I/O Statistics of a Power/AIX LPAR
ABSTRACT
These updated case-histories further illustrate the content presented in Part I:
Updated Concepts and Tactics -- How to Monitor and Analyze the VMM and Storage
I/O Statistics of a Power/AIX LPAR.
This presentation includes suggested ranges and ratios of AIX statistics to guide VMM
and storage IO performance and capacity analysis.
Each case is founded on a different real-world customer configuration and workload
that manifests characteristically in the AIX performance statistics -- as performing:
intensely in bursts, with hangs and releases, AIX:lrud constrained, AIX-buffer
constrained, freely unconstrained, inode-lock contended, consistently light,
atomic&synchronous, virtually nil IO workload, long avg-wait's, perfectly ideal, long
avg-serv's, mostly rawIO, etc.
© Copyright IBM Corporation 2012
3
Select Content Acknowledgement: IBM’s Technical Elite – ATS & EBC
• Damir Rubic, IBM Advanced Technical Skills
• Ralf Schmidt-Dannert, IBM Advanced Technical Skills
• Rebecca Ballough, IBM Advanced Technical Skills
• Steven Nasypany, IBM Advanced Technical Skills
• Dale Martin, IBM Advanced Technical Skills
• Dan Braden, IBM Advanced Technical Skills
• Patrick O’Rourke, IBM Executive Briefing Center
• Mark Olsen, WW Power Systems Offering Manager
© Copyright IBM Corporation 2012
4
Strategic Perspective: What is Workload Characterization?
• Power/AIX performance-tuning is based on continuous cycles of:
– workload characterization, i.e. monitoring for indicated issues
– implementing tactics to remedy indicated issues
• Workload characterization is determining an infrastructure’s resource capacities under load
• In other words, workload characterization examines:
– the readiness of instructions&data residing in SAN storage, main memory, Power7 L3/L2/L1 cache
– the latency&throughput of instruction&data transfers between the above, i.e. multipathing, blocked IOs
– the processing of instructions&data, i.e. CPUs simultaneously executing prioritized processes/threads
– the dynamic balance and relative exhaustion/surplus of above resources
• Workload characterization accounts an LPAR’s technology, implementation, size/count/bandwidth
– IBM Power CPU technology, i.e. Power5/5+, Power6/6+, Power7/7+
– Booted implementation, i.e. shared-pool vs dedicated CPU LPARs, SRAD affinity assignment
– Component implementation, i.e. dedicated IO adapters (traditional) vs. dual-VIOS (PowerVM), NPIV
– Size/count/bandwidth of component technologies to address the expected workload, i.e.:
•
Total LPAR gbRAM and the relative amounts of the four main sections AIX VMM memory
•
count of vCPU/eCPU/logicalCPU/FC-HBAs/LAN adapters/PCIe Gen2 slots/etc and the bandwidth of each
© Copyright IBM Corporation 2012
5
Formulations of AIX Tactics for Empirical Performance Analysis
•
This presentation will:
– explain the numbers presented by mundane AIX commands (vmstat,mpstat,iostat,ps,…)
– formulate the recognition and severity of indicated AIX performance issues hidden in these numbers
– offer tactics to remedy any indicated AIX performance issues
•
Formulated indicators in mundane AIX command output can distinguish areas of resource exhaustion, limitation and
over-commitment, as well as, resource under-utilization, surplus and over-allocation
•
Monitoring AIX: hardware->implementation->historical/accumulated stats->real-time/dynamic stats
–
–
–
–
Review component technology of the infrastructure, i.e. ensure proper tuning-by-hardware
Review implemented AIX structures, i.e. shared vs dedicated CPUs, SRADs, VIOS, NPIV, LVM/JFS2 constructs
Review historical/accumulated AIX events, usages, pendings, counts, blocks, exhaustion, etc.
Monitor real-time/dynamic AIX command behaviors, i.e. ps,vmstat,mpstat,iostat,ipcs, etc.
•
•
•
Interpret all indicators relative to the in-place technology, implementation and count/size/bandwidth of resources
Historical/cumulative indicators are judged by counts-per-scale over days-uptime since boot
Real-time/dynamic indicators are compared by ranges&ratios of system resources
•
Color-coded Severity-of-Indicators: blue/surplus, green/normal, orange/warning, red/critical
© Copyright IBM Corporation 2012
6
Considerations when Monitoring AIX Performance statistics
• Monitor dynamic AIX behaviors with 1 or 2 second sampling intervals (vs 10-600 secs.)
• Verify a stressful workload exists:
– “We can’t tune what is not being taxed”
• Discontinue active efforts when done:
– “If/when it runs fast enough, we’re tuned”
• Favor building track-able discrete structures:
– “We can’t tune what can’t be tracked”
• Discern workload spikes,peaks,bursts and burns:
– “We tune the intensities, not the sleepy-times”
• Establish dynamic baselines by monitoring real-time AIX behaviors with ranges&ratios
• Monitor AIX behaviors with the goal of characterizing the workload (vmstat –I 1)
© Copyright IBM Corporation 2012
7
A Metaphorical Palette of five observable workload characterizations
Move the Data
• More about sequential storage I/O and less about processing the data
• Proactively “tunable” workload using AIX VMM and JFS2 tactics
Think-Think
• Less about any storage I/O and more about intensive processing of the data
• More focused on SRAD, L1/L2/L3 cache, SMT-1/2/4 and eCPU/vCPU tuning tactics
Rapidly Repeatedly Doing (virtually) Nothing Until … – over&over, again&again, ad infinitum
• Polling/trapping/waiting for “something” to happen before doing real work
• A deceptive, challenging and perhaps violently-volatile workload
Rebuilding/Reburning Rome over&again
• Expends more than typical effort recreating-then-killing computational memory processes
• Legacy (curse) of running older software on any late-model computer architecture, i.e. Power7/7+
Bare-bones/comatose AIX background consciousness, i.e. no user-workload
Combinations of the above
© Copyright IBM Corporation 2012
8
Power/AIX Performance Monitoring and Tuning Tools table
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
9
Strategic Thoughts, Concepts, Considerations, and Tactics
• Monitoring AIX – Usage, Meaning and Interpretation
– Review component technology of the infrastructure, i.e. proper tuning-by-hardware
– Review implemented AIX constructs, i.e. “firm” near-static structures and settings
– Review historical/accumulated AIX events, i.e. usages, pendings, counts, blocks, etc.
– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, etc.
• Recognizing Common Performance-degrading Scenarios
– High Load Average relative to count-of-LCPUs, i.e. “over-threadedness”
– vmstat:memory:avm near-to or greater-than lruable-gbRAM, i.e. over-committed
– Continuous low vmstat:memory:fre with persistent lrud (fr:sr) activity
– Continuous high ratio of vmstat:kthr:b relative to vmstat:kthr:r
– Poor ratio of pages freed to pages examined (fr:sr ratio) in vmstat -s output
© Copyright IBM Corporation 2012
10
Strategic Thoughts, Concepts, Considerations, and Tactics
• Monitoring AIX – Usage, Meaning and Interpretation
– Review component technology of the infrastructure, i.e. proper tuning-by-hardware
• Introducing Power7+
• Active Memory Expansion (AME) enhanced with Power7+ HW-compression Accelerators
• AIX commands to review the component technology of the infrastructure
– Review implemented AIX constructs, i.e. “firm” near-static structures and settings
– Review historical/accumulated AIX events, i.e. usages, pendings, counts, blocks, etc.
– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, etc.
• Recognizing Common Performance-degrading Scenarios
© Copyright IBM Corporation 2012
11
IBM Power Processor Technology Roadmap
(content credit: IBM Executive Briefing Center)
© Copyright IBM Corporation 2012
12
Introducing POWER7+
(Content credit: IBM Executive Briefing Center)
Physical Design:
 Integrated Cache, Memory
Controllers and Accelerator
 32nm technology
Features:
 Higher Frequencies
 Larger L3 Cache
 Memory Compression Accelerator
• Active Memory Expansion
 Hardware encryption support for AIX
 Random Number Generator
 Improved RAS features
 Enhanced Energy / Power Gating
 Enhanced GX System Buses
 Enhanced Single Precision Floating
Point performance
 20 Virtual Machines per core
SMP Fabric
Core
Core
L2
L2
L3 Cache
M
C
Acc
Eng
Core
Core
L2
L2
L3 Cache
M
C
Power Bus
L3 Cache
L2
L2
Core
Core
G
X
B
u
s
L3 Cache
L2
L2
Core
Core
SMP Fabric
13
© Copyright IBM Corporation 2012
13
Transition from POWER6 to POWER7
(Content credit: IBM Executive Briefing Center)
L3
Ctrl
4 MB
L2
Memory
Cntrl
L3
Core
Alti
Core Vec
4 MB
L2
L3
Ctrl
Fabric Bus
Controller
Memory
Cntrl
Alti
Vec
GX Bus Cntrl
P
O
W
E
R
Core
Core
Core
Core
L2
L2
L2
L2
S
M
P
L3
G
X
B
U
S
L3 Cache
L2
L2
L2
L2
Core
Core
Core
Core
F
A
B
R
I
C
Memory Interface
GX+
Bridge
Memory+
Memory++
Memory+
POWER6
© Copyright IBM Corporation 2011
POWER7
© Copyright IBM Corporation 2012
14
IBM Power5Power7+ Comparative Architecture Designs
(content credit: IBM Executive Briefing Center)
© Copyright IBM Corporation 2012
15
Strategic Thoughts, Concepts, Considerations, and Tactics
• Monitoring AIX – Usage, Meaning and Interpretation
– Review component technology of the infrastructure, i.e. proper tuning-by-hardware
• Introducing Power7+
• Active Memory Expansion (AME) enhanced with Power7+ HW-compression Accelerators
• AIX commands to review the component technology of the infrastructure
– Review implemented AIX constructs, i.e. “firm” near-static structures and settings
– Review historical/accumulated AIX events, i.e. usages, pendings, counts, blocks, etc.
– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, etc.
• Recognizing Common Performance-degrading Scenarios
© Copyright IBM Corporation 2012
16
Practical Concept: Active Memory Expansion (AME)
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
17
Practical Concept: Active Memory Expansion (AME)
(content credit: IBM Executive Briefing Center)
© Copyright IBM Corporation 2012
18
Practical Concept: Active Memory Expansion (AME)
(content credit: IBM Executive Briefing Center)
© Copyright IBM Corporation 2012
19
Practical Concept: Active Memory Expansion (AME)
(content credit: IBM Executive Briefing Center)
© Copyright IBM Corporation 2012
20
Strategic Thoughts, Concepts, Considerations, and Tactics
• Monitoring AIX – Usage, Meaning and Interpretation
– Review component technology of the infrastructure, i.e. proper tuning-by-hardware
• Introducing Power7+
• Active Memory Expansion (AME) enhanced with Power7+ HW-compression Accelerators
• AIX commands to review the component technology of the infrastructure
– Review implemented AIX constructs, i.e. “firm” near-static structures and settings
– Review historical/accumulated AIX events, i.e. usages, pendings, counts, blocks, etc.
– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, etc.
• Recognizing Common Performance-degrading Scenarios
© Copyright IBM Corporation 2012
21
Note the size, scale, technology and implementation of the given LPAR
Note the LPAR’s ratio-of-resources, i.e. CPU-to-RAM-to-SAN I/O
$ date ; uname -a ; id ; oslevel –s; lparstat -i
Wed Sep 26 00:00:00 EDT 2012
AIX tsm03 1 6 00X555XX5X00
uid=0(root) gid=0(system) groups=2(bin),3(sys),7(security),8(cron),10(audit),11(lp)
6100-06-06-1140
Node Name
: tsm03
Partition Name
: TSM03
Partition Number
: 1
Type
: Shared-SMT-4
Mode
: Uncapped
Entitled Capacity
: 6.00
Partition Group-ID
: 32769
Shared Pool ID
: 0
Online Virtual CPUs
: 6
Maximum Virtual CPUs
: 7
Minimum Virtual CPUs
: 4
Online Memory
: 24064 MB
Maximum Memory
: 24064 MB
Minimum Memory
: 24064 MB
Variable Capacity Weight
: 128
Minimum Capacity
: 4.00
Maximum Capacity
: 7.00
Capacity Increment
: 0.01
Maximum Physical CPUs in system
: 16
Active Physical CPUs in system
: 16
Active CPUs in Pool
: 16
Shared Physical CPUs in system
: 16
Maximum Capacity of Pool
: 1600
Entitled Capacity of Pool
: 1600
Unallocated Capacity
: 0.00
Physical CPU Percentage
: 100.00%
Unallocated Weight
: 0
Memory Mode
: Dedicated
Total I/O Memory Entitlement
: Variable Memory Capacity Weight
: Memory Pool ID
: Physical Memory in the Pool
: Hypervisor Page Size
: Unallocated Variable Memory Capacity Weight: Unallocated I/O Memory entitlement
: Memory Group ID of LPAR
: Desired Virtual CPUs
: 6
Desired Memory
: 24064 MB
Desired Variable Capacity Weight
: 128
© Copyright IBM Corporation 2012
Desired Capacity
: 6.00
22
prtconf # note the component technology of the given LPAR
$ prtconf
System Model: IBM,8233-E8B
Machine Serial Number: 5555XXX
Processor Type: PowerPC_POWER7
Processor Implementation Mode: POWER 7
Processor Version: PV_7_Compat
Number Of Processors: 6
Processor Clock Speed: 3300 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
LPAR Info: 1 TSM03
Memory Size: 24064 MB
Good Memory Size: 24064 MB
Platform Firmware level: AL710_065
Firmware Version: IBM,AL710_065
Console Login: enable
Auto Restart: true
Full Core: false
Network Information
Host Name: tsm03
IP Address: 111.222.33.44
Sub Netmask: 255.255.255.128
Gateway: 111.222.33.1
Name Server: 111.222.166.17
Domain Name: customer.com
Paging Space Information
Total Paging Space: 60672MB
Percent Used: 24%
Volume Groups Information
==============================================================================
Inactive VGs
==============================================================================
heartbeat_vg
==============================================================================
Active VGs
==============================================================================
tsm_vg:
PV_NAME
PV STATE
TOTAL PPs
FREE PPs
FREE DISTRIBUTION
hdiskpower57
active
99
0
00..00..00..00..00
hdiskpower8
active
9
0
00..00..00..00..00
…
© Copyright IBM Corporation 2012
23
lscfg # note the placement of components in the implementation of the LPAR
$ lscfg
INSTALLED RESOURCE LIST
The following resources are installed on the machine.
+/- = Added or deleted from Resource List.
*
= Diagnostic support not available.
Model Architecture: chrp
Model Implementation: Multiple Processor, PCI bus
+
+
*
*
*
*
*
*
*
*
*
*
*
+
+
*
+
*
+
*
*
*
+
+
*
*
*
*
*
*
…
sys0
sysplanar0
vio0
vscsi2
vscsi1
vscsi0
hdisk3
hdisk2
hdisk1
hdisk0
vsa0
vty0
pci5
ent0
ent1
pci4
fcs6
fcnet6
fscsi6
sfwcomm6
rmt156
rmt157
fcs7
fscsi7
rmt74
rmt75
rmt76
rmt77
rmt78
rmt79
U8233.E8B.1009ADP-V1-C5-T1
U8233.E8B.1009ADP-V1-C3-T1
U8233.E8B.1009ADP-V1-C2-T1
U8233.E8B.1009ADP-V1-C2-T1-L8400000000000000
U8233.E8B.1009ADP-V1-C2-T1-L8300000000000000
U8233.E8B.1009ADP-V1-C2-T1-L8200000000000000
U8233.E8B.1009ADP-V1-C2-T1-L8100000000000000
U8233.E8B.1009ADP-V1-C0
U8233.E8B.1009ADP-V1-C0-L0
U5802.001.00H2615-P1
U5802.001.00H2615-P1-C6-T1
U5802.001.00H2615-P1-C6-T2
U5802.001.00H2615-P1
U5802.001.00H2615-P1-C5-T1
U5802.001.00H2615-P1-C5-T1
U5802.001.00H2615-P1-C5-T1
U5802.001.00H2615-P1-C5-T1-W0-L0
U5802.001.00H2615-P1-C5-T1-W500308C0022DD803-L0
U5802.001.00H2615-P1-C5-T1-W500308C0022DD803-L1000000000000
U5802.001.00H2615-P1-C5-T2
U5802.001.00H2615-P1-C5-T2
U5802.001.00H2615-P1-C5-T2-W21000024FF31B5B1-L9000000000000
U5802.001.00H2615-P1-C5-T2-W21000024FF31B5B1-LA000000000000
U5802.001.00H2615-P1-C5-T2-W21000024FF31B5B1-LB000000000000
U5802.001.00H2615-P1-C5-T2-W21000024FF31B5B1-LC000000000000
U5802.001.00H2615-P1-C5-T2-W21000024FF31B5B1-LD000000000000
U5802.001.00H2615-P1-C5-T2-W21000024FF31B5B1-LE000000000000
System Object
System Planar
Virtual I/O Bus
Virtual SCSI Client Adapter
Virtual SCSI Client Adapter
Virtual SCSI Client Adapter
Virtual SCSI Disk Drive
Virtual SCSI Disk Drive
Virtual SCSI Disk Drive
Virtual SCSI Disk Drive
LPAR Virtual Serial Adapter
Asynchronous Terminal
PCI Express Bus
2-Port 10/100/1000 Base-TX PCI-Express Ada
2-Port 10/100/1000 Base-TX PCI-Express Ada
PCI Express Bus
8Gb PCI Express Dual Port FC Adapter (df10
Fibre Channel Network Protocol Device
FC SCSI I/O Controller Protocol Device
Fibre Channel Storage Framework Comm
LTO Ultrium Tape Drive (FCP)
LTO Ultrium Tape Drive (FCP)
8Gb PCI Express Dual Port FC Adapter (df10
FC SCSI I/O Controller Protocol Device
LTO Ultrium Tape Drive (FCP)
LTO Ultrium Tape Drive (FCP)
LTO Ultrium Tape Drive (FCP)
LTO Ultrium Tape Drive (FCP)
LTO Ultrium Tape Drive (FCP)
LTO Ultrium Tape Drive (FCP)
© Copyright IBM Corporation 2012
24
lsdev note the count&capacity of the component technology of the LPAR
$ lsdev
L2cache0
cd0
en0
en1
en2
en3
en4
en5
ent0
ent1
ent2
ent3
et0
et1
et2
et3
et4
et5
fcnet0
fcnet1
fcnet2
fcnet3
fcnet4
fcnet5
fcnet6
fcnet7
fcs0
fcs1
fcs2
fcs3
fcs4
fcs5
fcs6
fcs7
fscsi0
fscsi1
fscsi2
fscsi3
fscsi4
fscsi5
fscsi6
fscsi7
hba0
…
Available
Defined
Defined
Defined
Defined
Available
Defined
Defined
Available
Available
Available
Available
Defined
Defined
Defined
Defined
Defined
Defined
Defined
Defined
Defined
Defined
Defined
Defined
Defined
Defined
Available
Available
Available
Available
Available
Available
Available
Available
Available
Available
Available
Available
Available
Available
Available
Available
Available
05-00
05-01
00-00-00
00-01
07-00-00
05-00
05-01
00-00-00
05-00
05-01
00-00-00
00-01
07-00-00
01-00-02
01-01-01
03-00-01
03-01-02
04-00-02
04-01-01
02-00-01
02-01-02
01-00
01-01
03-00
03-01
04-00
04-01
02-00
02-01
01-00-01
01-01-02
03-00-02
03-01-01
04-00-01
04-01-02
02-00-02
02-01-01
00-00
L2 Cache
Virtual SCSI Optical Served by VIO Server
Standard Ethernet Network Interface
Standard Ethernet Network Interface
Standard Ethernet Network Interface
Standard Ethernet Network Interface
Standard Ethernet Network Interface
Standard Ethernet Network Interface
2-Port 10/100/1000 Base-TX PCI-Express Adapter (14104003)
2-Port 10/100/1000 Base-TX PCI-Express Adapter (14104003)
10 Gigabit Ethernet Adapter (ct3)
EtherChannel / IEEE 802.3ad Link Aggregation
IEEE 802.3 Ethernet Network Interface
IEEE 802.3 Ethernet Network Interface
IEEE 802.3 Ethernet Network Interface
IEEE 802.3 Ethernet Network Interface
IEEE 802.3 Ethernet Network Interface
IEEE 802.3 Ethernet Network Interface
Fibre Channel Network Protocol Device
Fibre Channel Network Protocol Device
Fibre Channel Network Protocol Device
Fibre Channel Network Protocol Device
Fibre Channel Network Protocol Device
Fibre Channel Network Protocol Device
Fibre Channel Network Protocol Device
Fibre Channel Network Protocol Device
8Gb PCI Express Dual Port FC Adapter (df1000f114108a03)
8Gb PCI Express Dual Port FC Adapter (df1000f114108a03)
8Gb PCI Express Dual Port FC Adapter (df1000f114108a03)
8Gb PCI Express Dual Port FC Adapter (df1000f114108a03)
8Gb PCI Express Dual Port FC Adapter (df1000f114108a03)
8Gb PCI Express Dual Port FC Adapter (df1000f114108a03)
8Gb PCI Express Dual Port FC Adapter (df1000f114108a03)
8Gb PCI Express Dual Port FC Adapter (df1000f114108a03)
FC SCSI I/O Controller Protocol Device
FC SCSI I/O Controller Protocol Device
FC SCSI I/O Controller Protocol Device
FC SCSI I/O Controller Protocol Device
FC SCSI I/O Controller Protocol Device
FC SCSI I/O Controller Protocol Device
FC SCSI I/O Controller Protocol Device
FC SCSI I/O Controller Protocol Device
10 Gigabit Ethernet-SR PCI-Express Host Bus Adapter (2514300014108c03)
© Copyright IBM Corporation 2012
25
Strategic Thoughts, Concepts, Considerations, and Tactics
• Monitoring AIX – Usage, Meaning and Interpretation
– Review component technology of the infrastructure, i.e. proper tuning-by-hardware
– Review implemented AIX constructs, i.e. “firm” near-static structures and settings
– Review historical/accumulated AIX events, i.e. usages, pendings, counts, blocks, etc.
– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, etc.
• Recognizing Common Performance-degrading Scenarios
– High Load Average relative to count-of-LCPUs, i.e. “over-threadedness”
– vmstat:memory:avm near-to or greater-than lruable-gbRAM, i.e. over-committed
– Continuous low vmstat:memory:fre with persistent lrud (fr:sr) activity
– Continuous high ratio of vmstat:kthr:b relative to vmstat:kthr:r
– Poor ratio of pages freed to pages examined (fr:sr ratio) in vmstat -s output
© Copyright IBM Corporation 2012
26
Strategic Thoughts, Concepts, Considerations, and Tactics
• Monitoring AIX – Usage, Meaning and Interpretation
– Review component technology of the infrastructure, i.e. proper tuning-by-hardware
– Review implemented AIX constructs, i.e. “firm” near-static structures and settings
•
•
•
•
Power CPU’s PVL Hierarchy (Physical, Virtual and Logical)
“Socketizing Workloads” (p770, p780, p795)
Local, Near and Far memory access
How to interpret lssrad output
•
•
•
•
•
Four Main Sections of AIX VMM System Memory
AIX VMM parameter tuning (AIX 7.1, AIX 6.1, AIX 5.3)
AIX VMM automatic multiple page size support
Data Layout for Optimal I/O Performance
AIX commands to review the implemented AIX constructs, structures and settings
– Review historical/accumulated AIX events, i.e. usages, pendings, counts, blocks, etc.
– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, etc.
• Recognizing Common Performance-degrading Scenarios
© Copyright IBM Corporation 2012
27
Practical Concept: Hierarchy of PVL=Physical, Virtual and Logical CPUs
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
28
Strategic Thoughts, Concepts, Considerations, and Tactics
• Monitoring AIX – Usage, Meaning and Interpretation
– Review component technology of the infrastructure, i.e. proper tuning-by-hardware
– Review implemented AIX constructs, i.e. “firm” near-static structures and settings
•
•
•
•
Power CPU’s PVL Hierarchy (Physical, Virtual and Logical)
“Socketizing Workloads” (p770, p780, p795)
Local, Near and Far memory access
How to interpret lssrad output
•
•
•
•
•
Four Main Sections of AIX VMM System Memory
AIX VMM parameter tuning (AIX 7.1, AIX 6.1, AIX 5.3)
AIX VMM automatic multiple page size support
Data Layout for Optimal I/O Performance
AIX commands to review the implemented AIX constructs, structures and settings
– Review historical/accumulated AIX events, i.e. usages, pendings, counts, blocks, etc.
– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, etc.
• Recognizing Common Performance-degrading Scenarios
© Copyright IBM Corporation 2012
29
Practical Concept: “Socketizing” Workloads (p770, p780, p795)
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
30
Strategic Thoughts, Concepts, Considerations, and Tactics
• Monitoring AIX – Usage, Meaning and Interpretation
– Review component technology of the infrastructure, i.e. proper tuning-by-hardware
– Review implemented AIX constructs, i.e. “firm” near-static structures and settings
•
•
•
•
Power CPU’s PVL Hierarchy (Physical, Virtual and Logical)
“Socketizing Workloads” (p770, p780, p795)
Local, Near and Far memory access
How to interpret lssrad output
•
•
•
•
•
•
•
Four Main Sections of AIX VMM System Memory
IBM PowerVM Virtualization Architecture
AIX VMM parameter tuning (AIX 7.1, AIX 6.1, AIX 5.3)
AIX VMM automatic multiple page size support
Virtual Processor “Folding” mechanism
Data Layout for Optimal I/O Performance
AIX commands to review the implemented AIX constructs, structures and settings
– Review historical/accumulated AIX events, i.e. usages, pendings, counts, blocks, etc.
– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, etc.
• Recognizing Common Performance-degrading Scenarios
© Copyright IBM Corporation 2012
31
Strategic Thoughts, Concepts, Considerations, and Tactics
• Monitoring AIX – Usage, Meaning and Interpretation
– Review component technology of the infrastructure, i.e. proper tuning-by-hardware
– Review implemented AIX constructs, i.e. “firm” near-static structures and settings
•
•
•
•
Power CPU’s PVL Hierarchy (Physical, Virtual and Logical)
“Socketizing Workloads” (p770, p780, p795)
Local, Near and Far memory access
How to interpret lssrad output
•
•
•
•
•
Four Main Sections of AIX VMM System Memory
AIX VMM parameter tuning (AIX 7.1, AIX 6.1, AIX 5.3)
AIX VMM automatic multiple page size support
Data Layout for Optimal I/O Performance
AIX commands to review the implemented AIX constructs, structures and settings
– Review historical/accumulated AIX events, i.e. usages, pendings, counts, blocks, etc.
– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, etc.
• Recognizing Common Performance-degrading Scenarios
© Copyright IBM Corporation 2012
32
Practical Concept: Local, Near and Far memory access
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
33
Practical Concept: Local, Near and Far memory access
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
34
Practical Concept: Local, Near and Far memory access
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
35
Practical Concept: Displaying the Local, Near, Far memory access profile
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
36
Strategic Thoughts, Concepts, Considerations, and Tactics
• Monitoring AIX – Usage, Meaning and Interpretation
– Review component technology of the infrastructure, i.e. proper tuning-by-hardware
– Review implemented AIX constructs, i.e. “firm” near-static structures and settings
•
•
•
•
Power CPU’s PVL Hierarchy (Physical, Virtual and Logical)
“Socketizing Workloads” (p770, p780, p795)
Local, Near and Far memory access
How to interpret lssrad output
•
•
•
•
•
Four Main Sections of AIX VMM System Memory
AIX VMM parameter tuning (AIX 7.1, AIX 6.1, AIX 5.3)
AIX VMM automatic multiple page size support
Data Layout for Optimal I/O Performance
AIX commands to review the implemented AIX constructs, structures and settings
– Review historical/accumulated AIX events, i.e. usages, pendings, counts, blocks, etc.
– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, etc.
• Recognizing Common Performance-degrading Scenarios
© Copyright IBM Corporation 2012
37
Practical Concept: How to interpret lssrad output
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
38
Practical Concept: How to interpret lssrad output
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
39
Strategic Thoughts, Concepts, Considerations, and Tactics
• Monitoring AIX – Usage, Meaning and Interpretation
– Review component technology of the infrastructure, i.e. proper tuning-by-hardware
– Review implemented AIX constructs, i.e. “firm” near-static structures and settings
•
•
•
•
Power CPU’s PVL Hierarchy (Physical, Virtual and Logical)
“Socketizing Workloads” (p770, p780, p795)
Local, Near and Far memory access
How to interpret lssrad output
•
•
•
•
•
Four Main Sections of AIX VMM System Memory
AIX VMM parameter tuning (AIX 7.1, AIX 6.1, AIX 5.3)
AIX VMM automatic multiple page size support
Data Layout for Optimal I/O Performance
AIX commands to review the implemented AIX constructs, structures and settings
– Review historical/accumulated AIX events, i.e. usages, pendings, counts, blocks, etc.
– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, etc.
• Recognizing Common Performance-degrading Scenarios
© Copyright IBM Corporation 2012
40
Practical Concept: Four Main Sections of AIX VMM System Memory
• AIX itself and its set of buffers and other VMM-related structures
–
–
–
–
pbuf’s: allocated per hdisk/LUN and pooled in the LVM:VolumeGroup
fsbuf’s: allocated per JFS/JFS2 file system
psbuf’s: fixed allocation for pagingspace-paging I/O
nfsbuf’s: dynamically allocated for NFS I/O
– mbuf’s: dynamically allocated for network I/O
• Computational memory% – Ideal: up to 60%; acceptable tolerance: up to 70%
– User workload, i.e. application/rdbms binaries, processes&threads, shmemsegs, buffers
– Client TCP/IP connections, i.e. the count and size of each in resident memory (RSS)
• Non-computational memory% – Somewhat about [100 - (comp%)]
– JFS buffer cache
– NFS/GPFS/VxFS buffer cache
– JFS2 buffer cache
• Free Memory – Ideal: midrange-5 digits of freemem (fre) (can be up to low-6 digits)
© Copyright IBM Corporation 2012
41
Practical Concept: Four Main Sections of AIX VMM memory
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
42
What is Computational memory? What is File memory (aka Non-Computational memory)
Computational memory
Computational memory is used while your processes are actually working on computing information. These working segments are
temporary (transitory) and only exist up until the time a process terminates or the page is stolen. They have no real permanent
disk storage location. When a process terminates, both the physical and paging spaces are released in many cases. When
there is a large spike in available pages, you can actually see this happening while monitoring your system. When free physical
memory starts getting low, programs that have not used recently are moved from RAM to paging space to help release physical
memory for more real work.
File memory (aka Non-Computational memory)
File memory (unlike computational memory) uses persistent segments and has a permanent storage location on the disk. Data
files or executable programs are mapped to persistent segments rather than working segments. The data files can relate to
filesystems, such as JFS, JFS2, or NFS. They remain in memory until the file is unmounted, a page is stolen, or a file is
unlinked. After the data file is copied into RAM, VMM controls when these pages are overwritten or used to store other data.
Given the alternative, most people would much rather have file memory paged to disk rather than computational memory.
When a process references a page which is on disk, it must be paged, which could cause other pages to page out again. VMM is
constantly lurking and working in the background trying to steal frames that have not been recently referenced, using the page
replacement algorithm discussed earlier. It also helps detect thrashing, which can occur when memory is extremely low and
pages are constantly being paged in and out to support processing. VMM actually has a memory load control algorithm, which
can detect if the system is thrashing and actually tries to remedy the situation. Unabashed thrashing can literally cause a
system to come to a standstill, as the kernel becomes too concerned with making room for pages than actually doing anything
productive.
Source verbatim: Ken Milberg/Martin Brown http://www.ibm.com/developerworks/aix/library/au-aix7memoryoptimize1/index.html
© Copyright IBM Corporation 2012
43
Criteria for Creating a Write-Expedient pagingspace_vg
The first priority should be to preclude any pagingspace-pageouts. Thus, a write-expedient pagingspace is only needed
if you have any unavoidable pagingspace-pageout activity. Ultimately, if we must suffer any pagingspace-pageouts,
we want them to write-out to the pagingspace as quickly as possible (thus my term: write-expedient).
So, for the sake of prudence, we should always create a write-expedient pagingspace. The listed traits below are
optimal for write-expediency; include as many as you can (but always apply the key tuning tactic below):
•
Create a dedicated AIX:LVM:vg (VolumeGroup) called pagingspace_vg
•
Create the pagingspace_vg using FC-SAN storage LUNs (ideally RAID5 LUNs on SSD, FC or SAS technology disk drives,
and not on SATA disk drives (which are slower and employs RAID6), nor on any local/internal SAS disks)
•
The total size of the pagingspace in pagingspace_vg should match the size of installed LPAR gbRAM
•
Assign 3-to-8 LUN/hdisks to pagingspace_vg and size each LUN to be an even fraction of installed gbRAM. For instance, if
the LPAR has 18gbRAM, then assign three 6gb LUN/hdisks to pagingspace_vg
•
Configure one AIX:LVM:VG:lv (logical volume) for each LUN/hdisk in pagingspace_vg; do not deploy PP-striping
(because it messes-up discrete hdisk IO monitoring) –- just map one hdisk to one lv
•
The key tuning tactic: With root-user privileges, use AIX:lvmo to set pagingspace_vg:pv_pbuf_count=2048.
This will ensure pagingspace_vg:total_vg_pbufs will equal [<VGLUNcount> * pv_pbuf_count].
•
To set the pv_pbuf_count value to 2048, type the following:
lvmo -v pagingspace_vg -o pv_pbuf_count=2048
© Copyright IBM Corporation 2012
44
Strategic Thoughts, Concepts, Considerations, and Tactics
• Monitoring AIX – Usage, Meaning and Interpretation
– Review component technology of the infrastructure, i.e. proper tuning-by-hardware
– Review implemented AIX constructs, i.e. “firm” near-static structures and settings
•
•
•
•
Power CPU’s PVL Hierarchy (Physical, Virtual and Logical)
“Socketizing Workloads” (p770, p780, p795)
Local, Near and Far memory access
How to interpret lssrad output
•
•
•
•
•
Four Main Sections of AIX VMM System Memory
AIX VMM parameter tuning (AIX 7.1, AIX 6.1, AIX 5.3)
AIX VMM automatic multiple page size support
Data Layout for Optimal I/O Performance
AIX commands to review the implemented AIX constructs, structures and settings
– Review historical/accumulated AIX events, i.e. usages, pendings, counts, blocks, etc.
– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, etc.
• Recognizing Common Performance-degrading Scenarios
© Copyright IBM Corporation 2012
45
Practical Concept: AIX VMM Tuning – Page Replacement
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
46
Practical Concept: vmo tunable – page_steal_method
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
47
Practical Concept: AIX VMM parameter tuning (AIX 7.1, AIX 6.1, AIX 5.3)
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
48
Strategic Thoughts, Concepts, Considerations, and Tactics
• Monitoring AIX – Usage, Meaning and Interpretation
– Review component technology of the infrastructure, i.e. proper tuning-by-hardware
– Review implemented AIX constructs, i.e. “firm” near-static structures and settings
•
•
•
•
Power CPU’s PVL Hierarchy (Physical, Virtual and Logical)
“Socketizing Workloads” (p770, p780, p795)
Local, Near and Far memory access
How to interpret lssrad output
•
•
•
•
•
Four Main Sections of AIX VMM System Memory
AIX VMM parameter tuning (AIX 7.1, AIX 6.1, AIX 5.3)
AIX VMM automatic multiple page size support
Data Layout for Optimal I/O Performance
AIX commands to review the implemented AIX constructs, structures and settings
– Review historical/accumulated AIX events, i.e. usages, pendings, counts, blocks, etc.
– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, etc.
• Recognizing Common Performance-degrading Scenarios
© Copyright IBM Corporation 2012
49
Practical Concept: AIX VMM automatic multiple page size support
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
50
Practical Concept: AIX VMM multiple page size support table
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
51
Strategic Thoughts, Concepts, Considerations, and Tactics
• Monitoring AIX – Usage, Meaning and Interpretation
– Review component technology of the infrastructure, i.e. proper tuning-by-hardware
– Review implemented AIX constructs, i.e. “firm” near-static structures and settings
•
•
•
•
Power CPU’s PVL Hierarchy (Physical, Virtual and Logical)
“Socketizing Workloads” (p770, p780, p795)
Local, Near and Far memory access
How to interpret lssrad output
•
•
•
•
•
Four Main Sections of AIX VMM System Memory
AIX VMM parameter tuning (AIX 7.1, AIX 6.1, AIX 5.3)
AIX VMM automatic multiple page size support
Data Layout for Optimal I/O Performance
AIX commands to review the implemented AIX constructs, structures and settings
– Review historical/accumulated AIX events, i.e. usages, pendings, counts, blocks, etc.
– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, etc.
• Recognizing Common Performance-degrading Scenarios
© Copyright IBM Corporation 2012
52
Practical Concept: Data Layout for Optimal I/O Performance
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
53
Practical Concept: The AIX I/O Stack
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
54
Practical Concept: Data Layout for Optimal I/O Performance
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
55
Practical Concept: Data Layout for Optimal I/O Performance
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
56
Practical Concept: Data Layout for Optimal I/O Performance
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
57
Practical Concept: Characterization of Application IO
(content credit: IBM ATS team)
© Copyright IBM Corporation 2012
58
Strategic Thoughts, Concepts, Considerations, and Tactics
• Monitoring AIX – Usage, Meaning and Interpretation
– Review component technology of the infrastructure, i.e. proper tuning-by-hardware
– Review implemented AIX constructs, i.e. “firm” near-static structures and settings
•
•
•
•
Power CPU’s PVL Hierarchy (Physical, Virtual and Logical)
“Socketizing Workloads” (p770, p780, p795)
Local, Near and Far memory access
How to interpret lssrad output
•
•
•
•
•
Four Main Sections of AIX VMM System Memory
AIX VMM parameter tuning (AIX 7.1, AIX 6.1, AIX 5.3)
AIX VMM automatic multiple page size support
Data Layout for Optimal I/O Performance
AIX commands to review the implemented AIX constructs, structures and settings
– Review historical/accumulated AIX events, i.e. usages, pendings, counts, blocks, etc.
– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, etc.
• Recognizing Common Performance-degrading Scenarios
© Copyright IBM Corporation 2012
59
lsps ; mount ; df -k review the implemented construction of “firm” AIX structures
$ lsps –a ; lsps –s ; mount ; df -k
Page Space
Physical Volume
Volume Group Size %Used Active Auto Type Chksum
paging02
hdisk2
paging_vg
9216MB
39
yes
yes
lv
0
paging01
hdisk2
paging_vg
24576MB
15
yes
yes
lv
0
paging00
hdisk2
paging_vg
16384MB
22
yes
yes
lv
0
hd6
hdisk0
rootvg
10496MB
35
yes
yes
lv
0
Total Paging Space
Percent Used
60672MB
24%
node
mounted
mounted over
vfs
date
options
-------- --------------- --------------- ------ ------------ --------------/dev/hd4
/
jfs2
Sep 07 17:03 rw,log=/dev/hd8
/dev/hd2
/usr
jfs2
Sep 07 17:03 rw,log=/dev/hd8
/dev/hd9var
/var
jfs2
Sep 07 17:03 rw,log=/dev/hd8
/dev/hd3
/tmp
jfs2
Sep 07 17:03 rw,log=/dev/hd8
/dev/hd1
/home
jfs2
Sep 07 17:07 rw,log=/dev/hd8
/dev/hd11admin
/admin
jfs2
Sep 07 17:07 rw,log=/dev/hd8
/proc
/proc
procfs Sep 07 17:07 rw
/dev/hd10opt
/opt
jfs2
Sep 07 17:07 rw,log=/dev/hd8
/dev/livedump
/var/adm/ras/livedump jfs2
Sep 07 17:07 rw,log=/dev/hd8
/dev/install_sw_lv /install_sw
jfs2
Sep 07 17:07 rw,log=/dev/hd8
/dev/tsmlib1_lv /tsm/db2lib1
jfs2
Sep 07 17:22 rw,log=INLINE
/dev/tsm_db_lv /tsm/tsm
jfs2
Sep 07 17:22 rw,log=INLINE
/dev/tsm_arc_lv /tsm/tsm/arch jfs2
Sep 07 17:22 rw,log=INLINE
/dev/tsm_dat01_lv /tsm/tsm/data01 jfs2
Sep 07 17:22 rw,log=INLINE
/dev/tsm_dat02_lv /tsm/tsm/data02 jfs2
Sep 07 17:22 rw,log=INLINE
/dev/tsm_dat03_lv /tsm/tsm/data03 jfs2
Sep 07 17:22 rw,log=INLINE
/dev/tsm_lg_lv /tsm/tsm/log jfs2
Sep 07 17:22 rw,log=INLINE
/dev/lv01
/tsm/tsmb
jfs2
Sep 07 17:22 rw,log=INLINE
Filesystem
1024-blocks
Free %Used
Iused %Iused Mounted on
/dev/hd4
3145728
2605152
18%
31050
5% /
/dev/hd2
4390912
581548
87%
64251
29% /usr
/dev/hd9var
2097152
78452
97%
9844
24% /var
/dev/hd3
2097152
1035572
51%
2530
2% /tmp
/dev/hd1
1048576
250468
77%
1198
3% /home
/dev/hd11admin
131072
130692
1%
5
1% /admin
/proc
- /proc
/dev/hd10opt
5242880
1815992
66%
26774
6% /opt
/dev/livedump
262144
255344
3%
31
1% /var/adm/ras/livedump
/dev/install_sw_lv
20971520
7548932
65%
7944
1% /install_sw
/dev/tsmlib1_lv
51380224 21353496
59%
1818
1% /tsm/db2lib1
/dev/tsm_db_lv
513802240 209820276
60%
1695
1% /tsm/tsm
/dev/tsm_arc_lv
102760448 74128676
28%
73
1% /tsm/tsm/arch
/dev/tsm_dat01_lv
519045120
6434120
99%
25
1% /tsm/tsm/data01
/dev/tsm_dat02_lv
519045120 32034120
94%
23
1% /tsm/tsm/data02
…
© Copyright IBM Corporation 2012
60
df -k review the implemented construction of “firm” AIX structures;
observe count-of-inodes per GBs(used) of each application’s data filesystems
$ df -k
Filesystem
1024-blocks
Free %Used
Iused %Iused Mounted on
/dev/hd4
262144
129016
51%
3777
3% /
/dev/hd2
3932160
544280
87%
42721
5% /usr
/dev/hd9var
1048576
334980
69%
4293
2% /var
/dev/hd3
1048576
731832
31%
519
1% /tmp
/dev/hd1
262144
63632
76%
2622
5% /home
/proc
- /proc
/dev/hd10opt
262144
213832
19%
849
2% /opt
/dev/lvsapcds
2097152
456840
79%
1246
2% /sapcds
/dev/lvcnvbt
20480000 16993664
18%
715
1% /cnv
/dev/lvhrtmpbt
524288
506984
4%
30
1% /hrtmp
/dev/lvoraclebt
524288
436808
17%
2938
3% /oracle
/dev/lvorapr1bt
8978432
3838252
58%
21476
3% /oracle/PR1
/dev/lvmirrlogAp
3080192
2567348
17%
6
1% /oracle/PR1/mirrlogA
/dev/lvmirrlogBp
3080192
2567348
17%
6
1% /oracle/PR1/mirrlogB
/dev/lvoriglogAp
3080192
2567348
17%
6
1% /oracle/PR1/origlogA
/dev/lvoriglogBp
3080192
2567348
17%
6
1% /oracle/PR1/origlogB
/dev/lvsaparchbt
14680064 14296480
3%
7176
1% /oracle/PR1/saparch
/dev/lvsapdata1bt
268173312 73734764
73%
116
1% /oracle/PR1/sapdata1
/dev/lvsapdata18bt
268173312 73751196
73%
108
1% /oracle/PR1/sapdata10
/dev/lvsapdata11bt
268173312 77027948
72%
108
1% /oracle/PR1/sapdata11
/dev/lvsapdata24bt
268173312 75455208
72%
108
1% /oracle/PR1/sapdata12
/dev/lvsapdata2bt
268173312 76225148
72%
110
1% /oracle/PR1/sapdata2
/dev/lvsapdata3bt
268173312 75569716
72%
110
1% /oracle/PR1/sapdata3
/dev/lvsapdata14bt
268173312 74930816
73%
108
1% /oracle/PR1/sapdata4
/dev/lvsapdata23bt
268173312 77814376
71%
108
1% /oracle/PR1/sapdata5
/dev/lvsapdata16bt
268173312 79387368
71%
108
1% /oracle/PR1/sapdata6
/dev/lvsapdata7bt
268173312 74013420
73%
108
1% /oracle/PR1/sapdata7
/dev/lvsapdata8bt
268173312 75192876
72%
108
1% /oracle/PR1/sapdata8
/dev/lvsapdata19bt
268173312 74668728
73%
108
1% /oracle/PR1/sapdata9
/dev/lvsapreorgbt
25165824 19272876
24%
1153
1% /oracle/PR1/sapreorg
/dev/lvostage
2097152
1957092
7%
794
1% /oracle/stage
/dev/lvsapmntbt
2097152
1447736
31%
357
1% /sapmnt/PR1
…
…
© Copyright IBM Corporation 2012
61
ipcs -bm review the implemented construction of “firm” AIX structures;
computational memory includes allocated (vs authorized) shmemsegs
$ ipcs -bm
IPC status from /dev/mem as of Wed Sep 26 00:01:26 EDT 2012
T
ID
KEY
MODE
OWNER
GROUP
SEGSZ
Shared Memory:
m
1048576 0x78000166 --rw-rw-rwroot
system 33554432
m
1048577 0x7800010b --rw-rw-rwroot
system 33554432
m
1048578 0x21002002 --rw------- pconsole
system 10485760
m
3 0x6700b061 --rw-r--r-root
system
12
m
4 0x6800b061 --rw-r--r-root
system
377016
m
5 0x7000b061 --rw------root
system
3168
m 23068678 0xa7067574 --rw-rw-rw- db2prd1 db2srvrs 140871904
m
9437191 0xffffffff --rw------- db2lib1 db2srvrs 268435456
m 15728648 0xffffffff --rw------- db2prd1 db2srvrs 16106127360
m 10485770 0xffffffff --rw------- db2lib1 db2srvrs 3758096384
m 35651595 0xa7067561 --rw------- db2prd1 db2srvrs 51511296
m 14680076 0xffffffff --rw------- db2lib1 db2srvrs
131072
m
6291470 0x1b7fa074 --rw-rw-rw- db2lib1 db2srvrs 140871904
m 12582927 0xffffffff --rw------- db2lib1 db2srvrs 163905536
m
8388624 0xffffffff --rw------- db2prd1 db2srvrs 268435456
m
17 0xa7067668 --rw-rw---- db2prd1 db2srvrs 50331648
m 73400338 0x1b7fa168 --rw-rw---- db2lib1 db2srvrs 50331648
m 20971539 0xffffffff --rw------- db2prd1 db2srvrs 163905536
m
6291476 0xffffffff --rw------- db2prd1 db2srvrs
131072
m 13631509 0x1b7fa061 --rw------- db2lib1 db2srvrs 51511296
m 26214422 0xffffffff --rw------- db2prd1 db2srvrs 268435456
m 111149079 0xffffffff --rw------- db2prd1 db2srvrs 268435456
m 89128984 0xffffffff --rw------- db2lib1 db2srvrs 268435456
m 1067450393 0xffffffff --rw------- db2prd1 db2srvrs 268435456
m 115343386 0xffffffff --rw------- db2prd1 db2srvrs 268435456
m 894435355 0xffffffff --rw------- db2prd1 db2srvrs 268435456
m 311427100 0xffffffff --rw------- db2lib1 db2srvrs 268435456
m 371195933 0xffffffff --rw------- db2prd1 db2srvrs 268435456
m 547356703 0xffffffff --rw------- db2prd1 db2srvrs
131072
m 569376800 0xffffffff --rw------- db2prd1 db2srvrs
131072
m 576716833 0xffffffff --rw------- db2lib1 db2srvrs 268435456
…
© Copyright IBM Corporation 2012
62
vmo –L ; ioo -L # review the implemented construction of “firm” AIX structures
# vmo –L ; ioo –L
NAME
CUR
DEF
BOOT
MIN
MAX
UNIT
TYPE
DEPENDENCIES
-------------------------------------------------------------------------------ame_cpus_per_pool
n/a
8
8
1
1K
processors
B
-------------------------------------------------------------------------------ame_maxfree_mem
n/a
24M
24M
320K
16G
bytes
D
ame_minfree_mem
-------------------------------------------------------------------------------ame_min_ucpool_size
n/a
0
0
5
95
% memory
D
-------------------------------------------------------------------------------ame_minfree_mem
n/a
8M
8M
64K
16383M bytes
D
ame_maxfree_mem
-------------------------------------------------------------------------------ams_loan_policy
n/a
1
1
0
2
numeric
D
-------------------------------------------------------------------------------enhanced_affinity_affin_time
1
1
1
0
100
numeric
D
-------------------------------------------------------------------------------enhanced_affinity_vmpool_limit
10
10
10
-1
100
numeric
D
-------------------------------------------------------------------------------esid_allocator
0
0
0
0
1
boolean
D
-------------------------------------------------------------------------------force_relalias_lite
0
0
0
0
1
boolean
D
-------------------------------------------------------------------------------kernel_heap_psize
0
0
0
0
16M
bytes
B
-------------------------------------------------------------------------------lgpg_regions
0
0
0
0
8E-1
D
lgpg_size
-------------------------------------------------------------------------------…
NAME
CUR
DEF
BOOT
MIN
MAX
UNIT
TYPE
DEPENDENCIES
-------------------------------------------------------------------------------aio_active
1
1
boolean
S
-------------------------------------------------------------------------------aio_maxreqs
64K
64K
64K
4K
1M
numeric
D
-------------------------------------------------------------------------------aio_maxservers
30
30
30
1
20000 numeric
D
aio_minservers
-------------------------------------------------------------------------------aio_minservers
3
3
3
0
20000 numeric
D
aio_maxservers
------------------------------------------------ © Copyright IBM Corporation 2012
63
Strategic Thoughts, Concepts, Considerations, and Tactics
• Monitoring AIX – Usage, Meaning and Interpretation
– Review component technology of the infrastructure, i.e. proper tuning-by-hardware
– Review implemented AIX constructs, i.e. “firm” near-static structures and settings
– Review historical/accumulated AIX events, i.e. usages, pendings, counts, blocks, etc.
– Monitor dynamic AIX command behaviors, i.e. ps, vmstat, mpstat, iostat, etc.
• Recognizing Common Performance-degrading Scenarios
– High Load Average relative to count-of-LCPUs, i.e. “over-threadedness”
– vmstat:memory:avm near-to or greater-than lruable-gbRAM, i.e. over-committed
– Continuous low vmstat:memory:fre with persistent lrud (fr:sr) activity
– Continuous high ratio of vmstat:kthr:b relative to vmstat:kthr:r
– Poor ratio of pages freed to pages examined (fr:sr ratio) in vmstat -s output
© Copyright IBM Corporation 2012
64
2011
IBM Power Systems Technical University
October 10-14 | Fontainebleau Miami Beach | Miami, FL
IBM
Thank you
Earl Jew (earlj@us.ibm.com)
310-251-2907 cell
Senior IT Management Consultant - IBM Power Systems and IBM Systems Storage
IBM Lab Services and Training - US Power Systems (group/dept)
400 North Brand Blvd., c/o IBM 8th floor, Glendale, CA 91203
© Copyright IBM Corporation 2012
Materials may not be reproduced in whole or in part without the prior written permission of IBM.
5.3
Trademarks
The following are trademarks of the International Business Machines Corporation in the United States, other countries, or both.
Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not
actively marketed or is not significant within its relevant market.
Those trademarks followed by ® are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States.
For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:
*, AS/400®, e business(logo)®, DBE, ESCO, eServer, FICON, IBM®, IBM (logo)®, iSeries®, MVS, OS/390®, pSeries®, RS/6000®, S/30, VM/ESA®, VSE/ESA,
WebSphere®, xSeries®, z/OS®, zSeries®, z/VM®, System i, System i5, System p, System p5, System x, System z, System z9®, BladeCenter®
The following are trademarks or registered trademarks of other companies.
Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.
Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom.
Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.
Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel
Corporation or its subsidiaries in the United States and other countries.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.
ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.
IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce.
* All other products may be trademarks or registered trademarks of their respective companies.
Notes:
Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will
experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed.
Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here.
IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.
All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual
environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.
This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without
notice. Consult your local IBM business contact for information on the product or services available in your area.
All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.
Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance,
compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.
Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.
66
31-May-13
© Copyright IBM Corporation 2012
66
Disclaimers
No part of this document may be reproduced or transmitted in any form without written permission from IBM
Corporation.
Product data has been reviewed for accuracy as of the date of initial publication. Product data is subject to change
without notice. This information could include technical inaccuracies or typographical errors. IBM may make
improvements and/or changes in the product(s) and/or program(s) at any time without notice. Any statements
regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals
and objectives only.
The performance data contained herein was obtained in a controlled, isolated environment. Actual results that may be
obtained in other operating environments may vary significantly. While IBM has reviewed each item for accuracy in a
specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customer
experiences described herein are based upon information and opinions provided by the customer. The same results
may not be obtained by every user.
Reference in this document to IBM products, programs, or services does not imply that IBM intends to make such
products, programs or services available in all countries in which IBM operates or does business. Any reference to
an IBM Program Product in this document is not intended to state or imply that only that program product may be
used. Any functionally equivalent program, that does not infringe IBM's intellectual property rights, may be used
instead. It is the user's responsibility to evaluate and verify the operation on any non-IBM product, program or
service.
THE INFORMATION PROVIDED IN THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER
EXPRESS OR IMPLIED. IBM EXPRESSLY DISCLAIMS ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE OR INFRINGEMENT. IBM shall have no responsibility to update this information. IBM
products are warranted according to the terms and conditions of the agreements (e.g. IBM Customer Agreement,
Statement of Limited Warranty, International Program License Agreement, etc.) under which they are provided. IBM
is not responsible for the performance or interoperability of any non-IBM products discussed herein.
© Copyright IBM Corporation 2012
67
Disclaimers (Continued)
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products in connection with this
publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM
products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those
products.
The providing of the information contained herein is not intended to, and does not, grant any right or license under any
IBM patents or copyrights. Inquiries regarding patent or copyright licenses should be made, in writing, to:
IBM Director of Licensing
IBM Corporation
North Castle Drive
Armonk, NY 10504-1785
USA
IBM customers are responsible for ensuring their own compliance with legal requirements. It is the customer's sole
responsibility to obtain advice of competent legal counsel as to the identification and interpretation of any relevant
laws and regulatory requirements that may affect the customer's business and any actions the customer may need
to take to comply with such laws.
IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is
in compliance with any law.
The information contained in this documentation is provided for informational purposes only. While efforts were made
to verify the completeness and accuracy of the information provided, it is provided “as is” without warranty of any
kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related
to, this documentation or any other documentation. Nothing contained in this documentation is intended to, nor
shall have the effect of, creating any warranties or representations from IBM (or its suppliers or licensors), or altering
the terms and conditions of the applicable license agreement governing the use of IBM software.
© Copyright IBM Corporation 2012
68