Jim Mauro is a Senior Staff Engineer in the Performance, Architecture, and Applications Engineering group at Sun Microsystems, where his most recent efforts have focused on Solaris performance on Opteron platforms.

Brendan Gregg is a Solaris consultant and instructor teaching classes for Sun Microsystems across Australia and Asia. He is also an OpenSolaris contributor and community leader, and has written numerous software packages, including the DTraceToolkit.

Preface xxiii

About the Authors xxxi

Acknowledgments xxxiii

PART ONE: Observability Methods 1

Chapter 1: Introduction to Observability Tools 3

1.1 Observability Tools 4

1.2 Drill-Down Analysis 7

1.3 About Part One 8

Chapter 2: CPUs 11

2.1 Tools for CPU Analysis 11

2.2 vmstat Tool 13

2.3 CPU Utilization 14

2.4 CPU Saturation 15

2.5 psrinfo Command 15

2.6 uptime Command 15

2.7 sar Command 16

2.8 Clock Tick Woes 19

2.9 mpstat Command 20

2.10 Who Is Using the CPU? 23

2.11 CPU Run Queue Latency 24

2.12 CPU Statistics Internals 26

2.13 Using DTrace to Explain Events from Performance Tools 29

2.14 DTrace Versions of runq-sz, %runocc 31

2.15 DTrace Probes for CPU States 33

Chapter 3: Processes 35

3.1 Tools for Process Analysis 35

3.2 Process Statistics Summary: prstat 37

3.3 Process Status: ps 41

3.4 Tools for Listing and Controlling Processes 45

3.5 Process Introspection Commands 47

3.6 Examining User-Level Locks in a Process 52

3.7 Tracing Processes 53

3.8 Java Processes 60

Chapter 4: Disk Behavior and Analysis 67

4.1 Terms for Disk Analysis 67

4.2 Random vs. Sequential I/O 69

4.3 Storage Arrays 70

4.4 Sector Zoning 71

4.5 Max I/O Size 72

4.6 iostat Utility 73

4.7 Disk Utilization 74

4.8 Disk Saturation 75

4.9 Disk Throughput 76

4.10 iostat Reference 76

4.11 Reading iostat 82

4.12 iostat Internals 85

4.13 sar -d 87

4.14 Trace Normal Form (TNF) Tracing for I/O 88

4.15 DTrace for I/O 88

4.16 Disk I/O Time 97

4.17 DTraceToolkit Commands 101

4.18 DTraceTazTool 108

Chapter 5: File Systems 109

5.1 Layers of File System and I/O 109

5.2 Observing Physical I/O 111

5.3 File System Latency 112

5.4 Causes of Read/Write File System Latency 114

5.5 Observing File System “Top End” Activity 118

5.6 File System Caches 119

5.7 NFS Statistics 133

Chapter 6: Memory 135

6.1 Tools for Memory Analysis 135

6.2 vmstat(1M) Command 137

6.3 Types of Paging 138

6.4 Physical Memory Allocation 142

6.5 Relieving Memory Pressure 144

6.6 Scan Rate as a Memory Health Indicator 146

6.7 Process Virtual and Resident Set Size 148

6.8 Using pmap to Inspect Process Memory Usage 149

6.9 Calculating Process Memory Usage with ps and pmap 150

6.10 Displaying Page-Size Information with pmap 153

6.11 Using DTrace for Memory Analysis 154

6.12 Obtaining Memory Kstats 157

6.13 Using the Perl Kstat API to Look at Memory Statistics 158

6.14 System Memory Allocation Kstats 158

6.15 Kernel Memory with kstat 160

6.16 System Paging Kstats 161

6.17 Observing MMU Performance Impact with trapstat 163

6.18 Swap Space 164

Chapter 7: Networks 173

7.1 Terms for Network Analysis 173

7.2 Packets Are Not Bytes 175

7.3 Network Utilization 176

7.4 Network Saturation 177

7.5 Network Errors 177

7.6 Misconfigurations 177

7.7 Systemwide Statistics 178

7.8 Per-Process Network Statistics 189

7.9 TCP Statistics 191

7.10 IP Statistics 196

7.11 ICMP Statistics 199

Chapter 8: Performance Counters 201

8.1 Introducing CPU Caches 203

8.2 cpustat Command 206

8.3 cputrack Command 215

8.4 busstat Command 216

Chapter 9: Kernel Monitoring 221

9.1 Tools for Kernel Monitoring 221

9.2 Profiling the Kernel and Drivers 222

9.3 Analyzing Kernel Locks 223

9.4 DTrace lockstat Provider 227

9.5 DTrace Kernel Profiling 229

9.6 Interrupt Statistics: vmstat -i 230

9.7 Interrupt Analysis: intrstat 230

PART TWO: Observability Infrastructure 233

Chapter 10: Dynamic Tracing 235

10.1 Introduction to DTrace 235

10.2 The Basics 236

10.3 Inspecting Java Applications with DTrace 257

10.4 DTrace Architecture 265

10.5 Summary 271

10.6 Probe Reference 271

10.7 MDB Reference 294

Chapter 11: Kernel Statistics 295

11.1 C-Level Kstat Interface 295

11.2 Command-Line Interface 307

11.3 Using Perl to Access kstats

11.4 Snooping a Program’s kstat Use with DTrace 317

11.5 Adding Statistics to the Solaris Kernel 317

11.6 Additional Information 323

PART THREE: Debugging 325

Chapter 12: The Modular Debugger 327

12.1 Introduction to the Modular Debugger 327

12.2 MDB Concepts 330

Chapter 13: An MDB Tutorial 335

13.1 Invoking MDB 335

13.2 MDB Command Syntax 336

13.3 Working with Debugging Targets 353

13.4 GDB-to-MDB Reference 357

13.5 dcmd and Walker Reference 359

Chapter 14: Debugging Kernels 367

14.1 Working with Kernel Cores 367

14.2 Examining User Process Stacks within a Kernel Image 382

14.3 Switching MDB to Debug a Specific Process 385

14.4 kmdb, the Kernel Modular Debugger 388

14.5 Kernel Built-In MDB dcmds 395

APPENDICES

Appendix A Tunables and Settings 401

Appendix B DTrace One-Liners 407

Appendix C Java DTrace Scripts 409

Appendix D Sample Perl Kstat Utilities 413

Bibliography 429

Index 433

The New copy of this book will include any supplemental materials advertised. Please check the title of the book to determine if it should include any access cards, study guides, lab manuals, CDs, etc.

The Used, Rental and eBook copies of this book are not guaranteed to include any supplemental materials. Typically, only the book itself is included. This is true even if the title states it includes any access cards, study guides, lab manuals, CDs, etc.