Affinity

Contents

Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv

Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Layout of Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

The challenges. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Environmental Shifts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Business Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Organizational Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Consequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Production Changes Cost More . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Vertical Teams Lack Breadth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Process is an Afterthought . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Which Process? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Build Versus Buy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Buy More Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Hire More Employees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Return on Investment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

When All Else Fails, Kill It . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Common Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Tribal Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Manual Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Give Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Outsource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

The Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Solvable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Lay the Technical Foundation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Understand the Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Think about the Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Build Your Documentation Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Manage the Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Incorporate Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Perfect the Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

How . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Fundamental #1: Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Fundamental #2: Application Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Fundamental #3: Understand the Operating System . . . . . . . . . . . . . . . . . . . . . . . 35

Fundamental #4: Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Fundamental #5: Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

What is an application server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Getting to Know the Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

J2EE Connector Specification 1.5 (J2EE Connectors) . . . . . . . . . . . . . . . . . . . . . 43

J2EE Management Specification (JMX) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Enterprise

JavaBeans Specification (EJB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Java IDL API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Java Naming and Directory Interface (JNDI) Specification 1.2.1 . . . . . . . . . . . . . 45

Java Message Service (JMS) Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Java Servlet Specification (servlets) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Java Transaction API (JTA) Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

JDBC Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Java Server Pages (JSP) Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

RMI/IIOP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Contents vii

Components That Drive the SOA Initiative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Java API for XML Registries Specification 1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Java API’s for XML/SOAP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Designing an application for success . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

Why Design Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

Plan for Failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

Reach Out . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

Physical Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Production Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Service Level Agreements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Web Metrics Are Inaccurate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Build Process In . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Evolve or Die . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Application Traits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Developing Quality Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Copy and Paste . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Code Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Dependent Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Some Common Architectural patterns . . . . . . . . . . . . . . . . . . . . . . . . . 59

Some Architectural Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

MVC Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Client-Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

N-tier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Service-oriented architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Common Design Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

Why Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

MVC in Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Service Locator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Singleton . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

Struts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

viii Affinity

Application Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

Stabilize the Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

Lock Down the Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

Repeatable Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

Automation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

Implementation Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

Always Have a Backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Source Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

Configuration Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

Application Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

Kitchen Sink JARs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

Minimize the Footprint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

Text Editor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

Build the Application Locally . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

Automate Builds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

Automate Your Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

Cluster Early . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

Document the Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

Targeted Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

Load testing for success . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

Load-Test Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

Determine What You Are Going to Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

Customer-Usage Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

Generating the Load . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

Load-Test Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

Bench Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

Performance Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

Endurance Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

Destructive Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

Common Issues in Load Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

Production Load Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

Lessons Learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

Contents ix

Identify Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

Load-Test Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

Common areas to watch out for . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

Remote Calls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

Deployment Descriptors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

Session Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

Deployment Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

Automate Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

Load Balancers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

Affinity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

Sticky . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

EJB Load Balancing Versus Servlet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

Dynamic versus Static Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

Routing Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

Round-Robin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

Server Availability and Application Availability . . . . . . . . . . . . . . . . . . . . . . . . . . 101

Cluster Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

Large Versus Small Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

Multiple Applications One Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

Decouple Business from Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

Globally Distributed Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

Business Logic Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

Database Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

Production . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

Do You Know What Your Servers Did Last Night? . . . . . . . . . . . . . . . . . . . . . . . 109

Do You Know What Your Employees Did Last Night? . . . . . . . . . . . . . . . . . . . . 110

Four 9s is Difficult . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

Ex Post Facto (After the Fact) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

Ad Hoc Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

Root Cause . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

Determine Who Has Access to Production . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

Unauthorized Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

Controlled Chaos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

Multi-Homing Your Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

Log Level for Production . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

Large Cluster Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

Centralized Administration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

TroubleShooting the issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Troubleshooting 101 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Where to Start Looking for Trouble . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Ask the Right Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

Memory or Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Differences between Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Load Balancer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Java Virtual Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

Context Switching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

Operating Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

Java Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

Search Engines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

Involve the Vendors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

Escalate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

The War Room . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

Performance Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

Common-Denominator Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

Bottleneck . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

Validate before Making the Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

Outside the Box . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

Less is More . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

Out-of-the-Box Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

Contents xi

Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

Disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

CPU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

Java Virtual Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

Log Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

EJB Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

MTU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

Session Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

Web Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

Performance Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

Coarse-Grained/Fine-Grained . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

High-Performance Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

Keys to Success . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154

Where to Focus Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154

Performance Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

The Right Engineer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

Closing Thoughts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

Deployment Scripts

If you plan to keep an environment for any period of time, you will most likely want to automate how you get code into that environment. Having a repeatable process will better assure your chances of success when it comes time to deploy code; not automating the environment will cause inconsistent results. Review deployment scripts, and make sure you have redundancy in the process. You also will need to account for errors and, if possible, back up the code and any other artifacts, prior to deploying the new code and or scripts. Deployments happen at all hours of the day and night—doing a deployment in the middle of the night has its own challenges, but add a full day of work the day before to your being tired, and you can make all sorts of mistakes. When you determine your best practices approach for dealing with the clusters, keep in mind the growth and the possibility—you may need to add additional servers to support the growth of the company and/or applications.

Session Replication

Data may not be as predictable in production as you saw it in development, which may translate into the size of your session objects. The session objects are objects that contain data you use to store state (save the user session) on the server. You will most likely keep the username and password, some preferences, and, depending on the requirement, a whole range of other data sets. Session replication is when you take the session or state of a user on Server A and make a copy of that session for failover purpose. The idea is that if a user is logged on to a machine and it goes down, he can resume work on Server B without having to re-authenticate or lose the non-transient data before it is persisted to disk or database. When you move the application from a development environment, which has a small set of test data, into production, where the data may be ten times the size of the test, you may run into problems. The way the application is used may be different when you use it in production. Production may have sessions that are several megabytes in size. The challenge to understand is how your code is handling these large sets of data coming back from database, or downstream system. Let’s say you do a search and bring back a list of six hundred records. If persistence is on, this object is stored in session, and then the application server will replicate the data to another node in case of failure. Now, six hundred records may sound like a lot, but the process of serializing and sending them across to another node in the cluster does take some time and resources. You need to be aware of the data that is transient and can be thrown away, as well as what you need to persist (save) in the near and long term.

Deployment Descriptors

Given that the container or application servers has its own set of configuration files, called deployment descriptors, you need to be extremely cognizant of what they are and how they can impact the performance of your application. An application that runs perfectly fine in development and in test environments may not work in production. The deployment descriptors are a way to tell the container how to behave. When you change a deployment descriptor you tell the container how to behave differently. Be careful when making modifications to the deployment descriptors. If you make changes without knowing the consequences, you may introduce a performance issue, just by making the change.

Remote Calls

When making a remote call, keep in mind that the remote call will consume resources of the system. In a Java application server, you use a thread to wait for that resource and a socket on the operating system, which translates into an additional connection through the firewall. Remote calls can also cause additional network traffic if they are not designed properly. Remote calls have the disadvantage of overhead, of getting the remote connection. Be very cautious with any code that calls external resources. It is not just a matter of whether to release the resource but whether you also have a timeout in place to release the resource, in the event that the remote resource doesn’t respond. You also should be concerned with how often you are going to call a remote call and how much data you will get back. Be very aware of how fine grained your remote calls are. If you keep them too fine grained, you can bog down the network or servers with the amount of times you need to go to the back-end system for more data.

Lay the Technical Foundation

Building your technical systems requires a technical commitment to the projects and issues at hand; moreover, it sometimes is more important to show the support of the issues than to solve any single technical problem. The solutions to the problem are available at your fingertips; the challenge is to get people in the right mindset to find them. Doing this requires that technical issues be resolved so there is some breathing room to focus on operational excellence. To be successful, there must be technical initiatives that help reduce the amount of downtime in the environments. You may need to limit the number of projects in scope. Show the company that you are making the commitment by putting in the technical foundation to support the efforts. Systems today are so intertwined with multiple dependencies that it may be difficult to differentiate the root cause of a problem. The answer to technical solution may be found anywhere in the network infrastructure or dependent application and involves the various layers to include the networking enterprise server and, in some cases, the clients desktop. Understanding the problem can be difficult in itself, but if we lay a supportive technical foundation to fix these issues we improve our changes of success.

Solvable

The truth of the matter is that these challenges are solvable. There is a technical reason why the systems function the way they do. The biggest challenge is to get a process in place to tackle the issues, one by one, until at some point, you stop getting the late-night calls, and the systems start to become more stable. The second step is to change your process so that it includes the best practices in your development life cycle, which will help avoid these problems in the future. The more stability you bring into the environment, the more time you will have to focus on solving the IT challenges that will give the company the biggest return on investment. Changing how your systems work isn’t a straightforward process; it may vary from company to company.

If you can not think of a solution the solution will find you. It may be a mandate from above, or it could be just time for a change. I suspect that if you were to talk to your employees and coworkers, many would agree that it is time for a change—they just don’t know what that change is. Think back to some of the challenges you faced most recently. Is there a pattern to these challenges? Do people complain about the same issues over and over again? To solve the problems faced in the company, each team member may choose a different way to fix the issue—one may leave the department, get promoted, quit the company, or open that coffee shop in North Carolina he has always wanted. There is always a downside to letting things continue down the wrong path, and the biggest downside is that if you don’t find a way to change, someone else will find it. The path that person chooses may not be in the companies best interest. Some people may want to get as far away from a problem as possible, but the rest need to learn how to adapt to change. They need to focus on how to solve the problems, rather than how to escape them. I think there is a far greater sense of accomplishment when we tackle problems head on. Keep a positive attitude, these problems have solutions, and over time, you can shift from survival mode to growth mode—growth that can continue indefinitely as your company’s goals and aspirations grow.

THE SOLUTION

I’ve discussed some of the challenges that companies are facing today when changing technology and when just trying to keep up with their competitors seems insurmountable. The first thing to understand is that these systems weren’t built overnight. The problems weren’t created overnight either, and it takes time to make all the necessary changes. The longer it takes to get a sound strategy in place, the more confusing the application landscape can become. I once met an architect who said that it didn’t matter in what the application components were written—if he had a Java developer. He insisted, he would have that component written in Java; if not, he would let the developer write the application in whatever language the developer knew. Just think about the problems he created for the company! Such a strategy allows the corporate landscape to dictate what the applications will look like. At some point you need to take a side and say, “This is how we build out systems.” This is a challenge but if you want to build successful business applications, there needs to be a consistent approach.

Getting the applications back on track will take time—it takes time to know if you are successful. For each action you may not know the consequences of your changes for months afterward. The question is whether you have the time to wait—usually three- to six-month increments—to see if your changes are having an effect. What benchmark metrics do you use to see if you are taking the company further ahead or causing more problems?

Outsource

When a company considers outsourcing of an existing application, it is usually a long time coming, and in some cases, management can’t be faulted for this decision. If you have an internal IT department, but you choose to outsource the development of your applications, it’s generally because you either don’t have confidence in your internal teams or that the cost of the application would be more than the cost of getting in the market. The outsourcing usually isn’t a personal decision; it’s a business one. If you look under the hood, you’ll see that the culprit is poor financial performance. If you can get a high-performance application by outsourcing, then do it. You also may decide to outsource an application when you don’t have the in-house ability to build or support this application. The issue generally does not relate to the quality of employees. There are some situations that warrant placing blame on the employees. I would suggest that the issue is more deeply rooted in the organizational culture and its’ inability to effectively deliver applications and first-class support. With outsourcing, you want to get a better process in place; one that you believe will be better able to deliver the results you expect.

Give up

Companies tend to be cautious about adopting a new technology because they fear failure and think the application will be too complicated. For those companies that have failed at implementing their Java application servers, the only choice is to give up and return to what they know. Unless they are guaranteed to save money from a business perspective, they do not know if this system will help or hurt the bottom line; they are going to consider the applications ROI(return on investment). A properly designed application that reduces transaction costs and saves the company money is an asset to the business. If your systems are in a constant state of flux and you don’t know from one day to another how they are going to perform. By not fixing the issues you are impacting the ROI for the business and are seen as a liability. The best option may be to go back to what you understand. There are some companies that really shouldn’t have a high investment in technology—either they are interested in remaining small or there would be no lower cost with economies of scale.