Affinity

Manual Process

I have worked on teams where only a small percentage of the team actively contributed; the rest of the team performed manual tasks that could have been automated. I see situations where a few team members determine how to do the work and the rest simply repeat the process. Part of the reason for using technology is to help with the mundane and somewhat mindless tasks. If you automate these tasks, you can save a great deal of time, and that time saved could be used for other, more productive activities. For example, if someone writes an application that monitors and recycles the application, for an initial investment of $1,000 of development time and no maintenance cost, how much money is saved by not hiring a full-time employee to start and stop the application? The answer is simple. It is the cost of the employee’s salary minus the cost of the script. The time it takes to start and stop the servers, as well as monitor the servers all day long, really starts to add up. Why would you pay for a full-time employee when you can automate the process? Even outsourcing would not be as inexpensive, because the automated script can do the work for you. More importantly, the scripts that person wrote become a primitive form of documentation—you can go back through the script to understand how it works.

Tribal Knowledge

Tribal knowledge can be a detriment to any team, regardless of size. Tribal knowledge occurs when small groups of people become domain experts in the areas on which they are focusing. They become experts because they have been the only group working on these applications or components. They primarily have communicated via word of mouth and e-mail, and there isn’t a concrete document trail to follow. The problem with this type of communication is that to understand their process and procedure, you must become a tribe member. Becoming a tribe member means undergoing a time-consuming indoctrination process to get you up to speed on the ins and outs of the system. The tribal system is the simplest process in which to transition knowledge, but it’s the hardest system in which to get knowledge out of. That is why keeping your systems knowledge on a tribal system is bad for the company and hard on everyone. To determine if tribal knowledge is in use, ask any of the team members a question about a specific piece of functionality. If he refers you to the group who owns this functionality, this is your first clue they are using tribal knowledge. The second clue is when you are directed to the same person over and over again, and there is no one else in the group that can help. The third and final clue is when there is no process documentation for the answer once you get it. When only one person understands the information, you have a real problem, even if this one person is extremely helpful if you need to get an issue fixed. The question for management and others, however, is “Why do we have an entire team to support this process when only one person understands it?”

Tribal knowledge becomes extremely dangerous when you have a conflict between two tribes—it can cause a complete breakdown in communication, and these breakdowns in communication continue for months—sometimes years. It may reach the point where the person with the domain knowledge becomes even more guarded with their information. The team of engineers handling the mundane tasks leave at the end of the shift and have a fairly easy job, while the tribal leader on the other hand is off fighting fires and solving all the application inquiries. Managers, project managers, and team members have given up on the team, but as long as the domain expert is kept happy, everything is okay.

When confronted by management, those participating in the tribal transition will give you a wide range of excuses as to why they don’t understand the system—“ I wasn’t given the opportunity” or “It isn’t fair that I didn’t get all the opportunities that the other person got.” Now you are in trouble: You may have a team that is supposed to be saving the company money, but they are mired in conflict and trying to determine who they can blame.

Truthfully, everyone is to blame; people need and want direction. Not everyone has the ability to create a vision when sitting in a department, nor is it their job. There is a certain sense of security when someone comes up with a vision that gives a clear path toward success. If you do not have that leader in your group, then management must find a way to create that vision for the employees of the team. We all have goals in life, but many of us do not have a clear vision of how to reach them. When we do not meet our goals and reap rewards, we may feel like failures.

Documentation

A key to having a supportable environment is to track and document the issues that are faced on a regular basis, from the first day of development to the production rollout and beyond. The little “gotchas” that are learned, from the first day with the proof-of-concept to deployment, are the same ones that engineers will encounter years into the project. Successful projects have a good document trail that defines the issues encountered during all stages of the software-development life cycle, including the planning, architecture, proof-of-concept, development, and production implementation. For the average project, unless a process is adopted that forces the documentation and tracking of issues, the documentation is primarily tribal in nature if at all. The longer the project goes on in this state, the harder it will become to support and transition. The impact to engineers who are tightly coupled with the project is that unless they document and have a good transition plan, they may forever be tied to the project. This, in turn, will limit your growth and keep you from moving on to other departments. Even worse, if you abruptly leave the company without a transition plan it place it may negatively impact your future employment at other companies.

Common Approach

Regardless of whether you are a large company or a smaller company, the Java components you use or are going to use may be the same. The difference is the amount of times that you need to use them and the amount of integration between the dependent systems. Large or small, you can implement all the technologies, including SOA(service-oriented architecture). Some of the common components you may deal with are Web services, Java Message Service (JMS), Enterprise JavaBeans (EJB), Java Mail, J2EE connectors, and any number of other components. Depending on how you architect the solution, these components can be delivered in two-tier architecture or n-tier. In the enterprise, multiple architects may be designing different applications. It all depends on what new design the architect read about last week and whether or not they want to try it out as well as which direction they feel the application is going. It is important to have a common, consistent implementation when building your applications. Where does your domain knowledge reside? Do you keep that knowledge in the architectural units? Does this knowledge belong in the production arena? Production folks may be required to support these applications long-term. When the knowledge is kept among the development teams or architectural teams, they become key players in the success of the project. Now that they are key players, they also are locked into supporting this application until the knowledge transfers to another unit.

Complexity

The companies that use application servers range from small businesses to large Fortune 100 enterprises. The issues related to complexity are mostly found in large corporations, but can apply to the smaller companies as well. The risk of an overly complex system is magnified when yours is a small company. Some companies market only one product, and their entire existence depends on the success of the application they have deployed in production. An unstable or complex system can drain resources in a small company can put the whole shop out of business. Regardless of whether you have one application or five hundred applications, the problems are similar; problems in a large company are just magnified by the number of applications.

When it comes to the business application that is built and deployed on application servers the more complex the solution the harder it will be to support. Keep your application architecture simple. Use frameworks to build the custom application and keep applications simple. There are always multiple ways to solve a problem. In some cases, each developer will choose to solve it a little differently. Extremely complex problems can result if there is not a clearly defined, common approach to solving architectural challenges. This also will lead to word-of-mouth or tribal transition of source knowledge about the applications. The larger the projects, the more complex the systems are and the more complex the challenges are. Large companies need to have strong technical leadership that drives the best practices and a structured approach to developing applications.

When All Else Fails, Kill It

Java application servers, if configured correctly, should never have to be killed or shut down frequently to resolve performance issues. Java application servers have issues for a reason. The key is in knowing how to approach the servers from a tactical perspective, which will lead you to a more stable and more robust system. You might be surprised that a troubled application can be fixed relatively easily, without having to make too many code modifications or fixes. The worst system I worked on had fifty percent availability. Each of the Java Virtual Machines was recycled at least once every fifteen minutes. It took some time, but I got the system from fifty percent availability to over ninety-eight percent availability by adopting a definitive process for identification and resolution of issues. Recycling your application alleviates the immediate issue but does not fix persistent problems. If you have to recycle frequently, there is a systemic problem at the root of the issues. There may be more than just one problem causing the outages in the system. Understand that if something doesn’t feel right, then it probably isn’t. Kicking customers off of a system to clear up code issue is not acceptable if you are trying to run a for-profit business—or for any business, for that matter. It will take time to adapt the process changes that solve availability issues; but to be successful you must put them in place. With the right process in place, over time, you can incrementally gain an upper hand by defining a winning strategy to bring stability to your application servers and the business they are trying to support.

Return on Investment

A business is valued, in the market, based on how it performs financially. Management can look at the average cost for an employee and compare that to their peers. If all things are equal, then you will have the competitive edge over your competition if you can keep your costs under that of your competition. The flaw in this thinking is that not all technical folks are equal, and depending on the mix of talent in your company, the productivity can change from group to group. You may be above the curve for cost per unit of work yet your workers are ten times more productive than a similar company who is below the average cost per employee model. When you are trying to fit into the cost model and add the additional resources you are focusing on the metrics rather than efficiencies of operation. The metrics are showing that you need additional resources when what you really need is to become more effective. Thinking you are on the way to solve the problem you may have actually compounded them. You have now added overhead to your organization, with hardware licenses and desks for the new employees. It may result in a large organization that has only a few highly effective employees. This increases the likelihood that you will need to downsize your employee ranks or outsource your jobs to someone who can solve your problems. And this will be increasingly evident by the amount of work you are able to produce with the number of resources you have. The senior management level at most companies watches the numbers very closely. If you are unable to produce within the industry averages, then changes will be made.

Hire More Employees

The same truism about adding more servers can be applied when it comes to hiring employees. Some companies believe that hiring more employees will fix the issues. You can hire as many people as you want, but if you do not have the right technical leadership, you are destined for failure. If you don’t have a process in place that guides the company to successfully manage the systems, your shop will be a revolving door. I see this happen all the time; it’s a waste of resources. When you add new people, you will be trying to jump-start newly hired recruits to get up to speed quickly. The work environment will become very stressful and employee morale may be affected negatively. Solutions are developed to solve a challenges, and when those solutions are not implemented people may get frustrated and quit. The process is not in place to support the work that needs to be done. The continual practice of not listening to those who are trying to make the process better builds an apathetic view toward the systems and their stability. It ultimately creates a negative view toward the organizational culture. Individuals with domain knowledge may quit when they get frustrated with the issues and lack of stability of the systems. You can cut your liabilities by getting the systems stable, which improves employee morale and supports your employee retention plans.

Buy More Hardware

A common approach to solving performance problems and meeting capacity demands is to buy more hardware. The thinking is that one application server can handle one-hundred users. If you need to service five thousand concurrent users, then you should have fifty instances to support all of the users. This might be the case, but you should know for sure before making a big investment, and you may still have the same issue you had prior to getting additional hardware. Purchasing more hardware may solve the immediate business need to keep the systems running, but it isn’t the long-term solution to the challenges you face. The hardware is expensive to purchase and maintain. There are issues with adding capacity to the data centers and increased management cost to handle all the new servers, not to mention the licensing costs associated with the purchase. You must understand the application in its current architectural state, with the settings as they are today. Understand what the implications are when you add more resources. Get a handle on your system before you start purchasing more hardware. This is not to say that you never will need any hardware and will use one machine for all applications, but there may be inefficiencies in your code. No matter how many servers you add, you will still have issues.

In fact, in one case an architect solved a capacity issue by limiting the number of concurrent requests the server could run. He set the limit to two concurrent users. In this current design, how many instances do you think he needed to meet the demand of 480 concurrent requests? The system would have required 240 servers to run with that architecture. I had a one-on-one meeting with the CEO, and he said that he was spending ten million dollars a month on hardware. There was a far more efficient way to solve the performance issues this application was having—it took about three days—but when I was done they were able to handle the capacity they needed with only twenty servers, not 240. How much money do you think it would have cost the company to go with 240 servers?

Build vs. Buy

A custom business application offers all the functionality you choose to build and all the challenges of off-the-shelf packaged software in terms of the software development life cycle. You can build software that is custom-tailored to your business process. The integration may be easier than trying to customize an off the- shelf product to fit your business needs. The advantage of building it yourself is that you will get the application you want with the look, feel, and integration points you choose.

The off-the-shelf product has a standard set of features that has proven successful in other environments. It has a process in place to build and deploy its’ software. The product is most likely very specialized to meet a business objective making integration with your business process more complex. There may be a great deal of mapping between the off-the-shelf product and your current business model. There are most likely standard support and bug-fix procedures in place. The off-the-shelf packages also may provide you with references of customers that have similar implementations as yours.

When you choose to build your own application, you take on all the responsibilities of the off-the-shelf company. Some of the groups you may need to support this process are a product management group, development, and QA (quality assurance) support, each under their own leadership. With the home grown solutions the IT department takes on the ownership of the full software development life cycle for the application. It controls the applications’ logic, which hardware to use, the database schemas to deploy and in what technology you will build the application. The more complex you make the system, the greater number of variables required to deliver this application. The more variables you have in your application the more resources and bigger need to have a plan in place to support the application. This all makes a large impact in getting your applications and environments ready for production.

Now that you have ownership of all the areas that need to be developed in the application, add the complexity of defining the business process and a nearly impossible schedule, and the task of successfully building and supporting the application becomes even more daunting. Companies compensate for it by solving the same code challenges, multiple times in each department, and each in a slightly different way. There needs to be consistency when you start developing code for your environments. Identify the shared components to the applications, enforce your development standards, and do the peer reviews and code reviews of the systems.

There are advantages and disadvantages when deciding to build versus buy. The more off-the-shelf applications you try to integrate and support, the more complex the environment will become and the higher your maintenance cost for these applications will be. Whether you choose to build or buy your software, as long as they are both deployed on a Java application server the concepts detailed in this book should apply.