Client Engagement Case Studies in Performance Engineering

Case Studies

Some of Kobatera's Consulting Services engagements are listed below.

Provided performance engineering services to ensure a new multi-business unit customer servicing application could handle the expected user and transaction load. The application architecture is an Ajax based single page web application deployed on Amazon Web Services. Technologies involved were Apache 2.2, Tomcat 7.0, Java 1.8, Hystrix, and Hazelcast running on Linux servers. Server code is written using the Spring framework and interacts with a custom middleware layer using REST APIs. Identified and resolved issues with the client's performance testing approach that did not replicate real-world load. Improved the testing process to produce more useful results, rather than simply executing tests. Implemented robust monitoring with App Dynamics and Site Scope in the performance testing environment. Identified connection pool issues with the custom code using Spring and Apache connection pool libraries. Worked with the customer's development team to resolve those coding issues and increased capacity by 600%. Provided JVM, Hazelcast, and Linux tuning advice that increased capacity an additional 200%. Identified issues with the client's DNS configuration that made DNS a single point of failure for the application. Worked with the client to reconfigure DNS to be more resilient. Provided instance type and ELB configuration recommendations for AWS.
Provided performance engineering services for the migration of the client's 30 million customers from a vendor provided servicing platform to an in-house developed platform. Ensured the application met response time, capacity, reliability and scalability requirements through rigorous performance testing. The technologies utilized included Java, Tomcat, Apache, Nuxeo, Hazelcast, JMS, Splunk, HP Diagnostics, HP Loadrunner and HP Sitescope. The capacity requirements included 134 logins per second with 25,000 concurrent sessions, and were satisfied on the first day of production. Some of the major improvements identified prior to rollout were Hazelcast cache optimizations to maximize capacity and hit ratios, resolved threading issues with configuration file access that significantly reduced throughput, optimized JMS configuration to prevent disk space issues while maximizing fault tolerance, optimized Tomcat and JVM configurations to maximize capacity, identified the source of a memory leak causing out of memory crashes.
Provided performance engineering services to a client developing a new mobile application for customer account servicing. The application would have 1 million plus downloads. The mobile application utilized IBM Worklight, while the server components included Worklight, Websphere, and the IBM HTTP server, as well as a Weblogic based middleware layer with both REST and SOAP APIs, and Kapow Katalyst screen scraping software. All container configurations were optimized through performance tuning, including memory, garbage collection, threads, HTTP connections, DB connections, and JMS queues. Identified several code optimizations to improve JMS session creation, improve JDBC connection re-use, reduce object creation, improve socket reader throughput, and indentify a memory leak caused by improper handling of exceptions. Worked with IBM Engineers to provided production troubleshooting on several issues related to Worklight and Websphere configurations.
Provided performance engineering services to resolve throughput issues on a custom developed workflow application that processed auto loans. The application utilized BizLogic workflow software which was customized for the client. Identified row lock contention in the vendor's data base as the throughput issue. Worked with the vendor and the client to resolve the issue and increase throughput by 100%. Provided additional recommendations on code refactoring and architecture modifications that made another 100% improvement.
Provided performance engineering services for a new custom coded web service to display customer statements in an on-line account service application. The service was a custom modification of a vendor package that converts IBM AFP print stream data into PDF or HTML. It was running in a Weblogic container on Linux, and was accessed through BEA’s ALSB. Identified a memory leak in the vendor’s code, and worked with them to fix it. Executed an extensive series of tuning tests and implemented changes that improved response time from 12 seconds to less than .4 seconds.
Engaged to help the client ensure the capacity of a vendor provided internal social media site would meet requirements. Worked with the testing team to improve their testing approach and more closely mimic real-life usage. Identified numerous issues with row locking contention from the vendor's data base stored procedures. Worked closely with the vendor to replicate the issues in their environment and provide fixes. Provided sizing and configuration recommendations on the client's VMWare instances.
Engaged to Identify throughput issues with a vendor package ad campaign management software. Worked with the vendor to optimize their process threading configuration to greatly improve capacity. The vendor subsequently implemented the changes across all of their clients.
Provided performance engineering services to ensure a custom coded web site with 1 million visitors per day would meet response time and capacity requirements. The site was built using Weblogic Portal running in JRockit on Red Hat Linux and using Apache web servers. Capacity and performance goals were achieved through JVM and platform tuning. Identified issues with custom code for targeted advertising that was causing recursive object creation and out of memory crashes of the JVM.