/BACK

Outsourcing of the 2nd and 3rd line of IT systems support. Organization of the analytical systems support process. Success story

Problem statement
A few years ago, our company was tasked with coordinating SAP BI system support for several of our regular customers. The support included the analysis of incidents from users of the system with the level of performance SLA 95% seven days a week from 9 a.m. to 8 p.m. (2nd and 3rd line treatment applications), monitoring, and periodic tasks data loading, seven days a week, 24 hours a day, architectural control, and quality control development and documentation for acceptance of new functionality to support in a dynamic system.The contract stipulated a three-month stabilization period during which the SLA objective might be breached without the contractor facing penalties.
Problems
Several major issues were recognized when developing the work organization, without which the service would not be able to run properly for a long time:
  • Complex service availability schedule
Schedules of 10x7 (ten hours a day, seven days a week) and 24x7 (twenty-four hours a day, seven days a week) are unusual for a consulting firm. This is the first time we've been able to deliver services in this manner. Even without considering additional support issues, the process of scheduling the availability of the service on such a timetable is challenging. No one likes working late at night since it lowers one's quality of life and interferes with socializing.
  • Team Burnout
IT system implementation specialists are accustomed to working in a project-based environment. At various phases of the project, there are periods of strain and relaxation, mini vacations after accomplishing challenging tasks, and a sense of pleasure that something difficult has finally taken off despite all the challenges. On the other hand, the support process is routine and demotivating in and of itself. Every day, the specialist processes multiple applications of various complexity; despite all efforts, there may be even more applications on the group at the end of the day than there were at the start of the working day. And so it goes, month after month.
  • Lack of trained personnel on the market who want to support the system
Working in support necessitates the same system abilities as working in implementation/development. However, from the standpoint of a professional, the introduction and development of systems provides a variety of objective and subjective benefits:
  • Fills the CV with a wide range of projects and technological expertise.
  • Allows you to participate in the construction of architecture instead of dealing with existing structures.
  • For whatever reason, it is thought to provide more experience in the same length of time than support.
  • Support is, well, out of date.
As a result, professionals with the required abilities for project execution and development are rarely prepared to engage in routine support roles.
Workplace organization. A difficult timetable. Problem-solving
The arrangement of monitoring and restoring the operability of night-time periodic activities for downloading and calculating data was one of the primary challenges to be accomplished. The job of a highly competent specialist at night is costly, and only a few specialists will agree to undertake such work on a long-term basis. First and foremost, efforts were made to optimize and improve the stability of periodic tasks. This allowed night periodic duties to be condensed into the hours between 4:00 and 9:00 a.m., reducing their time. The labor expenditures of restoring task operability have decreased as night processes have become more stable. As a consequence, the time it took to recruit a support expert for monitoring was drastically reduced, and it was limited to the hours of 7 a.m. to 9 a.m., while system users were assured of receiving proper data in the system by the start of the working day (10 a.m.). Further work on the identification and classification of developing problems enabled the partial automation of process restarts and the partial description of problem solutions in comprehensive instructions. As a result, interns and younger specialists were drawn to this job, and they only sought help from a highly qualified specialist on duty if the problem was unique and not indicated in the instructions.
Another challenge was synchronizing weekend and holiday availability of the incident management service's second line. In this situation, the firm has decided to give the crew complete authority. The team assessed which mode would be the most enjoyable to work in based on the load distribution between weekdays and weekends. The final option was a mixed work schedule, with some staff working in 4x4 mode (four days of work from 9 a.m. to 8 p.m., followed by four days-off), and others working in normal mode, as determined by the production calendar. As a consequence, the service availability requirement of 9 a.m. to 8 p.m., seven days a week, was met.
When it comes to work organization, the Company took the position of giving the Team complete autonomy. The employees may be able to establish comfortable working conditions for themselves and have a direct influence on the service delivery process as a result of their decisionsThe only requirement was that at the end of the stabilization phase, the SLA performance targets be reached (3 months). As a result, the team was able to create a comfortable atmosphere, allowing them to focus on the work at hand. The Team has always regarded the implementation of the contract-mandated SLA as the sole criterion for service quality and job success. Based on this, the team was given a clear and straightforward financial incentive scheme, with a monthly bonus tied only to the fulfillment of the SLA service specified in the Customer's contract.
Burnout problem, resource search, staff rotation
One of the most pressing issues was noticed even during the planning stage: team fatigue and the requirement for personnel rotation. The service may last a year, or perhaps a year and a half, depending on the enthusiasm of the initial team, which was charged with developing the process. The crew would then become burnt out, and the service quality would deteriorate. As a result, it became important to plan for staff rotation. The work was divided into two categories:
  • Classification of occurrences and creation of specific software instructions for each type of issue for the 2nd line specialist. As a result, the time it took to add a new specialist to the project was cut in half.
  • The creation of an internship program with two major components. The learner learned broad theoretical and practical abilities for working with all supported systems and products in the first half. The second portion of the training focused on reviewing documentation and honing practical skills in the analysis of typical occurrences, which were mock-ups of problems from the production system that the student would see during his first days on the job.
The second "practical" part allowed for even faster onboarding of a new team member. As a consequence, it took one month of internship and two to four weeks of team work before a new individual was fully immersed in the job on an equal footing with the rest of the team.
With the introduction of the internship program, it became possible to organize the rotation of the team entirely at the expense of interns, rather than relying on market-based recruitment. The planned transition from support to consulting helped to prevent team fatigue and a drop in service quality.
Employees with hyper-responsiveness are in key roles
The precise selection of the initial team composition was the cornerstone of the effective organization of the support service. The men who recognized the key aspects of a process that expanded and became a significant business field were blessed with a valuable quality: a responsible and innovative work attitude. In hindsight, we can certainly state that this was the most significant attribute that enabled us to complete the assignments.
Three years of service operation results
The results have been largely positive, with the crucial metric of SLA performance following the stabilization phase never falling below 95% in three years of operation. In addition, the three-month stabilization time for connecting a new client has been lowered to one month. The technological stack of systems that are supported has also grown. We began by supporting data warehouses built on SAP BW technologies and SAP HANA databases, as well as visualizations built on SAP Analysis and SAP Business Objects. Greenplum and Clickhouse have now been included in the technical line.

/BACK

Outsourcing of the 2nd and 3rd line of IT systems support. Organization of the analytical systems support process. Success story

Problem statement
A few years ago, our company was tasked with coordinating SAP BI system support for several of our regular customers. The support included the analysis of incidents from users of the system with the level of performance SLA 95% seven days a week from 9 a.m. to 8 p.m. (2nd and 3rd line treatment applications), monitoring, and periodic tasks data loading, seven days a week, 24 hours a day, architectural control, and quality control development and documentation for acceptance of new functionality to support in a dynamic system.The contract stipulated a three-month stabilization period during which the SLA objective might be breached without the contractor facing penalties.
Problems
Several major issues were recognized when developing the work organization, without which the service would not be able to run properly for a long time:
  • Complex service availability schedule
Schedules of 10x7 (ten hours a day, seven days a week) and 24x7 (twenty-four hours a day, seven days a week) are unusual for a consulting firm. This is the first time we've been able to deliver services in this manner. Even without considering additional support issues, the process of scheduling the availability of the service on such a timetable is challenging. No one likes working late at night since it lowers one's quality of life and interferes with socializing.
  • Team Burnout
IT system implementation specialists are accustomed to working in a project-based environment. At various phases of the project, there are periods of strain and relaxation, mini vacations after accomplishing challenging tasks, and a sense of pleasure that something difficult has finally taken off despite all the challenges. On the other hand, the support process is routine and demotivating in and of itself. Every day, the specialist processes multiple applications of various complexity; despite all efforts, there may be even more applications on the group at the end of the day than there were at the start of the working day. And so it goes, month after month.
  • Lack of trained personnel on the market who want to support the system
Working in support necessitates the same system abilities as working in implementation/development. However, from the standpoint of a professional, the introduction and development of systems provides a variety of objective and subjective benefits:
  • Fills the CV with a wide range of projects and technological expertise.
  • Allows you to participate in the construction of architecture instead of dealing with existing structures.
  • For whatever reason, it is thought to provide more experience in the same length of time than support.
  • Support is, well, out of date.
As a result, professionals with the required abilities for project execution and development are rarely prepared to engage in routine support roles.
Workplace organization. A difficult timetable. Problem-solving
The arrangement of monitoring and restoring the operability of night-time periodic activities for downloading and calculating data was one of the primary challenges to be accomplished. The job of a highly competent specialist at night is costly, and only a few specialists will agree to undertake such work on a long-term basis. First and foremost, efforts were made to optimize and improve the stability of periodic tasks. This allowed night periodic duties to be condensed into the hours between 4:00 and 9:00 a.m., reducing their time. The labor expenditures of restoring task operability have decreased as night processes have become more stable. As a consequence, the time it took to recruit a support expert for monitoring was drastically reduced, and it was limited to the hours of 7 a.m. to 9 a.m., while system users were assured of receiving proper data in the system by the start of the working day (10 a.m.). Further work on the identification and classification of developing problems enabled the partial automation of process restarts and the partial description of problem solutions in comprehensive instructions. As a result, interns and younger specialists were drawn to this job, and they only sought help from a highly qualified specialist on duty if the problem was unique and not indicated in the instructions.
Another challenge was synchronizing weekend and holiday availability of the incident management service's second line. In this situation, the firm has decided to give the crew complete authority. The team assessed which mode would be the most enjoyable to work in based on the load distribution between weekdays and weekends. The final option was a mixed work schedule, with some staff working in 4x4 mode (four days of work from 9 a.m. to 8 p.m., followed by four days-off), and others working in normal mode, as determined by the production calendar. As a consequence, the service availability requirement of 9 a.m. to 8 p.m., seven days a week, was met.
When it comes to work organization, the Company took the position of giving the Team complete autonomy. The employees may be able to establish comfortable working conditions for themselves and have a direct influence on the service delivery process as a result of their decisionsThe only requirement was that at the end of the stabilization phase, the SLA performance targets be reached (3 months). As a result, the team was able to create a comfortable atmosphere, allowing them to focus on the work at hand. The Team has always regarded the implementation of the contract-mandated SLA as the sole criterion for service quality and job success. Based on this, the team was given a clear and straightforward financial incentive scheme, with a monthly bonus tied only to the fulfillment of the SLA service specified in the Customer's contract.
Burnout problem, resource search, staff rotation
One of the most pressing issues was noticed even during the planning stage: team fatigue and the requirement for personnel rotation. The service may last a year, or perhaps a year and a half, depending on the enthusiasm of the initial team, which was charged with developing the process. The crew would then become burnt out, and the service quality would deteriorate. As a result, it became important to plan for staff rotation. The work was divided into two categories:
  • Classification of occurrences and creation of specific software instructions for each type of issue for the 2nd line specialist. As a result, the time it took to add a new specialist to the project was cut in half.
  • The creation of an internship program with two major components. The learner learned broad theoretical and practical abilities for working with all supported systems and products in the first half. The second portion of the training focused on reviewing documentation and honing practical skills in the analysis of typical occurrences, which were mock-ups of problems from the production system that the student would see during his first days on the job.
The second "practical" part allowed for even faster onboarding of a new team member. As a consequence, it took one month of internship and two to four weeks of team work before a new individual was fully immersed in the job on an equal footing with the rest of the team.
With the introduction of the internship program, it became possible to organize the rotation of the team entirely at the expense of interns, rather than relying on market-based recruitment. The planned transition from support to consulting helped to prevent team fatigue and a drop in service quality.
Employees with hyper-responsiveness are in key roles
The precise selection of the initial team composition was the cornerstone of the effective organization of the support service. The men who recognized the key aspects of a process that expanded and became a significant business field were blessed with a valuable quality: a responsible and innovative work attitude. In hindsight, we can certainly state that this was the most significant attribute that enabled us to complete the assignments.
Three years of service operation results
The results have been largely positive, with the crucial metric of SLA performance following the stabilization phase never falling below 95% in three years of operation. In addition, the three-month stabilization time for connecting a new client has been lowered to one month. The technological stack of systems that are supported has also grown. We began by supporting data warehouses built on SAP BW technologies and SAP HANA databases, as well as visualizations built on SAP Analysis and SAP Business Objects. Greenplum and Clickhouse have now been included in the technical line.