by John Smith, CTP

I left the enterprise approximately 18 months ago after being a cubicle drone for the last 18 years.  I now work for a software company that makes a wire data analytics platform for providing operational intelligence to organizations around their applications, the data that traverses their wire and basically shines light on the somewhat opaque world of packet analysis.

I can honestly say, after sitting in “war rooms” off and on for the better part of the last 15 years (I did my 3 year stint in support like a lot of us did) I have concluded that perhaps it is time for a new methodology.  I say this as a person who has largely supported Citrix environments and has become all too familiar with the moving parts of tiered applications and the tiered infrastructure that supports Citrix.  We currently have KPIs, we collect logs and we have some fantastic PowerShell scripts that have given us some great visibility into our environments but many of these are very broad and lack the precision needed in today’s distributed application environments.  One KPI could be a measurement of a business transaction that involves over a half dozen packet-level conversations, any one of which could derail the performance.  

My first job out of College was with Maricopa County Environmental Health (I was the health inspector) and I was introduced to a concept called HACCP (Hazard Analysis Critical Control Point) and I think some of what I learned from it can be very relevant in analyzing today’s distributed and often problematic environments.

HACCP, pronounced “hassup”, is a methodology of ensuring food safety by the development of a series of processes that ensure, in most cases, that no one gets sick from eating your food.  It involves evaluating the ingredients of each dish and determining which food is potentially hazardous and what steps need to be taken to ensure that quality is ensured/maintained from food prep to serving. 

An example of a HACCP based SOP would be:

• The cooling of all pre-cooked foods will ensure that foods are cooled from 135 degrees to 70 degrees within two hours
• The entire cooling process from 135 degrees to 41 degrees will not take more than 6 hours.

So, I am taking away the “H” and putting in a “P” for PACCP. I am proposing that we do the same for our applications and systems that we support at the packet level.  Just as ingredients may have chicken, cheese and other potentially hazardous ingredients applications may have ICA Latency, CIFS Calls, Active Directory (LDAP, Kerberos) transactions and SQL Queries.   We need to understand each part of an infrastructure that could potentially derail the performance of an application and what an approved baseline is, what mitigation steps to take and who is responsible for maintaining it.  Let’s take a look at the 7 HACCP/PACCP principles. 

Principle 1 – Conduct a Hazard Packet Analysis
The application of this principle involves listing the steps in the process and identifying where significant degradation is likely to occur. The PACCP analysis will focus on hazards that can be prevented, eliminated or controlled by the PACCP plan. A justification for including or excluding the hazard is reported and possible control measures are identified.

Principle 2 – Identify the Critical Control Points
A critical control point (CCP) is a point, transaction or process at which control can be applied and poor performance can be avoided, eliminated or reduced to acceptable levels.

Principle 3 – Establish Critical Limits
A full understanding of acceptable thresholds, ports and protocols of specific transactions will help with identifying when CCP is outside an acceptable use. 

Principle 4 – Monitor Critical Control Point
Come up with monitoring procedures for specific critical control points.  For me, being a wire data person, I would look at the wire to find transaction times but there are many ways of monitoring critical control points both agent-based and non-agent based. 

Principle 5 – Establish Corrective Action
Part of this is not only understanding what to do when a specific critical control point is not within an acceptable threshold but to also establish who owns the systems involved in each CCP.  For example, a Critical Control Point for Citrix logons is Active Directory, something usually outside the Citrix team’s purview.  You would understand the team you would escalate to as part of the corrective action.  (Someone OTHER than the Citrix team) 

Principle 6 – Verification
Verification would be in the form of dashboards, reporting/alerting and basic operations.  While the Citrix teams may not own every specific Critical Control Point, they should at least be aware and verify that all CCPs are within thresholds if they are part of their supported applications.

Principle 7 – Recordkeeping
Recordkeeping is generally already a part of several regulatory frameworks in existence today.  Long term recordkeeping can result in the ability to intelligently forecast server workloads and general hardware requirements in the future.  Solid record keeping should position you and your team to be able to answer questions such as when the busiest times are, how many users will be on the system during work hours, during weekends when maintenance is performed, etc. 

So what would a PACCP profile look like?

So, let’s take an easy example of what a PACCP profile may look like, I don’t want the article to be 20 pages long so we will leave the Citrix infrastructure alone for the time being and just build out a PACCP profile for a simple application. 

Let’s say you have a 4-tier application that in addition to your VDA’s acting as ICA Servers the application includes a web tier accessed via a published browser, a back end web farm, a middle tier that leverages an external API call and a back end Database. 

So we have the following Critical Control Points (CCPs) for this application.

• The initial web transaction over port 80 or port 443.
• The Web servers making a call to the SOAP/REST tier.
• The SOAP/REST Tier making an external call to an API provider then interacting with a back end database server.
• The database server conducting database transactions.

In most scenarios, as many of your have become painfully aware, any failure within these Critical Control Points (CCPs) will be escalated as “a Citrix issue.”  This is something that we have been spending our time on for years.  A PACCP profile would outline the details of an application and would call out who is responsible for each tier of the application.  What I am proposing is that we use the PACCP principles to identify what is an acceptable performance metric for each one of these CCPs, find out who needs to be escalated to in the event they are outside their thresholds and what types of traffic we should be seeing.

As a result of principles 2/3 of PACCP, we will have identified what types of traffic should be traversing these CCPs and we will identify the process time that is acceptable.  If you note that the External API calls are taking 7 seconds than you might need to alert your external API provider.  Likewise, if you see a series of HTTP 500 errors at the SOAP/REST tier you would make sure you alert those system owners. 

So we looked at an example of a HACCP SOP around food prep, let’s see what a PACCP SOP may look like:

• CCP: Web Tier to the VDA
  • All Web transactions will occur in less than 3000ms (keep in mind, this is at the front end so you will wait for the rest of the back end to complete)
  • Error rates will be lower than 1% (HTTP 500 and above)
  • Access will occur over ports 80 and 443
• CCP: Web Tier to SOAP Tier
  • All SOAP Tier transactions will complete in under 500ms
  • SOAP errors will be under 1%
  • All SOAP Tier transactions will occur over port 8080
• CCP: SOAP Tier to External API Provider
  • All External API Transactions will occur over port 443
  • All External API transactions will complete in under 500ms
  • Error rates will be under 1%
• CCP: SOAP Tier to Database Server
  • All Database transactions will complete in under 500ms
  • Database errors will be under 1%
  • All Database transactions will occur over port 1433
  • Database queries will be made via stored procedures

In addition to the SOP that we have, we will also know who is responsible for each CCP and know which team to escalate to. 

Conclusion:
The time has come to stop “publishing” applications and start onboarding them.  In a recent study sponsored by eGinnovations and Doug Brown at dabcc.com, they took what was largely an anecdotal statement and provided some evidence around it.   The white paper surveyed a group of Citrix professionals on how much time they spend supporting issues outside of their Citrix environment.  65% of the responders stated that the issue is a NON-Citrix issue at least 50% of the time, nearly 40% of the respondents stated that issues were not Citrix related at least 70% of the time. 

We have all spent a great deal of our time doing someone else’s job by virtue of having applications published on our XenApp and XenDesktop environments.  The time has come for a new methodology for identifying Critical Control Points (CCPs) within the applications we are supporting to ensure that the time we spend doing support for issues outside our purview and core competency is limited to those who should be held responsible.  I am not implying that we should try to get out of work or not be proactive.  I understand the realities when a user says “Citrix” while on the phone with the helpdesk/service desk.  PACCP positions us to be proactive and at least have an understanding of the anatomy of the applications we are supporting which should keep the time we spend outside the Citrix environment to a minimum. I think that given the regularity that Citrix teams spend investigating issues outside their environment, it is fair to ask that before we publish or make an application available, that the application owners be able to identify the critical control points that are involved.

I plan to continue to develop this strategy and I would welcome any/all participation and assistance and I will ensure those who want to contribute are appropriately cited and credited. 

A special thanks to Doug Brown and eGinnovations for conducting a very informative study.  

Thanks for reading!!! 

John M. Smith