by Dan Schlimme, CUGC Steering Committee
My XA 7.6 upgrade and PVS Implementation
I recently performed an upgrade of our Epic application to Epic 2015 and our XA farm from XA 6.5 to XA 7.6 LTSR with the help of PVS and wanted to share with you my experience.
To start off, YES, we did two major upgrades at the exact same time, but for this post, I am going to focus on the XA migration and the PVS implementation from a high level. I will go over the pros and cons of what we experienced. Keep in mind this is strictly from my environment.
I will begin with a bit of background. My environment was XenApp 6.5 and ALL physical. We implemented Epic before they signed off on the use of virtual which put us at a big disadvantage for maintenance and as many of you know, the numerous SUs that Epic pushes out. We had a total of 192 physical servers. As you guessed, that meant 192 installs of Epic, third party applications, Windows patching, AV pushes, and the process of finding a needle in a haystack if a server was acting up.
I took the upgrade as a time to move away from the physical life and move to the virtual. This also opened the opportunity for me to implement PVS. We have two data centers which allowed for the perfect DR setup. I have a PVS environment running out of both data centers and segmented from one another by network segment, ESX hosts and storage. We utilized 56 ESX hosts at each data center, 112 in total. I also utilized 8TB of storage at each data center, 16TB in total for the image storage and the 15GB thin disk on each VDA for overflow. With this setup I was 440 guests at each data center, 880 total. This allowed for 8 guests per host with room to add one more guest per host and an extra host in the event we grew quicker than expected.
As you recall from earlier, we did have 96 physical servers per data center which would take our Epic team literally days to touch each server and install needed components. Once that was completed, we would point users to the data center that was completed, wait for users to drain off of the previously used data center, then update all of those. Obviously this is very time consuming and frustrating to plan around.
We now have the ability to simply update the image at each data center, set a time for it to be promoted to production and we are done. This took some getting used to for our Epic team but they very quickly got the hang of it and loved it. We went from days of work down to minutes of work with an immediate fallback solution of reverting to the old image which has actually been used and worked wonderfully. The mindset went from planning and dreading updates to it becoming something as simple as updating your own pc and they couldn’t be happier.
With the implementation of PVS we also migrated Epic from XA 6.5 to 7.6 LTSR. I built this out the same as I did for my PVS environment. The DDCs are separated by data center, ESX hosts and storage. I also updated the respected image to only utilize the DDCs located in that data center (Listed Below). This, just like our PVS environment, is fully redundant and built for DR.
There were some differences between XA 6.5 and XA 7.6 that are important to note.
- There are no Worker Groups, there are Delivery Groups
- This was crucial for us in XA 6.5 to move connections from one data center to another. We did find a way around it with a PowerShell script (Listed Below)
- There is no policy for server reboot schedules
- This was crucial for us in XA 6.5 to ensure that our servers were rebooted weekly. We did find a way around it by adding a tag to each server and running a PowerShell script (Listed Below)
- The policies are no longer broken up by Computer and User
- This wasn’t a big deal but just wanted to share
- Sessions talk to the database in order to launch an application
- This is called leasing and unlike earlier versions of XA, this puts a HUGE focus on the database in the event that it is down or you are a new session to the environment.
- This is the biggest gotcha that we came across and I will go more into that below.
During our go live, everything was going very smoothly until we began hitting a user count of around 4000 concurrent users. We spent most of the weekend on the phone with Citrix troubleshooting and it turns out that in 7.6 LTSR, there were some unknown issues with the database leasing. In short, when you hit a concurrent session amount of roughly 4000 the database began locking and not leasing out sessions to anyone past that. This resulted in all users past the 4000 mark stuck at the XA launch bar but would never connect. The database stores information for that user/session for a period of 14 days I believe. If that user has not launched a session in 14 days, that leasing record is removed from the database. With a flood of 15000 users accessing the environment at one time, the database was creating leasing records and simply couldn’t keep up. Unfortunately this isn’t something we could test for. Citrix provided us a number of scripts to run on the DB which resolved the issue for us. This prompted Citrix to release a KB with more information. (Listed Below). This is now resolved in CU1, which you should plan on rolling out into your environment before any go-lives.
Outside of our go-live hiccup and the the small differences in XA 7.6, the environment is very stable, robust and redundant. I really like the Director web portal for monitoring and managing the environment. It is a great centralized location for everything you need to know.
Items Mentioned in Article
- Force DDC Reg Change (On the VDA)
- Navigate to HKLM\Software\Citrix\VirtualDesktopAgent
- listofDDCs
- List the FQDN of the DDCs you want the VDA to register to with a space separating them
- RegistryOverridesAutoUpdate
- Set this to 1
- Modify the following entries
- listofDDCs
- Application Delivery Group Change Script
- I run this from the DDC itself by selecting Powershell from Studio
- If the App is located within a folder in Studio you will want to make note of that for the AppName
- Run the following two commands and this will point your app to the new Delivery Group and Remove it from the current Delivery Group
- Add-BrokerApplication -Name “AppName” -DesktopGroup “New Delivery Group“
- Remove-BrokerApplication -Name “AppName” -DesktopGroup “Current Delivery Group“
- Server Reboot Script
- I run this from the DDC itself as a scheduled task
- This can be found here: Server Reboot Script
Add-PSSnapin citrix*
$ServerList = Get-BrokerMachine -MaxRecordCount 1500 | where {($_.Tags -contains “Anything you want this to be“)}
$ServerList | ForEach-Object {
# Set the target machines to maintenance mode to drain sessions
Set-BrokerMachine -Machinename $_.MachineName -InMaintenanceMode 1
}
# Wait 1 hour before starting to reboot any machines
start-sleep 3600
$ServerList | ForEach-Object {
#Reboot the machine
new-brokerhostingpoweraction -Action Restart -MachineName $_.MachineName
#Remove maintenance mode
Set-BrokerMachine -MachineName $_.MachineName -InMaintenanceMode 0
#End Powershell Session
}
- XA Database KB
