by DJ Eshelman, CTA, Nashville CUGC Leader

I get this question a lot: What is the #1 thing you recommend?

As some quick background, in addition to independent consulting, I remain on the ‘virtual bench’ with Citrix Consulting (and have since 2011) so needless to say, I make a LOT of recommendations. In fact, I recently cataloged 335 recommendations in a search for the ‘Top 12’ that I have made over and over and over again… I call these the Dirty Dozen. After giving a presentation at E2EVC (Experts to Experts Virtualization Conference) it was clear I was on to something.

Well, lucky you, CUGC member! You are getting the one I make THE MOST OFTEN…and this time, for free. In fact, this is FIRST of a three-part series (Top 3) that I’m calling the #CitrixHero Top 3 Scavenger Hunt. You can go to my website ctxpro.com to learn where to find the other two tips or keep an eye on social media for the next location! I was really scared about this at first because, well, this is what I’m paid to do as a consultant…but then I realized: This is not proprietary information. Quite the contrary. So, if this saves your company a few thousand dollars, great…but I would much rather have a bigger impact:

I want you to be the #CitrixHero!

So let’s jump in, shall we?

Introduction – Default Settings are the #1 Citrix Mistake!

Did you know that the ‘out of the box’ configuration for every Microsoft OS is NOT optimized for virtual delivery? Microsoft builds the operating system for compatibility, not performance. Citrix Consulting has nearly from day one talked about these optimizations…and the need is actually increasing.

Recent tests by LoginVSI proved that Server 2016 is especially bad in this regard–unoptimized VMs can quite literally cost you thousands of dollars because fewer people can be logged into each VM. Add in the effects of the Meltdown and Spectre remediations and virtual hosted servers are able to do far less with the resources available. This means less users per blade which, at scale, is a huge problem.

This past year, my observations were that 90% of the companies I visited for assessments did not follow all of the Citrix recommended guidelines for optimization. When these companies implemented the optimization steps properly, some saw increased users per blade of over 30%. Add in the tips and tricks I share on my website and…you’ll be drastically increasing the performance overall!

Needless to say it matters. However, fixing it is easy and most tools I talk about are free!

Bottom line: saving your company several thousand dollars = #CitrixHero

Getting Optimized like a #CitrixHero

We know we need to optimize the Operating system for remote use, and in some cases for multi-user uses. Truth is, some of these optimizations can be used on the Control components as well, but for now we will focus on Resource Components (VDAs).

A key bit of information here: Citrix as a delivery mechanism affects the kind of performance we are talking about very little in these cases. I bring this up because recently, I was brought in (too late) to a VMware vs Citrix ‘bake off’ where I found that the people conducting the test ‘stacked the deck’ by optimizing the VMware image, but not the Citrix image. So guess who won the bake off? It isn’t the delivery technology – it is the underlying operating system and the policies by which we control it. Imagine their surprise when they found out just how tuning and some additional components we will talk about later made all the difference in the world!

Server 2016, though it hasn’t had a huge amount of adoption yet, is a challenge. Testing from Login VSI has indicated that unfortunately, 2016 just doesn’t scale anywhere near as well as 2012R2, even when optimized–though when optimized it is a lot closer. Sorry folks, just the way it is.

Hit the Easy Button: Citrix Optimizer (#CitrixHero Button)

Were I to have written this article six years ago, the system optimizations that I perform would take up the whole article. But soon, us consultants realized it was much easier to use Group Policy or scripts and three years ago, that was a thing. But there were problems there too–they didn’t always work and sometimes did more than was beneficial. What was needed was an interactive GUI solution. There are several options for this, primary of which is the Citrix Optimizer–a free tool that automatically detects variations from the recommended tunings and lets you select which ones to apply and which ones not to (for example, on some servers you may want Windows Search to run for Outlook – on others you may not, so optimize and test appropriately). It’s not even worth talking about the others any more!

The reality is that simply running this tool will disable a ton of unneeded services, remove built-in apps, and set certain key registry settings with a few clicks. In all, what this is doing for free is what I used to end up charging clients more than $1,500 worth of effort to complete. You’d be a fool not to take advantage of this tool! Hey, that has a certain ring to it! Don’t be a fool- use the Optimizer Tool! Go ahead and tweet that, I’ll wait, #CitrixHero. Tag me in it: @TheCitrixCoach.

Now, a few caveats:

• TEST. Always test before going into production with changes like this. Just because you hear me say to do something great doesn’t mean that every element of it is what is required for your users. Every use case has little differences which require your attention.
• Always make sure you have the latest templates. I have to call out Martin Zugec here. He does a fantastic job of not only creating this tool but keeping the templates up to date.
• Be careful with certain Optimizer options such as disabling Windows Search. Search may be something your users need for Outlook or other requirements and their user experience will be diminished without it.
• You *can* run Optimizer on Control servers (Broker/Controllers, StoreFront, PVS, etc) but be very aware when you do. Monitor event logs closely and as always… TEST BEFORE YOU DEPLOY.
• Run it from the Network. Here’s a fun tip: you don’t need to have the Optimizer installed to your machines, you can run it (and update it) from a Network share!
• Save your Templates. With the caveat to always check against the latest templates, if you are not using central image management like PVS or MCS, saving a template on the network share as well will save you a lot of time!

Special note here: if you sign up for Citrix Smart Tools, Optimizer Checks can be scheduled for you! Learn more about that and the other system checks at https://docs.citrix.com/en-us/smart-tools/checks/about-health-checks

Optimization Results

First- let me say that Optimization is a lot like pebbles on the beach – a little bit matters very little to a single VM. But over aggregate with many optimizations and many VMs- it matters a whole lot in the place where the true #CitrixHero shines- the CIO’s budget. As I mentioned earlier – I hear feedback from customers that just by doing this ONE THING some increased their Single Server Scalability by over 30%! Go ahead, do some quick math. What would your budget look like if you could do the same work with 30% less servers in your datacenter for VDI?
But do you know what I have NOT heard? I have not had one customer tell me that they’ve lost scalability by doing this. So, it’s absolutely worth your time.

Optional Optimizations

VMware Optimizer

The VMware OSOT can also be run for your workloads, but really this is only a great idea if your HOST is VMware. Even though it will say “Horizon View” the reality is that, once again- these are OS optimizations. So as you can imagine, there is a lot of overlap between settings that the Citrix Optimizer already does. But if you are running on a VMware host (as most people are) this is a good idea!

A caveat with OSOT is that it is community driven, so the odds of getting bad advice is always real. As always: TEST, TEST and then test again! 

Bonus Optimizations!

I’ll be honest- I was originally going to end the article there because honestly, those two things have had the biggest impact on what we call “Single Server Scalability” or, the ability for a single physical host server to house more users.
But there are a few other things that I just can’t ignore here. More stuff that if left at the ‘default’ setting hurts you in the long term, especially at scale.

Sealing the Deal: Image Creation the Right Way

Every time you make changes and updates to an image, you’ll want to reboot at least twice, then run the Windows Cleanup utility (cleanmgr) as an administrator to scan thru not only temp files but older updates that no longer need to be there. The process takes quite a while but it is not uncommon for me to see savings of 2-3 GB by doing this process.

Defragmentation – the Hidden Performance Thief

Another overlooked item, especially with Citrix deployments using a central image (MCS or PVS) is that while the new Cache in RAM with Overflow to Disk functionality, is defragmentation. ‘But I use SSDs,’ you say. Good for you. But the Cache in RAM does NOT cache files. It caches blocks. So if you have fragmented blocks, you can use between 2 and 8 times as much memory to cache. This is because the memory cannot fragment, so if only part of the block is used, too bad. It still uses the whole block. So when you have made changes to the base image and are getting ready to deploy the new snapshot or vDisk version… first make sure you defragment the image as part of your sealing process.

*Quick tip on PVS- a cool/advanced way to Defragment is to first clean up the image and shut it down. Mount the vDisk on the PVS server, where you can defragment it as an attached disk using the built-in tools. It is not only faster, but you are able to defragment files that normally would be in use.

Going Third Party: BIS-F

Confession time. I’m not very often allowed to recommend Third Party…anything. The reason is solid; Citrix can’t support it! And because I never want to walk a customer into a potential pitfall let me be clear: This solution is NOT SUPPORTED BY CITRIX. That being said, if you want to take your optimization and automation to the next level, I’d encourage you to test another tool from some people I know and trust. I’ve used their tools myself and it works very well. Sure, you’re on the hook to support it yourself but… trust me that it is worth it!

BIS-F stands for Base Image Script Framework. The goal was to have a single ‘master optimization’ script that does everything you should be doing when you ‘seal’ an image for distribution from either PVS or Machine Creation Services. The list of tasks the script performs is long and distinguished… but a few highlights are making sure Windows activation (KMS) is working properly, that the drive is optimized (defragmented). This is a script you run after making changes and updates to your master image, so nothing stays running! It is the last thing you end up doing before shutting it down for distribution.

Another cool feature is that the script can detect non-Citrix software running and optimize those too. Like running a purge of WEM cache and scanning AntiVirus to mark the disk as safe to a lot of newer programs that can take advantage of this feature (less files to scan = better performance). Also, something that is missed all too frequently: .Net Optimization. What this does is initiate a process that often runs at startup. The problem with this in a non-persistent desktop, it would happen every time the image starts up. I have seen this cause massive issues for host CPU in VDI environments. Again- important details that are frequently missed. Thank goodness for scripts.

Antivirus Defaults – the Secret Performance Killer

Just like Microsoft’s OS settings are made to fit a wide array of solutions ‘out of the box’ so also are AntiVirus and Anti-Malware programs. While they are getting better (check out Bitdefender’s Hypervisor Introspection with XenServer if you have doubts about that) there is still management that needs to be done.

I encourage you to look at this Citrix article, but at a high level:

• Make sure to exclude Citrix executables
• Exclude system files like the page file, print spooler and certain cache directories
• Set scanning to Write Only, especially on non-persistent MCS and PVS workloads
• Do not perform scheduled scans on MCS and PVS workloads (use the BIS-F tool or manually scan before sealing the disk)

There’s more here… but I have one more bonus tip for you today.

The Hidden Resource Hog: Ads and Tracking

My friend Dan Allen has been harping on this for a long time and he’s absolutely right: with everyone using web browsers all day long, the persistence of advertisements is inevitable even in the workplace. Once again, left to the defaults, the browser will simply consume all it is told to consume, even ads and tracking. But the good news is there are lot of ways this can be reduced or eliminated. Reduced to the tune of over 35% less resource consumption according to this whitepaper.

Here’s a list of options, in order of difficulty to deploy:

• Use a Group Policy Object to enable IE Tracking Protection
• Use the Ublock plugin (trust me when I tell you to not use Adblock plus – uBlock is more efficient)
• Attack the problem via DNS or hosts file (again- test this but I’ve found it VERY effective)
• Force your users to use Lynx (… I just dated myself I think.)

Summary 

So in summary, remember that the default settings are not there for your benefit, but they are there for your overall safety. They are the compromise draft–the deal no one was happy with, but work around it.

Test it, tweak it, then test again.

If you master these you are on your way to being a #CitrixHero!

Finally–I am planning on doing a live Q&A series on all of my top tips soon. If you would be interested in that, send me a direct message on Twitter and I’ll get you information on joining me!