How Positive Thinking Company approached Log4Shell remediation with Citrix ADC

Regular Sundays are made for reading sessions and long walks in the park. Unfortunately, Sunday December 5th turned out not to be a regular Sunday as our team members got notified about a potential huge vulnerability related to Apache Log4j Library. This is how the journey started with our Citrix ADC Managed Services team.

The immediate response of our team started 2 simultaneous tracks:

• Verify if the managed service platform was vulnerable
• Verify if any managed Citrix ADC products were vulnerable

Managed Services platform timeline

• 09 Dec 2021 [Security Information]: Vulnerability disclosed by Log4j project Github.
• 11 Dec 2021 [Manage Service Platform]: Ongoing analysis for all used components.
• 12 Dec 2021 [Citrix ADC]: Citrix confirms ADC itself is not vulnerable (except potentially for the WIonNS feature which is not used by any of our customers) We were able to figure this out in record time because our config analyzer’s rule set can be updated very quickly and a new analysis on all the configs can be done on the fly. Thanks to dashboards we can very quickly and swiftly analyze the results.
• 13 Dec 2021 [Managed Service Platform]: Elastic updated their mitigation page adding for Logstash that information leak could also be possible. However, it is important to note that our docker Logstash containers are not logging to file by default and should have no reason to log the malicious texts (Not directly reachable and customer ADC logs are not handled by log4j).
• 13 Dec 2021 [Managed Services Platform]: Applied the mitigation for Logstash in any case to the docker containers we use and activated the new containers.
• 13 Dec 2021 [Managed Services Platform] – 15:00: Impact analysis complete and continuing with follow up actions like clean up and report writing.

Citrix ADC timeline

In parallel there was also a track with our Managed Services customers. Although the Citrix ADCs that we manage for our customers were not vulnerable, they potentially reversed proxy a lot of applications that might have been vulnerable. We therefore developed a responder policy to detect and block possible malicious attempts to exploit this vulnerability. We gave every customer the choice to activate this additional protection. Some customers wanted it in logging only mode, because the Citrix ADC was fronted by a 4^th Gen Firewall that is supposed to catch it, other customers wanted us to put it in blocking mode all for different reasons.

Nevertheless, our Citrix ADC managed services customers were happy to receive additional protection from their ADC and our team giving them more time to:

• Verify 4^th Gen firewall functionality
• Deploy the patch to the vulnerable components
• Analyze the situation
• Verify if patching and mitigation went well

This situation illustrates perfectly how our customers benefit from the Citrix ADC Managed Service when things get rocky. In these many cases we can exploit Citrix ADCs’ features to the fullest in order to give your data and your applications the best protection possible. The whole security protection game is about layering. Every added layer, like our service, can help you build in additional protection that makes it more swift, accurate and robust.