Description
Citrix Software, Hardware Maintenance and Technical Support Control Number VA-23-00010721 2 JUSTIFICATION FOR AN EXCEPTION TO FAIR OPPORTUNITY 1. Contracting Activity: Department of Veterans Affairs (VA) Office of Procurement, Acquisition and Logistics Technology Acquisition Center 23 Christopher Way Eatontown, NJ 07724 2. Description of Action: This proposed action is for a firm-fixed-price delivery order issued under the National Aeronautics and Space Administration (NASA) Solutions for Enterprise-Wide Procurement (SEWP) V Governmentwide Acquisition Contract (GWAC) for brand name Citrix software licenses, associated maintenance and technical support to include Citrix Technical Account Manager (TAM) services for both software and hardware, Citrix Managed Services as well as consulting and training services. 3. Description of the Supplies or Services: VA Office of Information and Technology (OIT), Infrastructure Operations (IO) has a requirement for the renewal of maintenance support for existing brand name Citrix software licenses and hardware, Citrix analysis software, Citrix TAM services for both software and hardware, Citrix Managed Services as well as consulting and training services. Citrix software is currently installed throughout the existing VA virtual server infrastructure and provides virtual application and virtual desktop access to endpoint devices inside and outside VA s network. VA s current use of Citrix provides virtual secure accessibility to servers hosting Computerized Patient Records System (CPRS), Veterans Benefit Management System (VBMS) and Burial Operations Support System (BOSS) databases, as well as over 1,300 other VA applications published to users off of existing VA Citrix data center locations. This allows VA to centrally deploy and manage software applications while providing secure, on-demand access to critical business resources accessed 24 hours a day, 7 days a week by internal and external employees, contractors, as well as key external partners such as Veteran Service Organizations. In order to ensure continuity of operations for these critical systems and databases, VA OIT IO requires the renewal of 15,000 existing Citrix Virtual Apps and Desktops Premium Edition software licenses. Additionally, VA requires an optional 5,000 Citrix Virtual Apps and Desktops On-Premises Platinum Edition licenses in order to accommodate a potential surge in the number of remote users. Additionally, VA currently owns 110 Citrix NetScalers that will require continued maintenance. VA requires brand name Citrix software licenses and maintenance for the 110 existing VA Citrix NetScalers that are dedicated to Citrix application, Virtual Private network and Office 365 backend environments used for high availability in order to provide more than 95% uptime. The hardware maintenance will be a pooled capacity model to allow for scaling of capacity. In addition, VA requires the renewal and maintenance of 160 Citrix analysis and monitoring software licenses to monitor, manage and troubleshoot end to end from the client to Citrix servers and Citrix Application Delivery Controllers (ADCs). The associated maintenance support for both software and hardware includes downloads, minor software upgrades, troubleshooting features and 24 hours per day, 7 days per week, 365 days per year technical support for all required Citrix software licenses and hardware. Finally, VA requires technical support which includes dedicated Citrix TAM services for both software and hardware. VA also requires consulting and training services to support the existing and new Citrix software licenses and hardware products installed throughout VA. This shall include full-time dedicated software and hardware TAM services to assist VA staff in ensuring high availability of the Citrix environment while utilizing the least number of servers. This avoids unnecessary cost in underutilized hardware and software and ensures business operations are not negatively impacted by any unscheduled downtime. The TAMs have access to key internal Citrix resources that only Citrix or an authorized reseller of Citrix products and services can provide in addition to assigning correct priorities to any technical case VA opens to mitigate system issues. Because VA implements only a subset of Citrix s hardware and software capabilities, customized training classes are required to teach employees and contract staff maintaining Citrix environments in order to better maintain and improve their operation for end users. It would be inefficient to train VA staff in Citrix features that VA will not be utilizing. Only Citrix Education can provide customized Citrix course content, including the ability to skip standard course content that public training centers are required to teach, which allows VA to optimize the commitment of standard Citrix courses of 40 hours of training per attendee. Citrix Education provides training specific to the technology it offers and the curriculum stays current to the latest technology versions. Citrix offers the ability to hold dedicated sessions to only VA staff and schedule training sessions according to VA s needs. Only authorized Citrix resellers offer Citrix Education vouchers. The Citrix consulting services are required and provide for architecture, design and implementation support for the 10 Citrix environments located throughout the continental United States at VA data center locations. The consulting and training services are procured via Citrix Consulting Vouchers which are redeemed as the training classes and consulting engagements are scheduled. For Citrix Managed Services, the Contractor shall provide specific Citrix monitoring, expert Citrix specific ticket support guidance and collaboration with existing Citrix consultants, support and education contacts assigned to VA 24 hours a day, 7 days a week, 365 days a year. VA requires Managed Services to initially support VA s 15,000 Citrix concurrent user base and requires optional tasks for additional Managed Services, which can be exercised yearly, in increments of 1,000, and up to 20,000 total concurrent users per month to support telework surge efforts. Additionally, VA requires optional tasks of Citrix Consulting services to provide for architecture, design and implementation support for special projects such as telework surge events or resiliency and high availability planning for both the on-premises and cloud Citrix environments located throughout the continental United States at VA and cloud service provider data center locations. Capabilities of Citrix Consulting services include support of architecture design and build, both on-premises and on cloud, as well as rollout and end user support. Citrix Consulting services have Citrix specific knowledge on how to best deploy Citrix systems both on-premises and on cloud as well as the knowledge to provide ticket optimization and end user support techniques. The consulting and training services are procured via Citrix Consulting Vouchers which are redeemed as the consulting engagements are scheduled. In summary, the Citrix software and hardware maintenance and support and managed services will help VA meet the challenges of supporting up to a potential of 20,000 concurrent Citrix users and managing and responding to a potential large number of tickets outstanding in the queue due to any potential telework surge of remote users. The Citrix Managed Services will also provide sustainment support for any future tickets as well as regular maintenance of increased infrastructure. Citrix Consulting will assist VA with optimizing the infrastructure to support the influx of remote users as VA needs have accelerated due to the telework surge. The base period of performance shall be March 29, 2023, through March 28, 2024, with two 12-month option periods for the continuation of Citrix brand name software licenses, associated maintenance and technical support, including TAM services. In addition, there are optional tasks for Citrix Managed Services, consulting, training support and additional software licenses that may be exercised at any time during contract performance as the need arises. The total estimated price of the proposed action is $ inclusive of all options and optional tasks. 4. Statutory Authority: Pursuant to Federal Acquisition Regulation (FAR) 16.505(a)(4), the following brand name requirement is being justified and approved as an exception to fair opportunity under the statutory authority of Section 41 U.S.C. 4106(c)(2) as implemented by FAR 16.505(b)(2)(i)(B), entitled Only one awardee is capable of providing the supplies or services required at the level of quality required because the supplies or services ordered are unique or highly specialized. 5. Rationale Supporting Use of Authority Cited Above: Based on market research, as described in Section 8 of this document, it has been determined that limited competition is available among authorized resellers for the required brand name Citrix software licenses, maintenance, technical support and Citrix Managed Services. Only brand name Citrix items and services meet all of VA s requirements, specifically compatibility and interoperability with VA s existing virtual infrastructure and integration with critical business systems including CPRS, VBMS and BOSS, support of personal Windows and Macintosh endpoints, two-factor authentication exemptions, local printing and two-factor authentication with corporate workstations which connect through the VA Citrix Gateway. Until these capabilities are fully addressed with the new cloud native virtual desktop technologies, VA will need to sustain the use of Citrix to close these gaps. The Citrix software licenses provide the underlying core functionality necessary for application and desktop virtualization currently being utilized in support of VA s datacenter consolidation efforts. Additionally, VA s current remote access infrastructure, comprised of Citrix software licenses and hardware, is configured to only support Citrix software through Citrix ADC appliances deployed in VA s internet gateways. Since the telework surge began in 2020, VA has seen regular use of about 12,000 concurrent Citrix users due to staff working from home, the issuance of government furnished laptops and the introduction of new virtual desktop technologies. The current Citrix infrastructure consists of over 3,000 servers spread across the VA enterprise both on-premises and on the VA Enterprise Cloud that can only work with Citrix software inclusive of the solution providing parsing ability for proxy server sessions to assist in troubleshooting, network analysis and end-to-end performance for Virtual Apps and Desktop sessions to include historical data. The Citrix solution provides the capability of enabling traffic visibility, inclusive of management software, which is based on user application flow statistics in measuring and determining latency on the client side, network side and application side of the connection. Citrix has proprietary constraints within its software. No other brand name software would be able to communicate with these servers. If another brand name software was to be utilized, the current infrastructure would have to be completely replaced with a non-Citrix based solution which would require a similar number of servers to be procured or provisioned, along with all of the associated software to include remote access software. This solution would need to be deployed parallel to the existing Citrix solution to ensure there was seamless coverage. The Government estimates that this effort would take over one year to design, implement and test before it would be available to support production user load and would cost in excess of $82M to restructure the entire infrastructure, including a minimum of $20M in upfront hardware costs. Additionally, VA estimates it would take an additional three years to migrate user loads, which entails identifying a user group to gather requirements, build a prototype virtual desktop and/or application, then have users test, work on any requested revisions and plan to migrate to production. The migration would need to occur in a phased basis due to the complexity and size of the data included. This estimate of time is based on VA's experience with the Microsoft Azure Virtual Desktop (AVD) pilot and the Citrix cloud migration. Citrix is utilized for remote access to over 1,300 applications, which include CPRS, VBMS and BOSS. Users from non-VA furnished endpoint devices are given the option to utilize Citrix for remote access to these applications. Approximately 60,000 unique remote access users use Citrix on a quarterly basis from non-VA issued endpoints thus avoiding the need for VA to procure, deploy and maintain an equivalent number of VA issued endpoint devices and associated software licensure for these users at a sizable capital and operational expense to OIT. VA has approximately 200,000 Citrix remote access users enabled as of November 2022 with 60,000 remote Citrix users logged in during the last 90 days. The purchase of VA issued endpoints and associated software licenses for 60,000 new endpoint devices for remote users would cost VA in excess of $120M in duplicated costs. This duplicated cost estimate is based on previous efforts in obtaining VA issued endpoints and associated software licenses. VA uses Citrix NetScaler appliances to satisfy several security and functional requirements. No additional Citrix NetScalers are being procured at this time. These appliances need to be maintained and are the basis for VA s utilization of the Citrix Virtual Apps and Desktop Premium Edition software licenses. Specifically, VA utilizes Citrix NetScaler appliances in its Trusted Internet Connection gateway in providing remote accessibility from internet endpoints to 15 internal data center locations across the enterprise since 2009 and two VA Enterprise Cloud Service Providers. Load balancing features must be able to provide specific monitoring probes for Citrix XML service and the ability to provide data to a VA Application Delivery Management system for real-time client and network latency metrics, historical reports, end-to-end performance data and troubleshooting, to ensure that individual server software or hardware failures do not negatively impact user access to Citrix from inside or outside the VA network. VA s existing Citrix brand name software licenses and associated hardware require technical support in order to retain the use of the software, for future software version upgrades, and to ensure that software and hardware issues are addressed in a timely manner. Citrix has proprietary rights to the software licenses and associated technical support. Therefore, the required associated technical support must be compatible with the existing software licenses. Only Citrix, or an authorized Citrix reseller, has the rights to perform the associated technical support. Any other source must have access to the proprietary rights to ensure all the associated technical support provided is properly configured to both the software and infrastructure. Without access to the proprietary rights, another source would not be able to push through updates. Due to VA s large, complex Citrix environment, customized training is required. Only authorized Citrix resellers offer Citrix Education vouchers. Standard Citrix Authorized Learning Centers, offered by many Citrix Certified Instructors, are not permitted to customize proprietary course content. Citrix Education is the only authorized provider of Citrix Customized Training and is the only source that can offer the appropriate tailored training for VA, through a variety of methods, such as customized lectures and tailored hands-on labs to meet VA s unique technical training requirement in support of VA s Citrix environment. The tailored training, to include the lectures and hands-on labs that only Citrix Education can provide, will prepare VA staff to maintain and improve VA s complex Citrix environments in a manner that could not be achieved by taking generic, off-the-shelf training. Additionally, only through Citrix Education s customized training can VA optimize the commitment of standard Citrix courses of 40 hours of training per attendee. Managed Services provided by Citrix will allow immediate support for VA s 15,000 concurrent users with maximum flexibility to accommodate increases on a monthly basis during this unprecedented time since the COVID 19 pandemic. Only Citrix Managed Services provides for 24 hours per day, 7 days per week, 365 days per year coverage backed by Citrix Technical Support. Only Citrix Managed Services offer VA the capability of bridging Citrix Product Engineers, Professional Services and Technical Support thereby reducing troubleshooting and remediation times. Citrix Managed Services ensure detailed familiarity of the environment from design through support with internal feedback loops which will allow for a proactive approach which focuses attention on key areas to identify and remediate potentially harmful issues prior to user impact. Citrix Managed Services provides direct lines of communication with Citrix Engineering for necessary hotfixes as well as Citrix Product Management. Direct access to Citrix technical support services is critical to the success of not only ensuring the expansion of VA s remote access infrastructure occurs timely without interruption, but also in assisting VA in eliminating its ticket backlog and putting appropriate mechanisms in place to ensure tickets remain at a manageable level. Citrix Consulting services, which are only available through Citrix or its resellers, provide VA with the required Citrix specific knowledge on how to best deploy Citrix systems both on-premises and on cloud as well as the knowledge to provide ticket optimization and end user support techniques. VA is currently utilizing these services for consulting and training support and requires continuation of these services in order to ensure expansion efforts are successful as there can be no interruption in remote access as critical VA staff, including clinicians, rely on the Citrix Access Gateway environment to perform their duties. If Citrix software licenses were not renewed, VA s entire remote access infrastructure would need to be reengineered and revalidated or VA would have to purchase, deploy and maintain 60,000 additional endpoint devices leading to an inadequate solution and staffs lost work time. Failure to renew Citrix software licenses would impact 24 hours a day, 7 days a week application accessibility from outside VA s firewall to key, critical business applications including CPRS, VBMS and BOSS utilized by a large number of medical, benefits and burial personnel daily. In addition, Citrix virtual desktops are currently a key component of VA s telework and continuity of operations strategy and are utilized daily for both remote network and system access needs of staff and in emergent situations when weather or other disasters impact abilities of VA personnel to report to physical office spaces. Thus, renewal of licenses and technical support of existing Citrix software licenses is vital to ensure the security and availability of VA s information technology infrastructure for over 160,000 Citrix users, more than 60,000 of which access Citrix remotely. 6. Efforts to Obtain Competition: Market research was conducted, the details of which are in Section 8 of this justification. This effort did not yield any additional sources that can meet all of VA s requirements. However, the undersigned VA technical representative determined that limited competition amongst authorized Citrix resellers is available for this brand name acquisition. In accordance with FAR 16.505(a)(4)(iii)(A)(1), this justification will be provided with the solicitation to all appropriate NASA SEWP V GWAC holders. Furthermore, in accordance with FAR 5.301, 16.505(a)(4)(iii) and 16.505(b)(2)(ii)(D), the award notice for this action will be synopsized on the Contract Opportunities website and this justification will be made publicly available within 14 days of award. 7. Actions to Increase Competition: In order to remove or overcome barriers to competition in future acquisitions for this requirement, the Government will continue to perform market research to determine if there are any new products or maintenance services available in the marketplace that would enable future actions to be fully competed. VA is also currently piloting the use of Microsoft s AVD to the greatest extent possible while ensuring that customer needs and budgetary constraints are considered. 8. Market Research: Web-based Market Research was performed by Program Office experts from August through November 2022, during which time VA reviewed Cloud Native solutions from Microsoft AVD and Amazon AppStream and Workspaces. VA has been piloting the potential Microsoft AVD solution in VA for approximately one year. Limitations include two-factor authentication for Macintosh devices, local printing and high latency connections. The Amazon AppStream and Workspaces are not able to support Microsoft Office 365 due to licensing constraints, making it a non-viable solution unless VA identifies and implements a replacement for Microsoft Office 365 on an Enterprise level. The Project Manager also conducted additional market research to ascertain the extent of availability and competition among resellers of the required Citrix software, hardware, associated maintenance and technical support through the use of a Request for Information (RFI) posted to the NASA SEWP V GWAC for renewal of the Brand Name Citrix Requirement. One response was received from a Service-Disabled Veteran-Owned Small Business (SDVOSB) contract holder: Four Points Technology, LLC (Four Points). Additionally, in January 2023, the Contract Specialist conducted a brand agnostic RFI focused on the requirements in an attempt to identify viable substitutes. The RFI was posted to the System for Award Management (SAM) Contract Opportunities page, which also received one complete response for the Brand Name Citrix solution from Four Points, a NASA SEWP SDVOSB contract holder. A second vendor (Vinsys Information Technology Inc.) sent a message noting its interest in the effort but it was awaiting response from the Citrix distributor. After the close of the RFI on January 18, 2023, two other vendors (Sterling Computers and Jeskell Systems) reached out and provided responses. The response from Jeskell Systems on January 24, 2023, was not viable as it noted that the solution is hosted on-premises, which does not meet VA s requirement for a hybrid hosted solution. The response from Sterling Computers on February 2, 2023, was lacking specifics on how the virtual desktop and applications solutions would meet VA s administrative and technical needs beyond the scalability and security related questions asked. There was also no mention of consulting or training in the response. The transition time for any new solution would be multiple years to account for the time to build and transition user groups in phases. VA reached out to both Jeskell and Sterling on February 7, 2023, and notified them of the areas where their RFI responses did not meet VA's technical requirements. VA s technical POC offered to hold a meeting to discuss with the vendors on February 9, 2023. Jeskell accepted a meeting on February 16, 2023, to discuss technical information about its solution. While meeting with Jeskell, it was determined that the company would still be unable to meet VA s technical requirements for the current effort. Sterling Computers also inquired about a meeting on February 7, 2023, and after VA provided potential dates for meetings, Sterling Computers did not respond to VA s available meeting times. The NASA SEWP V Provider Look-up tool was also utilized and 18 SDVOSBs were identified as resellers of the required Citrix software, hardware and associated maintenance and technical support. Therefore, limited competition is anticipated. VA posted an RFI amendment to SAM Contract Opportunities on February 17, 2023, providing notice of intent to solicit a Brand Name Only requirement as an SDVOSB set-aside on the NASA SEWP V platform. 9. Other Facts: None.