Evaluation of New NIH Categorization Tool to Support RCDC

Notice of Intent Evaluation of New NIH Categorization Tool to Support RCDC In accordance with FAR Parts 12 and 15, the National Institutes of Health (NIH) Office of the Director (OD The Office of Extramural Research intends to procure software on a sole source basis from Recommind, Inc., 650 California Street, San Francisco, CA 94108 under the authority of FAR 6.302-1(a)(2) to provide product software and software installation and operational support to test its search and data extraction capabilities using large NIH data sets. Once the search and data extraction tests are completed, NIH Federal staff will compare the results to SIS software for the same data sets. The results of the comparison may lead NIH to replace SIS with Recommind for the RCDC search and data extraction applications. The North American Industry Classification System Code (NAICS) is 423430 Computer and Computer Peripheral Equipment and Software Merchant Wholesalers size standard of 100 employees. The period of performance shall be twelve (12) months from the Date of Award. Purpose/Objective: The Office of Extramural Research has been directed by NIH leadership to conduct a process evaluation to assess the NIH categorization program as it is supported by the Research, Condition and Disease Categorization system (RCDC), comparing to two alternative approaches for indexing and categorization. The evaluation will assess process endpoints, including program efficiency in terms of man hours devoted to research categorization, as well as the quality of the current categorization product, compared to results associated with the alternative semantic analysis approaches. The purpose of this requirement is to attempt to quantify the time staff members devote to categorization and determine whether an alternative approach involves less effort. In addition, some staff members believe that other software engines (used by RCDC) produces categorization results that are less accurate than might be possible with categorization software that relies upon a statistical categorization approach. The evaluation goal is to help the NIH decide whether it should continue to rely upon the existing technology, or incorporate new informatics technology, as a primary component of the research categorization program. Ultimately, the goal is for the NIH to support an efficient, increasingly precise and accurate categorization program that produces high quality results. The categorization engine that will be compared to the current engine is the Recommind engine. Recommind employs a proprietary statistical algorithm for indexing that is similar to probabilistic latent semantic analysis, and a machine-learning approach to categorization. In Recommind, the user identifies a set of documents as examples for the category. Recommind analyzes these documents, searches for others which are similar, then provides a sample of similar test documents, along with a sample of test documents it believes are dissimilar, and prompts the user to review and rate the documents as similar/dissimilar. Recommind repeats this process until a defined threshold level of correctness is achieved by the system. Contractor Tasks: The Contractor shall perform the following tasks: 1. The contractor shall provide software license(s), software maintenance for the base year, and on-site technical support to the Office of Research Information Systems (ORIS) to support the following activities. 2. Categorize the FY 2012 NIH research projects using a statistical categorization product for 16 selected categories (five new RCDC categories and 11 established categories) 3. Evaluate the accuracy of the statistical categorization product in comparison to the traditional RCDC fingerprint process and a "hybrid" process in which RCDC is utilized in a machine-learning environment. 4. Collect validity feedback on the project lists generated by the three approaches from the IC subject matter experts assigned to work on the 16 categories. 5. Collect user feedback from subject matter experts on the usability and acceptability of the three alternative approaches to categorization 6. Collect dairies from subject matter experts to calculate estimates of the effort required from IC subject matter experts to support the NIH research categorization program. 7. Optional Requirement - If required, the contractor shall provide software maintenance for option year 1 of the contract. Deliverables • Software licenses for two 4-processor servers • Software license maintenance support - for base contract year • On-site technical support for installing and configuring contractor software, and providing technical support in using the contractor-provided software • Optional - 2nd year of software maintenance support Period of Performance: The anticipated period of performance will be twelve (12) months from the Date of Award. Program Management and Control Requirements: The OBSSR Contracting Officer's Representative (COR) will exercise management control over the project and will communicate with the Contractor. Telephone and e-mail conversations as well as in person meetings between OBSSR staff and the Contractor are expected to identify unanticipated issues that will be documented and resolved. Inspection and Acceptance Requirements: The OBSSR COR will solicit a final written progress report for the performance period, provide feedback to the Contractor, and determine acceptability of progress updates and final reports. The OBSSR COR will approve Contractor payments upon completion and approval of the deliverable(s) and invoice(s). Sole Source Justification: Recommind is one of a very small number of companies that provide commercial off the shelf (COTS) search and data extraction software for research data. NIH conducted market research and did preliminary assessments on the leading software products and Recommind scored highest on all categories in comparison to the other vendors' products (excluding current engine). As such, Recommind has demonstrated the greatest potential for use at NIH if current software is replaced. The office currently has another software engine in place to provide software for our Research Condition & Disease Categorization (RCDC) system. At the direction of a committee headed by Dr. Tabak, a white paper analysis was prepared of other commercially available products that are similar to the current engine, because the committee was concerned that it may not be the best product for NIH and RCDC. The product that was judged to be the best, based on literature reviews and comments by non-NIH users, which is Recommind. Recommind will provide product software and software installation and operational support to test its search and data extraction capabilities using large NIH data sets. Once the search and data extraction tests are completed, NIH Federal staff will compare the results to SIS software for the same data sets. The results of the comparison may lead NIH to replace the current engine with Recommind for the RCDC search and data extraction applications. As part of the market research, NIH compared Recommind software costs to other vendors. The Recommind license costs are comparable or lower than and comparable to the other vendors It is recommended that this be awarded to Recommind based on the study. The following provisions and clauses apply to this acquisition and are incorporated by reference. Full text may be found at https://www.acquisition.gov/Far FEDERAL ACQUISITION REQULATION (FAR) CLAUSES FAR 52.224-1 Privacy Act Notification - This SOW requires the Contractor to do one or more of the following: design, develop, or operate a system of records on individuals to accomplish an agency function in accordance with the Privacy Act of 1974, Public Law 93-579, December 31, 1974 (5 USC 552a) and applicable agency regulations. Violation of the Act may involve the imposition of criminal penalties. The Privacy Act System of Records applicable to this project is Number 09-25-0106. FAR 52.224-2 Privacy Act FAR 52.212-4 Contract Terms and Conditions-Commercial Items FAR 52.212-5 Contract Terms and Conditions Required to Implement Statutes and Executive orders This is not a Request for Proposals (RFP), nor is a RFP available; however, all responsive sources may submit a proposal in a timely manner which will be considered by NLM. Firms interested in responding to this notice must be able to provide the referenced service as specified above. Responses must be in writing and must be received electronically at the Government infrastructure by 2:00 PM EST on Tuesday, September 11, 2012. Proposals must include pricing information and should reference Solicitation No. NIHLM2012551/EJB and should be submitted to [email protected].