This is the accessible text file for Library of Congress Office of the Inspector General Audit report number 2020-IT-101 entitled United States Copyright Office � Public Records Systems Development Audit which was released on November 17, 2021. Office of the Inspector General Library of Congress Memo Date November 17, 2021 To Dr. Carla Hayden Librarian of Congress From Kimberly Byrd Acting Inspector General Subject United States Copyright Office � Public Records Systems Development Audit report number 2020-IT-101 This transmits our final report for the Office of the Inspector General�s audit of the Library of Congress�s United States Copyright Office Public Records Systems Development. The report contains nine recommendations intended to improve management and minimize operational and cost inefficiencies for the Public Records systems development effort, as well as future information technology projects. Based on management's written responses to the draft report, we consider all of the nine recommendations resolved. Your response provided an action plan for the implementation for those recommendations, in accordance with LCR 9-160, Rights and Responsibilities of Employees to the Inspector General, section 6.A. We appreciate the cooperation and courtesies extended by the Copyright Office and the Office of the Chief Information Officer. cc Principal Deputy Librarian Chief Information Officer Register of Copyrights Chief Financial Officer General Counsel Summary The Office of the Inspector General (OIG) engaged an independent audit firm, Cotton and Company, LLP, to evaluate the United States Copyright Office�s (USCO) Information Technology Modernization efforts as supported by the Office of the Chief Information Officer�s (OCIO). USCO serves as a separate federal department within the Library, under the general oversight of the Librarian of Congress, pursuant to specific statutory authorities set forth in the U.S. Copyright Act. Under the direction of the Register of Copyrights, USCO administers a complex and dynamic set of laws, including registration, the recordation of title and licenses, a number of statutory licensing provisions, and other aspects of the 1976 Copyright Act and the 1998 Digital Millennium Copyright Act. To better serve its constituencies, USCO has dedicated itself to modernizing its systems and its administration of the nation�s copyright laws. Based on Congressional direction, USCO began the modernization process by conducting a detailed analysis and review of its systems, then laid out its vision for its future-state IT enterprise solution in two documents: Strategic Plan 2016�2020: Positioning the United States Copyright Office for the Future (issued December 1, 2015) and Provisional Information Technology Modernization Plan and Cost Analysis (issued February 29, 2016). USCO has dedicated significant resources and attention to modernization. Currently, the Office is in the midst of various business and technology modernization initiatives with key objectives of transforming all of USCO�s multiple IT systems to a single, improved, and integrated Enterprise Copyright System (ECS) and ensuring that non-IT activities align with the Office�s strategic goals. The ECS is comprised of functional components for Recordation, Registration, Online Public Records, and Licensing. The OIG requested Cotton & Company (Cotton) to conduct a performance audit of USCO and OCIO�s efforts to develop the new USCO IT environment and business applications, particularly with regard to the current development work related to the USCO Public Records System Development Project (OCIO PMO Project #546). The audit objectives were to evaluate: a) USCO and OCIO�s project management and software development practices using GAO�s Schedule Assessment Guide, Cost Estimating and Assessment Guide, and Software Development: Effective Practices and Federal Challenges in Applying Agile Methodologies; b) whether the representation of the project in Copyright Modernization Office�s (CMO) integrated master schedule (IMS) is comprehensive and sufficient; and c) USCO and OCIO�s progress in closing the 12 prior findings from the OIG�s August 2019 report entitled, Library Working Through Agile Delivery Method Challenges for Copyright IT Modernization Project (OIG Report No. 2018-IT-107). What the Audit Found In its report, Cotton provided conclusions for each audit objective and areas for improvement summarized as follows. Non-comprehensive Cost Estimate. The Library did not have a comprehensive cost estimate for the Public Records project. Developing reliable cost estimates is crucial for realistic program planning, budgeting, and management. A cost estimate is the summation of individual cost elements, using established methods and valid data, to estimate the future costs of a program, based on what is known to-date. Cotton attributed this to the OCIO Project Management Office (PMO) not having developed a well-documented cost estimating process. This contributed to the inability of the USCO Copyright Management Office (CMO) [Footnote 1] to baseline the IMS. The USCO CMO contracted a professional services firm to assist with completing an IMS, which ran for a year at a cost of almost $530,000. The final IMS document from the firm could not be baselined due to unreliable or missing cost information. Despite the missing Library information, the contractor used what it deemed to be an industry average to develop a cost estimate for the Public Records initiative. While the Library did not agree with the contractor�s conclusions, the estimate provided by the contractor was more than double the Library�s initial estimate ($15.3 million estimated by the contractor and $7.5 million estimated by the Library.) Incomplete Public Records Project Schedule. Despite USCO having begun its Public Records project in October 2019, the project schedule available during the review was not comprehensive or sufficient. Specifically, the project schedule only contained the completed and scheduled sprints for the first two years of the project, but did not include key release milestones and Work Breakdown Structures, as required by the OCIO PMO Scheduling Guidance. Unapproved IT Governance Policies and Procedures. The approved and published versions of Library of Congress Regulation 5-130, Information Technology Investment Management and Library of Congress Directive 5-130.1, IT Investment Management did not accurately reflect the current IT governance structure in place operation at the Library. In January 2019, OCIO implemented a revised IT governance structure in advance of receiving required Library approvals, as documented in LCR 1-710, Library of Congress Regulations (LCRs) and Directives (LCDs). Prior Findings and Recommendations. The Library closed seven of the 12 recommendations from the OIG�s August 2019 report entitled, Library Working Through Agile Delivery Method Challenges for Copyright IT Modernization Project (OIG Report No. 2018-IT-107). Recommendations Cotton made nine recommendations designed to assist the Library in improving its project planning, implementation, and oversight. Management Comments In response to the draft report, Library senior management agreed with all nine recommendations (OIG Appendix B). Cotton & Company Audit Report LIBRARY OF CONGRESS U.S. COPYRIGHT OFFICE�S PUBLIC RECORDS PERFORMANCE AUDIT REPORT November 10, 2021 Cotton & Company LLP 333 John Carlyle Street Suite 500 Alexandria, Virginia 22314 703.836.6701 [voice] 703.836.0941 [fax] www.cottoncpa.com Loren Schwartz, CPA, CISA, CISSP Partner lschwartz@cottoncpa.com Ms. Kimberly Byrd Inspector General Office of Inspector General U.S. Library of Congress Dear Ms. Byrd, Cotton & Company LLP is pleased to submit the attached audit report detailing the results of our performance audit of the U.S. Copyright Office�s (USCO�s) and Office of the Chief Information Officer�s (OCIO�s) efforts to develop the USCO Public Records System Development Project (OCIO Project Management Office [PMO] Project #546). The Library�s Office of Inspector General (OIG) engaged Cotton & Company to conduct this performance audit pursuant to Contract Number LCOIG20D0004. Cotton & Company performed the work from August 2020 through April 2021. As a result of the Coronavirus pandemic, all work was performed remotely in the Washington, D.C., metropolitan area. We conducted this performance audit in accordance with Generally Accepted Government Auditing Standards, 2018 Revision, issued by the Comptroller General of the U.S. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Sincerely, Cotton & Company LLP Loren Schwartz CPA, CISSP, CISA Partner EXECUTIVE SUMMARY The U.S. Copyright Office (USCO)�headed by the Register of Copyrights�is statutorily responsible for administering the nation�s copyright laws. The work of the USCO includes a number of public services, as well as extensive legal and policy portfolios. Among the services provided to the public are: examination and registration of copyright claims and documentation related to a broad range of creative output; administration of deposit requirements; certification of copyright ownership; recordation of transfers, assignments, licenses, and other copyright transactions; and provision of general copyright information to registrants and the public at large, including through the operation of a call center available to the public. The USCO also administers statutory licensing royalties and distributes hundreds of millions of dollars annually in royalty payments. Modernization of USCO�s information technology (IT) systems has been a top priority of the Office since 2011, beginning with a detailed analysis and review of USCO�s systems. The USCO laid out its vision for this future-state IT enterprise solution in two documents: the Strategic Plan 2016�2020: Positioning the United States Copyright Office for the Future and the Provisional Information Technology Modernization Plan and Cost Analysis. The USCO�s IT Modernization Plan is a multi-year effort to transform USCO�s IT systems into a single and improved Enterprise Copyright System (ECS). Products to be developed include: An enhanced recordation system. [Footnote 1] - An improved registration system. - A public records system. - Systems for USCO office workflow and creative community engagement. - A modern Licensing Division system. The Library Office of the Inspector General (OIG) awarded Cotton & Company LLP a contract to conduct a performance audit of USCO and Office of the Chief Information Officer�s (OCIO) efforts to develop the new USCO IT environment and business applications, particularly with regard to the current development work related to the USCO Public Records System Development Project (OCIO Project Management Office [PMO] Project #546). The objectives of this project were to: 1. Evaluate USCO and OCIO�s project management and software development practices. 2. Evaluate whether the representation of the project in the Copyright Modernization Office�s (CMO�s) integrated master schedule (IMS) is comprehensive and sufficient. 3. Evaluate USCO and OCIO�s progress in closing the 12 prior findings from the OIG�s August 2019 report entitled, Library Working Through Agile Delivery Method Challenges for Copyright IT Modernization Project (OIG Report No. 2018-IT-107). Summary of Results Based on the results of our audit work, we found that: - As it relates to USCO and OCIO�s project management and software development practices, the USCO Public Records Project was approved, as evidenced by a Project Charter signed by the Project Sponsor, OCIO Management, and other stakeholders. It is monitored by an active governance structure, as evidenced by regular status updates through Copyright Modernization Governance Board meeting minutes; however, we noted the following deficiencies: -- Non-Comprehensive Cost Estimate: The OCIO PMO does not have a comprehensive system development cost estimate for the USCO Public Records Project. Specifically, we noted for this project that: - The OCIO PMO did not complete the Cost Estimation Spreadsheet and Cost Estimate Document, as required in the initiation phase. - The Project Personnel Budget (PPB) spreadsheet contains estimated hours and personnel not supported with a required narrative and does not include costs for all personnel. - The Independent Government Cost Estimate (IGCE) contains the required estimated hours, personnel, labor rates, and inflation rates; however, they lack required supporting narratives. -- Implementation of Unapproved IT Governance Structure: The OCIO implemented a revised governance structure in advance of receiving required approvals as documented in LCR 1-710: Library of Congress Regulations (LCRs) and Directives (LCDs), section 7(A)5: �The Librarian shall give final approval to new or revised regulations.� -- Non-Comprehensive IMS: The CMO�s IMS is updated monthly and tracks CMO�s projects at the ECS level, which consists of the CMO�s registration, recordation, public records, and licensing IT applications; however, it is not comprehensive, as the IMS only includes work breakdown structures (WBS) through the second year of the USCO Public Records Project (September 16, 2020, to September 8, 2021), instead of the full life cycle of the project, as defined by the project charter (September 2019 to September 2024). - Prior Findings and Recommendations: The Library closed seven of the 12 recommendations in the OIG Report No. 2018-IT-107; see APPENDIX A � STATUS OF LOC OIG REPORT NO. 2018-IT-107 RECOMMENDATIONS for details on the statuses of recommendations. The following five recommendations remain open: -- Recommendation 1: Develop and implement guidance on tracking and resolving project health issues on development projects that follow an agile, hybrid, or other similar methodology. This guidance should follow the guidelines included in publications by Project Management Institute (PMI), Office of Management and Budget (OMB), and/or other risk management standard-setting bodies. The guidance should also identify critical characteristics of the earned value analysis (EVA) method for measuring the project budget and progress toward completion in coordination with the Financial Services Directorate (FSD), including establishing the project costing methodologies. Additionally, the OCIO should update its status dashboards to effectively convey project progress. -- Recommendation 2: Ensure all relevant stakeholders understand the development methodology used, implement a stakeholder engagement plan, assess the risks associated with the project before beginning the project, and document best practices for governance and status meetings, including best practices relating to the size and content of the meetings. -- Recommendation 7: The OCIO clearly define the purpose of a minimum viable product (MVP) in the Library�s instance of agile development and develop a process for ensuring that the Library incorporates the agreed-upon definition into its SDLC processes. The OCIO should coordinate with Contract Grants Directive (CGD) for issues related to contract management and with FSD for issues related to cost management. The Library should then codify the process in its policies. -- Recommendation 8: The OCIO�in conjunction with the Office of the General Counsel and CGD�develop guidance to align key activities and responsibilities defined in application development contracts with PMI�s Agile Practice Guide, or develop risk mitigation strategies for instances in which the Library chooses to deviate from agile best practices. These key activities and responsibilities may include items such as maintenance of the product roadmap. -- Recommendation 12: The OCIO develop contingency plans for each of the risks identified in the risk register and obtain appropriate management approval for the plans. BACKGROUND The Library is an agency under the Legislative Branch of the U.S. government, comprising several internal divisions of service units, including the Office of the Librarian, the Congressional Research Service, USCO, the Law Library of Congress, and Library Services. It is the world�s largest and most comprehensive library, maintaining a collection of more than 164 million items�many of them unique and irreplaceable�in more than 470 languages. The Library�s mission is to support Congress in fulfilling its constitutional duties and to further the progress of knowledge and creativity for the benefit of the American people. USCO serves as a separate federal department within the Library, under the general oversight of the Librarian of Congress, pursuant to specific statutory authorities set forth in the U.S. Copyright Act. Under the direction of the Register of Copyrights, USCO administers a complex and dynamic set of laws, including registration, the recordation of title and licenses, a number of statutory licensing provisions, and other aspects of the 1976 Copyright Act and the 1998 Digital Millennium Copyright Act. By statute, the Register of Copyrights is the principal advisor to Congress on national and international copyright matters, testifying upon request and providing ongoing leadership and impartial expertise on copyright law and policy. USCO IT Modernization To better serve its constituencies, USCO has dedicated itself to modernizing its systems and its administration of the nation�s copyright laws. Based on congressional direction, USCO began the modernization process by conducting a detailed analysis and review of its systems, then laid out its vision for its future-state IT enterprise solution in two documents: Strategic Plan 2016�2020: Positioning the United States Copyright Office for the Future (issued December 1, 2015) and Provisional Information Technology Modernization Plan and Cost Analysis (issued February 29, 2016). The Copyright IT Modernization Plan is a multi-year effort to transform USCO�s IT systems into a single, improved ECS. Products to be developed include: - An enhanced recordation system. - An improved registration system. - A public records system. - Systems for USCO office workflow and creative community engagement. - A modern Licensing Division system. Public Records System The USCO has the largest collection of copyright records in the world. Members of the public seek out these records to find copyright owners and to obtain copies of completed and in-process registration records, recordation documents, and registration deposits. Because many of these records pertain to works under copyright protection, it is vital that USCO provide accurate and timely data. Currently, USCO�s online Copyright Public Records Catalog hosts 29 million records of post-1978 registered works and copyright-related documents. The database that maintains these records is integrated with USCO�s registration system and other legacy systems that support the USCO�s mission. The database and associated search engine used for USCO�s current Copyright Public Records Catalog is Voyager ILS, a commercial off-the-shelf product designed specifically for libraries. The database currently holds 700 GB of digital assets, which is growing at a rate of 5.0 GB per month. Because the data included in the Copyright Public Records Catalog is limited by the sources from which it is drawn, the technological limits of the Registration and Recordation systems (e.g., limited fields to accept information related to recorded documents) affect the quality of these public records. In August 2019, the OCIO contracted a third-party vendor to assist in the development of a Copyright Public Records System. This system will allow users to see relationships between copyright records, including chain of title for copyrighted works. Additionally, the system will offer both simple and advanced search capabilities, bulk download, and increased interoperability with other USCO systems. The project charter was approved in October 2019 supporting the build of the Copyright Public Record System. The project charter outlined: (1) a planned project start date of September 16, 2019 and planned project completion of September 27, 2024; (2) the project estimated a total project cost of $7,523,570; and (3) the following project milestones and release plan: Table 1. Milestones for First Year (FY 2020) and Dates. Milestone: Project Initiation complete and development contractors onboarded Date: October 4, 2019 Milestone: Project Planning and Initial Strategy and Communications Date: November 8, 2019 Milestone: Initial Design & Development (Extract Transform Load [ETL], [Footnote 2] Application Programing Interface [API] [Footnote 3] (internal only), User Experience (UX), [Footnote 4] Web app/Search) Date: February 21, 2020 Milestone: Continuous Development (ETL, API [internal only], UX, Web app/Search) Date: May 15, 2020 Milestone: Continuous Development, with goal of a second ETL system (ETL, API [internal only], UX, Web app/Search) Date: August 28, 2020 Milestone: Proof of Concept Release Date: September 14, 2020 Table 2: Proposed Milestones for Subsequent Years and Date Proposed Milestone: Continuous Development (ETL, API [internal and/or external], UX, Web app/Search); interface with Virtual Card Catalog (VCC) images and text Date: 2021 Proposed Milestone: Continuous Development (ETL, API [internal and/or external], UX, Web app/Search); interface with microfilm records Date: 2022 Proposed Milestone: Continuous Development (ETL, API [internal and/or external], UX, Web app/Search); interface with digitized Catalog of Copyright Entries (CCE) Date: 2023 Proposed Milestone: Continuous Development (ETL, API [internal and/or external], UX, 2024 Web app/Search); interface with digitized copyright Record Books Date: 2024 Source: USCO Public Records System Development (PMO ID #546) Project Charter, dated October 11, 2019. AUDIT OBJECTIVE, SCOPE, METHODOLOGY Objective The Library awarded Cotton & Company a contract to conduct a performance audit of the USCO�s and the OCIO�s efforts to develop the new USCO IT environment and business applications, particularly with regard to the current development work related to the USCO Public Records System Development Project (OCIO PMO Project #546). The audit objectives were to: 1. Evaluate the USCO�s and the OCIO�s project management and software development practices using the Government Accountability Office�s (GAO�s) Schedule Assessment Guide, Cost Estimating and Assessment Guide, and Software Development: Effective Practices and Federal Challenges in Applying Agile Methodologies. At a minimum: a. Determine whether USCO and OCIO comprehensively and reasonably: estimated applicable Library-wide costs; established a baseline; appropriately issued a cost accumulation guide to all parties; and reported, monitored, and mitigated any risks related to comparisons of budgeted to actual costs. b. Determine whether USCO and OCIO formed a baseline for the complete schedule, matched the baseline schedule to the baseline costs, and sufficiently monitored this schedule and mitigated any risks. c. Evaluate and conclude on the management of requirements related to development work packages. This includes assessing and concluding on the levels of total work unit (i.e., user stories), additional work identified during the project, and issue resolution work compared to the capacity and capabilities of the development team. d. Evaluate and conclude on the organizational and governance structure of the project. Determine whether the structure includes executive management commitment and support, a clear project charter with appropriately delineated roles and accountability, and an appropriately active steering committee and project sponsor. e. Obtain and review the project plan, ensuring it includes key components (e.g., critical milestones, deliverables); requirements for each process area; clear parameters for scope, budget, resources, and time; appropriate and comprehensive performance measures, comprehensive and budgeted WBS, and a contingency plan. Confirm the proper use of baselining and change management procedures with regard to the project plan. f. Examine and evaluate project-to-date variances (including variances in budgeted to actual and forecast to complete) for adequacy, as well as for evidence of senior executive management review and approval. 2. Evaluate whether the representation of the project in CMO�s IMS is comprehensive and sufficient. 3. Evaluate USCO�s and OCIO�s progress in closing the 12 prior findings from the OIG�s August 2019 report, entitled Library Working Through Agile Delivery Method Challenges for Copyright IT Modernization Project (OIG Report No. 2018-IT-107). We conducted this performance audit in accordance with Generally Accepted Government Auditing Standards, 2018 Revision, as issued by the Comptroller General of the U.S. These standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Scope and Methodology In meeting these three objectives, we assessed internal controls that we considered relevant to nine of the 17 principles associated within the five components of internal control defined in the GAO�s Standards for Internal Control in the Federal Government (Green Book). The following table summarizes the principles we assessed: Control Environment Principle 2: Exercise Oversight Responsibility Principle 3: Establish Structure, Responsibility, and Authority Risk Assessment Principle 6: Define Objectives and Risk Tolerance Principle 7: Identify, Analyze, and Respond to Risks Principle 8: Assess Fraud Risk Control Activities Principle 12: Implement Control Activities Communication Principle 13: Use Quality Information Monitoring Principle 16: Perform Monitoring Activities Principle 17: Evaluate Issues and Remediate Deficiencies We assessed the design, implementation, and operating effectiveness of internal controls and identified deficiencies that we believe could affect the USCO and OCIO�s efforts to develop the USCO Public Records system. Below, we detail the procedures we performed to conduct our audit and assess internal controls relevant to the audit objectives: - Re-performed OCIO�s project cost estimation for the USCO Public Records Project�s using the supporting documentation for the basis of estimates, [Footnote 5] including the cost estimation spreadsheet, cost estimate document, project personnel budget spreadsheet, and IGCE. - Inspected the Public Records System Development Project Charter, [Footnote 6 ] Project Management Plan, [Footnote 7] and Risk Register [Footnote 8] supporting the Public Records baseline schedule. [Footnote 9] - Inquired with OCIO on its process to calculate earned value management [Footnote 10] (EVM) to evaluate OCIO�s management of requirements related to development work packages (i.e., user stories). [Footnote 11] - Inspected the USCO Public Records Project Charter and the Copyright Modernization Governance Board meeting minutes to determine whether the organizational and governance structure includes executive management commitment and support. - Inspected the USCO Public Records Project Management Plan to determine whether it included key components such as Project Classification Worksheet, Project Schedule, Project Status Reports, Risk Register, and Issue Log. - Calculated project variances between budgeted and actual costs for OCIO personnel by using actual hours recorded by Library employees working on the USCO Public Records Project and estimated hours from the cost estimate. - Inquired with USCO personnel about the CMO�s IMS and inspected the IMS to evaluate how the USCO Public Records Project is incorporated into the IMS. - Obtained and inspected documentation provided by OCIO to support the closure of seven out of 12 recommendations from OIG Report No. 2018-IT-107. - Performed inquiry with OCIO personnel to determine the status of in-progress corrective actions for five out of 12 recommendations from OIG Report No. 2018-IT-107. - Inspected the open recommendations in GAO Report, Library of Congress: Strong Leadership Needed to Address Serious Information Technology Management Weaknesses, dated March 2015, and evaluated the impact to our audit objectives. FINDINGS AND RECOMMENDATIONS We identified the following findings during the audit. Finding 1: Non-comprehensive Cost Estimate Background Developing reliable cost estimates is crucial for realistic program planning, budgeting, and management. A cost estimate is the summation of individual cost elements�using established methods and valid data�to estimate the future costs of a program, based on what is known to-date. The management of a cost estimate involves updating the estimate with actual data as it becomes available, revising the estimate to reflect program changes, and analyzing differences between estimated and actual cost (e.g., by using data from a reliable EVM system). Cost estimating is a critical element in any acquisition process and helps decision-makers evaluate resource requirements at milestones and other important decision points. It is the basis for establishing and defending budgets and drives affordability analyses. Cost estimates are integral to determining and communicating a realistic view of likely cost and schedule outcomes that can be used to plan the work necessary to develop, produce, operate, maintain, and dispose of a program. The Library OCIO PMO leverages several key deliverables to document its cost estimation of IT projects: - Cost Estimating Spreadsheet: Determine specific project costs categorized by types of IT expenditures. - Cost Estimating Document: Describes the process and assumptions. - IGCE: Price estimate independently developed based on a comparison and analysis of factors such as published catalogs prices, historical prices paid, market survey information, and vendor price quotes for goods and services routinely available on the open market at competitive prices. - Project Personnel Budget (PPB) Spreadsheet: Spreadsheet documenting estimated hours, labor rates, and costs for OCIO Personnel, Non-OCIO Personnel (i.e., U.S. Copyright personnel), and contracted labor costs. Additionally, the PPB is used to create the Project Baseline: Resources/Cost in the Project Charter. Additionally, in the Further Consolidated Appropriations Act, 2020 (H.R. 1865/Public Law 116�94), Congress requested the CMO develop an IMS that includes a comparison of the applicable planned cost of completed work to actual cost in order to measure the modernization cost and schedule performance on an ongoing basis. Specifically, H.R. 1865/P.L. 116-94 states: In order to measure the modernization cost and schedule performance on an ongoing basis, the integrated master schedule should also include a comparison of the applicable planned cost of completed work to actual cost incurred, to be updated quarterly. The baselined integrated master schedule should be completed and shared with the Committees within 60 days of enactment of this act. In May 2020, the USCO CMO worked with a contractor to develop an IMS that included a detailed narrative with the assumptions and methodology used by the contractor when developing the IMS. As part of the detailed IMS narrative, the contractor performed a cost estimate of the Copyright Modernization Program, including for the USCO Public Records Project. The primary inputs for the cost estimate used by the contractor were the contractor-prepared IMS and the scope, schedule, and resource information available for each individual project comprising the ECS program, including Public Records. We inspected the OCIO�s initial cost estimate calculated during the initiation phase of the USCO Public Records Project (October 2019) and the CMO contractor�s IMS and narrative, Version 10, dated April 30, 2021. We compared the cost estimate prepared by the CMO�s contractor for the USCO Public Records Project in the IMS (totaling $15,388,800) [Footnote 12] to the OCIO�s initial cost estimate in the project charter (totaling $7,523,570) and noted the CMO contractor�s estimated costs were $7,865,230 more than OCIO�s initial estimate. There are significant variances between the OCIO�s initial cost estimate and the cost estimate developed by CMO�s contractor. While we were unable to confirm the accuracy of the cost estimate performed by the contractor within the context of our audit objectives, we did note the cost estimate performed by the contractor used the USCO Public Records Project schedule, which we determined to be incomplete. Additionally, the current IMS narrative identifies the following caveats related to the accuracy of the IMS: - Does not currently reflect all project milestones, including internal and external releases. This will need to be addressed for the Copyright Modernization Program for the IMS to accurately portray the progress and overall Scope of Copyright Modernization. - The IMS is heavily dependent on the accuracy and details of underlying Sub-Projects maintained by USCO and/or OCIO for chartered projects. USCO requests to update information in the IMS may be dependent on an underlying Sub-Projects maintained by OCIO, which require engagement with the OCIO to address those requests. - Is not ready to be baselined due to the breadth and depth of items that need to be addressed as described above. In short, the contractor was unable to adequately estimate a reliable IMS and cost estimate for the USCO Public Records Project because it did not receive essential data (e.g., project velocity, cost baseline) from OCIO�even though the contractor worked on the IMS documentation for about a year and USCO has been working on USCO Public Records Project since October 2019. Without the data from OCIO, both the contractor and USCO determined any cost estimate would have significant limitations. Additionally, OCIO was unable to provide us an estimate on when it would be able to provide this mandatory information to USCO for its future planning efforts. Condition The OCIO PMO does not have a comprehensive cost estimate for the USCO Public Records Project. Specifically, we noted the following: 1. The OCIO PMO did not complete the required Cost Estimation Spreadsheet and Cost Estimate Document as part of the initiation phase of the USCO Public Records Project. Additionally, OCIO PMO provided a Cost Estimate Document that was prepared subsequent to the initiation phase. We noted that a sensitivity analysis and quantified risk assessment were not performed, as required by LOC guidance. OCIO stated that the cost estimating guidance was not updated to require a Cost Estimating Document and Cost Estimation Spreadsheet until October 2019, which is one month after the USCO Public Records Project began. 2. For the PPB spreadsheet that supports the USCO Public Records Project Charter project baseline cost: a. Support for the OCIO�s determination of required personnel and estimated hours of OCIO personnel provided by the OCIO Tower Owners is not documented. b. No labor rates were applied to estimated hours for OCIO-Indirect Personnel and Non-OCIO Personnel (i.e., USCO personnel); therefore, costs for these individuals are not included in the overall USCO Public Records Project cost estimate baseline. 3. During our fieldwork, the OCIO PMO provided an updated PPB that is currently in development to support an as-yet unapproved change request. Although not yet approved, we noted that the OCIO PMO: a. Does not have documented policies and procedures to support the calculation of all labor rates used in the cost estimation for both OCIO and Non-OCIO personnel. b. Did not apply the appropriate default labor rate for Non-OCIO personnel. c. Did not apply an inflation rate to the base year of contracted labor costs and applied a rate of 14 percent for another fiscal year (FY) without justification in the form of a short narrative. 4. For the IGCE, we noted: a. The OCIO did not document support for its determination of the required personnel, forecasted number of stories, and hours per sprint. b. The Library used a �blended hourly rate� of $125.00 for the Full Stack Developer role on the IGCE and stated in response to our written questions�dated January 26, 2021�that the $125.00 rate represents �a best guess at the logical average�; however, support for this determination was not documented. c. Inflation was considered on the initial PPB instead of the IGCE. We noted that the OCIO PMO applied an inflation rate of 4.09 percent for the FY 2022 estimated amount; however, it did not provide support for that inflation rate in a short narrative, as directed in the Library�s IGCE guidance. Criteria The GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Program Costs, �Chapter 3: The Characteristics of Credible Cost Estimates and a Reliable Process for Creating Them,� states a comprehensive cost estimate: - includes all life cycle costs - is based on a technical baseline description that completely defines the program, reflects the current schedule, and is technically reasonable - is based on a WBS that is product-oriented, traceable to the statement of work, and at an appropriate level of detail to ensure that cost elements are neither omitted nor double-counted - documents all cost-influencing ground rules and assumptions. LCD 5-310.1, Project Management Life Cycle (PMLC), effective August 2017, requires the Cost Estimating Spreadsheet and Cost Estimate Document be prepared during the initiation phase of the project. OCIO PMO Cost Estimating Guidance, last updated August 30, 2020, states: Cost Estimation Lead conducts Sensitivity Analysis and Quantified Risk Assessment. Sensitivity Analysis assesses the potential effect on costs based on changes to identified cost drivers and assumptions. Quantified Risk Assessment utilizes the cost risks identified as part of overall Risk Management to determine a range of estimated costs based on the risks. (See the GAO Cost Estimating and Assessment Guide, chapters 13 and 14, for detailed guidance on how to perform these analyses.) NOTE: These analyses are required for Large projects with High Risk and High Complexity scores (see Cost Estimation Spreadsheet). For such projects, conducting Sensitivity analysis and quantified risk assessments ensures that contingencies are put in place to identify a new cost estimate quickly in case a cost assumption is invalidated, a cost driver changes, or a cost risk is realized. The OCIO PMO ICGE Guidance states: A. The impact of inflation should be considered when developing your IGCE. The following information should provide basic information on how to use escalation techniques in your IGCE for base (or first) year as well as option years. 1. Develop the base year estimated costs as described above, inflating as necessary to provide a realistic price/cost. 2. To forecast the option year(s) cost, appropriate escalation factors are applied to the previous cost elements to bring them up to a realistic value. If the contract performance involves more than one year, different escalation factors may be applied dependent on the labor/material mix as appropriate. The following methodologies may be used as appropriate: (a) The Department of Labor Consumer Price Index (CPI) provides data and percentage of change in inflation/escalation factors. CPI information is available at http://www.bls.gov/cpi/. (b) Market trends should be considered when projecting option year(s) escalation rates and should be justified in a short narrative. A market survey will provide information on current market prices and potential volatility of prices in the market place. In addition, when computing a projected escalation rate a review of previous Treasury interest rates will provide an overall view of market prices. GAO Standards for Internal Control in the Federal Government (2014), Principle 13, states that management should use quality information from reliable sources that is appropriate, current, complete, accurate, accessible, and provided on a timely basis. Additionally, Principle 14 states quality information about the entity�s operational processes that flows up the reporting lines from personnel to help management achieve the entity�s objectives. Cause The OCIO PMO has not developed a well-documented cost estimating process. This is consistent with a currently open finding in GAO�s March 2015 report, Library of Congress: Strong Leadership Needed to Address Serious Information Technology Management Weaknesses, which states the Library has not fully established and implemented key IT acquisition practices, including cost estimating. GAO identified the underlying cause as the Library not having established an organization-wide policy for cost estimating. As such, the Library has not yet provided the information needed for USCO (or its contractor) to develop a reliable cost estimate for the USCO Public Records Project. The GAO recognizes the challenges of cost estimates and states: Even in the best of circumstances, cost estimating can be difficult. The cost estimator typically faces many challenges. These challenges often lead to unreliable estimates�for example, estimates that contain poorly defined assumptions, have no supporting documentation, are accompanied by no comparisons to similar programs, are characterized by inadequate data collection and inappropriate estimating methodologies, are sustained by irrelevant or out-of-date data, provide no basis or rationale for the estimate, or adhere to no defined process for generating the estimate. Effect Without a comprehensive, baselined cost estimate and documentation supporting the calculation of estimated costs, the Library is: (1) not able to effectively monitor the project using EVM; and (2) at risk of experiencing cost overruns, missed deadlines, and performance shortfalls for the USCO Public Records Project. Additionally, management will have difficulty successfully allocating resources, making informed decisions, and comparing the value of work accomplished in a given period with the actual cost of the work accomplished and the value of the work planned in that period. These risks are already being realized as evidenced by the significant gap in the initial OCIO cost estimate of $7,523,570 to the current CMO contractor cost estimate of $15,388,800. Recommendation In its audit report, Library of Congress: Strong Leadership Needed to Address Serious Information Technology Management Weaknesses, dated March 2015, GAO recommended: To effectively plan and manage its acquisitions of IT systems and increase the likelihood of delivering promised system capabilities on time and within budget, the Librarian should establish and implement an organization-wide policy for developing cost estimates that includes key practices as discussed in this report. This recommendation remains open. Additionally, we recommend: 1.1 The Library implement a quality review of the Cost Estimation Spreadsheet, Cost Estimate Document, the Project Personnel Budget Spreadsheet, and the IGCE, in order to ensure these documents are completed timely, accurately, and in accordance with applicable guidance. 1.2 The Library clearly document the determination of the required personnel and estimated labor hours for OCIO personnel. 1.3 The Library apply supported labor rates to the non-OCIO personnel in the Public Records cost estimation. 1.4 The Library document procedures to support the calculation of all labor rates used in the PPB cost estimation, for both OCIO and Non-OCIO personnel. 1.5 The Library apply an appropriate inflation rate to base year of contracted cost to the IGCE, as documented in guidance by the Library�s Contracting Office. 1.6 The Library document its support for the roles, hours per sprint, and labor rates applied to project cost estimates and contracting documents. 1.7 The Library ensures Public Records cost forecasts and supporting elements (e.g., epics, features, velocity)�along with dependencies�are accurately represented in the project documentation. It is recommended that this be completed with a sense of urgency, in order to allow costs to be baselined. Finding 2: Incomplete Public Records Project Schedule Background Project scheduling provides a detailed plan that represents how and when the project will deliver the required products, services, and results defined in the project scope and requirements documents. The schedule serves as a tool for communication, managing stakeholder�s expectations, and basis for performance reporting (i.e., baseline). A baseline schedule is the basis for managing the program scope, the time period for accomplishing it, and the required resources. The baseline schedule is designated the target schedule, and it is subject to a configuration management control process. Program performance is measured, monitored, and reported against the baseline schedule. The schedule should be continually monitored so as to reveal when forecasted completion dates differ from baseline dates and whether schedule variances affect downstream work. The OCIO PMO project managers are responsible for creating and maintaining project schedules. The project schedules are developed using OCIO Scheduling Guidance and are maintained in an OCIO-approved project management system. In May 2020, the USCO�s CMO engaged a contractor to�among other activities�develop an IMS for the Copyright IT Modernization effort. The IMS is defined as a schedule-based, detail-level plan that contains all the tasks necessary to support the key program events and accomplishments. It is accompanied by a detailed narrative describing the assumptions and methodology used by the contractor to develop the IMS. Additionally, it is heavily dependent on the accuracy and details of underlying Sub-Projects maintained by USCO and/or OCIO for chartered projects. The CMO and the contractor receive updates to the individual project schedules from OCIO project managers and use the updated schedules to update the IMS on a monthly basis. Condition Despite USCO having begun its USCO Public Records Project in October 2019, the most recent project schedule (dated November 3, 2020) is not comprehensive or sufficient. Specifically, the project schedule only contains the completed and scheduled sprints for the first 2 years of the project; however, it does not include key release milestones and detailed WBS (i.e., features), as required by the OCIO PMO Scheduling Guidance. Additionally, the project did not contain any information for years three, four, and five of the project. Criteria The OCIO PMO Scheduling Guidance�last updated August 30, 2020�states a schedule should: - Identify activities in the schedule based on all activities (government and contractor) of the WBS and/or release roadmap as well as near-term sprint objectives for Agile projects. - Identify and distinguish project/release milestones in the schedule. - At any given time, the schedule should be the list of instructions on how the program intends to execute. - Ensure that every activity within the schedule should be traceable to an appropriate WBS element, and every WBS element must have at least one associated activity that is necessary to complete that element. - As with updating the WBS, update the schedule progressively throughout the project as more detailed information is available for each activity. - Identify and distinguish project/release milestones in the schedule. GAO�s Schedule Assessment Guide: Best Practices for Project Schedules, Appendix III, Scheduling and Earned Value Management, states: It is important to establish and maintain a valid schedule baseline to ensure that earned value management (EVM) data being reported are reliable. Therefore, the entire schedule must be baselined because the IMS is the source of time-phasing for all control accounts and work packages that make up the project�s performance measurement baseline. GAO�s Agile Assessment Guide: Best Practices for Agile Adoption and Implementation, Chapter 7: Agile Program Monitoring and Control, states: Feature: A feature is a specific amount of work that can be developed within one or two reporting periods. It can be further segmented into a user story or stories. The functionality is described with enough detail that it can remain stable throughout its development and integration into working software. It is this level that should be tracked through program management products like the life cycle cost estimate and schedule. The features in the WBS should be fully traceable to the program�s road map. An integrated master schedule or similar artifact that includes Agile software development efforts should capture all the planned features needed to accomplish the program goals at an appropriate level of detail using rolling wave planning. This schedule should include all government and contractor activities. Developing an integrated master schedule for the whole program provides a comprehensive, end-to-end view of all the features necessary to accomplish the program�s goals. Including features enhances the utility of the schedule as a coordination and communication tool and allows for better performance tracking and measurement. For example, additional information in the schedule helps to ensure that it can serve as the summary, intermediate, and detailed schedule. Including high-level features in the schedule is also a foundational best practice for most other scheduling best practices, because if the schedule does not contain planning for all the features for the duration of the program, it will lack horizontal and vertical traceability, a valid critical path will not exist, and the schedule�s risk analysis will not be valid. Cause The Library has not developed and implemented a well-documented project scheduling process. This is consistent with GAO�s open finding in the March 2015 report, Library of Congress: Strong Leadership Needed to Address Serious Information Technology Management Weaknesses, which states the Library has not fully established and implemented key IT acquisition practices, including scheduling. GAO identified the underlying cause as the Library not having established an organization-wide policy for scheduling. USCO relies on OCIO for essential data needed to adequately estimate an IT project�s IMS; however, because the OCIO has not yet remedied the weaknesses identified in GAO�s 2015 audit report related to project scheduling, the OCIO has not been able to provide this information to USCO. Additionally, OCIO informed us that, as part of the path forward to address outstanding project management-related findings, a PMO Execution Roadmap is expected to be published by the end of FY 2021. Effect Without a comprehensive and sufficient schedule, the USCO Public Records Project is not accurately represented in the Copyright IT Modernization IMS. Both the Library and USCO management rely on the IMS to present a detail-level plan for tasks necessary to support the Copyright IT Modernization Program. Without comprehensive and sufficient project schedules, Library management is unable to properly baseline the IMS or determine the critical path within the IMS. The narrative accompanying the contractor-developed USCO Modernization IMS, Version 10 (dated April 30, 2021), identified the following deficiencies specific to the USCO Public Records Project: - The IMS does not currently reflect all internal dependencies (e.g., project charters, funding approval) and external dependencies (e.g., procurements, other agency involvement). Here are some examples: -- Public Records System Development - The IMS does not currently reflect all dependencies that would be expected between individual products (e.g., Registration, Recordation, Licensing, Public Records) and across shared services (e.g., Platform Services, UX, Financial Management). The contractor�s IMS narrative also identified several deficiencies related to the Copyright Modernization projects, including the USCO Public Records Project. Specifically, it states that the IMS: - Does not currently reflect all release management dependencies, including post-development approvals, which impact the actual release of software to staff/public users. - Does not currently reflect legacy process/system dependencies, including impacts of individual products feeding and/or referencing common legacy systems (e.g., Voyager, CIS). This will need to be addressed comprehensively for the Copyright Modernization Program with clear strategy which considers business impacts, technical considerations, and other dependencies for �turning on�/ �turning off� processes/systems. - Does not currently reflect all project milestones, including internal and external releases. This will need to be addressed for the Copyright Modernization Program for the IMS to accurately portray the progress and overall Scope of Copyright Modernization. - Is not ready to be baselined due to the breadth and depth of items that need to be addressed as described above. Additionally, without detailed WBS that include information at the feature level, Library management will not be able to monitor the project using EVM to determine whether actual completion dates differ from the planned dates. Recommendation In its audit report, Library of Congress: Strong Leadership Needed to Address Serious Information Technology Management Weaknesses (dated March 2015), GAO recommended: To effectively plan and manage its acquisitions of IT systems and increase the likelihood of delivering promised system capabilities on time and within budget, the Librarian should establish a time frame for finalizing and implementing an organization-wide policy for developing and maintaining project schedules that includes key practices as discussed in this report, and finalize and implement the policy within the established time frame. This recommendation remains open. Additionally, we recommend: 2.1 The Library update the USCO Public Records Project schedule to include key release milestones and adequate WBS level details (i.e., features) for all remaining years of the project, including the current year. Finding 3: Unapproved IT Governance Policies and Procedures Background The National Institute of Standards and Technology (NIST) defines governance as: [T]he set of responsibilities and practices exercised by those responsible for an organization (e.g., the board of directors and executive management in a corporation, the head of a federal agency) with the express goal of: 1. providing strategic direction; 2. ensuring that organizational mission and business objectives are achieved; 3. ascertaining that risks are managed appropriately; and (iv) verifying that the organization�s resources are used responsibly. Risks and resources can be associated with different organizational sectors (e.g., legal, finance, information technology, regulatory compliance, information security). LCR 5-130, Information Technology Investment Management (ITIM) provides the framework for IT governance at the Library, identifies roles and responsibilities, implements the Library�s ITIM policy, and centralizes ITIM activities. Supporting this regulation, LCD 5-130.1, Information Technology Investment Management, further defines roles and responsibilities, as well as provides more detailed descriptions of each step in the IT governance process. Condition The approved and published versions of LCR 5-130, Information Technology Investment Management (ITIM), and LCD 5-130.1, Information Technology Investment Management, do not accurately reflect the current IT governance structure in operation at the Library. Changes between the governance structure documented in LCR 5-130/LCD 5-130.1 and the current governance structure we observed in practice included the removal and addition of the following governance boards or working groups: Table 3: Approved and Published LCR 5-130 and LCD 5-130.1 - IT Steering Committee - Architecture Review Board Current IT Governance Structure In Place - Technology Strategy Board - IT Finance Working Group - IT Product Governance Board - Technical Architecture Board - Digital Strategy Working Group Criteria LCR 1-710, Library of Congress Regulations (LCRs) and Directives (LCDs), Section 7. Preparation of Regulations, Subsection A. Drafting, Review, and Approval, states: 1. A proposed new or revised policy and procedural statement shall be drafted by the appropriate responsible Library manager and forwarded through administrative channels, accompanied by the transmittal and routing form, to the General Counsel. The form shall state the purpose for the new regulation or summarize the changes in the revised regulation� 2. When revised regulations are forwarded to the General Counsel for review, they must be accompanied by a red-lined version showing the suggested changes as well as a clean version of the regulation. 3. Before submission to the Executive Committee, the General Counsel may submit a new or revised regulation to any management committee, team, or other Library managers for review and comment. The General Counsel will specify the time frame by which comments must be received for consideration. 4. The General Counsel shall forward the draft regulation, with any accompanying recommendations, to the EC� The reviewing officials shall initial and date any comments or editorial changes that they may make on the draft regulation and respond to the General Counsel who shall prepare the draft regulation for final review and approval. 5. The Librarian shall give final approval to new or revised regulations. GAO�s Standards for Internal Control in the Federal Government (Green Book) (September 2014 Revision), Sections 3.09, 3.10, and 3.11, Documentation of the Internal Control System, states: 3.09 Management develops and maintains documentation of its internal control system. 3.10 Effective documentation assists in management�s design of internal control by establishing and communicating the who, what, when, where, and why of internal control execution to personnel. Documentation also provides a means to retain organizational knowledge and mitigate the risk of having that knowledge limited to a few personnel, as well as a means to communicate that knowledge as needed to external parties, such as external auditors. 3.11 Management documents internal control to meet operational needs. Documentation of controls, including changes to controls, is evidence controls are identified, capable of being communicated to those responsible for their performance, and capable of being monitored and evaluated by the entity. Cause In January 2019, the OCIO implemented the revised IT governance structure in advance of receiving required Library approvals, as documented in LCR 1-710, Library of Congress Regulations (LCRs) and Directives (LCDs). As of January 2021, LCR 5-310 and LCD 5-310.1 have not been updated to reflect the current governance structure. Effect Investment governance structures without approved and published regulations/directives may obscure the scope, authority, policy, responsibilities, and definitions of the activities to the organizations participating in or effected by the objectives. Procedures for carrying out Library policy, or to implement a Library policy, may not be commonly understood or followed by those responsible for the activities. Recommendation We recommend: 3.1 The OCIO work with the Library�s Office of the General Counsel to obtain required approvals and to publish revised versions of LCR 5-130 and LCD-5-130.1 that reflect the current IT governance structure at the Library. APPENDIX A � STATUS OF LOC OIG REPORT NO. 2018-IT-107 RECOMMENDATIONS We evaluated and concluded on the U.S. Copyright Office�s (USCO�s) and the Office of the Chief Information Officer�s (OCIO�s) progress in closing the 12 prior findings from the Office of Inspector General�s (OIG�s) August 2019 report, entitled Library Working Through Agile Delivery Method Challenges for Copyright IT Modernization Project (OIG Report No. 2018-IT-107). The following table summarizes our determinations regarding the status of the recommendations. Table 4: Number: 1 Recommendation: Develop and implement guidance on tracking and resolving project health issues, on development projects that follow an agile, hybrid, or other similar methodology. This guidance should follow the guidelines included in publications by the Project Management Institute (PMI), Office of Management and Budget (OMB), and/or other risk management standard-setting bodies. The guidance should also identify critical characteristics of the EVA method for measuring the project budget and progress toward completion in coordination with the Financial Services Directorate (FSD), including establishing the project costing methodologies. Additionally, the OCIO should update its status dashboards to effectively convey project progress. Status: Open Summary of Status: The Library of Congress (Library) published an update to Library of Congress Directive (LCD) 5-310.2, Systems Development Life Cycle (SDLC), in February 2020, to include guidance for projects following hybrid methodology. Additionally, the OCIO is working with a third-party contractor to implement earned value management (EVM)/earned value analysis (EVA) and planning to run a pilot of a WebTA Labor module to incorporate actuals hours of OCIO personnel to monitor project health. The estimated completion date is fourth quarter of fiscal year (FY) 2021. Number: 2 Recommendation: Ensure all relevant stakeholders understand the development methodology used, implement a stakeholder engagement plan, assess the risks associated with the project before beginning the project, and document best practices for governance and status meetings, including best practices relating to the size and content of the meetings. Status: Open Summary of Status: Management provided documentation to support a �Request to Close� for this recommendation in January 2020. Based on the supporting documentation, management provided updated templates for a risk management plan and a risk register; however, detailed implementation guidance for performing risk assessments�including Monte Carlo Quantitative risk assessments�was not provided as an attachment in OCIO�s request for closure dated January 4, 2020. Additionally, guidance describing best practices for governance board and status meetings was not provided. In OCIO�s response to the request for closure (dated January 2020), it states it is currently working on documenting guidance for governance boards. Number: 3 Recommendation: Work with and obtain concurrence from project owners and, where applicable, development governance boards regarding the format and content of program and project reports. Ensure such content follows the guidelines published by PMI or other risk management standard-setting bodies. Status: Closed Summary of Status: Management provided example project and program status reports, and explained via inquiry that concurrence on report content and format from project owners and governance boards is obtained during project and program level status meetings. While the Project Management Body of Knowledge (PMBOK) does not provide specific criteria for what should be included in project status reports, the project and program reports include specific elements of the PMBOK�s Monitor Process Group, including: Monitor and Control Project Work, Control Scope, Control Schedule, Monitor Risks, Control Cost, and Monitor Stakeholder Engagement. Number: 4 Recommendation: OCIO develop and implement guidance (i.e., directives) for blending all SDLC approaches based on the characteristics of individual projects. Such directives should include obtaining the business owners� concurrence regarding the SDLC approach at the start of each project. The OCIO should also obtain the FSD�s concurrence regarding the SDLC approach as it relates to cost management. Status: Closed Summary of Status: OCIO updated the project charter template to include the new hybrid SDLC approach as an option and ensured the product and business owners sign off on all the project charters. OCIO added a line for FSD to sign on the project charter and invited FSD representatives to participate in the project charter development process. Number: 5 Recommendation: OCIO prepare a checklist of all required elements for starting a project before beginning system development for the project, including such items as obtaining approval of the SDLC methodology and completing an agile suitability scorecard. The Chief Information Officer (CIO) or Deputy CIO must approve the completed checklist before beginning the project. Status: Closed Summary of Status: Management provided an updated project charter template that includes a requirement to perform a project charter checklist. The project charter checklist includes requirements for identifying the SDLC methodology and for verifying the existence of an agile suitability scorecard, if applicable. Additionally, the project charter is signed by the Project Management Officer, who is responsible for ensuring the project charter satisfies the requirements identified in the project charter checklist. Additionally, the CIO�s and/or Deputy CIO�s approval of the project overall�including the content of the project charter�is obtained through signature on the project charter. Number: 6 Recommendation: OCIO map the current methodology used in developing the RMI minimum viable product (MVP) to the methodology required in LCD 5-310.2. Ensure that it identifies and implements appropriate risk mitigation steps for any substantive deviation from the required methodology. Obtain the FSD and Register of Copyright�s concurrence with regard to the mapping and any mitigation steps. Status: Closed Summary of Status: Management updated the LCD 5-310.2, Systems Development Lifecycle (SDLC), which establishes Library policy for executing systems development projects, to formally identify a hybrid methodology as one of the available SDLC methodologies for Library systems development projects. As stated, in LCR 1-170, Library of Congress Regulations (LCRs) and Directives (LCDs), published LCDs are approved by the Library General Counsel, the Library Executive Committee, and the Librarian of Congress. Number: 7 Recommendation: OCIO clearly define the purpose of an MVP in the Library�s instance of agile development and develop a process for ensuring the Library incorporates the agreed-upon definition into its SDLC processes. The OCIO should coordinate with CGD for issues related to contract management and with FSD for issues related to cost management. The Library should then codify the process in its policies. Status: Open Summary of Status: OCIO has updated the �Agile at the Library� portal page, which covers MVP in more detail. The Agile Contracts Working Group (ACWG) continues to work on the draft Agile Contracting Guide for the development of agile contracting practices. This guidance includes templates and examples to support the creation of standard contract language on the Library�s Agile information technology (IT) efforts. The estimated completion date is the fourth quarter of FY 2021. Number: 8 Recommendation: OCIO, in conjunction with the Office of the General Counsel and CGD, develop guidance to align key activities and responsibilities defined in application development contracts with PMI�s Agile Practice Guide, or develop risk mitigation strategies for instances in which the Library chooses to deviate from agile best practices. These key activities and responsibilities may include items such as maintenance of the product roadmap. Status: Open Summary of Status: OCIO has updated the �Agile at the Library� portal page, which covers MVP in more detail. The ACWG continues to work on the draft Agile Contracting Guide for the development of agile contracting practices. The guidance includes templates and examples to support the creation of standard contract language on the Library�s Agile IT efforts. The estimated completion date is the fourth quarter of FY 2021. Number: 9 Recommendation: Update the �Agile at the Library� Confluence site and any other relevant guidance to be consistent with The Scrum Guide and PMI guidance. Status: Closed Summary of Status: The condition for this recommendation states that Library�s policies did not implement industry best practices for executing Sprint Review Scrum Ceremonies (OIG Report No. 2018-IT-107). The Scrum Ceremonies section of the �Agile at the Library� guidance, which includes guidance for conducting Sprint Reviews, was updated to reflect industry best practices for conducting Scrum Ceremonies, as identified in The Scrum Guide. Number: 10 Recommendation: Develop a checklist and supervisory controls to ensure it uses updated guidance from the �Agile at the Library� Confluence site and the appropriate implementation methodologies on development projects. Status: Closed Summary of Status: During the initiation phase of a project, a project charter checklist is used to ensure that the appropriate SDLC methodology is selected. The project charter is signed by the Project Management Officer, who is responsible for ensuring the project charter satisfies the requirements identified in the project charter checklist. Additionally, the CIO�s and/or Deputy CIO�s approval of the project overall, including the content of the project charter, is obtained through signature on the project charter. Additionally, management stated via inquiry that adherence with Library Agile policies for individual projects is monitored through monthly Interim Progress Reviews (IPR) that include senior OCIO involvement. Number: 11 Recommendation: OCIO develop quality control mechanisms for ensuring it updates templates for project management deliverables (e.g., risk register) in a timely manner during ongoing projects, as well as for ensuring project teams keep project management deliverables current throughout the project. Status: Closed Summary of Status: OCIO holds regular Project Management Collaborative meetings that bring together project management professionals from across the Library. The Project Management Collaborative meetings provide a forum for announcing the Library�s project management-related news, including updates to the Library�s project management deliverable templates. Number: 12 Recommendation: OCIO develop contingency plans for each of the risks identified in the risk register and obtain appropriate management approval for the plans. Status: Open Summary of Status: While we inspected the Recordation risk register and noted that each risk has a corresponding contingency plan, OCIO has not implemented a corrective action to ensure contingency plans are developed for risks on project risk registers, as evidenced by the AVCMS risk register having risks without contingency plans. APPENDIX B � MANAGEMENT RESPONSE Cotton & Company provided Library management with our draft version of this report, and they provided the following responses. We have not audited management�s responses and therefore do not express an opinion on them. Library Library of Congress Office of the Librarian Memorandum Date October 29, 2021 To Kim Byrd, Acting Inspector General From J. Mark Sweeney, Principle Deputy Librarian of Congress Subject Management Response to OIG report 2020-IT-101, U.S. Copyright Office�s Public Records Performance Audit Thank you for providing the draft report on the U.S. Copyright Office's (USCO) public records system audit. The Library generally agrees with the recommendations and continues to improve its processes for cost estimating and scheduling. The Office of the Chief Information Officer (OCIO) has been working to update deficiencies noted in the initial phase of the public records project costing information and will coordinate with USCO and the Financial Services Directorate (FSD) to document and validate a comprehensive cost estimate. We again note that the FSD, in conjunction with OCIO, is working to develop a new indirect cost methodology. Currently, the Project Management Office (PMO) captures cost and schedule estimates in project charters and tracks certain actual project cost data via a spreadsheet that informs overall project management. With respect to labor, the PMO continues to mature processes for data collection, including completion of the pilot for the Web TA labor management module. The Library's October launch of the Web TA labor management module allows employees to more easily track and report on project hours. And in conjunction with the FSD methodologies that will be developed, Library PMs will be able to ensure more fulsome costing and tracking of Library IT modernization initiatives. In future, the cost estimation spreadsheet, cost estimate document, and project and portfolio management tool will directly feed the estimates used in the project charters. Therefore, OCIO and service unit approval will memorialize agreement of the estimated costs and resources for a given project, while FSD's approval will serve as validation of underlying budget estimates. In addition, OCIO will continue to refine the public records project schedule through updates to the integrated master plan that was previously provided in response to Congressional direction. The attached spreadsheet provides responses and target dates for addressing each of the recommendations as related to the USCO public records project. cc: Judith Conklin, Chief Information Officer Edward Jablonski, Chief Operating Officer Mary Klutts, Chief Financial Officer Shira Perlmutter, Register of Copyright Elizabeth Pugh, General Counsel John Rutledge, Deputy CIO Elizabeth Scheffler, Acting Comptroller Table 5. Management Comments on Draft OIG Report Number 2020-IT-101 U.S. Copyright Office�s Public Records Performance Audit Report Recommendation 1.1: The Library implement a quality review of the Cost Estimation Spreadsheet, Cost Estimate Document, the Project Personnel Budget Spreadsheet, and the IGCE, in order to ensure these documents are completed timely, accurately, and in accordance with applicable guidance. Responding Office: OCIO and FSD Comments: The Library implement a quality review of the Cost Estimation Spreadsheet, Cost Estimate Document, the ServiceNow Project & Portfolio Management Tool (as the Project Personnel Budget Spreadsheet is no longer being used to generate the cost summary), and the IGCE for the USCO public records project by having FSD, as third party validator, review cost estimate information as updated in response to recommendations 1.2 through 1. 7. Target completion: Q4 FY2022 Recommendation 1.2: The Library clearly document the determination of the required personnel and estimated labor hours for OCIO personnel. Responding Office: OCIO and FSD Comments: The Library will document the determination of the required personnel and estimated labor hours for OCJO personnel on the USCO public records project. Target completion: Q1 FY2022 Recommendation 1.3: The Library apply supported labor rates to the non-OCIO personnel in the Public Records cost estimation. Responding Office: OCIO and FSD Comments: The Library apply supported labor rates to the non-OCIO personnel in the USCO public records cost estimation. Target completion: Q4 FY2022 Recommendation 1.4: The Library document procedures to support the calculation of all labor rates used in the PPB cost estimation, for both OCIO and Non-OCIO personnel. Responding Office: OCIO and FSD Comments: The Library will document procedures to support the calculation of all labor rates used in the ServiceNow Project & Portfolio Management Tool (as the Project Personnel Budget Spreadsheet is no longer being used to generate the cost summary) for both OCIO and Non-OCIO personnel on the USCO public records project. Target completion: Q4 FY2022 Recommendation 1.5: The Library apply an appropriate inflation rate to base year of contracted cost to the IGCE, as documented in guidance by the Library�s Contracting Office. Responding Office: OCIO and FSD Comments: The Library will reconcile cost documentation in the IGCE for USCO public records to include an inflation rate adjustment, if any, as provided in the Contracting Office's IGCE guidance. Target completion: Q4 FY2022 Recommendation 1.6: The Library document its support for the roles, hours per sprint, and labor rates applied to project cost estimates and contracting documents. Responding Office: OCIO and FSD Comments: The Library will document its support for the roles, hours per sprint, and labor rates applied to USCO public records project cost estimates and contracting documents. Target completion: Q4 FY2022 Recommendation 1.7: The Library ensures Public Records cost forecasts and supporting elements (e.g., epics, features, velocity)�along with dependencies�are accurately represented in the project documentation. It is recommended that this be completed with a sense of urgency, in order to allow costs to be baselined. Responding Office: OCIO Comments: The Library will ensure USCO public records cost forecasts and supporting elements (epics, features, velocity), along with dependencies, are accurately represented in the project documentation. Target completion: Q2 FY2022 Recommendation 2.1: The Library update the USCO Public Records Project schedule to include key release milestones and adequate WBS level details (i.e., features) for all remaining years of the project, including the current year. Responding Office: OCIO and USCO Comments: The Library will update the USCO Public Records Project schedule to include key release milestones and feature details for the duration of the project. Target completion: Q2 FY2022 Recommendation 3.1: The OCIO work with the Library�s Office of the General Counsel to obtain required approvals and to publish revised versions of LCR 5-130 and LCD-5-130.1 that reflect the current IT governance structure at the Library. Responding Office: OCIO and OGC Comments: OGC circulated revised LCR 5-130 and LCD 5-130.1 to the LCR Working Group to update the IT governance structure. After OCIO and OGC coordination on comment resolution, the EC will review and Librarian will issue updated policies. Target completion: Q1 FY2022 OIG Footnotes: 1. Effective June, 2021, the Copyright Modernization Office is now the Product Management Division. Cotton Report Footnotes: 1. In 2019, Cotton & Company�under contract with the Library OIG�conducted a performance audit of both the OCIO�s and the USCO�s compliance with the Library�s systems development life cycle (SDLC) and project management life cycle (PMLC) policies and directives, as well as to determine whether the USCO�s Recordation IT System Modernization Project was on schedule and within budget (OIG Report No. 2018-IT-107). As part of our current audit, we evaluated the OCIO�s and the USCO�s progress in closing the 12 recommendations from this report. See APPENDIX A � STATUS OF LOC OIG REPORT NO. 2018-IT-107 RECOMMENDATIONS for our results. 2. ETL is a process that extracts, transforms, and loads data from multiple sources of data to a data warehouse or other unified data repository. 3. An API is a set of defined rules written in computer code that explain how computers or applications communicate with one another. APIs sit between an application and the web server, acting as an intermediary layer that processes data transfer between systems. 4. UX design is the process design teams use to create products that provide meaningful and relevant experiences to users. This involves the design of the entire process of acquiring and integrating the product, including aspects of branding, design, usability, and function. 5. The PMI Guide to The Project Management Body of Knowledge (PMBOK) Guide, Sixth Edition, defines �Basis of Estimates� as supporting documentation outlining the details used in establishing project estimates such as assumptions, constraints, level of detail, ranges, and confidence levels. 6. PMI defines �Project Charter� as a document issued by project initiator or sponsor that formally authorizes the existence of a project and provides the project manager with the authority to apply organizational resources to project activities. 7. PMI defines �Project Management Plan� as the document that describes how the project will be executed, monitored and controlled, and closed. 8. PMI defines �Risk Register� as a repository in which outputs of risk management processes are recorded. 9. PMI defines �Baseline� as the approved version of a work product that can be changed only through formal change control procedures and is used as a basis for comparison to actual results. 10. PMI defines �Earned Value Management� as a methodology that combines scope, schedule, and resource measurements to assess project performance and progress. 11. PMI defines �User Story� as a short, textual description of required functionality, often developed during a requirements workshop. 12 The estimated cost was determined by calculating the average of the Upper Range Cost (+30%) and the Lower Range Cost (-30%) in the USCO IMS, Version 10.