Review Quality Controls. Outline that will be expanded upon and further articulated as EDBP develops.
- Multiple expert review of key seed-set training documents using both subject matter experts (attorneys) and AI experts (technologists).
- Direct supervision and feedback by the responsible lawyer(s) (merits counsel) signing under 26(g).
- Extensive quality control methods, including training and more training, sampling, positive feedback loops, clever batching, and sometimes, quick reassignment or firing of reviewers who are not working well on the project.
- Experienced, well motivated human reviewers who know and like the AI agents (software tools) they work with.
- New tools and psychological techniques (e.g. game theory, story telling) to facilitate prolonged concentration (beyond just coffee, $, and fear) to keep attorney reviewers engaged and motivated to perform the complex legal judgment tasks required to correctly review thousands of usually boring documents for days on end (voyeurism will only take you so far). See Robots With A Story To Tell and Robot Games: the Gamification of Legal Review.
- Highly skilled project managers who know and understand their team, both human and computer, and the new tools and techniques under development to help coach the team.
- Strategic cooperation between opposing counsel with adequate disclosures to build trust and mutually acceptable relevancy standards. See Cooperation page of EDBP.
- Final, last-chance review of a production set before going out the door by spot checking, judgmental sampling (i.e. search for those attorney domains one more time), and random sampling.
See generally: The Sedona Conference® Commentary on Achieving Quality in the E-Discovery Process, May 2009.
For more on sampling and its use in both quality control and quality assurance, see the KPMG publications: Gabriel, Manfred, Quality Control For Predictive Coding In e-Discovery (2013); and, The Case For Statistical Sampling In e-Discovery (2012).
Also see: The U.S. District Court for the Northern District of California Guidelines for the Discovery of Electronically Stored Information at 2.02 on Rule 26(f) conferences. The guideline urges parties to discuss methoids to reduce costs and improve efficiencies, including “sampling methods to validate the search for relevant information.” The Northern California ESI checklist for use during the Rule 26(f) meet and confer process also includes a suggestion for attorneys to discuss:
The quality control method(s) the producing party will use to evaluate whether a production is missing relevant ESI or contains substantial amounts of irrelevant ESI.
Practicing attorneys and paralegals are invited to submit proposed additional best practices for review and quality control not included in the above, and to submit more detailed descriptions of the methods already listed, or links to helpful articles.