Computer Assisted Review

Computer Assisted Review (CAR).

Intro. Review of documents with the assistance of computers and specially designed legal search and review software is a well established best practice. If you have more than a thousand documents it simply makes no sense to print them all out and review and mark them by hand without computer assistance. But the term Computer Assisted Review, and the alternative phrase that has the same meaning, Technology Assisted Review, has come to mean more than simply reviewing and coding documents on a computer. It now indicates a process where attorneys use  computer software as indispensable tools to search and find relevant evidence in a big data setting. This is the needle in the haystack problem that is a core issue of e-discovery.

Due to the high volume of ESI that may be relevant to a dispute, even after the Bulk Culling processes described here, it is impractical for individual review of all custodian documents that might be relevant. CAR now means to use software for the review and coding documents, and also for the further search and culling down of a dataset to the most likely relevant documents for final attorney review.

Individual attorney review of all documents that survive Bulk Culling is not a best practice. It is not only exorbitantly expensive, even with the use of low paid contract review attorneys, it is also less effective. This is now well established, not only by extensive attorney experience, but my numerous scientific studies. Grossman, Cormack, Technology-assisted review in e-discovery can be more effective and more efficient than exhaustive manual review, Richmond Journal of Law and Technology, 17(3):11:1–48 (2011); Roitblat, Kershaw, and Oot, Document categorization in legal electronic discovery: computer classification vs. manual review, Journal of the American Society for Information Science and Technology, 61(1):70–80 (2010); Webber, W. Re-examining the Effectiveness of Manual Review (2011); Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, 36:5 Information Processing & Management 697, 701 (2000).

See NYSBA Best Practices In E-Discovery In New York State and Federal Courts (2011) (Guideline 8 contains a good high-level description of the review process; and 10 addresses scope of review and reasonability of search techniques utilized). Also see: The Sedona Conference® Best Practices Commentary on Search & Retrieval Methods, August 2007 (look for updated final version soon).

The CAR best practices are broken down into these sub-pages:

For more background on CARs, specifically those with predictive coding type search engines, see the e-Discovery Team CAR page. Also See Judge Paul Grimm’s standing Discovery Order, for cases before him in the District Court in Maryland. In mid 2015 he added this language in paragraph 6.b.i. on predictive coding:

Parties requesting ESI discovery and parties responding to such requests are expected to cooperate in the development of search methodology and criteria to achieve proportionality in ESI discovery, including appropriate use of computer-assisted search methodology, such as Technology Assisted Review, which employs advanced analytical software applications that can screen for relevant, privileged, or protected information in ways that are more accurate than manual review and involve far less expense.

Planning for a large review project is critical to its success, especially when predictive coding will be used. We suggest you consult this detailed Outline of a Form Plan for a Predictive Coding Project. Also see the excellent work of the Coalition of Technology Resources for Lawyers (CTRL), especially their Model Stipulation and Order for use of predictive coding.

It is Ralph Losey’s radically conservative view that the best practice in CAR is for the raw unreviewed data to be hosted by qualified vendors, and not kept on a law firm’s own computer system. Instead, he recommends that lawyers and law firms only have actual possession of the evidence obtained after review of the bulk collections. For details on his recommended best practice of outsourcing possession to qualified vendors, instead of law firm Litigation Support Departments, and the full rationale therefore, see Best Practices in e-Discovery for Handling Large Stores of Unreviewed Client Data.

If you have any suggestions and care to contribute to this project, or any questions (nothing case specific please), please leave a comment below.