Hybrid Multimodal

Hybrid Multimodal Search.  Although attorney use of software as tools for search is undoubtedly a better practice than manual review, it is still a best practice for an attorney’s legal judgment to control the search, rather than delegate the search entirely to software algorithms. It is a hybrid process, of Man and Machine, but with Man in control.

  • Rejection of the “Borg” Approach. Contrary to the assertion of some vendors and experts in the field information retrieval, EDBP does not accept the complete delegation of search to computers as a best practice, a process that Ralph Losey playfully refers to as the Borg approach. Although EDBP recognizes that legal search software is constantly improving, and greater reliance on software is now warranted using a hybrid approach, the consensus remains that attorney skill and judgment remains critical, and that attorneys must use and control these tools, not the other way around. See Eg.: Losey, R., Why the ‘Google Car’ Has No Place in Legal Search: Caution against over reliance on fully automated methods of active machine learning; Losey. R, Predictive Coding 4.0 – Nine Key Points of Legal Document Review and an Updated Statement of Our Workflow Part Two and Part Three of the seven part series. Losey, R., Day Two of a Predictive Coding Narrative: More Than A Random Stroll Down Memory Lane (2012); Losey, R., Analysis of the Official Report on the 2011 TREC Legal Track – Part Three (2012). Please feel free to express your dissenting or refining views in the comments sections below, as the EDBP is designed as an ever-open-process that will constantly evolve with the times and new technologies.
  • Multimodal. Search should not only be hybrid, with man working closely with computer, but in control, but search should be multimodal. This means that all types of search tools and methods should be used, not just one. EDBP at the present time rejects monomodal search methods as a best practice, regardless of the particular search method that may be advanced as the sole and exclusive search method required. See Losey. R, Predictive Coding 4.0 – Nine Key Points of Legal Document Review and an Updated Statement of Our Workflow Part Two and Part Three of the seven part series. EDBP believes that all tools should be used, as shown in the pyramid diagram below, and not just one to the exclusion of all others. EDBP thus does not accept the alternate view that Predictive Coding has now advanced to the point that other methods are obsolete and a Borg monomodal approach is the best practice. Again, dissenting or refining views are invited and EDBP remains open to constant review.

Despite the many well-known limitations of manual review and key word search, these search methods still have a place in multimodal review. See Losey. R, Predictive Coding 4.0 – Nine Key Points of Legal Document Review and an Updated Statement of Our Workflow Part Four of the seven part series. For discussion of the limits of keyword search see: Hedlin, Tomlinson, Baron, Oard, 2009 TREC Legal Track OverviewTREC legal track at §3.10.9. (2010) (Boolean searches had a recall average of less than 4%); Predictive Coding: Reading the Judicial Tea Leaves, (Law Tech. News, Oct. 17, 2011); Grossman, Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, Richmond Journal of Law and Technology,Vol. XVII, Issue 3, Article 11 (2011); Losey, R., The Information Explosion and a Great Article by Grossman and Cormack on Legal Search (2011).

Although keyword search is a powerful tool when used correctly, it is ineffective when used in the blind, as is a common practice today, without actually studying the documents with hits, both statistically and by content. Losey, R. Child’s Game of “Go Fish” is a Poor Model for e-Discovery Search (2009). But when done properly, with witness interviews and tested by a full study and refinement of proposed terms, keyword search can still be very effective, especially in certain simple projects. William A. Gross Construction Associates, Inc. v. American Manufacturers Mutual Insurance Co., 256 F.R.D. 134, 136 (S.D.N.Y. 2009). See Losey. R, Predictive Coding 4.0 – Nine Key Points of Legal Document Review and an Updated Statement of Our Workflow Part Four of the seven part series. Tested Keyword search is especially effective when you have unique terms and names, and when used with Boolean connectors and direction to particular metadata and document fields (which is the basic meaning of parametric).

Predictive coding (Intelligent Review) is the top of a larger search and review pyramid shown above. The foundation of the process remains Expert Manual Review of documents by skilled attorneys and paralegals. See Losey. R, Predictive Coding 4.0 – Nine Key Points of Legal Document Review and an Updated Statement of Our Workflow Part Four of the seven part series. They provide the requite judgments and initial input needed to train predictive coding search algorithms. Id. at Part Five (importance of good SMEs).

Keyword Search is an important first tool to find evidence, so long as it is combined with immediate metric reports and attorney evaluation of term effectiveness. Different parameters and Boolean logic searches may be quickly tried and tested. The defects of blind Go Fish type keyword methods are thus avoided. Still, unless you have a simple search project, as further explained below, you should never rely on keyword search alone. You should use all of the search tools. Unfortunately most attorneys today are over-relying on keyword search, along with simple linear review, because those are the only methods they know. The Search Pyramid to them looks more like this, which we call a Dunce Hat. You want to avoid wearing this.


3-factors_hybrid_prohumanStill, in some very limited circumstances, keyword search alone can be effective. Of course, it has to be done right, in a tested fashion using Boolean connectors and parametric features. The limited circumstances where keyword search alone may be effective is in simple lawsuits with clear search targets, non-complex data, and a good expert in control. In these circumstances it may be the only tool you need. This is explained further, along with the diagram right, in Part Four of the seven part series on Predictive Coding 4.0.

In addition, any software review platform today must also include powerful Similarity and Concept searches. All of these methods are part of the standard multimodal best practice protocols. Id. at  Part Four. For a good description of these admittedly powerful, albeit now somewhat dated search tools (at least compared to active machine learning), see the article by D4’s Tom Groom, The Three Groups of Discovery Analytics and When to Apply Them.

For another description of the multimodal approach from the perspective of using predictive coding and other techniques to cull out irrelevant documents, see Losey’s Two-Filter Document Culling – Part One and Part Two. The first filter is Bulk Culling, and the second filter refers to the CAR approach recommended as a best practice. It is summarized by the below diagram. Both filters are multimodal. Reliance on one search method alone is never a best practice.


Practicing attorneys and paralegals are invited to submit descriptions of particular search methods for inclusion here.

2 thoughts on “Hybrid Multimodal

  1. Anonymous comment:

    Not yet sure I like the pyramid graphic for your multimodal search description. Might be taken too much as a progressive workflow model that is not a best practice unless and until one gets to the top and uses IR. I see many variations in use of these tools for different types of projects. For example, clustering, similar, threading , used in conjunction with Boolean and regular expressions to batch docs out to a modified traditional team review workflow. I get that this is a diagram of an aspirational concept [no pun intended], but the problem with such is one size never fits all, and busy practitioners (or heaven forbid judges) without a deep hands-on user experience with these tools will take this upward pointy thing too literally in that this is “the” workflow that must be used when these tools are deployed on a project or they are outside of “best practices.”

    The best practice should be to use the workflow that meets the needs of a given matter (re budget, proportionality and all the other things you’ve helped me think about). And that should, in my mind’s eye, be more of a selection of different ways to use the tools in a hybrid review workflow model. Somewhere in such a diagram there needs to be a point of reference to scope analysis of the project and definition of which tools in what workflow would best serve the client and our ethical duties re discovery. Not sure yet what that might look like, but if I figure it out I’ll let you know. Perhaps it needs to be a three dimensional graphic to do justice to LSI in its space, [ ], the related tools in your multimodal approach, and the workflow selection. Intelligent Review as you lay that term out, for many matters depending on many factors, is not necessarily a best practice.

  2. Pingback: depo.com | Does Your CAR (“Computer Assisted Review”) Have a Full Tank of Gas?

Please leave a comment and help improve EDBP