Lib.rs

Extracting Citations

get_citations(), the main executable function, takes three parameters.

1. plain_text ==> str: The text to parse. Should be cleaned first.
2. remove_ambiguous ==> bool, default False: Whether to remove citations that might refer to more than one reporter and can't be narrowed down by date.
3. tokenizer ==> Tokenizer, default eyecite.tokenizers.default_tokenizer: An instance of a Tokenizer object (see "Tokenizers" below).

Cleaning Input Text

For a given citation text such as "... 1 Baldwin's Rep. 1 ...", eyecite expects that the text will be "clean" before being passed to get_citation. This means:

• Spaces will be single space characters, not multiple spaces or other whitespace.
• Quotes and hyphens will be standard quote and hyphen characters.
• No junk such as HTML tags inside the citation.

You can use clean_text to help with this:

from eyecite import clean_text, get_citations

source_text = '<p>foo   1  U.S.  1   </p>'
plain_text = clean_text(text, ['html', 'inline_whitespace', my_func])
found_citations = get_citations(plain_text)

See the Annotating Citations section for how to insert links into the original text using citations extracted from the cleaned text.

clean_text currently accepts these values as cleaners:

1. inline_whitespace: replace all runs of tab and space characters with a single space character
2. all_whitespace: replace all runs of any whitespace character with a single space character
3. underscores: remove two or more underscores, a common error in text extracted from PDFs
4. html: remove non-visible HTML content using the lxml library
5. Custom function: any function taking a string and returning a string.

Annotating Citations

For simple plain text, you can insert links to citations using the annotate function:

from eyecite import get_citations, annotate

plain_text = 'bob lissner v. test 1 U.S. 12, 347-348 (4th Cir. 1982)'
citations = get_citations(plain_text)
linked_text = annotate(plain_text, [[c.span(), "<a>", "</a>"] for c in citations])

returns:
'bob lissner v. test <a>1 U.S. 12</a>, 347-348 (4th Cir. 1982)'

Each citation returned by get_citations keeps track of where it was found in the source text. As a result, annotate must be called with the same cleaned text used by get_citations to extract citations. If you do not, the offsets returned by the citation's span method will not align with the text, and your annotations will be in the wrong place.

If you want to clean text and then insert annotations into the original text, you can pass the original text in as source_text:

from eyecite import get_citations, annotate, clean_text

source_text = '<p>bob lissner v. <i>test   1 U.S.</i> 12,   347-348 (4th Cir. 1982)</p>'
plain_text = clean_text(source_text, ['html', 'inline_whitespace'])
citations = get_citations(plain_text)
linked_text = annotate(plain_text, [[c.span(), "<a>", "</a>"] for c in citations], source_text=source_text)

returns:
'<p>bob lissner v. <i>test   <a>1 U.S.</i> 12</a>,   347-348 (4th Cir. 1982)</p>'

The above example extracts citations from plain_text and applies them to source_text, using a diffing algorithm to insert annotations in the correct locations in the original text.

def annotator(before, span_text, after):
    return before + span_text.lower() + after
linked_text = annotate(plain_text, [[c.span(), "<a>", "</a>"] for c in citations], annotator=annotator)

returns:
'bob lissner v. test <a>1 u.s. 12</a>, 347-348 (4th Cir. 1982)'

Resolving Citations

Once you have extracted citations from a document, you may wish to resolve them to their common references. To do so, just pass the results of get_citations() into resolve_citations(). This function will do its best to resolve each "full," "short form," "supra," and "id" citation to a common Resource object, returning a dictionary that maps resources to lists of associated citations:

from eyecite import get_citations, resolve_citations

text = 'first citation: 1 U.S. 12. second citation: 2 F.3d 2. third citation: Id.'
found_citations = get_citations(text)
resolved_citations = resolve_citations(found_citations)

returns (pseudo):
{
    <Resource object>: [FullCaseCitation('1 U.S. 12')],
    <Resource object>: [FullCaseCitation('2 F.3d 2'), IdCitation('Id.')]
}

Importantly, eyecite performs these resolutions using only its immanent knowledge about each citation's textual representation. If you want to perform more sophisticated resolution (e.g., by augmenting each citation with information from a third-party API), simply pass custom resolve_id_citation(), resolve_supra_citation(), resolve_shortcase_citation(), and resolve_full_citation() functions to resolve_citations() as keyword arguments. You can also configure those functions to return a more complex resource object (such as a Django model), so long as that object inherits the eyecite.models.ResourceType type (which simply requires hashability). For example, you might implement a custom full citation resolution function as follows, using the default resolution logic as a fallback:

def my_resolve(full_cite):
    # special handling for resolution of known cases in our database
    resource = MyOpinion.objects.get(full_cite)
    if resource:
        return resource
    # allow normal clustering of other citations
    return resolve_full_citation(full_cite)

resolve_citations(citations, resolve_full_citation=my_resolve)

returns (pseudo):
{
    <MyOpinion object>: [<full_cite>, <short_cite>, <id_cite>],
    <Resource object>: [<full cite>, <short cite>],
}

Tokenizers

Internally, eyecite works by applying a list of regular expressions to the source text to convert it to a list of tokens:

In [1]: from eyecite.tokenizers import default_tokenizer

In [2]: list(default_tokenizer.tokenize("Foo v. Bar, 123 U.S. 456 (2016). Id. at 457."))
Out[2]:
['Foo',
 StopWordToken(data='v.', ...),
 'Bar,',
 CitationToken(data='123 U.S. 456', volume='123', reporter='U.S.', page='456', ...),
 '(2016).',
 IdToken(data='Id.', ...),
 'at',
 '457.']

Tokens are then scanned to determine values like the citation year or case name for citation resolution.

Alternate tokenizers can be substituted by providing a tokenizer instance to get_citations():

from eyecite.tokenizers import HyperscanTokenizer
hyperscan_tokenizer = HyperscanTokenizer(cache_dir='.hyperscan')
cites = get_citations(text, tokenizer=hyperscan_tokenizer)

test_FindTest.py includes a simplified example of using a custom tokenizer that uses modified regular expressions to extract citations with OCR errors.

eyecite ships with two tokenizers:

hyperscan_tokenizer = HyperscanTokenizer(cache_dir='.hyperscan')

Debugging

If you want to see what metadata eyecite is able to extract for each citation, you can use dump_citations. This is primarily useful for developing eyecite, but may also be useful for exploring what data is available to you:

In [1]: from eyecite import dump_citations, get_citations

In [2]: text="Mass. Gen. Laws ch. 1, § 2. Foo v. Bar, 1 U.S. 2, 3-4 (1999). Id. at 3. Foo, supra, at 5."

In [3]: cites=get_citations(text)

In [4]: print(dump_citations(get_citations(text), text))
FullLawCitation: Mass. Gen. Laws ch. 1, § 2. Foo v. Bar, 1 U.S. 2, 3-4 (1
  * groups
    * reporter='Mass. Gen. Laws'
    * chapter='1'
    * section='2'
FullCaseCitation: Laws ch. 1, § 2. Foo v. Bar, 1 U.S. 2, 3-4 (1999). Id. at 3. Foo, s
  * groups
    * volume='1'
    * reporter='U.S.'
    * page='2'
  * metadata
    * pin_cite='3-4'
    * year='1999'
    * court='scotus'
    * plaintiff='Foo'
    * defendant='Bar,'
  * year=1999
IdCitation: v. Bar, 1 U.S. 2, 3-4 (1999). Id. at 3. Foo, supra, at 5.
  * metadata
    * pin_cite='at 3'
SupraCitation: 2, 3-4 (1999). Id. at 3. Foo, supra, at 5.
  * metadata
    * antecedent_guess='Foo'
    * pin_cite='at 5'

In the real terminal, the span() of each extracted citation will be highlighted. You can use the context_chars=30 parameter to control how much text is shown before and after.

Hyperscan installation

To use HyperscanTokenizer you must additionally install the python hyperscan library and its dependencies. python-hyperscan officially supports only x86 linux, though other configurations may be possible.

Hyperscan installation example on x86 Ubuntu 20.04:

apt install libhyperscan-dev
pip install hyperscan

Hyperscan installation example on x86 Debian Buster:

echo 'deb http://deb.debian.org/debian buster-backports main' > /etc/apt/sources.list.d/backports.list
apt install -t buster-backports libhyperscan-dev
pip install hyperscan

Hyperscan installation example with homebrew on x86 MacOS:

brew install hyperscan
pip install hyperscan