Scribd Logo
Upload

Welcome to Scribd!

  • 238 views

    Text Preprocessing: Information Retrieval

    Uploaded by

    Saad Bin Shahid
    Text preprocessing is an important step in information retrieval, natural language processing, and text mining. It involves techniques like tokenization, removing stop words, normalization, stemming, lemmatization, and acronym expansion. Tokenization breaks text into tokens or words, while removing stop words filters out common words. Lemmatization and stemming reduce inflected words to their base forms. Preprocessing helps improve the effectiveness of information retrieval and natural language understanding.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd

    Text Preprocessing: Information Retrieval

    Uploaded by

    Saad Bin Shahid
    238 views16 pages
    Text preprocessing is an important step in information retrieval, natural language processing, and text mining. It involves techniques like tokenization, removing stop words, normalization, stemming, lemmatization, and acronym expansion. Tokenization breaks text into tokens or words, while removing stop words filters out common words. Lemmatization and stemming reduce inflected words to their base forms. Preprocessing helps improve the effectiveness of information retrieval and natural language understanding.

    Original Description:

    Text Processing

    Original Title

    Text Preprocessing 02

    Copyright

    © © All Rights Reserved

    Available Formats

    PPTX, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Text preprocessing is an important step in information retrieval, natural language processing, and text mining. It involves techniques like tokenization, removing stop words, normalization, stemming, lemmatization, and acronym expansion. Tokenization breaks text into tokens or words, while removing stop words filters out common words. Lemmatization and stemming reduce inflected words to their base forms. Preprocessing helps improve the effectiveness of information retrieval and natural language understanding.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd
    Download as pptx, pdf, or txt
    238 views16 pages

    Text Preprocessing: Information Retrieval

    Uploaded by

    Saad Bin Shahid
    Text preprocessing is an important step in information retrieval, natural language processing, and text mining. It involves techniques like tokenization, removing stop words, normalization, stemming, lemmatization, and acronym expansion. Tokenization breaks text into tokens or words, while removing stop words filters out common words. Lemmatization and stemming reduce inflected words to their base forms. Preprocessing helps improve the effectiveness of information retrieval and natural language understanding.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd
    Download as pptx, pdf, or txt
    You are on page 1of 16

    TEXT

    PREPROCESSING
    Information Retrieval
    Role
    Preprocessing is an important task and critical step in Text
    mining, Natural Language Processing (NLP) and
    information retrieval (IR).
    In the area of Text Mining, data preprocessing used for
    extracting interesting and non-trivial and knowledge from
    unstructured text data.
    Information Retrieval (IR) is essentially a matter of
    deciding which documents in a collection should be
    retrieved to satisfy a user's need for information based
    upon query.
    Techniques
    Tokenization
    Stop Words (Common Words)
    Removal
    Normalization
    Stemming
    Lemmatization
    Acronym Expansion
    Tokenization
    Break the input in to words.
    Converts character stream -> token stream.
    Called tokenizer / lexer / scanner.
    The aim of the tokenization is the exploration of the
    words in a sentence.
    Tokenizer then hooks up to parser.
    A parser defines rules of grammar for these tokens
    and determines whether these statements are
    correct.
    Tokenizer then feeds the token to retrieval system for
    further processing.
    Identifying Tokens
    Divide on whitespace and throw away the
    punctuations?
    There are many documents that we see in day to
    day life that are structured using markup tags.
    HTML, XML, ePub etc
    What is the best way to deal with these tags?
    Use them as delimiter / tokens.
    Filter them our entirely.
    Challenges in Tokenization
    Challenges in tokenization depend on the type of
    language.
    Languages such as English and French are referred to as
    space delimited.
    Languages such as Chinese and Thai are referred to as
    unsegmented as words do not have clear boundaries.
    Tokenizing unsegmented language sentences requires
    additional lexical and morphological information.
    Advanced Tokenization
    Convert the character stream to tokens using any
    programming language.
    Libraries used in the programming language uses
    specific grammars to tokenize a text.
    They can easily identify comments, literals.
    Java has tokenization libraries
    java.util.Scanner
    Java.util.String.split()
    java.util.StringTokenizer
    Python provides you nltk.tokenize package which
    tokenizes the input text.
    Tweet Tokenizer
    This tokenizer tokenizes the tweets according to their writing context
    e.g links as a separate complete token.

    RT @Edourdoo: Shocking picture of the


    earthquake in Nepal, http://t.co/2N09Jz96lq

    RT @NewEarthquake: 4.7 earthquake, 22km W of Kodari, Nepal. Apr 25 14:05 at


    epicenter (31m ago, depth 10km). http://t.co/5I5vVQnCQe

    ['RT', ':', 'Shocking', 'picture', 'of', 'the', 'earthquake', 'in', 'Nepal', ',',
    'http://t.co/2N09Jz96lq']
    Dropping Common Terms (Stop Words)
    Stop words are insignificant or most common words in a
    language
    In English articles prepositions etc.
    To reduce vast amount of unnecessary information retrieved after
    search query.
    Some tools specifically avoid removing these stop words to
    support phrase search
    The general trend in IR systems over time has been from standard
    use of quite large stop lists (200-300 terms) to very small stop lists
    (7-12 terms) to no stop list whatsoever.
    Some stop words Accordingly, Across, Actually, After, Afterwards
    Dropping Common Terms (Stop Words)

    RT @hendhunu: #NepalQuakes More than 100 killed in powerful Nepal


    #earthquake, say govt. officials and police http://t.co/Bk50ubiZPB

    RT @hendhunu: #NepalQuakes 100 killed powerful Nepal #earthquake, say govt.


    official police http://t.co/Bk50ubiZPB
    Removed words are More, than, in, and (nltk.corpus stopwords)
    Normalization
    Transforming text into a single canonical form.
    Text normalization requires
    Awareness of type of text to be normalized
    How it is to be processed afterwards
    It is mostly used to:
    convert text to speech.
    Numbers and dates.
    Acronyms and abbreviations
    Non-standard "words" that need to be pronounced differently depending
    on context.

    Depending on the language:


    $200 two hundred dollars in English
    $200 lua selau tl in Samoan.
    Lemmatization
    Words appear in several inflected forms e.g the verb to
    walk may appear as walk, walked, walks, walking.
    Process of grouping together the different inflected forms
    of a word so they can be analyzed as a single item.
    Lemmatization reduces inflection or derivationally related
    forms of a word to a common base form.
    The base form of the word is called the lemma for the
    word.
    It can match different forms of verbs, adjectives and adverbs, and
    even for synonyms and wildcard matching.
    Sometimes refer to it as fuzzy matching.
    Lemmatization
    Example:
    am, are, is be
    car, cars, car's, cars car
    the boy's cars are different colors the boy car be differ color

    Using nltk.wordnet.WordNetLemmatizer

    24 hour Control Room for queries regarding the Nepal Earthquake.

    24 hour Control Room for query regard the Nepal Earthquake.


    Stemming vs. Lemmatization
    Stemmingusuallyreferstoacrudeheuristicprocessthatchops
    offtheendsofwords.
    Aimingtogetthewordbychoppingandoftenincludestheremovalof
    derivationalaffixes.
    Lemmatizationusuallyreferstodoingthingsproperlywiththe
    useofavocabularyandmorphologicalanalysisofwords.
    Aimingtoremoveinflectionalendingsonlyandtoreturnthebaseor
    dictionaryformofaword

    Saw s bystemming
    Sawseeorsawbylemmatization
    Dependsonwhethertheuseofthewordwasasaverboranoun.
    Acronym Expansion
    Word or name formed as an abbreviation from the initial
    components in a phrase or a word,
    Individual letters (as in NATO or laser) or syllables (as in Benelux).
    Words have multiple abbreviations depending on context
    Thursday could be abbreviated as any of Thurs,Thur, Thr, Th, or T
    Abbreviate library contains a dictionary of known
    abbreviations.(Python)
    Each word has preferred abbreviation (Thr for Thursday)
    The basic `abbreviate` method will only apply preferred
    abbreviations and no heuristics.
    For best possible matching target length and effort is given.
    Go SF Giants! Such an amaazzzing feelin!!!!
    \m/ :D
    Stopwords

    SF Giants! amaazzzing feelin!!!! \/ :D


    Special chars

    SF Giants amaazzzing feelin


    Spell check

    SF Giants amazing feeling

    Stemming

    SF Giants amazing feel me


    SF Giants amazing feel
    stopwords

    You might also like