✍️ Introduction to Information Retrieval — Handwritten-Style Study Notes

Lecture: Introduction to Information Retrieval
Purpose: A compact, memorable version of the lecture slides for revision + exam preparation.
Source: Lecture slides provided in this conversation.

🧠 Big idea: IR = finding information that is relevant to a user's information need from a collection of information resources.

🟦 1. What is Information Retrieval (IR)?

✍️ Definition

Information Retrieval (IR) is the activity of obtaining information resources relevant to an information need from a collection of information resources.

🔎 Search can be based on:

Metadata
Full-text indexing
Other content-based indexing

Why do we need IR?

Because we have too much information → information overload.

🧠 Remember:
Information Need → Search → Relevant Information

🟨 2. Why Information Retrieval?

🚨 Information Overload

Information overload = difficulty understanding an issue and making decisions because there is too much information.

As the amount of information grows, manually finding useful information becomes difficult.

🌐 Internet growth

The lecture uses the growth of the Internet as an example of the rapidly increasing amount of available information.

🧠 Main reason for IR

More information → harder to find what matters → need IR.

📦 Handling Unstructured Data

Structured data

A database system is a good choice for structured data.

Example idea:

ID | Name | Role
---|------|----------
1  | Jack | Professor
2  | David| Staff

The structure and meaning of fields are defined.

Unstructured data

Examples mentioned in the lecture:

Web document text
Emails
Images
Audio
Video

The lecture emphasizes that unstructured data is more dominant and that its semantic meaning can be unknown.

💡 Exam memory:
Structured → DB
Unstructured → IR

🟩 3. Why IR is Essential

IR is an essential tool for dealing with:

Information overload
Large collections of information
Unstructured information

🧠 One-line memory

IR helps us find useful information inside a huge amount of information.

🕰️ 4. History of Information Retrieval

🌱 Early idea — Vannevar Bush

The idea was popularized by Vannevar Bush's 1945 article “As We May Think”.

💭 Memex

Bush described a memex as a device where an individual could store:

Books
Records
Communications

and consult them with speed and flexibility.

The lecture connects this idea to the concept of a search engine.

🧠 Remember

Bush → 1945 → “As We May Think” → Memex

🏛️ 5. Major Research Milestones

1950s–1960s → Foundations

Important work included:

Luhn → automatic indexing
Cleverdon → Cranfield evaluation methodology and index experiments
Salton → SMART system and experiments

1970s–1980s → Retrieval Models

Major developments:

Vector Space Model
Probabilistic Models

1990s → New Models + New Tasks

Developments included:

Language Models
TREC evaluation
Web Search

2000s–Present → Applications & Interdisciplinary Work

Important developments:

Learning to Rank
Scalability — e.g. MapReduce
Real-time Search
More applications, especially Web Search
Interaction with other fields

🧠 Timeline shortcut

50s–60s  → Foundations
70s–80s  → Retrieval Models
90s      → Language Models + TREC + Web Search
2000s→   → Learning to Rank + Scalability + Real-time Search

🧪 6. Catalysts for IR

🎓 Academia — TREC

Text Retrieval Conference (TREC) began in 1992.

Purpose

TREC provided infrastructure for large-scale evaluation of text retrieval methodologies.

The lecture highlights TREC as an important test-bed for academic IR research.

🌍 Industry — Web Search Engines

The World Wide Web caused an explosion of published information.

This drove innovation in IR techniques.

The lecture mentions the early development of web search engines and later commercial search engines such as:

Lycos
Magellan
Excite
Infoseek
Inktomi
Northern Light
AltaVista
Yahoo!
Google
Bing

🧠 Remember:
WWW explosion → more information → stronger need for IR → search-engine innovation

🔍 7. Search Engine Market Example

The lecture shows historical global search-engine market examples for:

🖥️ Desktop

Fall 2017 figures shown in the slide include:

Google — 81.12%
Bing — 6.97%
Baidu — 5.82%
Yahoo — 4.74%
Ask — 0.17%
AOL — 0.05%
Excite — 0.01%
Other — 1.13%

📱 Mobile

Fall 2017 figures shown in the slide include:

Google — 96.38%
Yahoo — 1.21%
Bing — 0.67%
Baidu — 0.43%
Ask — 0.03%
AOL — 0%
Other — 1.27%

⚠️ These are historical figures from the lecture slide, not current market shares.

🧩 8. How to Perform Information Retrieval?

The lecture first points out that IR existed even before computers.

With computers, IR became increasingly automated.

⚙️ Basic IR workflow

Think:

Documents
   ↓
Document Analysis
   ↓
Parsing & Indexing
   ↓
Repository / Index
   ↑
User Query
   ↓
Retrieval
   ↓
Results
   ↓
User Judgments
   ↺
Learning

Key components

Crawler and Indexer
Document Analyzer
Parsing & Indexing
Repository
User Query
Results
Judgments
Learning

🧠 Memory trick

C → A → I → Q → R → J → L

Crawl → Analyze → Index → Query → Retrieve → Judge → Learn

🟪 9. Core Concepts in IR

There are three major concepts to remember:

        INFORMATION RETRIEVAL
                 │
       ┌─────────┼─────────┐
       ↓         ↓         ↓
     Query    Document   Retrieval
   Representation Representation  Model

9.1 🔎 Query Representation

A query represents the user's information need.

Two important gaps

Lexical Gap

Different words can express related information.

Example from the lecture:

say ↔ said

The words differ lexically even though they are closely related.

Semantic Gap

Different terms can refer to similar concepts.

Example from the lecture:

ranking model ↔ retrieval method

🧠 Remember:
Lexical = words
Semantic = meaning

9.2 📄 Document Representation

Documents need to be represented in a form that allows efficient retrieval.

Important idea

A special data structure is used for efficient access.

Document representation also has to deal with:

Lexical gap
Semantic gap

9.3 🎯 Retrieval Model

A retrieval model contains algorithms that find the most relevant documents for a given information need.

🧠 One-line definition

Retrieval model = algorithm for finding relevant documents.

🏎️ 10. A Glance at Modern Search Engines

The lecture compares older and modern search experiences.

Older search engines

The lecture shows examples such as Yahoo! with categories and directories.

Modern search

The lecture shows Google and emphasizes three user demands:

Efficiency
Convenience
Understanding

🧠 Search engines evolved toward:

Find information
      +
Find it efficiently
      +
Make it convenient
      +
Understand the user's need

🚫 11. IR is NOT Just Web Search

This is an important exam point.

Web Search is one important area of IR — but IR is not limited to Web Search.

IR also includes:

⭐ Recommendation

Example idea:

“Recommended based on your browsing history”

❓ Question Answering

Systems that retrieve information to answer questions.

⛏️ Text Mining

Extracting useful information/patterns from text.

📢 Online Advertising

IR techniques can be used in online advertising.

🏢 Enterprise Search

The lecture describes enterprise search as:

Web search + desktop search

🌐 12. IR and Related Areas

IR is a highly interdisciplinary area.

The lecture connects IR with:

Databases
Natural Language Processing
Machine Learning
HCI
Data Mining
Statistics
Optimization
Software Engineering
Computer Systems
Web Applications
Bioinformatics
Pattern Recognition
Library & Information Science

🧠 Big picture

             ┌── Databases
             ├── NLP
             ├── Machine Learning
       IR ───┼── HCI
             ├── Statistics
             ├── Data Mining
             ├── Optimization
             └── Computer Systems

🆚 13. Information Retrieval vs Database Systems

Information Retrieval	Database Systems
Unstructured data	Structured data
Object semantics are subjective	Object semantics are well defined
Simple keyword queries	Structured query languages
Example: keyword search	Example: SQL
Relevance-driven retrieval	Exact retrieval
Effectiveness is primary issue	Efficiency is emphasized
Efficiency is also important	Exact results are expected

🧠 Fast comparison

IR  → Unstructured → Keywords → Relevance
DB  → Structured   → SQL      → Exactness

🔄 14. IR and DBs Are Getting Closer

IR → DB

Database systems can provide:

Approximate search

Example mentioned:

SELECT * FROM articles
WHERE MATCH(title, body)
AGAINST ('database');

DB → IR

Information extraction can convert unstructured data → structured data.

Example:

Unstructured data
       ↓
Information extraction
       ↓
Structured representation
       ↓
Knowledge base

The lecture also mentions semi-structured representation, such as XML data, where queries can use structured information.

🆚 15. Information Retrieval vs NLP

Information Retrieval	Natural Language Processing
Computational approaches	Cognitive, symbolic & computational approaches
Statistical / shallow understanding	Semantic / deep understanding
Handles large-scale problems	Often handles smaller-scale problems

🧠 Easy memory

IR  → large scale + statistical/shallow
NLP → deeper semantic understanding

🔄 16. IR and NLP Are Getting Closer

IR → NLP

IR contributes:

Larger data collections
Scalable / robust NLP techniques
Translation models
Natural-language-based QA systems

NLP → IR

NLP contributes:

Deep analysis of text documents
Deep analysis of queries
Information extraction for structured IR tasks

🧠 Big picture

IR gives NLP scale; NLP gives IR deeper text understanding.

📚 17. Textbooks Mentioned

The lecture recommends:

📘 Introduction to Information Retrieval

Christopher D. Manning, Prabhakar Raghavan, Hinrich Schütze

Cambridge University Press, 2007.

📕 Search Engines: Information Retrieval in Practice

Bruce Croft, Donald Metzler, Trevor Strohman

Pearson Education, 2009.

🔮 18. IR in the Future

The lecture highlights several future directions.

📱 Mobile Search

Desktop search + location?

The lecture explicitly notes: “Not exactly!!”

🤝 Interactive Retrieval

Machine collaborates with humans for information access.

🧑‍💻 Personal Assistant

Proactive Information Retrieval

Instead of waiting for a query, systems can proactively help with information access.

The lecture connects this idea with the Knowledge Navigator concept.

➕ And many more...

“You name it!”

🧠 19. FINAL EXAM CHEAT SHEET

If you have only a few minutes, memorize this section.

⭐ What is IR?

IR = obtaining information resources relevant to an information need from a collection of information resources.

⭐ Why IR?

Information explosion
        ↓
Information overload
        ↓
Hard to find useful information
        ↓
Need Information Retrieval

⭐ What type of data?

Main focus: unstructured information

Examples:

Text + Web documents + Emails + Images + Audio + Video

⭐ Three Core Concepts

1. Query Representation
2. Document Representation
3. Retrieval Model

⭐ Two Important Gaps

Lexical Gap  → difference in WORDS
Semantic Gap → difference in MEANING

⭐ IR Applications

Remember:

W-R-Q-T-A-E

Web Search
Recommendation
Question Answering
Text Mining
Advertising
Enterprise Search

⭐ IR vs DB

IR → Unstructured → Keyword → Relevance
DB → Structured   → SQL     → Exact

⭐ IR vs NLP

IR  → Statistical / shallow + large scale
NLP → Semantic / deep understanding

⭐ Important History

1945 → Vannevar Bush → Memex
1950s–60s → Foundations
1970s–80s → Vector Space + Probabilistic Models
1990s → Language Models + TREC + Web Search
2000s→ → Learning to Rank + Scalability + Real-time Search

🎯 20. Active Recall — Test Yourself

Don't look above. Try answering these from memory.