Map Result Quality: Judge Guidelines

Updated 2021-10-12. Changes in this version are highlighted in BLUE. Previous changes in ORANGE.

1. Overview
This HIT app shows queries made by users of a map search website (e.g. bing.com/maps). Your task is to
read the query and understand the original user’s search “intent” (what they are searching for).

Then, you will rate the specificity of the result when compared to the MAIN THING that the user was
looking for: is the result Slightly too specific, Slightly too general (not specific enough), or is it Just
right? If none of these are true, then you will rate the specificity as Other / Bad.

Below, you see a HIT with an “Address” query. The result matches the street name from the query, but it
is Slightly too specific: the query is for an entire street, but result is a single street number that excludes
the rest of the street.

2. Judgment steps
Step 1: Understand the query
At the top left of the HIT app, Step 1 shows the user’s original Query and its Query Type.
The Query Type will help you understand the user’s search intent, especially when the query could be
interpreted in different ways. Query Types are:
• Address: User is searching for a street address, like “123 Main Street, San Francisco CA” or
“Interstate 80 Nevada”.
• Place: User is searching for a region, district, political entity (city/village/town, state/province,
country, etc.), such as “San Francisco CA”, or Postal Code.
• Business: User is searching for a business/organization by name, chain, product, category (type
of product or service), or by phone number. Includes (but not limited to) hotels, K-12 schools,
official buildings (not famous), business parks / industrial estates, and organizations of all kinds.
Note: some businesses have location words in their proper name, e.g. “Kentucky Fried Chicken”.
• POI (Point of Interest): User is searching for a specific, significant man-made "Point of Interest",
often with important cultural, historical, tourist, or other major public interest.
• Natural Feature: User is looking for natural area (e.g. parks, forests, ski areas etc.) or
geographical feature (e.g. mountains, rivers, lakes, beaches, deserts, islands etc.).

You must determine the MAIN THING the user was looking for. The HIT app will try to help by estimating
the MAIN THING, with the most specific part in bold face. (Example: for query “123 Main St, San
Francisco”, MAIN THING would appear as “123 Main St”.) However, you cannot completely rely on this.
It is only a hint / clue, and if it is wrong you will have to determine the true MAIN THING and its most
specific part.

N.B. The MAIN THING is often shown in simplified form (i.e. no punctuation) to make judging easier, but
sometimes you will need to refer to the original Query instead (to determine the true MAIN THING).

Step 2: Understand the result

In Step 2, the result is shown. Both the name and its full address are displayed (they may be identical). Is
it relevant to the query? Does it have the same level of specificity (not more specific, nor more general)?

Step 3: Research query and result

Select “Query” or “Result” to choose between
researching Query or Result.

Select default search engine for right research

panel or disable autoload of research.
Enable Google research link. Select if
research is blocked by Google Captcha.

The research pane will show search results for the query or result depending on whether “Query” or
“Result” button is selected. Usually, you will usually “Query” mode, but sometimes you need to research
the “Result” instead. “Result” mode is especially important when judging category queries to determine
if the result matches the query’s search intent for the category (e.g. Correct: “coffee shop” →
“Starbucks”; Correct: “hotel” → ”Hilton”; Wrong: “hotel” →“Starbucks”).
If the query is in a different language than yours, be sure to use the Translate button so you can
understand the user’s search intent.

CAUTION: Sometimes, you must use the “Bing Maps” or “Google Maps” buttons. The queries in this HIT
app were made by users of MAP search websites, such as bing.com/maps. This means that the queries
must be interpreted in a map context. Results from main search engines are often useful but are
sometimes misleading because they do NOT limit their results to map context. When in doubt, research
with a map search engine, and always interpret as a MAP query. For example, "Upland" on main search
engines (e.g. bing.com, google.com) returns “upland.com” as top result, but “Upland, CA” is the top
result on Maps, so “Upland, CA” is the correct result.

Ignore unimportant information and extra, non-conflicting information

Query: Ignore unimportant information, such as apartment # / unit # / suite # / floor #, or a person’s
name. It is not reasonable (and in some cases not ethical) for a search engine to return this detail.

Result: Ignore extra, non-conflicting information. Ignore extra info unless it conflicts with the query
and/or makes it more specific (or general). Often the extra information helps to describe the result more
completely, so it is more helpful to the user.

• When a query is incomplete, the result should have complete info, especially if the result has full
and proper names. Example: for query “222 15th Omaha”, extra (complete) information in the
result of “222 South 15th St, Omaha, NE 68102” does NOT make it too specific. It is Just right
• Extra info is good unless the extra information conflicts with the query (makes the result
something different than what the user was looking for). For example, an address query for “123
Main Street, San Francisco” that has a business result of “Macy’s, 123 Main St, San Francisco CA
94105” is OK (same address as query but extra, non-conflicting info). However, for place query
“SF, CA”, an address result of “123 Main St, San Francisco CA 94105” would be MUCH too
specific and should be rated as Other / Bad.

Step 4: Judgment. Is result too specific? Is it too general (not specific enough)?
Rate the specificity of the result compared to the user’s search intent (the MAIN THING). Result should
be very similar to the MAIN THING in query AND be as specific as the MAIN THING. That is what is meant
by “Just right.” Is result Slightly too specific, Slightly too general, or Just right? If none of those apply,
choose Other / Bad.

Step 3: Specificity Description

Slightly too specific Result is too specific but is only slightly different. Closely-related “child” of
most specific part of MAIN THING. Has a STRONG relationship with search
intent. Any extra info must not conflict too much with MAIN THING.
Slightly too general Result is too general but is only slightly different. Closely-related “parent” of
most specific part of MAIN THING. Has a STRONG relationship with search
intent. Any extra info must not conflict too much with MAIN THING.
Just right Result is very similar to the MAIN THING in query AND it is as specific as the
MAIN THING. It is not more specific and is not less specific (though it may have
other problems).
 Other / Bad Anything else. Includes results that are even more specific than Slightly too
specific, or more general than Slightly too general. It also includes BAD results
that simply have nothing at all to with the MAIN THING.

Close parent/child relationships

For a result to be considered as either Slightly too specific or Slightly too general, its most specific Entity
Type must have a close parent/child relationship with the most specific Entity Type of the MAIN THING.
That means it must belong to the same Entity Group as the MAIN THING, and that Entity Group must
allow close parent/child relationships. Parent and child may be in either the query or the result.

Entity Group: Place Entity Group: Address Entity Group: Other (Business,
POI, Natural Feature, category of
any type)
Parent/child relationships: Parent/child relationships: Parent/child relationships:
None Street name is always parent Any
Entity Types: Entity Types: Entity Types:
• Neighborhood / District • Street name (parent) • Business (name, chain, or
• City / Town / Village • Street number category), including
• State / Province • Street type government office / dept.
• Post Code • Street direction • POI
• Country / Continent • Other street info • Natural Feature

Notes:
• Address Entity Group: if result has different Street name than MAIN THING, specificity must be
rated as Other/ Bad.
• Most specific part of MAIN THING does not include any unimportant details.

EXAMPLE: Consider query for “Main St., SF” with result of “123 Main St, San Francisco CA 94105”.
• The result matches the street name from the query, but it is a specific street number instead of
the entire street.
• This street number is a part of the entire street, like the “child” of a “parent”, but for this HIT
app you must determine if it is an allowed close parent / child relationship.
• The most specific part of the query (“123”) has an Entity Type of Street number, and the most
specific part of the result (“Main”) has an Entity Type of Street name.
• Both Entity Types belong to the Address Entity Group, which allows close parent / child
relationships, but only if the parent is a Street name.
• In this case, the Street number is a child of the Street name – it is a smaller part of the parent.
Thus, they have a close parent / child relationship.
• Therefore, you should rate the result as Slightly too specific.

Special considerations
1. Category queries have results that match the intent of the query, not the text. Correct:
“coffee shop” → “Starbucks”; Correct: “hotel” → ”Hilton”; Wrong: “hotel” →“Starbucks”).
 2. Pay special attention to Category queries for real estate. Many results will list only an
address, which is correct (Just right). Example: for query “property for sale”, a result of “86
Morris Cove Rd, New Haven, CT 06512” is “Just right” because it is the address of a house
that may be for sale.
3. Results may make spelling corrections or use alternate names, valid abbreviations, or any
other reasonable alternate forms unless they affect specificity or correctness of the result.
4. Queries for most large places (e.g. a City) should not have a single address (street / street
number). Example: for query “Detroit, MI”, a result of “Detroit, MI, 2 Woodward Ave #1340,
Detroit, MI 48226” is MUCH too specific. The user was looking for the whole city, but result
has only one tiny portion. Such a result is not a Place, thus it is in a different Entity Group.
However, some large areas (especially parks) might have a valid address (as an access point)
which should not affect your judgment.
5. Finally, do NOT judge the quality of results in the research pane. Instead, judge the result
listed in Step 2.

3. Examples
Basic examples (with detail)
In examples below, MAIN THING in query is outlined, and any match in result is also outlined. The most
specific Entity Type of each is in bold face.

QUERY RESULT SPECIFICITY REASON

Main St, San Main St., San Just right Perfect match
Francisco Francisco CA
123 Main St., San Slightly too Address Entity Group allows parent/child
(Entity Group: Francisco CA specific relationships with Street Name as a
Address) parent. MAIN THING’s most specific Entity
Type is the Street Name, thus it is the
parent. Result’s most specific Entity Type
is Street number – it is the child.
→ Close parent/child relationship.
San Francisco CA Other - MAIN THING: Entity Type (Street name)
→ Address Entity Group
- Result: Entity Type (City)
→ Place Entity Group.
- Different Entity Groups.
→ Not a close relationship.

QUERY RESULT SPECIFICITY REASON

123 Main, San 123 Main St., San Just right Perfect match. The extra information is
Francisco Francisco CA 94105 non-conflicting.
123 King St., San Other For Address Entity Group, Street Name
(Entity Group: Francisco CA 94105 must be very similar.
Address) Main St., San Slightly too - MAIN THING: Entity Type (Street name)
Francisco CA general → Address Entity Group
- Result: Entity Type (Street number)
 → Address Entity Group.
- Parent: Street name; Child: Street
number.
→ Close parent/child relationship.
San Francisco CA Other - MAIN THING: Entity Type (Street number)
→ Address Entity Group
- Result: Entity Type (City)
→ Place Entity Group.
- Different Entity Groups.
→ Not a close relationship.

QUERY RESULT SPECIFICITY REASON

San Francisco, CA San Francisco CA Just Right Perfect match
New York, NY Other Result is NOT very similar to MAIN THING.
(Entity Group: California Other Place Entity Group does NOT allow
Place) parent/child relationships between
different kinds of Entity Types. MAIN
THING’s most specific Entity Type is City.
Result’s most specific Entity Type is State.
Entity Types are different.
→ Not a close relationship.
Main St., San Other - MAIN THING: Entity Type (City) → Place
Francisco CA Entity Group
- Result: Entity Type (Street name) →
Address Entity Group.
- Different Entity Groups
→ Not a close relationship.
San Francisco City Other - MAIN THING: Entity Type (City) → Place
Hall, 1 Dr Carlton B Entity Group
Goodlett Pl, San - Result: Entity Type (Business) → Other
Francisco, CA 94102, Entity Group. (“Business” includes any
United States government office / department.)
- Different Entity Groups
→ No parent/child relationships allowed
- N.B. City Hall represents the city
government, but not the city itself. Large
Place results should NOT have a street /
street number. It’s MUCH too specific.
Such a result is not a Place, thus it is in a
different Entity Group.

QUERY RESULT SPECIFICITY REASON

Hilton hotel San Hilton San Just Right Perfect match. The extra information is
Francisco Francisco Union non-conflicting.
Square, 333
(Entity Group: O'Farrell St, San
Other) Francisco, CA 94102
 The Lobby Bar, Slightly too Most specific parts of MAIN THING and
Hilton San Francisco specific result both belong to Other Entity Group,
Union Square, 333 which allows close parent/child
O'Farrell St, San relationships. The result is not an exact
Francisco, CA 94102 match to the MAIN THING, but it is a part
of it. The “Lobby Bar” (child) is a part of
the hotel (parent) and is owned by it.
→ Close parent/child relationship.
333 O'Farrell St, San Other - MAIN THING: Other Entity Group
Francisco, CA 94102 - Result: Address Entity Group.
- Different Entity Groups
→ Not a close relationship.
San Francisco CA Other - MAIN THING: Other Entity Group
- Result: Address Entity Group.
- Different Entity Groups
→ Not a close relationship.

QUERY RESULT SPECIFICITY REASON

hotel San Hilton San Just Right Result matches the query category
Francisco Francisco Union perfectly.
Square, 333
(Entity Group: O'Farrell St, San
Other) Francisco, CA 94102
The Lobby Bar, Slightly too - MAIN THING is a Business category
Hilton San Francisco specific query, which is in the Other Entity Group
Union Square, 333 that allows close parent/child
O'Farrell St, San relationships.
Francisco, CA 94102 - Result Entity Type is a Business, which is
also part of Other Entity Group.
- Result not an exact match to the MAIN
THING, but it is a part of it. The “Lobby
Bar” (child) is a part of the hotel (parent)
and is owned by it.
→ Close parent/child relationship.
333 O'Farrell St, San Other MAIN THING: Other Entity Group
Francisco, CA 94102 Result: Street Number → Address Entity
Group. Different Entity Groups
→ Not a close relationship.
San Francisco CA Other MAIN THING: Other Entity Group
Result: Street Number → Address Entity
Group. Different Entity Groups
→ Not a close relationship.

Other examples
In examples below, MAIN THING in query is outlined, and any match in result is also outlined. The most
specific Entity Type of each is in bold face.
QUERY RESULT SPECIFICITY REASON
london UK London, UK Just right Perfect match
london UK SoHo, London Other - MAIN THING: London. Entity Group: Place
UK - Result: Soho. Entity Group: Place
- No parent/child relationships allowed
→ Not a close parent/child relationship
London City Hall, Other - MAIN THING: London. Entity Group: Place
300 Dufferin Ave, - Result: London City Hall. Entity Group: Other
London, ON N6B (“Business” includes any government office /
1Z2 department.)
- No parent/child relationships allowed
- N.B. City Hall represents the city government,
but not the city itself. Large Place results should
NOT have a street / street number. It’s MUCH
too specific. Such a result is not a Place, thus it
is in a different Entity Group.
151 W 34th St, 151 W 34th St, Just right Perfect match.
NY New York, NY
10001
151 W 34th St, Macy’s, 151 W Just right “151 W 34th St” is a perfect match. Results are
NY 34th St, New allowed to include extra, non-conflicting
York, NY 10001 information (“Macy’s” is at the address and
does not conflict).
Unit #22, 151 151 W 34th St, Just right Ignore apartment # / suite # / unit # / floor # in
W 34th St, NY New York, NY query.
10001
151 W 34th St, Unit #22, 151 W Slightly too Ignore apartment # / suite # / unit # / floor # in
NY 34th St, New specific the query, but NOT in the result. “Unit # 22” is
York, NY 10001 at the requested address but it is for a specific
unit. Thus, result is too specific because it
excludes all other apartments at same address.
W 34th St, NY 151 W 34th St, Slightly too Address Entity Group allows close parent/child
New York, NY specific relationships with Street name as parent.
10001 → Close parent/child relationship
123 Main St, SF Main St, San Slightly too Result has correct street name, but is missing
CA Francisco CA general the street number. Address Entity Group allows
94105 close parent/child relationships with Street
name as parent.
→ Close parent/child relationship
123 Main St, SF San Francisco CA Other Result has a very little to do with the query.
CA Different Entity Groups (Address vs Place).
123 Main St, SF USA Other Result has a very little to do with the query.
CA Different Entity Groups (Address vs Place).
Apt #319, 123 123 Main St, San Just right Ignore apartment # / suite # / unit # / floor # in
Main St, SF CA Francisco CA the query, but not in the result. “123 Main St”
94105 is a perfect match.
Disney hotel Disneyland Just right Result has the full and proper name of the
anaheim Hotel, 1150 hotel plus the full address. This is extra, non-
Magic Way, conflicting information.
Anaheim, CA
92802
Edible Island Edible Island Just right Result has the full and proper name of the
courtenay Whole Foods business plus the full address. This is extra,
Market, 477 6th non-conflicting information.
St, Courtenay, BC
V9N 6V4, Canada
Edible Island Whole Foods Just right Result has valid, alternative name of the
courtenay Market, 477 6th business plus the full address.
St, Courtenay, BC
V9N 6V4, Canada
restaurants in The Harvest Just right As expected, this restaurant is more specific
Seattle Vine, 2701 E than the CATEGORY word “restaurant”, but it
Madison St, matches the search intent of the CATEGORY.
Seattle, WA
98112
hotel in seattle Parking service, Slightly too Result is more specific than the CATEGORY
Hyatt Hotel, specific (“hotel”), but it has a close parent/child
Seattle relationship (its “parent” is the Hyatt Hotel).
The service is provided by the hotel itself.
U Washington, University of Just right Perfect match (“U” is abbreviation of
Seattle, WA Washington, “University”).
Seattle WA
98105
U Washington, Dept. of Slightly too Other Entity Group allows parent/child
Seattle, WA Anthropology, specific relationships. Compare the full relevant portion
University of of result. It is slightly too specific (department
Washington, is one small part of the whole).
Seattle WA
Anthropology Department of Just right Perfect match. Query has abbreviation, result
Dept, UW, Anthropology, has full name (either is OK).
Seattle, WA University of
Washington,
Seattle WA
Anthropology University of Slightly too Other Entity Group allows parent/child
Dept, UW, Washington, general relationships. Result is slightly less specific
Seattle, WA Seattle WA (more general) than MAIN THING. The
98105 department (child) has a close relationship with
the University of Washington (parent).
Anthropology Seattle WA Other No part of the result corresponds to MAIN
Dept, UW, 98105 THING in the query. Different Entity Groups
Seattle, WA (Other vs Place).
San Francisco California, USA Other Place Entity Group does NOT allow parent/child
CA relationships between different Entity Types.
94105 CA San Francisco CA Just right Perfect match. “San Francisco CA” is extra, non-
94105 conflicting info. Note: 94105 is most specific
part of query because it is smaller than the
state of California. It is also smaller than the
city of San Francisco in result.
94105 CA San Francisco CA Other Does not match MAIN THING. Wrong Post
94106 Code.
Surrey, BC City of Surrey, Other - Result is a specific “business” office of the City
13450 104 Ave, of Surrey government. It does not represent
Surrey, BC V3T the city itself. Large Place results should NOT
1V8 have a street / street number. It’s MUCH too
specific. Such a result is not a Place, thus it is in
a different Entity Group.
- No parent/child relationships allowed
between “Place” and “Other” Entity Groups.
Walmart Walmart Vision Slightly too Other Entity Group allows parent/child
& Glasses, 11400 specific relationships. Result is too specific, but the
Hwy 99, Everett, Walmart department (child) is part of, and is
WA 98204 owned by, the Walmart store (parent). Thus,
they have a close parent/child relationship.
Mercy Hospital Muckerman Slightly too Other Entity Group allows parent/child
Garage, Mercy specific relationships. Result is too specific, but the
Hospital, parking garage (child) is part of, and is owned
McAuley Dr, by, the hospital (parent). Thus, they have a
Creve Coeur, MO close parent/child relationship.
63141
Hyatt Seattle Hyatt Room Slightly too Other Entity Group allows parent/child
Service, 808 specific relationships. Result is too specific, but this
Howell Street, kitchen (child) is part of the Hyatt hotel
Seattle, WA (parent) and is owned by it. They have a close
98101 parent/child relationship.
Hyatt Seattle Andare Kitchen Slightly too Other Entity Group allows parent/child
808 Howell specific relationships. Result is too specific. This
Street, Seattle, restaurant is located inside the Hyatt hotel and
WA 98101 is one of the services provided by the hotel.
They have a close parent/child relationship.
Safeway 15000 Safeway Deli, Slightly too Result is too specific, but the deli is a
NE 24th 15000 NE 24th St, specific department of the MAIN THING. Same Entity
Redmond Redmond WA Group, and they have a close parent/child
98052 relationship.
Safeway 15000 Starbucks, 15000 Other This Starbucks is inside the Safeway, but it is
NE 24th NE 24th St, not a department of Starbucks. It is a different
Redmond Redmond WA business. Same Entity Group, but they do NOT
98052 have a close parent/child relationship.
NY Tourist Empire State Just right Perfect match for the search intent of the
Attractions Building, 20 W category.
34th St, New
York, NY 10001
Empire State Empire State Just right Perfect match.
Building NY, NY Building, 20 W
34th St, New
York, NY 10001
Empire State Empire State Just right The result might seem too specific, but it is
Building NY, NY Building Ticket actually exactly what many users are searching
Office, 20 W for.
34th St, New
York, NY 10001
Empire State Empire State Slightly too Other Entity Group allows parent/child
Building NY, NY Building Gift specific relationships. Result is too specific, but the gift
Shop, 20 W 34th shop (child) is part of, and is owned by, the
St, New York, NY building (parent). There is a close relationship.
Empire State LinkedIn, Empire Other Other Entity Group allows parent/child
Building NY, NY State Building, relationships. Result is too specific. The
26th Floor, 20 W LinkedIn office is located inside the building
34th St, New but has no other relationship with it. They do
York, NY 10001 NOT have a close relationship.
NY, NY New York, NY Just right The outlined part of the query is a city (“NY”, an
10001 abbreviation of “New York”). Compare only the
relevant portion of result (city of “New York”).
The result is a perfect match.
NY, NY Empire State Other Result has a very little to do with the query.
Building, 20 W Different Entity Groups (Place vs Other).
34th St, New
York, NY 10001
NY, NY New York City Other - No parent/child relationships allowed
Hall, City Hall between “Place” and “Other” Entity Groups.
Park, New York, - N.B. City Hall represents the city government,
NY 10007, USA but not the city itself. Large Place results should
NOT have a street / street number. Such a
result is not a Place, thus it is in a different
Entity Group.
Yosemite Yosemite Just right The result has the full and proper name.
National Park, “National Park” is extra, non-conflicting
California information.
Houses for sale 60 Willemar Ave, Just right The result is the address of a house that may
courtenay bc Courtenay, BC be for sale.
canada V9N 3L1
Apartments for Apt #319, 123 Just right The result is the address of an apartment that
rent Main St, San may be for rent.
Francisco CA
94105
Ronald Ronald Just right Perfectly matches the MAIN THING the user
McDonald McDonald was looking for.
House near House New York,
Brooklyn 405 E 73rd St,
New York, NY
10021
Ronald Peter Luger Other Other Entity Group allows parent/child
McDonald Steak House, relationships, but result has no relation to the
House near 178 Broadway, user’s search intent.
Brooklyn Brooklyn, NY
emp seattle Museum of Pop Just right EMP (Experience Music Project) is the former
Culture, 325 5th name of this museum. The result has the
Ave N, Seattle, current name.
WA 98109
varanasi india Benares India Just right Benares is a valid alternative name for the
famous, holy city of Varanasi.