Project Callisto Search Quality Review Module 2 - Version 3
Project Callisto Search Quality Review Module 2 - Version 3
Project Callisto Search Quality Review Module 2 - Version 3
Lowest Quality Pages are untrustworthy, deceptive, harmful to people or society, or have other
highly undesirable characteristics. The considerations for Lowest quality include quality of the
Main Content (MC), reputation, E-E-A-T, etc., but there are special checks you will need to
complete first.
Click on the following to learn more about the different types of lowest quality pages:
Harmful
Pages that encourage depict, incite or directly cause physical, mental emotional or
financial harm.
Misleading Information, Pages that misinform people in a way that could cause harm
Untrustworthy
Spammy
Intentionally created without proper Main Content (MC) or gibberish MC, and can be
hacked or defaced.
Defined groups based on various characteristics, for example: age, caste, disability,
Ethnicity, gender Identity and Expression, Nationality, Race, Religion, Sex/Gender
Untrustworthy Webpages
The "Lowest" rating should be used for pages or websites you strongly suspect are engaging in
deceptive or malicious practices.
Inadequate information
Inadequate information about the website or content creator for its purpose
Lowest E-E-A-T
Deceptive purpose
Characteristics of scams
Manipulative Website
Any webpage or website designed to manipulate people into actions that benefit the website or
other organization while causing harm to self, others, or Specified Groups
Any inaccuracies on an informational page which would cause users to lose trust in the webpage as
a reliable source of information, would be rated with a "Lowest" rating
Examples of Lowest Quality: Untrustworthy Pages
Lowest E-E-A-T
This page appears to be an informational article about the animal the praying
mantis.
Factual inaccuracies and odd statements exist in the section titled “Praying
Mantises Are Not Poisonous”: "In contrast to praying mollies, which are mostly
solitary creatures that avoid human interaction, praying mollusks are more closely
related to God. There are no poisonous snakes, and there are no venomous
turtles."
● Statements about animals and their relation to God are not what one would
expect on an informational page like this
It is unlikely a human author would make these odd statements and obvious
mistakes. It's likely this content was auto-generated with no human editing.
No matter how this content was created, the odd statements and factual
inaccuracies make this article untrustworthy and therefore Lowest E-E-A-T and
Lowest quality.
Pages that mislead people in ways that can cause harm to people and society.
There is an especially high standard for accuracy on clear YMYL topics or other topics
where inaccurate information can cause harm.
Inadequate Information and E-E-A-T
For pages that require a high level of trust, information about who created the content and
who is responsible for the content is critical
YMYL pages or websites that handle sensitive data with absolutely no information about
the website or content creator should be rated Lowest
If the E-E-A-T of a page is low enough, people cannot or should not use the Main Content
(MC) of the page. If a page on YMYL topics is highly inexpert, it should be considered
Untrustworthy and rated Lowest
These pages or websites superficially appear to have one purpose, but in fact exist for
a different reason
All pages with a deceptive purpose and/or deceptive MC should be rated Lowest because
pages that engage in deception are Untrustworthy
Pages are untrustworthy if the Main Content (MC) is deliberately obstructed or obscured due to
Ads, Supplementary Content (SC), interstitial pages, download links or other content that is
beneficial to the website owner but not necessarily the website visitor. Attempts to manipulate or
coerce users away from the MC is evidence of untrustworthiness.
Important: Remember that many websites need monetization to share content with
users. The presence of Ads alone is not enough for Lowest.
Suspected Malicious Behavior
Use the Lowest rating if you strongly suspect a page is malicious or harmful even without having
proof. Any of the following should be considered untrustworthy:
Personal Information: Pages that ask for personal information without a legitimate reason
Phishing Pages: Pages that appear to “phish” for passwords to Facebook, Gmail, or other popular
online services
Malware Pages: Pages with links that you strongly suspect are malware downloads
Spammy Webpages
When we show up to the present moment with all of our senses, we invite the world to fill us with
joy. The pains of the past are behind us. The future has yet to unfold. But the now is full of beauty
simply waiting for our attention.
All pages should be created with sufficient Main Content (MC) quality so that the page can achieve
its purpose. Lowest rating for pages lacking clear purpose or MC.
Websites can become hacked, defaced, or filled with a large amount of distracting and unhelpful
content from bad actors. These pages should be rated Lowest because they fail to achieve their
original purpose.
Add Pages and websites made up of auto-generated content with no editing or manual curation,
and no original content or value added for users, should be rated Lowest.
Ads and SC are expected to be visible but shouldn’t make it difficult to use Main Content
A single pop-over Ad or interstitial page with a clear and easy-to-use close button is okay
Not okay: difficult to close ads, sexually suggestive images, shocking images
A lack of reputation information for ordinary people and lesser-known content creators is
expected and okay
Not enough information about creator
Pages that offer payment functionality or process other types of financial transactions
should receive a Low rating if there is an unsatisfying amount of customer service
information or contact information
Pages on YMYL topics and other pages that require a high level of user trust should receive
a Low rating if there is an unsatisfying amount of information about who is responsible for
the website or who created the content
For forums, social media pages, and other websites where people commonly exchange
opinions, a username is an acceptable identifier
Images from other sources (so you don’t have to create own), a lot of large pictures
Low Quality Main Content Examples
List of best vacuum cleaners with images and reviews from other sources with no signs of
effort or original content
Top 10 world destinations using pictures and existing lists of other people
Page with title “how many cm are in m” and there is a lot of filler information. It is difficult
to find an answer
Title "Pink Elephant: Part 2 coming soon!" with only speculations and rumors
Title "Is the World about to End? Mysterious Sightings of Sea Serpents Prompt Panic!" with
info about dead fish
There are low quality pages which do not meet the standards for the rating lowest. They have
similar though milder undesirable characteristics:
Info about energy sources/nuclear power can significantly impact global industries. YMYL topic.
Low E-E-A-T
There is no evidence that the author has medical expertise. Because this article gives advice on a
YMYL medical topic, lacking expertise is a reason for a Low rating.
Medium pages have a beneficial purpose and achieve their purpose. There is nothing wrong with
Medium quality pages. Expect to encounter many Medium quality pages in PQ rating tasks.
Identifying Page Content for Medium Quality
Medium quality pages have a beneficial or non-harmful
The purpose of the page
purpose.
The title of the page Medium quality pages have titles that summarize the page.
The following are the areas of the page we would take into consideration for medium quality:
Next, we want to assess the page to determine if the following "Medium" criteria applies:
Trustworthiness of the page: E-E-A-T Adequate level of E-E-A-T for the purpose of the page.
Types of Medium Quality Pages
Nothing wrong, but nothing special - All of the Medium Page Quality considerations and
criteria apply
Mixed, but with some redeeming qualities - The page or website has some signs of High
quality (E-E-A-T, quality of the MC, positive reputation), but also has one sign or mild signs
of Low quality.
High quality pages serve a beneficial purpose and achieve that purpose well.
The following are the areas of the page we would take into consideration for high quality:
The purpose of the page High quality pages have a beneficial purpose.
A page on any topic or any type of website may qualify for High.
The topic of the page, the type of website, and the extent to
which YMYL standards apply
A page on any topic or any type of website may qualify for High.
The title of the page High quality pages have titles that summarize the page.
Next, we want to assess the page to determine if the following "High" criteria applies:
Reputation of the website and content creator Positive reputation of the website for the topic of the page.
Trustworthiness of the page: E-E-A-T High level of E-E-A-T for the purpose of the page.
High Quality Main Content should satisfy people when they visit the page. For informational pages,
High quality MC must be accurate and consistent with well-established expert consensus when
such consensus exists.
If you aren't sure whether the content is high quality, try finding other pages on the same topic to
help calibrate your assessment: "typical" and "average" pages on a topic generally have Medium
(not High) quality Main Content (MC).
Positive reputation of the website can be a reason for a High rating if the website is
responsible for the MC
o If the website is not responsible for the MC (e.g., posts on social media), positive
reputation of creator(s) can be a reason for the High rating
•Experience is valuable for almost any topic. First-hand experience can make a page very high
quality. Social media posts and forum discussions are often High quality when they involve people
sharing their experience.
•Expertise is required for satisfying content on a variety of topics. Think about the topic of the
page and what expertise is needed to create satisfying, trustworthy content.
•Authoritative pages of all types can be found. Examples of these types of sites are government tax
websites, local businesses, and organizations. These pages are a "go-to" for local information.
When looking at a page or website, consider whether it is considered a go-to, authoritative source
for the type of information it is displaying.
•Trust is especially important for High quality pages that involve processing financial transactions
or cover YMYL topics. Even if the page topic is not YMYL, trust is still required even if it is not of
high level.
Highest Quality Pages
Section 8: Highest Quality Pages
Highest quality pages serve a beneficial purpose and achieve that purpose very well. The
distinction between High and Highest is based on the quality of MC, the reputation of the website
and content creator, and/or E-E-A-T.
The following are the areas of the page we would take into consideration for highest quality:
The purpose of the page Highest quality pages have a beneficial purpose.
Next, we want to assess the page to determine if the following "Highest" criteria applies:
Trustworthiness of the page: E-E-A-T Very high level of E-E-A-T for the purpose of the page.
Very High Quality Main Content (MC)
Very high quality MC should be highly satisfying for people visiting the page. The standards for
Highest quality MC may be very different depending on the purpose, topic, and type of website.
Very high quality MC shows evidence of a high level of effort, originality, talent, or skill. The Highest
rating may be justified for pages with very satisfying MC created with a very high level of effort,
originality, talent, or skill.
For news
Original reporting that provides information that would not otherwise have been known had the
article not revealed it. Accurate, original, in-depth, and investigative reporting requires a high level
of skill/talent and effort. Very high quality news content will include a description of primary
sources and other original reporting referenced during the content creation process. Very high
quality news content must be accurate and should meet professional journalistic standards.
Unique and original content created by highly skilled and talented artists or content creators. Such
artistic content requires a high level of skill/talent and effort. If the artistic content is related to a
YMYL topic (e.g., artistic content with the purpose of informing or swaying opinion about YMYL
topics), YMYL standards should apply.
Original, accurate, comprehensive, clearly communicated, and should reflect expert consensus as
appropriate. Expectations for different types of information may vary. For example, scientific
papers have a different set of expectations than a social media post sharing information about a
hobby such as stamp collecting. However, all types of very high quality informational content share
common characteristics of accuracy and clarity of communication, in addition to meeting
standards appropriate to the topic or field.
Reputation research is important when giving Highest ratings. For YMYL topics, very positive
reputation is often based on recommendations from known experts or professional societies
appropriate to the topic of the page. For non-YMYL topics, reputation information may be less
formal. Popularity, user engagement, and user reviews can be considered evidence of reputation
for non-YMYL websites.
Note: Many smaller websites and ordinary people have little reputation information. A page can
still receive a highest rating without reputation information.
Very High Level of E-E-A-T
Very high E-E-A-T is a distinguishing factor for Highest quality pages. Think about what E-E-A-T
means for the topic of the page. How important is first-hand experience? Who are the experts?
What makes a source highly authoritative for the topic? What makes a website or content creator
trustworthy for the topic? Standards for very high E-E-A-T will differ depending on the topic of the
page.
•A website or content creator who is the uniquely authoritative, go-to source for a topic has very
high E-E-A-T.
•A content creator with a wealth of experience may be considered to have very high E-E-A-T for
topics where experience is the primary factor in trust
•A very high level of expertise can justify a very high E-E-A-T assessment.
•Very high E-E-A-T websites and content creators are the most trusted sources on the internet for
a particular topic.
High and Highest quality ratings only for encyclopedias with very good reputations for
accuracy and expertise
o Less extensive main content and external references can be rated Medium
o Articles with little main content and factual inaccuracy is a sufficient reason for a
Low or even Lowest rating
In PQ rating tasks, you may encounter pages with error messages or other types of "broken" pages.
Please think about whether the page offers help for users or did the website owner make an effort
to ensure that users visiting the page have a good experience and get help finding what they are
looking for?
High rating may be used for the rare error message page that involves a high level of effort
and original content
Ratings for Forums and Q&A Pages
Ratings for forum and Q&A pages can be challenging, generally:
Main Content: The Main Content includes the question, the answers/responses, and the resulting
discussions
Rating: Rate from the point of view of a user who visits the page , rather than a participant
involved in the discussion
Users: Users who post answers/responses or comments are often identified only by a username or
alias. A page can be High or Highest quality with just usernames or aliases depending on other
criteria
E-E-A-T: The E-E-A-T of a discussion among users can often be judged by the posts or comments
themselves
For some topics, Experience is the most important dimension of Trust. For other topics,
assessing Expertise through the posts may be important. In some cases, the posters
themselves will highlight either their own Experience or Expertise, or other people will
comment on it
Pages on YMYL topics require more attention to Trust and more care in the assessment of
E-E-A-T
Have satisfying conversations that involve many participants, some of whom put a great deal of
effort into their posts and have a wealth of Experience and/or Expertise on the topic. Such
conversations can be very satisfying because of the depth of discussion, the unique insights, or the
sharing of experiences that many would not have access to in their real-world community
Often lack effort (few responses, surface-level rather than in-depth discussion), lack Experience or
Expertise, contain mild inaccuracies, or show a significant lack of respect or decorum among the
participants that might deter others from joining the discussion
May contain information or advice that is harmfully misleading, contradicts well-established expert
consensus, encourages harm towards self or other individuals/participants, etc
Page Quality Rating
Section 10: Page Quality Tasks
At first glance, rating Page Quality appears to be very involved and difficult as there are several
items to consider. However once you learn what to review, rating moves faster. It is VERY
important to give your best rating and move on. Though we want to ensure quality by choosing
the correct rating, we do not want to struggle with spending too much time on the task.
If you are having trouble deciding between two ratings, use the lower rating. If you are torn
between three ratings, choose the one in the middle.
Do not consider the country or location of the page or website for PQ rating.
The guidelines are specific to "regular" webpages. Occasionally, you may be asked to rate
a landing page that is not a webpage. In this case, please use your judgment.
Page Quality Rating Guideline does not completely cover every aspect of page quality. If
you find pages that you truly believe to be High or Low quality, please rate them as such,
even if the reason is based on something not covered in these guidelines. As always, use
your judgment.
2. A "PQ grid" to record your observations about PQ characteristics of the landing page
Some results to the initial questions will end the task early. If the page is Porn, Foreign Language,
Did Not Load, or consists of restricted or inaccessible MC (e.g., subscription is required to view
MC), you will not fill in the PQ grid or assign an overall rating.
The PQ grid is designed to be your "note pad." It allows you to record your observations about
the landing page and the website it belongs to.
Foreign Language should not be used when the language on the landing page is in the
task language, a language that is commonly used by a significant percentage of the
population in the task location, or English.
Did Not Load should be used for pages where there is absolutely no content on the page
created by the website. There is no MC, SC, or Ads on the page.
Section 11: Page Quality Rating FAQs
With practice, the amount of time needed for accurate PQ ratings will decrease. The steps are
important and are designed to help you assess many different aspects of PQ. Pages that initially
look Low quality may turn out to be Medium or High quality with careful inspection.
Are we just giving high quality ratings to pages that "look" good?
No! The goal is to do the exact opposite. These steps are designed to help you to analyze the page
without using a superficial “does it look good?” approach.
Is sharing a personal experience or opinion a beneficial purpose? What if the personal opinion is
upsetting or offensive?
Can life experience justify a Highest quality rating for a page on a YMYL topic?
Factual information and advice on YMYL topics should come from experts.
You talked about expertise when rating MC. Does expertise matter for all topics? Aren't there
some topics for which there are no experts?
Remember that we are not just talking about formal expertise. Informal expertise is equally
important, and for some topics may be a more common type of E-E-A-T
When I think about E-E-A-T, it seems like expertise and experience overlap a lot. What's the
difference?
● Expertise often involves objective, testable knowledge or skills, for example: can you calculate
the load bearing weight of a bridge?
● Experience can be more subjective and is often shared through personal narration, for example:
how does it feel to experience love for another person?
For PQ rating, it's not important to distinguish between expertise / experience. Instead, focus on
what kind of content is trustworthy and satisfying for the purpose of the page.
Aren't some types of pages always Low quality, such as celebrity gossip?
There are both High and Low quality celebrity gossip pages. A celebrity gossip page is High quality
if it has accurate, interesting information, coming from reliable source.
I've never seen a High quality page of this type. If there are no high quality pages of this type,
why are we giving existing pages a Low quality rating?
For some topics or types of pages, there may not be many (or any!) High quality pages now, but
there may be in the future. We need a uniform set of standards that apply to all pages, even for
pages that have not yet been created.
Some of these criteria seem unfair. For example, some art pages do not have a purpose. Are
these pages Low quality?
Art pages do have a purpose: artistic expression. Artistic expression, humor, entertainment, sharing
photos and videos, etc. are all valid and beneficial page purposes.
Sometimes clicking on the task URL will bring up an interstitial page. You can ignore this page in
your rating criteria if you can easily get to the MC. However, if the interstitial makes it extremely
hard (or impossible) to get to the MC and evaluate how well the page achieves its purpose, that
should factor into your PQ rating.