(19)

*EP003407221B1*
(11) EP 3 407 221 B1
(12) EUROPEAN PATENT SPECIFICATION

(45) Date of publication and mention (51) International Patent Classification (IPC):
of the grant of the patent: G06F 3/0482 (2013.01) G06F 16/00 (2019.01)
23.08.2023 Bulletin 2023/34 G06F 16/951 (2019.01)

(21) Application number: 18174459.0 (52) Cooperative Patent Classification (CPC):

G06F 3/0482; G06F 16/00; G06F 16/951
(22) Date of filing: 26.05.2018

(54) METHODS AND DEVICES FOR SEARCHING AND DISPLAYING INFORMATION ON A TERMINAL
VERFAHREN UND VORRICHTUNGEN ZUM SUCHEN UND ANZEIGEN VON INFORMATIONEN
AUF EINEM ENDGERÄT
PROCÉDÉS ET DISPOSITIFS POUR RECHERCHER ET AFFICHER DES INFORMATIONS SUR
UN TERMINAL

(84) Designated Contracting States: • LIU, Fang

AL AT BE BG CH CY CZ DE DK EE ES FI FR GB Haidian District, Beijing 100085 (CN)
GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO
PL PT RO RS SE SI SK SM TR (74) Representative: Hughes, Andrea Michelle et al
Dehns Germany
(30) Priority: 27.05.2017 CN 201710392518 Theresienstraße 6-8
80333 München (DE)
(43) Date of publication of application:
28.11.2018 Bulletin 2018/48 (56) References cited:
CN-A- 106 293 480 US-B1- 9 582 482
(73) Proprietor: Beijing Xiaomi Mobile Software Co.,
Ltd. • David Deutsch: "Responsive Demo - Owl
Beijing 100085 (CN) Carousel", , 26 May 2014 (2014-05-26),
XP055514432, Retrieved from the Internet:
(72) Inventors: URL:https://owlcarousel2.github.io/OwlCaro
• LU, Yizhan usel2/demos/responsive.html [retrieved on
Haidian District, Beijing 100085 (CN) 2018-10-11]
• LONG, Hai
Haidian District, Beijing 100085 (CN)
EP 3 407 221 B1

Note: Within nine months of the publication of the mention of the grant of the European patent in the European Patent
Bulletin, any person may give notice to the European Patent Office of opposition to that patent, in accordance with the
Implementing Regulations. Notice of opposition shall not be deemed to have been filed until the opposition fee has been
paid. (Art. 99(1) European Patent Convention).

Processed by Luminess, 75001 PARIS (FR)

 1 EP 3 407 221 B1 2

Description BRIEF DESCRIPTION OF THE DRAWINGS

TECHNICAL FIELD [0010] Hereinafter the drawing to be referred in the de-

scription of the embodiments will be briefly introduced.
[0001] The present disclosure relates to the field of hu- 5 Obviously, the drawings explained in the description as
man-machine interaction, particularly to methods and de- below merely illustrate several embodiments of the
vices for searching and displaying information on a ter- present disclosure. For those skilled in the art, other ones
minal. may be achieved based on these drawings without pay-
ing any inventive labor.
BACKGROUND 10
Fig. 1 is block diagram illustrating an information dis-
[0002] A user who utilizes an application program on play system provided by an aspect of the present
a terminal usually will search in an interested text dis- disclosure;
played on a user interface of the application program. Fig. 2 is a flow chart illustrating an information display
The terminal may be a mobile phone, a tablet PC, an e- 15 method provided by an aspect of the present disclo-
book reader, and the like. sure;
[0003] A typical searching process may include steps Fig. 3A shows schematic diagrams illustrating user
of: firstly, the user presses and holds on an interested interfaces when implementing the information dis-
text when catching sight of the text on the user interface play method provided by an aspect of the present
of the application program; secondly, the terminal pops 20 disclosure;
out a "copy" toolbar, upon receiving a press-and-hold Fig. 3B shows schematic diagrams illustrating user
signal, for the user to click on a "copy" button in the "copy" interfaces when implementing the information dis-
toolbar; thirdly, the user quits the application program play method provided by an aspect of the present
and returns to a desktop window; fourthly, the user finds disclosure;
and opens a browser in the desktop window; fifthly, the 25 Fig. 3C shows schematic diagrams illustrating user
user opens a searching website in the browser; sixthly, interfaces when implementing the information dis-
the user pastes the interested text into a searching box play method provided by an aspect of the present
of the browser for searching, so that the browser displays disclosure;
searching result information related to the interested text. Fig. 4 is a flow chart illustrating an information display
[0004] As shown above, the user has to perform a total 30 method provided by another aspect of the present
of six steps to complete a single searching process, which disclosure;
results in an extremely tedious operation flow for the user, Fig. 5A shows schematic diagrams illustrating user
with relatively poor efficiency of human-machine interac- interfaces when implementing an information display
tion. method provided by an aspect of the present disclo-
[0005] US9582482 B1 discloses capturing a terminal 35 sure;
screen upon detection of a user input, analysing the cap- Fig. 5B shows schematic diagrams illustrating user
tured screen to extract content information and key- interfaces when implementing an information display
words, and performing a search based on those informa- method provided by an aspect of the present disclo-
tion to find related content. The search result is then dis- sure;
played in a superposed manner over the current screen. 40 Fig. 6 is a block diagram illustrating a terminal pro-
The user can interact with the search result, vided by an aspect of the present disclosure;
Fig. 7 is a flow chart illustrating an information display
SUMMARY method provided by another aspect of the present
disclosure;
[0006] In order to solve problems in related technolo- 45 Fig. 8 is a flow chart illustrating an information display
gies that an operation flow of user is extremely tedious method provided by another aspect of the present
and that an efficiency of human-machine interaction is disclosure;
relatively poor, the present disclosure provides an infor- Fig. 9 is a block diagram illustrating a server provided
mation display method, an information display device, by an aspect of the present disclosure;
and a terminal. Technical solutions are described as be- 50 Fig. 10 is a flow chart illustrating an information dis-
low provide more efficient searching methods and better play method provided by another aspect of the
displaying methods. present disclosure;
[0007] According to a first aspect of the present disclo- Fig. 11 is a flow chart illustrating an information dis-
sure, a terminal is provided as claimed in claim 1. play method provided by another aspect of the
[0008] According to an aspect of the present disclo- 55 present disclosure;
sure, a method is provided as claimed in claim 13. Fig. 12 is a flow chart illustrating an information dis-
[0009] According to an aspect a computer-readable play method provided by another aspect of the
storage medium is provided as claimed in claim 14. present disclosure;

2
 3 EP 3 407 221 B1 4

Fig. 13 is a flow chart illustrating an information dis- work, a fiber optical network and the like. The wireless
play method provided by another aspect of the network may be a mobile communication network or a
present disclosure; wireless fidelity (WiFi) network.
Fig. 14 is a schematic diagram illustrating an inter- [0015] The server 140 may be a server for providing
face when implementing an information display 5 searching service and searching result information; and
method provided by an aspect of the present disclo- may also be an image processing server for providing
sure; image processing service at the same time. Optionally,
Fig. 15 is a flow chart illustrating an information dis- the server 140 includes a searching server 142 and a
play method provided by another aspect of the webpage server 144. The searching server 142 is pro-
present disclosure; 10 vided by a manufacturer corresponding to the operating
Fig. 16A shows schematic diagrams illustrating user system and used for providing searching service; the
interfaces when implementing an information display webpage server 144 is used for providing a webpage
method provided by another aspect of the present corresponding to the searching result information and
disclosure; may be provided by a manufacturer of a third-party ap-
Fig. 16B shows schematic diagrams illustrating user 15 plication program. The term "third-party" in the present
interfaces when implementing an information display disclosure refers to yet another party except the user and
method provided by another aspect of the present the operating system, without excluding the possibility
disclosure; that the third party and the provider of the operating sys-
Fig. 17 is a schematic diagram illustrating an inter- tem actually are a same organization. In case of multiple
face when implementing an information display 20 or multiple types of third-party application programs, the
method provided by another aspect of the present webpage server 144 may also be arranged in plural
disclosure; forms.
Fig. 18 is a flow chart illustrating an information dis- [0016] Fig. 2 is a flow chart illustrating an information
play method provided by another aspect of the display method provided by an aspect of the present dis-
present disclosure; 25 closure. Description will be given with reference to the
Fig. 19 is a schematic diagram illustrating an inter- information display method applied in a terminal including
face when implementing an information display an operating system and at least one application pro-
method provided by another aspect of the present gram, by way of example. The method includes steps as
disclosure; and below.
Fig. 20 is a block diagram illustrating an information 30 [0017] In step S201: displaying a user interface of the
display device provided by another aspect of the application program, the user interface includes at least
present disclosure. one interface element.
[0018] The application program installed in the termi-
DETAILED DESCRIPTION nal includes a native application program provided by the
35 operating system and/or a third-party application pro-
[0011] Hereinafter the embodiments of the present dis- gram installed by the user. In the present disclosure, an
closure will be described in further details in conjunction application program may refer to any application program
with the drawings, in order to make objectives, technical installed in the operating system or an application pro-
solutions and advantageous of the present disclosure gram belonging to a preset list.
more apparent. 40 [0019] An application program usually includes multi-
[0012] Fig. 1 is a block diagram illustrating an informa- ple user interfaces. The user interface of the application
tion display system provided by an aspect of the present program running in the foreground of the terminal will be
disclosure. The system includes a terminal 120 and a displayed on the display screen of the terminal preferen-
server 140. The terminal 120 and the server 140 are con- tially.
nected through a communication network 160. 45 [0020] The user interface includes at least one inter-
[0013] The terminal 120 may be a smartphone, a tablet face element thereon, and every interface element is of
PC, an e-book reader, a laptop PC, a desktop PC, and the portions constituting the user interface. The interface
the like. Optionally, the terminal 120 is provided with a element includes, but not limited to, at least one of a text,
touch screen, which is a common touch screen, or a sus- a picture, an audio and a video.
pension-touch screen, or a force-touch screen. The ter- 50 [0021] In step S202: receiving a first selecting opera-
minal 120 is installed with an operating system and at tion triggered on the user interface.
least one application program. The operating system [0022] The first selecting operation includes, but not
may be an Android operating system or an Apple (IOS) limited to, at least one of a press-and-hold operation, a
operating system. force-touch operation, a multi-click operation, a suspen-
[0014] The terminal 120 and the server 140 are con- 55 sion-touch operation and a double-finger touch opera-
nected through a communication network which may be tion.
a wired network or a wireless network. The wired network [0023] The press-and-hold operation is an operation
may be a metropolitan area network, a local area net- of pressing against a touch screen for a duration time

3
 5 EP 3 407 221 B1 6

longer than a first duration time. The force-touch opera- auxiliary screen is located.
tion is an operation of exerting a pressure larger than a [0031] Illustratively, when the local region is a bottom
threshold pressure. The multi-click operation is an oper- region of the display screen, it may reduce the coverage
ation of clicking for a preset amount of times within a of the local region relative to the user interface as far as
preset time interval. The suspension-touch operation is 5 possible, especially the coverage of the local region rel-
an operation of suspending above a touch screen for a ative to the target interface element. At the same time,
duration time longer than a second duration time. The compared to the solution in which the local region is a
double-finger touch operation is an operation of clicking top region, displaying the searching result information on
on the touch screen by using two fingers of the user at the bottom region may also prevent the user from being
the same time. 10 confused by the searching result information and a noti-
[0024] In step S203: selecting a target interface ele- fication normally displayed on the top region.
ment from the at least one interface element according [0032] When the local region is an adjacent region of
to the first selecting operation. the target interface element, it’s convenient for the user
[0025] The target interface element is one selected to read the searching result information which is matched
from the at least one interface element (without excluding 15 with the target interface element, without the need of fre-
the possibility of two or more target interface elements). quently switching between visual focuses of the user.
For example, the target interface element is a text and/or The adjacent region of the target interface element may
a picture. The target interface element is provided with be displayed directly next to the target interface element.
content information. When the target interface element [0033] When the local region is an adjacent region of
is a text, the content information is a character; when the 20 a position of the selecting operation, it’s convenient for
target interface element is a picture, the content informa- the user to take a next operation to the searching result
tion is image data. information; for example, after the selecting operation, it
[0026] In step S204, displaying searching result infor- may be possible to switch between different types of
mation in a superposed manner on a local region of the searching result information in a sliding manner, nearby,
user interface by the operating system, the searching 25 through a sliding operation. The adjacent region of the
result information is corresponding to the target interface position of the selecting operation may be displayed di-
element. For example, the searching result information rectly next to the position of the selecting operation.
may include search results using the content information [0034] Fig. 3A illustrates an example that the local re-
as input to a search engine. The searching result infor- gion is a bottom region. In Fig. 3A, a target interface el-
mation may include local results from the local storage 30 ement 22 that the user is interested in is shown in the
of the terminal, or from a local area network. Alternatively user interface 21. The target interface element 22 may
or additionally, the searching result information may in- be a text or a picture. When the user presses and holds
clude results from the a public network such as the In- on the target interface element 22, the terminal displays
ternet. a plurality of pieces of searching result information 23 on
[0027] The terminal displays the searching result infor- 35 the bottom region which is an adjacent region of a bottom
mation in a superposed manner on a local region of the edge of the display region. The searching result informa-
user interface by the operating system. The searching tion 23 may be provided in plural pieces, thus the terminal
result information is provided in an amount of n, in which may only display a part of the searching result information
at least two pieces of searching result information are 23 due to size restriction of the display region; for exam-
different from each other, and n is a positive integer. Op- 40 ple, the terminal may only display searching result infor-
tionally, n≥2. mation 1 and searching result information 2.
[0028] For example, the searching result information [0035] Fig. 3B illustrates an example that the local re-
may include at least one type of person name information, gion is a left border region. As shown in Fig. 3B, a target
site name information, WIKI information, navigation in- interface element 22 that the user is interested in is ex-
formation, journey information, user comment informa- 45 isted in the user interface 21. The target interface element
tion, shopping information and website information. 22 is a text or a picture. When the user presses and holds
[0029] Optionally, the searching result information is on the target interface element 22, the terminal displays
originated from at least two different websites. n pieces of searching result information 23 on the left
[0030] In one or more embodiments, the local region border region which is an adjacent region of a left edge
is any one of a bottom region, a top region, a left border 50 of the display region. The searching result information
region, a right border region, an adjacent region of the 23 may be provided in plural pieces, thus the terminal
target interface element and an adjacent region of an may only display a part of the searching result information
operating coordinate of the selecting operation. The local 23 due to size restriction of the display region; for exam-
region is a part of an entire display region of the display ple, the terminal may only display searching result infor-
screen, generally a portion of the entire display region 55 mation 1, searching result information 2 and searching
not covering the target interface element. Optionally, if result information 3.
the display screen includes a main screen and an auxil- [0036] In case that the local region is an adjacent re-
iary screen, then the local region is a region where the gion of the target interface element, by way of example,

4
 7 EP 3 407 221 B1 8

as it can be seen in conjunction with Fig. 3C, a target television work; a subscribing entrance of the film and
interface element 22 that the user is interested in is ex- television work in a local application program or on a
isted in the user interface 21. The target interface element network service; a sharing entrance of the film and tele-
22 is a text or a picture. When the user presses and holds vision work in the local application program or the network
on the target interface element 22, the terminal displays 5 service; a favorites entrance of the film and television
n pieces of searching result information 23 on the adja- work in the local application program; and a ticket-book-
cent region of the target interface element, which is an ing entrance of the film and television work in the network
adjacent region of an edge of the target interface element. service.
The searching result information 23 may be provided in [0042] If the content information of the target interface
plural pieces, thus the terminal may only display a part 10 element includes a title of literature work, then the search-
of the searching result information 23 due to size restric- ing result information includes, but not limited to, at least
tion of the display region; for example, the terminal may one of: introduction information of the literature work; rat-
only display searching result information 1 and searching ing information of the literature work; author information
result information 2. of the literature work; a subscribing entrance of the liter-
[0037] As mentioned above, the information display 15 ature work in the local application program; a sharing
method provided by the present disclosure is capable of entrance of the literature work in the local application
directly displaying n pieces of searching result informa- program; a favorites entrance of the literature work in the
tion in a superposed manner on a local region of a user local application program; a reading experience entrance
interface by an operating system, upon the terminal re- of the literature work in the network service; and a pur-
ceiving a selecting operation. In this way, from the point 20 chasing entrance of the literature work in the network
of view of the user, the searching result information cor- service.
responding to the target interface element can be ob- [0043] If the content information of the target interface
tained simply by a small number of operations, so that element includes information of a person, then the
the total six searching steps required in related technol- searching result information includes, but not limited to,
ogies may be simplified, which reduces the operation 25 at least one of: WIKI information of the person; a social
times during searching and improves the efficiency of network service homepage of the person in the network
human-machine interaction. service; latest news and trends of the person in the net-
[0038] Moreover, the n pieces of searching result in- work service; a dynamic subscribing entrance of the per-
formation are added by the operating system, thus the son in the network service; a sharing entrance of the per-
information display method provided by the present dis- 30 son in the local application program or the network serv-
closure always can execute the application program no ice; and a favorites entrance of the person in the local
matter it’s a native application program provided by the application program or the network service.
operating system or a third-party application program in- [0044] If the content information of the target interface
stalled by the user; in other words, the user is provided element includes information of a site, then the searching
with a capacity of globally searching within the terminal, 35 result information includes, but not limited to, at least one
which improves the application range of the present in- of: introduction information of the site; navigation infor-
formation display method. mation of the site in the local application program or the
[0039] Optionally, the above-mentioned n pieces of network service; map preview information of the site in
searching result information are obtained through the local application program or the network service; a
searching in the data of at least two different websites, 40 taxi-calling entrance with the site as a destination; a shar-
so that the preferred contents among different websites ing entrance of the site in the local application program
may be displayed in series on a local region of the appli- or the network service; and a favorites entrance of the
cation program by the operating system without the need site in the local application program or the network serv-
of switching and transiting between multiple application ice.
programs by the user constantly. 45 [0045] If the content information of the target interface
[0040] It should be explained that the information dis- element includes information of a merchant, then the
play method provided by the present disclosure may be searching result information includes, but not limited to,
applied in searching services of the user interface in any at least one of: introduction information of the merchant;
type of application programs. The searching result infor- a social network service homepage of the merchant in
mation includes at least one of text introduction informa- 50 the network service; latest news and trends of the mer-
tion, an entrance of local application program and an en- chant in the network service; a preferential subscribing
trance of network service. entrance of the merchant in the network service; contact
[0041] Illustratively, if the content information of the tar- information of the merchant; navigation information of
get interface element includes a title of film and television the merchant in the local application program or the net-
work, then the searching result information includes, but 55 work service; map preview information of the merchant
not limited to, at least one of: introduction information of in the local application program or the network service;
the film and television work; rating information of the film a taxi-calling entrance with the merchant as a destination;
and television work; author information of the film and a sharing entrance of the merchant in the local application

5
 9 EP 3 407 221 B1 10

program or the network service; and a favorites entrance tion triggered by the user through a touch screen. The
of the merchant in the local application program or the press-and-hold operation is used for selecting the target
network service. interface element on the user interface.
[0046] If the content information of the target interface [0054] Optionally, the press-and-hold operation may
element includes information of a commodity, then the 5 generate a touching event in the operating system. The
searching result information includes, but not limited to, touching event includes an operating coordinate of the
at least one of: introduction information of the commodity; press-and-hold operation, and the operating coordinate
a purchasing entrance of the commodity in the network is used for indicating a position of the selecting operation
service; price information of the commodity in at least on the display screen. A coordinate system where the
one shopping channel; a preferential subscribing en- 10 operating coordinate is located has a horizontal axis par-
trance of the commodity in the network service; a dis- allel to a horizontal edge of the display screen, and a
count reminding entrance of the commodity in the net- vertical axis parallel to a vertical edge of the display
work service; a history price trend checking entrance of screen.
the commodity in the local application program or the [0055] Optionally, the target interface element is one
network service; a sharing entrance of the commodity in 15 (or more than one) interface element on the user inter-
the local application program or the network service; and face. The target interface element usually is a text and/or
a favorites entrance of the commodity in the local appli- a picture.
cation program or the network service. [0056] In step S403: the terminal obtains content infor-
[0047] The above-mentioned searching result infor- mation of the target interface element according to the
mation may be obtained by searching through the server 20 first selecting operation.
upon the terminal sending the content information of the [0057] The operating system in the terminal obtains
target interface element to the server, without excluding the content information of the target interface element
the possibility that the terminal may perform searching according to the first selecting operation. When the target
in the data stored by itself according to the content infor- interface element is a text, the content information is a
mation. Illustratively, Fig. 4 is a flow chart illustrating an 25 character; when the target interface element is a picture,
information display method provided by another aspect the content information is image data.
of the present disclosure. The present embodiment will [0058] In step S404: the terminal sends a searching
be described with reference to the case where the infor- request to a server, the searching request includes the
mation display method is applied between the terminal content information of the target interface element.
and the server illustrated in Fig. 1, by way of example. 30 [0059] The terminal sends the searching request to the
The method includes steps as below. server over wired network or wireless network, the
[0048] In step S401: the terminal displays a user inter- searching request includes the content information of the
face of the application program, the user interface in- target interface element.
cludes at least one interface element. [0060] In step S405: the server receives the searching
[0049] When operated by a user, the terminal always 35 request.
can display a user interface of the application program [0061] In step S406: the server retrieves n pieces of
on the display screen thereof, no matter the user is op- searching result information according to the content in-
erating a native application program provided by the op- formation of the target interface element.
erating system or a third-party application program in- [0062] When the content information of the target in-
stalled by the user himself. 40 terface element is a text, the server decomposes the text
[0050] An application program usually includes multi- into several candidate items; selects a searching key-
ple user interfaces. The user interface of an application word from the several candidate items; and retrieves n
program running in the foreground of the terminal will be pieces of searching result information according to the
displayed in the display screen of the terminal preferen- searching keyword.
tially. 45 [0063] When the content information of the target in-
[0051] The user interface includes at least one inter- terface element is image data, the server performs an
face element thereon, and every interface element is one optical character recognition (OCR) to the image data;
of portions constituting the user interface. The interface decomposes a text recognized by the OCR into several
element includes, but not limited to, at least one of a text, candidate items; selects a searching keyword from the
a picture, an audio and a video. 50 several candidate items; and retrieves n pieces of
[0052] In step S402: the terminal receives a first se- searching result information according to the searching
lecting operation triggered on the user interface, the first keyword. Or, when the content information of the target
selecting operation is used for selecting a target interface interface element is image data, the server performs an
element from the at least one interface element. image recognition to the image data; obtains a recogni-
[0053] Referring to the first selecting operation which 55 tion result in a form of text according to at least one of
is a press-and-hold operation, by way of example, the an article name, an article category or an article property
user may press and hold on an interested interface ele- as recognized; and retrieves n pieces of searching result
ment. The terminal receives the press-and-hold opera- information by using the recognition result as a searching

6
 11 EP 3 407 221 B1 12

keyword. of an (i+1)th display element, i^2 and i≤n-1. The n display

[0064] In step S407: the server sends the n pieces of elements may also be displayed in a sliding manner ac-
searching result information to the terminal. cording to a sliding operation, to hide a portion of the list
[0065] The server sends the n pieces of searching re- items sliding off the display region of the display screen
sult information to the terminal over wired network or wire- 5 and display a portion of the list items sliding into the dis-
less network. play region of the display screen.
[0066] In step S408: the terminal receives the n pieces [0072] Optionally, in order to prevent from blocking a
of searching result information. main region of the user interface, an area of the display
[0067] In step S409: the terminal displays the n pieces elements is smaller than one half of an entire display
of searching result information in a superposed manner 10 region of the display screen.
on a local region of the user interface by the operating [0073] In step S410: the terminal receives a sliding op-
system, n is a positive integer. eration exerted on the display element.
[0068] The display element performs displaying under [0074] The user may slide the display element located
a control of the operating system, thus it has a display in the local region. The sliding operation includes a first
level higher than that of the user interface. Generally, 15 sliding operation of sliding along a first direction, and/or,
when a first display element has a higher display level a second sliding operation of sliding along a second di-
than a second display element, the first display element rection. The first direction is opposite to the second di-
will be shown in front of the second display element. In rection.
other words, the first display element may cover at least [0075] When the display element adopts a display
partially the second display element if they overlap. Every 20 mode illustrated in Fig. 3A, the first sliding operation is
display element is used for displaying at least one piece an operation of sliding from the left to the right, and the
of searching result information thereon. The display ele- second sliding operation is an operation of sliding from
ment may be in a form of card. Illustratively, the display the right to the left.
element may be presented in a shape of rectangular in [0076] In step S411: the terminal displays the display
which a text and/or an icon are/is displayed. Optionally, 25 element located in the local region in a sliding manner
the text is used for indicating the searching result infor- according to the sliding operation.
mation, and the icon is used for indicating an information [0077] Optionally, the terminal receives a first sliding
provider of the searching result information. operation triggered on an ith display element, and the first
[0069] Optionally, the local region is a bottom region. sliding operation is an operation of sliding along the first
Referring to Fig. 3A, a target interface element 22 that 30 direction. The terminal slides the ith display element along
the user is interested in is existed in the user interface the first direction according to the sliding operation to
21, and the target interface element 22 is a text or a pic- hide a portion sliding off the display screen; and slides
ture. When the user presses and holds on the target in- an (i-1)th display element along the first direction accord-
terface element 22, the terminal displays n display ele- ing to the sliding operation to display a portion sliding into
ments 23 on the bottom region, and each of the display 35 the display screen.
elements is used for displaying at least one piece of [0078] Referring to Fig. 5A, the terminal receives a
searching result information. The bottom region is an ad- rightward sliding operation triggered on a third display
jacent region of a bottom edge of the display region. The element 51. The terminal slides the third display element
searching result information may be provided in plural 51 along a rightward direction in the local region to hide
pieces, thus the terminal may only display a part of the 40 a portion sliding off the display region of the display
display elements 23 due to size restriction of the display screen, and slides a second display element 52 along a
region of the terminal. rightward direction in the local region to display the sec-
[0070] Optionally, the n display elements are arranged ond display element 52 sliding into the display region of
sequentially along a first direction on the local region of the display screen.
the user interface. In Fig. 3, the first direction is illustrated 45 [0079] Optionally, the terminal receives a second slid-
as a direction from the left to the right, and a tail of an ith ing operation triggered on the ith display element, and
display element is connected to a header of an (i+1)th the second sliding operation is an operation of sliding
display element, i is a positive integer and i<n-1. The along a second direction. The terminal slides the ith dis-
term "connected" may refer to that the tail of the ith display play element along the second direction according to the
element is overlapped with the header of the an (i+1)th 50 sliding operation to hide a portion sliding off the display
display element, or may be connected with a preset spac- screen; and slides the (i+1)th display element along the
ing. second direction according to the sliding operation to dis-
[0071] Optionally, the n display elements perform dis- play a portion sliding into the display screen.
playing by using a list-view control unit, and each of the [0080] Referring to Fig. 5B, the terminal receives a left-
display elements is one of list items in the list view. The 55 ward sliding operation triggered on a sixth display ele-
list view includes n display elements which are arranged ment 53. The terminal slides the sixth display element
horizontally and sequentially along a first direction, and 53 along a leftward direction in the local region to hide a
a tail of an ith display element is connected to a header portion (a fifth display element) sliding off the display re-

7
 13 EP 3 407 221 B1 14

gion of the display screen; and slides a seventh display to execute the program instructions in the memory 620
element 54 along a leftward direction in the local region so as to realize various functional applications and data
to display a portion (the seventh display element 54) slid- processes. Optionally, the program instructions stored in
ing into the display region of the display screen. the memory 620, when executed, are configured to im-
[0081] In some embodiments, the application program 5 plement steps in the information display method provided
running in the foreground of the terminal is a business by embodiments of the present disclosure which are per-
review APP, and the user interface running in the fore- formed by the terminal. The memory 620 may include
ground is a food introduction page of Beijing Ducky pro- high-speed random access memory (RAM), and may al-
vided by a merchant "X". When a user presses and holds so include nonvolatile memory; for example, at least one
on a target interface element that "Beijing ducky is one 10 disk storage, flash memory device or other volatile solid-
of traditional delicacies originated from Beijing. The Be- state memories.
ijing ducky prepared by merchant X is striving for tender [0087] The communication unit 630 is configured to
meat which is fat but not greasy", the terminal sends the realize a communication with the server. The communi-
text content of the target interface element as provided cation unit 630 may be a wireless communication unit
to the server, which feeds back four pieces of searching 15 such as radio frequency (RF) circuit, mobile communi-
result information, i.e., WIKI information of Beijing ducky, cation chip and WiFi communication chip. The commu-
introduction of shops under the name of merchant X, nav- nication unit 630 may also be wired communication unit
igation information concerning the merchant X, and on- such as Fiber, and RJ45 network card and interface.
line shopping information of Beijing ducky. Consequent- [0088] The display screen 640 is configured to display
ly, the terminal respectively displays the four pieces of 20 the user interface and receive touching operations which
searching result information through four display ele- are input from the outside. Optionally, the display screen
ments. Due to restriction of the width of the display is a touch screen which may acquire touching operations
screen, the terminal may display two pieces of searching exerted by the user on the touch screen or adjacent to
result information at the same time, and the user may the touch screen. Optionally, the touch screen is a touch
slide the respective pieces of searching result information 25 screen which supports a suspension touch and/or a force
for review by utilizing a sliding operation. touch.
[0082] As mentioned above, the information display [0089] Optionally, the program instructions stored in
method provided by embodiments of the present disclo- the memory 620 include an application layer 622 and a
sure determines the content information of the target in- kernel layer 624. The application layer 622 is located
terface element through the terminal and sends the 30 above the kernel layer 624; and the application layer 622
searching request carried with the content information to includes at least one application program. The kernel lay-
the server, which retrieves and feeds back n pieces of er 624 includes a screen driving program 21, other driving
searching result information corresponding to the target programs 22, an operating system kernel 23, and a con-
interface element; so as to provide diversified searching tent capturing program 24. Illustratively, the application
result information to the user by means of massive infor- 35 layer 622 includes a searching program 25, an applica-
mation on the server. Such searching result information tion program A, an application program B, an application
may be originated from variety of websites so that the program C and an application program D.
terminal may display variety of searching result informa- [0090] The screen driving program 21 is configured to
tion relatively rapidly, thereby improving the information package the touching operations received on the display
acquisition speed for the user. 40 screen as a touching event including an operating coor-
[0083] The information display method provided by dinate, and then report the touching event to the appli-
embodiments of the present disclosure further displays cation program in the application layer 622.
a display element located in the local region in a sliding [0091] Other driving programs 22 may be driving pro-
manner according to a sliding operation when receiving grams correlated with the processer 610, driving pro-
the sliding operation triggered on the display element 45 grams correlated with the memory 620, driving programs
through the terminal; in this way, several pieces of correlated with the communication unit 630, and the like.
searching result information may be displayed on the lo- [0092] The operating system kernel 23 is a kernel file
cal region with limited area to improve the display utili- of the operating system.
zation ratio of the local region. [0093] The content capturing program 24 has the ca-
[0084] Fig. 6 is a schematically structural view illustrat- 50 pacity of capturing the interface elements on the user
ing a terminal provided by an aspect of the present dis- interfaces of respective application programs in the ap-
closure. The terminal includes a processor 610, a mem- plication layer 622. Illustratively, each of the application
ory 620, a communication unit 630 and a display screen programs has at least one user interface, and each user
640. interface includes at least one interface element. Gener-
[0085] The processer 610 includes one or more 55 ally, n interface elements on a single user interface con-
processing kernels. stitute a DOM tree having nodes each denoting an inter-
[0086] The memory 620 is configured to store program face element. The content capturing program 24 has the
instructions and/or data; the processer 610 is configured capacity of capturing the interface elements on the user

8
 15 EP 3 407 221 B1 16

interfaces of respective application programs. Optionally, lecting operation; the touching event includes the oper-
the content capturing program 24 further stores a coor- ating coordinate (x, y) of the first selecting operation.
dinate range of respective interface elements in the user [0101] The screen driving program reports the operat-
interface; for example, the user interface includes a text ing coordinate (x, y) of the first selecting operation to the
view element; and the content capturing program 24 5 searching program. For example, the screen driving pro-
stores a coordinate range of the text view element in the gram reports the touching event to the searching pro-
user interface. gram.
[0094] The searching program 25 has the capacity of [0102] In step S403b, the searching program sends a
receiving the touching event reported by the screen driv- content acquiring request to the content capturing pro-
ing program 21 and the capacity of communicating with 10 gram, the content acquiring request is carried with the
the content capturing program 24. Optionally, the search- operating coordinate.
ing program 25 may also have the capacity of obtaining [0103] In step S403c, the content capturing program
an identifier of the application program running in the captures the content information of the target interface
foreground and the capacity of obtaining an identifier of element to which the operating coordinate corresponds.
the user interface running in the foreground. Optionally, 15 [0104] The content capturing program captures the
the searching program 25 further has the capacity of com- content information of several interface elements of the
municating with the server and the capacity of displaying user interface running in the foreground. Optionally, the
at least one piece of searching result information in a several interface elements may be represented by a
superposed manner on the user interface having been DOM tree. The content capturing program also stores a
displayed, upon obtaining the at least one piece of 20 coordinate range of respective interface elements in the
searching result information from the server. user interface.
[0095] Illustratively, the application program A is a na- [0105] The content capturing program selects the con-
tive application program provided by the operating sys- tent information corresponding to the target interface
tem; while the application program B, the application pro- from the content information of the several interface el-
gram C and the application program D each are a third- 25 ements in the user interface, according to the operating
party application program provided by a third-party man- coordinate (x, y) and the coordinate range corresponding
ufacture. The application program includes, but not lim- to each of the interface elements.
ited to, at least one of: News APP, navigation APP, busi- [0106] In step S403d, the content capturing program
ness review APP, instant communication APP, social sends the content information of the target interface el-
contact APP, blog APP and browser APP. 30 ement to the searching program.
[0096] The screen driving program 21, the other driving [0107] The searching program receives the content in-
programs 22, the operating system kernel 23, the content formation of the target interface element. Then the
capturing program 24, the searching program 25 and the searching program sends a searching request to the
application program A may be regarded as internal por- server, the searching request includes the content infor-
tions of the operating system; while the application pro- 35 mation of the target interface element; the searching pro-
gram B, the application program C, and the application gram receives at least one piece of searching result in-
program D may be regarded as external portions of the formation fed back by the server, and displays the at least
operating system. one piece of searching result information in a superposed
[0097] The above structure of the terminal is merely manner on the user interface having been displayed.
described by way of example. Those skilled in the art 40 [0108] As mentioned above, the information display
would appreciate that the terminal may include more or method provided by embodiments of the present disclo-
less components compared to that illustrated above. For sure achieves obtaining the content information of the
example, the terminal may further include loudspeaker, target interface element through a communication be-
microphone, input/output (I/O) component, power supply tween the searching program and the content capturing
and the like. 45 program, so as to realize cooperation between the ap-
[0098] In an alternative based on the embodiment il- plication layer and the kernel layer, thereby allowing the
lustrated in Fig. 4, the terminal determines the target in- searching program located in the application layer to ob-
terface element according to the operating coordinate of tain the content information of the target interface ele-
the first selecting operation, and captures the content ment located in other application programs running in the
information of the target interface element. The step 403 50 foreground.
in the flow chart illustrated in Fig. 4 may be replaced by [0109] In case that the first selecting operation is a
following steps S403a-403d, as illustrated in Fig. 7. press-and-hold operation, considering the press-and-
[0099] In step S403a, the screen driving program re- hold operation has other processing logics in certain ap-
ports the operating coordinate of the first selecting oper- plication programs, e.g., the press-and-hold operation is
ation to the searching program. 55 used for realizing a firing logic in a shooting game, the
[0100] Upon the user trigging the first selecting oper- content capturing program in some alternative embodi-
ation on the display screen, the screen driving program ments is stored with a preset list. The terminal detects
may generate a touching event corresponding to the se- whether context information of the application program

9
 17 EP 3 407 221 B1 18

satisfies a capturing condition according to the preset [0121] In case that the capturing condition indicates
list. The preset list is stored with context information sat- the application program where the context information
isfying the capturing condition and/or context information locates belongs to the preset white list, if the context in-
not satisfying the capturing condition. Illustratively, in an formation satisfies the capturing condition, i.e., the ap-
alternative based on the embodiment of Fig. 4, the step 5 plication program where the context information locates
S403 in the flow chart of the embodiment illustrated in belongs to the preset white list, then entering step S4036.
Fig.4 may be replaced by the following steps [0122] If the context information doesn’t satisfy the
S4031-4038, as illustrated in Fig. 8. capturing condition, i.e., the application program where
[0110] In step S4031, the screen driving program re- the context information locates doesn’t belong to the pre-
ports the operating coordinate of the first selecting oper- 10 set white list, then entering step S4039.
ation to the searching program. [0123] In step S4036, if the context information belongs
[0111] Upon the user trigging the first selecting oper- to the preset white list, then capturing the content infor-
ation on the display screen, the screen driving program mation of the interface element in the user interface.
may generate a touching event corresponding to the first [0124] The content capturing program captures the
selecting operation; the touching event includes the op- 15 content information of several interface elements of the
erating coordinate (x, y) of the first selecting operation. user interface running in the foreground. Optionally, the
[0112] The screen driving program reports the operat- several interface elements may be represented by a
ing coordinate (x, y) of the first selecting operation to the DOM tree. The content capturing program also stores a
searching program. For example, the screen driving pro- coordinate range corresponding to each of the interface
gram reports the touching event to the searching pro- 20 elements.
gram. [0125] In step S4037, the content capturing program
[0113] In step S4032, the searching program obtains selects the content information of the target interface el-
context information of the application program. ement according to the operating coordinate.
[0114] The searching program obtains context infor- [0126] The content capturing program selects the con-
mation of the application program running in the fore- 25 tent information corresponding to the target interface
ground. In case that the operating system is a mobile from the content information of the several interface el-
operating system, the operating system is provided with ements in the user interface, according to the operating
an active stack in which an activity located at a stack top coordinate (x, y) and the coordinate range corresponding
is corresponding to a user interface of the application to each of the interface elements.
program running in the foreground; the searching pro- 30 [0127] In step S4038, the content capturing program
gram obtains an identifier of the application program run- sends the content information of the target interface el-
ning in the foreground and an identifier of the user inter- ement to the searching program.
face from the activity located at the stack top. The iden- [0128] The searching program receives the content in-
tifier of the application program may be a package name formation of the target interface element. Then the
to which the APP corresponds, while the identifier of the 35 searching program sends the searching request to the
user interface may be a class name to which the user server, and the searching request includes the content
interface corresponds. information of the target interface element; the searching
[0115] The context information includes the identifier program receives at least one piece of searching result
of the application program and/or the identifier of the user information which is fed back by the server, and displays
interface. The application program is an application pro- 40 the at least one piece of searching result information in
gram running in the foreground, and the user interface a superposed manner on the user interface having been
is a user interface running in the foreground. displayed.
[0116] In step S4033, the searching program gener- [0129] In step S4039, the content capturing program
ates a content acquiring request including the operating sends acquisition failure information to the searching pro-
coordinate and the context information. 45 gram.
[0117] In step S4034, the searching program sends [0130] Herein, the searching program doesn’t need to
the content acquiring request to the content capturing send the searching request to the server.
program, the content acquiring request is carried with the [0131] In these optional embodiments, if an interface
operating coordinate and the context information. element in a user interface of an application program is
[0118] In step S4035, the content capturing program 50 not available for the content capturing program, then the
detects whether the context information satisfies a cap- identifier of the application program may be excluded
turing condition according to a preset list. from the preset white list or may be added to the preset
[0119] The preset list includes a preset white list and/or black list; if a certain user interface x of an application
a preset black list. program is not available for the content capturing pro-
[0120] The above-mentioned capturing condition indi- 55 gram, then the identifier of the user interface x may be
cates that the application program where the context in- excluded from the preset white list or may be added to
formation locates belongs to the preset white list, or the preset black list; if a user interface x of an application
doesn’t belong to the preset black list. program is not suitable to be captured by the content

10
 19 EP 3 407 221 B1 20

capturing program, e.g., the application is a game APP, the memory 920 include an application layer 922 and a
then the identifier of the game APP may be excluded kernel layer 924. The application layer 922 includes at
from the preset white list or may be added to the preset least one application program. Illustratively, the applica-
black list. tion layer 922 includes a database searching program
[0132] In the above-mentioned embodiments, the cap- 5 33, a database A, a database B and a database C.
turing condition of the content capturing program is de- [0139] The database searching program 33 has the
scribed with reference to the preset white list by way of capacity of extracting a searching keyword from the tar-
example; while in another optional embodiment, the con- get interface element and the capacity of retrieving the
tent capturing program is stored with a preset black list searching result information according to the searching
including an identifier of a black list APP and/or an iden- 10 keyword.
tifier of a blacklist user interface. In these optional em- [0140] More than one database may be configured to
bodiments, if an interface element in a user interface of store congruent relationships between the searching
an application program is not available for the content keyword and the searching result information. The
capturing program, then the identifier of the application searching result information may be varied depending
program may be added into the preset black list; if a cer- 15 on the database. Illustratively, the database A is config-
tain user interface x of an application program is not avail- ured to provide congruent relationships between person
able for the content capturing program, then the identifier name and person introduction information; the database
of the user interface x may be added into the preset black B is configured to provide congruent relationships be-
list; if a user interface x of an application program is not tween shop name and shop information; and database
suitable to be captured by the content capturing program, 20 C is configured to provide congruent relationships be-
e.g., the application is a game APP, then the identifier of tween commodity and commodity information.
the game APP may be added to the preset black list. [0141] Optionally, the application layer 922 may further
[0133] As mentioned above, the information display stores at least one of a decomposing program configured
method provided by the embodiments of the present dis- to decompose a text into several words; a word formation
closure stores a preset list in the content capturing pro- 25 recognizing program; a OCR program configured to per-
gram; in this way, it doesn’t need to obtain the target form OCR; and an image recognition program configured
interface element of the application program and the user to perform image recognition. These programs may be
interface which do not satisfy the capturing condition, integrated in the database searching program 33 as a
thereby preventing from performing unnecessary acqui- part thereof. When the content information of the target
sition tasks and avoiding any confliction with other exe- 30 interface element obtained by the database searching
cuted logics corresponding to the first selecting opera- program 33 is in a form of text, the decomposing program
tion. may decompose the content information into several can-
[0134] Fig. 9 is a schematically structural view illustrat- didate items each being a word, and the word formation
ing a server provided by an aspect of the present disclo- recognizing program is configured to recognize a word
sure. The server includes a processer 910, a memory 35 format of each of the candidate items. When the content
920 and a network interface 930. information of the target interface element obtained by
[0135] The processer 910 includes one or more the database searching program 33 is image data, the
processing kernel. OCR program is configured to recognize the characters
[0136] The memory 920 is configured to store program in the image, and instead use the characters in the image
instructions and/or data; the processer 910 is configured 40 as the content information of the target interface element;
to execute the program instructions in the memory 920 or the image recognition program is configured to recog-
so as to realize various functional applications and data nize an article name, an article category or an article at-
processes. Optionally, the program instructions stored in tribution from the image, and instead use the article
the memory 920 are configured to implement the steps name, the article category or the article attribution as the
of the information display method provided by embodi- 45 content information of the target interface element. These
ments of the present disclosure which are performed by recognition results may provide the user with the candi-
the server. The memory 920 may include high-speed ran- date item used as the searching keyword.
dom access memory (RAM), and may also include non- [0142] The above structure of the server is merely de-
volatile memory; for example, at least one disk storage, scribed by way of example. Those skilled in the art would
flash memory device or other volatile solid-state memo- 50 appreciate that the server may include more or less com-
ries. ponents compared to that illustrated above. For example,
[0137] The network interface 930 is configured to re- the server may further include loudspeaker, microphone,
alize a communication with the terminal or other servers. input/output (I/O) component, power supply and the like.
The network interface 930 may be a wired communica- [0143] In the embodiments illustrated in Fig. 7 or Fig.
tion unit such as Fiber interface, and RJ45 network card 55 8, the content information of the target interface element
and interface. The network interface 930 may also be a is described as a text directly captured by the content
wireless communication unit. capturing program by way of example. In some embod-
[0138] Optionally, the program instructions stored in iments, the target interface element may also be an im-

11
 21 EP 3 407 221 B1 22

age selected by the first selecting operation. Herein, the [0154] Considering the reading convenience of user,
step S403 in the embodiment illustrated in Fig. 4 may be the operating system of the terminal displays the inter-
replaced by the following step S403e or step S403f, as face element located in the region selecting control unit
illustrated in Fig. 10. in a magnified manner by using the operating position of
[0144] In step S403e, when the target interface ele- 5 the first selecting operation as the reference position. The
ment selected by the first selecting operation is a target magnification time may be any preset number.
image, determining image data of the target image as [0155] With reference to Fig. 13, an interface element
the content information of the target interface element. 132 is displayed on a user interface 131, and a user may
[0145] Under such implementation, the server may perform a force-touch operation (a press-and-hold oper-
perform OCR or image recognition to the image data of 10 ation is also possible) to the interface element 132 if
the target image, and perform searching according to the he/she is interested in the interface element 132. The
recognition result, to obtain at least one piece of search- operating system pops out a rectangular-shaped region
ing result information. selecting control unit 30 with the pressed point as the
[0146] In step S403f, when the target interface element central point. At the same time, the interface element in
selected by the first selecting operation is a target image, 15 the rectangular-shaped region selecting control unit 30
obtaining a recognition result corresponding to the target is also displayed with 1.5 times of magnification.
image and determining the recognition result as the con- [0156] In step s403i, receiving an adjusting operation
tent information of the target interface element. triggered on the region selecting control unit.
[0147] Under such implementation, the terminal may [0157] If a position and/or size of the region selecting
perform OCR to the image data of the target image, and 20 control unit cannot satisfy the user’s expectation, then
determine the recognized characters as the content in- the user may adjust the position and/or size of the region
formation of the target interface element; and/or, the ter- selecting control unit.
minal may perform image recognition to the target image, [0158] In case that the region selecting control unit is
and determine the recognized article name and/or article a rectangular-shaped region selecting control unit, by
attribution as the content information of the target inter- 25 way of example, when the user triggers a dragging op-
face element. The server may perform searching accord- eration inside the rectangular-shaped region selecting
ing to the recognition result of the terminal, to obtain at control unit, the dragging operation indicates an adjusting
least one piece of searching result information. operation for adjusting the position; when the user trig-
[0148] In the embodiment as illustrated in Fig.10, the gers a dragging operation at an edge of the rectangular-
content information of the target interface element is de- 30 shaped region selecting control unit, the dragging oper-
scribed as an image directly captured by the content cap- ation indicates an adjusting operation for adjusting the
turing program by way of example. In some embodi- size.
ments, the content information of the target interface el- [0159] In step S403j, changing the position and/or size
ement may also be a text obtained by capturing a picture of the region selecting control unit according to the ad-
of the user interface and analyzing according to the pic- 35 justing operation.
ture as captured. Herein, the step S403 in the embodi- [0160] When receiving the adjusting operation for ad-
ment of Fig. 4 may be replaced by the following steps justing the position, the operating system of the terminal
S403g-S4031, as illustrated in Fig. 11. changes the position of the region selecting control unit
[0149] In step S403g, displaying a region selecting according to the adjusting operation. For example, as it
control unit by using an operating position of the first se- 40 can be seen from Fig. 13, the user may drag the rectan-
lecting operation as a reference position. gular-shaped region selecting control unit to locate the
[0150] The region selecting control unit may be any interface element 132 at a center of the rectangular-
one of a rectangular-shaped region selecting control unit, shaped region selecting control unit 30.
a circle-shaped region selecting control unit and an ir- [0161] When receiving the adjusting operation for ad-
regular-shaped region selecting control unit. 45 justing the size, the operating system of the terminal
[0151] In case that the first selecting operation is a changes the size of the region selecting control unit ac-
force-touch operation by way of example, upon the user cording to the adjusting operation.
pressing a touch screen heavily, the operating system of [0162] In step S403k, upon receiving an acknowledge-
the terminal displays the region selecting control unit by ment signal, capturing a picture of the interface element
using the position at which the user presses heavily as 50 in the region selecting control unit to obtain a target im-
the reference position (or a central point). age.
[0152] The region selecting control unit may be dis- [0163] The acknowledgement signal may be a signal
played above the target interface element in a super- manually triggered by a user, or may be a signal auto-
posed manner. matically generated when the region selecting control
[0153] In step S403h, displaying the interface element 55 unit fails to receive any external operation within a preset
located in the region selecting control unit in a magnified time interval.
manner by using the operating position of the first select- [0164] Upon receiving the acknowledgement signal,
ing operation as a reference position. the operating system of the terminal captures a picture

12
 23 EP 3 407 221 B1 24

of the interface element in the region selecting control server; the searching request includes the recognition
unit and uses the image as captured as the target image. result obtained by performing OCR and/or image recog-
[0165] In step S4031, determining the content informa- nition to the image data. The server decomposes the
tion of the target interface element according to the target recognition result into words to obtain at least one can-
image. 5 didate item; determines a searching keyword from the at
[0166] The present step may be implemented as any least one candidate item; retrieves n pieces of searching
of the following three different solutions. result information according to the searching keyword;
[0167] As a first solution, the terminal determines im- and sends the n pieces of searching result information
age data of the target image as the content information to the terminal.
of the target interface element. 10 [0175] Consequently, the terminal receives the n piec-
[0168] The terminal sends a searching request to the es of searching result information and displays the n piec-
server, the searching request includes the image data of es of searching result information in a superposed man-
the target image. ner on the local region of the user interface through the
[0169] Optionally, the server performs an OCR to the operating system.
image data to obtain a text in the image data; decompos- 15 [0176] As mentioned above, the information display
es the text into words to obtain several candidate items; method provided by the present embodiment obtains the
determines a searching keyword from the several can- content information by means of capturing pictures, so
didate items; retrieves n pieces of searching result infor- that various types of interface elements such as pictures,
mation according to the searching keyword; and sends mixture of texts and pictures, videos and gifs may be
the n pieces of searching result information to the termi- 20 quickly searched and processed, which improves the
nal. Additionally or alternatively, the server performs an searching efficiency of a user when operating a terminal
image recognition to the image data; recognizes at least and simplifies the searching operation of the user.
one type of information of an article name, an article cat- [0177] In an alternative example, the step S406 may
egory or an article attribution from the image data for use be replaced by the following steps S406a-S406c, as il-
as at least one candidate item; determines a searching 25 lustrated in Fig. 12.
keyword from the at least one candidate item; retrieves [0178] In step S406a, decomposing the content infor-
n pieces of searching result information according to the mation of the target interface element to obtain at least
searching keyword; and sends the n pieces of searching one candidate item.
result information to the terminal. [0179] In case that the content information of the target
[0170] As a second solution, the terminal performs 30 interface element is a text, considering a word is the
OCR and/or image recognition to the image data of the smallest language element having independent mean-
target image, and uses the recognition result as the con- ing, the database searching program in the database de-
tent information of the target interface element. composes the content information into words to obtain
[0171] Optionally, the terminal performs OCR to the at least one candidate item.
image data to obtain a text in the image data, and uses 35 [0180] Optionally, the server may also recognize a
the text recognized by the OCR as the content informa- word format and/or an attribution of every candidate item.
tion; and/or, the terminal performs image recognition to The word format includes at least one of verb, noun, ad-
the image data to recognize at least one type of informa- jective, pronoun, predicate, numeral, classifier, interjec-
tion of an article name, an article category and an article tion and mimetic word. The attribution includes person
attribution from the image data for use as the content 40 name, site name, merchant name, commodity name,
information. website link, URL with predetermined format, and the
[0172] The terminal sends a searching request to the like.
server; the searching request includes the recognition [0181] For example, if the content information of the
result obtained by performing OCR or image recognition target interface element is "The Huaxia art gallery will
to the image data. The server decomposes the recogni- 45 hold an art exhibition on Day 19th", then the server de-
tion result into words to obtain at least one candidate composes the content information into five candidate
item; determines a searching keyword from the candidate items, which are "The Huaxia art gallery", "will", "hold",
item; retrieves n pieces of searching result information "an art exhibition" and "on Day 19th", respectively.
according to the searching keyword; and sends the n [0182] In case that the content information of the target
pieces of searching result information to the terminal. 50 interface element is image data, the server performs
[0173] As a third solution, the terminal sends the image OCR to the content information of the image data to ob-
data of the target image to the image processing server tain a text in the image data, and decomposes the text
which performs OCR and/or image recognition to the im- into words to obtain at least one candidate item; or, the
age data of the target image; then the terminal receives server performs image recognition to the image data to
a recognition result from the image processing server 55 recognize at least one type of information of an article
and determines the recognition result as the content in- name, an article category and an article attribution from
formation of the target interface element. the image data for use as the at least one candidate item.
[0174] The terminal sends a searching request to the [0183] In step S406b, determining a searching key-

13
 25 EP 3 407 221 B1 26

word from the at least one candidate item. [0194] In step S4042, the terminal sends a searching
[0184] Optionally, the database searching program de- request to the server, and the searching request includes
termines at least one of the person name, site name, the content information of the target interface element
merchant name, commodity name, website link and URL and the context information.
with predetermined format as the searching keyword. 5 [0195] The searching program in the terminal sends
[0185] In step S406c, retrieving the searching result the searching request to the server, and the searching
information from the database according to the searching request includes the content information and the context
keyword. , information of the target interface element.
[0186] The database searching program inputs the [0196] In step S405, the server receives the searching
searching keyword into the database, and retrieves the 10 request.
searching result information corresponding to the [0197] In step S4061, the server decomposes the con-
searching keyword. tent information of the target interface element to obtain
[0187] Optionally, the database searching program in- at least one candidate item.
puts the searching keyword into several databases, and [0198] In case that the content information of the target
retrieves n types of searching result information corre- 15 interface element is a text, considering a word is the
sponding to the searching keyword. smallest language element having independent mean-
[0188] As mentioned above, the information display ing, the database searching program in the database de-
method provided by the present embodiment decompos- composes the content information into words to obtain
es the content information through the server to obtain at least one candidate item.
at least one candidate item; determines a searching key- 20 [0199] Optionally, the server may also recognize a
word from the candidate item; and retrieves the searching word format and/or an attribution of every candidate item.
result information by searching using the searching key- The word format includes at least one of verb, noun, ad-
word. In this way, the meaning of the content information jective, pronoun, predicate, numeral, classifier, interjec-
may be accurately understood so as to retrieve the tion and mimetic word. The attribution includes person
searching result information expected by the user. 25 name, site name, merchant name, commodity name,
[0189] Alternatively, the step S404 in Fig. 4 may be website link, URL with predetermined format, and the
replaced by the following steps S4041 and S4042; and like.
the step S406 may be replaced by the following steps [0200] For example, if the content information of the
S4061- S4063, as illustrated in Fig. 13. target interface element is "The Huaxia art gallery will
[0190] In step S4041, the terminal obtains context in- 30 hold an art exhibition on Day 19th", then the server de-
formation of an application program. composes the content information into five candidate
[0191] The searching program in the terminal obtains items, which are "The Huaxia art gallery", "will", "hold",
context information of the application program running in "an art exhibition" and "on Day 19th", respectively.
the foreground. In case that the operating system is a [0201] In case that the content information of the target
mobile operating system, by way of example, the oper- 35 interface element is image data, the server performs
ating system is provided with an active stack in which an OCR to the content information of the image data to ob-
activity located at a stack top is corresponding to a user tain a text in the image data, and decomposes the text
interface of the application program running in the fore- into words to obtain at least one candidate item; or, the
ground; and the searching program obtains an identifier server performs image recognition to the image data to
of the application program and an identifier of the user 40 recognize at least one type of information of an article
interface running in the foreground from the activity lo- name, an article category and an article attribution from
cated at the stack top. The identifier of the application the image data for use as the at least one candidate item.
program may be a package name to which the APP cor- [0202] In step S4062, determining a searching key-
responds, while the identifier of the user interface may word from the at least one candidate item according to
be a class name to which the user interface corresponds. 45 the context information.
[0192] Optionally, the context information includes an [0203] Optionally, the database searching program
identifier of an application program and/or an identifier stores selection strategies corresponding to the context
of a user interface. The application program is an appli- information. The database searching program deter-
cation program running in the foreground, and the user mines a corresponding selection strategy according to
interface is a user interface running in the foreground. In 50 the context information, and determines the searching
other embodiments, the context information may also in- keyword from the at least one candidate item according
clude other information such as an identifier of a user to the selection strategy.
interface recently browsed and an identifier of a third- [0204] Illustratively, if the context information indicates
party application program having been installed, without that the application program running in the foreground is
limiting the embodiments thereto. 55 a news APP, then the selection strategy is to use all the
[0193] It should be explained that, if the terminal has nouns as the searching keyword.
obtained the context information in step S403, then the [0205] If the context information indicates that the ap-
step S4041 may be omitted. plication program running in the foreground is a user data

14
 27 EP 3 407 221 B1 28

page, then the selection strategy is to use the person display the search-adjusting element on the local region
name and the site name as the searching keyword. of the user interface. For a local region adopting a list-
[0206] If the context information indicates that the ap- view display mode, the search-adjusting element is dis-
plication program running in the foreground is a map played as one of the list items.
APP, then the selection strategy is to use the site name 5 [0218] Optionally, when the search-adjusting element
as the searching keyword. is fixedly displayed at a certain position (c.g., the right-
[0207] If the context information indicates that the ap- most side) on the local region, the step S412 and the
plication program running in the foreground is a shopping step S409 will be performed at the same time. Optionally,
APP, then the selection strategy is to use the merchant the search-adjusting element is displayed on the local
name, the commodity name, the website link and/or the 10 region as the last display element; when it’s turn to slide
URL with predetermined format as the searching key- the nth display element according to the sliding operation,
word. the search-adjusting element may be sliding into the local
[0208] If the context information indicates that the ap- region for display.
plication program running in the foreground is a movie [0219] For example, referring to Fig. 16A, eight display
APP, then the selection strategy is to use a movie name, 15 elements are arranged for displaying the searching result
a cinema name and/or actor/actress name as the search- information; when it is turn to slide the 8th display element
ing keyword. 151 according to the sliding operation, the terminal will
[0209] The selection strategies above are merely de- display the search-adjusting element, that is, a button
scribed by way of example, without limiting the embodi- 152 of "exact search", on the left side of thc8th display
ments of the present disclosure thereto. 20 element 151.
[0210] In step S4063, searching in the database ac- [0220] In step s413, receiving a triggering signal trig-
cording to the searching keyword to retrieve the search- gered on the search-adjusting element.
ing result information. [0221] Optionally, the triggering signal is a click-on sig-
[0211] The database searching program inputs the nal exerted on the search-adjusting element.
searching keyword into the database, and retrieves the 25 [0222] In step S414, displaying a candidate item ex-
searching result information corresponding to the hibiting window for the search-adjusting element, the
searching keyword. candidate item exhibiting window includes m candidate
[0212] Optionally, the database searching program in- items corresponding to the target interface element.
puts the searching keyword into several databases, and [0223] Optionally, the candidate item exhibiting win-
retrieves n pieces of searching result information corre- 30 dow is another superposed window which occupies a
sponding to the searching keyword. central region of the entire display region. On the candi-
[0213] Optionally, the database searching program se- date item exhibiting window, m candidate items are dis-
lects a target database according to the context informa- played, wherein m is a positive integer.
tion, and inputs the searching keyword into the target [0224] When displaying the candidate item exhibiting
database to obtain n pieces of searching result informa- 35 window, the n display elements and the search-adjusting
tion corresponding to the searching keyword. element superposed on the local region may be stayed
[0214] As mentioned above, in the information display at the original position, and may also be withdrawn from
method provided by the present embodiment, the search- display.
ing request is carried with context information, so that [0225] For example, referring to Fig. 16A, upon the us-
the server may select more suitable or more exact 40 er clicking on the button 152 of "exact search", the oper-
searching keyword from the at least one candidate item ating system in the terminal withdraws the display of the
according to the context information, and retrieves more n display elements and the button 152 of "exact search",
accurate searching result information according to the and displays the candidate item exhibiting window 153
searching keyword; thereby retrieving the searching re- in a superposed manner on the central region of the dis-
sult information which meet the user’s expectation in a 45 play screen. On the candidate item exhibiting window
better way by combining with the current usage scenario 153, four candidate items are displayed.
of the user, based on true understanding of the meaning [0226] Optionally, on the candidate item exhibiting win-
of the content information. dow 153, a button of "copy", a button of "search" and a
[0215] In an alternative of the example in Fig. 4, it may button of "select all" may be further displayed.
provide a searching keyword adjusting mechanism to the 50 [0227] In another example, referring to Fig. 16B, upon
user. In more details, the above-mentioned information the user clicking on the button 152 of "exact search," the
display method further includes steps S412-S416, as il- operating system in the terminal withdraws the display
lustrated in Fig. 15. of the n display elements and the button 152 of "exact
[0216] In step S412, displaying a search-adjusting el- search", and displays a candidate item exhibiting window
ement in a superposed manner on the local region of the 55 154 for picture in a superposed manner on the central
user interface. region of the display screen. On the candidate item ex-
[0217] When displaying the n pieces of searching re- hibiting window 154, three candidate items 154a, 154b
sult information on the local region, the terminal may also and 154c are displayed.

15
 29 EP 3 407 221 B1 30

[0228] Optionally, on the candidate item exhibiting win- at least one piece of first searching result information
dow 154, a button of "search" and a button of "select all" which includes an invoking interface for invoking a local
may be further displayed. application program; the invoking interface has an input
[0229] In step S415, receiving a second selecting op- parameter including a searching keyword corresponding
eration to the at least one candidate item. The second 5 to the target interface element. Steps S409-S411 may
selecting operation is used for determining a keyword, be replaced by the following steps S409a-S409b, or
to which a selected candidate item corresponds, as the steps S409c-S409d, as illustrated in Fig. 18.
searching keyword. [0240] In step S409a, receiving a third selecting oper-
[0230] The user may click on an interested candidate ation triggered on the first searching result information.
item on the candidate item exhibiting window, and then 10 [0241] The third selecting operation is used for select-
determines at least one candidate item as the searching ing a target first searching result information which in-
keyword. cludes an invoking interface of the local application pro-
[0231] For example, referring to Fig. 16A, the user se- gram. The local application program is an available APP
lects "candidate item 3" as the searching keyword. having been installed on the terminal, and the invoking
[0232] In Fig. 16B, the user selects a picture candidate 15 interface is a function instruction for invoking the local
154a as the searching keyword. application program or a URL address for invoking a web-
[0233] In step S416, displaying the searching result page corresponding to the local application program.
information corresponding to the searching keyword. [0242] In step S409b, displaying a user interface of the
[0234] Optionally, the terminal sends the above-men- local application program.
tioned searching keyword to the server, which feeds back 20 [0243] Receiving the invoking interface of local appli-
n pieces of searching result information corresponding cation program on the first searching result information
to the searching keyword to the terminal. according to third selecting operation as triggered, so as
[0235] Optionally, the terminal invokes a browser, uti- to invoke a user interface of the local application program.
lizes a searching engine in the browser to retrieve the The user interface includes processing results obtained
searching result information matched with the searching 25 through processing the searching keyword correspond-
keyword, and displays the searching result information ing to the first searching result information by the local
matched with the searching keyword in the browser. application program.
[0236] The browser invoked by the terminal may be a [0244] In an example, referring to Fig. 19, upon the
browser built in the operating system, and may also be user triggering the third selecting operation on the first
a default browser defined by the user. 30 searching result information 191, receiving the invoking
[0237] As mentioned above, in the information display interface of local application program in the first search-
method provided by the present embodiment, when n ing result information 191. For example, if the first search-
pieces of searching result information obtained by an in- ing result information contains a site name, then it con-
itial searching is relatively less or relatively more, the user tains an invoking interface of a local map APP; the in-
may perform a second searching by manually selecting 35 voking interface may be a function invoking instruction
a searching keyword through the search-adjusting ele- of the map APP or a URL address of a webpage to which
ment and the candidate item exhibiting window provided the map APP corresponds; upon triggering the third se-
to the user, thereby increasing the searching accuracy, lecting operation, receiving the function instruction for
and reducing the operation times of re-opening the invoking the map APP and displaying a map user inter-
browser and inputting the searching keyword. 40 face 192. The map user interface 192 as displayed in-
[0238] It should be explained herein that, the foregoing cludes processing results obtained by processing the
embodiments may be combined flexibly for implementa- searching keyword corresponding to the first searching
tion by those skilled in the art; moreover, the foregoing result information 191 by the map APP. For example, if
embodiments may also be flexibly combined with the ex- the searching keyword corresponding to the first search-
isting embodiments in the related art by those skilled in 45 ing result information 191 is "museum", then a content
the art. For example, in related art, usually, the press- displayed in the map user interface 192 includes a
and-hold operation may be used for triggering operations searching result interface obtained by searching in the
such as copying, pasting and searching. In case that the map APP with "museum" as the searching keyword.
foregoing embodiments are combined with existing em- [0245] The steps S409a-S409b in the above embodi-
bodiments in the related art, as illustrated in Fig. 17, if 50 ment may be replaced by steps S409c-S409d. The steps
the user presses and holds on the target interface ele- S409a-S409b and the steps S409c-S409d may be two
ment 22, then the terminal will not only display at least embodiments independently from each other. The steps
one piece of searching result information on the local S409a-S409b may be alternately performed with the
region 23 but also display a control unit 24 of "copy" and steps S409c-S409d, and may also be combined with any
a button 25 of "opening in the browser" on the central 55 one or more of the foregoing embodiments.
region of the display screen. [0246] In an alternative of the example illustrated in
[0239] In an alternative of the example illustrated in Fig. 4, the n pieces of searching result information include
Fig. 4, the n pieces of searching result information include at least one piece of second searching result information

16
 31 EP 3 407 221 B1 32

which includes an invoking interface for invoking a net- ing module 1720 is configured to display searching result
work service. The invoking interface has an input param- information in a superposed manner on a local region of
eter including a searching keyword corresponding to the the user interface by the operating system, the searching
target interface element. result information is corresponding to the target interface
[0247] In step S409c, receiving a fourth selecting op- 5 element.
eration triggered on the first searching result information. [0254] In one or more embodiments, the displaying
[0248] The fourth selecting operation is used for se- module 1720 is configured to display n display elements
lecting a target first searching result information. A con- in a superposed manner on a local region of the user
tent of the first searching result information includes an interface by the operating system, wherein n is a positive
invoking interface of network service. The network serv- 10 integer; a display level of the display element is higher
ice refers to an application program having not been in- than a display level of the user interface, and each of the
stalled in the terminal but available in a webpage, or other display elements is configured to display a first order in-
programs such as a searching engine; the invoking in- formation of at least one piece of searching result infor-
terface is a URL address for invoking a webpage corre- mation thereon.
sponding to the network service. 15 [0255] In one or more embodiments, the displaying
[0249] In step S409d, displaying a user interface of the module 1720 is configured to display a list-view control
network service. unit in a superposed manner on a local region of the user
[0250] Receiving the invoking interface of network interface by the operating system; the list-view control
service on the first searching result information according unit includes n display elements, wherein n≥2; the n dis-
to the fourth selecting operation as triggered, so as to 20 play elements are arranged sequentially along a first di-
invoke a user interface corresponding to the webpage of rection in the list-view control unit, and a tail of an ith
the network service. The user interface includes process- display element is connected to a header of an (i+1)th
ing results obtained through processing the searching display element, wherein i is a positive integer and i≤n-1.
keyword corresponding to the first searching result infor- [0256] In one or more embodiments, the local region
mation by a network application program. 25 is: a bottom region of the user interface; or an adjacent
[0251] In one or more embodiments, upon the user trig- region of the target interface element; or an adjacent re-
gering the fourth selecting operation on the first searching gion of an operating position of the selecting operation.
result information, receiving an invoking interface of net- [0257] In one or more embodiments, the receiving
work APP in the first searching result information. For module is further configured to receive a first sliding op-
example, if the first searching result information contains 30 eration on a jth searching result information, the first slid-
a title of book, then it contains a URL address of the ing operation is an operation of sliding along a first direc-
network APP. Upon triggering the fourth selecting oper- tion. When the receiving module receives a first sliding
ation, receiving a function instruction for invoking a operation on a jth searching result information, the dis-
browser APP and displaying a user interface. The user playing module is configured to display the jth searching
interface includes processing results obtained by 35 result information in a sliding manner along the first di-
processing the searching keyword corresponding to the rection by following the first sliding operation, to hide a
first searching result information through a browser portion of the jth searching result information which is
searching engine. For example, if the searching keyword sliding off the local region and display a portion of a (j-
corresponding to the first searching result information is 1)th searching result information which is sliding into the
"dictionary", then the content displayed in the browser 40 local region, where j is an integer greater than 1, and j is
user interface includes a searching result interface ob- smaller than or equal to n.
tained by searching in the searching engine with "diction- [0258] Additionally or alternatively, the receiving mod-
ary" as the searching keyword. ule is further configured to receive a second sliding op-
[0252] Fig. 20 is a block diagram illustrating an infor- eration on a kth searching result information, the second
mation display device provided by an aspect of the 45 sliding operation is an operation of sliding along a second
present disclosure. The information display device may direction. When the receiving module receives a second
be implemented with software, hardware and/or combi- sliding operation on a kth searching result information,
nation thereof to constitute an entire terminal or a portion the displaying module is configured to display the kth
of the terminal. The device is installed with an operating searching result information in a sliding manner along
system and an application program. 50 the second direction by following the second sliding op-
[0253] The device includes a displaying module 1720 eration, to hide a portion of the kth searching result infor-
configured to display a user interface of the application mation which is sliding off the local region and display a
program, the user interface includes at least one interface portion of a (k+1)th searching result information which is
element; a receiving module 1740 configured to receive sliding into the local region, wherein k is an integer small-
a first selecting operation triggered on the user interface; 55 er than or equal to n. The first direction is opposite to the
a selecting module 1760 configured to select a target second direction.
interface element from the at least one interface element [0259] In one embodiment according to the invention,
according to the first selecting operation; and the display- the device further includes: an obtaining module config-

17
 33 EP 3 407 221 B1 34

ured to obtain content information of the target interface obtaining module is configured to obtain a recognition
element according to the first selecting operation; a send- result corresponding to a target image and determine the
ing module configured to send a searching request to a recognition result as the content information of the target
server, the searching request includes the content infor- interface element when the target interface element se-
mation of the target interface element; and the receiving 5 lected by the first selecting operation is the target image.
module is further configured to receive the searching re- [0265] In one or more embodiments, the obtaining
sult information which is fed back by the server. module is configured to: display a region selecting control
[0260] In one embodiment according to the invention, unit by using an operating position of the first selecting
the obtaining module is configured to determine the tar- operation as a reference position; capture a screen pic-
get interface element according to an operating coordi- 10 ture of the interface element in the region selecting con-
nate of the first selecting operation, and capture the con- trol unit to obtain a target image upon receiving an ac-
tent information of the target interface element. knowledgement signal; and determine the target image
[0261] In one embodiment according to the invention, as the content information of the target interface element,
the obtaining module is configured to detect whether con- or, obtain a recognition result corresponding to the target
text information of the application program satisfies a 15 image and determine the recognition result as the content
capturing condition according to a preset list, the preset information of the target interface element.
list is stored with context information satisfying the cap- [0266] In one or more embodiments, the obtaining
turing condition and/or context information not satisfying module is configured to perform an optical character rec-
the capturing condition; and if the context information sat- ognition (OCR) or an image recognition to the target im-
isfies the capturing condition, then performing a step of 20 age, to obtain a recognition result corresponding to the
determining the target interface element according to the target image; or the obtaining module is configured to
operating coordinate of the first selecting operation and send the target image to an image processing server,
capturing the content information of the target interface and receive a recognition result corresponding to the tar-
element. get image which is fed back by the image processing
[0262] In one or more embodiments, the obtaining 25 server.
module includes: a screen driving program sub-module, [0267] In one or more embodiments, the displaying
a searching program sub-module, and a content captur- module is further configured to display the interface ele-
ing program sub-module. The screen driving program ment in the region selecting control unit in a magnified
sub-module is configured to report an operating coordi- manner, by using an operating position of the first select-
nate of the first selecting operation to the searching pro- 30 ing operation as a reference position.
gram; the searching program sub-module is configured [0268] In one or more embodiments, the device further
to send an interface element acquiring request to the includes an adjusting module. The receiving module is
content capturing program, the interface element acquir- further configured to receive an adjusting operation trig-
ing request is carried with the operating coordinate; the gered on the region selecting control unit; and the ad-
content capturing program sub-module is configured to 35 justing module is configured to change a position and/or
capture content information of the target interface ele- a size of the region selecting control unit according to the
ment to which the operating coordinate corresponds; and adjusting operation.
the content capturing program sub-module is configured [0269] In one or more embodiments, the device further
to send the content information of the target interface includes: the obtaining module is configured to obtain
element to the searching program by the content captur- 40 context information of the application program, the con-
ing program. text information includes an identifier of the application
[0263] In one or more embodiments, the obtaining program and/or an identifier of the user interface; the
module includes: a searching program sub-module and sending module is configured to send a searching re-
a content capturing program sub-module. The searching quest to the server, the searching request includes the
program sub-module is configured to obtain context in- 45 content information of the target interface element and
formation of the application program which includes an the context information. The server is configured to obtain
identifier of the application program and/or an identifier at least one candidate item according to the content in-
of the user interface, and is configured to generate the formation; determine a searching keyword from the at
interface element acquiring request which includes the least one candidate item according to the context infor-
operating coordinate and the context information. The 50 mation; and obtain the searching result information by
content capturing program sub-module is configured to searching according to the searching keyword.
detect whether the context information satisfies a cap- [0270] In one or more embodiments, the receiving
turing condition. module is further configured to receive a press-and-hold
[0264] In one or more embodiments, the obtaining operation triggered on the user interface, the press-and-
module is configured to determine image data of a target 55 hold operation is an operation with a touch-control dura-
image as the content information of the target interface tion time which is longer than a preset duration time; or
element when the target interface element selected by the receiving module is further configured to receive a
the first selecting operation is the target image; or the force-touch operation triggered on the user interface, the

18
 35 EP 3 407 221 B1 36

force-touch operation is an operation with a touch-control of the film and television work in the local application
pressure which is larger than a preset pressure. program or the network service; a favorites entrance of
[0271] In one or more embodiments, the displaying the film and television work in the local application pro-
module is configured to display a search-adjusting ele- gram; and a ticket-booking entrance of the film and tel-
ment in a superposed manner on the local region of the 5 evision work in the network service.
user interface; the receiving module is configured to re- [0275] Additionally or alternatively, if the content infor-
ceive a triggering signal triggered on the search-adjusting mation of the target interface element includes a title of
element; the displaying module is configured to display literature work, then the searching result information in-
a candidate item exhibiting window corresponding to the cludes, but not limited to, at least one of: introduction
search-adjusting element according to the triggering sig- 10 information of the literature work; rating information of
nal, the candidate item exhibiting window includes m can- the literature work; author information of the literature
didate items corresponding to the target interface ele- work; a subscribing entrance of the literature work in the
ment, wherein m is a positive integer; the receiving mod- local application program; a sharing entrance of the lit-
ule is configured to receive a second selecting operation erature work in the local application program; a favorites
for at least one of the candidate items, the second se- 15 entrance of the literature work in the local application
lecting operation is configured to determine a selected program; a reading experience entrance of the literature
candidate item as a searching keyword; and the display- work in the network service; and a purchasing entrance
ing module is configured to display searching result in- of the literature work in the network service.
formation corresponding to the searching keyword. [0276] Additionally or alternatively, if the content infor-
[0272] In one or more embodiments, the device further 20 mation of the target interface element includes informa-
includes an invoking module configured to invoke an in- tion of a person, then the searching result information
voking interface of local application program, wherein an includes, but not limited to, at least one of: WIKI informa-
input parameter of the invoking interface includes a tion of the person; a social network service homepage
searching keyword corresponding to the target interface of the person in the network service; latest news and
element; the receiving module is configured to receive a 25 trends of the person in the network service; a dynamic
third selecting operation triggered on the first searching subscribing entrance of the person in the network serv-
result information, the third selecting operation is config- ice; a sharing entrance of the person in the local appli-
ured to trigger invoking the local application program by cation program or the network service; and a favorites
the invoking interface to process the searching keyword; entrance of the person in the local application program
and the displaying module is configured to display a user 30 or the network service.
interface of the local application program, the user inter- [0277] Additionally or alternatively, if the content infor-
face includes a processing result obtained through mation of the target interface element includes informa-
processing the searching keyword by the local applica- tion of a site, then the searching result information in-
tion program. cludes, but not limited to, at least one of: introduction
[0273] In one or more embodiments, the device in- 35 information of the site; navigation information of the site
cludes: the invoking module is configured to invoke an in the local application program or the network service;
invoking interface of network application program, map preview information of the site in the local application
wherein an input parameter of the invoking interface in- program or the network service; a taxi-calling entrance
cludes a searching keyword corresponding to the target with the site as a destination; a sharing entrance of the
interface element; the receiving module is configured to 40 site in the local application program or the network serv-
receive a fourth selecting operation triggered on the sec- ice; and a favorites entrance of the site in the local ap-
ond searching result information, the fourth selecting op- plication program or the network service.
eration is configured to trigger invoking the network ap- [0278] Additionally or alternatively, if the content infor-
plication program by the invoking interface to process mation of the target interface element includes informa-
the searching keyword; and the displaying module is con- 45 tion of a merchant, then the searching result information
figured to display a user interface of the network appli- includes, but not limited to, at least one of: introduction
cation program, the user interface includes a processing information of the merchant; a social network service
result obtained through processing the searching key- homepage of the merchant in the network service; latest
word by the network application program. news and trends of the merchant in the network service;
[0274] In one or more embodiments, if the content in- 50 a preferential subscribing entrance of the merchant in
formation of the target interface element includes a title the network service; contact information of the merchant;
of film and television work, then the searching result in- navigation information of the merchant in the local appli-
formation includes, but not limited to, at least one of: in- cation program or the network service; map preview in-
troduction information of the film and television work; rat- formation of the merchant in the local application program
ing information of the film and television work; author 55 or the network service; a taxi-calling entrance with the
information of the film and television work; a subscribing merchant as a destination; a sharing entrance of the mer-
entrance of the film and television work in a local appli- chant in the local application program or the network serv-
cation program or a network service; a sharing entrance ice; and a favorites entrance of the merchant in the local

19
 37 EP 3 407 221 B1 38

application program or the network service. Thus, the appearances of the phrases "in one embodi-
[0279] Additionally or alternatively, if the content infor- ment" or "in an embodiment," "in an exemplary embodi-
mation of the target interface element includes informa- ment," or the like in the singular or plural in various places
tion of a commodity, then the searching result information throughout this specification are not necessarily all refer-
includes, but not limited to, at least one of: introduction 5 ring to the same embodiment. Furthermore, the particular
information of the commodity; a purchasing entrance of features, structures, or characteristics in one or more em-
the commodity in the network service; price information bodiments may be combined in any suitable manner.
of the commodity in at least one shopping channel; a
preferential subscribing entrance of the commodity in the
network service; a discount reminding entrance of the 10 Claims
commodity in the network service; a history price trend
checking entrance of the commodity in the local applica- 1. A terminal (120) installed with an operating system
tion program or the network service; a sharing entrance and an application program, the terminal (120) com-
of the commodity in the local application program or the prising a processor and a memory, wherein the proc-
network service; and a favorites entrance of the com- 15 essor is configured to:
modity in the local application program or the network
service. control a display to display a user interface (21,
[0280] The present disclosure further provides a com- 131) of the application program, the user inter-
puter-readable storage medium stored with at least one face (21, 131) comprising at least one interface
instruction which is loaded and executed by a processer 20 element;
to implement the information display method according receive a first selecting operation triggered on
to any of the foregoing embodiments. the user interface (21, 131);
[0281] Those skilled in the art would appreciate that all select a target interface element (22) from the
or part of the steps for realizing the foregoing embodi- at least one interface element according to the
ments may be implemented as hardware, or may be im- 25 first selecting operation; and
plemented as program instructions in combination with control the display to display searching result
related hardware. The program instructions may be information (23) in a superposed manner on a
stored in a computer-readable storage medium which local region of the user interface (21, 131) by
may be a read only memory, a magnetic disk, or an optical the operating system, and the searching result
disk. 30 information (23) corresponds to the target inter-
[0282] The terminology used in the present disclosure face element (22);
is for the purpose of describing exemplary embodiments wherein searching result information (23) is pro-
only and is not intended to limit the present disclosure. vided in an amount of n, in which at least two
As used in the present disclosure and the appended pieces of searching result information are differ-
claims, the singular forms "a," "an" and "the" are intended 35 ent from each other, and n is a positive integer;
to include the plural forms as well, unless the context and
clearly indicates otherwise. It shall also be understood wherein before displaying searching result infor-
that the terms "or" and "and/or" used herein are intended mation (23) in a superposed manner on a local
to signify and include any or all possible combinations of region of the user interface (21, 131) by the op-
one or more of the associated listed items, unless the 40 erating system, the processor is further config-
context clearly indicates otherwise. ured to:
[0283] It shall be understood that, although the terms
"first," "second," "third," etc. may be used herein to de- obtain content information of the target in-
scribe various information, the information should not be terface element (22) according to the first
limited by these terms. These terms are only used to 45 selecting operation;
distinguish one category of information from another. For send a searching request to a server (140),
example, without departing from the scope of the present wherein the searching request comprises
disclosure, first information may be termed as second the content information of the target inter-
information; and similarly, second information may also face element (22); and
be termed as first information. As used herein, the term 50 receive the searching result information
"if’ may be understood to mean "when" or "upon" or "in (23) fed back by the server (140),
response to" depending on the context. wherein obtaining content information of the
[0284] Reference throughout this specification to "one target interface element (22) according to
embodiment," "an embodiment," "exemplary embodi- the first selecting operation comprises:
ment," or the like in the singular or plural means that one 55
or more particular features, structures, or characteristics determining the target interface ele-
described in connection with an embodiment is included ment (22) according to an operating co-
in at least one embodiment of the present disclosure. ordinate of the first selecting operation,

20
 39 EP 3 407 221 B1 40

and capturing content information of to the searching program.

the target interface element (22);
wherein before determining the target 2. The terminal (120) according to claim 1, wherein the
interface element (22) according to an processor is further configured to:
operating coordinate of the first select- 5 control the display to display n display elements in
ing operation and capturing content in- a superposed manner on a local region of the user
formation of the target interface ele- interface (21, 131) by the operating system, wherein
ment (22), the processor is further con- n is a positive integer, a display level of the display
figured to: elements is higher than a display level of the user
10 interface (21, 131), and each of the display elements
obtain context information of the is configured to display information of at least one
application program, wherein the piece of searching result information (23) thereon.
application program is an applica-
tion program running in the fore- 3. The terminal (120) according to claim 2, wherein the
ground, the user interface (21, 131) 15 processor is further configured to:
is a user interface running in the control the display to display a list-view control unit
foreground of the application pro- in a superposed manner on a local region of the user
gram, the context information com- interface (21, 131) by the operating system, the list-
prises the identifier of the applica- view control unit comprising n display elements,
tion program and/or the identifier 20 wherein n is an integer greater than 1, the n display
of the user interface; the operating elements are arranged sequentially along a first di-
system is provided with an active rection in the list-view control unit, and a tail of an ith
stack in which an activity located display element is connected to a header of an (i+1)th
at a stack top is corresponding to display element, wherein i is a positive integer and i
the user interface (21, 131) of the 25 is less than n.
application program running in the
foreground; a searching program 4. The terminal (120) according to claim 1, wherein the
obtains an identifier of the applica- local region includes one of following regions:
tion program running in the fore-
ground and an identifier of the user 30 a bottom region of the user interface (21, 131);
interface from the activity located an adjacent region of the target interface ele-
at the stack top;detect whether ment (22); and
context information of the applica- an adjacent region of an operating position of
tion program satisfies a capturing the selecting operation.
condition according to a preset list 35
that comprises a preset white list 5. The terminal (120) according to any one of claims
and/or a preset black list, the pre- 1-4, wherein after displaying searching result infor-
set list being stored with context in- mation (23) in a superposed manner on a local region
formation satisfying the capturing of the user interface (21, 131) by the operating sys-
condition and/or context informa- 40 tem, the processor is further configured to:
tion not satisfying the capturing
condition; and when receiving a first sliding operation on a jth
if the context information satisfies searching result information, the first sliding op-
the capturing condition that indi- eration being an operation of sliding along a first
cates that the application program 45 direction, displaying the jth searching result in-
where the context information lo- formation in a sliding manner along the first di-
cates belongs to the preset white rection by following the first sliding operation, to
list, or doesn’t belong to the preset hide a portion of the jth searching result informa-
black list, then perform a step of tion which is sliding off the local region and dis-
determining the target interface el- 50 play a portion of a (j-1)th searching result infor-
ement (22) according to an operat- mation which is sliding into the local region,
ing coordinate of the first selecting wherein j is an integer greater than 1, and j is
operation and capturing content in- smaller than or equal to n; and
formation of the target interface el- when receiving a second sliding operation on a
ement (22); 55 kth searching result information, the second slid-
if the context information doesn’t ing operation is an operation of sliding along a
satisfy the capturing condition, second direction, displaying the kth searching
send acquisition failure information result information in a sliding manner along the

21
 41 EP 3 407 221 B1 42

second direction by following the second sliding content information of the target interface ele-
operation, to hide a portion of the kth searching ment (22);
result information which is sliding off the local when the target interface element (22) selected
region and display a portion of a (k+1)th search- by the selecting operation is the target image,
ing result information which is sliding into the 5 obtain a recognition result corresponding to the
local region, wherein k is an integer smaller than target image and determining the recognition re-
n, sult as the content information of the target in-
wherein the first direction is opposite to the sec- terface element (22);
ond direction. control the display to display a region selecting
10 control unit (30) by using an operating position
6. The terminal (120) according to any preceding claim, of the selecting operation as a reference posi-
wherein the operating system comprises: a screen tion;
driving program; a searching program and a content capture a screen picture of the interface element
capturing program; and wherein determining the tar- in the region selecting control unit (30) to obtain
get interface element (22) according to an operating 15 a target image upon receiving an acknowledge-
coordinate of the first selecting operation and cap- ment signal; and
turing content information of the target interface el- determine the target image as the content infor-
ement (22) comprises: mation of the target interface element (22); or,
obtain a recognition result corresponding to the
reporting an operating coordinate of the first se- 20 target image and determining the recognition re-
lecting operation to the searching program by sult as the content information of the target in-
the screen driving program; terface element (22).
sending an interface element acquiring request
to the content capturing program by the search- 8. The terminal (120) according to claim 7, wherein the
ing program, wherein the interface element ac- 25 processor is further configured to:
quiring request is carried with the operating co-
ordinate; perform an optical character recognition (OCR)
capturing content information of the target inter- or an image recognition to the target image to
face element (22), to which the operating coor- obtain a recognition result corresponding to the
dinate corresponds, by the content capturing 30 target image; and
program; and send the target image to an image processing
sending the content information of the target in- server; and receiving the recognition result cor-
terface element (22) to the searching program responding to the target image fed back by the
by the content capturing program; image processing server.
wherein the operating system comprises: a 35
searching program and a content capturing pro- 9. The terminal (120) according to any preceding claim,
gram; and wherein detecting whether context in- wherein the processor is further configured to:
formation of the application program satisfies a
capturing condition comprises: obtain context information of the application pro-
40 gram, the context information comprises an
obtaining context information of the appli- identifier of the application program and/or an
cation program by the searching program, identifier of the user interface (21, 131); and
the context information comprising an iden- send a searching request to the server (140),
tifier of the application program and/or an the searching request comprising content infor-
identifier of the user interface (21, 131); 45 mation of the target interface element (22) and
generating the interface element acquiring the context information;
request which comprises the operating co- wherein the server (140) is configured to obtain
ordinate and the context information; and at least one candidate item according to the con-
detecting whether the context information tent information, determine a searching keyword
satisfies the capturing condition by the con- 50 from the at least one candidate item according
tent capturing program. to the context information, and obtain the
searching result information (23) by searching
7. The terminal (120) according to any preceding claim, according to the searching keyword.
wherein the processor is further configured to:
55 10. The terminal (120) according to claim 1, wherein the
when the target interface element (22) selected processor is further configured to:
by the selecting operation is a target image, de-
termine image data of the target image as the control the display to display a search-adjusting

22
 43 EP 3 407 221 B1 44

element in a superposed manner on the local network service.

region of the user interface (21, 131);
receive a triggering signal triggered on the 12. The terminal (120) according to claim 1, wherein the
search-adjusting element; processor is configured to:
control the display to display a candidate item 5
exhibiting window (153) corresponding to the when the content information of the target inter-
search-adjusting element according to the trig- face element (22) comprises a title of film and
gering signal, wherein the candidate item exhib- television work, instruct the display to display
iting window (153) comprises m candidate items the searching result information (23) that com-
corresponding to the target interface element 10 prises at least one of following items: introduc-
(22), wherein m is a positive integer; tion information of the film and television work;
receive a second selecting operation for at least rating information of the film and television work;
one of the candidate items, wherein the second author information of the film and television
selecting operation is configured to select a tar- work; a subscribing entrance of the film and tel-
get candidate item; 15 evision work in a local application program or a
determine a searching keyword according to the network service; a sharing entrance of the film
target candidate item; and and television work in the local application pro-
control the display to display searching result gram or the network service; a favorites en-
information (23) corresponding to the searching trance of the film and television work in the local
keyword. 20 application program; and a ticket-booking en-
trance of the film and television work in the net-
11. The terminal (120) according to claim 1, wherein at work service;
least one piece of first searching result information when the content information of the target inter-
existed in the searching result information (23) com- face element (22) comprises a title of literature
prises: an invoking interface configured to invoke a 25 work, instruct the display to display the search-
local application program, wherein an input param- ing result information (23) that comprises at least
eter of the invoking interface comprises a searching one of following items: introduction information
keyword corresponding to the target interface ele- of the literature work; rating information of the
ment (22); and literature work; author information of the litera-
30 ture work; a subscribing entrance of the litera-
wherein the processor is further configured to: ture work in the local application program; a
sharing entrance of the literature work in the lo-
receive a third selecting operation triggered cal application program; a favorites entrance of
on the first searching result information, the literature work in the local application pro-
wherein the third selecting operation is con- 35 gram; a reading experience entrance of the lit-
figured to trigger invoking the local applica- erature work in the network service; and a pur-
tion program by the invoking interface to chasing entrance of the literature work in the net-
process the searching keyword; and work service;
control the display to display a user interface when the content information of the target inter-
(21, 131) of the local application program, 40 face element (22) comprises information of a
wherein the user interface (21, 131) com- person, instruct the display to display the
prises a processing result obtained through searching result information (23) that comprises
processing the searching keyword by the at least one of following items: WIKI information
local application program; of the person; a social network service
45 homepage of the person in the network service;
wherein the processor is further configured to: latest news and trends of the person in the net-
work service; a dynamic subscribing entrance
receive a fourth selecting operation trig- of the person in the network service; a sharing
gered on the second searching result infor- entrance of the person in the local application
mation, wherein the fourth selecting opera- 50 program or the network service; and a favorites
tion is configured to trigger invoking the net- entrance of the person in the local application
work service by the invoking interface to program or the network service;
process the searching keyword; and when the content information of the target inter-
control the display to display a user interface face element (22) comprises information of a
(21, 131) of the network service, wherein 55 site, instruct the display to display the searching
the user interface (21, 131) comprises a result information (23) that comprises at least
processing result obtained through one of following items: introduction information
processing the searching keyword by the of the site; navigation information of the site in

23
 45 EP 3 407 221 B1 46

the local application program or the network gered on the user interface (21, 131);
service; map preview information of the site in selecting (S203) a target interface element (22)
the local application program or the network from the at least one interface element accord-
service; a taxi-calling entrance with the site as ing to the first selecting operation; and
a destination; a sharing entrance of the site in 5 displaying (S204) searching result information
the local application program or the network (23) in a superposed manner on a local region
service; and a favorites entrance of the site in of the user interface (21, 131) by the operating
the local application program or the network system, the searching result information (23)
service; corresponding to the target interface element
when the content information of the target inter- 10 (22);
face element (22) comprises information of a wherein searching result information (23) is pro-
merchant, instruct the display to display the vided in an amount of n, in which at least two
searching result information (23) that comprises pieces of searching result information are differ-
at least one of following items: introduction in- ent from each other, and n is a positive integer;
formation of the merchant; a social network serv- 15 the method further comprising, before display-
ice homepage of the merchant in the network ing searching result information (23) in a super-
service; latest news and trends of the merchant posed manner on a local region of the user in-
in the network service; a preferential subscribing terface (21, 131) by the operating system:
entrance of the merchant in the network service;
contact information of the merchant; navigation 20 obtaining content information of the target
information of the merchant in the local applica- interface element (22) according to the first
tion program or the network service; map pre- selecting operation;
view information of the merchant in the local ap- sending a searching request to a server
plication program or the network service; a taxi- (140), wherein the searching request com-
calling entrance with the merchant as a destina- 25 prises the content information of the target
tion; a sharing entrance of the merchant in the interface element (22); and
local application program or the network service; receiving the searching result information
and a favorites entrance of the merchant in the (23) fed back by the server (140),
local application program or the network service; wherein obtaining content information of the
and 30 target interface element (22) according to
when the content information of the target inter- the first selecting operation comprises:
face element (22) comprises information of a
commodity, instruct the display to display the determining the target interface ele-
searching result information (23) that comprises ment (22) according to an operating co-
at least one of following items: introduction in- 35 ordinate of the first selecting operation,
formation of the commodity; a purchasing en- and capturing content information of
trance of the commodity in the network service; the target interface element (22);
price information of the commodity in at least wherein the method further comprises
one shopping channel; a preferential subscrib- before determining the target interface
ing entrance of the commodity in the network 40 element (22) according to an operating
service; a discount reminding entrance of the coordinate of the first selecting opera-
commodity in the network service; a history price tion and capturing content information
trend checking entrance of the commodity in the of the target interface element (22):
local application program or the network service;
a sharing entrance of the commodity in the local 45 obtain context information of the
application program or the network service; and application program, wherein the
a favorites entrance of the commodity in the local application program is an applica-
application program or the network service. tion program running in the fore-
ground, the user interface (21, 131)
13. An information display method applied in a terminal 50 is a user interface running in the
(120) installed with an operating system and an ap- foreground of the application pro-
plication program, the method comprising: gram, the context information com-
prises the identifier of the applica-
displaying (S201) a user interface (21, 131) of tion program and/or the identifier
the application program, wherein the user inter- 55 of the user interface; the operating
face (21, 131) comprises at least one interface system is provided with an active
element; stack in which an activity located
receiving (S202) a first selecting operation trig- at a stack top is corresponding to

24
 47 EP 3 407 221 B1 48

the user interface (21, 131) of the vided in an amount of n, in which at least two
application program running in the pieces of searching result information are differ-
foreground; a searching program ent from each other, and n is a positive integer;
obtains an identifier of the applica- the acts further comprising, before displaying
tion program running in the fore- 5 searching result information (23) in a super-
ground and an identifier of the user posed manner on a local region of the user in-
interface from the activity located terface (21, 131) by the operating system:
at the stack top;
detecting whether context informa- obtaining content information of the target
tion of the application program sat- 10 interface element (22) according to the first
isfies a capturing condition accord- selecting operation;
ing to a preset list that comprises sending a searching request to a server
a preset white list and/or a preset (140), wherein the searching request com-
black list, the preset list being prises the content information of the target
stored with context information sat- 15 interface element (22); and
isfying the capturing condition receiving the searching result information
and/or context information not sat- (23) fed back by the server (140),
isfying the capturing condition; and wherein obtaining content information of the
if the context information satisfies target interface element (22) according to
the capturing condition that indi- 20 the first selecting operation comprises:
cates that the application program
where the context information lo- determining the target interface ele-
cates belongs to the preset white ment (22) according to an operating co-
list, or doesn’t belong to the preset ordinate of the first selecting operation,
black list, then performing a step of 25 and capturing content information of
determining the target interface el- the target interface element (22);
ement (22) according to an operat- wherein the acts further comprise be-
ing coordinate of the first selecting fore determining the target interface el-
operation and capturing content in- ement (22) according to an operating
formation of the target interface el- 30 coordinate of the first selecting opera-
ement (22); tion and capturing content information
if the context information doesn’t of the target interface element (22):
satisfy the capturing condition,
send acquisition failure information obtain context information of the
to the searching program.. 35 application program, wherein the
application program is an applica-
14. A non-transitory computer-readable storage medi- tion program running in the fore-
um, stored with at least one instruction executable ground, the user interface (21, 131)
by a processer of a terminal (120), the at least one is a user interface running in the
instruction, when executed, causes the processor to 40 foreground of the application pro-
perform acts comprising: gram, the context information com-
prises the identifier of the applica-
instructing a display to display a user interface tion program and/or the identifier
(21, 131) of the application program, wherein of the user interface; the operating
the user interface (21, 131) comprises at least 45 system is provided with an active
one interface element; stack in which an activity located
receiving a first selecting operation triggered on at a stack top is corresponding to
the user interface (21, 131); the user interface (21, 131) of the
selecting a target interface element (22) from application program running in the
the at least one interface element according to 50 foreground; a searching program
the first selecting operation; and obtains an identifier of the applica-
instructing a display to display searching result tion program running in the fore-
information (23) in a superposed manner on a ground and an identifier of the user
local region of the user interface (21, 131) by interface from the activity located
the operating system, the searching result infor- 55 at the stack top;
mation (23) corresponding to the target interface detecting whether context informa-
element (22); tion of the application program sat-
wherein searching result information (23) is pro- isfies a capturing condition accord-

25
 49 EP 3 407 221 B1 50

ing to a preset list that comprises wobei der Prozessor vor dem Anzeigen von Su-
a preset white list and/or a preset chergebnisinformationen (23) in einer überla-
black list, the preset list being gerten Weise auf einem lokalen Bereich der Be-
stored with context information sat- nutzerschnittstelle (21, 131) durch das Betriebs-
isfying the capturing condition 5 system weiter konfiguriert ist zum:
and/or context information not sat-
isfying the capturing condition; and Erhalten von Inhaltsinformationen des Ziel-
if the context information satisfies Schnittstellenelements (22) gemäß der ers-
the capturing condition that indi- ten Auswahloperation;
cates that the application program 10 Senden einer Suchanfrage an einen Server
where the context information lo- (140), wobei die Suchanfrage die Inhaltsin-
cates belongs to the preset white formationen des Ziel-Schnittstellen-Ele-
list, or doesn’t belong to the preset ments (22) umfasst; und
black list, then performing a step of Empfangen der vom Server (140) zurück-
determining the target interface el- 15 gesendeten Suchergebnisinformationen
ement (22) according to an operat- (23),
ing coordinate of the first selecting wobei das Erhalten von Inhaltsinformatio-
operation and capturing content in- nen des Ziel-Schnittstellenelements (22)
formation of the target interface el- gemäß der ersten Auswahloperation Fol-
ement (22) ; 20 gendes umfasst:
if the context information doesn’t
satisfy the capturing condition, Bestimmen des Ziel-Schnittstellenele-
send acquisition failure information ments (22) gemäß einer Betriebskoor-
to the searching program. dinate der ersten Auswahloperation
25 und Aufnehmen von Inhaltsinformatio-
nen des Ziel-Schnittstellenelements
Patentansprüche (22);
wobei der Prozessor vor dem Bestim-
1. Endgerät (120), das mit einem Betriebssystem und men des Ziel-Schnittstellenelements
einem Anwendungsprogramm installiert ist, wobei 30 (22) gemäß einer Betriebskoordinate
das Endgerät (120) einen Prozessor und einen Spei- der ersten Auswahloperation und dem
cher umfasst, wobei der Prozessor konfiguriert ist Aufnehmen von Inhaltsinformationen
zum: des Ziel-Schnittstellenelements (22)
weiter konfiguriert ist zum:
Steuern eines Displays zur Anzeige einer Be- 35
nutzerschnittstelle (21, 131) des Anwendungs- Erhalten von Kontextinformatio-
programms, wobei die Benutzerschnittstelle nen des Anwendungsprogramms,
(21, 131) mindestens ein Oberflächenelement wobei das Anwendungsprogramm
umfasst; ein Anwendungsprogramm ist, das
Empfangen einer ersten Auswahloperation, die 40 im Vordergrund läuft, die Benut-
auf der Benutzerschnittstelle (21, 131) ausge- zerschnittstelle (21, 131) eine Be-
löst wird; nutzerschnittstelle ist, die im Vor-
Auswählen eines Ziel-Schnittstellenelements dergrund des Anwendungspro-
(22) aus dem mindestens einen Schnittstellen- gramms läuft, die Kontextinforma-
element gemäß der ersten Auswahloperation; 45 tionen die Kennung des Anwen-
und dungsprogramms und/oder die
Steuern der Anzeige, um Suchergebnisinforma- Kennung der Benutzerschnittstelle
tionen (23) in einer überlagerten Weise auf ei- umfassen; das Betriebssystem mit
nem lokalen Bereich der Benutzerschnittstelle einem aktiven Stapel bereitgestellt
(21, 131) durch das Betriebssystem anzuzei- 50 ist, in dem eine Aktivität, die sich
gen, und wobei die Suchergebnisinformationen an einer Stapelspitze befindet, der
(23) dem Ziel-Schnittstellenelement (22) ent- Benutzerschnittstelle (21, 131) des
sprechen; im Vordergrund laufenden Anwen-
wobei Suchergebnisinformationen (23) in einer dungsprogramms entspricht; ein
Menge von n bereitgestellt werden, wobei min- 55 Suchprogramm eine Kennung des
destens zwei Teile der Suchergebnisinformati- im Vordergrund laufenden Anwen-
onen voneinander verschieden sind und n eine dungsprogramms und eine Ken-
positive ganze Zahl ist; und nung der Benutzerschnittstelle von

26
 51 EP 3 407 221 B1 52

der an der Stapelspitze befindli- die n Anzeigeelemente sequentiell entlang einer ers-
chen Aktivität erhält; Erkennen, ob ten Richtung in der Listenansichts-Steuereinheit an-
Kontextinformationen des Anwen- geordnet sind, und ein Ende eines i-ten Anzeigeele-
dungsprogramms eine Aufnahme- ments mit einem Kopf eines (i+1)-ten Anzeigeele-
bedingung gemäß einer voreinge- 5 ments verbunden ist, wobei i eine positive ganze
stellten Liste erfüllen, die eine vor- Zahl ist und i kleiner als n ist.
eingestellte weiße Liste und/oder
eine voreingestellte schwarze Lis- 4. Endgerät (120) nach Anspruch 1, wobei die lokale
te umfasst, wobei die voreinge- Region eine der folgenden Regionen einschließt:
stellte Liste mit Kontextinformatio- 10
nen gespeichert wird, die die Auf- einen unteren Bereich der Benutzerschnittstelle
nahmebedingung erfüllen, (21, 131);
und/oder Kontextinformationen, einen angrenzenden Bereich des Ziel-Schnitt-
die die Aufnahmebedingung nicht stellenelements (22); und
erfüllen; und 15 einen angrenzenden Bereich einer Arbeitsposi-
wenn die Kontextinformationen die tion der Auswahloperation.
Aufnahmebedingung erfüllen, die
angeben, dass das Anwendungs- 5. Endgerät (120) nach einem der Ansprüche 1-4, wo-
programm, in dem sich die Kontex- bei nach dem Anzeigen von Suchergebnisinforma-
tinformationen befinden, zu der 20 tionen (23) in einer überlagerten Weise auf einem
voreingestellten weißen Liste ge- lokalen Bereich der Benutzerschnittstelle (21, 131)
hört oder nicht zu der voreingestell- durch das Betriebssystem der Prozessor weiter kon-
ten schwarzen Liste gehört, dann figuriert ist zum:
Durchführen eines Schritts des Be-
stimmens des Ziel-Schnittstellene- 25 wenn eine erste Gleitoperation auf j-ten Sucher-
lements (22) gemäß einer Be- gebnisinformationen empfangen wird, wobei die
triebskoordinate der ersten Aus- erste Gleitoperation eine Operation des Glei-
wahloperation und des Aufneh- tens entlang einer ersten Richtung ist, Anzeigen
mens von Inhaltsinformationen der j-ten Suchergebnisinformationen in gleiten-
des Ziel-Schnittstellenelements 30 der Weise entlang der ersten Richtung durch
(22); Folgen der ersten Gleitoperation, um einen Teil
wenn die Kontextinformationen die der j-ten Suchergebnisinformationen, die aus
Aufnahmebedingung nicht erfül- dem lokalen Bereich herausgleiten, zu verber-
len, Senden von Erfassungsfehle- gen und einen Teil einer (j-1)ten Suchergebnis-
rinformationen an das Suchpro- 35 informationen anzuzeigen, die in den lokalen
gramm. Bereich hineingleiten, wobei j eine ganze Zahl
größer als 1 ist und j kleiner oder gleich n ist; und
2. Endgerät (120) nach Anspruch 1, wobei der Prozes- wenn eine zweite Gleitoperation auf k-ten Such-
sor weiter konfiguriert ist zum: ergebnisinformationen empfangen wird, wobei
Steuern der Anzeige, um n Anzeigeelemente in einer 40 die zweite Gleitoperation eine Operation des
überlagerten Weise auf einem lokalen Bereich der Gleitens entlang einer zweiten Richtung ist, wo-
Benutzerschnittstelle (21, 131) durch das Betriebs- bei die k-ten Suchergebnisinformationen glei-
system anzuzeigen, wobei n eine positive ganze tend entlang der zweiten Richtung angezeigt
Zahl ist, eine Anzeigeebene der Anzeigeelemente werden, indem der zweiten Gleitoperation ge-
höher ist als eine Anzeigeebene der Benutzer- 45 folgt wird, um einen Teil der k-ten Suchergebni-
schnittstelle (21, 131), und jedes der Anzeigeele- sinformationen zu verbergen, der aus dem lo-
mente konfiguriert ist, um Informationen von mindes- kalen Bereich herausgleitet, und einen Teil einer
tens einem Teil der Suchergebnisinformationen (23) (k+1)ten Suchergebnisinformationen anzuzei-
darauf anzuzeigen. gen, der in den lokalen Bereich hineingleitet, wo-
50 bei k eine ganze Zahl kleiner als n ist,
3. Endgerät (120) nach Anspruch 2, wobei der Prozes- wobei die erste Richtung der zweiten Richtung
sor weiter konfiguriert ist zum: entgegengesetzt ist.
Steuern der Anzeige, um eine Listenansichts-Steu-
ereinheit in einer überlagerten Weise auf einem lo- 6. Endgerät (120) nach einem der vorstehenden An-
kalen Bereich der Benutzerschnittstelle (21, 131) 55 sprüche, wobei das Betriebssystem umfasst: ein
durch das Betriebssystem anzuzeigen, wobei die Bildschirmsteuerungsprogramm; ein Suchpro-
Listenansichts-Steuereinheit n Anzeigeelemente gramm und ein Inhaltsaufnahmeprogramm; und wo-
umfasst, wobei n eine ganze Zahl größer als 1 ist, bei das Bestimmen des Ziel-Schnittstellen-elements

27
 53 EP 3 407 221 B1 54

(22) gemäß einer Betriebskoordinate der ersten Referenzposition verwendet wird;

Auswahloperation und das Aufnehmen von Inhalts- Aufnehmen eines Bildschirmbildes des Schnitt-
informationen des Ziel-Schnittstellenelements (22) stellenelements in der Bereichsauswahlsteuer-
umfasst: einheit (30), um ein Zielbild zu erhalten, nach-
5 dem ein Bestätigungssignal empfangen wurde;
Melden einer Betriebskoordinate der ersten und
Auswahloperation an das Suchprogramm durch Bestimmen des Zielbildes als die Inhaltsinfor-
das Bildschirmsteuerungsprogramm; mationen des Ziel-Schnittstellenelements (22);
Senden einer Schnittstellenelement-Erfas- oder Erhalten eines Erkennungsergebnisses
sungsanforderung an das Inhaltsaufnahmepro- 10 entsprechend dem Zielbild und Bestimmen des
gramm durch das Suchprogramm, wobei die Erkennungsergebnisses als die Inhaltsinforma-
Schnittstellenelement-Erfassungsanforderung tionen des Ziel-Schnittstellenelements (22).
mit der Betriebskoordinate übertragen wird;
Aufnehmen von Inhaltsinformationen des Ziel- 8. Endgerät (120) nach Anspruch 7, wobei der Prozes-
Schnittstellenelements (22), dem die Be- 15 sor weiter konfiguriert ist zum:
triebskoordinate entspricht, durch das Inhalts-
aufnahmeprogramm; und Durchführen einer optischen Zeichenerken-
Senden der Inhaltsinformationen des Ziel- nung (OCR) oder einer Bilderkennung an dem
Schnittstellenelements (22) an das Suchpro- Zielbild, um ein Erkennungsergebnis zu erhal-
gramm durch das Inhaltsaufnahmeprogramm; 20 ten, das dem Zielbild entspricht; und
wobei das Betriebssystem umfasst: ein Such- Senden des Zielbildes an einen Bildverarbei-
programm und ein Inhaltsaufnahmeprogramm; tungsserver; und Empfangen des dem Zielbild
und wobei das Erkennen, ob die Kontextinfor- entsprechenden Erkennungsergebnisses, das
mationen des Anwendungsprogramms eine von dem Bildverarbeitungsserver zurückgesen-
Aufnahmebedingung erfüllen, Folgendes um- 25 det wird.
fasst:
9. Endgerät (120) nach einem vorstehenden Anspruch,
Erhalten von Kontextinformationen des An- wobei der Prozessor weiter konfiguriert ist zum:
wendungsprogramms durch das Suchpro-
gramm, wobei die Kontextinformationen ei- 30 Erhalten von Kontextinformationen des Anwen-
ne Kennung des Anwendungsprogramms dungsprogramms, wobei die Kontextinformati-
und/oder eine Kennung der Benutzer- onen eine Kennung des Anwendungspro-
schnittstelle (21, 131) umfassen; gramms und/oder eine Kennung der Benutzer-
Erzeugen der Anforderung zur Erfassung schnittstelle (21, 131) umfassen; und
des Schnittstellenelements, die die Be- 35 Senden einer Suchanfrage an den Server (140),
triebskoordinate und die Kontextinformati- wobei die Suchanfrage Inhaltsinformationen
onen umfasst; und des Ziel-Schnittstellenelements (22) und die
Erkennen, ob die Kontextinformationen die Kontextinformationen umfasst;
Aufnahmebedingung durch das Inhaltsauf- wobei der Server (140) so konfiguriert ist, dass
nahmeprogramm erfüllen. 40 er mindestens ein Kandidatenelement gemäß
den Inhaltsinformationen erhält, einen Suchbe-
7. Endgerät (120) nach einem vorstehenden Anspruch, griff aus dem mindestens einen Kandidatenele-
wobei der Prozessor weiter konfiguriert ist zum: ment gemäß den Kontextinformationen be-
stimmt und die Suchergebnisinformationen (23)
wenn das durch den Auswahlvorgang ausge- 45 durch Suchen gemäß dem Suchbegriff erhält.
wählte Ziel-Schnittstellenelement (22) ein Ziel-
bild ist, Bestimmen von Bilddaten des Zielbildes 10. Endgerät (120) nach Anspruch 1, wobei der Prozes-
als die Inhaltsinformationen des Ziel-Schnitt- sor weiter konfiguriert ist zum:
stellenelements (22);
wenn das durch den Auswahlvorgang ausge- 50 Steuern der Anzeige, um ein Sucheinstellungs-
wählte Ziel-Schnittstellenelement (22) das Ziel- element in einer überlagerten Weise auf dem
bild ist, Erhalten von einem dem Zielbild entspre- lokalen Bereich der Benutzerschnittstelle (21,
chenden Erkennungsergebnis und Bestimmen 131) anzuzeigen;
des Erkennungsergebnisses als Inhaltsinforma- Empfangen eines Auslösesignals, das auf dem
tionen des Ziel-Schnittstellenelements (22); 55 Sucheinstellungselement ausgelöst wird;
Steuern der Anzeige, um eine Bereichsaus- Steuern der Anzeige, um ein Kandidatenele-
wahl-Steuereinheit (30) anzuzeigen, indem eine ment-Anzeigefenster (153) anzuzeigen, das
Betriebsposition der Auswahloperation als eine dem Sucheinstellungselement gemäß dem

28
 55 EP 3 407 221 B1 56

Auslösesignal entspricht, wobei das Kandida- umfasst, das durch die Verarbeitung des
tenelement-Anzeigefenster (153) Kandidatene- Suchbegriffs durch den Netzwerkdienst er-
lemente umfasst, die dem Ziel-Schnittstellene- halten wurde.
lement (22) entsprechen, wobei m eine positive
ganze Zahl ist; 5 12. Endgerät (120) nach Anspruch 1, wobei der Prozes-
Empfangen einer zweiten Auswahloperation für sor konfiguriert ist zum:
mindestens eines der Kandidatenelemente, wo-
bei die zweite Auswahloperation so konfiguriert wenn die Inhaltsinformationen des Ziel-Schnitt-
ist, dass sie ein Zielkandidatenelement aus- stellenelements (22) einen Titel eines Film- und
wählt; 10 Fernsehwerks umfassen, Anweisen der Anzei-
Bestimmen eines Suchbegriffs entsprechend ge, die Suchergebnisinformationen (23) anzu-
dem Zielkandidatenelement; und zeigen, die mindestens eines der folgenden Ele-
Steuern des Displays zur Anzeige von Sucher- mente umfassen: Einführungsinformationen
gebnisinformationen (23), die dem Suchbegriff des Film- und Fernsehwerkes; Bewertungsin-
entsprechen. 15 formationen des Film- und Fernsehwerkes; Au-
toreninformationen des Film- und Fernsehwer-
11. Endgerät (120) nach Anspruch 1, wobei mindestens kes; einen Abonnement-Eingang des Film- und
ein Teil der ersten Suchergebnisinformationen, die Fernsehwerkes in einem lokalen Anwendungs-
in den Suchergebnisinformationen (23) vorhanden programm oder einem Netzwerkdienst; einen
sind, Folgendes umfasst: eine Aufrufschnittstelle, 20 Freigabe-Eingang des Film- und Fernsehwer-
die so konfiguriert ist, dass sie ein lokales Anwen- kes in dem lokalen Anwendungsprogramm oder
dungsprogramm aufruft, wobei ein Eingabeparame- dem Netzwerkdienst; einen Favoriten-Eingang
ter der Aufrufschnittstelle einen Suchbegriff umfasst, des Film- und Fernsehwerkes in dem lokalen
das dem Ziel-Schnittstellenelement (22) entspricht; Anwendungsprogramm; und einen Ticket-
und 25 Booking-Eingang des Film- und Fernsehwerkes
in dem Netzwerkdienst;
wobei der Prozessor weiter konfiguriert ist zum: wenn die Inhaltsinformationen des Ziel-Schnitt-
stellenelements (22) einen Titel eines literari-
Empfangen einer dritten Auswahloperation, schen Werks umfassen, Anweisen der Anzeige,
die durch die ersten Suchergebnisinforma- 30 die Suchergebnisinformationen (23) anzuzei-
tionen ausgelöst wird, wobei die dritte Aus- gen, die mindestens eines der folgenden Ele-
wahloperation so konfiguriert ist, dass sie mente umfassen: Einführungsinformationen
das Aufrufen des lokalen Anwendungspro- des literarischen Werks; Bewertungsinformati-
gramms durch die Aufrufschnittstelle aus- onen des literarischen Werks; Autoreninforma-
löst, um den Suchbegriff zu verarbeiten; 35 tionen des literarischen Werks; einen Abonne-
und menteingang des literarischen Werks in dem lo-
Steuern der Anzeige, um eine Benutzer- kalen Anwendungsprogramm; einen Freigabe-
schnittstelle (21, 131) des lokalen Anwen- Eingang des literarischen Werks in dem lokalen
dungsprogramms anzuzeigen, wobei die Anwendungsprogramm; einen Favoriten-Ein-
Benutzerschnittstelle (21, 131) ein Verar- 40 gang des literarischen Werks in dem lokalen An-
beitungsergebnis umfasst, das durch die wendungsprogramm; einen Leseerfahrungs-
Verarbeitung des Suchbegriffs durch das Eingang des literarischen Werks in dem Netz-
lokale Anwendungsprogramm erhalten werkdienst; und einen Kauf-Eingang des litera-
wurde; rischen Werks in dem Netzwerkdienst;
45 wenn die Inhaltsinformationen des Ziel-Schnitt-
wobei der Prozessor weiter konfiguriert ist zum: stellenelements (22) Informationen über eine
Person umfassen, Anweisen der Anzeige, die
Empfangen einer vierten Auswahloperati- Suchergebnisinformationen (23) anzuzeigen,
on, die durch die zweiten Suchergebnisin- die mindestens eines der folgenden Elemente
formationen ausgelöst wird, wobei die vierte 50 umfassen: WIKI-Informationen der Person; eine
Auswahloperation so konfiguriert ist, dass soziale Netzwerkdienst-Homepage der Person
sie das Aufrufen des Netzwerkdienstes in dem Netzwerkdienst; neueste Nachrichten
durch die Aufrufschnittstelle auslöst, um und Trends der Person in dem Netzwerkdienst;
den Suchbegriff zu verarbeiten; und einen dynamischen Anmeldungszugang der
Steuern der Anzeige, um eine Benutzer- 55 Person in dem Netzwerkdienst; einen Freigabe-
schnittstelle (21, 131) des Netzwerkdiens- zugang der Person in dem lokalen Anwen-
tes anzuzeigen, wobei die Benutzerschnitt- dungsprogramm oder dem Netzwerkdienst; und
stelle (21, 131) ein Verarbeitungsergebnis einen Favoritenzugang der Person in dem loka-

29
 57 EP 3 407 221 B1 58

len Anwendungsprogramm oder dem Netz- Anwendungsprogramm oder im Netzwerk-

werkdienst; dienst.
wenn die Inhaltsinformationen des Ziel-Schnitt-
stellenelements (22) Informationen eines Ortes 13. Informationsanzeigeverfahren, das in einem Endge-
umfassen, Anweisen der Anzeige, die Sucher- 5 rät (120) angewendet wird, das mit einem Betriebs-
gebnisinformationen (23) anzuzeigen, die min- system und einem Anwendungsprogramm installiert
destens eines der folgenden Elemente umfas- ist, wobei das Verfahren umfasst:
sen: Einführungsinformationen des Ortes; Na-
vigationsinformationen des Ortes in dem lokalen Anzeigen (S201) einer Benutzerschnittstelle
Anwendungsprogramm oder dem Netzwerk- 10 (21, 131) des Anwendungsprogramms, wobei
dienst; Kartenvorschauminformationen des Or- die Benutzerschnittstelle (21, 131) mindestens
tes in dem lokalen Anwendungsprogramm oder ein Oberflächenelement umfasst;
dem Netzwerkdienst; einen Taxiruf-Eingang mit Empfangen (S202) einer ersten Auswahlopera-
dem Ort als Bestimmungsort; einen Freigabe- tion, die auf der Benutzerschnittstelle (21, 131)
Eingang des Ortes in dem lokalen Anwendungs- 15 ausgelöst wurde;
programm oder dem Netzwerkdienst; und einen Auswählen (S203) eines Ziel-Schnittstellenele-
Favoriten-Eingang des Ortes in dem lokalen An- ments (22) aus dem mindestens einen Schnitt-
wendungsprogramm oder dem Netzwerkdienst; stellenelement gemäß der ersten Auswahlope-
wenn die Inhaltsinformationen des Ziel-Schnitt- ration; und
stellenelements (22) Informationen eines Händ- 20 Anzeigen (S204) von Suchergebnisinformatio-
lers umfassen, Anweisen der Anzeige, die Su- nen (23) in einer überlagerten Weise auf einem
chergebnisinformationen (23) anzuzeigen, die lokalen Bereich der Benutzerschnittstelle (21,
mindestens eines der folgenden Elemente um- 131) durch das Betriebssystem, wobei die Su-
fassen: Einführungsinformationen des Händ- chergebnisinformationen (23) dem Ziel-Schnitt-
lers; eine soziale Netzwerkdienst-Homepage 25 stellenelement (22) entsprechen;
des Händlers in dem Netzwerkdienst; neueste wobei Suchergebnisinformationen (23) in einer
Nachrichten und Trends des Händlers in dem Menge von n bereitgestellt werden, wobei min-
Netzwerkdienst; einen bevorzugten Abonnem- destens zwei Teile der Suchergebnisinformati-
entzugang des Händlers in dem Netzwerk- onen voneinander verschieden sind und n eine
dienst; Kontaktinformationen des Händlers; Na- 30 positive ganze Zahl ist;
vigationsinformationen des Händlers in dem lo- wobei das Verfahren weiter umfasst, vor dem
kalen Anwendungsprogramm oder dem Netz- Anzeigen von Suchergebnisinformationen (23)
werkdienst; Kartenvorschauminformationen in einer überlagerten Weise auf einem lokalen
des Händlers in dem lokalen Anwendungspro- Bereich der Benutzerschnittstelle (21, 131)
gramm oder dem Netzwerkdienst; einen Taxi- 35 durch das Betriebssystem:
ruf-Eingang mit dem Händler als Bestimmungs-
ort; einen Sharing-Eingang des Händlers in dem Erhalten von Inhaltsinformationen des Ziel-
lokalen Anwendungsprogramm oder dem Netz- Schnittstellenelements (22) gemäß der ers-
werkdienst; und einen Favoriten-Eingang des ten Auswahloperation;
Händlers in dem lokalen Anwendungspro- 40 Senden einer Suchanfrage an einen Server
gramm oder dem Netzwerkdienst; und (140), wobei die Suchanfrage die Inhaltsin-
wenn die Inhaltsinformationen des Ziel-Schnitt- formationen des Ziel-Schnittstellen-Ele-
stellenelements (22) Informationen über eine ments (22) umfasst; und
Ware umfassen, Anweisen der Anzeige, die Su- Empfangen der vom Server (140) zurück-
chergebnisinformationen (23) anzuzeigen, die 45 gesendeten Suchergebnisinformationen
mindestens eines der folgenden Elemente um- (23),
fassen: Einführungsinformationen der Ware; ei- wobei das Erhalten von Inhaltsinformatio-
nen Kaufeingang der Ware im Netzwerkdienst; nen des Ziel-Schnittstellenelements (22)
Preisinformationen der Ware in mindestens ei- gemäß der ersten Auswahloperation Fol-
nem Einkaufskanal; einen bevorzugten Abon- 50 gendes umfasst:
nementeingang der Ware im Netzwerkdienst;
einen Rabatt-Erinnerungseingang der Ware im Bestimmen des Ziel-Schnittstellenele-
Netzwerkdienst; einen Verlaufspreis-Trendprü- ments (22) gemäß einer Betriebskoor-
fungseingang der Ware im lokalen Anwen- dinate der ersten Auswahloperation
dungsprogramm oder im Netzwerkdienst; einen 55 und Aufnehmen von Inhaltsinformatio-
Freigabe-Eingang der Ware im lokalen Anwen- nen des Ziel-Schnittstellenelements
dungsprogramm oder im Netzwerkdienst; und (22);
einen Favoriten-Eingang der Ware im lokalen wobei das Verfahren weiter umfasst,

30
 59 EP 3 407 221 B1 60

vor der Bestimmung des Ziel-Schnitt- wahloperation und Aufnehmen

stellenelements (22) gemäß einer Be- von Inhaltsinformationen des Ziel-
triebskoordinate der ersten Auswahlo- Schnittstellenelements (22);
peration und des Aufnehmens von In- wenn die Kontextinformationen die
haltsinformationen des Ziel-Schnitt- 5 Aufnahmebedingung nicht erfül-
stellenelements (22): len, Senden einer Erfassungsfeh-
lerinformation an das Suchpro-
Erhalten von Kontextinformatio- gramm.
nen des Anwendungsprogramms,
wobei das Anwendungsprogramm 10 14. Nicht-transitorisches computerlesbares Speicher-
ein im Vordergrund laufendes An- medium, auf dem mindestens eine Anweisung ge-
wendungsprogramm ist, die Be- speichert ist, der von einem Prozessor eines Termi-
nutzerschnittstelle (21, 131) eine nals (120) ausgeführt werden kann, wobei die min-
im Vordergrund des Anwendungs- destens eine Anweisung, wenn sie ausgeführt wird,
programms laufende Benutzer- 15 den Prozessor veranlasst, Handlungen auszufüh-
schnittstelle ist, die Kontextinfor- ren, die Folgendes umfassen:
mationen die Kennung des An-
wendungsprogramms und/oder Anweisen einer Anzeige, eine Benutzerschnitt-
die Kennung der Benutzerschnitt- stelle (21, 131) des Anwendungsprogramms an-
stelle umfassen; das Betriebssys- 20 zuzeigen, wobei die Benutzerschnittstelle (21,
tem mit einem aktiven Stapel be- 131) mindestens ein Schnittstellenelement um-
reitgestellt ist, in dem eine Aktivität, fasst;
die sich an einer Stapelspitze be- Empfangen einer ersten Auswahloperation, der
findet, der Benutzerschnittstelle auf der Benutzerschnittstelle (21, 131) ausge-
(21, 131) des im Vordergrund lau- 25 löst wurde;
fenden Anwendungsprogramms Auswählen eines Ziel-Schnittstellenelements
entspricht; ein Suchprogramm ei- (22) aus dem mindestens einen Schnittstellen-
ne Kennung des im Vordergrund element gemäß der ersten Auswahloperation;
laufenden Anwendungspro- und
gramms und eine Kennung der Be- 30 Anweisen einer Anzeige, Suchergebnisinforma-
nutzerschnittstelle aus der Aktivi- tionen (23) in einer überlagerten Weise auf ei-
tät, die sich an der Stapelspitze be- nem lokalen Bereich der Benutzerschnittstelle
findet, erhält; (21, 131) durch das Betriebssystem anzuzei-
Erkennen, ob Kontextinformatio- gen, wobei die Suchergebnisinformationen (23)
nen des Anwendungsprogramms 35 dem Ziel-Schnittstellenelement (22) entspre-
eine Aufnahmebedingung gemäß chen;
einer voreingestellten Liste erfül- wobei Suchergebnisinformationen (23) in einer
len, die eine voreingestellte weiße Menge von n bereitgestellt werden, wobei min-
Liste und/oder eine voreingestellte destens zwei Teile der Suchergebnisinformati-
schwarze Liste umfasst, wobei die 40 onen voneinander verschieden sind und n eine
voreingestellte Liste mit Kontextin- positive ganze Zahl ist;
formationen, die die Aufnahmebe- wobei die Handlungen weiter umfassen, vor
dingung erfüllen, und/oder Kontex- dem Anzeigen von Suchergebnisinformationen
tinformationen, die die Aufnahme- (23) in einer überlagerten Weise auf einem lo-
bedingung nicht erfüllen, gespei- 45 kalen Bereich der Benutzerschnittstelle (21,
chert wird; und 131), durch das Betriebssystem:
wenn die Kontextinformationen die
Aufnahmebedingung erfüllen, die Erhalten von Inhaltsinformationen des Ziel-
angeben, dass das Anwendungs- Schnittstellenelements (22) gemäß der ers-
programm, in dem sich die Kontex- 50 ten Auswahloperation;
tinformationen befinden, zu der Senden einer Suchanfrage an einen Server
voreingestellten weißen Liste ge- (140), wobei die Suchanfrage die Inhaltsin-
hört oder nicht zu der voreingestell- formationen des Ziel-Schnittstellen-Ele-
ten schwarzen Liste gehört, dann ments (22) umfasst; und
Durchführen eines Schritts des Be- 55 Empfangen der vom Server (140) zurück-
stimmens des Ziel-Schnittstellene- gesendeten Suchergebnisinformationen
lements (22) gemäß einer Be- (23),
triebskoordinate der ersten Aus- wobei das Erhalten von Inhaltsinformatio-

31
 61 EP 3 407 221 B1 62

nen des Ziel-Schnittstellenelements (22) Aufnahmebedingung erfüllen, die

gemäß der ersten Auswahloperation Fol- angeben, dass das Anwendungs-
gendes umfasst: programm, in dem sich die Kontex-
tinformationen befinden, zu der
Bestimmen des Ziel-Schnittstellenele- 5 voreingestellten weißen Liste ge-
ments (22) gemäß einer Betriebskoor- hört oder nicht zu der voreingestell-
dinate der ersten Auswahloperation ten schwarzen Liste gehört, dann
und Aufnehmen von Inhaltsinformatio- Durchführen eines Schritts des Be-
nen des Ziel-Schnittstellenelements stimmens des Ziel-Schnittstellen-
(22); 10 elements (22) gemäß einer Be-
wobei die Handlungen weiter umfas- triebskoordinate der ersten Aus-
sen, vor dem Bestimmen des Ziel- wahloperation und Aufnehmen
Schnittstellenelements (22) gemäß ei- von Inhaltsinformationen des Ziel-
ner Betriebskoordinate der ersten Aus- Schnittstellenelements (22);
wahloperation und dem Aufnehmen 15 wenn die Kontextinformationen die
von Inhaltsinformationen des Ziel- Aufnahmebedingung nicht erfül-
Schnittstellenelements (22): len, Senden von Erfassungsfehle-
rinformationen an das Suchpro-
Erhalten von Kontextinformatio- gramm.
nen des Anwendungsprogramms, 20
wobei das Anwendungsprogramm
ein im Vordergrund laufendes An- Revendications
wendungsprogramm ist, die Be-
nutzerschnittstelle (21, 131) eine 1. Terminal (120) sur lequel sont installés système
im Vordergrund des Anwendungs- 25 d’exploitation et un programme d’application, le ter-
programms laufende Benutzer- minal (120) comprenant un processeur et une mé-
schnittstelle ist, die Kontextinfor- moire, dans lequel le processeur est configuré pour :
mationen die Kennung des An-
wendungsprogramms und/oder commander un dispositif d’affichage pour affi-
die Kennung der Benutzerschnitt- 30 cher une interface utilisateur (21, 131) du pro-
stelle umfassen; das Betriebssys- gramme d’application, l’interface utilisateur (21,
tem mit einem aktiven Stapel be- 131) comprenant au moins un élément
reitgestellt ist, in dem eine Aktivität, d’interface ;
die sich an einer Stapelspitze be- recevoir une première opération de sélection
findet, der Benutzerschnittstelle 35 déclenchée sur l’interface utilisateur (21, 131) ;
(21, 131) des im Vordergrund lau- sélectionner un élément d’interface cible (22) à
fenden Anwendungsprogramms partir de l’au moins un élément d’interface selon
entspricht; ein Suchprogramm ei- la première opération de sélection ; et
ne Kennung des im Vordergrund commander le dispositif d’affichage pour affi-
laufenden Anwendungspro- 40 cher des informations de résultat de recherche
gramms und eine Kennung der Be- (23) de manière superposée sur une région lo-
nutzerschnittstelle aus der Aktivi- cale de l’interface utilisateur (21, 131) à l’aide
tät, die sich an der Stapelspitze be- du système d’exploitation, et les informations de
findet, erhält; résultat de recherche (23) correspondent à l’élé-
Erkennen, ob Kontextinformatio- 45 ment d’interface cible (22) ;
nen des Anwendungsprogramms dans lequel les informations de résultat de re-
eine Aufnahmebedingung gemäß cherche (23) sont fournies en une quantité de
einer voreingestellten Liste erfül- n, dans lesquelles au moins deux parties d’in-
len, die eine voreingestellte weiße formations de résultat de recherche sont diffé-
Liste und/oder eine voreingestellte 50 rentes l’une de l’autre, et n est un nombre entier
schwarze Liste umfasst, wobei die positif ; et
voreingestellte Liste mit Kontextin- dans lequel, avant l’affichage des informations
formationen, die die Aufnahmebe- de résultat de recherche (23) de manière super-
dingung erfüllen, und/oder Kontex- posée sur une région locale de l’interface utili-
tinformationen, die die Aufnahme- 55 sateur (21, 131) à l’aide du système d’exploita-
bedingung nicht erfüllen, gespei- tion, le processeur est en outre configuré pour :
chert wird; und
wenn die Kontextinformationen die obtenir des informations de contenu de

32
 63 EP 3 407 221 B1 64

l’élément d’interface cible (22) selon la pre- prédéfinie qui comprend une liste
mière opération de sélection ; blanche prédéfinie et/ou une liste
envoyer une demande de recherche à un noire prédéfinie, la liste prédéfinie
serveur (140), dans lequel la demande de stockant dans celle-ci des informa-
recherche comprend les informations de 5 tions de contexte satisfaisant à la
contenu de l’élément d’interface cible (22) ; condition de capture et/ou des in-
et formations de contexte ne satisfai-
recevoir les informations de résultat de re- sant pas à la condition de capture ;
cherche (23) fournies en retour par le ser- et
veur (140), 10 si les informations de contexte sa-
dans lequel l’obtention d’informations de tisfont à la condition de capture, ce-
contenu de l’élément d’interface cible (22) ci indique que le programme d’ap-
selon la première opération de sélection plication où les informations de
comprend : contexte sont situées appartient à
15 la liste blanche prédéfinie, ou n’ap-
la détermination de l’élément d’interfa- partient pas à la liste noire prédé-
ce cible (22) selon une coordonnée finie, alors réaliser une étape de la
d’exploitation de la première opération détermination de l’élément d’inter-
de sélection, et la capture d’informa- face cible (22) selon une coordon-
tions de contenu de l’élément d’interfa- 20 née d’exploitation de la première
ce cible (22) ; opération de sélection et de la cap-
dans lequel, avant la détermination de ture d’informations de contenu de
l’élément d’interface cible (22) selon l’élément d’interface cible (22) ;
une coordonnée d’exploitation de la si les informations de contexte ne
première opération de sélection et la 25 satisfont pas à la condition de cap-
capture d’informations de contenu de ture, envoyer des informations
l’élément d’interface cible (22), le pro- d’échec d’acquisition au program-
cesseur est en outre configuré pour : me de recherche.

obtenir des informations de con- 30 2. Terminal (120) selon la revendication 1, dans lequel
texte du programme d’application, le processeur est en outre configuré pour :
dans lequel le programme d’appli- commander le dispositif d’affichage pour afficher n
cation est un programme d’appli- éléments d’affichage de manière superposée sur
cation en exécution dans l’avant- une région locale de l’interface utilisateur (21, 131)
plan, l’interface utilisateur (21, 35 à l’aide du système d’exploitation, dans lequel n est
131) est une interface utilisateur en un nombre entier positif, un niveau d’affichage des
exécution dans l’avant-plan du éléments d’affichage est plus haut qu’un niveau d’af-
programme d’application, les infor- fichage de l’interface utilisateur (21, 131), et chacun
mations de contexte comprennent des éléments d’affichage est configuré pour afficher
l’identifiant du programme d’appli- 40 des informations d’au moins une partie d’informa-
cation et/ou l’identifiant de l’interfa- tions de résultat de recherche (23) sur celui-ci.
ce utilisateur ; le système d’exploi-
tation est pourvu d’une pile active 3. Terminal (120) selon la revendication 2, dans lequel
dans laquelle une activité située en le processeur est en outre configuré pour :
haut d’une pile correspond à l’in- 45 commander le dispositif d’affichage pour afficher
terface utilisateur (21, 131) du pro- une unité de commande de vue de liste de manière
gramme d’application en exécu- superposée sur une région locale de l’interface uti-
tion dans l’avant-plan ; un pro- lisateur (21, 131) à l’aide du système d’exploitation,
gramme de recherche obtient un l’unité de commande de vue de liste comprenant n
identifiant du programme d’appli- 50 éléments d’affichage, dans lequel n est un nombre
cation en exécution dans l’avant- entier supérieur à 1, les n éléments d’affichage sont
plan et un identifiant de l’interface agencés séquentiellement le long d’une première di-
utilisateur à partir de l’activité si- rection dans l’unité de commande de vue de liste, et
tuée en haut de la pile ; détecter le une queue d’un iième élément d’affichage est con-
fait que des informations de con- 55 nectée à un en-tête d’un (i+1)ième élément d’afficha-
texte du programme d’application ge, dans lequel i est un nombre entier positif et i est
satisfont, ou ne satisfont pas, à une inférieur à n.
condition de capture selon une liste

33
 65 EP 3 407 221 B1 66

4. Terminal (120) selon la revendication 1, dans lequel lection et la capture d’informations de contenu de
la région locale inclut une des régions suivantes : l’élément d’interface cible (22) comprennent :

une région inférieure de l’interface utilisateur la transmission d’un rapport d’une coordonnée
(21, 131) ; 5 d’exploitation de la première opération de sélec-
une région adjacente de l’élément d’interface ci- tion au programme de recherche, par le pro-
ble (22) ; et gramme de pilotage d’écran ;
une région adjacente d’une position d’exploita- l’envoi d’une demande d’acquisition d’élément
tion de l’opération de sélection. d’interface au programme de capture de conte-
10 nu, par le programme de recherche, dans lequel
5. Terminal (120) selon l’une quelconque des revendi- la demande d’acquisition d’élément d’interface
cations 1-4, dans lequel, après l’affichage d’informa- est transportée avec la coordonnée
tions de résultat de recherche (23) de manière su- d’exploitation ;
perposée sur une région locale de l’interface utilisa- la capture d’informations de contenu de l’élé-
teur (21, 131) à l’aide du système d’exploitation, le 15 ment d’interface cible (22), auquel la coordon-
processeur est en outre configuré pour : née d’exploitation correspond, par le program-
me de capture de contenu ; et
lors de la réception d’une première opération de l’envoi des informations de contenu de l’élément
glisser sur une jième information de résultat de d’interface cible (22) au programme de recher-
recherche, la première opération de glisser 20 che, par le programme de capture de contenu ;
étant une opération consistant à glisser le long dans lequel le système d’exploitation
d’une première direction, l’affichage de la jième comprend : un programme de recherche et un
information de résultat de recherche de manière programme de capture de contenu ; et dans le-
glissante le long de la première direction en sui- quel la détection du fait que des informations de
vant la première opération de glisser, pour ca- 25 contexte du programme d’application satisfont,
cher une portion de la jième information de résul- ou ne satisfont pas, à une condition de capture
tat de recherche qui est en train de glisser pour comprend :
quitter la région locale et afficher une portion
d’une (j-1)ième information de résultat de recher- l’obtention d’informations de contexte du
che qui est en train de glisser pour entrer dans 30 programme d’application, par le program-
la région locale, dans lequel j est un nombre me de recherche, les informations de con-
entier supérieur à 1, et j est inférieur ou égal à texte comprenant un identifiant du program-
n ; et me d’application et/ou un identifiant de l’in-
lors de la réception d’une seconde opération de terface utilisateur (21, 131) ;
glisser sur une kième information de résultat de 35 la génération de la demande d’acquisition
recherche, la seconde opération de glisser est d’élément d’interface qui comprend la coor-
une opération consistant à glisser le long d’une donnée d’exploitation et les informations de
seconde direction, l’affichage de la kième infor- contexte ; et
mation de résultat de recherche de manière glis- la détection du fait que les informations de
sante le long de la seconde direction en suivant 40 contexte satisfont, ou ne satisfont pas, à la
la seconde opération de glisser, pour cacher une condition de capture, par le programme de
portion de la kième information de résultat de re- capture de contenu.
cherche qui est en train de glisser pour quitter
la région locale et afficher une portion d’une 7. Terminal (120) selon une quelconque revendication
(k+1)ième information de résultat de recherche 45 précédente, dans lequel le processeur est en outre
qui est en train de glisser pour entrer dans la configuré pour :
région locale, dans lequel k est un nombre entier
inférieur à n, lorsque l’élément d’interface cible (22) sélec-
dans lequel la première direction est opposée à tionné par l’opération de sélection est une image
la seconde direction. 50 cible, déterminer des données d’image de l’ima-
ge cible comme étant des informations de con-
6. Terminal (120) selon une quelconque revendication tenu de l’élément d’interface cible (22) ;
précédente, dans lequel le système d’exploitation lorsque l’élément d’interface cible (22) sélec-
comprend : un programme de pilotage d’écran ; un tionné par l’opération de sélection est l’image
programme de recherche et un programme de cap- 55 cible, obtenir un résultat de reconnaissance cor-
ture de contenu ; et dans lequel la détermination de respondant à l’image cible et la détermination
l’élément d’interface cible (22) selon une coordon- du résultat de reconnaissance comme étant les
née d’exploitation de la première opération de sé- informations de contenu de l’élément d’interface

34
 67 EP 3 407 221 B1 68

cible (22) ; cher un élément d’ajustement de recherche de

commander le dispositif d’affichage pour affi- manière superposée sur la région locale de l’in-
cher une unité de commande de sélection de terface utilisateur (21, 131) ;
région (30) en utilisant une position d’exploita- recevoir un signal de déclenchement, déclen-
tion de l’opération de sélection en tant que po- 5 ché sur l’élément d’ajustement de recherche ;
sition de référence ; commander le dispositif d’affichage pour affi-
capturer une représentation d’écran de l’élé- cher une fenêtre de présentation d’article can-
ment d’interface dans l’unité de commande de didat (153) correspondant à l’élément d’ajuste-
sélection de région (30) pour obtenir une image ment de recherche selon le signal de déclen-
cible lors de la réception d’un accusé de récep- 10 chement, dans lequel la fenêtre de présentation
tion signal ; et d’article candidat (153) comprend m articles
déterminer l’image cible comme étant les infor- candidats correspondant à l’élément d’interface
mations de contenu de l’élément d’interface ci- cible (22), dans lequel m est un nombre entier
ble (22) ; ou, obtenir un résultat de reconnais- positif ;
sance correspondant à l’image cible et la déter- 15 recevoir une deuxième opération de sélection
mination du résultat de reconnaissance comme pour au moins un des articles candidats, dans
étant les informations de contenu de l’élément lequel la deuxième opération de sélection est
d’interface cible (22). configurée pour sélectionner un article candidat
cible ;
8. Terminal (120) selon la revendication 7, dans lequel 20 déterminer un mot-clef de recherche selon l’ar-
le processeur est en outre configuré pour : ticle candidat cible ; et
commander le dispositif d’affichage pour affi-
réaliser une reconnaissance optique de carac- cher des informations de résultat de recherche
tères (OCR) ou une reconnaissance d’image sur (23) correspondant au mot-clef de recherche.
l’image cible pour obtenir un résultat de recon- 25
naissance correspondant à l’image cible ; et 11. Terminal (120) selon la revendication 1, dans lequel
envoyer l’image cible à un serveur de traitement au moins une partie de premières informations de
d’image ; et recevoir le résultat de reconnais- résultat de recherche existant dans les informations
sance correspondant à l’image cible fournie en de résultat de recherche (23) comprend : une inter-
retour par le serveur de traitement d’image. 30 face d’appel configurée pour appeler un programme
d’application local, dans lequel un paramètre d’en-
9. Terminal (120) selon une quelconque revendication trée de l’interface d’appel comprend un mot-clef de
précédente, dans lequel le processeur est en outre recherche correspondant à l’élément d’interface ci-
configuré pour : ble (22) ; et
35
obtenir des informations de contexte du pro- dans lequel le processeur est en outre configuré
gramme d’application, les informations de con- pour :
texte comprennent un identifiant du programme
d’application et/ou un identifiant de l’interface recevoir une troisième opération de sélec-
utilisateur (21, 131) ; et 40 tion déclenchée sur les première informa-
envoyer une demande de recherche au serveur tions de résultat de recherche, dans lequel
(140), la demande de recherche comprenant la troisième opération de sélection est con-
des informations de contenu de l’élément d’in- figurée pour déclencher l’appel du program-
terface cible (22) et les informations de me d’application local, à l’aide de l’interface
contexte ; 45 d’appel, pour traiter le mot-clef de
dans lequel le serveur (140) est configuré pour recherche ; et
obtenir au moins un article candidat selon les commander le dispositif d’affichage pour af-
informations de contenu, déterminer un mot-clef ficher une interface utilisateur (21, 131) du
de recherche à partir de l’au moins un article programme d’application local, dans lequel
candidat selon les informations de contexte, et 50 l’interface utilisateur (21, 131) comprend un
obtenir les informations de résultat de recherche résultat de traitement obtenu par le biais de
(23) en effectuant une recherche selon le mot- traitement du mot-clef de recherche par le
clef de recherche. programme d’application local ;

10. Terminal (120) selon la revendication 1, dans lequel 55 dans lequel le processeur est en outre configuré
le processeur est en outre configuré pour : pour :

commander le dispositif d’affichage pour affi- recevoir une quatrième opération de sélec-

35
 69 EP 3 407 221 B1 70

tion déclenchée sur les secondes informa- au dispositif d’affichage, d’afficher les informa-
tions de résultat de recherche, dans lequel tions de résultat de recherche (23) qui compren-
la quatrième opération de sélection est con- nent au moins un parmi les articles suivants :
figurée pour déclencher l’appel du service des informations WIKI de la personne ; une pa-
de réseau, à l’aide de l’interface d’appel, 5 ge d’accueil de service de réseau social de la
pour traiter le mot-clef de recherche ; et personne dans le service de réseau ; des actua-
commander le dispositif d’affichage pour af- lités et des tendances les plus récentes de la
ficher une interface utilisateur (21, 131) du personne dans le service de réseau ; une entrée
service de réseau, dans lequel l’interface d’abonnement dynamique de la personne dans
utilisateur (21, 131) comprend un résultat 10 le service de réseau ; une entrée de partage de
de traitement obtenu par le biais de traite- la personne dans le programme d’application lo-
ment du mot-clef de recherche par le servi- cal ou le service de réseau ; et une entrée de
ce de réseau. favoris de la personne dans le programme d’ap-
plication local ou le service de réseau ;
12. Terminal (120) selon la revendication 1, dans lequel 15 lorsque les informations de contenu de l’élément
le processeur est configuré pour : d’interface cible (22) comprend des informations
d’un site, transmettre l’instruction, au dispositif
lorsque les informations de contenu de l’élément d’affichage, d’afficher les informations de résul-
d’interface cible (22) comprennent un titre de tat de recherche (23) qui comprennent au moins
film et d’ouvrage télévisuel, transmettre l’ins- 20 un parmi les articles suivants : des informations
truction, au dispositif d’affichage, d’afficher les d’introduction du site ; des informations naviga-
informations de résultat de recherche (23) qui tion du site dans le programme d’application lo-
comprennent au moins un parmi les articles cal ou le service de réseau ; des informations
suivants : des informations d’introduction du film de prévisualisation de carte du site dans le pro-
et de l’ouvrage télévisuel ; des informations de 25 gramme d’application local ou du service de
classification du film et de l’ouvrage télévisuel ; réseau ; une entrée d’appel de taxi avec le site
des informations d’auteur du film et de l’ouvrage en tant que destination ; une entrée de partage
télévisuel ; une entrée d’abonnement du film et du site dans le programme d’application local
de l’ouvrage télévisuel dans un programme ou le service de réseau ; et une entrée de favoris
d’application local ou un service de réseau ; une 30 du site dans le programme d’application local
entrée de partage du film et de l’ouvrage télévi- ou le service de réseau ;
suel dans le programme d’application local ou lorsque les informations de contenu de l’élément
le service de réseau ; une entrée de favoris du d’interface cible (22) comprennent des informa-
film et de l’ouvrage télévisuel dans le program- tions d’un commerçant, transmettre l’instruc-
me d’application local ; et une entrée de réser- 35 tion, au dispositif d’affichage, d’afficher les in-
vation de billet du film et de l’ouvrage télévisuel formations de résultat de recherche (23) qui
dans le service de réseau ; comprennent au moins un parmi les articles
lorsque les informations de contenu de l’élément suivants : des informations d’introduction du
d’interface cible (22) comprennent un titre commerçant; une page d’accueil de service de
d’ouvrage littéraire, transmettre l’instruction, au 40 réseau social du commerçant dans le service
dispositif d’affichage, d’afficher les informations de réseau ; des actualités et des tendances les
de résultat de recherche (23) qui comprend au plus récentes du commerçant dans le service
moins un parmi les articles suivants : des infor- de réseau ; une entrée d’abonnement préféren-
mations d’introduction de l’ouvrage littéraire ; tiel du commerçant dans le service de réseau ;
des informations de classification de l’ouvrage 45 des informations de contact du commerçant ;
littéraire ; des informations d’auteur de l’ouvrage des informations de navigation du commerçant
littéraire ; une entrée d’abonnement de l’ouvra- dans le programme d’application local ou le ser-
ge littéraire dans le programme d’application vice de réseau ; des informations de prévisua-
local ; une entrée de partage de l’ouvrage litté- lisation de carte du commerçant dans le pro-
raire dans le programme d’application local ; 50 gramme d’application local ou le service de
une entrée de favoris de l’ouvrage littéraire dans réseau ; une entrée d’appel de taxi avec le com-
le programme d’application local ; une entrée merçant en tant que destination ; une entrée de
d’expérience de lecture de l’ouvrage littéraire partage du commerçant dans le programme
dans le service de réseau ; et une entrée d’achat d’application local ou le service de réseau ; et
de l’ouvrage littéraire dans le service de réseau ; 55 une entrée de favoris du commerçant dans le
lorsque les informations de contenu de l’élément programme d’application local ou le service de
d’interface cible (22) comprennent des informa- réseau ; et
tions d’une personne, transmettre l’instruction, lorsque les informations de contenu de l’élément

36
 71 EP 3 407 221 B1 72

d’interface cible (22) comprennent des informa- mière opération de sélection ;

tions d’une marchandise, transmettre l’instruc- l’envoi d’une demande de recherche à un
tion, au dispositif d’affichage, d’afficher les in- serveur (140), dans lequel la demande de
formations de résultat de recherche (23) qui recherche comprend les informations de
comprennent au moins un parmi les articles 5 contenu de l’élément d’interface cible (22) ;
suivants : des informations d’introduction de la et
marchandise ; une entrée d’achat de la mar- la réception des informations de résultat de
chandise dans le service de réseau ; des infor- recherche (23) fournies en retour par le ser-
mations de prix de la marchandise dans au veur (140),
moins une chaîne de téléachat; une entrée 10 dans lequel l’obtention d’informations de
d’abonnement préférentiel de la marchandise contenu de l’élément d’interface cible (22)
dans le service de réseau ; une entrée de rappel selon la première opération de sélection
de ristourne de la marchandise dans le service comprend :
de réseau ; une entrée de contrôle de tendance
de prix historique de la marchandise dans le pro- 15 la détermination de l’élément d’interfa-
gramme d’application local ou le service de ce cible (22) selon une coordonnée
réseau ; une entrée de partage de la marchan- d’exploitation de la première opération
dise dans le programme d’application local ou de sélection, et la capture d’informa-
le service de réseau ; et une entrée de favoris tions de contenu de l’élément d’interfa-
de la marchandise dans le programme d’appli- 20 ce cible (22) ;
cation local ou le service de réseau. dans lequel le procédé comprend en
outre, avant la détermination de l’élé-
13. Procédé d’affichage d’informations appliqué dans un ment d’interface cible (22) selon une
terminal (120) sur lequel sont installés un système coordonnée d’exploitation de la pre-
d’exploitation et un programme d’application, le pro- 25 mière opération de sélection et la cap-
cédé comprenant : ture d’informations de contenu de l’élé-
ment d’interface cible (22) ;
l’affichage (S201) d’une interface utilisateur (21, l’obtention d’informations de contexte
131) du programme d’application, dans lequel du programme d’application, dans le-
l’interface utilisateur (21, 131) comprend au 30 quel le programme d’application est un
moins un élément d’interface ; programme d’application en exécution
la réception (S202) d’une première opération de dans l’avant-plan, l’interface utilisateur
sélection déclenchée sur l’interface utilisateur (21, 131) est une interface utilisateur
(21, 131) ; en exécution dans l’avant-plan du pro-
la sélection (S203) d’un élément d’interface ci- 35 gramme d’application, les informations
ble (22) à partir de l’au moins un élément d’in- de contexte comprennent l’identifiant
terface selon la première opération de du programme d’application et/ou
sélection ; et l’identifiant de l’interface utilisateur ; le
l’affichage (S204) d’informations de résultat de système d’exploitation est pourvu
recherche (23) de manière superposée sur une 40 d’une pile active dans laquelle une ac-
région locale de l’interface utilisateur (21, 131) tivité située en haut d’une pile corres-
à l’aide du système d’exploitation, les informa- pond à l’interface utilisateur (21, 131)
tions de résultat de recherche (23) correspon- du programme d’application en exécu-
dant à l’élément d’interface cible (22) ; tion dans l’avant-plan ; un programme
dans lequel des informations de résultat de re- 45 de recherche obtient un identifiant du
cherche (23) sont fournies en une quantité de programme d’application en exécution
n, dans lesquelles au moins deux parties d’in- dans l’avant-plan et un identifiant de
formations de résultat de recherche sont diffé- l’interface utilisateur à partir de l’activité
rentes l’une de l’autre, et n est un nombre entier située en haut de la pile ;
positif ; 50 la détection du fait que des informations
le procédé comprenant en outre, avant l’afficha- de contexte du programme d’applica-
ge d’informations de résultat de recherche (23) tion satisfont, ou ne satisfont pas, à une
de manière superposée sur une région locale condition de capture selon une liste
de l’interface utilisateur (21, 131) à l’aide du sys- prédéfinie qui comprend une liste blan-
tème d’exploitation : 55 che prédéfinie et/ou une liste noire pré-
définie, la liste prédéfinie stockant dans
l’obtention d’informations de contenu de celle-ci des informations de contexte
l’élément d’interface cible (22) selon la pre- satisfaisant à la condition de capture

37
 73 EP 3 407 221 B1 74

et/ou des informations de contexte ne l’obtention d’informations de contenu de

satisfaisant pas à la condition de l’élément d’interface cible (22) selon la pre-
capture ; et mière opération de sélection ;
si les informations de contexte satisfont l’envoi d’une demande de recherche à un
à la condition de capture, ceci indique 5 serveur (140), dans lequel la demande de
que le programme d’application où les recherche comprend les informations de
informations de contexte sont situées contenu de l’élément d’interface cible (22) ;
appartient à la liste blanche prédéfinie, et
ou n’appartient pas à la liste noire pré- la réception des informations de résultat de
définie, puis la réalisation d’une étape 10 recherche (23) fournies en retour par le ser-
de la détermination de l’élément d’in- veur (140),
terface cible (22) selon une coordon- dans lequel l’obtention d’informations de
née d’exploitation de la première opé- contenu de l’élément d’interface cible (22)
ration de sélection et de la capture d’in- selon la première opération de sélection
formations de contenu de l’élément 15 comprend :
d’interface cible (22) ;
si les informations de contexte ne sa- la détermination de l’élément d’interfa-
tisfont pas à la condition de capture, ce cible (22) selon une coordonnée
l’envoi d’informations d’échec d’acqui- d’exploitation de la première opération
sition au programme de recherche. 20 de sélection, et la capture d’informa-
tions de contenu de l’élément d’interfa-
14. Support non transitoire de stockage lisible par ordi- ce cible (22) ;
nateur, stockant sur celui-ci au moins une instruction dans lequel les actions comprennent
exécutable par un processeur d’un terminal (120), en outre, avant la détermination de
l’au moins une instruction, lorsqu’elle est exécutée, 25 l’élément d’interface cible (22) selon
fait en sorte que le processeur réalise des actions une coordonnée d’exploitation de la
comprenant : première opération de sélection et la
capture d’informations de contenu de
la transmission de l’instruction, à un dispositif l’élément d’interface cible (22) :
d’affichage, d’afficher une interface utilisateur 30
(21, 131) du programme d’application, dans le- l’obtention d’informations de con-
quel l’interface utilisateur (21, 131) comprend texte du programme d’application,
au moins un élément d’interface ; dans lequel le programme d’appli-
la réception d’une première opération de sélec- cation est un programme d’appli-
tion déclenchée sur l’interface utilisateur (21, 35 cation en exécution dans l’avant-
131) ; plan, l’interface utilisateur (21,
la sélection d’un élément d’interface cible (22) 131) est une interface utilisateur en
à partir de l’au moins un élément d’interface se- exécution dans l’avant-plan du
lon la première opération de sélection ; et programme d’application, les infor-
la transmission de l’instruction, à un dispositif 40 mations de contexte comprennent
d’affichage, d’afficher des informations de résul- l’identifiant du programme d’appli-
tat de recherche (23) de manière superposée cation et/ou l’identifiant de l’interfa-
sur une région locale de l’interface utilisateur ce utilisateur ; le système d’exploi-
(21, 131) à l’aide du système d’exploitation, les tation est pourvu d’une pile active
informations de résultat de recherche (23) cor- 45 dans laquelle une activité située en
respondant à l’élément d’interface cible (22) ; haut d’une pile correspond à l’in-
dans lequel des informations de résultat de re- terface utilisateur (21, 131) du pro-
cherche (23) sont fournies en une quantité de gramme d’application en exécu-
n, dans lesquelles au moins deux parties d’in- tion dans l’avant-plan ; un pro-
formations de résultat de recherche sont diffé- 50 gramme de recherche obtient un
rentes l’une de l’autre, et n est un nombre entier identifiant du programme d’appli-
positif ; cation en exécution dans l’avant-
les actions comprenant en outre, avant l’afficha- plan et un identifiant de l’interface
ge d’informations de résultat de recherche (23) utilisateur à partir de l’activité si-
de manière superposée sur une région locale 55 tuée en haut de la pile ;
de l’interface utilisateur (21, 131) à l’aide du sys- la détection du fait que des infor-
tème d’exploitation : mations de contexte du program-
me d’application satisfont, ou ne

38
 75 EP 3 407 221 B1 76

satisfont pas, à une condition de

capture selon une liste prédéfinie
qui comprend une liste blanche
prédéfinie et/ou une liste noire pré-
définie, la liste prédéfinie stockant 5
dans celle-ci des informations de
contexte satisfaisant à la condition
de capture et/ou des informations
de contexte ne satisfaisant pas à
la condition de capture ; et 10
si les informations de contexte sa-
tisfont à la condition de capture, ce-
ci indique que le programme d’ap-
plication où les informations de
contexte sont situées appartient à 15
la liste blanche prédéfinie, ou n’ap-
partient pas à la liste noire prédé-
finie, puis la réalisation d’une étape
de la détermination de l’élément
d’interface cible (22) selon une 20
coordonnée d’exploitation de la
première opération de sélection et
de la capture d’informations de
contenu de l’élément d’interface ci-
ble (22) ; 25
si les informations de contexte ne
satisfont pas à la condition de cap-
ture, envoyer des informations
d’échec d’acquisition au program-
me de recherche. 30

35

40

45

50

55

39
EP 3 407 221 B1

40
EP 3 407 221 B1

41
EP 3 407 221 B1

42
EP 3 407 221 B1

43
EP 3 407 221 B1

44
EP 3 407 221 B1

45
EP 3 407 221 B1

46
EP 3 407 221 B1

47
EP 3 407 221 B1

48
EP 3 407 221 B1

49
EP 3 407 221 B1

50
EP 3 407 221 B1

51
EP 3 407 221 B1

52
EP 3 407 221 B1

53
EP 3 407 221 B1

54
EP 3 407 221 B1

55
EP 3 407 221 B1

56
EP 3 407 221 B1

57
EP 3 407 221 B1

58
EP 3 407 221 B1

59
 EP 3 407 221 B1

REFERENCES CITED IN THE DESCRIPTION

This list of references cited by the applicant is for the reader’s convenience only. It does not form part of the European
patent document. Even though great care has been taken in compiling the references, errors or omissions cannot be
excluded and the EPO disclaims all liability in this regard.

Patent documents cited in the description

• US 9582482 B1 [0005]

60