Standards-Based User Interface Technology For Universal Home Domination
Standards-Based User Interface Technology For Universal Home Domination
Standards-Based User Interface Technology For Universal Home Domination
Liang-Yen Lin, Ming-Chun Cheng, Shyan-Ming Yuan Dept. of Computer Science, National Chiao Tung University, Taiwan mick.cis89@nctu.edu.tw, native@cis.nctu.edu.tw, smyuan@cis.nctu.edu.tw Abstract
Based on AIAP-URC and OSGi standards, the authors developed a home control platform named SUIT which is suitable for any kind of home appliances and remote consoles. The SUIT platform provides an easy but standardized environment for both appliance and remote console manufacturers to develop their products. In addition, manufacturers are able to update software on the home gateway remotely and dynamically. In this paper, an IM remote console is implemented to demonstrate SUIT platform. Moreover, the products, including InsteOn, X10, UPnP, and IR Robot are developed successfully on the SUIT gateway. The authors want to ease the sufferings of both home appliance manufacturers and remote console manufacturers. In SUIT, the authors first integrate heterogeneous home control networks on the network layer, and then add user interface abstraction on those integrated services for the presentation layer. In addition, access control and dependency policy is required and they will be discussed in this paper.
2. Background
Two major parts of home control system are home appliances and remote consoles. In this section, these two technologies are discussed and they are foundations of SUIT. Besides, OSGi and AIAP-URC standards are the main bases of SUIT.
1. Introduction
Home control is more and more important in IT industry since information appliance appears. An information appliance is a network-enabled appliance which is different from traditional appliances. The number of standards and protocols for home control is increasing rapidly over the past years. Hence, we have different kinds of traditional home appliances and information appliances, and we also hold many different remote controls and consoles in our hands to dominate home devices. In other words, we need specific remote control for specific home appliance. This is not convenient. The meaning of universal home domination is that every remote control is capable of operating every home appliance, and also any kind of home appliance is operable by any kind of remote control. To achieve universal home domination goals, many problems have to be solved. For example, different control or console has different user interface or different language. Different appliances may have different protocols. In this paper, SUIT (Standardsbased User Interface Technology) is proposed to solve these problems by two approaches, They are user interface abstraction [6][19][20][21] and integration of heterogeneous home network [10][12][13][17].
2006 International Conference on Hybrid Information Technology (ICHIT'06) 0-7695-2674-8/06 $20.00 2006
Console) standards is to facilitate the development and deployment of a wide variety of devices that can act as Universal Remote Consoles (URC) for an equally varied range of devices and services. There are five standards in the URC standards set, below are some simple descriptions about those five standards: Universal Remote Console Standard [1], User Interface Socket Description Standard [2], Presentation Template Standard [3], Target Properties Sheet Standard [4] and Resource Description Standard [5].
Figure 1. AIAP-URC architecture Figure 1 illustrates the architecture of AIAP-URC, This architecture comprises four major components and two networks. The Universal Remote Console (URC), The Target, Supplemental User Interface Implementation Descriptions (UIIDs), Supplemental Resources, Target-URC Network (TUN): between the URC and the Target (required), Resource-URC Network (RUN): between the URC and additional resources. The interactions and communications between URCs and targets must follow the above standards.
2006 International Conference on Hybrid Information Technology (ICHIT'06) 0-7695-2674-8/06 $20.00 2006
The OSGi Alliance (formerly known as the Open Services Gateway initiative) is an open standards organization. Their mission is to specify, create, advance, and promote an open service platform for the delivery and management of multiple applications and services to all types of networked devices in home, vehicle, mobile and other environments. Over the past few years it has specified a Javabased service platform that can be remotely managed. The core part of the specifications is a framework that defines an application life cycle model and a service registry. Based on this framework, a large number of OSGi Services have been defined: Log, Configuration management, Preferences, Http Service (runs servlets), XML parsing, Device Access, Package Admin, Permission Admin, Start Level, User Admin, IO Connector, Wire Admin, Jini, UPnP Exporter, Application Tracking, Signed Bundles, Declarative Services, Power Management, Device Management, Security Policies, Diagnostic/Monitoring and Framework Layering, etc. Started from 2000, R1 release. The OSGi standard is now on the stage of R4 Release, 2005. It is widely accepted by the IT industry.
Home Appliance Manufacturers are responsible to provide control point bundles for their products, and they have to follow the standard bundle programming procedure defined by OSGi Alliance. Besides providing those control point software or drivers, the manufacturers need to give a default AIAP-URC description document which is used to present the default user interface to human beings. Remote Console Manufacturer are responsible to provide control point bundles for their products, and they have to follow the standard bundle programming procedure defined by OSGi Alliance. Developers refer to those people who have computer science background and programming skill, but they dont belong to any part of appliance manufacturer or console manufacturer. They are allowed to develop bundles and AIAP-URC documents. For instance, in figure 3, a developer is not an employee of IR Robot manufacturer, but he hopes to integrate the IR Robot on SUIT gateway. In this case, he can develop a control point which sends correct IR code to Robot through an IR transmitter connected to SUIT gateway with console line. In addition, he has to write some action documents according to AIAP-URC standards. Finally, he has to develop an MSN remote console bundle, so that he can control the robot with MSN.
Figure 3. Using IM to control IR Robot Figure 2. Depicts the relationship of logistics information with activities of an enterprise System Providers have to setup and manage the whole SUIT environment, and the majority job is taking good care of those bundles from both home appliance manufacturer and remote console manufacturer, including bundle installation, bundle upgrade and removing bundles. Supplemental Resource Providers provide external resource services for SUIT platform. Resources include images, sounds, labelsetc. For example, users can get multi-language description packs from supplemental resource services.
3. System Architecture
3.1. Roles
Figure 2 shows the relationship between each role in SUIT platform. IM user, Mobile user and Web user are typical type of users in SUIT system. Human can operate home appliances through instant messaging interface, mobile device, web page or some other remote console technologies that can be easily developed and deployed on SUIT platform. Besides controlling home devices, they can design user interfaces for themselves, or download user interface documents from internet to replace the old one.
2006 International Conference on Hybrid Information Technology (ICHIT'06) 0-7695-2674-8/06 $20.00 2006
manufacturer and remote console manufacturer, and SUIT eases their sufferings of deciding which protocol to follow.
Figure 4. SUIT Architecture On the SUIT platform, every home appliance can be controlled by any remote console, and any remote console can dominate any home appliance. Each manufacturer only has to follow OSGi and AIAP-URC international standards and their proprietary protocols. They dont have to study other manufacturers proprietary protocols, but others product can be accessed on SUIT platform. Figure 4 is the overall architecture of SUIT platform. In this section, the SUIT platform is divided into three parts for discussion. The First is core part of SUIT platform, the second is remote console part (including console bundles) and the last is the home network part (including appliance bundles).
backward. In other words, SUIT adapter is a bundle in charge of looking up all of related services from OSGi service repository and it is also responsible to pick up correct service objects for SUIT gateway. Controller is actually the commander in SUIT platform. Requests are passed to the controller. Then controller will perform related actions in response, and reply to the party who sent the request. For example, when one console requests to turn off TV, the controller will pick up the TV service object through SUIT adapter from the service repository. Then the controller will check up the dependency relationship of the action which is going to be performed on the TV through dependency management. If the access is permitted, controller will invoke the corresponding method Turn Off in the TV service object. After the invocation, the object may return some results message or null, and the controller takes care of the message and passes a proper result message to the remote console. Notification Service needs to pass messages to every remote console when any appliances status information is changed. It is implemented by two kinds of standard message passing approaches. The first one is JMS topic, every bundle that implements standard JMS [18][22] client is able to be notified. Each appliance has a JMS topic. The remote consoles may choose the topic to subscribe and will be notified once any changes happen. The second method to implement notification service is OSGi service listener standard. Every bundle listens to specific services in the appliance bundle, once the services are changed, notification happen. Notification services help to send messages to dynamic and heterogeneous remote console bundles. AIAP-URC engine is a preprocessor for AIAP-URC standards, the purpose of this engine is to process those standard documents provided by home appliance manufacturers or others. After the processing the URC documents, the URC engine provides two kinds of mechanisms for remote console bundles to get those metadata. The first one is for those remote console bundle which not only follows OGSi standard but also AIAP-URC standards. The engine just passes those original documents as metadata to the bundle because the bundle AIAP-URC compatible. It has own URC processing engine. The second way to pass metadata is by function calls. The remote console may ask for appliance list by function call provided by the URC engine then it returns the available appliances list. For action list and descriptions of specific home appliance, the URC engine replies the information to the bundle and so on.
2006 International Conference on Hybrid Information Technology (ICHIT'06) 0-7695-2674-8/06 $20.00 2006
Using instant messenger as a remote console is our brilliant innovative idea. IM remote console manufacturer develops an instant messenger robot as a bundle hosts on the SUIT gateway. The job of IM robot is reporting appliances status to other IM client, accepting requests from IM client and invoking the actions. Figure 5 displays an example of MSN remote console. We wrote a MSN robot names DCSLab House Keeper, wrapped it as a bundle, and installed it to SUIT gateway. I added the MSN robot to the buddy list of my personal MSN client which I used to use. After this procedure, I can always see the Robot on my buddy list. Then I am able to open a dialog and talk to the MSN robot on the SUIT gateway. On the right hand side of the screen capture as you see, when I ask for home appliance list, the Robot shows me the available and operable appliance list. IR robot is listed at the third place of the list MSN robot just gave me. If I want to know more details about the operations of the IR Robot, I make a request, and the MSN robot tells me the details as shown on the left hand side of the screen capture, now I can easily operate the IR robot. It seems quite convenient for end users to use IM remote console with any extra installation if they already an IM client, all they need to do is to add some robots as their friends.
Figure 5. MSN remote console The last one to introduce is IA remote console, IA here means intelligent agent. Intelligent devices will be a big research issue in the future, and this may conduct into SUIT platform easily. Imagine that when you are driving, the device generates voice user interface to prevent you from not paying attention on driving. When you are in the meeting place which requires extreme silent then the intelligent agent generates visual user interface for you. All these features cab be down in out platform regarding the abstraction of user interface layer. We can conclude this section that besides the integration of underlying home control, the user interface generation mechanism is more important in SUIT platform. Each remote console bundle gets URC documents or URC metadata from the URC engine in the core part of SUIT gateway. Then those bundles
2006 International Conference on Hybrid Information Technology (ICHIT'06) 0-7695-2674-8/06 $20.00 2006
know how to present the user interface of the target appliance to the users.
feedback any information because it is one-way communication. We take IR Robot as an interesting example to see how to integrate the IR Robot into SUIT platform. Actually it is just a toy that carries with an IR receiver, listening for IR signal in the air and then responds actions for correct IR code. As figure 7 shows, the first step to build the SUIT home appliance part is to find an IR transmitter like what kids hold in their hands. The IR transmitter is connected to the SUIT home gateway, and the IR Robot bundle transmits correct IR codes to the Robot through serial port. Sending the correct IR codes from the SUIT gateway is no different from pressing buttons on the kids remote controller, because the IR Robot doesnt care who sends the IR message, however it cares if there is any action corresponds to the IR code for it to perform gorgeous movements. The IR Robot bundle wraps those action commands like functions or methods, and registers the service objects to the service repository. So that now remote console bundles can see the services, get them and use them.
Figure 7. Home appliance bundle: IR Robot Besides developing IR Robot bundle, we also design AIAP-URC documents for the remote consoles. With the URC descriptions, we present lively action instructions to the end users, instead of expecting them to input correct IR code directly. It is also very easy to support multi-language choice in this architecture.
Figure 6. UPnP integration with SUIT system Traditional home appliances like TV, air conditioning are controlled by IR signal, and will not
2006 International Conference on Hybrid Information Technology (ICHIT'06) 0-7695-2674-8/06 $20.00 2006
In figure 1, the target is required to be manufactured following the AIAP URC standard. For example, if you are UPnP appliance manufacturer, the proprietary network is UPnP on Ethernet, or on wireless. With the underlying network layer, you have an abstraction user interface layer that follows the AIAP-URC standards. If you are remote console manufacturer, you have to build AIAP-URC engine on the UPnP network layer. This is the problem, not cost effective to both manufacturers. Another topic here is that AIAP-URC provides the universal solution for user interface, but we think real and complete universal home domination system must consider every related issue. Figure 10 is our solution to universal home domination, which we would like to build OSGi concept to solve the problem about the proprietary network mentioned above. As section 4.1 says, OSGi is a great choice for the integration of heterogeneous network in a home environment. The solution is that both home appliance developers and remote console developers need to learn how to wrap their programs into OSGi bundles and also how to write AIAP-URC profiles.
Figure 10. OSGi solution for AIAP-URC underlying network Figure 9. Control invocation procedure
3.8. Summary
In this section, we first gave an overview about SUIT platform, and then pointed out what kind of roles or users are related to the system. Then we talked about the system architecture, and separated it into three parts for more detail discussion. Three parts are SUIT core part, SUIT remote console part and SUIT home appliance part. We also discussed the user interface generation procedure and control invocation procedure. In next section, we will talk about design issues of SUIT platform.
4. Design Issues
4.1. AIAP-URC
We transform each proprietary network into OSGi services, so that different venders networks are able to communicate with each other now. Between the bundle that is installed on the SUIT gateway and the existing home appliance, the communication is over their proprietary networks. For example, UPnP on 802.11 Jini on Ethernet, etc. Between different venders bundles, the communication is over OSGi services definition. The AIAP-URC documents information are installed with the bundles and can be changed or replaced by others. Every bundle should provide one set of URC documents for each target. The URC document set includes Target Description User Interface Socket description Presentation Template and Resource Sheet. The URC support is optional for remote console manufacturers. If the remote console bundle supports
2006 International Conference on Hybrid Information Technology (ICHIT'06) 0-7695-2674-8/06 $20.00 2006
URC standards, then SUIT gateway is asked to pass those URC documents to the bundles and leaves the job to the bundles itself. If there is no URC support for the remote console bundle, which means the remote console manufacturer doesnt even know what the AIAP-URC standards are. Then SUIT gateway will read URC documents for the remote console bundle. The SUIT URC engine will parse and organize those documents, and expose methods for remote console bundles to call. When developers are working for bundles, they may read developers guide for useful methods that are provided by the URC engine. For example, the remote console bundles can ask for all available device list available device list by certain proprietary networkactions for specific device, etc. Those methods are contained in an object which registered to the service repository by the SUIT system, and remote console bundles can pick up objects from the service repository and use the methods as just mentioned which are carried with them.
interfaces, for final committee draft voting. If successful, this vote (closing in August 2006), will likely be the last vote on the subcommittee level. At that point, the draft standard will be passed on to JTC1 for final voting and publication as an ISO/IEC International Standard. We choose this future star to construct our abstraction user interface layer. People are able to write descriptions for integrated services, and remote consoles on the SUIT system will be aware of the user interface information, which mean they will generate user interface automatically. Themes for the appliances can be downloaded from somewhere and replace the old ones to keep them in fashion shapes. Now the SUIT platform is not only universal for machine, it is also universal for human. We think real universal home domination is both for machine and human beings.
4.2. OSGi
Using services on the OSGi platform is not very difficult. We just have to look up the services in the service repository, generate the objects and use them. This is a great contribution from OSGi. How about the world after integration? Some developers design user interfaces for those integrated services one by one, and some just design their own presentation languages for user interface auto generation. It is a big pity to construct user interface in non-standardized way after the standardized services.
Figure 11. OSGi with AIAP-URC AIAP-URC is basically user interface protocol by INCITS V2 group which is very possibly becoming the international standards in the future. The latest drafts of the ISO/IEC standard on a universal remote console have been recently distributed to the members of the pertaining subcommittee, ISO/IEC JTC1 SC35 user
2006 International Conference on Hybrid Information Technology (ICHIT'06) 0-7695-2674-8/06 $20.00 2006
information of each appliance on each remote console. The design of this issue on SUIT platform introduces JMS standards into the architecture. As figure 12 illustrates, each remote console implements the JMS subscriber if it needs to do synchronization. The SUIT core part is responsible to establish a JMS TOPIC there, and pastes update information on it. SUIT controller whose job is to invoke actins will publish message to the TOPIC once any of the remote consoles successfully completes an action invocation.
control anything on the SUIT gateway. SUIT platform is designed to serve people, instead of expecting people to compromise with it. In order to appraise the SUIT platform from manufacturers standpoints, we pretended ourselves as home appliances manufacturers and remote console manufacturers. After real cases implementation and evaluation, we prove that SUIT platform is quite satisfying for those manufacturers who wish to conduct their products into SUIT system.
Acknowledgments
The Ministry of Education of the Republic of China partially supported this work under grant nos. NSC942725-E009-006-PAE (Advanced Technologies and Applications for Next-Generation Information Networks (II)) and NSC94-2213-E009-026 (A Research on Next-Generation Massive Multiplayer Virtual Environment Platform). We thank the anonymous reviewers and editors for many useful comments.
[1] Universal Remote Console Standard, ANSI/INCITS 3892005, http://v2.incits.org/ [2] User Interface Socket Description ANSI/INCITS 390-2005, http://v2.incits.org/ Standard,
Figure 12. Synchronization and notification After the remote console bundle receives the update message from the JMS TOPIC, it is the proprietary network world. So what will be the next depends on the remote console venders who develop the bundle. Take MSN messenger as an example, when the MSN agent that is wrapped as a bundle receives messages, it will broadcast this message to remote consoles or you can say, friends, in its buddy list. Then users who have added this MSN agent in their buddy list will be notified, and the information for them is synchronized.
[3] Presentation Template Standard, ANSI/INCITS 3912005, http://v2.incits.org/ [4] Target Description Standard, ANSI/INCITS 392-2005, http://v2.incits.org/ [5] Resource Description Standard, ANSI/INCITS 393-2005, http://v2.incits.org/ [6] Vanderheiden, G., Zimmermann, G., Use of User Interface Sockets to Create Naturally Evolving Intelligent Environments, 2005, http://myurc.org/publications/2005HCI-Nat-Evol-Intel-Env.php [7] Vanderheiden, G., Zimmermann, G., Trewin, S., Interface Sockets, Remote Consoles, and Natural Language Agents, 2005, http://myurc.org/whitepaper.php [8] Zimmermann, G., Vanderheiden, G., Rich, C., Universal Control Hub & Pluggable User Interfaces, 2005, http://myurc.org/publications/2006-Univ-Ctrl-Hub.php [9] Zimmermann, G., Nixon, T., Beard, M., Sitnik, E., LaPlant, B., Trewin, S., Laskowski, S., Vanderheiden, G., Toward a unified universal remote console standard, Extended Abstracts on Human Factors in Computing Systems, CHI2003 Conference on Human Factors in Computing Systems, pp. 874-875.
5. Conclusion
We have introduced our solution to realize the universal home domination, the SUIT platform which is suitable for every appliance and remote console. SUIT platform eases the sufferings for both home appliance manufacturer and remote console manufacturer. They dont have to worry about what kind of network they must support, which one will be more competitive in the future. Proprietary technology plus OSGi bundle and AIAP-URC documents will be perfectly integrated with other proprietary networks, and also different kinds of remote consoles. More importantly, customers who use the SUIT system to manage their homes are likely to love it after trying it. They dont have to care numerous types of home control technologies, what they see are just simply appliances, with beautiful costumes, or say, with friendly and kindly user interface, not just cold machine instructions. In another point of view, customers are free to install the remote console theyre used to, and any kind of remote consoles are capable to
2006 International Conference on Hybrid Information Technology (ICHIT'06) 0-7695-2674-8/06 $20.00 2006
[10] Latvakoski, E.J., Paakkonen, P., Remote interaction with networked appliances attached in a mobile personal area network, IEEE International Conference on Communications (ICC 03), pp. 769-773. [11] Kushiro, N., Suzuki, S., Nakata, M., Takahara, H., Inoue, M., Integrated Residential Gateway Controller for Home Energy Management System engineering, IEEE Transactions on Consumer Electronics, Vol. 49(3), pp. 629636. [12] Zhaohui Ye, Yindong Ji, Shiyuan Yang, Home Automation Network Supporting Plug-and-Play, IEEE Transactions on Consumer Electronics, Vol. 50(1), pp. 173179. [13] Song Yean Cho, Dae Young Seo, Tai Yun Kim, Gateway framework for home appliances interoperability based on heterogeneous middleware in residential networks, International Conference on Consumer Electronics (ICCE 2002), pp. 98-99. [14] OSGi Alliance, http://www.osgi.org/ [15] UPnP Forum, http://www.upnp.org/ [16] Jini.org, http://www.jini.org/
[17] Chi-Huang Chiu, Hsien-Tang Lin, Ping-Jer Yeh, ShyanMing Yuan, An OSGi Platform Connecting Heterogeneous Smart Home Appliances from Mobile Devices, WSEAS Transactions on Computers, Vol. 5(7), pp. 1549-1555, 2006 [18] Ruey-Shyang Wu, Shyan-Ming Yuan, An Adaptive Architecture for Secure Message Oriented Middleware, WSEAS Transactions on Information Science and Applications, Vol. 3(7), pp. 1239-1246., 2006 [19] Tzu-Han Kao, Shyan-Ming Yuan, Automatic adaptation of mobile applications to different user devices using modular mobile agents, Software Practice and Experience, Vol. 35(14), pp. 1349-1391. [20] Tzu-Han Kao and Shyan-Ming Yuan, Designing an XML-based context-aware transformation framework for mobile execution environments using CC/PP and XSLT, Computer Standards & Interfaces, Vol. 26(5), pp. 377-399. [21] Ming-Chun Cheng, Shyan-Ming Yuan, An Adaptive Mobile Application Development Framework, Embedded and Ubiquitous Computing (LNCS 3824), pp.765-774 [22] Tsun-Yu Hsiao, Ming-Chun Cheng, Hsin-Ta Chiao, Shyan-Ming Yuan, FJM: A High Performance Java Message Library, IEEE International Conference on Cluster Computing 2003, pp. 460-463.
2006 International Conference on Hybrid Information Technology (ICHIT'06) 0-7695-2674-8/06 $20.00 2006