Virtual Assistant

CHAPTER 1 - INTRODUCTION

As we stand on the brink of the third wave of computing, the concept of a virtual
assistant (a digital service looking after a range of our needs) is fast becoming a reality. As artificial
intelligence and machine learning progress at pace, virtual assistants are set to become our gateway
to the internet and know more about us than we do ourselves. A virtual assistant is a software
agent that can perform tasks or services for an individual. As of 2017, the capabilities and usage
of virtual assistants are expanding rapidly, with new products entering the market and a strong
emphasis on voice user interfaces. The most widely used are Apple's Siri, Google
Assistant, Amazon Alexa and Microsoft Cortana.

Virtual assistants can provide a wide variety of services, and particularly those from
Amazon Alexa and Google Assistant grow by the day. These include:

 Provide information such as weather, facts from e.g. Wikipedia or IMDB, set an alarm, make
to-do lists and shopping lists.
 Play music from streaming services such as Spotify and Pandora, play radio stations,
read audiobooks.
 Play videos, TV shows or movies on televisions, streaming from e.g. Netflix.
 Conversational commerce.
 Complement and/or replace customer service by humans. One report estimated that an
automated online assistant produced a 30% decrease in the work-load for a human-
provided call center.
Home automation is also an application of virtual assistant. Home
automation or domotics is building automation for a home, called a smart home or smart house. A
home automation system will control lighting, climate, entertainment systems, and appliances. It
may also include home security such as access control and alarm systems. When connected with
the Internet, home devices are an important constituent of the Internet of Things. A home
automation system typically connects controlled devices to a central hub or "gateway". The user
interface for control of the system uses either wall-mounted terminals, tablet or desktop computers,
a mobile phone application, or a Web interface, that may also be accessible off-site through the
Internet.

Dept. of Electronics and Communication Engineering 1

Virtual Assistant

1.1 - ABOUT GREEN

The virtual assistant in this project is named ‘Green’. The system is named after
the Raspberry Pi board which is green in colour.

Initially when the project is run the system introduces itself by telling its name
and then it waits for the input from user. If the keyword ‘green’ is not present in the user input, it
goes into an infinite loop and waits for the keyword ‘green’. Once the user speaks the word ‘green’,
the system is enabled to perform certain tasks. After the completion of each task the system waits
for the keyword ‘green’ and the whole process repeats. Once the program is run, the system
outputs:

'Hi. I'm your virtual assistant. I'm called Green. Just call my name whenever you need me.'

through the output transducer (or simply the speaker). The system may be addressed as:

“hey green” or

“hello green” or

“good morning green” etc.

Here, the keyword that is of importance is ‘green’ and the other associated words are ignored.

Dept. of Electronics & Communication Engineering 2

Virtual Assistant

CHAPTER 2 - SOFTWARE TOOLS

Software is a generic term that refers to a collection of data or computer instructions that
tell the computer how to work, in contrast to the physical hardware from which the system is built,
that actually performs the work. In computer science and software engineering, computer software
is all information processed by computer systems, programs and data. Computer software includes
computer programs, libraries and related non-executable data, such as online documentation or
digital media. Computer hardware and software require each other and neither can be realistically
used on its own.

2.1 - Raspbian
Raspbian is a Debian-based computer operating system for Raspberry Pi [1]. There are
several versions of Raspbian including Raspbian Stretch and Raspbian Jessie. Raspbian is highly
optimized for the Raspberry Pi line's low-performance ARM CPUs.

Raspbian uses PIXEL, Pi Improved Xwindows Environment Lightweight as its main

desktop environment as of the latest update. It is composed of a modified LXDE desktop
environment and the Openbox stacking window manager with a new theme and few other changes.
Figure 3.1 shows how Raspbian OS looks.

Figure 3.1 – Raspbian OS

2.2 - Python
Python is an interpreted high-level programming language for general-purpose
programming [2]. Created by Guido van Rossum and first released in 1991, Python has a design

Dept. of Electronics & Communication Engineering 3

Virtual Assistant

philosophy that emphasizes code readability, notably using significant whitespace. It provides
constructs that enable clear programming on both small and large scales. In July 2018, Van
Rossum stepped down as the leader in the language community after 30 years.

Python features a dynamic type system and automatic memory management. It supports multiple
programming paradigms, including object-oriented, imperative, functional and procedural, and has
a large and comprehensive standard library. Here are the some of the salient features [3].

 Interpreted: There are no separate compilation and execution steps like C and
C++.Internally, Python converts the source code into an intermediate form called
bytecodes which is then translated into native language of specific computer to run it. There
is no need for linking and loading with libraries.

 Platform independent: Python programs can be developed and executed on multiple

operating system platforms.

 High-level language: In Python there is no need to take care about low-level details
such as managing the memory used by the program.

 Simple: It is closer to English language and easy to learn and. It emphasizes more on the
solution to the problem rather than the syntax.

 Embeddable: Python can be used within C/C++ program to give scripting capabilities
for the program’s users.

 Robust: It has exception handling features and in-built memory management techniques.

 Rich library support: The Python Standard Library is very vast.

2.3 - Libraries used

 SpeechRecognition: This has the ability to identify words and phrases in spoken
language and convert them to a human-readable format that is text. The speech recognition
software has a limited vocabulary of words and phrases, and it may only identify these if
they are spoken very clearly. More sophisticated software has the ability to accept natural
speech. It converts spoken language to human-readable with support of several engines and
APIs, online and offline [4].

 gTTS (Google Text-to-Speech): This Python library and CLI tool is to interface with
Google translator’s text-to-speech API. It writes spoken mp3 data to a file, a file-like object
(byte string). The gTTS API supports several languages including English, Hindi, Tamil,
French, German and many more. The speech can be delivered in any one of the two
available audio speeds, fast or slow. However, as of the latest update, it is not possible to
change the voice of the generated audio [4].

Dept. of Electronics & Communication Engineering 4

Virtual Assistant

 Wikipedia: Wikipedia is a Python library that makes it easy to access and parse data
from Wikipedia. Search Wikipedia, get article summaries, get data like links and images
from a page, and more. This library was designed for simple use and it wraps the
MediaWiki API [4].

 requests: Requests will allow us to send HTTP/1.1 requests using Python. With it, you
can add content like headers, form data, multipart files, and parameters via simple Python
libraries. It also allows you to access the response data of Python in the same way [4].

2.4 - Other software used

 PyAudio: With PyAudio, Python to play can be used and audio can be recorded on a
variety of platforms, such as GNU/Linux, Microsoft Windows, and Apple Mac OS [4].

 FLAC (Free Lossless Audio Codec): FLAC is an audio coding format for lossless
compression of digital audio, and is also the name of the free software project producing
the FLAC tools, the reference software package that includes a codec implementation.
Digital audio compressed by FLAC's algorithm can typically be reduced to between 50 and
70 percent of its original size and decompress to an identical copy of the original audio
data [4].

 mpg321: Is an audio player which can play mp3 format. It is a lightweight command line
MP3 player [4].

 Lame: LAME is a software encoder that converts audio to the MP3 file format [4].

Dept. of Electronics & Communication Engineering 5

Virtual Assistant

CHAPTER 3 - SETTING UP RASPBERRY PI

3.1 - Installing OS
Step 1: Format SD card

The first step to getting started with Raspberry Pi is to format the microSD card that will
be used to download the operating system [5]. Steps to format the SD card:

 Insert the microSD card into the USB card reader.

 Connect the card reader to a PC.
 Download and install SD Card Formatter on the computer.
 Open SD Card Formatter, select the SD card and format it.

Figure 3.2 – SD Card Formatter

Step 2: Install Raspbian to microSD card

The next step is to get Raspbian Stretch onto the microSD card. Once it's loaded, the SD
card can be plugged it into your Raspberry Pi and configure the operating system. The microSD
card should already be connected to your computer at this time. Steps to install the OS:

 Download the ZIP file of Raspbian OS from the official website.

 Unzip the file the Raspbian disc image. This can be done using 7-Zip application.
 Write the disc image to the microSD card with the help of Win32 Disk Imager.

Dept. of Electronics & Communication Engineering 6

Virtual Assistant

Figure 3.3 – Writing Disc image to microSD card

 Eject the microSD card.

Step 3: Setting up Raspberry Pi

 Insert the microSD card into the card slot on the underside of the Raspberry Pi.
 Plug the USB keyboard into one of the USB ports.
 Plug the USB mouse into one of the USB ports.
 Turn on your monitor or TV set and make sure it is set to the proper input (e.g. HDMI 1 or
Component.
 Plug the HDMI or video component cable into the monitor or TV set.
 Connect the other end of the cable into the Raspberry Pi.
 Plug the power supply into the power outlet. This will turn on and boot up Raspberry Pi.
A power indicator light will begin to glow, letting you know that you are connected.

Step 4: Configuring Raspberry Pi

When Raspbian begins to load a bunch of lines of code will appear. This will continue until
the boot process has completed. Then, the Raspbian Home screen will appear. Configure the
Raspberry Pi system in order to add your location, date, and time.

 Connect to the Wi-Fi.

 Click on the Menu in the upper left corner of the screen.
 Select Preferences in the dropdown menu.
 Select Raspberry Pi Configuration under Preferences.

Dept. of Electronics & Communication Engineering 7

Virtual Assistant

Figure 3.4 – Configuration

 When the configuration window appears, click on the Localisation tab.

Figure 3.5 – Configuration Window

 Click on Set Locale… to set the location.

 Click on Set time zone… to set your local time.
 Click on Set Keyboard… to set your keyboard language.
 Reconfiguring your Raspberry Pi will require a reboot. When the reboot window appears,
click Yes to continue.

Step 5: Connecting Raspberry Pi to Laptop via Wi-Fi

 Once Raspberry Pi is configured Raspberry Pi can be connected to laptop.

 Remove the microSD card from the Raspberry Pi and connect to the laptop via microSD
card reader.
 Open the microSD card driver.
 Create an empty file with the name “ssh”.

Dept. of Electronics & Communication Engineering 8

Virtual Assistant

 Create a file which contains the details of the Wi-Fi name and the Wi-Fi password with the
name “wpa_supplicant.config”. The code should be as follows:

network= {
ssid= “name of the network”
psk= “password of the network”
}

 Eject the microSD card and put it back into the Raspberry Pi.
 Power on the Pi board.
 Download and install PuTTY software.
 Open PuTTY and enter the IP addresses or Host name (Pi Host name) and click on “open”.

Figure 3.6 – PuTTY

 Now login with pi as the username and raspberry as the password.

Figure 3.7 – Login

 After login install tightvncserver by running sudo apt-get install tightvncserver command
and install xrdp by running sudo apt-get install xrdp command.

Dept. of Electronics & Communication Engineering 9

Virtual Assistant

 Open Remote Desktop Connection, enter Raspberry Pi 3B IP address and click Connect.

Figure 3.8 – Remote Desktop Connection

 Now enter “pi” as the username and “raspberry” as the password. Click OK. If everything
works fine, Raspberry Pi desktop will be loaded on the laptop.

Figure 3.9 – Login page

Dept. of Electronics & Communication Engineering 10

Virtual Assistant

CHAPTER 4 - PROTOTYPE DEVELOPMENT

This chapter details the development of the prototype starting with a description of
the process used and the structure of the prototype.

4.1 - Development Process

An agile process, with multiple phases, was used during the development of the
prototype. During each phase a number of functions were implemented and tested. The structure
of the functions in each phase were based on the order of the requirements presented. The
requirements were divided into three phases, where phase 1 consists of requirements that had to
be reached in order for the project to be considered successful. The subsequent phases dealt mostly
with improving performance and usability.

 Phase 1

– Understand a small set of defined commands.

– Always ready for new commands.
– Be able to control a device.

 Phase 2

– Improve the accuracy.

– Minimising the code complexity.
– Taking care of exceptions.

 Phase 3

– Status indication, e.g. with a LED-light.

– The prototype should give a response after a command.
During the development the main focus was on modularity in the source code. The goal
of the project was to develop a system with modules that should be easy to exchange. To this end,
the source code was divided into seven modules;

 speech recognition,
 response,
 core,
 search,
 weather,
 train details and
 hardware controller.

Dept. of Electronics & Communication Engineering 11

Virtual Assistant

4.2 - Hardware Setup

Due to the nature of the project a rather large amount of different types of hardware was
needed. The hardware ranges from the computing platform used to run the software to the
microphone used to record a user’s speech. The following subsections aims to provide an overview
of the hardware used throughout the project as well as provide arguments for why any specific
hardware was used. In order to fulfil the affordability property price was an important factor when
considering any hardware.

4.2.1 - Hardware Platform

The device used as a platform for the prototype had to be quite cheap, while still providing
the necessary computing power in order to enforce the affordability property. Furthermore, the
device should not be a disturbance to the user. An ordinary desktop computer provides the
necessary capabilities but could be hard to find a place for. This criterion extends further than just
physical space, a user should not constantly be reminded by the presence of the device, e.g. by it
creating a lot of noise.
These factors led to us choosing a small single-board computer as they are inexpensive
while still providing adequate performance. Their compact form factor also makes them
unobtrusive to the user. There exist several products to choose from but the final decision was to
use a Raspberry Pi, specifications presented in appendix B, for primarily two reasons. The
Raspberry Pi has good capabilities for attaching a number of varying sensors and transmitters,
which is a desirable property for controlling home equipment. The other reason was the active
online community that exists for the Raspberry Pi which we identified as a large potential source
for help during development.
The prototype should be fast and perform well which makes an operating system (OS)
that is lightweight and heavily configurable optimal. Based on this criterion the ARM version of
Arch Linux made the best fit. On the other hand, the prototype also needs to be easy to set up and
use. This characteristic is something that Arch is not known for and therefore another OS might
be more suitable. In this regard we found Raspbian, which is a Debian port optimised for the
Raspberry Pi, to be the preferred OS. Ultimately, we decided to leave this decision to the user by
not developing for any one particular OS. On the prototype Arch was primarily used in order to
have more control over the environment, but the software was also tested periodically on a
Raspbian installation. This way a user that is experienced in UNIX environments could use Arch
to optimise performance. Someone with less experience could use the more user friendly Raspbian,
or another OS of their choice.

Dept. of Electronics & Communication Engineering 12

Virtual Assistant

4.2.2 - Peripherals Hardware Units

Due to the time frame of the project it was decided that the prototype should focus on
one type of device to control. The type of device chosen were simple devices like LEDs and DC
Motor as they are easy to setup and can be used to control a large variety of different equipment.
In order to control LEDs and DC Motor which is used instead of light and Fan the use of
resistor, transistor, diode and an external power supply controlled by GPIO was done.
A very essential device needed for this project was a microphone in order to capture the
speech of the user. The choice of microphone can have large effects on the overall performance
since poor sound quality might result in low recognition accuracy. Ideally for the prototype one
would like a microphone that can capture speech from anywhere in a room, thus removing the
need for the user to be mindful of the positioning of the microphone. These microphones are
however quite expensive and since this project aims to create an affordable product, instead used
a lower-end microphone. The microphone used is “Anwesha 3.5mm Jack Plug Mic” as this had
3.5mm jack. USB sound card has to be used to connect it to Raspberry Pi. The sound card used is
“Quantum. QHM623 3D Virtual 5.1 USB Audio Controller Sound Card (Integrated 2 Channel)”.
As the prototype responded to the user commands through audio the use of speakers
which can be connected to the Raspberry Pi 3.5mm audio port were used.

Figure 4.1 – Block Diagram

Dept. of Electronics & Communication Engineering 13

 Virtual Assistant

CHAPTER 5 - THE CIRCUIT

The circuit implemented in the prototype is shown in Figure 5.1.

Output

Transducer

Input

Transduce
r

Figure 5.1 – Circuit Diagram

Dept. of Electronics & Communication Engineering 14

Virtual Assistant

The circuit mainly consists of:

 an input transducer
 a central control unit
 an output transducer
 resistors
 LEDs
 a diode
 a transistor
 a DC motor
Figure 5.2 – Circuit for interfacing DC motor

The input transducer or a microphone is connected to a sound card which is inserted into one of
the USB ports. This is used to control the GPIO pins.
The GPIOs are used to control the LEDs and the DC motor. The LEDs are named as
KITCHEN_LIGHT at GPIO 8, BEDROOM_LIGHT at GPIO 12, BATHROOM_LIGHT at
GPIO 16 and LIVINGROOM_LIGHT at GPIO 22. The LEDs are given to the ground through
resistors of value 1kΩ.
The DC motor is named as FAN and is connected to GPIO 26 through a resistor of value 1kΩ
and a npn transistor MPS2222A. The GPIO output is given to the base of the transistor. The DC
motor is connected across a 1N4148 diode and to the collector terminal of the transistor. The
other end of the DC motor is given to the positive terminal of a 9V battery. When a GPIO pin is
made HIGH the corresponding LED glows or the motor rotates.
The output transducer or a speaker is connected to the 3mm jack. The is used to convey a
message to the user regarding the status of the LEDs or the DC motor.

Dept. of Electronics & Communication Engineering 15

Virtual Assistant

CHAPTER 6 - PROGRAM FLOW AND CODE

6.1 - Flow Chart
The program make use of API’s like Weather API [6] and Train API [7]. The below
Figure 6.1 shows the flow of code.

Figure 6.1 – Flow Chart

Dept. of Electronics & Communication Engineering 16

Virtual Assistant

6.2 - Code
6.2.1 – initial.py
import text_to_speech as ts

import main

import speech_recognition as sr

import RPi.GPIO as GPIO

GPIO.setwarnings(False)

GPIO.setmode(GPIO.BOARD)

BLINK=7

GPIO.setup(BLINK,GPIO.OUT)

def loop():

while True:

r=sr.Recognizer()

with sr.Microphone() as scource:

print("Speak")

GPIO.output(BLINK,True)

audio = r.listen(scource)

GPIO.output(BLINK,False)

try:

print(r.recognize_google(audio))

if ('green' in r.recognize_google(audio)):

ts.voice('Hey there.. How can I help you?')

main.askme()

else:

loop()

except:

loop()

return

ts.voice('Hi. I\'m your virtual assistant. I\'m called Green. Just call my name whenever you need me.')

loop()

Dept. of Electronics & Communication Engineering 17

Virtual Assistant

6.2.2 – iopin.py
import RPi.GPIO as GPIO

import time

import text_to_speech as ts

GPIO.setmode(GPIO.BOARD)

GPIO.setwarnings(False)

GPIO.setup(8,GPIO.OUT)

GPIO.setup(12,GPIO.OUT)

GPIO.setup(16,GPIO.OUT)

GPIO.setup(22,GPIO.OUT)

KITCHEN_LIGHT = 8

BEDROOM_LIGHT = 12

BATHROOM_LIGHT = 16

HALL_LIGHT = 22

def devices(st):

if('bedroom' in st ):

if('on' in st):

ts.voice('Turning the berdroom light ON.')

GPIO.output(BEDROOM_LIGHT,True)

else:

ts.voice('Turning the bedroom light OFF.')

GPIO.output(BEDROOM_LIGHT,False)

elif('kitchen' in st):

if('on' in st):

ts.voice(' Turning the kitchen light ON.')

GPIO.output(KITCHEN_LIGHT,True)

else:

ts.voice('Turning the kitchen light OFF.')

GPIO.output(KITCHEN_LIGHT,False)

elif('bath' in st):

Dept. of Electronics & Communication Engineering 18

Virtual Assistant

if('on' in st):

ts.voice('Turning the bathroom light ON.')

GPIO.output(BATHROOM_LIGHT,True)

else:

ts.voice('Turning the bathroom light OFF.')

GPIO.output(BATHROOM_LIGHT,False)

elif('living' in st):

if('on' in st):

ts.voice('Turning the living room light OFF.')

GPIO.output(HALL_LIGHT,True)

else:

ts.voice('Turning the living room light OFF.')

GPIO.output(HALL_LIGHT,False)

else:

ts.voice('No such devices found.')

6.2.3 – main.py
import text_to_speech as ts

import train

import wiki

import weather

import iopin

from datetime import *

import speech_recognition as sr

import RPi.GPIO as GPIO

GPIO.setwarnings(False)

GPIO.setmode(GPIO.BOARD)

FAN=26

BLINK=7

GPIO.setup(FAN,GPIO.OUT)

GPIO.setup(BLINK,GPIO.OUT)

Dept. of Electronics & Communication Engineering 19

Virtual Assistant

now = datetime.now()

def askme():

r=sr.Recognizer()

with sr.Microphone() as scource:

print("Speak")

GPIO.output(BLINK,True)

audio = r.listen(scource)

GPIO.output(BLINK,False)

try:

print(r.recognize_google(audio))

if ("time" in r.recognize_google(audio)):

ts.voice('It is '+ now.strftime("%I:%M %p") +' on the clock')

print('It is '+ now.strftime("%I:%M %p") +' on the clock.')

elif ("date" in r.recognize_google(audio)):

ts.voice('Today is '+ now.strftime("%d") +' of '+ now.strftime("%B") + now.strftime("%Y"))

print('Today is '+ now.strftime(" %d ") +' of '+ now.strftime(" %B ") + now.strftime(" %Y ") + '.')

elif (" day" in r.recognize_google(audio)):

ts.voice('It is '+ now.strftime("%A"))

print('It is '+ now.strftime("%A") + '.')

elif ("weather" in r.recognize_google(audio)):

weather.Weather()

elif ("train" in r.recognize_google(audio)):

print(r.recognize_google(audio))

train.details()

elif ("search" in r.recognize_google(audio)):

wiki.Wiki()

elif ("you do" in r.recognize_google(audio)):

ts.voice('I can help you with a few things. You can ask me questions related to the current date, time,
weather or queries related to train data.'+

'Just tell me to search for information on wikipedia. I can also control all the lights and fans of a
building.'+

Dept. of Electronics & Communication Engineering 20

Virtual Assistant

'You can get started by saying my name out loud.')

elif ("name" in r.recognize_google(audio)):

ts.voice('You can call me Green.')

elif ("light" in r.recognize_google(audio)):

st=r.recognize_google(audio)

print(st)

iopin.devices(st)

elif ("fan" in r.recognize_google(audio)):

if('on' in r.recognize_google(audio)):

ts.voice('Turning the fan ON.')

GPIO.output(FAN,True)

elif('off' in r.recognize_google(audio)):

ts.voice('Turning the fan OFF.')

GPIO.output(FAN,False)

else:

ts.voice('I\'m sorry. I cannot help you with that yet.')

except:

ts.voice('I am sorry, I did not get you')

print('I am sorry, I did not get you')

return

6.2.4 - text_to_speech.py
from gtts import gTTS

import os

def voice(audio):

tts = gTTS(text=audio, lang ='en')

tts.save('audio.mp3')

os.system("mpg321 audio.mp3")

return

6.2.5 - train.py
import requests , json

Dept. of Electronics & Communication Engineering 21

Virtual Assistant

import text_to_speech as ts

import speech_recognition as sr

r=sr.Recognizer()

def details():

ts.voice('please tell the train number')

with sr.Microphone() as scource:

print("Speak")

audio = r.listen(scource)

x=r.recognize_google(audio)

api_key="d2mh5mt66z"

base_url="https://api.railwayapi.com/v2/live/train/"

train_number=x

current_date="06-07-2018"

complete_url=base_url + train_number + "/date/" + current_date + "/apikey/" + api_key + "/"

response_ob = requests.get(complete_url)

print(complete_url)

result = response_ob.json()

if result["response_code"] == 200 :

train_name = result["train"]["name"]

y = result["route"]

source_station = y[0]["station"]["name"]

destination_station = y[len(y)-1]["station"]["name"]

position = result["position"]

ts.voice("train name : " +str(train_name)+ " source station :" +str(source_station)

+ " destination station : " +str(destination_station) + " current station : "

+ str(position))

else:

ts.voice("record is not found for given request")

return

Dept. of Electronics & Communication Engineering 22

Virtual Assistant

6.2.6 - weather.py
import requests

import text_to_speech as ts

def Weather():

url =
'http://api.openweathermap.org/data/2.5/weather?appid=561470f211323497f57d99a4ae889ffd&q=bangalore'

json_data = requests.get(url).json()

forecast = json_data['weather'][0]['description']

temperature = json_data['main']['temp']

humidity = json_data['main']['humidity']

tmax = json_data['main']['temp_max']

tmin = json_data['main']['temp_min']

ts.voice('The Weather is like to be'+forecast + ' with a temperature of '

+ str(temperature-273.15) + 'degree Celsius and with an absolute humidity of '

+ str( humidity) + 'The temperature is estimated to go upto'

+str(tmax-273.15) + 'degree Celsius and might come down to '

+ str(tmin-273.15) + 'degree Celsius')

return

6.2.7 - wikipedia.py
import speech_recognition as sr

import text_to_speech as ts

import wikipedia

import RPi.GPIO as GPIO

GPIO.setwarnings(False)

GPIO.setmode(GPIO.BOARD)

GPIO.setup(7,GPIO.OUT)

def Wiki():

r=sr.Recognizer()

with sr.Microphone() as scource:

ts.voice('What do you want to know about?')

Dept. of Electronics & Communication Engineering 23

Virtual Assistant

print("What do you want to know about?")

GPIO.output(7,True)

audio = r.listen(scource)

GPIO.output(7,False)

print(r.recognize_google(audio))

try:

print(wikipedia.summary(r.recognize_google(audio),sentences=2))

s=(wikipedia.summary(r.recognize_google(audio),sentences=2))

ts.voice(s)

except:

ts.voice(r.recognize_google(audio)+' has multiple results. Please be more specific.')

Dept. of Electronics & Communication Engineering 24

Virtual Assistant

CHAPTER 7 – RESULT

Voice Command Speech Output

• Test case 1: • Output 1:
• What is the time? • The time is ‘SYSTEM_TIME’ on
the clock.
• Test case 2:
• Output 2:
• What is the date?
• Today is ‘SYSTEM_DATE’ of
‘SYSTEM_MONTH’
‘SYSTEM_YEAR’

• Test case 3: • Output 3:

• What day is it today? • It is ‘SYSTEM_DAY’.

• Test case 4: • Output 4:

• How is the weather? • The weather is likely to be

‘FORECAST’ with a temperature
of ‘TEMPERATURE’ °C and with
an absolute humidity of
‘HUMIDITY’. The temperature is
estimated to go up to
‘MAX_TEMPERATURE’ °C and
might come down to
‘MIN_TEMPERATURE’ °C.

Dept. of Electronics & Communication Engineering 25

Virtual Assistant

Voice Command Speech Output

• Test case 5: • Output 5:
• Can you search something • Rahul Dravid born on 11
for me?  Rahul Dravid January 1973, is a former
Indian cricketer and captain,
widely regarded as one of the
greatest batsmen in the history
of cricket. He is the current
Overseas Batting Consultant
for the Indian team, and also
the head coach for the Under-
19 and 'A' teams.

• Test case 6:
• Output 6:
• Turn the fan ON.
• Turning the fan ON.  (DC
motor is turned ON)

• Test case 7:
• Turn the ‘___ROOM’ light • Output 7:
ON/OFF.
• Turning the ‘___ROOM’ light
ON/OFF.  (LED is turned
ON/OFF)

Dept. of Electronics & Communication Engineering 26

Virtual Assistant

CHAPTER 8 - DISCUSSION

8.1 - Challenges

The challenges that were encountered during the project tenure and the solutions
to those challenges are as follows:

 Usage of gTTS instead of pyttsx: Initially python-text-to-speech package was made

use of for converting text to speech. The robotic voice output was not clear and the
pronunciation of certain English words were not indistinct. The voice output was
incomplete for long sentences. In order to set this right, gTTS or google text-to-speech is
being made use of. The voice output is clear along with the pronunciation.

 Usage of mpg321 instead of VLC player: Initially VLC was used to play the
output responses. For this, the user had to open VLC and click play manually. Python did
not support audios to be played on VLC directly using the OS package. mpg321, an mp3
player is made use of in place of VLC. The main advantage of using the player is that the
audio can be played directly while the program is on the run using the OS package in
python.

 Lack of a codec for mpg321: The mpg321 does not come with an in-built codec.
So, any audio file played gave an exception while the program was running. FLAC is an
audio coding format for lossless compression of digital audio. This had to be installed for
mpg321 to work as a regular audio player.

 Usage of a sound card for microphone: Initially a USB microphone was used as
an input transducer to input the voice commands from the user. But this microphone did
not have a good SNR and hence the added noise or the background noise was more evident
than the actual voice command. A microphone with 3.5mm jack is being used with a sound
card. This has a better noise cancellation when compared to the USB microphone. This
microphone is more accurate in converting the audio signal into text.

 Connecting the Pi to a laptop: Initially many devices like keyboard, mouse, monitor
etc. had to be interfaced with the Raspberry Pi. Hence the system was not portable. In order
to make the system portable the Pi was connected to a laptop using Remote Desktop
Connection application with the help of which the Pi’s display could be got on to the laptop.

Dept. of Electronics & Communication Engineering 27

Virtual Assistant

CHAPTER 9 – CONCLUSION

The project resulted in a prototype that is capable of controlling remote-controlled outlets

through them many different kinds of home equipment. The user can perform this control either
by using voice commands. The prototype is always ready to receive commands from the user.

The prototype developed during the project performed with satisfying results. The accuracy
reached with users that has distinct pronunciation resulted in a limited need to repeat commands.
The chosen hardware and framework did not have maximum compatibility which meant that an
optimal result could not be reached. To increase performance the prototype would need more
computational power, which would incur a higher cost. This increased cost would counter the
affordability property defined in the project purpose. It might have been better to use a
SpeechRecognition framework based on a lower-level language, for example C, and a more light-
weight framework with less demanding search algorithms. These changes would probably
optimize the execution of the software on the Raspberry Pi, which could lead to a more satisfying
result.

The properties defined by the project purpose were fulfilled. The prototype has a relatively
low cost and is easy to use. The implemented functionality corresponds well to the questions from
the user.

Dept. of Electronics & Communication Engineering 28

Virtual Assistant

CHAPTER 10 – FUTURE SCOPE

A virtual assistant can be an invaluable tool to a business, a startup, or brand. They can
handle tasks like scheduling, researching, social media, and more. Furthermore, they help in
thinking more creatively and thus driving the business forward. A virtual assistant can help ease
the tasks so that more emphasis can be laid for the future.
The use of virtual assistant for home automation provides great scope for technology as it
gives the user, access to control devices like lights, fans, and other appliances remotely. This not
only makes the work simpler but by adding some features like machine learning and artificial
intelligence it can be made more smart, secure and implemented for larger areas.
The use of virtual assistants for automation is not limited only to homes. Automation has
stirred the industrial sectors drastically changing the way of production, reducing the
manufacturing time, and also the production cost. These advances made in automation of various
activities, that were formerly carried out manually, have resulted in improved efficiency and saving
labour cost. This has reduced job opportunities. So, these virtual assistants have been affecting
mankind both positively and negatively.
On the whole, these virtual assistants can be of great assistance both professionally and
personally since it has the ability to be customized according to the requirements and has the
capability only to evolve with time.

Dept. of Electronics & Communication Engineering 29

Virtual Assistant

APPENDIX A

COMPONENTS

The main components used to build the assistant are:

 Raspberry Pi 3 model B
 Sound Card
 Microphone
 Speakers
 DC motor
 MPS2222A (Transistor)
 1N4148 (Diode)
 LED
 Resistors
 Bread Board
 Connecting wires

A.1 – Raspberry Pi
The Raspberry Pi is a series of small single-board computers developed in the United
Kingdom by the Raspberry Pi Foundation to promote the teaching of basic computer science in
schools and in developing countries. It does not include peripherals (such as keyboards and mice)
and cases.

A.1.1 - Raspberry Pi 3 model B

The Raspberry Pi 3 Model B is the earliest model of the third-generation Raspberry Pi. It
replaced the Raspberry Pi 2 Model B in February 2016. The specifications of Raspberry Pi 3B are:

 Quad Core 1.2GHz Broadcom BCM2837 64bit CPU

 1GB RAM
 BCM43438 wireless LAN and Bluetooth Low Energy (BLE) on board
 100 Base Ethernet
 40-pin extended GPIO
 4 USB 2 ports
 4 Pole stereo output and composite video port
 Full size HDMI
 CSI camera port for connecting a Raspberry Pi camera
 DSI display port for connecting a Raspberry Pi touchscreen display
 Micro SD port for loading your operating system and storing data

Dept. of Electronics & Communication Engineering 30

Virtual Assistant

 Upgraded switched Micro USB power source up to 2.5A

Figure 10.1 - Raspberry Pi 3 model B

10.1.2 - GPIO Header

A powerful feature of the Raspberry Pi is the row of GPIO (general-purpose

input/output) pins along the top edge of the board. A 40-pin GPIO header is found on all current
Raspberry Pi boards (unpopulated on Pi Zero and Pi Zero W). Prior to the Pi 1 Model B+ (2014),
boards comprised a shorter 26-pin header.

Any of the GPIO pins can be designated (in software) as an input or output pin and
used for a wide range of purposes.

Dept. of Electronics & Communication Engineering 31

Virtual Assistant

Figure 10.2 - GPIO Layout

Voltages:

Two 5V pins and two 3.3V pins are present on the board, as well as a number of ground
pins (0V), which are unconfigurable. The remaining pins are all general purpose 3V3 pins,
meaning outputs are set to 3V3 and inputs are 3V3-tolerant.

Outputs:

A GPIO pin designated as an output pin can be set to high (3V3) or low (0V).

Inputs:

A GPIO pin designated as an input pin can be read as high (3V3) or low (0V). This is
made easier with the use of internal pull-up or pull-down resistors. Pins GPIO2 and GPIO3 have
fixed pull-up resistors, but for other pins this can be configured in software.

PWM (pulse-width modulation):

The software PWM is available on all pins and hardware PWM is available on GPIO12,
GPIO13, GPIO18, GPIO19.

Driver Support:

The Foundation will not include a GPIO driver in the initial release, standard Linux
GPIO drivers should work with minimal modification. The community implemented SPI and I²C

Dept. of Electronics & Communication Engineering 32

Virtual Assistant

drivers which will be integrated with the new Linux pin control concept in a later version of the
kernel. The I²C and SPI driver uses the hardware modules of the microcontroller and interrupts for
low CPU usage, the 1-wire support uses bit banging on the GPIO ports, which results in higher
CPU usage.

MIPI CSI (Camera Serial Interface):

On the production board, the Raspberry Pi Foundation design brings out the MIPI CSI-2
(Camera Serial Interface) to a 15-way flat flex connector S5, between the Ethernet and HDMI
connectors. A compatible camera with 5 Megapixels and 1080p video resolution is released.

DSI (Display Serial Interface):

On the production board, the Raspberry Pi Foundation design brings out the DSI
(Display Serial Interface) to a 15-way flat flex connector labelled S2, next to Raspberry Pi logo. It
has two data lanes and a clock lane, to drive a possible future LCD screen device. Some smart
phone screens use DSI.

A.2 – Sound card

A sound card (also known as an audio card) is an internal expansion card that provides
input and output of audio signals to and from a computer under control of computer programs. The
term sound card is also applied to external audio interfaces used for professional audio
applications. Sound functionality can also be integrated onto the motherboard, using components
similar to those found on plug-in cards. The integrated sound system is often still referred to as a
sound card. Sound processing hardware is also present on modern video cards with HDMI to
output sound along with the video using that connector; previously they used a SPDIF connection
to the motherboard or sound card. Typical uses of sound cards or sound card functionality include
providing the audio component for multimedia applications such as music composition, editing
video or audio, presentation, education and entertainment (games) and video projection. Sound
cards are also used for computer-based communication such as voice over IP and teleconferencing.

Figure 10.3 - Sound Card

Dept. of Electronics & Communication Engineering 33

Virtual Assistant

A.3 – Microphone
A microphone is a transducer that converts sound into an electric signal. Microphones are
used in many applications such as telephones, hearing aids, public address systems for concert
halls and public events, motion picture production, live and recorded audio engineering, sound
recording, two-way radios, megaphones, radio and television broadcasting, and in computers for
recording voice, speech recognition, VoIP, and for non-acoustic purposes such as ultrasonic
sensors or knock sensors.

In the dynamic microphone, sound waves cause movement of a thin metallic diaphragm
and an attached coil of wire. A magnet produces a magnetic field which surrounds the coil, and
motion of the coil within this field causes current to flow. The principles are the same as those that
produce electricity at the utility company, realized in a pocket-sized scale. It is important to
remember that current is produced by the motion of the diaphragm, and the current in the coil is
channeled from the microphone along wires.

Figure 10.4 – Microphone

A.4 – Speaker
A speaker is an electroacoustic transducer which converts which converts an electric
signal into sound energy. The dynamic speaker operates on the same basic principle as a dynamic
microphone, but in reverse, to produce sound from an electrical signal. When an alternating current
electrical audio signal is applied to its voice coil, a coil of wire suspended in a circular gap between
the poles of a permanent magnet, the coil is forced to move rapidly back and forth due to Faraday's
law of induction, which causes a diaphragm (usually conically shaped) attached to the coil to move
back and forth, pushing on the air to create sound waves. Besides this most common method, there
are several alternative technologies that can be used to convert an electrical signal into sound. The
sound source (e.g., a sound recording or a microphone) must be amplified or strengthened with an
audio power amplifier before the signal is sent to the speaker.

Dept. of Electronics & Communication Engineering 34

Virtual Assistant

Figure 10.5 – Speaker

A.5 – DC motor
A DC motor is any of a class of rotary electrical machines that converts direct
current electrical energy into mechanical energy. The most common types rely on the forces
produced by magnetic fields. Nearly all types of DC motors have some internal mechanism, either
electromechanical or electronic; to periodically change the direction of current flow in part of the
motor.

This DC or direct current motor works on the principal which states that, ‘when
a current carrying conductor is placed in a magnetic field, it experiences a torque and tends to
move’. This is known as motoring action. If the direction of current in the wire is reversed, the
direction of rotation also reverses. When magnetic field and electric field interact, they produce a
mechanical force, and based on that the working principle of DC motor is established.

Figure 10.6 - DC Motor

A.6 – MPS2222A (Transistor)

MPS2222A is an NPN bipolar small signal transistor designed for use in linear
and switching applications. Transistor is a semiconductor device used to amplify or switch
electronic signals and electrical power. It is composed of semiconductor material usually with at
least three terminals for connection to an external circuit. A voltage or current applied to one pair
of the transistor's terminals controls the current through another pair of terminals. Because the
controlled (output) power can be higher than the controlling (input) power, a transistor can amplify
a signal.

Dept. of Electronics & Communication Engineering 35

Virtual Assistant

The MPS2222A transistor consists two PN diode connected back to back. It has
three terminals namely emitter, base and collector. The base is the middle section which is made
up of thin layers. The right part of the diode is called emitter diode and the left part is called
collector-base diode. These names are given as per the common terminal of the transistor. The
emitter-based junction of the transistor is connected to forward biased and the collector-base
junction is connected in reverse bias which offers a high resistance.

Figure 10.7 – MPS2222A

A.7 – 1N4148 (Diode)

The 1N4148 is a standard silicon switching signal diode. It is one of the most
popular and long-lived switching diodes because of its dependable specifications and low cost. Its
name follows the JEDEC nomenclature. The 1N4148 is useful in switching applications up to
about 100 MHz with a reverse-recovery time of no more than 4 ns.

A diode is a two-terminal electronic component that conducts current primarily

in one direction (asymmetric conductance); it has low (ideally zero) resistance in one direction,
and high (ideally infinite) resistance in the other. A diode vacuum tube or thermionic diode is a
vacuum tube with two electrodes, a heated cathode and a plate, in which electrons can flow in only
one direction, from cathode to plate. A semiconductor diode, the most common type today, is a
crystalline piece of semiconductor material with a p–n junction connected to two electrical
terminals.

Figure 10.8 – 1N4148 (Diode)

Dept. of Electronics & Communication Engineering 36

Virtual Assistant

A.8 – LED (Light Emitting Diode)

A light-emitting diode (LED) is a two-lead semiconductor light source. It is a p–

n junction diode that emits light when activated. When a suitable current is applied to the lead’s
electrons are able to recombine with electron holes within the device, releasing energy in the form
of photons. This effect is called electroluminescence, and the color of the light (corresponding to
the energy of the photon) is determined by the energy band gap of the semiconductor. LEDs are
typically small (less than 1 mm2) and integrated optical components may be used to shape the
radiation pattern.

LEDs have many advantages over incandescent light sources, including lower
energy consumption, longer lifetime, improved physical robustness, smaller size, and faster
switching. Light-emitting diodes are used in applications as diverse as aviation lighting,
automotive headlamps, advertising, general lighting, traffic signals, camera flashes, lighted
wallpaper and medical devices. They are also significantly more energy efficient and, arguably,
have fewer environmental concerns linked to their disposal.

Figure 10.9 – LED

Dept. of Electronics & Communication Engineering 37

Virtual Assistant

REFRERENCE
[1] Raspbian, [online]. Available: https://www.raspberrypi.org/downloads/raspbian/.
[2] Python, [online]. Available: https://www.python.org/.
[3] Python, [online]. Available: https://www.w3schools.com/python/.
[4] GIT, [online]. Available: https://github.com/.
[5] Install Raspbian OS, [online]. Available: https://maker.pro/raspberry-pi/tutorial/how-to-
install-raspbian-on-your-raspberry-pi-sd-card.
[6] Weather API, [online]. Available: https://openweathermap.org/api.
[7] Railway API, [online]. Available: https://railwayapi.com/.

Dept. of Electronics & Communication Engineering 38