Towards Richer Assisted Living Environments.

This paper describes an ongoing research project which explores the design and use of inexpensive robotics, artificial intelligence techniques, and human-computer interaction methods, to enrich assisted living environments. Such environments provide help to the inhabitants of a home or office, assisting them to perform daily activities, helping them to socialize and interact with others, and to provide enhanced levels of security and safeness. We present the development of an inexpensive robotic solution to help people with disabilities and/or older adults to perform their daily activities. It can be used as a remote controlled surveillance system, and also as a personal assistant. It is able to recognize each inhabitant, his/her emotions, and detect abnormal situations such as falls and health problems. The whole system is designed to operate solely within a local network and special attention is given to the privacy and data protection of the users.

Introduction

An assisted living environment offers a set of devices and tools to assist inhabitants to perform daily activities such as controlling the lighting, temperature, humidity, media centers, and all sorts of appliances. The automation of these tasks in a home environment is commonly referred as domotics and is accomplished by integrating sensors, actuators, and home appliances, in a coherent way under a network environment. In essence, this technology transforms a house into a so-called smart house.

Domotics can have a positive impact in people’s life, increasing their levels of comfort and quality of life. This is especially relevant in the case of people with limitations1, where the use of technology is of paramount importance for the execution of a variety of tasks that would not be possible otherwise. The benefits of technology tend to be emphasized when used by people with limitations. Consider for example a screen reader. Such a technology can be useful for drivers to listen to text messages while driving to work. The advantages for blind people, however, are of another order of magnitude, allowing them to have access to digital documents and navigate on the world wide web.

Although the proliferation of home automation solutions has been increasing over the years, the resulting living environments are still, for the most part, confined to perform relatively simple tasks. Moreover, they are often not designed with enough concern towards accessibility issues. Given the technological advances that have been occurring in the fields of Robotics, Artificial Intelligence (AI), Augmented and Virtual Reality (AR/VR), and Human–Computer Interaction (HCI), we envision a future with assisted living environments that are substantially richer than those that exist today. Our research work has been guided with this vision in mind, and we have the following goals:The rest of the article is organized as follows. Section 2 presents an overview of past work done in this area. Section 3 describes a prototype of a robotic assistant solution that was developed to test our initial ideas, along with informal observations collected from a group of users. Section 4 describes our subsequent efforts in understanding user needs, as well as concerns with privacy and data protection for this type of systems. Section 5 describes a further iteration of our user-centered development approach, namely the integration of AI techniques for recognizing people and their emotions as well as postures and abnormal situations. Finally, Sect. 6 presents ideas for future work, and Sect. 7 summarizes and concludes the paper.

Develop low cost robotic solutions to help people with limitations in performing daily tasks in an intelligent home environment.

Design and evaluate more natural interaction methods to allow this group of people to have a better user experience, not only with their surrounding environment but also with a remote one via telepresence.

Increase the autonomy of people with limitations, providing mechanisms that allow them to have a more independent life.

Previous Work

Domotics have been used to develop smart environments enabling people with limitations to interact with physical objects (e.g. a paraplegic person can use a remote control or mobile application to open a door or turn on/off a light), increasing their level of independence in their homes [1, 9, 12]. Delnevo et al. [12] presented a system that was designed to improve the accessibility of buildings, using Raspberry Pi and users’ mobile devices. Alrajhi et al. [4] developed a Brain Computer Interfacing based smart environment that allows a person with quadriplegia to open/close doors. Domotic technologies have also been used to build applications for improving the well-being of older adults [42]. For instance, fall detection systems have been developed to alert other inhabitants [13, 21].

A novel and inexpensive WIFI-based system for human monitoring was introduced by y Bassoli et al. [5]. This approach uses standard WIFI networks as an alternative to well established protocols such as Z-Wave or ZigBee [17]. A system based on WIFI connectivity has an advantage over the more traditional communication protocols: All devices can be easily connected to the Internet using a conventional router, without the need to acquire a specific home gateway to allow connectivity. However, as described by Bassoli et al. [5], the main issue of the WIFI connectivity is the higher power consumption. A system that combines a single-board micro-controller (Arduino UNO) as processing unit, a low-cost WIFI microchip (ESP8266) and many sensors (a dust sensor, aCO2 sensor, a light sensor, and a temperature and humidity sensor) was developed to monitor indoor air quality of assisted living environments [31].

An interesting study was published by Malasinghe et al. [30], in which they provide an extensive review of “advances in remote healthcare and monitoring” using both with-contact and contactless methods, namely heart and blood monitoring systems, diabetes monitoring systems, monitoring systems for neurological diseases, and fall detection systems. With these technologies, people with disabilities and/or older people can perform their daily activities at home. Fall detection systems are especially important in smart homes designed for people with limitations.

Shalom Greene [20] proposed a system that integrates fall detection into a smart home. His main goal was to provide a safe environment for older adults. As described by him [20, 21], a Tiva C Launchpad with a WIFI expansion and an accelerometer is used to detect the falls. When a fall is detected, it communicates with a Raspberry Pi via HTTP requests and then, using text-to-speech interactions with the person. Using a device with speech recognition capabilities, more specifically the Amazon Echo device, this system is able to receive the confirmation of the detected fall. In addition, researchers have been developing alternative interaction methods to allow people with severe coordination problems to control their home appliances [4, 6, 9, 14, 46].

When designing this type of domotic solutions, it is important to understand the needs and desires of all the inhabitants (with or without limitations) of a house, providing a rich experience for all. Another important aspect is to understand the way each individual interacts with the devices and with other residents, so that better solutions can be designed [6, 9].

Shalom Greene [9] designed a framework that allows people with disabilities and their family members to control home appliances using a mobile device. That work was inspired by a text-entry method for people with motor coordination problems which uses the metaphor of navigating through groups and sub-groups of sets of options [10, 18]. The concept of navigating through different groups seemed the prefect metaphor for walking between rooms to turn on/off home appliances. Based on previous studies [8, 18, 29], as well as on informal interviews conducted with people with cerebral palsy [9], an interface was designed allowing each user to adapt and use it according to her/his own needs.

The assisted living environment described in Condado and Lobo [9] was built and evaluated with an unconventional approach. The main goal was to study the proposed system in a real life scenario, learning more about the real interaction between the inhabitants and the home appliances. This was achieved using inexpensive and well-known domotic technologies, low-powered computers, single-board microcontrollers (e.g. Arduino boards), sensors and actuators.

Some researchers used neural network models to develop work to promote accessibility. Alam et al. [3] proposed a framework called SafeAccess that makes intelligent environments safer for people with disabilities.Abiyev and Arslan [2] also presented a head mouse system based on convolutional neural networks (CNN) to allow people with severe motor coordination problems to control computer devices.Rabhi et al. [35] describe another interaction method based on a model that uses neural networks and a specific image preprocessing step that allows people to control wheelchairs using their facial expressions. A gaze region estimation system, which also uses deep CNN to estimate gaze direction, was proposed by Kato et al. [26]. Ding and Wang [13] proposed a WIFI-based fall detection system, which uses a recurrent neural network (RNN) model to classify human movement and recognize a fall situation. Shimakawa et al. [39] developed a mobile application to assist people with visual impairments to walk. The application recognizes dangerous situations (e.g. obstacles) from images acquired by a camera and alerts the user.

Computer vision and deep learning have been used to build fall detection systems [34], as well as eHealth solutions, and these techniques are essential to develop a rich assisted environment.

A Summary and Perspective of the Literature

A variety of systems have been proposed for helping people with limitations in assisted living environments, namely special-purpose user interfaces [2, 4, 6, 9, 14, 26, 36, 46], monitoring systems [3, 5, 22, 30, 33, 40], fall detection systems [13, 21, 34], and devices based on different communication protocols [5, 17, 19]. Table 1 summarizes the literature in this area, highlighting the advantages and disadvantages of the various approaches.

Table 1

Summarization of domotic systems and assistive technologies for helping people with limitations.

Technology	Advantages and disadvantages
Interaction methods based on speech recognition used to control robotic solutions [25] and intelligent environments [21, 36, 46]	These methods can be easily used by people with and without disabilities. They are particularly useful for those who have quadriplegia. However, as described in Condado and Lobo [9] and in Koch Fager et al. [27], they are still not practical for those with severe speech disorders (e.g. people with cerebral palsy) because the recognition accuracy is still too low (see sect. 3)
Brain-Computer Interfaces for intelligent environments control [4, 14]	These methods can be used by people with quadriplegia. They are not contactless methods and that is a disadvantage, but they, as well as systems controlled by facial expression [35] and by hand gestures [6], can be used by people with severe speech disorders
Fall detection systems [13, 21, 34] and monitoring systems [3, 5, 22, 30, 33, 40]	These systems are essential to ensure the well-being of inhabitants with limitations. As described by Malasinghe et al. [30], there are systems based on both with-contact and contactless methods. Contactless methods appear to be better than with-contact systems because they are “transparent” for users
Adaptable user interfaces [9, 16]	These interfaces have the advantage of being adjustable to each user’ limitations. They appear to be suitable for both people with moderate limitations and well as those without limitations. However, they are not appropriate for people with severe disabilities
Systems based on cloud services [21, 32]	These systems are simple and easy-to-use, but there are privacy concerns associated with these technologies which shouldn’t be neglected lightly. For instance, the use of Amazon Echo can compromise users’ privacy [41]

Despite all these advances, a more holistic view is still missing in the area. In general, there is still lack of interoperability, excessive energy consumption, not much concern with the privacy and data protection of the users, and the technology introduced is still somewhat intrusive and unpersonalized. Below we elaborate on these problems.The implementation of gateways can solve the issues related to the lack of interoperability mentioned previously as well as avoid the need to be fully dependent on WIFI, by allowing the communication between equipments based on different protocols.

There is no standard protocol for domotic devices. This often makes the interoperability between devices of different brands to be difficult and costly.

WIFI has become ubiquitous, and WIFI-based systems can be easily connected to the Internet router present in the majority of homes [5]. Nonetheless, at the present time WIFI devices consume much more energy than those based on other protocols. Therefore, an intelligent house totally based on WIFI communication ends up having an unnecessarily high energy consumption, especially considering that many devices only need a tiny fraction of the bandwidth offered by WIFI networks.

Fall detection and monitoring systems are very important to promote an intelligent and safe environment for people with limitations. Many such systems use conventional cloud systems such as Amazon Echo or Google Home to do such monitoring and/or to do speech recognition [20, 21]. The utilization of cloud services for this task, however, raises serious concerns with respect to the privacy and data protection of the users, as referred by Sun et al. [41].

Several systems use contact-based methods for monitoring the users’ daily activity, which can be a bit intrusive and make users feel uncomfortable.

Regarding privacy and data protection, we advocate (see later in Section 4.1) that they are very important and have to be guaranteed in an assisted living environments. Data collected with respect to the daily activity of users in their living environment should not go to external servers without their explicit consent. Ideally, all such data should only be stored locally within the users’ building itself.

We advocate that monitoring systems should be as least intrusive as possible. Therefore, contactless forms of monitoring seem to be better suited for this endeavor. The advances in computer vision and deep learning, together with the reduced cost of cameras, make fall detection systems based on contactless forms doable and appealing [34].

Overall, the work that we are about to propose in this paper has similarities with the approach given by Pourazad et al. [34], but has a broader view. We advocate that an intelligent environment should be personalized and be able to not only detect falls and monitor health problems, but also recognize the user, know about his/her historical condition, limitations, and even his/her emotional state. The presence of monitoring cameras on the various rooms of a house can feel intrusive for those who live there, and the development of a small robot that act as a “personal assistant” has a certain appeal in that it can even be looked at as a companion to the user.

A First Prototype of a Robotic Assistant

Following well established development and design principles from the field of HCI, we used an iterative user-centered approach to develop and enhance the assisted living environment first described in Condado and Lobo [9]. Over the years, this environment has been improved to be more reliable, secure, and flexible. After all, it has been used in a real context to assist Paulo’s family since 2013. For instance, the system was adapted to work with Z-Wave, a protocol that has been widely used to build smart environments [19], and we also explored other communication protocols (e.g. ZigBee [17]) to develop a secure multi-protocol gateway.

Our first step towards a more intelligent and useful environment was the idea of having a small inexpensive robot that could act as an assistant to the inhabitants of the house. The idea is to give inhabitants the ability to have the robot move around the house while they can watch in real time what is happening in different parts of the house. The main goal is to allow people with severe motor disabilities (e.g. paraplegic people) to have an alternative low-cost surveillance system.

This first prototype was implemented with inexpensive electronic components. It has ultrasonic sensors to avoid collisions and to allow a person with severe hand coordination problems to use the robot in autonomous drive mode. A webcam is installed on the robot’s chassis to transmit real-time images for control and surveillance purposes.

A Raspberry Pi, a low-cost credit-card-sized computer, with a wireless dongle and an Arduino UNO are used as the “brain” of the robot. The Arduino UNO, an inexpensive microcontroller, is used to control two DC motors, get data from sensors, and communicate with the computer.

This rudimentary robot (see Fig. 1) is powered by two small batteries, has an autonomy of about 70 minutes, and was built for testing purposes to quickly try different ideas. To be of practical use, the robot should have a much better autonomy (say at least 12 hours) and have more sophisticated and powerful motors, not to mention being more aesthetically pleasing.

Fig. 1

A rudimentary prototype of the robotic assistant.

The robot is controlled remotely via a mobile application that we developed (see Fig. 2) with adjustable user interface components to suit users according to their own abilities, following the same ideas used in Condado and Lobo [9]. The user interface has six elements. The first one is placed at the top-center and allows users to watch real time images transmitted from the robot’s webcam. The other five elements are buttons to control the movement of the robot: move forward, move backward, turn left/right while moving forward, and move in autonomous mode.

Fig. 2

The first functional prototype of our mobile application. It allows to control the robot to move around the house.

At a first stage, there was no full integration between the robot and the system described in Condado and Lobo [9], even though, as we shall refer later in the paper, such integration is crucial in a truly smart environment. Although this first robotic solution had limitations, such as a low autonomy level, the prototype was tested by both Paulo (person with cerebral palsy) and his wife. Since it was also designed to work in standalone mode, other people were able to test it as well. Our informal study revealed that an adjustable user interface appears to be appropriate for those with moderate motor disorders.

We agree with an approach defended by Intille [24], in which smart environments should be implemented to empower people with appropriate mechanisms to perform their daily activities, and argue that a good user interface should be designed to allow both people with or without limitations to control the environment. Given the distinction between adaptable and adaptive systems [16], we advocate that adaptable user interfaces, also known as adjustable user interfaces [9], should be implemented to give users the ability to adapt them to their needs and preferences. Moreover, machine learning techniques [33, 40] should be used for activity recognition to suggest important actions to the inhabitants (e.g. suggest to open or close windows based on weather conditions and the users’ previous actions).

We also studied alternative interaction methods, such as electromyography (EMG)-based interfaces [45], to achieve a more natural way to control assistive robotic solutions. Muscle interfaces may provide an immersive way to interact with the surrounding world, allowing people to perform complex tasks [45]. Interaction methods based on speech recognition have also been used to control robotic solutions [25] and intelligent environments [36, 46], but, as described in Condado and Lobo [9] and in Koch Fager et al. [27], although there are some developments in speech recognition systems for people with speech disorders [23, 27], they are still not practical for those with severe speech disorders (e.g. people with cerebral palsy) because the recognition accuracy is still too low.

Observations and Informal Interviews

A small group of four people (with and without disabilities) tested the proposed robotic solution. This is the same group of people referenced in Condado and Lob [9]. We collected feedback from the group. In general they were satisfied with the user interface of the mobile application to control the robot, and liked the ability to adjust the user interfaces components to suit their abilities. Some members of the group suggested that we should try to design more natural interaction methods that could perhaps eliminate the need to use the mobile app in certain specific situations. For example, a certain agreed-upon gesture could be use to mean “turn on the light in the room the person is currently in”.

A common observation from all was that the robot was too fragile, and sometimes had trouble in moving around properly when faced with small obstacles on its way. For example, it had trouble going over a carpet.

Below we list a number of suggestions given by members of the group: We have been working on the implementation of these tasks, and we shall describe them in more details later in the paper.

Be capable of driving in a more rugged terrain

Recognize each inhabitant

Predict each inhabitant’s behavior

Detect abnormal situations such as falls or health problems

Recognize each inhabitant’s emotional state

Provide an appropriate environment based on that emotional state

Communicate with other smart devices

Assist inhabitant’s to perform their daily activities

Alert relatives or health care services about an abnormal event

Provide means to allow inhabitants (with and without disabilities) to communicate with others

Further Understanding User Needs and Desires

The number of users that tested our system was small. Nonetheless, they gave us important feedback and helped us to improve our work. In our quest to further develop assistive living environments, and similarly to what we have done previously in Condado and Lob [9], we tried to conduct interviews for requirement elicitation. After all, to design an assistive technology it is essential to understand the users’ needs, limitations and desires. Unfortunately, due to the COVID-19 pandemic, we were unable to do face-to-face interviews and had to do them remotely via an online questionnaire. We note that some research studies revealed that the COVID-19 infection causes a higher mortality rate in older adults and in people with chronic diseases [38]. This is a vulnerable group of people [28] that must be protected. On other hand, although there is no evidence that people with disabilities are more vulnerable than the average, it is important to protect them because many people with disabilities have comorbid conditions.

A total of 35 persons (with and without limitations), 16 female and 19 male, filled an online questionnaire with questions about their own thoughts and ideas of a personal assistant robot and an intelligent environment. 10 persons had some sort of disability (mostly cerebral palsy) and the other 25 did not. An interesting observation is that 24 out of the 25 persons without disabilities reported that they know someone with disabilities.

Table 2 shows the number of participants by age range. Only one person was over 65 years of age. That’s unfortunate because older adults are an important target group for our study and it would have been good to have had the opportunity to collect more data regarding this group of people. We hypothesize that the low number of older participants occurred because this group of people, as opposed to the younger ones, has the tendency to feel a bit uncomfortable with technology and computers, and so many of them showed no interest in filling the online questionnaire.

Table 2

Distribution of participants by age range

Age range	# of participants
15—24	1
25—34	9
35—44	9
45—54	9
55—64	6
Over 65	1

All participants answered that the introduction of assistive robotic solutions into smart environments is useful. They also considered that the development of low cost robotic solutions may improve the quality of life of people with limitations. 97% of the participants answered that they considered that assistive technologies, as well as assisted living environments, may increase the level of independence of people with disabilities and/or older adults, having an important role to assist people with limitations to perform their daily tasks.

The questionnaire had the following question: “In a scale of 1 (no impact) to 4 (big impact), what is the impact of the development of domotic systems to promote the autonomy of people with limitations?” Most participants (83%) answered that it can have a big impact, and the rest (17%) considered that it can have impact.

One of the goals of the questionnaire was to understand how people perceive smart environments, and so the following question was also asked: “Which of the following statements correspond to your idea of a smart home?”

An intelligent environment with the ability to predict each inhabitant’ desires, performing all tasks without human intervention.

An environment that offers tools to allow inhabitants to perform their tasks effectively, using various interaction methods. The ability to control a smart home is in the users’ hands, but the system is able to suggest which activities should be accomplished (e.g. turn on/off a home appliance). However, in exceptional cases (e.g. a fall or a health problem is detected), the environment has the ability to take decisions autonomously to ask for help.

A non-intrusive environment that, although has the ability to predict users’ behaviors, only gives suggestions to perform certain tasks when asked.

An environment in which it is only possible to control home appliances using a remote control or a smartphone.

The results obtained from the above question are shown in Table 3 and allowed us to take some interesting conclusions. On one hand, most people appear to share a common view about intelligent environments. From their point of view, a suitable smart environment should be able to predict people’s behaviors. However, although 14,3% of them answered that an appropriate framework should be able to perform all tasks without human intervention, 71,4% of the people replied that a suitable smart environment should only take decisions in exceptional situations, empowering inhabitants to perform daily tasks. On the other hand, it is interesting to notice that only 5,7% answered that their perception of an intelligent environment was a non-intrusive system.

Table 3

Participants’ answers regarding their perceptions of a smart home.

Statement	Percentage
1	14.3%
2	71.4%
3	5.7%
4	8.6%

Moreover, there was a small group of participants (8,6%) who showed a conservative point of view on the concept of smart environments. From their perspective, a smart home is an environment in which it is only possible to control home appliances using a smartphone or a remote control. Thus, according to them, the system should not be able to predict users’ behaviors.

Privacy and Data Protection

Privacy and data protection are important and should not be handled lightly, especially in the context of a home environment. The introduction of technology in our houses cannot disrespect the right of users to preserve their privacy. Users of our previous work utilized an assisted living environment in a real scenario [9]. In that work we took care in protecting the users’ privacy by not storing any information on remote servers on the cloud. As a matter of fact, the aforementioned users explicitly expressed that they did not want to use domotic technologies based on cloud computing because they did not trust them entirely. We recognize, however, that the behavior of these users is not the norm: they were technologically savvy and well informed of the risks of exposing personal data.

Most people, however, do not seem to worry a lot about this issue. Based on informal interviews and on conversations with a variety of people, we noticed that most use devices such as smartphones, tablets, and computers, with information stored on the cloud. These people believe that data centers are secure and tend to trust known private companies (e.g. Google, Amazon, among others) to keep their information. However, the Cambridge Analytica scandal revealed that, sometimes, our personal data is used without our consent. Should we trust these companies with our personal data? It is a difficult question to answer.

We advocate that it is very important to ensure that inhabitants can interact with the surrounding world, protecting their privacy. We argue that a rich intelligent environment should not rely on the cloud infrastructure, but rather on a private local network with a low cost local machine such as a mini-PC or even a Raspberry Pi acting as a gateway. This is essential to ensure a secure communication between mobile devices, domotic equipments, and assistive solutions, and to preserve the privacy of users.

We note that there may be advantages in having the devices connected to the Internet. For instance, Miniaoui et al. [32] describe the use of RFID tags to keep track and manage items on a smart fridge. A web application monitors the items inside the fridge and it able to suggest recipes based on the existing items, alert the user when items are close to reach the expiration date, and many other things. Such recommendations can be very useful for a user, but there is no doubt that there are privacy concerns associated with these technologies which shouldn’t be neglected lightly. Ultimately, we believe it is important to at least give users the freedom to choose the level of privacy and data protection that they want.

Enhancing the Robotic Assistant

This section describes our ongoing work to address the suggestions given by the test users and reported in Sect. 3.1. It is a challenge to develop an inexpensive multi-functional robot that integrates these functionalities to enrich assisted living environments. Our strategy is to build small subsystems and combine them incrementally to obtain an overall complex solution.

To build the features desired by the test users, it is essential to explore AI techniques, and in particular deep convolutional neural network (CNN)-based approaches, such as FaceNet [37] for face recognition and OpenPose [7] for pose estimation.

FaceNet uses deep-CNN together with triplet loss to have a high accuracy in face recognition, verification and clustering [37]. We trained a FaceNet model with images containing faces of a group of users. Then, the assistive robot used the resulting model to recognize each user. The idea is that the robot is able to identify each user, as well as his/her limitations and desires, and provide the appropriate support. For example, the system can be used to detect the user’s emotions and adapt the environmental conditions accordingly (e.g. change the colour and/or intensity of the lights, play a user’s favorite music, turn on/off the air-conditioning, or “chat” with the user, among other things). Moreover, it should be able to track and recognize the user’s movement to detect falls, abnormal behaviors and health problems, alerting his/her family if needed.

Recognizing People and Their Emotions

The test users expressed the desire to have a robot that could recognize abnormal behaviors, detect dangerous situations in the smart environment, and alert inhabitants, their relatives or health care services. Therefore, we combined traditional computer vision techniques with deep learning models to tackle this task.

From a basic point of view, we are developing a system that uses different techniques to recognize dangerous situations from images acquired by cameras. First of all, since it is essential to recognize each user/inhabitant to determinate his/her needs, we used the FaceNet model [37]. Then, we integrated it with other CNN-based models to determine what an inhabitant is feeling and/or to detect the inhabitant’s posture.

As shown in Fig. 3, we trained CNN models to recognize facial emotions, and integrated it with the FaceNet model [37]. We achieved interesting results, but there is still additional work to do to have a more stable and robust system, and we are currently improving some aspects of it.

Fig. 3

Recognizing people and their emotions. In the example shown, the robot recognizes the person as being Paulo, and recognizes that he is in a happy emotional state.

Recognizing Postures and Detecting Abnormal Situations

We incorporated other models that allow the system to recognize users’ postures and detect abnormal situations, such as falls or health problems. OpenPose, developed by Cao et al. [7], appears to be a good approach for real-time human pose estimation (see Fig. 4). Besides, it allows to determine the pose of multiple persons in an image frame.

Fig. 4

Using OpenPose for pose estimation.

Combining these models allow us to build a complex and useful assistive robotic solution. It is also important to train models that can be used to detect obstacles and avoid them. We plan to explore an approach proposed by Farias et al. [15].

Custom Robotic Structure

During the informal interviews, all test users mentioned that our robot was too fragile. Based on that, we decided to design a custom made aluminium robotic structure (see Fig. 5).

Fig. 5

The current robotic chassis.

This platform was planned for overcoming more rugged terrain with an adjustable spring suspension, increased size, enhanced battery control and capacity, better energy flow control and vertical modularity. The main goal on off-the-shelf modules and modularity of this construction is to give better affordability on the platform acquisition for learning and testing ideas.

The prototype is a six wheeled vehicle that uses skid steering, and applies differential drive for motion control. Such concept is widely used in robust vehicles, such as tracked tanks and bulldozers. This method engages differently each side of the wheels and turning is done by generating differential velocity between both sides. Thus, the vehicle can perform a turn, practically in the same spot.

Some of the advantages of skid steer drive are:However, this concept also presents some drawbacks. One is that differentially driven platforms have difficulty traveling precisely on a straight line as both motors are expected to drive at exactly the same speed. Another disadvantage is that skidding or slipping increases wheel wear.

Greater traction, especially good for rough terrain.

Can be used on both tracked robots and wheeled platforms.

No steering mechanism is required.

Each side of the assistive robot can be controlled by one powerful motor.

Avoid the problems caused by castor wheels on 3 wheeled platforms.

This platform was designed to overcome some limitations of our preliminary prototype. One of the issues detected in tests with the first prototype was its low traction. To solve this problem, the current platform uses six 4.5V DC uniaxial gear motors with plastic reduction gear and soft plastic tires, mounted on a swig arm directly below the bottom of the platform.

Similarly to our first prototype, the current one also has several sensors, a night vision camera, low cost credit-card-sized computers, and microcontrollers to suit our implementation needs. The purpose of the camera is not only to transmit real-time images for control and surveillance purposes, but also to allow us to implement ideas based on computer vision techniques.

Challenges to Overcome

There are several challenges to overcome to design and implement a suitable robotic solution to enrich a smart environment. First, it is crucial to understand users’ requirements to design appropriate interaction methods for both people with limitations and their family members. Although we developed an adjustable and easy-to-use user interface [9], we plan to explore other alternatives to provide a more immersive experience. For example, we have been studying how to use EMG signals to design a more natural way to control smart devices.

Another challenge that we observed is that it is necessary to design custom platforms to assemble a robust robotic solution to help people to perform their daily tasks. It is of course more expensive to obtain a custom-made platform than to use a ready-made chassis kit, but the custom one is more flexible to suit specific needs. Fortunately, nowadays, it is possible to use 3D printers to create custom components at a reasonable cost.

As described in Xing et al. [44], it is a challenging problem to run CNN models on a low-powered board such as Raspberry Pi 3. However, these small computers have been used together with an inexpensive and low-powered USB device, the Neural Compute Stick, which allows building smart systems based on CNNs. We also plan to explore another compact single-board solution, the NVIDIA Jetson Nano, to run deep learning models. This product has been developed by NVIDIA and is an alternative to other boards such as the Raspberry Pi.

Occasionally we faced minor issues in daily operation which were mainly related to SD card corruption in the Raspberry Pi. Although it has been shown to be a viable inexpensive solution, a more stable and robust alternative can be obtained by replacing the Raspberry Pi by a mini-PC, which is still relatively inexpensive and can be currently acquired under 150 euros.

It is also a challenge to design technology to assist both people with physical limitation and those without. After all, each person has his/her own needs and desires. Some user-focused design approaches provide a way to address this issue. Another challenge arises from the fact that the proposed robot acts as an interface between inhabitants and their smart environment.

Ideas for Future Work

This section highlights ideas that we plan to pursue for future work.

Design adjustable user interfaces [9] to allow both people with limitations and their relatives to control smart devices. We are exploring alternative interaction methods, such as the ones based on EMG signals [45], to design a more suitable user interface to control the robotic solution and the environment. Textile electrodes can be attached to clothes to allow a natural way to control the smart devices. We also plan to use augmented reality technology to provide immersive experiences and enhance user engagement.

Develop a multi-functional robotic solution, based on AI techniques, to enrich environments and assist inhabitants to perform their daily activities. As aforementioned, we developed an inexpensive robotic vehicle that has been used in a real scenario. It has been improved to recognize each inhabitant, as well as his/her emotions, but subsequent work is needed to automatically detect abnormal situations and alert inhabitants, their relatives, or even health care services. We want to preserve users’ privacy and our solution has been developed to guarantee that the robot will only alert health care services under certain conditions (e.g. a predefined authorization). Proper communication is a key ingredient to build a rich intelligent environment. The various devices and machines in the environment need to communicate and interact, not only among themselves, but also with the inhabitants of the environment. Part of the interaction can be triggered by artificial intelligence itself. For example, a certain type of music could be played (or suggested) based on the emotional state of a given user.

Use our platform to develop an immersive robotic telepresence system (RTS), allowing remote users to experience the sense of being at a remote location with theirs friends, relatives, or co-workers. There is recent work in this direction. For instance, research conducted by Young et al. [47] gave interesting results and led to concepts such as shared environments and shared mutual field of view. From our point of view, however, a really immersive telepresence experience, in which the user is able to interact with the remote environment, is only possible using robotics, something which has not been done as far as we know. Small robotic solutions have been developed to support people with limitations in their own homes (e.g. telepresence robots have been used to provide health care services [11]), offering them tools to better interact and communicate with others. However, there is no real integration between these equipments and other home appliances. We note that the development of an immersive RTS will be relevant not only for people with limitations but also for the rest of the population, contributing to support human mobility and to reduce, for instance, business travel. This can be especially important in the context of pandemics such as COVID-19 where a large fraction of the population is in self-isolation at their own home, and an immersive telepresence system would be very useful and helpful. We also highlight that a facial emotion recognition system can be integrated into the RTS to help each inhabitant maintain emotional well-being. Most people with disabilities and/or older adults are socially isolated and lonely, increasing the risk of depression. An immersive telepresence system can be extremely useful for this group of people, allowing them to have a better social life and interact with others.

With the current COVID-19 pandemic, it is too risky to perform usability tests with older adults and with people with vulnerable health conditions. Techniques for usability tests have themselves to be re-invented, and it is important to think of new ways of doing them. One idea is to conduct usability tests remotely, via a telepresence system itself, to achieve social distancing and protect vulnerable people.

Summary and Conclusions

This paper described an ongoing research project which explores a broad range of technologies to design intelligent assistive living environments, with special care devoted to assist people with limitations. We developed an inexpensive robotic solution and tested it in a real life scenario. We obtained valuable data and learned about the real interaction between the proposed solution, humans and home appliances. Step by step, we are designing a multi-functional robot that will be able to communicate with the smart environment, using a low powered gateway, detect abnormal situations, and provide suitable assistance for its inhabitants.

We designed an adjustable and flexible user interface that allows both people with and without limitations to control home appliances [9] and robotic solutions. Notwithstanding, we also studied and want to explore the use of textile electrodes and EMG signals to develop an alternative and more natural way to interact with physical objects. We also plan to explore augmented reality systems to allow people to control remote devices and objects in real-time.

A new robotic platform was built using a custom made aluminium structure. This offers a more stable and robust alternative to develop and test different ideas to assist people in their daily activities. One of our main goals is to develop inexpensive systems so that they are accessible to a large fraction of the population, including those with low incomes.

We combined different models to allow the proposed solution to recognize dangerous events from images acquired by a camera. We explored the FaceNet model [37] to recognize each inhabitant and integrated it with CNN-based models to determine each inhabitant feelings and detect his/her posture. These are first steps towards the development of a robot that can assist inhabitants and enrich people’s living environment.

We believe that the way to build an immersive intelligent environment requires the development of small and relatively simple subsystems, and then integrate them as building blocks to achieve an overall complex solution. The research work is challenging, requiring knowledge of a variety of topics including human–computer interaction, computer vision, artificial intelligence, domotics, augmented communication systems, augmented reality, security and computer networks.

This research is a starting point in the development of inexpensive smart devices to enrich intelligent environments, such as smart homes and smart offices, to assist people with limitations in their daily activities. The methodology and ideas presented herein are a step in that direction. The usefulness of such living (and working) environments could not be more relevant than now, when people around the world—especially older adults and people with disabilities—are forced to be more isolated than ever due to the current COVID-19 pandemic. The following quote from Mary Pat Radabaugh comes to mind.Finally, we note that the implications of this work are likely to have an impact beyond those with limitations. As mentioned by Wobbrock et al. [43], assistive technologies are often also useful for people without limitations and in contexts that were not anticipated beforehand.

“For most people, technology makes things easier. For people with disabilities, technology makes things possible. In some cases, especially in the workplace, technology becomes the great equalizer and provides the person with a disability a level playing field on which to compete.”

— Mary Pat Radabaugh, Director of IBM National Support Center for Persons with Disabilities, 1988