IJB-3193; No. of Pages 9
ARTICLE IN PRESS i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 5 ) xxx–xxx
journal homepage: www.ijmijournal.com
What device should be used for telementoring? Randomized controlled trial Andrius Budrionis a,∗ , Gunnar Hartvigsen a , Rolv-Ole Lindsetmo b , Johan Gustav Bellika a a b
Norwegian Centre for Integrated Care and Telemedicine University Hospital of North Norway, Tromsø, Norway Department of Gastroenterological Surgery University Hospital of North Norway, Tromsø, Norway
a r t i c l e
i n f o
a b s t r a c t
Article history:
Background and objective: The paper analyzes behavioral patterns of mentors while using dif-
Received 16 June 2014
ferent mentoring devices to demonstrate the feasibility of multi-platform mentoring. The
Received in revised form
fundamental differences of devices supporting telementoring create threats for the per-
13 May 2015
ception and interpretation of the transmitted video, highlighting the necessity of exploring
Accepted 15 May 2015
hardware usability aspects in a safety critical surgical mentoring scenario.
Keywords:
the randomized controlled trial. Streaming video recordings of a laparoscopic procedure to
Materials and methods: Three types of devices, based on the screen size, formed the arms for Telementoring
the mentors imitated the mentoring scenario. User preferences and response times were
Usability
recorded while participating in a session performed on all devices.
Touchscreen
Results: Median response to a mentoring request times were similar for mobile platforms;
Telestration
expected durations were considerably longer for stationary computer. Ability to perceive and
Regulations
identify anatomical structures was insignificantly lower on small sized devices. Stationary and tablet platforms were nearly equally preferred by the most of participants as default telementoring hardware. Discussion: As a side effect, incompatibility of daily duties of the surgeons in the hospital and telementoring responsibilities while implementing systems locally was identified. Scaling up the use of the service in combination with the organizational changes of clinical staff looks like a promising solution. Conclusion: The trial demonstrated the feasibility of using all three types of devices for the purpose of mentoring, allowing users to choose the preferred platform. The paper provided initial results on the quality assurance of telementoring systems imposed by the regulatory documents. © 2015 Elsevier Ireland Ltd. All rights reserved.
1.
Introduction
Telementoring systems used in surgery have been discussed extensively in research publications [1]. Nobody denies the
∗
Corresponding author. Tel.: +47 77 75 40 30. E-mail address: [email protected] (A. Budrionis).
benefits of successfully applying the approach in the Operating Room (OR). Numerous research groups report their experiences in developing such systems and remotely mentoring surgeons performing operations. Due to the global trend of positive results being published easier, negative outcomes (especially related to system failures, wrong decisions caused by immature implementations, etc.) are rare to be found. However, if we take a look at the case of mentoring a surgeon, the
http://dx.doi.org/10.1016/j.ijmedinf.2015.05.004 1386-5056/© 2015 Elsevier Ireland Ltd. All rights reserved.
Please cite this article in press as: A. Budrionis, et al., What device should be used for telementoring? Randomized controlled trial, Int. J. Med. Inform. (2015), http://dx.doi.org/10.1016/j.ijmedinf.2015.05.004
IJB-3193; No. of Pages 9
2
ARTICLE IN PRESS i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 5 ) xxx–xxx
chances of a regular scenario turning into critical are highly plausible. The purpose of using remote supervision on a complicated case is to enable the influence on the procedure by the remote expertise. If the remote expert has no influence on the progress in the OR, the approach, basically, looses its point: why would one need an expert following his steps but not being able to affect them? The described scenario has a literal match in the “Guidelines on the Qualification and Classification of Stand Alone Software Used in Healthcare Within the Regulatory Framework of Medical Devices” issued by the European Commission [2]. Systems “intended to influence the surgery procedure are qualified as medical devices”, meaning that the risk they introduce is too high to test prototypical implementations in the OR environment. However, numerous publications ignore the guidelines, putting the patients at unnecessary risk. In addition to the European regulations, American Food and Drug Administration (FDA) proposes a risk-based assessment of telemedical systems. Equipment, intended to be used for active patient monitoring, transmitting/receiving and storing of real-time video and audio as well as patient data is regarded as Class II (moderate risk) medical device. Communication modules fall within the same category, contrasting a more liberal European approach [3,4]. This paper advocates extensive testing of such safety critical systems and presents the initial evaluation results of the novel approach to the design of telementoring systems. Providing telementoring as a service in the health care network has changed the overall use of the technique. Instead of planning, preparing hardware and software for the oncoming supervision, participants are able to use the mentoring service the same way they use their cellphones: call for help, whenever it is needed, using the device they have on hand at a given moment [5]. However, the main difference from a regular call is the ability to use live video and telestration, features coming together with the mentoring service [5–7]. While video conferencing based system could qualify as a communication module used within surgery, ability to annotate video content pushes it over the threshold to become a medical device [2]. The presented enhancement to the emergency phone calls (common telementoring approach) has brought numerous advantages [5]. This paper does not address the benefits of employing telementoring in clinical workflows; we emphasize the experiences with the novel technology and the results from the comparative study, employing 3 types of devices for the purpose of mentoring. Usability properties of stationary computer, tablet and smartphone were studied in the trial. The purpose of the trial is to demonstrate the feasibility of performing telementoring on relatively small touch screen devices with respect to the established platform—the stationary computer. The study aims to clarify what influence the different properties of the used devices have on the mentoring process. Screen sizes, touchscreen/mouse input for annotating live video, portability are the properties, possibly having implications for the safety critical mentoring systems [2]. Moreover, end user experiences are also of high value ensuring that different devices maintain the same qualities of remote supervision [8]. The paper is structured as follows: introduction gives a brief understanding of the existing regulatory frameworks
with regards to the use of common hardware for surgical telementoring. Search for relevant research identifies the gap in literature, the study aims to cover. Method section elaborates on methodology to question the use of hardware in surgical telementoring scenario. Results present the findings from the study, followed by the discussion and limitations.
1.1.
Related work
The main focus of the study is usability of the devices supporting telementoring. While performing the search for relevant literature, it was expanded to cover the published research on comparison of touchscreen versus mouse inputs and studies on control of moving images on relatively small devices. Regardless the importance of the hypotheses of different inputs resulting in non-equivalent outcome while completing the same tasks, a surprisingly low interest in the topic was identified. None of the authors of telementoring related publications questioned the properties and user preferences on the employed devices, while considerably lower accuracy of task completion using touch screen instead of mouse was reported. Besides the higher error rate, touch input was considerably faster and was regarded as a more natural way of drawing [9,10]. The findings indicate the need for extra attention when developing precision critical touch interaction systems. Impact of age on the speed of manipulating different inputs was studied by Findlater et al. [11] Findings state generally longer task completion duration for older adults, however, the performance gap between older and younger participants was reduced using touch input [11]. It advocates touchscreen being a more natural input for drawing actions, ensuring a smaller performance difference between the age groups. When it comes to user preferences, touch interfaces were preferred to traditional mouse input for interactive tasks (drawing, gaming) [12], however, the application domain is of high importance in this case. A gap in research was identified in understanding user preferences for input in safety critical tasks, combining the need for extra accuracy and speed.
2.
Materials and methods
2.1.
Trial design
A crossover RCT was designed to investigate the impact of using 3 types of devices (stationary computer, tablet and smartphone) and record user experiences during an imitated telementoring session performed at the University Hospital of North Norway (UNN). Randomization voided the bias introduced by the order of the devices in the trial and possible learning effects. Every participant performed the same task on all three devices; measures taken while mentoring on stationary platform served as controls for the interventions (smartphone and tablet). A video recording, captured during a laparoscopic surgery, was broadcasted to the mentoring hardware, simulating a request from the OR. Telementoring task was broken down into two simultaneous parts: ability to perceive
Please cite this article in press as: A. Budrionis, et al., What device should be used for telementoring? Randomized controlled trial, Int. J. Med. Inform. (2015), http://dx.doi.org/10.1016/j.ijmedinf.2015.05.004
IJB-3193; No. of Pages 9
ARTICLE IN PRESS i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 5 ) xxx–xxx
3
Fig. 1 – Schematic Trial Design [13]—The selected crossover RCT design provided more consistent results including fewer participants. Moreover, it allowed minimizing the bias introduced by the human factor by discovering trends while the participants were using different devices (for instance, previous experience)
the information in the video, identify important internal structures and ability to produce annotations. Before starting the trial, participants attended a presentation, highlighting their role in the study. Personal instructions were given to everybody before starting with the first device. Pause/resume functionality for producing accurate annotations was emphasized. The study was carried out during regular workdays at UNN to imitate real-life use of the service [13]. After being instructed, participants were assigned a mentoring device (RTC arm) in a random order (Fig. 1). They were contacted by text messaging (sms) and email to join the imitated mentoring session during a regular workday. Due to the busy schedule of the surgeons and frequent unplanned changes in it, requesting mentoring at absolutely random time was often impossible. An approximate time for sending the mentoring request (within 2 h) was discussed with the participants before starting. Randomness in the predefined timeframe was maintained. The progress of the session was recorded together with the responses to the questionnaire, which was filled in by all users after each device. All questions were rated in a scale of 1 to 5, 1 representing no difficulties encountered, while 5—not being able to complete the task. Values between 1 and 2 were considered acceptable. Minimum washout period between devices was set to 3 days.
2.2.
Infrastructure
A web-based telementoring system, developed at Norwegian Centre for Integrated Care and Telemedicine (NST) provided the infrastructure for the experiment. Due to the technological properties of the selected implementation, telementoring was moved to web browser, voiding the need for dedicated client side mentoring hardware [7]. The following hardware was used in the study by the mentors:
(1) Screen size 15.4 laptop computer (Lenovo T61p, Core2 duo, 4GB RAM, Windows 7, screen resolution 1920 × 1200 pixel) located in the office of the surgeon,
representing stationary platform. Mouse was used for annotating. (2) Touchscreen size 10 tablet computer (Asus MeMO Pad, Full HD, Android 4.2, screen resolution 1280 × 800 pixel), representing middle-sized mobile devices. (3) Touchscreen size 5 smartphone (Samsung Galaxy S4, Android 4.3, screen resolution 1920 × 1080 pixel), representing small-sized mobile devices [13].
2.3.
Participants
Twelve surgeons (age 30–62) at the department of Gastroenterological Surgery were recruited to participate in the trial. All participants were experienced in using at least one type of the device on daily basis. Due to the goal of the trial to demonstrate the feasibility of multi-platform mentoring, non-inferior outcomes between the RCT arms were expected instead of superior [13].
2.4.
Measures, outcomes and data analysis
The following measurements were collected in the trial:
(1) Mentor response time (duration between the initiation of mentoring session and mentor being present online). Decrease in response time was expected while mentoring on mobile devices. (2) Mentor’s interaction with the device (coordinates of annotations, use of pause, resume and zoom functions). No significant changes in interaction patterns were expected while changing mentoring hardware. (3) Final outcome of mentoring (video and overlaid annotations [13]). No significant changes were expected.
Microsoft Excel:mac 2011 was used for data collection. Singificance of the results was calculated using Wilcoxon test in R v.3.1.1. Network related problems were registered, but not emphasized. Environment variables (lighting, level of noise, number of people in the same room while performing the task) were not recorded.
Please cite this article in press as: A. Budrionis, et al., What device should be used for telementoring? Randomized controlled trial, Int. J. Med. Inform. (2015), http://dx.doi.org/10.1016/j.ijmedinf.2015.05.004
ARTICLE IN PRESS
IJB-3193; No. of Pages 9
4
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 5 ) xxx–xxx
Fig. 2 – (A) Response times; (B) interactions between mentor and mentee devices per session.
3.
Results
Results from the trial were divided into quantitative (originating from automatic data logging) and qualitative (coming from the user questionnaires).
3.1.
Quantitative
Results revealed a trend of stationary mentoring post requiring more time to respond to a mentoring request. The measures ranged from 21 second to more that 3 h; distribution of response times for every platform is represented in Fig. 2A. If we look into the mean response values, differences between the mobile platforms were minor (p-value = 0.6499, Table 1, row 1); stationary platform required more time to respond to mentoring request, the difference between control and intervention was significant for tablet (p-value = 0.01546, Table 1, row 1) and slightly above the significance threshold for smartphone (p-value = 0.05078, Table 1, row 1). The expected (inside the box—between the 1st and 3rd quartiles) and maximal values were considerably longer in stationary case.
A represents mentor response times to a mentoring request on selected devices. Response time is defined as duration between the initiation of mentoring session and mentor being present online. B plots total information exchange massages travelling between the devices while performing the task. Messages are the representation participant–device interaction. Besides from the response time, we analyzed the use of functionality of the prototype in order to compare if differences of the devices call for behavioral changes in interaction for completing exactly the same task. Use of pause/resume and zoom functions were recorded and plotted for each device in Fig. 3. Pause/resume functionality was used in a similar manner on different platforms (high p-values, Table 1, row 3); use of zoom feature was more dependent on the size of the device, however differences were not statistically significant (Table 1, row 4). In addition to the use of certain functions, we looked into total number of interactions (“send/receive”) messages between mentor and mentee devices. Every move between mouse click and release (touch begin and touch end, respectively) produces a message. A number of interchanged messages is a representative measure of user-device interactions. The results revealed stationary platform requiring
Table 1 – p-Values and confidence intervals for the comparisons in the trial. Statistically significant values are highlighted. No 1 2 3 4 5 6 7
Measure Response time (Fig. 2A) Interactions (Fig. 2B) Pause/Resume (Fig. 3A) Zoom (Fig. 3B) Ability to identify all anatomical structures (Fig. 5A) Ability to annotate (Fig. 5B) Camera moved too much for producing accurate annotations (Fig. 6A)
Tablet vs PC
Smartphone vs PC
0.01546, 95% 0.001953, 95% 0.8652, 90% 0.9666, 95% 0.9623, 95%
0.05078, 95% 0.01367, 95% 0.8018, 95% 0.9609, 90% 0.9938, 95%
Tablet vs smartphone 0.6499, 95% 0.4609, 95% 0.785, 95% 0.3996, 95% 0.1325, 95%
0.6251, 95% 0.8016, 95%
0.8559, 95% 0.2931, 80%
0.1201, 95% 0.8977, 95%
Please cite this article in press as: A. Budrionis, et al., What device should be used for telementoring? Randomized controlled trial, Int. J. Med. Inform. (2015), http://dx.doi.org/10.1016/j.ijmedinf.2015.05.004
IJB-3193; No. of Pages 9
ARTICLE IN PRESS i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 5 ) xxx–xxx
5
Fig. 3 – Use of pause/resume and zoom functions—the number of pause/resume function calls was registered for every participant during the trial. The higher numbers for the devices equipped with smaller screens represent problems in perceiving video content and annotating it.
nearly twice as many interactions as mobiles to complete the same task. In other words, the number of moves the mentor performed using mouse input was considerably higher than touchscreen (p-values
ARTICLE IN PRESS i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 5 ) xxx–xxx
journal homepage: www.ijmijournal.com
What device should be used for telementoring? Randomized controlled trial Andrius Budrionis a,∗ , Gunnar Hartvigsen a , Rolv-Ole Lindsetmo b , Johan Gustav Bellika a a b
Norwegian Centre for Integrated Care and Telemedicine University Hospital of North Norway, Tromsø, Norway Department of Gastroenterological Surgery University Hospital of North Norway, Tromsø, Norway
a r t i c l e
i n f o
a b s t r a c t
Article history:
Background and objective: The paper analyzes behavioral patterns of mentors while using dif-
Received 16 June 2014
ferent mentoring devices to demonstrate the feasibility of multi-platform mentoring. The
Received in revised form
fundamental differences of devices supporting telementoring create threats for the per-
13 May 2015
ception and interpretation of the transmitted video, highlighting the necessity of exploring
Accepted 15 May 2015
hardware usability aspects in a safety critical surgical mentoring scenario.
Keywords:
the randomized controlled trial. Streaming video recordings of a laparoscopic procedure to
Materials and methods: Three types of devices, based on the screen size, formed the arms for Telementoring
the mentors imitated the mentoring scenario. User preferences and response times were
Usability
recorded while participating in a session performed on all devices.
Touchscreen
Results: Median response to a mentoring request times were similar for mobile platforms;
Telestration
expected durations were considerably longer for stationary computer. Ability to perceive and
Regulations
identify anatomical structures was insignificantly lower on small sized devices. Stationary and tablet platforms were nearly equally preferred by the most of participants as default telementoring hardware. Discussion: As a side effect, incompatibility of daily duties of the surgeons in the hospital and telementoring responsibilities while implementing systems locally was identified. Scaling up the use of the service in combination with the organizational changes of clinical staff looks like a promising solution. Conclusion: The trial demonstrated the feasibility of using all three types of devices for the purpose of mentoring, allowing users to choose the preferred platform. The paper provided initial results on the quality assurance of telementoring systems imposed by the regulatory documents. © 2015 Elsevier Ireland Ltd. All rights reserved.
1.
Introduction
Telementoring systems used in surgery have been discussed extensively in research publications [1]. Nobody denies the
∗
Corresponding author. Tel.: +47 77 75 40 30. E-mail address: [email protected] (A. Budrionis).
benefits of successfully applying the approach in the Operating Room (OR). Numerous research groups report their experiences in developing such systems and remotely mentoring surgeons performing operations. Due to the global trend of positive results being published easier, negative outcomes (especially related to system failures, wrong decisions caused by immature implementations, etc.) are rare to be found. However, if we take a look at the case of mentoring a surgeon, the
http://dx.doi.org/10.1016/j.ijmedinf.2015.05.004 1386-5056/© 2015 Elsevier Ireland Ltd. All rights reserved.
Please cite this article in press as: A. Budrionis, et al., What device should be used for telementoring? Randomized controlled trial, Int. J. Med. Inform. (2015), http://dx.doi.org/10.1016/j.ijmedinf.2015.05.004
IJB-3193; No. of Pages 9
2
ARTICLE IN PRESS i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 5 ) xxx–xxx
chances of a regular scenario turning into critical are highly plausible. The purpose of using remote supervision on a complicated case is to enable the influence on the procedure by the remote expertise. If the remote expert has no influence on the progress in the OR, the approach, basically, looses its point: why would one need an expert following his steps but not being able to affect them? The described scenario has a literal match in the “Guidelines on the Qualification and Classification of Stand Alone Software Used in Healthcare Within the Regulatory Framework of Medical Devices” issued by the European Commission [2]. Systems “intended to influence the surgery procedure are qualified as medical devices”, meaning that the risk they introduce is too high to test prototypical implementations in the OR environment. However, numerous publications ignore the guidelines, putting the patients at unnecessary risk. In addition to the European regulations, American Food and Drug Administration (FDA) proposes a risk-based assessment of telemedical systems. Equipment, intended to be used for active patient monitoring, transmitting/receiving and storing of real-time video and audio as well as patient data is regarded as Class II (moderate risk) medical device. Communication modules fall within the same category, contrasting a more liberal European approach [3,4]. This paper advocates extensive testing of such safety critical systems and presents the initial evaluation results of the novel approach to the design of telementoring systems. Providing telementoring as a service in the health care network has changed the overall use of the technique. Instead of planning, preparing hardware and software for the oncoming supervision, participants are able to use the mentoring service the same way they use their cellphones: call for help, whenever it is needed, using the device they have on hand at a given moment [5]. However, the main difference from a regular call is the ability to use live video and telestration, features coming together with the mentoring service [5–7]. While video conferencing based system could qualify as a communication module used within surgery, ability to annotate video content pushes it over the threshold to become a medical device [2]. The presented enhancement to the emergency phone calls (common telementoring approach) has brought numerous advantages [5]. This paper does not address the benefits of employing telementoring in clinical workflows; we emphasize the experiences with the novel technology and the results from the comparative study, employing 3 types of devices for the purpose of mentoring. Usability properties of stationary computer, tablet and smartphone were studied in the trial. The purpose of the trial is to demonstrate the feasibility of performing telementoring on relatively small touch screen devices with respect to the established platform—the stationary computer. The study aims to clarify what influence the different properties of the used devices have on the mentoring process. Screen sizes, touchscreen/mouse input for annotating live video, portability are the properties, possibly having implications for the safety critical mentoring systems [2]. Moreover, end user experiences are also of high value ensuring that different devices maintain the same qualities of remote supervision [8]. The paper is structured as follows: introduction gives a brief understanding of the existing regulatory frameworks
with regards to the use of common hardware for surgical telementoring. Search for relevant research identifies the gap in literature, the study aims to cover. Method section elaborates on methodology to question the use of hardware in surgical telementoring scenario. Results present the findings from the study, followed by the discussion and limitations.
1.1.
Related work
The main focus of the study is usability of the devices supporting telementoring. While performing the search for relevant literature, it was expanded to cover the published research on comparison of touchscreen versus mouse inputs and studies on control of moving images on relatively small devices. Regardless the importance of the hypotheses of different inputs resulting in non-equivalent outcome while completing the same tasks, a surprisingly low interest in the topic was identified. None of the authors of telementoring related publications questioned the properties and user preferences on the employed devices, while considerably lower accuracy of task completion using touch screen instead of mouse was reported. Besides the higher error rate, touch input was considerably faster and was regarded as a more natural way of drawing [9,10]. The findings indicate the need for extra attention when developing precision critical touch interaction systems. Impact of age on the speed of manipulating different inputs was studied by Findlater et al. [11] Findings state generally longer task completion duration for older adults, however, the performance gap between older and younger participants was reduced using touch input [11]. It advocates touchscreen being a more natural input for drawing actions, ensuring a smaller performance difference between the age groups. When it comes to user preferences, touch interfaces were preferred to traditional mouse input for interactive tasks (drawing, gaming) [12], however, the application domain is of high importance in this case. A gap in research was identified in understanding user preferences for input in safety critical tasks, combining the need for extra accuracy and speed.
2.
Materials and methods
2.1.
Trial design
A crossover RCT was designed to investigate the impact of using 3 types of devices (stationary computer, tablet and smartphone) and record user experiences during an imitated telementoring session performed at the University Hospital of North Norway (UNN). Randomization voided the bias introduced by the order of the devices in the trial and possible learning effects. Every participant performed the same task on all three devices; measures taken while mentoring on stationary platform served as controls for the interventions (smartphone and tablet). A video recording, captured during a laparoscopic surgery, was broadcasted to the mentoring hardware, simulating a request from the OR. Telementoring task was broken down into two simultaneous parts: ability to perceive
Please cite this article in press as: A. Budrionis, et al., What device should be used for telementoring? Randomized controlled trial, Int. J. Med. Inform. (2015), http://dx.doi.org/10.1016/j.ijmedinf.2015.05.004
IJB-3193; No. of Pages 9
ARTICLE IN PRESS i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 5 ) xxx–xxx
3
Fig. 1 – Schematic Trial Design [13]—The selected crossover RCT design provided more consistent results including fewer participants. Moreover, it allowed minimizing the bias introduced by the human factor by discovering trends while the participants were using different devices (for instance, previous experience)
the information in the video, identify important internal structures and ability to produce annotations. Before starting the trial, participants attended a presentation, highlighting their role in the study. Personal instructions were given to everybody before starting with the first device. Pause/resume functionality for producing accurate annotations was emphasized. The study was carried out during regular workdays at UNN to imitate real-life use of the service [13]. After being instructed, participants were assigned a mentoring device (RTC arm) in a random order (Fig. 1). They were contacted by text messaging (sms) and email to join the imitated mentoring session during a regular workday. Due to the busy schedule of the surgeons and frequent unplanned changes in it, requesting mentoring at absolutely random time was often impossible. An approximate time for sending the mentoring request (within 2 h) was discussed with the participants before starting. Randomness in the predefined timeframe was maintained. The progress of the session was recorded together with the responses to the questionnaire, which was filled in by all users after each device. All questions were rated in a scale of 1 to 5, 1 representing no difficulties encountered, while 5—not being able to complete the task. Values between 1 and 2 were considered acceptable. Minimum washout period between devices was set to 3 days.
2.2.
Infrastructure
A web-based telementoring system, developed at Norwegian Centre for Integrated Care and Telemedicine (NST) provided the infrastructure for the experiment. Due to the technological properties of the selected implementation, telementoring was moved to web browser, voiding the need for dedicated client side mentoring hardware [7]. The following hardware was used in the study by the mentors:
(1) Screen size 15.4 laptop computer (Lenovo T61p, Core2 duo, 4GB RAM, Windows 7, screen resolution 1920 × 1200 pixel) located in the office of the surgeon,
representing stationary platform. Mouse was used for annotating. (2) Touchscreen size 10 tablet computer (Asus MeMO Pad, Full HD, Android 4.2, screen resolution 1280 × 800 pixel), representing middle-sized mobile devices. (3) Touchscreen size 5 smartphone (Samsung Galaxy S4, Android 4.3, screen resolution 1920 × 1080 pixel), representing small-sized mobile devices [13].
2.3.
Participants
Twelve surgeons (age 30–62) at the department of Gastroenterological Surgery were recruited to participate in the trial. All participants were experienced in using at least one type of the device on daily basis. Due to the goal of the trial to demonstrate the feasibility of multi-platform mentoring, non-inferior outcomes between the RCT arms were expected instead of superior [13].
2.4.
Measures, outcomes and data analysis
The following measurements were collected in the trial:
(1) Mentor response time (duration between the initiation of mentoring session and mentor being present online). Decrease in response time was expected while mentoring on mobile devices. (2) Mentor’s interaction with the device (coordinates of annotations, use of pause, resume and zoom functions). No significant changes in interaction patterns were expected while changing mentoring hardware. (3) Final outcome of mentoring (video and overlaid annotations [13]). No significant changes were expected.
Microsoft Excel:mac 2011 was used for data collection. Singificance of the results was calculated using Wilcoxon test in R v.3.1.1. Network related problems were registered, but not emphasized. Environment variables (lighting, level of noise, number of people in the same room while performing the task) were not recorded.
Please cite this article in press as: A. Budrionis, et al., What device should be used for telementoring? Randomized controlled trial, Int. J. Med. Inform. (2015), http://dx.doi.org/10.1016/j.ijmedinf.2015.05.004
ARTICLE IN PRESS
IJB-3193; No. of Pages 9
4
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 5 ) xxx–xxx
Fig. 2 – (A) Response times; (B) interactions between mentor and mentee devices per session.
3.
Results
Results from the trial were divided into quantitative (originating from automatic data logging) and qualitative (coming from the user questionnaires).
3.1.
Quantitative
Results revealed a trend of stationary mentoring post requiring more time to respond to a mentoring request. The measures ranged from 21 second to more that 3 h; distribution of response times for every platform is represented in Fig. 2A. If we look into the mean response values, differences between the mobile platforms were minor (p-value = 0.6499, Table 1, row 1); stationary platform required more time to respond to mentoring request, the difference between control and intervention was significant for tablet (p-value = 0.01546, Table 1, row 1) and slightly above the significance threshold for smartphone (p-value = 0.05078, Table 1, row 1). The expected (inside the box—between the 1st and 3rd quartiles) and maximal values were considerably longer in stationary case.
A represents mentor response times to a mentoring request on selected devices. Response time is defined as duration between the initiation of mentoring session and mentor being present online. B plots total information exchange massages travelling between the devices while performing the task. Messages are the representation participant–device interaction. Besides from the response time, we analyzed the use of functionality of the prototype in order to compare if differences of the devices call for behavioral changes in interaction for completing exactly the same task. Use of pause/resume and zoom functions were recorded and plotted for each device in Fig. 3. Pause/resume functionality was used in a similar manner on different platforms (high p-values, Table 1, row 3); use of zoom feature was more dependent on the size of the device, however differences were not statistically significant (Table 1, row 4). In addition to the use of certain functions, we looked into total number of interactions (“send/receive”) messages between mentor and mentee devices. Every move between mouse click and release (touch begin and touch end, respectively) produces a message. A number of interchanged messages is a representative measure of user-device interactions. The results revealed stationary platform requiring
Table 1 – p-Values and confidence intervals for the comparisons in the trial. Statistically significant values are highlighted. No 1 2 3 4 5 6 7
Measure Response time (Fig. 2A) Interactions (Fig. 2B) Pause/Resume (Fig. 3A) Zoom (Fig. 3B) Ability to identify all anatomical structures (Fig. 5A) Ability to annotate (Fig. 5B) Camera moved too much for producing accurate annotations (Fig. 6A)
Tablet vs PC
Smartphone vs PC
0.01546, 95% 0.001953, 95% 0.8652, 90% 0.9666, 95% 0.9623, 95%
0.05078, 95% 0.01367, 95% 0.8018, 95% 0.9609, 90% 0.9938, 95%
Tablet vs smartphone 0.6499, 95% 0.4609, 95% 0.785, 95% 0.3996, 95% 0.1325, 95%
0.6251, 95% 0.8016, 95%
0.8559, 95% 0.2931, 80%
0.1201, 95% 0.8977, 95%
Please cite this article in press as: A. Budrionis, et al., What device should be used for telementoring? Randomized controlled trial, Int. J. Med. Inform. (2015), http://dx.doi.org/10.1016/j.ijmedinf.2015.05.004
IJB-3193; No. of Pages 9
ARTICLE IN PRESS i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 5 ) xxx–xxx
5
Fig. 3 – Use of pause/resume and zoom functions—the number of pause/resume function calls was registered for every participant during the trial. The higher numbers for the devices equipped with smaller screens represent problems in perceiving video content and annotating it.
nearly twice as many interactions as mobiles to complete the same task. In other words, the number of moves the mentor performed using mouse input was considerably higher than touchscreen (p-values
