1516

IEEE TRANSACTIONS ON CYBERNETICS, VOL. 43, NO. 6, DECEMBER 2013

Guest Editorial Introduction to the Special Issue on Modern Control for Computer Games Abstract—A typical gaming scenario, as developed in the past 20 years, involves a player interacting with a game using a specialized input device, such as a joystic, a mouse, a keyboard, etc. Recent technological advances and new sensors (for example, low cost commodity depth cameras) have enabled the introduction of more elaborated approaches in which the player is now able to interact with the game using his body pose, facial expressions, actions, and even his physiological signals. A new era of games has already started, employing computer vision techniques, brain–computer interfaces systems, haptic and wearable devices. The future lies in games that will be intelligent enough not only to extract the player’s commands provided by his speech and gestures but also his behavioral cues, as well as his/her emotional states, and adjust their game plot accordingly in order to ensure more realistic and satisfactory gameplay experience. This special issue on modern control for computer games discusses several interdisciplinary factors that influence a user’s input to a game, something directly linked to the gaming experience. These include, but are not limited to, the following: behavioral affective gaming, user satisfaction and perception, motion capture and scene modeling, and complete software frameworks that address several challenges risen in such scenarios.

The games industry has grown to be one of the mainstream markets in our days. At the beginning (around the 1970s) the games industry constituted a focused market, highly dependent on specialized input sensors to enable the interaction between a player and the game. Indeed, in typical games, the user had to be familiar with an input device, such as a keyboard, a mouse, a joystic, or a console in order to properly communicate with the game. Furthermore, the game had a predesigned plot that would progress along with the player’s actions in a predefined way, giving the feeling of a nonexistent, in reality, control over the game evolution. Moreover, in such a gaming scenario, several issues had to be tackled. The game had to be carefully designed and developed so as to allow real-time interaction with the player and to ensure high-quality environment so that the immersion of the player in the game environment would be as realistic as possible, and of course to employ devices that would be of affordable cost. Recent technological advances have opened new avenues toward more realistic human–machine interaction systems, thus enabling the introduction of games in which the player is not only able to control the game but also can control the gameplot and gameplay experience without using an input device, just by freely acting on his/her will. More specifically, the introduction of Microsoft Kinect has enabled the robust extraction of body poses and joint locations in real time, while Date of publication October 17, 2013; date of current version November 18, 2013. Digital Object Identifier 10.1109/TCYB.2013.2283551

at the same time being of low cost. This did not only make the gaming experience more realistic, but also opened new avenues in game design. Therefore, current games designers and developers can greatly benefit from its use to produce realistic systems. Emphasis has now been given toward affective gaming scenarios. The term affective gaming corresponds to the introduction of affect recognition in games. These approaches tried to incorporate emotion recognition in their systems to extract the emotional state of the player and to use it to control the gameplot and gameplay experience. In this special issue, we are pleased to present seven papers that are related to different aspects of modern control for computer games. All resulted from a careful review process (at least two rounds of reviews). Emotional states do not only play a crucial role in enhancing the game play but also in controlling the actual game (affective gaming). Further to this, the paper entitled, “Fusing Visual and Behavioral Cues for Modeling User Experience in Games,” investigates the creation of an accurate model of player’s experience based on a data corpus consisting of 58 subjects playing a popular game. Gameplay behavior, as well as player visual characteristics, was used to create models for engagement, frustration, and challenge. Subsequently, the derived player experience models were used to personalize the game level to desired levels of engagement, frustration, and challenge based on the assumption that game content can be mapped to player experience through the behavioral and expressivity patterns of each player. Surface motion capture (SurfCap) of a subject provides information useful for both game control and design. The paper entitled, “Animation Control of Surface Motion Capture,” introduces techniques for interactive animation control of SurfCap sequences that allow the flexibility in editing and interactive manipulation associated with existing tools for animation from skeletal motion capture (MoCap). To this end, Laplacian mesh editing is extended using a basis model learnt from SurfCap sequences to constrain the surface shape to reproduce natural deformation. Three novel approaches for animation control of SurfCap sequences are introduced and combined with high-level parametric control of SurfCap sequences in a hybrid surface and skeleton driven animation control framework to achieve natural surface deformation with an extended range of movement by exploiting existing skeletal motion capture archives. Later in this special issue, two papers that interestingly relate unobtrusive control acquired using audio and visual cues with games are presented. In more detail, the paper

c 2013 IEEE 2168-2267 

ARGYRIO et al.: GUEST EDITORIAL

entitled, “Multisignal Vision and Speech Interaction System in a Gaming Context,” introduces a novel system that integrates eye gaze estimation, head pose estimation, facial expression recognition, speech recognition, and text-to-speech components for use in real-time games. To this end, a novel head pose estimation algorithm is proposed combining scene flow with a generic head model, while a local binary patterns with three orthogonal planes (LBP-TOP) approach applied on the 2-D shape index domain is used to recognize facial expressions. The paper entitled, “Depth-Color Fusion Strategy for 3-D Scene Modeling With Kinect,” focuses on the use of Microsoft Kinect sensor and the inherent problems due to noise related to the provided depth data. The presence of noise significantly affects the game performance as in many games it considerably reduces the accuracy. In this paper, accurate depth and color models of the background elements are iteratively built, and used to detect moving objects in the scene. An adaptive jointbilateral filter that efficiently combines depth and color by analyzing an edge-uncertainty map and the detected foreground regions is designed and implemented to this end. The results verify that the proposed approach reduces the Kinect data problems such as spatial noise and temporal random fluctuations, thus better refining the objects depth boundaries. Visual and auditory cues are important facilitators of user engagement in virtual environments and video games. Not surprisingly, it has recently been suggested that auditory stimuli affects human perception of fidelity (quality). Hence, studying and understanding the exact effect of multimodal stimuli on visual fidelity perception can potentially impact the design of virtual environments and games and lead to the creation of more engaging virtual worlds and scenarios. The next paper entitled, “The Effect of Sound on Visual Fidelity Perception in Stereoscopic 3-D,” investigates the effect of auditory stimuli on human perception of visual fidelity within a stereoscopic 3-D environment. Another issue that is raised is which level of control provides the desirable result in games. This is discussed in the paper entitled, “How Much Control Is Enough?’ The authors investigate on whether the amount of control can be modulated so that a game can be fun without providing to the player perfect control. To this end, experiments are conducted with users trying to guide a hamster to the exit of a maze, with a varying level of control at each attempt. The correlation of frustration, fun, and control are thoroughly discussed. Finally, a general software framework for the development and research in the area of multimodal human–computer interaction (HCI) systems (such as games) is presented in the paper entitled, “HCI 2 Framework: A Software Framework for Multimodal Human–Computer Interaction Systems.” More precisely, the HCI 2 Framework is built upon publish/subscribe (P/S) architecture. It implements a shared-memory-based data transport protocol for message delivery and a TCP-based system management protocol. The authors also present its integrated development environment that provides a complete graphical environment to support every step in a typical MHCI system development process, including module development, debugging, packaging, and management, as well as the whole system management and testing. An example of a full system

1517

developed using the proposed HCI 2 framework, which represents a computer game based on hand-held marker(s) and low-cost camera(s), is also provided. Acknowledgment The Guest Editors would like to thank all the authors who have contributed to this special issue and all the reviewers who have provided valuable comments. They would like to thank the IEEE Transactions on Cybernetics (formerly IEEE Transactions on Systems, Man, and Cybernetics: Part B) for supporting this special issue, with special thanks to the Editor-in-Chief Prof. E. Santos for his support. They would also like to thank the Editorial Assistants L. Cullen and T. Martin, as well as the Journal Coordinator M. Rafferty, for their professional support with the editorial matters during the preparation of this special issue. Special Tribute to Prof. Maria Petrou During the preparation of this special issue, one of the Guest Editors, Prof. M. Petrou, Imperial College London, London, U.K., and the Director of the Information Technologies Institute, Centre for Research and Technology Hellas, Hellas, Greece, passed away. The Guest Editors all have been deeply saddened by the sudden loss of a great scholar, a colleague, and a friend. All the Editors are very much honored to have served with her as Guest Editors on this special issue. VASILEIOS ARGYRIOU, Member, IEEE, Guest Editor Department of Computing, Information Systems and Mathematics Kingston University, Kingston upon Thames Surrey KT1 2EE, U.K. (e-mail: [email protected]) IRENE KOTSIA, Guest Editor School of Science and Technology Middlesex University, The Burroughs London, NW4 4BT, U.K. (e-mail: [email protected]) STEFANOS ZAFEIRIOU, Guest Editor Department of Computing Imperial College London, London SW7 2AZ, U.K. (e-mail: [email protected]) MARIA PETROU, Senior Member, IEEE, Guest Editor Department of Electrical and Electronic Engineering Imperial College London, London, U.K. Informatics and Telematics Institute Centre of Research and Technology - Hellas, Thessaloniki 57001, Greece (e-mail: [email protected])

1518

IEEE TRANSACTIONS ON CYBERNETICS, VOL. 43, NO. 6, DECEMBER 2013

Vasileios Argyriou (M’10) received the B.Sc. degree in computer science from Aristotle University of Thessaloniki, Thessaloniki, Greece, in 2001, and the M.Sc. and Ph.D. degrees from the University of Surrey, Surrey, U.K., in 2003 and 2006, respectively, both in electrical engineering. From 2001 to 2002, he held a research position with the AIIA Laboratory, Aristotle University of Thessaloniki, involved in image and video watermarking. From 2002 and 2006, he participated in many European projects for archive file restoration (PrestoSpace) and subpixel motion estimation collaborating with Snell and Wilcox. He joined the Communications and Signal Processing Department, Imperial College, London, U.K., in 2007, where he was a Research Fellow involved in 3-D image reconstruction from photometric stereo. He is currently a Senior Lecturer with the Department of Computing, Information Systems and Mathematics, Kingston University, London, U.K, where he is engaged in action recognition and AI for computer games. Dr. Argyriou is a member of the IET.

Irene Kotsia received the B.Sc. and Ph.D. degrees from the Department of Informatics, Aristotle University of Thessaloniki, Thessaloniki, Greece, in 2002 and 2008, respectively. From 2008 to 2009, she was a Research Associate and Teaching Assistant with the Department of Informatics, Aristotle University of Thessaloniki. From 2009 to 2011, she was a Research Associate with the Department of Electronic Engineering and Computer Science, Queen Mary University of London, London, U.K., and since 2012, she has been a Research Associate with the Department of Computing, Imperial College, London, U.K., and a Lecturer in creative technology and digital creativity with the Department of Computing Science, Middlesex University, London, U.K. She has co-authored over 35 journal and conference publications.

Stefanos Zafeiriou received the B.Sc and Ph.D degrees from Department of Informatics, Aristotle University of Thessaloniki, in 2003 and 2007, respectively. He is currently a Lecturer (equivalent to Assistant Professor) in pattern recognition/statistical machine learning for computer vision with the Department of Computing, Imperial College, London, U.K. He has co-authored over 31 journal papers mainly on novel statistical machine learning methodologies applied to computer vision problems such as 2-D/3-D face and facial expression recognition, deformable object tracking, human behavior analysis, etc., published in the most prestigious journals in his field of research (such as the IEEE Transactions on Pattern Analysis and Machine Intelligence, International Journal of Computer Vision, the IEEE Transactions on Image Processing, the IEEE Transactions on Neural Networks and Learning Systems, the IEEE Transactions on Visualization and Computer Graphics, the IEEE Transactions on Information Forensics and Security, etc.). He has participated in more than ten European Union, British, and Greek research projects. His students are frequent recipients of very prestigious and highly competitive fellowships such as Google Fellowship, Intel Fellowship, and Qualcomm Fellowship. He has over 900 citations to his work. Dr. Zafeiriou has been awarded one of the prestigious Junior Research Fellowships from Imperial College London in 2011 to start his own independent research group. He currently serves as an Associate Editor in the IEEE Transactions on Cybernetics and Image and Vision Computing. He has been a Guest Editor in more than four special issues and co-organized more than five workshops and special sessions in top venues, such as CVPR/FG/ICCV/ECCV.

Maria Petrou (A’90–M’91–SM’05) received the degree in physics from Aristotle University of Thessaloniki, Thessaloniki, Greece, the degree in applied mathematics from University of Cambridge, Cambridge, U.K., the Ph.D. degree from the Institute of Astronomy, Cambridge, U.K., and the D.Sc. degree from Cambridge, in 2009. She was the Chair of Signal Processing with the Department of Electrical and Electronic Engineering, Imperial College London, London, U.K., and was the Director of the Informatics and Telematics Institute, Centre of Research and Technology, Hellas, Greece. She has published over 350 scientific papers on astronomy, remote sensing, computer vision, machine learning, color analysis, industrial inspection, medical signal, and image processing. She has co-authored two books, Image Processing: The Fundamentals (Wiley, first edition 1999 and second edition 2010) and Image Processing: Dealing With Texture (Wiley, 2006). She has also co-edited the book Next Generation Artificial Vision Systems: Reverse Engineering the Human Visual System. She has supervised to successful completion of 43 Ph.D. theses. Dr. Petrou was a fellow of the Royal Academy of Engineering, the City and Guilds Institute, the Institution of Engineering and Technology, the International Association for Pattern Recognition, the Institute of Physics, and a Distinguished Fellow of the British Machine Vision Association. She served as a Trustee of the IET from 2006 to 2009, the IAPR Newsletter Editor from 1994 to 1998, and the IAPR Treasurer from 2002 to 2006.

Guest editorial: Introduction to the special issue on modern control for computer games.

A typical gaming scenario, as developed in the past 20 years, involves a player interacting with a game using a specialized input device, such as a jo...
878KB Sizes 0 Downloads 0 Views