Real-time visual sensing system achieving high-speed 3D particle tracking with nanometer resolution Peng Cheng,1 Sissy M. Jhiang,2 and Chia-Hsiang Menq1,* 1
Department of Mechanical and Aerospace Engineering, The Ohio State University, Columbus, Ohio 43210, USA 2
Department of Physiology and Cell Biology, The Ohio State University, Columbus, Ohio 43210, USA *Corresponding author: [email protected] Received 2 August 2013; revised 4 October 2013; accepted 4 October 2013; posted 7 October 2013 (Doc. ID 195116); published 28 October 2013
This paper presents a real-time visual sensing system, which is created to achieve high-speed three-dimensional (3D) motion tracking of microscopic spherical particles in aqueous solutions with nanometer resolution. The system comprises a complementary metal–oxide-semiconductor (CMOS) camera, a field programmable gate array (FPGA), and real-time image processing programs. The CMOS camera has high photosensitivity and superior SNR. It acquires images of 128 × 120 pixels at a frame rate of up to 10,000 frames per second (fps) under the white light illumination from a standard 100 W halogen lamp. The real-time image stream is downloaded from the camera directly to the FPGA, wherein a 3D particle-tracking algorithm is implemented to calculate the 3D positions of the target particle in real time. Two important objectives, i.e., real-time estimation of the 3D position matches the maximum frame rate of the camera and the timing of the output data stream of the system is precisely controlled, are achieved. Two sets of experiments were conducted to demonstrate the performance of the system. First, the visual sensing system was used to track the motion of a 2 μm polystyrene bead, whose motion was controlled by a three-axis piezo motion stage. The ability to track long-range motion with nanometer resolution in all three axes is demonstrated. Second, it was used to measure the Brownian motion of the 2 μm polystyrene bead, which was stabilized in aqueous solution by a laser trapping system. © 2013 Optical Society of America OCIS codes: (150.6910) Three-dimensional sensing; (100.2000) Digital image processing; (330.4150) Motion detection. http://dx.doi.org/10.1364/AO.52.007530
1. Introduction
The ability to track the motion of microscopic particles in aqueous solutions has important applications in the development of many scientific instruments and modern research techniques, e.g., those using optical trapping [1–4] and magnetic tweezers [5–7] to enable motion measurement at the nanometer scale and force sensing at the subpiconewton level. Optical trapping has been instrumental in studying biological systems under physiological conditions [8–15]. It is well suited for quasi-static force measurement [16] 1559-128X/13/317530-10$15.00/0 © 2013 Optical Society of America 7530
APPLIED OPTICS / Vol. 52, No. 31 / 1 November 2013
and dynamic force sensing [17]. Magnetic tweezers have been used in a wide variety of applications, ranging from manipulating biological macromolecules [18,19], probing cell membranes [20,21], to characterizing intracellular properties [22–24]. Existing particle-tracking techniques are divided into two distinct categories, namely laser measurement techniques [25–27] and visual sensing based methods [28–32]. Laser measurement techniques, detecting the interference between the nonscattered light and either forward-scattered light [25] or backscattered light [26] signal, are commonly used in optical trapping systems to measure the threedimensional (3D) position of the trapped particle. In a typical system, the scattered interference pattern
at the conjugated focal plane of a condenser is projected onto a quadrant photodiode (QPD), by an auxiliary lens, the signal from which is used to determine the 3D position of the particle, allowing a bandwidth of more than 100 kHz [2]. The measurement range is, however, limited to several hundred nanometers in the two lateral directions and about 1 μm in the axial direction [26,27]. Moreover, the laser measurement technique always involves the establishment of a complicated optical system and alignment of many optical components, and a single laser beam cannot simultaneously track multiple targets. Visual sensing provides an easy-to-handle alternative solution, wherein a charge-coupled device (CCD) or complementary metal–oxide semiconductor (CMOS) camera is employed to acquire the images of the target/particle. Each frame of the incoming images is then processed by a particle-tracking algorithm implemented as a program in either a computer or a dedicated image-processing unit to calculate the 3D position of the particle. The spatial resolution of a visual sensing system is dictated by the quality of the image acquired by the camera and relies on the underlying principle of the particletracking algorithm implemented. The temporal resolution is limited by the maximum frame rate of the camera in conjunction with the image processing time. Using a CMOS camera and a novel intensitybased method, the particle-tracking system in [28] achieved subnanometer resolution in all three axes at 400 Hz when employed to track the 3D motion of a 4.5 μm particle. The method was later improved to be a shape-based method to track fluorescent particles in the presence of photobleaching and illumination variation [29]. Meanwhile, since multiple targets within the field of view of the camera could be acquired into the same image frame, parallel tracking of multiple particles at 40 Hz was realized [33]. The relative low measurement bandwidth (
Department of Mechanical and Aerospace Engineering, The Ohio State University, Columbus, Ohio 43210, USA 2
Department of Physiology and Cell Biology, The Ohio State University, Columbus, Ohio 43210, USA *Corresponding author: [email protected] Received 2 August 2013; revised 4 October 2013; accepted 4 October 2013; posted 7 October 2013 (Doc. ID 195116); published 28 October 2013
This paper presents a real-time visual sensing system, which is created to achieve high-speed three-dimensional (3D) motion tracking of microscopic spherical particles in aqueous solutions with nanometer resolution. The system comprises a complementary metal–oxide-semiconductor (CMOS) camera, a field programmable gate array (FPGA), and real-time image processing programs. The CMOS camera has high photosensitivity and superior SNR. It acquires images of 128 × 120 pixels at a frame rate of up to 10,000 frames per second (fps) under the white light illumination from a standard 100 W halogen lamp. The real-time image stream is downloaded from the camera directly to the FPGA, wherein a 3D particle-tracking algorithm is implemented to calculate the 3D positions of the target particle in real time. Two important objectives, i.e., real-time estimation of the 3D position matches the maximum frame rate of the camera and the timing of the output data stream of the system is precisely controlled, are achieved. Two sets of experiments were conducted to demonstrate the performance of the system. First, the visual sensing system was used to track the motion of a 2 μm polystyrene bead, whose motion was controlled by a three-axis piezo motion stage. The ability to track long-range motion with nanometer resolution in all three axes is demonstrated. Second, it was used to measure the Brownian motion of the 2 μm polystyrene bead, which was stabilized in aqueous solution by a laser trapping system. © 2013 Optical Society of America OCIS codes: (150.6910) Three-dimensional sensing; (100.2000) Digital image processing; (330.4150) Motion detection. http://dx.doi.org/10.1364/AO.52.007530
1. Introduction
The ability to track the motion of microscopic particles in aqueous solutions has important applications in the development of many scientific instruments and modern research techniques, e.g., those using optical trapping [1–4] and magnetic tweezers [5–7] to enable motion measurement at the nanometer scale and force sensing at the subpiconewton level. Optical trapping has been instrumental in studying biological systems under physiological conditions [8–15]. It is well suited for quasi-static force measurement [16] 1559-128X/13/317530-10$15.00/0 © 2013 Optical Society of America 7530
APPLIED OPTICS / Vol. 52, No. 31 / 1 November 2013
and dynamic force sensing [17]. Magnetic tweezers have been used in a wide variety of applications, ranging from manipulating biological macromolecules [18,19], probing cell membranes [20,21], to characterizing intracellular properties [22–24]. Existing particle-tracking techniques are divided into two distinct categories, namely laser measurement techniques [25–27] and visual sensing based methods [28–32]. Laser measurement techniques, detecting the interference between the nonscattered light and either forward-scattered light [25] or backscattered light [26] signal, are commonly used in optical trapping systems to measure the threedimensional (3D) position of the trapped particle. In a typical system, the scattered interference pattern
at the conjugated focal plane of a condenser is projected onto a quadrant photodiode (QPD), by an auxiliary lens, the signal from which is used to determine the 3D position of the particle, allowing a bandwidth of more than 100 kHz [2]. The measurement range is, however, limited to several hundred nanometers in the two lateral directions and about 1 μm in the axial direction [26,27]. Moreover, the laser measurement technique always involves the establishment of a complicated optical system and alignment of many optical components, and a single laser beam cannot simultaneously track multiple targets. Visual sensing provides an easy-to-handle alternative solution, wherein a charge-coupled device (CCD) or complementary metal–oxide semiconductor (CMOS) camera is employed to acquire the images of the target/particle. Each frame of the incoming images is then processed by a particle-tracking algorithm implemented as a program in either a computer or a dedicated image-processing unit to calculate the 3D position of the particle. The spatial resolution of a visual sensing system is dictated by the quality of the image acquired by the camera and relies on the underlying principle of the particletracking algorithm implemented. The temporal resolution is limited by the maximum frame rate of the camera in conjunction with the image processing time. Using a CMOS camera and a novel intensitybased method, the particle-tracking system in [28] achieved subnanometer resolution in all three axes at 400 Hz when employed to track the 3D motion of a 4.5 μm particle. The method was later improved to be a shape-based method to track fluorescent particles in the presence of photobleaching and illumination variation [29]. Meanwhile, since multiple targets within the field of view of the camera could be acquired into the same image frame, parallel tracking of multiple particles at 40 Hz was realized [33]. The relative low measurement bandwidth (
