Mobile Sensing V Motion Analysis. Spring 2015 Petteri Nurmi

Mobile Sensing V Motion Analysis Spring 2015 Petteri Nurmi 31.3.2015 1

Learning Objectives Understand the basic motion related forces, their relationships, and how they can be sensed Why the accelerometer is so important for motion analysis? How can true motion be extracted from accelerometer signals? What is pedestrian dead reckoning? How accelerometers can be used to estimate displacements? What alternatives there are to accelerometers in terms of motion analysis? 31.3.2015 2

Motion Analysis: Basics Motion defined as change in the position of object with respect to time Kinematics is a branch of classical mechanisms that describes motion of objects Motion analysis focuses on estimating kinematic primitives and relating those to behaviour of user Velocity: rate of change in position over time Acceleration: rate of change in the velocity Displacement: extent of change in the position of an object relative to a reference point/frame Translational: distance Rotational: orientation Further patterns can be extracted that characterize the velocity/acceleration/displacement over time 31.3.2015 3

Motion Analysis: Sensors Kinematic laws relate motion primitives to each other è extracting one dimension + time sufficient for motion analysis Displacement sensing: GPS and other positioning techniques enable extracting motion primitives through analysis of position changes Not covered on this course; see location-awareness course Velocity sensing: Cameras enable estimating rate of movement over time Acceleration sensing: Inertial sensors (accelerometer and gyroscope) enable capture of acceleration (and rotational) primitives Additionally, vertical and horizontal components of motion can be separated if angle of motion can be detected Barometer main sensor for this purpose 31.3.2015 4

Sensors: Barometer Device that measures atmospheric pressure Smartphones contain MEMS barometers which return current atmospheric pressure in millibars Main use in motion analysis: altitude estimation Pressure decreases as altitude increases, respectively pressure increases as altitude decreases Especially useful for detecting slopes, elevators, escalators etc. Main advantage position/placement independence Except if in air-tight environment (bags can distort) Disadvantages Requires temperature compensation (usually done on chip with additional sensor è increased power consumption) Reference level for comparing measurements depends on current weather è relative altitude 31.3.2015 5

Example Applications Fitness related applications Estimate total number of steps taken during a day or overall metabolic expenditure Pedestrian dead reckoning Incremental positioning using step counting together with step length estimation Sports medicine Estimate motion impacts and their severity, correlates with potential injuries Driving related applications Estimate driving skill, fuel consumption, transportation modality, or even the path taken by the car 31.3.2015 6

Velocity and Distance Importance of accelerometer comes from its capability to estimate traveled distance and velocity Acceleration a = dv/dt, i.e., change of velocity over time è velocity is integral of acceleration over time Velocity v = ds /dt, i.e., change in position over time è distance is the double integral of acceleration over time Note (!): only applicable to dynamic or linear acceleration where gravity has been eliminated Errors in acceleration values cumulative over time è estimates drift Drift compensation: other sensors need to be used to recalibrate accelerometer errors / distance estimates 31.3.2015 7

Practical Considerations: Numerical Integration Numerical integration refers to computational techniques for evaluating the value of an integral As accelerometer values discretely sampled, distance and velocity estimates need to be obtained numerically Trapezoidal integration: For velocity, x k is time and f(x k ) is acceleration For distance, x k is time and f(x k ) is velocity 31.3.2015 8

Numerical Integration: Example x y z -1.32 0.88-0.20-0.81 0.60-0.14-0.68 0.60-0.14-0.60 0.67-0.16-0.68 0.67-0.16 FX = (FX1 + FX2) + (FX2 + FX3) + = -6.18 FY = 5.28 FZ = -1.24 Integral given by: -3.09, 2.64, 0.62 31.3.2015 9

Tangential and Centripetal Acceleration When the user moves along a curved path, linear acceleration is divided into two components Tangential acceleration, change of tangential velocity over time Centripetal acceleration, change in the direction of the tangential acceleration Magnitude depends on the radius of the curve Different components need to be isolated to reliably obtain velocity and distance estimates Isolating components may also benefit applications, e.g., centripetal force tells about driving habits Requires fusing measurements from gyroscope and/or magnetometer 31.3.2015 10

Attitude Estimation (not altitude!) The problem of determining the orientation of object Widely studied, e.g., in aerospace, robotics, and commercial motion capture (movies, 3D modelling)...but most works rely on high quality sensors Reference frame Abstract coordinate system together with a set of physical reference points Local reference frame: relative to a local environment Global reference frame: globally unique In mobile sensing, THREE reference frames Phone, User, Global 31.3.2015 11

Attitude Estimation User reference frame defined by different forces: Longitudinal acceleration (x-axis): linear acceleration along longitudinal coordinates ( forward motion ) Lateral acceleration (y-axis): linear acceleration along the latitudinal coordinates ( sideways motion ) Gravity (z-axis): acceleration along vertical dimension Information about direction of movement required for transforming to global reference frame Latitude and longitude defined relative to magnetic north, discussed during next lecture 31.3.2015 12

Linear Acceleration Linear acceleration reflects dynamic changes in the velocity of an object, i.e., true motion Linear acceleration = Measured acceleration gravity As devices in arbitrary orientation, effects of gravity distributed along different axes Gravity estimation The process of estimating the coordinate-wise components of gravity from accelerometer Once an estimate of gravity obtained, measurements can be projected onto horizontal and vertical planes 31.3.2015 13

Linear Acceleration: Projections g y g g z g x Standard formula for vector projection We denote: Estimated gravity vector: g = (g x, g y, g z ) Measured acceleration: a = (a x,a y,a z ) Dynamic acceleration: d = (a x -g x,a y -g y,a z -g z ) Estimating true linear acceleration requires rotating the reference frame by projecting measurements Vertical projection of dynamic acceleration: p = (d g / g g) g (d g = dot product of d and g) Horizontal projection of dynamic acceleration: h = d - p 31.3.2015 14

Gravity Estimation: Mean Filter Gravity constant force exerting the sensor è in theory low-pass filter can be used to isolate gravity Respectively, high-pass filter would return gravity eliminated acceleration In practice, some movement usually always occurs within the low frequencies Hand tremors, minor foot movements Measurement fluctuations due to small changes in orientation è not a robust solution for estimating gravity Alternative, mean-filter based estimation: g = α g + (1 - α) a (α = 0.7 0.9) 31.3.2015 15

Gravity Estimation: Mizell and Variants Mizell: using mean over a 30-second window Good for kinematic/pedestrian motion Low frequency tremors affect gravity estimates but are uncorrelated over time Variants: using mean or median over x-s. window JigSaw: mean over 4 seconds Nericell: median over 10 seconds Long windows more robust, but have larger latency Issue if device orientation or user activity change Short windows can average too aggressively Problem during sustained acceleration, such as vehicular movement, where removes all motion related information 31.3.2015 16

Gravity Estimation: Example blue: actual measurement Black: mean (0Hz lowpass filter) red: mean-filter 31.3.2015 17

Gravity Estimation: Opportunistic Algorithms Intuition: when device is stationary, the only motion that is influencing the sensor is gravity Opportunistic gravity estimation 1. Identify (almost) stationary periods 2. Estimate gravity from measurements of these periods Main challenge determining when sensor is stationary, typically combine different heuristics: Threshold on variance of accelerometer magnitude (fixed or adaptive threshold) Constraint on magnitude, e.g., require that magnitude close to actual gravity Additionally gyroscopes can be used to filter out orientation changes and effects of centripetal forces 31.3.2015 18

Gravity Estimation: Opportunistic Algorithms 31.3.2015 19

Distance Estimation As discussed, in elapsed distance can be estimated (in theory) by double integrating linear acceleration In practice, this typically rather inaccurate Linear acceleration contains both forward and sideways movements (lateral and longitudinal) Errors in gravity estimation cause drift accumulation And there are many other sources of errors Applicable ONLY when recalibration available Stride-based distance estimation can be used as more accurate alternative during pedestrian motion: Step length contains relatively small intraperson variation è counting steps and estimating their length can be used to estimate distance 31.3.2015 20

Dead Reckoning and Pedestrian Dead Reckoning (PDR) Source: http://en.wikipedia. org/wiki/dead_rec koning Dead (or deduced) reckoning refers to extrapolation of position from last known position Also referred to as inertial navigation Requires information about Direction of motion Distance travelled since last known position Pedestrian Dead Reckoning Implementation of dead reckoning for pedestrian motion Distance travelled estimated by analyzing walking patterns (this lecture) Orientation estimated through multisensor fusion (next lecture) 31.3.2015 21

Pedestrian Dead Reckoning: Distance Estimation PDR systems a special case of dead reckoning targeted at pedestrians Displacement (i.e., elapsed distance) estimated by analysing walking patterns of the user Distance: d = i=1 n l i n = number of steps l i = length of step i Accordingly, requires mechanisms for (i) counting steps and (ii) estimating their length Widely studied in specialized domains using wearable sensors at fixed orientations E.g., emergency response and sports Typically either hip or foot-mounted sensing units 31.3.2015 22

Step Counting Apply to a mean filtered signal, otherwise too noisy! Threshold-based Simplest approach for step counting, considers any value exceeding a predefined threshold as step Windowed peak detection Detect local maxima in accelerometer measurements, consider each maxima as a step Require minimum distance between successive peaks e.g., with a sampling rate of 100Hz, steps should be at least 25 samples apart Minor fluctuations can also be reduced by requiring a minimum magnitude for the peak Mean crossing Each positive crossing of the mean considered a step Motivated by physical models of stride 31.3.2015 23

Step Counting: Example Mean Crossings Original data Mean filtering Peak Detection 31.3.2015 24

Step Counting: Autocorrelation Normalized autocorrelation peaks at a frequency equivalent to the gait cycle Fix a range τ MIN τ MAX and search for the lag τ that maximizes autocorrelation within this range Number of steps equal to the fraction of times the range appears within the window used for autocorrelation Note: if window length sufficiently long, can capture both the step cycle and gait cycle (i.e., step pair) Requires that walking estimation is applied on the measurements, otherwise pauses cause errors Generally a good approach as long as the cycle is continually reassessed 31.3.2015 25

Step Counting: Example of Autocorrelation Peaks Original data Autocorrelation Min Max Peaks τ = 42 and τ = 95 è Steps either 7 or 6.2, depending whether we use the step cycle or step pair cycle 31.3.2015 26

Step Detection: Other Approaches Dynamic Time Warping Template-based approach, extracts a model of the gait cycle and searches for it in the acceleration signal Transformation-based approaches Combine walk detection and step counting, operate using transformed signals Short Time Fourier Transform (STFT): variant of autocorrelation but applied on spectrograms Wavelet: mean crossings from wavelet transformed signals Classification approaches Hidden Markov Model (HMM), clustering: train classifiers/clusters for different phases of gait cycle 31.3.2015 27

Step Length Models Grieve: Constant model Average step length of a human between 60 75cm è constant step-size can be used for short distances And in practice results in smaller error than other possible sources of error If height of a person known, more accurate estimator can be obtained: step size / height 0.41 0.45 Biophysical models Step length, stride frequency, and walking speed determined to minimize metabolical energy Typically consider non-linear or power law relationships between walking speed and distance 31.3.2015 28

Step Length Models Two different linear models: Variance of accelerometer magnitude coefficients Linear model Step length generally depends on stride frequency Linear models represent step length as a function of frequency and other factors, such as overall magnitude of acceleration or height If ground truth distance measurements available, function can be learned automatically E.g., measure distances opportunistically using positioning systems and relate step count and frequency to distance using regression height frequency 31.3.2015 29

Zero Velocity Update Double integration of linear acceleration can be used when recalibration is available Foot-mounted inertial measurement units apply so-called zero velocity updates for recalibration Detect the impact of foot on the ground During this phase, velocity is momentarily zero, i.e., only gravity impacts sensor in a known orientation Accelerometer (and gyroscope) drift can be estimated during this phase and removed from measurements Also drift in velocity can be estimated Kalman (or extended Kalman) filter can be used to propagate drift correction and re-estimate state of system In theory, position of a phone constrained while in trouser pockets è similar idea can be applied, but less accurate 31.3.2015 30

Estimating Direction of Movement Linear acceleration indicates extent of motion, but not its direction Which way is forward? Attitude estimation can be used to estimate the orientation of the device But often there is an offset between the user s reference frame and device reference frame Determining the offset requires estimating the direction of motion Projection-based approaches (PCA): analyze latent structure of horizontal motion Tracking-based: detect a reference point where offset known and estimate changes from offset 31.3.2015 31

Estimating Direction of Movement: PCA The direction of movement can (in certain situations) determined using principal component analysis (PCA) Basic idea: variance in dynamic motion (usually) highest along the axis of motion Motion can thus be separated by applying PCA on the horizontal projection of linear acceleration Alternatively, can be detected also from raw measurements by applying PCA on them Gravity typically the dominant component (though not always), motion second component Main problem Requires high VARIANCE along the motion dimension è not work well during motorized movement or when device in bag or upper body 31.3.2015 32

PCA-based direction estimation: example Original data Gravity eliminated measurements Linear acceleration highest along x axis y-axis also contains high acceleration z-axis contains gravity, slight influence from y-axis 0.86 0.50 0.00-0.49 0.84 0.24 0.11-0.21 0.97 PCA 31.3.2015 33

Other Motion Primitives: Acceleration Signatures - Characterize individual acceleration (deceleration) events Extracted by applying windowed peak detection on horizontal acceleration Can be used to characterize vehicular motion, e.g., extent of braking, type of vehicle, etc. Each detected peak can be further represented using different features that relate to its shape Intensity, length, volume Skewness, excess kurtosis 31.3.2015 34

Estimating Direction of Movement: Other Solutions Quaternion/rotation matrix approach Bias estimated from changes in device orientation when angle between device and motion known at ONE point Zero velocity: motion approx. orthogonal during step impact Interaction: motion (typically) in forward direction when device is held in hand and interacted with Trilateration Estimate motion angle from changes in distance vectors to reference points Magnetometer-based: magnetic anomalies Positioning-based: differences in position estimates 31.3.2015 35

Vision-Based Motion Estimation Alternative to accelerometers is to estimate motion primitives using cameras Widely investigated problem within computer vision Basic principle: Images measure scene at a given moment Differences between successive images identify points that are moving as long as all camera parameters remain same: focal length, aperture, intrinsic calibration parameters etc. In practice, filtering steps needed to overcome noise and small scale variations in lightning etc. 31.3.2015 36

Vision-Based Motion Estimation Foreground vs. Background Objects that are moving in an image are referred to as foreground objects Objects that remain (approximately) constant are referred to as background objects First step in motion analysis is to separate background and foreground objects Displacement estimation If the camera location and parameters are known, displacements can be estimated by analysing changes in the positions of foreground objects 31.3.2015 37

Vision-Based Motion Estimation Vision-based sensing can also be applied for crowd activity sensing Instead of looking at individual object, aggregate changes over entire scene Pixel counting from difference images can be used for estimating overall level of activity Similarly to displacement tracking, requires background removal to be successful Need to also compensate for angle of camera and the position of the objects within the scene 31.3.2015 38

Summary Accelerometer most important sensor for motion analysis Usually we are interested in linear acceleration, which measures true dynamic motion Linear Acceleration = Measured Acceleration Gravity Gravity also essential for estimating orientation of sensor (attitude estimation) Gyroscope can be used to separate rotational effects (tangential and centripetal acceleration) Barometer can be used to assist in detecting vertical motion Pedestrian dead reckoning = technique for estimating changes in position using analysis of walking patterns Walking direction can be estimated using PCA and/or rotation tracking from a known reference point Vision-based algorithms provide an alternative for extracting motion patterns 31.3.2015 40

References Woodman, O. & Harle, R., Pedestrian localisation for indoor environments, Proceedings of the 10th international conference on Ubiquitous computing (UbiComp), ACM, 2008, 114-123 Brajdic, A. & Harle, R., Walk Detection and Step Counting on Unconstrained Smartphones, Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM, 2013, 225-234 Hemminki, S.; Nurmi, P. & Tarkoma, S., Accelerometer-Based Transportation Mode Detection on Smartphones, Proceedings of the 11th ACM Conference on Embedded Networked Sensor Systems (SenSys), ACM, 2013 Steinhoff, U. & Schiele, B., Dead Reckoning from the Pocket - An Experimental Study, Proceedings of the IEEE International Conference on Pervasive Computing and Communications (PerCom), IEEE, 2010, 162 170 Mizell, D., Using gravity to estimate accelerometer orientation, Proc. Seventh IEEE International Symposium on Wearable Computers, 2005, 252-253 Rai, A.; Chintalapudi, K. K.; Padmanabhan, V. N. & Sen, R., Zee: Zero-Effort Crowdsourcing for Indoor Localization, Proceedings of The 18th Annual International Conference on Mobile Computing and Networking (MobiCom), ACM, 2012, 293-304 31.3.2015 41

References Hemminki, S.; Nurmi, P. & Tarkoma, S., Gravity and Linear Acceleration Estimation on Mobile Devices, Proceedings of the 11th International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (Mobiquitous), 2014 Kunze, K. S.; Lukowicz, P.; Partridge, K. & Begole, B., Which Way Am I Facing: Inferring Horizontal Device Orientation from an Accelerometer Signal, Proceedings of the 13th IEEE International Symposium on Wearable Computers (ISWC), 2009, 149-150 Wang, H.; Sen, S.; Elgohary, A.; Farid, M.; Youssef, M. & Choudhury, R. R., No need to war-drive: unsupervised indoor localization, The 10th International Conference on Mobile Systems, Applications, and Services (MobiSys), ACM, 2012, 197-210 Cho, D.-K.; Mun, M.; Lee, U.; Kaiser, W. J. & Gerla, M., AutoGait: A mobile platform that accurately estimates the distance walked, Proceedings of the 8th Annual IEEE International Conference on Pervasive Computing and Communications (PerCom), IEEE Computer Society, 2010, 116-124 Grieve, D., Gait patterns and the speed of walking, Biomedical Engineering, 1968, 3, 119-122 Ma, R.; Li, L.; Huang, W. & Tian, Q., On pixel count based crowd density estimation for visual surveillance, IEEE Conference on Cybernetics and Intelligent Systems, 2004 31.3.2015 42