Cywee Group LTD v. Apple Inc.

Filing 1

COMPLAINT for Patent Infringement against Apple Inc. ( Filing fee $ 400, receipt number 0971-8556570.). Filed byCywee Group LTD. (Attachments: # 1 Exhibit A, # 2 Exhibit B, # 3 Civil Cover Sheet)(Kopeikin, Jill) (Filed on 4/22/2014)

EXHIBIT A

111111

1111111111111111111111111111111111111111111111111111111111111
US008552978B2

(54)

United States Patent

(10)

Ye et al.

(12)

(45)

3D POINTING DEVICE AND METHOD FOR
COMPENSATING ROTATIONS OF THE 3D
POINTING DEVICE THEREOF

(56)

(73)

Notice:

A
8/1992
8/1995
A
4/1999
A
112007
B2
B2
6/2007
B2
7/2007
B2
8/2007
B2
8/2008
200910262074 Al * 10/2009

Assignee: Cywee Group Limited, Tortola (VG)

( *)

5,138,154
5,440,326
5,898,421
7,158,118
7,236,156
7,239,301
7,262,760
7,414,611

Inventors: Zhou Ye, Foster City, CA (US);
Chin-Lung Li, Taoyuan County (TW);
Shun-Nan Liou, Kaohsiung (TW)

Subject to any disclaimer, the term of this
patent is extended or adjusted under 35
U.S.c. 154(b) by 144 days.

Filed:

Primary Examiner - Lun-Yi Lao
Assistant Examiner - Insa Sadio
(74) Attorney, Agent, or Firm - Ding Yu Tan

Jul. 6, 2011

(65)

Prior Publication Data
US 201110260968 Al

(57)

Oct. 27, 2011

(63)

Continuation-in-part of application No. 13/072,794,
filed on Mar. 28, 2011, which is a continuation-in-part
of application No. 12/943,934, filed on Nov. 11,201 O.

(60)

Provisional application No. 611292,558, filed on Jan.
6,2010.

(51)

Int. CI.
G06F 3/033
(2013.01)
(2006.01)
G09GS/08
U.S. CI.
USPC ........... 345/157; 3451156; 3451158; 3451173;
178118.01; 178118.03; 178119.01
Field of Classification Search
USPC ... 3451156-168,173-183; 178118.01-18.04,
178119.01-19.04
See application file for complete search history.

(58)

ABSTRACT

A 3D pointing device utilizing an orientation sensor, capable
of accurately transforming rotations and movements of the
3D pointing device into a movement pattern in the display
plane of a display device is provided. The 3D pointing device
includes the orientation sensor, a rotation sensor, and a computing processor. The orientation sensor generates an orientation output associated with the orientation of the 3D pointing device associated with three coordinate axes of a global
reference frame associated with the Earth. The rotation sensor
generates a rotation output associated with the rotation of the
3D pointing device associated with three coordinate axes of a
spatial reference frame associated with the 3D pointing
device itself The computing processor uses the orientation
output and the rotation output to generate a transformed output associated with a fixed reference frame associated with
the display device above. The transformed output represents
a segment of the movement pattern.

Related U.S. Application Data

(52)

18 Claims, 12 Drawing Sheets

342
346

348

304

926

I

/

,----

--------------~--l

I

:
I

Hotelling
Quinn
Quinn
Liberty
Liberty et al.
Liberty et al.
Liberty
Liberty
Nasiri et al. .................. 345/158

* cited by examiner

Appl. No.: 13/176,771

(22)

References Cited
U.S. PATENT DOCUMENTS

(75)

(21)

Patent No.:
US 8,552,978 B2
Date of Patent:
Oct. 8,2013

I

Data
Transmitting
Unit

Computing:
Processor:

L ______________________ J

I
I

910

u.s. Patent

Oct. 8,2013

Sheet 1 of 12

US 8,552,978 B2

120

122

Zp

FIG. 1 (RELATED ART)
ZD

120

122

FIG. 2 (RELATED ART)

u.s. Patent

Oct. 8,2013

Sheet 2 of 12

US 8,552,978 B2

310

330

320

FIG. 3

u.s. Patent

Oct. 8,2013

US 8,552,978 B2

Sheet 3 of 12

342

I
I
I
I
I
I

----rJ---1- j - 302
I

Rotation
Sensor

---!I

;344
Accelerometer

H-

;345
Magnetometer

I
I
I
I
I
I

346

348

i----~-----------~--fI
I
I
I
I

Data
T
ronsmitting
Unit

Computing
Processor

L _____________________

I
I
I
I
I
I

LI

L_________ J

I
I

FIG. 4

02

M
545

544

3~4

542

560~

522 520

FIG. 5

~

570

u.s. Patent

Oct. 8,2013

Sheet 4 of 12

US 8,552,978 B2

620

610

FIG. 6

u.s. Patent

Oct. 8,2013

US 8,552,978 B2

Sheet 5 of 12

705 "- Initialize an initial-value
set

1
710 "- Obtain a previous state

(1st quaternion) at T-1

1
715 "- Obtain measured angular
velocities at T

l
720 "- Obtain a current state

(2nd quaternion) at T

~

1

1

It

725 "-

Obtain measured axial
accelerations" of a
measured state at T

Obtain resultant
deviation including yaw, .....)745
pitch and roll angles

1
730 "- Calculate "predicted axial
accelerations" based on
current state at T

l
Obtain an updated state

735 "- (3rd quaternion) by

comparing current state
with measured state

Output 3rd quaternion r--1\-740
to 1st quaternion

r---

FIG. 7

u.s. Patent

Oct. 8,2013

US 8,552,978 B2

Sheet 6 of 12

7051'- Initialize an initial-value
set

1
710 ' - Obtain a previous state

(1 st quatemion) at T-1

....
~

1
715 ' - Obtain measured angular
velocities at T

~
720 ' - Obtain a current state
(2nd quaternion) at T

-----;;.

1

f---

1L740

1

Obtain "measured axial
accelerations" of a
725
'-- measured state at T

Obtain resultant
deviation including yaw,
pitch and roll angles

~

--,745

1

730 '-- Calculate "predicted axial

Obtain display data and
translate the resultant
lJ750
angles to movement
pattern in the display
reference frame

accelerations" based on
current state at T

1
Obtain an updated state
735 ' - (3rd quaternion) by
comparing current state
with measured state

Output 3rd quatemion
to 1st quaternion

f--

FIG. 8

u.s. Patent

Oct. 8,2013

Sheet 7 of 12

US 8,552,978 B2

926
........................

......

/

.-- - 946

............ .J..

924
Pm ax

--P

. . . "" ......
-

-

/--944 e. max -... -...
. . . . . "(
----... . . . . .......

-L. _

r-

942

--

_____ .-1._______

922

-9[---.. . . -.. . . .
--

d

910

FIG. 9

----::.::.~

c::::::y- 930

u.s. Patent
1005

Oct. 8,2013

US 8,552,978 B2

Sheet 8 of 12

Initialize an initial-value set J

~
1010I L Obtain a previous state
(1 st quaternion) at T-l

t
1015I L Obtain measured angular
velocities at T

t
1020I L Obtain a current state
(2nd quaternion) at T

-----?

Output 3rd quaternion
to 1st quaternion

1

t
1025' L Obtain "measured axial

Obtain resultant
deviation including yaw,
pitch and roll angles

accelerations" of a
measured state at T

~
Calculate "predicted axial

1030' L accelerations" based on
current state at T

~
1035I L

Obtain "measured
magnetism" of a
measured state at T

J
Calculate "predicted

1040' L magnetism" based on
current state at T

~
Obtain an updated state

1045' L (3rd quaternion) by

comparing current state
with measured state

-

FI G. 10

r-L1050

V

1060

u.s. Patent
1105

Oct. 8,2013

Initialize an initial-value set

US 8,552,978 B2

Sheet 9 of 12

J

~
1110I~ Obtain a previous state
(1st quaternion) at T-l

~
1115 ~ Obtain measured angular
velocities at T

~
1120I~ Obtain a current state
(2nd quaternion) at T

~
1125 ~

Obtain "measured axial
accelerations" of a
measured state at T

~
1130 ~

Calculate "predicted axial
accelerations" based on
current state at T

~
1135 ~

Obtain a 1st updated
state (3rd quaternion) by
comparing current state
with measured state

~

Output 4th quaternion
to 1st quaternion

1
Obtain resultant
deviation including yaw,
pitch and roll angles

~1155

---.r- 116O

1
Obtain display data and
translate the resultant
---.r- 1165
angles to movement
pattern in the display
reference frame

~
1140 ~ Obtain and calculate
"measured yaw angle" of
a measured state at T

~
1145 ~ Calculate "predicted
yaw angle" based on 1st
updated state at T

~
Obtain a 2nd updated
1150 ~ state (4th quaternion) by
comparing current state
with measured state

FIG. 11

u.s. Patent

Oct. 8,2013

Sheet 10 of 12

Obtain a previous state at T-1

US 8,552,978 B2

~-------------.

1210
Obtain a current state at T

i--------------l

I obtain measured angular I
I ______________ JI
L velocities at T

1220

1255
Perform 2n
1225
data association
Obtain 1st measured state at T
i--------------l
and determine whether
I obtain measured axial
I
the result of comparison
I accelerations at T
JI
falls within a
L- - - - - - - - - - - - - predetermined
1230
value
Calculate 1st predicted
No
measurement
1260
(predicted axial accelerations)
Obtain a 2nd updated
state
1265
Output updated state
to previous state
1270
Obtain resultant
deviation including
deviation angles in
spatial reference frame
1245
~---~~--~~

Obtain 2nd measured state at T
i--------------l

I obtain

measured magnetism I

l~~ '!?!! _ ~!____ J
a~~e_
1250
Calculate 2nd predicted
measurement tpredicted yaw angle)

FIG. 12

u.s. Patent

Oct. 8,2013

US 8,552,978 B2

Sheet 11 of 12

Begin

Generate an orientation output
associated with the 3D pointing
device associated with Earth

~

1320

Generate a rotation output
~1 340
associated with the 3D
pointing device

/'--1 360

I - -----------------~-I

I
I
I
I
I
I
I
I

I
I

\V

Obtain the orientation of
the display device
associated with Earth

---./i
I

1362

I
I
I
I
I

Obtain the orientation of the
3D pointing device
associated with the display
device

J! 1364

Generate a transformed
rotation associated with the
display device

J i 1366

Generate the transformed
output based on the
transformed rotation

LA- 1368
I
I
I

--------- ----------~
End

FI G. 13

u.s. Patent
342~

Rotation
Sensor

_F: 348
-,

,----------------I

344

,11

l- Accelerometer ~
I

Computing
Processor

I

:
L

US 8,552,978 B2

Sheet 12 of 12

Oct. 8,2013

Orientation Sensor

___________________

I
I

Computing
Processor

I

I
I

~./ -- -

Lr 1420

1410

~

FIG. 14
342~

Rotation
Sensor

_F: 348
-,

,----------------I

,11

344 '{--

345

Accelerometer

~

I
I
I
I

Computing
Processor

,

I
I

Computing
Processor

I

I
I

Lr 1420

~./---1510

l- Magnetometer
I
I
I ___________________
Orientation Sensor
L

I
I
I
I
I

~

FIG. 15
342~

Rotation
Sensor

_F: 348
-,

,-----------------

344

345

I

l- Accelerometer ~
I
I
I
I

Computing
Processor
I'

l- Magnetometer
I
I
I
L

Orientation Sensor

___________________

FIG. 16

I
I
I

I
I

....

Computing Lr 1420
Processor

~./---1610
I
I
I
I
I

~

US 8,552,978 B2
1
3D POINTING DEVICE AND METHOD FOR
COMPENSATING ROTATIONS OF THE 3D
POINTING DEVICE THEREOF

2

related to commercial vehicles or transportation such as ships
and airplanes. Conventionally, the yaw angle 111 may represent the rotation of the pointing device 11 0 about the Zp axis;
the pitch angle 112 may represent the rotation of the pointing
device 110 about the Ypaxis; the roll angle 113 may represent
CROSS-REFERENCE TO RELATED
the rotation of the pointing device 110 about the Xp axis.
APPLICATION
In a known related art as shown in FIG. 1, when the yaw
angle 111 of the pointing device 110 changes, the aforemenThis application is a continuation in part application of and
tioned pointer on the screen 122 must move horizontally or in
claims the priority benefit of U.S. application Ser. No. 13/072,
794, filed on Mar. 28, 2011, now pending. The prior applica- 10 a horizontal direction with reference to the ground in
response to the change of the yaw angle 111. FIG. 2 shows
tion Ser. No. 13/072,794 is a continuation in part application
what happens when the user rotates the pointing device 110
of and claims the priority benefit of U.S. application Ser. No.
counterclockwise by a degree such as a 90-degree about the
12/943,934, filed on Nov. 11, 2010, now pending, which
Xp axis. In another known related art as shown in FIG. 2,
claims the priority benefit of U.S. provisional application Ser.
No. 611292,558, filed on Jan. 6, 2010. The entirety of each of 15 when the yaw angle 111 changes, the aforementioned pointer
on the screen 122 is expected to move vertically in response.
the above-mentioned patent applications is hereby incorpoThe change of the yaw angle 111 can be detected by a gyrorated by reference herein and made a part of this specification.
sensor which detects the angular velocity Wx of the pointing
device 110 about the Xp axis. FIG. 1 and FIG. 2 show that the
BACKGROUND OF THE INVENTION
20 same change of the yaw angle 111 may be mapped to different
movements of the point on the screen 122. Therefore, a proper
1. Field of the Invention
The present invention generally relates to a 3D pointing
compensation mechanism for the orientation of the pointing
device, more particularly to a 3D pointing device for use in
device 110 is required such that corresponding mapping of
computers, motion detection or navigation utilizing a orienthe pointer on the screen 122 of the display 120 may be
tation sensor and a method for compensating signals of the 25 obtained correctly and desirably. The term compensation of
orientation sensor subject to movements and rotations of said
the prior arts by Liberty (U.S. Pat. Nos. 7,158,118, 7,262,760
3D pointing device.
and 7,414,611) refers to the correction and compensation of
2. Description of the Related Art
signals subject to gravity effects or extra rotations about the
axis related to "roll". The term of "comparison" of the present
FIG. 1 is a schematic diagram showing a user using a
portable electronic device 110, such as a 3D pointing device 30 invention may generally refer to the calculating and obtaining
or computer mouse, for detecting motions of the device and
of the actual deviation angles of the 3D pointing device 110
translating the detected motions to a cursor display such as a
with respect to the first reference frame or spatial pointing
cursor pointing on the screen 122 of a 2D display device 120.
frame Xp Y pZp utilizing signals generated by motion sensors
If the pointing device 110 emits a light beam, the correspondwhile reducing or eliminating noises associated with said
ing point would be the location where the light beam hits the 35 motion sensors; whereas the term mapping may refer to the
screen 122. For example, the pointing device 110 may be a
calculating and translating of said deviation angles in the
mouse of a computer or a pad of a video game console. The
spatial pointing frame Xp Y pZp onto the aforementioned
display device 120 may be a part of the computer or the video
pointer on the display plane associated with the 2D display
game console. There are two reference frames, such as the
device 120 of a second reference frame or display frame
spatial pointer reference frame and the display frame, asso- 40 XDYDZ D.
ciated with the pointing device 110 and the display device
It is known that a pointing device utilizing 5-axis motion
sensors, namely, Ax, Ay, Az, wyand wzmay be compensated.
120, respectively. The first reference frame or spatial pointer
reference frame associated with the pointing device 110 is
For example, U.S. Pat. No. 7,158,118 by Liberty, U.S. Pat.
defined by the coordinate axes X p, Y p and Zp as shown in
No. 7,262,760 by Liberty and U.S. Pat. No. 7,414,611 by
FIG. 1. The second reference frame or display frame associ- 45 Liberty provide such pointing device having a 5-axis motion
ated with the display device 120 is defined by the coordinate
sensor and discloses a compensation using two gyro-sensors
wyand W z to detect rotation about the Yp and Zp axes, and
axes X D,Y D and ZD as shown in FIG. 1. The screen 122 of the
display device 120 is a subset of the X D YD plane of the
accelerometers Ax, Ay andAz to detect the acceleration of the
reference frame X DY DZD associated with the display device
pointing device along the three axes of the reference frame
120. Therefore, the X D YD plane is also known as the display 50 Xp Y pZp. The pointing device by Liberty utilizing a 5-axis
plane associated with the display device 120.
motion sensor may not output deviation angles of the pointing
device in, for example, a 3D reference frame; in other words,
A user may perform control actions and movements utilizdue to due to the limitation of the 5-axis motion sensor of
ing the pointing device for certain purposes including entertainment such as playing a video game, on the display device
accelerometers and gyro-sensors utilized therein, the point120 through the aforementioned pointer on the screen 122. 55 ing device by Liberty caunot output deviation angles readily
For proper interaction with the use of the pointing device,
in 3D reference frame but rather a 2D reference frame only
when the user moves the pointing device 110, the pointer on
and the output of such device having 5-axis motion sensors is
the screen 122 is expected to move along with the orientation,
a planar pattern in 2D reference frame only. In addition, it has
been found that the pointing device and compensation disdirection and distance travelled by the pointing device 110
and the display 120 shall display such movement of the 60 closed therein cannot accurately or properly calculate or
pointer to a new location on the screen 122 of the display 120.
obtain movements, angles and directions of the pointing
The orientation of the pointing device 110 may be represented
device while being subject to undesirable interferences,
external or internal, in the dynamic environment during the
by three deviation angles of the 3D pointing device 110 with
respect to the reference frame XpYpZp, namely, the yaw
obtaining of the signals generated by the motion sensors, in
angle 111, the pitch angle 112 and the roll angle 113. The yaw, 65 particular, during unexpected drifting movements and/or
accelerations along with the direction of gravity. In other
pitch and roll angles 111, 112, 113 may be best understood in
relation to the universal standard definition of spatial angles
words, it has been found that dynamic actions or extra accel-

US 8,552,978 B2

3

4

erations including additional accelerations, in particular the
one acted upon the direction substantially parallel to or along
with the gravity imposed on the pointing device with the
compensation methods provided by Liberty, said pointing
device by Liberty cannot properly or accurately output the
actual yaw, pitch and roll angles in the spatial reference frame
Xp Y pZp and following which, consequently, the mapping of
the spatial angles onto any 2D display reference frame such as
X D YDZD may be greatly affected and erred. To be more
specific, as the 5-axis compensation by Liberty cannot detect
or compensate rotation about the Xp axis directly or accurately, the rotation about the Xp axis has to be derived from the
gravitational acceleration detected by the accelerometer. Furthennore, the reading of the accelerometer may be accurate
only when the pointing device is static since due to the limitation on known accelerometers that these sensors may not
distinguish the gravitational acceleration from the accelerationofthe forces including centrifugal forces or other types of
additional accelerations imposed or exerted by the user.
Furthermore, it has been found that known prior arts may
only be able to output a "relative" movement pattern in a 2D
reference frame based on the result calculated from the signals of motion sensors. For example, the abovementioned
prior arts by Liberty may only output a 2D movement pattern
in a relative marmer and a pointer on a display screen to show
such corresponding 2D relative movement pattern. To be
more specific, the pointer moves from a first location to a
second new location relative to said first location only. Such
relative movement from the previous location to the next
location with respect to time cannot accurately detennine
and/or output the next location, particularly in situations
where the previous location may have been an erred location
or have been faultily detennined as an incorrect reference
point for the next location that is to be calculated therefrom
and obtained based on their relative relationship adapted. One
illustration of such defect of known prior arts adapting a
relative relationship in obtaining a movement pattern may be
clearly illustrated by an example showing the faultily outputted movements of a pointer intended to move out of a bonndary or an edge of display screen. It has been fonnd that as the
pointer of known prior arts reaches the edge of a display and
continues to move out of the boundary or edge at a certain
extra extent beyond said boundary, the pointer fails to demonstrate a correct or "absolute" pattern as it moves to a new
location either within the display or remaining outside of the
bonndary; in other words, instead of returning to a new location by taking into acconnt said certain extra extend beyond
the bonndary made earlier in an "absolute" manner, the
pointer of known arts discards such virtual distance of the
extra extend beyond the bonndary already made and an erred
next position is faultily outputted due to the relative relationship adapted and utilized by the pointer.
Therefore, it is clear that an improved device for use in for
example motion detection, computers or navigation with
enhanced calculating or comparison method capable of accurately obtaining and calculating actual deviation angles in the
spatial pointer frame is needed. For applications of navigations or computers including portable communication
devices integrated with displays therein, the electronic device
may too include the mapping of such actual angles onto a
cursor, pointer or position infonnation on the display frame in
dynamic environments and conditions including undesirable
external interferences. In addition, as the trend of 3D technology advances and is applicable to various fields including
displays, interactive systems and navigation, there is a significant need for an electronic device, including for example
a motion detector, a 3D pointing device, a navigation equip-

ment, or a commnnication device integrated with motion
sensors therein, capable of accurately outputting a deviation
of such device readily useful in a 3D or spatial reference
frame. Furthermore, there is a need to provide an enhanced
comparison method and/or model applicable to the processing of signals of motion sensors such that errors and/or noises
associated with such signals or fusion of signals from the
motions sensors may be corrected or eliminated. In addition,
according to the field of application, such output of deviation
in 3D reference frame may too be further mapped or translated to a pattern useful in a 2D reference frame.

10

SUMMARY OF THE INVENTION
15

20

25

30

35

40

45

50

55

60

65

According to one aspect of an exemplary embodiment of
the present invention, an electronic device utilizing a nineaxis motion sensor module for use in for example computers,
motion detection or navigation is provided. The electronic
device comprises an accelerometer to measure or detect axial
accelerations Ax, Ay, Az, a magnetometer to measure or
detect magnetism Mx, My, Mz and a rotation sensor to measure or detect angular velocities Cllx ' Clly ' Cllz such that resulting
deviation including resultant angles comprising yaw, pitch
and roll angles in a spatial pointer frame of the electronic
device subject to movements and rotations in dynamic environments may be obtained and such that said resulting deviation including said resultant angles may be obtained and
outputted in an absolute marmer reflecting or associating with
the actual movements and rotations of the electronic device of
the present invention in said spatial pointer reference frame
and preferably excluding undesirable external interferences
in the dynamic environments.
According to another aspect of the present invention, the
present invention provides an enhanced comparison method
and/or model to eliminate the accumulated errors as well as
noises over time associated with signals generated by a combination of motion sensors, including the ones generated by
accelerometers Ax, A y , A z' the ones generated by magnetometers Mx ' My, Mz and the ones generated by gyroscopes Cllx '
Clly ' Cllx in dynamic environments. In other words, accumulated
errors associated with a fusion of signals from a motions
sensor module comprising a plurality of motion sensors to
detect movements on and rotations about different axes of a
reference frame may be eliminated or corrected.
According to still another aspect of the present invention,
the present invention provides an enhanced comparison
method to correctly calculating and outputting a resulting
deviation comprising a set of resultant angles including yaw,
pitch and roll angles in a spatial pointer frame, preferably
about each of three orthogonal coordinate axes of the spatial
pointer reference frame, by comparing signals of rotation
sensor related to angular velocities or rates with the ones of
accelerometer related to axial accelerations and the ones of
magnetometer related to magnetism such that these angles
may be accurately outputted and obtained, which may too be
further mapping to another reference frame different from
said spatial pointer frame.
In the event of interferences including external interferences introduced by either the device user or the surrounding
environment, such as external electromagnetic fields, according to still another aspect of the present invention, the present
invention provides a unique update program comprising a
data association model to intelligently process signals
received from a motion sensor module to output a resultant
deviation preferably in 3D reference frame such that the
adverse effects caused by the interferences may be advantageously reduced or compensated.

US 8,552,978 B2

5

6

According to still another aspect of the present invention,
the present invention further provides a mapping of the
abovementioned resultant angles, preferably about each of
three orthogonal coordinate axes of the spatial pointer reference frame, including yaw, pitch and roll angles in a spatial
pointer reference frame onto a display frame either external to
the device of the present invention or integrated therein such
that a movement pattern in a display frame different from the
spatial pointer reference frame may be obtained according to
the mapping or translation of the resultant angles of the resultant deviation onto said movement pattern.
According to another example embodiment of the present
invention, an electronic device capable of generating 3D
deviation angles and for use in for example computers,
motion detection or navigation is provided. The electronic
device may utilize a nine-axis motion sensor module with an
enhanced comparison method or model for eliminating accumulated errors of said nine-axis motion sensor module to
obtain deviation angles corresponding to movements and
rotations of said electronic device in a spatial pointer reference frame. The comparison method or model may be advantageously provided by comparing signals from the abovementioned nine-axis motion sensor module capable of
detecting rotation rates or angular velocities of the electronic
device about all of the X p, Y p and Zp axes as well as axial
accelerations and ambient magnetism including such as
Earth's magnetic field or that of other planets of the electronic
device along all of the X p, Y p and Zp axes such that deviation
angles of the resultant deviation of the electronic device of the
present invention may be preferably obtained or outputted in
an absolute marmer. In other words, the present invention is
capable of accurately outputting the abovementioned deviation angles including yaw, pitch and roll angles in a 3D spatial
pointer reference frame of the 3D pointing device to eliminate
or reduce accumulated errors and noises generated over time
in a dynamic environment including conditions such as being
subject to a combination of continuous movements, rotations,
external gravity forces, magnetic field and additional extra
accelerations in multiple directions or movement and rotations that are continuously nonlinear with respect to time; and
furthermore, based on the deviation angles being compensated and accurately outputted in 3D spatial reference frame
may be further mapped onto or translated into another reference frame such as the abovementioned display frame, for
example a reference in two-dimension (2D).
According to another example embodiment of the present
invention, a 3D pointing device utilizing a nine-axis motion
sensor module is provided; wherein the nine-axis motion
sensor module of the 3D pointing device comprises at least
one gyroscope, at least one accelerometer and at least one
magnetometer. In one preferred embodiment of the present
invention, the nine-axis motion sensor module comprises a
rotation sensor capable of detecting and generating angular
velocities of Cllx ' Clly ' Cllz ' an accelerometer capable of detecting
and generating axial accelerations of Ax, Ay, Az, and a magnetometer capable of detecting and generating magnetism of
Mx, My, Mz. It can be understood that in another embodiment, the abovementioned rotation sensor may comprise
three gyroscopes corresponding to each of the said angular
velocities of Cllx ' Clly ' Cllz in a 3D spatial reference frame of the
3D pointing device; whereas the abovementioned accelerometer may comprise three accelerometers corresponding to
each of the said axial accelerations Ax, Ay, Az in a 3D spatial
reference frame of the 3D pointing device; and whereas the
abovementioned magnetometer may comprise three magnetic sensors such as magneto-impedance (MI) sensors or
magneto-resistive (MR) sensors corresponding to each of the

said magnetism Mx, My, Mz in a 3D spatial reference frame
of the electronic device. The rotation sensor detects the rotation of the 3D pointing device with respect to a reference
frame associated with the 3D pointing device and provides a
rotation rate or angular velocity output. The angular velocity
output includes three components corresponding to the rotation rate or angular velocities Cllx ' Clly ' Cllz of the 3D pointing
device about the first axis, the second axis and the third axis of
the reference frame, namely, Xp, Yp and Zp of the 3D spatial
frame. The accelerometer detects the axial accelerations of
the 3D pointing device with respect to the spatial reference
frame such as a 3D-pointer reference frame and provides an
acceleration output. The acceleration output includes three
components corresponding to the accelerations, Ax, Az, Ay of
the 3D pointing device along the first axis, the second axis and
the third axis of the reference frame, namely, Xp, Yp and Zp
of the 3D spatial reference frame. The magnetometer detects
the magnetism of the electronic device with respect to the
spatial reference frame such as a 3D reference frame and
provides an magnetism output. The magnetism output
includes three components corresponding to the magnetism,
Mx, My, Mz of the 3D pointing device along the first axis, the
second axis and the third axis of the reference frame, namely,
Xp, Yp and Zp of the 3D spatial frame. It can, however, be
understood that the axes ofXp, Yp and Zp of the 3D spatial
reference frame may too be represented simply by the denotation of X, Y and Z.
According to another example embodiment of the present
invention, a method for compensating accumulated errors of
signals of the abovementioned nine-axis motion sensor module in dynamic environments associated in a spatial reference
frame is provided. In one embodiment, the method may be
performed or handled by a hardware processor. The processor
is capable of compensating the accumulated errors associated
with the resultant deviation in relation to the signals of the
abovementioned nine-axis motion sensor module of the 3D
pointing device subject to movements and rotations in a spatial reference frame and in a dynamic environment by performing a data comparison to compare signals of rotation
sensor related to angular velocities with the ones of accelerometer related to axial accelerations and the ones of magnetometer related to magnetism such that the resultant deviation
corresponding to the movements and rotations of the 3D
pointing device in the 3D spatial reference frame may be
obtained accurately over time in the dynamic environments.
According to another embodiment of the present invention,
a method for obtaining a resulting deviation including resultant angles in a spatial reference frame of a three-dimensional
(3D) pointing device utilizing a nine-axis motion sensor module therein and subject to movements and rotations in
dynamic environments in said spatial reference frame is provided. Said method comprises the steps of: obtaining a previous state associated with previous angular velocities Cllx ' Clly '
Cllz gained from the motion sensor signals of the nine-axis
motion sensor module at a previous time T-l; obtaining a
current state of the nine-axis motion sensor module by obtaining measured angular velocities Cllx ' Clly ' Cllz gained from the
motion sensor signals at a current time T; obtaining a measured state of the nine-axis motion sensor module by obtaining measured axial accelerations Ax, Ay, Az and measured
magnetism M x' My, Mz gained from the motion sensor signals
at the current time T and calculating predicted axial accelerations Ax', Ay', Az' and predicted magnetism Mx', My', Mz '
based on the measured angular velocities Cllx ' Clly ' Cllz of the
current state; obtaining an updated state of the nine-axis
motion sensor module by comparing the current state with the
measured state of the nine-axis motion sensor module; and

10

15

20

25

30

35

40

45

50

55

60

65

US 8,552,978 B2

7

8

calculating and converting the updated state of the nine-axis
motion sensor module to said resulting deviation comprising
said resultant angles in said spatial reference frame of the 3D
pointing device.
According to another aspect of the present invention, a
method for mapping deviation angles associated with movements and rotations of a 3D pointing device in a spatial
reference frame onto a display frame of a display having a
predetermined screen size is provided. In one embodiment,
the method for mapping or translating deviation angles
including yaw, pitch and roll angles in a spatial reference
frame to an pointing object, such as a pointer, having movements in a display frame, preferably a 2D reference frame,
comprises the steps of obtaining boundary information of the
display frame by calculating a predefined sensitivity associated with the display frame and performing angle and distance translation in the display frame based on said deviation
angles and boundary information.
According to another embodiment of the present invention,
a method for obtaining a resulting deviation including resultant angles in a spatial reference frame of a three-dimensional
pointing device utilizing a nine-axis motion sensor module
therein and subject to movements and rotations in dynamic
environments in said spatial reference frame is provided. Said
method comprises the steps of: obtaining a previous state of
the nine-axis motion sensor module; wherein the previous
state includes an initial-value set associated with at least
previous angular velocities gained from the motion sensor
signals of the nine-axis motion sensor module at a previous
time T -1; obtaining a current state of the nine-axis motion
sensor module by obtaining measured angular velocities W x '
Wy, Wz gained from the motion sensor signals of the nine-axis
motion sensor module at a current time T; obtaining a measured state of the nine-axis motion sensor module by obtaining measured axial accelerations Ax, Ay, Az gained from the
motion sensor signals of the nine-axis motion sensor module
at the current time T and calculating predicted axial accelerations Ax', Ay', Az' based on the measured angular velocities
wx, wy, wz of the current state of the nine-axis motion sensor
module; obtaining a first updated state of the nine-axis motion
sensor module by comparing the current state with the measured state of the nine-axis motion sensor module; obtaining
the measured state of the nine-axis motion sensor module by
obtaining and calculating a measured yaw angle gained from
the motion sensor signals of the nine-axis motion sensor
module at the current time T and calculating a predicted yaw
angle based on the first updated state of the nine-axis motion
sensor module; obtaining a second updated state of the nineaxis motion sensor module by comparing the current state
with the measured state of the nine-axis motion sensor module; and calculating and converting the second updated state
of the nine-axis motion sensor module to said resulting deviation comprising said resultant angles in said spatial reference
frame of the electronic device.
According to another aspect of the present invention, a 3D
pointing device is provided, which includes an orientation
sensor, a rotation sensor, and a computing processor. The
orientation sensor generates an orientation output associated
with an orientation of the 3D pointing device associated with
three coordinate axes of a global reference frame associated
with the Earth. The rotation sensor generates a rotation output
associated with a rotation of the 3D pointing device associated with three coordinate axes of a spatial reference frame
associated with the 3D pointing device. The computing processor uses the orientation output and the rotation output to
generate a transformed output associated with a fixed reference frame associated with a display device.

According to another aspect of the present invention, a
method for compensating the rotations of a 3D pointing
device is provided. The method includes the following steps.
Generate an orientation output associated with an orientation
of the 3D pointing device associated with three coordinate
axes of a global reference frame associated with the Earth.
Generate a rotation output associated with the rotation of the
3D pointing device associated with three coordinate axes of a
spatial reference frame associated with the 3D pointing
device. Use the orientation output and the rotation output to
generate a transformed output associated with a fixed reference frame associated with a display device.

10

BRIEF DESCRIPTION OF THE DRAWINGS
15

20

25

30

35

40

45

50

55

60

65

The accompanying drawings are included to provide a
further understanding of the invention, and are incorporated
herein for illustrative purposes only. The drawings illustrate
embodiments of the invention and, together with the description, serve to only illustrate the principles of the invention.
FIG. 1 shows a known related art having a 5-axis motion
sensor in 2D reference frame.
FIG. 2 shows the known related art having a 5-axis motion
sensor as shown in FIG. 1 being rotated or rolled about Xp
axis and is subject to further dynamic interactions or environment.
FIG. 3 is an exploded diagram showing an electronic
device of the present invention, such as a pointing device,
utilizing a nine-axis motion sensor module according to one
embodiment of the present invention.
FIG. 4 is a schematic block diagram illustrating hardware
components of an electronic device according to one embodiment of the present invention.
FIG. 5 is a schematic diagram showing another embodiment of an electronic device of the present invention, such as
a pointing device, utilizing a nine-axis motion sensor module
as well as an external processor.
FIG. 6 is an exploded diagram showing still another
embodiment of an electronic device of the present invention,
such as a smartphone or navigation equipment, utilizing a
nine-axis motion sensor module according to anther embodiment of the present invention.
FIG. 7 is a flow chart illustrating a method for obtaining a
resultant deviation of an electronic device of the present
invention subject to movements and rotations in a spatial
reference frame.
FIG. 8 shows another exemplary flow chart illustrating a
method for obtaining resultant deviation including mapping
of said deviation to a display of an electronic device according
to another embodiment of the present invention.
FIG. 9 is a schematic diagram showing the mapping of the
resultant angles of the resultant deviation according to an
embodiment of the present invention.
FIG. 10 is an exemplary flow chart illustrating another
embodiment of a method for obtaining a resultant deviation of
an electronic device of the present invention.
FIG. 11 shows an exemplary flow chart illustrating another
embodiment of a method for obtaining a resultant deviation
including mapping of such deviation to a display of an electronic device of the present invention.
FIG. 12 shows an exemplary flow chart illustrating a
method for obtaining resultant deviation of an electronic
device according to still another embodiment of the present
invention.
FIG. 13 is a flow chart of a method for compensating
rotations of a 3D pointing device according to an embodiment
of the present invention.

US 8,552,978 B2
9
FIG. 14, FIG. 15 and FIG. 16 are schematic diagrams
showing three 3D pointing devices according to three different embodiments of the present invention.

10

My, Mz. The rotation sensor 342 of the nine-motion sensor
module 302 detects and generates the first signal set including
angular velocities W x ' Wy, W z associated with the movements
and rotations of the electronic device 300 about each of three
orthogonal coordinate axes XpYpZp of the spatial reference
DESCRIPTION OF PREFERRED
EMBODIMENTS
frame. The angular velocities W x ' wyand W z are corresponding
to the coordinate axes X p , Yp and Zp respectively. The accelDetailed descriptions of preferred embodiments of the
erometer 344 detects and generates the second signal set
present invention recited herein are provided for illustrative
including axial accelerations Ax, Ay, Az associated with the
purposes only; examples of which are too illustrated in the 10 movements and rotations of the electronic device 300 along
accompanying drawings. In addition, similar reference numeach of the three orthogonal coordinate axes XpYpZp of the
bers in the drawings and the description may too refer to
spatial reference frame. The axial accelerations Ax, Ay and
Az are corresponding to the coordinate axes X p , Y p and Zp
similar parts or components.
respectively. The magnetometer 345 of the nine-motion senFIG. 3 is an exploded diagram showing an electronic
device 300 according to one embodiment of the present 15 sor module 302 detects and generates the third signal set
invention, such as a pointing device. The electronic device
including magnetism Mx, My, Mz associated with the movements and rotations of the electronic device 300 along each of
300 is subject to movements and rotations in dynamic envithe three orthogonal coordinate axes Xp Y pZp of the spatial
ronments in a spatial reference frame such as a 3D reference
frame. The spatial reference frame is analogous to the referreference frame. The magnetism Mx, My and Mz represent
ence frame Xp Y pZp also shown in FIG. 1 and FIG. 2. The 20 the strength and/or direction of ambient magnetic field (such
movements and rotations of the electronic device 300, such as
as the magnetic field of the Earth) of the electronic device
300. The magnetism Mx, My and Mz are corresponding to the
a pointing device, in the aforementioned dynamic environcoordinate axes X p , Y p and Zp respectively. It too can be
ments in the spatial reference frame may be continuously
nonlinear with respect to time. The term of "dynamic" recited
understood that the abovementioned nine axes of Xp Y pZp
herein may refer to moving or subject to motions in general. 25 may not need to be orthogonal in a specific orientation and
The electronic device 300 includes a top cover 310, a
they may be rotated in different orientations; the present
printed circuit board (PCB) 340, a rotation sensor 342, an
invention discloses such coordinate system for illustrative
accelerometer 344, a magnetometer 345, a data transmitting
purposes only and any coordinates in different orientation
unit 346, a computing processor 348, a bottom cover 320, and
and/or denotations may too be possible.
a battery pack 322. The top cover 310 may include a few 30
Furthermore, in one embodiment of the present invention,
the motion sensor module or nine-axis motion sensor module
control buttons 312 for a user to issue predefined commands
302 of the electronic device 300 may refer to a Micro-Electrofor remote control. In one embodiment, the housing 330 may
Mechanical-System (MEMS) type of sensor. In an explanacomprise the top cover 310 and the bottom cover 320. The
housing 330 may move and rotate in the spatial reference
tory example, the abovementioned rotation sensor 342 of the
frame according to user manipulation or any external forces 35 nine-axis motion sensor module 302 may further comprise at
least one resonating mass such that a movement of said at
in any direction and/or under the abovementioned dynamic
least one resonating mass along an axis of said spatial referenvironments. As shown in the FIG. 3, in one embodiment,
the rotation sensor 342, the accelerometer 344, the magneence frame may be detected and measured by said rotation
tometer 345, the data transmitting unit 346, and the computsensor using the Coriolis acceleration effect to generate said
ing processor 348 may be all attached to the PCB 340. The 40 first signal set comprising angular velocities wx, wy, wz in
PCB 340 is enclosed by the housing 330. The PCB 340
said spatial reference frame. It can be understood that for a
includes at least one substrate having a longitudinal side
three-axis rotation sensor of a MEMS type sensor, there may
configured to be substantially parallel to the longitudinal
be positioned three resonating masses along each of X, Y and
surface of the housing 330. An additional battery pack 322
Z axes of the spatial reference frame to generate and obtain
45 movements or displacements of the three resonating masses
provides electrical power for the electronic device 300.
thereof. It can too be understood that the nine-axis motion
Furthermore, in one embodiment, the abovementioned
sensor 302 of the present invention may also include a threedynamic environments, in which the electronic device 300 of
axis accelerometer, a three-axis rotation sensor and a threethe present invention may be present or subject to, may
axis magnetometer in a MEMS structure.
include undesirable external interferences to the electronic
The data transmitting unit 346 is electrically counected to
device 300 of the present invention. In one example, the 50
the nine-axis motion sensor module 302 for transmitting the
undesirable external interferences may refer to or include
undesirable axial accelerations caused by undesirable exterfirst, second and third signal sets. The data transmitting unit
nal forces other than a force of gravity. In another example,
346 transmits the first, second and third signal sets of the
the undesirable external interferences may also refer to or
nine-axis motion sensor module 302 to the computing proinclude undesirable magnetism caused by undesirable elec- ss cessor 348 preferably via electronic connections configured
on the PCB 340. The computing processor 348 receives and
tromagnetic fields.
FIG. 4 is a schematic block diagram illustrating hardware
calculates the first, second and third signal sets from the data
components of the electronic device 300. The electronic
transmitting unit 346. The computing processor 348 further
device 300 includes a nine-axis motion sensor module 302
communicates with the nine-axis motion sensor module 302
and a processing and transmitting module 304. The nine-axis 60 to calculate the resulting deviation of the electronic device
300 including three resultant angles preferably about each of
motion sensor module 302 includes the rotation sensor 342,
the three axes of the spatial reference frame. The resultant
the accelerometer 344 and the magnetometer 345. The proangles include the yaw angle 111, the pitch angle 112 and the
cessing and transmitting module 304 includes the data transroll angle 113 as shown in FIG. 1 and FIG. 2. In order to
mitting unit 346 and the computing processor 348.
The term "nine-axis" recited herein may refer to and gen- 65 calculate the resulting deviation, the computing processor
348 may utilize a comparison or algorithm to eliminate accuerally include the three angular velocities W x ' Wy, Wz ' the three
mulated errors of the first, second and/or third signal sets of
axial accelerations Ax, Ay, Az, and the three magnetism Mx,

US 8,552,978 B2
11

12

the nine-axis motion sensor module 302, whereby the resultant angles in the spatial reference frame, preferably about
each of three orthogonal coordinate axes of the spatial reference frame, of the resulting deviation of the nine-axis motion
sensor module 302 of the electronic device 300 is obtained
under the aforementioned dynamic environments excluding
the abovementioned undesirable external interferences and
such that it is preferably obtained and outputted in an absolute
manner reflecting or associating with the actual movements
and rotations of the electronic device 300, including such as a
pointing device, of the present invention in said spatial reference frame. In addition, said comparison utilized by the computing processor 348 may further comprise an update program to obtain an updated state of the nine-axis motion sensor
module based on a previous state associated with a first signal
set in relation to the angular velocities wx ' Wy, W z and a
measured state associated with both said second and third
signal sets in relation to the axial accelerations Ax, Ay, Az as
well as magnetism Mx, My, Mz. The abovementioned measured state may include a measurement of said second signal
set or measured Ax, Ay, Az and a predicted measurement of
Ax', Ay' and Az' obtained based on or calculated from a
current state of the motion sensor module 302. In addition, the
abovementioned measured state may too include a measurement of said third signal set or measured Mx, My, Mz and a
predicted measurement of Mx', My' and Mz' obtained based
on or calculated from the current state of the motion sensor
module 302. Details of different "states" of the nine-axis
motion sensor module 302 of the electronic device 300 of the
present invention are provided in the later content.
In one embodiment, the computing processor 348 of the
processing and transmitting module 304 may further include
a mapping program for translating the resultant angles of the
resulting deviation in the spatial reference frame to a movement pattern in a display reference frame different from the
spatial reference frame. The display reference frame is analogous to the reference frame X D YDZD in FIG. 1 and FIG. 2.
The movement pattern may be displayed on a screen of a 2D
display device similar to the display device 120 in FIG. 1 and
FIG. 2. The mapping program translates the resultant angles,
preferably about each of the three orthogonal coordinate axes
of the spatial reference frame to the movement pattern
according to a sensitivity input correlated to the display reference frame.
FI G. 5 is a schematic diagram showing an electronic device
500 utilizing a nine-axis motion sensor module according to
anther embodiment of the present invention in a 3D spatial
reference frame. As shown in FIG. 5, the electronic device
500 may comprise two parts 560 and 570 in data communication with each other. In one embodiment, the first part 560
includes a top cover (not shown), a PCB 540, a nine-axis
motion sensor module 502 comprising a rotation sensor 542,
an accelerometer 544 and a magnetometer 545, a data transmitting unit 546, a bottom cover 520, and a battery pack 522.
The data transmitting unit 546 transmits the first signal set
(wx' Wy, wz ) generated by the rotation sensor 542 of the
nine-motion sensor module 502 and the second signal set
(Ax, Ay, Az) generated by the accelerometer 544 as well as the
third signal set (Mx, My, Mz) generated by the magnetometer
545 of the nine-motion sensor module 502 to the data receiving unit 552 of the second part 570 via wireless communication or connection including wireless local area network
(WLAN) based on IEEE 802.11 standards or BluetoothTM. It
can be understood that in another embodiment, wired communication or connection via a physical cable or electrical
wires connecting the first part 560 and the second part 570
may too be possible. In one embodiment of the present inven-

tion, the motion sensor module or nine-axis motion sensor
module 502 of the electronic device 500 may refer to a
MEMS type of sensor. In an explanatory example, the abovementioned rotation sensor 542 of the nine-axis motion sensor
module 502 may further comprise at least one resonating
mass such that a movement of said at least one resonating
mass along an axis of said spatial reference frame may be
detected and measured by said rotation sensor using the
Corio lis acceleration effect to generate said first signal set
comprising angular velocities wx, wy, wz in said spatial reference frame. It can be understood that for a three-axis rotation sensor of a MEMS type sensor, there may be positioned
three resonating masses along each of X, Y and Z axes of the
spatial reference frame to generate and obtain movements or
displacements of the three resonating masses thereof It can
too be understood that the nine-axis motion sensor 502 of the
present invention may also include a three-axis accelerometer, a three-axis rotation sensor and a three-axis magnetometer in a MEMS structure.
In one embodiment, the second part 570 may be an external
processing device to be adapted to another electronic computing apparatus or system such as a standalone personal
computer or server 580; for instance, the second part 570 may
be coupled or adapted to an laptop computer via a standard
interface, such as the universal serial bus (USB) interface
depicted as shown in FIG. 5. The first part 560 and the second
part 570 communicate via the data transmitting unit 546 and
the data receiving unit 552. As previously mentioned, the data
transmitting unit 546 and the data receiving unit 552 may
communicate through wireless connection or wired connection. In other words, in terms of hardware configuration and
data transmission, in one embodiment of the present invention, the nine-axis motion sensor module 502 comprising the
rotation sensor 542, the accelerometer 544 and the magnetometer 545 may be disposed distally from the processing
unit or computing processor 554; the signals from the nineaxis motion sensor module 502 may then be transmitted via
the data transmitting units 546, 552 to the computing processor 554 via wired or wireless communication including for
example IEEE 802.11 standards or Bluetooth™.
The second part 570 of the electronic device 500 according
to one embodiment of the present invention comprises the
data transmitting unit 552 and the processor 554. The data
transmitting unit 552 of the second part 570 may be in data
communication with the other data transmitting unit 546 disposed distally therefrom in the first part 560 as previously
mentioned. The data transmitting unit 552 in the second part
570 receives the first, second and third signal sets from the
data transmitting unit 546 in the first part 560 and transmits
the first, second and third signal sets to the computing processor 554. In one embodiment, the computing processor 554
performs the aforementioned calculation as well as comparison of signals. In one embodiment, said comparison utilized
by the computing processor 554 may further comprise an
update program to obtain an updated state based on or from a
previous state associated with said first signal set and a measured state associated with said second and third signal sets.
The measured state may further include a measurement of
said second and third signal sets and predicted measurements
obtained based on the first signal set or based on a current
state of the motion sensor module 502. The computing processor 554 is external to the housing of the 3D pointing device
as depicted in FIG. 5. In one embodiment, the computing
processor 554 also performs mapping by translating the
resultant angles of the resulting deviation of the electronic
device in the spatial pointer reference frame, preferably about
each of three orthogonal coordinate axes of the spatial refer-

10

15

20

25

30

35

40

45

50

55

60

65

US 8,552,978 B2
13

14

ence frame, to a movement pattern in a display reference
frame associated with the notebook computer 580. The movement pattern may be displayed on the screen 582 of the
notebook computer 580.
FIG. 6 is an exploded diagram showing a portable electronic device 600, such as for example a 3D pointing device,
utilizing a nine-axis motion sensor module according to
anther embodiment of the present invention in a 3D spatial
reference frame. The portable electronic device 600 may
further comprises a built-in display 682; examples of the
portable electronic device 600 as an explanatory embodiment
of the present invention may include such as smartphone,
tablet PC or navigation equipment. In other words, the abovementioned display reference frame associated with a display
may need not to be external to the spatial reference frame in
terms of the hardware configuration of the present invention.
In one embodiment, the electronic device 600 comprises a
bottom cover 620, a PCB 640, a battery pack 622, a rotation
sensor 642, an accelerometer 644, a magnetometer 645, a data
transmitting unit 646, a computing processor 648, a display
682, and a top cover 610. Likewise, in one embodiment, the
housing 630 may comprise the top and bottom covers 610,
620. A built-in display 682 may too be integrated on the
housing 630; the nine-axis motion sensor module 602 may
comprise the rotation sensor 642, the accelerometer 644 and
the magnetometer 645. The data transmitting unit 646 and the
computing processor 648 may also be integrated as a processing and transmitting module 604 of the electronic device 600.
In one embodiment of the present invention, the motion sensor module or nine-axis motion sensor module 602 of the
portable electronic device 600 may refer to a MEMS type of
sensor. In an explanatory example, the abovementioned rotation sensor 642 of the nine-axis motion sensor module 602
may further comprise at least one resonating mass such that a
movement of said at least one resonating mass along an axis
of said spatial reference frame may be detected and measured
by said rotation sensor using the Corio lis acceleration effect
to generate said first signal set comprising angular velocities
wx, wy, wz in said spatial reference frame. It can be understood that for a three-axis rotation sensor of a MEMS type
sensor, there may be positioned three resonating masses
along each of X, Y and Z axes of the spatial reference frame
to generate and obtain movements or displacements of the
three resonating masses thereof. It can too be understood that
the nine-axis motion sensor 602 of the present invention may
also include a three-axis accelerometer, a three-axis rotation
sensor and a three-axis magnetometer in a MEMS structure.
The computing processor 648 of the processing and transmitting module 604 may too perform the mapping of resultant
deviation from or in said spatial reference frame or 3D reference frame to a display reference frame such as a 2D reference frame by translating the resultant angles of the resulting
deviation of the electronic device 600 in the spatial reference
frame, preferably about each of three orthogonal coordinate
axes of the spatial reference frame to a movement pattern in a
display reference frame associated with the electronic device
600 itself. The display 682 displays the aforementioned
movement pattern. The top cover 610 includes a transparent
area 614 for the user to see the display 682.
FIG. 7 is an explanatory flow chart illustrating a method for
obtaining and/or outputting a resultant deviation including
deviation angles in a spatial reference frame of an electronic
device, including such as a pointing device, navigation equipment or smartphone, having movements and rotations in a 3D
spatial reference frame and in dynamic environments according to an embodiment of the present invention. The method in
FIG. 7 may be a program such as an algorithm or a compari-

son model to be embedded or performed by the processing
unit or computing processor 348,554,648 of the processing
and transmitting module according to different embodiments
of the present invention recited herein for illustrative purposes.
Accordingly, in one embodiment of the present invention,
a method for obtaining a resultant deviation including deviation angles in a spatial reference frame of an electronic device
utilizing a nine-axis motion sensor module therein in
dynamic environments and preferably excluding undesirable
external interferences thereof is provided. As the electronic
device may be subject to movements and rotations in the
dynamic environments, undesirable interferences may cause
the measurements, calculations or outputs of the motion sensor module thereof to be errors orne. In one embodiment, said
method may comprise the following steps. First of all, as
shown in FIG. 7, different states including "previous state",
"current state", "measured state" and "update state" of the
nine-axis motion sensor module may be provided to represent
a step or a set of steps utilized by the method for obtaining the
resulting deviation in 3D reference frame, and preferably in
the abovementioned "absolute" manner. In one exemplary
embodiment, the method comprises the steps of obtaining a
previous state of the nine-axis motion sensor module (such as
steps 705, 710); and wherein the previous state may too
include an initial-value set predetermined to initialize said
previous state of the nine-axis motion sensor module at a
beginning of the method. The initial-value set may preferably
be utilized at said beginning of the method or a start of the
method where a previous state is not available to be obtained
from an updated state (to be recited hereafter). In another
embodiment where previous state may be obtained or updated
from an updated state, said previous state may be a first
quaternion inducing values associated with at least previous
angular velocities W x ' Wy, W z gained from the motion sensor
signals of the nine-axis motion sensor module at a previous
time T -1. A current state of the nine-axis motion sensor
module may then be subsequently obtained by obtaining
measured angular velocities W x ' Wy, W z gained from the
motion sensor signals of the nine-axis motion sensor module
at a current time T (such as steps 715, 720). A measured state
of the nine-axis motion sensor module may then be obtained
by obtaining measured axial accelerations Ax, Ay, Az gained
from the motion sensor signals of the nine-axis motion sensor
module at the current time T (such as step 725). Furthermore,
the step of calculating predicted axial accelerations Ax', Ay',
Az' based on the measured angular velocities wx ' Wy, W z of the
current state of the nine-axis motion sensor module (such as
step 730); obtaining an updated state of the nine-axis motion
sensor module by comparing the current state with the measured state of the nine-axis motion sensor module (such as
step 735); and calculating and converting the updated state of
the nine-axis motion sensor module to said resulting deviation comprising said resultant angles in said spatial reference
frame of the electronic device (745) may then be performed
and obtained; and whereby the resultant deviation comprising
deviation angles associated with the updated state of the
nine-axis motion module may be obtained excluding said
undesirable external interferences in the dynamic environments. In order to provide a continuous loop such as performed in a looped manner, the result of the updated state of
the nine-axis motion sensor module may preferably be outputted to the previous state; in one embodiment, the updated
state may be a quaternion, namely third quaternion as shown
in the figure, such that it may be directly outputted to the

10

15

20

25

30

35

40

45

50

55

60

65

US 8,552,978 B2
15

16

abovementioned previous state of another quatemion,
namely the abovementioned first quatemion and as shown in
the figure (such as step 740).
In addition, it can be understood that the abovementioned
comparison utilized by the processing and transmitting module and comprising the update program may too make reference to said different states of the nine-axis motion sensor
module as shown in FIGS. 7 and 8. As mentioned previously,
the update program may be utilized by the processor to obtain
the updated state of the nine-axis motion sensor module based
on the previous state associated with a first signal set in
relation to the angular velocities W x ' Wy, Wz and the measured
state associated with said second signal set in relation to the
axial accelerations Ax, Ay, Az. The abovementioned measured state may include a measurement of said second signal
set or measured Ax, Ay, Az and a predicted measurement of
Ax', Ay' andAz' obtained based on or calculated from the first
signal set. Details of each of the abovementioned states of the
nine-axis motion sensor module and the related steps of the
method for obtaining the resultant deviation of the electronic
device in 3D reference frame are as follows.
Referring to FIG. 7 again, the method for obtaining a
resultant deviation including resultant angles in a spatial reference frame of electronic device utilizing a nine-axis motion
sensor module according to one embodiment of the present
invention may begin at the obtaining of a previous state of the
nine-axis motion sensor module. In one embedment, the previous state of the nine-axis motion sensor module may preferably be in a form of a first quaternion, and the first quaternion may be preferably initialized (step 705) at a very
beginning of the process or method and as part of the obtaining of the previous state thereof. In other words, according to
one embodiment of the present invention, the signals of the
nine-axis motion sensor are preferably to be initialized
according to a predetennined value set or quaternion including such as zeros and in particular, the signal or value associated with the yaw angle in tenns of a quaternion value. The
four elements of the first quaternion may be initialized with
predetermined initial values. Alternatively, the first quaternion may be initialized or replaced by another signal sets
generated by the rotation sensor and the accelerometer at a
next time frame such that the method as shown in FIG. 7 is a
continuous loop between a previous time frame T -1 and a
present time frame T; details on the replacement of the first
quaternion at T -1 with the later outputted quaternion at T is to
be provided in the later content. It can be understood that one
may make reference to Euler Angles for definition on quaternion. Similarly, it can be easily comprehended that the abovementioned previous time T -1 and present time T may too be
substitute by a present time T and a next time T + 1 respectively
and shall too fall within the scope and spirit of the present
invention.
In addition, the abovementioned dynamic environments
may include undesirable external interferences to the present
invention as mentioned previously. For instance, the undesirable external interferences nay refer to or include undesirable
axial accelerations caused by undesirable external forces
other than a force of gravity and they may too include or refer
to include undesirable magnetism caused by undesirable
electromagnetic fields. In a preferred embodiment of the
present invention, one of the technical effects of the perfonn
of the method as shown in FIG. 7 include that the abovementioned updated state of the nine-axis motion sensor module to
said resulting deviation comprising said resultant angles in
said spatial reference frame of the electronic device (step 745)
may be preferably obtained excluding undesirable interferences in the dynamic environments, such as decoupling of

undesirable external forces from the force of gravity to
exclude undesirable axial accelerations and exclusion of
undesirable external magnetism caused or induced by undesirable electromagnetic fields in the dynamic environments.
The method illustrated in FIG. 7 may be perfonned in
consecutive time frames. According to one embodiment of
the present invention, steps 710-745 may be performed in a
looped mauner by such as a data processing unit of an electronic device of the present invention. In another embodiment, multiple steps may be perfonned simultaneously, such
as the obtaining of signals from the nine-axis motion sensor
module may be performed simultaneously instead of one
after another. It can therefore be understood that the steps
recited herein are for illustrative purposes only and any other
sequential orders or simultaneous steps are possible and shall
too be considered to be within the scope of the present invention. The first quaternion with respect to the previous time T
is obtained as shown in the figure as step 710. When step 710
is perfonned, the first quaternion initialized in step 705 is
obtained. Otherwise, the first quaternion used in the present
time T is generated in the previous time T -1. In other words,
the step 71 0 may generally refer to or represented by the
abovementioned "previous state" of the nine-axis motion sensor module; according to another embodiment, the previous
state may refer to the steps of 705 and 710.
The next may be to obtain the first signal set generated by
the rotation sensor, which includes the measured angular
velocities wx ' wyand W z as shown in step 715 according to an
exemplary embodiment of the present invention. In step 720,
the second quaternion with respect to a present time T is
calculated and obtained based on the angular velocities W x ' Wy
and W z . The step 715 and 720 may generally refer to or may be
represented by the abovementioned "current state" of the
nine-axis motion sensor module. In one embodiment, the
computing processor may use a data conversion utility
including such as an algorithm to convert the angular velocities W x ' wY ' W z and first quaternion into the second quaternion.
This data conversion utility may be a program or instruction
represented by the following equation (1).

10

15

20

25

30

35

40

go

45

0

-Wx

-w y

-w,

g)

Wx

0

w,

-W y

q)

Wy

-w,

0

Wx

q2

W,

Wy

-Wx

0

q3

g2
g3

50

55

60

65

2:

qo

(1)

Equation (1) is a differential equation. The quaternion on the
left side of the equal sign is the first order derivative with
respect to time of the quaternion (qo, ql' q2' 'b) on the right
side of the equal sign. The data conversion utility uses the first
quaternion as the initial values for the differential equation (1)
and calculates the solution of the differential equation (1).
The second quaternion may be represented by a solution of
the differential equation (1).
As shown in the figure, the "measured state" of the nineaxis motion sensor module according to one embodiment of
the present invention may generally refer or may be represented by steps 725 and 730. In step 725, the second signal set
generated by the accelerometer may be obtained, which
includes measured axial accelerations Ax, Ay and Az; or Ax,
Ay and Az may refer to the measurement of the axial accelerations obtained. In order to obtain said measured state of the
nine-axis motion sensor of the present invention, according to
one embodiment, predicted axial accelerations Ax', Ay' and
Az' may too be calculated and obtained based on the above-

US 8,552,978 B2
17

18

mentioned current state of the nine-axis motion sensor module or the second quatemion as shown in step 730. In other
words, two sets of axial accelerations may be obtained for the
measured state of the nine-axis motion sensor module; one
may be the measured axial accelerations Ax, Ay, Az in step
725 and the other may be the predicted axial accelerations
Ax', Ay', Az' in step 730 calculated based on the abovementioned current state or second quaternion in relation to the
measured angular velocities thereof. Furthermore, in one
embodiment, the computing processor may use a data conversion utility to convert quaternion into the predicted axial
accelerations Ax', Ay' and Az'. This data conversion utility
may be a software program represented by the following
equations (2), (3) and (4).

Likewise, the measured state correlated to the abovementioned predicted axial accelerations and in relation to the axial
accelerations of accelerometers and current state may be
obtained based on an exemplary equation of:
z,(tlt-l)~h(x(tlt-l»

(8)

Preferably, a second probability (measurement probability)
associated with the measured state may be further obtained
based on an exemplary equation of:
10
(9)

Hx=

8h(x(tl t-l»
8x(tlt-l)

(10)

15
(2)

(3)

qo2_Q12_Q22+Q32=Az'

(4) 20

The computing processor calculates the solution (Ax', Ay',
Az') of the equations (2), (3) and (4).
According to an exemplary embodiment of the method for
obtaining a resultant deviation including deviation angles in a
spatial reference frame of an electronic device, including
such as a 3D pointing device, a portable electronic device, a
navigation equipment or a smartphone, utilizing a nine-axis
motion sensor module, it may be preferable to compare the
current state of the nine-axis motion sensor module with the
measured state thereof with respect to the present time frame
T by utilizing a comparison model. In other words, in one
embodiment as shown in step 735, it is preferable to compare
the second quaternion in relation to the measured angular
velocities of the current state at present time T with the
measured axial accelerations Ax, Ay, Az as well as the predicted axial accelerations Ax', Ay', Az' also at present time T.
Following which, a result may be advantageously obtained as
an updated state of the nine-axis motion sensor module,
excluding the abovementioned undesirable external interferences of the dynamic environments. In an explanatory
example, the updated state may generally refer to the update
of the current state of the nine-axis motion sensor module at
preset time T. Instructions including equations related to the
abovementioned current state, measured state and updated
state may be illustrated in the following.
According to an exemplary embodiment of the comparison
model utilized by the present invention in relation to step 735
as shown in the figure, the current state correlated to the
abovementioned second quaternion and in relation to the
angular velocities of gyroscope(s) may be obtained based on
an exemplary equation of:

D,~{[z,-h(x(tlt-l »]P(z,lx,)[z,-h(x(tlt-l)Jr 1}l!2

25

30

35

40

45

50

(5)

Preferably, a first probability (state transition probability)
associated with the said current state may be further obtained
based on an exemplary equation of:

55

P(x, I X,-l, u,) = FxP(X,-l I x'-lJF; + FuP(U,-l I u,_IlF~ + Q,
8 !(X,-l, u,)

60

Fx

=

(6)

8Xt-l

(7)

65

wherein Qt=additional motion model noise

wherein Rt=measurement model noise
As an illustrative example, the abovementioned first and second probabilities may be further utilized to obtain the updated
state of the nine-axis motion sensor module based on an
exemplary method of data association of an exemplary equation of:
(11)

In one embodiment, the result of the updated state of the
nine-axis motion sensor module, preferably involving comparison or data association represented by the equations, may
be a third quaternion as shown in the figure. Furthermore, the
result may then be further outputted and utilized to obtain a
resultant deviation, excluding undesirable interferences of
the dynamic environments under which the present invention
is subject to, but including deviation angles in a spatial reference frame in the following steps as shown in the figure. In a
preferred embodiment of the present invention, said undesirable external interferences may further comprise or refer to
undesirable axial accelerations caused by undesirable external forces other than a force of gravity; in another preferred
embodiment, said undesirable external interferences may further comprise or refer to undesirable magnetism caused by
undesirable electromagnetic fields. In other words, the
method and algorithm provided by the present invention may
preferably generate or provide an output of the resultant
deviation of the nine-axis motion sensor module excluding
the abovementioned undesirable interferences. In one
example, external forces exerted to cause axial accelerations
of a nine-axis motion sensor of an electronic device of the
present invention may be decoupled or separated from a force
of gravity; and in another example, the undesirable magnetism caused by such as electromagnetic fields external or
internal to an electronic device the present invention may be
excluded. It can be understood that the examples of current
state, measured state, state update, data association and probabilities of the comparison model and method of the present
invention recited herein are provided for illustrative purposes
only.
As mentioned previously, it may be preferable to output the
result of the updated state, preferably in a form of third
quaternion, to the previous state of the nine-axis motion sensor module as shown in step 740 in FIG.7. In a preferred
embodiment, the updated state may further comprise a first
data association model; and wherein the abovementioned and
related data association model may be provided for comparing the measured state associated with said second signal set
with a predicted measurement obtained from said current
state. In other words, in one embodiment, the first quaternion
may be replaced by the abovementioned third quaternion or
substitute directly any previous values of first quaternion in

US 8,552,978 B2
19

20

the previous time T for further process in a loop. In other
words, the third quaternion with respect to the present time T
becomes the first quaternion with respect to the next time such
as T + 1; or, the third quaternion at previous time frame T-l
outputted may now be the first quaternion at present time
frame T.
In step 745, the updated state of the nine-axis motion
sensor module of the present invention may be further calculated and converted to the resultant deviation including deviation angles associated with the spatial reference frame,
wherein the deviation angles includes the yaw angle, pitch
angle and roll angle of the electronic device associated with
the spatial reference frame, preferably about each of three
orthogonal coordinate axes of the spatial reference frame; and
whereby the resultant deviation comprising deviation angles
associated with the updated state of the nine-axis motion
module may be preferably obtained excluding said undesirable external interferences in the dynamic environments. In
an explanatory example, said undesirable external interferences may refer to or further comprise undesirable axial
accelerations caused by undesirable external forces other
than a force of gravity. In another explanatory example, said
undesirable external interferences may refer to or further
comprise undesirable magnetism caused by undesirable electromagnetic fields. In one embodiment, the computing processor may use a data conversion utility to convert the third
quaternion of the updated state of the nine-axis motion sensor
module into the yaw, pitch and roll angles thereof This data
conversion utility may be a program or instruction represented by the following equations (12), (13) and (14).

an electronic device according to this embodiment. For illustrative purposes, the difference between FIG. 7 and FIG. 8
may be represented by the additional mapping step 750 as
shown in FIG. 8. Steps 705-745 in FIG. 8 are the same as their
counterparts in FIG. 7, which perfonn the comparison process for the 3D pointing device. Step 750 perfonns the mapping process for the electronic device. The computing processor may include a mapping program that performs the
mapping step 750. At step 750, the processing and transmitting module may obtain display data including for example,
display screen size such as boundary infonnation, and translates the deviation angles of the resultant deviation associated
with the spatial reference frame, preferably about each of
three orthogonal coordinate axes of the spatial reference
frame, to a movement pattern in a mapping area in a display
reference frame based on a sensitivity input correlated to the
display reference frame. It can be understood that the abovementioned display data may too include or refer to the type of
display such as LED, LCD, touch panel or 3D display as well
as frequency rate of display such as 120 Hz or 240 Hz. In one
embodiment, the display reference frame associated with the
display to be mapped may be a 2D display reference frame; in
another embodiment, the display reference frame may be a
3D display reference frame of a 3D display.
The aforementioned display data may further include a
sensitivity input. The aforementioned sensitivity input is a
parameter which may be inputted and adjusted by a user
through control buttons attached on the housing of the electronic device. The sensitivity input may represent the sensitivity of the display device with respect to the movement of
the electronic device. For details of the mapping process,
please refer to FIG. 9. In one embodiment, the sensitivity
input is a parameter representing the relationship between the
display to be mapped with deviation to a movement pattern in
2D display reference frame and the electronic device of the
present invention outputted with said deviation including
yaw, pitch and roll angles in 3D reference frame; wherein the
relationship may be a distance relationship. In another
embodiment, the sensitivity input may be a display screen
size including boundary information predetermined by a
user; wherein the boundary infonnation may be obtained
based on a user input or manual input data from the user. In
still another embodiment, the sensitivity input may be predefined or preset in the mapping program such that the parameter of the sensitivity input is a preset value for either increase
or decrease the movement patterns including distance or
number of pixels to be moved or mapped from said resultant
deviation of the electronic device of the present invention.
FIG. 9 is a bird's-eye view of an electronic device 930
according to one embodiment of the present invention
directed to a display screen 910 of a display device. The
display screen has a central point 922, a target point 924 and
a boundary point 926. The central point 922 is the geometric
center of the display screen 910. The target point 924 is the
position that the electronic device 930 is aiming at. The
boundary point 926 is a point on the right boundary of the
display screen 910. The points 922, 924, 926 and the electronic device 930 are on a common plane parallel to both the
X D axis and the ZD axis of the display reference frame X D yDZD' Virtual beams 942, 944 and 946 are imaginary light
beams from the electronic device 930 to the central point 922,
the target point 924 and the boundary point 926, respectively.
The distance P is the distance between the central point 922
and the target point 924, while the distance P max is the distance between the central point 922 and the boundary point
926. The distance d is the distance between the central point
922 and the electronic device 930. The aforementioned yaw

10

15

20

25

30

(12)

35

pitch = arcsin(2(qoq2 - q3qIl)

(13)
(14)

40

The variables qo, ql' q2 andq3 in equations (12), (13) and (14)
are the four elements of the third quaternion.
For a looped method continuous with respect to time, in
one embodiment of the present invention, the method utilized
by for example the computing processor communicated with
the nine-axis motion sensor module may return to step 71 0 to
perform the comparison process or method with respect to the
next time T + 1. In addition, the abovementioned resultant
deviation including deviation angles comprising yaw, pitch
and roll angles in the spatial reference frame converted from
the third quaternion is preferably obtained and outputted in an
absolute manner reflecting or associating with the actual
movements and rotations of the electronic device of the
present invention in said spatial reference frame. It can be
understood that said actual movements and rotations of the
electronic device of the present invention in the spatial reference frame or 3D reference frame may refer to real-time
movements and rotations associated with vectors having both
magnitudes and directions along or about orthogonal axes in
the spatial reference frame under the dynamic environments.
FIG. 8 shows a flow chart illustrating a method of mapping
resultant deviation angles of an electronic device having
movements and rotations in a 3D spatial reference frame and
in a dynamic environment onto a display reference frame
according to another embodiment of the present invention.
FIG. 9 is a schematic diagram showing the aforementioned
mapping of the resultant angles of the resultant deviation of

45

50

55

60

65

US 8,552,978 B2

21

22

angle of the resultant deviation of the electronic device 930 of
the present invention is the angle 8 between the virtual beams
942 and 944, while the angle 8max is the angle between the
virtual beams 942 and 946. The aforementioned mapping
area is a plane including the display surface of the display
screen 910 in the display reference frame. The display surface
of the display screen 910 is a subset of the mapping area.
In this embodiment, the aforementioned sensitivity input is
provided by the user of the electronic device 930. The sensitivity ~ is defined by the following equation (15).

may be a lopped method. For a looped method continuous
with respect to time, in one embodiment, the method utilized
by for example the computing processor communicating with
the nine-axis motion sensor module may return to step 71 0 to
perfonn the comparison process or method with respect to the
next time T + 1, following which the comparison and mapping
process with respect to the next time frame may then be
perfonned.
FIG. 10 shows another embodiment of the comparison
method of the present invention. The flow chart illustrates a
method of obtaining resultant deviation including deviation
angles in a spatial reference frame of an electronic device
utilizing a nine-axis motion sensor module therein and subject to movements and rotations in dynamic environments in
the spatial reference frame and mapping resultant deviation of
the electronic device of the present invention having movements and rotations in a 3D spatial reference frame and in a
dynamic environment onto a display reference frame according to another embodiment of the present invention; and
whereby the resultant deviation comprising deviation angles
associated an output or state such as an updated state (details
below) of the nine-axis motion module may be preferably
obtained excluding said undesirable external interferences in
the dynamic environments. In an explanatory example, said
undesirable external interferences may refer to or further
comprise undesirable axial accelerations caused by undesirable external forces other than a force of gravity. In another
explanatory example, said undesirable external interferences
may refer to or further comprise undesirable magnetism
caused by undesirable electromagnetic fields. The steps
1005-1030 in FIG. 10 may make reference to the ones shown
in another embodiment of the present invention as shown in
FIG. 7.
For an electronic device, including such as a pointing
device, a navigation equipment, a smartphone or other portable electronic apparatus, utilizing a nine-axis motion sensor
module, the signals of the magnetometer of the motion sensor
module may be preferably be used to facilitate the obtaining
of the resultant deviation including deviation angles in 3D
reference and preferably in an absolute mauner. The third
signal set generated by the magnetometer may be obtained as
shown in step 1035 in FIG. 10, which includes measured
magnetism Mx, My and Mz. In one embodiment, the Mx, My
and Mz may refer to the measurement of the magnetism
obtained. In order to obtain said measured state of the nineaxis motion sensor, according to one embodiment of the
present invention, predicted magnetism Mx', My' and Mz'
may too be calculated and obtained based on the abovementioned current state of the nine-axis motion sensor module or
the second quaternion as shown in step 1040. In other words,
two sets of magnetism may be obtained for the measured state
of the nine-axis motion sensor module; one may be the measured magnetism Mx, My, Mz in step 1035 and the other may
be the predicted magnetism Mx', My', Mz' in step 1040 calculated based on the abovementioned current state or second
quaternion in relation to the measured angular velocities
thereof. Furthennore, in one embodiment, the computing
processor may use a data conversion utility to convert the
current state or second quaternion into predicted magnetism
Mx', My' and Mz' and vice versa. This data conversion utility
may be a software program represented by the following
equations (18), (19) and (20).

10

(15)

15

The variable ~ in equation (16) is the sensitivity input
defined by user.
The following equation (16) may be derived from equation
(15) and geometry.

20

(16)

25

The following equation (17) may be derived from equations (16).
30

P = f(8) = d xtan8 = Pm= xtan8
Pmax
tan- )
(

(17)

f3

35

In equation (17), the distance Pmax may be obtained from
the width of the display screen of the display data obtained at
step 750; the angle 8 is the yaw angle obtained at step 745; the
sensitivity input ~ is provided by the user. Therefore, the
computing processor of the electronic device 930 can calculate the distance P according to equation (17). Next, the
computing processor can easily obtain the horizontal coordinate of the target point 924 on the display screen 910 according to the distance P and the width of the display screen 910.
In addition, the computing processor can easily obtains the
vertical coordinate of the target point 924 on the display
screen 910 according to the pitch angle in a similar way.
The mapping process performed at step 750 may be exemplified by the process of translating the yaw angle and the
pitch angle o[[he resultant angles to the 2D coordinates o[[he
target point 924 on the display screen 910 discussed above.
Now the computing processor has the coordinates of the
target point 924 of the present time frame. The computing
processor subtracts the coordinates of the target point 924 of
the previous time frame from the coordinates of the target
point 924 of the present time frame. The result of the subtraction is the horizontal offset and the vertical offset of the target
point 924 in the present time frame. The horizontal and vertical offsets may be transmitted to the display device so that
the display device can track the position of the target point
924. The display device may display a cursor or some video
effect on the display screen 91 0 to highlight the position of the
target point 924. The cursor or video effect may exhibit a
movement pattern on the display screen 910 when the user
moves the electronic device 930 of the present invention.
Likewise, according to another embodiment of the present
invention, the comparison method of the present invention

40

45

50

55

60

(18)
65

(19)
(20)

US 8,552,978 B2
23

24

The variable A in equations (18), (19) and (20) is the dip angle
between the direction of the ambient magnetic field measured
by the magnetometer and a horizontal plane in the spatial
reference frame. The dip angle A may be measured or calculated through an initial calibration process of the electronic
device of the present invention and then be used as a parameter. The computing processor calculates the solution (Mx',
My', Mz') of the equations (18), (19) and (20).
According to an exemplary embodiment of the method for
obtaining a resultant deviation including deviation angles in a
spatial reference frame of an electronic device, including
such as a pointing device, a navigation equipment, a smartphone or other portable electronic apparatus, utilizing a nineaxis motion sensor module, it may be preferable to compare
the current state of the nine-axis motion sensor module with
the measured state thereof with respect to the present time
frame T by utilizing a comparison model. In other words, in
one embodiment as shown in step 1045, it is preferable to
compare the second quaternion in relation to the measured
angular velocities of the current state at present time T with
the measured axial accelerations Ax, Ay, Az, the predicted
axial accelerations Ax', Ay', Az', the measured magnetism
Mx, My, Mz, and the predicted magnetism Mx', My', Mz' also
at present time T. Following which, a result may be obtained
as an updated state of the nine-axis motion sensor module. In
general and in an explanatory example of the present invention, the updated state may generally refer to the update of the
previous state of the nine-axis motion sensor module at a
previous time T -1 with reference to the current state and/or
measured state thereof. The comparison model in step 1045
utilizes the measured axial accelerations Ax, Ay, Az and measured magnetism Mx, My, Mz, as well as the predicted axial
accelerations Ax', Ay', Az' and the predicted magnetism Mx',
My',Mz'.
In one embodiment, the result of the updated state of the
nine-axis motion sensor module, preferably involving comparison or data association represented by the equations associated to the comparison model, may be a third quaternion as
shown in the figure. Furthermore, as shown in steps
1050-1060, the result may then be further outputted and
utilized to obtain a resultant deviation including deviation
angles in a spatial reference frame in the steps as shown in the
figure. It can be understood that the examples of current state,
measured state, state update, data association and probabilities of the comparison model and method of the present
invention are provided for illustrative purposes only.
FIG. 11 shows a further exemplary embodiment of the
comparison method of the present invention. The flow chart
illustrates a method of obtaining resulting deviation including
resultant angles in a spatial reference frame of an electronic
device, including such as a pointing device, a navigation
equipment, a smartphone or other portable electronic device,
utilizing a nine-axis motion sensor module therein and subject to movements and rotations in dynamic environments in
the spatial reference frame and mapping resultant deviation
angles of said electronic device having movements and rotations in the 3D spatial reference frame and in a dynamic
environment onto a display reference frame according to
another embodiment of the present invention. Likewise, steps
1105-1130 may include obtaining a previous state and a
current state of the motion sensor module as well as a measured state related to the axial accelerations of the motion
sensor module. Additionally, in step 1135, it may be preferable to compare the current state of the nine-axis motion
sensor module with the measured state thereof with respect to
the present time frame T by utilizing a comparison model. In
other words, as shown in step 1135, it is preferable to compare

the second quaternion in relation to the measured angular
velocities of the current state at present time T with the
measured axial accelerations Ax, Ay, Az as well as the predicted axial accelerations Ax', Ay', Az' also at present time T.
Following which, a result may be obtained as the first updated
state of the nine-axis motion sensor module. In an explanatory example, the first updated state may generally refer to the
first update of the current state of the nine-axis motion sensor
module at preset time T. Furthermore, one of the technical
effects of the present invention may too be obtained or
achieved. In step 1135, one of the advantages or effects by
performing steps from 1105-1135 may be that the first
updated state or third quaternion as shown in FIG. 11 may be
advantageously obtained excluding undesirable axial accelerations caused by for example undesirable external forces
such as the ones decoupled from a force of gravity.
In one embodiment, the result of the first updated state of
the nine-axis motion sensor module, preferably involving
comparison or data association represented by the equations
associated to the comparison model, may be a third quaternion as shown in the figure. In addition, one of the technical
effects of the present invention may include the exclusion of
undesirable external interferences in the dynamic environment as previously mentioned; and wherein the undesirable
external interferences may refer to or further include undesirable axial accelerations caused by undesirable external
forces, preferably decoupled from a force of gravity, and/or
undesirable magnetism caused by for example undesirable
electromagnetic fields either adjacent to the motion sensor
module. As shown in step 1140 of FIG. 11, the first updated
state of the nine-axis motion sensor module of the present
invention may be further calculated and converted to a temporary pitch angle and a temporary roll angle based on the
third quaternion. The first updated state, as shown in the
figure, may be advantageously obtained such that or whereby
undesirable axial accelerations associated with said undesirable external interferences in the dynamic environments may
be preferably excluded; in an explanatory example, the first
updated state may be preferably obtained excluding the
abovementioned undesirable axial accelerations caused by
undesirable external forces, such as external forces decoupled
from a force of gravity. The third signal set generated by the
magnetometer may be obtained, which includes measured
magnetism Mx, My and Mz. The measured state of the nineaxis motion sensor module may be obtained by obtaining and
calculating a measured yaw angle gained from the motion
sensor signals of the nine-axis motion sensor module at the
current time T according to the following equation (21).

10

15

20

25

30

35

40

45

50
-Mycos(Tr) + Mzsin(Tr)
Ty = Mxcos(Tp) + Mysin(Tp)cos(Tr) + Mzsin(Tp)cos(Tr)

55

60

65

(21)

In equation (21), Ty is the measured yaw angle, Tp is the
temporary pitch angle and Tr is the temporary roll angle.
In order to obtain said measured state of the nine-axis
motion sensor, according to one embodiment of the present
invention, a predicted yaw angle may be calculated and
obtained based on the abovementioned first updated state of
the nine-axis motion sensor module or the third quaternion at
present time T as shown in step 1145. In other words, the
measured yaw angle in step 1140 and the predicted yaw angle
in step 1145 may be obtained for the measured state of the
nine-axis motion sensor module.
Furthermore, it may be preferable to compare the current
state of the nine-axis motion sensor module with the mea-

US 8,552,978 B2

25

26

sured state thereof with respect to the present time frame T by
utilizing a comparison model. In other words, as shown in
step 1150, it is preferable to compare the second quaternion in
relation to the measured angular velocities of the current state
at present time T with the measured axial accelerations Ax,
Ay, Az, the predicted axial accelerations Ax', Ay', Az', the
measured yaw angle and the predicted yaw angle also at
present time T. Following which, a result may be obtained as
the second updated state of the nine-axis motion sensor module. In an explanatory example, the second updated state may
generally refer to the second update of the current state of the
nine-axis motion sensor module at preset time T. The comparison model in step 1150 is very similar to the abovementioned comparison models. Related details are omitted here
for brevity. In one embodiment, the result of the second
updated state of the nine-axis motion sensor module may be
a fourth quatemion as shown in the figure. Furthermore, the
result may then be further outputted and utilized to obtain a
resulting deviation including resultant angles in a spatial reference frame in the following steps as shown in the figure. In
addition to the abovementioned technical effects of the
present invention in which undesirable axial accelerations of
undesirable external interferences in the dynamic environments may be advantageously excluded as a result of the first
updated state of the motion sensor module in step 1135,
another technical effect or merit may too be obtained along
with result of the second updated state of the motion sensor as
shown in step 1150 of FIG. 11. One of the advantages or
effects by performing steps such as from 1140-1150 may be
that the second updated state or fourth quaternion as shown in
FIG. 11 may be advantageously obtained excluding undesirable magnetism such as the ones caused by for example
undesirable external or internal electromagnetic fields adjacent to the motion sensor module in the dynamic environments of the present invention.
It may be preferable to output the result of the second
updated state, preferably in a form of the fourth quaternion, to
the previous state of the nine-axis motion sensor module as
shown in step 1155 in the figure. In other words, in one
embodiment, the first quaternion may be replaced by the
abovementioned fourth quaternion or substitute directly any
previous values of first quaternion in the previous time T for
further process in a loop. In other words, the fourth quaternion
with respect to the present time T becomes the first quaternion
with respect to the next time such as T + 1; or, the fourth
quaternion at previous time frame T -1 outputted may now be
the first quaternion at present time frame T.
In step 1160, the second updated state of the nine-axis
motion sensor module of the present invention may be further
calculated and converted to the resulting deviation including
resultant angles associated with the spatial reference frame,
wherein the resultant angles includes the yaw angle, pitch
angle and roll angle of the 3D pointing device associated with
the spatial reference frame, preferably about each of three
orthogonal coordinate axes of the spatial reference frame. In
addition, the second updated state, as shown in the figure, may
be advantageously obtained such that or whereby undesirable
magnetism associated with said undesirable external interferences in the dynamic environments may be preferably
excluded; in an explanatory example, the second updated
state may be preferably obtained excluding the abovementioned undesirable magnetism caused by for example undesirable electromagnetic fields, or magnetism other than the
planetary geomagnetism, adjacent or of a magnitude influencing magnetometer of the motion sensor module. The
resultant angles may be calculated according to equations
(12), (13) and (14), wherein the variables qo, ql' q2 and q3 in

equations (12), (13) and (14) are the four elements of the
fourth quaternion. Furthermore, the resultant deviation in
step 1160 may be advantageously obtained excluding undesirable interferences including such as the ones of undesirable
axial accelerations caused by undesirable external forces
decoupled from a force of gravity mentioned previously in
step 1135 and the ones of undesirable magnetism caused by
for example undesirable electromagnetic fields as mentioned
previously in step 1150. Likewise, in step 1165 as shown in
FIG. 11, the resultant deviation including deviation angles in
3D pointer reference may be further mapped to a display
reference such as a 2D display reference of a display.
As illustrated by FIG. 12, in one preferred embodiment, the
first and second updated states may further comprise a first
data association model and a second data association model
respectively. The first data association model may be advantageously provided for comparing the first measured state
associated with said second signal set with a first predicted
measurement obtained from said current state; in addition, the
second data association model may too be advantageously
provided for comparing the second measured state associated
with said third signal set with a second predicted measurement obtained from said first updated state. Furthermore, in
another preferred embodiment, the first and second updated
states may further comprise a first data association model and
a second data association model respectively; and wherein the
first data association model may be advantageously provided
for comparing the first measured state associated with said
second signal set with a first predicted measurement obtained
from said current state; and wherein the second data association model may be advantageously provided for comparing
the second measured state associated with said third signal set
with a second predicted measurement obtained from said
current state. Details on the differences of the obtaining of
said second predicted measurement based on either the first
updated state or the current state of the motion sensor module
depending upon a comparison result are further described in
FIG. 12, as the routes presented or denoted by "Yes" and "No"
shown therein.
FIG. 12 shows an exemplary flow chart of another embodiment of the present invention of a method for obtaining a
resultant deviation comprising deviation angles of an electronic device, including such as a pointing device, navigation
equipment, a smartphone or other portable electronic apparatus. Accordingly, the method for obtaining a resultant
deviation including deviation angles in a spatial reference
frame of an electronic device utilizing a nine-axis motion
sensor module therein and subject to movements and rotations in dynamic environments in said spatial reference frame
includes the following steps. As shown in the figure, at step
1210, a previous state of the nine-axis motion sensor module
may be obtained; and wherein the previous state is associated
with at least previous angular velocities gained from the
motion sensor signals of the nine-axis motion sensor module
at a previous time T -1. In another embodiment, the previous
state is associated with previous angular velocities, previous
axial accelerations and previous magnetism gained from the
motion sensor signals of the nine-axis motion sensor module
at a previous time T -1. Next, at step 1220, it may be to obtain
a current state of the nine-axis motion sensor module by
obtaining measured angular velocities Cllx ' CllY ' Cllz gained from
the motion sensor signals of the nine-axis motion sensor
module at a current time T. At step 1225, it may be to obtain
a first measured state of the nine-axis motion sensor module
by obtaining measured axial accelerations Ax, Ay, Az gained
from the motion sensor signals of the nine-axis motion sensor
module at the current time T. Following which and at step

10

15

20

25

30

35

40

45

50

55

60

65

US 8,552,978 B2
27

28

1230 is calculating and obtaining a first predicted measurement of the nine-axis motion sensor module based on said
current state thereof. At step 1235, a comparison may be
performed to determine whether the signals related to the
measured state including such as measured axial accelerations and/or measured magnetism are "good enough" to be
used to compensate the current state of the motion sensor
module and therefore an updated state thereof can be
obtained.
According to the previously mentioned objectives of the
present invention, it is preferable to provide an advantageous
comparison or compensation method capable of outputting
resultant deviation of a motion sensor module of a relatively
high accuracy in the presence of external or internal interferences including such as electromagnetic fields generated by
other electronic components adjacent to the motion sensor
modules or of a magnitude strong enough to distort or affect
the normal operations or signals of motion sensor module.
Under such circumstances, a comparison utilizing data association may be advantageously provided or used to compare
measured state of the motion sensor modules with an
expected or predicted measurement thereof to determine the
compensation for updating an updated state of the previous
state. In step 1235 as previously mentioned, the data association may also include a predetermined value preset or preselected in accordance with for example the performance of
motion sensor module utilized, and such that the comparison
result of the measured state and the predicted measurement
may make reference to the data association and the predetermined value or range to determine the compensation needed
to take place to update the state of the motion sensor module
including such as the previous and/or current states thereof.
Accordingly, updated state(s) of the motion sensor module
may be obtained based on the result of the data association( s).
As shown in the figure, if the result of abovementioned comparison falls within the predetermined result of for example a
predetermined value or range of the data association, then in
one embodiment of the present invention, in step 1240, a first
updated state of the nine-axis motion sensor module may be
obtained based on a first comparison between said first predicted measurement and said first measured state of the nineaxis motion sensor module. Otherwise, if the result is not
within the predetermined value of range of the data association, then the first updated state may not be performed or
obtained. Such method of the use of data association and
comparison may be particularly useful in the abovementioned scenario of external or internal "interferences" such as
the ones caused by undesired electromagnetic fields. In the
case where the result falls outside of expected range, or
denoted by "No" as shown in FIG. 12, the next step would be
to obtain another measured state or the second measured state
of the motion sensor module to determine whether another
data association may be utilized to obtain the second updated
state. It can however be understood that the second updated
state may be provided as an additional step to the method of
the present invention. One may only perform the abovementioned steps and obtain a result of the first updated state based
on the measured state including the measured axial accelerations associated with the motion sensor module; in other
words, either performing steps to obtain only the first updated
state, steps to obtain only the second updated state, and/or
steps to obtain both the first and second updated states as
shown in the figure, shall all be considered to be within the
scope of the present invention and within the spirit of the
present invention. Furthermore and likewise, one of the technical effects of the present invention may too be obtained or
achieved. In step 1240, one of the advantages or effects by

performing steps from 1210-1240 may be that the first
updated state as shown in FIG. 12 may be advantageously
obtained excluding undesirable axial accelerations caused by
for example undesirable external forces such as the ones
decoupled from a force of gravity.
In another embodiment of the present invention or in the
case where the abovementioned second updated state is to be
obtained, one may further perform steps 1245-1260 as shown
in FIG. 12. In step 1245, one may obtain a second measured
state of the nine-axis motion sensor module by obtaining a
measured yaw angle based on measured magnetism Mx, My,
Mz gained from the motion sensor signals of the nine-axis
motion sensor module at the current time T. Furthermore, as
shown in step 1250, a second predicted measurement of the
nine-axis motion sensor module may be calculated and
obtained, following which a predicted yaw angle may too be
obtained based on said first updated state thereof depending
upon the result of the comparison such as the route denoted by
"Yes" in FIG. 12. In another embodiment, the predicted yaw
angle may be obtained based on said current state of the
motion sensor module depending upon the result of the comparison such as a result or route of "No" denoted in FIG. 12.
Once the measured state and the predicted measurements are
obtained and available, a second comparison may be performed to determine whether compensation may be carried
out based on the result of the comparison and the second data
association. As shown in step 1255, the second data association including a predetermined value or range may be performed to determine whether the comparison falls within said
predetermined value or range. If the result falls within said
value or range, then a second updated state may be obtained
and compensation may too take place as shown in step 1260
with the denotation of "Yes". Otherwise, if the result is not
within the predetermined value, then step 1265 shall be carried out, following the direction shown and denoted by "No"
in the figure; and in other words, compensation may utilize
the second predicted measurement of the motion sensor module thereof for updating instead of using the second measured
state thereof Likewise, in addition to the abovementioned
technical effects of the present invention in which undesirable
axial accelerations of undesirable external interferences in
the dynamic environments may be advantageously excluded
as a result of the first updated state of the motion sensor
module in step 1240, another technical effect or merit may too
be obtained along with result of the second updated state of
the motion sensor as shown in step 1260 of FIG. 12. One of
the advantages or effects by performing steps such as from
1245-1260 may be that the second updated state as shown in
FIG. 12 may be advantageously obtained excluding undesirable magnetism such as the ones caused by for example
undesirable external or internal electromagnetic fields adjacent to the motion sensor module in the dynamic environments of the present invention.
Following the above steps, in one embodiment of the
present invention in which said comparison method may be
provided in a continuous loop or a looped marmer with
respect to time, the result of the updated state at present time
T may then be outputted to the previous state at previous time
T -1 and become another beginning of the loop for the abovementioned steps to carry out again. The terminology of time
(s) T, T -1 or T + 1 shall be clear and apparent and shall too fall
within the scope and spirit of the present invention. For
example, in step 1260 as shown in FIG. 12, the second
updated state of the nine-axis motion sensor module may be
obtained by, or may too include, the updating said first
updated state thereofbased on a second comparison between
said second predicted measurement and said second mea-

10

15

20

25

30

35

40

45

50

55

60

65

US 8,552,978 B2

29

30

sured state of the nine-axis motion sensor module; and in step
1265, the result of the second updated state thereof may be
further outputted to the previous state in a looped manner with
respect to time.
After step 1265, the resultant deviation including the
deviation angles in the spatial reference frame, namely the
yaw, pitch and roll angles, may be obtained in step 1270 in a
similar way as those in steps 745, 1060 and 1160. Furthermore, the resultant deviation in step 1270 may be advantageously obtained excluding undesirable interferences including such as the ones of undesirable axial accelerations caused
by undesirable external forces decoupled from a force of
gravity mentioned previously in step 1240 and the ones of
undesirable magnetism caused by for example undesirable
electromagnetic fields as mentioned previously in step 1260.
As mentioned previously, in one embodiment of the
present invention, the method for obtaining a resultant deviation of an electronic device utilizing a nine-axis motion sensor module, data associations may be provided to obtain a
relatively accurate result nnder for example the existence of
external or internal interferences to the sensor module.
Accordingly, the abovementioned step of obtaining the first
updated state of the nine-axis motion sensor module may
further comprise perfonning a first data association to determine whether said first comparison between said first predicted measurement and said first measured state thereof falls
within a first predetermined value of the nine-axis motion
sensor module; and wherein the step of obtaining the second
updated state of the nine-axis motion sensor module may too
further comprise performing a second data association to
determine whether said second comparison between said second predicted measurement and said second measured state
thereof falls within a second predetennined value of the nineaxis motion sensor module.
Likewise, in accordance to the abovementioned continuous loop of the method of the present invention with respect to
time and in one embodiment, the method for obtaining a
resultant deviation of an electronic device utilizing a nineaxis motion sensor module may further comprise outputting
said second updated state of the nine-axis motion sensor
module to said previous state thereof; and wherein said previous state of the nine-axis motion sensor module may be a
first quaternion with respect to said previous time T -1; and
wherein said current state of the nine-axis motion sensor
module may be a second quaternion with respect to said
current time T; and wherein said first and second updated
states of the nine-axis motion sensor module may too be a
third and a fourth quaternion with respect to said current time
T respectively.
In summary, the present invention also provides a nine-axis
comparison method that compares the detected signals generated by and converted from the rotation of the electronic
device, utilizing a nine-axis motion sensor module, about all
of the three axes with the detected signals generated by and
converted from the acceleration of the device along all of the
three axes. In one embodiment, the nine-axis comparison
method may then output the resultant deviation including
yaw, pitch and roll angles in a spatial reference frame such as
a 3D reference frame of the device. In another embodiment,
the nine-axis comparison method may also include the mapping of the resultant deviation including yaw, pitch and roll
angles in the spatial reference to a display reference frame
such as a 2D display reference frame of a display screen of a
display device. The nine-axis comparison method involving
the comparison of different states of the motion sensor module and the utilization of data association of the present invention in order to output a resultant deviation having yaw, pitch

and roll angles in for example 3D reference frame is novel and
cannot be easily achieved by any know arts or their combinations thereof.
In view of the above, it is clear that such obtaining and
outputting of deviation including 3D angles in a spatial reference frame in an "absolute" manner of the present invention
is too novel, and the fact that the electronic device utilizing a
motion sensor module therein having a novel comparison
method and program of the present invention to obtain and
output such deviation in "absolute" manner cannot be easily
achieved by any known arts or their combination thereof. The
tenn "absolute" associated with the resulting deviation
including resultant angles such as yaw, pitch and roll in a
spatial reference frame or 3D reference frame obtained and
outputted by the device of the present invention may refer to
the "actual" movements and rotations of the 3D pointer
device of the present invention in said spatial reference frame.
Moreover, the nine-axis comparison method of the present
invention may accurately output said deviation including
angles in 3D reference frame as noises associated with the
nine-axis motion sensor module subject to movement and
rotations in dynamic environments and accumulated over
time may be effectively eliminated or compensated. Furthermore, the tenn "a", "an" or "one" recited herein as well as in
the claims hereafter may refer to and include the meaning of
"at least one" or "more than one". It can be understood that, as
previously mentioned, the term of "dynamic" recited herein
may refer to moving or subject to motions in general. It too
can be nnderstood that the tenn "excluding" recited herein to
describe the exclusion of undesirable interferences is provided for illustrative purposes and shall not be limited to a
certain or specific degree or magnitude of the effect of exclusion; any degree or magnitude associated thereto shall be
considered to be within the spirit and scope of the present
invention.
FIG. 13 is a flow chart of a method for compensating
rotations of a 3D pointing device according to an embodiment
of the present invention. The purpose of the method is transforming rotations and movements of the 3D pointing device
to a movement pattern in the display plane of a display device
(such as the plane X D YD of the display device 120 shown in
FIG. 1 and FIG. 2). The method may be executed by the 3D
pointing device shown in FIG. 14. FIG. 14 is a schematic
diagram showing a 3D pointing device according to an
embodiment of the present invention. The 3D pointing device
in FIG. 14 includes a rotation sensor 342, an orientation
sensor 1410, and a computing processor 1420. The orientation sensor 1410 includes an accelerometer 344 and another
computing processor 348.
The flow in FIG. 13 is discussed as follows. In step 1320,
the orientation sensor 1410 generates an orientation output
associated with the orientation of the 3D pointing device
associated with three coordinate axes of a global reference
frame associated with the Earth. The computing processor
348 of the orientation sensor 1410 may generate the aforementioned orientation output by executing steps 710 to 745
illustrated in FIG. 7 and FIG. 8. In brief, in steps 710 to 745
illustrated in FIG. 7 and FIG. 8, the rotation sensor 342
generates a rotation output (Cllx , CllY ' Cllz ) associated with the
rotation of the 3D pointing device associated with the three
coordinate axes of a spatial reference frame associated with
the 3D pointing device (such as the reference frame Xp Y pZp
shown in FIG. 1 and FIG. 2), the accelerometer 344 generates
a first signal set including axial accelerations Ax, Ay and Az
associated with the movements and rotations of the 3D pointing device in the spatial reference frame, and then the computing processor 348 generates the orientation output based

10

15

20

25

30

35

40

45

50

55

60

65

US 8,552,978 B2
31

32

on the first signal set and the rotation output. For more details,
please refer to the discussions related to FIG. 7 and FIG. 8
above.
The computing processor 348 may generate the aforementioned orientation output in the form of a rotation matrix, a
quaternion, a rotation vector, or in a fonn including the three
orientation angles yaw, pitch and roll. The orientation output
in the quaternion form may be the third quaternion generated
in step 740 in FIG. 7 and FIG. 8. The three orientation angles,
namely yaw, pitch and roll, may be generated in step 745 in
FIG. 7 and FIG. 8. The computing processor 348 may obtain
the rotation matrix from the orientation angles according to
the following equation (22).

pointing device may constitute a movement pattern in the
display plane and the display device may be controlled to
move a virtual object or a cursor along the movement pattern.
The step 1360 includes four sub-steps 1362, 1364, 1366
and 1368. In step 1362, the computing processor 1420 obtains
the orientation of the display device associated with the global reference frame associated with the Earth. For example,
the 3D pointing device may include a reset button. The user
may point the 3D pointing device at the display device and
press the reset button. The reset button may transmit a reset
signal to the computing processor 1420. The computing processor 1420 may record the current orientation output generated by the orientation sensor 1410 as the orientation of the
display device associated with the global reference frame
upon receiving the reset signal. The orientation of the display
device associated with the global reference frame associated
with the Earth may be recorded as a reset yaw angle.
In step 1364, the computing processor 1420 obtains the
orientation of the 3D pointing device associated with the fixed
reference frame associated with the display device based on
the orientation output and the orientation of the display device
associated with the global reference frame associated with the
Earth. As mentioned above, the current orientation output
recorded by the computing processor 1420 may include a
reset yaw angle associated with one of the three coordinate
axes (such as the Z axis) of the global reference frame associated with the Earth. The computing processor 1420 may
obtain the orientation of the 3D pointing device associated
with the fixed reference frame associated with the display
device by subtracting the reset yaw angle from the orientation
output.
Step 705 shown in FIG. 7 and FIG. 8 is equivalent to steps
1362 and 1364. In an alternative embodiment of the present
invention, in order to generate the orientation output in step
1320, the computing processor 348 may execute steps 705 to
745 shown in FIG. 7 and FIG. 8. In this case, the orientation
output generated by the orientation sensor 1410 represents
the orientation of the 3D pointing device associated with the
fixed reference frame associated with the display device.
Therefore, the computing processor 1420 skips steps 1362
and 1364 in the alternative embodiment.
In step 1366, the computing processor 1420 generates a
transfonned rotation associated with the fixed reference
frame associated with the display device based on the orientation of the 3D pointing device associated with the fixed
reference frame associated with the display device and the
rotation output. For example, the computing processor 1420
may generate the transfonned rotation according to the following equation (25).

10

15

cos8cosifJ
[Rhx3 =

cos8sinifJ
-sinO

sin8sin¢cosifJ -

sin8cos¢cosifJ +

cos¢sinifJ

sin¢sinifJ

sin8sin¢sinifJ +

(22)

sin8cos¢sinifJ -

cos¢cosifJ

sin¢cosifJ

cos8sin¢

cos8cos¢

20

[R]3X3 is the orientation output in the fonn of a rotation
matrix, 8 is the pitch angle, <p is the roll angle and 1jJ is the yaw
angle.
The computing processor 348 may also obtain the orientation output in the rotation matrix fonn from the quaternion
form according to the following equation (23), wherein the
quaternion fonn is represented as <eo, e u e 2 , e 3 >.

25

30

e6+ei-e~-e~
2(e) e2

+ eOe3)

2(e) e2

[Rhx3 =

+ eOe3)

2(e) e2 - eOe3)

e6 -ei +e~ -e~
2(e2e3

+ eOe)

2(e)e3 +eOe2)

(23)

2(e2e3 - eoell

e6 -ei -e~ +e~

35

Assume that the orientation output in the rotation vector
form is represented as <e 1 , e 2 , e3 > and the orientation output
in the quaternion form is represented as <eo, e 1 , e 2 , e 3 >. The
computing processor 348 may convert the rotation vector
form to the quaternion form and convert the quaternion fonn
to the rotation vector fonn according to the following equation (24).

40

45
(24)

One of the four forms of the orientation output may be converted to another of the four forms easily based on equations
(22), (23) and (24).
In step 1340, the rotation sensor 342 generates a rotation
output associated with the rotation of the 3D pointing device
associated with the three coordinate axes of a spatial reference frame associated with the 3D pointing device itself (such
as the reference frame Xp Y pZp shown in FIG. 1 and FIG. 2).
In step 1360, the computing processor 1420 uses the orientation output and the rotation output to generate a transformed
output <dx , dy > associated with the fixed reference frame
associated with the display device. The transfonned output
<dx , dy > represents a 2-dimensional movement in a display
plane in the fixed reference frame parallel to the screen of the
display device, such as the display plane X D YD of the display
device 120 shown in FIG. 1 and FIG. 2, whereindJepresents
the movement along the X D axis and dy represents the movement along the YD axis. In addition, the transformed output
<dx , dy > may represent a segment of movement in the display
plane. Multiple segments of movement plotted by the 3D

50
(25)

55

60

65

R ll -R 13 , R21-R23 and R31-R33 are the elements of the 3x3
rotation matrix obtained from the orientation of the 3 D pointing device associated with the fixed reference frame associated with the display device. [wx Wy wzb is the transformed
rotation associated with the fixed reference frame associated
with the display device. [wx Wy wz]p is the rotation output
generated by the rotation sensor 342. The transformed rotation [wx Wy wZ]D includes three angular velocities W x' wyand
W z respectively associated with the three coordinate axes X D ,
YD and ZD of the fixed reference frame associated with the
display device. The rotation output [wx Wy wz]p includes three

US 8,552,978 B2

33

34

angular velocities W x ' wyand W z respectively associated with
the three coordinate axes X p, Yp and Zp of the spatial reference frame associated with the 3D pointing device.
In step 1368, the computing processor 1420 generates the
transfonned output <dx ' dy > based on the transfonned rotation [wx Wy wzb. ~ is the first movement component of the
transfonned output associated with the coordinate axis X D of
the fixed reference frame associated with the display device,
while dy is the second movement component of the transformed output associated with the coordinate axis YD of the
fixed reference frame associated with the display device. For
example, the computing processor 1420 may multiply the
angular velocity Wy of the transfonned rotation by a predetermined scale factor to generate the second movement component dy and multiply the angular velocity W z of the transformed rotation by the same scale factor to generate the first
movement component dx ' The value of the scale factor may be
set by the user.
The method for compensating rotations of a 3D pointing
device shown in FIG. 13 may be executed by the 3D pointing
device shown in FIG. 15 as well. FIG. 15 is a schematic
diagram showing another 3D pointing device according to an
embodiment of the present invention. The 3 D pointing device
in FIG. 15 includes a rotation sensor 342, an orientation
sensor 1510, and a computing processor 1420. The orientation sensor 1510 includes an accelerometer 344, a magnetometer 345, and a computing processor 348.
In step 1320, the orientation sensor 1510 generates an
orientation output associated with the orientation of the 3D
pointing device associated with the three coordinate axes of
the global reference frame associated with the Earth. The
computing processor 348 of the orientation sensor 1510 may
generate the aforementioned orientation output by executing
steps 1010 to 1060 illustrated in FIG. 10, steps 1110 to 1160
illustrated in FIG. 11, or steps 1210 to 1270 illustrated in FIG.
12. In brief, in the aforementioned steps shown in FIG. 10,
FIG. 11 or FIG. 12, the rotation sensor 342 generates a rotation output (wx, Wy, wz) associated with the rotation of the 3D
pointing device associated with the three coordinate axes of
the spatial reference frame associated with the 3D pointing
device (such as the reference frame Xp Y pZp shown in FIG. 1
and FIG. 2), the accelerometer 344 generates a first signal set
including axial accelerations Ax, Ay and Az associated with
the movements and rotations of the 3D pointing device in the
spatial reference frame, the magnetometer 345 generates a
second signal set (Mx, My, Mz) associated with the magnetism of the Earth, and then the computing processor 348
generates the orientation output based on the first signal set,
the second signal set and the rotation output. For more details,
please refer to the discussions regarding FIG. 10, FIG. 11 and
FIG. 12 above.
The computing processor 348 may generate the aforementioned orientation output in the form of a rotation matrix, a
quaternion, a rotation vector, or in a fonn including the three
orientation angles yaw, pitch and roll. The orientation output
in the quaternion form may be the third quaternion generated
in step 1050 in FIG. 10, the fourth quaternion generated in
step 1155 in FIG. 11, or may be obtained from the updated
state in step 1265 in FIG. 12. The three orientation angles,
namely yaw, pitch and roll, may be generated in step 1060 in
FIG. 10, step 1160 in FIG. 11, or step 1270 in FIG. 12. The
rotation matrix form and the rotation vector form may be
obtained according to the aforementioned equations (22),
(23) and (24).
In this embodiment, the rotation sensor 342 in FIG. 15
executes step 1340 as the rotation sensor 342 in FIG. 14 does,

and the computing processor 1420 in FIG. 15 executes step
1360 as the computing processor 1420 in FIG. 14 does.

10

15

20

25

30

35

40

45

50

55

Step 1005 shown in FIG. 10 and step 1105 shown FIG. 11
are equivalent to steps 1362 and 1364. In an alternative
embodiment of the present invention, in order to generate the
orientation output in step 1320, the computing processor 348
in FIG. 15 may execute steps 1005 to 1060 shown in FIG. 10
or steps 1105 to 1160 shown in FIG. 8. In this case, the
orientation output generated by the orientation sensor 1510
represents the orientation of the 3D pointing device associated with the fixed reference frame associated with the display device. Therefore, the computing processor 1420 in FIG.
15 skips steps 1362 and 1364 in the alternative embodiment.
The method for compensating rotations of a 3D pointing
device shown in FIG. 13 may be executed by the 3D pointing
device shown in FIG. 16 as well. FIG. 16 is a schematic
diagram showing another 3D pointing device according to an
embodiment of the present invention. The 3D pointing device
in FIG. 16 includes a rotation sensor 342, an orientation
sensor 1610, and a computing processor 1420. The orientation sensor 1610 includes an accelerometer 344, a magnetometer 345, and a computing processor 348.
In step 1320, the orientation sensor 1610 generates an
orientation output associated with the orientation of the 3D
pointing device associated with the three coordinate axes of
the global reference frame associated with the Earth. In order
to generate the orientation output, the accelerometer 344 generates a first signal set including axial accelerations Ax, Ay
andAz associated with the movements and rotations of the 3 D
pointing device in the spatial reference frame associated with
the 3D pointing device itself, the magnetometer 345 generates a second signal set (Mx, My, Mz) associated with the
magnetism of the Earth, and then the computing processor
348 in FIG. 16 generates the orientation output based on the
first signal set and the second signal set. The details of the
generation of the orientation output are discussed below.
The orientation output may be provided in a form including
a yaw angle 1jJ, a pitch angle 8, and a roll angle <p respectively
associated with the three coordinate axes of the global reference frame associated with the Earth. The first signal set
includes axial accelerations Ax, Ay andAz respectively associated with the movements and rotations of the 3D pointing
device along the X p, Yp and Zp axes of the spatial reference
frame associated with the 3D pointing device itself The second signal set includes magnetism Mx, My, Mz associated
with the movements and rotations of the 3D pointing device
along each of the three orthogonal coordinate axes Xp Y pZp
of the spatial reference frame associated with the 3D pointing
device, respectively.
The computing processor 348 may calculate the pitch
angle 8 and the roll angle <p according to the following equations (26) and (27).

(26)

60

65

¢ = sin- 1 (

AYe) or ¢ = cos- 1 ( gcos
AXe)

(27)

gcos

In equations (26) and (27), Ax, Ay andAz are axial accelerations of the first signal set, while g is the gravitational acceleration. In addition, the computing processor 348 may calculate the yaw angle 1jJ according to the following equation (28).

US 8,552,978 B2

36

35
1/1 = tan

-l(

-Mycos¢ + Mzsin¢
)
. . .
M xcos8 + MYS1ll8s111¢ + M ZS1ll8cos¢

(28)

In equation (28), Mx, My and Mz are the aforementioned
elements of the second signal set. The computing processor
348 in FIG. 16 may generate the orientation output including
the yaw angle 1.jJ, the pitch angle 8, and the roll angle <p
according to equations (26), (27) and (28).
In this embodiment, the rotation sensor 342 in FIG. 16
executes step 1340 as the rotation sensor 342 in FIG. 14 does,
and the computing processor 1420 in FIG. 16 executes step
1360 as the computing processor 1420 in FIG. 14 does.
The method shown in FIG. 13 and the 3D pointing devices
shown in FIG. 14 to FIG. 16 can transform 3D rotations and
movements of the pointing devices into a 2D movement pattern in the display plane of the display device. The transformation of conventional pointing devices fails to take every
one of the yaw, pitch and roll angles into account and their
transformation is erroneous in some circumstances. For
example, a conventional pointing device may transform its
rotation to movement along an exactly opposite direction
when the conventional pointing device is flipped over. On the
other hand, the method shown in FIG. 13 and the 3D pointing
devices shown in FIG. 14 to FIG. 16 can transform the rotations and movements of the 3D pointing devices correctly no
matter how they are oriented or turned because the transformation of the embodiments of the present invention takes
every one of the yaw, pitch and roll angles into account.
It may be understood that the present invention may be
applied to various scenarios and application fields including
such as gaming, computers and navigation. It may too be
understood that the scope of the present invention shall be
determined by the accompanying claims and shall include
variations of applications of the present invention as well as
differences in the term definitions used or related to including
such as pointing devices, navigation equipment, smartphone
and/or electronic devices.
What is claimed is:
1. A 3D pointing device, comprising:
an orientation sensor, generating an orientation output
associated with an orientation of the 3D pointing device
associated with three coordinate axes of a global reference frame associated with Earth;
a rotation sensor, generating a rotation output associated
with a rotation of the 3D pointing device associated with
three coordinate axes of a spatial reference frame associated with the 3D pointing device; and
a first computing processor, using the orientation output
and the rotation output to generate a transformed output
associated with a fixed reference frame associated with a
display device,
wherein the orientation sensor comprises:
an accelerometer, generating a first signal set comprising
axial accelerations associated with movements and rotations of the 3D pointing device in the spatial reference
frame;
a magnetometer, generating a second signal set associated
with Earth's magnetism; and
a second computing processor, generating the orientation
output based on the first signal set, the second signal set
and the rotation output or based on the first signal set and
the second signal set;
wherein the rotation sensor, the accelerometer, and the
magnetometer forming a nine-axis motion sensor module; the 3D pointing device is configured for obtaining

10

15

20

25

30

35

40

45

50

55

60

65

one or more resultant deviation including a plurality of
deviation angles using a plurality of measured magnetisms Mx, My, Mz and a plurality of predicted magnetism
Mx', My' and Mz'.
2. The 3D pointing device of claim 1, wherein the orientation output comprises a yaw angle, a pitch angle, and a roll
angle associated with the three coordinate axes of the global
reference frame; the first signal set comprises a first axial
acceleration, a second axial acceleration, and a third axial
acceleration; the second computing processor calculates the
pitch angle based on the first axial acceleration, calculates the
roll angle based on the second axial acceleration and the pitch
angle or based on the third axial acceleration and the pitch
angle, and calculates the yaw angle based on the pitch angle,
the roll angle, and the second signal set.
3. The 3D pointing device of claim 1, wherein the orientation output provided by the orientation sensor is a rotation
matrix, a quaternion, a rotation vector, or comprises three
orientation angles.
4. The 3D pointing device of claim 1, wherein the transformed output represents a segment of a movement in a plane
in the fixed reference frame parallel to a screen of the display
device.
5. The 3D pointing device of claim 1, wherein the first
computing processor obtains an orientation of the display
device associated with the global reference frame, obtains an
orientation of the 3D pointing device associated with the fixed
reference frame based on the orientation output and the orientation of the display device associated with the global reference frame, generates a transformed rotation associated
with the fixed reference frame based on the orientation of the
3D pointing device associated with the fixed reference frame
and the rotation output, and generates the transformed output
based on the transformed rotation.
6. The 3D pointing device of claim 5, wherein when the
first computing processor receives a reset signal, the first
computing processor records a current orientation output
generated by the orientation sensor as the orientation of the
display device associated with the global reference frame.
7. The 3D pointing device of claim 6, wherein the current
orientation output comprises a yaw angle associated with one
of the three coordinate axes of the global reference frame, the
first computing processor obtains the orientation of the 3D
pointing device associated with the fixed reference frame by
subtracting the yaw angle from the orientation output.
8. The 3D pointing device of claim 5, wherein the first
computing processor obtains a rotation matrix from the orientation of the 3D pointing device associated with the fixed
reference frame, and multiplies the rotation matrix and the
rotation output together to generate the transformed rotation.
9. The 3D pointing device of claim 8, wherein the transformed rotation comprises a first angular velocity, a second
angular velocity, and a third angular velocity associated with
three coordinate axes of the fixed reference frame; the transformed output comprises a first movement component and a
second movement component associated with two of the
three coordinate axes of the fixed reference frame; the first
computing processor multiplies the second angular velocity
by a scale factor to generate the second movement component
and multiplies the third angular velocity by the scale factor to
generate the first movement component.
10. A method for compensating rotations of a 3 D pointing
device, comprising:
generating an orientation output associated with an orientation of the 3D pointing device associated with three
coordinate axes of a global reference frame associated
with Earth;

US 8,552,978 B2

37

38

generatinq a first signal set comprising axial accelerations
associated with movements and rotations of the 3D
pointing device in the spatial reference frame;
generating a second signal set associated with Earth's magnetism; generating the orientation output based on the
first signal set, the second signal set and the rotation
output or based on the first signal set and the second
signal set;
generating a rotation output associated with a rotation of
the 3D pointing device associated with three coordinate
axes of a spatial reference frame associated with the 3D
pointing device; and
using the orientation output and the rotation output to generate a transformed output associated with a fixed reference frame associated with a display device, wherein the
orientation output and the rotation output is generated by
a nine-axis motion sensor module; obtaining one or
more resultant deviation including a plurality of deviation angles using a plurality of measured magnetisms
Mx, My, Mz and a plurality of predicted magnetism Mx',
My' and Mz' for the second signal set.
11. ~he method of claim 10, wherein the orientation output
c?mpns~s a yaw angle, a pitch angle, and a roll angle assocmted wIth the three coordinate axes of the global reference
frame; the first signal set comprises a first axial acceleration
a second axial acceleration, and a third axial acceleration' th~
step of generating the orientation output based on the 'first
signal set and the second signal set comprises:
calculating the pitch angle based on the first axial acceleration;
calculating the roll angle based on the second axial acceleration and the pitch angle or based on the third axial
acceleration and the pitch angle; and
calculating the yaw angle based on the pitch angle, the roll
angle, and the second signal set.
12. The method of claim 10, wherein the orientation output
is a rotation matrix, a quaternion, a rotation vector or comprises three orientation angles.
'
13. The method of claim 10, wherein the transfonned output represents a segment of a movement in a plane in the fixed
reference frame parallel to a screen of the display device.
14. The method of claim 10, wherein the step ofgenerating
the transformed output comprises:
obtaining an orientation of the display device associated
with the global reference frame;

obtaining an orientation of the 3D pointing device associated with the fixed reference frame based on the orientation output and the orientation of the display device
asso~iated with the global reference frame;
generatmg a transfonned rotation associated with the fixed
reference frame based on the orientation of the 3D pointing device associated with the fixed reference frame and
the rotation output; and
generating the transformed output based on the transfonned rotation.
15. The method of claim 14, wherein the step of generating
the transfonned rotation comprises:
obtaining a rotation matrix from the orientation of the 3D
pointing device associated with the fixed reference
frame; and
multiplying the rotation matrix and the rotation output
together to generate the transformed rotation.
16. The method of claim 15, wherein the transformed rotation ~omprises a. first angular velocity, a second angular
velocIty, and a thIrd angular velocity associated with three
coordinate axes of the fixed reference frame; the transformed
output comprises a first movement component and a second
movement component associated with two of the three coordinate axes of the fixed reference frame; the step of generating
the transformed output based on the transformed rotation
comprises:
multiplying the second angular velocity by a scale factor to
generate the second movement component; and
multiplying the third angular velocity by the scale factor to
generate the first movement component.
17 ..The ~ethod of claim 14, wherein the step of obtaining
the onentatlOn of the display device associated with the global reference frame comprises:
recording a current orientation output as the orientation of
the display device associated with the global reference
frame in response to a reset signal.
18. The method of claim 17, wherein the current orientation output comprises a yaw angle associated with one of the
three coordinate axes of the global reference frame, and the
step ~f obtai~ing the orientation of the 3D pointing device
assocmted wIth the fixed reference frame comprises: obtaining the orientation of the 3D pointing device associated with
the fixed reference frame by subtracting the yaw angle from
the orientation output.

10

15

20

25

30

35

40

* * * * *