Cywee Group LTD v. Apple Inc.

Filing 1

COMPLAINT for Patent Infringement against Apple Inc. ( Filing fee $ 400, receipt number 0971-8556570.). Filed byCywee Group LTD. (Attachments: # 1 Exhibit A, # 2 Exhibit B, # 3 Civil Cover Sheet)(Kopeikin, Jill) (Filed on 4/22/2014)

EXHIBIT B

US00844143 8B2

(12) United States Patent
Ye et al.
(54)

(10) Patent No.:

US 8,441,438 B2

(45) Date of Patent:

May 14, 2013

3D POINTING DEVICE AND METHOD FOR

7,414,611 B2

8/2008 Liberty

COMPENSATING MOVEMENT THEREOF

7,489,298 B2
7,535,456 B2

2/2009 hbe?y er a1~
5/2009 Liberty et a1.

7,774,155 B2*

8/2010

.

(75) Inventors: Zhf’“ Ye’ F051“ C113” CA (Us);

345/158

7,924,264 132*
8,010,313 B2 *

ChlIl-Lllllg L1, Taoyuan County (TW);
Shun-Nan Liou, Kaohsiung (TW)

345/157
702/141

2008/0096654 A1*
'

.

-

-

(73)

Asslgnee- CyWee GFOIIP Llmlted, TOITOla (VG)

(*)

Notice:

Subject to any disclaimer, the term of this
patent is extended or adjusted under 35
U_S_C_ 154(1)) by 218 days_

Sato et a1. ................... .. 702/127

7,817,134 B2* 10/2010 Huang et a1.

2009/0262074 A1*

4/2011 Ohta ............ ..
8/2011 Mathews et a1.
4/2008 Mondesir et a1. ..
10/2009

Nasiri et a1.

..... ..

463/31
345/158

2011/0307173 A1,, 12/2011 Riley ““““““““““““““ “ 701/220

OTHER PUBLICATIONS

AZuma, Ronald et a1. Improving Static and Dynamic Registration in
an Optical See-Through HMD. Proceedings of SIGGRAPH ’94

21

APP 1. No.: 12/943 s 934

(Orlando, Fla., Jul. 24 29, 1994), Computer Graphics, Annual Con
ference Series, 1994, 197 204*

(22) Filed:

Nov. 11, 2010
* cited by examiner

(65)

Prior Publication Data
Us 2011/0163950 A1

Jul 7 2011

Primary Examiner * William Boddie

’

Assistant Examiner * Bryan E Earles

Related US. Application Data

(74) Attorney, Agent, or Firm * Ding Yu Tan

(60) l6’ro2\(/)i1s(i)onal application No. 61/292,558, ?led on Jan.
’

i

G09G 5/00
us CL

ABSTRACT

A three-dimensional (3D) pointing device capable of accu

(51) Int, Cl,
(52)

(57)

rately outputting a deviation including yaW, pitch and roll
(200601)

angles in a 3D reference frame and preferably in an absolute
manner is provided. Said 3D pointing device comprises a

USPC ........................................................ .. 345/156

six'axis motion Sensor module including a rotation Sensor

Field of Classi?cation Search ...................... .. None

andanaccelerometen andaprocessing andtransmimng mod‘

See application ?le for Complete Search history

(58)

ule. The six-axis motion sensor module generates a ?rst sig
nal set comprising angular velocities and a second signal set
comprising axial accelerations associated With said move
ments and rotations of the 3D pointing device in the 3D

(56)

References Cited

reference frame. The processing and transmitting module

U.S. PATENT DOCUMENTS
5,138,154 A
8/1992 Hotelling
5,440,326 A
8/1995 Quinn
5,898,421 A
4/1999 Quinn

utilizes a comparison method to compare the ?rst signal set
With the second signal set to obtain an updated state of the

6,993,451 B2*
6,061,611 A *

measured state thereof in order to output the resulting devia
tion in the 3D reference frame and preferably in an absolute

six-axis motion sensor module based on a current state and a

5/2000 Whitmorea1.
1/2006 Chang et

7,158,118 B2

1/2007 Liberty

7,236,156 B2
7,239,301 B2

6/2007 Liberty et a1.
7/2007 Liberty et a1.

7,262,760 B2

8/2007 Liberty

manner.

19 Claims, 7 Drawing Sheets
705

Initialize an initial-value
set

710

Obtain a previous state

715

(1st quatemion) at T-l
Obtain measured angular
velocities at T

720

Obtain a current state

(2nd quatemian) at ‘l
Obtain ‘measured axial

725

accelerations‘ of a

measured state at T

730

CGICUMG 'predicted axial
accelerations’ of a
measured state at T

Output 3rd quaternion
to is! quatemion
Obtain resultant

deviation including yaw,
pitch and roll angles
Obtain display data and
translate the resultant

angles to movement
pattern in the display
reference frame

Obtain an updated state

735

(3rd quaternion) by

comparing current state
with measured state

US. Patent

May 14, 2013

Sheet 1 017

2

US 8,441,438 B2

D

120

X0
122

Yo
XP
110

113

YP
112

111

ZP

FIG. 1 (RELATED ART)
Zn

120

X0
122
Y

YP

D

112

110

XP
113

ZP
111

FIG. 2 (RELATED ART)

US. Patent

May 14, 2013

Sheet 2 on

US 8,441,438 B2

310\

320)

FIG. 3

US. Patent

May 14, 2013

Sheet 3 of7

US 8,441,438 B2

Rotation
Data

Transmitting
Unit

FIG 0

4

Computing
Processor

US. Patent

May 14, 2013

Sheet 4 on

US 8,441,438 B2

620“

FIG. 6

US. Patent

May 14, 2013

Sheet 5 0f 7

r

7051

US 8,441,438 B2

\

lnitialize an initial-value

set
a

J

Obtain a previous state

7101

(1st quaternion) at T-l
L

J

F

\

Obtain measured
7151 velocities at T angular

720 1 Obtain a current state

Output 3rd quaternion

(2nd quaternion) at T

to 1st quaternion

740

,

Obtain "measured axial

Obtain resultant

measured state at T

deviation including yaw,
pitch and roll angles

accelerations of a

7251
L

J

W

‘\

Calculate "predicted axial
accelerations of a

measured state at T

Obtain an updated state

7351/ (3rd quaternion) by

comparing current state
with measured state

FIG. 7

745

US. Patent

705

May 14, 2013

Sheet 6 of7

US 8,441,438 B2

Initialize an initial-value

set

710

Obtain a previous state

(1st quaternion) at T-l J

f

\

715x Obtain measured angular
velocities at T
k

J

720x Obtain a current state

.

~

t(2nd quatermon) at T a

Output 3rd quaternion

to lst quaternion

1740

__—__J

fObtain "measured axial

Obtain resultant

725x accelerations” of a
measured state at T
O

J

730x Calculate "predicted axial
accelerations” of a
measured state at T

deviation including yaw, $745
pitch and roll angles
%
Obtain display data and
translate the resultant

k

j

angles to movement I750
pattern in the display
reference frame

Obtain an updated state

735x (3rd quaternion) by

comparing current state
with measured state

\

J

FIG. 8

US. Patent

May 14, 2013

Sheet 7 0f 7

Pmox

FIG. 9

US 8,441,438 B2

US 8,441,438 B2
1

2

3D POINTING DEVICE AND METHOD FOR
COMPENSATING MOVEMENT THEREOF

device 110 about theYP axis; the roll angle 113 may represent
the rotation of the pointing device 110 about the XP axis.
In a knoWn related art as shoWn in FIG. 1, When the yaW

angle 111 of the pointing device 110 changes, the aforemen

CROSS-REFERENCE TO RELATED
APPLICATION

tioned pointer on the screen 122 must move horizontally or in

a horizontal direction With reference to the ground in
response to the change of the yaW angle 111. FIG. 2 shoWs
What happens When the user rotates the pointing device 110
counterclockwise by a degree such as a 90-degree about the

This application claims priority bene?ts of Us. Patent
Provisional Application No. 61/292,558, ?led on Jan. 6,
2010. The entirety of the above-mentioned patent applica
tions is hereby incorporated by reference herein and made a
part of this speci?cation.

XP axis.

In another knoWn related art as shoWn in FIG. 2, When the

yaW angle 111 changes, the aforementioned pointer on the
screen 122 is expected to move vertically in response. The
change of the yaW angle 111 can be detected by a gyro-sensor

BACKGROUND OF THE INVENTION

Which detects the angular velocity 00x of the pointing device

1. Field of the Invention
The present invention generally relates to a three-dimen
sional (3D) pointing device utilizing a motion sensor module

110 about the XP axis. FIG. 1 and FIG. 2 shoW that the same

change of the yaW angle 111 may be mapped to different
movements of the point on the screen 122. Therefore, a proper

and method of compensating and mapping signals of the
motion sensor module subject to movements and rotations of

20

compensation mechanism for the orientation of the pointing
device 110 is required such that corresponding mapping of

said 3D pointing device. More particularly, the present inven

the pointer on the screen 122 of the display 120 may be

tion relates to a 3D pointing device utilizing a six-axis motion
sensor module With an enhanced comparison to calculate and
compensate accumulated errors associated With the motion
sensor module and to obtain actual resulting deviation angles

obtained correctly and desirably. The term compensation of
the prior arts by Liberty (US. Pat. No. 7,158,118, U.S. Pat.
No. 7,262,760 and Us. Pat. No. 7,414,611) refers to the
25

in spatial reference frame and under dynamic environments.
2. Description of the Related Art
FIG. 1 is a schematic diagram shoWing a user using a
handheld 3D pointing device 110 to point at a point on the
screen 122 of a 2D display device 120. If the pointing device

effects or extra rotations about the axis related to “roll”. The

30

110 emits a light beam, the point Would be the location Where
the light beam hits the screen 122. For example, the pointing
device 110 may be a mouse of a computer or a pad of a video

game console. The display device 120 may be a part of the
computer or the video game console. There are tWo reference

correction and compensation of signals subject to gravity

35

frames, such as the spatial pointer reference frame and the
display frame, associated With the pointing device 110 and
the display device 120, respectively. The ?rst reference frame
or spatial pointer reference frame associated With the pointing

term of “comparison” of the present invention may generally
refer to the calculating and obtaining of the actual deviation
angles of the 3D pointing device 110 With respect to the ?rst
reference frame or spatial pointing frame XPYPZP utilizing
signals generated by motion sensors While reducing or elimi
nating noises associated With said motion sensors; Whereas
the term mapping may refer to the calculating and translating
of said deviation angles in the sptatial pointing frame
XPYPZP onto the aforementioned pointer on the display
plane associated With the 2D display device 120 of a second
reference frame or display frame XDYDZD.

It is knoWn that a pointing device utilizing 5-axis motion

shoWn in FIG. 1. The second reference frame or display frame
associated With the display device 120 is de?ned by the coor

sensors, namely, Ax, Ay, Az, uuYand 002 may be compensated.
For example, U.S. Pat. No. 7,158,118 by Liberty, U.S. Pat.
No. 7,262,760 by Liberty and Us. Pat. No. 7,414,611 by
Liberty provide such pointing device having a 5-axis motion

dinate axes XD, YD and ZD as shoWn in FIG. 1. The screen 122

sensor and discloses a compensation using tWo gyro-sensors

device 110 is de?ned by the coordinate axes XP, YP and ZP as

ofthe display device 120 is a subset ofthe XDYD plane ofthe
reference frame XDYDZD associated With the display device
120. Therefore, the XDYD plane is also knoWn as the display
plane associated With the display device 120.

40

45

XPYPZP. The pointing device by Liberty utilizing a 5-axis
motion sensor may not output deviation angles of the pointing
device in, for example, a 3D reference frame; in other Words,

A user may perform control actions and movements utiliz

ing the pointing device for certain purposes including enter
tainment such as playing a video game, on the display device
120 through the aforementioned pointer on the screen 122.

50

due to due to the limitation of the 5-axis motion sensor of

accelerometers and gyro-sensors utilized therein, the point

ing device by Liberty cannot output deviation angles readily

For proper interaction With the use of the pointing device,
When the user moves the pointing device 110, the pointer on
the screen 122 is expected to move along With the orientation,

direction and distance travelled by the pointing device 110
and the display 120 shall display such movement of the

my and 002 to detect rotation about the Yp and Zp axes, and
accelerometers Ax, Ay and Az to detect the acceleration of the
pointing device along the three axes of the reference frame

55

in 3D reference frame but rather a 2D reference frame only
and the output of such device having 5-axis motion sensors is
a planar pattern in 2D reference frame only. In addition, it has

been found that the pointing device and compensation dis

pointer to a neW location on the screen 122 of the display 120.

closed therein cannot accurately or properly calculate or

The orientation of the pointing device 1 10 may be represented
by three deviation angles of the 3D pointing device 110 With
respect to the reference frame XPYPZP, namely, the yaW
angle 1 11, the pitch angle 1 12 and the roll angle 1 13. The yaW,
pitch and roll angles 111, 112, 113 may be best understood in
relation to the universal standard de?nition of spatial angles

obtain movements, angles and directions of the pointing
device While being subject to unexpected dynamic movement
60

and/or accelerations along With the direction of gravity. In
other Words, it has been found that dynamic actions or extra

accelerations including additional accelerations, in particular

related to commercial vehicles or transportation such as ships

and airplanes. Conventionally, the yaW angle 111 may repre
sent the rotation of the pointing device 110 about the Z1, axis;
the pitch angle 112 may represent the rotation of the pointing

during the obtaining of the signals generated by the motion
sensors, in particular, during unexpected drifting movements

65

the one acted upon the direction substantially parallel to or

along With the gravity imposed on the pointing device With

the compensation methods provided by Liberty, said pointing

US 8,441,438 B2
3

4

device by Liberty cannot properly or accurately output the
actual yaw, pitch and roll angles in the spatial reference frame
XPYPZP and following which, consequently, the mapping of

frame may too be further mapped or translated to a pattern
useful in a 2D reference frame.

the spatial angles onto any 2D display reference frame such as

SUMMARY OF THE INVENTION

XDYDZD may be greatly affected and erred. To be more

speci?c, as the 5-axis compensation by Liberty cannot detect

According to one aspect of an example embodiment of the
present invention, a 3D pointing device utilizing a six-axis
motion sensor module is provided. The 3D pointing device

or compensate rotation about the XP axis directly or accu
rately, the rotation about the XP axis has to be derived from the

gravitational acceleration detected by the accelerometer. Fur
thermore, the reading of the accelerometer may be accurate
only when the pointing device is static since due to the limi

comprises an accelerometer to measure or detect axial accel
erations Ax, AZ, Ay and a rotation sensor to measure or detect

tation on known accelerometers that these sensors may not

including resultant angles comprising yaw, pitch and roll

distinguish the gravitational acceleration from the accelera
tion of the forces including centrifugal forces or other types of
additional accelerations imposed or exerted by the user.
Furthermore, it has been found that known prior arts may

angles in a spatial pointer frame of the 3D pointing device
subject to movements and rotations in dynamic environments
may be obtained and such that said resulting deviation includ
ing said resultant angles may be obtained and outputted in an

only be able to output a “relative” movement pattern in a 2D
reference frame based on the result calculated from the sig
nals of motion sensors. For example, the abovementioned

absolute manner re?ecting or associating with the actual
movements and rotations of the 3D pointer device of the

angular velocities 00x, my, 002 such that resulting deviation

in a relative manner and a pointer on a display screen to show

present invention in said spatial pointer reference frame.
According to another aspect of the present invention, the
present invention provides an enhanced comparison method

such corresponding 2D relative movement pattern. To be

to eliminate the accumulated errors as well as noises over

more speci?c, the pointer moves from a ?rst location to a

time associated with signals generated by a combination of
motion sensors, including the ones generated by accelerom
eters Ax, Ay, AZ and the ones generated by gyroscopes 00x, my,
002 in dynamic environments. In other words, accumulated

prior arts by Liberty may only output a 2D movement pattern

second new location relative to said ?rst location only. Such
relative movement from the previous location to the next
location with respect to time cannot accurately determine
and/ or output the next location, particularly in situations
where the previous location may have been an erred location

20

25

or have been faultily determined as an incorrect reference

point for the next location that is to be calculated therefrom
and obtained based on their relative relationship adapted. One

30

errors associated with a fusion of signals from a motions
sensor module comprising a plurality of motion sensors to
detect movements on and rotations about different axes of a
reference frame may be eliminated or corrected.

illustration of such defect of known prior arts adapting a
relative relationship in obtaining a movement pattern may be

According to still another aspect of the present invention,
the present invention provides an enhanced comparison
method to correctly calculating and outputting a resulting

clearly illustrated by an example showing the faultily output

deviation comprising a set of resultant angles including yaw,

ted movements of a pointer intended to move out of a bound 35 pitch and roll angles in a spatial pointer frame, preferably
about each of three orthogonal coordinate axes of the spatial
ary or an edge of display screen. It has been found that as the

pointer reference frame, by comparing signals of rotation

pointer of known prior arts reaches the edge of a display and
continues to move out of the boundary or edge at a certain

sensor related to angular velocities or rates with the ones of

extra extent beyond said boundary, the pointer fails to dem

accelerometer related to axial accelerations such that these

onstrate a correct or “absolute” pattern as it moves to a new 40

angles may be accurately outputted and obtained, which may

location either within the display or remaining outside of the
boundary; in other words, instead of returning to a new loca
tion by taking into account said certain extra extend beyond
the boundary made earlier in an “absolute” manner, the
pointer of known arts discards such virtual distance of the
extra extend beyond the boundary already made and an erred
next position is faultily outputted due to the relative relation
ship adapted and utiliZed by the pointer. may be never calcu
lated or processed due to the faultily obtained location at the
edge or boundary of the display as well as the relative rela
tionship adapted to obtain its next location therefrom.
Therefore, it is clear that an improved pointing device with
enhanced calculating or comparison method capable of accu

too be further mapping to another reference frame different

from said spatial pointer frame.
According to still another aspect of the present invention,
45

reference frame onto a display frame such that a movement

pattern in a display frame different from the spatial pointer
50

reference frame may be obtained according to the mapping or
translation of the resultant angles of the resultant deviation
onto said movement pattern.

According to another example embodiment of the present

rately obtaining and calculating actual deviation angles in the
spatial pointer frame as well as mapping of such angles onto
a pointer on the display frame in dynamic environments and
conditions is needed. In addition, as the trend of 3D technol
ogy advances and is applicable to various ?elds including
displays and interactive systems, there is a signi?cant need for
a 3D pointing device capable of accurately outputting a devia
tion of such device readily useful in a 3D or spatial reference

the present invention provides a mapping of the abovemen
tioned resultant angles, preferably about each of three
orthogonal coordinate axes of the spatial pointer reference
frame, including yaw, pitch and roll angles in a spatial pointer

invention, a 3D pointing device utiliZing a six-axis motion
55

sensor module with an enhanced comparison method for
eliminating accumulated errors of said six-axis motion sensor
module to obtain deviation angles corresponding to move

ments and rotations of said 3D pointing device in a spatial
60

pointer reference frame is provided. The 3D pointing device
and the comparison method provided by the present invention
by comparing signals from the abovementioned six-axis
motion sensor module capable of detecting rotation rates or

frame. Furthermore, there is a need to provide an enhanced

comparison method applicable to the processing of signals of

angular velocities of the 3D pointing device about all of the

motion sensors such that errors and/ or noises associated with

XP, YP and ZP axes as well as axial accelerations of the 3D
pointing device along all of the XP, YP and ZP axes. In other

such signals or fusion of signals from the motions sensors
may be corrected or eliminated. In addition, according to the
?eld of application, such output of deviation in 3D reference

65

words, the present invention is capable of accurately output
ting the abovementioned deviation angles including yaw,

US 8,441,438 B2
5

6

pitch and roll angles in a 3D spatial pointer reference frame of

According to another embodiment of the present invention,
a method for obtaining a resulting deviation including result

the 3D pointing device to eliminate or reduce accumulated
errors and noises generated over time in a dynamic environ
ment including conditions such as being subject to a combi

ant angles in a spatial pointer reference frame of a three

dimensional (3D) pointing device utiliZing a six-axis motion

nation of continuous movements, rotations, external gravity

sensor module therein and subject to movements and rota

forces and additional extra accelerations in multiple direc

tions in dynamic environments in said spatial pointer refer

tions or movement and rotations that are continuously non

ence frame is provided. Said method comprises the steps of:
obtaining a previous state associated With previous angular
velocities 00x, my, 002 gained from the motion sensor signals of

linear With respect to time; and furthermore, based on the

deviation angles being compensated and accurately outputted
in 3D spatial pointer reference frame may be further mapped

the six-axis motion sensor module at a previous time T-l;
obtaining a current state of the six-axis motion sensor module

onto or translated into another reference frame such as the

abovementioned display frame, for example a reference in

by obtaining measured angular velocities 00x, my, 002 gained

tWo-dimension (2D).

from the motion sensor signals at a current time T; obtaining
a measured state of the six-axis motion sensor module by

According to another example embodiment of the present
invention, a 3D pointing device utiliZing a six-axis motion

obtaining measured axial accelerations Ax, Ay, AZ gained

sensor module is provided; Wherein the six-axis motion sen
sor module of the 3D pointing device comprises at least one
gyroscope and at least one accelerometer. In one preferred

from the motion sensor signals at the current time T and

calculating predicted axial accelerations Ax', Ay', AZ' based

embodiment of the present invention, the six-axis motion
sensor module comprises a rotation sensor capable of detect

20

ing and generating angular velocities of 00x, my, 002 and an

accelerometer capable of detecting and generating axial
accelerations of Ax, Ay, AZ. It can be understood that in

another preferred embodiment, the abovementioned rotation
sensor may comprise three gyroscopes corresponding to each
of the said angular velocities of 00x, my, 002 in a 3D spatial

25

According to another aspect of the present invention, a
method for mapping deviation angles associated With move
30

the 3D pointing device. The rotation sensor detects the rota

tion of the 3D pointing device With respect to a reference
frame associated With the 3D pointing device and provides a

angles including yaW, pitch and roll angles in a spatial pointer
35

reference frame to an pointing object, such as a pointer, hav

ing movements in a display frame, preferably a 2D reference

frame, comprises the steps of obtaining boundary informa
tion of the display frame by calculating a prede?ned sensitiv
ity associated With the display frame and performing angle

the reference frame, namely, Xp, Yp and Zp of the 3D spatial
pointer frame. The accelerometer detects the axial accelera
tions of the 3D pointing device With respect to the spatial

ments and rotations of a 3D pointing device in a spatial
pointer reference frame onto a display frame of a display
having a predetermined screen siZe is provided. In one

embodiment, the method for mapping or translating deviation

rotation rate or angular velocity output. The angular velocity
output includes three components corresponding to the rota
tion rate or angular velocities 00x, my, 002 of the 3D pointing
device about the ?rst axis, the second axis and the third axis of

module to said resulting deviation comprising said resultant
angles in said spatial pointer reference frame of the 3D point

ing device.

pointer reference frame of the 3D pointing device; Whereas
the abovementioned accelerometer may comprise three
accelerometers corresponding to each of the said axial accel
erations Ax, Ay, AZ in a 3D spatial pointer reference frame of

on the measured angular velocities 00x, my, 002 of the current
state; obtaining an updated state of the six-axis motion sensor
module by comparing the current state With the measured
state of the six-axis motion sensor module; and calculating
and converting the updated state of the six axis motion sensor

40

and distance translation in the display frame based on said

deviation angles and boundary information.

pointer reference frame such as a 3D-pointer reference frame
and provides an acceleration output. The acceleration output

BRIEF DESCRIPTION OF THE DRAWINGS

includes three components corresponding to the accelera

tions, Ax, AZ, Ay of the 3D pointing device along the ?rst axis,
the second axis and the third axis of the reference frame,

45

The accompanying draWings are included to provide a
further understanding of the invention, and are incorporated
in and constitute a part of this speci?cation. The draWings
illustrate embodiments of the invention and, together With the
description, serve to explain the principles of the invention.

50

FIG. 1 shoWs a knoWn related art having a 5-axis motion
sensor in 2D reference frame.

namely, Xp,Yp and Zp of the 3D spatial pointer frame. It can,
hoWever, be understood that the axes of Xp, Yp and Zp of the
3D spatial pointer frame may too be represented simply by the
denotation of X, Y and Z.

According to another example embodiment of the present
invention, a method for compensating accumulated errors of
signals of the abovementioned six-axis motion sensor module

FIG. 2 shoWs the knoWn related art having a 5-axis motion
sensor as shoWn in FIG. 1 being rotated or rolled about Xp

in dynamic environments associated in a spatial pointer ref
erence frame is provided. In one embodiment, the method
may be performed or handled by a hardWare processor. The
processor is capable of compensating the accumulated errors
associated With the resultant deviation in relation to the sig
nals of the above-mentioned six-axis motion sensor module
of the 3D pointing device subject to movements and rotations
in a spatial pointer reference frame and in a dynamic envi

axis and is subject to further dynamic interactions or environ
55

FIG. 3 is an exploded diagram shoWing a 3D pointing
device utiliZing a six-axis motion sensor module according to
one embodiment of the present invention in a 3D spatial
60

ronment by performing a data comparison to compare signals
of rotation sensor related to angular velocities With the ones of
accelerometer related to axial accelerations such that the
resultant deviation corresponding to the movements and rota

tions of the 3D pointing device in the 3D spatial pointer frame
may be obtained accurately over time in the dynamic envi
ronments.

ment.

65

pointer reference frame.
FIG. 4 is a schematic block diagram illustrating hardWare
components of a 3D pointing device according to one
embodiment of the present invention.
FIG. 5 is a schematic diagram shoWing a 3D pointing
device utiliZing a six-axis motion sensor module according to
anther embodiment of the present invention in a 3D spatial

pointer reference frame.

US 8,441,438 B2
8

7
FIG. 6 is an exploded diagram showing a 3D pointing

orthogonal coordinate axes XPYPZP of the spatial pointer

device utilizing a six-axis motion sensor module according to
anther embodiment of the present invention in a 3D spatial

reference frame. The angular velocities 00x, my and 002 are

pointer reference frame.
ing deviation angles of a 3D pointing device having move
ments and rotations in a 3D spatial pointer reference frame

tively. The accelerometer 344 detects and generates the sec
ond signal set including axial accelerations Ax, Ay, AZ asso
ciated With the movements and rotations of the 3D pointing
device 300 along each of the three orthogonal coordinate axes

and in a dynamic environment according to an embodiment of

XPYPZP of the spatial pointer reference frame. The axial

the present invention.

accelerations Ax, Ay and AZ are corresponding to the coordi
nate axes XP, YP and ZP respectively. The term “six-axis”
means the three angular velocities 00x, my, 002 and the three
axial accelerations Ax, Ay, AZ. It can therefore be understood

corresponding to the coordinate axes XP, YP and ZP respec

FIG. 7 is a How chart illustrating a method for compensat

FIG. 8 shoWs a How chart illustrating a method of mapping

deviation angles of a 3D pointing device having movements
and rotations in a 3D spatial pointer reference frame and in a
dynamic environment onto a display reference frame accord
ing to another embodiment of the present invention.
FIG. 9 is a schematic diagram shoWing the mapping of the
resultant angles of the resultant deviation of a 3D pointing
device according to an embodiment of the present invention.

that the abovementioned six axes of XPYPZP may not need to

be orthogonal in a speci?c orientation and they may be rotated
in different orientations; the present invention discloses such
coordinate system for illustrative purposes only and any coor
dinates in different orientation and/or denotations may too be

possible.
The data transmitting unit 346 is electrically connected to

DESCRIPTION OF THE EMBODIMENTS
20

Reference Will noW be made in detail to the present

embodiments of the invention, examples of Which are illus

trated in the accompanying draWings. Wherever possible, the
same reference numbers are used in the draWings and the
description to refer to the same or like parts.

25

FIG. 3 is an exploded diagram shoWing a 3D pointing
device 300 according to an embodiment of the present inven
tion. The 3D pointing device 300 is subject to movements and
rotations in dynamic environments in a 3D spatial pointer
reference frame. The spatial pointer reference frame is analo

30

gous to the reference frame XPYPZP in FIG. 1 and FIG. 2. The

movements and rotations of the 3D pointing device 300 in the

aforementioned dynamic environments in the spatial pointer
reference frame may be continuously nonlinear With respect
to time.
The 3D pointing device 300 includes a top cover 310, a

35

spatial pointer reference frame, of the resulting deviation of
the six-axis motion sensor module 302 of the 3D pointing
device 300 is obtained under the aforementioned dynamic
environments and such that it is preferably obtained and out
putted in an ab solute manner re?ecting or associating With the
actual movements and rotations of the 3D pointer device of

45

the present invention in said spatial pointer reference frame.
In addition, said comparison utiliZed by the computing pro

ting unit 346, and the computing processor 348 may be all

having a longitudinal side con?gured to be substantially par
allel to the longitudinal surface of the housing 330. An addi
tional battery pack 322 provides electrical poWer for the
entire 3D pointing device 300.

50

accelerations Ax, Ay, AZ. The abovementioned measured
state may include a measurement of said second signal set or

measured Ax, Ay, AZ and a predicted measurement ofAx', Ay'
55

device 300 includes a six-axis motion sensor module 302 and

a processing and transmitting module 304. The six-axis
60

provided in the later content.
In this embodiment, the computing processor 348 of the
processing and transmitting module 304 further includes a

resulting deviation in the spatial pointer reference frame to a
movement pattern in a display reference frame different from

the spatial pointer reference frame. The display reference

The rotation sensor 342 of the six-motion sensor module

velocities 00x, my, 002 associated With the movements and rota
tions of the 3D pointing device 300 about each of three

and AZ' obtained based on or calculated from the ?rst signal
set. Details of different states of the six-axis motion sensor
module of the 3D pointing device of the present invention are

mapping program for translating the resultant angles of the

and the accelerometer 344. The processing and transmitting
module 304 includes the data transmitting unit 346 and the
computing processor 348.

302 detects and generates the ?rst signal set including angular

cessor 348 may further comprise an update program to obtain
an updated state of the six-axis motion sensor module based
on a previous state associated With a ?rst signal set in relation
to the angular velocities cox, coy, oZ and a measured state

associated With said second signal set in relation to the axial

FIG. 4 is a schematic block diagram illustrating hardWare
components of the 3D pointing device 300. The 3D pointing

motion sensor module 302 includes the rotation sensor 342

FIG. 1 and FIG. 2. In order to calculate the resulting devia
tion, the computing processor 348 utiliZes a comparison to
eliminate accumulated errors of the ?rst and second signal
sets of the six-axis motion sensor module 302, Whereby the

40

one embodiment, the housing 330 may comprise the top

attached to the PCB 340. The PCB 340 is enclosed by the
housing 330. The PCB 340 includes at least one substrate

of the 3D pointing device 300 including three resultant angles
preferably about each of the three axes of the spatial pointer
reference frame. The resultant angles include the yaW angle
111, the pitch angle 112 and the roll angle 113 as shoWn in

erably about each of three orthogonal coordinate axes of the

cover 310 and the bottom cover 320. The housing 330 may

move and rotate in the spatial pointer reference frame accord
ing to user manipulation or any external forces in any direc
tion and/ or under the abovementioned dynamic environ
ments. As shoWn in the FIG. 3, in one embodiment, the
rotation sensor 342, the accelerometer 344, the data transmit

?rst and second signal sets. The data transmitting unit 346
transmits the ?rst and second signal sets of the six-axis
motion sensor module 302 to the computing processor 348
via electronic connections on the PCB 340. The computing
processor 348 receives and calculates the ?rst and second
signal sets from the data transmitting unit 346. The comput
ing processor 348 further communicates With the six-axis
motion sensor module 302 to calculate the resulting deviation

resultant angles in the spatial pointer reference frame, pref

printed circuit board (PCB) 340, a rotation sensor 342, an
accelerometer 344, a data transmitting unit 346, a computing
processor 348, a bottom cover 320, and a battery pack 322.
The top cover 310 may include a feW control buttons 312 for
a user to issue prede?ned commands for remote control. In

the six-axis motion sensor module 302 for transmitting the

65

frame is analogous to the reference frame XDYDZD in FIG. 1
and FIG. 2. The movement pattern may be displayed on a
screen of a 2D display device similar to the display device 120

US 8,441,438 B2
9

10

in FIG. 1 and FIG. 2. The mapping program translates the

ment, the computing processor 554 also performs mapping by
translating the resultant angles of the resulting deviation of
the 3D pointing device in the spatial pointer reference frame,

resultant angles, preferably about each of three orthogonal
coordinate axes of the spatial pointer reference frame to the
movement pattern according to a sensitivity input correlated
to the display reference frame.
FIG. 5 is a schematic diagram shoWing a 3D pointing

preferably about each of three orthogonal coordinate axes of
the spatial pointer reference frame, to a movement pattern in
a display reference frame associated With the notebook com
puter 580. The movement pattern may be displayed on the
screen 582 of the notebook computer 580.
FIG. 6 is an exploded diagram shoWing a 3D pointing

device 500 utiliZing a six-axis motion sensor module accord
ing to anther embodiment of the present invention in a 3D

spatial pointer reference frame. As shoWn in FIG. 5, the 3D
pointing device 500 may comprise tWo parts 560 and 570 in

device 600 utiliZing a six-axis motion sensor module accord
ing to anther embodiment of the present invention in a 3D

data communication With each other. In one embodiment, the
?rst part 560 includes a top cover (not shoWn), a PCB 540, a
six-axis motion sensor module 502 comprising a rotation
sensor 542 and an accelerometer 544, a data transmitting unit
546, a bottom cover 520, and a battery pack 522. The data

spatial pointer reference frame. The 3D pointing device 600
may further comprises a built-in display 682. In other Words,
the abovementioned display reference frame associated With
a display may need not to be external to the spatial pointer
reference frame in terms of the hardWare con?guration of the
present invention. In one embodiment, the 3D pointing device
600 comprises a bottom cover 620, a PCB 640, a battery pack

transmitting unit 546 transmits the ?rst signal set (00x, my, 002)
generated by the rotation sensor 542 of the six-motion sensor

module 502 and the second signal set (Ax, Ay, AZ) generated
by the accelerometer 544 of the six-motion sensor module
502 to the data receiving unit 552 of the second part 570 via
Wireless communication or connection including Wireless
local area netWork (WLAN) based on IEEE 802.1 1 standards
or BluetoothTM. It can be understood that in another embodi
ment, Wired communication or connection via a physical
cable or electrical Wires connecting the ?rst part 560 and the

622, a rotation sensor 642, an accelerometer 644, a data
20

682, and a top cover 610. LikeWise, in one embodiment, the

housing 630 may comprise the top and bottom covers 610,

25

620. A built-in display 682 may too be integrated on the
housing 630; the six-axis motion sensor module 602 may
comprise the rotation sensor 642 and the accelerometer 644.

30

The data transmitting unit 646 and the computing processor
648 may also be integrated as a processing and transmitting
module 604 of the 3D pointing device 600.
The computing processor 648 of the processing and trans
mitting module 604 may too perform the mapping of resultant

second part 570 may too be possible.
In one embodiment, the second part 570 may be an external
processing device to be adapted to another electronic com

puting apparatus or system such as a personal computer 580;
for instance, the second part 570 may be coupled or adapted

deviation from or in said spatial reference frame or 3D refer
ence frame to a display reference frame such as a 2D refer

to an laptop computer via a standard interface, such as the

universal serial bus (USB) interface depicted as shoWn in
FIG. 5. The ?rst part 560 and the second part 570 communi
cate via the data transmitting unit 546 and the data receiving

unit 552. As previously mentioned, the data transmitting unit

35

546 and the data receiving unit 552 may communicate
through Wireless connection or Wired connection. In other
Words, in terms of hardWare con?guration and data transmis
sion, in one embodiment of the present invention, the six-axis
motion sensor module 502 comprising the rotation sensor 542

transmitting unit 646, a computing processor 648, a display

ence frame by translating the resultant angles of the resulting
deviation of the 3D pointing device 600 in the spatial pointer
reference frame, preferably about each of three orthogonal
coordinate axes of the spatial pointer reference frame to a
movement pattern in a display reference frame associated

With the 3D pointing device 600 itself. The display 682 dis
plays the aforementioned movement pattern. The top cover
40

610 includes a transparent area 614 for the user to see the

and the accelerometer 544 may be disposed distally from the
processing unit or computing processor 554; the signals from

display 682.

the six-axis motion sensor module 502 may then be transmit

obtaining and/or outputting a resulting deviation including

FIG. 7 is an explanatory ?oW chart illustrating a method for

ted via the data transmitting units 546, 552 to the computing
processor 554 via Wired or Wireless communication including
for example IEEE 802.11 standards or BluetoothTM.
The second part 570 of the 3D pointing device 500 accord
ing to one embodiment of the present invention comprises the
data transmitting unit 552 and the processor 554. The data
transmitting unit 552 of the second part 570 may be in data
communication With the other data transmitting unit 546 dis

posed distally therefrom in the ?rst part 560 as previously
mentioned. The data transmitting unit 552 in the second part
570 receives the ?rst and second signal sets from the data
transmitting unit 546 in the ?rst part 560 and transmits the
?rst and second signal sets to the computing processor 554. In
one embodiment, the computing processor 554 performs the
aforementioned calculation as Well as comparison of signals.
In one embodiment, said comparison utiliZed by the comput
ing processor 554 may further comprise an update program to

45

spatial pointer reference frame and in dynamic environments
according to an embodiment of the present invention. The
method in FIG. 7 may be a program or comparison model to

be embedded or performed by the processing unit or comput
50

55

60

subject to movements and rotations in dynamic environments
in said spatial pointer reference frame is provided; and said
method may comprise the folloWing steps. First of all, as
shoWn in FIG. 7, different states including “previous state”,
“current state”, “measured state” and “update state” of the
six-axis motion sensor module may be provided to represent
a step or a set of steps utiliZed by the method for obtaining the

said second signal set. The measured state may further

set. The computing processor 554 is external to the housing of
the 3D pointing device as depicted in FIG. 5. In one embodi

ing processor 348, 554, 648 of the processing and transmit
ting module according to different embodiments of the
present invention recited herein for illustrative purposes.
Accordingly, in one embodiment of the present invention,
a method for obtaining a resulting deviation including result
ant angles in a spatial pointer reference frame of a 3D pointing
device utiliZing a six-axis motion sensor module therein and

obtain an updated state based on a previous state associated
With said ?rst signal set and a measured state associated With

include a measurement of said second signal set and a pre
dicted measurement obtained based on the ?rst second signal

resultant angles in a spatial pointer reference frame of a 3D
pointing device having movements and rotations in a 3D

resulting deviation in 3D reference frame, and preferably in
65

the above-mentioned absolute manner. In one exemplary

embodiment, the method comprises obtaining a previous
state of the six-axis motion sensor module (such as steps 705,

US 8,441 ,438 B2
11

12

710); and wherein the previous state includes an initial -value
set or a ?rst quatemion associated With previous angular
velocities 00x, my, 002 gained from the motion sensor signals of
the six-axis motion sensor module at a previous time T-l;

continuous loop betWeen a previous time frame T-l and a
present time frame T; details on the replacement of the ?rst
quatemion at T-l With the later outputted quatemion at T is to
be provided in the later content. It can be understood that one
may make reference to Euler Angles for dentition on quater

obtaining a current state of the six-axis motion sensor module

by obtaining measured angular velocities 00x, my, 002 gained

nion. Similarly, it can be easily comprehended that the above
mentioned previous time T-l and present time T may too be

from the motion sensor signals of the six-axis motion sensor

module at a current time T (such as steps 715, 720); obtaining
a measured state of the six-axis motion sensor module by

obtaining measured axial accelerations Ax, Ay, AZ gained
from the motion sensor signals of the six-axis motion sensor
module at the current time T (such as step 725) and calculat

ing predicted axial accelerations Ax', Ay', AZ' based on the
measured angular velocities 00x, my, 002 of the current state of
the six-axis motion sensor module (such as step 73 0); obtain
ing an updated state of the six-axis motion sensor module by
comparing the current state With the measured state of the
six-axis motion sensor module (such as step 735); and calcu

5

be performed simultaneously, such as the obtaining of signals

lating and converting the updated state of the six axis motion
sensor module to said resulting deviation comprising said

20

resultant angles in said spatial pointer reference frame of the
3D pointing device (745). In order to provide a continuous
loop, the result of the updated state of the six-axis motion
sensor module may preferably be outputted to the previous
state; in one embodiment, the updated state may be a quater
nion, namely third quaternion as shoWn in the ?gure, such that
it may be directly outputted to the abovementioned previous
state of another quaternion, namely the abovementioned ?rst
quaternion and as shoWn in the ?gure (such as step 740).
In addition, it can be understood that the abovementioned

25

30

from the six-axis motion sensor module may be performed
simultaneously instead of one after another. It can therefore
be understood that the steps recited herein are for illustrative
purposes only and any other sequential orders or simulta
neous steps are possible and shall be Within the scope of the

present invention. When step 710 is performed for the ?rst
time, the ?rst quaternion initialiZed in step 705 is obtained.
Otherwise, the ?rst quatemion used in the present time T is
generated in the previous time T-l. In other Words, the step
710 may generally refer to or represented by the abovemen
tioned “previous state” of the six-axis motion sensor module;
according to another embodiment, the previous state may
refer to the steps of 705 and 710.

comparison utiliZed by the processing and transmitting mod

The next may be to obtain the ?rst signal set generated by
the rotation sensor, Which includes the measured angular
velocities 00x, my and 002 as shoWn in step 715 according to an

ule and comprising the update program may too make refer
ence to said different states of the six-axis motion sensor

module as shoWn in FIGS. 7 and 8. As mentioned previously,
the update program may be utiliZed by the processor to obtain
the updated state of the six-axis motion sensor module based
on the previous state associated With a ?rst signal set in
relation to the angular velocities 00x, my, 002 and the measured
state associated With said second signal set in relation to the
axial accelerations Ax, Ay, AZ. The abovementioned mea
sured state may include a measurement of said second signal
set or measured Ax, Ay, AZ and a predicted measurement of
Ax', Ay' and AZ' obtained based on or calculated from the ?rst
signal set. Details of each of the abovementioned states of the
six-axis motion sensor module and the related steps of the

substitute by a present time T and a next time T+l respectively
and shall too fall Within the scope and spirit of the present
invention.
The ?rst quatemion With respect to the previous time T is
obtained as shoWn in the ?gure as step 710. The method
illustrated in FIG. 7 may be performed in consecutive time
frames. According to one embodiment of the present inven
tion, steps 710-745 may be in a loop that may be performed
one step at a time. In another embodiment, multiple steps may

35

exemplary embodiment of the present invention. In step 720,
the second quaternion With respect to a present time T is
calculated and obtained based on the angular velocities 00x, my
and 002. The step 715 and 720 may generally refer to or may be

represented by the abovementioned “current state” of the
40

six-axis motion sensor module. In one embodiment, the com

puting processor may use a data conversion utility to convert

the angular velocities 00x, my and 002 into the second quater
nion. This data conversion utility may be a program or

instruction represented by the folloWing equation (1).
45

method for obtaining the resulting deviation of the 3D point
ing device in 3D reference frame are as folloWs.

Referring to FIG. 7 again, the method for obtaining a

resulting deviation including resultant angles in a spatial
pointer reference frame of 3D pointing device utiliZing a

50

six-axis motion sensor module according to one embodiment

of the present invention may begin at the obtaining of a
previous state of the six-axis motion sensor module. In one

embedment, the previous state of the six-axis motion sensor
module may preferably be in a form of a ?rst quaternion, and

55

the ?rst quaternion may be preferably initialiZed (step 705) at
a very beginning of the process or method and as part of the

obtaining of the previous state thereof. In other Words,
according to one embodiment of the present invention, the
signals of the six-axis motion sensor are preferably to be
initialiZed to Zero and in particular, the signal or value asso

60

motion sensor module according to one embodiment of the

predetermined initial values. Alternatively, the ?rst quater
generated by the rotation sensor and the accelerometer at a
next time frame such that the method as shoWn in FIG. 7 is a

the differential equation (1).
As shoWn in the ?gure, the “measured state” of the six-axis

ciated With the yaW angle in terms of a quatemion value. The
four elements of the ?rst quatemion may be initialiZed With

nion may be initialiZed or replaced by another signal sets

Equation (1) is a differential equation. The quatemion on
the left side of the equal sign is the ?rst order derivative With
respect to time of the quaternion (qO, ql, q2, q3) on the right
side of the equal sign. The data conversion utility uses the ?rst
quatemion as the initial values for the differential equation (1)
and calculates the solution of the differential equation (1).
The second quatemion may be represented by a solution of

65

present invention may generally refer or may be represented
by steps 725 and 730. In step 725, the second signal set
generated by the accelerometer may be obtained, Which
includes measured axial accelerations Ax, Ay and AZ; or Ax,
Ay and AZ may refer to the measurement of the axial accel

US 8,441,438 B2
13

14

erations obtained. In order to obtain said measured state of the

accelerations of accelerometers and current state may be
obtained based on an exemplary equation of:

six-axis motion sensor of the present invention, according to

one embodiment, predicted axial accelerations Ax', Ay' and
AZ' may too be calculated and obtained based on the above
mentioned current state of the six-axis motion sensor module

Preferably, a second probability (measurement probability)

or the second quaternion as shoWn in step 73 0. In other Words,

associated With the measured state may be further obtained
based on an exemplary equation of:

tWo sets of axial accelerations may be obtained for the mea
sured state of the six-axis motion sensor module; one may be

the measured axial accelerations Ax, Ay, As in step 725 and
the other may be the predicted axial accelerations Ax', Ay', AZ'
in step 730 calculated based on the abovementioned current
state or second quaternion in relation to the measured angular
velocities thereof. Furthermore, in one embodiment, the com
puting processor may use a data conversion utility to convert

(10)

the measured axial accelerations Ax, Ay and AZ into a quater
nion. This data conversion utility may be a softWare program

Wherein RtImeasurement noise

represented by the folloWing equations (2), (3) and (4).

As an illustrative example, the abovementioned ?rst and sec

ond probabilities may be further utiliZed to obtain the updated
state of the six-axis motion sensor module based on an exem
20

plary method of data association of an exemplary equation of:

25

In one embodiment, the result of the updated state of the
six-axis motion sensor module, preferably involving com

(4)

The computing processor calculates the solution (qO, ql, q2,

q3) of the equations (2), (3) and (4).
According to an exemplary embodiment of the method for
obtaining a resulting deviation including resultant angles in a
spatial pointer reference frame of a 3D pointing device uti
liZing a six-axis motion sensor module, it may be preferable

resulting deviation including resultant angles in a spatial

to compare the current state of the six-axis motion sensor

module With the measured state thereof With respect to the

30

present time frame T by utiliZing a comparison model. In
other Words, in one embodiment as shoWn in step 735, it is
preferable to compare the second quaternion in relation to the
measured angular velocities of the current state at present
time T With the measured axial accelerations Ax, Ay, AZ as
Well as the predicted axial accelerations Ax', Ay', AZ' also at
present time T. FolloWing Which, a result may be obtained as

35

an updated state of the six-axis motion sensor module. In an
40

module at preset time T. Instructions including equations
related to the abovementioned current state, measured state

and updated state may be illustrated in the folloWing.
According to an exemplary embodiment of the comparison
model utiliZed by the present invention in relation to step 735

pointer reference frame in the folloWing steps as shoWn in the
?gure. It can be understood that the examples of current state,

measured state, state update, data association and probabili

explanatory example, the updated state may generally refer to
the update of the current state of the six-axis motion sensor

parison or data association represented by the equations, may
be a third quaternion as shoWn in the ?gure. Furthermore, the
result may then be further outputted and utiliZed to obtain a

45

ties of the comparison model and method of the present
invention are provided for illustrative purposes only.
As mentioned previously, it may be preferable to output the
result of the updated state, preferably in a form of third
quaternion, to the previous state of the six-axis motion sensor
module as shoWn in step 740 in the ?gure. In other Words, in
one embodiment, the ?rst quaternion may be replaced by the
abovementioned third quaternion or substitute directly any
previous values of ?rst quaternion in the previous time T for
further process in a loop. In other Words, the third quaternion
With respect to the present time T becomes the ?rst quaternion
With respect to the next time such as T+l ; or, the third quater
nion at previous time frame T-l outputted may noW be the

?rst quaternion at present time frame T.
In step 745, the updated state of the six-axis motion sensor
module of the present invention may be further calculated and
convert to the resulting deviation including resultant angles

as shoWn in the ?gure, the current state correlated to the

abovementioned second quaternion and in relation to the
angular velocities of gyroscope(s) may be obtained based on
an exemplary equation of:
50

associated With the spatial pointer reference frame, Wherein
the resultant angles includes the yaW angle, pitch angle and
roll angle of the 3D pointing device associated With the spatial
pointer reference frame, preferably about each of three
orthogonal coordinate axes of the spatial pointer reference

55

frame. In one embodiment, the computing processor may use
a data conversion utility to convert the third quaternion of the
updated state of the six-axis motion sensor module into the

Preferably, a ?rst probability (state transition probability)
associated With the said current state may be further obtained
based on an exemplary equation of:

yaW, pitch and roll angles thereof. This data conversion utility
FX : a?xH, 14!)

635141
F” : ?fwii, 14!)

may be a program or instruction represented by the folloWing

(6)
60

equations (l2), (l3) and (14).

(7)

1314,

Z(qoq3 +4142)

(12)

yaW : arctan W
‘10 + £11 — £12 — £13

Wherein Qfadditional motion noise
Likewise, the measured state correlated to the abovemen
tioned second axial accelerations and in relation to the axial

65
Pitch = aICSiIIQUIoQz — £13 £11 ))

(13)

US 8,441,438 B2
15

16

-continued

display to be mapped With deviation to a movement pattern in
14

roll : arctan[

2D display reference frame and the 3D pointing device of the

( )

present invention outputted With said deviation including
yaW, pitch and roll angles in 3D pointer reference frame;
Wherein the relationship may be a distance relationship. In

The variables qO, ql, q2 and q3 in equations (12), (13) and (14)

another embodiment, the sensitivity input may be a display
screen siZe including boundary information predetermined

are the four elements of the third quatemion.
For a looped method continuous With respect to time, in
one embodiment of the present invention, the method utilized

by a user; Wherein the boundary information may be obtained
based on a user input or manual input data from the user. In

by for example the computing processor communicated With

still another embodiment, the sensitivity input may be pre

the six-axis motion sensor module may return to step 710 to

de?ned or preset in the mapping program such that the param
eter of the sensitivity input is a preset value for either increase

perform the comparison process or method With respect to the
next time T+1. In addition, the abovementioned resulting

or decrease the movement patterns including distance or

deviation including resultant angles comprising yaW, pitch
and roll angles in the spatial reference frame converted from
the third quatemion is preferably obtained and outputted in an
absolute manner re?ecting or associating With the actual
movements and rotations of the 3D pointer device of the

present invention in said spatial pointer reference frame. It
can be understood that said actual movements and rotations of

20

the 3D pointer device of the present invention in the spatial
pointer reference frame or 3D reference frame may refer to
real-time movements and rotations associated With vectors

having both magnitudes and directions along or about
orthogonal axes in the spatial pointer reference frame under
the dynamic environments.

the 3D pointing device 930 is pointing at. The boundary point
25

FIG. 8 shoWs a How chart illustrating a method of mapping

resultant deviation angles of a 3D pointing device having
movements and rotations in a 3D spatial pointer reference
frame and in a dynamic environment onto a display reference

frame according to another embodiment of the present inven
tion. FIG. 9 is a schematic diagram showing the aforemen
tioned mapping of the resultant angles of the resultant devia
tion of a 3D pointing device according to this embodiment.
For illustrative purposes, the difference betWeen FIG. 7 and
FIG. 8 may be represented by the additional mapping step 750

number of pixels to be moved or mapped from said deviation
of the 3D pointing device.
FIG. 9 is a bird’s-eye vieW ofa 3D pointing device 930 and
the display screen 910 of a display device according to an
embodiment of the present invention. The display screen has
a central point 922, a target point 924 and a boundary point
926. The central point 922 is the geometric center of the
display screen 910. The target point 924 is the position that

30

926 is a point on the right boundary of the display screen 910.
The points 922, 924, 926 and the 3D pointing device 930 are
on a common plane parallel to both the XD axis and the ZD
axis of the display reference frame XDYDZD. Virtual beams
942, 944 and 946 are imaginary light beams from the 3D

pointing device 930 to the central point 922, the target point
924 and the boundary point 926, respectively. The distance P
is the distance betWeen the central point 922 and the target
point 924, While the distance Pmax is the distance betWeen the
central point 922 and the boundary point 926. The distance d

processor may include a mapping program that performs the

is the distance betWeen the central point 922 and the 3D
pointing device 930. The aforementioned yaW angle of the
resultant deviation of the 3D pointing device 930 is the angle
6 betWeen the virtual beams 942 and 944, While the angle
6m,C is the angle betWeen the virtual beams 942 and 946. The
aforementioned mapping area is a plane including the display
surface of the display screen 910 in the display reference

mapping step 750. At step 750, the processing and transmit
ting module may obtain display data including for example,

frame. The display surface of the display screen 910 is a
subset of the mapping area.

35

as shoWn in FIG. 8. Steps 705-745 in FIG. 8 are the same as

their counterparts in FIG. 7, Which perform the comparison
process for the 3D pointing device. Step 750 performs the
mapping process for the 3D pointing device. The computing

40

display screen siZe such as boundary information, and trans

lates the resultant angles of the resulting deviation associated

45

With the spatial pointer reference frame, preferably about

sensitivity [3 is de?ned by the folloWing equation (15).

each of three orthogonal coordinate axes of the spatial pointer
reference frame, to a movement pattern in a mapping area in
a display reference frame based on a sensitivity input corre
lated to the display reference frame. It can be understood that
the above-mentioned display data may too include or refer to

In this embodiment, the aforementioned sensitivity input is
provided by the user of the 3D pointing device 930. The

50

Pmax
9%

the type of display such as LED, LCD, touch panel or 3D

(15)

display as Well as frequency rate of display such as 120 HZ or

240 HZ. In one embodiment, the display reference frame
associated With the display to be mapped may be a 2D display

55

reference frame; in another embodiment, the display refer

The variable [3 in equation (16) is the sensitivity input
de?ned by user.

ence frame may be a 3D display reference frame of a 3D

display.
The aforementioned display data may further include a

sensitivity input. The aforementioned sensitivity input is a

60

parameter Which may be inputted and adjusted by a user
through control buttons attached on the housing of the 3D

pointing device. The sensitivity input may represent the sen
sitivity of the display device With respect to the movement of
the 3D pointing device. For details of the mapping process,
please refer to FIG. 9. In one embodiment, the sensitivity

input is a parameter representing the relationship betWeen the

The folloWing equation (16) may be derived from equation
(15) and geometry.
(16)

65

US 8,441,438 B2
17

18

The following equation (17) may be derived from equa

In vieW of the above, it is clear that such obtaining and

outputting of deviation including 3D angles in a spatial

tions (16).

pointer reference frame in an “absolute” manner of the
Pm ><tan0

present invention is novel, and the fact that the enhanced 3D
pointing device having a novel comparison method and pro
gram of the present invention to obtain and output such devia
tion in “absolute” manner cannot be easily achieved by any

(17)

knoWn arts or their combination thereof. The term “absolute”

associated With the resulting deviation including resultant
angles such as yaW, pitch and roll in a spatial pointer reference
frame or 3D reference frame obtained and outputted by the
enhanced 3D pointing device of the present invention may

In equation (17), the distance Pmax may be obtained from
the Width of the display screen of the display data obtained at

step 750; the angle 6 is the yaW angle obtained at step 745; the

sensitivity input [3 is provided by the user. Therefore, the

refer to the “actual” movements and rotations of the 3D

computing processor of the 3D pointing device 930 can cal

pointer device of the present invention in said spatial pointer

culate the distance P according to equation (17). Next, the

reference frame. It is clear that knoWn arts capable of only
outputting planar angles or relative movements, in for
example 2D reference frame, are devoid of providing a result
ing deviation in such absolute manner provided by the present
invention. Moreover, the six-axis comparison method of the
present invention may accurately output said deviation
including angles in 3D reference frame as noises associated

computing processor can easily obtain the horizontal coordi
nate of the target point 924 on the display screen 910 accord
ing to the distance P and the Width of the display screen 910.
In addition, the computing processor can easily obtains the
vertical coordinate of the target point 924 on the display
screen 910 according to the pitch angle in a similar Way.
The mapping process performed at step 750 may be exem

20

With the six-motion sensor module subject to movement and
rotations in dynamic environments and accumulated over
time may be effectively eliminated or compensated. The cur

pli?ed by the process of translating the yaW angle and the
pitch angle of the resultant angles to the 2D coordinates of the
target point 924 on the display screen 910 discussed above.
NoW the computing processor has the coordinates of the

target point 924 of the present time frame. The computing
processor subtracts the coordinates of the target point 924 of
the previous time frame from the coordinates of the target
point 924 of the present time frame. The result of the subtrac
tion is the horizontal offset and the vertical offset of the target

25

30

ing resultant angles in the spatial pointer reference frame or

point 924 in the present time frame. The horizontal and ver
tical offsets may be transmitted to the display device so that

the display device can track the position of the target point

35

924. The display device may display a cursor or some video

effect on the display screen 910 to highlight the position of the
target point 924. The cursor or video effect may exhibit a
movement pattern on the display screen 910 When the user
moves the 3D pointing device 930.

modi?cations and variations can be made to the structure of

the present invention Without departing from the scope or
spirit of the invention. In vieW of the foregoing, it is intended
that the present invention cover modi?cations and variations
of this invention provided they fall Within the scope of the

45

“a”, “an” or “one” recited herein as Well as in the claims

utilized by for example the computing processor communi

folloWing claims and their equivalents. Furthermore, the term

cated With the six-axis motion sensor module may return to

hereafter may refer to and include the meaning of “at least
one” or “more than one”. For example, it can be understood

that a printed circuit board (PCB) recited herein may refer to
50

more than one PCBs such that motion sensors such as rotation
sensors or gyroscopes and/or accelerometers of the six-mo
tion sensor module may be attached to more than one PCBs.

What is claimed is:
55

1. A three-dimensional (3D) pointing device subject to
movements and rotations in dynamic environments, compris

ing:
a housing associated With said movements and rotations of

the 3D pointing device in a spatial pointer reference

resultant deviation including yaW, pitch and roll angles in the
spatial pointer reference to a display reference frame such as
a 2D display reference frame of a display screen of a display
device. The six-axis comparison method involving the com
parison of motion sensor signals, the calculation and conver
sion of quaternion of the present invention in order to output
a resultant deviation having yaW, pitch and roll angles in for
example 3D reference frame is novel and cannot be easily
achieved by any knoW arts or their combinations thereof.

3D reference frame of the present invention can be further
mapped to another display reference frame or 2D reference
frame and such mapping of “absolute” movements and rota
tions of the enhanced 3D pointing device of the present inven
tion onto the display reference frame is novel and cannot be
easily achieved by knoWn arts or their combination thereof.
It Will be apparent to those skilled in the art that various

40

Likewise, for a looped method continuous With respect to
time, in one embodiment of the present invention, the method
step 710 to perform the comparison process or method With
respect to the next time T+1.to perform the comparison and
mapping process With respect to the next time frame.
In summary, the present invention also provides a six-axis
comparison method that compares the detected signals gen
erated by and converted from the rotation of the pointing
device about all of the three axes With the detected signals
generated by and converted from the acceleration of the
pointing device along all of the three axes. In one embodi
ment, The six-axis comparison method may then output the
resultant deviation including yaW, pitch and roll angles in a
spatial pointer reference frame such as a 3D reference frame
of the 3D pointing device. In another embodiment, the six
axis comparison method may also include the mapping of the

rent state, measured state, updated state of the six-axis motion
sensor module utilized in the method for obtaining the result
ing deviation and to eliminate the accumulated errors of the
motion sensor module of the 3D pointing device of the
present invention are novel and cannot be easily achieved by
the knoWn arts. Additionally, the resulting deviation includ

frame;
60

a printed circuit board (PCB) enclosed by the housing;
a six-axis motion sensor module attached to the PCB,

comprising a rotation sensor for detecting and generat

ing a ?rst signal set comprising angular velocities 00x,
my, 002 associated With said movements and rotations of
65

the 3D pointing device in the spatial pointer reference
frame, an accelerometer for detecting and generating a

second signal set comprising axial accelerations Ax, Ay,

US 8,441,438 B2
19

20

AZ associated With said movements and rotations of the

frame different from said spatial pointer reference frame and

3D pointing device in the spatial pointer reference
frame; and

based on a sensitivity input correlated to said display refer

a processing and transmitting module, comprising a data
transmitting unit electrically connected to the six-axis

9. The 3D pointing device of claim 8, Wherein the sensi
tivity input correlated to the display reference frame is deter

motion sensor module for transmitting said ?rst and
second signal sets thereof and a computing processor for

mined based on a user input and is associated With boundary

ence frame.

information of a display apparatus having a corresponding
mapping area in said display reference frame.
10. A three-dimensional (3D) pointing device subject to

receiving and calculating said ?rst and second signal sets
from the data transmitting unit, communicating With the

movements and rotations in dynamic environments in a
3D-pointer reference frame and associated With a movement

six-axis motion sensor module to calculate a resulting

deviation comprising resultant angles in said spatial
pointer reference frame by utiliZing a comparison to

pattern in a tWo-dimensional (2D)-display reference frame,

comprising:

compare the ?rst signal set With the second signal set

Whereby said resultant angles in the spatial pointer ref

a housing associated With said movements and rotations of
15

the 3D pointing device in the 3D-pointer reference

erence frame of the resulting deviation of the six-axis
motion sensor module of the 3D pointing device are

a printed circuit board (PCB) enclosed by the housing;

obtained under said dynamic environments, Wherein the

a six-axis motion sensor module attached to the PCB,

frame;

comparison utiliZed by the processing and transmitting
module further comprises an update program to obtain

comprising a rotation sensor for detecting and generat
20

ing a ?rst signal set comprising angular velocities 00x,

an updated state based on a previous state associated

my, 002 associated With said movements and rotations of

With said ?rst signal set and a measured state associated

the 3D pointing device in the 3D-pointer reference
frame, an accelerometer for detecting and generating a
second signal set comprising axial accelerations Ax, Ay,

With said second signal set; Wherein the measured state
includes a measurement of said second signal set and a

predicted measurement obtained based on the ?rst signal
set Without using any derivatives of the ?rst signal set.

25

3D pointing device in the 3D-pointer reference frame;
and

2. The 3D pointing device of claim 1, Wherein the dynamic

a processing and transmitting module, comprising a data
transmitting unit electrically connected to the six-axis

environments include a condition in Which said movements

and rotations of the 3D pointing device in the spatial pointer
reference frame are continuously nonlinear With respect to
time.
3. The 3D pointing device of claim 1, Wherein the PCB

enclosed by the housing comprises at least one substrate
having a ?rst longitudinal side con?gured to be substantially
parallel to a longitudinal surface of the housing.
4. The 3D pointing device of claim 1, Wherein the spatial
pointer reference frame is a reference frame in three dimen
sions; and Wherein saidresultant angles of the resulting devia
tion includes yaW, pitch and roll angles about each of three
orthogonal coordinate axes of the spatial pointer reference
frame.
5. The 3D pointing device of claim 1, Wherein the data

30

six-axis motion sensor module to calculate a resulting
35

40

resulting deviation of the six-axis motion sensor module

of the 3D pointing device in the 3D-pointer reference
frame to said movement pattern in the 2D-display refer
ence frame based on a sensitivity input correlated to said
45

2D-display reference frame, Wherein the comparison
utiliZed by the processing and transmitting module fur
ther comprises an update program to obtain an updated

tions on the PCB.
50

external to the housing and receives said ?rst and second
signal sets of the six-axis motion sensor module Wirelessly
from said data transmitting unit.
7. The 3D pointing device of claim 1, Wherein the com

parison utiliZed by the processing and transmitting module

deviation comprising resultant angles in said 3D-pointer
reference frame by utiliZing a comparison to compare
the ?rst signal set With the second signal set; and Wherein
the computing processor further comprises a mapping
program for translating said resultant angles of the

module to the computing processor via electronic connec

6. The 3D pointing device of claim 1, Wherein the comput
ing processor of the processing and transmitting module is

motion sensor module for transmitting said ?rst and
second signal sets thereof and a computing processor for

receiving and calculating said ?rst and second signal sets
from the data transmitting unit, communicating With the

transmitting unit of the processing and transmitting module is
attached to the PCB enclosed by the housing and transmits
said ?rst and second signal of the six-axis motion sensor

AZ associated With said movements and rotations of the

55

state based on a previous state associated With said ?rst
signal set and a measured state associated With said
second signal set; Wherein the measured state includes a
measurement of said second signal set and a predicted
measurement obtained based on the ?rst signal set With

out using any derivatives of the ?rst signal set; and
Wherein said resultant angles of the resulting deviation
includes yaW, pitch and roll angles about each of three
orthogonal coordinate axes of the spatial pointer refer

further comprises a data conversion utility for converting
quaternion values associated With said ?rst and second signal

11. The 3D pointing device of claim 10, Wherein the data

sets of the six-axis motion module as Well as an integrated

transmitting unit of the processing and transmitting module is

result to the resultant angles of the resulting deviation of the
six-axis motion sensor module of the 3D pointing device in

ence frame.

60

attached to the PCB enclosed by the housing and transmits
said ?rst and second signal of the six-axis motion sensor

the spatial pointer reference frame.

module to the computing processor via electronic connec

8. The 3D pointing device of claim 1, Wherein the comput
ing processor of the processing and transmitting module fur
ther comprises a mapping program for translating said result
ant angles of the resulting deviation of the six-axis motion
sensor module of the 3D pointing device in the spatial pointer

tions on the PCB.

reference frame to a movement pattern in a display reference

12. The 3D pointing device of claim 10, Wherein the com

puting processor of the processing and transmitting module is
65

external to the housing and receives said ?rst and second
signal sets of the six-axis motion sensor module Wirelessly
from said data transmitting unit.

US 8,441,438 B2
21

22

13. The 3D pointing device of claim 10, Wherein the sen

six-axis motion sensor module is a ?rst quatemion With

sitivity input correlated to the display reference frame is

respect to said previous time T-l; and said updated state of

predetermined by said mapping program of the computing
processor of the processing and transmitting module utiliZing

the six-axis motion sensor module is a third quaternion With
respect to said current time T.

17. The method for obtaining a resulting deviation of 3D

boundary information of a display apparatus having a corre

pointing device of claim 14, Wherein the obtaining of said

sponding mapping area in said 2D-display reference frame
and de?ned by a user input.

previous state of the six-axis motion sensor module further

14. A method for obtaining a resulting deviation including
resultant angles in a spatial pointer reference frame of a
three-dimensional (3D) pointing device utiliZing a six-axis

pointing device of claim 14, further comprises a mapping step

comprises initialiZing said initial-value set.
18. The method for obtaining a resulting deviation of 3D

comprising translating said angles of the resulting deviation

motion sensor module therein and subject to movements and

in said spatial pointer reference frame to a movement pattern

rotations in dynamic environments in said spatial pointer
reference frame, comprising the steps of:

in a display reference frame; and the mapping step further
comprises obtaining a sensitivity input correlated to said dis
play reference frame different from said spatial pointer ref

obtaining a previous state of the six-axis motion sensor
module; Wherein the previous state includes an initial

erence frame.

value set associated With previous angular velocities
gained from the motion sensor signals of the six-axis

19. A method for obtaining a resulting deviation including
resultant angles in a spatial pointer reference frame of a

motion sensor module at a previous time T-l;
obtaining a current state of the six-axis motion sensor

module by obtaining measured angular velocities 00x,

three-dimensional (3D) pointing device utiliZing a six-axis
20

my, 002 gained from the motion sensor signals of the
six-axis motion sensor module at a current time T;
obtaining a measured state of the six-axis motion sensor

obtaining a previous state of the six-axis motion sensor
module; Wherein the previous state includes an initial

module by obtaining measured axial accelerations Ax,
Ay, AZ gained from the motion sensor signals of the

25

six-axis motion sensor module at the current time T and

module by obtaining measured angular velocities 00x,

the current state of the six-axis motion sensor module
30

obtaining a measured state of the six-axis motion sensor

module by obtaining measured axial accelerations Ax,
Ay, AZ gained from the motion sensor signals of the

respect to said current time T; comparing the second
quatemion in relation to the measured angular velocities
35

based on the measured angular velocities 00x, my, 002 of
the current state of the six-axis motion sensor module

time T;

Without using any derivatives of the measured angular
40

motion sensor module to said resulting deviation com

00x, my, 002 of the current state at current time T With the

prising said resultant angles in said spatial pointer ref
45

outputting the updated state of the six-axis motion sensor
module to the previous state of the six-axis motion sensor

ence frame.

16. The method for obtaining a resulting deviation of a 3D

pointing device of claim 14, Wherein saidprevious state of the

measured axial accelerations Ax, Ay, AZ and the pre
dicted axial accelerations Ax', Ay', AZ' also at current

time T;

pointing device of claim 14, further comprises the step of
module; and Wherein said resultant angles of the resulting
deviation includes yaW, pitch and roll angles about each of
three orthogonal coordinate axes of the spatial pointer refer

velocities 00x, my, 002; said current state of the six-axis
motion sensor module is a second quatemion With

respect to said current time T; comparing the second
quatemion in relation to the measured angular velocities

calculating and converting the updated state of the six axis
erence frame of the 3D pointing device.
15. The method for obtaining a resulting deviation of a 3D

six-axis motion sensor module at the current time T and

calculating predicted axial accelerations Ax’, Ay', AZ,

measured axial accelerations Ax, Ay, AZ and the pre
dicted axial accelerations Ax', Ay', AZ' also at current
obtaining an updated state of the six-axis motion sensor
module by comparing the current state With the mea
sured state of the six-axis motion sensor module; and

my, 002 gained from the motion sensor signals of the
six-axis motion sensor module at a current time T;

velocities uux, my, uuZ; said current state of the six-axis
motion sensor module is a second quaternion With

00x, my, 002 of the current state at current time T With the

value set associated With previous angular velocities
gained from the motion sensor signals of the six-axis
motion sensor module at a previous time T-l;
obtaining a current state of the six-axis motion sensor

calculating predicted axial accelerations Ax', Ay', AZ'
based on the measured angular velocities 00x, my, 002 of

Without using any derivatives of the measured angular

motion sensor module therein and subject to movements and

rotations in dynamic environments in said spatial pointer
reference frame, comprising the steps of:

50

obtaining an updated state of the six-axis motion sensor
module by comparing the current state With the mea
sured state of the six-axis motion sensor module; and

calculating and converting the updated state of the six axis
motion sensor module to said resulting deviation com

prising said resultant angles in said spatial pointer ref
erence frame of the 3D pointing device.
*

*

*

*

*