During threedimensional vibration displacement measurement based on optical devices, the reference point of the optical measurement system must remain stationary. It is difficult to satisfy this condition in field tests. To solve this problem, a compensation method was presented to amend the influence of small vibrations of the reference point. The displacement of the reference point can be obtained from its acceleration signal at first, and the correction matrixes for target displacement are derived for compensating the measurement errors caused by the shaking of the reference point. The data processing algorithm to convert acceleration signal to displacement signal is presented, and the effectiveness of this algorithm is verified by simulation of composite sinusoidal signals and damped decaying vibration signals with random white noise. The theory and calculation procedure for correction matrixes derived from displacement of the reference point are comprehensively discussed. The experimental measurement system was built using two sets of NDI Optotrak Certus System and one 3D capacitive accelerometer, which is used to verify the theory and method proposed in the paper. Measurement results with data processing show that this method can effectively compensate and correct the measurement errors due to vibrations of the reference point, thus improved measurement accuracy is expected.