Purpose: Currently effort is taken to use radiation therapy to cure heart diseases like arrhythmia. This approach requires high accuracy localisation and tracking of the pulmonary veins. Because of the high speed of motion of the heart, fluoroscopic tracking of fiducials or anatomical structures as in IGRT would on the one hand require high frame rates and, on the other hand, it would be dangerous to place fiducials near the target. We propose to use live 3D ultrasound to perform the landmark localization and tracking. Methods and materials: We have modified a GE Vivid7 dimension 3D cardiovascular ultrasound station for real‐time volume processing and target localisation. It is capable of providing ultrasound volume scans of the target region with more than 20 fps. A framework was established to upload and run image‐processing algorithms directly on the ultrasound machine which is necessary to handle the high amount of data. This prevents the bottleneck of Ethernet data streaming and external processing. We propose to localise the pulmonary veins using a template matching algorithm with multiple templates. Approximately 20 templates are manually generated during one heart beat cycle. To increase the speed and accuracy of the matching process, electrical pulse signals were recorded by the ultrasound station. This allows selecting two or three pulse‐dependent templates in the live matching stage. Results: The accuracy of the localization process is highly dependent on the templates chosen. The best results were achieved providing a full heart cycle as template data. As a compromise between speed and accuracy, we used 9×9×9 points as template, corresponding to 4.5×4.5×4.5mm3. Conclusion: The presented approach is a new and robust approach to semi‐automatically track small substructures in the beating heart. Furthermore, the generated signal is suitable as input to numerous prediction algorithms currently used to compensate for breathing motion in radiosurgery.