In order to turn digital representations into actual digital twins, they need to be extended to (i) cover all relevant parts of their environment (covering possibly a large area of the real world), (ii) be kept in sync with changes in the real world, and (iii) represent the semantics of objects in order to link sensor data to functionalities. While it is feasible to use high-end equipment for the initial capture, continuous updates need to be obtained in a cost-effective way, using the numerous but heterogeneous sensors already deployed. Automated methods based on artificial intelligence are needed in order to fuse information from these sensors, analyse the semantics of the scene and make the appropriate changes to the digital twin.

The capture of scenes at scale, as well as using cameras and other sensor data for synchronising the digital representation, bears the risk of capturing personal and sensitive data. Hence, the technologies resulting from DIDYMOS-XR must be and will be ethical and privacy-aware by design. DIDYMOS-XR will research and develop robust and scalable methods for 3D scene reconstruction from heterogeneous cameras and sensor data (e.g., lidar), integrating data captured at different times and under different environmental conditions and creating accurate maps of the scene. The semantics of elements in the scene will be automatically annotated using deep learning approaches trained with weakly or self-supervised methods able to scale and adapt to a large set of relevant objects.