ESR10 - Wiktor Mucha

Under this project, state-of-the-art research on egocentric vision and lifelogging from a computer vision perspective was initially conducted, focusing on segmentation, action recognition, food-related scenes, and social interaction tracking. A comprehensive database of available datasets in the field was also created. This initial research identified several open challenges that have received limited attention from the scientific community, leading to research exploring the potential of egocentric video-based lifelogging systems for tracking actions and behaviours that impact health and well-being. Additionally, the feasibility of processing egocentric images to assist with health-related tasks, such as rehabilitation and detecting struggles to provide timely assistance, was investigated. Building on these questions, significant contributions have been made through the development of novel approaches for 2D hand pose estimation, achieving superior accuracy on public benchmarks. A 3D hand pose estimation method was also introduced, enhancing results using pseudo-depth data derived from RGB images. Further improvements were made through a camera-agnostic approach for zero-shot 3D hand pose estimation, significantly reducing mean pose error across unseen domains. These advancements contribute to bridging the gap between laboratory conditions and real-world scenarios, improving the generalisability and reliability of experimental results. Further investigations were conducted to analyse the usability of 2D hand and object poses for egocentric action recognition tasks and to evaluate performance variations when different types of pose input are used. These theoretical advancements have been successfully translated into practical and novel AAL applications, including an intelligent reading assistant for visually impaired users that integrates smart glasses with LLMs and the study for stroke patients' rehabilitation, which establishes benchmarks for exercise recognition, form evaluation, and repetition counting. This research demonstrates how egocentric vision can meaningfully support individuals in need by bridging the gap between advanced computer vision techniques and real-world assistive technologies.