(a), classifying pixel (red circle) is difficult by watching local space-time context (green boxes) due to limited inter-class variance between kidney and covered kidney.Įxtending space-time view (yellow boxes) to see the folded fascia helps accurately categorize the pixel to covered kidney. The perceived context in these works is insufficient to achieve accurate segmentation for complicated surgical scenes. TheĬonventional temporal modelling module of convolutional LSTM is just employed in a recent study, which is also limited by the local reception field, otherwise suffers from model training difficulty. Meanwhile they solely model the spatial information while ignore the sequential dynamics in surgical videos. We identify two main limitations in current state-of-the-art literature to tackle this task.įirstly, most recent works rely on traditional feature aggregation modules, such as atrous convolution, whose encoded visual context is still relatively local for segmenting surgical scenes. We consider both relations in model learning to boost segmentation. (a) High discrimination of each pixel embedding via intra-video relation modelling: aggregating richer space-time context to each pixel.(b) Well-structured semantic embedding space via inter-video relation modelling: computing pixel-to-pixel contrast by pulling pixels from the same class closer while repelling pixels of different classes. 1: Two core factors for accurate surgical scene segmentation. Motion blur, lighting changes, occlusions from smoke and blood, further increase the challenges that segmentation models need to overcome. thread) and rarely used instruments are difficult to identify. Class imbalance also exists in surgical scenes, in which tiny objects ( e.g. different instruments) and high intra-class variance ( e.g. However, precisely parsing the entire scene from surgical video is highly challenging because complicated surgical scenes lead to limited inter-class variance ( e.g. Precisely identifying instruments and their location is also a central CAI theme with work on tool pose estimation, tool tracking and control and surgical task automation Semantic labels can facilitate cognitive assistance, by providing pixel-wise context awareness of tissues and instruments, which is fundamentally required for supporting several downstream tasks, such as surgical decision making, surgical navigation and skill assessment Semantic segmentation of the entire surgical scene in the field of view of the surgical camera is an essential prerequisite for modern CAI systems. Code will beĬomputer assisted interventions (CAI) are revolutionizing surgical procedures to achieve enhanced patient safety with improved operative quality, reduced adverse events and shorter recovery periods. Surgical video benchmarks, including EndoVis18 Challenge and CaDIS dataset.Įxperimental results demonstrate the promising performance of our method, whichĬonsistently exceeds previous state-of-the-art approaches. We extensively validate our approach on two public With the ground-truth guidance, which is crucial for learning the global Training objective is developed to group the pixel embeddings across videos Which well structures the global embedding space. Then, we explore inter-video relation via pixel-to-pixel contrastive learning, Proposed to efficiently aggregate these two cues into each pixel embedding. A joint space-time window shift scheme is Intra-video relation that includes richer spatial and temporal cues from We firstly develop a hierarchy Transformer to capture Relations to boost segmentation performance, by progressively capturing the Novel framework STswinCL that explores the complementary intra- and inter-video LSTM), which only make use of the local context. Previous works rely onĬonventional aggregation modules (e.g., dilated convolution, convolutional Automatic surgical scene segmentation is fundamental for facilitatingĬognitive intelligence in the modern operating theatre.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |