Seoyeon Kim

Hi there! I'm a passionate PhD student delving into the intersection of computer vision and computational imaging.

During my master's at Computer Vision Lab, I focused on vision-language multimodal tasks under the guidance of Prof. Suha Kwak.

Now, for my PhD with Prof. Seunghwan Baek, I'm combining computer vision and computational imaging, leveraging end-to-end optimization of visual perception models and metasurfaces to create cutting-edge imaging systems.

If you are interested in my work, feel free to reach out.

Education

Sept, 2022 - Aug, 2024	POSTECH, Pohang, South Korea M.S. in Computer Science and Engineering GPA 4.06/4.30 Advised by Prof. Suha Kwak
Feb, 2018 - Aug, 2022	POSTECH, Pohang, South Korea B.S. in Computer Science and Engineering Summa Cum Laude Advised by Prof. Jaesik Park
Sept, 2013 - Aug, 2017	City of London Freemen's School, Surrey, United Kingdom 4 A*s in A-level (Maths, Further Maths, Physics, Chemistry)

Experience

June, 2021 - Sept, 2021	Hyundai Motor Group, Seoul, South Korea Research Intern Researched Object Detection and Neural Style Transfer.
Feb, 2021 - Aug, 2022	Computer Vision Lab, POSTECH, Pohang, South Korea Undergraduate Research Intern Developed Sample-Dependent Occlusion Network for Intelligent Data Augmentation. Proposed Patch-wise Matching Framework for Self-Supervised Learning in Vision Transformers.

Publications

Bootstrapping Top-down Information for Self-modulating Slot Attention

Dongwon Kim, Seoyeon Kim, Suha Kwak

In Proceedings of Neural Information Processing Systems (NeurIPS), 2024

Abstract Paper

Object-centric learning (OCL) aims to learn representations of individual objects within visual scenes without manual supervision, facilitating efficient and effective visual reasoning. Traditional OCL methods primarily employ bottom-up approaches that aggregate homogeneous visual features to represent objects. However, in complex visual environments, these methods often fall short due to the heterogeneous nature of visual features within an object. To address this, we propose a novel OCL framework incorporating a top-down pathway. This pathway first bootstraps the semantics of individual objects and then modulates the model to prioritize features relevant to these semantics. By dynamically modulating the model based on its own output, our top-down pathway enhances the representational quality of objects. Our framework achieves state-of-the-art performance across multiple synthetic and real-world object-discovery benchmarks.

Extending CLIP’s Image-Text Alignment to Referring Image Segmentation

Seoyeon Kim, Minguk Kang, Dongwon Kim, Jaesik Park, Suha Kwak

In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL), 2024

Abstract Paper

Referring Image Segmentation (RIS) is a cross-modal task that aims to segment an instance described by a natural language expression. Recent methods leverage large-scale pretrained unimodal models as backbones along with fusion techniques for joint reasoning across modalities. However, the inherent cross-modal nature of RIS raises questions about the effectiveness of unimodal backbones. We propose RISCLIP, a novel framework that effectively leverages the cross-modal nature of CLIP for RIS. Observing CLIP's inherent alignment between image and text features, we capitalize on this starting point and introduce simple but strong modules that enhance unimodal feature extraction and leverage rich alignment knowledge in CLIP's image-text shared-embedding space. RISCLIP exhibits outstanding results on all three major RIS benchmarks and also outperforms previous CLIP-based methods, demonstrating the efficacy of our strategy in extending CLIP's image-text alignment to RIS.

PAM: Patch Aware Matching for Vision Transformer based Self-Supervised Learning Frameworks

Seoyeon Kim, Minguk Kang, and Jaesik Park

34th Workshop on Image Processing and Image Understanding (IPIU), Bronze Award, 2023

Abstract

Self-supervised Learning (SSL) is a field that learns meaningful representations from large unlabeled datasets and finetunes such representations to target downstream tasks. Recently, many SSL methods that employ the Vision Transformer based Teacher-Student framework with different losses have been proven effective—even effective enough to outperform their supervised counterparts trained with labeled data. In line with this effective ViT-based Teacher-Student framework, we propose a new loss named “Patch Aware Matching (PAM)” which performs patch token feature distillation across the local view features output from the student and global view features output from the teacher. We hypothesize that such an approach learns “local-to-global” correspondence along with local, structural information on the patch level. When trained and evaluated under the k-NN protocol with a ViT-Small/16 backbone, our approach outperforms state-of-the-art methods on the 100 easiest classes of ImageNet-1K but falls behind on 10% of ImageNet. Also, we come across interesting results that suggest that easier classes of ImageNet-1K are more sample efficient. As a work in progress, we aim to further develop our method and investigate sample efficiency of different ImageNet-1K classes.

Awards and Scholarships

First Place in Research Project I Course, POSTECH (2021)

Won Best Award for outstanding research effort

National Science and Engineering Scholarship, Korea Student Aid Foundation (2022)

Received two years full scholarhip for academic excellence

Global Leadership Program, POSTECH (2020, 2021)

Received CSE scholarship twice for academic potential (4K USD)