Recent advances in tuning-free personalized image generation based on diffusion models are impres... more Recent advances in tuning-free personalized image generation based on diffusion models are impressive. However, to improve subject fidelity, existing methods either retrain the diffusion model or infuse it with dense visual embeddings, both of which suffer from poor generalization and efficiency. Also, these methods falter in multi-subject image generation due to the unconstrained cross-attention mechanism. In this paper, we propose MM-Diff, a unified and tuning-free image personalization framework capable of generating high-fidelity images of both single and multiple subjects in seconds. Specifically, to simultaneously enhance text consistency and subject fidelity, MM-Diff employs a vision encoder to transform the input image into CLS and patch embeddings. CLS embeddings are used on the one hand to augment the text embeddings, and on the other hand together with patch embeddings to derive a small number of detail-rich subject embeddings, both of which are efficiently integrated into the diffusion model through the welldesigned multimodal cross-attention mechanism. Additionally, MM-Diff introduces cross-attention map constraints during the training phase, ensuring flexible multi-subject image sampling during inference without any predefined inputs (e.g., layout). Extensive experiments demonstrate the superior performance of MM-Diff over other leading methods.
State-of-the-art text-video retrieval (TVR) methods typically utilize CLIP and cosine similarity ... more State-of-the-art text-video retrieval (TVR) methods typically utilize CLIP and cosine similarity for efficient retrieval. Meanwhile, cross attention methods, which employ a transformer decoder to compute attention between each text query and all frames in a video, offer a more comprehensive interaction between text and videos. However, these methods lack important fine-grained spatial information as they directly compute attention between text and video-level tokens. To address this issue, we propose CrossTVR, a two-stage text-video retrieval architecture. In the first stage, we leverage existing TVR methods with cosine similarity network for efficient text/video candidate selection. In the second stage, we propose a novel decoupled video text cross attention module to capture fine-grained multimodal information in spatial and temporal dimensions. Additionally, we employ the frozen CLIP model strategy in fine-grained retrieval, enabling scalability to larger pre-trained vision models like ViT-G, resulting in improved retrieval performance. Experiments on text video retrieval datasets demonstrate the effectiveness and scalability of our proposed CrossTVR compared to state-of-the-art approaches.
With the increasing volumes of digital image data and the rapid development of internet technolog... more With the increasing volumes of digital image data and the rapid development of internet technologies, it becomes vital to efficiently and accurately retrieve inquired images from the vast available data resources. In this context, content-based image retrieval has been intensively researched in the past decades. In this work, we propose to use contrast and luminance distribution, abbreviated as CoLD, to describe the textural and luminance characteristics of a digital image. The CoLD descriptor is rotation invariant and scale invariant. In addition, it is very simple to compute. Experimental results demonstrate that, when used for content-based image retrieval, the CoLD descriptor yields significantly higher retrieval precision when compared with the classical methods including gray level co-occurrence matrix and Hu's seven moment invariants.
In this work we introduce 2D-Dragger, a unified touch-based target acquisition technique that ena... more In this work we introduce 2D-Dragger, a unified touch-based target acquisition technique that enables easy access to small targets in dense regions or distant targets on screens of various sizes. The effective width of a target is constant with our tool, allowing a fixed scale of finger movement for capturing a new target. Our tool is thus insensitive to the distribution and size of the selectable targets, and consistently works well for screens of different sizes, from mobile to wall-sized screens. Our user studies show that overall 2D-Dragger performs the best compared to the state-of-the-art techniques for selecting both near and distant targets of various sizes in different densities.
IEEE Transactions on Visualization and Computer Graphics, Jun 1, 2016
Reconstructed building models using stereo-based methods inevitably suffer from noise, leading to... more Reconstructed building models using stereo-based methods inevitably suffer from noise, leading to the lack of regularity which is characterized by straightness of structural linear features and smoothness of homogeneous regions. We leverage the structural linear features embedded in the mesh to construct a novel surface scaffold structure for model regularization. The regularization comprises two iterative stages: (1) the linear features are semi-automatically proposed from images by exploiting photometric and geometric clues jointly; (2) the scaffold topology represented by spatial relations among the linear features is optimized according to data fidelity and topological rules, then the mesh is refined by adjusting itself to the consolidated scaffold. Our method has two advantages. First, the proposed scaffold representation is able to concisely describe semantic building structures. Second, the scaffold structure is embedded in the mesh, which can preserve the mesh connectivity and avoid stitching or intersecting surfaces in challenging cases. We demonstrate that our method can enhance structural characteristics and suppress irregularities in the building models robustly in some challenging datasets. Moreover, the regularization can significantly improve the results of general applications such as simplification and non-photorealistic rendering.
Figure 1: The proposed system automatically refines sketch lines (a, c, e) (created by different ... more Figure 1: The proposed system automatically refines sketch lines (a, c, e) (created by different users) roughly traced over a single image in a three-level optimization framework. The refined sketches (b, d, f) show closer resemblance to the traced images and are often aesthetically more pleasing, as confirmed by the user study.
Freehand sketching is a dynamic process where points are sequentially sampled and grouped as stro... more Freehand sketching is a dynamic process where points are sequentially sampled and grouped as strokes for sketch acquisition on electronic devices. To recognize a sketched object, most existing methods discard such important temporal ordering and grouping information from human and simply rasterize sketches into binary images for classification. In this paper, we propose a novel singlebranch attentive network architecture RNN-Rasterization-CNN (Sketch-R2CNN for short) to fully leverage the dynamics in sketches for recognition. Sketch-R2CNN takes as input only a vector sketch with grouped sequences of points, and uses an RNN for stroke attention estimation in the vector space and a CNN for 2D feature extraction in the pixel space respectively. To bridge the gap between these two spaces in neural networks, we propose a neural line rasterization module to convert the vector sketch along with the attention estimated by RNN into a bitmap image, which is subsequently consumed by CNN. The neural line rasterization module is designed in a differentiable way to yield a unified pipeline for end-to-end learning. We perform experiments on existing large-scale sketch recognition benchmarks and show that by exploiting the sketch dynamics with the attention mechanism, our method is more robust and achieves better performance than the state-of-the-art methods.
2021 International Conference on 3D Vision (3DV), 2021
Deep learning based 3D shape generation methods generally utilize latent features extracted from ... more Deep learning based 3D shape generation methods generally utilize latent features extracted from color images to encode the semantics of objects and guide the shape generation process. These color image semantics only implicitly encode 3D information, potentially limiting the accuracy of the generated shapes. In this paper we propose a multi-view mesh generation method which incorporates geometry information explicitly by using the features from intermediate depth representations of multi-view stereo and regularizing the 3D shapes against these depth images. First, our system predicts a coarse 3D volume from the color images by probabilistically merging voxel occupancy grids from the prediction of individual views. Then the depth images from multi-view stereo along with the rendered depth images of the coarse shape are used as a contrastive input whose features guide the refinement of the coarse shape through a series of graph convolution networks. Notably, we achieve superior results than state-of-the-art multi-view shape generation methods with 34% decrease in Chamfer distance to ground truth and 14% increase in F1-score on ShapeNet dataset. Our source code is available at
IEEE Transactions on Visualization and Computer Graphics, 2021
Sketches in existing large-scale datasets like the recent QuickDraw collection are often stored i... more Sketches in existing large-scale datasets like the recent QuickDraw collection are often stored in a vector format, with strokes consisting of sequentially sampled points. However, most existing sketch recognition methods rasterize vector sketches as binary images and then adopt image classification techniques. In this paper, we propose a novel end-to-end single-branch network architecture RNN-Rasterization-CNN (Sketch-R2CNN for short) to fully leverage the vector format of sketches for recognition. Sketch-R2CNN takes a vector sketch as input and uses an RNN for extracting per-point features in the vector space. We then develop a neural line rasterization module to convert the vector sketch and the per-point features to multi-channel point feature maps, which are subsequently fed to a CNN for extracting convolutional features in the pixel space. Our neural line rasterization module is designed in a differentiable way for end-to-end learning. We perform experiments on existing large-scale sketch recognition datasets and show that the RNN-Rasterization design brings consistent improvement over CNN baselines and that Sketch-R2CNN substantially outperforms the state-of-the-art methods.
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018
Input Sketch & Video Mo on Transfer Mo on Extrac on (c) (b) (a) (d) Figure 1. We present Live Ske... more Input Sketch & Video Mo on Transfer Mo on Extrac on (c) (b) (a) (d) Figure 1. We present Live Sketch, an interactive system for users with no or little animation skills to create vivid dynamic deformation of a sketch (a) by first extracting object motion from a video sequence (bottom), and then transferring the motion to the sketch (top). Please see the animation in the accompanying video. Video credit: Albert Ziganshin.
Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services, 2016
In this work we introduce 2D-Dragger, a unified touch-based target acquisition technique that ena... more In this work we introduce 2D-Dragger, a unified touch-based target acquisition technique that enables easy access to small targets in dense regions or distant targets on screens of various sizes. The effective width of a target is constant with our tool, allowing a fixed scale of finger movement for capturing a new target. Our tool is thus insensitive to the distribution and size of the selectable targets, and consistently works well for screens of different sizes, from mobile to wall-sized screens. Our user studies show that overall 2D-Dragger performs the best compared to the state-of-the-art techniques for selecting both near and distant targets of various sizes in different densities.
IEEE Transactions on Visualization and Computer Graphics, 2016
Reconstructed building models using stereo-based methods inevitably suffer from noise, leading to... more Reconstructed building models using stereo-based methods inevitably suffer from noise, leading to the lack of regularity which is characterized by straightness of structural linear features and smoothness of homogeneous regions. We leverage the structural linear features embedded in the mesh to construct a novel surface scaffold structure for model regularization. The regularization comprises two iterative stages: (1) the linear features are semi-automatically proposed from images by exploiting photometric and geometric clues jointly; (2) the scaffold topology represented by spatial relations among the linear features is optimized according to data fidelity and topological rules, then the mesh is refined by adjusting itself to the consolidated scaffold. Our method has two advantages. First, the proposed scaffold representation is able to concisely describe semantic building structures. Second, the scaffold structure is embedded in the mesh, which can preserve the mesh connectivity and avoid stitching or intersecting surfaces in challenging cases. We demonstrate that our method can enhance structural characteristics and suppress irregularities in the building models robustly in some challenging datasets. Moreover, the regularization can significantly improve the results of general applications such as simplification and non-photorealistic rendering.
2011 International Conference on Image Analysis and Signal Processing, 2011
With the increasing volumes of digital image data and the rapid development of internet technolog... more With the increasing volumes of digital image data and the rapid development of internet technologies, it becomes vital to efficiently and accurately retrieve inquired images from the vast available data resources. In this context, content-based image retrieval has been intensively researched in the past decades. In this work, we propose to use contrast and luminance distribution, abbreviated as CoLD, to describe the textural and luminance characteristics of a digital image. The CoLD descriptor is rotation invariant and scale invariant. In addition, it is very simple to compute. Experimental results demonstrate that, when used for content-based image retrieval, the CoLD descriptor yields significantly higher retrieval precision when compared with the classical methods including gray level co-occurrence matrix and Hu's seven moment invariants.
We present a new image-guided drawing interface called EZ-Sketching , which uses a tracing paradi... more We present a new image-guided drawing interface called EZ-Sketching , which uses a tracing paradigm and automatically corrects sketch lines roughly traced over an image by analyzing and utilizing the image features being traced. While previous edge snapping methods aim at optimizing individual strokes, we show that a co-analysis of multiple roughly placed nearby strokes better captures the user's intent. We formulate automatic sketch improvement as a three-level optimization problem and present an efficient solution to it. EZ-Sketching can tolerate errors from various sources such as indirect control and inherently inaccurate input, and works well for sketching on touch devices with small screens using fingers. Our user study confirms that the drawings our approach helped generate show closer resemblance to the traced images, and are often aesthetically more pleasing.
Recent advances in tuning-free personalized image generation based on diffusion models are impres... more Recent advances in tuning-free personalized image generation based on diffusion models are impressive. However, to improve subject fidelity, existing methods either retrain the diffusion model or infuse it with dense visual embeddings, both of which suffer from poor generalization and efficiency. Also, these methods falter in multi-subject image generation due to the unconstrained cross-attention mechanism. In this paper, we propose MM-Diff, a unified and tuning-free image personalization framework capable of generating high-fidelity images of both single and multiple subjects in seconds. Specifically, to simultaneously enhance text consistency and subject fidelity, MM-Diff employs a vision encoder to transform the input image into CLS and patch embeddings. CLS embeddings are used on the one hand to augment the text embeddings, and on the other hand together with patch embeddings to derive a small number of detail-rich subject embeddings, both of which are efficiently integrated into the diffusion model through the welldesigned multimodal cross-attention mechanism. Additionally, MM-Diff introduces cross-attention map constraints during the training phase, ensuring flexible multi-subject image sampling during inference without any predefined inputs (e.g., layout). Extensive experiments demonstrate the superior performance of MM-Diff over other leading methods.
State-of-the-art text-video retrieval (TVR) methods typically utilize CLIP and cosine similarity ... more State-of-the-art text-video retrieval (TVR) methods typically utilize CLIP and cosine similarity for efficient retrieval. Meanwhile, cross attention methods, which employ a transformer decoder to compute attention between each text query and all frames in a video, offer a more comprehensive interaction between text and videos. However, these methods lack important fine-grained spatial information as they directly compute attention between text and video-level tokens. To address this issue, we propose CrossTVR, a two-stage text-video retrieval architecture. In the first stage, we leverage existing TVR methods with cosine similarity network for efficient text/video candidate selection. In the second stage, we propose a novel decoupled video text cross attention module to capture fine-grained multimodal information in spatial and temporal dimensions. Additionally, we employ the frozen CLIP model strategy in fine-grained retrieval, enabling scalability to larger pre-trained vision models like ViT-G, resulting in improved retrieval performance. Experiments on text video retrieval datasets demonstrate the effectiveness and scalability of our proposed CrossTVR compared to state-of-the-art approaches.
With the increasing volumes of digital image data and the rapid development of internet technolog... more With the increasing volumes of digital image data and the rapid development of internet technologies, it becomes vital to efficiently and accurately retrieve inquired images from the vast available data resources. In this context, content-based image retrieval has been intensively researched in the past decades. In this work, we propose to use contrast and luminance distribution, abbreviated as CoLD, to describe the textural and luminance characteristics of a digital image. The CoLD descriptor is rotation invariant and scale invariant. In addition, it is very simple to compute. Experimental results demonstrate that, when used for content-based image retrieval, the CoLD descriptor yields significantly higher retrieval precision when compared with the classical methods including gray level co-occurrence matrix and Hu's seven moment invariants.
In this work we introduce 2D-Dragger, a unified touch-based target acquisition technique that ena... more In this work we introduce 2D-Dragger, a unified touch-based target acquisition technique that enables easy access to small targets in dense regions or distant targets on screens of various sizes. The effective width of a target is constant with our tool, allowing a fixed scale of finger movement for capturing a new target. Our tool is thus insensitive to the distribution and size of the selectable targets, and consistently works well for screens of different sizes, from mobile to wall-sized screens. Our user studies show that overall 2D-Dragger performs the best compared to the state-of-the-art techniques for selecting both near and distant targets of various sizes in different densities.
IEEE Transactions on Visualization and Computer Graphics, Jun 1, 2016
Reconstructed building models using stereo-based methods inevitably suffer from noise, leading to... more Reconstructed building models using stereo-based methods inevitably suffer from noise, leading to the lack of regularity which is characterized by straightness of structural linear features and smoothness of homogeneous regions. We leverage the structural linear features embedded in the mesh to construct a novel surface scaffold structure for model regularization. The regularization comprises two iterative stages: (1) the linear features are semi-automatically proposed from images by exploiting photometric and geometric clues jointly; (2) the scaffold topology represented by spatial relations among the linear features is optimized according to data fidelity and topological rules, then the mesh is refined by adjusting itself to the consolidated scaffold. Our method has two advantages. First, the proposed scaffold representation is able to concisely describe semantic building structures. Second, the scaffold structure is embedded in the mesh, which can preserve the mesh connectivity and avoid stitching or intersecting surfaces in challenging cases. We demonstrate that our method can enhance structural characteristics and suppress irregularities in the building models robustly in some challenging datasets. Moreover, the regularization can significantly improve the results of general applications such as simplification and non-photorealistic rendering.
Figure 1: The proposed system automatically refines sketch lines (a, c, e) (created by different ... more Figure 1: The proposed system automatically refines sketch lines (a, c, e) (created by different users) roughly traced over a single image in a three-level optimization framework. The refined sketches (b, d, f) show closer resemblance to the traced images and are often aesthetically more pleasing, as confirmed by the user study.
Freehand sketching is a dynamic process where points are sequentially sampled and grouped as stro... more Freehand sketching is a dynamic process where points are sequentially sampled and grouped as strokes for sketch acquisition on electronic devices. To recognize a sketched object, most existing methods discard such important temporal ordering and grouping information from human and simply rasterize sketches into binary images for classification. In this paper, we propose a novel singlebranch attentive network architecture RNN-Rasterization-CNN (Sketch-R2CNN for short) to fully leverage the dynamics in sketches for recognition. Sketch-R2CNN takes as input only a vector sketch with grouped sequences of points, and uses an RNN for stroke attention estimation in the vector space and a CNN for 2D feature extraction in the pixel space respectively. To bridge the gap between these two spaces in neural networks, we propose a neural line rasterization module to convert the vector sketch along with the attention estimated by RNN into a bitmap image, which is subsequently consumed by CNN. The neural line rasterization module is designed in a differentiable way to yield a unified pipeline for end-to-end learning. We perform experiments on existing large-scale sketch recognition benchmarks and show that by exploiting the sketch dynamics with the attention mechanism, our method is more robust and achieves better performance than the state-of-the-art methods.
2021 International Conference on 3D Vision (3DV), 2021
Deep learning based 3D shape generation methods generally utilize latent features extracted from ... more Deep learning based 3D shape generation methods generally utilize latent features extracted from color images to encode the semantics of objects and guide the shape generation process. These color image semantics only implicitly encode 3D information, potentially limiting the accuracy of the generated shapes. In this paper we propose a multi-view mesh generation method which incorporates geometry information explicitly by using the features from intermediate depth representations of multi-view stereo and regularizing the 3D shapes against these depth images. First, our system predicts a coarse 3D volume from the color images by probabilistically merging voxel occupancy grids from the prediction of individual views. Then the depth images from multi-view stereo along with the rendered depth images of the coarse shape are used as a contrastive input whose features guide the refinement of the coarse shape through a series of graph convolution networks. Notably, we achieve superior results than state-of-the-art multi-view shape generation methods with 34% decrease in Chamfer distance to ground truth and 14% increase in F1-score on ShapeNet dataset. Our source code is available at
IEEE Transactions on Visualization and Computer Graphics, 2021
Sketches in existing large-scale datasets like the recent QuickDraw collection are often stored i... more Sketches in existing large-scale datasets like the recent QuickDraw collection are often stored in a vector format, with strokes consisting of sequentially sampled points. However, most existing sketch recognition methods rasterize vector sketches as binary images and then adopt image classification techniques. In this paper, we propose a novel end-to-end single-branch network architecture RNN-Rasterization-CNN (Sketch-R2CNN for short) to fully leverage the vector format of sketches for recognition. Sketch-R2CNN takes a vector sketch as input and uses an RNN for extracting per-point features in the vector space. We then develop a neural line rasterization module to convert the vector sketch and the per-point features to multi-channel point feature maps, which are subsequently fed to a CNN for extracting convolutional features in the pixel space. Our neural line rasterization module is designed in a differentiable way for end-to-end learning. We perform experiments on existing large-scale sketch recognition datasets and show that the RNN-Rasterization design brings consistent improvement over CNN baselines and that Sketch-R2CNN substantially outperforms the state-of-the-art methods.
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018
Input Sketch & Video Mo on Transfer Mo on Extrac on (c) (b) (a) (d) Figure 1. We present Live Ske... more Input Sketch & Video Mo on Transfer Mo on Extrac on (c) (b) (a) (d) Figure 1. We present Live Sketch, an interactive system for users with no or little animation skills to create vivid dynamic deformation of a sketch (a) by first extracting object motion from a video sequence (bottom), and then transferring the motion to the sketch (top). Please see the animation in the accompanying video. Video credit: Albert Ziganshin.
Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services, 2016
In this work we introduce 2D-Dragger, a unified touch-based target acquisition technique that ena... more In this work we introduce 2D-Dragger, a unified touch-based target acquisition technique that enables easy access to small targets in dense regions or distant targets on screens of various sizes. The effective width of a target is constant with our tool, allowing a fixed scale of finger movement for capturing a new target. Our tool is thus insensitive to the distribution and size of the selectable targets, and consistently works well for screens of different sizes, from mobile to wall-sized screens. Our user studies show that overall 2D-Dragger performs the best compared to the state-of-the-art techniques for selecting both near and distant targets of various sizes in different densities.
IEEE Transactions on Visualization and Computer Graphics, 2016
Reconstructed building models using stereo-based methods inevitably suffer from noise, leading to... more Reconstructed building models using stereo-based methods inevitably suffer from noise, leading to the lack of regularity which is characterized by straightness of structural linear features and smoothness of homogeneous regions. We leverage the structural linear features embedded in the mesh to construct a novel surface scaffold structure for model regularization. The regularization comprises two iterative stages: (1) the linear features are semi-automatically proposed from images by exploiting photometric and geometric clues jointly; (2) the scaffold topology represented by spatial relations among the linear features is optimized according to data fidelity and topological rules, then the mesh is refined by adjusting itself to the consolidated scaffold. Our method has two advantages. First, the proposed scaffold representation is able to concisely describe semantic building structures. Second, the scaffold structure is embedded in the mesh, which can preserve the mesh connectivity and avoid stitching or intersecting surfaces in challenging cases. We demonstrate that our method can enhance structural characteristics and suppress irregularities in the building models robustly in some challenging datasets. Moreover, the regularization can significantly improve the results of general applications such as simplification and non-photorealistic rendering.
2011 International Conference on Image Analysis and Signal Processing, 2011
With the increasing volumes of digital image data and the rapid development of internet technolog... more With the increasing volumes of digital image data and the rapid development of internet technologies, it becomes vital to efficiently and accurately retrieve inquired images from the vast available data resources. In this context, content-based image retrieval has been intensively researched in the past decades. In this work, we propose to use contrast and luminance distribution, abbreviated as CoLD, to describe the textural and luminance characteristics of a digital image. The CoLD descriptor is rotation invariant and scale invariant. In addition, it is very simple to compute. Experimental results demonstrate that, when used for content-based image retrieval, the CoLD descriptor yields significantly higher retrieval precision when compared with the classical methods including gray level co-occurrence matrix and Hu's seven moment invariants.
We present a new image-guided drawing interface called EZ-Sketching , which uses a tracing paradi... more We present a new image-guided drawing interface called EZ-Sketching , which uses a tracing paradigm and automatically corrects sketch lines roughly traced over an image by analyzing and utilizing the image features being traced. While previous edge snapping methods aim at optimizing individual strokes, we show that a co-analysis of multiple roughly placed nearby strokes better captures the user's intent. We formulate automatic sketch improvement as a three-level optimization problem and present an efficient solution to it. EZ-Sketching can tolerate errors from various sources such as indirect control and inherently inaccurate input, and works well for sketching on touch devices with small screens using fingers. Our user study confirms that the drawings our approach helped generate show closer resemblance to the traced images, and are often aesthetically more pleasing.
Uploads
Papers by Qingkun SU