Engineering Sketch Generation for Computer-Aided Design

Karl D.D. Willis Pradeep Kumar Jayaraman Joseph G. Lambourne Hang Chu Yewen Pu
Autodesk Research

Engineering Sketch
Generator

SKETCH GENERATION 3D SYNTHESIS 3D COMPOSITION

Figure 1: We tackle the problem of learning based engineering sketch generation as a first step towards synthesis and com-
position of solid models with an editable parametric CAD history.

Abstract ric CAD, and refer specifically to composite curves made

up of 2D geometric primitives (e.g. lines, arcs, circles),
Engineering sketches form the 2D basis of paramet- topological information about how the primitives connect
ric Computer-Aided Design (CAD), the foremost model- together, and constraints defined using topology (e.g. co-
ing paradigm for manufactured objects. In this paper we incidence, tangency, symmetry). Engineering sketches can
tackle the problem of learning based engineering sketch be extruded or revolved to generate simple 3D solid bodies
generation as a first step towards synthesis and compo- (Figure 1), and in turn combined using Boolean operations
sition of parametric CAD models. We propose two gen- to build up complex shapes [22]. This workflow is com-
erative models, CurveGen and TurtleGen, for engineering mon to all parametric CAD software and supported by all
sketch generation. Both models generate curve primitives the major solid modeling kernels. Consequently, the abil-
without the need for a sketch constraint solver and explic- ity to generate high quality engineering sketches is a major
itly consider topology for downstream use with constraints enabling technology for the automatic generation of solid
and 3D CAD modeling operations. We find in our percep- models with an editable parametric CAD history.
tual evaluation using human subjects that both CurveGen Engineering sketch generation can be applied in a num-
and TurtleGen produce more realistic engineering sketches ber of CAD workflows. For example, a long sought-after
when compared with the current state-of-the-art for engi- goal is the ability to automatically reverse engineer a para-
neering sketch generation. metric CAD model from noisy 3D scan data [1]. One
way to realize this goal is to generate 2D CAD sketches
from sparse scan data, just like human designers would,
1. Introduction and apply suitable modeling operations to reconstruct the
3D CAD model. Engineering sketch generation can also be
Parametric Computer-Aided Design (CAD) is the fore- applied to auto-completion of user input. The ability to in-
most 3D modeling paradigm used to design manufactured fer repetitive commands based on visual or geometric input
objects from automobile parts, to electronic devices, to fur- could significantly ease user burden when producing com-
niture. Engineering sketches form the 2D basis of paramet- plex engineering sketches. Another sought-after capabil-
ity is the generation of engineering sketches from approxi- 2. Related Work
mate geometry like free-hand drawings. Often referred to as
beautification, a generative model for engineering sketches With the advent of deep learning a vast body of work
could potentially improve user workflows over traditional has focused on generation of novel raster images. By con-
approaches [9]. trast, significantly fewer works have tackled image genera-
Despite recent advances with 2D vector graphic gener- tion using parametric curves such as lines, arcs, or Bézier
ation using data-driven approaches [21, 3, 26], there exists curves. Parametric curves are widely used in 2D applica-
limited research on synthesizing engineering sketches di- tions including vector graphics, technical drawings, floor
rectly. This is a challenging problem because engineering plans, and engineering sketches—the focus of this paper. In
sketches contain disparate 2D geometric primitives coupled this section we review work related to engineering sketch
with topological information on how these primitives are generation and its application to the synthesis and composi-
connected together. The topology information is critical to tion of solid CAD models.
ensure: 1) geometric primitives can be grouped into closed
profile loops and lifted to 3D with modeling operations, and Vector Graphics Vector graphics are used extensively
2) constraints can be correctly defined using the topology, in commercial software to enable the resolution indepen-
for example, two lines can be constrained to intersect at 90◦ dent design of fonts, logos, animations, and illustrations.
if their endpoints are known to coincide. With the availabil- Although a rich body of work has focused on freeform
ity of large-scale engineering sketch and parametric CAD sketches [36, 37], we narrow our focus to structured vec-
datasets [27, 35], we believe a data-driven approach that tor graphics that consider shape topology, connectivity, or
learns a generative model is a promising avenue to explore. hierarchy relevant to engineering sketches.
In this paper, we propose two generative models, Curve- Fonts are described with curves that form closed loop
Gen and TurtleGen, for the task of engineering sketch gen- profiles and can contain interior holes to represent letters of
eration. CurveGen is an adaptation of PolyGen [23], where different genus (e.g. b or B). SVG-VAE [21] is the first work
Transformer networks are used to predict the sketches au- to learn a latent representation of fonts using a sequential
toregressively. While PolyGen generates polygonal mesh generative model for vector graphic output. DeepSVG [3]
vertices and indexed face sets, CurveGen generates ver- considers the structured nature of both fonts and icons with
tices lying on individual curves and indexed hyperedge sets a hierarchical Transformer architecture. BézierGAN [4]
that group the vertices into different curve primitives (line, synthesizes smooth curves using a generative adversarial
arc, circle). This approach generates not only the indi- network [12] and applies it to 2D airfoil profiles. More re-
vidual curve geometry, but also the topology linking dif- cently, Im2Vec [26] leverages differentiable rendering [20]
ferent curves, a vital requirement of engineering sketches. for vector graphic generation trained only on raster images.
TurtleGen is an extension of TurtleGraphics [11], where Common in prior research is the use of Bézier curves.
an autoregressive neural network generates a sequence of Line, arc, and circle primitives are preferred in engineer-
pen_down, pen_draw, pen_up commands, creating a series ing sketches as they are easier to control parametrically,
of closed loops which forms the engineering sketch. We less prone to numerical issues (e.g. when computing in-
find in our perceptual evaluation using human subjects that tersections, offsetting, etc.) and can be lifted to 3D pris-
both CurveGen and TurtleGen produce more realistic engi- matic shapes like planes and cylinders rather than NURBS
neering sketches when compared with the current state-of- surfaces. In contrast to prior work, we pursue engineering
the-art for engineering sketch generation. Importantly, both sketch generation with line, arc, and circle primitives and
CurveGen and TurtleGen can generate geometry and topol- explicitly consider topology for downstream application of
ogy directly without the expense of using a sketch constraint constraints and use with 3D CAD modeling operations.
solver. This paper makes the following contributions:
• We introduce two generative models which tackle the Technical Drawings and Layout Technical drawings
problem of engineering sketch generation without the take the form of 2D projections of 3D CAD models, of-
use of a sketch constraint solver. ten with important details marked by dimensions, notes,
or section views. Han et al. [13] present the SPARE3D
• We use a novel sketch representation with our Curve-
dataset, a collection of technical drawings generated from
Gen model that implicitly encodes the sketch primitive
3D objects with three axis-aligned and one isometric pro-
type based on hyperedge cardinality.
jection. The dataset is primarily aimed at spatial reasoning
• We show quantitative and qualitative results for en- tasks. Pu et al. [25] show how freehand 2D sketches can
gineering sketch generation, including the results of be used to assist in the retrieval of 3D models from large
a perceptual study comparing our generative models object databases. Their system allows designers to make
with a state-of-the-art baseline. freehand sketches approximating the engineering sketches
which would be used in the construction of a 3D model. (86,213) (255,213) Vertex
Egiazarian et al. [5] address the long standing problem of
image vectorization for technical drawings. They predict (86,170) Line
the parameters of lines and Bézier curves with a Trans-
(0,128) (43,128) (128,128)
former based network, then refine them using optimization. Arc
In contrast, our system focuses on the generation of new (86,85)

sketch geometry suitable for use with 3D CAD modeling

Circle
operations. (86,43) (255,43)

An emerging area of research considers learning based

approaches to generative layout for graphic design [18, 38, Figure 2: A hypergraph representation of a sketch consist-
17] and floor plans [24, 15]. Although a different domain ing of three lines (2 vertices), an arc (3 vertices), and a circle
than engineering sketches, these works output high level (4 vertices).
primitives (e.g. rooms in a floor plan) that must be correctly
connected to the overall layout and obey relevant constraints
(e.g. number of bedrooms). Layout problems have simi- eration. The SketchGraphs dataset consists of engineering
larities to engineering sketches where connectivity between sketches made by users of the Onshape CAD software. The
parts of the sketch and relationships, such as parallel, sym- SketchGraphs generative model works by predicting the un-
metric, or perpendicular curves, are critical when designing. derlying geometric constraint graph, and relies on a sketch
constraint solver to determine the final configuration of the
Program Synthesis Another potential approach to engi- sketch. Unlike SketchGraphs we directly predict geome-
neering sketch generation is program synthesis. Recent try and are not reliant on a sketch constraint solver. We
work in this domain leverages neural networks in combi- observe qualitatively that our methods naturally generate
nation with techniques from the programming language lit- geometry which conforms to the regular patterns seen in
erature to generate or infer programs capable of represent- engineering sketches, such as horizontal and vertical lines
ing 2D [7, 8, 28] or 3D [28, 32, 6, 16] geometry. One can and symmetries. The network can be considered to encode
view engineering sketch generation as a program synthe- the constraint information implicitly in the geometric coor-
sis task where the geometry is represented as a sequence dinates, allowing constrained sketches to be recovered in a
of programmatic commands that draw vertices and curves, post-processing step if required. We compare our work di-
constructing the sketch one piece at a time. Future applica- rectly to SketchGraphs and present results that show we are
tions of program synthesis would allow us to model more able to produce more realistic engineering sketch output.
complex operations with programmatic constructs, such as
repetitions (loops), symmetries (constraints), and modifica- 3. Method
tions (refactoring and debugging).
3.1. Engineering Sketch Representation
Reverse engineering Another important field is reverse To create an engineering sketch representation we con-
engineering 3D models from scan data or triangle meshes sider a number of factors. 1) Engineering sketches are com-
[2]. Typically 2D poly-linear profiles are generated by in- posed primarily from lines, arcs, and circles, while ellipses
tersecting the mesh data with a plane, and then line and arc and splines are used less frequently [27, 35]. 2) Engineer-
primitives can be least squares fitted in a way which respects ing sketches must obey constraints such as forming closed
a series of constraints [1]. Generative models which can be profiles, coinciding the end points, and forming 90 degree
conditioned on approximate data provide an approach to- angles. 3) As the constraints remove many degrees of free-
wards automating this part of the reverse engineering pro- dom, engineering sketches are both structured and sparse
cedure. Recent learning-based approaches have tackled the when compared to free-form sketches or vector graphics.
challenge of reverse engineering parametric curves [10, 34], We present two representations of engineering sketches that
paving the way for reconstruction of trimmed surface mod- account for these design considerations: Sketch Hypergraph
els [19, 30, 31, 29]. In contrast to these works, we focus representation (used by CurveGen) and Turtle Graphics rep-
on 2D sketch generation as a building block towards the resentation (used by TurtleGen).
synthesis and composition of solid models, with parametric
CAD history, using common modeling operations. Sketch Hypergraph Representation Under this repre-
sentation, a sketch is represented as a hypergraph G =
Engineering Sketches Most closely related to our work (V, E) where V = {v1 , v2 , . . . , vn } denotes the set of n
is SketchGraphs [27], a recently released dataset and base- vertices, each vertex vi = (xi , yi ) consists of a vertex id i
line generative model for the task of engineering sketch gen- and its corresponding location xi , yi . E denotes a set of hy-
peredges that connects 2 or more vertices to form geometric 3.2.1 CurveGen
primitives. The primitive curve type is implicitly defined
CurveGen is our adaptation of the PolyGen [23] architec-
by the cardinality of the hyperedge: line (2), arc (3), circle
ture applied to engineering sketch generation. CurveGen
(4). Arcs are recovered by finding the circle that uniquely
generates the sketch hypergraph representation directly. As
passes through the 3 vertices. Circles are recovered by a
with the original PolyGen implementation, we break the
least squares fitting due to the over-parameterization by 4
generation of G into two steps based on the chain rule: 1)
points. Figure 2 shows the sketch hypergraph representa-
generate the sketch vertices V , 2) generate the sketch hy-
tion, consisting of 9 vertices and 5 hyperedges. In addition,
peredges E conditioned on the vertices:
the vertices are quantized to a 256×256 grid to bias the gen-
erative model into generating few distinct coordinates by p(G) = p(E|V ) p(V ) ,
learning to produce repeated (x, y) values. | {z } | {z }
Curve Vertex
model model

Turtle Graphics Representation The Turtle Graphics where p(·) are probability distributions. Figure 3 illus-
representation uses a sequence of drawing commands, trates the two step generation process starting with the ver-
which can be executed to form an engineering sketch in the tex model (left) and then the curve model (right). The fi-
hypergraph representation. Intuitively, the turtle represen- nal stage of recovering the curve primitives from the hyper-
tation can be thought of as a sequence of pen-up, pen-move edges is done as a post-process. We use the vertex model di-
actions, which iteratively draws the engineering sketch one rectly from PolyGen with 2D vertex coordinates and adapt
loop at a time. It is used by Ellis et al. [8] to generate com- our curve model from the PolyGen face model to work with
positional and symmetric sketches. Specifically, the draw- 2D curves. We use 3 Transformer blocks in the vertex de-
ing commands of turtle graphics are specified by the follow- coder, curve encoder, and curve decoder Transformer mod-
ing grammar: els [33]. Each block includes a multihead attention with
8 heads, layer normalization and a 2-layer multilayer per-
T urtle ⊢ [ Loop ]
ceptron (MLP) with 512D hidden, and 128D output dimen-
Loop ⊢ LoopStart [Draw]
sions. Dropout with rates 0.2 and 0.5 are applied in each
Draw ⊢ Line | Arc | Circle
block of the vertex and curve models, respectively, right af-
LoopStart ⊢ loopstart(∆)
ter the MLP. The vertex and curve models are both trained
Line ⊢ line(∆)
by negative log likelihood loss against the ground truth data.
Arc ⊢ arc(∆, ∆)
Once the neural network is trained, we perform nucleus
Circle ⊢ circle(∆, ∆, ∆, ∆)
sampling [14] to directly generate samples in the hyper-
∆ ⊢ (int, int)
graph sketch representation. We refer the reader to Nash
et al. [23] for further details.
Here, a T urtle program consists of a sequence of Loops,
each loop consists of a start command LoopStart, followed
by a sequence of [Draw] commands. The pen initially 3.2.2 TurtleGen
starts at (0, 0). The LoopStart command starts a new loop The neural network for generating a program in the Turtle
by lifting the current pen, displacing/teleporting it by ∆ and representation is a sequence generator. The sequence of tur-
putting it back down. The Draw command can be one of tle commands are encoded as a sequence of discrete valued
three primitives Line, Arc, and Circle, each parameterized tokens, where each of the commands loopstart, line, arc,
by a different number of ∆ displacements, which extend the circle, along with two tokens for sequence start and end,
current loop without lifting the pen, displacing it relative are represented as 1-hot vectors. The quantized integer co-
to the current pen location. After a loop is completed, the ordinates are encoded as two 1-hot vectors for x and y each.
pen returns/teleports back to (0, 0). As with the hypergraph For any given sketch in the hypergraph representation, we
representation, ∆ values are quantized to a 256×256 grid. randomize the turtle sequence generation by randomly se-
The loops are ordered so that ones closest to (0, 0), where lecting the loop order, loop’s starting vertex, and the direc-
distance is measured between the loop’s closest vertex to tion of drawing the loop. We discard long sequences over
(0, 0), are drawn first. We provide an example program in 100 turtle commands.
Section A.1 of the Supplementary Material. The neural network is a simple 9-layer Transformer with
512D hidden and 128D output dimensions. The network
3.2. Generative Models
has seven linear branches of input and output, where the
We design and compare two different neural architec- first branch corresponds to the type of command, and the
tures on the task of engineering sketch generation: Curve- remaining six branches correspond to three x and y coor-
Gen and TurtleGen. dinates. Commands with less than three points are padded
 (y,x) (y,x)
PREDICTION

Next
Curve

Vertex Transformer Vertex Transformer Curve Transformer Curve Transformer Curve Transformer Curve Transformer
CONDITIONING

(y,x)

Vertex Model Curve Model

Figure 3: CurveGen, our variant of the PolyGen [23] architecture, first generates sketch vertices (left), and then generates
sketch curves conditioned on the vertices (right).

with zeros. The input branches are concatenated and added curves. We use the official training split and after filtering
with the conventional positional embedding after a linear have 1,106,328 / 39,407 / 39,147 sketches for the train, val-
layer, before they are fed into the Transformer network. The idation, and test sets respectively.
output branches are connected to the Transformer encod-
ing at the previous sequence step. We store the first three 4.2. Experiment Setup
ground-truth turtle steps from the training set as a fixed dic- We train the CurveGen model on the SketchGraphs train-
tionary, which we randomly sample from to precondition ing set. Unlike the ShapeNet dataset used to train the orig-
the auto-regressive model at sampling time. Each sampled inal Polygen model, the SketchGraphs data does not have
sequence is then executed to recover the sketch hypergraph any notion of classes and represents a more challenging,
representation, and define the geometry and topology. but realistic scenario where human annotated labels are un-
available. We train for 2 million iterations with a batch size
4. Results of 8 using 4 Nvidia V100 GPUs. We use the Adam opti-
mizer with learning rate 5e-4, and apply to the curve model
We now present quantitative and qualitative results data a jitter augmentation with the amount sampled from a
on the task of engineering sketch generation, comparing truncated normal distribution with mean 0, variance 0.1 and
the CurveGen and TurtleGen generative models with the truncated by the bounding box of the sketch vertices. Train-
SketchGraphs [27] generative model. ing time takes approximately 48 hours. We save the model
with the lowest overall validation loss for evaluation.
4.1. Data Preparation
We train the TurtleGen model on the same data. To en-
We use the pre-filtered version of the SketchGraphs [27] sure fair comparison, we train the model with a batch size
dataset that contains 9.8 million engineering sketches with of 128 for a total of 0.5 million iterations, which exposes
16 or fewer curves. We remove duplicates from the dataset the model to the same number of training data samples as
to promote data diversity during training, and ensure evalua- CurveGen. Training is done with a single Nvidia V100
tion on unseen data at test time. We consider two sketches to GPU and takes around 48 hours. We use the Adam opti-
be duplicates if they have identical topology and similar ge- mizer with a learning rate of 5e-4, as well as a learning rate
ometry. To detect similar geometry, sketches are uniformly scheduler that decreases the learning rate by a factor of 0.5
scaled and the coordinates are quantized into a 9×9 grid. when the validation loss plateaus. Validation is conducted
Vertices are considered identical if they lie in the same grid once every 500 training iterations. We save the model with
square. The same quantization is also applied to the radii the lowest overall validation loss for evaluation.
of circles and arcs. Lines are considered identical if the end We train the SketchGraphs generative model using the
points are identical. Arcs additionally check for an identical official implementation with and without duplicates. We
quantized radius. Circles check the center point and radius. train for 150 epochs, as in the original paper, on a single
We do not consider a sketch unique if both the topology and Quadro RTX 6000 GPU. Training time takes approximately
geometry match, but curve or vertex order is different. Us- 27 hours. Following the advice of the SketchGraphs au-
ing this approach we find that duplicates make up 87.01% thors, we adjust the learning rate scheduler to reduce the
of the SketchGraphs data. We remove all duplicates, invalid learning rate at epochs 50 and 100. All other hyperparam-
data (e.g. sketches with only points) and omit construction eters and settings follow the official implementation, in-
Table 1: Quantitative sketch generation results. Bits per Vertex and Bits per Sketch are the negative log-likelihood calculated
over the test set; both are not directly comparable and reported separately for the vertex/curve models used in CurveGen.
Unique, Valid, and Novel are calculated over 1000 generated sketches.

Model Parameters Bits per Vertex Bits per Sketch Unique % Valid % Novel %
CurveGen 2,155,542 1.75 / 0.20 176.69 / 30.64 99.90 81.50 90.90
TurtleGen 2,690,310 2.27 54.54 86.40 42.90 80.60
SketchGraphs 18,621,560 - 99.38 76.20 65.80 69.10
SketchGraphs (w/ Duplicates) 18,621,560 - 94.30 58.70 74.00 49.70

cluding the prediction of numerical node features. These sketches from SketchGraphs are commonly due to arcs of
provide improved geometry initialization before the data is near zero length. For the novel metric, it is reasonable to ex-
passed to the OnShape constraint solver. pect generative models to produce some identical sketches
to those in the training data, such as simple circles, rect-
4.3. Quantitative Results angles, and combinations thereof. The low percentage of
4.3.1 Metrics novel sketches generated by SketchGraphs when trained on
the dataset with duplicates suggests that the model memo-
For quantitative evaluation we report the following met- rizes sketches which are duplicated in the training data. Re-
rics. Bits per Vertex is the negative log-likelihood of test moving duplicates from the data helps improve variety in
examples averaged per-vertex and converted from nats to the output. For the remainder of the paper we report results
bits; lower is better. Bits per Sketch is the negative log- from all models trained without duplicates.
likelihood of test examples averaged per-sketch as a whole
in bits; lower is better. For CurveGen we report the bits for 4.4. Perceptual Evaluation
both the vertex and curve models. Unique is the percentage
To understand how engineering sketches generated by
of unique sketches generated within the sample set. We use
each model compare to human designed sketches, we per-
the duplicate detection method described in Section 4.1 to
form a perceptual evaluation using human subjects. In our
find the percentage of unique sketches. A lower value indi-
two-alternative forced choice study, each participant is pre-
cates the model outputs more duplicate sketches. Valid is
sented with one human designed and one generated engi-
the percentage of valid sketches generated. Invalid sketches
neering sketch and asked: “Which sketch is more realis-
include curve fitting failures, curves generated with >4 ver-
tic?". Brief instructions are provided, including an illustra-
tices, or identical vertices within a curve. Novel is the per-
tion of an engineering sketch used in context, similar to Fig-
centage of novel sketches generated that are not identical
ure 1. We evaluate 1000 unique generative sketches from
to sketches in the training set. We again use the duplicate
each model, with a consistent set of 1000 human designed
detection method described in Section 4.1. We evaluate
sketches from the SketchGraphs test set. For each pair of
the bits per vertex/sketch metrics on the withheld test set.
sketches, we log the responses of three human subjects and
All other metrics are evaluated on 1000 generated samples.
We include non-Valid sketches, which often contain valid
curves, when calculating the Unique and Novel metrics. 50% 80%
Indistinguishable from Human Design
Realistic Sketch % vs. Human Designed

70%
40%
60%
Inter-rater Agreement %

4.3.2 Quantitative Comparison

30% 50%
Table 1 shows the results comparing CurveGen and Turtle- 40%
Gen with the SketchGraphs generative model. The param- 20% 30%
eter count for each model is provided for reference. Due to
20%
differences in sketch representation and terms in the neg- 10%
10%
ative log likelihood loss, the bits per vertex and bits per
0% 0%
sketch results are not directly comparable between mod- CurveGen TurtleGen SketchGraphs CurveGen TurtleGen SketchGraphs
els, and only provided here for reference. For the Sketch-
Graphs model, the prediction of numerical node features Figure 4: Results from our perceptual evaluation using
adds an additional term into the loss, giving a higher bits per human subjects to identify the most realistic engineering
sketch value than reported in the SketchGraphs paper. In- sketch. Left: The percentage of generated sketches classed
valid sketches occur most frequently with TurtleGen, where as more realistic than human designed sketches. Right: The
identical vertices within a curve are often predicted. Invalid percentage of inter-rater agreement.
 Human Designed CurveGen TurtleGen SketchGraphs

Figure 5: Qualitative sketch generation results. From left to right: human designed sketches from the SketchGraphs dataset,
randomly selected sketches generated using the CurveGen, TurtleGen, and SketchGraphs generative models.
 Figure 6: Examples of 3D geometry produced by extruding the closed profiles of sketches generated from CurveGen.

use the majority answer. We conduct the study using work- SketchGraphs generative models. We observe that Curve-
ers from Amazon Mechanical Turk. Figure 4, left shows Gen in particular is able to consistently produce sketches
the percentage of generated sketches classed as more real- with closed loops, symmetrical features, perpendicular
istic than human designed sketches; higher values are bet- lines, and parallel lines. We provide additional qualitative
ter. A value of 50% indicates the generated sketches are results in Section A.2 of the supplementary material.
indistinguishable from human design. Figure 4, right shows
the inter-rater agreement calculated as a percentage between Sketch to Solid CAD Models A key motivation of the
each of the three human subjects. A lower value indicates current work is to enable the synthesis and composition of
there is more confusion between the generated and human solid CAD models. In Figure 6 we demonstrate how engi-
designed sketches. The study results show that human sub- neering sketches generated by CurveGen with closed loop
jects find CurveGen output to be the most realistic of the profiles can be lifted into 3D using the extrude modeling
generated engineering sketches. operation in a procedural manner.

4.5. Qualitative Results

Sketch Constraints The geometric output of our gener-
Figure 5 shows human designed sketches from the ative models can be post-processed to apply sketch con-
SketchGraphs dataset beside randomly selected valid straints and build a constraint graph. Figure 7 shows how
sketches generated using the CurveGen, TurtleGen, and the auto-constrain functionality in Autodesk AutoCAD can
enforce parallel and perpendicular lines within a given tol-
erance. The unconstrained output from CurveGen is shown
on the top row, and has a number of lines that are close to
Unconstrained

perpendicular. The auto-constrained output on the bottom

row snaps these lines to perpendicular and establishes con-
straints for further editing.

5. Conclusion
Auto-constrained

In this paper we presented the CurveGen and TurtleGen

generative models for the task of engineering sketch gener-
ation and demonstrated that they produce more realistic out-
put when compared with the current state-of-the art. We be-
lieve engineering sketches are an important building block
on the path to synthesis and composition of solid models
Figure 7: Comparison of sketches generated by CurveGen with an editable parametric CAD history. Promising future
before (top) and after (bottom) applying automatic con- directions include modeling higher-order constructs such as
straints in Autodesk AutoCAD. constraints and repetitions in the underlying design.
References [14] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin
Choi. The curious case of neural text degeneration. Inter-
[1] Pál Benko;, Géza Kós, Tamás Várady, László Andor, and national Conference on Learning Representations (ICLR),
Ralph Martin. Constrained fitting in reverse engineering. 2019.
Computer Aided Geometric Design, 19(3):173–205, 2002.
[15] Ruizhen Hu, Zeyu Huang, Yuhan Tang, Oliver Van Kaick,
[2] Francesco Buonamici, Monica Carfagni, Rocco Furferi, Hao Zhang, and Hui Huang. Graph2plan: Learning floor-
Lapo Governi, Alessandro Lapini, and Yary Volpe. Re- plan generation from layout graphs. ACM Transactions on
verse engineering modeling methods and tools: a survey. Graphics (TOG), 39(4):118–1, 2020.
Computer-Aided Design and Applications, 15(3):443–464, [16] Kacper Kania, Maciej Zi˛eba, and Tomasz Kajdanowicz.
2018. Ucsg-net–unsupervised discovering of constructive solid ge-
[3] Alexandre Carlier, Martin Danelljan, Alexandre Alahi, and ometry tree. arXiv:2006.09102, 2020.
Radu Timofte. DeepSVG: A Hierarchical Generative Net- [17] Hsin-Ying Lee, Weilong Yang, Lu Jiang, Madison Le, Irfan
work for Vector Graphics Animation. In Advances in Neural Essa, Haifeng Gong, and Ming-Hsuan Yang. Neural design
Information Processing Systems (NeurIPS), 2020. network: Graphic layout generation with constraints. Euro-
[4] Wei Chen, Kevin Chiu, and Mark D Fuge. Airfoil Design pean Conference on Computer Vision (ECCV), 2020.
Parameterization and Optimization Using Bézier Generative [18] Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang,
Adversarial Networks. AIAA Journal, 58(11):4723–4735, and Tingfa Xu. Layoutgan: Generating graphic layouts with
2020. wireframe discriminators. In International Conference on
[5] Vage Egiazarian, Oleg Voynov, Alexey Artemov, Denis Learning Representations (ICLR), 2019.
Volkhonskiy, Aleksandr Safin, Maria Taktasheva, Denis [19] Lingxiao Li, Minhyuk Sung, Anastasia Dubrovina, Li Yi,
Zorin, and Evgeny Burnaev. Deep vectorization of techni- and Leonidas J Guibas. Supervised fitting of geometric prim-
cal drawings. In European Conference on Computer Vision, itives to 3d point clouds. In IEEE Conference on Computer
pages 582–598. Springer, 2020. Vision and Pattern Recognition (CVPR), pages 2652–2660,
[6] Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh 2019.
Tenenbaum, and Armando Solar-Lezama. Write, execute, [20] Tzu-Mao Li, Michal Lukáč, Michaël Gharbi, and Jonathan
assess: Program synthesis with a repl. In Advances in Neu- Ragan-Kelley. Differentiable vector graphics rasterization
ral Information Processing Systems (NeurIPS), pages 9169– for editing and learning. ACM Transactions on Graphics
9178, 2019. (TOG), 39(6):1–15, 2020.
[7] Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and [21] Raphael Gontijo Lopes, David Ha, Douglas Eck, and
Joshua B Tenenbaum. Learning to infer graphics programs Jonathon Shlens. A learned representation for scalable vec-
from hand-drawn images. arXiv preprint arXiv:1707.09627, tor graphics. In IEEE Conference on Computer Vision and
2017. Pattern Recognition (CVPR), pages 7930–7939, 2019.
[8] Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sable- [22] H. Masuda. Topological operators and boolean opera-
Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando tions for complex-based nonmanifold geometric models.
Solar-Lezama, and Joshua B Tenenbaum. Dreamcoder: Computer-Aided Design, 25(2):119–129, 1993.
Growing generalizable, interpretable knowledge with wake- [23] Charlie Nash, Yaroslav Ganin, SM Ali Eslami, and Peter
sleep bayesian program learning. arXiv:2006.08381, 2020. Battaglia. Polygen: An autoregressive generative model of
[9] Jakub Fišer, Paul Asente, Stephen Schiller, and Daniel 3d meshes. In International Conference on Machine Learn-
Sýkora. Advanced drawing beautification with shipshape. ing (ICML), pages 7220–7229. PMLR, 2020.
Computers & Graphics, 56:46–58, 2016. [24] Nelson Nauata, Kai-Hung Chang, Chin-Yi Cheng, Greg
[10] Jun Gao, Chengcheng Tang, Vignesh Ganapathi- Mori, and Yasutaka Furukawa. House-gan: Relational gener-
Subramanian, Jiahui Huang, Hao Su, and Leonidas J ative adversarial networks for graph-constrained house lay-
Guibas. Deepspline: Data-driven reconstruction of paramet- out generation. In European Conference on Computer Vision
ric curves and surfaces. arXiv:1901.03781, 2019. (ECCV), pages 162–177. Springer, 2020.
[11] Ron Goldman, Scott Schaefer, and Tao Ju. Turtle geometry [25] Jiantao Pu, Kuiyang Lou, and Karthik Ramani. A 2d sketch-
in computer graphics and computer-aided design. Computer- based user interface for 3d cad model retrieval. Computer-
Aided Design, 36(14):1471–1482, 2004. aided Design - CAD, 2, 01 2005.
[12] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing [26] Pradyumna Reddy, Michael Gharbi, Michal Lukac, and
Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Niloy J Mitra. Im2vec: Synthesizing vector graphics without
Yoshua Bengio. Generative adversarial networks. In Ad- vector supervision. arXiv:2102.02798, 2021.
vances in Neural Information Processing Systems (NeurIPS), [27] Ari Seff, Yaniv Ovadia, Wenda Zhou, and Ryan P Adams.
page 2672–2680, 2014. Sketchgraphs: A large-scale dataset for modeling relational
[13] Wenyu Han, Siyuan Xiang, Chenhui Liu, Ruoyu Wang, and geometry in computer-aided design. arXiv:2007.08506,
Chen Feng. Spare3d: A dataset for spatial reasoning on 2020.
three-view line drawings. In IEEE Conference on Com- [28] Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos
puter Vision and Pattern Recognition (CVPR), pages 14690– Kalogerakis, and Subhransu Maji. Csgnet: Neural shape
14699, 2020. parser for constructive solid geometry. In IEEE Conference
 on Computer Vision and Pattern Recognition (CVPR), pages
5515–5523, 2018.
[29] Gopal Sharma, Difan Liu, Subhransu Maji, Evangelos
Kalogerakis, Siddhartha Chaudhuri, and Radomír Měch.
Parsenet: A parametric surface fitting network for 3d point
clouds. In European Conference on Computer Vision
(ECCV), pages 261–276. Springer, 2020.
[30] Dmitriy Smirnov, Mikhail Bessmeltsev, and Justin Solomon.
Learning manifold patch-based representations of man-made
shapes. arXiv:1906.12337, 2019.
[31] Dmitriy Smirnov, Matthew Fisher, Vladimir G Kim, Richard
Zhang, and Justin Solomon. Deep parametric shape predic-
tions using distance fields. In IEEE Conference on Com-
puter Vision and Pattern Recognition (CVPR), pages 561–
570, 2020.
[32] Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis,
William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu.
Learning to infer and execute 3d shape programs. In Inter-
national Conference on Learning Representations (ICLR),
2019.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko-
reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia
Polosukhin. Attention is all you need. Advances in neural
information processing systems, pages 5998–6008, 2017.
[34] Xiaogang Wang, Yuelang Xu, Kai Xu, Andrea Tagliasacchi,
Bin Zhou, Ali Mahdavi-Amiri, and Hao Zhang. Pie-net:
Parametric inference of point cloud edges. In Advances in
Neural Information Processing Systems (NeurIPS), 2020.
[35] Karl D. D. Willis, Yewen Pu, Jieliang Luo, Hang Chu,
Tao Du, Joseph G. Lambourne, Armando Solar-Lezama,
and Wojciech Matusik. Fusion 360 gallery: A dataset
and environment for programmatic cad reconstruction.
arXiv:2010.02392, 2020.
[36] Peng Xu. Deep learning for free-hand sketch: A survey.
arXiv:2001.02600, 2020.
[37] Chuan Yan, David Vanderhaeghe, and Yotam Gingold. A
benchmark for rough sketch cleanup. ACM Trans. Graph.,
39(6), Nov. 2020.
[38] Xinru Zheng, Xiaotian Qiao, Ying Cao, and Rynson WH
Lau. Content-aware generative modeling of graphic design
layouts. ACM Transactions on Graphics (TOG), 38(4):1–15,
2019.