Yes I tried to trace the model before. At the moment it seems like the timm module is not 100% compatible.
Will look into it in the future.

Hi, did you trace the whole model (encoder+decoder)? And what's your problem? Maybe we can have a discussion. In addition, I have a question and hope you can give some idea. [x_transformers] provides both encoder and decoder, why did you use the encoder from [timm] and the decoder from [x_transformers]. Is there any special reason?

Hi, I've traced the model for my deployment exercise.
First of all is the encoder
`

encoder.eval()
 img = cv2.imread('path_to_my_example_image') #image size is 464 x 112 
 dummy_img = test_transform(image=img)['image'][:1].unsqueeze(0) #shape now is [1, 1, 112, 464]

 with torch.no_grad():
     torch.onnx.export(
         encoder, 
         dummy_img, 
         f = "encoder.onnx",
         opset_version=16, 
         input_names=['input_image'], 
         output_names=['output_context'],
         dynamic_axes={
             'context': {0: 'batch_size', 1: 'channel', 2: 'height', 3: 'width'},
             'output': {0: 'batch_size', 1: 'output_context'}
             },
         export_params=True,
         verbose=True
     )

Everything is ok for the encoder. But when it comes to the decoder, here's the code:

decoder.eval()

dummy_context = encoder(dummy_img)
dummy_tgt_seq = torch.rand(1, 512)

dummy_input = {
    'context': dummy_context,
    'tgt_seq': dummy_tgt_seq
}

with torch.no_grad():
    torch.onnx.export(
        decoder,
        args = (
            dummy_input["tgt_seq"],
            dummy_input["context"]
        ),
        f = "decoder.onnx",
        opset_version=16,
        input_names=['input_seq', 'input_context'],
        output_names=['output_seq'],
        dynamic_axes={
            'input_context': {0: 'batch_size', 1: 'sequence', '2': 'encoded_context'},
            'output_seq': {0: 'batch_size', 1: 'output_seq'}
            },
        # export_params=True,
        verbose=True
    )

I've got the error like this:

Traceback (most recent call last):
  File "d:\service-ml-api-server\flask_app\Im2Tex\pix2tex\convert_onnxmodel.py", line 105, in <module>
    torch.onnx.export(
  File "C:\Users\Admins\AppData\Local\Programs\Python\Python38\lib\site-packages\torch-1.12.1-py3.8-win-amd64.egg\torch\onnx\__init__.py", line 350, in export
    return utils.export(
  File "C:\Users\Admins\AppData\Local\Programs\Python\Python38\lib\site-packages\torch-1.12.1-py3.8-win-amd64.egg\torch\onnx\utils.py", line 163, in export
    _export(
  File "C:\Users\Admins\AppData\Local\Programs\Python\Python38\lib\site-packages\torch-1.12.1-py3.8-win-amd64.egg\torch\onnx\utils.py", line 1074, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "C:\Users\Admins\AppData\Local\Programs\Python\Python38\lib\site-packages\torch-1.12.1-py3.8-win-amd64.egg\torch\onnx\utils.py", line 727, in _model_to_graph
    graph, params, torch_out, module = _create_jit_graph(model, args)
  File "C:\Users\Admins\AppData\Local\Programs\Python\Python38\lib\site-packages\torch-1.12.1-py3.8-win-amd64.egg\torch\onnx\utils.py", line 602, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "C:\Users\Admins\AppData\Local\Programs\Python\Python38\lib\site-packages\torch-1.12.1-py3.8-win-amd64.egg\torch\onnx\utils.py", line 517, in _trace_and_get_graph_from_model
    trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
  File "C:\Users\Admins\AppData\Local\Programs\Python\Python38\lib\site-packages\torch-1.12.1-py3.8-win-amd64.egg\torch\jit\_trace.py", line 1175, in _get_trace_graph
    outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "C:\Users\Admins\AppData\Local\Programs\Python\Python38\lib\site-packages\torch-1.12.1-py3.8-win-amd64.egg\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\Admins\AppData\Local\Programs\Python\Python38\lib\site-packages\torch-1.12.1-py3.8-win-amd64.egg\torch\jit\_trace.py", line 127, in forward
    graph, out = torch._C._create_graph_by_tracing(
  File "C:\Users\Admins\AppData\Local\Programs\Python\Python38\lib\site-packages\torch-1.12.1-py3.8-win-amd64.egg\torch\jit\_trace.py", line 118, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "C:\Users\Admins\AppData\Local\Programs\Python\Python38\lib\site-packages\torch-1.12.1-py3.8-win-amd64.egg\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\Admins\AppData\Local\Programs\Python\Python38\lib\site-packages\torch-1.12.1-py3.8-win-amd64.egg\torch\nn\modules\module.py", line 1118, in _slow_forward
    result = self.forward(*input, **kwargs)

TypeError: forward() takes 2 positional arguments but 3 were given

Is my solution got something went wrong or i have to do this in another way? Thanks.

Convert to onnx model #2

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions