Skip to content

[UI] Jupyter Notebook export in Frontend #343

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 20 commits into from
Jul 1, 2023
Merged

[UI] Jupyter Notebook export in Frontend #343

merged 20 commits into from
Jul 1, 2023

Conversation

senwang86
Copy link
Collaborator

@senwang86 senwang86 commented Jun 27, 2023 •
edited
Loading

To export the Codepod repo as one Jupyter Notebook

  • In summary,
    • After discussion, the export functionality is implemented in ui server, rather than API server.
    • Add new button "Jupyter Notebook" in the sidebar
    • Build the Jupyter notebook by parsing Pod object, ignoring the "result" and "stdout" fields; append the scope structure at the head of each cell.
    • Export cells in geographical order(top-bottom, left-right), following PR #240
    • Convert Remirror editor content to markdown, ref
  • Test
    • 2 pods have 2 scopes, 1 pod w/o scope
    • The exported notebook is rendered by Colab
  • Bug to fix [e5b9d1a]
    • pod.content is not able to split by newline character
  • Follow up
    • More types of repos should be tested
    • Should test in other platforms
    • Add support for rich-text cell

exportJupyterNotebook

Note, the export functionality requires the Amazon S3 bucket setup.

@senwang86 senwang86 marked this pull request as ready for review June 27, 2023 22:07
@forrestbao
Copy link
Collaborator

Cells in Jupyter notebook are linear whereas Codepod is hierarchical and 2D. In what order do we map 2D, hierarchical pods and scopes to linear cells?

@senwang86
Copy link
Collaborator Author

senwang86 commented Jun 28, 2023 •
edited
Loading

Cells in Jupyter notebook are linear whereas Codepod is hierarchical and 2D. In what order do we map 2D, hierarchical pods and scopes to linear cells?

If the pods/scopes are connected by "edges", we can order the pods by the flow of edges; otherwise, pods are not ordered.

@forrestbao
Copy link
Collaborator

forrestbao commented Jun 29, 2023 •
edited
Loading

Cells in Jupyter notebook are linear whereas Codepod is hierarchical and 2D. In what order do we map 2D, hierarchical pods and scopes to linear cells?

If the pods/scopes are connected by "edges", we can order the pods by the flow of edges; otherwise, pods are not ordered.

Un-ordered is not a good idea as a notebook, in many cases, is designated as executable top-down.
How about this simple strategy?

  1. Members of a scope should be placed above a pod in exported Jupyter notebook.
  2. Within a scope, including the global scope, the order of pods after exporting is: top-down and then left-right? In your screenshot at the top, pod_1 should be exported first, then pod_2 and finally pod_wo_scope?
    @lihebi

@lihebi
Copy link
Collaborator

lihebi commented Jun 30, 2023

  1. top-down and then left-right

Yes, this is what I have in my mind.

  1. Members of a scope should be placed above a pod

Say we have this:

pod 1
scope A
  pod A.1
  pod A.2
pod 2

Do you mean that we should put pod A.1 above pod 1? That doesn't make too much sense to me.

Or we could use markdown sections with different levels of titles to indicate the scope. Jupyter has a ToC section to display the levels. However, when we export rich text nodes to markdown cells, they will mess up with the ToC and levels.

image

@senwang86 senwang86 changed the title [WIP] Jupyter Notebook export [UI] Jupyter Notebook export in Frontend Jun 30, 2023
lihebi
lihebi reviewed Jul 1, 2023
View reviewed changes
Copy link
Collaborator

@lihebi lihebi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Sen! Overall looks great. A couple of minor comments.

ui/src/lib/store/podSlice.tsx Outdated
return;
}
pod.richContent = richContent;
pod.dirty = true;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to set pod.dirty=true here, because pod.richContent doesn't need to be saved to DB. Also, this doesn't change the behavior, as pod.dirty is already set to true in setPodContent.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 1ceac12

ui/src/components/Sidebar.tsx
Comment on lines +667 to +668
nbformat: 4,
nbformat_minor: 0,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest nbformat version seems to be 5.9.0. Should we use that instead?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Colab still uses nbformat:4 in their exported notebook, and it doesn't support the 5.9.0 yet.

Screen Shot 2023-06-30 at 9 19 24 PM

Copy link
Collaborator

@forrestbao forrestbao Aug 2, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the key names we use, I think we (along with CoLab and DeepNote) are already using NBFormat 5.9 despite that the value of #/nbformat is 4. See here and pay attention to the new key names, that we are using, under "Changed in version nbformat: 4.0"

@forrestbao
Copy link
Collaborator

  1. Members of a scope should be placed above a pod

Say we have this:

pod 1
scope A
  pod A.1
  pod A.2
pod 2

Do you mean that we should put pod A.1 above pod 1? That doesn't make too much sense to me.

Sorry, I made a typo.

Say we have this :

pod 1
scope A
   pod A.1
   pod A.2
   scope A.B
      pod A.B.1
      pod A.B.2
 pod 2

then pod A.B.1 and pod A.B.2 should be above pod A.1 and pod A.2 in the exported Jupyter notebook. Makes sense?

@senwang86
Copy link
Collaborator Author

  1. Members of a scope should be placed above a pod

Say we have this:

pod 1
scope A
  pod A.1
  pod A.2
pod 2

Do you mean that we should put pod A.1 above pod 1? That doesn't make too much sense to me.

Sorry, I made a typo.

Say we have this :

pod 1
scope A
   pod A.1
   pod A.2
   scope A.B
      pod A.B.1
      pod A.B.2
 pod 2

then pod A.B.1 and pod A.B.2 should be above pod A.1 and pod A.2 in the exported Jupyter notebook. Makes sense?

exportJupyterNotebook

Updated in 1ceac12.

lihebi
lihebi approved these changes Jul 1, 2023
View reviewed changes
Copy link
Collaborator

@lihebi lihebi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, thanks! Merging.

@lihebi lihebi merged commit 012ecc8 into codepod-io:main Jul 1, 2023
li-xin-yi pushed a commit to li-xin-yi/codepod-1 that referenced this pull request Jul 4, 2023
@senwang86 @li-xin-yi
feat: [UI] Jupyter Notebook export in Frontend (codepod-io#343)
5f2fd0b
@forrestbao forrestbao mentioned this pull request Jul 20, 2023
Closed
@senwang86 senwang86 mentioned this pull request Jul 20, 2023
Open
@forrestbao forrestbao mentioned this pull request Aug 2, 2023
Closed
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@forrestbao forrestbao forrestbao left review comments

@lihebi lihebi lihebi approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@senwang86 @forrestbao @lihebi