fix: add backfill migration for routing response denormalized tables #21474

eunjae-lee · 2025-05-22T14:27:31Z

What does this PR do?

This PR adds backfill migration for the RoutingFormResponseDenormalized and RoutingFormResponseField tables.

Mandatory Tasks (DO NOT REMOVE)

I have self-reviewed the code (A decent size PR without self-review might be rejected).
N/A - I have updated the developer docs in /docs if this PR makes changes that would require a documentation change. If N/A, write N/A here and check the checkbox.
I confirm automated tests are in place that prove my fix is effective or that my feature works.

vercel · 2025-05-22T14:27:44Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

2 Skipped Deployments

Name	Status	Preview	Comments	Updated (UTC)
cal	⬜️ Ignored (Inspect)	Visit Preview		Jul 28, 2025 1:15pm
cal-eu	⬜️ Ignored (Inspect)	Visit Preview		Jul 28, 2025 1:15pm

github-actions · 2025-06-07T00:24:51Z

This PR is being marked as stale due to inactivity.

eunjae-lee · 2025-06-23T09:09:25Z

Note to myself:

To skip these migrations:

insert into
    _prisma_migrations (
        id,
        migration_name,
        checksum,
        started_at,
        finished_at,
        applied_steps_count
    )
values (
        gen_random_uuid(),
        '20250618093846_routing_form_response_denormalize_backfill',
        '<INSERT-CHECKSUM1>',
        now(),
        now(),
        1
    ),
    (
        gen_random_uuid(),
        '20250618093923_routing_form_response_field_backfill',
        '<INSERT-CHECKSUM2>',
        now(),
        now(),
        1
    );

CHECKSUM1 ↓

sha256sum packages/prisma/migrations/20250618093846_routing_form_response_denormalize_backfill/migration.sql

CHECKSUM2 ↓

sha256sum packages/prisma/migrations/20250618093923_routing_form_response_field_backfill/migration.sql

github-actions · 2025-07-08T00:25:50Z

This PR is being marked as stale due to inactivity.

emrysal

Will be released as part of tonights release.

coderabbitai · 2025-07-28T11:35:27Z

Caution

Review failed

The head commit changed during the review from a989026 to e3fdfb2.

Walkthrough

This pull request adds two standalone SQL migration scripts for backfilling denormalized data related to routing form responses. The first script performs a batch upsert on the "RoutingFormResponseDenormalized" table by processing records in chunks of 1000 from the "App_RoutingForms_FormResponse" table, joining with related tables, and inserting or updating records as needed while logging progress and throttling load. The second script processes the same source table in chunks, identifying routing form response fields requiring updates by comparing existing denormalized data with expected values derived from JSON responses and field definitions, then calls a reprocessing function for each such response, handling errors and logging progress similarly.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~15–20 minutes

Possibly related PRs

fix: routing form response denormalized triggers #22518: This PR sets up triggers and functions to maintain the "RoutingFormResponseDenormalized" table in real-time, complementing the batch backfill migration script in the current PR that operates on the same table but through batch processing.

✨ Finishing Touches

🧪 Generate unit tests

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch fix/backfill-denorm-tables

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Explain this complex logic.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai explain this code block.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and explain its main purpose.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
@coderabbitai generate unit tests to generate unit tests for this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

coderabbitai

Actionable comments posted: 4

🧹 Nitpick comments (3)

packages/prisma/migrations/20250618093846_routing_form_response_denormalize_backfill/migration.sql (2)
4-8: Consider deriving start_id from the data instead of hard-coding 1.
If the smallest App_RoutingForms_FormResponse.id is greater than 1, the first chunk does an unnecessary DELETE on a range that cannot hold rows, slightly inflating processed_count and wasting time.
-start_id INTEGER := 1;      -- Starting ID
+SELECT COALESCE(MIN(id), 1) INTO start_id FROM "App_RoutingForms_FormResponse";
23-29: processed_count is over-counted when gaps exist in IDs.
processed_count := processed_count + chunk_size; assumes every BETWEEN range contains exactly chunk_size responses, which is false when IDs are sparse or at the tail of the table. Prefer COUNT(*) of the actually inserted rows (or use GET DIAGNOSTICS) to keep the progress log trustworthy.
packages/prisma/migrations/20250618093923_routing_form_response_field_backfill/migration.sql (1)
30-33: WHEN OTHERS swallows useful detail; include SQLSTATE.
Capturing SQLSTATE helps triage repeated failures.
-RAISE WARNING 'Failed to process responseId %: %', response_record.id, SQLERRM;
+RAISE WARNING 'Failed responseId % (SQLSTATE %, message %)', 
+              response_record.id, SQLSTATE, SQLERRM;

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between a745d8a and ecce399.

📒 Files selected for processing (2)

packages/prisma/migrations/20250618093846_routing_form_response_denormalize_backfill/migration.sql (1 hunks)
packages/prisma/migrations/20250618093923_routing_form_response_field_backfill/migration.sql (1 hunks)

🔇 Additional comments (1)

packages/prisma/migrations/20250618093846_routing_form_response_denormalize_backfill/migration.sql (1)

58-99: LEFT JOINs can duplicate the target id, risking unique_violation.
If a booking has multiple Tracking rows, the SELECT will return >1 row for the same r.id, breaking the presumed PK on RoutingFormResponseDenormalized.id. Add DISTINCT ON (r.id) or aggregate the tracking columns (e.g., MAX) before insertion.

...es/prisma/migrations/20250618093846_routing_form_response_denormalize_backfill/migration.sql

packages/prisma/migrations/20250618093923_routing_form_response_field_backfill/migration.sql

graphite-app · 2025-07-28T11:40:11Z

Graphite Automations

"Add ready-for-e2e label" took an action on this PR • (07/28/25)

1 label was added to this PR based on Keith Williams's automation.

"Add consumer team as reviewer" took an action on this PR • (07/28/25)

1 reviewer was added to this PR based on Keith Williams's automation.

coderabbitai

Actionable comments posted: 1

♻️ Duplicate comments (3)

packages/prisma/migrations/20250618093846_routing_form_response_denormalize_backfill/migration.sql (2)

1-26: Entire backfill still runs in one monolithic transaction

The DO block is a single implicit transaction, so every DELETE/UPSERT lock it takes is held until the very end. On large tables this can block production traffic, create bloat, and make the migration nearly un-cancelable. Please commit per chunk (e.g. move the chunk logic to a helper function invoked with PERFORM …; COMMIT;) or use autonomous transactions (CALL) instead.

194-195: pg_sleep executed while locks are held

Because no commit happens before the sleep, row/share locks stay active for the whole second, defeating the throttling purpose. Move the sleep after a per-chunk commit or drop it.

packages/prisma/migrations/20250618093923_routing_form_response_field_backfill/migration.sql (1)

124-126: pg_sleep holds locks for the entire pause

Without a preceding commit the sleep simply lengthens the lock window. Commit (or split into separate transactions) before calling pg_sleep, or delete the sleep.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ecce399 and e59fc85.

📒 Files selected for processing (2)

packages/prisma/migrations/20250618093846_routing_form_response_denormalize_backfill/migration.sql (1 hunks)
packages/prisma/migrations/20250618093923_routing_form_response_field_backfill/migration.sql (1 hunks)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

GitHub Check: Install dependencies / Yarn install & cache

packages/prisma/migrations/20250618093923_routing_form_response_field_backfill/migration.sql

eunjae-lee · 2025-07-28T13:18:34Z

@coderabbitai review

coderabbitai · 2025-07-28T13:18:41Z

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

emrysal

LGTM, prod is prepared. 👍

fix: add backfill migration for routing response denormalized tables

07d1345

github-actions bot added the ❗️ migrations contains migration files label May 22, 2025

keithwillcode added consumer core area: core, team members only labels May 22, 2025

vercel bot deployed to Preview – dev May 22, 2025 14:49 View deployment

update script

30c18dd

vercel bot deployed to Preview – dev May 23, 2025 10:02 View deployment

github-actions bot added the Stale label Jun 7, 2025

implement select type

18c5299

vercel bot deployed to Preview – dev June 17, 2025 15:39 View deployment

improve number handling

ec20422

vercel bot deployed to Preview – dev June 17, 2025 16:21 View deployment

github-actions bot removed the Stale label Jun 18, 2025

Merge branch 'main' into fix/backfill-denorm-tables

2d9f3e8

vercel bot deployed to Preview – dev June 18, 2025 09:09 View deployment

eunjae-lee added 8 commits June 18, 2025 11:18

clean up queries

452a14a

rename

1d45197

handle zero record case

bba1431

revert

94d26bd

rename

b59d348

fix division problem

69d08ee

log clean up

a29ab43

force adding new rows after removing existing rows

ccef367

vercel bot deployed to Preview – dev June 18, 2025 10:45 View deployment

eunjae-lee force-pushed the fix/backfill-denorm-tables branch from a48b1c5 to ccef367 Compare June 18, 2025 14:58

eunjae-lee added 2 commits June 20, 2025 16:57

Merge branch 'main' into fix/backfill-denorm-tables

e9728dc

Merge branch 'main' into fix/backfill-denorm-tables

ecce399

vercel bot deployed to Preview – dev June 23, 2025 09:56 View deployment

github-actions bot added the Stale label Jul 8, 2025

emrysal previously approved these changes Jul 28, 2025

View reviewed changes

eunjae-lee marked this pull request as ready for review July 28, 2025 11:35

eunjae-lee requested a review from a team as a code owner July 28, 2025 11:35

graphite-app bot added the ready-for-e2e label Jul 28, 2025

graphite-app bot requested a review from a team July 28, 2025 11:35

dosubot bot added the routing-forms area: routing forms, routing, forms label Jul 28, 2025

coderabbitai bot reviewed Jul 28, 2025

View reviewed changes

eunjae-lee marked this pull request as draft July 28, 2025 11:48

eunjae-lee requested a review from emrysal July 28, 2025 11:52

eunjae-lee marked this pull request as ready for review July 28, 2025 12:12

eunjae-lee dismissed emrysal’s stale review via e59fc85 July 28, 2025 12:39

coderabbitai bot reviewed Jul 28, 2025

View reviewed changes

packages/prisma/migrations/20250618093923_routing_form_response_field_backfill/migration.sql Outdated Show resolved Hide resolved

backfill only missing or wrong data

a989026

eunjae-lee force-pushed the fix/backfill-denorm-tables branch from e59fc85 to a989026 Compare July 28, 2025 13:09

eunjae-lee added 2 commits July 28, 2025 15:09

Merge branch 'main' into fix/backfill-denorm-tables

e3fdfb2

remove sleep

afabf5e

vercel bot deployed to Preview – dev July 28, 2025 13:52 View deployment

github-actions bot removed the Stale label Jul 30, 2025

emrysal approved these changes Aug 5, 2025

View reviewed changes

emrysal merged commit 006aecb into main Aug 5, 2025
87 of 92 checks passed

emrysal deleted the fix/backfill-denorm-tables branch August 5, 2025 00:40

fix: add backfill migration for routing response denormalized tables #21474

fix: add backfill migration for routing response denormalized tables #21474

Uh oh!

Conversation

eunjae-lee commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Mandatory Tasks (DO NOT REMOVE)

Uh oh!

vercel bot commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 7, 2025

Uh oh!

eunjae-lee commented Jun 23, 2025

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

emrysal left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review failed

Walkthrough

Estimated code review effort

Possibly related PRs

Chat

Support

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

graphite-app bot commented Jul 28, 2025

Graphite Automations

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

eunjae-lee commented Jul 28, 2025

Uh oh!

coderabbitai bot commented Jul 28, 2025

Uh oh!

emrysal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eunjae-lee commented May 22, 2025 •

edited

Loading

vercel bot commented May 22, 2025 •

edited

Loading

coderabbitai bot commented Jul 28, 2025 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)