Skip to content

[BUG] Malformed UTF-8 Characters in Query Parameters starting from v3.5.6 #854

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
lamminsalo opened this issue May 6, 2025 · 4 comments
Open

[BUG] Malformed UTF-8 Characters in Query Parameters starting from v3.5.6 #854

lamminsalo opened this issue May 6, 2025 · 4 comments

Comments

@lamminsalo
Copy link

lamminsalo commented May 6, 2025 •
edited
Loading

Bug Report: Malformed UTF-8 Characters in Query Parameters in v3.5.6

Description
After v3.5.6, UTF-8 characters (e.g., Nordic characters like äöå) in query parameters are being malformed into � in deployed environments. This issue only occurs when navigating within the application to a cached page with UTF-8 encoded query parameters. Reverting to v3.5.5 resolves the issue.

The issue is specific to deployed environments and is not reproducible locally, making debugging and reproduction more challenging.

Expected Behavior
When navigating to a page with UTF-8 encoded URL parameters, the request should return:

  • HTTP 304: "Not Modified" with the query parameter q=äää.

Example:

GET /search?q=äää
Response: 304 Not Modified
Query: q=äää

Actual Behavior
Instead of returning the cached page, navigating to a cached page results in a redirect along with malformed query parameters:

  • HTTP 308: Permanent redirect with query parameters encoded incorrectly.

Malformed Example:

GET /search?q=äää
Response: 308 Permanent Redirect
To: /_next/data/[hash]/search.json?q=%C3%A4%C3%A4%C3%A4
Actual Query: q=��� (Malformed)

Steps to Reproduce

  1. Deploy the reproduction example linked below.
  2. Navigate to /search?q=äöå€ using the bottom button (direct navigation from URL works)
  3. Observe the malformed query parameters after a redirect.

Minimal Reproduction

Discord Discussion
For additional context, see the ongoing thread on Discord:
https://discord.com/channels/1283128968140161065/1286094576788177059/1367919144694972569

Reproduction Environment

  • Version: opennext v3.6.0

Manually tested that v3.5.6 is the breaking update.

EDIT: Added a more minimal reproduction repo and deployed it. Links and steps updated.
@sommeeeer
Copy link
Contributor

As stated in the Discord thread the issue is the Location header returned here:

opennextjs-aws/packages/open-next/src/core/routing/matcher.ts

Line 343 in 9411b7a

Location: internalEvent.url,

It should be encoded, which it is not >= 3.5.6. We need to figure out what in this commit broke it.

@sommeeeer
Copy link
Contributor

It seems to be this function:

opennextjs-aws/packages/open-next/src/core/routing/util.ts

Lines 126 to 137 in 9411b7a

export function convertToQueryString(query: Record<string, string | string[]>) {
const queryStrings: string[] = [];
Object.entries(query).forEach(([key, value]) => {
if (Array.isArray(value)) {
value.forEach((entry) => queryStrings.push(`${key}=${entry}`));
} else {
queryStrings.push(`${key}=${value}`);
}
});
return queryStrings.length > 0 ? `?${queryStrings.join("&")}` : "";
}

In the versions prior to < 3.5.6 it was using URLSearchParams which will automatically encode the query parameters. What I think we need to do now is actually wrap encodeURIComponent() around the value and entry in that function, however im not sure if that would possibly break 533 and 537 in the opennextjs-cloudflare repo.

@conico974
Copy link
Contributor

Yeah, you can't just revert to URLSearchParams without breaking those. BTW this was not only broken in cloudflare

@sommeeeer
Copy link
Contributor

sommeeeer commented May 6, 2025 •
edited
Loading

This PR had this line:

opennextjs-aws/packages/open-next/src/core/routing/matcher.ts

Line 208 in 77d87e7

const encodePlusQueryString = queryString.replaceAll("+", "%20");

If I add it back and also have this as convertToQueryString():

export function convertToQueryString(query: Record<string, string | string[]>) {
  const searchParams = new URLSearchParams();
  Object.entries(query).forEach(([key, value]) => {
    if (Array.isArray(value)) {
      value.forEach((entry) => searchParams.append(key, decodeURIComponent(entry)));
    } else {
      searchParams.append(key, decodeURIComponent(value));
    }
  });

  const queryString = searchParams.toString();
  return queryString.length > 0 ? `?${queryString}` : "";
}

E2E runs fine, but one unit test will fail:

FAIL  packages/tests-unit/tests/core/routing/util.test.ts > convertToQueryString > should respect existing query encoding                              
                                                                                                                                                  
Expected: "?key=value%201&key=value2+something+else&another=value3"                                                                               
Received: "?key=value+1&key=value2%2Bsomething%2Belse&another=value3"

However, if you have a redirect with this:

const nextConfig: NextConfig = {
  redirects: async () => [
    {
      source: '/baby',
      destination: '/foo?bar=value%201&bar=value2+something+else&baz=value3',
      permanent: false,
    },
  ],
};

it will give you a Location with http://localhost:3000/foo?bar=value+1&bar=value2+something+else&baz=value3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@conico974 @lamminsalo @sommeeeer