Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove limit in IdListBuilder #39603

Open
wants to merge 1 commit into
base: 2.4-develop
Choose a base branch
from

Conversation

sanderjongsma
Copy link
Contributor

@sanderjongsma sanderjongsma commented Feb 5, 2025

There should be no limit on the IdListBuilder since it is chunked in: vendor/magento/module-sales/Model/ResourceModel/Grid.php:

    public function refreshBySchedule()
    {
        $lastUpdatedAt = null;
        $notSyncedIds = $this->notSyncedDataProvider->getIds($this->mainTableName, $this->gridTableName);
        foreach (array_chunk($notSyncedIds, self::BATCH_SIZE) as $bunch) {
            $select = $this->getGridOriginSelect()->where($this->mainTableName . '.entity_id IN (?)', $bunch);

Description (*)

The limit in the select is removed so it will fetch all results and later it will be chuncked.

Related Pull Requests

Manual testing scenarios (*)

  1. Make sure you have more than 100 orders
  2. Make sure your sales_order_grid is also filled with this orders
  3. Truncate sales_order_grid
  4. Make sure async indexing of grids is true
  5. run magerun2 sys:cron:run sales_grid_order_async_insert

vendor/magento/module-sales/Model/ResourceModel/Grid.php

    public function refreshBySchedule()
    {
        $lastUpdatedAt = null;
        $notSyncedIds = $this->notSyncedDataProvider->getIds($this->mainTableName, $this->gridTableName);
        foreach (array_chunk($notSyncedIds, self::BATCH_SIZE) as $bunch) {
            $select = $this->getGridOriginSelect()->where($this->mainTableName . '.entity_id IN (?)', $bunch);

The code above will be run and fetch $notSyncedIds and chunck it in batches. The provider used is: vendor/magento/module-sales/Model/ResourceModel/Provider/UpdatedIdListProvider.php and does this:

    public function getIds($mainTableName, $gridTableName)
    {
        $mainTableName = $this->resourceConnection->getTableName($mainTableName);
        $gridTableName = $this->resourceConnection->getTableName($gridTableName);
        $select = $this->idListQueryBuilder->build($mainTableName, $gridTableName);
        return $this->getConnection()->fetchAll($select, [], \Zend_Db::FETCH_COLUMN);

and builds a select with the idListQueryBuilder:

public function build(string $mainTableName, string $gridTableName): Select
    {
        $select = $this->getConnection()->select()
            ->from(['main_table' => $mainTableName], ['main_table.entity_id'])
            ->joinLeft(
                ['grid_table' => $this->resourceConnection->getTableName($gridTableName)],
                'main_table.entity_id = grid_table.entity_id',
                []
            );

        $select->where('grid_table.entity_id IS NULL');
        $select->limit(Grid::BATCH_SIZE); // should be removed
        foreach ($this->additionalGridTables as $table) {
            $select->joinLeft(
                [$table => $table],
                sprintf(
                    '%s.%s = %s.%s',
                    'main_table',
                    'entity_id',
                    $table,
                    'entity_id'
                ),
                []
            )
                ->where($table . '.entity_id IS NULL');
        }
        return $select;
    }

In here there is limit: $select->limit(Grid::BATCH_SIZE); the same as the chunk batch size in refreshBySchedule. This looks like a bug to me, because you will never get more items than 100. Now this will probably only occur when the sales_order_grid is truncated or more than 100 orders are placed between the cron execution.

Questions or comments

Eventually the grid will be filled, because the cron is run every minute. But it takes a lot of time if you have a lot of orders.

Contribution checklist (*)

  • Pull request has a meaningful description of its purpose
  • All commits are accompanied by meaningful commit messages
  • All new or changed code is covered with unit/integration tests (if applicable)
  • README.md files for modified modules are updated and included in the pull request if any README.md predefined sections require an update
  • All automated tests passed successfully (all builds are green)

Resolved issues:

  1. resolves [Issue] Remove limit in IdListBuilder #39621: Remove limit in IdListBuilder

Copy link

m2-assistant bot commented Feb 5, 2025

Hi @sanderjongsma. Thank you for your contribution!
Here are some useful tips on how you can test your changes using Magento test environment.
❗ Automated tests can be triggered manually with an appropriate comment:

  • @magento run all tests - run or re-run all required tests against the PR changes
  • @magento run <test-build(s)> - run or re-run specific test build(s)
    For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names.

Allowed build names are:
  1. Database Compare
  2. Functional Tests CE
  3. Functional Tests EE
  4. Functional Tests B2B
  5. Integration Tests
  6. Magento Health Index
  7. Sample Data Tests CE
  8. Sample Data Tests EE
  9. Sample Data Tests B2B
  10. Static Tests
  11. Unit Tests
  12. WebAPI Tests
  13. Semantic Version Checker

You can find more information about the builds here
ℹ️ Run only required test builds during development. Run all test builds before sending your pull request for review.


For more details, review the Code Contributions documentation.
Join Magento Community Engineering Slack and ask your questions in #github channel.

@sanderjongsma
Copy link
Contributor Author

@magento run all tests

@engcom-Hotel
Copy link
Contributor

@magento create issue

@engcom-Hotel engcom-Hotel added the Priority: P3 May be fixed according to the position in the backlog. label Feb 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Priority: P3 May be fixed according to the position in the backlog.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Issue] Remove limit in IdListBuilder
2 participants