Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LINSTOR creates volumes larger than requested size #421

Open
kvaps opened this issue Oct 11, 2024 · 7 comments
Open

LINSTOR creates volumes larger than requested size #421

kvaps opened this issue Oct 11, 2024 · 7 comments

Comments

@kvaps
Copy link

kvaps commented Oct 11, 2024 •
edited
Loading

I create PersistentVolumeClaim with size of 1Gi:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: block-pvc
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Block
  resources:
    requests:
      storage: 1Gi

1Gi*1024*1024*1024 = 1073741824 bytes

But inside the pod the size of volume is only 1073750016:

/ # lsblk -bno SIZE /dev/drbd1014
1073750016
/ # qemu-img info /dev/drbd1014
image: /dev/drbd1014
file format: raw
virtual size: 1 GiB (1073750016 bytes)
disk size: 0 B
Child node '/file':
    filename: /dev/drbd1014
    protocol type: host_device
    file length: 1 GiB (1073750016 bytes)
    disk size: 0 B

link kubevirt/containerized-data-importer#3159
@kvaps kvaps mentioned this issue Oct 11, 2024
Closed
@kvaps
Copy link
Author

kvaps commented Oct 11, 2024

In LINSTOR the volumes have different

                "allocated_size_kib": 20976040,
                "usable_size_kib": 20971528

Is it possible to order volumes in LINSTOR using usable_size, not allocated_size?

@kvaps
Copy link
Author

kvaps commented Oct 11, 2024

I just tried to use --gross but created volume even smaller than without this flag

@kvaps
Copy link
Author

kvaps commented Oct 11, 2024 •
edited
Loading

Is there ant reliable way to request LINSTOR create a DRBD volume with exact size 1Gi (1073741824 bytes) that can be 100% usable by the client application?

@ghernadi
Copy link
Contributor

Correct me if I'm wrong, but reading the linked issue it does not look like the question / request here is that LINSTOR should not give you a little bit more space, but exactly the space you asked for, right?

If so, please compare the sizes from linstor -m vd l | grep size_kib and linstor -m v l | grep '"allocated_size_kib":'. The volume-definition's size is exactly the size the user asked for, whereas the allocated_size_kib as well as the usable_size_kib might be a bit larger than the user wanted due to LVM / ZFS rounding up to the next extent size.

Getting back to your example, if you ask LINSTOR for a 1GiB large volume (i.e. 1073741824 bytes), you will get a bit more than 1GiB volume, in your case 1073750016. 1073750016 - 1073741824 = 8192, which means you got 8KiB more than requested.

Am I missing something or is the title of this issue the wrong way around?

@kvaps
Copy link
Author

kvaps commented Oct 11, 2024 •
edited
Loading

Hi @ghernadi thank you for investigating, but seems I was wrong here, here is my pvc:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prime-dbabbeef-a3f4-412e-aec6-cdbc8acd2c05
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: "1073741824"
  storageClassName: replicated
  volumeMode: Block
  volumeName: pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a
# linstor vd l -r pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a
+-------------------------------------------------------------------------------------------+
| ResourceName                             | VolumeNr | VolumeMinor | Size  | Gross | State |
|===========================================================================================|
| pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a | 0        | 1000        | 1 GiB |       | ok    |
+-------------------------------------------------------------------------------------------+

# linstor v l -r pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a
+---------------------------------------------------------------------------------------------------------------------------------+
| Node | Resource                                 | StoragePool | VolNr | MinorNr | DeviceName    | Allocated | InUse  |    State |
|=================================================================================================================================|
| srv1 | pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a | data        |     0 |    1000 | /dev/drbd1000 |  1.00 GiB | Unused | UpToDate |
| srv2 | pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a | data        |     0 |    1000 | /dev/drbd1000 |  1.00 GiB | InUse  | UpToDate |
| srv3 | pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a | data        |     0 |    1000 | /dev/drbd1000 |  1.00 GiB | Unused | UpToDate |
+---------------------------------------------------------------------------------------------------------------------------------+

# linstor -m v l -r pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a
[
  [
    {
      "name": "pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a",
      "node_name": "srv1",
      "props": {
        "StorPoolName": "data"
      },
      "layer_object": {
        "children": [
          {
            "type": "STORAGE",
            "storage": {
              "storage_volumes": [
                {
                  "volume_number": 0,
                  "device_path": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
                  "allocated_size_kib": 1048848,
                  "usable_size_kib": 1048848,
                  "disk_state": "[]"
                }
              ]
            }
          }
        ],
        "type": "DRBD",
        "drbd": {
          "drbd_resource_definition": {
            "peer_slots": 7,
            "al_stripes": 1,
            "al_stripe_size_kib": 32,
            "port": 7001,
            "transport_type": "IP",
            "secret": "uV4aAOXbn/YtSIryS9yf",
            "down": false
          },
          "node_id": 1,
          "peer_slots": 7,
          "al_stripes": 1,
          "al_size": 32,
          "flags": [
            "INITIALIZED"
          ],
          "drbd_volumes": [
            {
              "drbd_volume_definition": {
                "volume_number": 0,
                "minor_number": 1000
              },
              "device_path": "/dev/drbd1000",
              "backing_device": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
              "allocated_size_kib": 1048840,
              "usable_size_kib": 1048584
            }
          ],
          "connections": {
            "srv3": {
              "connected": true,
              "message": "Connected"
            },
            "srv2": {
              "connected": true,
              "message": "Connected"
            }
          },
          "promotion_score": 10103,
          "may_promote": false
        }
      },
      "state": {
        "in_use": false
      },
      "uuid": "98460b7d-97cb-44d7-8022-2970006aa361",
      "create_timestamp": 1728646426440,
      "volumes": [
        {
          "volume_number": 0,
          "storage_pool_name": "data",
          "provider_kind": "ZFS",
          "device_path": "/dev/drbd1000",
          "allocated_size_kib": 1048848,
          "state": {
            "disk_state": "UpToDate"
          },
          "layer_data_list": [
            {
              "type": "DRBD",
              "data": {
                "drbd_volume_definition": {
                  "volume_number": 0,
                  "minor_number": 1000
                },
                "device_path": "/dev/drbd1000",
                "backing_device": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
                "allocated_size_kib": 1048840,
                "usable_size_kib": 1048584
              }
            },
            {
              "type": "STORAGE",
              "data": {
                "volume_number": 0,
                "device_path": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
                "allocated_size_kib": 1048848,
                "usable_size_kib": 1048848,
                "disk_state": "[]"
              }
            }
          ],
          "uuid": "293063cd-fdd3-4e45-9178-802c61832290"
        }
      ]
    },
    {
      "name": "pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a",
      "node_name": "srv2",
      "props": {
        "StorPoolName": "data"
      },
      "layer_object": {
        "children": [
          {
            "type": "STORAGE",
            "storage": {
              "storage_volumes": [
                {
                  "volume_number": 0,
                  "device_path": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
                  "allocated_size_kib": 1048848,
                  "usable_size_kib": 1048848,
                  "disk_state": "[]"
                }
              ]
            }
          }
        ],
        "type": "DRBD",
        "drbd": {
          "drbd_resource_definition": {
            "peer_slots": 7,
            "al_stripes": 1,
            "al_stripe_size_kib": 32,
            "port": 7001,
            "transport_type": "IP",
            "secret": "uV4aAOXbn/YtSIryS9yf",
            "down": false
          },
          "node_id": 0,
          "peer_slots": 7,
          "al_stripes": 1,
          "al_size": 32,
          "flags": [
            "INITIALIZED"
          ],
          "drbd_volumes": [
            {
              "drbd_volume_definition": {
                "volume_number": 0,
                "minor_number": 1000
              },
              "device_path": "/dev/drbd1000",
              "backing_device": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
              "allocated_size_kib": 1048840,
              "usable_size_kib": 1048584
            }
          ],
          "connections": {
            "srv3": {
              "connected": true,
              "message": "Connected"
            },
            "srv1": {
              "connected": true,
              "message": "Connected"
            }
          },
          "promotion_score": 10103,
          "may_promote": false
        }
      },
      "state": {
        "in_use": true
      },
      "uuid": "9cffb7f6-bea8-48a0-9906-d74e8eb2694c",
      "create_timestamp": 1728646425721,
      "volumes": [
        {
          "volume_number": 0,
          "storage_pool_name": "data",
          "provider_kind": "ZFS",
          "device_path": "/dev/drbd1000",
          "allocated_size_kib": 1048848,
          "state": {
            "disk_state": "UpToDate"
          },
          "layer_data_list": [
            {
              "type": "DRBD",
              "data": {
                "drbd_volume_definition": {
                  "volume_number": 0,
                  "minor_number": 1000
                },
                "device_path": "/dev/drbd1000",
                "backing_device": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
                "allocated_size_kib": 1048840,
                "usable_size_kib": 1048584
              }
            },
            {
              "type": "STORAGE",
              "data": {
                "volume_number": 0,
                "device_path": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
                "allocated_size_kib": 1048848,
                "usable_size_kib": 1048848,
                "disk_state": "[]"
              }
            }
          ],
          "uuid": "66fbfb01-a4da-4643-8b50-dc33e97cf18d"
        }
      ]
    },
    {
      "name": "pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a",
      "node_name": "srv3",
      "props": {
        "StorPoolName": "data"
      },
      "layer_object": {
        "children": [
          {
            "type": "STORAGE",
            "storage": {
              "storage_volumes": [
                {
                  "volume_number": 0,
                  "device_path": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
                  "allocated_size_kib": 1048848,
                  "usable_size_kib": 1048848,
                  "disk_state": "[]"
                }
              ]
            }
          }
        ],
        "type": "DRBD",
        "drbd": {
          "drbd_resource_definition": {
            "peer_slots": 7,
            "al_stripes": 1,
            "al_stripe_size_kib": 32,
            "port": 7001,
            "transport_type": "IP",
            "secret": "uV4aAOXbn/YtSIryS9yf",
            "down": false
          },
          "node_id": 2,
          "peer_slots": 7,
          "al_stripes": 1,
          "al_size": 32,
          "flags": [
            "INITIALIZED"
          ],
          "drbd_volumes": [
            {
              "drbd_volume_definition": {
                "volume_number": 0,
                "minor_number": 1000
              },
              "device_path": "/dev/drbd1000",
              "backing_device": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
              "allocated_size_kib": 1048840,
              "usable_size_kib": 1048584
            }
          ],
          "connections": {
            "srv1": {
              "connected": true,
              "message": "Connected"
            },
            "srv2": {
              "connected": true,
              "message": "Connected"
            }
          },
          "promotion_score": 10103,
          "may_promote": false
        }
      },
      "state": {
        "in_use": false
      },
      "uuid": "b9441c59-3f16-413e-b8b1-b73918d098f0",
      "create_timestamp": 1728646426396,
      "volumes": [
        {
          "volume_number": 0,
          "storage_pool_name": "data",
          "provider_kind": "ZFS",
          "device_path": "/dev/drbd1000",
          "allocated_size_kib": 1048848,
          "state": {
            "disk_state": "UpToDate"
          },
          "layer_data_list": [
            {
              "type": "DRBD",
              "data": {
                "drbd_volume_definition": {
                  "volume_number": 0,
                  "minor_number": 1000
                },
                "device_path": "/dev/drbd1000",
                "backing_device": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
                "allocated_size_kib": 1048840,
                "usable_size_kib": 1048584
              }
            },
            {
              "type": "STORAGE",
              "data": {
                "volume_number": 0,
                "device_path": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
                "allocated_size_kib": 1048848,
                "usable_size_kib": 1048848,
                "disk_state": "[]"
              }
            }
          ],
          "uuid": "7fa0181c-3522-4a44-826c-ea027b35558e"
        }
      ]
    }
  ]
]

For DRBD layer it is:

                "allocated_size_kib": 1048840,
                "usable_size_kib": 1048584

/ # lsblk -bno SIZE /dev/drbd1000
1073750016

>>> 1073750016/1024/1024/1024
1.0000076293945312

It seems CDI issue, I will continue investigating, and let you know.
/close for now

@kvaps kvaps closed this as completed Oct 11, 2024
@kvaps
Copy link
Author

kvaps commented Oct 12, 2024 •
edited
Loading

It seems there is a bug (or feature?) in CDI. It expects that volume will have exact size that requested in PVC.

Personally I don't see the problem if the drive will be a n
bit larger than expected, but it might affect some mechanics of copying PVCs between each other.

I didn't dig so far, but I see a potential issue, if CDI creates a new PVC with the same size as origin (and it reads this size from the PVC spec), but in fact that origin have larger volume inside it might block copying operations.

@kvaps
Copy link
Author

kvaps commented Oct 12, 2024 •
edited
Loading

I think LINSTOR should create drives with exact requested size from PVC, if it is not possible due to technical limitations, it should at least report correct usable_size into created PV after the volume is created

@kvaps kvaps changed the title LINSTOR creates volumes smaller than requested: LINSTOR creates volumes larger than requested size Oct 12, 2024
@kvaps kvaps reopened this Oct 12, 2024
@kvaps kvaps mentioned this issue Oct 14, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kvaps @ghernadi