ex-pod-affinity sample failed after follow the guide #8122
Description
In which file did you encounter the issue?
Did you change the file? If so, how?
No
Describe the issue
My Composer info:
Zone: asia-southeast1
Composer version: 1.19.0
Airflow version: 2.2.5
I follow the guide https://cloud.google.com/composer/docs/how-to/using/using-kubernetes-pod-operator
on the section of Pod Affinity Configuration, I created a node-pool name pool-0, the node has been created with return message below.
NAME MACHINE_TYPE DISK_SIZE_GB NODE_VERSION
pool-0 n1-standard-1 100 1.21.11-gke.1900I back to the Composer and run again(as the guide suggested) but got the error below in the Composer log.
*** Reading remote log from gs://asia-southeast1-cc1-2e842d2e-bucket/logs/composer_sample_kubernetes_pod/ex-pod-affinity/2022-06-24T08:20:54.389986+00:00/12.log.
[2022-06-26, 08:07:05 +07] {taskinstance.py:1044} INFO - Dependencies all met for <TaskInstance: composer_sample_kubernetes_pod.ex-pod-affinity scheduled__2022-06-24T08:20:54.389986+00:00 [queued]>
[2022-06-26, 08:07:05 +07] {taskinstance.py:1044} INFO - Dependencies all met for <TaskInstance: composer_sample_kubernetes_pod.ex-pod-affinity scheduled__2022-06-24T08:20:54.389986+00:00 [queued]>
[2022-06-26, 08:07:05 +07] {taskinstance.py:1250} INFO -
--------------------------------------------------------------------------------
[2022-06-26, 08:07:05 +07] {taskinstance.py:1251} INFO - Starting attempt 12 of 14
[2022-06-26, 08:07:05 +07] {taskinstance.py:1252} INFO -
--------------------------------------------------------------------------------
[2022-06-26, 08:07:05 +07] {taskinstance.py:1271} INFO - Executing <Task(KubernetesPodOperator): ex-pod-affinity> on 2022-06-24 08:20:54.389986+00:00
[2022-06-26, 08:07:05 +07] {standard_task_runner.py:52} INFO - Started process 47413 to run task
[2022-06-26, 08:07:05 +07] {standard_task_runner.py:79} INFO - Running: ['airflow', 'tasks', 'run', 'composer_sample_kubernetes_pod', 'ex-pod-affinity', 'scheduled__2022-06-24T08:20:54.389986+00:00', '--job-id', '700', '--raw', '--subdir', 'DAGS_FOLDER/kubernetes_pod_operator.py', '--cfg-path', '/tmp/tmpkgdujdr7', '--error-file', '/tmp/tmp8bz_8iwy']
[2022-06-26, 08:07:05 +07] {standard_task_runner.py:80} INFO - Job 700: Subtask ex-pod-affinity
[2022-06-26, 08:07:06 +07] {task_command.py:298} INFO - Running <TaskInstance: composer_sample_kubernetes_pod.ex-pod-affinity scheduled__2022-06-24T08:20:54.389986+00:00 [running]> on host airflow-worker-8696b4747f-n59j4
[2022-06-26, 08:07:06 +07] {taskinstance.py:1448} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=composer_sample_kubernetes_pod
AIRFLOW_CTX_TASK_ID=ex-pod-affinity
AIRFLOW_CTX_EXECUTION_DATE=2022-06-24T08:20:54.389986+00:00
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-06-24T08:20:54.389986+00:00
[2022-06-26, 08:07:06 +07] {kubernetes_pod.py:566} INFO - Creating pod ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69 with labels: {'dag_id': 'composer_sample_kubernetes_pod', 'task_id': 'ex-pod-affinity', 'execution_date': '2022-06-24T082054.3899860000-0c9c100e3', 'try_number': '12'}
[2022-06-26, 08:07:07 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:08 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:09 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:10 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:11 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:12 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:13 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:14 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:15 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:16 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:17 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:18 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:19 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:20 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:21 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:22 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:23 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:24 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:25 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:26 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:32 +07] {pod_manager.py:197} INFO - Can't use an undefined value as an ARRAY reference at /usr/local/lib/perl5/5.36.0/Math/BigInt/Calc.pm line 1049.
[2022-06-26, 08:07:35 +07] {kubernetes_pod.py:459} INFO - Deleting pod: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:35 +07] {taskinstance.py:1776} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 430, in execute
self.cleanup(
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 452, in cleanup
raise AirflowException(f'Pod {pod and pod.metadata.name} returned a failure: {remote_pod}')
airflow.exceptions.AirflowException: Pod ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69 returned a failure: {'api_version': 'v1',
'kind': 'Pod',
'metadata': {'annotations': None,
'cluster_name': None,
'creation_timestamp': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal()),
'deletion_grace_period_seconds': None,
'deletion_timestamp': None,
'finalizers': None,
'generate_name': None,
'generation': None,
'initializers': None,
'labels': {'airflow_version': '2.2.5-composer',
'dag_id': 'composer_sample_kubernetes_pod',
'execution_date': '2022-06-24T082054.3899860000-0c9c100e3',
'kubernetes_pod_operator': 'True',
'task_id': 'ex-pod-affinity',
'try_number': '12'},
'managed_fields': [{'api_version': 'v1',
'fields': None,
'manager': 'OpenAPI-Generator',
'operation': 'Update',
'time': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal())},
{'api_version': 'v1',
'fields': None,
'manager': 'kubelet',
'operation': 'Update',
'time': datetime.datetime(2022, 6, 26, 1, 7, 34, tzinfo=tzlocal())}],
'name': 'ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69',
'namespace': 'default',
'owner_references': None,
'resource_version': '981544',
'self_link': None,
'uid': '80edd551-f55b-46d1-972c-bff97473d074'},
'spec': {'active_deadline_seconds': None,
'affinity': {'node_affinity': {'preferred_during_scheduling_ignored_during_execution': None,
'required_during_scheduling_ignored_during_execution': {'node_selector_terms': [{'match_expressions': [{'key': 'cloud.google.com/gke-nodepool',
'operator': 'In',
'values': ['pool-0',
'pool-1']}],
'match_fields': None}]}},
'pod_affinity': None,
'pod_anti_affinity': None},
'automount_service_account_token': None,
'containers': [{'args': ['-Mbignum=bpi', '-wle', 'print bpi(2000)'],
'command': ['perl'],
'env': None,
'env_from': None,
'image': 'perl',
'image_pull_policy': 'Always',
'lifecycle': None,
'liveness_probe': None,
'name': 'base',
'ports': None,
'readiness_probe': None,
'resources': {'limits': None, 'requests': None},
'security_context': None,
'stdin': None,
'stdin_once': None,
'termination_message_path': '/dev/termination-log',
'termination_message_policy': 'File',
'tty': None,
'volume_devices': None,
'volume_mounts': [{'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',
'mount_propagation': None,
'name': 'kube-api-access-lw9pt',
'read_only': True,
'sub_path': None,
'sub_path_expr': None}],
'working_dir': None}],
'dns_config': None,
'dns_policy': 'ClusterFirst',
'enable_service_links': True,
'host_aliases': None,
'host_ipc': None,
'host_network': None,
'host_pid': None,
'hostname': None,
'image_pull_secrets': None,
'init_containers': None,
'node_name': 'gke-asia-southeast1-cc1-2e842d-pool-0-32a57203-s34t',
'node_selector': None,
'preemption_policy': 'PreemptLowerPriority',
'priority': 0,
'priority_class_name': None,
'readiness_gates': None,
'restart_policy': 'Never',
'runtime_class_name': None,
'scheduler_name': 'default-scheduler',
'security_context': {'fs_group': None,
'run_as_group': None,
'run_as_non_root': None,
'run_as_user': None,
'se_linux_options': None,
'supplemental_groups': None,
'sysctls': None,
'windows_options': None},
'service_account': 'default',
'service_account_name': 'default',
'share_process_namespace': None,
'subdomain': None,
'termination_grace_period_seconds': 30,
'tolerations': [{'effect': 'NoExecute',
'key': 'node.kubernetes.io/not-ready',
'operator': 'Exists',
'toleration_seconds': 300,
'value': None},
{'effect': 'NoExecute',
'key': 'node.kubernetes.io/unreachable',
'operator': 'Exists',
'toleration_seconds': 300,
'value': None}],
'volumes': [{'aws_elastic_block_store': None,
'azure_disk': None,
'azure_file': None,
'cephfs': None,
'cinder': None,
'config_map': None,
'csi': None,
'downward_api': None,
'empty_dir': None,
'fc': None,
'flex_volume': None,
'flocker': None,
'gce_persistent_disk': None,
'git_repo': None,
'glusterfs': None,
'host_path': None,
'iscsi': None,
'name': 'kube-api-access-lw9pt',
'nfs': None,
'persistent_volume_claim': None,
'photon_persistent_disk': None,
'portworx_volume': None,
'projected': {'default_mode': 420,
'sources': [{'config_map': None,
'downward_api': None,
'secret': None,
'service_account_token': {'audience': None,
'expiration_seconds': 3607,
'path': 'token'}},
{'config_map': {'items': [{'key': 'ca.crt',
'mode': None,
'path': 'ca.crt'}],
'name': 'kube-root-ca.crt',
'optional': None},
'downward_api': None,
'secret': None,
'service_account_token': None},
{'config_map': None,
'downward_api': {'items': [{'field_ref': {'api_version': 'v1',
'field_path': 'metadata.namespace'},
'mode': None,
'path': 'namespace',
'resource_field_ref': None}]},
'secret': None,
'service_account_token': None}]},
'quobyte': None,
'rbd': None,
'scale_io': None,
'secret': None,
'storageos': None,
'vsphere_volume': None}]},
'status': {'conditions': [{'last_probe_time': None,
'last_transition_time': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal()),
'message': None,
'reason': None,
'status': 'True',
'type': 'Initialized'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2022, 6, 26, 1, 7, 34, tzinfo=tzlocal()),
'message': 'containers with unready status: [base]',
'reason': 'ContainersNotReady',
'status': 'False',
'type': 'Ready'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2022, 6, 26, 1, 7, 34, tzinfo=tzlocal()),
'message': 'containers with unready status: [base]',
'reason': 'ContainersNotReady',
'status': 'False',
'type': 'ContainersReady'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal()),
'message': None,
'reason': None,
'status': 'True',
'type': 'PodScheduled'}],
'container_statuses': [{'container_id': 'containerd://265b273c470d3e0243913d01440d8d28353d5036916a39809ce03fb3d49d4c4f',
'image': 'docker.io/library/perl:latest',
'image_id': 'docker.io/library/perl@sha256:b83504159488e1ae88e6705f34b78fac623b2cd55ca0304de31bfb09d59f7bfd',
'last_state': {'running': None,
'terminated': None,
'waiting': None},
'name': 'base',
'ready': False,
'restart_count': 0,
'state': {'running': None,
'terminated': {'container_id': 'containerd://265b273c470d3e0243913d01440d8d28353d5036916a39809ce03fb3d49d4c4f',
'exit_code': 255,
'finished_at': datetime.datetime(2022, 6, 26, 1, 7, 32, tzinfo=tzlocal()),
'message': None,
'reason': 'Error',
'signal': None,
'started_at': datetime.datetime(2022, 6, 26, 1, 7, 26, tzinfo=tzlocal())},
'waiting': None}}],
'host_ip': '10.148.0.17',
'init_container_statuses': None,
'message': None,
'nominated_node_name': None,
'phase': 'Failed',
'pod_ip': '10.124.3.2',
'qos_class': 'BestEffort',
'reason': None,
'start_time': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal())}}
[2022-06-26, 08:07:35 +07] {taskinstance.py:1279} INFO - Marking task as UP_FOR_RETRY. dag_id=composer_sample_kubernetes_pod, task_id=ex-pod-affinity, execution_date=20220624T082054, start_date=20220626T010705, end_date=20220626T010735
[2022-06-26, 08:07:35 +07] {standard_task_runner.py:93} ERROR - Failed to execute job 700 for task ex-pod-affinity (Pod ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69 returned a failure: {'api_version': 'v1',
'kind': 'Pod',
'metadata': {'annotations': None,
'cluster_name': None,
'creation_timestamp': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal()),
'deletion_grace_period_seconds': None,
'deletion_timestamp': None,
'finalizers': None,
'generate_name': None,
'generation': None,
'initializers': None,
'labels': {'airflow_version': '2.2.5-composer',
'dag_id': 'composer_sample_kubernetes_pod',
'execution_date': '2022-06-24T082054.3899860000-0c9c100e3',
'kubernetes_pod_operator': 'True',
'task_id': 'ex-pod-affinity',
'try_number': '12'},
'managed_fields': [{'api_version': 'v1',
'fields': None,
'manager': 'OpenAPI-Generator',
'operation': 'Update',
'time': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal())},
{'api_version': 'v1',
'fields': None,
'manager': 'kubelet',
'operation': 'Update',
'time': datetime.datetime(2022, 6, 26, 1, 7, 34, tzinfo=tzlocal())}],
'name': 'ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69',
'namespace': 'default',
'owner_references': None,
'resource_version': '981544',
'self_link': None,
'uid': '80edd551-f55b-46d1-972c-bff97473d074'},
'spec': {'active_deadline_seconds': None,
'affinity': {'node_affinity': {'preferred_during_scheduling_ignored_during_execution': None,
'required_during_scheduling_ignored_during_execution': {'node_selector_terms': [{'match_expressions': [{'key': 'cloud.google.com/gke-nodepool',
'operator': 'In',
'values': ['pool-0',
'pool-1']}],
'match_fields': None}]}},
'pod_affinity': None,
'pod_anti_affinity': None},
'automount_service_account_token': None,
'containers': [{'args': ['-Mbignum=bpi', '-wle', 'print bpi(2000)'],
'command': ['perl'],
'env': None,
'env_from': None,
'image': 'perl',
'image_pull_policy': 'Always',
'lifecycle': None,
'liveness_probe': None,
'name': 'base',
'ports': None,
'readiness_probe': None,
'resources': {'limits': None, 'requests': None},
'security_context': None,
'stdin': None,
'stdin_once': None,
'termination_message_path': '/dev/termination-log',
'termination_message_policy': 'File',
'tty': None,
'volume_devices': None,
'volume_mounts': [{'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',
'mount_propagation': None,
'name': 'kube-api-access-lw9pt',
'read_only': True,
'sub_path': None,
'sub_path_expr': None}],
'working_dir': None}],
'dns_config': None,
'dns_policy': 'ClusterFirst',
'enable_service_links': True,
'host_aliases': None,
'host_ipc': None,
'host_network': None,
'host_pid': None,
'hostname': None,
'image_pull_secrets': None,
'init_containers': None,
'node_name': 'gke-asia-southeast1-cc1-2e842d-pool-0-32a57203-s34t',
'node_selector': None,
'preemption_policy': 'PreemptLowerPriority',
'priority': 0,
'priority_class_name': None,
'readiness_gates': None,
'restart_policy': 'Never',
'runtime_class_name': None,
'scheduler_name': 'default-scheduler',
'security_context': {'fs_group': None,
'run_as_group': None,
'run_as_non_root': None,
'run_as_user': None,
'se_linux_options': None,
'supplemental_groups': None,
'sysctls': None,
'windows_options': None},
'service_account': 'default',
'service_account_name': 'default',
'share_process_namespace': None,
'subdomain': None,
'termination_grace_period_seconds': 30,
'tolerations': [{'effect': 'NoExecute',
'key': 'node.kubernetes.io/not-ready',
'operator': 'Exists',
'toleration_seconds': 300,
'value': None},
{'effect': 'NoExecute',
'key': 'node.kubernetes.io/unreachable',
'operator': 'Exists',
'toleration_seconds': 300,
'value': None}],
'volumes': [{'aws_elastic_block_store': None,
'azure_disk': None,
'azure_file': None,
'cephfs': None,
'cinder': None,
'config_map': None,
'csi': None,
'downward_api': None,
'empty_dir': None,
'fc': None,
'flex_volume': None,
'flocker': None,
'gce_persistent_disk': None,
'git_repo': None,
'glusterfs': None,
'host_path': None,
'iscsi': None,
'name': 'kube-api-access-lw9pt',
'nfs': None,
'persistent_volume_claim': None,
'photon_persistent_disk': None,
'portworx_volume': None,
'projected': {'default_mode': 420,
'sources': [{'config_map': None,
'downward_api': None,
'secret': None,
'service_account_token': {'audience': None,
'expiration_seconds': 3607,
'path': 'token'}},
{'config_map': {'items': [{'key': 'ca.crt',
'mode': None,
'path': 'ca.crt'}],
'name': 'kube-root-ca.crt',
'optional': None},
'downward_api': None,
'secret': None,
'service_account_token': None},
{'config_map': None,
'downward_api': {'items': [{'field_ref': {'api_version': 'v1',
'field_path': 'metadata.namespace'},
'mode': None,
'path': 'namespace',
'resource_field_ref': None}]},
'secret': None,
'service_account_token': None}]},
'quobyte': None,
'rbd': None,
'scale_io': None,
'secret': None,
'storageos': None,
'vsphere_volume': None}]},
'status': {'conditions': [{'last_probe_time': None,
'last_transition_time': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal()),
'message': None,
'reason': None,
'status': 'True',
'type': 'Initialized'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2022, 6, 26, 1, 7, 34, tzinfo=tzlocal()),
'message': 'containers with unready status: [base]',
'reason': 'ContainersNotReady',
'status': 'False',
'type': 'Ready'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2022, 6, 26, 1, 7, 34, tzinfo=tzlocal()),
'message': 'containers with unready status: [base]',
'reason': 'ContainersNotReady',
'status': 'False',
'type': 'ContainersReady'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal()),
'message': None,
'reason': None,
'status': 'True',
'type': 'PodScheduled'}],
'container_statuses': [{'container_id': 'containerd://265b273c470d3e0243913d01440d8d28353d5036916a39809ce03fb3d49d4c4f',
'image': 'docker.io/library/perl:latest',
'image_id': 'docker.io/library/perl@sha256:b83504159488e1ae88e6705f34b78fac623b2cd55ca0304de31bfb09d59f7bfd',
'last_state': {'running': None,
'terminated': None,
'waiting': None},
'name': 'base',
'ready': False,
'restart_count': 0,
'state': {'running': None,
'terminated': {'container_id': 'containerd://265b273c470d3e0243913d01440d8d28353d5036916a39809ce03fb3d49d4c4f',
'exit_code': 255,
'finished_at': datetime.datetime(2022, 6, 26, 1, 7, 32, tzinfo=tzlocal()),
'message': None,
'reason': 'Error',
'signal': None,
'started_at': datetime.datetime(2022, 6, 26, 1, 7, 26, tzinfo=tzlocal())},
'waiting': None}}],
'host_ip': '10.148.0.17',
'init_container_statuses': None,
'message': None,
'nominated_node_name': None,
'phase': 'Failed',
'pod_ip': '10.124.3.2',
'qos_class': 'BestEffort',
'reason': None,
'start_time': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal())}}; 47413)