Skip to content

When writing a batch to an implicit bucket upon schema conflict, I receive a 400. The logs seems to be f-printed. #693

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
BernardClabots opened this issue Apr 25, 2025 · 2 comments
Labels
bug Something isn't working

Comments

@BernardClabots
Copy link

Specifications

  • Client Version: 1.40.0
  • InfluxDB Version: Cloud 2.0 TSM
  • Platform: GCP

I am extracting some data from BQ and pushing it to Influx. That part is working well.
Next, I write a point in an Influx bucket to mark the task completed. This was written by someone else and it uses a different call.
In this particular case, I have a conflict for the second parameter. In the GCP logs, I can see the error as "default" on top of the log generated by loguru (the log appears twice).

Code sample to reproduce problem

in the calling function

    points = [
        Point.from_dict(
            {
                "measurement": __DATA_MEASUREMENT,
                "fields": {state: value},
                "tags": {"type": "status", **tags},
            }
        ),
        Point.from_dict(
            {
                "measurement": __DATA_MEASUREMENT,
                "fields": {state: datetime.now().isoformat()},
                "tags": {"type": "timestamp", **tags},
            }
        ),
    ]

    write_points(
        __INFLUX_URL,
        __INFLUX_TOKEN,
        __INFLUX_BUCKET,
        __INFLUX_ORGANISATION,
        points,
    )
```
in the lib to write_points:
``````python
def write_points(
    influx_url: str,
    influx_token: str,
    influx_bucket: str,
    influx_organization: str,
    points: List[Union[Point, Dict[str, Any]]],
) -> None:
    logger.info("Writing to InfluxDB, Points length={}", len(points))
    client = InfluxDBClient(url=influx_url, token=influx_token, timeout=300_000)
    try:
        write_api = client.write_api(
            success_callback=callback.success,
            error_callback=callback.error,
            retry_callback=callback.retry,
            write_options=WriteOptions(max_retries=10, max_retry_time=280_000)
        )
        
        try:
            write_api.write(
                batch_size=200,
                bucket=influx_bucket,
                org=influx_organization,
                record=points
            )
            logger.debug("Write operation completed")
            
        except InfluxDBError as e:
            logger.error("InfluxDB Asynchronous write error: {}", str(e))
            if hasattr(e, 'response'):
                logger.error("Response status: {}", e.response.status_code)
                logger.error("Response headers: {}", dict(e.response.headers))
                logger.error("Response body: {}", e.response.text)
            raise

        except Exception as ex:
            logger.error("Unexpected error during write operation: {}", str(ex))
            raise

    finally:
        client.close()

```
I know this is not correctly configured to do SYNCHRONOUS write, but that is how I see the issue.
The callback object is just implementing calls to logger.{info|error|warning} resp.

### Expected behavior

The logs are formatted as per my loguru logger settings

### Actual behavior

some logs (error 400 schema conflict) appear twice.

### Additional info

(DEBUG) 2025-04-25 15:15:06.130 CEST
Write operation completed
(DEFAULT) 2025-04-25 15:15:10.914 CEST
The batch item wasn't processed successfully because: (400)
(DEFAULT) 2025-04-25 15:15:10.914 CEST
Reason: Bad Request
(DEFAULT) 2025-04-25 15:15:10.914 CEST
HTTP response headers: HTTPHeaderDict({'Date': 'Fri, 25 Apr 2025 13:15:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '247', 'Connection': 'keep-alive', 'influxdata-partial-write': 'true', 'trace-id': 'xxxxxxxxxxxxxxxx', 'trace-sampled': 'false', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'X-Influxdb-Request-ID': 'zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz', 'X-Influxdb-Build': 'Cloud'})
(DEFAULT) 2025-04-25 15:15:10.914 CEST
HTTP response body: {"code":"invalid","message":"partial write has occurred, errors encountered on line(s): line 2: batch schema conflict - line has column 'inserted_data' of type iox::column_type::field::string but type is iox::column_type::field::integer","line":2}
(ERROR) 2025-04-25 15:15:10.915 CEST
Cannot write batch: ('{bucket_name}', '{org_name}', 'ns'), data: b'GCP_Data_monitoring,app=myapp,stage=preprod,type=status,zone=eu-west-01 inserted_data=1i\nGCP_Data_monitoring,app=myapp,stage=preprod,type=timestamp,zone=eu-west-01 inserted_data="2025-04-25T13:15:06.127660"' due: (400) Reason: Bad Request HTTP response headers: HTTPHeaderDict({'Date': 'Fri, 25 Apr 2025 13:15:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '247', 'Connection': 'keep-alive', 'influxdata-partial-write': 'true', 'trace-id': 'xxxxxxxxxxxxxxxx', 'trace-sampled': 'false', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'X-Influxdb-Request-ID': 'zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz', 'X-Influxdb-Build': 'Cloud'}) HTTP response body: {"code":"invalid","message":"partial write has occurred, errors encountered on line(s): line 2: batch schema conflict - line has column 'inserted_data' of type iox::column_type::field::string but type is iox::column_type::field::integer","line":2}
@BernardClabots BernardClabots added the bug Something isn't working label Apr 25, 2025
@BernardClabots
Copy link
Author

The (DEFAULT) lines are useless and polute my logs. looks like they are debug statement that were not removed.

@suyashcjoshi
Copy link

Bernard shared this issue in our community slack as well, just an fyi.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants