Maybe we should just use the tool provided by Airflow itself? db clean --clean-before-timestamp for example.

Related eventgate schema: https://schema.wikimedia.org/repositories//secondary/jsonschema/analytics/legacy/editattemptstep/latest.yaml

Related Airflow-DAGs MR: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/943

@brouberol got it. You'll mount Ceph as a file system local to the Airflow instance, and HDFS sync will write to Ceph - effectively, to Airflow, this will look like a local fs directory being updated.

@brouberol automatic sync from Ceph every 5 minutes might cause some issues with the HDFS synchronizer. We need to think about the scenario where your 5 minute sync starts in the middle of HDFS synchronization to Ceph - in that case, you will only get a part of airflow-dag repository and Airflow will most likely choke. Sure, it would fix itself on the next run within 5 minutes but it would be much better if we could mitigate this scenario somehow.

Update: We've refactored the library to support cache_key_fn config parameter, which enabled us to get rid of FsVersionedArtifactCache in favor of simply having one class FsArtifactCache:

Cool! How will this be used via artifact.yaml config?

But for this MR, what do you think of limiting the change to just restricting to fsspec, perhaps, perhaps by just renaming FsArtifactSource and FsArtifactCache to ArtifactSource and ArtifactCache and removing those abstract base classes? Then I think your refactor which removes the ArtifactCache.open abstract method will just work as is.

@Ottomata @mforns I think we should expand the scope of this refactor to include redefining the relationships between Artifact, ArtifactLocator, ArtifactSource and ArtifactCache, too. The current design is cumbersome and unintuitive IMHO, for the following reasons:

• ArtifactSource is defined/constructed through class name and base_uri which is optional, but in practice base_uri is not optional and points to either a directory URI or to an actual instance of Artifact. However, since it extends ArtifactLocator, all of its methods require an Artifact instance as a parameter.
• ArtifactCache is defined through class name and base_uri which is again defined as optional, but in practice it's actually required. All of its methods also require an instance of Artifact as a parameter.
• Even though ArtifactLocators all de-facto depend on an Artifact, the Artifact class itself introduces a hard dependency on ArtifactLocators through its source and caches arguments.

I've published a draft MR that, as far as I can tell, implements most of support for the produced_by configuration of a dataset. The code correctly recognizes, parses and models the produced_by section, and inserts a DatasetProducer object into a Dataset object. This DatasetProducer object supports its own implementation of the get_sensor_for method that takes precedence over Dataset's normal method, if the DatasetProducer is available.

In T372647#10183283, @Ottomata wrote:

it does not care about whether there's an actual file in the s3 bucket, or whether the contents of the file are correct.

Ah! This was the bit I was missing. I was trying to understand how a Hive dataset would work here.

So, there is no data aware sensing using Airflow Datasets, it is 100% airflow task state.

In T368033#10184240, @brouberol wrote:

@amastilovic I'm interested! Do you have something I could read to know more?

How would Kerberos authentication play out when it comes to different Airflow instances communicating with each other? Would each instance get a Kerberos ticket automatically, and then use that to communicate to other instances?

In T368033#10172189, @brouberol wrote:

Additional thought: once we migrate to KubernetesExecutor instead of LocalExecutor, the dags repo would get cloned by git-sync at the start of each task (ie Pod), causing us to have a hard reliance on gitlab being up. I think we should have an asynchronous process that clones the repo via git-sync (using a sha, and not a branch name), and exports the data to, say, s3. Then "pulling the DAGs repo" would basically become a s3 cp operation, causing us to move the dependency from gitlab to Ceph/RadosGW.

In T375878#10183064, @brouberol wrote:

@amastilovic this is part of the answer as to how we keep PG as lowly loaded as possible.

As an experienced Airflow WMF user, would you have an idea as to what a good retention would look like for you? 30 days? More?

Things are not settled for sure. The version of Airflow we use already supports the notion of Dataset (Data Asset is basically a rename+upgrade of a Dataset) and I think even as of right now we could use that for "event-driven pipelines" that react to Datasets being updated.

Currently, each instance specifies its own datasets.yaml files in its “config” directory. This approach is problematic due to a couple of reasons:

OK, so the schema I proposed in the Google Doc looks like this:

I've posted a draft proposal for the implementation design of ideas/needs described in this ticket: https://docs.google.com/document/d/1lapUHpWY2rm9si1iupRYHTcsDY-_QSbLcL03MXOSOlE/edit

Project migrated to GitLab: https://gitlab.wikimedia.org/repos/data-engineering/gobblin-wmf/-/merge_requests/1

This was resolved some time ago when SRE released updated Airflow Debian package.

Maybe just change the name of the file to data_dependencies.yaml and the module to DataDependency?
And then the user would use it like data_dependency("data-dependency-name").get_sensor_for(dag)?
With this, the semantic weirdness would be solved, no?

I talked to @BTullis about obtaining a functional test environment that would mimic the real world this service would be operating in, and he kindly provided a list of things to do in order to build such an environment. The list is in the subtask ticket https://phabricator.wikimedia.org/T371994

Also, could we add a settings.xml file with the following contents to the Docker image? It's necessary for Maven release plugin to interact with GitLab:

I've considered the option of pulling from the git origin into the destination HDFS, albeit not using a systemd timer. I've actually done something similar before in previous jobs/roles, by mounting HDFS onto a local file system, but I don't think this is a viable solution for a number of reasons:

Seconded about the .test in the group ID - do we really need that? I believe the group ID should simply be org.wikimedia and then we can have the test part in the artifact ID.

In T368033#9912923, @BTullis wrote:

In some ways, the fundamental question is: do we want to move to a continuous deployment model, or do we want to retain manual deployment.

In T368033#9909148, @BTullis wrote:

My feeling is that, at our scale and given its proximity to the dse-k8s cluster, Ceph will be just fine for these DAG volumes.

Merged and applied - done

In T360968#9865611, @Ottomata wrote:

Q: Have we discussed these ideas with Release Engineering folks? They are currently working on a similar CD project, but it might be MediaWiki focused only.

This ticket has been resolved, the tasks from the ticket definition have been performed on an-launcher1002 (an-launcher1001 has been decommissioned).

@lbowmaker this task can be closed.

@lbowmaker this task can be closed.

@lbowmaker this task can be closed.

@lbowmaker this task can be closed.

We are attempting to resolve this issue in this ticket: T365382

No update still - let's wait for a bit and see what happens, the sync refreshment period might be daily as opposed to hourly.

This task can be closed as the issue has been fixed and changes to the DAG have been merged.

In T364487#9786155, @BTullis wrote:

What's the longer-term location for the log4j properties file name? Presumably we don't want to leave the file name as aqu-log4j.properties within any folder?

MR to switch to using DagProperties: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/689

The issue was in the path to the configured log4j.properties file in Airflow UI, hdfs:///user/aqu/aqu-log4j.properties was not accessible by the Airflow user analytics.

In T362699#9722376, @Aklapper wrote:

Hi @amastilovic, can you please associate one or more active project tags with this task (via the Add Action... → Change Project Tags dropdown)? That will allow to see a task when looking at project workboards or searching for tasks in certain projects, and get notified about a task when watching a related project tag. Thanks!

I need access to the following (from the wiki page you provided):

@Arnoldokoth @Dzahn thank you!