fs/operations: add support for unix and unixnano time formats #8726

dmotte · 2025-08-04T23:35:15Z

What is the purpose of this change?

First of all, thank you so much for this amazing project! 🙂 I found Rclone extremely useful and very well done.

In this very simple PR I'm adding support for two new time formats: unix and unixnano. The reason behind this is that sometimes I use the rclone lsf command like this:

$ rclone lsf -R --format=pst --time-format=RFC3339 myremote:/
myfiles/;-1;2025-08-04T13:00:00+02:00
myfiles/file01.txt;123;2025-08-04T13:10:00+02:00
myfiles/file02.txt;456;2025-08-04T13:20:00+02:00
myfiles/file03.txt;789;2025-08-04T13:30:00+02:00

to generate a sort of CSV-based "tree" of a directory (either local or remote) with file sizes and last modification times. But the output of such command is timezone-dependent (note the +02:00s). So, if I run the same command on one of my servers, which has a copy of the same folder but the system time zone set to UTC, I get this:

$ rclone lsf -R --format=pst --time-format=RFC3339 myremote:/
myfiles/;-1;2025-08-04T11:00:00Z
myfiles/file01.txt;123;2025-08-04T11:10:00Z
myfiles/file02.txt;456;2025-08-04T11:20:00Z
myfiles/file03.txt;789;2025-08-04T11:30:00Z

So, even if the content of the directory is exactly the same, I'm unable to programmatically compare the two outputs, e.g. using the diff command.

Since Unix timestamps are timezone-agnostic, with this PR I'm introducing the possibility to do this:

$ rclone lsf -R --format=pst --time-format=unix myremote:/
myfiles/;-1;1754305200
myfiles/file01.txt;123;1754305800
myfiles/file02.txt;456;1754306400
myfiles/file03.txt;789;1754307000

And:

$ rclone lsf -R --format=pst --time-format=unixnano myremote:/
myfiles/;-1;1754305200123456789
myfiles/file01.txt;123;1754305800123456789
myfiles/file02.txt;456;1754306400123456789
myfiles/file03.txt;789;1754307000123456789

I hope this is useful :) let me know what you think. Thanks in advance!

Was the change discussed in an issue or in the forum before?

I don't think so. I tried to search but couldn't find anything related :)

Checklist

I have read the contribution guidelines.
I have added tests for all changes in this PR if appropriate.
I have added documentation for the changes if appropriate.
All commit messages are in house style.
I'm done, this Pull Request is ready for review :-)

albertony

Thanks. First of all you should revert changes in autogenerated files. These would have been flagged by our linter, if we had allowed the workflows to run.

docs/content/commands/rclone_lsf.md

MANUAL.md

MANUAL.txt

dmotte · 2025-08-10T12:12:00Z

@albertony thank you so much for the review! And sorry for the oversight. I reverted the changes to the autogenerated files in 4a67257. I hope it's OK now. Let me know :)

albertony · 2025-08-10T12:31:44Z

Great, lets run the workflows and see. 👍

dmotte · 2025-08-11T11:00:13Z

Hey @albertony JFYI it seems like all the workflows passed 🙂

albertony · 2025-08-11T11:05:45Z

Great. Looks good as far as I can tell, good job!

However; I'll leave the final review up to Nick.

ncw · 2025-08-13T15:45:11Z

But the output of such command is timezone-dependent (note the +02:00s).

There are several ways to fix that, but the first one which comes to mind is this

TZ=UTC rclone lsf -R --format=pst --time-format=RFC3339 myremote:/

I hope this is useful

Yes it does look useful, thank you. rclone lsf's purpose is to make reports and unix time is a sensible time format to do that.

I'll review the code now

ncw

Looking good :-)

See comments inline

ncw · 2025-08-13T16:36:56Z

cmd/lsf/lsf_test.go

@@ -264,6 +265,84 @@ subdir/file3_+_111_+_`+expectedOutput[6]+`
 	dirSlash = false
 }

+func TestTimeFormatUnix(t *testing.T) {


I think these tests would be more sensibly in fs/operations/operations_test.go

Search for AddModTime in there - we don't test with any time formats so it would be a good idea to add some.

Thank you, done in 1647bf7. I hope I got everything right :) let me know

ncw · 2025-08-13T16:40:34Z

fs/operations/operationsflags/operationsflags.go

@@ -62,7 +62,7 @@ func AddLoggerFlags(cmdFlags *pflag.FlagSet, opt *operations.LoggerOpt, flagsOpt

 	// lsf flags for destAfter
 	flags.StringVarP(cmdFlags, &opt.Format, "format", "F", "p", "Output format - see lsf help for details", "Sync")
-	flags.StringVarP(cmdFlags, &opt.TimeFormat, "timeformat", "t", "", "Specify a custom time format, or 'max' for max precision supported by remote (default: 2006-01-02 15:04:05)", "")
+	flags.StringVarP(cmdFlags, &opt.TimeFormat, "timeformat", "t", "", "Specify a custom time format, 'unix' / 'unixnano' for timestamp, or 'max' for max precision supported by remote (default: 2006-01-02 15:04:05)", "")


Same comment as above.

@nielash was there a reason this flag is called --timeformat but the lsf flag is called --time-format?

@ncw thank you for the idea. Addressed in f89cd9e

I don't specifically remember now, but probably an accidental oversight. I think the logger flags came first and adding the same idea to lsf was something we did later.

ncw · 2025-08-13T16:46:02Z

cmd/lsf/lsf.go

@@ -33,7 +33,7 @@ func init() {
 	cmd.Root.AddCommand(commandDefinition)
 	cmdFlags := commandDefinition.Flags()
 	flags.StringVarP(cmdFlags, &format, "format", "F", "p", "Output format - see  help for details", "")
-	flags.StringVarP(cmdFlags, &timeFormat, "time-format", "t", "", "Specify a custom time format, or 'max' for max precision supported by remote (default: 2006-01-02 15:04:05)", "")
+	flags.StringVarP(cmdFlags, &timeFormat, "time-format", "t", "", "Specify a custom time format, 'unix' / 'unixnano' for timestamp, or 'max' for max precision supported by remote (default: 2006-01-02 15:04:05)", "")


This is the command line help, so try to keep it short. It leaves out an awful lot so maybe just this?

"Specify a custom time format - see docs for details"

Note the docs here also need updating.

rclone/cmd/lsf/lsf.go

Lines 149 to 154 in fc5bd21

rclone lsf remote:path --format pt --time-format 'Jan 2, 2006 at 3:04pm (MST)'

rclone lsf remote:path --format pt --time-format '2006-01-02 15:04:05.000000000'

rclone lsf remote:path --format pt --time-format '2006-01-02T15:04:05.999999999Z07:00'

rclone lsf remote:path --format pt --time-format RFC3339

rclone lsf remote:path --format pt --time-format DateOnly

rclone lsf remote:path --format pt --time-format max

@ncw thank you for the idea. Addressed in f89cd9e

dmotte · 2025-08-13T20:24:12Z

TZ=UTC

@ncw already tried that, but for some reason it didn't work. But thanks for pointing it out anyway 🙂 nevermind, unix timestamps will be more than enough for me.

Now I'm gonna address all your comments

nielash · 2025-08-13T21:00:30Z

TZ=UTC

@ncw already tried that, but for some reason it didn't work.

Are you on Windows maybe? It doesn't work on Windows. I ran into this myself just the other day! fc5bd21

dmotte · 2025-08-13T21:11:11Z

It doesn't work on Windows

Yep @nielash I can confirm. I was trying it from a Windows PC and TZ didn't work, but it works on my Linux server. Thank you for the clarification!

@ncw I addressed all your comments 👍 I hope I did things correctly now. Sorry but it's my first contribution to this project and I don't have much experience with the codebase yet :)

dmotte · 2025-08-28T12:19:42Z

Hey guys @albertony @ncw @nielash any news on this? I just resolved some conflicts with master, and I personally think it's ready to be merged, but let me know what you think 🙂

dmotte · 2025-08-28T13:16:50Z

Mmmh, I can see a CI failure here: https://github.com/rclone/rclone/actions/runs/17295515406/job/49094283747?pr=8726

But the failure is only in the linux job. The test gets stuck retrying "failed to allocate memory" and then times out after 10 minutes. So to me it seems like a flaky issue in the pool test rather than something caused by this PR. WDYT?

Update: yep, I can see the same failure in a completely different commit from master: https://github.com/rclone/rclone/actions/runs/17160919578/job/48689720090

albertony · 2025-08-28T13:22:41Z

Yes, it is a bit flaky. I'll rerun it and hopefully it gets all green..

dmotte · 2025-08-28T13:56:45Z

Thank you @albertony, now it's green :)

ncw

This looks great now - thank you :-)

I'll merge it now.

dmotte added 3 commits August 5, 2025 00:10

fs/operations: added support for unix and unixnano time formats

bf0b264

docs: updated docs with support for unix and unixnano time formats

591d736

cmd/lsf: added unit tests for unix and unixnano time formats

56f0c3a

albertony reviewed Aug 10, 2025

View reviewed changes

docs/content/commands/rclone_lsf.md Outdated Show resolved Hide resolved

MANUAL.md Outdated Show resolved Hide resolved

MANUAL.txt Outdated Show resolved Hide resolved

docs: reverted changes to autogenerated files

4a67257

dmotte requested a review from albertony August 10, 2025 12:12

albertony requested review from ncw and removed request for albertony August 11, 2025 11:05

ncw reviewed Aug 13, 2025

View reviewed changes

dmotte added 2 commits August 13, 2025 22:35

cmd/lsf: shortened command help and updated docs

f89cd9e

fs/operations: simplified tests for unix and unixnano time formats

1647bf7

dmotte requested a review from ncw August 13, 2025 21:12

Merge branch 'master' into pr-timeformat-unix

7ce3a09

ncw approved these changes Aug 28, 2025

View reviewed changes

ncw merged commit 89a8ea7 into rclone:master Aug 28, 2025
17 of 18 checks passed

	rclone lsf remote:path --format pt --time-format 'Jan 2, 2006 at 3:04pm (MST)'
	rclone lsf remote:path --format pt --time-format '2006-01-02 15:04:05.000000000'
	rclone lsf remote:path --format pt --time-format '2006-01-02T15:04:05.999999999Z07:00'
	rclone lsf remote:path --format pt --time-format RFC3339
	rclone lsf remote:path --format pt --time-format DateOnly
	rclone lsf remote:path --format pt --time-format max

Uh oh!

fs/operations: add support for unix and unixnano time formats #8726

fs/operations: add support for unix and unixnano time formats #8726

Uh oh!

Conversation

dmotte commented Aug 4, 2025

What is the purpose of this change?

Was the change discussed in an issue or in the forum before?

Checklist

Uh oh!

albertony left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dmotte commented Aug 10, 2025

Uh oh!

albertony commented Aug 10, 2025

Uh oh!

dmotte commented Aug 11, 2025

Uh oh!

albertony commented Aug 11, 2025

Uh oh!

ncw commented Aug 13, 2025

Uh oh!

ncw left a comment

Choose a reason for hiding this comment

Uh oh!

ncw Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

dmotte Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

ncw Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

dmotte Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

nielash Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

ncw Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

dmotte Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

dmotte commented Aug 13, 2025

Uh oh!

nielash commented Aug 13, 2025

Uh oh!

dmotte commented Aug 13, 2025

Uh oh!

dmotte commented Aug 28, 2025

Uh oh!

dmotte commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

albertony commented Aug 28, 2025

Uh oh!

dmotte commented Aug 28, 2025

Uh oh!

ncw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dmotte commented Aug 28, 2025 •

edited

Loading