Skip to content

Commit 54067bf

Browse files
committed
Replace the testcases script with a Vagrant VM
See the README section for more details. Basically, with this way, we can store a bunch of existing valid exa outputs, change a VM's environment to match our values, then check that exa still works by comparing outputs.
1 parent d321434 commit 54067bf

28 files changed

+558
-217
lines changed

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
11
*~
22
target
33
testcases
4+
.vagrant

.travis.yml

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,4 @@ rust:
88
- stable
99
script:
1010
- cargo build --verbose
11-
- yes | sudo ./generate-testcases.sh
1211
- cargo test --verbose
13-
- cargo run
14-
- cargo run -- --long
15-
- cargo run -- --long --grid
16-
- cargo run -- --tree
17-
- cargo run -- --tree --long
18-
- cargo run -- --recurse
19-
- cargo run -- --recurse --long --extended

README.md

Lines changed: 33 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -66,15 +66,43 @@ If you’re unable to compile libgit2, you can opt out of Git support by running
6666

6767
### Cargo Install
6868

69-
If you're using a recent version of Cargo (0.5.0 or higher), you can
70-
use the `cargo install` command:
69+
If you’re using a recent version of Cargo (0.5.0 or higher), you can use the `cargo install` command:
7170

7271
cargo install --git https://github.com/ogham/exa
7372

7473
or:
7574

7675
cargo install --no-default-features --git https://github.com/ogham/exa
7776

78-
Cargo will clone the repository to a temporary directory, build it
79-
there and place the `exa` binary to: `$HOME/.cargo` (and can be
80-
overridden by setting the `--root` option).
77+
Cargo will clone the repository to a temporary directory, build it there and place the `exa` binary to: `$HOME/.cargo` (and can be overridden by setting the `--root` option).
78+
79+
80+
## Testing with Vagrant
81+
82+
exa uses [Vagrant][] to configure virtual machines for testing.
83+
84+
Programs such as exa that are basically interfaces to the system are [notoriously difficult to test][testing]. Although the internal components have unit tests, it’s impossible to do a complete end-to-end test without mandating the current user’s name, the time zone, the locale, and directory structure to test. (And yes, these tests are worth doing. I have missed an edge case on more than one occasion.)
85+
86+
The initial attempt to solve the problem was just to create a directory of “awkward” test cases, run exa on it, and make sure it produced the correct output. But even this output would change if, say, the user’s locale formats dates in a different way. These can be mocked inside the code, but at the cost of making that code more complicated to read and understand.
87+
88+
An alternative solution is to fake *everything*: create a virtual machine with a known state and run the tests on *that*. This is what Vagrant does. Although it takes a while to download and set up, it gives everyone the same development environment to test for any obvious regressions.
89+
90+
[Vagrant]: https://www.vagrantup.com/docs/why-vagrant/
91+
[testing]: https://eev.ee/blog/2016/08/22/testing-for-people-who-hate-testing/#troublesome-cases
92+
93+
First, initialise the VM:
94+
95+
host$ vagrant up
96+
97+
The first command downloads the virtual machine image, and then runs our provisioning script, which installs Rust, exa’s dependencies, configures the environment, and generates some awkward files and folders to use as test cases. This takes some time, but it does write to output occasionally. Once this is done, you can SSH in, and build and test:
98+
99+
host$ vagrant ssh
100+
vm$ cd /vagrant
101+
vm$ cargo build
102+
vm$ ./xtests/run
103+
All the tests passed!
104+
105+
106+
### Running without Vagrant
107+
108+
Of course, the drawback of having a standard development environment is that you stop noticing bugs that occur outside of it. For this reason, Vagrant isn’t a *necessary* development step — it’s there if you’d like to use it, but exa still gets used and tested on other platforms. It can still be built and compiled on any target triple that it supports, VM or no VM, with `cargo build` and `cargo test`.

Vagrantfile

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
Vagrant.configure("2") do |config|
2+
config.vm.provider "virtualbox" do |v|
3+
v.memory = 1024
4+
v.cpus = 1
5+
end
6+
7+
config.vm.box = "debian/contrib-jessie64"
8+
config.vm.hostname = "exa"
9+
10+
# Install the dependencies needed for exa to build.
11+
config.vm.provision :shell, privileged: true, inline:
12+
%[apt-get install -y git cmake libgit2-dev libssh2-1-dev curl attr]
13+
14+
# Guarantee that the timezone is UTC -- some of the tests
15+
# depend on this (for now).
16+
config.vm.provision :shell, privileged: true, inline:
17+
%[timedatectl set-timezone UTC]
18+
19+
# Install Rust.
20+
# This is done as vagrant, not root, because it’s vagrant
21+
# who actually uses it. Sent to /dev/null because the progress
22+
# bar produces a lot of output.
23+
config.vm.provision :shell, privileged: false, inline:
24+
%[hash rustc &>/dev/null || curl -sSf https://static.rust-lang.org/rustup.sh | sh &> /dev/null]
25+
26+
# Use a different ‘target’ directory on the VM than on the host.
27+
# By default it just uses the one in /vagrant/target, which can
28+
# cause problems if it has different permissions than the other
29+
# directories, or contains object files compiled for the host.
30+
config.vm.provision :shell, privileged: false, inline:
31+
%[echo "export CARGO_TARGET_DIR=/home/vagrant/target" >> ~/.bashrc]
32+
33+
# Test that wide columns work with a really long username.
34+
# The benefit of Vagrant is that we don’t need to set this up
35+
# on the *actual* system!
36+
longuser = "antidisestablishmentarienism"
37+
config.vm.provision :shell, privileged: true, inline:
38+
%[id -u #{longuser} &>/dev/null || useradd #{longuser}]
39+
40+
test_dir = "/home/vagrant/testcases"
41+
invalid_uid = 666
42+
invalid_gid = 616
43+
some_date = "201601011234.56" # 1st January 2016, 12:34:56
44+
45+
# Delete old testcases if they exist already.
46+
# This needs root because the generator does some sudo-ing.
47+
config.vm.provision :shell, privileged: true, inline:
48+
%[rm -rfv #{test_dir}]
49+
50+
# Generate our awkward testcases.
51+
config.vm.provision :shell, privileged: false, inline:
52+
%[mkdir #{test_dir}]
53+
54+
# Awkward file size testcases.
55+
config.vm.provision :shell, privileged: false, inline: <<-EOF
56+
set -xe
57+
mkdir "#{test_dir}/files"
58+
for i in {1..13}; do
59+
fallocate -l "$i" "#{test_dir}/files/$i"_bytes
60+
fallocate -l "$i"KiB "#{test_dir}/files/$i"_KiB
61+
fallocate -l "$i"MiB "#{test_dir}/files/$i"_MiB
62+
done
63+
touch -t #{some_date} "#{test_dir}/files/"*
64+
EOF
65+
66+
# Awkward symlink testcases.
67+
config.vm.provision :shell, privileged: false, inline: <<-EOF
68+
set -xe
69+
mkdir "#{test_dir}/links"
70+
ln -s / "#{test_dir}/links/root"
71+
ln -s /usr "#{test_dir}/links/usr"
72+
ln -s nowhere "#{test_dir}/links/broken"
73+
EOF
74+
75+
# Awkward passwd testcases.
76+
# sudo is needed for these because we technically aren’t a member
77+
# of the groups (because they don’t exist), and chown and chgrp
78+
# are smart enough to disallow it!
79+
config.vm.provision :shell, privileged: false, inline: <<-EOF
80+
set -xe
81+
mkdir "#{test_dir}/passwd"
82+
83+
touch -t #{some_date} "#{test_dir}/passwd/unknown-uid"
84+
sudo chown #{invalid_uid} "#{test_dir}/passwd/unknown-uid"
85+
86+
touch -t #{some_date} "#{test_dir}/passwd/unknown-gid"
87+
sudo chgrp #{invalid_gid} "#{test_dir}/passwd/unknown-gid"
88+
EOF
89+
90+
# Awkward permission testcases.
91+
config.vm.provision :shell, privileged: false, inline: <<-EOF
92+
set -xe
93+
mkdir "#{test_dir}/permissions"
94+
95+
touch "#{test_dir}/permissions/all-permissions"
96+
chmod 777 "#{test_dir}/permissions/all-permissions"
97+
98+
touch "#{test_dir}/permissions/no-permissions"
99+
chmod 000 "#{test_dir}/permissions/no-permissions"
100+
101+
mkdir "#{test_dir}/permissions/forbidden-directory"
102+
chmod 000 "#{test_dir}/permissions/forbidden-directory"
103+
104+
touch -t #{some_date} "#{test_dir}/permissions/"*
105+
EOF
106+
107+
108+
# Awkward extended attribute testcases.
109+
config.vm.provision :shell, privileged: false, inline: <<-EOF
110+
set -xe
111+
mkdir "#{test_dir}/attributes"
112+
113+
touch "#{test_dir}/attributes/none"
114+
115+
touch "#{test_dir}/attributes/one"
116+
setfattr -n user.greeting -v hello "#{test_dir}/attributes/one"
117+
118+
touch "#{test_dir}/attributes/two"
119+
setfattr -n user.greeting -v hello "#{test_dir}/attributes/two"
120+
setfattr -n user.another_greeting -v hi "#{test_dir}/attributes/two"
121+
122+
#touch "#{test_dir}/attributes/forbidden"
123+
#setfattr -n user.greeting -v hello "#{test_dir}/attributes/forbidden"
124+
#chmod +a "$YOU deny readextattr" "#{test_dir}/attributes/forbidden"
125+
126+
mkdir "#{test_dir}/attributes/dirs"
127+
128+
mkdir "#{test_dir}/attributes/dirs/empty-with-attribute"
129+
setfattr -n user.greeting -v hello "#{test_dir}/attributes/dirs/empty-with-attribute"
130+
131+
mkdir "#{test_dir}/attributes/dirs/full-with-attribute"
132+
touch "#{test_dir}/attributes/dirs/full-with-attribute/file"
133+
setfattr -n user.greeting -v hello "#{test_dir}/attributes/dirs/full-with-attribute"
134+
135+
mkdir "#{test_dir}/attributes/dirs/full-but-forbidden"
136+
touch "#{test_dir}/attributes/dirs/full-but-forbidden/file"
137+
#setfattr -n user.greeting -v hello "#{test_dir}/attributes/dirs/full-but-forbidden"
138+
#chmod 000 "#{test_dir}/attributes/dirs/full-but-forbidden"
139+
#chmod +a "$YOU deny readextattr" "#{test_dir}/attributes/dirs/full-but-forbidden"
140+
141+
touch -t #{some_date} "#{test_dir}/attributes"
142+
touch -t #{some_date} "#{test_dir}/attributes/"*
143+
touch -t #{some_date} "#{test_dir}/attributes/dirs/"*
144+
touch -t #{some_date} "#{test_dir}/attributes/dirs/"*/*
145+
EOF
146+
end

generate-testcases.sh

Lines changed: 0 additions & 138 deletions
This file was deleted.

tests/directories.rs

Lines changed: 0 additions & 24 deletions
This file was deleted.

tests/links.rs

Lines changed: 0 additions & 21 deletions
This file was deleted.

0 commit comments

Comments
 (0)