Skip to content

Wide arg read csv excel (issue #574) #588

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions doc/source/changes/version_0_28.rst.inc
Original file line number Diff line number Diff line change
Expand Up @@ -289,6 +289,17 @@ Miscellaneous improvements
Argument `transpose` has a different purpose than `wide` and is mainly useful to allow multiple axes as header
when exporting arrays with more than 2 dimensions. Closes :issue:`575` and :issue:`371`.

* added argument `wide` to `read_csv` and `read_excel` functions. If False, the array to be loaded is assumed to
be stored in "narrow" format:

>>> # assuming the array was saved using command: arr.to_excel('my_file.xlsx', wide=False)
>>> read_excel('my_file.xlsx', wide=False)
a\b b0 b1 b2
a0 0 1 2
a1 3 4 5

Closes :issue:`574`.

* added argument `name` to `to_series` method allowing to set a name to the Pandas Series returned by the method.

* added argument `value_name` to `to_csv` and `to_excel` allowing to change the default name ('value') to
Expand Down
200 changes: 148 additions & 52 deletions larray/inout/array.py

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions larray/inout/excel.py
Original file line number Diff line number Diff line change
Expand Up @@ -418,9 +418,9 @@ def __setattr__(self, key, value):
setattr(self.xw_sheet, key, value)

def load(self, header=True, convert_float=True, nb_index=None, index_col=None, fill_value=np.nan,
sort_rows=False, sort_columns=False):
sort_rows=False, sort_columns=False, wide=True):
return self[:].load(header=header, convert_float=convert_float, nb_index=nb_index, index_col=index_col,
sort_rows=sort_rows, sort_columns=sort_columns, fill_value=fill_value)
fill_value=fill_value, sort_rows=sort_rows, sort_columns=sort_columns, wide=wide)

# TODO: generalize to more than 2 dimensions or scrap it
def array(self, data, row_labels=None, column_labels=None, names=None):
Expand Down Expand Up @@ -547,15 +547,15 @@ def __str__(self):
__repr__ = __str__

def load(self, header=True, convert_float=True, nb_index=None, index_col=None, fill_value=np.nan,
sort_rows=False, sort_columns=False):
sort_rows=False, sort_columns=False, wide=True):
if not self.ndim:
return LArray([])

list_data = self._converted_value(convert_float=convert_float)

if header:
return from_lists(list_data, nb_index=nb_index, index_col=index_col, fill_value=fill_value,
sort_rows=sort_rows, sort_columns=sort_columns)
sort_rows=sort_rows, sort_columns=sort_columns, wide=wide)
else:
return LArray(list_data)

Expand Down
Binary file modified larray/tests/data/test.xlsx
Binary file not shown.
2 changes: 1 addition & 1 deletion larray/tests/data/test1d.csv
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
time,2007,2010,2013
a,a0,a1,a2
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using a0 etc... is probably a good idea, but then we need another test for "int" labels (to check they are parsed to int correctly and do not stay strings). This could use from_string or from_list instead of a file though.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't already the case for from_lists and from_string?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the from_list test does not count as the year labels already are of the correct type (it uses 1991; not '1991'), and there is no test in from_string doctest with int-like labels in that last axis, and the tests do not check the type of the labels.

,0,1,2
4 changes: 4 additions & 0 deletions larray/tests/data/test1d_narrow.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
a,value
a0,0
a1,1
a2,2
7 changes: 4 additions & 3 deletions larray/tests/data/test2d.csv
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
a\b,0,1,2
0,0,1,2
1,3,4,5
a\b,b0,b1
1,0,1
2,2,3
3,4,5
10 changes: 4 additions & 6 deletions larray/tests/data/test2d_classic.csv
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
age,2007,2010,2013
0,3722,3395,3347
1,338,316,323
2,2878,2791,2822
3,1121,1037,976
4,4073,4161,4429
a,b0,b1,b2
a0,0,1,2
a1,3,4,5
a2,6,7,8
7 changes: 7 additions & 0 deletions larray/tests/data/test2d_narrow.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
a,b,value
1,b0,0
1,b1,1
2,b0,2
2,b1,3
3,b0,4
3,b1,5
16 changes: 7 additions & 9 deletions larray/tests/data/test3d.csv
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
age,sex\time,2015,2016,2017
0,F,0.5,1.5,2.5
0,M,3.5,4.5,5.5
1,F,6.5,7.5,8.5
1,M,9.5,10.5,11.5
2,F,12.5,13.5,14.5
2,M,15.5,16.5,17.5
3,F,18.5,19.5,20.5
3,M,21.5,22.5,23.5
a,b\c,c0,c1,c2
1,b0,0,1,2
1,b1,3,4,5
2,b0,6,7,8
2,b1,9,10,11
3,b0,12,13,14
3,b1,15,16,17
19 changes: 19 additions & 0 deletions larray/tests/data/test3d_narrow.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
a,b,c,value
1,b0,c0,0
1,b0,c1,1
1,b0,c2,2
1,b1,c0,3
1,b1,c1,4
1,b1,c2,5
2,b0,c0,6
2,b0,c1,7
2,b0,c2,8
2,b1,c0,9
2,b1,c1,10
2,b1,c2,11
3,b0,c0,12
3,b0,c1,13
3,b0,c2,14
3,b1,c0,15
3,b1,c1,16
3,b1,c2,17
41 changes: 0 additions & 41 deletions larray/tests/data/test5d.csv

This file was deleted.

Binary file added larray/tests/data/test_narrow.xlsx
Binary file not shown.
10 changes: 10 additions & 0 deletions larray/tests/data/testint_labels.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
a,b\c,0,1,2
0,0,0,1,2
0,1,3,4,5
0,2,6,7,8
1,0,9,10,11
1,1,12,13,14
1,2,15,16,17
2,0,18,19,20
2,1,21,22,23
2,2,24,25,26
5 changes: 5 additions & 0 deletions larray/tests/data/testmissing_values.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
a,b\c,c0,c1,c2
1,b0,0,1,2
1,b1,3,4,5
2,b1,9,10,11
3,b0,12,13,14
12 changes: 12 additions & 0 deletions larray/tests/data/testmissing_values_narrow.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
a,b,c,value
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be nice if this test also included missing values for c

1,b0,c0,0
1,b0,c1,1
1,b0,c2,2
1,b1,c0,3
1,b1,c1,4
1,b1,c2,5
2,b1,c0,9
2,b1,c2,11
3,b0,c0,12
3,b0,c1,13
3,b0,c2,14
19 changes: 19 additions & 0 deletions larray/tests/data/testunsorted_narrow.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
a,b,c,value
3,b1,c2,0
3,b1,c1,1
3,b1,c0,2
3,b0,c2,3
3,b0,c1,4
3,b0,c0,5
2,b1,c2,6
2,b1,c1,7
2,b1,c0,8
2,b0,c2,9
2,b0,c1,10
2,b0,c0,11
1,b1,c2,12
1,b1,c1,13
1,b1,c0,14
1,b0,c2,15
1,b0,c1,16
1,b0,c0,17
Loading