Refac rolling (blue-yonder#173)

MaxBenChrist · web-flow · commit 89be37b0e226 · 2017-03-29T13:49:57.000+02:00
* we should track the requirements of the notebook files * added github ISSUE_template * updated question about rolling in FAQ * refactored page about rolling mechanism * refactored docstring of rolling utility * it is reproduce instead of produce * linked to data formats in faq * fixed two typos in industry 4.0 examples * clarified that rolling=0 does nothing * polished first paragrapoh of rolling page * now the medical example should be more clear * verb was missing in roll_time_series method * parameter should be checked directly * we do not have a shift parameter for the rolling * rolling direction of 0 should just return ts container * Revert "rolling direction of 0 should just return ts container" This reverts commit 7278060. * a maximum length of time series of NaN should break rolling * adapted unti tests to value error on single row rolling * added possible reason for nan maximum length * removed paratnethesis in DataFrame constructor
diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md
@@ -0,0 +1,10 @@
+Oh no, you encountered a problem while using *tsfesh*.
+
+We, the maintainers, are happy to help you. When opening an issue, please provide the following information to us:
+
+1. Your operating system
+2. The version of *tsfresh* that you are using
+3. The data on which the problem occurred (please do not upload 1000s of time series but try to boil the problem down to a small group or even a singular one)
+4. A minimal code snippet which reproduces the problem/bug
+
+For questions, you can also use our [gitter chatroom](https://gitter.im/tsfresh/)
diff --git a/docs/text/faq.rst b/docs/text/faq.rst
@@ -2,13 +2,15 @@ FAQ
 ===
 
 
-    1. *Does tsfresh support different time series lengths?*
+    1. **Does tsfresh support different time series lengths?**
        Yes, it supports different time series lengths. However, some feature calculators can demand a minimal length
        of the time series. If a shorter time series is passed to the calculator, normally a NaN is returned.
 
 
-    2. *Is it possible to extract features from rolling/shifted time series?*
-       Yes, there is the option `rolling` for the :func:`tsfresh.feature_extraction.extract_features` function.
-       Set it to a non-zero value to enable rolling. In the moment, this just rolls the input data into
-       as many time series as there are time steps - so there is no internal optimization for rolling calculations.
-       Please see :ref:`rolling-label` for more information.
+    2. **Is it possible to extract features from rolling/shifted time series?**
+       Yes, the :func:`tsfresh.dataframe_functions.roll_time_series` function allows to conviniently create a rolled
+       time series datframe from your data. You just have to transform your data into one of the supported tsfresh
+       :ref:`data-formats-label`.
+       Then, the :func:`tsfresh.dataframe_functions.roll_time_series` give you a DataFrame with the rolled time series,
+       that you can pass to tsfresh.
+       On the following page you can find a detailed description: :ref:`rolling-label`.
diff --git a/docs/text/rolling.rst b/docs/text/rolling.rst
@@ -3,26 +3,32 @@
 How to handle rolling time series
 =================================
 
-In many application with time series on real-world problems, the "time" column
-(we will call it time in the following, although it can be anything)
-gives a certain sequential order to the data. We can exploit this sequence to generate
-more input data out of single time series, by *rolling* over the data.
-
-Imagine the following situation: you have the data of EEG measurements, that
-you want to use to classify patients into healthy and not healthy (we oversimplify the problem here).
-You have e.g. 100 time steps of data, so you can extract features that may forecast the healthiness
-of the patients. But what would happen if you had only the recorded measurement for 50 time steps?
-The patients would be as healthy as with 100 time steps. So you can easily increase the amount of
-training data by reusing time series cut into smaller pieces.
-
-Another example is streaming data, e.g. in Industry 4.0 applications. Here you typically get one
-new data row at a time and use this to predict machine failures for example. To train you model,
+Lets assume that we have a DataFrame of one of the tsfresh :ref:`data-formats-label`.
+The "sort" column of such a container gives a sequential state to the individual measurements.
+In the case of time series this can be the *time* dimension while in the case of spectra the order is given by the
+*wavelength* or *frequency* dimensions.
+We can exploit this sequence to generate more input data out of single time series, by *rolling* over the data.
+
+Imagine the following situation:
+You have the data of certain sensors (e.g. EEG measurements) as the base to classify patients into a healthy and not
+healthy group (we oversimplify the problem here).
+Lets say you have sensor data of 100 time steps, so you may extract features for the forecasting of the patients
+healthiness by a classification algorithm.
+If you also have measurements of the healthiness for those 100 time steps (this is the target vector), then you could
+predict the healthiness of the patient in every time step, which essentially states a time series forecasting problem.
+So, to do that, you want to extract features in every time step of the original time series while for example looking at
+the last 10 steps.
+A rolling mechanism creates such time series for every time step by creating sub time series of the sensor data of the
+last 10 time steps.
+
+Another example can be found in streaming data, e.g. in Industry 4.0 applications.
+Here you typically get one new data row at a time and use this to for example predict machine failures. To train your model,
 you could act as if you would stream the data, by feeding your classifier the data after one time step,
 the data after the first two time steps etc.
 
 Both examples imply, that you extract the features not only on the full data set, but also
-on all temporal coherent subsets of data, which is the process of *rolling*. You can do this easily,
-by calling the function :func:`tsfresh.utilities.dataframe_functions.roll_time_series`.
+on all temporal coherent subsets of data, which is the process of *rolling*. In tsfresh, this is implemented in the
+function :func:`tsfresh.utilities.dataframe_functions.roll_time_series`.
 
 The rolling mechanism takes a time series :math:`x` with its data rows :math:`[x_1, x_2, x_3, ..., x_n]`
 and creates :math:`n` new time series :math:`\hat x^k`, each of them with a different consecutive part
@@ -31,8 +37,7 @@ of :math:`x`:
 .. math::
     \hat x^k = [x_k, x_{k-1}, x_{k-2}, ..., x_1]
 
-To see what this does in real-world applications, we look into the following example data frame (we show only one possible data format,
-but rolling works on all 3 data formats :ref:`data-formats-label`):
+To see what this does in real-world applications, we look into the following example flat DataFrame in tsfresh format
 
 +----+------+----+----+
 | id | time | x  | y  |
@@ -50,9 +55,13 @@ but rolling works on all 3 data formats :ref:`data-formats-label`):
 | 2  | t9   | 11 | 13 |
 +----+------+----+----+
 
-where you have measured two values (x and y) for two different entities (1 and 2) in 4 or 2 time steps.
+where you have measured the values from two sensors x and y for two different entities (id 1 and 2) in 4 or 2 time
+steps (t1 to t9).
 
-If you set `rolling` to 0, the feature extraction works on
+Now, we can use :func:`tsfresh.utilities.dataframe_functions.roll_time_series` to get consecutive sub-time series.
+E.g. if you set `rolling` to 0, the feature extraction works on the original time series without any rolling.
+
+So it extracts 2 set of features,
 
 +----+------+----+----+
 | id | time | x  | y  |
@@ -76,8 +85,6 @@ and
 | 2  | t9   | 11 | 13 |
 +----+------+----+----+
 
-So it extracts 2 set of features.
-
 If you set rolling to 1, the feature extraction works with all of the following time series:
 
 +----+------+----+----+
@@ -164,4 +171,7 @@ If you set rolling to -1, you end up with features for the time series, rolled i
 | 2  | t8   | 10 | 12 |
 +----+------+----+----+
 | 2  | t9   | 11 | 13 |
-+----+------+----+----+
++----+------+----+----+
+
+We only gave an example for the flat DataFrame format, but rolling actually works on all 3 :ref:`data-formats-label`
+that are supported by tsfresh.
diff --git a/notebooks-requirements.txt b/notebooks-requirements.txt
@@ -0,0 +1,4 @@
+matplotlib==2.0.0
+seaborn==0.7.1
+ipython==5.3.0
+notebook==4.4.1
diff --git a/tests/utilities/test_dataframe_functions.py b/tests/utilities/test_dataframe_functions.py
@@ -164,13 +164,13 @@ def test_with_wrong_input(self):
 
 class RollingTestCase(TestCase):
     def test_with_wrong_input(self):
-        test_df = pd.DataFrame([{"id": 0, "kind": "a", "value": 3, "sort": np.NaN}])
+        test_df = pd.DataFrame({"id": [0, 0], "kind": ["a", "b"], "value": [3, 3], "sort": [np.NaN, np.NaN]})
         self.assertRaises(ValueError, dataframe_functions.roll_time_series,
                           df_or_dict=test_df, column_id="id",
                           column_sort="sort", column_kind="kind",
                           rolling_direction=1)
 
-        test_df = pd.DataFrame([{"id": 0, "kind": "a", "value": 3, "sort": 1}])
+        test_df = pd.DataFrame({"id": [0, 0], "kind": ["a", "b"], "value": [3, 3], "sort": [1, 1]})
         self.assertRaises(AttributeError, dataframe_functions.roll_time_series,
                           df_or_dict=test_df, column_id="strange_id",
                           column_sort="sort", column_kind="kind",
@@ -197,12 +197,12 @@ def test_with_wrong_input(self):
                           column_sort=None, column_kind=None,
                           rolling_direction=0)
 
-    def test_single_row(self):
+    def test_assert_single_row(self):
         test_df = pd.DataFrame([{"id": np.NaN, "kind": "a", "value": 3, "sort": 1}])
-        dataframe_functions.roll_time_series(
-            df_or_dict=test_df, column_id="id",
-            column_sort="sort", column_kind="kind",
-            rolling_direction=1)
+        self.assertRaises(ValueError, dataframe_functions.roll_time_series,
+                          df_or_dict=test_df, column_id="id",
+                          column_sort="sort", column_kind="kind",
+                          rolling_direction=1)
 
     def test_positive_rolling(self):
         first_class = pd.DataFrame({"a": [1, 2, 3, 4], "b": [5, 6, 7, 8], "time": range(4)})
diff --git a/tsfresh/utilities/dataframe_functions.py b/tsfresh/utilities/dataframe_functions.py
@@ -330,12 +330,19 @@ def normalize_input_to_internal_representation(df_or_dict, column_id, column_sor
 
 def roll_time_series(df_or_dict, column_id, column_sort, column_kind, rolling_direction):
     """
-    Roll the (sorted) data frames for each kind and each id separately in "time"
-    (time is here the abstract sort order defined by the sort column). For each rolling step a new id will be
-    created, with the name "id={id}, shift={shift}" where the id is the former id of the column and shift is the
-    amount of "time" shifts. ATTENTION: This will (obviously) create new IDs! The sign of rolling defines the
-    direction of time rolling.
-    For more information, please see :ref:`rolling-label`.
+    Roll the (sorted) data frames for each kind and each id separately in the "time" domain
+    (which is represented by the sort order of the sort column given by `column_sort`).
+
+    For each rolling step, a new id is created by the scheme "id={id}, shift={shift}", here id is the former id of the
+    column and shift is the amount of "time" shifts.
+
+    A few remarks:
+
+     * This method will create new IDs!
+     * The sign of rolling defines the direction of time rolling, a positive value means we are going back in time
+     * It is possible to shift time series of different lenghts but
+     * We assume that the time series are uniformly sampled
+     * For more information, please see :ref:`rolling-label`.
 
     :param df_or_dict: a pandas DataFrame or a dictionary. The required shape/form of the object depends on the rest of
         the passed arguments.
@@ -358,6 +365,9 @@ def roll_time_series(df_or_dict, column_id, column_sort, column_kind, rolling_di
     :rtype: the one from df_or_dict
     """
 
+    if rolling_direction == 0:
+        raise ValueError("Rolling direction of 0 is not possible")
+
     if isinstance(df_or_dict, dict):
         if column_kind is not None:
             raise ValueError("You passed in a dictionary and gave a column name for the kind. Both are not possible.")
@@ -405,14 +415,12 @@ def roll_time_series(df_or_dict, column_id, column_sort, column_kind, rolling_di
     # Roll the data frames if requested
     rolling_direction = np.sign(rolling_direction)
 
-    if rolling_direction == 0:
-        raise ValueError("Rolling direction of 0 is not possible")
-
     grouped_data = df.groupby(grouper)
     maximum_number_of_timeshifts = grouped_data.count().max().max()
 
     if np.isnan(maximum_number_of_timeshifts):
-        maximum_number_of_timeshifts = 0
+        raise ValueError("Somehow the maximum length of your time series is NaN (Does your time series container have "
+                         "only one row?). Can not perform rolling.")
 
     if rolling_direction > 0:
         range_of_shifts = range(maximum_number_of_timeshifts, -1, -1)
