Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

4.1 Hourly Model #537

Merged
merged 145 commits into from
Jan 28, 2025
Merged
Changes from 1 commit
Commits
Show all changes
145 commits
Select commit Hold shift + click to select a range
b8a6060
initial implement of new hourly model on meter level
ArminRecurve Mar 13, 2024
eadaf2c
add errors and pre population run
ArminRecurve Mar 15, 2024
07b3331
added hourly settings, serialization, and baseline metrics
travis-recurve Mar 15, 2024
7edd44a
added safe_divide to metrics
travis-recurve Mar 19, 2024
7206752
minor changes to settings etc
travis-recurve Mar 19, 2024
8e76252
Population run set, error on to_json
ArminRecurve Mar 21, 2024
24bb188
Added ReportingMetrics
travis-recurve Mar 21, 2024
6874735
Merge: adding ReportingMetrics
travis-recurve Mar 21, 2024
8d62b2e
isolating testing changes in metrics.py
travis-recurve Mar 21, 2024
8f9c81f
population run before hyper-parameter optimization
ArminRecurve Mar 26, 2024
fc169ec
population run with saved features for hyperparameter tunning
ArminRecurve Mar 27, 2024
a65f3ad
Population optimizaiton with ghi
ArminRecurve Apr 2, 2024
bb399f3
adding prepare_dataframe to hourly
travis-recurve Apr 8, 2024
d226319
add HourlyData class draft, add supplemental data
ArminRecurve Apr 19, 2024
8421bc2
bug fixing with temporary files
travis-recurve Apr 23, 2024
9b4fbec
supplemental figures add
ArminRecurve Apr 23, 2024
4d9a8c5
plot change for ads retreat presentation
ArminRecurve May 7, 2024
353ce06
inital interpolation with RBFInterpolator
ArminRecurve May 8, 2024
bc2e6cb
initial Interpolation class with multiple input
ArminRecurve May 9, 2024
bcca425
Interpolation class searching scnearios fix
ArminRecurve May 10, 2024
9bad148
interpolation in the hourly data class
ArminRecurve May 14, 2024
00f4a58
add time series and categorical supplemetal data to hourly model, rem…
ArminRecurve May 15, 2024
f6c2ad0
Merge branch 'master' into feature/solar_hourly_model
travis-recurve May 17, 2024
a281870
update branch to master
travis-recurve May 17, 2024
19af22d
Point test data to correct branch
jason-recurve May 6, 2024
84865dc
Convert datetime indexes to nanosecond resolution to prevent datafram…
jason-recurve May 6, 2024
b8a090e
Resample coverage from consistent origin
jason-recurve May 6, 2024
fe47b34
Created new branch for refactoring caltrack tests. Added revert for h…
joydeep-recurve Mar 5, 2024
7611e22
update branch to master
travis-recurve May 17, 2024
9f7f6de
Merge branch 'feature/solar_hourly_model' of github.com:openeemeter/e…
travis-recurve May 17, 2024
d2be840
initiate regimeshift analysis for PV installation date
ArminRecurve May 17, 2024
95bfeec
ready for final hourly model optimization
ArminRecurve May 20, 2024
59ec2fa
Merge branch 'feature/solar_hourly_model' of github.com:openeemeter/e…
travis-recurve May 20, 2024
99cc382
removed hourly opt from eemeter
travis-recurve May 21, 2024
29bd483
updating default alpha and L1_ratio values
travis-recurve May 31, 2024
cc80553
updating hourly data
travis-recurve Jun 2, 2024
a2e3462
removing cli
travis-recurve Jun 24, 2024
9591747
add data class and interpolator class to data.py in hourly model
ArminRecurve Jun 24, 2024
1a0fd2d
HourlySetting fix in init
ArminRecurve Jun 24, 2024
0e9652c
add temperature bins in the setting and model
ArminRecurve Jul 18, 2024
eea2177
minor modifications
travis-recurve Jul 19, 2024
f12af64
clustering data and model class ready for optimization
ArminRecurve Jul 20, 2024
5231a92
removing season categorical variable
travis-recurve Aug 7, 2024
97caaec
refactoring hourly model, fixing serialization,
travis-recurve Aug 15, 2024
04cf8da
adding optimized settings
travis-recurve Aug 21, 2024
2b67eea
forgot a comma
travis-recurve Aug 21, 2024
0b6b0a4
settings immutable and change baselinemetric repr
travis-recurve Aug 21, 2024
26d45ea
Rough merge of hourly dataclass into solar branch
jason-recurve Aug 22, 2024
0413504
logic fix for incomplete last day
travis-recurve Aug 22, 2024
984a8c7
Remove API key/info
jason-recurve Aug 24, 2024
f510e93
Account for daylight savings when creating feature matrices
jason-recurve Aug 27, 2024
4823f37
Fix hourly references in serialization test; remove unused cli test
jason-recurve Aug 27, 2024
5ce4088
Add unimplemented exception to from_series
jason-recurve Aug 27, 2024
ab54e3e
Blacken
jason-recurve Aug 27, 2024
cbb99ab
Handle midnight DST
jason-recurve Aug 27, 2024
6db7f51
Cleanup before alpha release
jason-recurve Aug 27, 2024
60fc176
Merge branch 'master' into feature/solar_hourly_model
jason-recurve Aug 27, 2024
60cf848
Add pyarrow requirement; put ceiling on numpy version
jason-recurve Aug 27, 2024
4da7ba0
Bump alpha version
jason-recurve Aug 27, 2024
fd2ec07
Update legacy hourly json test reference
jason-recurve Aug 27, 2024
7676486
Re-add __init__.py to previously deleted utilities module
jason-recurve Aug 28, 2024
6049b65
Use Optional[] syntax for pydantic model to support 3.9
jason-recurve Aug 28, 2024
6d3c92b
Bump alpha version
jason-recurve Aug 29, 2024
1b81d69
updating settings and new temperature input
travis-recurve Sep 13, 2024
0e9c242
changing settings
travis-recurve Sep 13, 2024
299e198
hourly update for opt
travis-recurve Sep 16, 2024
371b088
new temperature binning option
travis-recurve Sep 17, 2024
06ba64b
Handle DST index shifts for days removed during fit due to interpolation
jason-recurve Sep 19, 2024
4fd2cc8
Bump alpha version
jason-recurve Sep 19, 2024
d07d025
Readjust predictions to align with real datetimes
jason-recurve Sep 23, 2024
546e1a6
Bump alpha version
jason-recurve Sep 23, 2024
759ee99
integrating model changes and exposing data used to make feature matr…
travis-recurve Sep 25, 2024
7411ef1
removed defunct files
travis-recurve Sep 25, 2024
b0dbe70
_init_.py fixed
travis-recurve Sep 25, 2024
0894d1a
adding ghi interpolation
travis-recurve Sep 25, 2024
6581950
fixing supplemental data
travis-recurve Sep 25, 2024
3729732
updating model coeffs
travis-recurve Oct 1, 2024
50c50bd
updated interpolation (slightly faster) introduced edge bin functiona…
travis-recurve Oct 11, 2024
0a4f12d
changed clustering to FPCA/bisecting kmeans
travis-recurve Oct 12, 2024
e404c91
new clustering methodology
travis-recurve Oct 14, 2024
f84cd95
minor opt changes
travis-recurve Oct 15, 2024
998d691
updating dependencies
travis-recurve Oct 15, 2024
d9584fc
moving updated temporal clustering to main model
travis-recurve Oct 17, 2024
86c840c
updating settings - optimized
travis-recurve Oct 22, 2024
5877efb
Merge branch 'master' into feature/solar_hourly_model
jason-recurve Oct 24, 2024
a6f4e6b
Add __init__.py to clustering subpackage
jason-recurve Oct 24, 2024
0bda5e1
fixing interpolation to work for less data
travis-recurve Oct 25, 2024
15d6b37
altering interpolation logic slightly
travis-recurve Oct 26, 2024
c5fb71d
fix DST for billing
jason-recurve Nov 4, 2024
4680ade
new clustering method
travis-recurve Nov 8, 2024
b12fc61
hourly temp bins
travis-recurve Nov 18, 2024
33ef36c
temporal bins explained variance change
travis-recurve Nov 18, 2024
a253e6a
Check whether GHI is present before adding to interpolated columns
jason-recurve Nov 18, 2024
8b24714
X inputs to sparse matrices
travis-recurve Nov 19, 2024
a776bd5
alter default edge bin behavior
travis-recurve Nov 19, 2024
bd9b447
changing edge bin amount
travis-recurve Nov 19, 2024
bb5af19
Set extreme temp function with numpy
jason-recurve Nov 19, 2024
3d903de
fixed serialization
travis-recurve Nov 20, 2024
acd7a4d
remove unique from temperature sort
travis-recurve Nov 20, 2024
57ad530
removing redundant copying and making edge bin to pct
travis-recurve Nov 21, 2024
be1291b
extending edge bin slop
travis-recurve Nov 21, 2024
8e35523
update hourly settings and start on drmeter
travis-recurve Dec 8, 2024
855512a
working DRmodel
travis-recurve Dec 9, 2024
1bf9a47
simplifying DR model by altering EE hourly model
travis-recurve Dec 12, 2024
f0e7fb5
Add serialization/deserialization to hourly test
jason-recurve Nov 20, 2024
482ad16
Fix r_squared_adj when _safe_divide returns None
jason-recurve Nov 20, 2024
54afa18
Bump alpha version
jason-recurve Nov 20, 2024
6bbdb08
Add base hourly documentation, make some attrs private
jason-recurve Dec 17, 2024
b8bb857
Copy Steve's docs fix onto solar branch
jason-recurve Dec 17, 2024
31fd919
Add pywt to setup.py
jason-recurve Jan 13, 2025
c73e311
Merge branch 'master' into feature/solar_hourly_model
jason-recurve Jan 13, 2025
d61931a
Add hourly data tests
jason-recurve Jan 22, 2025
e4503a8
Update hourly serialization
jason-recurve Jan 23, 2025
ffc6191
Move sufficiency check to pydantic, begin refactor
jason-recurve Jan 23, 2025
274451a
Detect solar/nonsolar during fit()
jason-recurve Jan 25, 2025
454f571
Make hourly settings keys lowercase for consistency with daily model
jason-recurve Jan 25, 2025
dff74ba
Override pydantic __str__ in EEMeterWarning
jason-recurve Jan 25, 2025
e790992
Allow dict or list in EEMeterWarning data
jason-recurve Jan 25, 2025
5aa766e
Make pv_start first class parameter
jason-recurve Jan 25, 2025
aa054a3
Add hourly from_series for observed and temperature only
jason-recurve Jan 25, 2025
6308c02
Revert hourly from_series
jason-recurve Jan 25, 2025
c690296
Add CVRMSE/PNRMSE disqualification
jason-recurve Jan 25, 2025
32cba4e
Update hourly percentage tests
jason-recurve Jan 26, 2025
4b46e16
Make a few functions private inside of hourly.model
jason-recurve Jan 26, 2025
136d3da
Correct hourly frequency label
jason-recurve Jan 27, 2025
ba20fa2
Replace deprecated pydantic config
jason-recurve Jan 27, 2025
1d15d82
Fix data download path
jason-recurve Jan 27, 2025
7bdc570
Explicitly set seed datatype for windows compatibility
jason-recurve Jan 27, 2025
c7eeed5
Drop 3.8 from supported runtimes
jason-recurve Jan 27, 2025
91cde8b
Blacken
jason-recurve Jan 27, 2025
9795321
Avoid SettingWithCopyWarning
jason-recurve Jan 27, 2025
6825576
Suppress pywt warning with notes
jason-recurve Jan 27, 2025
c539f7d
Fix y_scaler named feature warning and numpy implicit scalar cast
jason-recurve Jan 27, 2025
a5f45a1
Suppress sklearn deprecation warning with note; exclude suppressed wa…
jason-recurve Jan 27, 2025
8b5f349
Explicitly set masked values to np.nan
jason-recurve Jan 27, 2025
bf5cd6a
Skip extreme value check for empty observed
jason-recurve Jan 27, 2025
c0a3d50
Suppress warning instead of manually setting masked values to nan
jason-recurve Jan 27, 2025
6096391
Remove default error filter from pytest for now
jason-recurve Jan 28, 2025
43882d2
Remove new_hourly module from initial update
jason-recurve Jan 28, 2025
a1298dd
Remove unused private attrs in hourly data
jason-recurve Jan 28, 2025
ddf1060
Suppress warnings
jason-recurve Jan 28, 2025
9212bf4
Allow using dict settings in HourlyModel
jason-recurve Jan 28, 2025
5ebfdd9
Merge branch 'master' into feature/solar_hourly_model
jason-recurve Jan 28, 2025
918a4cd
Blacken
jason-recurve Jan 28, 2025
0a60284
Update changelog and bump version
jason-recurve Jan 28, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Suppress warning instead of manually setting masked values to nan
Signed-off-by: Jason Chulock <jason@recurve.com>
  • Loading branch information
jason-recurve committed Jan 28, 2025
commit c0a3d50d20dd6fc405680b3f2de129aec136040e
9 changes: 4 additions & 5 deletions eemeter/common/hourly_interpolation.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,10 @@ def autocorr_fcn(x, lags, exclude_0=True):
xp = x_msk - mean
corr = [1.0 if l == 0 else ma.sum(xp[l:] * xp[:-l]) / len(x) / var for l in lags]

# explicitly cast masked values to nan to avoid warning
corr = [np.nan if type(v) == ma.core.MaskedConstant else v for v in corr]

# combine the lags, the correlation values, and mirror to get leads/lags
res = np.vstack((lags, corr)).T
with warnings.catch_warnings():
warnings.filterwarnings("ignore", "Warning: converting a masked element to nan")
# combine the lags, the correlation values, and mirror to get leads/lags
res = np.vstack((lags, corr)).T
if exclude_0: # remove the 0 lag
res = res[1:]
rev_res = copy(res)[::-1]
Expand Down