Skip to content

Fix test framework running against real devices #1235

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Nov 11, 2024
Merged

Conversation

sdb9696
Copy link
Collaborator

@sdb9696 sdb9696 commented Nov 8, 2024

No description provided.

Copy link

codecov bot commented Nov 8, 2024

Codecov Report

Attention: Patch coverage is 76.92308% with 3 lines in your changes missing coverage. Please review.

Project coverage is 92.57%. Comparing base (32671da) to head (420f428).
Report is 193 commits behind head on master.

Files with missing lines Patch % Lines
kasa/smart/smartdevice.py 66.66% 1 Missing and 1 partial ⚠️
kasa/iot/iotdevice.py 75.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1235      +/-   ##
==========================================
- Coverage   92.60%   92.57%   -0.04%     
==========================================
  Files         101      101              
  Lines        6644     6657      +13     
  Branches      706      707       +1     
==========================================
+ Hits         6153     6163      +10     
- Misses        370      372       +2     
- Partials      121      122       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@rytilahti rytilahti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested on L530 and it did seemingly work (i.e., I saw the state of the bulb changing), but got some errors from a completely unrelated model:

FAILED kasa/tests/test_bulb.py::test_bulb_sysinfo[L510B(EU)_3.0_1.0.5.json-SMART] - voluptuous.error.MultipleInvalid: expected float for dictionary value @ data['longitude']
FAILED kasa/tests/test_bulb.py::test_hsv_on_non_color[L510B(EU)_3.0_1.0.5.json-SMART] - assert not True
FAILED kasa/tests/test_bulb.py::test_non_variable_temp[L510B(EU)_3.0_1.0.5.json-SMART] - Failed: DID NOT RAISE <class 'kasa.exceptions.KasaException'>

@ryenitcher
Copy link
Contributor

ryenitcher commented Nov 9, 2024

I can confirm that the tests now no longer crash for the EP40M, however the following errors remain:

======================================================================================== short test summary info ========================================================================================= 
FAILED tests\test_childdevice.py::test_child_time[P300(EU)_1.0_1.0.13.json-SMART] - AssertionError: assert FakeDatetime(2024, 11, 8, 17, 56, 32, tzinfo=zoneinfo.ZoneInfo(key='America/Denver')) != FakeDatetime(2024, 11, 8, 17, 56, 32, tzinfo=tzlocal())
FAILED tests\test_readme_examples.py::test_tutorial_examples - ValueError: Cannot find module=docs/tutorial.py

Results (103.99s (0:01:43)):
     458 passed
       2 failed
         - kasa\tests/test_childdevice.py:127 test_child_time[P300(EU)_1.0_1.0.13.json-SMART]
         - kasa\tests/test_readme_examples.py:140 test_tutorial_examples
    13290 skipped

I believe that the docs failure is due to the test being run from the within ./kasa directory, not the top level directory where the docs folder is. The python path would need updated to include the top level directory in order for this to run properly from the ./kasa or ./kasa/tests directory. (I forgot to activate the venv before running pytest, but the test fails even with the venv activated)

Meanwhile it appears that the time test may be failing because the fallback time is the same as the current time, which may be the same as the actual device time. I suggest using the unix epoch instead as the fallback time. Running the tests multiple times in a row shows that it is an intermittent failure, meaning it depends on if you get lucky and how bad your network latency is.

======================================================================================== short test summary info ========================================================================================= 
FAILED tests\test_readme_examples.py::test_tutorial_examples - ValueError: Cannot find module=docs/tutorial.py

Results (109.10s (0:01:49)):
     459 passed
       1 failed
         - kasa\tests/test_readme_examples.py:140 test_tutorial_examples
    13290 skipped

@sdb9696
Copy link
Collaborator Author

sdb9696 commented Nov 11, 2024

Tested on L530 and it did seemingly work (i.e., I saw the state of the bulb changing), but got some errors from a completely unrelated model:

This should now be fixed

I believe that the docs failure is due to the test being run from the within ./kasa directory, not the top level directory where the docs folder is.

That's correct. You should now be able to run this with uv run pytest --ip=<ip address> --username=<username> --password=<password> from the root project directory. Remember to include the = between option and value. Works ok for me on windows.

Meanwhile it appears that the time test may be failing because the fallback time is the same as the current time, which may be the same as the actual device time.

I've now excluded this from physical device testing so it should be skipped.

@sdb9696 sdb9696 requested a review from rytilahti November 11, 2024 15:28
@rytilahti
Copy link
Member

A couple of errors, but the model is now correct. Copying a couple of lines from the source for context:

FAILED tests/smart/modules/test_lighttransition.py::test_module_v1[L530E(EU)_3.0_1.0.6.json-SMART] - AssertionError: assert 'smooth_transitions' in {'smooth_transition_off': Smooth transition off (smooth_transition_off): 0 (range: 0-60), 'smooth_transition_on': Smooth transition on (smooth_transition_on): 0 (range...
>       assert "smooth_transitions" in light_transition._module_features
E       AssertionError: assert 'smooth_transitions' in {'smooth_transition_off': Smooth transition off (smooth_transition_off): 0 (range: 0-60), 'smooth_transition_on': Smooth transition on (smooth_transition_on): 0 (range: 0-60)}
E        +  where {'smooth_transition_off': Smooth transition off (smooth_transition_off): 0 (range: 0-60), 'smooth_transition_on': Smooth transition on (smooth_transition_on): 0 (range: 0-60)} = <Module LightTransition (on_off_gradually) for 

FAILED tests/test_common_modules.py::test_light_effect_brightness[L530E(EU)_3.0_1.0.6.json-SMART] - assert 1 == 50
        await light_module.set_brightness(50)
        await dev.update()
        assert light_effect.effect == light_effect.LIGHT_EFFECTS_OFF
>       assert light_module.brightness == 50
E       assert 1 == 50
E        +  where 1 = <Module Light (light) for 192.168.xx.xx>.brightness

@sdb9696 sdb9696 requested a review from rytilahti November 11, 2024 17:22
@sdb9696 sdb9696 added the maintenance Project improvements and maintenance label Nov 11, 2024
@sdb9696 sdb9696 added this to the 0.8.0 milestone Nov 11, 2024
@sdb9696 sdb9696 merged commit 71ae06f into master Nov 11, 2024
26 of 28 checks passed
@sdb9696 sdb9696 deleted the janitor/fix_test_ip branch November 11, 2024 17:41
@ryenitcher
Copy link
Contributor

I'm a bit late on the follow up, but for the sake of the records, I can confirm that all device tests now pass when run against the EP40M for the final merged commit 71ae06f (including when run the top level directory).

Results (78.55s (0:01:18)):
     459 passed
    14331 skipped

@sdb9696
Copy link
Collaborator Author

sdb9696 commented Nov 14, 2024

Many thanks @ryenitcher!

@sdb9696 sdb9696 mentioned this pull request Nov 25, 2024
sdb9696 added a commit that referenced this pull request Nov 26, 2024
## [0.8.0](https://github.com/python-kasa/python-kasa/tree/0.8.0) (2024-11-26)

[Full Changelog](0.7.7...0.8.0)

**Release highlights:**

- **Initial support for devices using the Tapo camera protocol, i.e. Tapo cameras and the Tapo H200 hub.**
- New camera functionality such as exposing RTSP streaming urls and camera pan/tilt.
- New way of testing module support for individual features with `has_feature` and `get_feature`.
- Adding voltage and current monitoring to `smart` devices.
- Migration from pydantic to mashumaro for serialization.

Special thanks to @ryenitcher and @Puxtril for their new contributions to the improvement of the project! Also thanks to everyone who has helped with testing, contributing fixtures, and reporting issues!

**Breaking change notes:**

- Removed support for python <3.11. If you haven't got a compatible version try [uv](https://docs.astral.sh/uv/).
- Renamed `device_config.to_dict()` to `device_config.to_dict_control_credentials()`. `to_dict()` is still available but takes no parameters.
- From the `iot.Cloud` module the `iot.CloudInfo` class attributes have been converted to snake case.


**Breaking changes:**

- Migrate iot cloud module to mashumaro [\#1282](#1282) (@sdb9696)
- Replace custom deviceconfig serialization with mashumaru [\#1274](#1274) (@sdb9696)
- Remove support for python \<3.11 [\#1273](#1273) (@sdb9696)

**Implemented enhancements:**

- Update cli modify presets to support smart devices [\#1295](#1295) (@sdb9696)
- Use credentials\_hash for smartcamera rtsp url [\#1293](#1293) (@sdb9696)
- Add voltage and current monitoring to smart Devices [\#1281](#1281) (@ryenitcher)
- Update cli feature command for actions not to require a value [\#1264](#1264) (@sdb9696)
- Add pan tilt camera module [\#1261](#1261) (@sdb9696)
- Add alarm module for smartcamera hubs [\#1258](#1258) (@sdb9696)
- Move TAPO smartcamera out of experimental package [\#1255](#1255) (@sdb9696)
- Add SmartCamera Led Module [\#1249](#1249) (@sdb9696)
- Use component queries to select smartcamera modules [\#1248](#1248) (@sdb9696)
- Print formatting for IotLightPreset [\#1216](#1216) (@Puxtril)
- Allow getting Annotated features from modules [\#1018](#1018) (@sdb9696)
- Add common Thermostat module [\#977](#977) (@sdb9696)

**Fixed bugs:**

- TP-Link Tapo S505D cannot disable gradual on/off  [\#1309](#1309)
- Inconsistent emeter information between features and emeter cli [\#1308](#1308)
- How to dump power usage after latest updates? [\#1306](#1306)
- kasa.discover: Got unsupported connection type: 'device\_family': 'SMART.IPCAMERA' [\#1267](#1267)
- device \_\_repr\_\_ fails if no sys\_info [\#1262](#1262)
- Tapo P110M: Error processing Energy for device, module will be unavailable: get\_energy\_usage for Energy  [\#1243](#1243)
- Listing light presets throws error [\#1201](#1201)
- Include duration when disabling smooth transition on/off [\#1313](#1313) (@rytilahti)
- Expose energy command to cli [\#1307](#1307) (@rytilahti)
- Make discovery on unsupported devices less noisy [\#1291](#1291) (@rytilahti)
- Fix repr for device created with no sysinfo or discovery info" [\#1266](#1266) (@sdb9696)
- Fix discovery by alias for smart devices [\#1260](#1260) (@sdb9696)
- Make \_\_repr\_\_ work on discovery info [\#1233](#1233) (@rytilahti)

**Added support for devices:**

- Add HS200 \(US\) Smart Fixture [\#1303](#1303) (@ZeliardM)
- Add smartcamera devices to supported docs [\#1257](#1257) (@sdb9696)
- Add P110M\(AU\) fixture [\#1244](#1244) (@rytilahti)
- Add L630 fixture [\#1240](#1240) (@rytilahti)
- Add EP40M Fixture [\#1238](#1238) (@ryenitcher)
- Add KS220 Fixture [\#1237](#1237) (@ryenitcher)

**Documentation updates:**

- Use markdown footnotes in supported.md [\#1310](#1310) (@sdb9696)
- Update docs for the new module attributes has/get feature [\#1301](#1301) (@sdb9696)
- Fixup contributing.md for running test against a real device [\#1236](#1236) (@sdb9696)

**Project maintenance:**

- Rename tests/smartcamera to tests/smartcam [\#1315](#1315) (@sdb9696)
- Do not error on smartcam hub attached smartcam child devices [\#1314](#1314) (@sdb9696)
- Add P110M\(EU\) fixture [\#1305](#1305) (@sdb9696)
- Run tests with caplog in a single worker [\#1304](#1304) (@sdb9696)
- Rename smartcamera to smartcam [\#1300](#1300) (@sdb9696)
- Move iot fixtures into iot subfolder [\#1299](#1299) (@sdb9696)
- Annotate fan\_speed\_level of Fan interface [\#1298](#1298) (@sdb9696)
- Add PIR ADC Values to Test Fixtures [\#1296](#1296) (@ryenitcher)
- Exclude \_\_getattr\_\_ for deprecated attributes from type checkers [\#1294](#1294) (@sdb9696)
- Simplify omit http\_client in DeviceConfig serialization [\#1292](#1292) (@sdb9696)
- Add SMART Voltage Monitoring to Fixtures [\#1290](#1290) (@ryenitcher)
- Remove pydantic dependency [\#1289](#1289) (@sdb9696)
- Do not print out all the fixture names at the start of test runs [\#1287](#1287) (@sdb9696)
- dump\_devinfo: iot light strip commands [\#1286](#1286) (@sdb9696)
- Migrate TurnOnBehaviours to mashumaro [\#1285](#1285) (@sdb9696)
- dump\_devinfo: query smartlife.iot.common.cloud for fw updates [\#1284](#1284) (@rytilahti)
- Migrate RuleModule to mashumaro [\#1283](#1283) (@sdb9696)
- Update sphinx dependency to 6.2 to fix docs build [\#1280](#1280) (@sdb9696)
- Update DiscoveryResult to use mashu Annotated Alias [\#1279](#1279) (@sdb9696)
- Extend dump\_devinfo iot queries [\#1278](#1278) (@sdb9696)
- Migrate triggerlogs to mashumaru [\#1277](#1277) (@sdb9696)
- Migrate smart firmware module to mashumaro [\#1276](#1276) (@sdb9696)
- Migrate IotLightPreset to mashumaru [\#1275](#1275) (@sdb9696)
- Allow callable coroutines for feature setters [\#1272](#1272) (@sdb9696)
- Fix deprecated SSLContext\(\) usage [\#1271](#1271) (@sdb9696)
- Use \_get\_device\_info methods for smart and iot devs in devtools [\#1265](#1265) (@sdb9696)
- Remove experimental support [\#1256](#1256) (@sdb9696)
- Move protocol modules into protocols package [\#1254](#1254) (@sdb9696)
- Add linkcheck to readthedocs CI [\#1253](#1253) (@rytilahti)
- Update cli energy command to use energy module [\#1252](#1252) (@sdb9696)
- Consolidate warnings for fixtures missing child devices [\#1251](#1251) (@sdb9696)
- Update smartcamera fixtures with components [\#1250](#1250) (@sdb9696)
- Move transports into their own package [\#1247](#1247) (@rytilahti)
- Fix warnings in our test suite [\#1246](#1246) (@rytilahti)
- Move tests folder to top level of project [\#1242](#1242) (@sdb9696)
- Fix test framework running against real devices [\#1235](#1235) (@sdb9696)
- Add Additional Firmware Test Fixures [\#1234](#1234) (@ryenitcher)
- Update DiscoveryResult to use Mashumaro instead of pydantic [\#1231](#1231) (@sdb9696)
- Update fixture for ES20M 1.0.11 [\#1215](#1215) (@rytilahti)
- Enable ruff check for ANN [\#1139](#1139) (@rytilahti)

**Closed issues:**

- Expose Fan speed range from the library [\#1008](#1008)
- \[META\] 0.7 series - module support for SMART devices, support for introspectable device features and refactoring the library [\#783](#783)
rytilahti pushed a commit that referenced this pull request Nov 29, 2024
## [0.8.0](https://github.com/python-kasa/python-kasa/tree/0.8.0) (2024-11-26)

[Full Changelog](0.7.7...0.8.0)

**Release highlights:**

- **Initial support for devices using the Tapo camera protocol, i.e. Tapo cameras and the Tapo H200 hub.**
- New camera functionality such as exposing RTSP streaming urls and camera pan/tilt.
- New way of testing module support for individual features with `has_feature` and `get_feature`.
- Adding voltage and current monitoring to `smart` devices.
- Migration from pydantic to mashumaro for serialization.

Special thanks to @ryenitcher and @Puxtril for their new contributions to the improvement of the project! Also thanks to everyone who has helped with testing, contributing fixtures, and reporting issues!

**Breaking change notes:**

- Removed support for python <3.11. If you haven't got a compatible version try [uv](https://docs.astral.sh/uv/).
- Renamed `device_config.to_dict()` to `device_config.to_dict_control_credentials()`. `to_dict()` is still available but takes no parameters.
- From the `iot.Cloud` module the `iot.CloudInfo` class attributes have been converted to snake case.


**Breaking changes:**

- Migrate iot cloud module to mashumaro [\#1282](#1282) (@sdb9696)
- Replace custom deviceconfig serialization with mashumaru [\#1274](#1274) (@sdb9696)
- Remove support for python \<3.11 [\#1273](#1273) (@sdb9696)

**Implemented enhancements:**

- Update cli modify presets to support smart devices [\#1295](#1295) (@sdb9696)
- Use credentials\_hash for smartcamera rtsp url [\#1293](#1293) (@sdb9696)
- Add voltage and current monitoring to smart Devices [\#1281](#1281) (@ryenitcher)
- Update cli feature command for actions not to require a value [\#1264](#1264) (@sdb9696)
- Add pan tilt camera module [\#1261](#1261) (@sdb9696)
- Add alarm module for smartcamera hubs [\#1258](#1258) (@sdb9696)
- Move TAPO smartcamera out of experimental package [\#1255](#1255) (@sdb9696)
- Add SmartCamera Led Module [\#1249](#1249) (@sdb9696)
- Use component queries to select smartcamera modules [\#1248](#1248) (@sdb9696)
- Print formatting for IotLightPreset [\#1216](#1216) (@Puxtril)
- Allow getting Annotated features from modules [\#1018](#1018) (@sdb9696)
- Add common Thermostat module [\#977](#977) (@sdb9696)

**Fixed bugs:**

- TP-Link Tapo S505D cannot disable gradual on/off  [\#1309](#1309)
- Inconsistent emeter information between features and emeter cli [\#1308](#1308)
- How to dump power usage after latest updates? [\#1306](#1306)
- kasa.discover: Got unsupported connection type: 'device\_family': 'SMART.IPCAMERA' [\#1267](#1267)
- device \_\_repr\_\_ fails if no sys\_info [\#1262](#1262)
- Tapo P110M: Error processing Energy for device, module will be unavailable: get\_energy\_usage for Energy  [\#1243](#1243)
- Listing light presets throws error [\#1201](#1201)
- Include duration when disabling smooth transition on/off [\#1313](#1313) (@rytilahti)
- Expose energy command to cli [\#1307](#1307) (@rytilahti)
- Make discovery on unsupported devices less noisy [\#1291](#1291) (@rytilahti)
- Fix repr for device created with no sysinfo or discovery info" [\#1266](#1266) (@sdb9696)
- Fix discovery by alias for smart devices [\#1260](#1260) (@sdb9696)
- Make \_\_repr\_\_ work on discovery info [\#1233](#1233) (@rytilahti)

**Added support for devices:**

- Add HS200 \(US\) Smart Fixture [\#1303](#1303) (@ZeliardM)
- Add smartcamera devices to supported docs [\#1257](#1257) (@sdb9696)
- Add P110M\(AU\) fixture [\#1244](#1244) (@rytilahti)
- Add L630 fixture [\#1240](#1240) (@rytilahti)
- Add EP40M Fixture [\#1238](#1238) (@ryenitcher)
- Add KS220 Fixture [\#1237](#1237) (@ryenitcher)

**Documentation updates:**

- Use markdown footnotes in supported.md [\#1310](#1310) (@sdb9696)
- Update docs for the new module attributes has/get feature [\#1301](#1301) (@sdb9696)
- Fixup contributing.md for running test against a real device [\#1236](#1236) (@sdb9696)

**Project maintenance:**

- Rename tests/smartcamera to tests/smartcam [\#1315](#1315) (@sdb9696)
- Do not error on smartcam hub attached smartcam child devices [\#1314](#1314) (@sdb9696)
- Add P110M\(EU\) fixture [\#1305](#1305) (@sdb9696)
- Run tests with caplog in a single worker [\#1304](#1304) (@sdb9696)
- Rename smartcamera to smartcam [\#1300](#1300) (@sdb9696)
- Move iot fixtures into iot subfolder [\#1299](#1299) (@sdb9696)
- Annotate fan\_speed\_level of Fan interface [\#1298](#1298) (@sdb9696)
- Add PIR ADC Values to Test Fixtures [\#1296](#1296) (@ryenitcher)
- Exclude \_\_getattr\_\_ for deprecated attributes from type checkers [\#1294](#1294) (@sdb9696)
- Simplify omit http\_client in DeviceConfig serialization [\#1292](#1292) (@sdb9696)
- Add SMART Voltage Monitoring to Fixtures [\#1290](#1290) (@ryenitcher)
- Remove pydantic dependency [\#1289](#1289) (@sdb9696)
- Do not print out all the fixture names at the start of test runs [\#1287](#1287) (@sdb9696)
- dump\_devinfo: iot light strip commands [\#1286](#1286) (@sdb9696)
- Migrate TurnOnBehaviours to mashumaro [\#1285](#1285) (@sdb9696)
- dump\_devinfo: query smartlife.iot.common.cloud for fw updates [\#1284](#1284) (@rytilahti)
- Migrate RuleModule to mashumaro [\#1283](#1283) (@sdb9696)
- Update sphinx dependency to 6.2 to fix docs build [\#1280](#1280) (@sdb9696)
- Update DiscoveryResult to use mashu Annotated Alias [\#1279](#1279) (@sdb9696)
- Extend dump\_devinfo iot queries [\#1278](#1278) (@sdb9696)
- Migrate triggerlogs to mashumaru [\#1277](#1277) (@sdb9696)
- Migrate smart firmware module to mashumaro [\#1276](#1276) (@sdb9696)
- Migrate IotLightPreset to mashumaru [\#1275](#1275) (@sdb9696)
- Allow callable coroutines for feature setters [\#1272](#1272) (@sdb9696)
- Fix deprecated SSLContext\(\) usage [\#1271](#1271) (@sdb9696)
- Use \_get\_device\_info methods for smart and iot devs in devtools [\#1265](#1265) (@sdb9696)
- Remove experimental support [\#1256](#1256) (@sdb9696)
- Move protocol modules into protocols package [\#1254](#1254) (@sdb9696)
- Add linkcheck to readthedocs CI [\#1253](#1253) (@rytilahti)
- Update cli energy command to use energy module [\#1252](#1252) (@sdb9696)
- Consolidate warnings for fixtures missing child devices [\#1251](#1251) (@sdb9696)
- Update smartcamera fixtures with components [\#1250](#1250) (@sdb9696)
- Move transports into their own package [\#1247](#1247) (@rytilahti)
- Fix warnings in our test suite [\#1246](#1246) (@rytilahti)
- Move tests folder to top level of project [\#1242](#1242) (@sdb9696)
- Fix test framework running against real devices [\#1235](#1235) (@sdb9696)
- Add Additional Firmware Test Fixures [\#1234](#1234) (@ryenitcher)
- Update DiscoveryResult to use Mashumaro instead of pydantic [\#1231](#1231) (@sdb9696)
- Update fixture for ES20M 1.0.11 [\#1215](#1215) (@rytilahti)
- Enable ruff check for ANN [\#1139](#1139) (@rytilahti)

**Closed issues:**

- Expose Fan speed range from the library [\#1008](#1008)
- \[META\] 0.7 series - module support for SMART devices, support for introspectable device features and refactoring the library [\#783](#783)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintenance Project improvements and maintenance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants