Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • aqdl/denoising
1 result
Show changes
Commits on Source (5)
Expand all Hide whitespace changes
Inline Side-by-side
Showing
with 5404 additions and 58 deletions
README.md
View file @ 26dcca8e
# denoising
## Installation
```cfgrlanguage
pip install -r requirements.txt
```
Older versions of `pywavelet` were able to handle `discrete wavelet transform` using the `cwt()` function mean for
continuous wavelets. To get `pywavelet` library working with the (discrete) `haar` wavelet we need to make two small
modifications to the source of the `pywavelet` library. To do so, locate the installation directory: Let's say
`<some_path>/lib/python3.10/site-packages/pywt`. Open the `/_cwt.py` module. Replace the line
`dt_out = dt_cplx if wavelet.complex_cwt else dt` by `dt_out = dt` and comment
`int_psi = np.conj(int_psi) if wavelet.complex_cwt else int_psi`.
## Getting started
[//]: # ()
[//]: # ()
[//]: # (## Getting started)
To make it easy for you to get started with GitLab, here's a list of recommended next steps.
[//]: # ()
[//]: # (To make it easy for you to get started with GitLab, here's a list of recommended next steps.)
Already a pro? Just edit this README.md and make it your own. Want to make it easy? [Use the template at the bottom](#editing-this-readme)!
[//]: # ()
[//]: # (Already a pro? Just edit this README.md and make it your own. Want to make it easy? [Use the template at the bottom]&#40;#editing-this-readme&#41;!)
## Add your files
[//]: # ()
[//]: # (## Add your files)
- [ ] [Create](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#create-a-file) or [upload](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#upload-a-file) files
- [ ] [Add files using the command line](https://docs.gitlab.com/ee/gitlab-basics/add-file.html#add-a-file-using-the-command-line) or push an existing Git repository with the following command:
[//]: # ()
[//]: # (- [ ] [Create]&#40;https://docs.gitlab.com/ee/user/project/repository/web_editor.html#create-a-file&#41; or [upload]&#40;https://docs.gitlab.com/ee/user/project/repository/web_editor.html#upload-a-file&#41; files)
```
cd existing_repo
git remote add origin https://git.nilu.no/aqdl/denoising.git
git branch -M main
git push -uf origin main
```
[//]: # (- [ ] [Add files using the command line]&#40;https://docs.gitlab.com/ee/gitlab-basics/add-file.html#add-a-file-using-the-command-line&#41; or push an existing Git repository with the following command:)
## Integrate with your tools
[//]: # ()
[//]: # (```)
- [ ] [Set up project integrations](https://git.nilu.no/aqdl/denoising/-/settings/integrations)
[//]: # (cd existing_repo)
## Collaborate with your team
[//]: # (git remote add origin https://git.nilu.no/aqdl/denoising.git)
- [ ] [Invite team members and collaborators](https://docs.gitlab.com/ee/user/project/members/)
- [ ] [Create a new merge request](https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html)
- [ ] [Automatically close issues from merge requests](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically)
- [ ] [Enable merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/approvals/)
- [ ] [Automatically merge when pipeline succeeds](https://docs.gitlab.com/ee/user/project/merge_requests/merge_when_pipeline_succeeds.html)
[//]: # (git branch -M main)
## Test and Deploy
[//]: # (git push -uf origin main)
Use the built-in continuous integration in GitLab.
[//]: # (```)
- [ ] [Get started with GitLab CI/CD](https://docs.gitlab.com/ee/ci/quick_start/index.html)
- [ ] [Analyze your code for known vulnerabilities with Static Application Security Testing(SAST)](https://docs.gitlab.com/ee/user/application_security/sast/)
- [ ] [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/requirements.html)
- [ ] [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/ee/user/clusters/agent/)
- [ ] [Set up protected environments](https://docs.gitlab.com/ee/ci/environments/protected_environments.html)
[//]: # ()
[//]: # (## Integrate with your tools)
***
[//]: # ()
[//]: # (- [ ] [Set up project integrations]&#40;https://git.nilu.no/aqdl/denoising/-/settings/integrations&#41;)
# Editing this README
[//]: # ()
[//]: # (## Collaborate with your team)
When you're ready to make this README your own, just edit this file and use the handy template below (or feel free to structure it however you want - this is just a starting point!). Thank you to [makeareadme.com](https://www.makeareadme.com/) for this template.
[//]: # ()
[//]: # (- [ ] [Invite team members and collaborators]&#40;https://docs.gitlab.com/ee/user/project/members/&#41;)
## Suggestions for a good README
Every project is different, so consider which of these sections apply to yours. The sections used in the template are suggestions for most open source projects. Also keep in mind that while a README can be too long and detailed, too long is better than too short. If you think your README is too long, consider utilizing another form of documentation rather than cutting out information.
[//]: # (- [ ] [Create a new merge request]&#40;https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html&#41;)
## Name
Choose a self-explaining name for your project.
[//]: # (- [ ] [Automatically close issues from merge requests]&#40;https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically&#41;)
## Description
Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.
[//]: # (- [ ] [Enable merge request approvals]&#40;https://docs.gitlab.com/ee/user/project/merge_requests/approvals/&#41;)
## Badges
On some READMEs, you may see small images that convey metadata, such as whether or not all the tests are passing for the project. You can use Shields to add some to your README. Many services also have instructions for adding a badge.
[//]: # (- [ ] [Automatically merge when pipeline succeeds]&#40;https://docs.gitlab.com/ee/user/project/merge_requests/merge_when_pipeline_succeeds.html&#41;)
## Visuals
Depending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method.
[//]: # ()
[//]: # (## Test and Deploy)
## Installation
Within a particular ecosystem, there may be a common way of installing things, such as using Yarn, NuGet, or Homebrew. However, consider the possibility that whoever is reading your README is a novice and would like more guidance. Listing specific steps helps remove ambiguity and gets people to using your project as quickly as possible. If it only runs in a specific context like a particular programming language version or operating system or has dependencies that have to be installed manually, also add a Requirements subsection.
[//]: # ()
[//]: # (Use the built-in continuous integration in GitLab.)
[//]: # ()
[//]: # (- [ ] [Get started with GitLab CI/CD]&#40;https://docs.gitlab.com/ee/ci/quick_start/index.html&#41;)
[//]: # (- [ ] [Analyze your code for known vulnerabilities with Static Application Security Testing&#40;SAST&#41;]&#40;https://docs.gitlab.com/ee/user/application_security/sast/&#41;)
[//]: # (- [ ] [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy]&#40;https://docs.gitlab.com/ee/topics/autodevops/requirements.html&#41;)
[//]: # (- [ ] [Use pull-based deployments for improved Kubernetes management]&#40;https://docs.gitlab.com/ee/user/clusters/agent/&#41;)
[//]: # (- [ ] [Set up protected environments]&#40;https://docs.gitlab.com/ee/ci/environments/protected_environments.html&#41;)
[//]: # ()
[//]: # (***)
[//]: # ()
[//]: # (# Editing this README)
[//]: # ()
[//]: # (When you're ready to make this README your own, just edit this file and use the handy template below &#40;or feel free to structure it however you want - this is just a starting point!&#41;. Thank you to [makeareadme.com]&#40;https://www.makeareadme.com/&#41; for this template.)
[//]: # ()
[//]: # (## Suggestions for a good README)
[//]: # (Every project is different, so consider which of these sections apply to yours. The sections used in the template are suggestions for most open source projects. Also keep in mind that while a README can be too long and detailed, too long is better than too short. If you think your README is too long, consider utilizing another form of documentation rather than cutting out information.)
[//]: # ()
[//]: # (## Name)
[//]: # (Choose a self-explaining name for your project.)
[//]: # ()
[//]: # (## Description)
[//]: # (Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.)
[//]: # ()
[//]: # (## Badges)
[//]: # (On some READMEs, you may see small images that convey metadata, such as whether or not all the tests are passing for the project. You can use Shields to add some to your README. Many services also have instructions for adding a badge.)
[//]: # ()
[//]: # (## Visuals)
[//]: # (Depending on what you are making, it can be a good idea to include screenshots or even a video &#40;you'll frequently see GIFs rather than actual videos&#41;. Tools like ttygif can help, but check out Asciinema for a more sophisticated method.)
[//]: # ()
[//]: # (## Installation)
[//]: # (Within a particular ecosystem, there may be a common way of installing things, such as using Yarn, NuGet, or Homebrew. However, consider the possibility that whoever is reading your README is a novice and would like more guidance. Listing specific steps helps remove ambiguity and gets people to using your project as quickly as possible. If it only runs in a specific context like a particular programming language version or operating system or has dependencies that have to be installed manually, also add a Requirements subsection.)
[//]: # ()
[//]: # (## Usage)
[//]: # (Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.)
[//]: # ()
[//]: # (## Support)
[//]: # (Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.)
[//]: # ()
[//]: # (## Roadmap)
[//]: # (If you have ideas for releases in the future, it is a good idea to list them in the README.)
[//]: # ()
[//]: # (## Contributing)
## Usage
Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.
[//]: # (State if you are open to contributions and what your requirements are for accepting them.)
## Support
Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.
[//]: # ()
[//]: # (For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.)
## Roadmap
If you have ideas for releases in the future, it is a good idea to list them in the README.
[//]: # ()
[//]: # (You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.)
## Contributing
State if you are open to contributions and what your requirements are for accepting them.
[//]: # ()
[//]: # (## Authors and acknowledgment)
For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.
[//]: # (Show your appreciation to those who have contributed to the project.)
You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.
[//]: # ()
[//]: # (## License)
## Authors and acknowledgment
Show your appreciation to those who have contributed to the project.
[//]: # (For open source projects, say how it is licensed.)
## License
For open source projects, say how it is licensed.
[//]: # ()
[//]: # (## Project status)
## Project status
If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.
[//]: # (If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.)
all.ipynb 0 → 100644
View file @ 26dcca8e
This diff is collapsed.
data/sensor_data.txt 0 → 100644
View file @ 26dcca8e
This diff is collapsed.
data/sensor_data_dataset_2.txt 0 → 100644
View file @ 26dcca8e
This diff is collapsed.
data/weather_station_data.txt 0 → 100644
View file @ 26dcca8e
------------weather station data -----------------
timestamp rh008 pa008 pm10 pm1 pm2_5 no2 temp008
--------------------------------------------------
738532.0416666666 76.20 1032.300 39.3 38.5 38.9 46.9000000 -1.40
738532.0833333334 72.70 1032.500 34.3 33.8 34.1 39.0000000 -0.30
738532.1250000000 71.30 1032.800 24.0 23.7 23.9 23.8000000 0.20
738532.1666666666 72.10 1032.900 13.9 13.7 13.8 9.0000000 0.50
738532.2083333334 75.70 1033.000 13.0 12.8 12.9 8.5000000 -0.20
738532.2500000000 79.10 1033.000 12.7 12.5 12.6 12.2000000 -0.60
738532.2916666666 81.60 1033.100 14.4 14.0 14.2 32.8000000 -1.20
738532.3333333334 81.70 1033.800 14.4 13.8 14.0 45.0000000 -0.80
738532.3750000000 81.70 1034.500 18.2 17.2 17.6 41.7000000 -0.30
738532.4166666666 80.90 1035.000 30.8 28.9 29.6 36.6000000 0.30
738532.4583333334 81.30 1035.200 54.0 42.5 44.7 47.5000000 0.70
738532.5000000000 82.10 1035.000 54.9 40.9 43.8 54.0000000 0.80
738532.5416666666 80.80 1035.100 62.7 39.2 44.7 58.8000000 1.30
738532.5833333334 77.70 1034.900 47.4 35.6 37.9 53.3000000 1.60
738532.6250000000 76.30 1034.800 39.1 30.1 31.9 48.1000000 1.70
738532.6666666666 77.40 1034.800 34.1 31.3 32.1 47.9000000 1.40
738532.7083333334 78.90 1034.800 34.9 32.8 33.5 45.7000000 0.90
738532.7500000000 79.50 1034.900 23.1 22.6 22.8 30.4000000 0.60
738532.7916666666 78.80 1035.000 36.4 35.0 35.4 36.9000000 0.40
738532.8333333334 77.70 1035.000 47.0 45.4 45.9 40.8000000 0.40
738532.8750000000 76.10 1035.000 38.9 37.9 38.3 45.6000000 0.30
738532.9166666666 73.90 1035.000 52.1 51.0 51.5 47.9000000 0.10
738532.9583333334 74.40 1034.900 47.6 46.8 47.2 40.9000000 -0.50
738533.0000000000 74.90 1034.700 43.6 42.5 43.1 34.6000000 -1.10
738533.0416666666 74.80 1034.400 29.5 28.9 29.3 18.7000000 -1.30
738533.0833333334 75.00 1034.200 21.3 21.0 21.2 11.3000000 -0.80
738533.1250000000 76.20 1034.000 16.0 15.7 15.9 9.7000000 -1.40
738533.1666666666 75.20 1033.600 15.4 15.1 15.3 8.7000000 -1.50
738533.2083333334 73.40 1033.400 19.7 17.6 18.2 15.9000000 -1.40
738533.2500000000 74.00 1033.200 22.1 20.3 20.8 23.3000000 -1.60
738533.2916666666 73.10 1033.000 41.8 35.8 37.5 42.7000000 -1.80
738533.3333333334 73.60 1032.600 42.3 37.4 38.7 41.5000000 -2.20
738533.3750000000 74.70 1032.300 48.2 40.8 44.1 42.0000000 -2.30
738533.4166666666 72.20 1032.500 46.4 38.7 40.8 40.9000000 -1.50
738533.4583333334 71.80 1032.400 31.7 30.5 31.0 19.9000000 -1.20
738533.5000000000 72.30 1031.600 33.7 32.2 32.8 19.3000000 -0.50
738533.5416666666 73.30 1030.900 32.6 30.7 31.3 19.7000000 0.00
738533.5833333334 73.50 1031.100 33.9 31.8 32.5 23.2000000 0.30
738533.6250000000 74.70 1030.900 32.4 30.6 31.2 27.5000000 0.30
738533.6666666666 78.60 1030.400 34.3 31.3 32.8 30.4000000 0.60
738533.7083333334 80.20 1030.100 52.0 45.1 48.4 39.6000000 0.60
738533.7500000000 80.00 1029.800 48.6 38.7 41.0 37.5000000 1.10
738533.7916666666 79.40 1029.500 14.6 14.2 14.4 13.8000000 1.90
738533.8333333334 78.90 1029.300 14.1 13.6 13.9 12.3000000 1.90
738533.8750000000 77.90 1029.000 11.7 11.2 11.5 9.7000000 2.30
738533.9166666666 80.70 1028.700 10.1 9.6 9.9 8.1000000 2.50
738533.9583333334 81.60 1027.700 8.3 8.0 8.2 NaN 2.60
738534.0000000000 81.70 1026.800 6.4 6.3 6.4 6.8000000 2.60
738534.0416666666 82.10 1025.500 5.5 5.3 5.5 5.9000000 3.30
738534.0833333334 82.90 1025.200 4.6 4.5 4.5 5.0000000 3.70
738534.1250000000 82.10 1024.200 3.2 3.1 3.1 4.6000000 3.90
738534.1666666666 82.20 1022.700 2.6 2.5 2.6 4.8000000 4.30
738534.2083333334 81.90 1021.800 2.5 2.4 2.5 4.8000000 5.10
738534.2500000000 81.80 1021.300 2.6 2.5 2.6 4.8000000 5.60
738534.2916666666 81.40 1020.800 3.3 3.2 3.3 5.7000000 5.90
738534.3333333334 81.00 1021.500 3.3 3.1 3.2 7.3000000 6.00
738534.3750000000 82.50 1021.900 3.3 2.9 3.1 9.0000000 6.00
738534.4166666666 82.00 1021.600 4.1 3.6 3.9 8.5000000 6.60
738534.4583333334 82.30 1021.300 5.3 4.8 5.1 7.4000000 6.70
738534.5000000000 80.70 1020.800 4.5 4.1 4.4 7.7000000 7.00
738534.5416666666 80.70 1020.500 4.6 4.1 4.4 7.6000000 7.00
738534.5833333334 80.40 1019.900 4.4 3.9 4.2 7.4000000 7.20
738534.6250000000 80.00 1019.100 4.8 4.3 4.6 6.9000000 7.20
738534.6666666666 78.50 1018.600 5.0 4.5 4.8 6.9000000 7.30
738534.7083333334 77.20 1018.300 5.1 4.5 4.9 6.5000000 7.40
738534.7500000000 75.80 1018.400 4.6 4.0 4.4 6.0000000 7.80
738534.7916666666 74.90 1018.200 4.5 3.9 4.3 5.2000000 7.80
738534.8333333334 74.30 1017.600 4.4 3.8 4.2 4.7000000 7.80
738534.8750000000 74.20 1017.000 4.5 3.7 4.2 4.4000000 7.80
738534.9166666666 74.00 1016.900 4.2 3.6 4.0 4.1000000 7.60
738534.9583333334 74.70 1016.800 3.9 3.5 3.8 3.5000000 7.50
738535.0000000000 74.30 1016.700 3.6 3.2 3.5 3.3000000 7.50
738535.0416666666 75.20 1015.500 3.6 3.1 3.4 2.9000000 7.40
738535.0833333334 77.30 1014.500 3.0 2.7 2.9 3.0000000 7.20
738535.1250000000 77.30 1013.900 2.6 2.3 2.5 2.9000000 7.30
738535.1666666666 77.70 1012.900 2.4 2.2 2.3 2.9000000 7.50
738535.2083333334 77.80 1012.300 2.2 1.9 2.1 2.9000000 7.60
738535.2500000000 78.40 1011.700 2.6 2.3 2.5 3.3000000 7.80
738535.2916666666 77.70 1011.300 3.1 2.7 3.0 4.1000000 8.00
738535.3333333334 77.20 1011.300 3.6 3.0 3.4 4.6000000 8.10
738535.3750000000 71.50 1011.500 4.6 2.9 3.5 4.4000000 9.10
738535.4166666666 67.60 1011.600 4.5 3.0 3.5 3.7000000 9.40
738535.4583333334 67.30 1011.800 4.6 3.1 3.7 3.8000000 9.20
738535.5000000000 66.20 1012.400 5.4 3.1 4.0 3.9000000 9.00
738535.5416666666 62.90 1013.100 6.4 3.2 4.2 4.2000000 8.80
738535.5833333334 68.30 1014.500 3.1 2.1 2.5 3.7000000 6.20
738535.6250000000 59.20 1015.100 3.8 2.8 3.3 6.5000000 6.30
738535.6666666666 47.20 1016.400 3.8 2.7 3.2 6.0000000 6.70
738535.7083333334 48.60 1017.300 4.3 3.0 3.6 5.3000000 6.10
738535.7500000000 50.00 1018.200 5.2 3.5 4.2 5.4000000 5.60
738535.7916666666 48.80 1018.900 4.8 3.3 4.0 4.1000000 5.40
738535.8333333334 47.40 1019.400 4.9 3.3 3.9 3.5000000 5.30
738535.8750000000 46.70 1019.800 4.7 3.0 3.6 3.0000000 5.10
738535.9166666666 48.10 1020.100 4.3 2.8 3.5 2.9000000 5.00
738535.9583333334 52.20 1020.500 4.6 2.9 3.7 3.3000000 4.60
738536.0000000000 54.80 1020.700 4.8 3.0 3.8 2.7000000 4.20
738536.0416666666 57.40 1020.800 4.3 2.9 3.6 2.3000000 3.70
738536.0833333334 59.60 1020.700 3.8 2.7 3.3 2.3000000 3.20
738536.1250000000 62.10 1020.800 3.4 2.6 3.1 2.4000000 2.70
738536.1666666666 64.50 1020.600 3.6 2.6 3.2 2.5000000 2.40
738536.2083333334 65.80 1020.800 3.7 2.6 3.2 2.6000000 2.20
738536.2500000000 67.40 1021.400 3.6 2.6 3.2 2.7000000 1.70
738536.2916666666 69.60 1021.700 3.9 3.0 3.6 6.3000000 1.20
738536.3333333334 71.90 1022.100 4.9 3.4 4.2 8.4000000 0.70
738536.3750000000 72.40 1022.400 6.1 4.3 5.2 11.5000000 0.80
738536.4166666666 68.90 1022.800 7.6 5.2 6.1 11.1000000 2.00
738536.4583333334 65.70 1023.400 6.1 4.3 5.1 5.6000000 3.20
738536.5000000000 62.90 1023.500 5.7 4.0 4.8 5.2000000 4.00
738536.5416666666 61.50 1023.800 6.6 4.4 5.2 5.2000000 4.40
738536.5833333334 60.40 1023.900 5.5 3.6 4.3 4.9000000 4.50
738536.6250000000 61.20 1024.000 5.4 3.5 4.2 5.9000000 4.00
738536.6666666666 64.60 1024.000 9.6 4.8 5.7 7.7000000 3.00
738536.7083333334 68.90 1023.900 9.1 7.1 7.9 10.8000000 2.00
738536.7500000000 70.60 1023.800 11.1 9.3 10.1 12.2000000 1.80
738536.7916666666 73.50 1023.400 9.3 8.2 8.8 9.9000000 2.00
738536.8333333334 75.70 1023.300 9.1 7.8 8.6 10.3000000 2.10
738536.8750000000 75.90 1022.700 12.0 10.6 11.5 8.3000000 1.90
738536.9166666666 75.70 1022.600 11.9 10.5 11.4 NaN 1.60
738536.9583333334 76.50 1022.700 10.3 9.5 10.1 5.5000000 1.10
738537.0000000000 76.50 1022.600 11.7 11.0 11.5 4.5000000 0.50
738537.0416666666 76.50 1022.300 14.3 13.2 14.0 5.0000000 0.10
738537.0833333334 76.10 1021.800 15.3 14.5 15.1 7.4000000 -0.30
738537.1250000000 75.70 1021.500 12.9 12.3 12.7 3.6000000 -0.40
738537.1666666666 75.30 1020.200 12.1 11.5 11.9 4.4000000 -0.40
738537.2083333334 76.50 1019.800 12.1 11.4 11.9 6.0000000 -0.20
738537.2500000000 78.80 1018.900 12.8 12.0 12.5 7.6000000 -0.20
738537.2916666666 78.40 1018.200 11.2 10.6 11.1 4.7000000 0.00
738537.3333333334 79.70 1017.500 12.1 11.4 11.9 9.3000000 0.00
738537.3750000000 81.30 1017.200 15.4 14.1 14.7 13.6000000 -0.10
738537.4166666666 81.30 1016.800 14.8 13.6 14.3 17.5000000 0.20
738537.4583333334 79.40 1015.700 14.7 13.1 13.9 16.4000000 0.70
738537.5000000000 74.80 1014.900 11.6 10.3 11.1 7.4000000 1.30
738537.5416666666 73.70 1014.000 12.5 10.9 11.7 4.8000000 1.30
738537.5833333334 72.30 1012.800 12.5 11.0 11.9 4.5000000 1.40
738537.6250000000 72.00 1011.400 14.8 13.2 14.2 10.0000000 1.40
738537.6666666666 75.90 1010.900 15.4 14.1 15.0 10.8000000 1.30
738537.7083333334 79.00 1010.600 17.3 16.0 16.9 8.4000000 1.30
738537.7500000000 79.50 1010.100 17.0 15.8 16.7 8.5000000 1.60
738537.7916666666 81.20 1009.400 13.9 13.4 13.8 9.2000000 2.00
738537.8333333334 81.00 1008.900 11.7 11.6 11.7 9.5000000 2.60
738537.8750000000 82.30 1008.100 12.4 12.3 12.4 10.9000000 2.80
738537.9166666666 81.50 1007.000 9.7 9.6 9.6 11.0000000 3.30
738537.9583333334 82.60 1005.900 10.0 9.8 9.9 9.6000000 3.70
738538.0000000000 81.10 1004.500 8.6 8.4 8.5 7.0000000 4.60
738538.0416666666 72.90 1003.100 7.4 6.9 7.2 4.2000000 5.70
738538.0833333334 68.50 1002.300 5.4 4.5 5.0 3.1000000 5.70
738538.1250000000 67.90 1000.900 4.6 3.3 4.0 2.2000000 5.60
738538.1666666666 70.10 999.600 4.3 3.2 3.9 2.1000000 5.70
738538.2083333334 76.20 997.800 3.1 2.7 3.0 2.4000000 5.40
738538.2500000000 76.60 996.300 2.7 2.4 2.7 2.8000000 5.80
738538.2916666666 78.00 996.300 2.2 2.0 2.1 3.4000000 3.80
738538.3333333334 79.50 997.700 2.3 2.0 2.2 5.2000000 1.30
738538.3750000000 68.20 999.100 3.0 2.4 2.8 5.7000000 2.30
738538.4166666666 55.80 1000.800 3.6 2.5 3.0 5.1000000 2.90
738538.4583333334 48.60 1002.100 3.4 2.4 2.9 3.7000000 3.80
738538.5000000000 45.20 1003.700 3.6 2.5 3.0 4.1000000 4.00
738538.5416666666 47.70 1005.400 3.7 2.5 3.0 5.2000000 3.70
738538.5833333334 50.00 1007.500 4.0 3.0 3.4 5.0000000 3.60
738538.6250000000 50.50 1009.500 3.5 2.7 3.1 4.2000000 3.60
738538.6666666666 51.40 1011.500 3.7 2.9 3.3 5.1000000 3.30
738538.7083333334 51.70 1013.600 5.4 4.3 4.8 6.6000000 3.20
738538.7500000000 53.90 1015.500 3.9 3.0 3.4 5.0000000 3.00
738538.7916666666 55.30 1017.000 4.4 3.7 4.0 4.1000000 3.00
738538.8333333334 55.20 1018.100 4.3 3.3 3.8 5.0000000 3.20
738538.8750000000 53.30 1019.100 4.6 3.2 3.8 6.5000000 3.10
738538.9166666666 53.30 1019.700 4.7 3.4 4.0 3.4000000 2.60
738538.9583333334 57.30 1020.700 4.8 3.9 4.3 3.9000000 1.60
738539.0000000000 58.20 1021.100 4.3 3.6 3.9 3.4000000 1.40
738539.0416666666 61.40 1021.700 4.3 3.6 4.0 2.8000000 1.30
738539.0833333334 65.60 1022.200 4.3 3.7 4.0 2.6000000 1.10
738539.1250000000 67.90 1022.700 4.0 3.4 3.8 2.7000000 1.10
738539.1666666666 69.30 1023.500 3.3 2.7 3.1 2.2000000 1.10
738539.2083333334 69.70 1024.000 3.2 2.5 3.0 2.5000000 1.20
738539.2500000000 69.50 1024.700 3.2 2.5 3.0 4.1000000 1.30
738539.2916666666 70.40 1025.500 3.5 2.8 3.2 7.1000000 1.10
738539.3333333334 70.40 1026.500 3.8 2.9 3.4 12.8000000 1.50
738539.3750000000 71.00 1027.500 5.2 3.5 4.3 18.9000000 1.40
738539.4166666666 71.00 1028.400 8.7 5.0 6.6 19.9000000 1.60
738539.4583333334 67.90 1029.200 6.9 4.6 5.4 13.7000000 2.80
738539.5000000000 66.40 1029.500 7.6 5.0 5.7 11.0000000 3.50
738539.5416666666 63.60 1029.500 6.1 3.8 4.5 7.1000000 4.00
738539.5833333334 61.20 1029.400 6.0 3.9 4.5 9.7000000 4.40
738539.6250000000 61.40 1029.800 10.4 4.3 5.5 9.2000000 4.10
738539.6666666666 61.50 1029.700 11.3 4.5 5.7 13.4000000 3.40
738539.7083333334 65.20 1029.900 14.0 7.8 9.2 19.2000000 2.80
738539.7500000000 66.70 1029.700 11.6 9.6 10.3 22.0000000 2.10
738539.7916666666 69.10 1029.200 14.0 12.6 13.3 27.2000000 1.70
738539.8333333334 74.30 1028.800 12.1 10.5 11.3 9.9000000 1.80
738539.8750000000 74.10 1028.500 7.3 6.2 6.9 NaN 2.10
738539.9166666666 76.60 1027.800 7.6 6.7 7.3 7.7000000 2.40
738539.9583333334 81.30 1027.100 9.0 8.3 8.8 7.3000000 2.30
738540.0000000000 80.30 1026.300 8.5 8.0 8.3 6.7000000 2.80
data/weather_station_dataset_2.txt 0 → 100644
View file @ 26dcca8e
This diff is collapsed.
denoising/__init__.py 0 → 100644
View file @ 26dcca8e
denoising/__main__.py 0 → 100644
View file @ 26dcca8e
import matplotlib.pyplot as plt
import numpy as np
from pathlib import Path
import sys
from scipy.signal import savgol_filter
import copy
import plotly.graph_objects as go
import json
from scipy.interpolate import PchipInterpolator
from denoising import (dataset, outliers)
def main():
# Settings
srcdir = Path('/home/wrfadmin/PycharmProjects/hapads/denoising/data')
dstdir = Path('/home/wrfadmin/test')
sensor_data_1_txt = srcdir.joinpath('sensor_data.txt')
sensor_data_2_txt = srcdir.joinpath('sensor_data_dataset_2.txt')
reference_data_1_txt = srcdir.joinpath('weather_station_data.txt')
reference_data_2_txt = srcdir.joinpath('weather_station_dataset_2.txt')
# Reference offset (one hour)
refence_offset = -0.04218
# Threshold for evident peaks
# TODO: Rename
thd = 75 * 5;
# Threshold for delta-based peaks
# TODO: Rename
thd2 = 0.05;
sfr = 1
wid = np.array([1.245, 1.25])
option = 'reset'
# Factor that multiplies [cost of comission]/[cost of omission].
# For most practical purposes -0.2 <= L <= 0.2. Larger L --> omissions
# likely, smaller L --> false positives likely. For unsupervised
# detection, the suggested value of L is close to 0.
L = -0.1
wname = 'haar'
# Load datasets
reference_data = dataset.from_csv(paths=[reference_data_1_txt, reference_data_2_txt])
# Apply offset to reference data
reference_data = dataset.apply_offset(reference_data, refence_offset)
sensor_data = dataset.from_csv(paths=[sensor_data_1_txt, sensor_data_2_txt])
scor = sensor_data[['pm10', 'pm1', 'pm2_5']].values
# # OK
# dataset.check_values(scor)
scor2 = copy.deepcopy(scor)
# Clean sensor data using threshold
scor2 = outliers.apply_threshold(scor2, threshold=thd)
####################################################################################################################
####################################################################################################################
####################################################################################################################
# Set minumum target PM value based on cumulative data
reftst = scor2.min(axis=1)
# TODO: Not exactly how it is done in the notebook. Lets check that this is the same as scor3 = deepcopy(scor2[:, -1]) !!!!!
scor3 = copy.deepcopy(scor2)
scor3 = outliers.apply_delta_relative_threshold(dataset=scor3[:, -1], reftst=reftst, threshold=thd2)
####################################################################################################################
####################################################################################################################
####################################################################################################################
signal = np.expand_dims(scor3, axis=0)
# OK !!!
tE = outliers.get_wavelet_spikes(signal, sfr, wid, L, wname)
scor4 = copy.deepcopy(scor3)
for k in range(len(tE)):
scor4[tE[k]] = (np.array([scor4[tE[k] - 1], scor4[tE[k] + 1]])).mean()
# dataset.check_values(scor4)
# Interpolate reference at sensor time
reference_interpolator = PchipInterpolator(x=reference_data.values[:, 0], y=reference_data.values[:, 5])
refint = reference_interpolator(sensor_data.values[:, 0])
# dataset.check_values(refint)
# NOTE: Call this scor5 to avoid overwritting scor4 for control porposes
scor5 = savgol_filter(x=scor4,
window_length=5, # window size used for filtering
polyorder=2) # order of fitted polynomial
# dataset.check_values(scor5)
corr_matrix = np.corrcoef(refint, scor5)
corr = corr_matrix[0, 1]
R_sq = corr ** 2
# How well does the reint match the observations scor5
print(f'Coefficient of determination: {R_sq}')
rmse5 = dataset.rmse(refint, scor5)
rmse3 = dataset.rmse(refint, scor3)
rmse2 = dataset.rmse(refint, scor2[:, 2])
rmse1 = dataset.rmse(refint, sensor_data.values[:, 5])
print(f'RMSE - scor5: {rmse5}')
print(f'RMSE - scor3: {rmse3}')
print(f'RMSE - scor2: {rmse2}')
print(f'RMSE - scor1: {rmse1}')
# Display results
plt.figure(figsize=(30, 8))
plt.plot(sensor_data.values[:, 0][0:1140], reftst[0:1140], 'blue', linewidth=2)
plt.plot(sensor_data.values[:, 0][0:1140], scor3[0:1140], 'r', linewidth=0.5)
plt.plot(sensor_data.values[tE, 0][0:48], scor3[tE][0:48], 'ro', mfc='none')
plt.plot(sensor_data.values[:, 0][0:1140], scor4[0:1140], 'c')
plt.xlabel('Time')
plt.ylabel('Concentration')
plt.savefig(dstdir.joinpath('figure_1.png'))
plt.show()
# f1 = go.Figure(
# data=[
# go.Scatter(x=sensor_data.values[:, 0][0:1140], y=reftst[0:1140], name="reftst"),
# go.Scatter(x=sensor_data.values[:, 0][0:1140], y=scor3[0:1140], name="scor3"),
# go.Scatter(x=sensor_data.values[tE, 0][0:48], y=scor3[tE][0:48], name="scor3-peaks", mode='markers', marker=dict(color='red')),
# go.Scatter(x=sensor_data.values[:, 0][0:1140], y=scor4[0:1140], name="scor3")
# ],
# layout={"xaxis": {"title": "Time"}, "yaxis": {"title": "Concentration"}, "title": ""}
# )
# f1.show()
plt.figure(figsize=(30, 8))
plt.plot(sensor_data.values[:, 0][0:1140], scor2[:, 0][0:1140], 'k', linewidth=2)
plt.plot(sensor_data.values[:, 0][0:1140], scor2[:, 1][0:1140], 'cyan', linewidth=1)
plt.plot(sensor_data.values[:, 0][0:1140], scor3[0:1140], 'b', linewidth=0.5)
plt.plot(sensor_data.values[:, 0][0:1140], scor2[:, 2][0:1140], 'r')
plt.xlabel('Time')
plt.ylabel('Concentration')
plt.savefig(dstdir.joinpath('figure_2.png'))
plt.show()
plt.figure(figsize=(30, 8))
plt.plot(sensor_data.values[:, 0][0:1140], scor3[0:1140], 'b', linewidth=2)
plt.plot(sensor_data.values[:, 0][0:1140], scor5[0:1140], 'g', linewidth=2)
plt.plot(sensor_data.values[:, 0][0:1140], refint[0:1140], 'r')
plt.xlabel('Time')
plt.ylabel('Concentration')
plt.savefig(dstdir.joinpath('figure_3.png'))
plt.show()
plt.figure(figsize=(15, 15))
plt.plot(refint, scor5, 'o')
plt.xlim([0, refint.max()])
plt.ylim([0, scor5.max()])
plt.xlabel('Interpolated')
plt.ylabel('Despiked')
plt.title(f'Coefficient of determination R²: {R_sq}')
plt.savefig(dstdir.joinpath('figure_4.png'))
plt.show()
# Save corrected results
tstamp = copy.deepcopy(sensor_data.values[:, 0])
xs = [sensor_data.values[:, 8], sensor_data.values[:, 1], sensor_data.values[:, 2]/100.0, scor5]
xr = np.zeros((len(tstamp), 4))
xr[:, 0] = PchipInterpolator(reference_data.values[:, 0], reference_data.values[:, 7])(tstamp)
xr[:, 1] = PchipInterpolator(reference_data.values[:, 0], reference_data.values[:, 1])(tstamp)
xr[:, 2] = PchipInterpolator(reference_data.values[:, 0], reference_data.values[:, 2])(tstamp)
xr[:, 3] = refint
scor1 = sensor_data.values[:, 3:5]
res = dict(xs=xs, xr=xr, scor1=scor1, scor2=scor2, ref=refint, env=reftst, rmse1=rmse1, rmse2=rmse2, rmse3=rmse3, rmse4=rmse5)
# with open(dstdir.joinpath('results.json'), 'w') as outfile:
# outfile.write(json.dumps(res, indent=4))
pass
if __name__ == '__main__':
sys.exit(main())
denoising/dataset.py 0 → 100644
View file @ 26dcca8e
import math
import numpy as np
from pathlib import Path
import pandas as pd
def check_values(dataset):
print(f'{dataset.min(axis=0)=}')
print(f'{dataset.mean(axis=0)=}')
print(f'{dataset.max(axis=0)=}')
print(f'{dataset.std(axis=0, ddof=1)=}')
print(f'{dataset.sum(axis=0)=}')
def from_csv(paths: list[Path], names=['timestamp', 'rh', 'pa', 'pm10', 'pm1', 'pm2_5', 'no2', 'temp0', 'temp1']) -> pd.DataFrame:
dataset_list = [pd.read_csv(path, skiprows=3, header=None, sep=' ', names=names) for path in paths]
# Merge datasets
dataset = pd.concat(dataset_list)
for col in dataset.columns.tolist():
dataset[col] = dataset[col].astype(float)
# Reset the shit index, otherwise you will get duplicate!
return dataset.reset_index(drop=True)
def apply_offset(dataset, offset):
dataset.timestamp += offset
return dataset
def rmse(y_actual, y_predicted):
MSE = np.square(np.subtract(y_actual,y_predicted)).mean()
return math.sqrt(MSE)
def evaluate_results(refint, scor5, scor4, scor3, scor2, sensor_data, reftst, tE, save_to=None):
corr_matrix = np.corrcoef(refint, scor5)
corr = corr_matrix[0, 1]
R_sq = corr ** 2
# How well does the reint match the observations scor5
print(f'Coefficient of determination: {R_sq}')
denoising/outliers.py 0 → 100644
View file @ 26dcca8e
This diff is collapsed.
denoising/wavelets.py 0 → 100644
View file @ 26dcca8e
import numpy as np
def get_scales(sfr, wid):
# define relevant scales for detection - A = {a1, ..., aJ} || Section II.B
w1 = round(wid[0] * sfr) + 1
w2 = round(wid[1] * sfr) + 1
# Make sure w1 is even
w1 = w1 - w1 % 2
# Filters should be of even length
return np.arange(w1, w2 + 1, 2)
\ No newline at end of file
requirements.txt 0 → 100644
View file @ 26dcca8e
numpy
pandas
pywavelets==1.4.1
scipy