Compare commits

..
40 changed files with 670 additions and 1824 deletions
+2 -7
View File
@@ -13,7 +13,7 @@ Steps to reproduce the behavior:
(e.g. Instaloader command line) (e.g. Instaloader command line)
**Expected behavior** **Expected behavior**
A clear and concise description of what you expected to happen (even if it seems obvious). A clear and concise description of what you expected to happen (if not obvious).
**Error messages and tracebacks** **Error messages and tracebacks**
If applicable, add error messages and tracebacks to help explain your problem. If applicable, add error messages and tracebacks to help explain your problem.
@@ -24,9 +24,4 @@ If applicable, add error messages and tracebacks to help explain your problem.
**Additional context** **Additional context**
Add any other context about the problem here. Add any other context about the problem here.
<!-- <!-- please also see https://instaloader.github.io/contributing.html for how to report a bug -->
Also see https://instaloader.github.io/contributing.html for how to report a bug.
Please understand that a bug issue that lacks the required information cannot be
processed.
-->
+2 -9
View File
@@ -6,15 +6,13 @@ labels: "feature suggestion"
--- ---
**Provide us a use case of the feature** **Provide us a use case of the feature**
How could the user invoke the new function? Which problem would it solve? How could the user invoke the new function? Which problem would it solve? If new information is obtained, how would it be further processed?
If new information is obtained, how would it be further processed?
**Describe the solution you'd like** **Describe the solution you'd like**
A clear and concise description of what you want to happen. A clear and concise description of what you want to happen.
**Describe alternatives you've considered** **Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered. A clear and concise description of any alternative solutions or features you've considered.
How do you achieve the here-described goal without the feature you are requesting?
**If the feature request is accepted, would you be willing to submit a pull request?** **If the feature request is accepted, would you be willing to submit a pull request?**
Yes / No Yes / No
@@ -23,9 +21,4 @@ Yes / No
**Additional context** **Additional context**
Add any other context about the feature request here. Add any other context about the feature request here.
<!-- <!-- please also see https://instaloader.github.io/contributing.html for how to suggest a feature -->
Also see https://instaloader.github.io/contributing.html for how to suggest a feature.
Please understand that a feature suggestion issue that lacks the required information cannot be
processed.
-->
+2 -8
View File
@@ -7,12 +7,6 @@ labels: question
Your question here... Your question here...
<!-- <!-- Please see our documentation: https://instaloader.github.io/ -->
People will be better able and more motivated to provide help if your question
- has a concise and meaningful title, <!-- Basic questions regarding Instaloader's usage might be more appropriate to ask on Stack Overflow -->
- is not covered by the documentation: https://instaloader.github.io/,
- is actually related to Instaloader (this is not a Python help forum),
- has not already been answered here or on other sites such as Stack Overflow: https://stackoverflow.com/questions/tagged/instaloader,
- is written in a polite and welcoming tone.
-->
-2
View File
@@ -1,2 +0,0 @@
github: [aandergr]
custom: "https://instaloader.github.io/sponsors.html"
+1 -2
View File
@@ -1,8 +1,7 @@
<!-- <!--
Thanks for your willingness to contribute to Instaloader! Please briefly Please describe:
describe
- A motivation for this change, e.g. - A motivation for this change, e.g.
- Fixes # . - Fixes # .
+2 -3
View File
@@ -8,7 +8,7 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
strategy: strategy:
matrix: matrix:
python-version: ["3.8", "3.9", "3.10"] python-version: [3.8]
steps: steps:
- name: Checkout Instaloader Repository - name: Checkout Instaloader Repository
uses: actions/checkout@v2 uses: actions/checkout@v2
@@ -20,12 +20,11 @@ jobs:
python-version: ${{ matrix.python-version }} python-version: ${{ matrix.python-version }}
- name: Install Dependencies - name: Install Dependencies
run: | run: |
python -m pip install pipenv==2020.11.15 python -m pip install pipenv==2020.6.2
pipenv sync --dev pipenv sync --dev
- name: PyLint - name: PyLint
run: pipenv run pylint instaloader run: pipenv run pylint instaloader
- name: MyPy - name: MyPy
run: pipenv run mypy -m instaloader run: pipenv run mypy -m instaloader
- name: Build Documentation - name: Build Documentation
if: matrix.python-version == '3.8'
run: pipenv run make -C docs html SPHINXOPTS="-W -n" run: pipenv run make -C docs html SPHINXOPTS="-W -n"
+1 -1
View File
@@ -30,5 +30,5 @@ jobs:
stale-pr-label: 'stale' stale-pr-label: 'stale'
exempt-issue-label: 'leave open' exempt-issue-label: 'leave open'
exempt-pr-label: 'leave open' exempt-pr-label: 'leave open'
days-before-stale: 135 days-before-stale: 90
days-before-close: 14 days-before-close: 14
+2 -2
View File
@@ -10,9 +10,9 @@ jobs:
- name: Checkout Instaloader repository - name: Checkout Instaloader repository
uses: actions/checkout@v2 uses: actions/checkout@v2
- name: Setup Python - name: Setup Python
uses: actions/setup-python@v2 uses: actions/setup-python@v1
with: with:
python-version: "3.9" python-version: 3.8
architecture: x64 architecture: x64
- name: Get the tagged version - name: Get the tagged version
id: get_version id: get_version
+1
View File
@@ -1,6 +1,7 @@
dist: xenial dist: xenial
language: python language: python
python: python:
- "3.5"
- "3.6" - "3.6"
- "3.7" - "3.7"
install: install:
-1
View File
@@ -6,4 +6,3 @@ Instaloader is written by
- Alexander Graf (@aandergr) - Alexander Graf (@aandergr)
- André Koch-Kramer (@Thammus) - André Koch-Kramer (@Thammus)
- Lars Lindqvist (@e5150) - Lars Lindqvist (@e5150)
- ... and many more, see https://github.com/instaloader/instaloader/graphs/contributors
-135
View File
@@ -1,135 +0,0 @@
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, religion, or sexual identity
and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the
overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or
advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email
address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement by opening an
issue or contacting one or more of the project maintainers.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series
of actions.
**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or
permanent ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within
the community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.0, available at
[https://www.contributor-covenant.org/version/2/0/code_of_conduct.html][v2.0].
Community Impact Guidelines were inspired by
[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
For answers to common questions about this code of conduct, see the FAQ at
[https://www.contributor-covenant.org/faq][FAQ]. Translations are available
at [https://www.contributor-covenant.org/translations][translations].
[homepage]: https://www.contributor-covenant.org
[v2.0]: https://www.contributor-covenant.org/version/2/0/code_of_conduct.html
[Mozilla CoC]: https://github.com/mozilla/diversity
[FAQ]: https://www.contributor-covenant.org/faq
[translations]: https://www.contributor-covenant.org/translations
+1 -1
View File
@@ -1,6 +1,6 @@
The MIT License (MIT) The MIT License (MIT)
Copyright (c) 2016-2020 Alexander Graf and André Koch-Kramer. Copyright (c) 2016-2019 Alexander Graf and André Koch-Kramer.
Permission is hereby granted, free of charge, to any person obtaining a copy Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal of this software and associated documentation files (the "Software"), to deal
-1
View File
@@ -11,7 +11,6 @@ pyinstaller = "*"
pefile = "*" pefile = "*"
pywin32-ctypes = "*" pywin32-ctypes = "*"
psutil = "*" psutil = "*"
typing-extensions = "*"
[packages] [packages]
requests = "*" requests = "*"
Generated
+183 -221
View File
@@ -1,7 +1,7 @@
{ {
"_meta": { "_meta": {
"hash": { "hash": {
"sha256": "9704331c2f9e76af9e5ee386bc359a46ee66d1e2f93aa880d9d148da7ba0e6f5" "sha256": "58caf1be034bed9aa7ebb8c78e75ffa7d7abe820d014b6398c444a5db566b306"
}, },
"pipfile-spec": 6, "pipfile-spec": 6,
"requires": {}, "requires": {},
@@ -16,17 +16,17 @@
"default": { "default": {
"certifi": { "certifi": {
"hashes": [ "hashes": [
"sha256:2bbf76fd432960138b3ef6dda3dde0544f27cbf8546c458e60baf371917ba9ee", "sha256:5930595817496dd21bb8dc35dad090f1c2cd0adfaf21204bf6732ca5d8ee34d3",
"sha256:50b1e4f8446b06f41be7dd6338db18e0990601dce795c2b1686458aa7e8fa7d8" "sha256:8fc0819f1f30ba15bdb34cceffb9ef04d99f420f68eb75d901e9560b8749fc41"
], ],
"version": "==2021.5.30" "version": "==2020.6.20"
}, },
"chardet": { "chardet": {
"hashes": [ "hashes": [
"sha256:0d6f53a15db4120f2b08c94f11e7d93d2c911ee118b6b30a04ec3ee8310179fa", "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",
"sha256:f864054d66fd9118f2e67044ac8981a54775ec5b67aed0441892edb553d21da5" "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"
], ],
"version": "==4.0.0" "version": "==3.0.4"
}, },
"idna": { "idna": {
"hashes": [ "hashes": [
@@ -37,19 +37,18 @@
}, },
"requests": { "requests": {
"hashes": [ "hashes": [
"sha256:27973dd4a904a4f13b263a19c866c13b92a39ed1c964655f025f3f8d3d75b804", "sha256:b3559a131db72c33ee969480840fff4bb6dd111de7dd27c8ee1f820f4f00231b",
"sha256:c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e" "sha256:fe75cc94a9443b9246fc7049224f75604b113c36acb93f87b80ed42c44cbb898"
], ],
"index": "pypi", "index": "pypi",
"version": "==2.25.1" "version": "==2.24.0"
}, },
"urllib3": { "urllib3": {
"hashes": [ "hashes": [
"sha256:753a0374df26658f99d826cfe40394a686d05985786d946fbe4165b5148f5a7c", "sha256:91056c15fa70756691db97756772bb1eb9678fa585d9184f24534b100dc60f4a",
"sha256:a7acd0977125325f516bda9735fa7142b909a8d01e8b2e4c8108d0984e6e0098" "sha256:e7983572181f5e1522d9c98453462384ee92a0be7fac5f1413a1e35c56cc0461"
], ],
"index": "pypi", "version": "==1.25.10"
"version": "==1.26.5"
} }
}, },
"develop": { "develop": {
@@ -69,38 +68,38 @@
}, },
"astroid": { "astroid": {
"hashes": [ "hashes": [
"sha256:3c9a2d84354185d13213ff2640ec03d39168dbcd13648abc84fb13ca3b2e2761", "sha256:2f4078c2a41bf377eea06d71c9d2ba4eb8f6b1af2135bec27bbbb7d8f12bb703",
"sha256:d66a600e1602736a0f24f725a511b0e50d12eb18f54b31ec276d2c26a0a62c6a" "sha256:bc58d83eb610252fd8de6363e39d4f1d0619c894b0ed24603b881c02e64c7386"
], ],
"version": "==2.5.7" "version": "==2.4.2"
}, },
"babel": { "babel": {
"hashes": [ "hashes": [
"sha256:ab49e12b91d937cd11f0b67cb259a57ab4ad2b59ac7a3b41d6c06c0ac5b0def9", "sha256:1aac2ae2d0d8ea368fa90906567f5c08463d98ade155c0c4bfedd6a0f7160e38",
"sha256:bc0c176f9f6a994582230df350aa6e05ba2ebe4b3ac317eab29d9be5d2768da0" "sha256:d670ea0b10f8b723672d3a6abeb87b565b244da220d76b4dba1b66269ec152d4"
], ],
"version": "==2.9.1" "version": "==2.8.0"
}, },
"certifi": { "certifi": {
"hashes": [ "hashes": [
"sha256:2bbf76fd432960138b3ef6dda3dde0544f27cbf8546c458e60baf371917ba9ee", "sha256:5930595817496dd21bb8dc35dad090f1c2cd0adfaf21204bf6732ca5d8ee34d3",
"sha256:50b1e4f8446b06f41be7dd6338db18e0990601dce795c2b1686458aa7e8fa7d8" "sha256:8fc0819f1f30ba15bdb34cceffb9ef04d99f420f68eb75d901e9560b8749fc41"
], ],
"version": "==2021.5.30" "version": "==2020.6.20"
}, },
"chardet": { "chardet": {
"hashes": [ "hashes": [
"sha256:0d6f53a15db4120f2b08c94f11e7d93d2c911ee118b6b30a04ec3ee8310179fa", "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",
"sha256:f864054d66fd9118f2e67044ac8981a54775ec5b67aed0441892edb553d21da5" "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"
], ],
"version": "==4.0.0" "version": "==3.0.4"
}, },
"docutils": { "docutils": {
"hashes": [ "hashes": [
"sha256:686577d2e4c32380bb50cbb22f575ed742d58168cee37e99117a854bcd88f125", "sha256:0c5b78adfbf7762415433f5515cd5c9e762339e23369dbe8000d84a4bf4ab3af",
"sha256:cf316c8370a737a022b72b56874f6602acf974a37a9fba42ec2876387549fc61" "sha256:c2de3a60e9e7d07be26b7f2b00ca0309c207e06c100f9cc2a94931fc75a478fc"
], ],
"version": "==0.17.1" "version": "==0.16"
}, },
"future": { "future": {
"hashes": [ "hashes": [
@@ -124,83 +123,81 @@
}, },
"isort": { "isort": {
"hashes": [ "hashes": [
"sha256:0a943902919f65c5684ac4e0154b1ad4fac6dcaa5d9f3426b732f1c8b5419be6", "sha256:54da7e92468955c4fceacd0c86bd0ec997b0e1ee80d97f67c35a78b719dccab1",
"sha256:2bb1680aad211e3c9944dbce1d4ba09a989f04e238296c87fe2139faa26d655d" "sha256:6e811fcb295968434526407adb8796944f1988c5b65e8139058f2014cbe100fd"
], ],
"version": "==5.8.0" "version": "==4.3.21"
}, },
"jinja2": { "jinja2": {
"hashes": [ "hashes": [
"sha256:1f06f2da51e7b56b8f238affdd6b4e2c61e39598a378cc49345bc1bd42a978a4", "sha256:89aab215427ef59c34ad58735269eb58b1a5808103067f7bb9d5836c651b3bb0",
"sha256:703f484b47a6af502e743c9122595cc812b0271f661722403114f71a79d0f5a4" "sha256:f0a4641d3cf955324a89c04f3d94663aa4d638abe8f733ecd3582848e1c37035"
], ],
"version": "==3.0.1" "version": "==2.11.2"
}, },
"lazy-object-proxy": { "lazy-object-proxy": {
"hashes": [ "hashes": [
"sha256:17e0967ba374fc24141738c69736da90e94419338fd4c7c7bef01ee26b339653", "sha256:0c4b206227a8097f05c4dbdd323c50edf81f15db3b8dc064d08c62d37e1a504d",
"sha256:1fee665d2638491f4d6e55bd483e15ef21f6c8c2095f235fef72601021e64f61", "sha256:194d092e6f246b906e8f70884e620e459fc54db3259e60cf69a4d66c3fda3449",
"sha256:22ddd618cefe54305df49e4c069fa65715be4ad0e78e8d252a33debf00f6ede2", "sha256:1be7e4c9f96948003609aa6c974ae59830a6baecc5376c25c92d7d697e684c08",
"sha256:24a5045889cc2729033b3e604d496c2b6f588c754f7a62027ad4437a7ecc4837", "sha256:4677f594e474c91da97f489fea5b7daa17b5517190899cf213697e48d3902f5a",
"sha256:410283732af311b51b837894fa2f24f2c0039aa7f220135192b38fcc42bd43d3", "sha256:48dab84ebd4831077b150572aec802f303117c8cc5c871e182447281ebf3ac50",
"sha256:4732c765372bd78a2d6b2150a6e99d00a78ec963375f236979c0626b97ed8e43", "sha256:5541cada25cd173702dbd99f8e22434105456314462326f06dba3e180f203dfd",
"sha256:489000d368377571c6f982fba6497f2aa13c6d1facc40660963da62f5c379726", "sha256:59f79fef100b09564bc2df42ea2d8d21a64fdcda64979c0fa3db7bdaabaf6239",
"sha256:4f60460e9f1eb632584c9685bccea152f4ac2130e299784dbaf9fae9f49891b3", "sha256:8d859b89baf8ef7f8bc6b00aa20316483d67f0b1cbf422f5b4dc56701c8f2ffb",
"sha256:5743a5ab42ae40caa8421b320ebf3a998f89c85cdc8376d6b2e00bd12bd1b587", "sha256:9254f4358b9b541e3441b007a0ea0764b9d056afdeafc1a5569eee1cc6c1b9ea",
"sha256:85fb7608121fd5621cc4377a8961d0b32ccf84a7285b4f1d21988b2eae2868e8", "sha256:9651375199045a358eb6741df3e02a651e0330be090b3bc79f6d0de31a80ec3e",
"sha256:9698110e36e2df951c7c36b6729e96429c9c32b3331989ef19976592c5f3c77a", "sha256:97bb5884f6f1cdce0099f86b907aa41c970c3c672ac8b9c8352789e103cf3156",
"sha256:9d397bf41caad3f489e10774667310d73cb9c4258e9aed94b9ec734b34b495fd", "sha256:9b15f3f4c0f35727d3a0fba4b770b3c4ebbb1fa907dbcc046a1d2799f3edd142",
"sha256:b579f8acbf2bdd9ea200b1d5dea36abd93cabf56cf626ab9c744a432e15c815f", "sha256:a2238e9d1bb71a56cd710611a1614d1194dc10a175c1e08d75e1a7bcc250d442",
"sha256:b865b01a2e7f96db0c5d12cfea590f98d8c5ba64ad222300d93ce6ff9138bcad", "sha256:a6ae12d08c0bf9909ce12385803a543bfe99b95fe01e752536a60af2b7797c62",
"sha256:bf34e368e8dd976423396555078def5cfc3039ebc6fc06d1ae2c5a65eebbcde4", "sha256:ca0a928a3ddbc5725be2dd1cf895ec0a254798915fb3a36af0964a0a4149e3db",
"sha256:c6938967f8528b3668622a9ed3b31d145fab161a32f5891ea7b84f6b790be05b", "sha256:cb2c7c57005a6804ab66f106ceb8482da55f5314b7fcb06551db1edae4ad1531",
"sha256:d1c2676e3d840852a2de7c7d5d76407c772927addff8d742b9808fe0afccebdf", "sha256:d74bb8693bf9cf75ac3b47a54d716bbb1a92648d5f781fc799347cfc95952383",
"sha256:d7124f52f3bd259f510651450e18e0fd081ed82f3c08541dffc7b94b883aa981", "sha256:d945239a5639b3ff35b70a88c5f2f491913eb94871780ebfabb2568bd58afc5a",
"sha256:d900d949b707778696fdf01036f58c9876a0d8bfe116e8d220cfd4b15f14e741", "sha256:eba7011090323c1dadf18b3b689845fd96a61ba0a1dfbd7f24b921398affc357",
"sha256:ebfd274dcd5133e0afae738e6d9da4323c3eb021b3e13052d8cbd0e457b1256e", "sha256:efa1909120ce98bbb3777e8b6f92237f5d5c8ea6758efea36a473e1d38f7d3e4",
"sha256:ed361bb83436f117f9917d282a456f9e5009ea12fd6de8742d1a4752c3017e93", "sha256:f3900e8a5de27447acbf900b4750b0ddfd7ec1ea7fbaf11dfa911141bc522af0"
"sha256:f5144c75445ae3ca2057faac03fda5a902eff196702b0a24daf1d6ce0650514b"
], ],
"version": "==1.6.0" "version": "==1.4.3"
}, },
"markupsafe": { "markupsafe": {
"hashes": [ "hashes": [
"sha256:01a9b8ea66f1658938f65b93a85ebe8bc016e6769611be228d797c9d998dd298", "sha256:00bc623926325b26bb9605ae9eae8a215691f33cae5df11ca5424f06f2d1f473",
"sha256:023cb26ec21ece8dc3907c0e8320058b2e0cb3c55cf9564da612bc325bed5e64", "sha256:09027a7803a62ca78792ad89403b1b7a73a01c8cb65909cd876f7fcebd79b161",
"sha256:0446679737af14f45767963a1a9ef7620189912317d095f2d9ffa183a4d25d2b", "sha256:09c4b7f37d6c648cb13f9230d847adf22f8171b1ccc4d5682398e77f40309235",
"sha256:0717a7390a68be14b8c793ba258e075c6f4ca819f15edfc2a3a027c823718567", "sha256:1027c282dad077d0bae18be6794e6b6b8c91d58ed8a8d89a89d59693b9131db5",
"sha256:0955295dd5eec6cb6cc2fe1698f4c6d84af2e92de33fbcac4111913cd100a6ff", "sha256:13d3144e1e340870b25e7b10b98d779608c02016d5184cfb9927a9f10c689f42",
"sha256:10f82115e21dc0dfec9ab5c0223652f7197feb168c940f3ef61563fc2d6beb74", "sha256:24982cc2533820871eba85ba648cd53d8623687ff11cbb805be4ff7b4c971aff",
"sha256:1d609f577dc6e1aa17d746f8bd3c31aa4d258f4070d61b2aa5c4166c1539de35", "sha256:29872e92839765e546828bb7754a68c418d927cd064fd4708fab9fe9c8bb116b",
"sha256:2ef54abee730b502252bcdf31b10dacb0a416229b72c18b19e24a4509f273d26", "sha256:43a55c2930bbc139570ac2452adf3d70cdbb3cfe5912c71cdce1c2c6bbd9c5d1",
"sha256:3c112550557578c26af18a1ccc9e090bfe03832ae994343cfdacd287db6a6ae7", "sha256:46c99d2de99945ec5cb54f23c8cd5689f6d7177305ebff350a58ce5f8de1669e",
"sha256:47ab1e7b91c098ab893b828deafa1203de86d0bc6ab587b160f78fe6c4011f75", "sha256:500d4957e52ddc3351cabf489e79c91c17f6e0899158447047588650b5e69183",
"sha256:49e3ceeabbfb9d66c3aef5af3a60cc43b85c33df25ce03d0031a608b0a8b2e3f", "sha256:535f6fc4d397c1563d08b88e485c3496cf5784e927af890fb3c3aac7f933ec66",
"sha256:4efca8f86c54b22348a5467704e3fec767b2db12fc39c6d963168ab1d3fc9135", "sha256:596510de112c685489095da617b5bcbbac7dd6384aeebeda4df6025d0256a81b",
"sha256:53edb4da6925ad13c07b6d26c2a852bd81e364f95301c66e930ab2aef5b5ddd8", "sha256:62fe6c95e3ec8a7fad637b7f3d372c15ec1caa01ab47926cfdf7a75b40e0eac1",
"sha256:594c67807fb16238b30c44bdf74f36c02cdf22d1c8cda91ef8a0ed8dabf5620a", "sha256:6788b695d50a51edb699cb55e35487e430fa21f1ed838122d722e0ff0ac5ba15",
"sha256:611d1ad9a4288cf3e3c16014564df047fe08410e628f89805e475368bd304914", "sha256:6dd73240d2af64df90aa7c4e7481e23825ea70af4b4922f8ede5b9e35f78a3b1",
"sha256:6557b31b5e2c9ddf0de32a691f2312a32f77cd7681d8af66c2692efdbef84c18", "sha256:717ba8fe3ae9cc0006d7c451f0bb265ee07739daf76355d06366154ee68d221e",
"sha256:693ce3f9e70a6cf7d2fb9e6c9d8b204b6b39897a2c4a1aa65728d5ac97dcc1d8", "sha256:79855e1c5b8da654cf486b830bd42c06e8780cea587384cf6545b7d9ac013a0b",
"sha256:6a7fae0dd14cf60ad5ff42baa2e95727c3d81ded453457771d02b7d2b3f9c0c2", "sha256:7c1699dfe0cf8ff607dbdcc1e9b9af1755371f92a68f706051cc8c37d447c905",
"sha256:6c4ca60fa24e85fe25b912b01e62cb969d69a23a5d5867682dd3e80b5b02581d", "sha256:88e5fcfb52ee7b911e8bb6d6aa2fd21fbecc674eadd44118a9cc3863f938e735",
"sha256:7d91275b0245b1da4d4cfa07e0faedd5b0812efc15b702576d103293e252af1b", "sha256:8defac2f2ccd6805ebf65f5eeb132adcf2ab57aa11fdf4c0dd5169a004710e7d",
"sha256:905fec760bd2fa1388bb5b489ee8ee5f7291d692638ea5f67982d968366bef9f", "sha256:98c7086708b163d425c67c7a91bad6e466bb99d797aa64f965e9d25c12111a5e",
"sha256:97383d78eb34da7e1fa37dd273c20ad4320929af65d156e35a5e2d89566d9dfb", "sha256:9add70b36c5666a2ed02b43b335fe19002ee5235efd4b8a89bfcf9005bebac0d",
"sha256:984d76483eb32f1bcb536dc27e4ad56bba4baa70be32fa87152832cdd9db0833", "sha256:9bf40443012702a1d2070043cb6291650a0841ece432556f784f004937f0f32c",
"sha256:a30e67a65b53ea0a5e62fe23682cfe22712e01f453b95233b25502f7c61cb415", "sha256:ade5e387d2ad0d7ebf59146cc00c8044acbd863725f887353a10df825fc8ae21",
"sha256:ab3ef638ace319fa26553db0624c4699e31a28bb2a835c5faca8f8acf6a5a902", "sha256:b00c1de48212e4cc9603895652c5c410df699856a2853135b3967591e4beebc2",
"sha256:b2f4bf27480f5e5e8ce285a8c8fd176c0b03e93dcc6646477d4630e83440c6a9", "sha256:b1282f8c00509d99fef04d8ba936b156d419be841854fe901d8ae224c59f0be5",
"sha256:b7f2d075102dc8c794cbde1947378051c4e5180d52d276987b8d28a3bd58c17d", "sha256:b2051432115498d3562c084a49bba65d97cf251f5a331c64a12ee7e04dacc51b",
"sha256:be98f628055368795d818ebf93da628541e10b75b41c559fdf36d104c5787066", "sha256:ba59edeaa2fc6114428f1637ffff42da1e311e29382d81b339c1817d37ec93c6",
"sha256:d7f9850398e85aba693bb640262d3611788b1f29a79f0c93c565694658f4071f", "sha256:c8716a48d94b06bb3b2524c2b77e055fb313aeb4ea620c8dd03a105574ba704f",
"sha256:f5653a225f31e113b152e56f154ccbe59eeb1c7487b39b9d9f9cdb58e6c79dc5", "sha256:cd5df75523866410809ca100dc9681e301e3c27567cf498077e8551b6d20e42f",
"sha256:f826e31d18b516f653fe296d967d700fddad5901ae07c622bb3705955e1faa94", "sha256:cdb132fc825c38e1aeec2c8aa9338310d29d337bebbd7baa06889d09a60a1fa2",
"sha256:f8ba0e8349a38d3001fae7eadded3f6606f0da5d748ee53cc1dab1d6527b9509", "sha256:e249096428b3ae81b08327a63a485ad0878de3fb939049038579ac0ef61e17e7",
"sha256:f9081981fe268bd86831e5c75f7de206ef275defcb82bc70740ae6dc507aee51", "sha256:e8313f01ba26fbbe36c7be1966a7b7424942f670f38e666995b88d012765b9be"
"sha256:fa130dd50c57d53368c9d59395cb5526eda596d3ffe36666cd81a44d56e48872"
], ],
"version": "==2.0.1" "version": "==1.1.1"
}, },
"mccabe": { "mccabe": {
"hashes": [ "hashes": [
@@ -211,31 +208,23 @@
}, },
"mypy": { "mypy": {
"hashes": [ "hashes": [
"sha256:0d0a87c0e7e3a9becdfbe936c981d32e5ee0ccda3e0f07e1ef2c3d1a817cf73e", "sha256:2c6cde8aa3426c1682d35190b59b71f661237d74b053822ea3d748e2c9578a7c",
"sha256:25adde9b862f8f9aac9d2d11971f226bd4c8fbaa89fb76bdadb267ef22d10064", "sha256:3fdda71c067d3ddfb21da4b80e2686b71e9e5c72cca65fa216d207a358827f86",
"sha256:28fb5479c494b1bab244620685e2eb3c3f988d71fd5d64cc753195e8ed53df7c", "sha256:5dd13ff1f2a97f94540fd37a49e5d255950ebcdf446fb597463a40d0df3fac8b",
"sha256:2f9b3407c58347a452fc0736861593e105139b905cca7d097e413453a1d650b4", "sha256:6731603dfe0ce4352c555c6284c6db0dc935b685e9ce2e4cf220abe1e14386fd",
"sha256:33f159443db0829d16f0a8d83d94df3109bb6dd801975fe86bacb9bf71628e97", "sha256:6bb93479caa6619d21d6e7160c552c1193f6952f0668cdda2f851156e85186fc",
"sha256:3f2aca7f68580dc2508289c729bd49ee929a436208d2b2b6aab15745a70a57df", "sha256:81c7908b94239c4010e16642c9102bfc958ab14e36048fa77d0be3289dda76ea",
"sha256:499c798053cdebcaa916eef8cd733e5584b5909f789de856b482cd7d069bdad8", "sha256:9c7a9a7ceb2871ba4bac1cf7217a7dd9ccd44c27c2950edbc6dc08530f32ad4e",
"sha256:4eec37370483331d13514c3f55f446fc5248d6373e7029a29ecb7b7494851e7a", "sha256:a4a2cbcfc4cbf45cd126f531dedda8485671545b43107ded25ce952aac6fb308",
"sha256:552a815579aa1e995f39fd05dde6cd378e191b063f031f2acfe73ce9fb7f9e56", "sha256:b7fbfabdbcc78c4f6fc4712544b9b0d6bf171069c6e0e3cb82440dd10ced3406",
"sha256:5873888fff1c7cf5b71efbe80e0e73153fe9212fafdf8e44adfe4c20ec9f82d7", "sha256:c05b9e4fb1d8a41d41dec8786c94f3b95d3c5f528298d769eb8e73d293abc48d",
"sha256:61a3d5b97955422964be6b3baf05ff2ce7f26f52c85dd88db11d5e03e146a3a6", "sha256:d7df6eddb6054d21ca4d3c6249cae5578cb4602951fd2b6ee2f5510ffb098707",
"sha256:674e822aa665b9fd75130c6c5f5ed9564a38c6cea6a6432ce47eafb68ee578c5", "sha256:e0b61738ab504e656d1fe4ff0c0601387a5489ca122d55390ade31f9ca0e252d",
"sha256:7ce3175801d0ae5fdfa79b4f0cfed08807af4d075b402b7e294e6aa72af9aa2a", "sha256:eff7d4a85e9eea55afa34888dfeaccde99e7520b51f867ac28a48492c0b1130c",
"sha256:9743c91088d396c1a5a3c9978354b61b0382b4e3c440ce83cf77994a43e8c521", "sha256:f05644db6779387ccdb468cc47a44b4356fc2ffa9287135d05b70a98dc83b89a"
"sha256:9f94aac67a2045ec719ffe6111df543bac7874cee01f41928f6969756e030564",
"sha256:a26f8ec704e5a7423c8824d425086705e381b4f1dfdef6e3a1edab7ba174ec49",
"sha256:abf7e0c3cf117c44d9285cc6128856106183938c68fd4944763003decdcfeb66",
"sha256:b09669bcda124e83708f34a94606e01b614fa71931d356c1f1a5297ba11f110a",
"sha256:cd07039aa5df222037005b08fbbfd69b3ab0b0bd7a07d7906de75ae52c4e3119",
"sha256:d23e0ea196702d918b60c8288561e722bf437d82cb7ef2edcd98cfa38905d506",
"sha256:d65cc1df038ef55a99e617431f0553cd77763869eebdf9042403e16089fe746c",
"sha256:d7da2e1d5f558c37d6e8c1246f1aec1e7349e4913d8fb3cb289a35de573fe2eb"
], ],
"index": "pypi", "index": "pypi",
"version": "==0.812" "version": "==0.782"
}, },
"mypy-extensions": { "mypy-extensions": {
"hashes": [ "hashes": [
@@ -246,10 +235,10 @@
}, },
"packaging": { "packaging": {
"hashes": [ "hashes": [
"sha256:5b327ac1320dc863dca72f4514ecc086f31186744b84a230374cc1fd776feae5", "sha256:4357f74f47b9c12db93624a82154e9b120fa8293699949152b22065d556079f8",
"sha256:67714da7f7bc052e064859c05c595155bd1ee9f69f76557e21f051443c20947a" "sha256:998416ba6962ae7fbd6596850b80e17859a5753ba17c32284f67bfff33784181"
], ],
"version": "==20.9" "version": "==20.4"
}, },
"pefile": { "pefile": {
"hashes": [ "hashes": [
@@ -260,66 +249,42 @@
}, },
"psutil": { "psutil": {
"hashes": [ "hashes": [
"sha256:0066a82f7b1b37d334e68697faba68e5ad5e858279fd6351c8ca6024e8d6ba64", "sha256:0ee3c36428f160d2d8fce3c583a0353e848abb7de9732c50cf3356dd49ad63f8",
"sha256:02b8292609b1f7fcb34173b25e48d0da8667bc85f81d7476584d889c6e0f2131", "sha256:10512b46c95b02842c225f58fa00385c08fa00c68bac7da2d9a58ebe2c517498",
"sha256:0ae6f386d8d297177fd288be6e8d1afc05966878704dad9847719650e44fc49c", "sha256:4080869ed93cce662905b029a1770fe89c98787e543fa7347f075ade761b19d6",
"sha256:0c9ccb99ab76025f2f0bbecf341d4656e9c1351db8cc8a03ccd62e318ab4b5c6", "sha256:5e9d0f26d4194479a13d5f4b3798260c20cecf9ac9a461e718eb59ea520a360c",
"sha256:0dd4465a039d343925cdc29023bb6960ccf4e74a65ad53e768403746a9207023", "sha256:66c18ca7680a31bf16ee22b1d21b6397869dda8059dbdb57d9f27efa6615f195",
"sha256:12d844996d6c2b1d3881cfa6fa201fd635971869a9da945cf6756105af73d2df", "sha256:68d36986ded5dac7c2dcd42f2682af1db80d4bce3faa126a6145c1637e1b559f",
"sha256:1bff0d07e76114ec24ee32e7f7f8d0c4b0514b3fae93e3d2aaafd65d22502394", "sha256:90990af1c3c67195c44c9a889184f84f5b2320dce3ee3acbd054e3ba0b4a7beb",
"sha256:245b5509968ac0bd179287d91210cd3f37add77dad385ef238b275bad35fa1c4", "sha256:a5b120bb3c0c71dfe27551f9da2f3209a8257a178ed6c628a819037a8df487f1",
"sha256:28ff7c95293ae74bf1ca1a79e8805fcde005c18a122ca983abf676ea3466362b", "sha256:d8a82162f23c53b8525cf5f14a355f5d1eea86fa8edde27287dd3a98399e4fdf",
"sha256:36b3b6c9e2a34b7d7fbae330a85bf72c30b1c827a4366a07443fc4b6270449e2", "sha256:f2018461733b23f308c298653c8903d32aaad7873d25e1d228765e91ae42c3f2",
"sha256:52de075468cd394ac98c66f9ca33b2f54ae1d9bff1ef6b67a212ee8f639ec06d", "sha256:ff1977ba1a5f71f89166d5145c3da1cea89a0fdb044075a12c720ee9123ec818"
"sha256:5da29e394bdedd9144c7331192e20c1f79283fb03b06e6abd3a8ae45ffecee65",
"sha256:61f05864b42fedc0771d6d8e49c35f07efd209ade09a5afe6a5059e7bb7bf83d",
"sha256:6223d07a1ae93f86451d0198a0c361032c4c93ebd4bf6d25e2fb3edfad9571ef",
"sha256:6323d5d845c2785efb20aded4726636546b26d3b577aded22492908f7c1bdda7",
"sha256:6ffe81843131ee0ffa02c317186ed1e759a145267d54fdef1bc4ea5f5931ab60",
"sha256:74f2d0be88db96ada78756cb3a3e1b107ce8ab79f65aa885f76d7664e56928f6",
"sha256:74fb2557d1430fff18ff0d72613c5ca30c45cdbfcddd6a5773e9fc1fe9364be8",
"sha256:90d4091c2d30ddd0a03e0b97e6a33a48628469b99585e2ad6bf21f17423b112b",
"sha256:90f31c34d25b1b3ed6c40cdd34ff122b1887a825297c017e4cbd6796dd8b672d",
"sha256:99de3e8739258b3c3e8669cb9757c9a861b2a25ad0955f8e53ac662d66de61ac",
"sha256:c6a5fd10ce6b6344e616cf01cc5b849fa8103fbb5ba507b6b2dee4c11e84c935",
"sha256:ce8b867423291cb65cfc6d9c4955ee9bfc1e21fe03bb50e177f2b957f1c2469d",
"sha256:d225cd8319aa1d3c85bf195c4e07d17d3cd68636b8fc97e6cf198f782f99af28",
"sha256:ea313bb02e5e25224e518e4352af4bf5e062755160f77e4b1767dd5ccb65f876",
"sha256:ea372bcc129394485824ae3e3ddabe67dc0b118d262c568b4d2602a7070afdb0",
"sha256:f4634b033faf0d968bb9220dd1c793b897ab7f1189956e1aa9eae752527127d3",
"sha256:fcc01e900c1d7bee2a37e5d6e4f9194760a93597c97fee89c4ae51701de03563"
], ],
"index": "pypi", "index": "pypi",
"version": "==5.8.0" "version": "==5.7.2"
}, },
"pygments": { "pygments": {
"hashes": [ "hashes": [
"sha256:a18f47b506a429f6f4b9df81bb02beab9ca21d0a5fee38ed15aef65f0545519f", "sha256:647344a061c249a3b74e230c739f434d7ea4d8b1d5f3721bc0f3558049b38f44",
"sha256:d66e804411278594d764fc69ec36ec13d9ae9147193a1740cd34d272ca383b8e" "sha256:ff7a40b4860b727ab48fad6360eb351cc1b33cbf9b15a0f689ca5353e9463324"
], ],
"version": "==2.9.0" "version": "==2.6.1"
}, },
"pyinstaller": { "pyinstaller": {
"hashes": [ "hashes": [
"sha256:f5c0eeb2aa663cce9a5404292c0195011fa500a6501c873a466b2e8cad3c950c" "sha256:3730fa80d088f8bb7084d32480eb87cbb4ddb64123363763cf8f2a1378c1c4b7"
], ],
"index": "pypi", "index": "pypi",
"version": "==4.2" "version": "==3.6"
},
"pyinstaller-hooks-contrib": {
"hashes": [
"sha256:27558072021857d89524c42136feaa2ffe4f003f1bdf0278f9b24f6902c1759c",
"sha256:892310e6363655838485ee748bf1c5e5cade7963686d9af8650ee218a3e0b031"
],
"version": "==2021.1"
}, },
"pylint": { "pylint": {
"hashes": [ "hashes": [
"sha256:0e21d3b80b96740909d77206d741aa3ce0b06b41be375d92e1f3244a274c1f8a", "sha256:7dd78437f2d8d019717dbf287772d0b2dbdfd13fc016aa7faa08d67bccc46adc",
"sha256:d09b0b07ba06bcdff463958f53f23df25e740ecd81895f7d2699ec04bbd8dc3b" "sha256:d0ece7d223fe422088b0e8f13fa0a1e8eb745ebffcb8ed53d3e95394b6101a1c"
], ],
"index": "pypi", "index": "pypi",
"version": "==2.7.2" "version": "==2.5.3"
}, },
"pyparsing": { "pyparsing": {
"hashes": [ "hashes": [
@@ -330,10 +295,10 @@
}, },
"pytz": { "pytz": {
"hashes": [ "hashes": [
"sha256:83a4a90894bf38e243cf052c8b58f381bfe9a7a483f6a9cab140bc7f702ac4da", "sha256:a494d53b6d39c3c6e44c3bec237336e14305e4f29bbf800b599253057fbb79ed",
"sha256:eb10ce3e7736052ed3623d49975ce333bcd712c7bb19a58b9e2089d4057d0798" "sha256:c35965d010ce31b23eeb663ed3cc8c906275d6be1a34393a1d73a41febf4a048"
], ],
"version": "==2021.1" "version": "==2020.1"
}, },
"pywin32-ctypes": { "pywin32-ctypes": {
"hashes": [ "hashes": [
@@ -345,18 +310,25 @@
}, },
"requests": { "requests": {
"hashes": [ "hashes": [
"sha256:27973dd4a904a4f13b263a19c866c13b92a39ed1c964655f025f3f8d3d75b804", "sha256:b3559a131db72c33ee969480840fff4bb6dd111de7dd27c8ee1f820f4f00231b",
"sha256:c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e" "sha256:fe75cc94a9443b9246fc7049224f75604b113c36acb93f87b80ed42c44cbb898"
], ],
"index": "pypi", "index": "pypi",
"version": "==2.25.1" "version": "==2.24.0"
},
"six": {
"hashes": [
"sha256:30639c035cdb23534cd4aa2dd52c3bf48f06e5f4a941509c8bafd8ce11080259",
"sha256:8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced"
],
"version": "==1.15.0"
}, },
"snowballstemmer": { "snowballstemmer": {
"hashes": [ "hashes": [
"sha256:b51b447bea85f9968c13b650126a888aabd4cb4463fca868ec596826325dedc2", "sha256:209f257d7533fdb3cb73bdbd24f436239ca3b2fa67d56f6ff88e86be08cc5ef0",
"sha256:e997baa4f2e9139951b6f4c631bad912dfd3c792467e2f03d7239464af90e914" "sha256:df3bac3df4c2c01363f3dd2cfa78cce2840a79b9f1c2d2de9ce8d31683992f52"
], ],
"version": "==2.1.0" "version": "==2.0.0"
}, },
"sphinx": { "sphinx": {
"hashes": [ "hashes": [
@@ -382,10 +354,10 @@
}, },
"sphinxcontrib-htmlhelp": { "sphinxcontrib-htmlhelp": {
"hashes": [ "hashes": [
"sha256:d412243dfb797ae3ec2b59eca0e52dac12e75a241bf0e4eb861e450d06c6ed07", "sha256:3c0bc24a2c41e340ac37c85ced6dafc879ab485c095b1d65d2461ac2f7cca86f",
"sha256:f5f8bb2d0d629f398bf47d0d69c07bc13b65f75a81ad9e2f71a63d4b7a2f6db2" "sha256:e8f5bb7e31b2dbb25b9cc435c8ab7a79787ebf7f906155729338f3156d93659b"
], ],
"version": "==2.0.0" "version": "==1.0.3"
}, },
"sphinxcontrib-jsmath": { "sphinxcontrib-jsmath": {
"hashes": [ "hashes": [
@@ -403,69 +375,59 @@
}, },
"sphinxcontrib-serializinghtml": { "sphinxcontrib-serializinghtml": {
"hashes": [ "hashes": [
"sha256:352a9a00ae864471d3a7ead8d7d79f5fc0b57e8b3f95e9867eb9eb28999b92fd", "sha256:eaa0eccc86e982a9b939b2b82d12cc5d013385ba5eadcc7e4fed23f4405f77bc",
"sha256:aa5f6de5dfdf809ef505c4895e51ef5c9eac17d0f287933eb49ec495280b6952" "sha256:f242a81d423f59617a8e5cf16f5d4d74e28ee9a66f9e5b637a18082991db5a9a"
], ],
"version": "==1.1.5" "version": "==1.1.4"
}, },
"toml": { "toml": {
"hashes": [ "hashes": [
"sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b", "sha256:926b612be1e5ce0634a2ca03470f95169cf16f939018233a670519cb4ac58b0f",
"sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f" "sha256:bda89d5935c2eac546d648028b9901107a595863cb36bae0c73ac804a9b4ce88"
], ],
"version": "==0.10.2" "version": "==0.10.1"
}, },
"typed-ast": { "typed-ast": {
"hashes": [ "hashes": [
"sha256:01ae5f73431d21eead5015997ab41afa53aa1fbe252f9da060be5dad2c730ace", "sha256:0666aa36131496aed8f7be0410ff974562ab7eeac11ef351def9ea6fa28f6355",
"sha256:067a74454df670dcaa4e59349a2e5c81e567d8d65458d480a5b3dfecec08c5ff", "sha256:0c2c07682d61a629b68433afb159376e24e5b2fd4641d35424e462169c0a7919",
"sha256:0fb71b8c643187d7492c1f8352f2c15b4c4af3f6338f21681d3681b3dc31a266", "sha256:249862707802d40f7f29f6e1aad8d84b5aa9e44552d2cc17384b209f091276aa",
"sha256:1b3ead4a96c9101bef08f9f7d1217c096f31667617b58de957f690c92378b528", "sha256:24995c843eb0ad11a4527b026b4dde3da70e1f2d8806c99b7b4a7cf491612652",
"sha256:2068531575a125b87a41802130fa7e29f26c09a2833fea68d9a40cf33902eba6", "sha256:269151951236b0f9a6f04015a9004084a5ab0d5f19b57de779f908621e7d8b75",
"sha256:209596a4ec71d990d71d5e0d312ac935d86930e6eecff6ccc7007fe54d703808", "sha256:4083861b0aa07990b619bd7ddc365eb7fa4b817e99cf5f8d9cf21a42780f6e01",
"sha256:2c726c276d09fc5c414693a2de063f521052d9ea7c240ce553316f70656c84d4", "sha256:498b0f36cc7054c1fead3d7fc59d2150f4d5c6c56ba7fb150c013fbc683a8d2d",
"sha256:398e44cd480f4d2b7ee8d98385ca104e35c81525dd98c519acff1b79bdaac363", "sha256:4e3e5da80ccbebfff202a67bf900d081906c358ccc3d5e3c8aea42fdfdfd51c1",
"sha256:52b1eb8c83f178ab787f3a4283f68258525f8d70f778a2f6dd54d3b5e5fb4341", "sha256:6daac9731f172c2a22ade6ed0c00197ee7cc1221aa84cfdf9c31defeb059a907",
"sha256:5feca99c17af94057417d744607b82dd0a664fd5e4ca98061480fd8b14b18d04", "sha256:715ff2f2df46121071622063fc7543d9b1fd19ebfc4f5c8895af64a77a8c852c",
"sha256:7538e495704e2ccda9b234b82423a4038f324f3a10c43bc088a1636180f11a41", "sha256:73d785a950fc82dd2a25897d525d003f6378d1cb23ab305578394694202a58c3",
"sha256:760ad187b1041a154f0e4d0f6aae3e40fdb51d6de16e5c99aedadd9246450e9e", "sha256:8c8aaad94455178e3187ab22c8b01a3837f8ee50e09cf31f1ba129eb293ec30b",
"sha256:777a26c84bea6cd934422ac2e3b78863a37017618b6e5c08f92ef69853e765d3", "sha256:8ce678dbaf790dbdb3eba24056d5364fb45944f33553dd5869b7580cdbb83614",
"sha256:95431a26309a21874005845c21118c83991c63ea800dd44843e42a916aec5899", "sha256:aaee9905aee35ba5905cfb3c62f3e83b3bec7b39413f0a7f19be4e547ea01ebb",
"sha256:9ad2c92ec681e02baf81fdfa056fe0d818645efa9af1f1cd5fd6f1bd2bdfd805", "sha256:bcd3b13b56ea479b3650b82cabd6b5343a625b0ced5429e4ccad28a8973f301b",
"sha256:9c6d1a54552b5330bc657b7ef0eae25d00ba7ffe85d9ea8ae6540d2197a3788c", "sha256:c9e348e02e4d2b4a8b2eedb48210430658df6951fa484e59de33ff773fbd4b41",
"sha256:aee0c1256be6c07bd3e1263ff920c325b59849dc95392a05f258bb9b259cf39c", "sha256:d205b1b46085271b4e15f670058ce182bd1199e56b317bf2ec004b6a44f911f6",
"sha256:af3d4a73793725138d6b334d9d247ce7e5f084d96284ed23f22ee626a7b88e39", "sha256:d43943ef777f9a1c42bf4e552ba23ac77a6351de620aa9acf64ad54933ad4d34",
"sha256:b36b4f3920103a25e1d5d024d155c504080959582b928e91cb608a65c3a49e1a", "sha256:d5d33e9e7af3b34a40dc05f498939f0ebf187f07c385fd58d591c533ad8562fe",
"sha256:b9574c6f03f685070d859e75c7f9eeca02d6933273b5e69572e5ff9d5e3931c3", "sha256:fc0fea399acb12edbf8a628ba8d2312f583bdbdb3335635db062fa98cf71fca4",
"sha256:bff6ad71c81b3bba8fa35f0f1921fb24ff4476235a6e94a26ada2e54370e6da7", "sha256:fe460b922ec15dd205595c9b5b99e2f056fd98ae8f9f56b888e7a17dc2b757e7"
"sha256:c190f0899e9f9f8b6b7863debfb739abcb21a5c054f911ca3596d12b8a4c4c7f",
"sha256:c907f561b1e83e93fad565bac5ba9c22d96a54e7ea0267c708bffe863cbe4075",
"sha256:cae53c389825d3b46fb37538441f75d6aecc4174f615d048321b716df2757fb0",
"sha256:dd4a21253f42b8d2b48410cb31fe501d32f8b9fbeb1f55063ad102fe9c425e40",
"sha256:dde816ca9dac1d9c01dd504ea5967821606f02e510438120091b84e852367428",
"sha256:f2362f3cb0f3172c42938946dbc5b7843c2a28aec307c49100c8b38764eb6927",
"sha256:f328adcfebed9f11301eaedfa48e15bdece9b519fb27e6a8c01aa52a17ec31b3",
"sha256:f8afcf15cc511ada719a88e013cec87c11aff7b91f019295eb4530f96fe5ef2f",
"sha256:fb1bbeac803adea29cedd70781399c99138358c26d05fcbd23c13016b7f5ec65"
], ],
"version": "==1.4.3" "markers": "implementation_name == 'cpython' and python_version < '3.8'",
"version": "==1.4.1"
}, },
"typing-extensions": { "typing-extensions": {
"hashes": [ "hashes": [
"sha256:7cb407020f00f7bfc3cb3e7881628838e69d8f3fcab2f64742a5e76b2f841918", "sha256:6e95524d8a547a91e08f404ae485bbb71962de46967e1b71a0cb89af24e761c5",
"sha256:99d4073b617d30288f569d3f13d2bd7548c3a7e4c8de87db09a9d29bb3a4a60c", "sha256:79ee589a3caca649a9bfd2a8de4709837400dfa00b6cc81962a1e6a1815969ae",
"sha256:dafc7639cde7f1b6e1acc0f457842a83e722ccca8eef5270af2d74792619a89f" "sha256:f8d2bd89d25bc39dabe7d23df520442fa1d8969b82544370e03d88b5a591c392"
], ],
"index": "pypi", "version": "==3.7.4.2"
"version": "==3.7.4.3"
}, },
"urllib3": { "urllib3": {
"hashes": [ "hashes": [
"sha256:753a0374df26658f99d826cfe40394a686d05985786d946fbe4165b5148f5a7c", "sha256:91056c15fa70756691db97756772bb1eb9678fa585d9184f24534b100dc60f4a",
"sha256:a7acd0977125325f516bda9735fa7142b909a8d01e8b2e4c8108d0984e6e0098" "sha256:e7983572181f5e1522d9c98453462384ee92a0be7fac5f1413a1e35c56cc0461"
], ],
"index": "pypi", "version": "==1.25.10"
"version": "==1.26.5"
}, },
"wrapt": { "wrapt": {
"hashes": [ "hashes": [
+20 -42
View File
@@ -29,7 +29,7 @@
.. |downloads| image:: https://pepy.tech/badge/instaloader/month .. |downloads| image:: https://pepy.tech/badge/instaloader/month
:alt: PyPI Download Count :alt: PyPI Download Count
:target: https://pepy.tech/project/instaloader :target: https://pepy.tech/project/instaloader/month
.. badges-end .. badges-end
@@ -84,19 +84,7 @@ To later **update your local copy** of that profiles, you may run
instaloader --fast-update profile [profile ...] instaloader --fast-update profile [profile ...]
If ``--fast-update`` is given, Instaloader stops when arriving at the If ``--fast-update`` is given, Instaloader stops when arriving at the
first already-downloaded picture. first already-downloaded picture. When updating profiles, Instaloader
Alternatively, you can use ``--latest-stamps`` to have Instaloader store
the time each profile was last downloaded and only download newer media:
::
instaloader --latest-stamps -- profile [profile ...]
With this option it's possible to move or delete downloaded media and still keep
the archive updated.
When updating profiles, Instaloader
automatically **detects profile name changes** and renames the target directory automatically **detects profile name changes** and renames the target directory
accordingly. accordingly.
@@ -114,34 +102,6 @@ already have a valid session cookie file.
`Instaloader Documentation <https://instaloader.github.io/basic-usage.html>`__ `Instaloader Documentation <https://instaloader.github.io/basic-usage.html>`__
Contributing
------------
As an open source project, Instaloader heavily depends on the contributions from
its community. See
`contributing <https://instaloader.github.io/contributing.html>`__
for how you may help Instaloader to become an even greater tool.
Supporters
----------
.. current-sponsors-start
See `Alex' GitHub Sponsors <https://github.com/sponsors/aandergr>`__ page for
how you can sponsor the development of Instaloader!
.. current-sponsors-end
It is a pleasure for us to share our Instaloader to the world, and we are proud
to have attracted such an active and motivating community, with so many users
who share their suggestions and ideas with us. Buying a community-sponsored beer
or coffee from time to time is very likely to further raise our passion for the
development of Instaloader.
| For Donations, we provide GitHub Sponsors page, a PayPal.Me link and a Bitcoin address.
| GitHub Sponsors: `Sponsor @aandergr on GitHub Sponsors <https://github.com/sponsors/aandergr>`__
| PayPal: `PayPal.me/aandergr <https://www.paypal.me/aandergr>`__
| BTC: 1Nst4LoadeYzrKjJ1DX9CpbLXBYE9RKLwY
Disclaimer Disclaimer
---------- ----------
@@ -154,3 +114,21 @@ subsidiaries. This is an independent and unofficial project. Use at your own ris
Instaloader is licensed under an MIT license. Refer to ``LICENSE`` file for more information. Instaloader is licensed under an MIT license. Refer to ``LICENSE`` file for more information.
.. disclaimer-end .. disclaimer-end
Contributing
------------
As an open source project, Instaloader heavily depends on the contributions from
its community. See
`contributing <https://instaloader.github.io/contributing.html>`__
for how you may help Instaloader to become an even greater tool.
It is a pleasure for us to share our Instaloader to the world, and we are proud
to have attracted such an active and motivating community, with so many users
who share their suggestions and ideas with us. Buying a community-sponsored beer
or coffee from time to time is very likely to further raise our passion for the
development of Instaloader.
| For Donations, we provide a PayPal.Me link and a Bitcoin address.
| PayPal: `PayPal.me/aandergr <https://www.paypal.me/aandergr>`__
| BTC: 1Nst4LoadeYzrKjJ1DX9CpbLXBYE9RKLwY
+1 -1
View File
@@ -6,7 +6,7 @@ pkgbase = instaloader
arch = any arch = any
license = MIT license = MIT
makedepends = python-setuptools makedepends = python-setuptools
depends = python>=3.6 depends = python>=3.5
depends = python-requests>=2.4 depends = python-requests>=2.4
options = !emptydirs options = !emptydirs
source = instaloader-{{version}}.tar.gz::https://codeload.github.com/instaloader/instaloader/tar.gz/v{{version}} source = instaloader-{{version}}.tar.gz::https://codeload.github.com/instaloader/instaloader/tar.gz/v{{version}}
+1 -1
View File
@@ -8,7 +8,7 @@ arch=('any')
url="https://instaloader.github.io/" url="https://instaloader.github.io/"
license=('MIT') license=('MIT')
groups=() groups=()
depends=('python>=3.6' 'python-requests>=2.4') depends=('python>=3.5' 'python-requests>=2.4')
makedepends=('python-setuptools') makedepends=('python-setuptools')
options=('!emptydirs') options=('!emptydirs')
source=($pkgname-$pkgver.tar.gz::https://codeload.github.com/instaloader/instaloader/tar.gz/v$pkgver) source=($pkgname-$pkgver.tar.gz::https://codeload.github.com/instaloader/instaloader/tar.gz/v$pkgver)
+2 -2
View File
@@ -44,8 +44,8 @@ with open('__main__.py', 'w+') as f:
f.writelines(lines) f.writelines(lines)
# install dependencies and invoke PyInstaller # install dependencies and invoke PyInstaller
commands = ["pip install pipenv==2020.11.15", commands = ["pip install pipenv==2020.6.2",
f"pipenv --python {sys.version_info.major}.{sys.version_info.minor} sync --dev", "pipenv sync --dev",
"pipenv run pyinstaller --log-level=DEBUG instaloader.spec"] "pipenv run pyinstaller --log-level=DEBUG instaloader.spec"]
for command in commands: for command in commands:
+7 -34
View File
@@ -39,16 +39,6 @@ To later **update your local copy** of that profiles, you may run
If :option:`--fast-update` is given, Instaloader stops when arriving at the If :option:`--fast-update` is given, Instaloader stops when arriving at the
first already-downloaded picture. first already-downloaded picture.
Alternatively, you can use :option:`--latest-stamps` to have Instaloader store
the time each profile was last downloaded and only download newer media:
::
instaloader --latest-stamps -- profile [profile ...]
With this option it's possible to move or delete downloaded media and still keep
the archive updated.
When updating profiles, Instaloader When updating profiles, Instaloader
automatically **detects profile name changes** and renames the target directory automatically **detects profile name changes** and renames the target directory
accordingly. accordingly.
@@ -159,40 +149,27 @@ target. The default is ``--dirname-pattern={target}``. In the dirname
pattern, the token ``{target}`` is replaced by the target name, and pattern, the token ``{target}`` is replaced by the target name, and
``{profile}`` is replaced by the owner of the post which is downloaded. ``{profile}`` is replaced by the owner of the post which is downloaded.
:option:`--filename-pattern` configures the path of the post and story's files relative :option:`--filename-pattern` configures the path of the post's files relative
to the target directory that is specified with :option:`--dirname-pattern`. to the target directory that is specified with :option:`--dirname-pattern`.
The default is ``--filename-pattern={date_utc}_UTC``. The default is ``--filename-pattern={date_utc}_UTC``.
The tokens ``{target}`` and ``{profile}`` are replaced like in the The tokens ``{target}`` and ``{profile}`` are replaced like in the
dirname pattern. dirname pattern. The following tokens are defined for usage with
:option:`--filename-pattern`:
:option:`--title-pattern` is similar to :option:`--filename-pattern`, but for profile
pics, hashtag profile pics, and highlight covers. The default is
``{date_utc}_UTC_{typename}`` if :option:`--dirname-pattern` contains ``{target}`` or
``{profile}``, or ``{target}_{date_utc}_UTC_{typename}`` if it does not. Some tokens
are not supported for this option, see below for details.
The following tokens are defined for usage with
:option:`--filename-pattern` and :option:`--title-pattern`:
- ``{target}`` - ``{target}``
Target name (as given in Instaloader command line) Target name (as given in Instaloader command line)
- ``{profile}`` (same as ``{owner_username}``) - ``{profile}`` (same as ``{owner_username}``)
Owner of the Post / StoryItem / ProfilePic. For hashtag profile pics and Owner of the Post / StoryItem.
highlight covers, equivalent to ``{target}``.
- ``{owner_id}`` - ``{owner_id}``
Unique integer ID of owner profile. For hashtag profile pics, equivalent to Unique integer ID of owner profile.
``{target}``.
- ``{shortcode}`` - ``{shortcode}``
Shortcode (identifier string). Not available for :option:`--title-pattern`. Shortcode (identifier string).
- ``{mediaid}`` - ``{mediaid}``
Integer representation of shortcode. Not available for :option:`--title-pattern`. Integer representation of shortcode.
- ``{filename}``
Instagram's internal filename.
- ``{date_utc}`` (same as ``{date}``) - ``{date_utc}`` (same as ``{date}``)
Creation time in UTC timezone. Creation time in UTC timezone.
@@ -202,10 +179,6 @@ The following tokens are defined for usage with
{date_utc:%Y-%m-%d_%H-%M-%S} {date_utc:%Y-%m-%d_%H-%M-%S}
- ``{typename}``
Type of media being saved, such as GraphImage, GraphStoryVideo, profile_pic,
etc.
For example, encode the poster's profile name in the filenames with:: For example, encode the poster's profile name in the filenames with::
instaloader --filename-pattern={date_utc}_UTC_{profile} "#hashtag" instaloader --filename-pattern={date_utc}_UTC_{profile} "#hashtag"
+9 -74
View File
@@ -82,16 +82,6 @@ What to Download of each Post
Template to write in txt file for each StoryItem. See Template to write in txt file for each StoryItem. See
:ref:`metadata-text-files`. :ref:`metadata-text-files`.
.. option:: --slide
Download only selected images of a sidecar. You can select single images using their
index in the sidecar starting with the leftmost or you can specify a range of images
with the following syntax: ``start_index-end_index``. Example:
``--slide 1`` will select only the first image, ``--slide last`` only the last one and ``--slide 1-3`` will select only
the first three images.
.. versionadded:: 4.6
.. option:: --no-metadata-json .. option:: --no-metadata-json
Do not create a JSON file containing the metadata of each post. Do not create a JSON file containing the metadata of each post.
@@ -100,6 +90,7 @@ What to Download of each Post
Do not xz compress JSON files, rather create pretty formatted JSONs. Do not xz compress JSON files, rather create pretty formatted JSONs.
What to Download of each Profile What to Download of each Profile
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -146,23 +137,6 @@ Which Posts to Download
This flag is recommended when you use Instaloader to update your personal This flag is recommended when you use Instaloader to update your personal
Instagram archive. Instagram archive.
.. option:: --latest-stamps [STAMPSFILE]
Works similarly to :option:`--fast-update`, but instead of relying on already
downloaded media, the time each profile was downloaded is stored, and only
media newer than the last download is fetched. This allows updating your
personal Instagram archive while emptying the target directories.
Only works for media associated with a specific profile, and that is returned
in chronological order: profile posts, profile stories, profile IGTV posts
and profile tagged posts.
By default, the information is stored in
``~/.config/instaloader/latest-stamps.ini``, but you can specify an
alternative location.
.. versionadded:: 4.8
.. option:: --post-filter filter, --only-if filter .. option:: --post-filter filter, --only-if filter
Expression that, if given, must evaluate to True for each post to be Expression that, if given, must evaluate to True for each post to be
@@ -180,8 +154,8 @@ Which Posts to Download
.. option:: --count COUNT, -c .. option:: --count COUNT, -c
Do not attempt to download more than COUNT posts. Applies to Do not attempt to download more than COUNT posts. Applies only to
``#hashtag``, ``%location_id``, ``:feed``, and ``:saved``. ``#hashtag``, ``%location id``, and ``:feed``.
Login (Download Private Profiles) Login (Download Private Profiles)
@@ -222,23 +196,12 @@ How to Download
.. option:: --filename-pattern FILENAME_PATTERN .. option:: --filename-pattern FILENAME_PATTERN
Prefix of filenames for posts and stories, relative to the directory given with Prefix of filenames, relative to the directory given with
:option:`--dirname-pattern`. ``{profile}`` is replaced by the profile name, :option:`--dirname-pattern`. ``{profile}`` is replaced by the profile name,
``{target}`` is replaced by the target you specified, i.e. either ``:feed``, ``{target}`` is replaced by the target you specified, i.e. either ``:feed``,
``#hashtag`` or the profile name. Defaults to ``{date_utc}_UTC``. ``#hashtag`` or the profile name. Defaults to ``{date_utc}_UTC``.
See :ref:`filename-specification` for a list of supported tokens. See :ref:`filename-specification` for a list of supported tokens.
.. option:: --title-pattern TITLE_PATTERN
Prefix of filenames for profile pics, hashtag profile pics, and highlight
covers, relative to the directory given with :option:`--dirname-pattern`.
Defaults to ``{date_utc}_UTC_{typename}`` if :option:`--dirname-pattern`
contains ``{target}`` or ``{profile}``, otherwise defaults to
``{target}_{date_utc}_UTC_{typename}``.
See :ref:`filename-specification` for a list of supported tokens.
.. versionadded:: 4.8
.. option:: --resume-prefix prefix .. option:: --resume-prefix prefix
For many targets, Instaloader is capable of resuming a previously-aborted For many targets, Instaloader is capable of resuming a previously-aborted
@@ -247,20 +210,12 @@ How to Download
used to save the information to resume an interrupted download. The default used to save the information to resume an interrupted download. The default
prefix is ``iterator``. prefix is ``iterator``.
Resuming an interrupted download is supported for the following targets: Resuming an interrupted download is supported for most, but not all targets.
- Profile posts,
- Profile IGTV posts (:option:`--igtv`),
- Profile tagged posts (:option:`--tagged`),
- Saved posts (``:saved``).
This feature is enabled by default for targets where it is supported;
:option:`--resume-prefix` only changes the name of the iterator files.
To turn this feature off, use :option:`--no-resume`.
JSON files with resume information are always compressed, regardless of JSON files with resume information are always compressed, regardless of
:option:`--no-compress-json`. :option:`--no-compress-json`.
This feature is turned off entirely with :option:`--no-resume`.
.. versionadded:: 4.5 .. versionadded:: 4.5
.. option:: --no-resume .. option:: --no-resume
@@ -273,7 +228,7 @@ How to Download
.. option:: --user-agent USER_AGENT .. option:: --user-agent USER_AGENT
User Agent to use for HTTP requests. Per default, Instaloader pretends being User Agent to use for HTTP requests. Per default, Instaloader pretends being
Chrome/89 on Linux. Chrome/51.
.. option:: --max-connection-attempts N .. option:: --max-connection-attempts N
@@ -283,30 +238,10 @@ How to Download
.. option:: --request-timeout N .. option:: --request-timeout N
Seconds to wait before timing out a connection request. Defaults to 300. Seconds to wait before timing out a connection request.
.. versionadded:: 4.3 .. versionadded:: 4.3
.. versionchanged:: 4.6
Enabled this option by default with a timeout of 300 seconds.
.. option:: --abort-on STATUS_CODE_LIST
Comma-separated list of HTTP status codes that cause Instaloader to abort,
bypassing all retry logic.
For example, with ``--abort-on=302,400,429``, Instaloader will stop if a
request is responded with a 302 redirect, a Bad Request error, or a Too Many
Requests error.
.. versionadded:: 4.7
.. option:: --no-iphone
Do not attempt to download iPhone version of images and videos.
.. versionadded:: 4.8
Miscellaneous Options Miscellaneous Options
^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^
@@ -23,7 +23,7 @@ def get_cookiefile():
def import_session(cookiefile, sessionfile): def import_session(cookiefile, sessionfile):
print("Using cookies from {}.".format(cookiefile)) print("Using cookies from {}.".format(cookiefile))
conn = connect(f"file:{cookiefile}?immutable=1", uri=True) conn = connect(cookiefile)
try: try:
cookie_data = conn.execute( cookie_data = conn.execute(
"SELECT name, value FROM moz_cookies WHERE baseDomain='instagram.com'" "SELECT name, value FROM moz_cookies WHERE baseDomain='instagram.com'"
+1 -1
View File
@@ -65,7 +65,7 @@ extlinks = {
'Issue #' 'Issue #'
), ),
'example': ( 'example': (
'https://raw.githubusercontent.com/instaloader/instaloader/master/docs/codesnippets/%s', 'https://raw.githubusercontent.com/instaloader/instaloader/v' + current_release + '/docs/codesnippets/%s',
'Example ' 'Example '
), ),
} }
+31 -53
View File
@@ -11,23 +11,6 @@ Instaloader's development is organized on
`GitHub <https://github.com/instaloader/instaloader>`__, where Issues and Pull `GitHub <https://github.com/instaloader/instaloader>`__, where Issues and Pull
Requests are discussed. Requests are discussed.
Contributing to an open source project is an unmatchable opportunity to improve
and manifest programming skills. For issues that are good starting points to get
involved in Instaloader development, as they do not require deep insight into
Instaloader's code base, have a look at our
`Issues labelled 'good first issue' <https://github.com/instaloader/instaloader/contribute>`__.
This document covers several ways of how you can contribute back to Instaloader.
Answering Questions
-------------------
The easiest way to help out is to answer questions. If you are interested in
answering questions regarding Instaloader, good places to begin are
- `Questions tagged 'instaloader' on Stack Overflow <https://stackoverflow.com/questions/tagged/instaloader>`__,
- `Instaloader Issues labeled 'question' <https://github.com/instaloader/instaloader/issues?q=is%3Aissue+is%3Aopen+label%3Aquestion>`__.
Reporting Bugs Reporting Bugs
-------------- --------------
@@ -49,7 +32,7 @@ reporting a problem, please keep the following in mind:
- Include all **error messages and tracebacks** in the report. - Include all **error messages and tracebacks** in the report.
- Even if it seems obvious, describe **which behavior you expected** - If not obvious, describe **which behavior you expected**
instead of what actually happened. instead of what actually happened.
- If we have closed an issue apparently inadvertently or inappropriately, please - If we have closed an issue apparently inadvertently or inappropriately, please
@@ -58,39 +41,22 @@ reporting a problem, please keep the following in mind:
Writing Code or Improving the Documentation Writing Code or Improving the Documentation
------------------------------------------- -------------------------------------------
Improvements of the Instaloader source or its documentation can be proposed as a Changes of the Instaloader source can be proposed as a
`Pull Request <https://github.com/instaloader/instaloader/pulls>`__. `Pull Request <https://github.com/instaloader/instaloader/pulls>`__. There are only
few things to consider:
- Please base your Pull Request on - Base your Pull Request on the ``master`` branch if it fixes a bug,
or the ``upcoming/v4.X`` branch (if it exists at the moment of submitting the PR)
otherwise.
- ``master``, which will be released with the next minor release, if it is - We use `Pylint <https://www.pylint.org/>`__ for error and syntax checking of
the source and `MyPy <https://github.com/python/mypy>`__ for type checking.
Beware that sometimes it might be better to disable a warning rather than
adapting the code to a tool's desires.
- a bug fix that does not require extensive testing, - The documentation source is located in the ``docs`` folder. The file
- an improvement to the documentation, ``cli-options.rst`` is merely an RST-formatted copy of ``instaloader --help``
output, of which the source is in ``instaloader/__main__.py``.
- ``upcoming/v4.X``, if it is
- a new feature,
- a bug fix that does require thorough testing before being released to a
final version.
- All Pull Requests are analyzed by `Pylint <https://www.pylint.org/>`__ for
error and syntax checking of the source and
`Mypy <https://github.com/python/mypy>`__ for type checking. You can run them
locally with::
pylint instaloader
mypy -m instaloader
- Improvements to the documentation are very welcome. The documentation is
created with `Sphinx <https://www.sphinx-doc.org/en/2.0/>`__, version 2,
and can be build locally using::
make -C docs html
- Feel free to create an issue to make sure someone from the Instaloader team
agrees that the change might be an improvement, or if you want to discuss
your basic proposal, before working on a pull request.
Proposing Features Proposing Features
------------------ ------------------
@@ -132,9 +98,21 @@ details. If you feel the need to do so, please create a "feature suggestion".
be implemented and the underlying problem could be solved. Also **describe be implemented and the underlying problem could be solved. Also **describe
alternatives** that you have considered. alternatives** that you have considered.
Sponsoring Donations
---------- ---------
.. include:: sponsors.rst .. donations-start
:start-after: donations-start
:end-before: donations-end It is a pleasure for us to share our Instaloader to the world, and we are proud
to have attracted such an active and motivating community, with so many users
who share their suggestions and ideas with us. Buying a community-sponsored beer
or coffee from time to time is very likely to further raise our passion for the
development of Instaloader.
| For donations, we provide a PayPal.Me link and a Bitcoin address.
| `PayPal.me/aandergr <https://www.paypal.me/aandergr>`__
| BTC: 1Nst4LoadeYzrKjJ1DX9CpbLXBYE9RKLwY
.. donations-end
.. (Discussion in :issue:`130`)
+11 -14
View File
@@ -78,20 +78,6 @@ Useful Links
- `Issue Tracker / Bug Tracker <https://github.com/instaloader/instaloader/issues>`__ - `Issue Tracker / Bug Tracker <https://github.com/instaloader/instaloader/issues>`__
- `Version History <https://github.com/instaloader/instaloader/releases>`__ - `Version History <https://github.com/instaloader/instaloader/releases>`__
Contributing
------------
As an open source project, Instaloader heavily depends on the contributions from
its community. See :ref:`contributing` for how you may help Instaloader to
become an even greater tool.
Supporters
----------
.. include:: sponsors.rst
:start-after: donations-start
:end-before: donations-end
Disclaimer Disclaimer
---------- ----------
@@ -99,6 +85,17 @@ Disclaimer
:start-after: disclaimer-start :start-after: disclaimer-start
:end-before: disclaimer-end :end-before: disclaimer-end
Contributing
------------
As an open source project, Instaloader heavily depends on the contributions from
its community. See :ref:`contributing` for how you may help Instaloader to
become an even greater tool.
.. include:: contributing.rst
:start-after: donations-start
:end-before: donations-end
.. ..
* :ref:`genindex` * :ref:`genindex`
* :ref:`modindex` * :ref:`modindex`
+1 -4
View File
@@ -8,7 +8,7 @@ Install Instaloader
To **install Instaloader**, To **install Instaloader**,
#. Ensure that you have `Python <https://www.python.org/>`__, at least #. Ensure that you have `Python <https://www.python.org/>`__, at least
version 3.6, and `pip <https://pypi.python.org/pypi/pip>`__ version 3.5, and `pip <https://pypi.python.org/pypi/pip>`__
installed. installed.
#. Then, install Instaloader using:: #. Then, install Instaloader using::
@@ -33,9 +33,6 @@ To **upgrade Instaloader** to its current version, do::
- On **Windows 10**, you may download the standalone executable from the - On **Windows 10**, you may download the standalone executable from the
`current release page <https://github.com/instaloader/instaloader/releases/latest>`__. `current release page <https://github.com/instaloader/instaloader/releases/latest>`__.
- On **Android**, you can use Instaloader with `Termux <https://play.google.com/store/apps/details?id=com.termux>`__
after typing ``pkg install python`` and ``pip3 install instaloader``.
- To test the most current **pre-release** version of Instaloader:: - To test the most current **pre-release** version of Instaloader::
pip3 install --pre instaloader pip3 install --pre instaloader
-2
View File
@@ -47,5 +47,3 @@ Exceptions
.. autoexception:: TooManyRequestsException .. autoexception:: TooManyRequestsException
.. autoexception:: AbortDownloadException
-7
View File
@@ -25,13 +25,6 @@ Iterator :class:`NodeIterator` and a context manager
.. autoclass:: FrozenNodeIterator .. autoclass:: FrozenNodeIterator
:no-show-inheritance: :no-show-inheritance:
A serializable representation of a :class:`NodeIterator` instance, saving
its iteration state.
It can be serialized and deserialized with :func:`save_structure_to_file`
and :func:`load_structure_from_file`, as well as with :mod:`json` and
:mod:`pickle` thanks to being a :func:`~collections.namedtuple`.
``resumable_iteration`` ``resumable_iteration``
""""""""""""""""""""""" """""""""""""""""""""""
-10
View File
@@ -77,16 +77,6 @@ Loading and Saving
:class:`Post`, :class:`StoryItem`, :class:`Profile`, :class:`Hashtag` and :class:`Post`, :class:`StoryItem`, :class:`Profile`, :class:`Hashtag` and
:class:`FrozenNodeIterator` can be saved and loaded to/from JSON files. :class:`FrozenNodeIterator` can be saved and loaded to/from JSON files.
.. autofunction:: load_structure
.. autofunction:: load_structure_from_file .. autofunction:: load_structure_from_file
.. autofunction:: get_json_structure
.. autofunction:: save_structure_to_file .. autofunction:: save_structure_to_file
LatestStamps
""""""""""""
.. autoclass:: LatestStamps
:no-show-inheritance:
-23
View File
@@ -1,23 +0,0 @@
.. _sponsors:
Sponsoring the development of Instaloader
=========================================
.. donations-start
.. include:: ../README.rst
:start-after: current-sponsors-start
:end-before: current-sponsors-end
It is a pleasure for us to share our Instaloader to the world, and we are proud
to have attracted such an active and motivating community, with so many users
who share their suggestions and ideas with us. Buying a community-sponsored beer
or coffee from time to time is very likely to further raise our passion for the
development of Instaloader.
| For Donations, we provide GitHub Sponsors page, a PayPal.Me link and a Bitcoin address.
| GitHub Sponsors: `Sponsor @aandergr on GitHub Sponsors <https://github.com/sponsors/aandergr>`__
| PayPal: `PayPal.me/aandergr <https://www.paypal.me/aandergr>`__
| BTC: 1Nst4LoadeYzrKjJ1DX9CpbLXBYE9RKLwY
.. donations-end
+11 -19
View File
@@ -9,7 +9,10 @@ Troubleshooting
--------------------- ---------------------
Instaloader has a logic to keep track of its requests to Instagram and to obey Instaloader has a logic to keep track of its requests to Instagram and to obey
their rate limits. The rate controller assumes that their rate limits. Since they are nowhere documented, we try them out
experimentally. We have a daily cron job running to confirm that Instaloader
still stays within the rate limits. Nevertheless, the rate control logic assumes
that
- at one time, Instaloader is the only application that consumes requests, i.e. - at one time, Instaloader is the only application that consumes requests, i.e.
neither the Instagram browser interface, nor a mobile app, nor another neither the Instagram browser interface, nor a mobile app, nor another
@@ -18,13 +21,7 @@ their rate limits. The rate controller assumes that
- no requests had been consumed when Instaloader starts. - no requests had been consumed when Instaloader starts.
The latter one implies that restarting or reinstantiating Instaloader often The latter one implies that restarting or reinstantiating Instaloader often
within short time is prone to cause a 429. within short time is prone to cause a 429. If a request is denied with a 429,
Since the behavior of the rate controller might change between different
versions of Instaloader, make sure to use the current version of Instaloader,
especially when encountering many 429 errors.
If a request is denied with a 429,
Instaloader retries the request as soon as the temporary ban is assumed to be Instaloader retries the request as soon as the temporary ban is assumed to be
expired. In case the retry continuously fails for some reason, which should not expired. In case the retry continuously fails for some reason, which should not
happen under normal conditions, consider adjusting the happen under normal conditions, consider adjusting the
@@ -35,18 +32,13 @@ promiscuous IP addresses, such as cloud, VPN and public proxy services, might be
subject to significantly stricter limits for anonymous access. However, subject to significantly stricter limits for anonymous access. However,
logged-in accesses (see :option:`--login`) do not seem to be affected. logged-in accesses (see :option:`--login`) do not seem to be affected.
Instaloader allows to adjust the rate controlling behavior by overriding
:class:`instaloader.RateController`.
Too many queries in the last time Too many queries in the last time
--------------------------------- ---------------------------------
**"Too many queries in the last time"** is not an error. It is a notice that the **"Too many queries in the last time"** is not an error. It is a notice that the
rate limit has almost been reached, according to Instaloader's own rate rate limit has almost been reached, according to Instaloader's own rate
accounting mechanism. accounting mechanism. We regularly adjust this mechanism to match Instagram's
current rate limiting.
Instaloader allows to adjust the rate controlling behavior by overriding
:class:`instaloader.RateController`.
Private but not followed Private but not followed
------------------------ ------------------------
@@ -62,12 +54,12 @@ Two-Factor-Authentication. It also supports handling the *checkpoint challenge*,
issued when Instagram suspects authentication activity on your account, by issued when Instagram suspects authentication activity on your account, by
pointing the user to an URL to be opened in a browser. pointing the user to an URL to be opened in a browser.
Nevertheless, in :issue:`92`, :issue:`615`, :issue:`1150` and :issue:`1217`, Nevertheless, in :issue:`92` and :issue:`615` users reported problems with
users reported problems with
logging in. We recommend to always keep the session file which Instaloader logging in. We recommend to always keep the session file which Instaloader
creates when using :option:`--login`. If a session file is present, creates when using :option:`--login`. If a session file is present,
:option:`--login` does not make use of the failure-prone login procedure. :option:`--login` does not make make use of the failure-prone login procedure.
Also, session files usually do not expire. Also, session files usually do not expire and can be copied between different
computers without any problems.
If you do not have a session file present, you may use the following script If you do not have a session file present, you may use the following script
(:example:`615_import_firefox_session.py`) to workaround login problems by (:example:`615_import_firefox_session.py`) to workaround login problems by
+2 -4
View File
@@ -1,7 +1,7 @@
"""Download pictures (or videos) along with their captions and other metadata from Instagram.""" """Download pictures (or videos) along with their captions and other metadata from Instagram."""
__version__ = '4.8.2' __version__ = '4.5'
try: try:
@@ -15,8 +15,6 @@ else:
from .exceptions import * from .exceptions import *
from .instaloader import Instaloader from .instaloader import Instaloader
from .instaloadercontext import InstaloaderContext, RateController from .instaloadercontext import InstaloaderContext, RateController
from .lateststamps import LatestStamps
from .nodeiterator import NodeIterator, FrozenNodeIterator, resumable_iteration from .nodeiterator import NodeIterator, FrozenNodeIterator, resumable_iteration
from .structures import (Hashtag, Highlight, Post, PostSidecarNode, PostComment, PostCommentAnswer, PostLocation, from .structures import (Hashtag, Highlight, Post, PostSidecarNode, PostComment, PostCommentAnswer, PostLocation,
Profile, Story, StoryItem, TopSearchResults, load_structure_from_file, save_structure_to_file, Profile, Story, StoryItem, TopSearchResults, load_structure_from_file, save_structure_to_file)
load_structure, get_json_structure)
+22 -61
View File
@@ -5,15 +5,14 @@ import datetime
import os import os
import re import re
import sys import sys
from argparse import ArgumentParser, ArgumentTypeError, SUPPRESS from argparse import ArgumentParser, SUPPRESS
from typing import List, Optional from typing import List, Optional
from . import (AbortDownloadException, BadCredentialsException, Instaloader, InstaloaderException, from . import (Instaloader, InstaloaderException, InvalidArgumentException, Post, Profile, ProfileNotExistsException,
InvalidArgumentException, Post, Profile, ProfileNotExistsException, StoryItem, StoryItem, __version__, load_structure_from_file, TwoFactorAuthRequiredException,
TwoFactorAuthRequiredException, __version__, load_structure_from_file) BadCredentialsException)
from .instaloader import (get_default_session_filename, get_default_stamps_filename) from .instaloader import get_default_session_filename
from .instaloadercontext import default_user_agent from .instaloadercontext import default_user_agent
from .lateststamps import LatestStamps
def usage_string(): def usage_string():
@@ -28,14 +27,6 @@ def usage_string():
{0} --help""".format(argv0, len(argv0), '') {0} --help""".format(argv0, len(argv0), '')
def http_status_code_list(code_list_str: str) -> List[int]:
codes = [int(s) for s in code_list_str.split(',')]
for code in codes:
if not 100 <= code <= 599:
raise ArgumentTypeError("Invalid HTTP status code: {}".format(code))
return codes
def filterstr_to_filterfunc(filter_str: str, item_type: type): def filterstr_to_filterfunc(filter_str: str, item_type: type):
"""Takes an --post-filter=... or --storyitem-filter=... filter """Takes an --post-filter=... or --storyitem-filter=... filter
specification and makes a filter_func Callable out of it.""" specification and makes a filter_func Callable out of it."""
@@ -77,7 +68,6 @@ def _main(instaloader: Instaloader, targetlist: List[str],
download_tagged: bool = False, download_tagged: bool = False,
download_igtv: bool = False, download_igtv: bool = False,
fast_update: bool = False, fast_update: bool = False,
latest_stamps_file: Optional[str] = None,
max_count: Optional[int] = None, post_filter_str: Optional[str] = None, max_count: Optional[int] = None, post_filter_str: Optional[str] = None,
storyitem_filter_str: Optional[str] = None) -> None: storyitem_filter_str: Optional[str] = None) -> None:
"""Download set of profiles, hashtags etc. and handle logging in and session files if desired.""" """Download set of profiles, hashtags etc. and handle logging in and session files if desired."""
@@ -90,10 +80,6 @@ def _main(instaloader: Instaloader, targetlist: List[str],
if storyitem_filter_str is not None: if storyitem_filter_str is not None:
storyitem_filter = filterstr_to_filterfunc(storyitem_filter_str, StoryItem) storyitem_filter = filterstr_to_filterfunc(storyitem_filter_str, StoryItem)
instaloader.context.log('Only download storyitems with property "{}".'.format(storyitem_filter_str)) instaloader.context.log('Only download storyitems with property "{}".'.format(storyitem_filter_str))
latest_stamps = None
if latest_stamps_file is not None:
latest_stamps = LatestStamps(latest_stamps_file)
instaloader.context.log(f"Using latest stamps from {latest_stamps_file}.")
# Login, if desired # Login, if desired
if username is not None: if username is not None:
if not re.match(r"^[A-Za-z0-9._]+$", username): if not re.match(r"^[A-Za-z0-9._]+$", username):
@@ -114,8 +100,7 @@ def _main(instaloader: Instaloader, targetlist: List[str],
code = input("Enter 2FA verification code: ") code = input("Enter 2FA verification code: ")
instaloader.two_factor_login(code) instaloader.two_factor_login(code)
break break
except BadCredentialsException as err: except BadCredentialsException:
print(err, file=sys.stderr)
pass pass
else: else:
instaloader.interactive_login(username) instaloader.interactive_login(username)
@@ -179,7 +164,7 @@ def _main(instaloader: Instaloader, targetlist: List[str],
post_filter=post_filter) post_filter=post_filter)
elif re.match(r"^[A-Za-z0-9._]+$", target): elif re.match(r"^[A-Za-z0-9._]+$", target):
try: try:
profile = instaloader.check_profile_id(target, latest_stamps) profile = instaloader.check_profile_id(target)
if instaloader.context.is_logged_in and profile.has_blocked_viewer: if instaloader.context.is_logged_in and profile.has_blocked_viewer:
if download_profile_pic or ((download_posts or download_tagged or download_igtv) if download_profile_pic or ((download_posts or download_tagged or download_igtv)
and not profile.is_private): and not profile.is_private):
@@ -198,8 +183,7 @@ def _main(instaloader: Instaloader, targetlist: List[str],
instaloader.context.log("Trying again anonymously, helps in case you are just blocked.") instaloader.context.log("Trying again anonymously, helps in case you are just blocked.")
with instaloader.anonymous_copy() as anonymous_loader: with instaloader.anonymous_copy() as anonymous_loader:
with instaloader.context.error_catcher(): with instaloader.context.error_catcher():
anonymous_retry_profiles.add(anonymous_loader.check_profile_id(target, anonymous_retry_profiles.add(anonymous_loader.check_profile_id(target))
latest_stamps))
instaloader.context.error("Warning: {} will be downloaded anonymously (\"{}\")." instaloader.context.error("Warning: {} will be downloaded anonymously (\"{}\")."
.format(target, err)) .format(target, err))
else: else:
@@ -214,25 +198,21 @@ def _main(instaloader: Instaloader, targetlist: List[str],
if len(profiles) > 1: if len(profiles) > 1:
instaloader.context.log("Downloading {} profiles: {}".format(len(profiles), instaloader.context.log("Downloading {} profiles: {}".format(len(profiles),
' '.join([p.username for p in profiles]))) ' '.join([p.username for p in profiles])))
if instaloader.context.iphone_support and profiles and (download_profile_pic or download_posts) and \ if profiles and (download_profile_pic or download_posts) and not instaloader.context.is_logged_in:
not instaloader.context.is_logged_in: instaloader.context.error("Warning: Use --login to download higher-quality versions of pictures.")
instaloader.context.log("Hint: Use --login to download higher-quality versions of pictures.")
instaloader.download_profiles(profiles, instaloader.download_profiles(profiles,
download_profile_pic, download_posts, download_tagged, download_igtv, download_profile_pic, download_posts, download_tagged, download_igtv,
download_highlights, download_stories, download_highlights, download_stories,
fast_update, post_filter, storyitem_filter, latest_stamps=latest_stamps) fast_update, post_filter, storyitem_filter)
if anonymous_retry_profiles: if anonymous_retry_profiles:
instaloader.context.log("Downloading anonymously: {}" instaloader.context.log("Downloading anonymously: {}"
.format(' '.join([p.username for p in anonymous_retry_profiles]))) .format(' '.join([p.username for p in anonymous_retry_profiles])))
with instaloader.anonymous_copy() as anonymous_loader: with instaloader.anonymous_copy() as anonymous_loader:
anonymous_loader.download_profiles(anonymous_retry_profiles, anonymous_loader.download_profiles(anonymous_retry_profiles,
download_profile_pic, download_posts, download_tagged, download_igtv, download_profile_pic, download_posts, download_tagged, download_igtv,
fast_update=fast_update, post_filter=post_filter, fast_update=fast_update, post_filter=post_filter)
latest_stamps=latest_stamps)
except KeyboardInterrupt: except KeyboardInterrupt:
print("\nInterrupted by user.", file=sys.stderr) print("\nInterrupted by user.", file=sys.stderr)
except AbortDownloadException as exc:
print("\nDownload aborted: {}.".format(exc), file=sys.stderr)
# Save session if it is useful # Save session if it is useful
if instaloader.context.is_logged_in: if instaloader.context.is_logged_in:
instaloader.save_session_to_file(sessionfile) instaloader.save_session_to_file(sessionfile)
@@ -242,13 +222,14 @@ def _main(instaloader: Instaloader, targetlist: List[str],
# Instaloader did at least save a session file # Instaloader did at least save a session file
instaloader.context.log("No targets were specified, thus nothing has been downloaded.") instaloader.context.log("No targets were specified, thus nothing has been downloaded.")
else: else:
# Instaloader did not do anything # Instloader did not do anything
instaloader.context.log("usage:" + usage_string()) instaloader.context.log("usage:" + usage_string())
def main(): def main():
parser = ArgumentParser(description=__doc__, add_help=False, usage=usage_string(), parser = ArgumentParser(description=__doc__, add_help=False, usage=usage_string(),
epilog="The complete documentation can be found at " epilog="Report issues at https://github.com/instaloader/instaloader/issues. "
"The complete documentation can be found at "
"https://instaloader.github.io/.", "https://instaloader.github.io/.",
fromfile_prefix_chars='+') fromfile_prefix_chars='+')
@@ -287,8 +268,6 @@ def main():
help="Do not download regular posts.") help="Do not download regular posts.")
g_prof.add_argument('--no-profile-pic', action='store_true', g_prof.add_argument('--no-profile-pic', action='store_true',
help='Do not download profile picture.') help='Do not download profile picture.')
g_post.add_argument('--slide', action='store',
help='Set what image/interval of a sidecar you want to download.')
g_post.add_argument('--no-pictures', action='store_true', g_post.add_argument('--no-pictures', action='store_true',
help='Do not download post pictures. Cannot be used together with --fast-update. ' help='Do not download post pictures. Cannot be used together with --fast-update. '
'Implies --no-video-thumbnails, does not imply --no-videos.') 'Implies --no-video-thumbnails, does not imply --no-videos.')
@@ -333,10 +312,7 @@ def main():
g_cond.add_argument('-F', '--fast-update', action='store_true', g_cond.add_argument('-F', '--fast-update', action='store_true',
help='For each target, stop when encountering the first already-downloaded picture. This ' help='For each target, stop when encountering the first already-downloaded picture. This '
'flag is recommended when you use Instaloader to update your personal Instagram archive.') 'flag is recommended when you use Instaloader to update your personal Instagram archive.')
g_cond.add_argument('--latest-stamps', nargs='?', metavar='STAMPSFILE', const=get_default_stamps_filename(),
help='Store the timestamps of latest media scraped for each profile. This allows updating '
'your personal Instagram archive even if you delete the destination directories. '
'If STAMPSFILE is not provided, defaults to ' + get_default_stamps_filename())
g_cond.add_argument('--post-filter', '--only-if', metavar='filter', g_cond.add_argument('--post-filter', '--only-if', metavar='filter',
help='Expression that, if given, must evaluate to True for each post to be downloaded. Must be ' help='Expression that, if given, must evaluate to True for each post to be downloaded. Must be '
'a syntactically valid python expression. Variables are evaluated to ' 'a syntactically valid python expression. Variables are evaluated to '
@@ -348,7 +324,7 @@ def main():
g_cond.add_argument('-c', '--count', g_cond.add_argument('-c', '--count',
help='Do not attempt to download more than COUNT posts. ' help='Do not attempt to download more than COUNT posts. '
'Applies to #hashtag, %%location_id, :feed, and :saved.') 'Applies only to #hashtag and :feed.')
g_login = parser.add_argument_group('Login (Download Private Profiles)', g_login = parser.add_argument_group('Login (Download Private Profiles)',
'Instaloader can login to Instagram. This allows downloading private profiles. ' 'Instaloader can login to Instagram. This allows downloading private profiles. '
@@ -371,15 +347,10 @@ def main():
'{target} is replaced by the target you specified, i.e. either :feed, #hashtag or the ' '{target} is replaced by the target you specified, i.e. either :feed, #hashtag or the '
'profile name. Defaults to \'{target}\'.') 'profile name. Defaults to \'{target}\'.')
g_how.add_argument('--filename-pattern', g_how.add_argument('--filename-pattern',
help='Prefix of filenames for posts and stories, relative to the directory given with ' help='Prefix of filenames, relative to the directory given with '
'--dirname-pattern. {profile} is replaced by the profile name,' '--dirname-pattern. {profile} is replaced by the profile name,'
'{target} is replaced by the target you specified, i.e. either :feed' '{target} is replaced by the target you specified, i.e. either :feed'
'#hashtag or the profile name. Defaults to \'{date_utc}_UTC\'') '#hashtag or the profile name. Defaults to \'{date_utc}_UTC\'')
g_how.add_argument('--title-pattern',
help='Prefix of filenames for profile pics, hashtag profile pics, and highlight covers. '
'Defaults to \'{date_utc}_UTC_{typename}\' if --dirname-pattern contains \'{target}\' '
'or \'{dirname}\', or if --dirname-pattern is not specified. Otherwise defaults to '
'\'{target}_{date_utc}_UTC_{typename}\'.')
g_how.add_argument('--resume-prefix', metavar='PREFIX', g_how.add_argument('--resume-prefix', metavar='PREFIX',
help='Prefix for filenames that are used to save the information to resume an interrupted ' help='Prefix for filenames that are used to save the information to resume an interrupted '
'download.') 'download.')
@@ -395,13 +366,8 @@ def main():
'connection fails, it can be manually skipped by hitting CTRL+C. Set this to 0 to retry ' 'connection fails, it can be manually skipped by hitting CTRL+C. Set this to 0 to retry '
'infinitely.') 'infinitely.')
g_how.add_argument('--commit-mode', action='store_true', help=SUPPRESS) g_how.add_argument('--commit-mode', action='store_true', help=SUPPRESS)
g_how.add_argument('--request-timeout', metavar='N', type=float, default=300.0, g_how.add_argument('--request-timeout', metavar='N', type=float,
help='Seconds to wait before timing out a connection request. Defaults to 300.') help='seconds to wait before timing out a connection request')
g_how.add_argument('--abort-on', type=http_status_code_list, metavar="STATUS_CODES",
help='Comma-separated list of HTTP status codes that cause Instaloader to abort, bypassing all '
'retry logic.')
g_how.add_argument('--no-iphone', action='store_true',
help='Do not attempt to download iPhone version of images and videos.')
g_misc = parser.add_argument_group('Miscellaneous Options') g_misc = parser.add_argument_group('Miscellaneous Options')
g_misc.add_argument('-q', '--quiet', action='store_true', g_misc.add_argument('-q', '--quiet', action='store_true',
@@ -459,11 +425,7 @@ def main():
max_connection_attempts=args.max_connection_attempts, max_connection_attempts=args.max_connection_attempts,
request_timeout=args.request_timeout, request_timeout=args.request_timeout,
resume_prefix=resume_prefix, resume_prefix=resume_prefix,
check_resume_bbd=not args.use_aged_resume_files, check_resume_bbd=not args.use_aged_resume_files)
slide=args.slide,
fatal_status_codes=args.abort_on,
iphone_support=not args.no_iphone,
title_pattern=args.title_pattern)
_main(loader, _main(loader,
args.profile, args.profile,
username=args.login.lower() if args.login is not None else None, username=args.login.lower() if args.login is not None else None,
@@ -476,13 +438,12 @@ def main():
download_tagged=args.tagged, download_tagged=args.tagged,
download_igtv=args.igtv, download_igtv=args.igtv,
fast_update=args.fast_update, fast_update=args.fast_update,
latest_stamps_file=args.latest_stamps,
max_count=int(args.count) if args.count is not None else None, max_count=int(args.count) if args.count is not None else None,
post_filter_str=args.post_filter, post_filter_str=args.post_filter,
storyitem_filter_str=args.storyitem_filter) storyitem_filter_str=args.storyitem_filter)
loader.close() loader.close()
except InstaloaderException as err: except InstaloaderException as err:
raise SystemExit("Fatal error: %s" % err) from err raise SystemExit("Fatal error: %s" % err)
if __name__ == "__main__": if __name__ == "__main__":
-14
View File
@@ -64,17 +64,3 @@ class QueryReturnedNotFoundException(ConnectionException):
class TooManyRequestsException(ConnectionException): class TooManyRequestsException(ConnectionException):
pass pass
class IPhoneSupportDisabledException(InstaloaderException):
pass
class AbortDownloadException(Exception):
"""
Exception that is not catched in the error catchers inside the download loop and so aborts the
download loop.
This exception is not a subclass of ``InstaloaderException``.
.. versionadded:: 4.7
"""
pass
+121 -398
View File
@@ -10,39 +10,33 @@ import tempfile
from contextlib import contextmanager, suppress from contextlib import contextmanager, suppress
from datetime import datetime, timezone from datetime import datetime, timezone
from functools import wraps from functools import wraps
from hashlib import md5
from io import BytesIO from io import BytesIO
from pathlib import Path from pathlib import Path
from typing import Any, Callable, IO, Iterator, List, Optional, Set, Union, cast from typing import Any, Callable, ContextManager, IO, Iterator, List, Optional, Set, Union, cast
from urllib.parse import urlparse
import requests import requests
import urllib3 # type: ignore import urllib3 # type: ignore
from .exceptions import * from .exceptions import *
from .instaloadercontext import InstaloaderContext, RateController from .instaloadercontext import InstaloaderContext, RateController
from .lateststamps import LatestStamps
from .nodeiterator import NodeIterator, resumable_iteration from .nodeiterator import NodeIterator, resumable_iteration
from .structures import (Hashtag, Highlight, JsonExportable, Post, PostLocation, Profile, Story, StoryItem, from .structures import (Hashtag, Highlight, JsonExportable, Post, PostLocation, Profile, Story, StoryItem,
load_structure_from_file, save_structure_to_file, PostSidecarNode, TitlePic) load_structure_from_file, save_structure_to_file)
def _get_config_dir() -> str:
if platform.system() == "Windows":
# on Windows, use %LOCALAPPDATA%\Instaloader
localappdata = os.getenv("LOCALAPPDATA")
if localappdata is not None:
return os.path.join(localappdata, "Instaloader")
# legacy fallback - store in temp dir if %LOCALAPPDATA% is not set
return os.path.join(tempfile.gettempdir(), ".instaloader-" + getpass.getuser())
# on Unix, use ~/.config/instaloader
return os.path.join(os.getenv("XDG_CONFIG_HOME", os.path.expanduser("~/.config")), "instaloader")
def get_default_session_filename(username: str) -> str: def get_default_session_filename(username: str) -> str:
"""Returns default session filename for given username.""" """Returns default session filename for given username."""
configdir = _get_config_dir()
sessionfilename = "session-{}".format(username) sessionfilename = "session-{}".format(username)
return os.path.join(configdir, sessionfilename) if platform.system() == "Windows":
# on Windows, use %LOCALAPPDATA%\Instaloader\session-USERNAME
localappdata = os.getenv("LOCALAPPDATA")
if localappdata is not None:
return os.path.join(localappdata, "Instaloader", sessionfilename)
# legacy fallback - store in temp dir if %LOCALAPPDATA% is not set
return os.path.join(tempfile.gettempdir(), ".instaloader-" + getpass.getuser(), sessionfilename)
# on Unix, use ~/.config/instaloader/session-USERNAME
return os.path.join(os.getenv("XDG_CONFIG_HOME", os.path.expanduser("~/.config")), "instaloader", sessionfilename)
def get_legacy_session_filename(username: str) -> str: def get_legacy_session_filename(username: str) -> str:
@@ -52,17 +46,6 @@ def get_legacy_session_filename(username: str) -> str:
return filename.lower() return filename.lower()
def get_default_stamps_filename() -> str:
"""
Returns default filename for latest stamps database.
.. versionadded:: 4.8
"""
configdir = _get_config_dir()
return os.path.join(configdir, "latest-stamps.ini")
def format_string_contains_key(format_string: str, key: str) -> bool: def format_string_contains_key(format_string: str, key: str) -> bool:
# pylint:disable=unused-variable # pylint:disable=unused-variable
for literal_text, field_name, format_spec, conversion in string.Formatter().parse(format_string): for literal_text, field_name, format_spec, conversion in string.Formatter().parse(format_string):
@@ -118,8 +101,6 @@ class _ArbitraryItemFormatter(string.Formatter):
def get_value(self, key, args, kwargs): def get_value(self, key, args, kwargs):
"""Override to substitute {ATTRIBUTE} by attributes of our _item.""" """Override to substitute {ATTRIBUTE} by attributes of our _item."""
if key == 'filename' and isinstance(self._item, (Post, StoryItem, PostSidecarNode, TitlePic)):
return "{filename}"
if hasattr(self._item, key): if hasattr(self._item, key):
return getattr(self._item, key) return getattr(self._item, key)
return super().get_value(key, args, kwargs) return super().get_value(key, args, kwargs)
@@ -160,9 +141,6 @@ class Instaloader:
:param user_agent: :option:`--user-agent` :param user_agent: :option:`--user-agent`
:param dirname_pattern: :option:`--dirname-pattern`, default is ``{target}`` :param dirname_pattern: :option:`--dirname-pattern`, default is ``{target}``
:param filename_pattern: :option:`--filename-pattern`, default is ``{date_utc}_UTC`` :param filename_pattern: :option:`--filename-pattern`, default is ``{date_utc}_UTC``
:param title_pattern:
:option:`--title-pattern`, default is ``{date_utc}_UTC_{typename}`` if ``dirname_pattern`` contains
``{target}`` or ``{profile}``, ``{target}_{date_utc}_UTC_{typename}`` otherwise.
:param download_pictures: not :option:`--no-pictures` :param download_pictures: not :option:`--no-pictures`
:param download_videos: not :option:`--no-videos` :param download_videos: not :option:`--no-videos`
:param download_video_thumbnails: not :option:`--no-video-thumbnails` :param download_video_thumbnails: not :option:`--no-video-thumbnails`
@@ -179,9 +157,6 @@ class Instaloader:
:param rate_controller: Generator for a :class:`RateController` to override rate controlling behavior :param rate_controller: Generator for a :class:`RateController` to override rate controlling behavior
:param resume_prefix: :option:`--resume-prefix`, or None for :option:`--no-resume`. :param resume_prefix: :option:`--resume-prefix`, or None for :option:`--no-resume`.
:param check_resume_bbd: Whether to check the date of expiry of resume files and reject them if expired. :param check_resume_bbd: Whether to check the date of expiry of resume files and reject them if expired.
:param slide: :option:`--slide`
:param fatal_status_codes: :option:`--abort-on`
:param iphone_support: not :option:`--no-iphone`
.. attribute:: context .. attribute:: context
@@ -197,37 +172,24 @@ class Instaloader:
download_pictures=True, download_pictures=True,
download_videos: bool = True, download_videos: bool = True,
download_video_thumbnails: bool = True, download_video_thumbnails: bool = True,
download_geotags: bool = False, download_geotags: bool = True,
download_comments: bool = False, download_comments: bool = True,
save_metadata: bool = True, save_metadata: bool = True,
compress_json: bool = True, compress_json: bool = True,
post_metadata_txt_pattern: str = None, post_metadata_txt_pattern: str = None,
storyitem_metadata_txt_pattern: str = None, storyitem_metadata_txt_pattern: str = None,
max_connection_attempts: int = 3, max_connection_attempts: int = 3,
request_timeout: float = 300.0, request_timeout: Optional[float] = None,
rate_controller: Optional[Callable[[InstaloaderContext], RateController]] = None, rate_controller: Optional[Callable[[InstaloaderContext], RateController]] = None,
resume_prefix: Optional[str] = "iterator", resume_prefix: Optional[str] = "iterator",
check_resume_bbd: bool = True, check_resume_bbd: bool = True):
slide: Optional[str] = None,
fatal_status_codes: Optional[List[int]] = None,
iphone_support: bool = True,
title_pattern: Optional[str] = None):
self.context = InstaloaderContext(sleep, quiet, user_agent, max_connection_attempts, self.context = InstaloaderContext(sleep, quiet, user_agent, max_connection_attempts,
request_timeout, rate_controller, fatal_status_codes, request_timeout, rate_controller)
iphone_support)
# configuration parameters # configuration parameters
self.dirname_pattern = dirname_pattern or "{target}" self.dirname_pattern = dirname_pattern or "{target}"
self.filename_pattern = filename_pattern or "{date_utc}_UTC" self.filename_pattern = filename_pattern or "{date_utc}_UTC"
if title_pattern is not None:
self.title_pattern = title_pattern
else:
if (format_string_contains_key(self.dirname_pattern, 'profile') or
format_string_contains_key(self.dirname_pattern, 'target')):
self.title_pattern = '{date_utc}_UTC_{typename}'
else:
self.title_pattern = '{target}_{date_utc}_UTC_{typename}'
self.download_pictures = download_pictures self.download_pictures = download_pictures
self.download_videos = download_videos self.download_videos = download_videos
self.download_video_thumbnails = download_video_thumbnails self.download_video_thumbnails = download_video_thumbnails
@@ -242,31 +204,6 @@ class Instaloader:
self.resume_prefix = resume_prefix self.resume_prefix = resume_prefix
self.check_resume_bbd = check_resume_bbd self.check_resume_bbd = check_resume_bbd
self.slide = slide or ""
self.slide_start = 0
self.slide_end = -1
if self.slide != "":
splitted = self.slide.split('-')
if len(splitted) == 1:
if splitted[0] == 'last':
# download only last image of a sidecar
self.slide_start = -1
else:
if int(splitted[0]) > 0:
self.slide_start = self.slide_end = int(splitted[0])-1
else:
raise InvalidArgumentException("--slide parameter must be greater than 0.")
elif len(splitted) == 2:
if splitted[1] == 'last':
self.slide_start = int(splitted[0])-1
elif 0 < int(splitted[0]) < int(splitted[1]):
self.slide_start = int(splitted[0])-1
self.slide_end = int(splitted[1])-1
else:
raise InvalidArgumentException("Invalid data for --slide parameter.")
else:
raise InvalidArgumentException("Invalid data for --slide parameter.")
@contextmanager @contextmanager
def anonymous_copy(self): def anonymous_copy(self):
"""Yield an anonymous, otherwise equally-configured copy of an Instaloader instance; Then copy its error log.""" """Yield an anonymous, otherwise equally-configured copy of an Instaloader instance; Then copy its error log."""
@@ -288,10 +225,7 @@ class Instaloader:
max_connection_attempts=self.context.max_connection_attempts, max_connection_attempts=self.context.max_connection_attempts,
request_timeout=self.context.request_timeout, request_timeout=self.context.request_timeout,
resume_prefix=self.resume_prefix, resume_prefix=self.resume_prefix,
check_resume_bbd=self.check_resume_bbd, check_resume_bbd=self.check_resume_bbd)
slide=self.slide,
fatal_status_codes=self.context.fatal_status_codes,
iphone_support=self.context.iphone_support)
yield new_loader yield new_loader
self.context.error_log.extend(new_loader.context.error_log) self.context.error_log.extend(new_loader.context.error_log)
new_loader.context.error_log = [] # avoid double-printing of errors new_loader.context.error_log = [] # avoid double-printing of errors
@@ -367,47 +301,18 @@ class Instaloader:
combined_answers.extend(y['answers']) combined_answers.extend(y['answers'])
unique_comments_list[-1]['answers'] = get_unique_comments(combined_answers) unique_comments_list[-1]['answers'] = get_unique_comments(combined_answers)
return unique_comments_list return unique_comments_list
def get_new_comments(new_comments, start):
for idx, comment in enumerate(new_comments, start=start+1):
if idx % 250 == 0:
self.context.log('{}'.format(idx), end='…', flush=True)
yield comment
def save_comments(extended_comments):
unique_comments = get_unique_comments(extended_comments, combine_answers=True)
answer_ids = set(int(answer['id']) for comment in unique_comments for answer in comment.get('answers', []))
with open(filename, 'w') as file:
file.write(json.dumps(list(filter(lambda t: int(t['id']) not in answer_ids, unique_comments)),
indent=4))
base_filename = filename
filename += '_comments.json' filename += '_comments.json'
try: try:
with open(filename) as fp: with open(filename) as fp:
comments = json.load(fp) comments = json.load(fp)
except (FileNotFoundError, json.decoder.JSONDecodeError): except (FileNotFoundError, json.decoder.JSONDecodeError):
comments = list() comments = list()
comments.extend(_postcomment_asdict(comment) for comment in post.get_comments())
comments_iterator = post.get_comments()
try:
with resumable_iteration(
context=self.context,
iterator=comments_iterator,
load=load_structure_from_file,
save=save_structure_to_file,
format_path=lambda magic: "{}_{}_{}.json.xz".format(base_filename, self.resume_prefix, magic),
check_bbd=self.check_resume_bbd,
enabled=self.resume_prefix is not None
) as (_is_resuming, start_index):
comments.extend(_postcomment_asdict(comment)
for comment in get_new_comments(comments_iterator, start_index))
except (KeyboardInterrupt, AbortDownloadException):
if comments:
save_comments(comments)
raise
if comments: if comments:
save_comments(comments) comments = get_unique_comments(comments, combine_answers=True)
answer_ids = set(int(answer['id']) for comment in comments for answer in comment.get('answers', []))
with open(filename, 'w') as file:
file.write(json.dumps(list(filter(lambda t: int(t['id']) not in answer_ids, comments)), indent=4))
self.context.log('comments', end=' ', flush=True) self.context.log('comments', end=' ', flush=True)
def save_caption(self, filename: str, mtime: datetime, caption: str) -> None: def save_caption(self, filename: str, mtime: datetime, caption: str) -> None:
@@ -453,12 +358,9 @@ class Instaloader:
def save_location(self, filename: str, location: PostLocation, mtime: datetime) -> None: def save_location(self, filename: str, location: PostLocation, mtime: datetime) -> None:
"""Save post location name and Google Maps link.""" """Save post location name and Google Maps link."""
filename += '_location.txt' filename += '_location.txt'
if location.lat is not None and location.lng is not None: location_string = (location.name + "\n" +
location_string = (location.name + "\n" + "https://maps.google.com/maps?q={0},{1}&ll={0},{1}\n".format(location.lat,
"https://maps.google.com/maps?q={0},{1}&ll={0},{1}\n".format(location.lat, location.lng))
location.lng))
else:
location_string = location.name
with open(filename, 'wb') as text_file: with open(filename, 'wb') as text_file:
with BytesIO(location_string.encode()) as bio: with BytesIO(location_string.encode()) as bio:
shutil.copyfileobj(cast(IO, bio), text_file) shutil.copyfileobj(cast(IO, bio), text_file)
@@ -491,51 +393,31 @@ class Instaloader:
.. versionadded:: 4.3""" .. versionadded:: 4.3"""
def _epoch_to_string(epoch: datetime) -> str:
return epoch.strftime('%Y-%m-%d_%H-%M-%S_UTC')
http_response = self.context.get_raw(url) http_response = self.context.get_raw(url)
date_object = None # type: Optional[datetime] date_object = None # type: Optional[datetime]
if 'Last-Modified' in http_response.headers: if 'Last-Modified' in http_response.headers:
date_object = datetime.strptime(http_response.headers["Last-Modified"], '%a, %d %b %Y %H:%M:%S GMT') date_object = datetime.strptime(http_response.headers["Last-Modified"], '%a, %d %b %Y %H:%M:%S GMT')
date_object = date_object.replace(tzinfo=timezone.utc)
pic_bytes = None pic_bytes = None
pic_identifier = _epoch_to_string(date_object)
else: else:
pic_bytes = http_response.content pic_bytes = http_response.content
ig_filename = url.split('/')[-1].split('?')[0] pic_identifier = md5(pic_bytes).hexdigest()[:16]
pic_data = TitlePic(owner_profile, target, name_suffix, ig_filename, date_object) filename = self.format_filename_within_target_path(target, owner_profile, pic_identifier, name_suffix, 'jpg')
dirname = _PostPathFormatter(pic_data).format(self.dirname_pattern, target=target)
filename_template = os.path.join(dirname,
_PostPathFormatter(pic_data).format(self.title_pattern, target=target))
filename = self.__prepare_filename(filename_template, lambda: url) + ".jpg"
content_length = http_response.headers.get('Content-Length', None) content_length = http_response.headers.get('Content-Length', None)
if os.path.isfile(filename) and (not self.context.is_logged_in or if os.path.isfile(filename) and (not self.context.is_logged_in or
(content_length is not None and (content_length is not None and
os.path.getsize(filename) >= int(content_length))): os.path.getsize(filename) >= int(content_length))):
self.context.log(filename + ' already exists') self.context.log(filename + ' already exists')
return return None
os.makedirs(os.path.dirname(filename), exist_ok=True) os.makedirs(os.path.dirname(filename), exist_ok=True)
self.context.write_raw(pic_bytes if pic_bytes else http_response, filename) self.context.write_raw(pic_bytes if pic_bytes else http_response, filename)
if date_object: if date_object:
os.utime(filename, (datetime.now().timestamp(), date_object.timestamp())) os.utime(filename, (datetime.now().timestamp(), date_object.timestamp()))
self.context.log('') # log output of _get_and_write_raw() does not produce \n self.context.log('') # log output of _get_and_write_raw() does not produce \n
def download_profilepic_if_new(self, profile: Profile, latest_stamps: Optional[LatestStamps]) -> None:
"""
Downloads and saves profile pic if it has not been downloaded before.
:param latest_stamps: Database with the last downloaded data. If not present,
the profile pic is downloaded unless it already exists
.. versionadded:: 4.8
"""
if latest_stamps is None:
self.download_profilepic(profile)
return
profile_pic_basename = profile.profile_pic_url.split('/')[-1].split('?')[0]
saved_basename = latest_stamps.get_profile_pic(profile.username)
if saved_basename == profile_pic_basename:
return
self.download_profilepic(profile)
latest_stamps.set_profile_pic(profile.username, profile_pic_basename)
def download_profilepic(self, profile: Profile) -> None: def download_profilepic(self, profile: Profile) -> None:
"""Downloads and saves profile pic.""" """Downloads and saves profile pic."""
self.download_title_pic(profile.profile_pic_url, profile.username.lower(), 'profile_pic', profile) self.download_title_pic(profile.profile_pic_url, profile.username.lower(), 'profile_pic', profile)
@@ -588,7 +470,13 @@ class Instaloader:
def test_login(self) -> Optional[str]: def test_login(self) -> Optional[str]:
"""Returns the Instagram username to which given :class:`requests.Session` object belongs, or None.""" """Returns the Instagram username to which given :class:`requests.Session` object belongs, or None."""
return self.context.test_login() data = self.context.graphql_query("d6f4427fbe92d846298cf93df0b937d3", {}).get('data', {}).get('user')
if data is not None:
profile = Profile.from_node(self.context, {'id': data.get('id'),
'username': data.get('username'),
'profile_pic_url_hd': data.get('profile_pic_url')})
return profile.username
return None
def login(self, user: str, passwd: str) -> None: def login(self, user: str, passwd: str) -> None:
"""Log in to instagram with given username and password and internally store session object. """Log in to instagram with given username and password and internally store session object.
@@ -610,21 +498,7 @@ class Instaloader:
.. versionadded:: 4.2""" .. versionadded:: 4.2"""
self.context.two_factor_login(two_factor_code) self.context.two_factor_login(two_factor_code)
@staticmethod def format_filename(self, item: Union[Post, StoryItem], target: Optional[Union[str, Path]] = None):
def __prepare_filename(filename_template: str, url: Callable[[], str]) -> str:
"""Replace filename token inside filename_template with url's filename and assure the directories exist.
.. versionadded:: 4.6"""
if "{filename}" in filename_template:
filename = filename_template.replace("{filename}",
os.path.splitext(os.path.basename(urlparse(url()).path))[0])
else:
filename = filename_template
os.makedirs(os.path.dirname(filename), exist_ok=True)
return filename
def format_filename(self, item: Union[Post, StoryItem, PostSidecarNode, TitlePic],
target: Optional[Union[str, Path]] = None):
"""Format filename of a :class:`Post` or :class:`StoryItem` according to ``filename-pattern`` parameter. """Format filename of a :class:`Post` or :class:`StoryItem` according to ``filename-pattern`` parameter.
.. versionadded:: 4.1""" .. versionadded:: 4.1"""
@@ -639,78 +513,32 @@ class Instaloader:
:return: True if something was downloaded, False otherwise, i.e. file was already there :return: True if something was downloaded, False otherwise, i.e. file was already there
""" """
def _already_downloaded(path: str) -> bool:
if not os.path.isfile(path):
return False
else:
self.context.log(path + ' exists', end=' ', flush=True)
return True
def _all_already_downloaded(path_base, is_videos_enumerated) -> bool:
if '{filename}' in self.filename_pattern:
# full URL needed to evaluate actual filename, cannot determine at
# this point if all sidecar nodes were already downloaded.
return False
for idx, is_video in is_videos_enumerated:
if self.download_pictures and (not is_video or self.download_video_thumbnails):
if not _already_downloaded("{0}_{1}.jpg".format(path_base, idx)):
return False
if is_video and self.download_videos:
if not _already_downloaded("{0}_{1}.mp4".format(path_base, idx)):
return False
return True
dirname = _PostPathFormatter(post).format(self.dirname_pattern, target=target) dirname = _PostPathFormatter(post).format(self.dirname_pattern, target=target)
filename_template = os.path.join(dirname, self.format_filename(post, target=target)) filename = os.path.join(dirname, self.format_filename(post, target=target))
filename = self.__prepare_filename(filename_template, lambda: post.url) os.makedirs(os.path.dirname(filename), exist_ok=True)
# Download the image(s) / video thumbnail and videos within sidecars if desired # Download the image(s) / video thumbnail and videos within sidecars if desired
downloaded = True downloaded = True
if post.typename == 'GraphSidecar': if self.download_pictures:
if self.download_pictures or self.download_videos: if post.typename == 'GraphSidecar':
if not _all_already_downloaded( edge_number = 1
filename_template, enumerate( for sidecar_node in post.get_sidecar_nodes():
(post.get_is_videos()[i] # Download picture or video thumbnail
for i in range(self.slide_start % post.mediacount, self.slide_end % post.mediacount + 1)), if not sidecar_node.is_video or self.download_video_thumbnails is True:
start=self.slide_start % post.mediacount + 1 downloaded &= self.download_pic(filename=filename, url=sidecar_node.display_url,
) mtime=post.date_local, filename_suffix=str(edge_number))
): # Additionally download video if available and desired
for edge_number, sidecar_node in enumerate( if sidecar_node.is_video and self.download_videos is True:
post.get_sidecar_nodes(self.slide_start, self.slide_end), downloaded &= self.download_pic(filename=filename, url=sidecar_node.video_url,
start=self.slide_start % post.mediacount + 1 mtime=post.date_local, filename_suffix=str(edge_number))
): edge_number += 1
suffix = str(edge_number) # type: Optional[str] elif post.typename == 'GraphImage':
if '{filename}' in self.filename_pattern: downloaded = self.download_pic(filename=filename, url=post.url, mtime=post.date_local)
suffix = None elif post.typename == 'GraphVideo':
if self.download_pictures and (not sidecar_node.is_video or self.download_video_thumbnails): if self.download_video_thumbnails is True:
# pylint:disable=cell-var-from-loop downloaded = self.download_pic(filename=filename, url=post.url, mtime=post.date_local)
sidecar_filename = self.__prepare_filename(filename_template, else:
lambda: sidecar_node.display_url) self.context.error("Warning: {0} has unknown typename: {1}".format(post, post.typename))
# Download sidecar picture or video thumbnail (--no-pictures implies --no-video-thumbnails)
downloaded &= self.download_pic(filename=sidecar_filename, url=sidecar_node.display_url,
mtime=post.date_local, filename_suffix=suffix)
if sidecar_node.is_video and self.download_videos:
# pylint:disable=cell-var-from-loop
sidecar_filename = self.__prepare_filename(filename_template,
lambda: sidecar_node.video_url)
# Download sidecar video if desired
downloaded &= self.download_pic(filename=sidecar_filename, url=sidecar_node.video_url,
mtime=post.date_local, filename_suffix=suffix)
else:
downloaded = False
elif post.typename == 'GraphImage':
# Download picture
if self.download_pictures:
downloaded = (not _already_downloaded(filename + ".jpg") and
self.download_pic(filename=filename, url=post.url, mtime=post.date_local))
elif post.typename == 'GraphVideo':
# Download video thumbnail (--no-pictures implies --no-video-thumbnails)
if self.download_pictures and self.download_video_thumbnails:
with self.context.error_catcher("Video thumbnail of {}".format(post)):
downloaded = (not _already_downloaded(filename + ".jpg") and
self.download_pic(filename=filename, url=post.url, mtime=post.date_local))
else:
self.context.error("Warning: {0} has unknown typename: {1}".format(post, post.typename))
# Save caption if desired # Save caption if desired
metadata_string = _ArbitraryItemFormatter(post).format(self.post_metadata_txt_pattern).strip() metadata_string = _ArbitraryItemFormatter(post).format(self.post_metadata_txt_pattern).strip()
@@ -718,20 +546,19 @@ class Instaloader:
self.save_caption(filename=filename, mtime=post.date_local, caption=metadata_string) self.save_caption(filename=filename, mtime=post.date_local, caption=metadata_string)
# Download video if desired # Download video if desired
if post.is_video and self.download_videos: if post.is_video and self.download_videos is True:
downloaded &= (not _already_downloaded(filename + ".mp4") and downloaded &= self.download_pic(filename=filename, url=post.video_url, mtime=post.date_local)
self.download_pic(filename=filename, url=post.video_url, mtime=post.date_local))
# Download geotags if desired # Download geotags if desired
if self.download_geotags and post.location: if self.download_geotags and post.location:
self.save_location(filename, post.location, post.date_local) self.save_location(filename, post.location, post.date_local)
# Update comments if desired # Update comments if desired
if self.download_comments: if self.download_comments is True:
self.update_comments(filename=filename, post=post) self.update_comments(filename=filename, post=post)
# Save metadata as JSON if desired. # Save metadata as JSON if desired.
if self.save_metadata: if self.save_metadata is not False:
self.save_metadata_json(filename, post) self.save_metadata_json(filename, post)
self.context.log() self.context.log()
@@ -756,12 +583,12 @@ class Instaloader:
def _userid_chunks(): def _userid_chunks():
assert userids is not None assert userids is not None
userids_per_query = 50 userids_per_query = 100
for i in range(0, len(userids), userids_per_query): for i in range(0, len(userids), userids_per_query):
yield userids[i:i + userids_per_query] yield userids[i:i + userids_per_query]
for userid_chunk in _userid_chunks(): for userid_chunk in _userid_chunks():
stories = self.context.graphql_query("303a4ae99711322310f25250d988f3b7", stories = self.context.graphql_query("bf41e22b1c4ba4c9f31b844ebb7d9056",
{"reel_ids": userid_chunk, "precomposed_overlay": False})["data"] {"reel_ids": userid_chunk, "precomposed_overlay": False})["data"]
yield from (Story(self.context, media) for media in stories['reels_media']) yield from (Story(self.context, media) for media in stories['reels_media'])
@@ -770,8 +597,7 @@ class Instaloader:
userids: Optional[List[Union[int, Profile]]] = None, userids: Optional[List[Union[int, Profile]]] = None,
fast_update: bool = False, fast_update: bool = False,
filename_target: Optional[str] = ':stories', filename_target: Optional[str] = ':stories',
storyitem_filter: Optional[Callable[[StoryItem], bool]] = None, storyitem_filter: Optional[Callable[[StoryItem], bool]] = None) -> None:
latest_stamps: Optional[LatestStamps] = None) -> None:
""" """
Download available stories from user followees or all stories of users whose ID are given. Download available stories from user followees or all stories of users whose ID are given.
Does not mark stories as seen. Does not mark stories as seen.
@@ -782,37 +608,20 @@ class Instaloader:
:param filename_target: Replacement for {target} in dirname_pattern and filename_pattern :param filename_target: Replacement for {target} in dirname_pattern and filename_pattern
or None if profile name should be used instead or None if profile name should be used instead
:param storyitem_filter: function(storyitem), which returns True if given StoryItem should be downloaded :param storyitem_filter: function(storyitem), which returns True if given StoryItem should be downloaded
:param latest_stamps: Database with the last times each user was scraped
:raises LoginRequiredException: If called without being logged in. :raises LoginRequiredException: If called without being logged in.
.. versionchanged:: 4.8
Add `latest_stamps` parameter.
""" """
if not userids: if not userids:
self.context.log("Retrieving all visible stories...") self.context.log("Retrieving all visible stories...")
profile_count = None
else: else:
userids = [p if isinstance(p, int) else p.userid for p in userids] userids = [p if isinstance(p, int) else p.userid for p in userids]
profile_count = len(userids)
for i, user_story in enumerate(self.get_stories(userids), start=1): for user_story in self.get_stories(userids):
name = user_story.owner_username name = user_story.owner_username
if profile_count is not None: self.context.log("Retrieving stories from profile {}.".format(name))
msg = "[{0:{w}d}/{1:{w}d}] Retrieving stories from profile {2}.".format(i, profile_count, name,
w=len(str(profile_count)))
else:
msg = "[{:3d}] Retrieving stories from profile {}.".format(i, name)
self.context.log(msg)
totalcount = user_story.itemcount totalcount = user_story.itemcount
count = 1 count = 1
if latest_stamps is not None:
# pylint:disable=cell-var-from-loop
last_scraped = latest_stamps.get_last_story_timestamp(name)
scraped_timestamp = datetime.now().astimezone()
for item in user_story.get_items(): for item in user_story.get_items():
if latest_stamps is not None and item.date_utc.replace(tzinfo=timezone.utc) <= last_scraped:
break
if storyitem_filter is not None and not storyitem_filter(item): if storyitem_filter is not None and not storyitem_filter(item):
self.context.log("<{} skipped>".format(item), flush=True) self.context.log("<{} skipped>".format(item), flush=True)
continue continue
@@ -822,8 +631,6 @@ class Instaloader:
downloaded = self.download_storyitem(item, filename_target if filename_target else name) downloaded = self.download_storyitem(item, filename_target if filename_target else name)
if fast_update and not downloaded: if fast_update and not downloaded:
break break
if latest_stamps is not None:
latest_stamps.set_last_story_timestamp(name, scraped_timestamp)
def download_storyitem(self, item: StoryItem, target: Union[str, Path]) -> bool: def download_storyitem(self, item: StoryItem, target: Union[str, Path]) -> bool:
"""Download one user story. """Download one user story.
@@ -833,25 +640,16 @@ class Instaloader:
:return: True if something was downloaded, False otherwise, i.e. file was already there :return: True if something was downloaded, False otherwise, i.e. file was already there
""" """
def _already_downloaded(path: str) -> bool:
if not os.path.isfile(path):
return False
else:
self.context.log(path + ' exists', end=' ', flush=True)
return True
date_local = item.date_local date_local = item.date_local
dirname = _PostPathFormatter(item).format(self.dirname_pattern, target=target) dirname = _PostPathFormatter(item).format(self.dirname_pattern, target=target)
filename_template = os.path.join(dirname, self.format_filename(item, target=target)) filename = os.path.join(dirname, self.format_filename(item, target=target))
filename = self.__prepare_filename(filename_template, lambda: item.url) os.makedirs(os.path.dirname(filename), exist_ok=True)
downloaded = False downloaded = False
if not item.is_video or self.download_video_thumbnails is True: if not item.is_video or self.download_video_thumbnails is True:
downloaded = (not _already_downloaded(filename + ".jpg") and url = item.url
self.download_pic(filename=filename, url=item.url, mtime=date_local)) downloaded = self.download_pic(filename=filename, url=url, mtime=date_local)
if item.is_video and self.download_videos is True: if item.is_video and self.download_videos is True:
filename = self.__prepare_filename(filename_template, lambda: str(item.video_url)) downloaded |= self.download_pic(filename=filename, url=item.video_url, mtime=date_local)
downloaded |= (not _already_downloaded(filename + ".mp4") and
self.download_pic(filename=filename, url=item.video_url, mtime=date_local))
# Save caption if desired # Save caption if desired
metadata_string = _ArbitraryItemFormatter(item).format(self.storyitem_metadata_txt_pattern).strip() metadata_string = _ArbitraryItemFormatter(item).format(self.storyitem_metadata_txt_pattern).strip()
if metadata_string: if metadata_string:
@@ -934,8 +732,7 @@ class Instaloader:
post_filter: Optional[Callable[[Post], bool]] = None, post_filter: Optional[Callable[[Post], bool]] = None,
max_count: Optional[int] = None, max_count: Optional[int] = None,
total_count: Optional[int] = None, total_count: Optional[int] = None,
owner_profile: Optional[Profile] = None, owner_profile: Optional[Profile] = None) -> None:
takewhile: Optional[Callable[[Post], bool]] = None) -> None:
""" """
Download the Posts returned by given Post Iterator. Download the Posts returned by given Post Iterator.
@@ -944,9 +741,6 @@ class Instaloader:
.. versionchanged:: 4.5 .. versionchanged:: 4.5
Transparently resume an aborted operation if `posts` is a :class:`NodeIterator`. Transparently resume an aborted operation if `posts` is a :class:`NodeIterator`.
.. versionchanged:: 4.8
Add `takewhile` parameter.
:param posts: Post Iterator to loop through. :param posts: Post Iterator to loop through.
:param target: Target name. :param target: Target name.
:param fast_update: :option:`--fast-update`. :param fast_update: :option:`--fast-update`.
@@ -954,15 +748,12 @@ class Instaloader:
:param max_count: Maximum count of Posts to download (:option:`--count`). :param max_count: Maximum count of Posts to download (:option:`--count`).
:param total_count: Total number of posts returned by given iterator. :param total_count: Total number of posts returned by given iterator.
:param owner_profile: Associated profile, if any. :param owner_profile: Associated profile, if any.
:param takewhile: Expression evaluated for each post. Once it returns false, downloading stops.
""" """
displayed_count = (max_count if total_count is None or max_count is not None and max_count < total_count displayed_count = (max_count if total_count is None or max_count is not None and max_count < total_count
else total_count) else total_count)
sanitized_target = target sanitized_target = target
if isinstance(target, str): if isinstance(target, str):
sanitized_target = _PostPathFormatter.sanitize_path(target) sanitized_target = _PostPathFormatter.sanitize_path(target)
if takewhile is None:
takewhile = lambda _: True
with resumable_iteration( with resumable_iteration(
context=self.context, context=self.context,
iterator=posts, iterator=posts,
@@ -975,7 +766,7 @@ class Instaloader:
enabled=self.resume_prefix is not None enabled=self.resume_prefix is not None
) as (is_resuming, start_index): ) as (is_resuming, start_index):
for number, post in enumerate(posts, start=start_index + 1): for number, post in enumerate(posts, start=start_index + 1):
if (max_count is not None and number > max_count) or not takewhile(post): if max_count is not None and number > max_count:
break break
if displayed_count is not None: if displayed_count is not None:
self.context.log("[{0:{w}d}/{1:{w}d}] ".format(number, displayed_count, self.context.log("[{0:{w}d}/{1:{w}d}] ".format(number, displayed_count,
@@ -1070,7 +861,7 @@ class Instaloader:
""" """
self.context.log("Retrieving saved posts...") self.context.log("Retrieving saved posts...")
assert self.context.username is not None # safe due to @_requires_login; required by typechecker assert self.context.username is not None # safe due to @_requires_login; required by typechecker
node_iterator = Profile.own_profile(self.context).get_saved_posts() node_iterator = Profile.from_username(self.context, self.context.username).get_saved_posts()
self.posts_download_loop(node_iterator, ":saved", self.posts_download_loop(node_iterator, ":saved",
fast_update, post_filter, fast_update, post_filter,
max_count=max_count, total_count=node_iterator.count) max_count=max_count, total_count=node_iterator.count)
@@ -1194,47 +985,25 @@ class Instaloader:
def download_tagged(self, profile: Profile, fast_update: bool = False, def download_tagged(self, profile: Profile, fast_update: bool = False,
target: Optional[str] = None, target: Optional[str] = None,
post_filter: Optional[Callable[[Post], bool]] = None, post_filter: Optional[Callable[[Post], bool]] = None) -> None:
latest_stamps: Optional[LatestStamps] = None) -> None:
"""Download all posts where a profile is tagged. """Download all posts where a profile is tagged.
.. versionadded:: 4.1 .. versionadded:: 4.1"""
.. versionchanged:: 4.8
Add `latest_stamps` parameter."""
self.context.log("Retrieving tagged posts for profile {}.".format(profile.username)) self.context.log("Retrieving tagged posts for profile {}.".format(profile.username))
posts_takewhile: Optional[Callable[[Post], bool]] = None self.posts_download_loop(profile.get_tagged_posts(),
if latest_stamps is not None:
last_scraped = latest_stamps.get_last_tagged_timestamp(profile.username)
posts_takewhile = lambda p: p.date_utc.replace(tzinfo=timezone.utc) > last_scraped
tagged_posts = profile.get_tagged_posts()
self.posts_download_loop(tagged_posts,
target if target target if target
else (Path(_PostPathFormatter.sanitize_path(profile.username)) / else (Path(_PostPathFormatter.sanitize_path(profile.username)) /
_PostPathFormatter.sanitize_path(':tagged')), _PostPathFormatter.sanitize_path(':tagged')),
fast_update, post_filter, takewhile=posts_takewhile) fast_update, post_filter)
if latest_stamps is not None and tagged_posts.first_item is not None:
latest_stamps.set_last_tagged_timestamp(profile.username, tagged_posts.first_item.date_local.astimezone())
def download_igtv(self, profile: Profile, fast_update: bool = False, def download_igtv(self, profile: Profile, fast_update: bool = False,
post_filter: Optional[Callable[[Post], bool]] = None, post_filter: Optional[Callable[[Post], bool]] = None) -> None:
latest_stamps: Optional[LatestStamps] = None) -> None:
"""Download IGTV videos of a profile. """Download IGTV videos of a profile.
.. versionadded:: 4.3 .. versionadded:: 4.3"""
.. versionchanged:: 4.8
Add `latest_stamps` parameter."""
self.context.log("Retrieving IGTV videos for profile {}.".format(profile.username)) self.context.log("Retrieving IGTV videos for profile {}.".format(profile.username))
posts_takewhile: Optional[Callable[[Post], bool]] = None self.posts_download_loop(profile.get_igtv_posts(), profile.username, fast_update, post_filter,
if latest_stamps is not None: total_count=profile.igtvcount, owner_profile=profile)
last_scraped = latest_stamps.get_last_igtv_timestamp(profile.username)
posts_takewhile = lambda p: p.date_utc.replace(tzinfo=timezone.utc) > last_scraped
igtv_posts = profile.get_igtv_posts()
self.posts_download_loop(igtv_posts, profile.username, fast_update, post_filter,
total_count=profile.igtvcount, owner_profile=profile, takewhile=posts_takewhile)
if latest_stamps is not None and igtv_posts.first_item is not None:
latest_stamps.set_last_igtv_timestamp(profile.username, igtv_posts.first_item.date_local.astimezone())
def _get_id_filename(self, profile_name: str) -> str: def _get_id_filename(self, profile_name: str) -> str:
if ((format_string_contains_key(self.dirname_pattern, 'profile') or if ((format_string_contains_key(self.dirname_pattern, 'profile') or
@@ -1246,22 +1015,9 @@ class Instaloader:
return os.path.join(self.dirname_pattern.format(), return os.path.join(self.dirname_pattern.format(),
'{0}_id'.format(profile_name.lower())) '{0}_id'.format(profile_name.lower()))
def load_profile_id(self, profile_name: str) -> Optional[int]:
"""
Load ID of profile from profile directory.
.. versionadded:: 4.8
"""
id_filename = self._get_id_filename(profile_name)
try:
with open(id_filename, 'rb') as id_file:
return int(id_file.read())
except (FileNotFoundError, ValueError):
return None
def save_profile_id(self, profile: Profile): def save_profile_id(self, profile: Profile):
""" """
Store ID of profile on profile directory. Store ID of profile locally.
.. versionadded:: 4.0.6 .. versionadded:: 4.0.6
""" """
@@ -1271,18 +1027,13 @@ class Instaloader:
text_file.write(str(profile.userid) + "\n") text_file.write(str(profile.userid) + "\n")
self.context.log("Stored ID {0} for profile {1}.".format(profile.userid, profile.username)) self.context.log("Stored ID {0} for profile {1}.".format(profile.userid, profile.username))
def check_profile_id(self, profile_name: str, latest_stamps: Optional[LatestStamps] = None) -> Profile: def check_profile_id(self, profile_name: str) -> Profile:
""" """
Consult locally stored ID of profile with given name, check whether ID matches and whether name Consult locally stored ID of profile with given name, check whether ID matches and whether name
has changed and return current name of the profile, and store ID of profile. has changed and return current name of the profile, and store ID of profile.
:param profile_name: Profile name :param profile_name: Profile name
:param latest_stamps: Database of downloaded data. If present, IDs are retrieved from it,
otherwise from the target directory
:return: Instance of current profile :return: Instance of current profile
.. versionchanged:: 4.8
Add `latest_stamps` parameter.
""" """
profile = None profile = None
profile_name_not_exists_err = None profile_name_not_exists_err = None
@@ -1290,11 +1041,10 @@ class Instaloader:
profile = Profile.from_username(self.context, profile_name) profile = Profile.from_username(self.context, profile_name)
except ProfileNotExistsException as err: except ProfileNotExistsException as err:
profile_name_not_exists_err = err profile_name_not_exists_err = err
if latest_stamps is None: id_filename = self._get_id_filename(profile_name)
profile_id = self.load_profile_id(profile_name) try:
else: with open(id_filename, 'rb') as id_file:
profile_id = latest_stamps.get_profile_id(profile_name) profile_id = int(id_file.read())
if profile_id is not None:
if (profile is None) or \ if (profile is None) or \
(profile_id != profile.userid): (profile_id != profile.userid):
if profile is not None: if profile is not None:
@@ -1305,27 +1055,23 @@ class Instaloader:
profile_id)) profile_id))
profile_from_id = Profile.from_id(self.context, profile_id) profile_from_id = Profile.from_id(self.context, profile_id)
newname = profile_from_id.username newname = profile_from_id.username
self.context.error("Profile {0} has changed its name to {1}.".format(profile_name, newname)) self.context.log("Profile {0} has changed its name to {1}.".format(profile_name, newname))
if latest_stamps is None: if ((format_string_contains_key(self.dirname_pattern, 'profile') or
if ((format_string_contains_key(self.dirname_pattern, 'profile') or format_string_contains_key(self.dirname_pattern, 'target'))):
format_string_contains_key(self.dirname_pattern, 'target'))): os.rename(self.dirname_pattern.format(profile=profile_name.lower(),
os.rename(self.dirname_pattern.format(profile=profile_name.lower(), target=profile_name.lower()),
target=profile_name.lower()), self.dirname_pattern.format(profile=newname.lower(),
self.dirname_pattern.format(profile=newname.lower(), target=newname.lower()))
target=newname.lower()))
else:
os.rename('{0}/{1}_id'.format(self.dirname_pattern.format(), profile_name.lower()),
'{0}/{1}_id'.format(self.dirname_pattern.format(), newname.lower()))
else: else:
latest_stamps.rename_profile(profile_name, newname) os.rename('{0}/{1}_id'.format(self.dirname_pattern.format(), profile_name.lower()),
'{0}/{1}_id'.format(self.dirname_pattern.format(), newname.lower()))
return profile_from_id return profile_from_id
# profile exists and profile id matches saved id # profile exists and profile id matches saved id
return profile return profile
except (FileNotFoundError, ValueError):
pass
if profile is not None: if profile is not None:
if latest_stamps is None: self.save_profile_id(profile)
self.save_profile_id(profile)
else:
latest_stamps.save_profile_id(profile.username, profile.userid)
return profile return profile
if profile_name_not_exists_err: if profile_name_not_exists_err:
raise profile_name_not_exists_err raise profile_name_not_exists_err
@@ -1340,8 +1086,7 @@ class Instaloader:
fast_update: bool = False, fast_update: bool = False,
post_filter: Optional[Callable[[Post], bool]] = None, post_filter: Optional[Callable[[Post], bool]] = None,
storyitem_filter: Optional[Callable[[Post], bool]] = None, storyitem_filter: Optional[Callable[[Post], bool]] = None,
raise_errors: bool = False, raise_errors: bool = False):
latest_stamps: Optional[LatestStamps] = None):
"""High-level method to download set of profiles. """High-level method to download set of profiles.
:param profiles: Set of profiles to download. :param profiles: Set of profiles to download.
@@ -1357,34 +1102,28 @@ class Instaloader:
:param raise_errors: :param raise_errors:
Whether :exc:`LoginRequiredException` and :exc:`PrivateProfileNotFollowedException` should be raised or Whether :exc:`LoginRequiredException` and :exc:`PrivateProfileNotFollowedException` should be raised or
catched and printed with :meth:`InstaloaderContext.error_catcher`. catched and printed with :meth:`InstaloaderContext.error_catcher`.
:param latest_stamps: :option:`--latest-stamps`.
.. versionadded:: 4.1 .. versionadded:: 4.1
.. versionchanged:: 4.3 .. versionchanged:: 4.3
Add `igtv` parameter. Add `igtv` parameter.
.. versionchanged:: 4.8
Add `latest_stamps` parameter.
""" """
@contextmanager @contextmanager
def _error_raiser(_str): def _error_raiser(_str):
yield yield
# error_handler type is Callable[[Optional[str]], ContextManager[None]] (not supported with Python 3.5.0..3.5.3) error_handler = cast(Callable[[Optional[str]], ContextManager[None]],
error_handler = _error_raiser if raise_errors else self.context.error_catcher _error_raiser if raise_errors else self.context.error_catcher)
for i, profile in enumerate(profiles, start=1): for profile in profiles:
self.context.log("[{0:{w}d}/{1:{w}d}] Downloading profile {2}".format(i, len(profiles), profile.username, with error_handler(profile.username):
w=len(str(len(profiles)))))
with error_handler(profile.username): # type: ignore # (ignore type for Python 3.5 support)
profile_name = profile.username profile_name = profile.username
# Download profile picture # Download profile picture
if profile_pic: if profile_pic:
with self.context.error_catcher('Download profile picture of {}'.format(profile_name)): with self.context.error_catcher('Download profile picture of {}'.format(profile_name)):
self.download_profilepic_if_new(profile, latest_stamps) self.download_profilepic(profile)
# Save metadata as JSON if desired. # Save metadata as JSON if desired.
if self.save_metadata: if self.save_metadata:
@@ -1394,26 +1133,21 @@ class Instaloader:
self.save_metadata_json(json_filename, profile) self.save_metadata_json(json_filename, profile)
# Catch some errors # Catch some errors
if tagged or igtv or highlights or posts: if profile.is_private and (tagged or igtv or highlights or posts):
if (not self.context.is_logged_in and if not self.context.is_logged_in:
profile.is_private):
raise LoginRequiredException("--login=USERNAME required.") raise LoginRequiredException("--login=USERNAME required.")
if (self.context.username != profile.username and if not profile.followed_by_viewer and self.context.username != profile.username:
profile.is_private and
not profile.followed_by_viewer):
raise PrivateProfileNotFollowedException("Private but not followed.") raise PrivateProfileNotFollowedException("Private but not followed.")
# Download tagged, if requested # Download tagged, if requested
if tagged: if tagged:
with self.context.error_catcher('Download tagged of {}'.format(profile_name)): with self.context.error_catcher('Download tagged of {}'.format(profile_name)):
self.download_tagged(profile, fast_update=fast_update, post_filter=post_filter, self.download_tagged(profile, fast_update=fast_update, post_filter=post_filter)
latest_stamps=latest_stamps)
# Download IGTV, if requested # Download IGTV, if requested
if igtv: if igtv:
with self.context.error_catcher('Download IGTV of {}'.format(profile_name)): with self.context.error_catcher('Download IGTV of {}'.format(profile_name)):
self.download_igtv(profile, fast_update=fast_update, post_filter=post_filter, self.download_igtv(profile, fast_update=fast_update, post_filter=post_filter)
latest_stamps=latest_stamps)
# Download highlights, if requested # Download highlights, if requested
if highlights: if highlights:
@@ -1423,24 +1157,14 @@ class Instaloader:
# Iterate over pictures and download them # Iterate over pictures and download them
if posts: if posts:
self.context.log("Retrieving posts from profile {}.".format(profile_name)) self.context.log("Retrieving posts from profile {}.".format(profile_name))
posts_takewhile: Optional[Callable[[Post], bool]] = None self.posts_download_loop(profile.get_posts(), profile_name, fast_update, post_filter,
if latest_stamps is not None: total_count=profile.mediacount, owner_profile=profile)
# pylint:disable=cell-var-from-loop
last_scraped = latest_stamps.get_last_post_timestamp(profile_name)
posts_takewhile = lambda p: p.date_utc.replace(tzinfo=timezone.utc) > last_scraped
posts_to_download = profile.get_posts()
self.posts_download_loop(posts_to_download, profile_name, fast_update, post_filter,
total_count=profile.mediacount, owner_profile=profile,
takewhile=posts_takewhile)
if latest_stamps is not None and posts_to_download.first_item is not None:
latest_stamps.set_last_post_timestamp(profile_name,
posts_to_download.first_item.date_local.astimezone())
if stories and profiles: if stories and profiles:
with self.context.error_catcher("Download stories"): with self.context.error_catcher("Download stories"):
self.context.log("Downloading stories") self.context.log("Downloading stories")
self.download_stories(userids=list(profiles), fast_update=fast_update, filename_target=None, self.download_stories(userids=list(profiles), fast_update=fast_update, filename_target=None,
storyitem_filter=storyitem_filter, latest_stamps=latest_stamps) storyitem_filter=storyitem_filter)
def download_profile(self, profile_name: Union[str, Profile], def download_profile(self, profile_name: Union[str, Profile],
profile_pic: bool = True, profile_pic_only: bool = False, profile_pic: bool = True, profile_pic_only: bool = False,
@@ -1539,6 +1263,5 @@ class Instaloader:
code = input("Enter 2FA verification code: ") code = input("Enter 2FA verification code: ")
self.two_factor_login(code) self.two_factor_login(code)
break break
except BadCredentialsException as err: except BadCredentialsException:
print(err, file=sys.stderr)
pass pass
+86 -176
View File
@@ -25,15 +25,16 @@ def copy_session(session: requests.Session, request_timeout: Optional[float] = N
new = requests.Session() new = requests.Session()
new.cookies = requests.utils.cookiejar_from_dict(requests.utils.dict_from_cookiejar(session.cookies)) new.cookies = requests.utils.cookiejar_from_dict(requests.utils.dict_from_cookiejar(session.cookies))
new.headers = session.headers.copy() new.headers = session.headers.copy()
# Override default timeout behavior. if request_timeout is not None:
# Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427 # Override default timeout behavior.
new.request = partial(new.request, timeout=request_timeout) # type: ignore # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
new.request = partial(new.request, timeout=request_timeout) # type: ignore
return new return new
def default_user_agent() -> str: def default_user_agent() -> str:
return 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ' \ return 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ' \
'(KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36' '(KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36'
class InstaloaderContext: class InstaloaderContext:
@@ -52,10 +53,8 @@ class InstaloaderContext:
""" """
def __init__(self, sleep: bool = True, quiet: bool = False, user_agent: Optional[str] = None, def __init__(self, sleep: bool = True, quiet: bool = False, user_agent: Optional[str] = None,
max_connection_attempts: int = 3, request_timeout: float = 300.0, max_connection_attempts: int = 3, request_timeout: Optional[float] = None,
rate_controller: Optional[Callable[["InstaloaderContext"], "RateController"]] = None, rate_controller: Optional[Callable[["InstaloaderContext"], "RateController"]] = None):
fatal_status_codes: Optional[List[int]] = None,
iphone_support: bool = True):
self.user_agent = user_agent if user_agent is not None else default_user_agent() self.user_agent = user_agent if user_agent is not None else default_user_agent()
self.request_timeout = request_timeout self.request_timeout = request_timeout
@@ -67,7 +66,6 @@ class InstaloaderContext:
self._graphql_page_length = 50 self._graphql_page_length = 50
self._root_rhx_gis = None self._root_rhx_gis = None
self.two_factor_auth_pending = None self.two_factor_auth_pending = None
self.iphone_support = iphone_support
# error log, filled with error() and printed at the end of Instaloader.main() # error log, filled with error() and printed at the end of Instaloader.main()
self.error_log = [] # type: List[str] self.error_log = [] # type: List[str]
@@ -77,12 +75,12 @@ class InstaloaderContext:
# Can be set to True for testing, disables supression of InstaloaderContext._error_catcher # Can be set to True for testing, disables supression of InstaloaderContext._error_catcher
self.raise_all_errors = False self.raise_all_errors = False
# HTTP status codes that should cause an AbortDownloadException
self.fatal_status_codes = fatal_status_codes or []
# Cache profile from id (mapping from id to Profile) # Cache profile from id (mapping from id to Profile)
self.profile_id_cache = dict() # type: Dict[int, Any] self.profile_id_cache = dict() # type: Dict[int, Any]
# Cache profile from name (mapping from username to Profile)
self.profile_name_cache = dict() # type: Dict[str, Any]
@contextmanager @contextmanager
def anonymous_copy(self): def anonymous_copy(self):
session = self._session session = self._session
@@ -154,6 +152,7 @@ class InstaloaderContext:
if empty_session_only: if empty_session_only:
del header['Host'] del header['Host']
del header['Origin'] del header['Origin']
del header['Referer']
del header['X-Instagram-AJAX'] del header['X-Instagram-AJAX']
del header['X-Requested-With'] del header['X-Requested-With']
return header return header
@@ -165,9 +164,10 @@ class InstaloaderContext:
'ig_vw': '1920', 'csrftoken': '', 'ig_vw': '1920', 'csrftoken': '',
's_network': '', 'ds_user_id': ''}) 's_network': '', 'ds_user_id': ''})
session.headers.update(self._default_http_header(empty_session_only=True)) session.headers.update(self._default_http_header(empty_session_only=True))
# Override default timeout behavior. if self.request_timeout is not None:
# Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427 # Override default timeout behavior.
session.request = partial(session.request, timeout=self.request_timeout) # type: ignore # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
return session return session
def save_session_to_file(self, sessionfile): def save_session_to_file(self, sessionfile):
@@ -180,14 +180,20 @@ class InstaloaderContext:
session.cookies = requests.utils.cookiejar_from_dict(pickle.load(sessionfile)) session.cookies = requests.utils.cookiejar_from_dict(pickle.load(sessionfile))
session.headers.update(self._default_http_header()) session.headers.update(self._default_http_header())
session.headers.update({'X-CSRFToken': session.cookies.get_dict()['csrftoken']}) session.headers.update({'X-CSRFToken': session.cookies.get_dict()['csrftoken']})
# Override default timeout behavior. if self.request_timeout is not None:
# Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427 # Override default timeout behavior.
session.request = partial(session.request, timeout=self.request_timeout) # type: ignore # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
self._session = session self._session = session
self.username = username self.username = username
def test_login(self) -> Optional[str]: def test_login(self) -> Optional[str]:
"""Not meant to be used directly, use :meth:`Instaloader.test_login`.""" """Not meant to be used directly, use :meth:`Instaloader.test_login`.
.. deprecated:: 4.5
:meth:`Instaloader.test_login` now calls :meth:`InstaloaderContext.graphql_query` directly
without using this function.
"""
data = self.graphql_query("d6f4427fbe92d846298cf93df0b937d3", {}) data = self.graphql_query("d6f4427fbe92d846298cf93df0b937d3", {})
return data["data"]["user"]["username"] if data["data"]["user"] is not None else None return data["data"]["user"]["username"] if data["data"]["user"] is not None else None
@@ -208,11 +214,12 @@ class InstaloaderContext:
'ig_vw': '1920', 'ig_cb': '1', 'csrftoken': '', 'ig_vw': '1920', 'ig_cb': '1', 'csrftoken': '',
's_network': '', 'ds_user_id': ''}) 's_network': '', 'ds_user_id': ''})
session.headers.update(self._default_http_header()) session.headers.update(self._default_http_header())
# Override default timeout behavior. if self.request_timeout is not None:
# Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427 # Override default timeout behavior.
session.request = partial(session.request, timeout=self.request_timeout) # type: ignore # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
csrf_json = self.get_json('accounts/login/', {}, session=session) session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
csrf_token = csrf_json['config']['csrf_token'] session.get('https://www.instagram.com/web/__mid/')
csrf_token = session.cookies.get_dict()['csrftoken']
session.headers.update({'X-CSRFToken': csrf_token}) session.headers.update({'X-CSRFToken': csrf_token})
# Not using self.get_json() here, because we need to access csrftoken cookie # Not using self.get_json() here, because we need to access csrftoken cookie
self.do_sleep() self.do_sleep()
@@ -223,10 +230,8 @@ class InstaloaderContext:
data={'enc_password': enc_password, 'username': user}, allow_redirects=True) data={'enc_password': enc_password, 'username': user}, allow_redirects=True)
try: try:
resp_json = login.json() resp_json = login.json()
except json.decoder.JSONDecodeError as err: except json.decoder.JSONDecodeError:
raise ConnectionException( raise ConnectionException("Login error: JSON decode fail, {} - {}.".format(login.status_code, login.reason))
"Login error: JSON decode fail, {} - {}.".format(login.status_code, login.reason)
) from err
if resp_json.get('two_factor_required'): if resp_json.get('two_factor_required'):
two_factor_session = copy_session(session, self.request_timeout) two_factor_session = copy_session(session, self.request_timeout)
two_factor_session.headers.update({'X-CSRFToken': csrf_token}) two_factor_session.headers.update({'X-CSRFToken': csrf_token})
@@ -284,9 +289,9 @@ class InstaloaderContext:
resp_json = login.json() resp_json = login.json()
if resp_json['status'] != 'ok': if resp_json['status'] != 'ok':
if 'message' in resp_json: if 'message' in resp_json:
raise BadCredentialsException("2FA error: {}".format(resp_json['message'])) raise BadCredentialsException("Login error: {}".format(resp_json['message']))
else: else:
raise BadCredentialsException("2FA error: \"{}\" status.".format(resp_json['status'])) raise BadCredentialsException("Login error: \"{}\" status.".format(resp_json['status']))
session.headers.update({'X-CSRFToken': login.cookies['csrftoken']}) session.headers.update({'X-CSRFToken': login.cookies['csrftoken']})
self._session = session self._session = session
self.username = user self.username = user
@@ -295,7 +300,7 @@ class InstaloaderContext:
def do_sleep(self): def do_sleep(self):
"""Sleep a short time if self.sleep is set. Called before each request to instagram.com.""" """Sleep a short time if self.sleep is set. Called before each request to instagram.com."""
if self.sleep: if self.sleep:
time.sleep(min(random.expovariate(0.6), 15.0)) time.sleep(min(random.expovariate(0.7), 5.0))
def get_json(self, path: str, params: Dict[str, Any], host: str = 'www.instagram.com', def get_json(self, path: str, params: Dict[str, Any], host: str = 'www.instagram.com',
session: Optional[requests.Session] = None, _attempt=1) -> Dict[str, Any]: session: Optional[requests.Session] = None, _attempt=1) -> Dict[str, Any]:
@@ -323,19 +328,12 @@ class InstaloaderContext:
if is_other_query: if is_other_query:
self._rate_controller.wait_before_query('other') self._rate_controller.wait_before_query('other')
resp = sess.get('https://{0}/{1}'.format(host, path), params=params, allow_redirects=False) resp = sess.get('https://{0}/{1}'.format(host, path), params=params, allow_redirects=False)
if resp.status_code in self.fatal_status_codes:
redirect = " redirect to {}".format(resp.headers['location']) if 'location' in resp.headers else ""
raise AbortDownloadException("Query to https://{}/{} responded with \"{} {}\"{}".format(
host, path, resp.status_code, resp.reason, redirect
))
while resp.is_redirect: while resp.is_redirect:
redirect_url = resp.headers['location'] redirect_url = resp.headers['location']
self.log('\nHTTP redirect from https://{0}/{1} to {2}'.format(host, path, redirect_url)) self.log('\nHTTP redirect from https://{0}/{1} to {2}'.format(host, path, redirect_url))
if redirect_url.startswith('https://www.instagram.com/accounts/login'): if redirect_url.startswith('https://www.instagram.com/accounts/login'):
if not self.is_logged_in:
raise LoginRequiredException("Redirected to login page. Use --login.")
# alternate rate limit exceeded behavior # alternate rate limit exceeded behavior
raise TooManyRequestsException("Redirected to login") raise TooManyRequestsException("429 Too Many Requests: redirected to login")
if redirect_url.startswith('https://{}/'.format(host)): if redirect_url.startswith('https://{}/'.format(host)):
resp = sess.get(redirect_url if redirect_url.endswith('/') else redirect_url + '/', resp = sess.get(redirect_url if redirect_url.endswith('/') else redirect_url + '/',
params=params, allow_redirects=False) params=params, allow_redirects=False)
@@ -361,10 +359,10 @@ class InstaloaderContext:
raise ConnectionException("\"window._sharedData\" does not contain required keys.") raise ConnectionException("\"window._sharedData\" does not contain required keys.")
# If GraphQL data is missing in `window._sharedData`, search for it in `__additionalDataLoaded`. # If GraphQL data is missing in `window._sharedData`, search for it in `__additionalDataLoaded`.
if 'graphql' not in post_or_profile_page[0]: if 'graphql' not in post_or_profile_page[0]:
match = re.search(r'window\.__additionalDataLoaded\(.*?({.*"graphql":.*})\);</script>', match = re.search(r'window\.__additionalDataLoaded\([^{]+{"graphql":({.*})}\);</script>',
resp.text) resp.text)
if match is not None: if match is not None:
post_or_profile_page[0]['graphql'] = json.loads(match.group(1))['graphql'] post_or_profile_page[0]['graphql'] = json.loads(match.group(1))
return resp_json return resp_json
else: else:
resp_json = resp.json() resp_json = resp.json()
@@ -481,7 +479,7 @@ class InstaloaderContext:
.. versionadded:: 4.2.1""" .. versionadded:: 4.2.1"""
with copy_session(self._session, self.request_timeout) as tempsession: with copy_session(self._session, self.request_timeout) as tempsession:
tempsession.headers['User-Agent'] = 'Instagram 146.0.0.27.125 (iPhone12,1; iOS 13_3; en_US; en-US; ' \ tempsession.headers['User-Agent'] = 'Instagram 123.1.0.26.115 (iPhone12,1; iOS 13_3; en_US; en-US; ' \
'scale=2.00; 1656x3584; 190542906)' 'scale=2.00; 1656x3584; 190542906)'
for header in ['Host', 'Origin', 'X-Instagram-AJAX', 'X-Requested-With']: for header in ['Host', 'Origin', 'X-Instagram-AJAX', 'X-Requested-With']:
tempsession.headers.pop(header, None) tempsession.headers.pop(header, None)
@@ -497,7 +495,7 @@ class InstaloaderContext:
shutil.copyfileobj(resp.raw, file) shutil.copyfileobj(resp.raw, file)
else: else:
file.write(resp) file.write(resp)
os.replace(filename + '.temp', filename) os.rename(filename + '.temp', filename)
def get_raw(self, url: str, _attempt=1) -> requests.Response: def get_raw(self, url: str, _attempt=1) -> requests.Response:
"""Downloads a file anonymously. """Downloads a file anonymously.
@@ -529,28 +527,6 @@ class InstaloaderContext:
:raises ConnectionException: When download repeatedly failed.""" :raises ConnectionException: When download repeatedly failed."""
self.write_raw(self.get_raw(url), filename) self.write_raw(self.get_raw(url), filename)
def head(self, url: str, allow_redirects: bool = False) -> requests.Response:
"""HEAD a URL anonymously.
:raises QueryReturnedNotFoundException: When the server responds with a 404.
:raises QueryReturnedForbiddenException: When the server responds with a 403.
:raises ConnectionException: When request failed.
.. versionadded:: 4.7.6
"""
with self.get_anonymous_session() as anonymous_session:
resp = anonymous_session.head(url, allow_redirects=allow_redirects)
if resp.status_code == 200:
return resp
else:
if resp.status_code == 403:
# suspected invalid URL signature
raise QueryReturnedForbiddenException("403 when accessing {}.".format(url))
if resp.status_code == 404:
# 404 not worth retrying.
raise QueryReturnedNotFoundException("404 when accessing {}.".format(url))
raise ConnectionException("HTTP error code {}.".format(resp.status_code))
@property @property
def root_rhx_gis(self) -> Optional[str]: def root_rhx_gis(self) -> Optional[str]:
"""rhx_gis string returned in the / query.""" """rhx_gis string returned in the / query."""
@@ -581,9 +557,8 @@ class RateController:
def __init__(self, context: InstaloaderContext): def __init__(self, context: InstaloaderContext):
self._context = context self._context = context
self._query_timestamps = dict() # type: Dict[str, List[float]] self._graphql_query_timestamps = dict() # type: Dict[str, List[float]]
self._earliest_next_request_time = 0.0 self._graphql_earliest_next_request_time = 0.0
self._iphone_earliest_next_request_time = 0.0
def sleep(self, secs: float): def sleep(self, secs: float):
"""Wait given number of seconds.""" """Wait given number of seconds."""
@@ -592,132 +567,67 @@ class RateController:
# pylint:disable=no-self-use # pylint:disable=no-self-use
time.sleep(secs) time.sleep(secs)
def _dump_query_timestamps(self, current_time: float, failed_query_type: str): def _dump_query_timestamps(self, current_time: float):
windows = [10, 11, 20, 22, 30, 60] windows = [10, 11, 15, 20, 30, 60]
self._context.error("Number of requests within last {} minutes grouped by type:" print("GraphQL requests:", file=sys.stderr)
.format('/'.join(str(w) for w in windows)), for query_hash, times in self._graphql_query_timestamps.items():
repeat_at_end=False) print(" {}".format(query_hash), file=sys.stderr)
for query_type, times in self._query_timestamps.items(): for window in windows:
reqs_in_sliding_window = [sum(t > current_time - w * 60 for t in times) for w in windows] reqs_in_sliding_window = sum(t > current_time - window * 60 for t in times)
self._context.error(" {} {:>32}: {}".format( print(" last {} minutes: {} requests".format(window, reqs_in_sliding_window), file=sys.stderr)
"*" if query_type == failed_query_type else " ",
query_type,
" ".join("{:4}".format(reqs) for reqs in reqs_in_sliding_window)
), repeat_at_end=False)
def count_per_sliding_window(self, query_type: str) -> int: def count_per_sliding_window(self, query_type: str) -> int:
"""Return how many requests of the given type can be done within a sliding window of 11 minutes. """Return how many GraphQL requests can be done within the sliding window."""
This is called by :meth:`RateController.query_waittime` and allows to simply customize wait times before queries
at query_type granularity. Consider overriding :meth:`RateController.query_waittime` directly if you need more
control."""
# Not static, to allow for the count_per_sliding_window to depend on context-inherent properties, such as # Not static, to allow for the count_per_sliding_window to depend on context-inherent properties, such as
# whether we are logged in. # whether we are logged in.
# pylint:disable=no-self-use # pylint:disable=no-self-use,unused-argument
return 75 if query_type == 'other' else 200 return 200
def _reqs_in_sliding_window(self, query_type: Optional[str], current_time: float, window: float) -> List[float]:
if query_type is not None:
# timestamps of type query_type
relevant_timestamps = self._query_timestamps[query_type]
else:
# all GraphQL queries, i.e. not 'iphone' or 'other'
graphql_query_timestamps = filter(lambda tp: tp[0] not in ['iphone', 'other'],
self._query_timestamps.items())
relevant_timestamps = [t for times in (tp[1] for tp in graphql_query_timestamps) for t in times]
return list(filter(lambda t: t > current_time - window, relevant_timestamps))
def query_waittime(self, query_type: str, current_time: float, untracked_queries: bool = False) -> float: def query_waittime(self, query_type: str, current_time: float, untracked_queries: bool = False) -> float:
"""Calculate time needed to wait before query can be executed.""" """Calculate time needed to wait before GraphQL query can be executed."""
per_type_sliding_window = 660 sliding_window = 660
iphone_sliding_window = 1800 if query_type not in self._graphql_query_timestamps:
if query_type not in self._query_timestamps: self._graphql_query_timestamps[query_type] = []
self._query_timestamps[query_type] = [] self._graphql_query_timestamps[query_type] = list(filter(lambda t: t > current_time - 60 * 60,
self._query_timestamps[query_type] = list(filter(lambda t: t > current_time - 60 * 60, self._graphql_query_timestamps[query_type]))
self._query_timestamps[query_type])) reqs_in_sliding_window = list(filter(lambda t: t > current_time - sliding_window,
self._graphql_query_timestamps[query_type]))
def per_type_next_request_time(): count_per_sliding_window = self.count_per_sliding_window(query_type)
reqs_in_sliding_window = self._reqs_in_sliding_window(query_type, current_time, per_type_sliding_window) if len(reqs_in_sliding_window) < count_per_sliding_window and not untracked_queries:
if len(reqs_in_sliding_window) < self.count_per_sliding_window(query_type): return max(0.0, self._graphql_earliest_next_request_time - current_time)
return 0.0 next_request_time = min(reqs_in_sliding_window) + sliding_window + 6
else: if untracked_queries:
return min(reqs_in_sliding_window) + per_type_sliding_window + 6 self._graphql_earliest_next_request_time = next_request_time
return max(next_request_time, self._graphql_earliest_next_request_time) - current_time
def gql_accumulated_next_request_time():
if query_type in ['iphone', 'other']:
return 0.0
gql_accumulated_sliding_window = 600
gql_accumulated_max_count = 275
reqs_in_sliding_window = self._reqs_in_sliding_window(None, current_time, gql_accumulated_sliding_window)
if len(reqs_in_sliding_window) < gql_accumulated_max_count:
return 0.0
else:
return min(reqs_in_sliding_window) + gql_accumulated_sliding_window
def untracked_next_request_time():
if untracked_queries:
if query_type == "iphone":
reqs_in_sliding_window = self._reqs_in_sliding_window(query_type, current_time,
iphone_sliding_window)
self._iphone_earliest_next_request_time = min(reqs_in_sliding_window) + iphone_sliding_window + 18
else:
reqs_in_sliding_window = self._reqs_in_sliding_window(query_type, current_time,
per_type_sliding_window)
self._earliest_next_request_time = min(reqs_in_sliding_window) + per_type_sliding_window + 6
return max(self._iphone_earliest_next_request_time, self._earliest_next_request_time)
def iphone_next_request():
if query_type == "iphone":
reqs_in_sliding_window = self._reqs_in_sliding_window(query_type, current_time, iphone_sliding_window)
if len(reqs_in_sliding_window) >= 199:
return min(reqs_in_sliding_window) + iphone_sliding_window + 18
return 0.0
return max(0.0,
max(
per_type_next_request_time(),
gql_accumulated_next_request_time(),
untracked_next_request_time(),
iphone_next_request(),
) - current_time)
def wait_before_query(self, query_type: str) -> None: def wait_before_query(self, query_type: str) -> None:
"""This method is called before a query to Instagram. """This method is called before a query to Instagram. It calls :meth:`RateController.sleep` to wait
until the request can be made."""
It calls :meth:`RateController.query_waittime` to determine the time needed to wait and then calls
:meth:`RateController.sleep` to wait until the request can be made."""
waittime = self.query_waittime(query_type, time.monotonic(), False) waittime = self.query_waittime(query_type, time.monotonic(), False)
assert waittime >= 0 assert waittime >= 0
if waittime > 15: if waittime > 15:
formatted_waittime = ("{} seconds".format(round(waittime)) if waittime <= 666 else self._context.log("\nToo many queries in the last time. Need to wait {} seconds, until {:%H:%M}."
"{} minutes".format(round(waittime / 60))) .format(round(waittime), datetime.now() + timedelta(seconds=waittime)))
self._context.log("\nToo many queries in the last time. Need to wait {}, until {:%H:%M}."
.format(formatted_waittime, datetime.now() + timedelta(seconds=waittime)))
if waittime > 0: if waittime > 0:
self.sleep(waittime) self.sleep(waittime)
if query_type not in self._query_timestamps: if query_type not in self._graphql_query_timestamps:
self._query_timestamps[query_type] = [time.monotonic()] self._graphql_query_timestamps[query_type] = [time.monotonic()]
else: else:
self._query_timestamps[query_type].append(time.monotonic()) self._graphql_query_timestamps[query_type].append(time.monotonic())
def handle_429(self, query_type: str) -> None: def handle_429(self, query_type: str) -> None:
"""This method is called to handle a 429 Too Many Requests response. """This method is called to handle a 429 Too Many Requests response. It calls :meth:`RateController.sleep` to
wait until we can repeat the same request."""
It calls :meth:`RateController.query_waittime` to determine the time needed to wait and then calls text_for_429 = ("HTTP error code 429 was returned because too many queries occurred in the last time. "
:meth:`RateController.sleep` to wait until we can repeat the same request.""" "Please do not use Instagram in your browser or run multiple instances of Instaloader "
"in parallel.")
print(textwrap.fill(text_for_429), file=sys.stderr)
current_time = time.monotonic() current_time = time.monotonic()
waittime = self.query_waittime(query_type, current_time, True) waittime = self.query_waittime(query_type, current_time, True)
assert waittime >= 0 assert waittime >= 0
self._dump_query_timestamps(current_time, query_type) if waittime > 15:
text_for_429 = ("Instagram responded with HTTP error \"429 - Too Many Requests\". Please do not run multiple " self._context.log("The request will be retried in {} seconds, at {:%H:%M}."
"instances of Instaloader in parallel or within short sequence. Also, do not use any Instagram " .format(round(waittime), datetime.now() + timedelta(seconds=waittime)))
"App while Instaloader is running.") self._dump_query_timestamps(current_time)
self._context.error(textwrap.fill(text_for_429), repeat_at_end=False)
if waittime > 1.5:
formatted_waittime = ("{} seconds".format(round(waittime)) if waittime <= 666 else
"{} minutes".format(round(waittime / 60)))
self._context.error("The request will be retried in {}, at {:%H:%M}."
.format(formatted_waittime, datetime.now() + timedelta(seconds=waittime)),
repeat_at_end=False)
if waittime > 0: if waittime > 0:
self.sleep(waittime) self.sleep(waittime)
-113
View File
@@ -1,113 +0,0 @@
import configparser
from datetime import datetime, timezone
from typing import Optional
class LatestStamps:
"""LatestStamps class.
Convenience class for retrieving and storing data from the :option:`--latest-stamps` file.
:param latest_stamps_file: path to file.
.. versionadded:: 4.8"""
PROFILE_ID = 'profile-id'
PROFILE_PIC = 'profile-pic'
POST_TIMESTAMP = 'post-timestamp'
TAGGED_TIMESTAMP = 'tagged-timestamp'
IGTV_TIMESTAMP = 'igtv-timestamp'
STORY_TIMESTAMP = 'story-timestamp'
ISO_FORMAT = '%Y-%m-%dT%H:%M:%S.%f%z'
def __init__(self, latest_stamps_file):
self.file = latest_stamps_file
self.data = configparser.ConfigParser()
self.data.read(latest_stamps_file)
def _save(self):
with open(self.file, 'w') as f:
self.data.write(f)
def _ensure_section(self, section: str):
if not self.data.has_section(section):
self.data.add_section(section)
def get_profile_id(self, profile_name: str) -> Optional[int]:
"""Returns stored ID of profile."""
try:
return self.data.getint(profile_name, self.PROFILE_ID)
except (configparser.Error, ValueError):
return None
def save_profile_id(self, profile_name: str, profile_id: int):
"""Stores ID of profile."""
self._ensure_section(profile_name)
self.data.set(profile_name, self.PROFILE_ID, str(profile_id))
self._save()
def rename_profile(self, old_profile: str, new_profile: str):
"""Renames a profile."""
self._ensure_section(new_profile)
for option in [self.PROFILE_ID, self.PROFILE_PIC, self.POST_TIMESTAMP,
self.TAGGED_TIMESTAMP, self.IGTV_TIMESTAMP, self.STORY_TIMESTAMP]:
if self.data.has_option(old_profile, option):
value = self.data.get(old_profile, option)
self.data.set(new_profile, option, value)
self.data.remove_section(old_profile)
self._save()
def _get_timestamp(self, section: str, key: str) -> datetime:
try:
return datetime.strptime(self.data.get(section, key), self.ISO_FORMAT)
except (configparser.Error, ValueError):
return datetime.fromtimestamp(0, timezone.utc)
def _set_timestamp(self, section: str, key: str, timestamp: datetime):
self._ensure_section(section)
self.data.set(section, key, timestamp.strftime(self.ISO_FORMAT))
self._save()
def get_last_post_timestamp(self, profile_name: str) -> datetime:
"""Returns timestamp of last download of a profile's posts."""
return self._get_timestamp(profile_name, self.POST_TIMESTAMP)
def set_last_post_timestamp(self, profile_name: str, timestamp: datetime):
"""Sets timestamp of last download of a profile's posts."""
self._set_timestamp(profile_name, self.POST_TIMESTAMP, timestamp)
def get_last_tagged_timestamp(self, profile_name: str) -> datetime:
"""Returns timestamp of last download of a profile's tagged posts."""
return self._get_timestamp(profile_name, self.TAGGED_TIMESTAMP)
def set_last_tagged_timestamp(self, profile_name: str, timestamp: datetime):
"""Sets timestamp of last download of a profile's tagged posts."""
self._set_timestamp(profile_name, self.TAGGED_TIMESTAMP, timestamp)
def get_last_igtv_timestamp(self, profile_name: str) -> datetime:
"""Returns timestamp of last download of a profile's igtv posts."""
return self._get_timestamp(profile_name, self.IGTV_TIMESTAMP)
def set_last_igtv_timestamp(self, profile_name: str, timestamp: datetime):
"""Sets timestamp of last download of a profile's igtv posts."""
self._set_timestamp(profile_name, self.IGTV_TIMESTAMP, timestamp)
def get_last_story_timestamp(self, profile_name: str) -> datetime:
"""Returns timestamp of last download of a profile's stories."""
return self._get_timestamp(profile_name, self.STORY_TIMESTAMP)
def set_last_story_timestamp(self, profile_name: str, timestamp: datetime):
"""Sets timestamp of last download of a profile's stories."""
self._set_timestamp(profile_name, self.STORY_TIMESTAMP, timestamp)
def get_profile_pic(self, profile_name: str) -> str:
"""Returns filename of profile's last downloaded profile pic."""
try:
return self.data.get(profile_name, self.PROFILE_PIC)
except configparser.Error:
return ""
def set_profile_pic(self, profile_name: str, profile_pic: str):
"""Sets filename of profile's last downloaded profile pic."""
self._ensure_section(profile_name)
self.data.set(profile_name, self.PROFILE_PIC, profile_pic)
self._save()
+30 -54
View File
@@ -5,9 +5,9 @@ import os
from contextlib import contextmanager from contextlib import contextmanager
from datetime import datetime, timedelta from datetime import datetime, timedelta
from lzma import LZMAError from lzma import LZMAError
from typing import Any, Callable, Dict, Iterable, Iterator, NamedTuple, Optional, Tuple, TypeVar from typing import Any, Callable, Dict, Iterator, NamedTuple, Optional, Tuple, TypeVar
from .exceptions import AbortDownloadException, InvalidArgumentException, QueryReturnedBadRequestException from .exceptions import InvalidArgumentException, QueryReturnedBadRequestException
from .instaloadercontext import InstaloaderContext from .instaloadercontext import InstaloaderContext
FrozenNodeIterator = NamedTuple('FrozenNodeIterator', FrozenNodeIterator = NamedTuple('FrozenNodeIterator',
@@ -17,8 +17,9 @@ FrozenNodeIterator = NamedTuple('FrozenNodeIterator',
('context_username', Optional[str]), ('context_username', Optional[str]),
('total_index', int), ('total_index', int),
('best_before', Optional[float]), ('best_before', Optional[float]),
('remaining_data', Optional[Dict]), ('remaining_data', Optional[Dict])])
('first_node', Optional[Dict])]) FrozenNodeIterator.__doc__ = \
"""A serializable representation of a :class:`NodeIterator` instance, saving its iteration state."""
FrozenNodeIterator.query_hash.__doc__ = """The GraphQL ``query_hash`` parameter.""" FrozenNodeIterator.query_hash.__doc__ = """The GraphQL ``query_hash`` parameter."""
FrozenNodeIterator.query_variables.__doc__ = """The GraphQL ``query_variables`` parameter.""" FrozenNodeIterator.query_variables.__doc__ = """The GraphQL ``query_variables`` parameter."""
FrozenNodeIterator.query_referer.__doc__ = """The HTTP referer used for the GraphQL query.""" FrozenNodeIterator.query_referer.__doc__ = """The HTTP referer used for the GraphQL query."""
@@ -27,7 +28,7 @@ FrozenNodeIterator.total_index.__doc__ = """Number of items that have already be
FrozenNodeIterator.best_before.__doc__ = """Date when parts of the stored nodes might have expired.""" FrozenNodeIterator.best_before.__doc__ = """Date when parts of the stored nodes might have expired."""
FrozenNodeIterator.remaining_data.__doc__ = \ FrozenNodeIterator.remaining_data.__doc__ = \
"""The already-retrieved, yet-unprocessed ``edges`` and the ``page_info`` at time of freezing.""" """The already-retrieved, yet-unprocessed ``edges`` and the ``page_info`` at time of freezing."""
FrozenNodeIterator.first_node.__doc__ = """Node data of the first item, if an item has been produced."""
T = TypeVar('T') T = TypeVar('T')
@@ -54,9 +55,6 @@ class NodeIterator(Iterator[T]):
post_iterator = profile.get_posts() post_iterator = profile.get_posts()
post_iterator.thaw(load("resume_information.json")) post_iterator.thaw(load("resume_information.json"))
(an appropriate method to load and save the :class:`FrozenNodeIterator` is e.g.
:func:`load_structure_from_file` and :func:`save_structure_to_file`.)
A :class:`FrozenNodeIterator` can only be thawn with a matching NodeIterator, i.e. a NodeIterator instance that has A :class:`FrozenNodeIterator` can only be thawn with a matching NodeIterator, i.e. a NodeIterator instance that has
been constructed with the same parameters as the instance that is represented by the :class:`FrozenNodeIterator` in been constructed with the same parameters as the instance that is represented by the :class:`FrozenNodeIterator` in
question. This is to ensure that an iteration cannot be resumed in a wrong, unmatching loop. As a quick way to question. This is to ensure that an iteration cannot be resumed in a wrong, unmatching loop. As a quick way to
@@ -83,14 +81,11 @@ class NodeIterator(Iterator[T]):
self._node_wrapper = node_wrapper self._node_wrapper = node_wrapper
self._query_variables = query_variables if query_variables is not None else {} self._query_variables = query_variables if query_variables is not None else {}
self._query_referer = query_referer self._query_referer = query_referer
self._data = first_data
self._page_index = 0 self._page_index = 0
self._total_index = 0 self._total_index = 0
if first_data is not None: self._best_before = (None if first_data is None else
self._data = first_data datetime.now() + NodeIterator._shelf_life)
self._best_before = datetime.now() + NodeIterator._shelf_life
else:
self._data = self._query()
self._first_node: Optional[Dict] = None
def _query(self, after: Optional[str] = None) -> Dict: def _query(self, after: Optional[str] = None) -> Dict:
pagination_variables = {'first': NodeIterator._graphql_page_length} # type: Dict[str, Any] pagination_variables = {'first': NodeIterator._graphql_page_length} # type: Dict[str, Any]
@@ -118,6 +113,8 @@ class NodeIterator(Iterator[T]):
return self return self
def __next__(self) -> T: def __next__(self) -> T:
if self._data is None:
self._data = self._query()
if self._page_index < len(self._data['edges']): if self._page_index < len(self._data['edges']):
node = self._data['edges'][self._page_index]['node'] node = self._data['edges'][self._page_index]['node']
page_index, total_index = self._page_index, self._total_index page_index, total_index = self._page_index, self._total_index
@@ -127,21 +124,17 @@ class NodeIterator(Iterator[T]):
except KeyboardInterrupt: except KeyboardInterrupt:
self._page_index, self._total_index = page_index, total_index self._page_index, self._total_index = page_index, total_index
raise raise
item = self._node_wrapper(node) return self._node_wrapper(node)
if self._first_node is None:
self._first_node = node
return item
if self._data['page_info']['has_next_page']: if self._data['page_info']['has_next_page']:
query_response = self._query(self._data['page_info']['end_cursor']) query_response = self._query(self._data['page_info']['end_cursor'])
if self._data['edges'] != query_response['edges']: page_index, data = self._page_index, self._data
page_index, data = self._page_index, self._data try:
try: self._page_index = 0
self._page_index = 0 self._data = query_response
self._data = query_response except KeyboardInterrupt:
except KeyboardInterrupt: self._page_index, self._data = page_index, data
self._page_index, self._data = page_index, data raise
raise return self.__next__()
return self.__next__()
raise StopIteration() raise StopIteration()
@property @property
@@ -157,21 +150,16 @@ class NodeIterator(Iterator[T]):
@property @property
def magic(self) -> str: def magic(self) -> str:
"""Magic string for easily identifying a matching iterator file for resuming (hash of some parameters).""" """Magic string for easily identifying a matching iterator file for resuming (hash of some parameters)."""
magic_hash = hashlib.blake2b(digest_size=6) if 'blake2b' not in hashlib.algorithms_available:
magic_hash = hashlib.new('sha224')
else:
# Use blake2b when possible, i.e. on Python >= 3.6.
magic_hash = hashlib.blake2b(digest_size=6) # type:ignore # pylint: disable=no-member
magic_hash.update(json.dumps( magic_hash.update(json.dumps(
[self._query_hash, self._query_variables, self._query_referer, self._context.username] [self._query_hash, self._query_variables, self._query_referer, self._context.username]
).encode()) ).encode())
return base64.urlsafe_b64encode(magic_hash.digest()).decode() return base64.urlsafe_b64encode(magic_hash.digest()).decode()
@property
def first_item(self) -> Optional[T]:
"""
If this iterator has produced any items, returns the first item produced.
.. versionadded:: 4.8
"""
return self._node_wrapper(self._first_node) if self._first_node is not None else None
def freeze(self) -> FrozenNodeIterator: def freeze(self) -> FrozenNodeIterator:
"""Freeze the iterator for later resuming.""" """Freeze the iterator for later resuming."""
remaining_data = None remaining_data = None
@@ -186,7 +174,6 @@ class NodeIterator(Iterator[T]):
total_index=max(self.total_index - 1, 0), total_index=max(self.total_index - 1, 0),
best_before=self._best_before.timestamp() if self._best_before else None, best_before=self._best_before.timestamp() if self._best_before else None,
remaining_data=remaining_data, remaining_data=remaining_data,
first_node=self._first_node,
) )
def thaw(self, frozen: FrozenNodeIterator) -> None: def thaw(self, frozen: FrozenNodeIterator) -> None:
@@ -206,28 +193,21 @@ class NodeIterator(Iterator[T]):
self._query_referer != frozen.query_referer or self._query_referer != frozen.query_referer or
self._context.username != frozen.context_username): self._context.username != frozen.context_username):
raise InvalidArgumentException("Mismatching resume information.") raise InvalidArgumentException("Mismatching resume information.")
if not frozen.best_before:
raise InvalidArgumentException("\"best before\" date missing.")
if frozen.remaining_data is None:
raise InvalidArgumentException("\"remaining_data\" missing.")
self._total_index = frozen.total_index self._total_index = frozen.total_index
self._best_before = datetime.fromtimestamp(frozen.best_before) self._best_before = datetime.fromtimestamp(frozen.best_before) if frozen.best_before else None
self._data = frozen.remaining_data self._data = frozen.remaining_data
if frozen.first_node is not None:
self._first_node = frozen.first_node
@contextmanager @contextmanager
def resumable_iteration(context: InstaloaderContext, def resumable_iteration(context: InstaloaderContext,
iterator: Iterable, iterator: Iterator,
load: Callable[[InstaloaderContext, str], Any], load: Callable[[InstaloaderContext, str], Any],
save: Callable[[FrozenNodeIterator, str], None], save: Callable[[FrozenNodeIterator, str], None],
format_path: Callable[[str], str], format_path: Callable[[str], str],
check_bbd: bool = True, check_bbd: bool = True,
enabled: bool = True) -> Iterator[Tuple[bool, int]]: enabled: bool = True) -> Iterator[Tuple[bool, int]]:
""" """
High-level context manager to handle a resumable iteration that can be interrupted High-level context manager to handle a resumable iteration that can be interrupted with a KeyboardInterrupt.
with a :class:`KeyboardInterrupt` or an :class:`AbortDownloadException`.
It can be used as follows to automatically load a previously-saved state into the iterator, save the iterator's It can be used as follows to automatically load a previously-saved state into the iterator, save the iterator's
state when interrupted, and delete the resume file upon completion:: state when interrupted, and delete the resume file upon completion::
@@ -255,9 +235,6 @@ def resumable_iteration(context: InstaloaderContext,
:param format_path: Returns the path to the resume file for the given magic. :param format_path: Returns the path to the resume file for the given magic.
:param check_bbd: Whether to check the best before date and reject an expired FrozenNodeIterator. :param check_bbd: Whether to check the best before date and reject an expired FrozenNodeIterator.
:param enabled: Set to False to disable all functionality and simply execute the inner body. :param enabled: Set to False to disable all functionality and simply execute the inner body.
.. versionchanged:: 4.7
Also interrupt on :class:`AbortDownloadException`.
""" """
if not enabled or not isinstance(iterator, NodeIterator): if not enabled or not isinstance(iterator, NodeIterator):
yield False, 0 yield False, 0
@@ -281,9 +258,8 @@ def resumable_iteration(context: InstaloaderContext,
context.error("Warning: Not resuming from {}: {}".format(resume_file_path, exc)) context.error("Warning: Not resuming from {}: {}".format(resume_file_path, exc))
try: try:
yield is_resuming, start_index yield is_resuming, start_index
except (KeyboardInterrupt, AbortDownloadException): except KeyboardInterrupt:
if os.path.dirname(resume_file_path): os.makedirs(os.path.dirname(resume_file_path), exist_ok=True)
os.makedirs(os.path.dirname(resume_file_path), exist_ok=True)
save(iterator.freeze(), resume_file_path) save(iterator.freeze(), resume_file_path)
context.log("\nSaved resume information to {}.".format(resume_file_path)) context.log("\nSaved resume information to {}.".format(resume_file_path))
raise raise
View File
+107 -317
View File
@@ -4,8 +4,7 @@ import re
from base64 import b64decode, b64encode from base64 import b64decode, b64encode
from collections import namedtuple from collections import namedtuple
from datetime import datetime from datetime import datetime
from pathlib import Path from typing import Any, Dict, Iterator, List, Optional, Union
from typing import Any, Dict, Iterable, Iterator, List, Optional, Tuple, Union
from . import __version__ from . import __version__
from .exceptions import * from .exceptions import *
@@ -35,8 +34,8 @@ PostLocation.id.__doc__ = "ID number of location."
PostLocation.name.__doc__ = "Location name." PostLocation.name.__doc__ = "Location name."
PostLocation.slug.__doc__ = "URL friendly variant of location name." PostLocation.slug.__doc__ = "URL friendly variant of location name."
PostLocation.has_public_page.__doc__ = "Whether location has a public page." PostLocation.has_public_page.__doc__ = "Whether location has a public page."
PostLocation.lat.__doc__ = "Latitude (:class:`float` or None)." PostLocation.lat.__doc__ = "Latitude (:class:`float`)."
PostLocation.lng.__doc__ = "Longitude (:class:`float` or None)." PostLocation.lng.__doc__ = "Longitude (:class:`float`)."
class Post: class Post:
@@ -129,14 +128,6 @@ class Post:
"""The mediaid is a decimal representation of the media shortcode.""" """The mediaid is a decimal representation of the media shortcode."""
return int(self._node['id']) return int(self._node['id'])
@property
def title(self) -> Optional[str]:
"""Title of post"""
try:
return self._field('title')
except KeyError:
return None
def __repr__(self): def __repr__(self):
return '<Post {}>'.format(self.shortcode) return '<Post {}>'.format(self.shortcode)
@@ -156,6 +147,10 @@ class Post:
) )
self._full_metadata_dict = pic_json['data']['shortcode_media'] self._full_metadata_dict = pic_json['data']['shortcode_media']
if self._full_metadata_dict is None: if self._full_metadata_dict is None:
# issue #449
self._context.error("Fetching Post metadata failed (issue #449). "
"The following data has been returned:\n"
+ json.dumps(pic_json['entry_data'], indent=2))
raise BadResponseException("Fetching Post metadata failed.") raise BadResponseException("Fetching Post metadata failed.")
if self.shortcode != self._full_metadata_dict['shortcode']: if self.shortcode != self._full_metadata_dict['shortcode']:
self._node.update(self._full_metadata_dict) self._node.update(self._full_metadata_dict)
@@ -169,8 +164,6 @@ class Post:
@property @property
def _iphone_struct(self) -> Dict[str, Any]: def _iphone_struct(self) -> Dict[str, Any]:
if not self._context.iphone_support:
raise IPhoneSupportDisabledException("iPhone support is disabled.")
if not self._context.is_logged_in: if not self._context.is_logged_in:
raise LoginRequiredException("--login required to access iPhone media info endpoint.") raise LoginRequiredException("--login required to access iPhone media info endpoint.")
if not self._iphone_struct_: if not self._iphone_struct_:
@@ -203,7 +196,7 @@ class Post:
# is contained. # is contained.
# Note that we cannot use Profile.from_id() here since that would lead us into a recursion. # Note that we cannot use Profile.from_id() here since that would lead us into a recursion.
owner_struct = self._full_metadata['owner'] owner_struct = self._full_metadata['owner']
self._owner_profile = Profile(self._context, owner_struct) self._owner_profile = Profile.from_node(self._context, owner_struct)
return self._owner_profile return self._owner_profile
@property @property
@@ -214,13 +207,7 @@ class Post:
@property @property
def owner_id(self) -> int: def owner_id(self) -> int:
"""The ID of the Post's owner.""" """The ID of the Post's owner."""
# The ID may already be available, e.g. if the post instance was created return self.owner_profile.userid
# from an `hashtag.get_posts()` iterator, so no need to make another
# http request.
if 'owner' in self._node and 'id' in self._node['owner']:
return self._node['owner']['id']
else:
return self.owner_profile.userid
@property @property
def date_local(self) -> datetime: def date_local(self) -> datetime:
@@ -249,10 +236,10 @@ class Post:
@property @property
def url(self) -> str: def url(self) -> str:
"""URL of the picture / video thumbnail of the post""" """URL of the picture / video thumbnail of the post"""
if self.typename == "GraphImage" and self._context.iphone_support and self._context.is_logged_in: if self.typename == "GraphImage" and self._context.is_logged_in:
try: try:
orig_url = self._iphone_struct['image_versions2']['candidates'][0]['url'] orig_url = self._iphone_struct['image_versions2']['candidates'][0]['url']
url = re.sub(r'([?&])se=\d+&?', r'\1', orig_url).rstrip('&') url = re.sub(r'&se=\d+(&?)', r'\1', orig_url)
return url return url
except (InstaloaderException, KeyError, IndexError) as err: except (InstaloaderException, KeyError, IndexError) as err:
self._context.error('{} Unable to fetch high quality image version of {}.'.format(err, self)) self._context.error('{} Unable to fetch high quality image version of {}.'.format(err, self))
@@ -263,60 +250,26 @@ class Post:
"""Type of post, GraphImage, GraphVideo or GraphSidecar""" """Type of post, GraphImage, GraphVideo or GraphSidecar"""
return self._field('__typename') return self._field('__typename')
@property def get_sidecar_nodes(self) -> Iterator[PostSidecarNode]:
def mediacount(self) -> int: """Sidecar nodes of a Post with typename==GraphSidecar."""
"""
The number of media in a sidecar Post, or 1 if the Post it not a sidecar.
.. versionadded:: 4.6
"""
if self.typename == 'GraphSidecar': if self.typename == 'GraphSidecar':
edges = self._field('edge_sidecar_to_children', 'edges') edges = self._field('edge_sidecar_to_children', 'edges')
return len(edges) if any(edge['node']['is_video'] for edge in edges):
return 1
def get_is_videos(self) -> List[bool]:
"""
Return a list containing the ``is_video`` property for each media in the post.
.. versionadded:: 4.7
"""
if self.typename == 'GraphSidecar':
edges = self._field('edge_sidecar_to_children', 'edges')
return [edge['node']['is_video'] for edge in edges]
return [self.is_video]
def get_sidecar_nodes(self, start=0, end=-1) -> Iterator[PostSidecarNode]:
"""
Sidecar nodes of a Post with typename==GraphSidecar.
.. versionchanged:: 4.6
Added parameters *start* and *end* to specify a slice of sidecar media.
"""
if self.typename == 'GraphSidecar':
edges = self._field('edge_sidecar_to_children', 'edges')
if end < 0:
end = len(edges)-1
if start < 0:
start = len(edges)-1
if any(edge['node']['is_video'] and 'video_url' not in edge['node'] for edge in edges[start:(end+1)]):
# video_url is only present in full metadata, issue #558. # video_url is only present in full metadata, issue #558.
edges = self._full_metadata['edge_sidecar_to_children']['edges'] edges = self._full_metadata['edge_sidecar_to_children']['edges']
for idx, edge in enumerate(edges): for idx, edge in enumerate(edges):
if start <= idx <= end: node = edge['node']
node = edge['node'] is_video = node['is_video']
is_video = node['is_video'] display_url = node['display_url']
display_url = node['display_url'] if not is_video and self._context.is_logged_in:
if not is_video and self._context.iphone_support and self._context.is_logged_in: try:
try: carousel_media = self._iphone_struct['carousel_media']
carousel_media = self._iphone_struct['carousel_media'] orig_url = carousel_media[idx]['image_versions2']['candidates'][0]['url']
orig_url = carousel_media[idx]['image_versions2']['candidates'][0]['url'] display_url = re.sub(r'&se=\d+(&?)', r'\1', orig_url)
display_url = re.sub(r'([?&])se=\d+&?', r'\1', orig_url).rstrip('&') except (InstaloaderException, KeyError, IndexError) as err:
except (InstaloaderException, KeyError, IndexError) as err: self._context.error('{} Unable to fetch high quality image version of {}.'.format(err, self))
self._context.error('{} Unable to fetch high quality image version of {}.'.format( yield PostSidecarNode(is_video=is_video, display_url=display_url,
err, self)) video_url=node['video_url'] if is_video else None)
yield PostSidecarNode(is_video=is_video, display_url=display_url,
video_url=node['video_url'] if is_video else None)
@property @property
def caption(self) -> Optional[str]: def caption(self) -> Optional[str]:
@@ -342,11 +295,9 @@ class Post:
"""List of all lowercased profiles that are mentioned in the Post's caption, without preceeding @.""" """List of all lowercased profiles that are mentioned in the Post's caption, without preceeding @."""
if not self.caption: if not self.caption:
return [] return []
# This regular expression is modified from jStassen, adjusted to use Python's \w to # This regular expression is from jStassen, adjusted to use Python's \w to support Unicode
# support Unicode and a word/beginning of string delimiter at the beginning to ensure
# that no email addresses join the list of mentions.
# http://blog.jstassen.com/2016/03/code-regex-for-instagram-username-and-hashtags/ # http://blog.jstassen.com/2016/03/code-regex-for-instagram-username-and-hashtags/
mention_regex = re.compile(r"(?:^|\W|_)(?:@)(\w(?:(?:\w|(?:\.(?!\.))){0,28}(?:\w))?)") mention_regex = re.compile(r"(?:@)(\w(?:(?:\w|(?:\.(?!\.))){0,28}(?:\w))?)")
return re.findall(mention_regex, self.caption.lower()) return re.findall(mention_regex, self.caption.lower())
@property @property
@@ -377,35 +328,7 @@ class Post:
def video_url(self) -> Optional[str]: def video_url(self) -> Optional[str]:
"""URL of the video, or None.""" """URL of the video, or None."""
if self.is_video: if self.is_video:
version_urls = [] return self._field('video_url')
try:
version_urls.append(self._field('video_url'))
except (InstaloaderException, KeyError, IndexError) as err:
self._context.error(f"Warning: Unable to fetch video from graphql of {self}: {err}")
if self._context.iphone_support and self._context.is_logged_in:
try:
version_urls.extend(version['url'] for version in self._iphone_struct['video_versions'])
except (InstaloaderException, KeyError, IndexError) as err:
self._context.error(f"Unable to fetch high-quality video version of {self}: {err}")
version_urls = list(dict.fromkeys(version_urls))
if len(version_urls) == 0:
return None
if len(version_urls) == 1:
return version_urls[0]
url_candidates: List[Tuple[int, str]] = []
for idx, version_url in enumerate(version_urls):
try:
url_candidates.append((
int(self._context.head(version_url, allow_redirects=True).headers.get('Content-Length', 0)),
version_url
))
except (InstaloaderException, KeyError, IndexError) as err:
self._context.error(f"Video URL candidate {idx+1}/{len(version_urls)} for {self}: {err}")
if not url_candidates:
# All candidates fail: Fallback to default URL and handle errors later at the actual download attempt
return version_urls[0]
url_candidates.sort()
return url_candidates[-1][1]
return None return None
@property @property
@@ -453,21 +376,18 @@ class Post:
except KeyError: except KeyError:
return self._field('edge_media_to_comment', 'count') return self._field('edge_media_to_comment', 'count')
def get_comments(self) -> Iterable[PostComment]: def get_comments(self) -> Iterator[PostComment]:
r"""Iterate over all comments of the post. r"""Iterate over all comments of the post.
Each comment is represented by a PostComment namedtuple with fields text (string), created_at (datetime), Each comment is represented by a PostComment namedtuple with fields text (string), created_at (datetime),
id (int), owner (:class:`Profile`) and answers (:class:`~typing.Iterator`\ [:class:`PostCommentAnswer`]) id (int), owner (:class:`Profile`) and answers (:class:`~typing.Iterator`\ [:class:`PostCommentAnswer`])
if available. if available.
.. versionchanged:: 4.7
Change return type to ``Iterable``.
""" """
def _postcommentanswer(node): def _postcommentanswer(node):
return PostCommentAnswer(id=int(node['id']), return PostCommentAnswer(id=int(node['id']),
created_at_utc=datetime.utcfromtimestamp(node['created_at']), created_at_utc=datetime.utcfromtimestamp(node['created_at']),
text=node['text'], text=node['text'],
owner=Profile(self._context, node['owner']), owner=Profile.from_node(self._context, node['owner']),
likes_count=node.get('edge_liked_by', {}).get('count', 0)) likes_count=node.get('edge_liked_by', {}).get('count', 0))
def _postcommentanswers(node): def _postcommentanswers(node):
@@ -496,15 +416,16 @@ class Post:
answers=_postcommentanswers(node)) answers=_postcommentanswers(node))
if self.comments == 0: if self.comments == 0:
# Avoid doing additional requests if there are no comments # Avoid doing additional requests if there are no comments
return [] return
comment_edges = self._field('edge_media_to_comment', 'edges') comment_edges = self._field('edge_media_to_comment', 'edges')
answers_count = sum([edge['node'].get('edge_threaded_comments', {}).get('count', 0) for edge in comment_edges]) answers_count = sum([edge['node'].get('edge_threaded_comments', {}).get('count', 0) for edge in comment_edges])
if self.comments == len(comment_edges) + answers_count: if self.comments == len(comment_edges) + answers_count:
# If the Post's metadata already contains all parent comments, don't do GraphQL requests to obtain them # If the Post's metadata already contains all parent comments, don't do GraphQL requests to obtain them
return [_postcomment(comment['node']) for comment in comment_edges] yield from (_postcomment(comment['node']) for comment in comment_edges)
return NodeIterator( return
yield from NodeIterator(
self._context, self._context,
'97b41c52301f77ce508f55e66d17620e', '97b41c52301f77ce508f55e66d17620e',
lambda d: d['data']['shortcode_media']['edge_media_to_parent_comment'], lambda d: d['data']['shortcode_media']['edge_media_to_parent_comment'],
@@ -514,27 +435,20 @@ class Post:
) )
def get_likes(self) -> Iterator['Profile']: def get_likes(self) -> Iterator['Profile']:
""" """Iterate over all likes of the post. A :class:`Profile` instance of each likee is yielded."""
Iterate over all likes of the post. A :class:`Profile` instance of each likee is yielded.
.. versionchanged:: 4.5.4
Require being logged in (as required by Instagram).
"""
if not self._context.is_logged_in:
raise LoginRequiredException("--login required to access likes of a post.")
if self.likes == 0: if self.likes == 0:
# Avoid doing additional requests if there are no comments # Avoid doing additional requests if there are no comments
return return
likes_edges = self._field('edge_media_preview_like', 'edges') likes_edges = self._field('edge_media_preview_like', 'edges')
if self.likes == len(likes_edges): if self.likes == len(likes_edges):
# If the Post's metadata already contains all likes, don't do GraphQL requests to obtain them # If the Post's metadata already contains all likes, don't do GraphQL requests to obtain them
yield from (Profile(self._context, like['node']) for like in likes_edges) yield from (Profile.from_node(self._context, like['node']) for like in likes_edges)
return return
yield from NodeIterator( yield from NodeIterator(
self._context, self._context,
'1cb6ec562846122743b61e492c85999f', '1cb6ec562846122743b61e492c85999f',
lambda d: d['data']['shortcode_media']['edge_liked_by'], lambda d: d['data']['shortcode_media']['edge_liked_by'],
lambda n: Profile(self._context, n), lambda n: Profile.from_node(self._context, n),
{'shortcode': self.shortcode}, {'shortcode': self.shortcode},
'https://www.instagram.com/p/{0}/'.format(self.shortcode), 'https://www.instagram.com/p/{0}/'.format(self.shortcode),
) )
@@ -560,7 +474,7 @@ class Post:
.. versionadded:: 4.4 .. versionadded:: 4.4
""" """
return ([] if not self.is_sponsored else return ([] if not self.is_sponsored else
[Profile(self._context, edge['node']['sponsor']) for edge in [Profile.from_node(self._context, edge['node']['sponsor']) for edge in
self._field('edge_media_to_sponsor_user', 'edges')]) self._field('edge_media_to_sponsor_user', 'edges')])
@property @property
@@ -578,10 +492,10 @@ class Post:
return None return None
location_id = int(loc['id']) location_id = int(loc['id'])
if any(k not in loc for k in ('name', 'slug', 'has_public_page', 'lat', 'lng')): if any(k not in loc for k in ('name', 'slug', 'has_public_page', 'lat', 'lng')):
loc.update(self._context.get_json("explore/locations/{0}/".format(location_id), loc = self._context.get_json("explore/locations/{0}/".format(location_id),
params={'__a': 1})['native_location_data']['location_info']) params={'__a': 1})['graphql']['location']
self._location = PostLocation(location_id, loc['name'], loc['slug'], loc['has_public_page'], self._location = PostLocation(location_id, loc['name'], loc['slug'], loc['has_public_page'],
loc.get('lat'), loc.get('lng')) loc['lat'], loc['lng'])
return self._location return self._location
@@ -623,6 +537,17 @@ class Profile:
# if loaded from JSON with load_structure_from_file() # if loaded from JSON with load_structure_from_file()
self._iphone_struct_ = node['iphone_struct'] self._iphone_struct_ = node['iphone_struct']
@classmethod
def from_node(cls, context: InstaloaderContext, node: Dict[str, Any]) -> 'Profile':
assert 'username' in node
if node['username'] in context.profile_name_cache:
return context.profile_name_cache[node['username']]
profile = cls(context, node)
if 'id' in node:
context.profile_id_cache[node['id']] = profile
context.profile_name_cache[profile.username] = profile
return profile
@classmethod @classmethod
def from_username(cls, context: InstaloaderContext, username: str): def from_username(cls, context: InstaloaderContext, username: str):
"""Create a Profile instance from a given username, raise exception if it does not exist. """Create a Profile instance from a given username, raise exception if it does not exist.
@@ -634,8 +559,11 @@ class Profile:
:raises: :class:`ProfileNotExistsException` :raises: :class:`ProfileNotExistsException`
""" """
# pylint:disable=protected-access # pylint:disable=protected-access
profile = cls(context, {'username': username.lower()}) if username.lower() in context.profile_name_cache:
profile._obtain_metadata() # to raise ProfileNotExistsException now in case username is invalid return context.profile_name_cache[username.lower()]
profile = cls.from_node(context, {'username': username.lower()})
profile._obtain_metadata() # to raise ProfileNotExistException now in case username is invalid
context.profile_id_cache[profile.userid] = profile
return profile return profile
@classmethod @classmethod
@@ -658,23 +586,10 @@ class Profile:
'include_highlight_reels': False}, 'include_highlight_reels': False},
rhx_gis=context.root_rhx_gis)['data']['user'] rhx_gis=context.root_rhx_gis)['data']['user']
if data: if data:
profile = cls(context, data['reel']['owner']) return cls.from_node(context, data['reel']['owner'])
else: else:
raise ProfileNotExistsException("No profile found, the user may have blocked you (ID: " + raise ProfileNotExistsException("No profile found, the user may have blocked you (ID: " +
str(profile_id) + ").") str(profile_id) + ").")
context.profile_id_cache[profile_id] = profile
return profile
@classmethod
def own_profile(cls, context: InstaloaderContext):
"""Return own profile if logged-in.
:param context: :attr:`Instaloader.context`
.. versionadded:: 4.5.2"""
if not context.is_logged_in:
raise LoginRequiredException("--login required to access own profile.")
return cls(context, context.graphql_query("d6f4427fbe92d846298cf93df0b937d3", {})["data"]["user"])
def _asdict(self): def _asdict(self):
json_node = self._node.copy() json_node = self._node.copy()
@@ -718,8 +633,6 @@ class Profile:
@property @property
def _iphone_struct(self) -> Dict[str, Any]: def _iphone_struct(self) -> Dict[str, Any]:
if not self._context.iphone_support:
raise IPhoneSupportDisabledException("iPhone support is disabled.")
if not self._context.is_logged_in: if not self._context.is_logged_in:
raise LoginRequiredException("--login required to access iPhone profile info endpoint.") raise LoginRequiredException("--login required to access iPhone profile info endpoint.")
if not self._iphone_struct_: if not self._iphone_struct_:
@@ -820,13 +733,14 @@ class Profile:
def has_public_story(self) -> bool: def has_public_story(self) -> bool:
if not self._has_public_story: if not self._has_public_story:
self._obtain_metadata() self._obtain_metadata()
# query rate might be limited: # query not rate limited if invoked anonymously:
data = self._context.graphql_query('9ca88e465c3f866a76f7adee3871bdd8', with self._context.anonymous_copy() as anonymous_context:
{'user_id': self.userid, 'include_chaining': False, data = anonymous_context.graphql_query('9ca88e465c3f866a76f7adee3871bdd8',
'include_reel': False, 'include_suggested_users': False, {'user_id': self.userid, 'include_chaining': False,
'include_logged_out_extras': True, 'include_reel': False, 'include_suggested_users': False,
'include_highlight_reels': False}, 'include_logged_out_extras': True,
'https://www.instagram.com/{}/'.format(self.username)) 'include_highlight_reels': False},
'https://www.instagram.com/{}/'.format(self.username))
self._has_public_story = data['data']['user']['has_public_story'] self._has_public_story = data['data']['user']['has_public_story']
assert self._has_public_story is not None assert self._has_public_story is not None
return self._has_public_story return self._has_public_story
@@ -862,7 +776,7 @@ class Profile:
.. versionchanged:: 4.2.1 .. versionchanged:: 4.2.1
Require being logged in for HD version (as required by Instagram).""" Require being logged in for HD version (as required by Instagram)."""
if self._context.iphone_support and self._context.is_logged_in: if self._context.is_logged_in:
try: try:
return self._iphone_struct['hd_profile_pic_url_info']['url'] return self._iphone_struct['hd_profile_pic_url_info']['url']
except (InstaloaderException, KeyError) as err: except (InstaloaderException, KeyError) as err:
@@ -884,7 +798,7 @@ class Profile:
self._obtain_metadata() self._obtain_metadata()
return NodeIterator( return NodeIterator(
self._context, self._context,
'003056d32c2554def87228bc3fd9668a', '472f257a40c653c64c666ce877d59d2b',
lambda d: d['data']['user']['edge_owner_to_timeline_media'], lambda d: d['data']['user']['edge_owner_to_timeline_media'],
lambda n: Post(self._context, n, self), lambda n: Post(self._context, n, self),
{'id': self.userid}, {'id': self.userid},
@@ -900,6 +814,7 @@ class Profile:
if self.username != self._context.username: if self.username != self._context.username:
raise LoginRequiredException("--login={} required to get that profile's saved posts.".format(self.username)) raise LoginRequiredException("--login={} required to get that profile's saved posts.".format(self.username))
self._obtain_metadata()
return NodeIterator( return NodeIterator(
self._context, self._context,
'f883d95537fbcd400f466f63d42bd8a1', 'f883d95537fbcd400f466f63d42bd8a1',
@@ -907,6 +822,7 @@ class Profile:
lambda n: Post(self._context, n), lambda n: Post(self._context, n),
{'id': self.userid}, {'id': self.userid},
'https://www.instagram.com/{0}/'.format(self.username), 'https://www.instagram.com/{0}/'.format(self.username),
self._metadata('edge_saved_media'),
) )
def get_tagged_posts(self) -> NodeIterator[Post]: def get_tagged_posts(self) -> NodeIterator[Post]:
@@ -956,7 +872,7 @@ class Profile:
self._context, self._context,
'37479f2b8209594dde7facb0d904896a', '37479f2b8209594dde7facb0d904896a',
lambda d: d['data']['user']['edge_followed_by'], lambda d: d['data']['user']['edge_followed_by'],
lambda n: Profile(self._context, n), lambda n: Profile.from_node(self._context, n),
{'id': str(self.userid)}, {'id': str(self.userid)},
'https://www.instagram.com/{0}/'.format(self.username), 'https://www.instagram.com/{0}/'.format(self.username),
) )
@@ -975,7 +891,7 @@ class Profile:
self._context, self._context,
'58712303d941c6855d4e888c5f0cd22f', '58712303d941c6855d4e888c5f0cd22f',
lambda d: d['data']['user']['edge_follow'], lambda d: d['data']['user']['edge_follow'],
lambda n: Profile(self._context, n), lambda n: Profile.from_node(self._context, n),
{'id': str(self.userid)}, {'id': str(self.userid)},
'https://www.instagram.com/{0}/'.format(self.username), 'https://www.instagram.com/{0}/'.format(self.username),
) )
@@ -990,7 +906,7 @@ class Profile:
if not self._context.is_logged_in: if not self._context.is_logged_in:
raise LoginRequiredException("--login required to get a profile's similar accounts.") raise LoginRequiredException("--login required to get a profile's similar accounts.")
self._obtain_metadata() self._obtain_metadata()
yield from (Profile(self._context, edge["node"]) for edge in yield from (Profile.from_node(self._context, edge["node"]) for edge in
self._context.graphql_query("ad99dd9d3646cc3c0dda65debcd266a7", self._context.graphql_query("ad99dd9d3646cc3c0dda65debcd266a7",
{"user_id": str(self.userid), "include_chaining": True}, {"user_id": str(self.userid), "include_chaining": True},
"https://www.instagram.com/{0}/" "https://www.instagram.com/{0}/"
@@ -1012,17 +928,11 @@ class StoryItem:
self._context = context self._context = context
self._node = node self._node = node
self._owner_profile = owner_profile self._owner_profile = owner_profile
self._iphone_struct_ = None
if 'iphone_struct' in node:
# if loaded from JSON with load_structure_from_file()
self._iphone_struct_ = node['iphone_struct']
def _asdict(self): def _asdict(self):
node = self._node node = self._node
if self._owner_profile: if self._owner_profile:
node['owner'] = self._owner_profile._asdict() node['owner'] = self._owner_profile._asdict()
if self._iphone_struct_:
node['iphone_struct'] = self._iphone_struct_
return node return node
@property @property
@@ -1047,17 +957,6 @@ class StoryItem:
def __hash__(self) -> int: def __hash__(self) -> int:
return hash(self.mediaid) return hash(self.mediaid)
@property
def _iphone_struct(self) -> Dict[str, Any]:
if not self._context.iphone_support:
raise IPhoneSupportDisabledException("iPhone support is disabled.")
if not self._context.is_logged_in:
raise LoginRequiredException("--login required to access iPhone media info endpoint.")
if not self._iphone_struct_:
data = self._context.get_iphone_json(path='api/v1/media/{}/info/'.format(self.mediaid), params={})
self._iphone_struct_ = data['items'][0]
return self._iphone_struct_
@property @property
def owner_profile(self) -> Profile: def owner_profile(self) -> Profile:
""":class:`Profile` instance of the story item's owner.""" """:class:`Profile` instance of the story item's owner."""
@@ -1109,13 +1008,6 @@ class StoryItem:
@property @property
def url(self) -> str: def url(self) -> str:
"""URL of the picture / video thumbnail of the StoryItem""" """URL of the picture / video thumbnail of the StoryItem"""
if self.typename == "GraphStoryImage" and self._context.iphone_support and self._context.is_logged_in:
try:
orig_url = self._iphone_struct['image_versions2']['candidates'][0]['url']
url = re.sub(r'([?&])se=\d+&?', r'\1', orig_url).rstrip('&')
return url
except (InstaloaderException, KeyError, IndexError) as err:
self._context.error('{} Unable to fetch high quality image version of {}.'.format(err, self))
return self._node['display_resources'][-1]['src'] return self._node['display_resources'][-1]['src']
@property @property
@@ -1132,35 +1024,7 @@ class StoryItem:
def video_url(self) -> Optional[str]: def video_url(self) -> Optional[str]:
"""URL of the video, or None.""" """URL of the video, or None."""
if self.is_video: if self.is_video:
version_urls = [] return self._node['video_resources'][-1]['src']
try:
version_urls.append(self._node['video_resources'][-1]['src'])
except (InstaloaderException, KeyError, IndexError) as err:
self._context.error(f"Warning: Unable to fetch video from graphql of {self}: {err}")
if self._context.iphone_support and self._context.is_logged_in:
try:
version_urls.extend(version['url'] for version in self._iphone_struct['video_versions'])
except (InstaloaderException, KeyError, IndexError) as err:
self._context.error(f"Unable to fetch high-quality video version of {self}: {err}")
version_urls = list(dict.fromkeys(version_urls))
if len(version_urls) == 0:
return None
if len(version_urls) == 1:
return version_urls[0]
url_candidates: List[Tuple[int, str]] = []
for idx, version_url in enumerate(version_urls):
try:
url_candidates.append((
int(self._context.head(version_url, allow_redirects=True).headers.get('Content-Length', 0)),
version_url
))
except (InstaloaderException, KeyError, IndexError) as err:
self._context.error(f"Video URL candidate {idx+1}/{len(version_urls)} for {self}: {err}")
if not url_candidates:
# All candidates fail: Fallback to default URL and handle errors later at the actual download attempt
return version_urls[0]
url_candidates.sort()
return url_candidates[-1][1]
return None return None
@@ -1216,14 +1080,14 @@ class Story:
@property @property
def last_seen_local(self) -> Optional[datetime]: def last_seen_local(self) -> Optional[datetime]:
"""Timestamp of the most recent StoryItem that has been watched or None (local time zone).""" """Timestamp when the story has last been watched or None (local time zone)."""
if self._node['seen']: if self._node['seen']:
return datetime.fromtimestamp(self._node['seen']) return datetime.fromtimestamp(self._node['seen'])
return None return None
@property @property
def last_seen_utc(self) -> Optional[datetime]: def last_seen_utc(self) -> Optional[datetime]:
"""Timestamp of the most recent StoryItem that has been watched or None (UTC).""" """Timestamp when the story has last been watched or None (UTC)."""
if self._node['seen']: if self._node['seen']:
return datetime.utcfromtimestamp(self._node['seen']) return datetime.utcfromtimestamp(self._node['seen'])
return None return None
@@ -1247,7 +1111,7 @@ class Story:
def owner_profile(self) -> Profile: def owner_profile(self) -> Profile:
""":class:`Profile` instance of the story owner.""" """:class:`Profile` instance of the story owner."""
if not self._owner_profile: if not self._owner_profile:
self._owner_profile = Profile(self._context, self._node['user']) self._owner_profile = Profile.from_node(self._context, self._node['user'])
return self._owner_profile return self._owner_profile
@property @property
@@ -1305,7 +1169,7 @@ class Highlight(Story):
def owner_profile(self) -> Profile: def owner_profile(self) -> Profile:
""":class:`Profile` instance of the highlights' owner.""" """:class:`Profile` instance of the highlights' owner."""
if not self._owner_profile: if not self._owner_profile:
self._owner_profile = Profile(self._context, self._node['owner']) self._owner_profile = Profile.from_node(self._context, self._node['owner'])
return self._owner_profile return self._owner_profile
@property @property
@@ -1489,13 +1353,9 @@ class Hashtag:
next_other = next(other_posts, None) next_other = next(other_posts, None)
while next_top is not None or next_other is not None: while next_top is not None or next_other is not None:
if next_other is None: if next_other is None:
assert next_top is not None
yield next_top
yield from sorted_top_posts yield from sorted_top_posts
break break
if next_top is None: if next_top is None:
assert next_other is not None
yield next_other
yield from other_posts yield from other_posts
break break
if next_top == next_other: if next_top == next_other:
@@ -1540,7 +1400,7 @@ class TopSearchResults:
user_node = user['user'] user_node = user['user']
if 'pk' in user_node: if 'pk' in user_node:
user_node['id'] = user_node['pk'] user_node['id'] = user_node['pk']
yield Profile(self._context, user_node) yield Profile.from_node(self._context, user_node)
def get_prefixed_usernames(self) -> Iterator[str]: def get_prefixed_usernames(self) -> Iterator[str]:
""" """
@@ -1559,7 +1419,7 @@ class TopSearchResults:
place = location.get('place', {}) place = location.get('place', {})
slug = place.get('slug') slug = place.get('slug')
loc = place.get('location', {}) loc = place.get('location', {})
yield PostLocation(int(loc['pk']), loc['name'], slug, None, loc.get('lat'), loc.get('lng')) yield PostLocation(int(loc['pk']), loc['name'], slug, None, loc['lat'], loc['lng'])
def get_hashtag_strings(self) -> Iterator[str]: def get_hashtag_strings(self) -> Iterator[str]:
""" """
@@ -1589,72 +1449,12 @@ class TopSearchResults:
return self._searchstring return self._searchstring
class TitlePic:
def __init__(self, profile: Optional[Profile], target: Union[str, Path], typename: str,
filename: str, date_utc: Optional[datetime]):
self._profile = profile
self._target = target
self._typename = typename
self._filename = filename
self._date_utc = date_utc
@property
def profile(self) -> Union[str, Path]:
return self._profile.username.lower() if self._profile is not None else self._target
@property
def owner_username(self) -> Union[str, Path]:
return self.profile
@property
def owner_id(self) -> Union[str, Path]:
return str(self._profile.userid) if self._profile is not None else self._target
@property
def target(self) -> Union[str, Path]:
return self._target
@property
def typename(self) -> str:
return self._typename
@property
def filename(self) -> str:
return self._filename
@property
def date_utc(self) -> Optional[datetime]:
return self._date_utc
@property
def date(self) -> Optional[datetime]:
return self.date_utc
@property
def date_local(self) -> Optional[datetime]:
return self._date_utc.astimezone() if self._date_utc is not None else None
JsonExportable = Union[Post, Profile, StoryItem, Hashtag, FrozenNodeIterator] JsonExportable = Union[Post, Profile, StoryItem, Hashtag, FrozenNodeIterator]
def get_json_structure(structure: JsonExportable) -> dict:
"""Returns Instaloader JSON structure for a :class:`Post`, :class:`Profile`, :class:`StoryItem`, :class:`Hashtag`
or :class:`FrozenNodeIterator` so that it can be loaded by :func:`load_structure`.
:param structure: :class:`Post`, :class:`Profile`, :class:`StoryItem` or :class:`Hashtag`
.. versionadded:: 4.8
"""
return {
'node': structure._asdict(),
'instaloader': {'version': __version__, 'node_type': structure.__class__.__name__}
}
def save_structure_to_file(structure: JsonExportable, filename: str) -> None: def save_structure_to_file(structure: JsonExportable, filename: str) -> None:
"""Saves a :class:`Post`, :class:`Profile`, :class:`StoryItem`, :class:`Hashtag` or :class:`FrozenNodeIterator` to a """Saves a :class:`Post`, :class:`Profile`, :class:`StoryItem` or :class:`Hashtag` to a '.json' or '.json.xz' file
'.json' or '.json.xz' file such that it can later be loaded by :func:`load_structure_from_file`. such that it can later be loaded by :func:`load_structure_from_file`.
If the specified filename ends in '.xz', the file will be LZMA compressed. Otherwise, a pretty-printed JSON file If the specified filename ends in '.xz', the file will be LZMA compressed. Otherwise, a pretty-printed JSON file
will be created. will be created.
@@ -1662,7 +1462,8 @@ def save_structure_to_file(structure: JsonExportable, filename: str) -> None:
:param structure: :class:`Post`, :class:`Profile`, :class:`StoryItem` or :class:`Hashtag` :param structure: :class:`Post`, :class:`Profile`, :class:`StoryItem` or :class:`Hashtag`
:param filename: Filename, ends in '.json' or '.json.xz' :param filename: Filename, ends in '.json' or '.json.xz'
""" """
json_structure = get_json_structure(structure) json_structure = {'node': structure._asdict(),
'instaloader': {'version': __version__, 'node_type': structure.__class__.__name__}}
compress = filename.endswith('.xz') compress = filename.endswith('.xz')
if compress: if compress:
with lzma.open(filename, 'wt', check=lzma.CHECK_NONE) as fp: with lzma.open(filename, 'wt', check=lzma.CHECK_NONE) as fp:
@@ -1672,39 +1473,9 @@ def save_structure_to_file(structure: JsonExportable, filename: str) -> None:
json.dump(json_structure, fp=fp, indent=4, sort_keys=True) json.dump(json_structure, fp=fp, indent=4, sort_keys=True)
def load_structure(context: InstaloaderContext, json_structure: dict) -> JsonExportable:
"""Loads a :class:`Post`, :class:`Profile`, :class:`StoryItem`, :class:`Hashtag` or :class:`FrozenNodeIterator` from
a json structure.
:param context: :attr:`Instaloader.context` linked to the new object, used for additional queries if neccessary.
:param json_structure: Instaloader JSON structure
.. versionadded:: 4.8
"""
if 'node' in json_structure and 'instaloader' in json_structure and \
'node_type' in json_structure['instaloader']:
node_type = json_structure['instaloader']['node_type']
if node_type == "Post":
return Post(context, json_structure['node'])
elif node_type == "Profile":
return Profile(context, json_structure['node'])
elif node_type == "StoryItem":
return StoryItem(context, json_structure['node'])
elif node_type == "Hashtag":
return Hashtag(context, json_structure['node'])
elif node_type == "FrozenNodeIterator":
if not 'first_node' in json_structure['node']:
json_structure['node']['first_node'] = None
return FrozenNodeIterator(**json_structure['node'])
elif 'shortcode' in json_structure:
# Post JSON created with Instaloader v3
return Post.from_shortcode(context, json_structure['shortcode'])
raise InvalidArgumentException("Passed json structure is not an Instaloader JSON")
def load_structure_from_file(context: InstaloaderContext, filename: str) -> JsonExportable: def load_structure_from_file(context: InstaloaderContext, filename: str) -> JsonExportable:
"""Loads a :class:`Post`, :class:`Profile`, :class:`StoryItem`, :class:`Hashtag` or :class:`FrozenNodeIterator` from """Loads a :class:`Post`, :class:`Profile`, :class:`StoryItem` or :class:`Hashtag` from a '.json' or '.json.xz' file
a '.json' or '.json.xz' file that has been saved by :func:`save_structure_to_file`. that has been saved by :func:`save_structure_to_file`.
:param context: :attr:`Instaloader.context` linked to the new object, used for additional queries if neccessary. :param context: :attr:`Instaloader.context` linked to the new object, used for additional queries if neccessary.
:param filename: Filename, ends in '.json' or '.json.xz' :param filename: Filename, ends in '.json' or '.json.xz'
@@ -1716,4 +1487,23 @@ def load_structure_from_file(context: InstaloaderContext, filename: str) -> Json
fp = open(filename, 'rt') fp = open(filename, 'rt')
json_structure = json.load(fp) json_structure = json.load(fp)
fp.close() fp.close()
return load_structure(context, json_structure) if 'node' in json_structure and 'instaloader' in json_structure and \
'node_type' in json_structure['instaloader']:
node_type = json_structure['instaloader']['node_type']
if node_type == "Post":
return Post(context, json_structure['node'])
elif node_type == "Profile":
return Profile.from_node(context, json_structure['node'])
elif node_type == "StoryItem":
return StoryItem(context, json_structure['node'])
elif node_type == "Hashtag":
return Hashtag(context, json_structure['node'])
elif node_type == "FrozenNodeIterator":
return FrozenNodeIterator(**json_structure['node'])
else:
raise InvalidArgumentException("{}: Not an Instaloader JSON.".format(filename))
elif 'shortcode' in json_structure:
# Post JSON created with Instaloader v3
return Post.from_shortcode(context, json_structure['shortcode'])
else:
raise InvalidArgumentException("{}: Not an Instaloader JSON.".format(filename))
+9 -6
View File
@@ -3,6 +3,7 @@
import re import re
import sys import sys
import os import os
import platform
from setuptools import setup from setuptools import setup
@@ -18,11 +19,14 @@ def get_version():
raise SystemExit("Could not find version string.") raise SystemExit("Could not find version string.")
if sys.version_info < (3, 6): if sys.version_info < (3, 5):
sys.exit('Instaloader requires Python >= 3.6.') sys.exit('Instaloader requires Python >= 3.5.')
requirements = ['requests>=2.4'] requirements = ['requests>=2.4']
if platform.system() == 'Windows' and sys.version_info < (3, 6):
requirements.append('win_unicode_console')
keywords = (['instagram', 'instagram-scraper', 'instagram-client', 'instagram-feed', 'downloader', 'videos', 'photos', keywords = (['instagram', 'instagram-scraper', 'instagram-client', 'instagram-feed', 'downloader', 'videos', 'photos',
'pictures', 'instagram-user-photos', 'instagram-photos', 'instagram-metadata', 'instagram-downloader', 'pictures', 'instagram-user-photos', 'instagram-photos', 'instagram-metadata', 'instagram-downloader',
'instagram-stories']) 'instagram-stories'])
@@ -34,7 +38,6 @@ setup(
name='instaloader', name='instaloader',
version=get_version(), version=get_version(),
packages=['instaloader'], packages=['instaloader'],
package_data={'instaloader': ['py.typed']},
url='https://instaloader.github.io/', url='https://instaloader.github.io/',
license='MIT', license='MIT',
author='Alexander Graf, André Koch-Kramer', author='Alexander Graf, André Koch-Kramer',
@@ -43,9 +46,9 @@ setup(
'from Instagram.', 'from Instagram.',
long_description=open(os.path.join(SRC, 'README.rst')).read(), long_description=open(os.path.join(SRC, 'README.rst')).read(),
install_requires=requirements, install_requires=requirements,
python_requires='>=3.6', python_requires='>=3.5',
entry_points={'console_scripts': ['instaloader=instaloader.__main__:main']}, entry_points={'console_scripts': ['instaloader=instaloader.__main__:main']},
zip_safe=False, zip_safe=True,
keywords=keywords, keywords=keywords,
classifiers=[ classifiers=[
'Development Status :: 5 - Production/Stable', 'Development Status :: 5 - Production/Stable',
@@ -54,10 +57,10 @@ setup(
'Intended Audience :: Developers', 'Intended Audience :: Developers',
'License :: OSI Approved :: MIT License', 'License :: OSI Approved :: MIT License',
'Operating System :: OS Independent', 'Operating System :: OS Independent',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7', 'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3 :: Only', 'Programming Language :: Python :: 3 :: Only',
'Topic :: Internet', 'Topic :: Internet',
'Topic :: Multimedia :: Graphics' 'Topic :: Multimedia :: Graphics'