Release of Version 4.5.1

Improve 429 error message
This text_for_429 error message is expected to be less confusing for new users. Further improve the output formatting of _dump_query_timestamps().
2026-10-16 21:52:17 -07:00 · 2020-08-20 11:08:08 +02:00 · 2020-08-20 10:38:06 +02:00 · 2020-08-17 18:29:37 +02:00 · 2020-08-17 18:25:22 +02:00 · 2020-08-17 18:23:24 +02:00
47 changed files with 1366 additions and 1079 deletions
@@ -24,4 +24,4 @@ If applicable, add error messages and tracebacks to help explain your problem.
 **Additional context**
 Add any other context about the problem here.

-(please also see https://instaloader.github.io/contributing.html for how to report a bug)
+<!-- please also see https://instaloader.github.io/contributing.html for how to report a bug -->
@@ -0,0 +1,11 @@
+blank_issues_enabled: false
+contact_links:
+  - name: Contributing to Instaloader
+    url: https://instaloader.github.io/contributing.html
+    about: Guidelines for how to report an issue.
+  - name: Instaloader Troubleshooting
+    url: https://instaloader.github.io/troubleshooting.html
+    about: Many common problems and their workarounds are described here.
+  - name: Instaloader Documentation
+    url: https://instaloader.github.io/
+    about: Many questions might be already answered here.
@@ -14,11 +14,11 @@ A clear and concise description of what you want to happen.
 **Describe alternatives you've considered**
 A clear and concise description of any alternative solutions or features you've considered.

-**If the feature request is accepted, would you be willing to submit a PR?**
-
-Yes / No _(Help can be provided if you need assistance submitting a PR)_
+**If the feature request is accepted, would you be willing to submit a pull request?**
+Yes / No
+<!-- Help can be provided if you need assistance submitting a PR -->

 **Additional context**
 Add any other context about the feature request here.

-(please also see https://instaloader.github.io/contributing.html for how to suggest a feature)
+<!-- please also see https://instaloader.github.io/contributing.html for how to suggest a feature -->
@@ -6,3 +6,7 @@ labels: question
 ---

 Your question here...
+
+<!-- Please see our documentation: https://instaloader.github.io/ -->
+
+<!-- Basic questions regarding Instaloader's usage might be more appropriate to ask on Stack Overflow -->
@@ -0,0 +1,17 @@
+
+
+<!--
+Please describe:
+
+- A motivation for this change, e.g.
+  - Fixes # .
+  - More general: What problem does the pull request solve?
+
+- The changes proposed in this pull request
+
+- The completeness of this change
+  - Is it just a proof of concept?
+  - Is the documentation updated (if appropriate)?
+  - Do you consider it ready to be merged or is it a draft?
+  - Can we help you at some point?
+-->
@@ -0,0 +1,30 @@
+name: PyLint and MyPy
+
+on: [push, pull_request]
+
+jobs:
+  lint:
+    name: PyLint and MyPy
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.8]
+    steps:
+      - name: Checkout Instaloader Repository
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0  # needed for building docs
+      - name: Setup Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install Dependencies
+        run: |
+          python -m pip install pipenv==2020.6.2
+          pipenv sync --dev
+      - name: PyLint
+        run: pipenv run pylint instaloader
+      - name: MyPy
+        run: pipenv run mypy -m instaloader
+      - name: Build Documentation
+        run: pipenv run make -C docs html SPHINXOPTS="-W -n"
@@ -7,6 +7,7 @@ on:
 jobs:
  stale:

+    name: Mark stale issues and pull requests
    runs-on: ubuntu-latest
    
    steps:
@@ -4,6 +4,7 @@ on: [push, pull_request]

 jobs:
  build:
+    name: Windows EXE build
    runs-on: windows-latest
    steps:
      - name: Checkout Instaloader repository
@@ -27,7 +27,7 @@ deploy:
    repo: instaloader/instaloader.github.io
    target-branch: master
    on:
-      tags: true
+      branch: master
      python: 3.6
  - provider: script
    script: deploy/arch/deploy.sh $TRAVIS_TAG
@@ -6,7 +6,7 @@ verify_ssl = true
 [dev-packages]
 pylint = "*"
 mypy = "*"
-sphinx = "*"
+sphinx = "~=2.3"
 pyinstaller = "*"
 pefile = "*"
 pywin32-ctypes = "*"
@@ -1,7 +1,7 @@
 {
    "_meta": {
        "hash": {
-            "sha256": "33408a60fda1a56bf680462fad21643ae0159bc13f3b85d8f10ebf3eef7d8725"
+            "sha256": "58caf1be034bed9aa7ebb8c78e75ffa7d7abe820d014b6398c444a5db566b306"
        },
        "pipfile-spec": 6,
        "requires": {},
@@ -16,10 +16,10 @@
    "default": {
        "certifi": {
            "hashes": [
-                "sha256:017c25db2a153ce562900032d5bc68e9f191e44e9a0f762f373977de9df1fbb3",
-                "sha256:25b64c7da4cd7479594d035c08c2d809eb4aab3a26e5a990ea98cc450c320f1f"
+                "sha256:5930595817496dd21bb8dc35dad090f1c2cd0adfaf21204bf6732ca5d8ee34d3",
+                "sha256:8fc0819f1f30ba15bdb34cceffb9ef04d99f420f68eb75d901e9560b8749fc41"
            ],
-            "version": "==2019.11.28"
+            "version": "==2020.6.20"
        },
        "chardet": {
            "hashes": [
@@ -30,25 +30,25 @@
        },
        "idna": {
            "hashes": [
-                "sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407",
-                "sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c"
+                "sha256:b307872f855b18632ce0c21c5e45be78c0ea7ae4c15c828c20788b26921eb3f6",
+                "sha256:b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0"
            ],
-            "version": "==2.8"
+            "version": "==2.10"
        },
        "requests": {
            "hashes": [
-                "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4",
-                "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31"
+                "sha256:b3559a131db72c33ee969480840fff4bb6dd111de7dd27c8ee1f820f4f00231b",
+                "sha256:fe75cc94a9443b9246fc7049224f75604b113c36acb93f87b80ed42c44cbb898"
            ],
            "index": "pypi",
-            "version": "==2.22.0"
+            "version": "==2.24.0"
        },
        "urllib3": {
            "hashes": [
-                "sha256:2f3db8b19923a873b3e5256dc9c2dedfa883e33d87c690d9c7913e1f40673cdc",
-                "sha256:87716c2d2a7121198ebcb7ce7cccf6ce5e9ba539041cfbaeecfb641dc0bf6acc"
+                "sha256:91056c15fa70756691db97756772bb1eb9678fa585d9184f24534b100dc60f4a",
+                "sha256:e7983572181f5e1522d9c98453462384ee92a0be7fac5f1413a1e35c56cc0461"
            ],
-            "version": "==1.25.8"
+            "version": "==1.25.10"
        }
    },
    "develop": {
@@ -68,10 +68,10 @@
        },
        "astroid": {
            "hashes": [
-                "sha256:71ea07f44df9568a75d0f354c49143a4575d90645e9fead6dfb52c26a85ed13a",
-                "sha256:840947ebfa8b58f318d42301cf8c0a20fd794a33b61cc4638e28e9e61ba32f42"
+                "sha256:2f4078c2a41bf377eea06d71c9d2ba4eb8f6b1af2135bec27bbbb7d8f12bb703",
+                "sha256:bc58d83eb610252fd8de6363e39d4f1d0619c894b0ed24603b881c02e64c7386"
            ],
-            "version": "==2.3.3"
+            "version": "==2.4.2"
        },
        "babel": {
            "hashes": [
@@ -82,10 +82,10 @@
        },
        "certifi": {
            "hashes": [
-                "sha256:017c25db2a153ce562900032d5bc68e9f191e44e9a0f762f373977de9df1fbb3",
-                "sha256:25b64c7da4cd7479594d035c08c2d809eb4aab3a26e5a990ea98cc450c320f1f"
+                "sha256:5930595817496dd21bb8dc35dad090f1c2cd0adfaf21204bf6732ca5d8ee34d3",
+                "sha256:8fc0819f1f30ba15bdb34cceffb9ef04d99f420f68eb75d901e9560b8749fc41"
            ],
-            "version": "==2019.11.28"
+            "version": "==2020.6.20"
        },
        "chardet": {
            "hashes": [
@@ -109,10 +109,10 @@
        },
        "idna": {
            "hashes": [
-                "sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407",
-                "sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c"
+                "sha256:b307872f855b18632ce0c21c5e45be78c0ea7ae4c15c828c20788b26921eb3f6",
+                "sha256:b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0"
            ],
-            "version": "==2.8"
+            "version": "==2.10"
        },
        "imagesize": {
            "hashes": [
@@ -130,10 +130,10 @@
        },
        "jinja2": {
            "hashes": [
-                "sha256:93187ffbc7808079673ef52771baa950426fd664d3aad1d0fa3e95644360e250",
-                "sha256:b0eaf100007721b5c16c1fc1eecb87409464edc10469ddc9a22a27a99123be49"
+                "sha256:89aab215427ef59c34ad58735269eb58b1a5808103067f7bb9d5836c651b3bb0",
+                "sha256:f0a4641d3cf955324a89c04f3d94663aa4d638abe8f733ecd3582848e1c37035"
            ],
-            "version": "==2.11.1"
+            "version": "==2.11.2"
        },
        "lazy-object-proxy": {
            "hashes": [
@@ -208,23 +208,23 @@
        },
        "mypy": {
            "hashes": [
-                "sha256:0a9a45157e532da06fe56adcfef8a74629566b607fa2c1ac0122d1ff995c748a",
-                "sha256:2c35cae79ceb20d47facfad51f952df16c2ae9f45db6cb38405a3da1cf8fc0a7",
-                "sha256:4b9365ade157794cef9685791032521233729cb00ce76b0ddc78749abea463d2",
-                "sha256:53ea810ae3f83f9c9b452582261ea859828a9ed666f2e1ca840300b69322c474",
-                "sha256:634aef60b4ff0f650d3e59d4374626ca6153fcaff96ec075b215b568e6ee3cb0",
-                "sha256:7e396ce53cacd5596ff6d191b47ab0ea18f8e0ec04e15d69728d530e86d4c217",
-                "sha256:7eadc91af8270455e0d73565b8964da1642fe226665dd5c9560067cd64d56749",
-                "sha256:7f672d02fffcbace4db2b05369142e0506cdcde20cea0e07c7c2171c4fd11dd6",
-                "sha256:85baab8d74ec601e86134afe2bcccd87820f79d2f8d5798c889507d1088287bf",
-                "sha256:87c556fb85d709dacd4b4cb6167eecc5bbb4f0a9864b69136a0d4640fdc76a36",
-                "sha256:a6bd44efee4dc8c3324c13785a9dc3519b3ee3a92cada42d2b57762b7053b49b",
-                "sha256:c6d27bd20c3ba60d5b02f20bd28e20091d6286a699174dfad515636cb09b5a72",
-                "sha256:e2bb577d10d09a2d8822a042a23b8d62bc3b269667c9eb8e60a6edfa000211b1",
-                "sha256:f97a605d7c8bc2c6d1172c2f0d5a65b24142e11a58de689046e62c2d632ca8c1"
+                "sha256:2c6cde8aa3426c1682d35190b59b71f661237d74b053822ea3d748e2c9578a7c",
+                "sha256:3fdda71c067d3ddfb21da4b80e2686b71e9e5c72cca65fa216d207a358827f86",
+                "sha256:5dd13ff1f2a97f94540fd37a49e5d255950ebcdf446fb597463a40d0df3fac8b",
+                "sha256:6731603dfe0ce4352c555c6284c6db0dc935b685e9ce2e4cf220abe1e14386fd",
+                "sha256:6bb93479caa6619d21d6e7160c552c1193f6952f0668cdda2f851156e85186fc",
+                "sha256:81c7908b94239c4010e16642c9102bfc958ab14e36048fa77d0be3289dda76ea",
+                "sha256:9c7a9a7ceb2871ba4bac1cf7217a7dd9ccd44c27c2950edbc6dc08530f32ad4e",
+                "sha256:a4a2cbcfc4cbf45cd126f531dedda8485671545b43107ded25ce952aac6fb308",
+                "sha256:b7fbfabdbcc78c4f6fc4712544b9b0d6bf171069c6e0e3cb82440dd10ced3406",
+                "sha256:c05b9e4fb1d8a41d41dec8786c94f3b95d3c5f528298d769eb8e73d293abc48d",
+                "sha256:d7df6eddb6054d21ca4d3c6249cae5578cb4602951fd2b6ee2f5510ffb098707",
+                "sha256:e0b61738ab504e656d1fe4ff0c0601387a5489ca122d55390ade31f9ca0e252d",
+                "sha256:eff7d4a85e9eea55afa34888dfeaccde99e7520b51f867ac28a48492c0b1130c",
+                "sha256:f05644db6779387ccdb468cc47a44b4356fc2ffa9287135d05b70a98dc83b89a"
            ],
            "index": "pypi",
-            "version": "==0.761"
+            "version": "==0.782"
        },
        "mypy-extensions": {
            "hashes": [
@@ -235,10 +235,10 @@
        },
        "packaging": {
            "hashes": [
-                "sha256:170748228214b70b672c581a3dd610ee51f733018650740e98c7df862a583f73",
-                "sha256:e665345f9eef0c621aa0bf2f8d78cf6d21904eef16a93f020240b704a57f1334"
+                "sha256:4357f74f47b9c12db93624a82154e9b120fa8293699949152b22065d556079f8",
+                "sha256:998416ba6962ae7fbd6596850b80e17859a5753ba17c32284f67bfff33784181"
            ],
-            "version": "==20.1"
+            "version": "==20.4"
        },
        "pefile": {
            "hashes": [
@@ -249,27 +249,27 @@
        },
        "psutil": {
            "hashes": [
-                "sha256:094f899ac3ef72422b7e00411b4ed174e3c5a2e04c267db6643937ddba67a05b",
-                "sha256:10b7f75cc8bd676cfc6fa40cd7d5c25b3f45a0e06d43becd7c2d2871cbb5e806",
-                "sha256:1b1575240ca9a90b437e5a40db662acd87bbf181f6aa02f0204978737b913c6b",
-                "sha256:21231ef1c1a89728e29b98a885b8e0a8e00d09018f6da5cdc1f43f988471a995",
-                "sha256:28f771129bfee9fc6b63d83a15d857663bbdcae3828e1cb926e91320a9b5b5cd",
-                "sha256:70387772f84fa5c3bb6a106915a2445e20ac8f9821c5914d7cbde148f4d7ff73",
-                "sha256:b560f5cd86cf8df7bcd258a851ca1ad98f0d5b8b98748e877a0aec4e9032b465",
-                "sha256:b74b43fecce384a57094a83d2778cdfc2e2d9a6afaadd1ebecb2e75e0d34e10d",
-                "sha256:e85f727ffb21539849e6012f47b12f6dd4c44965e56591d8dec6e8bc9ab96f4a",
-                "sha256:fd2e09bb593ad9bdd7429e779699d2d47c1268cbde4dda95fcd1bd17544a0217",
-                "sha256:ffad8eb2ac614518bbe3c0b8eb9dffdb3a8d2e3a7d5da51c5b974fb723a5c5aa"
+                "sha256:0ee3c36428f160d2d8fce3c583a0353e848abb7de9732c50cf3356dd49ad63f8",
+                "sha256:10512b46c95b02842c225f58fa00385c08fa00c68bac7da2d9a58ebe2c517498",
+                "sha256:4080869ed93cce662905b029a1770fe89c98787e543fa7347f075ade761b19d6",
+                "sha256:5e9d0f26d4194479a13d5f4b3798260c20cecf9ac9a461e718eb59ea520a360c",
+                "sha256:66c18ca7680a31bf16ee22b1d21b6397869dda8059dbdb57d9f27efa6615f195",
+                "sha256:68d36986ded5dac7c2dcd42f2682af1db80d4bce3faa126a6145c1637e1b559f",
+                "sha256:90990af1c3c67195c44c9a889184f84f5b2320dce3ee3acbd054e3ba0b4a7beb",
+                "sha256:a5b120bb3c0c71dfe27551f9da2f3209a8257a178ed6c628a819037a8df487f1",
+                "sha256:d8a82162f23c53b8525cf5f14a355f5d1eea86fa8edde27287dd3a98399e4fdf",
+                "sha256:f2018461733b23f308c298653c8903d32aaad7873d25e1d228765e91ae42c3f2",
+                "sha256:ff1977ba1a5f71f89166d5145c3da1cea89a0fdb044075a12c720ee9123ec818"
            ],
            "index": "pypi",
-            "version": "==5.6.7"
+            "version": "==5.7.2"
        },
        "pygments": {
            "hashes": [
-                "sha256:2a3fe295e54a20164a9df49c75fa58526d3be48e14aceba6d6b1e8ac0bfd6f1b",
-                "sha256:98c8aa5a9f778fcd1026a17361ddaf7330d1b7c62ae97c3bb0ae73e0b9b6b0fe"
+                "sha256:647344a061c249a3b74e230c739f434d7ea4d8b1d5f3721bc0f3558049b38f44",
+                "sha256:ff7a40b4860b727ab48fad6360eb351cc1b33cbf9b15a0f689ca5353e9463324"
            ],
-            "version": "==2.5.2"
+            "version": "==2.6.1"
        },
        "pyinstaller": {
            "hashes": [
@@ -280,25 +280,25 @@
        },
        "pylint": {
            "hashes": [
-                "sha256:3db5468ad013380e987410a8d6956226963aed94ecb5f9d3a28acca6d9ac36cd",
-                "sha256:886e6afc935ea2590b462664b161ca9a5e40168ea99e5300935f6591ad467df4"
+                "sha256:7dd78437f2d8d019717dbf287772d0b2dbdfd13fc016aa7faa08d67bccc46adc",
+                "sha256:d0ece7d223fe422088b0e8f13fa0a1e8eb745ebffcb8ed53d3e95394b6101a1c"
            ],
            "index": "pypi",
-            "version": "==2.4.4"
+            "version": "==2.5.3"
        },
        "pyparsing": {
            "hashes": [
-                "sha256:4c830582a84fb022400b85429791bc551f1f4871c33f23e44f353119e92f969f",
-                "sha256:c342dccb5250c08d45fd6f8b4a559613ca603b57498511740e65cd11a2e7dcec"
+                "sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1",
+                "sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"
            ],
-            "version": "==2.4.6"
+            "version": "==2.4.7"
        },
        "pytz": {
            "hashes": [
-                "sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d",
-                "sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be"
+                "sha256:a494d53b6d39c3c6e44c3bec237336e14305e4f29bbf800b599253057fbb79ed",
+                "sha256:c35965d010ce31b23eeb663ed3cc8c906275d6be1a34393a1d73a41febf4a048"
            ],
-            "version": "==2019.3"
+            "version": "==2020.1"
        },
        "pywin32-ctypes": {
            "hashes": [
@@ -310,18 +310,18 @@
        },
        "requests": {
            "hashes": [
-                "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4",
-                "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31"
+                "sha256:b3559a131db72c33ee969480840fff4bb6dd111de7dd27c8ee1f820f4f00231b",
+                "sha256:fe75cc94a9443b9246fc7049224f75604b113c36acb93f87b80ed42c44cbb898"
            ],
            "index": "pypi",
-            "version": "==2.22.0"
+            "version": "==2.24.0"
        },
        "six": {
            "hashes": [
-                "sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a",
-                "sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c"
+                "sha256:30639c035cdb23534cd4aa2dd52c3bf48f06e5f4a941509c8bafd8ce11080259",
+                "sha256:8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced"
            ],
-            "version": "==1.14.0"
+            "version": "==1.15.0"
        },
        "snowballstemmer": {
            "hashes": [
@@ -332,32 +332,32 @@
        },
        "sphinx": {
            "hashes": [
-                "sha256:298537cb3234578b2d954ff18c5608468229e116a9757af3b831c2b2b4819159",
-                "sha256:e6e766b74f85f37a5f3e0773a1e1be8db3fcb799deb58ca6d18b70b0b44542a5"
+                "sha256:b4c750d546ab6d7e05bdff6ac24db8ae3e8b8253a3569b754e445110a0a12b66",
+                "sha256:fc312670b56cb54920d6cc2ced455a22a547910de10b3142276495ced49231cb"
            ],
            "index": "pypi",
-            "version": "==2.3.1"
+            "version": "==2.4.4"
        },
        "sphinxcontrib-applehelp": {
            "hashes": [
-                "sha256:edaa0ab2b2bc74403149cb0209d6775c96de797dfd5b5e2a71981309efab3897",
-                "sha256:fb8dee85af95e5c30c91f10e7eb3c8967308518e0f7488a2828ef7bc191d0d5d"
+                "sha256:806111e5e962be97c29ec4c1e7fe277bfd19e9652fb1a4392105b43e01af885a",
+                "sha256:a072735ec80e7675e3f432fcae8610ecf509c5f1869d17e2eecff44389cdbc58"
            ],
-            "version": "==1.0.1"
+            "version": "==1.0.2"
        },
        "sphinxcontrib-devhelp": {
            "hashes": [
-                "sha256:6c64b077937330a9128a4da74586e8c2130262f014689b4b89e2d08ee7294a34",
-                "sha256:9512ecb00a2b0821a146736b39f7aeb90759834b07e81e8cc23a9c70bacb9981"
+                "sha256:8165223f9a335cc1af7ffe1ed31d2871f325254c0423bc0c4c7cd1c1e4734a2e",
+                "sha256:ff7f1afa7b9642e7060379360a67e9c41e8f3121f2ce9164266f61b9f4b338e4"
            ],
-            "version": "==1.0.1"
+            "version": "==1.0.2"
        },
        "sphinxcontrib-htmlhelp": {
            "hashes": [
-                "sha256:4670f99f8951bd78cd4ad2ab962f798f5618b17675c35c5ac3b2132a14ea8422",
-                "sha256:d4fd39a65a625c9df86d7fa8a2d9f3cd8299a3a4b15db63b50aac9e161d8eff7"
+                "sha256:3c0bc24a2c41e340ac37c85ced6dafc879ab485c095b1d65d2461ac2f7cca86f",
+                "sha256:e8f5bb7e31b2dbb25b9cc435c8ab7a79787ebf7f906155729338f3156d93659b"
            ],
-            "version": "==1.0.2"
+            "version": "==1.0.3"
        },
        "sphinxcontrib-jsmath": {
            "hashes": [
@@ -368,17 +368,24 @@
        },
        "sphinxcontrib-qthelp": {
            "hashes": [
-                "sha256:513049b93031beb1f57d4daea74068a4feb77aa5630f856fcff2e50de14e9a20",
-                "sha256:79465ce11ae5694ff165becda529a600c754f4bc459778778c7017374d4d406f"
+                "sha256:4c33767ee058b70dba89a6fc5c1892c0d57a54be67ddd3e7875a18d14cba5a72",
+                "sha256:bd9fc24bcb748a8d51fd4ecaade681350aa63009a347a8c14e637895444dfab6"
            ],
-            "version": "==1.0.2"
+            "version": "==1.0.3"
        },
        "sphinxcontrib-serializinghtml": {
            "hashes": [
-                "sha256:c0efb33f8052c04fd7a26c0a07f1678e8512e0faec19f4aa8f2473a8b81d5227",
-                "sha256:db6615af393650bf1151a6cd39120c29abaf93cc60db8c48eb2dddbfdc3a9768"
+                "sha256:eaa0eccc86e982a9b939b2b82d12cc5d013385ba5eadcc7e4fed23f4405f77bc",
+                "sha256:f242a81d423f59617a8e5cf16f5d4d74e28ee9a66f9e5b637a18082991db5a9a"
            ],
-            "version": "==1.1.3"
+            "version": "==1.1.4"
+        },
+        "toml": {
+            "hashes": [
+                "sha256:926b612be1e5ce0634a2ca03470f95169cf16f939018233a670519cb4ac58b0f",
+                "sha256:bda89d5935c2eac546d648028b9901107a595863cb36bae0c73ac804a9b4ce88"
+            ],
+            "version": "==0.10.1"
        },
        "typed-ast": {
            "hashes": [
@@ -404,28 +411,29 @@
                "sha256:fc0fea399acb12edbf8a628ba8d2312f583bdbdb3335635db062fa98cf71fca4",
                "sha256:fe460b922ec15dd205595c9b5b99e2f056fd98ae8f9f56b888e7a17dc2b757e7"
            ],
+            "markers": "implementation_name == 'cpython' and python_version < '3.8'",
            "version": "==1.4.1"
        },
        "typing-extensions": {
            "hashes": [
-                "sha256:091ecc894d5e908ac75209f10d5b4f118fbdb2eb1ede6a63544054bb1edb41f2",
-                "sha256:910f4656f54de5993ad9304959ce9bb903f90aadc7c67a0bef07e678014e892d",
-                "sha256:cf8b63fedea4d89bab840ecbb93e75578af28f76f66c35889bd7065f5af88575"
+                "sha256:6e95524d8a547a91e08f404ae485bbb71962de46967e1b71a0cb89af24e761c5",
+                "sha256:79ee589a3caca649a9bfd2a8de4709837400dfa00b6cc81962a1e6a1815969ae",
+                "sha256:f8d2bd89d25bc39dabe7d23df520442fa1d8969b82544370e03d88b5a591c392"
            ],
-            "version": "==3.7.4.1"
+            "version": "==3.7.4.2"
        },
        "urllib3": {
            "hashes": [
-                "sha256:2f3db8b19923a873b3e5256dc9c2dedfa883e33d87c690d9c7913e1f40673cdc",
-                "sha256:87716c2d2a7121198ebcb7ce7cccf6ce5e9ba539041cfbaeecfb641dc0bf6acc"
+                "sha256:91056c15fa70756691db97756772bb1eb9678fa585d9184f24534b100dc60f4a",
+                "sha256:e7983572181f5e1522d9c98453462384ee92a0be7fac5f1413a1e35c56cc0461"
            ],
-            "version": "==1.25.8"
+            "version": "==1.25.10"
        },
        "wrapt": {
            "hashes": [
-                "sha256:565a021fd19419476b9362b05eeaa094178de64f8361e44468f9e9d7843901e1"
+                "sha256:b62ffa81fb85f4332a4f609cab4ac40709470da05643a082ec1eb88e6d9b97d7"
            ],
-            "version": "==1.11.2"
+            "version": "==1.12.1"
        }
    }
 }
@@ -27,8 +27,9 @@
   :alt: Arch User Repository Package
   :target: https://aur.archlinux.org/packages/instaloader/

-.. |downloads| image:: https://img.shields.io/pypi/dm/instaloader.svg
+.. |downloads| image:: https://pepy.tech/badge/instaloader/month
   :alt: PyPI Download Count
+   :target: https://pepy.tech/project/instaloader/month

 .. badges-end

@@ -49,7 +50,9 @@
  directory accordingly,

 - allows **fine-grained customization** of filters and where to store
-  downloaded media.
+  downloaded media,
+
+- automatically **resumes previously-interrupted** download iterations.

 ::

@@ -17,7 +17,7 @@ import subprocess
 def __main():
    with contextlib.suppress(AttributeError, psutil.Error):
        if psutil.Process().parent().parent().name() == "explorer.exe":
-            subprocess.Popen("powershell -NoExit -Command \\\"& {0}\\\"".format(sys.argv[0]))
+            subprocess.Popen("powershell -NoExit -Command \\\"& '{0}'\\\"".format(sys.argv[0]))
            return
    main()

@@ -44,7 +44,7 @@ with open('__main__.py', 'w+') as f:
    f.writelines(lines)

 # install dependencies and invoke PyInstaller
-commands = ["pip install pipenv==2018.11.26",
+commands = ["pip install pipenv==2020.6.2",
            "pipenv sync --dev",
            "pipenv run pyinstaller --log-level=DEBUG instaloader.spec"]

@@ -1,76 +0,0 @@
-@import url(bootstrap-4.1.3.min.css);
-a {
-  color: #008d06; }
-  a:hover, a.current, a.active {
-    color: #f48400; }
-  a.current, a.active {
-    font-weight: 500; }
-
-.doc-sidebar a:hover, .modal-body a:hover {
-  text-decoration: none; }
-
-code {
-  color: #212529; }
-
-.highlight {
-  border-radius: 0.3em; }
-  .highlight pre {
-    padding: 0.7em;
-    color: #fff; }
-  .highlight .c1, .highlight .k, .highlight .kn, .highlight .ow {
-    color: #008d06; }
-  .highlight .nb, .highlight .ne, .highlight .nf, .highlight .vm {
-    color: #f48400; }
-  .highlight .s1, .highlight .s2, .highlight .sa, .highlight .se, .highlight .si, .highlight .mi {
-    color: #ff7efd; }
-  .highlight .c1, .highlight .nf, .highlight .sa, .highlight .se, .highlight .si {
-    font-style: italic; }
-  .highlight .k, .highlight .kn, .highlight .ow {
-    font-weight: bold; }
-
-.doc-content {
-  padding-top: 0.5em;
-  top: 3.5625em;
-  position: fixed;
-  bottom: 0;
-  overflow-y: auto; }
-
-.doc-sidebar {
-  position: fixed;
-  padding-top: 0.5em;
-  top: 3.5625em;
-  bottom: 0;
-  right: 0;
-  overflow-y: auto; }
-  .doc-sidebar ul {
-    list-style-type: none;
-    padding-left: 1em; }
-    .doc-sidebar ul.current {
-      padding-left: 0; }
-  .doc-sidebar li {
-    padding-top: 0.2em;
-    padding-bottom: 0.2em; }
-
-.admonition {
-  border-radius: 0.3em;
-  border: 1px solid #ccc;
-  background-color: #f8f9fa; }
-  .admonition .admonition-title {
-    color: #ff7efd; }
-
-.topic {
-  border-radius: 0.3em;
-  background-color: #f8f9fa; }
-
-.nav-link {
-  padding-top: 0;
-  padding-bottom: 0; }
-
-.btn-success {
-  background-color: #008d06;
-  border-color: #008d06; }
-  .btn-success:hover {
-    background-color: #f48400;
-    border-color: #f48400; }
-
-/*# sourceMappingURL=instaloader.css.map */
@@ -1,22 +0,0 @@
-$(function () {
-    $('[data-toggle="tooltip"]').tooltip();
-
-    $('.doc-sidebar > ul > li.current').attr("id", "localtoc");
-    $('#localtoc ul').addClass("nav flex-column");
-    $('#localtoc ul li').addClass("nav-item");
-    $('#localtoc ul li a').addClass("nav-link");
-    $('.doc-content').scrollspy({target: '#localtoc'});
-
-    const top_href = '#' + $('.section:first').attr("id");
-    $('#localtoc > a.current').attr("href", top_href);
-
-    $('#navbarToc a').on("click", function () {
-        const href = $(this).attr("href");
-        if (href === '#') {
-            window.location.href = top_href;
-        } else {
-            window.location.href = href;
-        }
-        $('#navbarToc').modal('hide');
-    });
-});
@@ -1,120 +0,0 @@
-@import 'bootstrap-4.1.3.min.css';
-
-$color_instaloader_main: #008d06;
-$color_instaloader_accent: #f48400;
-$color_instaloader_hyper: #ff7efd;
-$color_normal_text: #212529;
-$color_light_bg: #f8f9fa;
-
-a {
-  color: $color_instaloader_main;
-
-  &:hover, &.current, &.active {
-    color: $color_instaloader_accent;
-  }
-
-  &.current, &.active {
-    font-weight: 500;
-  }
-}
-
-.doc-sidebar, .modal-body {
-  a:hover {
-    text-decoration: none
-  }
-}
-
-code {
-  color: $color_normal_text;
-}
-
-.highlight {
-  border-radius: 0.3em;
-
-  pre {
-    padding: 0.7em;
-    color: #fff;
-  }
-
-  .c1, .k, .kn, .ow {
-    color: $color_instaloader_main
-  }
-
-  .nb, .ne, .nf, .vm {
-    color: $color_instaloader_accent
-  }
-
-  .s1, .s2, .sa, .se, .si, .mi {
-    color: $color_instaloader_hyper
-  }
-
-  .c1, .nf, .sa, .se, .si {
-    font-style: italic;
-  }
-
-  .k, .kn, .ow {
-    font-weight: bold
-  }
-}
-
-.doc-content {
-  padding-top: 0.5em;
-  top: 3.5625em;
-  position: fixed;
-  bottom: 0;
-  overflow-y: auto;
-}
-
-.doc-sidebar {
-  position: fixed;
-  padding-top: 0.5em;
-  top: 3.5625em;
-  bottom: 0;
-  right: 0;
-  overflow-y: auto;
-
-  ul {
-    list-style-type: none;
-    padding-left: 1em;
-
-    &.current {
-      padding-left: 0;
-    }
-  }
-
-  li {
-    padding-top: 0.2em;
-    padding-bottom: 0.2em;
-
-  }
-}
-
-.admonition {
-  border-radius: 0.3em;
-  border: 1px solid #ccc;
-  background-color: $color_light_bg;
-
-  .admonition-title {
-    color: $color_instaloader_hyper;
-  }
-}
-
-.topic {
-  border-radius: 0.3em;
-  background-color: $color_light_bg;
-}
-
-.nav-link {
-  padding-top: 0;
-  padding-bottom: 0;
-}
-
-.btn-success {
-  background-color: $color_instaloader_main;
-  border-color: $color_instaloader_main;
-
-  &:hover {
-    background-color: $color_instaloader_accent;
-    border-color: $color_instaloader_accent;
-  }
-}
@@ -1,9 +1,6 @@
 {% extends "!layout.html" %}
-
 {% set html5_doctype = True %}
-
-{% set script_files = script_files + ['_static/bootstrap-4.1.3.bundle.min.js', '_static/instaloader.js'] %}
-
+{% set script_files = script_files + ['_static/instaloaderdoc.js'] %}
 {% block htmltitle %}
  {% if pagename == "index" %}
    <title>{{ title|striptags|e }}</title>
@@ -11,7 +8,6 @@
    <title>{{ title|striptags|e }}{{ titlesuffix }}</title>
  {% endif %}
 {% endblock %}
-
 {% block extrahead %}
  {{ super() }}
  <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
@@ -22,94 +18,66 @@
    <link rel="canonical" href="https://instaloader.github.io/{{ pagename }}.html"/>
  {% endif %}
 {% endblock %}
-
 {% block header %}
-  <nav class="navbar navbar-expand-sm navbar-light bg-light fixed-top border-bottom">
+  <nav class="navbar navbar-light navbar-expand bg-light border-bottom">
    <a class="navbar-brand" href="{{ pathto(master_doc) }}">
      <img src="{{ pathto('_static/' + logo, 1) }}" width="30" height="30" class="d-inline-block align-top" alt="logo">
-      Instaloader
+      <span class="ml-3">Instaloader</span>
    </a>
    <ul class="navbar-nav mr-auto">
-      <li class="nav-item ml-0 ml-sm-2 ml-md-0">
-        <a target="_blank" class="nav-link"
+      <li class="nav-item ml-0 ml-sm-2">
+        <a class="nav-link"
           href="https://github.com/instaloader/instaloader/releases/tag/v{{ current_release }}"
-           title="Released on {{ current_release_date }}" data-toggle="tooltip">v<b>{{ current_release }}</b></a>
+           title="Released on {{ current_release_date }}" data-toggle="tooltip">
+          v<b>{{ current_release }}</b>
+        </a>
      </li>
    </ul>
-    <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarLinks"
-            aria-controls="navbarLinks" aria-expanded="false" aria-label="Toggle navigation">
+    <ul class="navbar-nav ml-auto d-none d-sm-flex">
+      <li class="nav-item">
+        <a class="nav-link" href="https://github.com/instaloader/instaloader/issues">
+          Issues
+        </a>
+      </li>
+      <li class="nav-item">
+        <a class="nav-link" href="https://github.com/instaloader/instaloader/releases">
+          Releases
+        </a>
+      </li>
+      <li class="nav-item">
+        <a class="nav-link" href="https://github.com/instaloader/instaloader">
+          GitHub
+        </a>
+      </li>
+    </ul>
+    <button class="d-inline-block d-md-none btn btn-outline-primary btn-sm" type="button"
+            data-toggle="collapse" data-target="#instaloaderdocSidebar" aria-controls="instaloaderdocSidebar"
+            aria-expanded="false" aria-label="Toggle navigation">
      <span class="navbar-toggler-icon"></span>
    </button>
-    <div class="collapse navbar-collapse" id="navbarLinks">
-      <ul class="navbar-nav ml-auto">
-        <li class="nav-item">
-          <a target="_blank" class="nav-link" href="https://github.com/instaloader/instaloader/issues">Issues</a>
-        </li>
-        <li class="nav-item">
-          <a target="_blank" class="nav-link" href="https://github.com/instaloader/instaloader/releases">Releases</a>
-        </li>
-        <li class="nav-item">
-          <a target="_blank" class="nav-link" href="https://github.com/instaloader/instaloader">GitHub</a>
-        </li>
-      </ul>
-      <form class="form-inline" action="search.html" method="get">
-        <div class="input-group my-2 my-sm-0 ml-2">
-          <input class="form-control" type="search" placeholder="Search" aria-label="Search" name="q">
-          <div class="input-group-append">
-            <button class="btn btn-success btn-small" type="submit">Go</button>
-          </div>
-        </div>
-      </form>
-    </div>
-    {% if pagename != "index" %}
-      <button type="button" class="btn btn-success d-md-none btn-small ml-2" data-toggle="modal"
-              data-target="#navbarToc">
-        TOC
-      </button>
-    {% endif %}
  </nav>
-  {% if pagename != "index" %}
-    <div class="modal fade" id="navbarToc" tabindex="-1" role="dialog"
-         aria-hidden="true" aria-labelledby="navbarTocTitle">
-      <div class="modal-dialog modal-dialog-centered" role="document">
-        <div class="modal-content">
-          <div class="modal-header">
-            <h5 class="modal-title" id="navbarTocTitle">Contents</h5>
-            <button type="button" class="close" data-dismiss="modal" aria-label="Close">
-              <span aria-hidden="true">&times;</span>
-            </button>
-          </div>
-          <div class="modal-body">
-            {{ toctree(maxdepth=-1) }}
-          </div>
-        </div>
-      </div>
-    </div>
-  {% endif %}
 {% endblock %}
-
 {% block relbar1 %}{% endblock %}
 {% block relbar2 %}{% endblock %}
 {% block footer %}{% endblock %}
 {% block content %}
-  <div class="container-fluid row">
-    <div class="col-12 {% if pagename != "index" %}col-md-9 col-xl-10{% endif %} doc-content">
-      {% block body %}{% endblock %}
-    </div>
-    {% if pagename != "index" %}
-      <div class="d-none col-md-3 col-xl-2 bg-light doc-sidebar d-md-flex flex-column border-left">
-
-        <h3>{{ _('Table of Contents') }}</h3>
-        {{ toctree(maxdepth=-1) }}
-
-        <div class="d-flex justify-content-between small mt-auto mb-2">
-          <a class="reference internal{% if pagename == "genindex" %} current{% endif %}" href="genindex.html">Index</a>
-          {% if pagename != "genindex" and pagename != "search" %}
-            <a class="reference external"
-               href="https://github.com/instaloader/instaloader/edit/master/docs/{{ pagename }}.rst">Edit this page</a>
-          {% endif %}
-        </div>
+  <div class="container-fluid">
+    <div class="row">
+      <div class="instaloaderdoc-sidebar collapse d-md-block col-md-3 col-xl-2 bg-light"
+           id="instaloaderdocSidebar">
+        <nav role="navigation" aria-label="Table of Contents">
+          <form action="{{ pathto('search') }}" method="get" role="search" class="border-bottom mr-2 pb-2 mb-2 px-1">
+            <label class="sr-only" for="searchQueryInput">Search</label>
+            <input type="search" name="q" id="searchQueryInput" class="form-control my-2"
+                   placeholder="Search..." aria-label="Search">
+          </form>
+          <h4>Contents</h4>
+          {{ toctree(maxdepth=-1) }}
+        </nav>
      </div>
-    {% endif %}
+      <main class="col-12 col-md-9 col-xl-10 pt-2" role="main">
+        {% block body %}{% endblock %}
+      </main>
+    </div>
  </div>
-{% endblock %}
+{% endblock %}
@@ -12,11 +12,8 @@ Python Module :mod:`instaloader`

 .. highlight:: python

-.. contents::
-  :backlinks: none
-
 Instaloader exposes its internally used methods and structures as a Python
-module, making it a **powerful and intuitive Python API for Instagram**,
+module, making it a powerful and intuitive Python API for Instagram,
 allowing to further customize obtaining media and metadata.

 Start with getting an instance of :class:`Instaloader`::
@@ -85,149 +82,26 @@ metadata of a Profile. :class:`Profile` instances can be created with:
 - :meth:`Profile.get_followers`
   Profiles that follow given user.

- :attr:`Post.owner_profile`, :attr:`Story.owner_profile` and :attr:`StoryItem.owner_profile`
+- :attr:`Post.owner_profile`, :attr:`Story.owner_profile`, :attr:`StoryItem.owner_profile`
   Owner profile of particular object.

 - :meth:`Post.get_likes`
   Profiles that liked a given :class:`Post`

- :attr:`PostComment.owner` attribute for comment in :meth:`Post.get_comments`
+- :attr:`PostComment.owner`
   Profile of a Post comment.

-A reference of the many methods provided by the :mod:`instaloader` module is
-provided in the remainder of this document.
+For a list of a few code examples that use the Instaloader module, see
+:ref:`codesnippets`.

-For a list of real code examples using the Instaloader module for advanced
-tasks, see :ref:`codesnippets`.
+The reference of the classes and functions provided by the :mod:`instaloader` module is
+divided into the following subsections:

-``Instaloader`` (Main Class)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. toctree::
+   :maxdepth: 2

-.. autoclass:: Instaloader
-   :no-show-inheritance:
-
-Instagram Structures
-^^^^^^^^^^^^^^^^^^^^
-
-Posts
-"""""
-
-.. autoclass:: Post
-   :no-show-inheritance:
-
-Additionally, the following trivial structures are defined:
-
-.. autoclass:: PostSidecarNode
-   :no-show-inheritance:
-
-.. autoclass:: PostComment
-   :no-show-inheritance:
-
-.. autoclass:: PostCommentAnswer
-   :no-show-inheritance:
-
-.. autoclass:: PostLocation
-   :no-show-inheritance:
-
-User Stories
-""""""""""""
-
-.. autoclass:: Story
-   :no-show-inheritance:
-
-.. autoclass:: StoryItem
-   :no-show-inheritance:
-
-Highlights
-""""""""""
-
-.. autoclass:: Highlight
-   :no-show-inheritance:
-   :inherited-members:
-
-   Bases: :class:`Story`
-
-   .. versionadded:: 4.1
-
-Profiles
-""""""""
-
-.. autoclass:: Profile
-   :no-show-inheritance:
-
-Hashtags
-""""""""
-
-.. autoclass:: Hashtag
-   :no-show-inheritance:
-
-   .. versionadded:: 4.4
-
-TopSearchResults
-""""""""""""""""
-
-.. autoclass:: TopSearchResults
-   :no-show-inheritance:
-
-   .. versionadded:: 4.3
-
-Loading and Saving
-""""""""""""""""""
-
-:class:`Post`, :class:`StoryItem`, :class:`Profile` and :class:`Hashtag` can be saved and loaded
-to/from JSON files.
-
-.. autofunction:: load_structure_from_file
-
-.. autofunction:: save_structure_to_file
-
-Exceptions
-^^^^^^^^^^
-
-.. currentmodule:: instaloader.exceptions
-
-.. autoexception:: InstaloaderException
-   :no-show-inheritance:
-
-.. autoexception:: ConnectionException
-
-.. currentmodule:: instaloader
-
-.. autoexception:: QueryReturnedBadRequestException
-
-.. autoexception:: QueryReturnedForbiddenException
-
-.. autoexception:: ProfileNotExistsException
-
-.. autoexception:: ProfileHasNoPicsException
-
-.. autoexception:: PrivateProfileNotFollowedException
-
-.. autoexception:: LoginRequiredException
-
-.. autoexception:: TwoFactorAuthRequiredException
-
-   .. versionadded:: 4.2
-
-.. autoexception:: InvalidArgumentException
-
-.. autoexception:: BadResponseException
-
-.. autoexception:: BadCredentialsException
-
-.. autoexception:: PostChangedException
-
-.. autoexception:: QueryReturnedNotFoundException
-
-   .. versionchanged:: 4.3
-      QueryReturnedNotFoundException now inherits ConnectionException
-      to retry on 404 errors.
-
-
-.. autoexception:: TooManyRequestsException
-
-``InstaloaderContext`` (Low-level functions)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. autoclass:: InstaloaderContext
-   :no-show-inheritance:
+   module/instaloader
+   module/structures
+   module/nodeiterator
+   module/instaloadercontext
+   module/exceptions
@@ -77,7 +77,7 @@ Instaloader supports the following targets:
      instructs Instaloader to also **download the user's stories**,

   - :option:`--highlights`
-      to **download highlights of each profile that is downloaded**,
+      to **download the highlights of that profile**,

   - :option:`--tagged`
      to **download posts where the user is tagged**, and
@@ -86,7 +86,7 @@ Instaloader supports the following targets:
      to **download IGTV videos**.

 - ``"#hashtag"``
-   Posts with a certain **hashtag** (the quotes are usually necessary),
+   Posts with a certain **hashtag** (the quotes are usually necessary).

 - ``%location id``
   Posts tagged with a given location; the location ID is the numerical ID
@@ -98,21 +98,25 @@ Instaloader supports the following targets:

 - ``:stories``
   The currently-visible **stories** of your followees (requires
-   :option:`--login`),
+   :option:`--login`).

 - ``:feed``
-   Your **feed** (requires :option:`--login`),
+   Your **feed** (requires :option:`--login`).

 - ``:saved``
-   Posts which are marked as **saved** (requires :option:`--login`),
+   Posts which are marked as **saved** (requires :option:`--login`).

 - ``@profile``
   All profiles that are followed by ``profile``, i.e. the *followees* of
   ``profile`` (requires :option:`--login`).

 - ``-post``
-   The single **post** with the given shortcode. Must be preceeded by ``--`` in
-   the argument list to not be mistaken as an option flag.
+   Replace **post** with the post's shortcode to download single post. Must be preceded by ``--`` in
+   the argument list to not be mistaken as an option flag::
+
+    instaloader -- -B_K4CykAOtf
+
+

   .. versionadded:: 4.1

@@ -192,7 +196,7 @@ Filter Posts
 .. py:currentmodule:: instaloader

 The options :option:`--post-filter` and :option:`--storyitem-filter`
-allows to specify criteria that posts or story items have to
+allow to specify criteria that posts or story items have to
 meet to be downloaded. If not given, all posts are downloaded.

 The filter string must be a
@@ -204,7 +208,7 @@ Id est, the following attributes can be used with both
 :option:`--post-filter` and :option:`--storyitem-filter`:

 - :attr:`~Post.owner_username` (str), :attr:`~Post.owner_id` (int)
-   Owner profile username / userid.
+   Owner profile username / user ID.

 - :attr:`~Post.date_utc` (datetime), :attr:`~Post.date_local` (datetime)
   Creation timestamp. Since :class:`~datetime.datetime` objects can be created
@@ -213,7 +217,7 @@ Id est, the following attributes can be used with both
      instaloader --post-filter="date_utc <= datetime(2018, 5, 31)" target

 - :attr:`~Post.is_video` (bool)
-   Post/StoryItem is a video. For example, you may skip videos::
+   Whether Post/StoryItem is a video. For example, you may skip videos::

      instaloader --post-filter="not is_video" target

@@ -230,8 +234,8 @@ As :option:`--post-filter`, the following attributes can be used additionally:
      instaloader --login=your_username --post-filter=viewer_has_liked :feed

 - :attr:`~Post.likes` (int), :attr:`~Post.comments` (int)
-   Likes count / Comments count. You might only want to download posts that
-   either you liked or were liked by many others**::
+   Likes count / comments count. You might only want to download posts that
+   were either liked by yourself or by many others::

      instaloader --login=your_username --post-filter="likes>100 or viewer_has_liked" profile

@@ -261,7 +265,7 @@ along with each post where the Post's caption is saved.
 You can customize what metadata to save for each Post or StoryItem with
 :option:`--post-metadata-txt` and :option:`--storyitem-metadata-txt`. The
 default is ``--post-metadata-txt={caption}`` and no storyitem metadata txt.
-These strings are formatted similar as in the :ref:`filename-specification` and
+These strings are formatted similar as the path patterns described in :ref:`filename-specification` and
 the result is saved in text files, unless it is empty.

 Specifying these options multiple times results in output having multiple lines,
@@ -295,10 +299,10 @@ Instaloader logged-in, create a session file::

   instaloader --login=your_username

-Then use the same parameter in your cronjob to load the session and download
+Then use the same username in your cronjob to load the session and download
 the given targets::

-   instaloader --login=your_username --quiet <target> [...]
+   instaloader --login=your_username --quiet target [...]

 Instaloader saves the session file to
 ``~/.config/instaloader/session-YOUR-USERNAME``. See
@@ -313,5 +317,6 @@ Instaloader exposes its internally used methods and structures, making it a
 powerful and intuitive Python API for Instagram, allowing to further customize
 obtaining media and metadata.

-Also see :ref:`codesnippets`, where we collect example scripts that use
-Instaloader to achieve more complex tasks.
+Also see :ref:`codesnippets`, where we collect a few example scripts that use
+Instaloader for simple tasks that cannot be done with the command line
+interface.
@@ -36,7 +36,7 @@ stores all posts along with the pictures's captions there.
   Re-Download the given object

 - ``+args.txt``
-   Read targets (and options) from given textfile. See :option:`+args.txt`.
+   Read targets (and options) from given text file. See :option:`+args.txt`.

 What to Download of each Post
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -59,7 +59,7 @@ What to Download of each Post

 .. option:: --geotags, -G

-   **Download geotags** when available. Geotags are stored as a text file with
+   Download geotags when available. Geotags are stored as a text file with
   the location's name and a Google Maps link. This requires an additional
   request to the Instagram server for each picture. Requires :option:`--login`.

@@ -94,13 +94,6 @@ What to Download of each Post
 What to Download of each Profile
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

-.. option:: --profile-pic-only, -P
-
-   .. deprecated:: 4.1
-      Use :option:`--no-posts`.
-
-   Only download profile picture.
-
 .. option:: --no-posts

   Do not download regular posts.
@@ -113,12 +106,12 @@ What to Download of each Profile

 .. option:: --stories, -s

-   Also **download stories** of each profile that is downloaded. Requires
+   Also download stories of each profile that is downloaded. Requires
   :option:`--login`.

 .. option:: --highlights

-   Also **download highlights** of each profile that is downloaded. Requires
+   Also download highlights of each profile that is downloaded. Requires
   :option:`--login`.

   .. versionadded:: 4.1
@@ -135,20 +128,6 @@ What to Download of each Profile

   .. versionadded:: 4.3

-.. option:: --stories-only
-
-   .. deprecated:: 4.1
-      Use :option:`--stories` :option:`--no-posts`.
-
-   Rather than downloading regular posts of each specified profile, only
-   download stories.  Requires :option:`--login`. Does not imply
-   :option:`--no-profile-pic`.
-
-   .. note::
-
-      If possible, use ``:stories`` target rather than :option:`--stories-only`
-      with all your followees. ``:stories`` uses fewer API requests.
-
 Which Posts to Download
 ^^^^^^^^^^^^^^^^^^^^^^^

@@ -182,7 +161,7 @@ Which Posts to Download
 Login (Download Private Profiles)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

-Instaloader can **login to Instagram**. This allows downloading private
+Instaloader can login to Instagram. This allows downloading private
 profiles. To login, pass the :option:`--login` option. Your session cookie (not your
 password!) will be saved to a local file to be reused next time you want
 Instaloader to login.
@@ -223,6 +202,29 @@ How to Download
   ``#hashtag`` or the profile name. Defaults to ``{date_utc}_UTC``.
   See :ref:`filename-specification` for a list of supported tokens.

+.. option:: --resume-prefix prefix
+
+   For many targets, Instaloader is capable of resuming a previously-aborted
+   download loop.  To do so, it creates a JSON file within the target directory
+   when interrupted.  This option controls the prefix for filenames that are
+   used to save the information to resume an interrupted download.  The default
+   prefix is ``iterator``.
+
+   Resuming an interrupted download is supported for most, but not all targets.
+   JSON files with resume information are always compressed, regardless of
+   :option:`--no-compress-json`.
+
+   This feature is turned off entirely with :option:`--no-resume`.
+
+   .. versionadded:: 4.5
+
+.. option:: --no-resume
+
+   Do not resume a previously-aborted download iteration, and do not save such
+   information when interrupted.
+
+   .. versionadded:: 4.5
+
 .. option:: --user-agent USER_AGENT

   User Agent to use for HTTP requests. Per default, Instaloader pretends being
@@ -234,19 +236,6 @@ How to Download
   to ``3``. If a connection fails, it can be manually skipped by hitting
   :kbd:`Control-c`. Set this to ``0`` to retry infinitely.

-.. option:: --commit-mode
-
-   Tries to ensure downloaded images avoid corruption in case of unexpected
-   interruption. If the last picture is corrupted, Instaloader will fix the
-   picture the next time it is run.
-
-   .. versionadded:: 4.2
-
-   .. warning::
-      We discourage from using the "commit mode" due to its bugs when being used
-      with other command line options (e.g. :issue:`257` with
-      :option:`--fast-update` and :issue:`483` with :option:`--tagged`).
-
 .. option:: --request-timeout N

   Seconds to wait before timing out a connection request.
@@ -266,18 +255,18 @@ Miscellaneous Options

   Read arguments from file `args.txt`, a shortcut to provide arguments from
   file rather than command-line. This provides a convenient way to hide login
-   info from CLI, and can also be used to simplify managment of long arguments.
+   info from CLI, and can also be used to simplify management of long arguments.

   .. note::

      Text file should separate arguments with line breaks.

-      args.txt example::
+   args.txt example::

-         --login=MYUSERNAME
-         --password=MYPASSWORD
-         --fast-update
-         profile1
-         profile2
+      --login=MYUSERNAME
+      --password=MYPASSWORD
+      --fast-update
+      profile1
+      profile2

   .. versionadded:: 4.1
@@ -28,8 +28,9 @@ Download Posts in a Specific Period
 -----------------------------------

 To only download Instagram pictures (and metadata) that are within a specific
-period, you can play around with :func:`~itertools.dropwhile` and
-:func:`~itertools.takewhile` from :mod:`itertools` like in this snippet.
+period, you can simply use :func:`~itertools.dropwhile` and
+:func:`~itertools.takewhile` from :mod:`itertools` on a generator that returns
+Posts in **exact chronological order**, such as :meth:`Profile.get_posts`.

 .. literalinclude:: codesnippets/121_since_until.py

@@ -37,11 +38,19 @@ See also :class:`Post`, :meth:`Instaloader.download_post`.

 Discussed in :issue:`121`.

+The code example with :func:`~itertools.dropwhile` and
+:func:`~itertools.takewhile` makes the assumption that the post iterator returns
+posts in exact chronological order.  As discussed in :issue:`666`, the following
+approach fits for an **almost chronological order**, where up to *k* older posts
+are inserted into an otherwise chronological order, such as a Hashtag feed.
+
+.. literalinclude:: codesnippets/666_historical_hashtag_data.py
+
 Likes of a Profile / Ghost Followers
 ------------------------------------

 To obtain a list of your inactive followers, i.e. followers that did not like
-any of your pictures, into a file you can use this approach.
+any of your pictures, you can use this approach.

 .. literalinclude:: codesnippets/120_ghost_followers.py

@@ -69,11 +78,11 @@ Discussed in :issue:`56`.
 Only one Post per User
 ----------------------

-To download only the one most recent post from each user, this snippet creates a
-:class:`set` that contains the users of which a post has already been
-downloaded. While iterating the posts, it checks whether the post's owner
-already is in the set. If not, the post is downloaded from Instagram and the
-user is added to that set.
+To download only the single most recent post per user within a hashtag feed,
+this snippet uses a :class:`set` that contains the users of whom a post has
+already been downloaded. For each post, it checks whether the post's creator is
+already contained in that set. If not, the post is downloaded from Instagram and
+the user is added to that set.

 .. literalinclude:: codesnippets/113_only_one_per_user.py

@@ -85,37 +94,12 @@ Discussed in :issue:`113`.
 Top X Posts of User
 -------------------

-With Instaloader, it is easy to download the few most-liked pictres of a user.
+With Instaloader, it is easy to download the few most-liked pictures of a user.

 .. literalinclude:: codesnippets/194_top_x_of_user.py

 Discussed in :issue:`194`.

-Upgrade Images by Local Copies
------------------------------
-
-The following script finds local versions of images fetched by Instaloader, in
-order to upgrade the downloaded images by locally-found versions with better
-quality. It uses image hashing to identify similar images.
-
-`updgrade-instaloader-images.py <https://gist.github.com/pavelkryukov/15f93d19a99428a284a8bcec27e0187b>`__ (external link to GitHub Gist)
-
-Discussed in :issue:`46`.
-
-Add Captions to Images
----------------------
-
-Instaloader does not modify the downloaded JPEG file. However, one could combine
-it with an imaging library such as Pillow or PIL to render the caption on
-Instagram pictures.  The following shows an approach.
-
-.. literalinclude:: codesnippets/110_pil_captions.py
-
-See also :attr:`Post.caption`, :attr:`Post.url`, :meth:`Post.from_shortcode`,
-:func:`load_structure_from_file`.
-
-Discussed in :issue:`110`.
-
 Metadata JSON Files
 -------------------

@@ -1,20 +0,0 @@
-from io import BytesIO
-
-from requests import get
-from PIL import Image, ImageDraw
-from instaloader import *
-
-L = Instaloader()
-
-# Load Post instance
-post = load_structure_from_file(L.context, '2017-10-01_18-53-03_UTC.json.xz')
-# or post = Post.from_shortcode(L.context, SHORTCODE)
-
-# Render caption
-image = Image.open(BytesIO(get(post.url).content))
-draw = ImageDraw.Draw(image)
-color = 'rgb(0, 0, 0)'  # black color
-draw.text((300,100), post.caption.encode('latin1', errors='ignore'), fill=color)
-
-# Save image
-image.save('test.jpg')
@@ -2,26 +2,26 @@ import instaloader

 L = instaloader.Instaloader()

-USER = 'your_account'
+USER = "your_account"
 PROFILE = USER

-# Your preferred way of logging in:
+# Load session previously saved with `instaloader -l USERNAME`:
 L.load_session_from_file(USER)

 profile = instaloader.Profile.from_username(L.context, PROFILE)

 likes = set()
-print('Fetching likes of all posts of profile {}.'.format(profile.username))
+print("Fetching likes of all posts of profile {}.".format(profile.username))
 for post in profile.get_posts():
    print(post)
    likes = likes | set(post.get_likes())

-print('Fetching followers of profile {}.'.format(profile.username))
+print("Fetching followers of profile {}.".format(profile.username))
 followers = set(profile.get_followers())

 ghosts = followers - likes

-print('Storing ghosts into file.')
-with open('/YOUR PATH/inactive-users.txt', 'w') as f:
+print("Storing ghosts into file.")
+with open("inactive-users.txt", 'w') as f:
    for ghost in ghosts:
        print(ghost.username, file=f)
@@ -5,13 +5,11 @@ import instaloader

 L = instaloader.Instaloader()

-posts = instaloader.Hashtag.from_name(L.context, 'urbanphotography').get_posts()
-# or
-# posts = instaloader.Profile.from_username(L.context, PROFILE).get_posts()
+posts = instaloader.Profile.from_username(L.context, "instagram").get_posts()

 SINCE = datetime(2015, 5, 1)
 UNTIL = datetime(2015, 3, 1)

 for post in takewhile(lambda p: p.date > UNTIL, dropwhile(lambda p: p.date > SINCE, posts)):
    print(post.date)
-    L.download_post(post, '#urbanphotography')
+    L.download_post(post, "instagram")
@@ -0,0 +1,31 @@
+from datetime import datetime
+import instaloader
+
+L = instaloader.Instaloader()
+
+posts = instaloader.Hashtag.from_name(L.context, "urbanphotography").get_posts()
+
+SINCE = datetime(2020, 5, 10)  # further from today, inclusive
+UNTIL = datetime(2020, 5, 11)  # closer to today, not inclusive
+
+k = 0  # initiate k
+#k_list = []  # uncomment this to tune k
+
+for post in posts:
+    postdate = post.date
+
+    if postdate > UNTIL:
+        continue
+    elif postdate <= SINCE:
+        k += 1
+        if k == 50:
+            break
+        else:
+            continue
+    else:
+        L.download_post(post, "#urbanphotography")
+        # if you want to tune k, uncomment below to get your k max
+        #k_list.append(k)
+        k = 0  # set k to 0
+
+#max(k_list)
@@ -18,6 +18,7 @@
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
 import os
+import platform
 import subprocess
 import sys
 import typing
@@ -49,9 +50,12 @@ autodoc_member_order = 'bysource'
 intersphinx_mapping = {'python': ('https://docs.python.org/3', None),
                       'requests': ('https://requests.kennethreitz.org/en/master/', None)}

+nitpick_ignore = [('py:class', 'typing.Tuple')]
+
 current_release = subprocess.check_output(["git", "describe", "--abbrev=0"]).decode("ascii")[1:-1]
+date_format = "%e %b %Y" if platform.system() != "Windows" else "%d %b %Y"
 current_release_date = subprocess.check_output(
-    ["git", "log", "-1", "--tags", "--format=%ad", "--date=format:%e %b %Y"]).decode("ascii")[:-1]
+    ["git", "log", "-1", "--tags", "--format=%ad", "--date=format:"+date_format]).decode("ascii")[:-1]

 html_context = {'current_release': current_release, 'current_release_date': current_release_date}

@@ -370,6 +374,6 @@ texinfo_documents = [

 def setup(app):
    typing.TYPE_CHECKING = True
-    app.add_stylesheet("instaloader.css")
+    app.add_stylesheet("instaloaderdoc.css")
    app.connect('autodoc-process-signature', sphinx_autodoc_typehints.process_signature)
    app.connect('autodoc-process-docstring', sphinx_autodoc_typehints.process_docstring)
@@ -35,8 +35,8 @@ reporting a problem, please keep the following in mind:
 - If not obvious, describe **which behavior you expected**
  instead of what actually happened.

- If you think an issue has been closed accidentally or inappropriately, feel
-  free to reopen it.
+- If we have closed an issue apparently inadvertently or inappropriately, please
+  let us know.

 Writing Code or Improving the Documentation
 -------------------------------------------
@@ -45,28 +45,25 @@ Changes of the Instaloader source can be proposed as a
 `Pull Request <https://github.com/instaloader/instaloader/pulls>`__. There are only
 few things to consider:

- Sometimes, the most current code is not in the ``master`` branch. Check that
-  you forked from the most recent branch.
+- Base your Pull Request on the ``master`` branch if it fixes a bug,
+  or the ``upcoming/v4.X`` branch (if it exists at the moment of submitting the PR)
+  otherwise.

 - We use `Pylint <https://www.pylint.org/>`__ for error and syntax checking of
-  the source. The file ``.travis.yml`` in the project's root directory
-  shows how it is invoked. Note that sometimes it might be better to disable a
-  warning rather than adapting the code to Pylint's desires.
+  the source and `MyPy <https://github.com/python/mypy>`__ for type checking.
+  Beware that sometimes it might be better to disable a warning rather than
+  adapting the code to a tool's desires.

 - The documentation source is located in the ``docs`` folder. The file
  ``cli-options.rst`` is merely an RST-formatted copy of ``instaloader --help``
  output, of which the source is in ``instaloader/__main__.py``.

- Feel free to contact us, even if you "only" have Proof-of-Concepts or
-  not-fully integrated changes. They already might be an advance for the
-  project.
-
-Suggesting Features
-------------------
+Proposing Features
+------------------

 .. goal-start

-Instaloader's goal is to mimick the browser's behavior to access the data that
+Instaloader's goal is to mimic the browser's behavior to access the data that
 is available through the Instagram web interface, pack this data into complete
 and easily-(re)usable python objects, and provide a user interface for the most
 common downloading and metadata collection tasks, without changing any of the
@@ -74,10 +71,9 @@ online data.

 .. goal-end

-If you have an idea of how Instaloader should be enhanced, but do not want to
-implement the feature yourself, feel free to open a ticket in our 
-`Issue Tracker <https://github.com/instaloader/instaloader/issues>`__.
-Please consider the following:
+Prior spending effort on implementing a new feature, it might be appropriate to
+clarify how it could fit into the project's scope or discuss implementation
+details. If you feel the need to do so, please create a "feature suggestion".

 - Instaloader already has plenty of features. **Check the documentation**
  beforehand to ensure your desired suggestion is not already implemented.
@@ -86,16 +82,21 @@ Please consider the following:
  an issue suggesting the same or a similar feature, share your thoughts in a
  comment there, instead of opening a new issue.

- If possible, provide us a **use case of the feature**: How could the user
-  invoke the new function? Which problem would it solve? If new information is
-  obtained, how would it be further processed?
+- **Motivate the feature**, i.e.

- If not obvious, briefly motivate how your suggested feature **conforms with
-  Instaloader's project goal**.
+  - Provide us a **use case of the feature**: How could the user
+    invoke the new function? Which problem would it solve? If new information is
+    obtained, how would it be further processed?

- **Be patient**. Naturally, bugs and pull requests have a higher priority than
-  feature suggestions. Keep in mind that this is a free software project, and
-  unfortunately we only have limited time to work on it.
+  - Describe already-working **alternatives of the feature** and how they
+    compare to your proposed feature.
+
+  - Briefly describe how your suggested feature **conforms with Instaloader's
+    project goal**.
+
+- Explain your **solution ideas**. Describe your ideas on how the feature could
+  be implemented and the underlying problem could be solved. Also **describe
+  alternatives** that you have considered.

 Donations
 ---------
@@ -40,6 +40,8 @@ See :ref:`install` for more options on how to install Instaloader.
 - allows **fine-grained customization** of filters and where to store
  downloaded media,

+- automatically **resumes previously-interrupted** download iterations,
+
 - is free `open source <https://github.com/instaloader/instaloader>`__
  software written in Python.

@@ -5,27 +5,27 @@ Install Instaloader

 .. highlight:: none

-**To install Instaloader**,
+To **install Instaloader**,

 #. Ensure that you have `Python <https://www.python.org/>`__, at least
-   version 3.5 and `pip <https://pypi.python.org/pypi/pip>`__
-   installed
+   version 3.5, and `pip <https://pypi.python.org/pypi/pip>`__
+   installed.

 #. Then, install Instaloader using::

      pip3 install instaloader

-**To upgrade Instaloader**, do::
+To **upgrade Instaloader** to its current version, do::

   pip3 install --upgrade instaloader


-**Alternative methods for installing Instaloader:**
+**Alternative methods** for installing Instaloader:

 - If you do not want to use pip, even though it is highly recommended,
  and prefer to **install Instaloader manually**,
  `Download the Source <https://github.com/instaloader/instaloader/releases/latest>`__,
-  extract the Zip or Tarball and execute ``instaloader.py`` from there.
+  extract the Zip or Tarball and run ``setup.py`` from there.

 - On **Arch Linux**, you may install Instaloader using the
  `Instaloader AUR package <https://aur.archlinux.org/packages/instaloader/>`__.
@@ -33,6 +33,6 @@ Install Instaloader
 - On **Windows 10**, you may download the standalone executable from the
  `current release page <https://github.com/instaloader/instaloader/releases/latest>`__.

- To test the most current pre-release or development version of Instaloader::
+- To test the most current **pre-release** version of Instaloader::

     pip3 install --pre instaloader
@@ -0,0 +1,49 @@
+Exceptions
+^^^^^^^^^^
+
+.. module:: instaloader
+   :noindex:
+
+.. highlight:: python
+
+.. currentmodule:: instaloader.exceptions
+
+.. autoexception:: InstaloaderException
+   :no-show-inheritance:
+
+.. autoexception:: ConnectionException
+
+.. currentmodule:: instaloader
+
+.. autoexception:: QueryReturnedBadRequestException
+
+.. autoexception:: QueryReturnedForbiddenException
+
+.. autoexception:: ProfileNotExistsException
+
+.. autoexception:: ProfileHasNoPicsException
+
+.. autoexception:: PrivateProfileNotFollowedException
+
+.. autoexception:: LoginRequiredException
+
+.. autoexception:: TwoFactorAuthRequiredException
+
+   .. versionadded:: 4.2
+
+.. autoexception:: InvalidArgumentException
+
+.. autoexception:: BadResponseException
+
+.. autoexception:: BadCredentialsException
+
+.. autoexception:: PostChangedException
+
+.. autoexception:: QueryReturnedNotFoundException
+
+   .. versionchanged:: 4.3
+      QueryReturnedNotFoundException now inherits ConnectionException
+      to retry on 404 errors.
+
+
+.. autoexception:: TooManyRequestsException
@@ -0,0 +1,10 @@
+``Instaloader`` (Main Class)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. module:: instaloader
+   :noindex:
+
+.. highlight:: python
+
+.. autoclass:: Instaloader
+   :no-show-inheritance:
@@ -0,0 +1,24 @@
+``InstaloaderContext`` (Low-level functions)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. module:: instaloader
+   :noindex:
+
+.. highlight:: python
+
+.. contents::
+   :backlinks: none
+
+``InstaloaderContext``
+""""""""""""""""""""""
+
+.. autoclass:: InstaloaderContext
+   :no-show-inheritance:
+
+``RateController``
+""""""""""""""""""
+
+.. autoclass:: RateController
+   :no-show-inheritance:
+
+   .. versionadded:: 4.5
@@ -0,0 +1,31 @@
+Resumable Iterations
+^^^^^^^^^^^^^^^^^^^^
+
+.. module:: instaloader
+   :noindex:
+
+.. highlight:: python
+
+.. contents::
+   :backlinks: none
+
+For many download targets, Instaloader is able to resume a
+previously-interrupted iteration. It provides an interruptible
+Iterator :class:`NodeIterator` and a context manager
+:func:`resumable_iteration`, which we both present here.
+
+.. versionadded:: 4.5
+
+``NodeIterator``
+""""""""""""""""
+
+.. autoclass:: NodeIterator
+   :no-show-inheritance:
+
+.. autoclass:: FrozenNodeIterator
+   :no-show-inheritance:
+
+``resumable_iteration``
+"""""""""""""""""""""""
+
+.. autofunction:: resumable_iteration
@@ -0,0 +1,82 @@
+Instagram Structures
+^^^^^^^^^^^^^^^^^^^^
+
+.. module:: instaloader
+   :noindex:
+
+.. highlight:: python
+
+.. contents::
+   :backlinks: none
+
+Posts
+"""""
+
+.. autoclass:: Post
+   :no-show-inheritance:
+
+Additionally, the following trivial structures are defined:
+
+.. autoclass:: PostSidecarNode
+   :no-show-inheritance:
+
+.. autoclass:: PostComment
+   :no-show-inheritance:
+
+.. autoclass:: PostCommentAnswer
+   :no-show-inheritance:
+
+.. autoclass:: PostLocation
+   :no-show-inheritance:
+
+User Stories
+""""""""""""
+
+.. autoclass:: Story
+   :no-show-inheritance:
+
+.. autoclass:: StoryItem
+   :no-show-inheritance:
+
+Highlights
+""""""""""
+
+.. autoclass:: Highlight
+   :no-show-inheritance:
+   :inherited-members:
+
+   Bases: :class:`Story`
+
+   .. versionadded:: 4.1
+
+Profiles
+""""""""
+
+.. autoclass:: Profile
+   :no-show-inheritance:
+
+Hashtags
+""""""""
+
+.. autoclass:: Hashtag
+   :no-show-inheritance:
+
+   .. versionadded:: 4.4
+
+TopSearchResults
+""""""""""""""""
+
+.. autoclass:: TopSearchResults
+   :no-show-inheritance:
+
+   .. versionadded:: 4.3
+
+Loading and Saving
+""""""""""""""""""
+
+:class:`Post`, :class:`StoryItem`, :class:`Profile`, :class:`Hashtag` and
+:class:`FrozenNodeIterator` can be saved and loaded to/from JSON files.
+
+.. autofunction:: load_structure_from_file
+
+.. autofunction:: save_structure_to_file
@@ -1,2 +1,2 @@
 requests
-sphinx!=1.7.7
+sphinx~=2.3
@@ -5,8 +5,8 @@ Troubleshooting

 .. highlight:: python

-429 - Too Many Requests
-----------------------
+429 Too Many Requests
+---------------------

 Instaloader has a logic to keep track of its requests to Instagram and to obey
 their rate limits. Since they are nowhere documented, we try them out
@@ -14,25 +14,39 @@ experimentally. We have a daily cron job running to confirm that Instaloader
 still stays within the rate limits. Nevertheless, the rate control logic assumes
 that

- at one time, Instaloader is the only application that consumes requests. I.e.
+- at one time, Instaloader is the only application that consumes requests, i.e.
  neither the Instagram browser interface, nor a mobile app, nor another
-  Instaloader instance is running in parallel,
+  Instaloader instance is running in parallel, and

 - no requests had been consumed when Instaloader starts.

 The latter one implies that restarting or reinstantiating Instaloader often
-within short time is prone to cause a 429. When a request is denied with a 429,
+within short time is prone to cause a 429. If a request is denied with a 429,
 Instaloader retries the request as soon as the temporary ban is assumed to be
 expired. In case the retry continuously fails for some reason, which should not
-happen in normal conditions, consider adjusting the
+happen under normal conditions, consider adjusting the
 :option:`--max-connection-attempts` option.

+There have been observations that services, that in their nature offer
+promiscuous IP addresses, such as cloud, VPN and public proxy services, might be
+subject to significantly stricter limits for anonymous access. However,
+logged-in accesses (see :option:`--login`) do not seem to be affected.
+
+Too many queries in the last time
+---------------------------------
+
 **"Too many queries in the last time"** is not an error. It is a notice that the
 rate limit has almost been reached, according to Instaloader's own rate
 accounting mechanism. We regularly adjust this mechanism to match Instagram's
 current rate limiting.

-Login Error
+Private but not followed
+------------------------
+
+You have to follow a private account to access most of its associated
+information.
+
+Login error
 -----------

 Instaloader's login *should* work fine, both with and without
@@ -41,13 +55,13 @@ issued when Instagram suspects authentication activity on your account, by
 pointing the user to an URL to be opened in a browser.

 Nevertheless, in :issue:`92` and :issue:`615` users reported problems with
-logging in. We recommend to always keep the sessionfile which Instaloader
-creates when using :option:`--login`. If a sessionfile is present,
+logging in. We recommend to always keep the session file which Instaloader
+creates when using :option:`--login`. If a session file is present,
 :option:`--login` does not make make use of the failure-prone login procedure.
-Also, sessionfiles usually do not expire and can be copied between different
+Also, session files usually do not expire and can be copied between different
 computers without any problems.

-If you do not have a sessionfile present, you may use the following script
+If you do not have a session file present, you may use the following script
 (:example:`615_import_firefox_session.py`) to workaround login problems by
 importing the session cookies from Firefox and bypassing Instaloader's login and
 so still use Instaloader's logged-in functionality.
@@ -64,7 +78,7 @@ To use this script,

 #. Then, ``instaloader -l USERNAME`` should work fine.

-This script also supports specifying a cookiefile path, which may be useful if
+This script also supports specifying a cookie file path, which may be useful if
 you use multiple Firefox profiles or if your operating system has the directory
-structure differently set up. Also, you can specify an alternative sessionfile
+structure differently set up. Also, you can specify an alternative session file
 path.
@@ -1,7 +1,7 @@
 """Download pictures (or videos) along with their captions and other metadata from Instagram."""


-__version__ = '4.4.4rc3'
+__version__ = '4.5.1'


 try:
@@ -14,6 +14,7 @@ else:

 from .exceptions import *
 from .instaloader import Instaloader
-from .instaloadercontext import InstaloaderContext
+from .instaloadercontext import InstaloaderContext, RateController
+from .nodeiterator import NodeIterator, FrozenNodeIterator, resumable_iteration
 from .structures import (Hashtag, Highlight, Post, PostSidecarNode, PostComment, PostCommentAnswer, PostLocation,
                         Profile, Story, StoryItem, TopSearchResults, load_structure_from_file, save_structure_to_file)
@@ -82,6 +82,8 @@ def _main(instaloader: Instaloader, targetlist: List[str],
        instaloader.context.log('Only download storyitems with property "{}".'.format(storyitem_filter_str))
    # Login, if desired
    if username is not None:
+        if not re.match(r"^[A-Za-z0-9._]+$", username):
+            instaloader.context.error("Warning: Parameter \"{}\" for --login is not a valid username.".format(username))
        try:
            instaloader.load_session_from_file(username, sessionfile)
        except FileNotFoundError as err:
@@ -349,6 +351,13 @@ def main():
                            '--dirname-pattern. {profile} is replaced by the profile name,'
                            '{target} is replaced by the target you specified, i.e. either :feed'
                            '#hashtag or the profile name. Defaults to \'{date_utc}_UTC\'')
+    g_how.add_argument('--resume-prefix', metavar='PREFIX',
+                       help='Prefix for filenames that are used to save the information to resume an interrupted '
+                            'download.')
+    g_how.add_argument('--no-resume', action='store_true',
+                       help='Do not resume a previously-aborted download iteration, and do not save such information '
+                            'when interrupted.')
+    g_how.add_argument('--use-aged-resume-files', action='store_true', help=SUPPRESS)
    g_how.add_argument('--user-agent',
                       help='User Agent to use for HTTP requests. Defaults to \'{}\'.'.format(default_user_agent()))
    g_how.add_argument('-S', '--no-sleep', action='store_true', help=SUPPRESS)
@@ -356,10 +365,7 @@ def main():
                       help='Maximum number of connection attempts until a request is aborted. Defaults to 3. If a '
                            'connection fails, it can be manually skipped by hitting CTRL+C. Set this to 0 to retry '
                            'infinitely.')
-    g_how.add_argument('--commit-mode', action='store_true',
-                       help='Tries to ensure downloaded images avoid corruption in case of unexpected interruption. '
-                       'If the last picture is corrupted, Instaloader will fix the picture the next time it is run. '
-                       'Requires the JSON metadata to be saved.')
+    g_how.add_argument('--commit-mode', action='store_true', help=SUPPRESS)
    g_how.add_argument('--request-timeout', metavar='N', type=float,
                       help='seconds to wait before timing out a connection request')

@@ -395,6 +401,10 @@ def main():
                raise SystemExit("--no-captions and --post-metadata-txt or --storyitem-metadata-txt given; "
                                 "That contradicts.")

+        if args.no_resume and args.resume_prefix:
+            raise SystemExit("--no-resume and --resume-prefix given; That contradicts.")
+        resume_prefix = (args.resume_prefix if args.resume_prefix else 'iterator') if not args.no_resume else None
+
        if args.no_pictures and args.fast_update:
            raise SystemExit('--no-pictures and --fast-update cannot be used together.')

@@ -403,9 +413,6 @@ def main():
        download_posts = not (args.no_posts or args.stories_only or args.profile_pic_only)
        download_stories = args.stories or args.stories_only

-        if args.commit_mode and args.no_metadata_json:
-            raise SystemExit('--commit-mode requires JSON metadata to be saved.')
-
        loader = Instaloader(sleep=not args.no_sleep, quiet=args.quiet, user_agent=args.user_agent,
                             dirname_pattern=args.dirname_pattern, filename_pattern=args.filename_pattern,
                             download_pictures=not args.no_pictures,
@@ -417,7 +424,8 @@ def main():
                             storyitem_metadata_txt_pattern=storyitem_metadata_txt_pattern,
                             max_connection_attempts=args.max_connection_attempts,
                             request_timeout=args.request_timeout,
-                             commit_mode=args.commit_mode)
+                             resume_prefix=resume_prefix,
+                             check_resume_bbd=not args.use_aged_resume_files)
        _main(loader,
              args.profile,
              username=args.login.lower() if args.login is not None else None,
@@ -1,6 +1,5 @@
 import getpass
 import json
-import lzma
 import os
 import platform
 import re
@@ -14,15 +13,16 @@ from functools import wraps
 from hashlib import md5
 from io import BytesIO
 from pathlib import Path
-from typing import Any, Callable, Iterator, List, Optional, Set, Union
+from typing import Any, Callable, IO, Iterator, List, Optional, Set, Union, cast

 import requests
 import urllib3  # type: ignore

 from .exceptions import *
-from .instaloadercontext import InstaloaderContext
+from .instaloadercontext import InstaloaderContext, RateController
+from .nodeiterator import NodeIterator, resumable_iteration
 from .structures import (Hashtag, Highlight, JsonExportable, Post, PostLocation, Profile, Story, StoryItem,
-                         save_structure_to_file, load_structure_from_file)
+                         load_structure_from_file, save_structure_to_file)


 def get_default_session_filename(username: str) -> str:
@@ -153,8 +153,10 @@ class Instaloader:
       txt file.
    :param storyitem_metadata_txt_pattern: :option:`--storyitem-metadata-txt`, default is empty (=none)
    :param max_connection_attempts: :option:`--max-connection-attempts`
-    :param commit_mode: :option:`--commit-mode`
    :param request_timeout: :option:`--request-timeout`, set per-request timeout (seconds)
+    :param rate_controller: Generator for a :class:`RateController` to override rate controlling behavior
+    :param resume_prefix: :option:`--resume-prefix`, or None for :option:`--no-resume`.
+    :param check_resume_bbd: Whether to check the date of expiry of resume files and reject them if expired.

    .. attribute:: context

@@ -178,9 +180,12 @@ class Instaloader:
                 storyitem_metadata_txt_pattern: str = None,
                 max_connection_attempts: int = 3,
                 request_timeout: Optional[float] = None,
-                 commit_mode: bool = False):
+                 rate_controller: Optional[Callable[[InstaloaderContext], RateController]] = None,
+                 resume_prefix: Optional[str] = "iterator",
+                 check_resume_bbd: bool = True):

-        self.context = InstaloaderContext(sleep, quiet, user_agent, max_connection_attempts, request_timeout)
+        self.context = InstaloaderContext(sleep, quiet, user_agent, max_connection_attempts,
+                                          request_timeout, rate_controller)

        # configuration parameters
        self.dirname_pattern = dirname_pattern or "{target}"
@@ -196,12 +201,8 @@ class Instaloader:
            else post_metadata_txt_pattern
        self.storyitem_metadata_txt_pattern = '' if storyitem_metadata_txt_pattern is None \
            else storyitem_metadata_txt_pattern
-        self.commit_mode = commit_mode
-        if self.commit_mode and not self.save_metadata:
-            raise InvalidArgumentException("Commit mode requires JSON metadata to be saved.")
-
-        # Used to keep state in commit mode
-        self._committed = None  # type: Optional[bool]
+        self.resume_prefix = resume_prefix
+        self.check_resume_bbd = check_resume_bbd

    @contextmanager
    def anonymous_copy(self):
@@ -223,7 +224,8 @@ class Instaloader:
            storyitem_metadata_txt_pattern=self.storyitem_metadata_txt_pattern,
            max_connection_attempts=self.context.max_connection_attempts,
            request_timeout=self.context.request_timeout,
-            commit_mode=self.commit_mode)
+            resume_prefix=self.resume_prefix,
+            check_resume_bbd=self.check_resume_bbd)
        yield new_loader
        self.context.error_log.extend(new_loader.context.error_log)
        new_loader.context.error_log = []  # avoid double-printing of errors
@@ -249,15 +251,9 @@ class Instaloader:
        if filename_suffix is not None:
            filename += '_' + filename_suffix
        filename += '.' + file_extension
-        # A post is considered "commited" if the json file exists and is not malformed.
-        if self.commit_mode:
-            if self._committed and os.path.isfile(filename):
-                self.context.log(filename + ' exists', end=' ', flush=True)
-                return False
-        else:
-            if os.path.isfile(filename):
-                self.context.log(filename + ' exists', end=' ', flush=True)
-                return False
+        if os.path.isfile(filename):
+            self.context.log(filename + ' exists', end=' ', flush=True)
+            return False
        self.context.get_and_write_raw(url, filename)
        os.utime(filename, (datetime.now().timestamp(), mtime.timestamp()))
        return True
@@ -355,7 +351,8 @@ class Instaloader:
        except UnicodeEncodeError:
            self.context.log('txt', end=' ', flush=True)
        with open(filename, 'wb') as text_file:
-            shutil.copyfileobj(BytesIO(bcaption), text_file)
+            with BytesIO(bcaption) as bio:
+                shutil.copyfileobj(cast(IO, bio), text_file)
        os.utime(filename, (datetime.now().timestamp(), mtime.timestamp()))

    def save_location(self, filename: str, location: PostLocation, mtime: datetime) -> None:
@@ -365,10 +362,29 @@ class Instaloader:
                           "https://maps.google.com/maps?q={0},{1}&ll={0},{1}\n".format(location.lat,
                                                                                        location.lng))
        with open(filename, 'wb') as text_file:
-            shutil.copyfileobj(BytesIO(location_string.encode()), text_file)
+            with BytesIO(location_string.encode()) as bio:
+                shutil.copyfileobj(cast(IO, bio), text_file)
        os.utime(filename, (datetime.now().timestamp(), mtime.timestamp()))
        self.context.log('geo', end=' ', flush=True)

+    def format_filename_within_target_path(self,
+                                           target: Union[str, Path],
+                                           owner_profile: Optional[Profile],
+                                           identifier: str,
+                                           name_suffix: str,
+                                           extension: str):
+        """Returns a filename within the target path.
+
+        .. versionadded:: 4.5"""
+        if ((format_string_contains_key(self.dirname_pattern, 'profile') or
+             format_string_contains_key(self.dirname_pattern, 'target'))):
+            profile_str = owner_profile.username.lower() if owner_profile is not None else target
+            return os.path.join(self.dirname_pattern.format(profile=profile_str, target=target),
+                                '{0}_{1}.{2}'.format(identifier, name_suffix, extension))
+        else:
+            return os.path.join(self.dirname_pattern.format(),
+                                '{0}_{1}_{2}.{3}'.format(target, identifier, name_suffix, extension))
+
    @_retry_on_connection_error
    def download_title_pic(self, url: str, target: Union[str, Path], name_suffix: str, owner_profile: Optional[Profile],
                           _attempt: int = 1) -> None:
@@ -389,16 +405,7 @@ class Instaloader:
        else:
            pic_bytes = http_response.content
            pic_identifier = md5(pic_bytes).hexdigest()[:16]
-        pic_extension = 'jpg'
-        if ((format_string_contains_key(self.dirname_pattern, 'profile') or
-             format_string_contains_key(self.dirname_pattern, 'target'))):
-            profile_str = owner_profile.username.lower() if owner_profile is not None else target
-            filename = '{0}/{1}_{2}.{3}'.format(self.dirname_pattern.format(profile=profile_str,
-                                                                            target=target),
-                                                pic_identifier, name_suffix, pic_extension)
-        else:
-            filename = '{0}/{1}_{2}_{3}.{4}'.format(self.dirname_pattern.format(), target,
-                                                    pic_identifier, name_suffix, pic_extension)
+        filename = self.format_filename_within_target_path(target, owner_profile, pic_identifier, name_suffix, 'jpg')
        content_length = http_response.headers.get('Content-Length', None)
        if os.path.isfile(filename) and (not self.context.is_logged_in or
                                         (content_length is not None and
@@ -501,32 +508,32 @@ class Instaloader:
        """

        dirname = _PostPathFormatter(post).format(self.dirname_pattern, target=target)
-        filename = dirname + '/' + self.format_filename(post, target=target)
+        filename = os.path.join(dirname, self.format_filename(post, target=target))
        os.makedirs(os.path.dirname(filename), exist_ok=True)

        # Download the image(s) / video thumbnail and videos within sidecars if desired
        downloaded = True
-        self._committed = self.check_if_committed(filename)
-        if self.download_pictures:
-            if post.typename == 'GraphSidecar':
-                edge_number = 1
-                for sidecar_node in post.get_sidecar_nodes():
-                    # Download picture or video thumbnail
-                    if not sidecar_node.is_video or self.download_video_thumbnails is True:
-                        downloaded &= self.download_pic(filename=filename, url=sidecar_node.display_url,
-                                                        mtime=post.date_local, filename_suffix=str(edge_number))
-                    # Additionally download video if available and desired
-                    if sidecar_node.is_video and self.download_videos is True:
-                        downloaded &= self.download_pic(filename=filename, url=sidecar_node.video_url,
-                                                        mtime=post.date_local, filename_suffix=str(edge_number))
-                    edge_number += 1
-            elif post.typename == 'GraphImage':
+        if post.typename == 'GraphSidecar':
+            for edge_number, sidecar_node in enumerate(post.get_sidecar_nodes(), start=1):
+                if self.download_pictures and (not sidecar_node.is_video or self.download_video_thumbnails):
+                    # Download sidecar picture or video thumbnail (--no-pictures implies --no-video-thumbnails)
+                    downloaded &= self.download_pic(filename=filename, url=sidecar_node.display_url,
+                                                    mtime=post.date_local, filename_suffix=str(edge_number))
+                if sidecar_node.is_video and self.download_videos:
+                    # Download sidecar video if desired
+                    downloaded &= self.download_pic(filename=filename, url=sidecar_node.video_url,
+                                                    mtime=post.date_local, filename_suffix=str(edge_number))
+        elif post.typename == 'GraphImage':
+            # Download picture
+            if self.download_pictures:
                downloaded = self.download_pic(filename=filename, url=post.url, mtime=post.date_local)
-            elif post.typename == 'GraphVideo':
-                if self.download_video_thumbnails is True:
+        elif post.typename == 'GraphVideo':
+            # Download video thumbnail (--no-pictures implies --no-video-thumbnails)
+            if self.download_pictures and self.download_video_thumbnails:
+                with self.context.error_catcher("Video thumbnail of {}".format(post)):
                    downloaded = self.download_pic(filename=filename, url=post.url, mtime=post.date_local)
-            else:
-                self.context.error("Warning: {0} has unknown typename: {1}".format(post, post.typename))
+        else:
+            self.context.error("Warning: {0} has unknown typename: {1}".format(post, post.typename))

        # Save caption if desired
        metadata_string = _ArbitraryItemFormatter(post).format(self.post_metadata_txt_pattern).strip()
@@ -534,7 +541,7 @@ class Instaloader:
            self.save_caption(filename=filename, mtime=post.date_local, caption=metadata_string)

        # Download video if desired
-        if post.is_video and self.download_videos is True:
+        if post.is_video and self.download_videos:
            downloaded &= self.download_pic(filename=filename, url=post.video_url, mtime=post.date_local)

        # Download geotags if desired
@@ -542,11 +549,11 @@ class Instaloader:
            self.save_location(filename, post.location, post.date_local)

        # Update comments if desired
-        if self.download_comments is True:
+        if self.download_comments:
            self.update_comments(filename=filename, post=post)

        # Save metadata as JSON if desired.
-        if self.save_metadata is not False:
+        if self.save_metadata:
            self.save_metadata_json(filename, post)

        self.context.log()
@@ -630,7 +637,7 @@ class Instaloader:

        date_local = item.date_local
        dirname = _PostPathFormatter(item).format(self.dirname_pattern, target=target)
-        filename = dirname + '/' + self.format_filename(item, target=target)
+        filename = os.path.join(dirname, self.format_filename(item, target=target))
        os.makedirs(os.path.dirname(filename), exist_ok=True)
        downloaded = False
        if not item.is_video or self.download_video_thumbnails is True:
@@ -719,59 +726,77 @@ class Instaloader:
                            fast_update: bool = False,
                            post_filter: Optional[Callable[[Post], bool]] = None,
                            max_count: Optional[int] = None,
-                            total_count: Optional[int] = None) -> None:
+                            total_count: Optional[int] = None,
+                            owner_profile: Optional[Profile] = None) -> None:
        """
        Download the Posts returned by given Post Iterator.

-        ..versionadded:: 4.4
+        .. versionadded:: 4.4
+
+        .. versionchanged:: 4.5
+           Transparently resume an aborted operation if `posts` is a :class:`NodeIterator`.

        :param posts: Post Iterator to loop through.
-        :param target: Target name
-        :param fast_update: :option:`--fast-update`
-        :param post_filter: :option:`--post-filter`
-        :param max_count: Maximum count of Posts to download (:option:`--count`)
-        :param total_count: Total number of posts returned by given iterator
+        :param target: Target name.
+        :param fast_update: :option:`--fast-update`.
+        :param post_filter: :option:`--post-filter`.
+        :param max_count: Maximum count of Posts to download (:option:`--count`).
+        :param total_count: Total number of posts returned by given iterator.
+        :param owner_profile: Associated profile, if any.
        """
-        for number, post in enumerate(posts):
-            if max_count is not None and number >= max_count:
-                break
-            if total_count is not None:
-                self.context.log("[{0:{w}d}/{1:{w}d}] ".format(number + 1, total_count,
-                                                               w=len(str(total_count))),
-                                 end="", flush=True)
-            else:
-                if max_count is not None:
-                    self.context.log("[{0:{w}d}/{1:{w}d}] ".format(number + 1, max_count,
-                                                                   w=len(str(max_count))),
+        displayed_count = (max_count if total_count is None or max_count is not None and max_count < total_count
+                           else total_count)
+        sanitized_target = target
+        if isinstance(target, str):
+            sanitized_target = _PostPathFormatter.sanitize_path(target)
+        with resumable_iteration(
+                context=self.context,
+                iterator=posts,
+                load=load_structure_from_file,
+                save=save_structure_to_file,
+                format_path=lambda magic: self.format_filename_within_target_path(
+                    sanitized_target, owner_profile, self.resume_prefix or '', magic, 'json.xz'
+                ),
+                check_bbd=self.check_resume_bbd,
+                enabled=self.resume_prefix is not None
+        ) as (is_resuming, start_index):
+            for number, post in enumerate(posts, start=start_index + 1):
+                if max_count is not None and number > max_count:
+                    break
+                if displayed_count is not None:
+                    self.context.log("[{0:{w}d}/{1:{w}d}] ".format(number, displayed_count,
+                                                                   w=len(str(displayed_count))),
                                     end="", flush=True)
                else:
-                    self.context.log("[{:3d}] ".format(number + 1), end="", flush=True)
-            if post_filter is not None:
-                try:
-                    if not post_filter(post):
-                        self.context.log("{} skipped".format(post))
-                        continue
-                except (InstaloaderException, KeyError, TypeError) as err:
-                    self.context.error("{} skipped. Filter evaluation failed: {}".format(post, err))
-                    continue
-            with self.context.error_catcher("Download {} of {}".format(post, target)):
-                # The PostChangedException gets raised if the Post's id/shortcode changed while obtaining
-                # additional metadata. This is most likely the case if a HTTP redirect takes place while
-                # resolving the shortcode URL.
-                # The `post_changed` variable keeps the fast-update functionality alive: A Post which is
-                # obained after a redirect has probably already been downloaded as a previous Post of the
-                # same Profile.
-                # Observed in issue #225: https://github.com/instaloader/instaloader/issues/225
-                post_changed = False
-                while True:
+                    self.context.log("[{:3d}] ".format(number), end="", flush=True)
+                if post_filter is not None:
                    try:
-                        downloaded = self.download_post(post, target=target)
-                        break
-                    except PostChangedException:
-                        post_changed = True
+                        if not post_filter(post):
+                            self.context.log("{} skipped".format(post))
+                            continue
+                    except (InstaloaderException, KeyError, TypeError) as err:
+                        self.context.error("{} skipped. Filter evaluation failed: {}".format(post, err))
                        continue
-                if fast_update and not downloaded and not post_changed:
-                    break
+                with self.context.error_catcher("Download {} of {}".format(post, target)):
+                    # The PostChangedException gets raised if the Post's id/shortcode changed while obtaining
+                    # additional metadata. This is most likely the case if a HTTP redirect takes place while
+                    # resolving the shortcode URL.
+                    # The `post_changed` variable keeps the fast-update functionality alive: A Post which is
+                    # obained after a redirect has probably already been downloaded as a previous Post of the
+                    # same Profile.
+                    # Observed in issue #225: https://github.com/instaloader/instaloader/issues/225
+                    post_changed = False
+                    while True:
+                        try:
+                            downloaded = self.download_post(post, target=target)
+                            break
+                        except PostChangedException:
+                            post_changed = True
+                            continue
+                    if fast_update and not downloaded and not post_changed:
+                        # disengage fast_update for first post when resuming
+                        if not is_resuming or number > 0:
+                            break

    @_requires_login
    def get_feed_posts(self) -> Iterator[Post]:
@@ -831,8 +856,10 @@ class Instaloader:
        """
        self.context.log("Retrieving saved posts...")
        assert self.context.username is not None  # safe due to @_requires_login; required by typechecker
-        self.posts_download_loop(Profile.from_username(self.context, self.context.username).get_saved_posts(), ":saved",
-                                 fast_update, post_filter, max_count=max_count)
+        node_iterator = Profile.from_username(self.context, self.context.username).get_saved_posts()
+        self.posts_download_loop(node_iterator, ":saved",
+                                 fast_update, post_filter,
+                                 max_count=max_count, total_count=node_iterator.count)

    @_requires_login
    def get_location_posts(self, location: str) -> Iterator[Post]:
@@ -887,18 +914,20 @@ class Instaloader:
                                 max_count=max_count)

    @_requires_login
-    def get_explore_posts(self) -> Iterator[Post]:
+    def get_explore_posts(self) -> NodeIterator[Post]:
        """Get Posts which are worthy of exploring suggested by Instagram.

        :return: Iterator over Posts of the user's suggested posts.
+        :rtype: NodeIterator[Post]
        :raises LoginRequiredException: If called without being logged in.
        """
-        data = self.context.get_json('explore/', {})
-        yield from (Post(self.context, node)
-                    for node in self.context.graphql_node_list("df0dcc250c2b18d9fd27c5581ef33c7c",
-                                                               {}, 'https://www.instagram.com/explore/',
-                                                               lambda d: d['data']['user']['edge_web_discover_media'],
-                                                               data.get('rhx_gis')))
+        return NodeIterator(
+            self.context,
+            'df0dcc250c2b18d9fd27c5581ef33c7c',
+            lambda d: d['data']['user']['edge_web_discover_media'],
+            lambda n: Post(self.context, n),
+            query_referer='https://www.instagram.com/explore/',
+        )

    def get_hashtag_posts(self, hashtag: str) -> Iterator[Post]:
        """Get Posts associated with a #hashtag.
@@ -969,15 +998,17 @@ class Instaloader:
        .. versionadded:: 4.3"""
        self.context.log("Retrieving IGTV videos for profile {}.".format(profile.username))
        self.posts_download_loop(profile.get_igtv_posts(), profile.username, fast_update, post_filter,
-                                 total_count=profile.igtvcount)
+                                 total_count=profile.igtvcount, owner_profile=profile)

    def _get_id_filename(self, profile_name: str) -> str:
        if ((format_string_contains_key(self.dirname_pattern, 'profile') or
             format_string_contains_key(self.dirname_pattern, 'target'))):
-            return '{0}/id'.format(self.dirname_pattern.format(profile=profile_name.lower(),
-                                                               target=profile_name.lower()))
+            return os.path.join(self.dirname_pattern.format(profile=profile_name.lower(),
+                                                            target=profile_name.lower()),
+                                'id')
        else:
-            return '{0}/{1}_id'.format(self.dirname_pattern.format(), profile_name.lower())
+            return os.path.join(self.dirname_pattern.format(),
+                                '{0}_id'.format(profile_name.lower()))

    def save_profile_id(self, profile: Profile):
        """
@@ -1077,11 +1108,11 @@ class Instaloader:
        def _error_raiser(_str):
            yield

-        # error_handler type is Callable[[Optional[str]], ContextManager[None]] (not supported with Python 3.5)
+        # error_handler type is Callable[[Optional[str]], ContextManager[None]] (not supported with Python 3.5.0..3.5.3)
        error_handler = _error_raiser if raise_errors else self.context.error_catcher

        for profile in profiles:
-            with error_handler(profile.username):  # type: ignore
+            with error_handler(profile.username):  # type: ignore # (ignore type for Python 3.5 support)
                profile_name = profile.username

                # Download profile picture
@@ -1091,9 +1122,9 @@ class Instaloader:

                # Save metadata as JSON if desired.
                if self.save_metadata:
-                    json_filename = '{0}/{1}_{2}'.format(self.dirname_pattern.format(profile=profile_name,
-                                                                                     target=profile_name),
-                                                         profile_name, profile.userid)
+                    json_filename = os.path.join(self.dirname_pattern.format(profile=profile_name,
+                                                                             target=profile_name),
+                                                 '{0}_{1}'.format(profile_name, profile.userid))
                    self.save_metadata_json(json_filename, profile)

                # Catch some errors
@@ -1122,7 +1153,7 @@ class Instaloader:
                if posts:
                    self.context.log("Retrieving posts from profile {}.".format(profile_name))
                    self.posts_download_loop(profile.get_posts(), profile_name, fast_update, post_filter,
-                                             total_count=profile.mediacount)
+                                             total_count=profile.mediacount, owner_profile=profile)

        if stories and profiles:
            with self.context.error_catcher("Download stories"):
@@ -1202,26 +1233,7 @@ class Instaloader:
        # Iterate over pictures and download them
        self.context.log("Retrieving posts from profile {}.".format(profile_name))
        self.posts_download_loop(profile.get_posts(), profile_name, fast_update, post_filter,
-                                 total_count=profile.mediacount)
-
-    def check_if_committed(self, filename: str) -> bool:
-        """Checks to see if the current post has been committed.
-
-        A post is considered committed if its json metadata file exists and is not malformed.
-
-        .. versionadded:: 4.2
-        """
-        if os.path.isfile(filename + '.json.xz'):
-            filename += '.json.xz'
-        elif os.path.isfile(filename + '.json'):
-            filename += '.json'
-        else:
-            return False
-        try:
-            load_structure_from_file(self.context, filename)
-            return True
-        except (FileNotFoundError, lzma.LZMAError, json.decoder.JSONDecodeError):
-            return False
+                                 total_count=profile.mediacount, owner_profile=profile)

    def interactive_login(self, username: str) -> None:
        """Logs in and internally stores session, asking user for password interactively.
@@ -1,5 +1,6 @@
 import hashlib
 import json
+import os
 import pickle
 import random
 import re
@@ -23,7 +24,7 @@ def copy_session(session: requests.Session, request_timeout: Optional[float] = N
    """Duplicates a requests.Session."""
    new = requests.Session()
    new.cookies = requests.utils.cookiejar_from_dict(requests.utils.dict_from_cookiejar(session.cookies))
-    new.headers = session.headers.copy() # type: ignore
+    new.headers = session.headers.copy()
    if request_timeout is not None:
        # Override default timeout behavior.
        # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
@@ -52,7 +53,8 @@ class InstaloaderContext:
    """

    def __init__(self, sleep: bool = True, quiet: bool = False, user_agent: Optional[str] = None,
-                 max_connection_attempts: int = 3, request_timeout: Optional[float] = None):
+                 max_connection_attempts: int = 3, request_timeout: Optional[float] = None,
+                 rate_controller: Optional[Callable[["InstaloaderContext"], "RateController"]] = None):

        self.user_agent = user_agent if user_agent is not None else default_user_agent()
        self.request_timeout = request_timeout
@@ -68,9 +70,7 @@ class InstaloaderContext:
        # error log, filled with error() and printed at the end of Instaloader.main()
        self.error_log = []                      # type: List[str]

-        # For the adaption of sleep intervals (rate control)
-        self._graphql_query_timestamps = dict()  # type: Dict[str, List[float]]
-        self._graphql_earliest_next_request_time = 0.0
+        self._rate_controller = rate_controller(self) if rate_controller is not None else RateController(self)

        # Can be set to True for testing, disables supression of InstaloaderContext._error_catcher
        self.raise_all_errors = False
@@ -294,77 +294,6 @@ class InstaloaderContext:
        if self.sleep:
            time.sleep(min(random.expovariate(0.7), 5.0))

-    def _dump_query_timestamps(self, current_time: float):
-        """Output the number of GraphQL queries grouped by their query_hash within the last time."""
-        windows = [10, 11, 15, 20, 30, 60]
-        print("GraphQL requests:", file=sys.stderr)
-        for query_hash, times in self._graphql_query_timestamps.items():
-            print("  {}".format(query_hash), file=sys.stderr)
-            for window in windows:
-                reqs_in_sliding_window = sum(t > current_time - window * 60 for t in times)
-                print("    last {} minutes: {} requests".format(window, reqs_in_sliding_window), file=sys.stderr)
-
-    def _graphql_request_count_per_sliding_window(self, query_hash: str) -> int:
-        """Return how many GraphQL requests can be done within the sliding window."""
-        if self.is_logged_in:
-            max_reqs = {'1cb6ec562846122743b61e492c85999f': 20,
-                        '33ba35852cb50da46f5b5e889df7d159': 20,
-                        'iphone': 100,
-                        'other': 100}
-        else:
-            max_reqs = {'1cb6ec562846122743b61e492c85999f': 200,
-                        '33ba35852cb50da46f5b5e889df7d159': 200,
-                        'other': 200}
-        return max_reqs.get(query_hash) or min(max_reqs.values())
-
-    def _graphql_query_waittime(self, query_hash: str, current_time: float, untracked_queries: bool = False) -> float:
-        """Calculate time needed to wait before GraphQL query can be executed."""
-        sliding_window = 660
-        if query_hash not in self._graphql_query_timestamps:
-            self._graphql_query_timestamps[query_hash] = []
-        self._graphql_query_timestamps[query_hash] = list(filter(lambda t: t > current_time - 60 * 60,
-                                                                 self._graphql_query_timestamps[query_hash]))
-        reqs_in_sliding_window = list(filter(lambda t: t > current_time - sliding_window,
-                                             self._graphql_query_timestamps[query_hash]))
-        count_per_sliding_window = self._graphql_request_count_per_sliding_window(query_hash)
-        if len(reqs_in_sliding_window) < count_per_sliding_window and not untracked_queries:
-            return max(0, self._graphql_earliest_next_request_time - current_time)
-        next_request_time = min(reqs_in_sliding_window) + sliding_window + 6
-        if untracked_queries:
-            self._graphql_earliest_next_request_time = next_request_time
-        return round(max(next_request_time, self._graphql_earliest_next_request_time) - current_time)
-
-    def _ratecontrol_graphql_query(self, query_hash: str, untracked_queries: bool = False):
-        """Called before a GraphQL query is made in order to stay within Instagram's rate limits.
-
-        :param query_hash: The query_hash parameter of the query.
-        :param untracked_queries: True, if 429 has been returned to apply 429 logic.
-        """
-        if not untracked_queries:
-            waittime = self._graphql_query_waittime(query_hash, time.monotonic(), untracked_queries)
-            assert waittime >= 0
-            if waittime > 10:
-                self.log('\nToo many queries in the last time. Need to wait {} seconds, until {:%H:%M}.'
-                         .format(waittime, datetime.now() + timedelta(seconds=waittime)))
-            time.sleep(waittime)
-            if query_hash not in self._graphql_query_timestamps:
-                self._graphql_query_timestamps[query_hash] = [time.monotonic()]
-            else:
-                self._graphql_query_timestamps[query_hash].append(time.monotonic())
-        else:
-            text_for_429 = ("HTTP error code 429 was returned because too many queries occurred in the last time. "
-                            "Please do not use Instagram in your browser or run multiple instances of Instaloader "
-                            "in parallel.")
-            print(textwrap.fill(text_for_429), file=sys.stderr)
-            current_time = time.monotonic()
-            waittime = self._graphql_query_waittime(query_hash, current_time, untracked_queries)
-            assert waittime >= 0
-            if waittime > 10:
-                self.log('The request will be retried in {} seconds, at {:%H:%M}.'
-                         .format(waittime, datetime.now() + timedelta(seconds=waittime)))
-            self._dump_query_timestamps(current_time)
-            time.sleep(waittime)
-
    def get_json(self, path: str, params: Dict[str, Any], host: str = 'www.instagram.com',
                 session: Optional[requests.Session] = None, _attempt=1) -> Dict[str, Any]:
        """JSON request to Instagram.
@@ -385,11 +314,11 @@ class InstaloaderContext:
        try:
            self.do_sleep()
            if is_graphql_query:
-                self._ratecontrol_graphql_query(params['query_hash'])
+                self._rate_controller.wait_before_query(params['query_hash'])
            if is_iphone_query:
-                self._ratecontrol_graphql_query('iphone')
+                self._rate_controller.wait_before_query('iphone')
            if is_other_query:
-                self._ratecontrol_graphql_query('other')
+                self._rate_controller.wait_before_query('other')
            resp = sess.get('https://{0}/{1}'.format(host, path), params=params, allow_redirects=False)
            while resp.is_redirect:
                redirect_url = resp.headers['location']
@@ -445,12 +374,13 @@ class InstaloaderContext:
                    raise ConnectionException(error_string) from err
            self.error(error_string + " [retrying; skip with ^C]", repeat_at_end=False)
            try:
-                if is_graphql_query and isinstance(err, TooManyRequestsException):
-                    self._ratecontrol_graphql_query(params['query_hash'], untracked_queries=True)
-                if is_iphone_query and isinstance(err, TooManyRequestsException):
-                    self._ratecontrol_graphql_query('iphone', untracked_queries=True)
-                if is_other_query and isinstance(err, TooManyRequestsException):
-                    self._ratecontrol_graphql_query('other', untracked_queries=True)
+                if isinstance(err, TooManyRequestsException):
+                    if is_graphql_query:
+                        self._rate_controller.handle_429(params['query_hash'])
+                    if is_iphone_query:
+                        self._rate_controller.handle_429('iphone')
+                    if is_other_query:
+                        self._rate_controller.handle_429('other')
                return self.get_json(path=path, params=params, host=host, session=sess, _attempt=_attempt + 1)
            except KeyboardInterrupt:
                self.error("[skipped by user]", repeat_at_end=False)
@@ -498,7 +428,12 @@ class InstaloaderContext:
                          edge_extractor: Callable[[Dict[str, Any]], Dict[str, Any]],
                          rhx_gis: Optional[str] = None,
                          first_data: Optional[Dict[str, Any]] = None) -> Iterator[Dict[str, Any]]:
-        """Retrieve a list of GraphQL nodes."""
+        """
+        Retrieve a list of GraphQL nodes.
+
+        .. deprecated:: 4.5
+           Use :class:`NodeIterator` instead, which provides more functionality.
+        """

        def _query():
            query_variables['first'] = self._graphql_page_length
@@ -547,11 +482,12 @@ class InstaloaderContext:

        .. versionadded:: 4.2.1"""
        self.log(filename, end=' ', flush=True)
-        with open(filename, 'wb') as file:
+        with open(filename + '.temp', 'wb') as file:
            if isinstance(resp, requests.Response):
                shutil.copyfileobj(resp.raw, file)
            else:
                file.write(resp)
+        os.replace(filename + '.temp', filename)

    def get_raw(self, url: str, _attempt=1) -> requests.Response:
        """Downloads a file anonymously.
@@ -593,3 +529,102 @@ class InstaloaderContext:
        if self._root_rhx_gis is None:
            self._root_rhx_gis = self.get_json('', {}).get('rhx_gis', '')
        return self._root_rhx_gis or None
+
+
+class RateController:
+    """
+    Class providing request tracking and rate controlling to stay within rate limits.
+
+    It can be overridden to change Instaloader's behavior regarding rate limits, for example to raise a custom
+    exception when the rate limit is hit::
+
+       import instaloader
+
+       class MyRateController(instaloader.RateController):
+           def sleep(self, secs):
+               raise MyCustomException()
+
+       L = instaloader.Instaloader(rate_controller=lambda ctx: MyRateController(ctx))
+    """
+
+    def __init__(self, context: InstaloaderContext):
+        self._context = context
+        self._graphql_query_timestamps = dict()  # type: Dict[str, List[float]]
+        self._graphql_earliest_next_request_time = 0.0
+
+    def sleep(self, secs: float):
+        """Wait given number of seconds."""
+        # Not static, to allow for the behavior of this method to depend on context-inherent properties, such as
+        # whether we are logged in.
+        # pylint:disable=no-self-use
+        time.sleep(secs)
+
+    def _dump_query_timestamps(self, current_time: float, failed_query_type: str):
+        windows = [10, 11, 15, 20, 30, 60]
+        self._context.error("Requests within last {} minutes grouped by type:"
+                            .format('/'.join(str(w) for w in windows)),
+                            repeat_at_end=False)
+        for query_type, times in self._graphql_query_timestamps.items():
+            reqs_in_sliding_window = [sum(t > current_time - w * 60 for t in times) for w in windows]
+            self._context.error(" {} {:>32}: {}".format(
+                "*" if query_type == failed_query_type else " ",
+                query_type,
+                " ".join("{:4}".format(reqs) for reqs in reqs_in_sliding_window)
+            ), repeat_at_end=False)
+
+    def count_per_sliding_window(self, query_type: str) -> int:
+        """Return how many GraphQL requests can be done within the sliding window."""
+        # Not static, to allow for the count_per_sliding_window to depend on context-inherent properties, such as
+        # whether we are logged in.
+        # pylint:disable=no-self-use,unused-argument
+        return 200
+
+    def query_waittime(self, query_type: str, current_time: float, untracked_queries: bool = False) -> float:
+        """Calculate time needed to wait before GraphQL query can be executed."""
+        sliding_window = 660
+        if query_type not in self._graphql_query_timestamps:
+            self._graphql_query_timestamps[query_type] = []
+        self._graphql_query_timestamps[query_type] = list(filter(lambda t: t > current_time - 60 * 60,
+                                                                 self._graphql_query_timestamps[query_type]))
+        reqs_in_sliding_window = list(filter(lambda t: t > current_time - sliding_window,
+                                             self._graphql_query_timestamps[query_type]))
+        count_per_sliding_window = self.count_per_sliding_window(query_type)
+        if len(reqs_in_sliding_window) < count_per_sliding_window and not untracked_queries:
+            return max(0.0, self._graphql_earliest_next_request_time - current_time)
+        next_request_time = min(reqs_in_sliding_window) + sliding_window + 6
+        if untracked_queries:
+            self._graphql_earliest_next_request_time = next_request_time
+        return max(next_request_time, self._graphql_earliest_next_request_time) - current_time
+
+    def wait_before_query(self, query_type: str) -> None:
+        """This method is called before a query to Instagram. It calls :meth:`RateController.sleep` to wait
+        until the request can be made."""
+        waittime = self.query_waittime(query_type, time.monotonic(), False)
+        assert waittime >= 0
+        if waittime > 15:
+            self._context.log("\nToo many queries in the last time. Need to wait {} seconds, until {:%H:%M}."
+                              .format(round(waittime), datetime.now() + timedelta(seconds=waittime)))
+        if waittime > 0:
+            self.sleep(waittime)
+        if query_type not in self._graphql_query_timestamps:
+            self._graphql_query_timestamps[query_type] = [time.monotonic()]
+        else:
+            self._graphql_query_timestamps[query_type].append(time.monotonic())
+
+    def handle_429(self, query_type: str) -> None:
+        """This method is called to handle a 429 Too Many Requests response. It calls :meth:`RateController.sleep` to
+         wait until we can repeat the same request."""
+        current_time = time.monotonic()
+        waittime = self.query_waittime(query_type, current_time, True)
+        assert waittime >= 0
+        self._dump_query_timestamps(current_time, query_type)
+        text_for_429 = ("Instagram responded with HTTP error \"429 - Too Many Requests\". Please do not run multiple "
+                        "instances of Instaloader in parallel or within short sequence. Also, do not use any Instagram "
+                        "App while Instaloader is running.")
+        self._context.error(textwrap.fill(text_for_429), repeat_at_end=False)
+        if waittime > 1.5:
+            self._context.error("The request will be retried in {} seconds, at {:%H:%M}."
+                                .format(round(waittime), datetime.now() + timedelta(seconds=waittime)),
+                                repeat_at_end=False)
+        if waittime > 0:
+            self.sleep(waittime)
@@ -0,0 +1,268 @@
+import base64
+import hashlib
+import json
+import os
+from contextlib import contextmanager
+from datetime import datetime, timedelta
+from lzma import LZMAError
+from typing import Any, Callable, Dict, Iterator, NamedTuple, Optional, Tuple, TypeVar
+
+from .exceptions import InvalidArgumentException, QueryReturnedBadRequestException
+from .instaloadercontext import InstaloaderContext
+
+FrozenNodeIterator = NamedTuple('FrozenNodeIterator',
+                                [('query_hash', str),
+                                 ('query_variables', Dict),
+                                 ('query_referer', Optional[str]),
+                                 ('context_username', Optional[str]),
+                                 ('total_index', int),
+                                 ('best_before', Optional[float]),
+                                 ('remaining_data', Optional[Dict])])
+FrozenNodeIterator.__doc__ = \
+    """A serializable representation of a :class:`NodeIterator` instance, saving its iteration state."""
+FrozenNodeIterator.query_hash.__doc__ = """The GraphQL ``query_hash`` parameter."""
+FrozenNodeIterator.query_variables.__doc__ = """The GraphQL ``query_variables`` parameter."""
+FrozenNodeIterator.query_referer.__doc__ = """The HTTP referer used for the GraphQL query."""
+FrozenNodeIterator.context_username.__doc__ = """The username who created the iterator, or ``None``."""
+FrozenNodeIterator.total_index.__doc__ = """Number of items that have already been returned."""
+FrozenNodeIterator.best_before.__doc__ = """Date when parts of the stored nodes might have expired."""
+FrozenNodeIterator.remaining_data.__doc__ = \
+    """The already-retrieved, yet-unprocessed ``edges`` and the ``page_info`` at time of freezing."""
+
+
+T = TypeVar('T')
+
+
+class NodeIterator(Iterator[T]):
+    """
+    Iterate the nodes within edges in a GraphQL pagination. Instances of this class are returned by many (but not all)
+    of Instaloader's :class:`Post`-returning functions (such as :meth:`Profile.get_posts` etc.).
+
+    What makes this iterator special is its ability to freeze/store its current state, e.g. to interrupt an iteration,
+    and later thaw/resume from where it left off.
+
+    You can freeze a NodeIterator with :meth:`NodeIterator.freeze`::
+
+       post_iterator = profile.get_posts()
+       try:
+           for post in post_iterator:
+               do_something_with(post)
+       except KeyboardInterrupt:
+           save("resume_information.json", post_iterator.freeze())
+
+    and later reuse it with :meth:`NodeIterator.thaw` on an equally-constructed NodeIterator::
+
+       post_iterator = profile.get_posts()
+       post_iterator.thaw(load("resume_information.json"))
+
+    A :class:`FrozenNodeIterator` can only be thawn with a matching NodeIterator, i.e. a NodeIterator instance that has
+    been constructed with the same parameters as the instance that is represented by the :class:`FrozenNodeIterator` in
+    question. This is to ensure that an iteration cannot be resumed in a wrong, unmatching loop. As a quick way to
+    distinguish iterators that are saved e.g. in files, there is the :attr:`NodeIterator.magic` string: Two
+    NodeIterators are matching if and only if they have the same magic.
+
+    See also :func:`resumable_iteration` for a high-level context manager that handles a resumable iteration.
+    """
+
+    _graphql_page_length = 50
+    _shelf_life = timedelta(days=29)
+
+    def __init__(self,
+                 context: InstaloaderContext,
+                 query_hash: str,
+                 edge_extractor: Callable[[Dict[str, Any]], Dict[str, Any]],
+                 node_wrapper: Callable[[Dict], T],
+                 query_variables: Optional[Dict[str, Any]] = None,
+                 query_referer: Optional[str] = None,
+                 first_data: Optional[Dict[str, Any]] = None):
+        self._context = context
+        self._query_hash = query_hash
+        self._edge_extractor = edge_extractor
+        self._node_wrapper = node_wrapper
+        self._query_variables = query_variables if query_variables is not None else {}
+        self._query_referer = query_referer
+        self._data = first_data
+        self._page_index = 0
+        self._total_index = 0
+        self._best_before = (None if first_data is None else
+                             datetime.now() + NodeIterator._shelf_life)
+
+    def _query(self, after: Optional[str] = None) -> Dict:
+        pagination_variables = {'first': NodeIterator._graphql_page_length}  # type: Dict[str, Any]
+        if after is not None:
+            pagination_variables['after'] = after
+        try:
+            data = self._edge_extractor(
+                self._context.graphql_query(
+                    self._query_hash, {**self._query_variables, **pagination_variables}, self._query_referer
+                )
+            )
+            self._best_before = datetime.now() + NodeIterator._shelf_life
+            return data
+        except QueryReturnedBadRequestException:
+            new_page_length = int(NodeIterator._graphql_page_length / 2)
+            if new_page_length >= 12:
+                NodeIterator._graphql_page_length = new_page_length
+                self._context.error("HTTP Error 400 (Bad Request) on GraphQL Query. Retrying with shorter page length.",
+                                    repeat_at_end=False)
+                return self._query(after)
+            else:
+                raise
+
+    def __iter__(self):
+        return self
+
+    def __next__(self) -> T:
+        if self._data is None:
+            self._data = self._query()
+        if self._page_index < len(self._data['edges']):
+            node = self._data['edges'][self._page_index]['node']
+            page_index, total_index = self._page_index, self._total_index
+            try:
+                self._page_index += 1
+                self._total_index += 1
+            except KeyboardInterrupt:
+                self._page_index, self._total_index = page_index, total_index
+                raise
+            return self._node_wrapper(node)
+        if self._data['page_info']['has_next_page']:
+            query_response = self._query(self._data['page_info']['end_cursor'])
+            page_index, data = self._page_index, self._data
+            try:
+                self._page_index = 0
+                self._data = query_response
+            except KeyboardInterrupt:
+                self._page_index, self._data = page_index, data
+                raise
+            return self.__next__()
+        raise StopIteration()
+
+    @property
+    def count(self) -> Optional[int]:
+        """The ``count`` as returned by Instagram. This is not always the total count this iterator will yield."""
+        return self._data.get('count') if self._data is not None else None
+
+    @property
+    def total_index(self) -> int:
+        """Number of items that have already been returned."""
+        return self._total_index
+
+    @property
+    def magic(self) -> str:
+        """Magic string for easily identifying a matching iterator file for resuming (hash of some parameters)."""
+        if 'blake2b' not in hashlib.algorithms_available:
+            magic_hash = hashlib.new('sha224')
+        else:
+            # Use blake2b when possible, i.e. on Python >= 3.6.
+            magic_hash = hashlib.blake2b(digest_size=6)  # type:ignore  # pylint: disable=no-member
+        magic_hash.update(json.dumps(
+            [self._query_hash, self._query_variables, self._query_referer, self._context.username]
+        ).encode())
+        return base64.urlsafe_b64encode(magic_hash.digest()).decode()
+
+    def freeze(self) -> FrozenNodeIterator:
+        """Freeze the iterator for later resuming."""
+        remaining_data = None
+        if self._data is not None:
+            remaining_data = {**self._data,
+                              'edges': (self._data['edges'][(max(self._page_index - 1, 0)):])}
+        return FrozenNodeIterator(
+            query_hash=self._query_hash,
+            query_variables=self._query_variables,
+            query_referer=self._query_referer,
+            context_username=self._context.username,
+            total_index=max(self.total_index - 1, 0),
+            best_before=self._best_before.timestamp() if self._best_before else None,
+            remaining_data=remaining_data,
+        )
+
+    def thaw(self, frozen: FrozenNodeIterator) -> None:
+        """
+        Use this iterator for resuming from earlier iteration.
+
+        :raises InvalidArgumentException:
+           If
+
+           - the iterator on which this method is called has already been used, or
+           - the given :class:`FrozenNodeIterator` does not match, i.e. belongs to a different iteration.
+        """
+        if self._total_index or self._page_index:
+            raise InvalidArgumentException("thaw() called on already-used iterator.")
+        if (self._query_hash != frozen.query_hash or
+                self._query_variables != frozen.query_variables or
+                self._query_referer != frozen.query_referer or
+                self._context.username != frozen.context_username):
+            raise InvalidArgumentException("Mismatching resume information.")
+        self._total_index = frozen.total_index
+        self._best_before = datetime.fromtimestamp(frozen.best_before) if frozen.best_before else None
+        self._data = frozen.remaining_data
+
+
+@contextmanager
+def resumable_iteration(context: InstaloaderContext,
+                        iterator: Iterator,
+                        load: Callable[[InstaloaderContext, str], Any],
+                        save: Callable[[FrozenNodeIterator, str], None],
+                        format_path: Callable[[str], str],
+                        check_bbd: bool = True,
+                        enabled: bool = True) -> Iterator[Tuple[bool, int]]:
+    """
+    High-level context manager to handle a resumable iteration that can be interrupted with a KeyboardInterrupt.
+
+    It can be used as follows to automatically load a previously-saved state into the iterator, save the iterator's
+    state when interrupted, and delete the resume file upon completion::
+
+       post_iterator = profile.get_posts()
+       with resumable_iteration(
+               context=L.context,
+               iterator=post_iterator,
+               load=lambda _, path: FrozenNodeIterator(**json.load(open(path))),
+               save=lambda fni, path: json.dump(fni._asdict(), open(path, 'w')),
+               format_path=lambda magic: "resume_info_{}.json".format(magic)
+       ) as (is_resuming, start_index):
+           for post in post_iterator:
+               do_something_with(post)
+
+    It yields a tuple (is_resuming, start_index).
+
+    When the passed iterator is not a :class:`NodeIterator`, it behaves as if ``resumable_iteration`` was not used,
+    just executing the inner body.
+
+    :param context: The :class:`InstaloaderContext`.
+    :param iterator: The fresh :class:`NodeIterator`.
+    :param load: Loads a FrozenNodeIterator from given path. The object is ignored if it has a different type.
+    :param save: Saves the given FrozenNodeIterator to the given path.
+    :param format_path: Returns the path to the resume file for the given magic.
+    :param check_bbd: Whether to check the best before date and reject an expired FrozenNodeIterator.
+    :param enabled: Set to False to disable all functionality and simply execute the inner body.
+    """
+    if not enabled or not isinstance(iterator, NodeIterator):
+        yield False, 0
+        return
+    is_resuming = False
+    start_index = 0
+    resume_file_path = format_path(iterator.magic)
+    resume_file_exists = os.path.isfile(resume_file_path)
+    if resume_file_exists:
+        try:
+            fni = load(context, resume_file_path)
+            if not isinstance(fni, FrozenNodeIterator):
+                raise InvalidArgumentException("Invalid type.")
+            if check_bbd and fni.best_before and datetime.fromtimestamp(fni.best_before) < datetime.now():
+                raise InvalidArgumentException("\"Best before\" date exceeded.")
+            iterator.thaw(fni)
+            is_resuming = True
+            start_index = iterator.total_index
+            context.log("Resuming from {}.".format(resume_file_path))
+        except (InvalidArgumentException, LZMAError, json.decoder.JSONDecodeError) as exc:
+            context.error("Warning: Not resuming from {}: {}".format(resume_file_path, exc))
+    try:
+        yield is_resuming, start_index
+    except KeyboardInterrupt:
+        os.makedirs(os.path.dirname(resume_file_path), exist_ok=True)
+        save(iterator.freeze(), resume_file_path)
+        context.log("\nSaved resume information to {}.".format(resume_file_path))
+        raise
+    if resume_file_exists:
+        os.unlink(resume_file_path)
+        context.log("Iteration complete, deleted resume information file {}.".format(resume_file_path))
@@ -9,7 +9,7 @@ from typing import Any, Dict, Iterator, List, Optional, Union
 from . import __version__
 from .exceptions import *
 from .instaloadercontext import InstaloaderContext
-
+from .nodeiterator import FrozenNodeIterator, NodeIterator

 PostSidecarNode = namedtuple('PostSidecarNode', ['is_video', 'display_url', 'video_url'])
 PostSidecarNode.__doc__ = "Item of a Sidecar Post."
@@ -26,7 +26,7 @@ PostCommentAnswer.likes_count.__doc__ = "Number of likes on comment."

 PostComment = namedtuple('PostComment', (*PostCommentAnswer._fields, 'answers')) # type: ignore
 for field in PostCommentAnswer._fields:
-    getattr(PostComment, field).__doc__ = getattr(PostCommentAnswer, field).__doc__
+    getattr(PostComment, field).__doc__ = getattr(PostCommentAnswer, field).__doc__  # pylint: disable=no-member
 PostComment.answers.__doc__ = r"Iterator which yields all :class:`PostCommentAnswer`\ s for the comment." # type: ignore

 PostLocation = namedtuple('PostLocation', ['id', 'name', 'slug', 'has_public_page', 'lat', 'lng'])
@@ -69,7 +69,6 @@ class Post:
        self._node = node
        self._owner_profile = owner_profile
        self._full_metadata_dict = None  # type: Optional[Dict[str, Any]]
-        self._rhx_gis_str = None         # type: Optional[str]
        self._location = None            # type: Optional[PostLocation]
        self._iphone_struct_ = None
        if 'iphone_struct' in node:
@@ -142,9 +141,11 @@ class Post:

    def _obtain_metadata(self):
        if not self._full_metadata_dict:
-            pic_json = self._context.get_json("p/{0}/".format(self.shortcode), params={})
-            self._full_metadata_dict = pic_json['entry_data']['PostPage'][0]['graphql']['shortcode_media']
-            self._rhx_gis_str = pic_json.get('rhx_gis')
+            pic_json = self._context.graphql_query(
+                '2b0673e0dc4580674a88d426fe00ea90',
+                {'shortcode': self.shortcode}
+            )
+            self._full_metadata_dict = pic_json['data']['shortcode_media']
            if self._full_metadata_dict is None:
                # issue #449
                self._context.error("Fetching Post metadata failed (issue #449). "
@@ -161,11 +162,6 @@ class Post:
        assert self._full_metadata_dict is not None
        return self._full_metadata_dict

-    @property
-    def _rhx_gis(self) -> Optional[str]:
-        self._obtain_metadata()
-        return self._rhx_gis_str
-
    @property
    def _iphone_struct(self) -> Dict[str, Any]:
        if not self._context.is_logged_in:
@@ -392,7 +388,7 @@ class Post:
                                     created_at_utc=datetime.utcfromtimestamp(node['created_at']),
                                     text=node['text'],
                                     owner=Profile(self._context, node['owner']),
-                                     likes_count=node['edge_liked_by']['count'])
+                                     likes_count=node.get('edge_liked_by', {}).get('count', 0))

        def _postcommentanswers(node):
            if 'edge_threaded_comments' not in node:
@@ -406,11 +402,14 @@ class Post:
                # If the answer's metadata already contains all comments, don't do GraphQL requests to obtain them
                yield from (_postcommentanswer(comment['node']) for comment in answer_edges)
                return
-            yield from (_postcommentanswer(answer_node) for answer_node in
-                        self._context.graphql_node_list("51fdd02b67508306ad4484ff574a0b62",
-                                                        {'comment_id': node['id']},
-                                                        'https://www.instagram.com/p/' + self.shortcode + '/',
-                                                        lambda d: d['data']['comment']['edge_threaded_comments']))
+            yield from NodeIterator(
+                self._context,
+                '51fdd02b67508306ad4484ff574a0b62',
+                lambda d: d['data']['comment']['edge_threaded_comments'],
+                _postcommentanswer,
+                {'comment_id': node['id']},
+                'https://www.instagram.com/p/{0}/'.format(self.shortcode),
+            )

        def _postcomment(node):
            return PostComment(*_postcommentanswer(node),
@@ -418,29 +417,22 @@ class Post:
        if self.comments == 0:
            # Avoid doing additional requests if there are no comments
            return
-        try:
-            comment_edges = self._field('edge_media_to_parent_comment', 'edges')
-            answers_count = sum([edge['node']['edge_threaded_comments']['count'] for edge in comment_edges])
-            threaded_comments_available = True
-        except KeyError:
-            comment_edges = self._field('edge_media_to_comment', 'edges')
-            answers_count = 0
-            threaded_comments_available = False
+
+        comment_edges = self._field('edge_media_to_comment', 'edges')
+        answers_count = sum([edge['node'].get('edge_threaded_comments', {}).get('count', 0) for edge in comment_edges])

        if self.comments == len(comment_edges) + answers_count:
            # If the Post's metadata already contains all parent comments, don't do GraphQL requests to obtain them
            yield from (_postcomment(comment['node']) for comment in comment_edges)
            return
-        yield from (_postcomment(node) for node in
-                    self._context.graphql_node_list(
-                        "97b41c52301f77ce508f55e66d17620e" if threaded_comments_available
-                        else "f0986789a5c5d17c2400faebf16efd0d",
-                        {'shortcode': self.shortcode},
-                        'https://www.instagram.com/p/' + self.shortcode + '/',
-                        lambda d:
-                        d['data']['shortcode_media'][
-                            'edge_media_to_parent_comment' if threaded_comments_available else 'edge_media_to_comment'],
-                        self._rhx_gis))
+        yield from NodeIterator(
+            self._context,
+            '97b41c52301f77ce508f55e66d17620e',
+            lambda d: d['data']['shortcode_media']['edge_media_to_parent_comment'],
+            _postcomment,
+            {'shortcode': self.shortcode},
+            'https://www.instagram.com/p/{0}/'.format(self.shortcode),
+        )

    def get_likes(self) -> Iterator['Profile']:
        """Iterate over all likes of the post. A :class:`Profile` instance of each likee is yielded."""
@@ -452,11 +444,14 @@ class Post:
            # If the Post's metadata already contains all likes, don't do GraphQL requests to obtain them
            yield from (Profile(self._context, like['node']) for like in likes_edges)
            return
-        yield from (Profile(self._context, node) for node in
-                    self._context.graphql_node_list("1cb6ec562846122743b61e492c85999f", {'shortcode': self.shortcode},
-                                                    'https://www.instagram.com/p/' + self.shortcode + '/',
-                                                    lambda d: d['data']['shortcode_media']['edge_liked_by'],
-                                                    self._rhx_gis))
+        yield from NodeIterator(
+            self._context,
+            '1cb6ec562846122743b61e492c85999f',
+            lambda d: d['data']['shortcode_media']['edge_liked_by'],
+            lambda n: Profile(self._context, n),
+            {'shortcode': self.shortcode},
+            'https://www.instagram.com/p/{0}/'.format(self.shortcode),
+        )

    @property
    def is_sponsored(self) -> bool:
@@ -537,7 +532,6 @@ class Profile:
        self._has_public_story = None  # type: Optional[bool]
        self._node = node
        self._has_full_metadata = False
-        self._rhx_gis = None
        self._iphone_struct_ = None
        if 'iphone_struct' in node:
            # if loaded from JSON with load_structure_from_file()
@@ -599,10 +593,9 @@ class Profile:
    def _obtain_metadata(self):
        try:
            if not self._has_full_metadata:
-                metadata = self._context.get_json('{}/'.format(self.username), params={})
+                metadata = self._context.get_json('{}/feed/'.format(self.username), params={})
                self._node = metadata['entry_data']['ProfilePage'][0]['graphql']['user']
                self._has_full_metadata = True
-                self._rhx_gis = metadata.get('rhx_gis')
        except (QueryReturnedNotFoundException, KeyError) as err:
            top_search_results = TopSearchResults(self._context, self.username)
            similar_profiles = [profile.username for profile in top_search_results.get_profiles()]
@@ -735,8 +728,7 @@ class Profile:
                                                        'include_reel': False, 'include_suggested_users': False,
                                                        'include_logged_out_extras': True,
                                                        'include_highlight_reels': False},
-                                                       'https://www.instagram.com/{}/'.format(self.username),
-                                                       self._rhx_gis)
+                                                       'https://www.instagram.com/{}/'.format(self.username))
            self._has_public_story = data['data']['user']['has_public_story']
        assert self._has_public_story is not None
        return self._has_public_story
@@ -787,86 +779,110 @@ class Profile:
 	   Use :attr:`profile_pic_url`."""
        return self.profile_pic_url

-    def get_posts(self) -> Iterator[Post]:
-        """Retrieve all posts from a profile."""
-        self._obtain_metadata()
-        yield from (Post(self._context, node, self) for node in
-                    self._context.graphql_node_list("472f257a40c653c64c666ce877d59d2b",
-                                                    {'id': self.userid},
-                                                    'https://www.instagram.com/{0}/'.format(self.username),
-                                                    lambda d: d['data']['user']['edge_owner_to_timeline_media'],
-                                                    self._rhx_gis,
-                                                    self._metadata('edge_owner_to_timeline_media')))
+    def get_posts(self) -> NodeIterator[Post]:
+        """Retrieve all posts from a profile.

-    def get_saved_posts(self) -> Iterator[Post]:
-        """Get Posts that are marked as saved by the user."""
+        :rtype: NodeIterator[Post]"""
+        self._obtain_metadata()
+        return NodeIterator(
+            self._context,
+            '472f257a40c653c64c666ce877d59d2b',
+            lambda d: d['data']['user']['edge_owner_to_timeline_media'],
+            lambda n: Post(self._context, n, self),
+            {'id': self.userid},
+            'https://www.instagram.com/{0}/'.format(self.username),
+            self._metadata('edge_owner_to_timeline_media'),
+        )
+
+    def get_saved_posts(self) -> NodeIterator[Post]:
+        """Get Posts that are marked as saved by the user.
+
+        :rtype: NodeIterator[Post]"""

        if self.username != self._context.username:
            raise LoginRequiredException("--login={} required to get that profile's saved posts.".format(self.username))

        self._obtain_metadata()
-        yield from (Post(self._context, node) for node in
-                    self._context.graphql_node_list("f883d95537fbcd400f466f63d42bd8a1",
-                                                    {'id': self.userid},
-                                                    'https://www.instagram.com/{0}/'.format(self.username),
-                                                    lambda d: d['data']['user']['edge_saved_media'],
-                                                    self._rhx_gis,
-                                                    self._metadata('edge_saved_media')))
+        return NodeIterator(
+            self._context,
+            'f883d95537fbcd400f466f63d42bd8a1',
+            lambda d: d['data']['user']['edge_saved_media'],
+            lambda n: Post(self._context, n),
+            {'id': self.userid},
+            'https://www.instagram.com/{0}/'.format(self.username),
+            self._metadata('edge_saved_media'),
+        )

-    def get_tagged_posts(self) -> Iterator[Post]:
+    def get_tagged_posts(self) -> NodeIterator[Post]:
        """Retrieve all posts where a profile is tagged.

+        :rtype: NodeIterator[Post]
+
        .. versionadded:: 4.0.7"""
        self._obtain_metadata()
-        yield from (Post(self._context, node, self if int(node['owner']['id']) == self.userid else None) for node in
-                    self._context.graphql_node_list("e31a871f7301132ceaab56507a66bbb7",
-                                                    {'id': self.userid},
-                                                    'https://www.instagram.com/{0}/'.format(self.username),
-                                                    lambda d: d['data']['user']['edge_user_to_photos_of_you'],
-                                                    self._rhx_gis))
+        return NodeIterator(
+            self._context,
+            'e31a871f7301132ceaab56507a66bbb7',
+            lambda d: d['data']['user']['edge_user_to_photos_of_you'],
+            lambda n: Post(self._context, n, self if int(n['owner']['id']) == self.userid else None),
+            {'id': self.userid},
+            'https://www.instagram.com/{0}/'.format(self.username),
+        )

-    def get_igtv_posts(self) -> Iterator[Post]:
+    def get_igtv_posts(self) -> NodeIterator[Post]:
        """Retrieve all IGTV posts.

+        :rtype: NodeIterator[Post]
+
        .. versionadded:: 4.3"""
        self._obtain_metadata()
-        yield from (Post(self._context, node, self) for node in
-                    self._context.graphql_node_list('bc78b344a68ed16dd5d7f264681c4c76',
-                                                    {'id': self.userid},
-                                                    'https://www.instagram.com/{0}/channel/'.format(self.username),
-                                                    lambda d: d['data']['user']['edge_felix_video_timeline'],
-                                                    self._rhx_gis,
-                                                    self._metadata('edge_felix_video_timeline')))
+        return NodeIterator(
+            self._context,
+            'bc78b344a68ed16dd5d7f264681c4c76',
+            lambda d: d['data']['user']['edge_felix_video_timeline'],
+            lambda n: Post(self._context, n, self),
+            {'id': self.userid},
+            'https://www.instagram.com/{0}/channel/'.format(self.username),
+            self._metadata('edge_felix_video_timeline'),
+        )

-    def get_followers(self) -> Iterator['Profile']:
+    def get_followers(self) -> NodeIterator['Profile']:
        """
        Retrieve list of followers of given profile.
        To use this, one needs to be logged in and private profiles has to be followed.
+
+        :rtype: NodeIterator[Profile]
        """
        if not self._context.is_logged_in:
            raise LoginRequiredException("--login required to get a profile's followers.")
        self._obtain_metadata()
-        yield from (Profile(self._context, node) for node in
-                    self._context.graphql_node_list("37479f2b8209594dde7facb0d904896a",
-                                                    {'id': str(self.userid)},
-                                                    'https://www.instagram.com/' + self.username + '/',
-                                                    lambda d: d['data']['user']['edge_followed_by'],
-                                                    self._rhx_gis))
+        return NodeIterator(
+            self._context,
+            '37479f2b8209594dde7facb0d904896a',
+            lambda d: d['data']['user']['edge_followed_by'],
+            lambda n: Profile(self._context, n),
+            {'id': str(self.userid)},
+            'https://www.instagram.com/{0}/'.format(self.username),
+        )

-    def get_followees(self) -> Iterator['Profile']:
+    def get_followees(self) -> NodeIterator['Profile']:
        """
        Retrieve list of followees (followings) of given profile.
        To use this, one needs to be logged in and private profiles has to be followed.
+
+        :rtype: NodeIterator[Profile]
        """
        if not self._context.is_logged_in:
            raise LoginRequiredException("--login required to get a profile's followees.")
        self._obtain_metadata()
-        yield from (Profile(self._context, node) for node in
-                    self._context.graphql_node_list("58712303d941c6855d4e888c5f0cd22f",
-                                                    {'id': str(self.userid)},
-                                                    'https://www.instagram.com/' + self.username + '/',
-                                                    lambda d: d['data']['user']['edge_follow'],
-                                                    self._rhx_gis))
+        return NodeIterator(
+            self._context,
+            '58712303d941c6855d4e888c5f0cd22f',
+            lambda d: d['data']['user']['edge_follow'],
+            lambda n: Profile(self._context, n),
+            {'id': str(self.userid)},
+            'https://www.instagram.com/{0}/'.format(self.username),
+        )

    def get_similar_accounts(self) -> Iterator['Profile']:
        """
@@ -881,8 +897,8 @@ class Profile:
        yield from (Profile(self._context, edge["node"]) for edge in
                    self._context.graphql_query("ad99dd9d3646cc3c0dda65debcd266a7",
                                                {"user_id": str(self.userid), "include_chaining": True},
-                                                "https://www.instagram.com/{0}/".format(self.username),
-                                                self._rhx_gis)["data"]["user"]["edge_chaining"]["edges"])
+                                                "https://www.instagram.com/{0}/"
+                                                .format(self.username))["data"]["user"]["edge_chaining"]["edges"])


 class StoryItem:
@@ -1318,7 +1334,7 @@ class Hashtag:
            yield from (Post(self._context, edge["node"]) for edge in conn["edges"])

    def get_all_posts(self) -> Iterator[Post]:
-        """Yields all posts, i.e. all most recent posts and the top posts, in chronological order."""
+        """Yields all posts, i.e. all most recent posts and the top posts, in almost-chronological order."""
        sorted_top_posts = iter(sorted(self.get_top_posts(), key=lambda p: p.date_utc, reverse=True))
        other_posts = self.get_posts()
        next_top = next(sorted_top_posts, None)
@@ -1421,7 +1437,7 @@ class TopSearchResults:
        return self._searchstring


-JsonExportable = Union[Post, Profile, StoryItem, Hashtag]
+JsonExportable = Union[Post, Profile, StoryItem, Hashtag, FrozenNodeIterator]


 def save_structure_to_file(structure: JsonExportable, filename: str) -> None:
@@ -1470,6 +1486,8 @@ def load_structure_from_file(context: InstaloaderContext, filename: str) -> Json
            return StoryItem(context, json_structure['node'])
        elif node_type == "Hashtag":
            return Hashtag(context, json_structure['node'])
+        elif node_type == "FrozenNodeIterator":
+            return FrozenNodeIterator(**json_structure['node'])
        else:
            raise InvalidArgumentException("{}: Not an Instaloader JSON.".format(filename))
    elif 'shortcode' in json_structure:
@@ -5,7 +5,7 @@ import shutil
 import tempfile
 import unittest
 from itertools import islice
-from typing import Dict, List
+from typing import Optional

 import instaloader

@@ -23,8 +23,7 @@ PRIVATE_PROFILE_ID = 1706625676
 EMPTY_PROFILE = "not_public"
 EMPTY_PROFILE_ID = 1928659031

-# Preserve query timestamps (rate control) between tests to not get rate limited
-instaloadercontext_query_timestamps = dict()  # type: Dict[str, List[float]]
+ratecontroller = None  # type: Optional[instaloader.RateController]


 class TestInstaloaderAnonymously(unittest.TestCase):
@@ -37,13 +36,15 @@ class TestInstaloaderAnonymously(unittest.TestCase):
                                         download_comments=True,
                                         save_metadata=True)
        self.L.context.raise_all_errors = True
-        # pylint:disable=protected-access
-        self.L.context._graphql_query_timestamps = instaloadercontext_query_timestamps.copy()
+        if ratecontroller is not None:
+            # pylint:disable=protected-access
+            ratecontroller._context = self.L.context
+            self.L.context._rate_controller = ratecontroller

    def tearDown(self):
        # pylint:disable=global-statement,protected-access
-        global instaloadercontext_query_timestamps
-        instaloadercontext_query_timestamps = self.L.context._graphql_query_timestamps.copy()
+        global ratecontroller
+        ratecontroller = self.L.context._rate_controller
        self.L.close()
        os.chdir('/')
        print("Removing {}".format(self.dir))
@@ -94,6 +95,10 @@ class TestInstaloaderAnonymously(unittest.TestCase):
        self.assertEqual(EMPTY_PROFILE.lower(),
                         instaloader.Profile.from_id(self.L.context, EMPTY_PROFILE_ID).username)

+    def test_get_username_by_name_empty(self):
+        self.assertEqual(EMPTY_PROFILE_ID,
+                         instaloader.Profile.from_username(self.L.context, EMPTY_PROFILE).userid)
+
    def test_post_from_mediaid(self):
        for post in instaloader.Profile.from_username(self.L.context, PUBLIC_PROFILE).get_posts():
            post2 = instaloader.Post.from_mediaid(self.L.context, post.mediaid)
Author	SHA1	Message	Date
Alexander Graf	3a6ba6485e	Release of Version 4.5.1	2020-08-20 11:08:08 +02:00
Alexander Graf	177a299ced	Improve 429 error message This text_for_429 error message is expected to be less confusing for new users. Further improve the output formatting of _dump_query_timestamps().	2020-08-20 10:38:06 +02:00
Alexander Graf	4a1fe035af	First Release Candidate for Version 4.5.1rc1	2020-08-17 18:29:37 +02:00
Alexander Graf	db86f81be7	Fix support for Python 3.5 prior Python 3.5.4 (#777 )	2020-08-17 18:25:22 +02:00
Alexander Graf	020d412342	Fix some video download edge cases (#776 ) Fixes unintended sidecar video skipping if --no-pictures is set. Fixes video download if video thumbnail access fails with 410 error.	2020-08-17 18:23:24 +02:00
lapourgagner	6f57197afb	Fixed issue #769 by using os.replace instead os.rename (#775 )	2020-08-17 18:17:23 +02:00
Alexander Graf	57393f619d	doc: Don't force links to be opened in new tab	2020-08-09 08:08:17 +02:00
Alexander Graf	d5dd8cec00	Fix Downloads/Month Badge Change to pepy.tech as source for this badge.	2020-08-06 09:43:22 +02:00
Alexander Graf	3652e009d9	Release of Version 4.5	2020-08-02 10:32:44 +02:00
Alexander Graf	ecde92d05e	Minor fix of Hashtag.get_all_posts() docstring	2020-08-02 10:23:09 +02:00
Alexander Graf	a8d3851956	Nicer syntax for resumable_iteration usage	2020-08-02 10:22:45 +02:00
Alexander Graf	c71873231d	Minor Documentation Fixes (mostly spelling&grammar)	2020-07-31 22:12:27 +02:00
Alexander Graf	3823b1e593	Documentation Template: Use ARIA roles This fixes the full text functionality of the search.	2020-07-31 11:41:22 +02:00
Alexander Graf	6c779e8d4b	First Beta Release for Version 4.5	2020-07-30 17:17:55 +02:00
Alexander Graf	c24015a93d	Merge branch 'master' into upcoming/v4.5	2020-07-30 17:16:00 +02:00
Alexander Graf	2c56e6994a	Enhance Documentation Theme Improvements on our bootstrap 4 theme for sphinx.	2020-07-30 17:02:43 +02:00
Alexander Graf	6791e9a2f8	Textual Improvements on Documentation - Fix spelling errors / typos, - Remove deprecated command line options from doc.	2020-07-30 16:41:45 +02:00
Alexander Graf	64652f3994	Feature Request template: PR -> pull request	2020-07-30 13:20:02 +02:00
Alexander Graf	4f5b223b60	Minor docs fixes	2020-07-26 19:00:01 +02:00
Alexander Graf	2a3711687f	Update deps used in CI (esp. MyPy and PyLint)	2020-07-26 18:54:23 +02:00
Alexander Graf	612842f255	Minor nodeiterator-related docs improvements	2020-07-26 18:05:46 +02:00
Alexander Graf	fe9cd653e7	docs: Improvements to branded syntax highlighting	2020-07-26 18:01:33 +02:00
Alexander Graf	82e7b19d6d	docs: Split as-module into smaller documents	2020-07-26 18:00:37 +02:00
Alexander Graf	5d7c36399f	Windows: Sanitize Path of Resume Files	2020-07-24 18:35:10 +02:00
Alexander Graf	f0e66ef465	Windows EXE: Fix CommandNotFoundException Fixes CommandNotFoundException that occured when started from within a directory whose name contains a space.	2020-07-24 17:19:48 +02:00
Alexander Graf	f71ea562af	First Alpha Release for Version 4.5	2020-07-21 17:37:45 +02:00
Alexander Graf	bc40b82f94	Resume a previously-aborted post download loop (#732 ) With this change, Instaloader is capable of resuming a previously-aborted download loop. To do so, it creates a JSON file within the target directory when interrupted, that contains all the necessary information to later resume that operation. Resuming an interrupted download is supported for most, but not all targets. It is supported for: - Regular profile posts, - IGTV posts - Saved posts, - Tagged posts, - Explore posts.	2020-07-21 17:28:55 +02:00
Alexander Graf	c817d1901a	Merge branch 'master' into upcoming/v4.5	2020-07-18 15:32:05 +02:00
Alexander Graf	e232c82d5e	Release of Version 4.4.5	2020-07-18 15:24:31 +02:00
Alexander Graf	601dd0b106	First Release Candidate for Version 4.4.5	2020-07-15 10:26:22 +02:00
Alexander Graf	4c20b573e2	Fix KeyError on Posts with very few comments	2020-07-15 10:17:20 +02:00
Cadence EmberandAlexander Graf	db63e5c7e5	Bypass IP-based redirects to /accounts/login (#727 ) * Bypass IP-based redirects to /accounts/login There are two main changes made: For users, we request /{username}/feed/ instead of /{username}/. For some reason, this completely bypasses the login redirect. This page doesn't work in browser while blocked, but fortunately all the data we need is just present in the HTML page. For posts, we change from using the /p/ page to using the graphql endpoint for the same data, which is still subject to graphql rate limits, but is not subject to login redirects. The data is identical between the two pages, apart from the object keys being sorted differently and rhx_gis being missing on graphql. Yes, this now unblocks access from VPNs, Tor, cloud servers, etc. * Apply requested patch to fix comments * Remove rhx_gis from Post and Profile Co-authored-by: Alexander Graf <17130992+aandergr@users.noreply.github.com>	2020-07-15 09:46:39 +02:00
Alexander Graf	fbe05a1add	More flexible Rate Controller (#716 ) Move InstaloaderContext's rate controlling logic into a class RateController with fine-grained methods to enable easily changing Instaloader's behavior regarding rate limits.	2020-07-11 13:54:31 +02:00
Alexander Graf	e21d34124d	Add codesnippet for almost chronological order Such as for downloading hashtag feeds, as discussed in #666 and contributed by @e2tovar. Also change comment color to grey in codesnippets in documentation.	2020-06-21 19:38:16 +02:00
Lars Lindqvist	fa62025ea3	Deprecate commit-mode (#697 ) Rather than checking the json file to make sure posts have been successfully downloaded, data is stored in a temporary file which is renamed when downloading has finished, as suggested in #257.	2020-06-20 16:23:31 +02:00
michalens	66b3ec6928	Add explanation on downloading single post (#694 )	2020-06-19 19:39:08 +02:00
Alexander Graf	f7d82495bd	Consistently use native '\' path separator on Windows Use os.path.join() for joining paths rather than +'/'+. os.path.join() uses '/' on Unix-ish and '\' on Windows. On Windows, this makes the following improvements: - Visual consistency with / and \ not being mixed-up in one path (os.path.join() has already been used in many places). - Allow using the \\?\ bypass method, e.g. with --dirname-pattern=\\?\D:\{target} The \\?\ bypass method fails when '/' is used as path separator. With the \\?\ bypass method, we can create directories whose name would otherwise be considered invalid, such as directories starting with "aux.".	2020-06-19 19:33:46 +02:00
Alexander Graf	b0a430051e	Restrict GH Action lint workflow on Python 3.8 Now that Travis-CI status reporting works again, there is no need to run the tests twice.	2020-06-18 08:33:25 +02:00
Alexander Graf	8b56b90e72	Less intrusive PR template	2020-06-18 08:32:46 +02:00
Alexander Graf	0c93e63615	Update troubleshooting.rst, Minor fixes to contributing.rst	2020-06-16 21:44:22 +02:00
Alexander Graf	65025e61d9	Update issue/PR templates, contributing.rst, CI - Keep documentation in sync with master branch - Run PyLint and MyPy on Github Actions - Update Issue templates, enforce usage of issue templates, add links to resources in Instaloader Documentation - Update contributing.rst	2020-06-16 20:26:58 +02:00
Alexander Graf	11baf493db	Release of Version 4.4.4	2020-06-11 20:10:52 +02:00
Alexander Graf	2345d90e0e	Fix docs building on Windows	2020-06-09 13:03:48 +02:00
Alexander Graf	71b09aed0f	Warn if --login parameter is not a username Closes #624.	2020-06-09 11:44:06 +02:00
Alexander Graf	fd6d8542c9	Add a unit test for an empty private profile	2020-06-09 11:44:06 +02:00