Release of Version 4.12

Fix GitHub Actions Node.js deprecation warnings.

Use actions/setup-python cache where appropriate.

.github/ISSUE_TEMPLATE/bug_report.md

@@ -24,4 +24,9 @@ If applicable, add error messages and tracebacks to help explain your problem.
 **Additional context**
 Add any other context about the problem here.
 
-<!-- please also see https://instaloader.github.io/contributing.html for how to report a bug -->
+<!--
+Also see https://instaloader.github.io/contributing.html for how to report a bug.
+
+Please understand that a bug issue that lacks the required information cannot be
+processed.
+-->

.github/ISSUE_TEMPLATE/feature_request.md

@@ -6,13 +6,15 @@ labels: "feature suggestion"
 ---
 
 **Provide us a use case of the feature**
-How could the user invoke the new function? Which problem would it solve? If new information is obtained, how would it be further processed?
+How could the user invoke the new function? Which problem would it solve?
+If new information is obtained, how would it be further processed?
 
 **Describe the solution you'd like**
 A clear and concise description of what you want to happen.
 
 **Describe alternatives you've considered**
 A clear and concise description of any alternative solutions or features you've considered.
+How do you achieve the here-described goal without the feature you are requesting?
 
 **If the feature request is accepted, would you be willing to submit a pull request?**
 Yes / No
@@ -21,4 +23,9 @@ Yes / No
 **Additional context**
 Add any other context about the feature request here.
 
-<!-- please also see https://instaloader.github.io/contributing.html for how to suggest a feature -->
+<!-- 
+Also see https://instaloader.github.io/contributing.html for how to suggest a feature.
+
+Please understand that a feature suggestion issue that lacks the required information cannot be
+processed.
+-->

.github/ISSUE_TEMPLATE/question.md

@@ -7,6 +7,12 @@ labels: question
 
 Your question here...
 
-<!-- Please see our documentation: https://instaloader.github.io/ -->
+<!--
+People will be better able and more motivated to provide help if your question
 
-<!-- Basic questions regarding Instaloader's usage might be more appropriate to ask on Stack Overflow -->
+ - has a concise and meaningful title,
+ - is not covered by the documentation: https://instaloader.github.io/,
+ - is actually related to Instaloader (this is not a Python help forum),
+ - has not already been answered here or on other sites such as Stack Overflow: https://stackoverflow.com/questions/tagged/instaloader,
+ - is written in a polite and welcoming tone.
+-->

.github/funding.yml

@@ -0,0 +1,2 @@
+github: [aandergr]
+custom: "https://instaloader.github.io/sponsors.html"

.github/pull_request_template.md

@@ -1,7 +1,8 @@
 
 
 <!--
-Please describe:
+Thanks for your willingness to contribute to Instaloader! Please briefly
+describe
 
 - A motivation for this change, e.g.
   - Fixes # .

.github/workflows/documentation.yml

@@ -0,0 +1,32 @@
+name: "Documentation"
+
+on: [push, pull_request]
+
+jobs:
+  docs:
+    name: "Documentation"
+    runs-on: ubuntu-latest
+    steps:
+      - name: "Checkout Instaloader Repository"
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0  # needed for building docs
+      - name: "Setup Python"
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.12
+          cache: 'pipenv'
+      - name: "Install Dependencies"
+        run: |
+          python -m pip install pipenv==2023.12.1
+          pipenv --python `python --version | grep -Eo '3\.[0-9]+'` sync --dev
+      - name: "Build Documentation"
+        run: pipenv --python `python --version | grep -Eo '3\.[0-9]+'` run make -C docs html SPHINXOPTS="-W -n"
+      - name: "Deploy Documentation"
+        if: github.event_name == 'push' && github.ref == 'refs/heads/master'
+        uses: JamesIves/github-pages-deploy-action@v4
+        with:
+          branch: master
+          repository-name: instaloader/instaloader.github.io
+          folder: docs/_build/html
+          token: ${{ secrets.PAGES_DEPLOYMENT_TOKEN }}

.github/workflows/lint.yml

@@ -7,24 +7,22 @@ jobs:
     name: PyLint and MyPy
     runs-on: ubuntu-latest
     strategy:
+      fail-fast: false
       matrix:
-        python-version: [3.8]
+        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
     steps:
       - name: Checkout Instaloader Repository
-        uses: actions/checkout@v2
-        with:
-          fetch-depth: 0  # needed for building docs
+        uses: actions/checkout@v4
       - name: Setup Python
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
+          cache: 'pipenv'
       - name: Install Dependencies
         run: |
-          python -m pip install pipenv==2020.6.2
-          pipenv sync --dev
+          python -m pip install pipenv==2023.12.1
+          pipenv --python `python --version | grep -Eo '3\.[0-9]+'` sync --dev
       - name: PyLint
-        run: pipenv run pylint instaloader
+        run: pipenv --python `python --version | grep -Eo '3\.[0-9]+'` run pylint instaloader
       - name: MyPy
-        run: pipenv run mypy -m instaloader
-      - name: Build Documentation
-        run: pipenv run make -C docs html SPHINXOPTS="-W -n"
+        run: pipenv --python `python --version | grep -Eo '3\.[0-9]+'` run mypy -m instaloader

.github/workflows/packages.yml

@@ -0,0 +1,38 @@
+name: "Publish Packages"
+
+on:
+  push:
+    tags:
+      - v*
+
+jobs:
+  publish:
+    name: "Publish Package"
+    runs-on: ubuntu-latest
+    steps:
+      - name: "Checkout Repository"
+        uses: actions/checkout@v4
+      - name: "Setup Python"
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.12
+      - name: "Get the tagged version"
+        id: get_version
+        env:
+          GITHUB_REF: ${{ github.ref }}
+        run: echo "VERSION=${GITHUB_REF/refs\/tags\/v/}" >> $GITHUB_OUTPUT
+        shell: bash
+      - name: "Install Dependencies"
+        run: python -m pip install setuptools
+      - name: "Generate Python Package"
+        run: python setup.py sdist
+      - name: "Publish Distribution to PyPI"
+        uses: pypa/gh-action-pypi-publish@master
+        with:
+          password: ${{ secrets.PYPI_PASSWORD }}
+      - name: "Publish Distribution to AUR"
+        run: deploy/arch/deploy.sh $VERSION_TAG
+        env:
+          VERSION_TAG: v${{ steps.get_version.outputs.VERSION }}
+          AUR_KEY: ${{ secrets.AUR_KEY }}
+          AUR_IV: ${{ secrets.AUR_IV }}

.github/workflows/stale.yml

@@ -2,7 +2,7 @@ name: Mark stale issues and pull requests
 
 on:
   schedule:
-  - cron: "10 14 * * *"
+  - cron: "55 7 * * *"
 
 jobs:
   stale:
@@ -11,24 +11,24 @@ jobs:
     runs-on: ubuntu-latest
     
     steps:
-    - uses: actions/stale@v3
+    - uses: actions/stale@v9
       with:
         repo-token: ${{ secrets.GITHUB_TOKEN }}
         only-labels: 'question'
         stale-issue-message: 'There has been no activity on this question for an extended period of time. This issue will be closed after further 14 days of inactivity.'
         stale-issue-label: 'stale'
         exempt-issue-labels: 'leave open'
-        days-before-stale: 15
+        days-before-stale: 21
         days-before-close: -1
         remove-stale-when-updated: false
-    - uses: actions/stale@v1
+    - uses: actions/stale@v9
       with:
         repo-token: ${{ secrets.GITHUB_TOKEN }}
         stale-issue-message: 'There has been no activity on this issue for an extended period of time. This issue will be closed after further 14 days of inactivity.'
         stale-pr-message: 'There has been no activity on this Pull Request for an extended period of time. This Pull Request will be closed after further 14 days of inactivity.'
         stale-issue-label: 'stale'
         stale-pr-label: 'stale'
-        exempt-issue-label: 'leave open'
-        exempt-pr-label: 'leave open'
-        days-before-stale: 90
+        exempt-issue-labels: 'leave open'
+        exempt-pr-labels: 'leave open'
+        days-before-stale: 189
         days-before-close: 14

.github/workflows/windows_exe.yml

@@ -8,17 +8,18 @@ jobs:
     runs-on: windows-latest
     steps:
       - name: Checkout Instaloader repository
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
       - name: Setup Python
-        uses: actions/setup-python@v1
+        uses: actions/setup-python@v5
         with:
-          python-version: 3.8
+          python-version: "3.12"
+          cache: 'pipenv'
           architecture: x64
       - name: Get the tagged version
         id: get_version
         env:
           GITHUB_REF: ${{ github.ref }}
-        run: echo ::set-output name=VERSION::${GITHUB_REF/refs\/tags\/v/}
+        run: echo "VERSION=${GITHUB_REF/refs\/tags\/v/}" >> $GITHUB_OUTPUT
         shell: bash
       - name: Run custom python script for EXE creation
         env:

.travis.yml

@@ -1,36 +0,0 @@
-dist: xenial
-language: python
-python:
- - "3.5"
- - "3.6"
- - "3.7"
-install:
- - pipenv sync --dev
-script:
- - pylint instaloader
- - mypy -m instaloader
- - make -C docs html SPHINXOPTS="-W -n"
-addons:
-  ssh_known_hosts: aur.archlinux.org
-deploy:
-  - provider: pypi
-    user: aandergr
-    on:
-      tags: true
-      python: 3.6
-    password:
-      secure: KBsATtXNnEzqr0X2ALdyx2zaYj2EUfK8YjP4jEOilBELpJ4evHYXVgRkn9CKXGsD7LKbWmR3gz2rr8NZ6U42pHw0fpQHizz91MpJSKbNY7RZZh4UVWMDh81izeglchWAF+m3Wdpxpo/GBDbrZIbRQFhPgB+kjSzJo3uiCS+yVXtGG+6csJHzd0WkJjsVHz+XSdxyOiLy/qFF3nmY6ZrVrgw6WbX2YLbx9je6DxK3YRrnBqgMgH7Ui2xWWbgJCk3KKpQTMiMKnTa9mdeMJAd26YMCz1gG/X7ck6VNly/6bs5+Zq7/bU0dSolhPBhNthyccYdS0IeGPdPF3YqatblJ4fLGTfHn8oJs3/T11yIW9pvOT3UftEynPOUX+en2Qnb/3WfgwUfVts8i8DAYgEFLGEzbF5XMGy4gwAD/bZ0JW0pmgAG2PZsdOokAt+ydlNj/uxq1GFW2doZbul6k4t03p1mik6itYscvuhKoEm8Yw6wr1P9aDvMR6FFpSu4gDt2XrKmRmWkSXxpNeCfZujTjyAnISxLVQGSY7BG5K0Z2HbDtqqNz9nj4WgNGzZmngiRofCevWR+vPIDUDW7NCX03csfcFS9RRllBO/BrL7NVUV+Qp9dfTQYp+8SqYLWgHtDoXdgyMREjJaAmbx2u/TYTY8bADZ5F3vGUpv2j5hP3NTU=
-  - provider: pages
-    skip-cleanup: true
-    github-token: $GITHUB_TOKEN
-    local-dir: docs/_build/html
-    repo: instaloader/instaloader.github.io
-    target-branch: master
-    on:
-      branch: master
-      python: 3.6
-  - provider: script
-    script: deploy/arch/deploy.sh $TRAVIS_TAG
-    on:
-      tags: true
-      python: 3.6

AUTHORS.md

@@ -6,3 +6,5 @@ Instaloader is written by
 - Alexander Graf (@aandergr)
 - André Koch-Kramer (@Thammus)
 - Lars Lindqvist (@e5150)
+- Eduardo M Kalinowksi (@ekalin)
+- ... and many more, see https://github.com/instaloader/instaloader/graphs/contributors

CODE_OF_CONDUCT.md

@@ -0,0 +1,135 @@
+
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, religion, or sexual identity
+and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the
+  overall community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or
+  advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email
+  address, without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official e-mail address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement by opening an
+issue or contacting one or more of the project maintainers.
+
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series
+of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or
+permanent ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior,  harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within
+the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.0, available at
+[https://www.contributor-covenant.org/version/2/0/code_of_conduct.html][v2.0].
+
+Community Impact Guidelines were inspired by 
+[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at
+[https://www.contributor-covenant.org/faq][FAQ]. Translations are available 
+at [https://www.contributor-covenant.org/translations][translations].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.0]: https://www.contributor-covenant.org/version/2/0/code_of_conduct.html
+[Mozilla CoC]: https://github.com/mozilla/diversity
+[FAQ]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations
+

LICENSE

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2016-2019 Alexander Graf and André Koch-Kramer.
+Copyright (c) 2016-2020 Alexander Graf and André Koch-Kramer.
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

Pipfile

@@ -6,11 +6,17 @@ verify_ssl = true
 [dev-packages]
 pylint = "*"
 mypy = "*"
-sphinx = "~=2.3"
+sphinx = "*"
 pyinstaller = "*"
-pefile = "*"
-pywin32-ctypes = "*"
 psutil = "*"
+types-requests = "*"
+sphinx-autodoc-typehints = "*"
+sphinxcontrib-jquery = "*"
+tomli = "*"           # Needed for Pylint for "python_version < '3.11'"
+dill = "*"            # Needed for Pylint for "python_version < '3.11'"
+pywin32-ctypes = "*"  # Needed for pyinstaller on Windows
+pefile = "*"          # Needed for pyinstaller on Windows
 
 [packages]
 requests = "*"
+browser-cookie3 = "*"

README.rst

@@ -2,11 +2,7 @@
 
 .. badges-start
 
-|travis| |pypi| |pyversion| |license| |aur| |contributors| |downloads|
-
-.. |travis| image:: https://img.shields.io/travis/instaloader/instaloader/master.svg
-   :alt: Travis-CI Build Status
-   :target: https://travis-ci.org/instaloader/instaloader
+|pypi| |pyversion| |license| |aur| |contributors| |downloads|
 
 .. |pypi| image:: https://img.shields.io/pypi/v/instaloader.svg
    :alt: Instaloader PyPI Project Page
@@ -29,7 +25,7 @@
 
 .. |downloads| image:: https://pepy.tech/badge/instaloader/month
    :alt: PyPI Download Count
-   :target: https://pepy.tech/project/instaloader/month
+   :target: https://pepy.tech/project/instaloader
 
 .. badges-end
 
@@ -84,7 +80,19 @@ To later **update your local copy** of that profiles, you may run
     instaloader --fast-update profile [profile ...]
 
 If ``--fast-update`` is given, Instaloader stops when arriving at the
-first already-downloaded picture. When updating profiles, Instaloader
+first already-downloaded picture.
+
+Alternatively, you can use ``--latest-stamps`` to have Instaloader store
+the time each profile was last downloaded and only download newer media:
+
+::
+
+    instaloader --latest-stamps -- profile [profile ...]
+
+With this option it's possible to move or delete downloaded media and still keep
+the archive updated.
+
+When updating profiles, Instaloader
 automatically **detects profile name changes** and renames the target directory
 accordingly.
 
@@ -102,6 +110,37 @@ already have a valid session cookie file.
 
 `Instaloader Documentation <https://instaloader.github.io/basic-usage.html>`__
 
+Contributing
+------------
+
+As an open source project, Instaloader heavily depends on the contributions from
+its community. See
+`contributing <https://instaloader.github.io/contributing.html>`__
+for how you may help Instaloader to become an even greater tool.
+
+Supporters
+----------
+
+.. current-sponsors-start
+
+| Instaloader is proudly sponsored by
+|  `@rocketapi-io <https://github.com/rocketapi-io>`__
+
+See `Alex' GitHub Sponsors <https://github.com/sponsors/aandergr>`__ page for
+how you can sponsor the development of Instaloader!
+
+.. current-sponsors-end
+
+It is a pleasure for us to share our Instaloader to the world, and we are proud
+to have attracted such an active and motivating community, with so many users
+who share their suggestions and ideas with us. Buying a community-sponsored beer
+or coffee from time to time is very likely to further raise our passion for the
+development of Instaloader.
+
+| For Donations, we provide GitHub Sponsors page, a PayPal.Me link and a Bitcoin address.
+|  GitHub Sponsors: `Sponsor @aandergr on GitHub Sponsors <https://github.com/sponsors/aandergr>`__
+|  PayPal: `PayPal.me/aandergr <https://www.paypal.me/aandergr>`__
+|  BTC: 1Nst4LoadeYzrKjJ1DX9CpbLXBYE9RKLwY
 
 Disclaimer
 ----------
@@ -114,21 +153,3 @@ subsidiaries. This is an independent and unofficial project. Use at your own ris
 Instaloader is licensed under an MIT license. Refer to ``LICENSE`` file for more information.
 
 .. disclaimer-end
-
-Contributing
-------------
-
-As an open source project, Instaloader heavily depends on the contributions from
-its community. See
-`contributing <https://instaloader.github.io/contributing.html>`__
-for how you may help Instaloader to become an even greater tool.
-
-It is a pleasure for us to share our Instaloader to the world, and we are proud
-to have attracted such an active and motivating community, with so many users
-who share their suggestions and ideas with us. Buying a community-sponsored beer
-or coffee from time to time is very likely to further raise our passion for the
-development of Instaloader.
-
-| For Donations, we provide a PayPal.Me link and a Bitcoin address.
-|  PayPal: `PayPal.me/aandergr <https://www.paypal.me/aandergr>`__
-|  BTC: 1Nst4LoadeYzrKjJ1DX9CpbLXBYE9RKLwY

deploy/arch/.SRCINFO.template

@@ -6,7 +6,7 @@ pkgbase = instaloader
 	arch = any
 	license = MIT
 	makedepends = python-setuptools
-	depends = python>=3.5
+	depends = python>=3.8
 	depends = python-requests>=2.4
 	options = !emptydirs
 	source = instaloader-{{version}}.tar.gz::https://codeload.github.com/instaloader/instaloader/tar.gz/v{{version}}

deploy/arch/PKGBUILD.template

@@ -8,7 +8,7 @@ arch=('any')
 url="https://instaloader.github.io/"
 license=('MIT')
 groups=()
-depends=('python>=3.5' 'python-requests>=2.4')
+depends=('python>=3.8' 'python-requests>=2.4')
 makedepends=('python-setuptools')
 options=('!emptydirs')
 source=($pkgname-$pkgver.tar.gz::https://codeload.github.com/instaloader/instaloader/tar.gz/v$pkgver)

deploy/arch/deploy.sh

@@ -8,10 +8,12 @@ echo $VERSION | grep -qe "[abc]" && exit 0
 cd $(dirname $0)
 
 # decrypt and add ssh key
-openssl aes-256-cbc -K $encrypted_354637631c28_key -iv $encrypted_354637631c28_iv -in id_rsa_AUR.enc -out /tmp/AUR_openssh -d
+openssl aes-256-cbc -K $AUR_KEY -iv $AUR_IV -in id_rsa_AUR.enc -out /tmp/AUR_openssh -d
 eval "$(ssh-agent -s)"
 chmod 600 /tmp/AUR_openssh
 ssh-add /tmp/AUR_openssh
+mkdir -p ~/.ssh
+ssh-keyscan -t rsa aur.archlinux.org >> ~/.ssh/known_hosts
 
 # clone and modify AUR repo
 git clone --depth 1 ssh://aur@aur.archlinux.org/instaloader.git

deploy/windows/create_exe.py

@@ -44,8 +44,8 @@ with open('__main__.py', 'w+') as f:
     f.writelines(lines)
 
 # install dependencies and invoke PyInstaller
-commands = ["pip install pipenv==2020.6.2",
-            "pipenv sync --dev",
+commands = ["pip install pipenv==2023.12.1",
+            f"pipenv --python {sys.version_info.major}.{sys.version_info.minor} sync --dev",
             "pipenv run pyinstaller --log-level=DEBUG instaloader.spec"]
 
 for command in commands:

docs/_templates/layout.html

@@ -21,7 +21,7 @@
 {% block header %}
   <nav class="navbar navbar-light navbar-expand bg-light border-bottom">
     <a class="navbar-brand" href="{{ pathto(master_doc) }}">
-      <img src="{{ pathto('_static/' + logo, 1) }}" width="30" height="30" class="d-inline-block align-top" alt="logo">
+      <img src="{{ pathto('_static/logo.png', 1) }}" width="30" height="30" class="d-inline-block align-top" alt="logo">
       <span class="ml-3">Instaloader</span>
     </a>
     <ul class="navbar-nav mr-auto">

docs/basic-usage.rst

@@ -39,6 +39,16 @@ To later **update your local copy** of that profiles, you may run
 If :option:`--fast-update` is given, Instaloader stops when arriving at the
 first already-downloaded picture.
 
+Alternatively, you can use :option:`--latest-stamps` to have Instaloader store
+the time each profile was last downloaded and only download newer media:
+
+::
+
+    instaloader --latest-stamps -- profile [profile ...]
+
+With this option it's possible to move or delete downloaded media and still keep
+the archive updated.
+
 When updating profiles, Instaloader
 automatically **detects profile name changes** and renames the target directory
 accordingly.
@@ -65,7 +75,7 @@ What to Download
 Instaloader supports the following targets:
 
 - ``profile``
-   Public profile, or private profile with :option:`--login`.
+   Public profile, or private profile with :ref:`login<login>`.
 
    If an already-downloaded profile has been renamed, Instaloader automatically
    finds it by its unique ID and renames the folder accordingly.
@@ -92,23 +102,23 @@ Instaloader supports the following targets:
    Posts tagged with a given location; the location ID is the numerical ID
    Instagram labels a location with (e.g.
    \https://www.instagram.com/explore/locations/**362629379**/plymouth-naval-memorial/).
-   Requires :option:`--login`.
+   Requires :ref:`login<login>`.
 
    .. versionadded:: 4.2
 
 - ``:stories``
    The currently-visible **stories** of your followees (requires
-   :option:`--login`).
+   :ref:`login<login>`).
 
 - ``:feed``
-   Your **feed** (requires :option:`--login`).
+   Your **feed** (requires :ref:`login<login>`).
 
 - ``:saved``
-   Posts which are marked as **saved** (requires :option:`--login`).
+   Posts which are marked as **saved** (requires :ref:`login<login>`).
 
 - ``@profile``
    All profiles that are followed by ``profile``, i.e. the *followees* of
-   ``profile`` (requires :option:`--login`).
+   ``profile`` (requires :ref:`login<login>`).
 
 - ``-post``
    Replace **post** with the post's shortcode to download single post. Must be preceded by ``--`` in
@@ -130,7 +140,7 @@ downloads the pictures and videos and their captions. You can specify
 
 - :option:`--geotags`
    **download geotags** of each post and save them as
-   Google Maps link (requires :option:`--login`),
+   Google Maps link (requires :ref:`login<login>`),
 
 For a reference of all supported command line options, see
 :ref:`command-line-options`.
@@ -149,27 +159,40 @@ target. The default is ``--dirname-pattern={target}``. In the dirname
 pattern, the token ``{target}`` is replaced by the target name, and
 ``{profile}`` is replaced by the owner of the post which is downloaded.
 
-:option:`--filename-pattern` configures the path of the post's files relative
+:option:`--filename-pattern` configures the path of the post and story's files relative
 to the target directory that is specified with :option:`--dirname-pattern`.
 The default is ``--filename-pattern={date_utc}_UTC``.
 The tokens ``{target}`` and ``{profile}`` are replaced like in the
-dirname pattern. The following tokens are defined for usage with
-:option:`--filename-pattern`:
+dirname pattern.
+
+:option:`--title-pattern` is similar to :option:`--filename-pattern`, but for profile
+pics, hashtag profile pics, and highlight covers. The default is
+``{date_utc}_UTC_{typename}`` if :option:`--dirname-pattern` contains ``{target}`` or
+``{profile}``, or ``{target}_{date_utc}_UTC_{typename}`` if it does not. Some tokens
+are not supported for this option, see below for details.
+
+The following tokens are defined for usage with
+:option:`--filename-pattern` and :option:`--title-pattern`:
 
 - ``{target}``
    Target name (as given in Instaloader command line)
 
 - ``{profile}`` (same as ``{owner_username}``)
-   Owner of the Post / StoryItem.
+   Owner of the Post / StoryItem / ProfilePic. For hashtag profile pics and
+   highlight covers, equivalent to ``{target}``.
 
 - ``{owner_id}``
-   Unique integer ID of owner profile.
+   Unique integer ID of owner profile. For hashtag profile pics, equivalent to
+   ``{target}``.
 
 - ``{shortcode}``
-   Shortcode (identifier string).
+   Shortcode (identifier string). Not available for :option:`--title-pattern`.
 
 - ``{mediaid}``
-   Integer representation of shortcode.
+   Integer representation of shortcode. Not available for :option:`--title-pattern`.
+
+- ``{filename}``
+   Instagram's internal filename.
 
 - ``{date_utc}`` (same as ``{date}``)
    Creation time in UTC timezone.
@@ -179,6 +202,10 @@ dirname pattern. The following tokens are defined for usage with
 
       {date_utc:%Y-%m-%d_%H-%M-%S}
 
+- ``{typename}``
+   Type of media being saved, such as GraphImage, GraphStoryVideo, profile_pic,
+   etc.
+
 For example, encode the poster's profile name in the filenames with::
 
     instaloader --filename-pattern={date_utc}_UTC_{profile} "#hashtag"
@@ -228,7 +255,7 @@ Id est, the following attributes can be used with both
 As :option:`--post-filter`, the following attributes can be used additionally:
 
 - :attr:`~Post.viewer_has_liked` (bool)
-   Whether user (with :option:`--login`) has liked given post. To download the
+   Whether user (with :ref:`login<login>`) has liked given post. To download the
    pictures from your feed that you have liked::
 
       instaloader --login=your_username --post-filter=viewer_has_liked :feed
@@ -280,13 +307,47 @@ the post's caption::
 
    instaloader --post-metadata-txt="{likes} likes." <target>
 
-Note that with this feature, it is possible to easily and fastly extract
+Note that with this feature, it is possible to easily and quickly extract
 additional metadata of already-downloaded posts, by reimporting their JSON
 files. Say, you now also want to export the number of comments the Posts had
 when they were downloaded::
 
    instaloader --post-metadata-txt="{likes} likes, {comments} comments." <target>/*.json.xz
 
+.. _exit_codes:
+
+Exit codes
+^^^^^^^^^^
+
+Different exit codes are used to indicate different kinds of error:
+
+0
+  No error, all downloads were successful.
+
+1
+  A non-fatal error happened. One or more posts, or even one or more
+  profiles could not be downloaded, but execution was not stopped. The
+  errors are repeated at the end of the log for easy access.
+
+2
+  Command-line error. An unrecognized option was passed, or an invalid
+  combination of options, for example. No interaction with Instagram
+  was made.
+
+3
+  Login error. It was not possible to login. Downloads were not
+  attempted.
+
+4
+  Fatal download error. Downloads were interrupted and no further
+  attempts were made. Happens when a response with one of the status
+  codes in the :option:`--abort-on` option were passed, or when
+  Instagram logs the user out during downloads.
+
+5
+  Interrupted by the user. Happens when the user presses Control-C or
+  sends SIGINT to the process.
+
 .. _instaloader-as-cronjob:
 
 Instaloader as Cronjob

docs/cli-options.rst

@@ -26,7 +26,7 @@ Targets
 ^^^^^^^
 
 Specify a list of targets. For each of these, Instaloader creates a folder and
-stores all posts along with the pictures's captions there.
+stores all posts along with the pictures' captions there.
 
 .. include:: basic-usage.rst
    :start-after: targets-start
@@ -61,13 +61,13 @@ What to Download of each Post
 
    Download geotags when available. Geotags are stored as a text file with
    the location's name and a Google Maps link. This requires an additional
-   request to the Instagram server for each picture. Requires :option:`--login`.
+   request to the Instagram server for each picture. Requires :ref:`login<login>`.
 
 .. option:: --comments, -C
 
    Download and update comments for each post. This requires an additional
    request to the Instagram server for each post, which is why it is disabled by
-   default.
+   default. Requires :ref:`login<login>`.
 
 .. option:: --no-captions
 
@@ -82,6 +82,16 @@ What to Download of each Post
    Template to write in txt file for each StoryItem. See
    :ref:`metadata-text-files`.
 
+.. option:: --slide
+
+   Download only selected images of a sidecar. You can select single images using their
+   index in the sidecar starting with the leftmost or you can specify a range of images
+   with the following syntax: ``start_index-end_index``. Example:
+   ``--slide 1`` will select only the first image, ``--slide last`` only the last one and ``--slide 1-3`` will select only
+   the first three images.
+
+   .. versionadded:: 4.6
+
 .. option:: --no-metadata-json
 
    Do not create a JSON file containing the metadata of each post.
@@ -90,7 +100,6 @@ What to Download of each Post
 
    Do not xz compress JSON files, rather create pretty formatted JSONs.
 
-
 What to Download of each Profile
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -107,12 +116,12 @@ What to Download of each Profile
 .. option:: --stories, -s
 
    Also download stories of each profile that is downloaded. Requires
-   :option:`--login`.
+   :ref:`login<login>`.
 
 .. option:: --highlights
 
    Also download highlights of each profile that is downloaded. Requires
-   :option:`--login`.
+   :ref:`login<login>`.
 
    .. versionadded:: 4.1
 
@@ -137,6 +146,23 @@ Which Posts to Download
    This flag is recommended when you use Instaloader to update your personal
    Instagram archive.
 
+.. option:: --latest-stamps [STAMPSFILE]
+
+   Works similarly to :option:`--fast-update`, but instead of relying on already
+   downloaded media, the time each profile was downloaded is stored, and only
+   media newer than the last download is fetched. This allows updating your
+   personal Instagram archive while emptying the target directories.
+
+   Only works for media associated with a specific profile, and that is returned
+   in chronological order: profile posts, profile stories, profile IGTV posts
+   and profile tagged posts.
+
+   By default, the information is stored in
+   ``~/.config/instaloader/latest-stamps.ini``, but you can specify an
+   alternative location.
+
+   .. versionadded:: 4.8
+
 .. option:: --post-filter filter, --only-if filter
 
    Expression that, if given, must evaluate to True for each post to be
@@ -154,9 +180,10 @@ Which Posts to Download
 
 .. option:: --count COUNT, -c
 
-   Do not attempt to download more than COUNT posts.  Applies only to
-   ``#hashtag``, ``%location id``, and ``:feed``.
+   Do not attempt to download more than COUNT posts.  Applies to
+   ``#hashtag``, ``%location_id``, ``:feed``, and ``:saved``.
 
+.. _login:
 
 Login (Download Private Profiles)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -166,10 +193,32 @@ profiles. To login, pass the :option:`--login` option. Your session cookie (not
 password!) will be saved to a local file to be reused next time you want
 Instaloader to login.
 
+Instead of :option:`--login`, it is possible to use
+:option:`--load-cookies` to import a session from a browser.
+
 .. option:: --login YOUR-USERNAME, -l YOUR-USERNAME
 
    Login name (profile name) for your Instagram account.
 
+.. option:: --load-cookies BROWSER-NAME, -b BROWSER-NAME
+
+   Use Instagram cookie in your browser to login.
+   This feature requires the browser_cookie3 library.
+   Compatible with :option:`--cookiefile` if you want to load cookies from browser profiles.
+   Incompatible with :option:`--login` due to potential username mismatch between user input and browser login.
+   Supported browsers: Brave, Chrome, Chromium, Edge, Firefox, LibreWolf, Opera, Opera_GX, Safari and Vivaldi.
+
+   In subsequent runs, you can just use :option:`--login` to reuse the
+   same session, which is saved by Instaloader.
+
+   .. versionadded:: 4.11
+
+.. option:: --cookiefile COOKIE-FILE, -B COOKIE-FILE
+
+   Cookie file path of a browser profile to load cookies from.
+
+  .. versionadded:: 4.11
+
 .. option:: --sessionfile SESSIONFILE, -f SESSIONFILE
 
    Path for loading and storing session key file.  Defaults to
@@ -196,12 +245,30 @@ How to Download
 
 .. option:: --filename-pattern FILENAME_PATTERN
 
-   Prefix of filenames, relative to the directory given with
+   Prefix of filenames for posts and stories, relative to the directory given with
    :option:`--dirname-pattern`. ``{profile}`` is replaced by the profile name,
    ``{target}`` is replaced by the target you specified, i.e.  either ``:feed``,
    ``#hashtag`` or the profile name. Defaults to ``{date_utc}_UTC``.
    See :ref:`filename-specification` for a list of supported tokens.
 
+.. option:: --title-pattern TITLE_PATTERN
+
+   Prefix of filenames for profile pics, hashtag profile pics, and highlight
+   covers, relative to the directory given with :option:`--dirname-pattern`.
+   Defaults to ``{date_utc}_UTC_{typename}`` if :option:`--dirname-pattern`
+   contains ``{target}`` or ``{profile}``, otherwise defaults to
+   ``{target}_{date_utc}_UTC_{typename}``.
+   See :ref:`filename-specification` for a list of supported tokens.
+
+   .. versionadded:: 4.8
+
+.. option:: --sanitize-paths
+
+   Force sanitization of paths so that the resulting file and directory names
+   are valid on both Windows and Unix.
+
+   .. versionadded:: 4.9
+
 .. option:: --resume-prefix prefix
 
    For many targets, Instaloader is capable of resuming a previously-aborted
@@ -210,12 +277,21 @@ How to Download
    used to save the information to resume an interrupted download.  The default
    prefix is ``iterator``.
 
-   Resuming an interrupted download is supported for most, but not all targets.
+   Resuming an interrupted download is supported for the following targets:
+    - Profile posts,
+    - Profile IGTV posts (:option:`--igtv`),
+    - Profile tagged posts (:option:`--tagged`),
+    - Saved posts (``:saved``),
+    - Hashtags.
+
+   This feature is enabled by default for targets where it is supported;
+   :option:`--resume-prefix` only changes the name of the iterator files.
+
+   To turn this feature off, use :option:`--no-resume`.
+
    JSON files with resume information are always compressed, regardless of
    :option:`--no-compress-json`.
 
-   This feature is turned off entirely with :option:`--no-resume`.
-
    .. versionadded:: 4.5
 
 .. option:: --no-resume
@@ -228,7 +304,7 @@ How to Download
 .. option:: --user-agent USER_AGENT
 
    User Agent to use for HTTP requests. Per default, Instaloader pretends being
-   Chrome/51.
+   Chrome/120 on Linux.
 
 .. option:: --max-connection-attempts N
 
@@ -238,10 +314,30 @@ How to Download
 
 .. option:: --request-timeout N
 
-   Seconds to wait before timing out a connection request.
+   Seconds to wait before timing out a connection request. Defaults to 300.
 
    .. versionadded:: 4.3
 
+   .. versionchanged:: 4.6
+      Enabled this option by default with a timeout of 300 seconds.
+
+.. option:: --abort-on STATUS_CODE_LIST
+
+   Comma-separated list of HTTP status codes that cause Instaloader to abort,
+   bypassing all retry logic.
+
+   For example, with ``--abort-on=302,400,429``, Instaloader will stop if a
+   request is responded with a 302 redirect, a Bad Request error, or a Too Many
+   Requests error.
+
+   .. versionadded:: 4.7
+
+.. option:: --no-iphone
+
+   Do not attempt to download iPhone version of images and videos.
+
+   .. versionadded:: 4.8
+
 Miscellaneous Options
 ^^^^^^^^^^^^^^^^^^^^^
 
@@ -256,12 +352,13 @@ Miscellaneous Options
    Read arguments from file `args.txt`, a shortcut to provide arguments from
    file rather than command-line. This provides a convenient way to hide login
    info from CLI, and can also be used to simplify management of long arguments.
+   You can provide more than one file at once, e.g.: ``+args1.txt +args2.txt``.
 
    .. note::
 
       Text file should separate arguments with line breaks.
 
-   args.txt example::
+   `args.txt` example::
 
       --login=MYUSERNAME
       --password=MYPASSWORD

docs/codesnippets/615_import_firefox_session.py

@@ -23,7 +23,7 @@ def get_cookiefile():
 
 def import_session(cookiefile, sessionfile):
     print("Using cookies from {}.".format(cookiefile))
-    conn = connect(cookiefile)
+    conn = connect(f"file:{cookiefile}?immutable=1", uri=True)
     try:
         cookie_data = conn.execute(
             "SELECT name, value FROM moz_cookies WHERE baseDomain='instagram.com'"

docs/conf.py

@@ -1,56 +1,84 @@
-#!/usr/bin/env python3
-# -*- coding: utf-8 -*-
+# Configuration file for the Sphinx documentation builder.
 #
-# Instaloader documentation build configuration file, created by
-# sphinx-quickstart on Fri Aug 25 15:38:26 2017.
-#
-# This file is execfile()d with the current directory set to its
-# containing dir.
-#
-# Note that not all possible configuration values are present in this
-# autogenerated file.
-#
-# All configuration values have a default; values that are commented out
-# serve to show the default.
+# For the full list of built-in configuration values, see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
 
-# If extensions (or modules to document with autodoc) are in another directory,
-# add these directories to sys.path here. If the directory is relative to the
-# documentation root, use os.path.abspath to make it absolute, like shown here.
-#
 import os
 import platform
 import subprocess
 import sys
-import typing
 
 sys.path.insert(0, os.path.abspath('..'))
 
-import docs.sphinx_autodoc_typehints as sphinx_autodoc_typehints
 
-# -- General configuration ------------------------------------------------
+# -- Project information -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
 
-# If your documentation needs a minimal Sphinx version, state it here.
-#
-# needs_sphinx = '1.0'
+project = 'Instaloader'
+copyright = '2023, Alexander Graf and Andre Koch-Kramer'
+author = 'Alexander Graf and Andre Koch-Kramer'
+
+# -- General configuration ---------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
 
-# Add any Sphinx extension module names here, as strings. They can be
-# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
-# ones.
 extensions = [
-    'sphinx.ext.autodoc',
-    'sphinx.ext.githubpages',
-    'sphinx.ext.intersphinx',
-    'sphinx.ext.extlinks'
+    "sphinx.ext.autodoc",
+    "sphinx.ext.extlinks",
+    "sphinx.ext.githubpages",
+    "sphinx.ext.intersphinx",
+    "sphinxcontrib.jquery",
 ]
 
-autodoc_default_flags = ['show-inheritance', 'members', 'undoc-members']
+# autodoc options
 
-autodoc_member_order = 'bysource'
+autodoc_default_options = {
+    'show-inheritance': True,
+    'members': True,
+    'undoc-members': True,
+}
 
-intersphinx_mapping = {'python': ('https://docs.python.org/3', None),
-                       'requests': ('https://requests.kennethreitz.org/en/master/', None)}
+# extlinks options
 
-nitpick_ignore = [('py:class', 'typing.Tuple')]
+extlinks = {
+    'issue': (
+        'https://github.com/instaloader/instaloader/issues/%s',
+        'Issue #%s'
+    ),
+    'example': (
+        'https://raw.githubusercontent.com/instaloader/instaloader/master/docs/codesnippets/%s',
+        'Example %s'
+    ),
+}
+
+# intersphinx options
+
+intersphinx_mapping = {
+    'python': ('https://docs.python.org/3', None),
+    'requests': ('https://requests.readthedocs.io/en/latest/', None),
+}
+
+# other options
+
+templates_path = ['_templates']
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+# ignore TypeVar as it is currently unsupported by Sphinx
+nitpick_ignore = [('py:class', 'instaloader.nodeiterator.T')]
+
+# -- Options for HTML output -------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
+
+pygments_style = 'fruity'
+
+html_theme = 'basic'
+html_static_path = ['_static']
+
+html_logo = "logo.png"
+
+html_css_files = ['instaloaderdoc.css']
+html_js_files = ['instaloaderdoc.js']
+
+# get release number
 
 current_release = subprocess.check_output(["git", "describe", "--abbrev=0"]).decode("ascii")[1:-1]
 date_format = "%e %b %Y" if platform.system() != "Windows" else "%d %b %Y"
@@ -58,322 +86,3 @@ current_release_date = subprocess.check_output(
     ["git", "log", "-1", "--tags", "--format=%ad", "--date=format:"+date_format]).decode("ascii")[:-1]
 
 html_context = {'current_release': current_release, 'current_release_date': current_release_date}
-
-extlinks = {
-    'issue': (
-        'https://github.com/instaloader/instaloader/issues/%s',
-        'Issue #'
-    ),
-    'example': (
-        'https://raw.githubusercontent.com/instaloader/instaloader/v' + current_release + '/docs/codesnippets/%s',
-        'Example '
-    ),
-}
-
-# Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
-
-# The suffix(es) of source filenames.
-# You can specify multiple suffix as a list of string:
-#
-# source_suffix = ['.rst', '.md']
-source_suffix = '.rst'
-
-# The encoding of source files.
-#
-# source_encoding = 'utf-8-sig'
-
-# The master toctree document.
-master_doc = 'index'
-
-# General information about the project.
-project = 'Instaloader'
-copyright = '2017-2020, Alexander Graf and Andre Koch-Kramer'
-author = 'Alexander Graf and Andre Koch-Kramer'
-
-# The version info for the project you're documenting, acts as replacement for
-# |version| and |release|, also used in various other places throughout the
-# built documents.
-#
-# The short X.Y version.
-#version = '3.0'
-# The full version, including alpha/beta/rc tags.
-#release = '3.0rc0'
-
-# The language for content autogenerated by Sphinx. Refer to documentation
-# for a list of supported languages.
-#
-# This is also used if you do content translation via gettext catalogs.
-# Usually you set "language" from the command line for these cases.
-language = "en"
-
-# There are two options for replacing |today|: either, you set today to some
-# non-false value, then it is used:
-#
-# today = ''
-#
-# Else, today_fmt is used as the format for a strftime call.
-#
-# today_fmt = '%B %d, %Y'
-
-# List of patterns, relative to source directory, that match files and
-# directories to ignore when looking for source files.
-# This patterns also effect to html_static_path and html_extra_path
-exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
-
-# The reST default role (used for this markup: `text`) to use for all
-# documents.
-#
-# default_role = None
-
-# If true, '()' will be appended to :func: etc. cross-reference text.
-#
-# add_function_parentheses = True
-
-# If true, the current module name will be prepended to all description
-# unit titles (such as .. function::).
-#
-add_module_names = False
-
-# If true, sectionauthor and moduleauthor directives will be shown in the
-# output. They are ignored by default.
-#
-# show_authors = False
-
-# The name of the Pygments (syntax highlighting) style to use.
-pygments_style = 'fruity'
-
-# A list of ignored prefixes for module index sorting.
-# modindex_common_prefix = []
-
-# If true, keep warnings as "system message" paragraphs in the built documents.
-# keep_warnings = False
-
-# If true, `todo` and `todoList` produce output, else they produce nothing.
-todo_include_todos = False
-
-
-# -- Options for HTML output ----------------------------------------------
-
-# The theme to use for HTML and HTML Help pages.  See the documentation for
-# a list of builtin themes.
-#
-html_theme = 'basic'
-
-# Add any paths that contain custom themes here, relative to this directory.
-# html_theme_path = []
-
-# The name for this set of Sphinx documents.
-# "<project> v<release> documentation" by default.
-#
-# html_title = 'Instaloader v3.0rc0'
-
-# A shorter title for the navigation bar.  Default is the same as html_title.
-#
-# html_short_title = None
-
-# The name of an image file (relative to this directory) to place at the top
-# of the sidebar.
-#
-html_logo = "logo.png"
-
-# The name of an image file (relative to this directory) to use as a favicon of
-# the docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
-# pixels large.
-#
-html_favicon = "favicon.ico"
-
-# Add any paths that contain custom static files (such as style sheets) here,
-# relative to this directory. They are copied after the builtin static files,
-# so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
-
-# Add any extra paths that contain custom files (such as robots.txt or
-# .htaccess) here, relative to this directory. These files are copied
-# directly to the root of the documentation.
-#
-# html_extra_path = []
-
-# If not None, a 'Last updated on:' timestamp is inserted at every page
-# bottom, using the given strftime format.
-# The empty string is equivalent to '%b %d, %Y'.
-#
-# html_last_updated_fmt = None
-
-# If true, SmartyPants will be used to convert quotes and dashes to
-# typographically correct entities.
-#
-# html_use_smartypants = True
-
-# Custom sidebar templates, maps document names to template names.
-#
-#html_sidebars = {'**': ["relations.html", "links.html"] }
-html_sidebars = {}
-
-# Additional templates that should be rendered to pages, maps page names to
-# template names.
-#
-# html_additional_pages = {}
-
-# If false, no module index is generated.
-#
-html_domain_indices = False
-
-# If false, no index is generated.
-#
-html_use_index = True
-
-# If true, the index is split into individual pages for each letter.
-#
-# html_split_index = False
-
-# If true, links to the reST sources are added to the pages.
-#
-html_show_sourcelink = False
-
-# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
-#
-html_show_sphinx = False
-
-# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
-#
-html_show_copyright = False
-
-# If true, an OpenSearch description file will be output, and all pages will
-# contain a <link> tag referring to it.  The value of this option must be the
-# base URL from which the finished HTML is served.
-#
-# html_use_opensearch = ''
-
-# This is the file name suffix for HTML files (e.g. ".xhtml").
-# html_file_suffix = None
-
-# Language to be used for generating the HTML full-text search index.
-# Sphinx supports the following languages:
-#   'da', 'de', 'en', 'es', 'fi', 'fr', 'h', 'it', 'ja'
-#   'nl', 'no', 'pt', 'ro', 'r', 'sv', 'tr', 'zh'
-#
-# html_search_language = 'en'
-
-# A dictionary with options for the search language support, empty by default.
-# 'ja' uses this config value.
-# 'zh' user can custom change `jieba` dictionary path.
-#
-# html_search_options = {'type': 'default'}
-
-# The name of a javascript file (relative to the configuration directory) that
-# implements a search results scorer. If empty, the default will be used.
-#
-# html_search_scorer = 'scorer.js'
-
-# Output file base name for HTML help builder.
-htmlhelp_basename = 'Instaloaderdoc'
-
-# -- Options for LaTeX output ---------------------------------------------
-
-latex_elements = {
-     # The paper size ('letterpaper' or 'a4paper').
-     #
-     # 'papersize': 'letterpaper',
-
-     # The font size ('10pt', '11pt' or '12pt').
-     #
-     # 'pointsize': '10pt',
-
-     # Additional stuff for the LaTeX preamble.
-     #
-     # 'preamble': '',
-
-     # Latex figure (float) alignment
-     #
-     # 'figure_align': 'htbp',
-}
-
-# Grouping the document tree into LaTeX files. List of tuples
-# (source start file, target name, title,
-#  author, documentclass [howto, manual, or own class]).
-latex_documents = [
-    (master_doc, 'Instaloader.tex', 'Instaloader Documentation',
-     'Alexander Graf, André Koch-Kramer', 'manual'),
-]
-
-# The name of an image file (relative to this directory) to place at the top of
-# the title page.
-#
-# latex_logo = None
-
-# For "manual" documents, if this is true, then toplevel headings are parts,
-# not chapters.
-#
-# latex_use_parts = False
-
-# If true, show page references after internal links.
-#
-# latex_show_pagerefs = False
-
-# If true, show URL addresses after external links.
-#
-# latex_show_urls = False
-
-# Documents to append as an appendix to all manuals.
-#
-# latex_appendices = []
-
-# It false, will not define \strong, \code, 	itleref, \crossref ... but only
-# \sphinxstrong, ..., \sphinxtitleref, ... To help avoid clash with user added
-# packages.
-#
-# latex_keep_old_macro_names = True
-
-# If false, no module index is generated.
-#
-# latex_domain_indices = True
-
-
-# -- Options for manual page output ---------------------------------------
-
-# One entry per manual page. List of tuples
-# (source start file, name, description, authors, manual section).
-man_pages = [
-    (master_doc, 'instaloader', 'Instaloader Documentation',
-     [author], 1)
-]
-
-# If true, show URL addresses after external links.
-#
-# man_show_urls = False
-
-
-# -- Options for Texinfo output -------------------------------------------
-
-# Grouping the document tree into Texinfo files. List of tuples
-# (source start file, target name, title, author,
-#  dir menu entry, description, category)
-texinfo_documents = [
-    (master_doc, 'Instaloader', 'Instaloader Documentation',
-     author, 'Instaloader', 'One line description of project.',
-     'Miscellaneous'),
-]
-
-# Documents to append as an appendix to all manuals.
-#
-# texinfo_appendices = []
-
-# If false, no module index is generated.
-#
-# texinfo_domain_indices = True
-
-# How to display URL addresses: 'footnote', 'no', or 'inline'.
-#
-# texinfo_show_urls = 'footnote'
-
-# If true, do not generate a @detailmenu in the "Top" node's menu.
-#
-# texinfo_no_detailmenu = False
-
-
-def setup(app):
-    typing.TYPE_CHECKING = True
-    app.add_stylesheet("instaloaderdoc.css")
-    app.connect('autodoc-process-signature', sphinx_autodoc_typehints.process_signature)
-    app.connect('autodoc-process-docstring', sphinx_autodoc_typehints.process_docstring)

docs/contributing.rst

@@ -11,6 +11,23 @@ Instaloader's development is organized on
 `GitHub <https://github.com/instaloader/instaloader>`__, where Issues and Pull
 Requests are discussed.
 
+Contributing to an open source project is an unmatchable opportunity to improve
+and manifest programming skills. For issues that are good starting points to get
+involved in Instaloader development, as they do not require deep insight into
+Instaloader's code base, have a look at our
+`Issues labelled 'good first issue' <https://github.com/instaloader/instaloader/contribute>`__.
+
+This document covers several ways of how you can contribute back to Instaloader.
+
+Answering Questions
+-------------------
+
+The easiest way to help out is to answer questions. If you are interested in
+answering questions regarding Instaloader, good places to begin are
+
+- `Questions tagged 'instaloader' on Stack Overflow <https://stackoverflow.com/questions/tagged/instaloader>`__,
+- `Instaloader Issues labeled 'question' <https://github.com/instaloader/instaloader/issues?q=is%3Aissue+is%3Aopen+label%3Aquestion>`__.
+
 Reporting Bugs
 --------------
 
@@ -41,22 +58,39 @@ reporting a problem, please keep the following in mind:
 Writing Code or Improving the Documentation
 -------------------------------------------
 
-Changes of the Instaloader source can be proposed as a
-`Pull Request <https://github.com/instaloader/instaloader/pulls>`__. There are only
-few things to consider:
+Improvements of the Instaloader source or its documentation can be proposed as a
+`Pull Request <https://github.com/instaloader/instaloader/pulls>`__.
 
-- Base your Pull Request on the ``master`` branch if it fixes a bug,
-  or the ``upcoming/v4.X`` branch (if it exists at the moment of submitting the PR)
-  otherwise.
+- Please base your Pull Request on
 
-- We use `Pylint <https://www.pylint.org/>`__ for error and syntax checking of
-  the source and `MyPy <https://github.com/python/mypy>`__ for type checking.
-  Beware that sometimes it might be better to disable a warning rather than
-  adapting the code to a tool's desires.
+  - ``master``, which will be released with the next minor release, if it is
 
-- The documentation source is located in the ``docs`` folder. The file
-  ``cli-options.rst`` is merely an RST-formatted copy of ``instaloader --help``
-  output, of which the source is in ``instaloader/__main__.py``.
+    - a bug fix that does not require extensive testing,
+    - an improvement to the documentation,
+
+  - ``upcoming/v4.X``, if it is
+
+    - a new feature,
+    - a bug fix that does require thorough testing before being released to a
+      final version.
+
+- All Pull Requests are analyzed by `Pylint <https://www.pylint.org/>`__ for
+  error and syntax checking of the source and
+  `Mypy <https://github.com/python/mypy>`__ for type checking. You can run them
+  locally with::
+
+     pylint instaloader
+     mypy -m instaloader
+
+- Improvements to the documentation are very welcome. The documentation is
+  created with `Sphinx <https://www.sphinx-doc.org/en/2.0/>`__, version 2,
+  and can be build locally using::
+
+     make -C docs html
+
+- Feel free to create an issue to make sure someone from the Instaloader team
+  agrees that the change might be an improvement, or if you want to discuss
+  your basic proposal, before working on a pull request.
 
 Proposing Features
 ------------------
@@ -98,21 +132,9 @@ details. If you feel the need to do so, please create a "feature suggestion".
   be implemented and the underlying problem could be solved. Also **describe
   alternatives** that you have considered.
 
-Donations
----------
+Sponsoring
+----------
 
-.. donations-start
-
-It is a pleasure for us to share our Instaloader to the world, and we are proud
-to have attracted such an active and motivating community, with so many users
-who share their suggestions and ideas with us. Buying a community-sponsored beer
-or coffee from time to time is very likely to further raise our passion for the
-development of Instaloader.
-
-| For donations, we provide a PayPal.Me link and a Bitcoin address.
-|  `PayPal.me/aandergr <https://www.paypal.me/aandergr>`__
-|  BTC: 1Nst4LoadeYzrKjJ1DX9CpbLXBYE9RKLwY
-
-.. donations-end
-
-.. (Discussion in :issue:`130`)
+.. include:: sponsors.rst
+   :start-after: donations-start
+   :end-before: donations-end

docs/index.rst

@@ -78,13 +78,6 @@ Useful Links
 - `Issue Tracker / Bug Tracker <https://github.com/instaloader/instaloader/issues>`__
 - `Version History <https://github.com/instaloader/instaloader/releases>`__
 
-Disclaimer
-----------
-
-.. include:: ../README.rst
-   :start-after: disclaimer-start
-   :end-before: disclaimer-end
-
 Contributing
 ------------
 
@@ -92,10 +85,20 @@ As an open source project, Instaloader heavily depends on the contributions from
 its community. See :ref:`contributing` for how you may help Instaloader to
 become an even greater tool.
 
-.. include:: contributing.rst
+Supporters
+----------
+
+.. include:: sponsors.rst
    :start-after: donations-start
    :end-before: donations-end
 
+Disclaimer
+----------
+
+.. include:: ../README.rst
+   :start-after: disclaimer-start
+   :end-before: disclaimer-end
+
 ..
    * :ref:`genindex`
    * :ref:`modindex`

docs/installation.rst

@@ -8,7 +8,7 @@ Install Instaloader
 To **install Instaloader**,
 
 #. Ensure that you have `Python <https://www.python.org/>`__, at least
-   version 3.5, and `pip <https://pypi.python.org/pypi/pip>`__
+   version 3.8, and `pip <https://pypi.python.org/pypi/pip>`__
    installed.
 
 #. Then, install Instaloader using::
@@ -33,6 +33,9 @@ To **upgrade Instaloader** to its current version, do::
 - On **Windows 10**, you may download the standalone executable from the
   `current release page <https://github.com/instaloader/instaloader/releases/latest>`__.
 
+- On **Android**, you can use Instaloader with `Termux <https://play.google.com/store/apps/details?id=com.termux>`__
+  after typing ``pkg install python`` and ``pip3 install instaloader``.
+
 - To test the most current **pre-release** version of Instaloader::
 
      pip3 install --pre instaloader

docs/module/exceptions.rst

@@ -27,16 +27,26 @@ Exceptions
 
 .. autoexception:: LoginRequiredException
 
+.. autoexception:: LoginException
+
+   .. versionadded:: 4.12
+
 .. autoexception:: TwoFactorAuthRequiredException
 
    .. versionadded:: 4.2
 
+   .. versionchanged:: 4.12
+      Inherits LoginException
+
 .. autoexception:: InvalidArgumentException
 
 .. autoexception:: BadResponseException
 
 .. autoexception:: BadCredentialsException
 
+   .. versionchanged:: 4.12
+      Inherits LoginException
+
 .. autoexception:: PostChangedException
 
 .. autoexception:: QueryReturnedNotFoundException
@@ -47,3 +57,5 @@ Exceptions
 
 
 .. autoexception:: TooManyRequestsException
+
+.. autoexception:: AbortDownloadException

docs/module/nodeiterator.rst

@@ -25,6 +25,13 @@ Iterator :class:`NodeIterator` and a context manager
 .. autoclass:: FrozenNodeIterator
    :no-show-inheritance:
 
+   A serializable representation of a :class:`NodeIterator` instance, saving
+   its iteration state.
+
+   It can be serialized and deserialized with :func:`save_structure_to_file`
+   and :func:`load_structure_from_file`, as well as with :mod:`json` and
+   :mod:`pickle` thanks to being a :class:`~typing.NamedTuple`.
+
 ``resumable_iteration``
 """""""""""""""""""""""
 

docs/module/structures.rst

@@ -71,12 +71,30 @@ TopSearchResults
 
    .. versionadded:: 4.3
 
+TitlePic
+""""""""
+
+.. autoclass:: TitlePic
+   :no-show-inheritance:
+
+   .. versionadded:: 4.8
+
 Loading and Saving
 """"""""""""""""""
 
 :class:`Post`, :class:`StoryItem`, :class:`Profile`, :class:`Hashtag` and
 :class:`FrozenNodeIterator` can be saved and loaded to/from JSON files.
 
+.. autofunction:: load_structure
+
 .. autofunction:: load_structure_from_file
 
+.. autofunction:: get_json_structure
+
 .. autofunction:: save_structure_to_file
+
+LatestStamps
+""""""""""""
+
+.. autoclass:: LatestStamps
+   :no-show-inheritance:

docs/sphinx_autodoc_typehints.py

@@ -1,251 +0,0 @@
-# sphinx_autodoc_typehints from Nov 20 2018 (version 1.5.2).
-#  https://github.com/agronholm/sphinx-autodoc-typehints
-# (slightly modified to fix class links, maybe related to agronholm/sphinx-autodoc-typehints#38)
-#
-# This is the MIT license: http://www.opensource.org/licenses/mit-license.php
-#
-# Copyright (c) Alex Grönholm
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy of this
-# software and associated documentation files (the "Software"), to deal in the Software
-# without restriction, including without limitation the rights to use, copy, modify, merge,
-# publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons
-# to whom the Software is furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in all copies or
-# substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
-# INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
-# PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE
-# FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
-# OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
-# DEALINGS IN THE SOFTWARE.
-#
-
-
-import inspect
-from typing import get_type_hints, TypeVar, Any, AnyStr, Generic, Union
-
-from sphinx.util import logging
-from sphinx.util.inspect import Signature
-
-try:
-    from inspect import unwrap
-except ImportError:
-    def unwrap(func, *, stop=None):
-        """This is the inspect.unwrap() method copied from Python 3.5's standard library."""
-        if stop is None:
-            def _is_wrapper(f):
-                return hasattr(f, '__wrapped__')
-        else:
-            def _is_wrapper(f):
-                return hasattr(f, '__wrapped__') and not stop(f)
-        f = func  # remember the original func for error reporting
-        memo = {id(f)}  # Memoise by id to tolerate non-hashable objects
-        while _is_wrapper(func):
-            func = func.__wrapped__
-            id_func = id(func)
-            if id_func in memo:
-                raise ValueError('wrapper loop when unwrapping {!r}'.format(f))
-            memo.add(id_func)
-        return func
-
-logger = logging.getLogger(__name__)
-
-
-def format_annotation(annotation):
-    if inspect.isclass(annotation) and annotation.__module__ == 'builtins':
-        if annotation.__qualname__ == 'NoneType':
-            return '``None``'
-        else:
-            return ':py:class:`{}`'.format(annotation.__qualname__)
-
-    annotation_cls = annotation if inspect.isclass(annotation) else type(annotation)
-    class_name = None
-    if annotation_cls.__module__ == 'typing':
-        params = None
-        prefix = ':py:class:'
-        module = 'typing'
-        extra = ''
-
-        if inspect.isclass(getattr(annotation, '__origin__', None)):
-            annotation_cls = annotation.__origin__
-            try:
-                if Generic in annotation_cls.mro():
-                    module = annotation_cls.__module__
-            except TypeError:
-                pass  # annotation_cls was either the "type" object or typing.Type
-
-        if annotation is Any:
-            return ':py:data:`~typing.Any`'
-        elif annotation is AnyStr:
-            return ':py:data:`~typing.AnyStr`'
-        elif isinstance(annotation, TypeVar):
-            return '\\%r' % annotation
-        elif (annotation is Union or getattr(annotation, '__origin__', None) is Union or
-              hasattr(annotation, '__union_params__')):
-            prefix = ':py:data:'
-            class_name = 'Union'
-            if hasattr(annotation, '__union_params__'):
-                params = annotation.__union_params__
-            elif hasattr(annotation, '__args__'):
-                params = annotation.__args__
-
-            if params and len(params) == 2 and (hasattr(params[1], '__qualname__') and
-                                                params[1].__qualname__ == 'NoneType'):
-                class_name = 'Optional'
-                params = (params[0],)
-        elif annotation_cls.__qualname__ == 'Tuple' and hasattr(annotation, '__tuple_params__'):
-            params = annotation.__tuple_params__
-            if annotation.__tuple_use_ellipsis__:
-                params += (Ellipsis,)
-        elif annotation_cls.__qualname__ == 'Callable':
-            prefix = ':py:data:'
-            arg_annotations = result_annotation = None
-            if hasattr(annotation, '__result__'):
-                arg_annotations = annotation.__args__
-                result_annotation = annotation.__result__
-            elif getattr(annotation, '__args__', None):
-                arg_annotations = annotation.__args__[:-1]
-                result_annotation = annotation.__args__[-1]
-
-            if arg_annotations in (Ellipsis, (Ellipsis,)):
-                params = [Ellipsis, result_annotation]
-            elif arg_annotations is not None:
-                params = [
-                    '\\[{}]'.format(
-                        ', '.join(format_annotation(param) for param in arg_annotations)),
-                    result_annotation
-                ]
-        elif hasattr(annotation, 'type_var'):
-            # Type alias
-            class_name = annotation.name
-            params = (annotation.type_var,)
-        elif getattr(annotation, '__args__', None) is not None:
-            params = annotation.__args__
-        elif hasattr(annotation, '__parameters__'):
-            params = annotation.__parameters__
-
-        if params:
-            extra = '\\[{}]'.format(', '.join(format_annotation(param) for param in params))
-
-        if not class_name:
-            class_name = annotation_cls.__qualname__.title()
-
-        return '{}`~{}.{}`{}'.format(prefix, module, class_name, extra)
-    elif annotation is Ellipsis:
-        return '...'
-    elif inspect.isclass(annotation) or inspect.isclass(getattr(annotation, '__origin__', None)):
-        if not inspect.isclass(annotation):
-            annotation_cls = annotation.__origin__
-
-        extra = ''
-        if Generic in annotation_cls.mro():
-            params = (getattr(annotation, '__parameters__', None) or
-                      getattr(annotation, '__args__', None))
-            extra = '\\[{}]'.format(', '.join(format_annotation(param) for param in params))
-
-        module = annotation.__module__.split('.')[0]    # hack to 'fix' class linking for Instaloader project
-        return ':py:class:`~{}.{}`{}'.format(module, annotation_cls.__qualname__,
-                                             extra)
-
-    return str(annotation)
-
-
-def process_signature(app, what: str, name: str, obj, options, signature, return_annotation):
-    if not callable(obj):
-        return
-
-    if what in ('class', 'exception'):
-        obj = getattr(obj, '__init__', getattr(obj, '__new__', None))
-
-    if not getattr(obj, '__annotations__', None):
-        return
-
-    obj = unwrap(obj)
-    signature = Signature(obj)
-    parameters = [
-        param.replace(annotation=inspect.Parameter.empty)
-        for param in signature.signature.parameters.values()
-    ]
-
-    if parameters:
-        if what in ('class', 'exception'):
-            del parameters[0]
-        elif what == 'method':
-            outer = inspect.getmodule(obj)
-            for clsname in obj.__qualname__.split('.')[:-1]:
-                outer = getattr(outer, clsname)
-
-            method_name = obj.__name__
-            if method_name.startswith("__") and not method_name.endswith("__"):
-                # If the method starts with double underscore (dunder)
-                # Python applies mangling so we need to prepend the class name.
-                # This doesn't happen if it always ends with double underscore.
-                class_name = obj.__qualname__.split('.')[-2]
-                method_name = "_{c}{m}".format(c=class_name, m=method_name)
-
-            method_object = outer.__dict__[method_name]
-            if not isinstance(method_object, (classmethod, staticmethod)):
-                del parameters[0]
-
-    signature.signature = signature.signature.replace(
-        parameters=parameters,
-        return_annotation=inspect.Signature.empty)
-
-    return signature.format_args().replace('\\', '\\\\'), None
-
-
-def process_docstring(app, what, name, obj, options, lines):
-    if isinstance(obj, property):
-        obj = obj.fget
-
-    if callable(obj):
-        if what in ('class', 'exception'):
-            obj = getattr(obj, '__init__')
-
-        obj = unwrap(obj)
-        try:
-            type_hints = get_type_hints(obj)
-        except (AttributeError, TypeError):
-            # Introspecting a slot wrapper will raise TypeError
-            return
-        except NameError as exc:
-            logger.warning('Cannot resolve forward reference in type annotations of "%s": %s',
-                           name, exc)
-            type_hints = obj.__annotations__
-
-        for argname, annotation in type_hints.items():
-            if argname.endswith('_'):
-                argname = '{}\\_'.format(argname[:-1])
-
-            formatted_annotation = format_annotation(annotation)
-
-            if argname == 'return':
-                if what in ('class', 'exception'):
-                    # Don't add return type None from __init__()
-                    continue
-
-                insert_index = len(lines)
-                for i, line in enumerate(lines):
-                    if line.startswith(':rtype:'):
-                        insert_index = None
-                        break
-                    elif line.startswith(':return:') or line.startswith(':returns:'):
-                        insert_index = i
-
-                if insert_index is not None:
-                    if insert_index == len(lines):
-                        # Ensure that :rtype: doesn't get joined with a paragraph of text, which
-                        # prevents it being interpreted.
-                        lines.append('')
-                        insert_index += 1
-
-                    lines.insert(insert_index, ':rtype: {}'.format(formatted_annotation))
-            else:
-                searchfor = ':param {}:'.format(argname)
-                for i, line in enumerate(lines):
-                    if line.startswith(searchfor):
-                        lines.insert(i, ':type {}: {}'.format(argname, formatted_annotation))
-                        break

docs/sponsors.rst

@@ -0,0 +1,23 @@
+.. _sponsors:
+
+Sponsoring the development of Instaloader
+=========================================
+
+.. donations-start
+
+.. include:: ../README.rst
+   :start-after: current-sponsors-start
+   :end-before: current-sponsors-end
+
+It is a pleasure for us to share our Instaloader to the world, and we are proud
+to have attracted such an active and motivating community, with so many users
+who share their suggestions and ideas with us. Buying a community-sponsored beer
+or coffee from time to time is very likely to further raise our passion for the
+development of Instaloader.
+
+| For Donations, we provide GitHub Sponsors page, a PayPal.Me link and a Bitcoin address.
+|  GitHub Sponsors: `Sponsor @aandergr on GitHub Sponsors <https://github.com/sponsors/aandergr>`__
+|  PayPal: `PayPal.me/aandergr <https://www.paypal.me/aandergr>`__
+|  BTC: 1Nst4LoadeYzrKjJ1DX9CpbLXBYE9RKLwY
+
+.. donations-end

docs/troubleshooting.rst

@@ -9,10 +9,7 @@ Troubleshooting
 ---------------------
 
 Instaloader has a logic to keep track of its requests to Instagram and to obey
-their rate limits. Since they are nowhere documented, we try them out
-experimentally. We have a daily cron job running to confirm that Instaloader
-still stays within the rate limits. Nevertheless, the rate control logic assumes
-that
+their rate limits. The rate controller assumes that
 
 - at one time, Instaloader is the only application that consumes requests, i.e.
   neither the Instagram browser interface, nor a mobile app, nor another
@@ -21,7 +18,13 @@ that
 - no requests had been consumed when Instaloader starts.
 
 The latter one implies that restarting or reinstantiating Instaloader often
-within short time is prone to cause a 429. If a request is denied with a 429,
+within short time is prone to cause a 429.
+
+Since the behavior of the rate controller might change between different
+versions of Instaloader, make sure to use the current version of Instaloader,
+especially when encountering many 429 errors.
+
+If a request is denied with a 429,
 Instaloader retries the request as soon as the temporary ban is assumed to be
 expired. In case the retry continuously fails for some reason, which should not
 happen under normal conditions, consider adjusting the
@@ -30,15 +33,20 @@ happen under normal conditions, consider adjusting the
 There have been observations that services, that in their nature offer
 promiscuous IP addresses, such as cloud, VPN and public proxy services, might be
 subject to significantly stricter limits for anonymous access. However,
-logged-in accesses (see :option:`--login`) do not seem to be affected.
+:ref:`logged-in accesses<login>` do not seem to be affected.
+
+Instaloader allows to adjust the rate controlling behavior by overriding
+:class:`instaloader.RateController`.
 
 Too many queries in the last time
 ---------------------------------
 
 **"Too many queries in the last time"** is not an error. It is a notice that the
 rate limit has almost been reached, according to Instaloader's own rate
-accounting mechanism. We regularly adjust this mechanism to match Instagram's
-current rate limiting.
+accounting mechanism.
+
+Instaloader allows to adjust the rate controlling behavior by overriding
+:class:`instaloader.RateController`.
 
 Private but not followed
 ------------------------
@@ -54,12 +62,12 @@ Two-Factor-Authentication. It also supports handling the *checkpoint challenge*,
 issued when Instagram suspects authentication activity on your account, by
 pointing the user to an URL to be opened in a browser.
 
-Nevertheless, in :issue:`92` and :issue:`615` users reported problems with
+Nevertheless, in :issue:`92`, :issue:`615`, :issue:`1150` and :issue:`1217`,
+users reported problems with
 logging in. We recommend to always keep the session file which Instaloader
 creates when using :option:`--login`. If a session file is present,
-:option:`--login` does not make make use of the failure-prone login procedure.
-Also, session files usually do not expire and can be copied between different
-computers without any problems.
+:option:`--login` does not make use of the failure-prone login procedure.
+Also, session files usually do not expire.
 
 If you do not have a session file present, you may use the following script
 (:example:`615_import_firefox_session.py`) to workaround login problems by

instaloader/__init__.py

@@ -1,7 +1,7 @@
 """Download pictures (or videos) along with their captions and other metadata from Instagram."""
 
 
-__version__ = '4.5.3'
+__version__ = '4.12'
 
 
 try:
@@ -15,6 +15,9 @@ else:
 from .exceptions import *
 from .instaloader import Instaloader
 from .instaloadercontext import InstaloaderContext, RateController
+from .lateststamps import LatestStamps
 from .nodeiterator import NodeIterator, FrozenNodeIterator, resumable_iteration
 from .structures import (Hashtag, Highlight, Post, PostSidecarNode, PostComment, PostCommentAnswer, PostLocation,
-                         Profile, Story, StoryItem, TopSearchResults, load_structure_from_file, save_structure_to_file)
+                         Profile, Story, StoryItem, TopSearchResults, TitlePic,
+                         load_structure_from_file, save_structure_to_file,
+                         load_structure, get_json_structure)

instaloader/__main__.py

@@ -5,16 +5,32 @@ import datetime
 import os
 import re
 import sys
-from argparse import ArgumentParser, SUPPRESS
+from argparse import ArgumentParser, ArgumentTypeError, SUPPRESS
+from enum import IntEnum
 from typing import List, Optional
 
-from . import (Instaloader, InstaloaderException, InvalidArgumentException, Post, Profile, ProfileNotExistsException,
-               StoryItem, __version__, load_structure_from_file, TwoFactorAuthRequiredException,
-               BadCredentialsException)
-from .instaloader import get_default_session_filename
+from . import (AbortDownloadException, BadCredentialsException, Instaloader, InstaloaderException,
+               InvalidArgumentException, LoginException, Post, Profile, ProfileNotExistsException, StoryItem,
+               TwoFactorAuthRequiredException, __version__, load_structure_from_file)
+from .instaloader import (get_default_session_filename, get_default_stamps_filename)
 from .instaloadercontext import default_user_agent
+from .lateststamps import LatestStamps
+try:
+    import browser_cookie3
+    bc3_library = True
+except ImportError:
+    bc3_library = False
 
 
+class ExitCode(IntEnum):
+    SUCCESS = 0
+    NON_FATAL_ERROR = 1
+    INIT_FAILURE = 2
+    LOGIN_FAILURE = 3
+    DOWNLOAD_ABORTED = 4
+    USER_ABORTED = 5
+    UNEXPECTED_ERROR = 99
+
 def usage_string():
     # NOTE: duplicated in README.rst and docs/index.rst
     argv0 = os.path.basename(sys.argv[0])
@@ -27,6 +43,14 @@ def usage_string():
 {0} --help""".format(argv0, len(argv0), '')
 
 
+def http_status_code_list(code_list_str: str) -> List[int]:
+    codes = [int(s) for s in code_list_str.split(',')]
+    for code in codes:
+        if not 100 <= code <= 599:
+            raise ArgumentTypeError("Invalid HTTP status code: {}".format(code))
+    return codes
+
+
 def filterstr_to_filterfunc(filter_str: str, item_type: type):
     """Takes an --post-filter=... or --storyitem-filter=... filter
      specification and makes a filter_func Callable out of it."""
@@ -36,7 +60,6 @@ def filterstr_to_filterfunc(filter_str: str, item_type: type):
 
     class TransformFilterAst(ast.NodeTransformer):
         def visit_Name(self, node: ast.Name):
-            # pylint:disable=no-self-use
             if not isinstance(node.ctx, ast.Load):
                 raise InvalidArgumentException("Invalid filter: Modifying variables ({}) not allowed.".format(node.id))
             if node.id == "datetime":
@@ -59,6 +82,56 @@ def filterstr_to_filterfunc(filter_str: str, item_type: type):
     return filterfunc
 
 
+def get_cookies_from_instagram(domain, browser, cookie_file='', cookie_name=''):
+    supported_browsers = {
+        "brave": browser_cookie3.brave,
+        "chrome": browser_cookie3.chrome,
+        "chromium": browser_cookie3.chromium,
+        "edge": browser_cookie3.edge,
+        "firefox": browser_cookie3.firefox,
+        "librewolf": browser_cookie3.librewolf,
+        "opera": browser_cookie3.opera,
+        "opera_gx": browser_cookie3.opera_gx,
+        "safari": browser_cookie3.safari,
+        "vivaldi": browser_cookie3.vivaldi,
+    }
+
+    if browser not in supported_browsers:
+        raise InvalidArgumentException("Loading cookies from the specified browser failed\n"
+                                       "Supported browsers are Brave, Chrome, Chromium, Edge, Firefox, LibreWolf, "
+                                       "Opera, Opera_GX, Safari and Vivaldi")
+
+    cookies = {}
+    browser_cookies = list(supported_browsers[browser](cookie_file=cookie_file))
+
+    for cookie in browser_cookies:
+        if domain in cookie.domain:
+            cookies[cookie.name] = cookie.value
+
+    if cookies:
+        print(f"Cookies loaded successfully from {browser}")
+    else:
+        raise LoginException(f"No cookies found for Instagram in {browser}, "
+                             f"Are you logged in succesfully in {browser}?")
+
+    if cookie_name:
+        return cookies.get(cookie_name, {})
+    else:
+        return cookies
+
+
+def import_session(browser, instaloader, cookiefile):
+    cookie = get_cookies_from_instagram('instagram', browser, cookiefile)
+    if cookie is not None:
+        instaloader.context.update_cookies(cookie)
+        username = instaloader.test_login()
+        if not username:
+            raise LoginException(f"Not logged in. Are you logged in successfully in {browser}?")
+        instaloader.context.username = username
+        print(f"{username} has been successfully logged in.")
+        print(f"Next time use --login={username} to reuse the same session.")
+
+
 def _main(instaloader: Instaloader, targetlist: List[str],
           username: Optional[str] = None, password: Optional[str] = None,
           sessionfile: Optional[str] = None,
@@ -68,8 +141,11 @@ def _main(instaloader: Instaloader, targetlist: List[str],
           download_tagged: bool = False,
           download_igtv: bool = False,
           fast_update: bool = False,
+          latest_stamps_file: Optional[str] = None,
           max_count: Optional[int] = None, post_filter_str: Optional[str] = None,
-          storyitem_filter_str: Optional[str] = None) -> None:
+          storyitem_filter_str: Optional[str] = None,
+          browser: Optional[str] = None,
+          cookiefile: Optional[str] = None) -> ExitCode:
     """Download set of profiles, hashtags etc. and handle logging in and session files if desired."""
     # Parse and generate filter function
     post_filter = None
@@ -80,6 +156,15 @@ def _main(instaloader: Instaloader, targetlist: List[str],
     if storyitem_filter_str is not None:
         storyitem_filter = filterstr_to_filterfunc(storyitem_filter_str, StoryItem)
         instaloader.context.log('Only download storyitems with property "{}".'.format(storyitem_filter_str))
+    latest_stamps = None
+    if latest_stamps_file is not None:
+        latest_stamps = LatestStamps(latest_stamps_file)
+        instaloader.context.log(f"Using latest stamps from {latest_stamps_file}.")
+    # load cookies if browser is not None
+    if browser and bc3_library:
+        import_session(browser.lower(), instaloader, cookiefile)
+    elif browser and not bc3_library:
+        raise InvalidArgumentException("browser_cookie3 library is needed to load cookies from browsers")
     # Login, if desired
     if username is not None:
         if not re.match(r"^[A-Za-z0-9._]+$", username):
@@ -95,22 +180,32 @@ def _main(instaloader: Instaloader, targetlist: List[str],
                 try:
                     instaloader.login(username, password)
                 except TwoFactorAuthRequiredException:
+                    # https://github.com/instaloader/instaloader/issues/1217
+                    instaloader.context.error("Warning: There have been reports of 2FA currently not working. "
+                                              "Consider importing session cookies from your browser with "
+                                              "--load-cookies.")
                     while True:
                         try:
                             code = input("Enter 2FA verification code: ")
                             instaloader.two_factor_login(code)
                             break
-                        except BadCredentialsException:
+                        except BadCredentialsException as err:
+                            print(err, file=sys.stderr)
                             pass
             else:
-                instaloader.interactive_login(username)
+                try:
+                    instaloader.interactive_login(username)
+                except KeyboardInterrupt:
+                    print("\nInterrupted by user.", file=sys.stderr)
+                    return ExitCode.USER_ABORTED
         instaloader.context.log("Logged in as %s." % username)
     # since 4.2.9 login is required for geotags
     if instaloader.download_geotags and not instaloader.context.is_logged_in:
-        instaloader.context.error("Warning: Use --login to download geotags of posts.")
+        instaloader.context.error("Warning: Login is required to download geotags of posts.")
     # Try block for KeyboardInterrupt (save session on ^C)
     profiles = set()
     anonymous_retry_profiles = set()
+    exit_code = ExitCode.SUCCESS
     try:
         # Generate set of profiles, already downloading non-profile targets
         for target in targetlist:
@@ -164,7 +259,7 @@ def _main(instaloader: Instaloader, targetlist: List[str],
                                                      post_filter=post_filter)
                 elif re.match(r"^[A-Za-z0-9._]+$", target):
                     try:
-                        profile = instaloader.check_profile_id(target)
+                        profile = instaloader.check_profile_id(target, latest_stamps)
                         if instaloader.context.is_logged_in and profile.has_blocked_viewer:
                             if download_profile_pic or ((download_posts or download_tagged or download_igtv)
                                                         and not profile.is_private):
@@ -183,7 +278,8 @@ def _main(instaloader: Instaloader, targetlist: List[str],
                             instaloader.context.log("Trying again anonymously, helps in case you are just blocked.")
                             with instaloader.anonymous_copy() as anonymous_loader:
                                 with instaloader.context.error_catcher():
-                                    anonymous_retry_profiles.add(anonymous_loader.check_profile_id(target))
+                                    anonymous_retry_profiles.add(anonymous_loader.check_profile_id(target,
+                                                                                                   latest_stamps))
                                     instaloader.context.error("Warning: {} will be downloaded anonymously (\"{}\")."
                                                               .format(target, err))
                         else:
@@ -198,21 +294,27 @@ def _main(instaloader: Instaloader, targetlist: List[str],
         if len(profiles) > 1:
             instaloader.context.log("Downloading {} profiles: {}".format(len(profiles),
                                                                          ' '.join([p.username for p in profiles])))
-        if profiles and (download_profile_pic or download_posts) and not instaloader.context.is_logged_in:
-            instaloader.context.error("Warning: Use --login to download higher-quality versions of pictures.")
+        if instaloader.context.iphone_support and profiles and (download_profile_pic or download_posts) and \
+           not instaloader.context.is_logged_in:
+            instaloader.context.log("Hint: Login to download higher-quality versions of pictures.")
         instaloader.download_profiles(profiles,
                                       download_profile_pic, download_posts, download_tagged, download_igtv,
                                       download_highlights, download_stories,
-                                      fast_update, post_filter, storyitem_filter)
+                                      fast_update, post_filter, storyitem_filter, latest_stamps=latest_stamps)
         if anonymous_retry_profiles:
             instaloader.context.log("Downloading anonymously: {}"
                                     .format(' '.join([p.username for p in anonymous_retry_profiles])))
             with instaloader.anonymous_copy() as anonymous_loader:
                 anonymous_loader.download_profiles(anonymous_retry_profiles,
                                                    download_profile_pic, download_posts, download_tagged, download_igtv,
-                                                   fast_update=fast_update, post_filter=post_filter)
+                                                   fast_update=fast_update, post_filter=post_filter,
+                                                   latest_stamps=latest_stamps)
     except KeyboardInterrupt:
         print("\nInterrupted by user.", file=sys.stderr)
+        exit_code = ExitCode.USER_ABORTED
+    except AbortDownloadException as exc:
+        print("\nDownload aborted: {}.".format(exc), file=sys.stderr)
+        exit_code = ExitCode.DOWNLOAD_ABORTED
     # Save session if it is useful
     if instaloader.context.is_logged_in:
         instaloader.save_session_to_file(sessionfile)
@@ -222,14 +324,15 @@ def _main(instaloader: Instaloader, targetlist: List[str],
             # Instaloader did at least save a session file
             instaloader.context.log("No targets were specified, thus nothing has been downloaded.")
         else:
-            # Instloader did not do anything
+            # Instaloader did not do anything
             instaloader.context.log("usage:" + usage_string())
+            exit_code = ExitCode.INIT_FAILURE
+    return exit_code
 
 
 def main():
     parser = ArgumentParser(description=__doc__, add_help=False, usage=usage_string(),
-                            epilog="Report issues at https://github.com/instaloader/instaloader/issues. "
-                                   "The complete documentation can be found at "
+                            epilog="The complete documentation can be found at "
                                    "https://instaloader.github.io/.",
                             fromfile_prefix_chars='+')
 
@@ -240,17 +343,17 @@ def main():
                            help="Download profile. If an already-downloaded profile has been renamed, Instaloader "
                                 "automatically finds it by its unique ID and renames the folder likewise.")
     g_targets.add_argument('_at_profile', nargs='*', metavar="@profile",
-                           help="Download all followees of profile. Requires --login. "
+                           help="Download all followees of profile. Requires login. "
                                 "Consider using :feed rather than @yourself.")
     g_targets.add_argument('_hashtag', nargs='*', metavar='"#hashtag"', help="Download #hashtag.")
     g_targets.add_argument('_location', nargs='*', metavar='%location_id',
-                           help="Download %%location_id. Requires --login.")
+                           help="Download %%location_id. Requires login.")
     g_targets.add_argument('_feed', nargs='*', metavar=":feed",
-                           help="Download pictures from your feed. Requires --login.")
+                           help="Download pictures from your feed. Requires login.")
     g_targets.add_argument('_stories', nargs='*', metavar=":stories",
-                           help="Download the stories of your followees. Requires --login.")
+                           help="Download the stories of your followees. Requires login.")
     g_targets.add_argument('_saved', nargs='*', metavar=":saved",
-                           help="Download the posts that you marked as saved. Requires --login.")
+                           help="Download the posts that you marked as saved. Requires login.")
     g_targets.add_argument('_singlepost', nargs='*', metavar="-- -shortcode",
                            help="Download the post with the given shortcode")
     g_targets.add_argument('_json', nargs='*', metavar="filename.json[.xz]",
@@ -268,6 +371,8 @@ def main():
                         help="Do not download regular posts.")
     g_prof.add_argument('--no-profile-pic', action='store_true',
                         help='Do not download profile picture.')
+    g_post.add_argument('--slide', action='store',
+                        help='Set what image/interval of a sidecar you want to download.')
     g_post.add_argument('--no-pictures', action='store_true',
                         help='Do not download post pictures. Cannot be used together with --fast-update. '
                              'Implies --no-video-thumbnails, does not imply --no-videos.')
@@ -279,11 +384,11 @@ def main():
                         help='Download geotags when available. Geotags are stored as a '
                              'text file with the location\'s name and a Google Maps link. '
                              'This requires an additional request to the Instagram '
-                             'server for each picture. Requires --login.')
+                             'server for each picture. Requires login.')
     g_post.add_argument('-C', '--comments', action='store_true',
                         help='Download and update comments for each post. '
                              'This requires an additional request to the Instagram '
-                             'server for each post, which is why it is disabled by default.')
+                             'server for each post, which is why it is disabled by default. Requires login.')
     g_post.add_argument('--no-captions', action='store_true',
                         help='Do not create txt files.')
     g_post.add_argument('--post-metadata-txt', action='append',
@@ -297,11 +402,11 @@ def main():
     g_post.add_argument('--no-compress-json', action='store_true',
                         help='Do not xz compress JSON files, rather create pretty formatted JSONs.')
     g_prof.add_argument('-s', '--stories', action='store_true',
-                        help='Also download stories of each profile that is downloaded. Requires --login.')
+                        help='Also download stories of each profile that is downloaded. Requires login.')
     g_prof.add_argument('--stories-only', action='store_true',
                         help=SUPPRESS)
     g_prof.add_argument('--highlights', action='store_true',
-                        help='Also download highlights of each profile that is downloaded. Requires --login.')
+                        help='Also download highlights of each profile that is downloaded. Requires login.')
     g_prof.add_argument('--tagged', action='store_true',
                         help='Also download posts where each profile is tagged.')
     g_prof.add_argument('--igtv', action='store_true',
@@ -312,7 +417,10 @@ def main():
     g_cond.add_argument('-F', '--fast-update', action='store_true',
                         help='For each target, stop when encountering the first already-downloaded picture. This '
                              'flag is recommended when you use Instaloader to update your personal Instagram archive.')
-
+    g_cond.add_argument('--latest-stamps', nargs='?', metavar='STAMPSFILE', const=get_default_stamps_filename(),
+                        help='Store the timestamps of latest media scraped for each profile. This allows updating '
+                             'your personal Instagram archive even if you delete the destination directories. '
+                             'If STAMPSFILE is not provided, defaults to ' + get_default_stamps_filename())
     g_cond.add_argument('--post-filter', '--only-if', metavar='filter',
                         help='Expression that, if given, must evaluate to True for each post to be downloaded. Must be '
                              'a syntactically valid python expression. Variables are evaluated to '
@@ -324,15 +432,20 @@ def main():
 
     g_cond.add_argument('-c', '--count',
                         help='Do not attempt to download more than COUNT posts. '
-                             'Applies only to #hashtag and :feed.')
+                             'Applies to #hashtag, %%location_id, :feed, and :saved.')
 
     g_login = parser.add_argument_group('Login (Download Private Profiles)',
                                         'Instaloader can login to Instagram. This allows downloading private profiles. '
                                         'To login, pass the --login option. Your session cookie (not your password!) '
                                         'will be saved to a local file to be reused next time you want Instaloader '
-                                        'to login.')
+                                        'to login. Instead of --login, the --load-cookies option can be used to '
+                                        'import a session from a browser.')
     g_login.add_argument('-l', '--login', metavar='YOUR-USERNAME',
                          help='Login name (profile name) for your Instagram account.')
+    g_login.add_argument('-b', '--load-cookies', metavar='BROWSER-NAME',
+                         help='Browser name to load cookies from Instagram')
+    g_login.add_argument('-B', '--cookiefile', metavar='COOKIE-FILE',
+                         help='Cookie file of a profile to load cookies')
     g_login.add_argument('-f', '--sessionfile',
                          help='Path for loading and storing session key file. '
                               'Defaults to ' + get_default_session_filename("<login_name>"))
@@ -347,13 +460,21 @@ def main():
                             '{target} is replaced by the target you specified, i.e. either :feed, #hashtag or the '
                             'profile name. Defaults to \'{target}\'.')
     g_how.add_argument('--filename-pattern',
-                       help='Prefix of filenames, relative to the directory given with '
+                       help='Prefix of filenames for posts and stories, relative to the directory given with '
                             '--dirname-pattern. {profile} is replaced by the profile name,'
                             '{target} is replaced by the target you specified, i.e. either :feed'
                             '#hashtag or the profile name. Defaults to \'{date_utc}_UTC\'')
+    g_how.add_argument('--title-pattern',
+                       help='Prefix of filenames for profile pics, hashtag profile pics, and highlight covers. '
+                            'Defaults to \'{date_utc}_UTC_{typename}\' if --dirname-pattern contains \'{target}\' '
+                            'or \'{dirname}\', or if --dirname-pattern is not specified. Otherwise defaults to '
+                            '\'{target}_{date_utc}_UTC_{typename}\'.')
     g_how.add_argument('--resume-prefix', metavar='PREFIX',
                        help='Prefix for filenames that are used to save the information to resume an interrupted '
                             'download.')
+    g_how.add_argument('--sanitize-paths', action='store_true',
+                       help='Sanitize paths so that the resulting file and directory names are valid on both '
+                            'Windows and Unix.')
     g_how.add_argument('--no-resume', action='store_true',
                        help='Do not resume a previously-aborted download iteration, and do not save such information '
                             'when interrupted.')
@@ -366,8 +487,13 @@ def main():
                             'connection fails, it can be manually skipped by hitting CTRL+C. Set this to 0 to retry '
                             'infinitely.')
     g_how.add_argument('--commit-mode', action='store_true', help=SUPPRESS)
-    g_how.add_argument('--request-timeout', metavar='N', type=float,
-                       help='seconds to wait before timing out a connection request')
+    g_how.add_argument('--request-timeout', metavar='N', type=float, default=300.0,
+                       help='Seconds to wait before timing out a connection request. Defaults to 300.')
+    g_how.add_argument('--abort-on', type=http_status_code_list, metavar="STATUS_CODES",
+                       help='Comma-separated list of HTTP status codes that cause Instaloader to abort, bypassing all '
+                            'retry logic.')
+    g_how.add_argument('--no-iphone', action='store_true',
+                        help='Do not attempt to download iPhone version of images and videos.')
 
     g_misc = parser.add_argument_group('Miscellaneous Options')
     g_misc.add_argument('-q', '--quiet', action='store_true',
@@ -380,15 +506,15 @@ def main():
 
     args = parser.parse_args()
     try:
-        if args.login is None and (args.stories or args.stories_only):
-            print("--login=USERNAME required to download stories.", file=sys.stderr)
+        if (args.login is None and args.load_cookies is None) and (args.stories or args.stories_only):
+            print("Login is required to download stories.", file=sys.stderr)
             args.stories = False
             if args.stories_only:
-                raise SystemExit(1)
+                raise InvalidArgumentException()
 
         if ':feed-all' in args.profile or ':feed-liked' in args.profile:
-            raise SystemExit(":feed-all and :feed-liked were removed. Use :feed as target and "
-                             "eventually --post-filter=viewer_has_liked.")
+            raise InvalidArgumentException(":feed-all and :feed-liked were removed. Use :feed as target and "
+                                           "eventually --post-filter=viewer_has_liked.")
 
         post_metadata_txt_pattern = '\n'.join(args.post_metadata_txt) if args.post_metadata_txt else None
         storyitem_metadata_txt_pattern = '\n'.join(args.storyitem_metadata_txt) if args.storyitem_metadata_txt else None
@@ -398,15 +524,18 @@ def main():
                 post_metadata_txt_pattern = ''
                 storyitem_metadata_txt_pattern = ''
             else:
-                raise SystemExit("--no-captions and --post-metadata-txt or --storyitem-metadata-txt given; "
-                                 "That contradicts.")
+                raise InvalidArgumentException("--no-captions and --post-metadata-txt or --storyitem-metadata-txt "
+                                               "given; That contradicts.")
 
         if args.no_resume and args.resume_prefix:
-            raise SystemExit("--no-resume and --resume-prefix given; That contradicts.")
+            raise InvalidArgumentException("--no-resume and --resume-prefix given; That contradicts.")
         resume_prefix = (args.resume_prefix if args.resume_prefix else 'iterator') if not args.no_resume else None
 
         if args.no_pictures and args.fast_update:
-            raise SystemExit('--no-pictures and --fast-update cannot be used together.')
+            raise InvalidArgumentException('--no-pictures and --fast-update cannot be used together.')
+
+        if args.login and args.load_cookies:
+            raise InvalidArgumentException('--load-cookies and --login cannot be used together.')
 
         # Determine what to download
         download_profile_pic = not args.no_profile_pic or args.profile_pic_only
@@ -425,25 +554,43 @@ def main():
                              max_connection_attempts=args.max_connection_attempts,
                              request_timeout=args.request_timeout,
                              resume_prefix=resume_prefix,
-                             check_resume_bbd=not args.use_aged_resume_files)
-        _main(loader,
-              args.profile,
-              username=args.login.lower() if args.login is not None else None,
-              password=args.password,
-              sessionfile=args.sessionfile,
-              download_profile_pic=download_profile_pic,
-              download_posts=download_posts,
-              download_stories=download_stories,
-              download_highlights=args.highlights,
-              download_tagged=args.tagged,
-              download_igtv=args.igtv,
-              fast_update=args.fast_update,
-              max_count=int(args.count) if args.count is not None else None,
-              post_filter_str=args.post_filter,
-              storyitem_filter_str=args.storyitem_filter)
+                             check_resume_bbd=not args.use_aged_resume_files,
+                             slide=args.slide,
+                             fatal_status_codes=args.abort_on,
+                             iphone_support=not args.no_iphone,
+                             title_pattern=args.title_pattern,
+                             sanitize_paths=args.sanitize_paths)
+        exit_code = _main(loader,
+                          args.profile,
+                          username=args.login.lower() if args.login is not None else None,
+                          password=args.password,
+                          sessionfile=args.sessionfile,
+                          download_profile_pic=download_profile_pic,
+                          download_posts=download_posts,
+                          download_stories=download_stories,
+                          download_highlights=args.highlights,
+                          download_tagged=args.tagged,
+                          download_igtv=args.igtv,
+                          fast_update=args.fast_update,
+                          latest_stamps_file=args.latest_stamps,
+                          max_count=int(args.count) if args.count is not None else None,
+                          post_filter_str=args.post_filter,
+                          storyitem_filter_str=args.storyitem_filter,
+                          browser=args.load_cookies,
+                          cookiefile=args.cookiefile)
         loader.close()
+        if loader.has_stored_errors:
+            exit_code = ExitCode.NON_FATAL_ERROR
+    except InvalidArgumentException as err:
+        print(err, file=sys.stderr)
+        exit_code = ExitCode.INIT_FAILURE
+    except LoginException as err:
+        print(err, file=sys.stderr)
+        exit_code = ExitCode.LOGIN_FAILURE
     except InstaloaderException as err:
-        raise SystemExit("Fatal error: %s" % err)
+        print("Fatal error: %s" % err)
+        exit_code = ExitCode.UNEXPECTED_ERROR
+    sys.exit(exit_code)
 
 
 if __name__ == "__main__":

instaloader/exceptions.py

@@ -33,7 +33,11 @@ class LoginRequiredException(InstaloaderException):
     pass
 
 
-class TwoFactorAuthRequiredException(InstaloaderException):
+class LoginException(InstaloaderException):
+    pass
+
+
+class TwoFactorAuthRequiredException(LoginException):
     pass
 
 
@@ -45,7 +49,7 @@ class BadResponseException(InstaloaderException):
     pass
 
 
-class BadCredentialsException(InstaloaderException):
+class BadCredentialsException(LoginException):
     pass
 
 
@@ -64,3 +68,17 @@ class QueryReturnedNotFoundException(ConnectionException):
 
 class TooManyRequestsException(ConnectionException):
     pass
+
+class IPhoneSupportDisabledException(InstaloaderException):
+    pass
+
+class AbortDownloadException(Exception):
+    """
+    Exception that is not catched in the error catchers inside the download loop and so aborts the
+    download loop.
+
+    This exception is not a subclass of ``InstaloaderException``.
+
+    .. versionadded:: 4.7
+    """
+    pass

instaloader/instaloadercontext.py

@@ -3,13 +3,13 @@ import json
 import os
 import pickle
 import random
-import re
 import shutil
 import sys
 import textwrap
 import time
 import urllib.parse
-from contextlib import contextmanager
+import uuid
+from contextlib import contextmanager, suppress
 from datetime import datetime, timedelta
 from functools import partial
 from typing import Any, Callable, Dict, Iterator, List, Optional, Union
@@ -24,17 +24,44 @@ def copy_session(session: requests.Session, request_timeout: Optional[float] = N
     """Duplicates a requests.Session."""
     new = requests.Session()
     new.cookies = requests.utils.cookiejar_from_dict(requests.utils.dict_from_cookiejar(session.cookies))
-    new.headers = session.headers.copy()
-    if request_timeout is not None:
-        # Override default timeout behavior.
-        # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
-        new.request = partial(new.request, timeout=request_timeout) # type: ignore
+    new.headers = session.headers.copy()  # type: ignore
+    # Override default timeout behavior.
+    # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
+    new.request = partial(new.request, timeout=request_timeout)  # type: ignore
     return new
 
 
 def default_user_agent() -> str:
-    return 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ' \
-           '(KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36'
+    return ('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 '
+            '(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36')
+
+
+def default_iphone_headers() -> Dict[str, Any]:
+    return {'User-Agent': 'Instagram 273.0.0.16.70 (iPad13,8; iOS 16_3; en_US; en-US; ' \
+                          'scale=2.00; 2048x2732; 452417278) AppleWebKit/420+',
+            'x-ads-opt-out': '1',
+            'x-bloks-is-panorama-enabled': 'true',
+            'x-bloks-version-id': '01507c21540f73e2216b6f62a11a5b5e51aa85491b72475c080da35b1228ddd6',
+            'x-fb-client-ip': 'True',
+            'x-fb-connection-type': 'wifi',
+            'x-fb-http-engine': 'Liger',
+            'x-fb-server-cluster': 'True',
+            'x-fb': '1',
+            'x-ig-abr-connection-speed-kbps': '2',
+            'x-ig-app-id': '124024574287414',
+            'x-ig-app-locale': 'en-US',
+            'x-ig-app-startup-country': 'US',
+            'x-ig-bandwidth-speed-kbps': '0.000',
+            'x-ig-capabilities': '36r/F/8=',
+            'x-ig-connection-speed': '{}kbps'.format(random.randint(1000, 20000)),
+            'x-ig-connection-type': 'WiFi',
+            'x-ig-device-locale': 'en-US',
+            'x-ig-mapped-locale': 'en-US',
+            'x-ig-timezone-offset': str((datetime.now().astimezone().utcoffset() or timedelta(seconds=0)).seconds),
+            'x-ig-www-claim': '0',
+            'x-pigeon-session-id': str(uuid.uuid4()),
+            'x-tigon-is-retry': 'False',
+            'x-whatsapp': '0'}
 
 
 class InstaloaderContext:
@@ -53,43 +80,57 @@ class InstaloaderContext:
     """
 
     def __init__(self, sleep: bool = True, quiet: bool = False, user_agent: Optional[str] = None,
-                 max_connection_attempts: int = 3, request_timeout: Optional[float] = None,
-                 rate_controller: Optional[Callable[["InstaloaderContext"], "RateController"]] = None):
+                 max_connection_attempts: int = 3, request_timeout: float = 300.0,
+                 rate_controller: Optional[Callable[["InstaloaderContext"], "RateController"]] = None,
+                 fatal_status_codes: Optional[List[int]] = None,
+                 iphone_support: bool = True):
 
         self.user_agent = user_agent if user_agent is not None else default_user_agent()
         self.request_timeout = request_timeout
         self._session = self.get_anonymous_session()
         self.username = None
+        self.user_id = None
         self.sleep = sleep
         self.quiet = quiet
         self.max_connection_attempts = max_connection_attempts
         self._graphql_page_length = 50
         self._root_rhx_gis = None
         self.two_factor_auth_pending = None
+        self.iphone_support = iphone_support
+        self.iphone_headers = default_iphone_headers()
 
         # error log, filled with error() and printed at the end of Instaloader.main()
-        self.error_log = []                      # type: List[str]
+        self.error_log: List[str] = []
 
         self._rate_controller = rate_controller(self) if rate_controller is not None else RateController(self)
 
         # Can be set to True for testing, disables supression of InstaloaderContext._error_catcher
         self.raise_all_errors = False
 
+        # HTTP status codes that should cause an AbortDownloadException
+        self.fatal_status_codes = fatal_status_codes or []
+
         # Cache profile from id (mapping from id to Profile)
-        self.profile_id_cache = dict()           # type: Dict[int, Any]
+        self.profile_id_cache: Dict[int, Any] = dict()
 
     @contextmanager
     def anonymous_copy(self):
         session = self._session
         username = self.username
+        user_id = self.user_id
+        iphone_headers = self.iphone_headers
         self._session = self.get_anonymous_session()
         self.username = None
+        self.user_id = None
+        self.iphone_headers = default_iphone_headers()
         try:
             yield self
         finally:
             self._session.close()
             self.username = username
             self._session = session
+            self.user_id = user_id
+            self.iphone_headers = iphone_headers
 
     @property
     def is_logged_in(self) -> bool:
@@ -110,6 +151,13 @@ class InstaloaderContext:
         if repeat_at_end:
             self.error_log.append(msg)
 
+    @property
+    def has_stored_errors(self) -> bool:
+        """Returns whether any error has been reported and stored to be repeated at program termination.
+
+        .. versionadded: 4.12"""
+        return bool(self.error_log)
+
     def close(self):
         """Print error log and close session"""
         if self.error_log and not self.quiet:
@@ -149,7 +197,6 @@ class InstaloaderContext:
         if empty_session_only:
             del header['Host']
             del header['Origin']
-            del header['Referer']
             del header['X-Instagram-AJAX']
             del header['X-Requested-With']
         return header
@@ -161,42 +208,61 @@ class InstaloaderContext:
                                 'ig_vw': '1920', 'csrftoken': '',
                                 's_network': '', 'ds_user_id': ''})
         session.headers.update(self._default_http_header(empty_session_only=True))
-        if self.request_timeout is not None:
-            # Override default timeout behavior.
-            # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
-            session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
+        # Override default timeout behavior.
+        # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
+        session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
         return session
 
-    def save_session_to_file(self, sessionfile):
-        """Not meant to be used directly, use :meth:`Instaloader.save_session_to_file`."""
-        pickle.dump(requests.utils.dict_from_cookiejar(self._session.cookies), sessionfile)
+    def save_session(self):
+        """Not meant to be used directly, use :meth:`Instaloader.save_session`."""
+        return requests.utils.dict_from_cookiejar(self._session.cookies)
 
-    def load_session_from_file(self, username, sessionfile):
-        """Not meant to be used directly, use :meth:`Instaloader.load_session_from_file`."""
+    def update_cookies(self, cookie):
+        """.. versionadded:: 4.11"""
+        self._session.cookies.update(cookie)
+
+    def load_session(self, username, sessiondata):
+        """Not meant to be used directly, use :meth:`Instaloader.load_session`."""
         session = requests.Session()
-        session.cookies = requests.utils.cookiejar_from_dict(pickle.load(sessionfile))
+        session.cookies = requests.utils.cookiejar_from_dict(sessiondata)
         session.headers.update(self._default_http_header())
         session.headers.update({'X-CSRFToken': session.cookies.get_dict()['csrftoken']})
-        if self.request_timeout is not None:
-            # Override default timeout behavior.
-            # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
-            session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
+        # Override default timeout behavior.
+        # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
+        session.request = partial(session.request, timeout=self.request_timeout)  # type: ignore
         self._session = session
         self.username = username
 
+    def save_session_to_file(self, sessionfile):
+        """Not meant to be used directly, use :meth:`Instaloader.save_session_to_file`."""
+        pickle.dump(self.save_session(), sessionfile)
+
+    def load_session_from_file(self, username, sessionfile):
+        """Not meant to be used directly, use :meth:`Instaloader.load_session_from_file`."""
+        self.load_session(username, pickle.load(sessionfile))
+
     def test_login(self) -> Optional[str]:
         """Not meant to be used directly, use :meth:`Instaloader.test_login`."""
-        data = self.graphql_query("d6f4427fbe92d846298cf93df0b937d3", {})
-        return data["data"]["user"]["username"] if data["data"]["user"] is not None else None
+        try:
+            data = self.graphql_query("d6f4427fbe92d846298cf93df0b937d3", {})
+            return data["data"]["user"]["username"] if data["data"]["user"] is not None else None
+        except (AbortDownloadException, ConnectionException) as err:
+            self.error(f"Error when checking if logged in: {err}")
+            return None
 
     def login(self, user, passwd):
         """Not meant to be used directly, use :meth:`Instaloader.login`.
 
-        :raises InvalidArgumentException: If the provided username does not exist.
         :raises BadCredentialsException: If the provided password is wrong.
-        :raises ConnectionException: If connection to Instagram failed.
         :raises TwoFactorAuthRequiredException: First step of 2FA login done, now call
-           :meth:`Instaloader.two_factor_login`."""
+           :meth:`Instaloader.two_factor_login`.
+        :raises LoginException: An error happened during login (for example, and invalid response).
+           Or if the provided username does not exist.
+
+        .. versionchanged:: 4.12
+           Raises LoginException instead of ConnectionException when an error happens.
+           Raises LoginException instead of InvalidArgumentException when the username does not exist.
+        """
         # pylint:disable=import-outside-toplevel
         import http.client
         # pylint:disable=protected-access
@@ -206,24 +272,30 @@ class InstaloaderContext:
                                 'ig_vw': '1920', 'ig_cb': '1', 'csrftoken': '',
                                 's_network': '', 'ds_user_id': ''})
         session.headers.update(self._default_http_header())
-        if self.request_timeout is not None:
-            # Override default timeout behavior.
-            # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
-            session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
-        session.get('https://www.instagram.com/web/__mid/')
+        # Override default timeout behavior.
+        # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
+        session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
+
+        # Make a request to Instagram's root URL, which will set the session's csrftoken cookie
+        # Not using self.get_json() here, because we need to access the cookie
+        session.get('https://www.instagram.com/')
+        # Add session's csrftoken cookie to session headers
         csrf_token = session.cookies.get_dict()['csrftoken']
         session.headers.update({'X-CSRFToken': csrf_token})
-        # Not using self.get_json() here, because we need to access csrftoken cookie
+
         self.do_sleep()
         # Workaround credits to pgrimaud.
         # See: https://github.com/pgrimaud/instagram-user-feed/commit/96ad4cf54d1ad331b337f325c73e664999a6d066
         enc_password = '#PWD_INSTAGRAM_BROWSER:0:{}:{}'.format(int(datetime.now().timestamp()), passwd)
-        login = session.post('https://www.instagram.com/accounts/login/ajax/',
+        login = session.post('https://www.instagram.com/api/v1/web/accounts/login/ajax/',
                              data={'enc_password': enc_password, 'username': user}, allow_redirects=True)
         try:
             resp_json = login.json()
-        except json.decoder.JSONDecodeError:
-            raise ConnectionException("Login error: JSON decode fail, {} - {}.".format(login.status_code, login.reason))
+
+        except json.decoder.JSONDecodeError as err:
+            raise LoginException(
+                "Login error: JSON decode fail, {} - {}.".format(login.status_code, login.reason)
+            ) from err
         if resp_json.get('two_factor_required'):
             two_factor_session = copy_session(session, self.request_timeout)
             two_factor_session.headers.update({'X-CSRFToken': csrf_token})
@@ -233,35 +305,37 @@ class InstaloaderContext:
                                             resp_json['two_factor_info']['two_factor_identifier'])
             raise TwoFactorAuthRequiredException("Login error: two-factor authentication required.")
         if resp_json.get('checkpoint_url'):
-            raise ConnectionException("Login: Checkpoint required. Point your browser to "
-                                      "https://www.instagram.com{} - "
-                                      "follow the instructions, then retry.".format(resp_json.get('checkpoint_url')))
+            raise LoginException(
+                f"Login: Checkpoint required. Point your browser to {resp_json.get('checkpoint_url')} - "
+                f"follow the instructions, then retry."
+            )
         if resp_json['status'] != 'ok':
             if 'message' in resp_json:
-                raise ConnectionException("Login error: \"{}\" status, message \"{}\".".format(resp_json['status'],
-                                                                                               resp_json['message']))
+                raise LoginException("Login error: \"{}\" status, message \"{}\".".format(resp_json['status'],
+                                                                                          resp_json['message']))
             else:
-                raise ConnectionException("Login error: \"{}\" status.".format(resp_json['status']))
+                raise LoginException("Login error: \"{}\" status.".format(resp_json['status']))
         if 'authenticated' not in resp_json:
             # Issue #472
             if 'message' in resp_json:
-                raise ConnectionException("Login error: Unexpected response, \"{}\".".format(resp_json['message']))
+                raise LoginException("Login error: Unexpected response, \"{}\".".format(resp_json['message']))
             else:
-                raise ConnectionException("Login error: Unexpected response, this might indicate a blocked IP.")
+                raise LoginException("Login error: Unexpected response, this might indicate a blocked IP.")
         if not resp_json['authenticated']:
             if resp_json['user']:
                 # '{"authenticated": false, "user": true, "status": "ok"}'
                 raise BadCredentialsException('Login error: Wrong password.')
             else:
                 # '{"authenticated": false, "user": false, "status": "ok"}'
-                # Raise InvalidArgumentException rather than BadCredentialException, because BadCredentialException
+                # Raise LoginException rather than BadCredentialException, because BadCredentialException
                 # triggers re-asking of password in Instaloader.interactive_login(), which makes no sense if the
                 # username is invalid.
-                raise InvalidArgumentException('Login error: User {} does not exist.'.format(user))
+                raise LoginException('Login error: User {} does not exist.'.format(user))
         # '{"authenticated": true, "user": true, "userId": ..., "oneTapPrompt": false, "status": "ok"}'
         session.headers.update({'X-CSRFToken': login.cookies['csrftoken']})
         self._session = session
         self.username = user
+        self.user_id = resp_json['userId']
 
     def two_factor_login(self, two_factor_code):
         """Second step of login if 2FA is enabled.
@@ -281,9 +355,9 @@ class InstaloaderContext:
         resp_json = login.json()
         if resp_json['status'] != 'ok':
             if 'message' in resp_json:
-                raise BadCredentialsException("Login error: {}".format(resp_json['message']))
+                raise BadCredentialsException("2FA error: {}".format(resp_json['message']))
             else:
-                raise BadCredentialsException("Login error: \"{}\" status.".format(resp_json['status']))
+                raise BadCredentialsException("2FA error: \"{}\" status.".format(resp_json['status']))
         session.headers.update({'X-CSRFToken': login.cookies['csrftoken']})
         self._session = session
         self.username = user
@@ -292,10 +366,28 @@ class InstaloaderContext:
     def do_sleep(self):
         """Sleep a short time if self.sleep is set. Called before each request to instagram.com."""
         if self.sleep:
-            time.sleep(min(random.expovariate(0.7), 5.0))
+            time.sleep(min(random.expovariate(0.6), 15.0))
+
+    @staticmethod
+    def _response_error(resp: requests.Response) -> str:
+        extra_from_json: Optional[str] = None
+        with suppress(json.decoder.JSONDecodeError):
+            resp_json = resp.json()
+            if "status" in resp_json:
+                extra_from_json = (
+                    f"\"{resp_json['status']}\" status, message \"{resp_json['message']}\""
+                    if "message" in resp_json
+                    else f"\"{resp_json['status']}\" status"
+                )
+        return (
+            f"{resp.status_code} {resp.reason}"
+            f"{f' - {extra_from_json}' if extra_from_json is not None else ''}"
+            f" when accessing {resp.url}"
+        )
 
     def get_json(self, path: str, params: Dict[str, Any], host: str = 'www.instagram.com',
-                 session: Optional[requests.Session] = None, _attempt=1) -> Dict[str, Any]:
+                 session: Optional[requests.Session] = None, _attempt=1,
+                 response_headers: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
         """JSON request to Instagram.
 
         :param path: URL, relative to the given domain which defaults to www.instagram.com/
@@ -320,50 +412,43 @@ class InstaloaderContext:
             if is_other_query:
                 self._rate_controller.wait_before_query('other')
             resp = sess.get('https://{0}/{1}'.format(host, path), params=params, allow_redirects=False)
+            if resp.status_code in self.fatal_status_codes:
+                redirect = " redirect to {}".format(resp.headers['location']) if 'location' in resp.headers else ""
+                body = ""
+                if resp.headers['Content-Type'].startswith('application/json'):
+                    body = ': ' + resp.text[:500] + ('…' if len(resp.text) > 501 else '')
+                raise AbortDownloadException("Query to https://{}/{} responded with \"{} {}\"{}{}".format(
+                    host, path, resp.status_code, resp.reason, redirect, body
+                ))
             while resp.is_redirect:
                 redirect_url = resp.headers['location']
                 self.log('\nHTTP redirect from https://{0}/{1} to {2}'.format(host, path, redirect_url))
-                if redirect_url.startswith('https://www.instagram.com/accounts/login'):
-                    # alternate rate limit exceeded behavior
-                    raise TooManyRequestsException("429 Too Many Requests: redirected to login")
+                if (redirect_url.startswith('https://www.instagram.com/accounts/login') or
+                    redirect_url.startswith('https://i.instagram.com/accounts/login')):
+                    if not self.is_logged_in:
+                        raise LoginRequiredException("Redirected to login page. Use --login or --load-cookies.")
+                    raise AbortDownloadException("Redirected to login page. You've been logged out, please wait " +
+                                                 "some time, recreate the session and try again")
                 if redirect_url.startswith('https://{}/'.format(host)):
                     resp = sess.get(redirect_url if redirect_url.endswith('/') else redirect_url + '/',
                                     params=params, allow_redirects=False)
                 else:
                     break
+            if response_headers is not None:
+                response_headers.clear()
+                response_headers.update(resp.headers)
             if resp.status_code == 400:
-                raise QueryReturnedBadRequestException("400 Bad Request")
+                raise QueryReturnedBadRequestException(self._response_error(resp))
             if resp.status_code == 404:
-                raise QueryReturnedNotFoundException("404 Not Found")
+                raise QueryReturnedNotFoundException(self._response_error(resp))
             if resp.status_code == 429:
-                raise TooManyRequestsException("429 Too Many Requests")
+                raise TooManyRequestsException(self._response_error(resp))
             if resp.status_code != 200:
-                raise ConnectionException("HTTP error code {}.".format(resp.status_code))
-            is_html_query = not is_graphql_query and not "__a" in params and host == "www.instagram.com"
-            if is_html_query:
-                match = re.search(r'window\._sharedData = (.*);</script>', resp.text)
-                if match is None:
-                    raise QueryReturnedNotFoundException("Could not find \"window._sharedData\" in html response.")
-                resp_json = json.loads(match.group(1))
-                entry_data = resp_json.get('entry_data')
-                post_or_profile_page = list(entry_data.values())[0] if entry_data is not None else None
-                if post_or_profile_page is None:
-                    raise ConnectionException("\"window._sharedData\" does not contain required keys.")
-                # If GraphQL data is missing in `window._sharedData`, search for it in `__additionalDataLoaded`.
-                if 'graphql' not in post_or_profile_page[0]:
-                    match = re.search(r'window\.__additionalDataLoaded\(.*?({.*"graphql":.*})\);</script>',
-                                      resp.text)
-                    if match is not None:
-                        post_or_profile_page[0]['graphql'] = json.loads(match.group(1))['graphql']
-                return resp_json
+                raise ConnectionException(self._response_error(resp))
             else:
                 resp_json = resp.json()
             if 'status' in resp_json and resp_json['status'] != "ok":
-                if 'message' in resp_json:
-                    raise ConnectionException("Returned \"{}\" status, message \"{}\".".format(resp_json['status'],
-                                                                                               resp_json['message']))
-                else:
-                    raise ConnectionException("Returned \"{}\" status.".format(resp_json['status']))
+                raise ConnectionException(self._response_error(resp))
             return resp_json
         except (ConnectionException, json.decoder.JSONDecodeError, requests.exceptions.RequestException) as err:
             error_string = "JSON Query to {}: {}".format(path, err)
@@ -381,7 +466,8 @@ class InstaloaderContext:
                         self._rate_controller.handle_429('iphone')
                     if is_other_query:
                         self._rate_controller.handle_429('other')
-                return self.get_json(path=path, params=params, host=host, session=sess, _attempt=_attempt + 1)
+                return self.get_json(path=path, params=params, host=host, session=sess, _attempt=_attempt + 1,
+                                     response_headers=response_headers)
             except KeyboardInterrupt:
                 self.error("[skipped by user]", repeat_at_end=False)
                 raise ConnectionException(error_string) from err
@@ -471,11 +557,57 @@ class InstaloaderContext:
 
         .. versionadded:: 4.2.1"""
         with copy_session(self._session, self.request_timeout) as tempsession:
-            tempsession.headers['User-Agent'] = 'Instagram 123.1.0.26.115 (iPhone12,1; iOS 13_3; en_US; en-US; ' \
-                                                'scale=2.00; 1656x3584; 190542906)'
-            for header in ['Host', 'Origin', 'X-Instagram-AJAX', 'X-Requested-With']:
+            # Set headers to simulate an API request from iPad
+            tempsession.headers['ig-intended-user-id'] = str(self.user_id)
+            tempsession.headers['x-pigeon-rawclienttime'] = '{:.6f}'.format(time.time())
+
+            # Add headers obtained from previous iPad request
+            tempsession.headers.update(self.iphone_headers)
+
+            # Extract key information from cookies if we haven't got it already from a previous request
+            header_cookies_mapping = {'x-mid': 'mid',
+                                     'ig-u-ds-user-id': 'ds_user_id',
+                                     'x-ig-device-id': 'ig_did',
+                                     'x-ig-family-device-id': 'ig_did',
+                                     'family_device_id': 'ig_did'}
+
+            # Map the cookie value to the matching HTTP request header
+            cookies = tempsession.cookies.get_dict().copy()
+            for key, value in header_cookies_mapping.items():
+                if value in cookies:
+                    if key not in tempsession.headers:
+                        tempsession.headers[key] = cookies[value]
+                    else:
+                        # Remove the cookie value if it's already specified as a header
+                        tempsession.cookies.pop(value, None)
+
+            # Edge case for ig-u-rur header due to special string encoding in cookie
+            if 'rur' in cookies:
+                if 'ig-u-rur' not in tempsession.headers:
+                    tempsession.headers['ig-u-rur'] = cookies['rur'].strip('\"').encode('utf-8') \
+                                                                    .decode('unicode_escape')
+                else:
+                    tempsession.cookies.pop('rur', None)
+
+            # Remove headers specific to Desktop version
+            for header in ['Host', 'Origin', 'X-Instagram-AJAX', 'X-Requested-With', 'Referer']:
                 tempsession.headers.pop(header, None)
-            return self.get_json(path, params, 'i.instagram.com', tempsession)
+
+            # No need for cookies if we have a bearer token
+            if 'authorization' in tempsession.headers:
+                tempsession.cookies.clear()
+
+            response_headers = dict()    # type: Dict[str, Any]
+            response = self.get_json(path, params, 'i.instagram.com', tempsession, response_headers=response_headers)
+
+            # Extract the ig-set-* headers and use them in the next request
+            for key, value in response_headers.items():
+                if key.startswith('ig-set-'):
+                    self.iphone_headers[key.replace('ig-set-', '')] = value
+                elif key.startswith('x-ig-set-'):
+                    self.iphone_headers[key.replace('x-ig-set-', 'x-ig-')] = value
+
+            return response
 
     def write_raw(self, resp: Union[bytes, requests.Response], filename: str) -> None:
         """Write raw response data into a file.
@@ -505,11 +637,11 @@ class InstaloaderContext:
         else:
             if resp.status_code == 403:
                 # suspected invalid URL signature
-                raise QueryReturnedForbiddenException("403 when accessing {}.".format(url))
+                raise QueryReturnedForbiddenException(self._response_error(resp))
             if resp.status_code == 404:
                 # 404 not worth retrying.
-                raise QueryReturnedNotFoundException("404 when accessing {}.".format(url))
-            raise ConnectionException("HTTP error code {}.".format(resp.status_code))
+                raise QueryReturnedNotFoundException(self._response_error(resp))
+            raise ConnectionException(self._response_error(resp))
 
     def get_and_write_raw(self, url: str, filename: str) -> None:
         """Downloads and writes anonymously-requested raw data into a file.
@@ -519,6 +651,28 @@ class InstaloaderContext:
         :raises ConnectionException: When download repeatedly failed."""
         self.write_raw(self.get_raw(url), filename)
 
+    def head(self, url: str, allow_redirects: bool = False) -> requests.Response:
+        """HEAD a URL anonymously.
+
+        :raises QueryReturnedNotFoundException: When the server responds with a 404.
+        :raises QueryReturnedForbiddenException: When the server responds with a 403.
+        :raises ConnectionException: When request failed.
+
+        .. versionadded:: 4.7.6
+        """
+        with self.get_anonymous_session() as anonymous_session:
+            resp = anonymous_session.head(url, allow_redirects=allow_redirects)
+        if resp.status_code == 200:
+            return resp
+        else:
+            if resp.status_code == 403:
+                # suspected invalid URL signature
+                raise QueryReturnedForbiddenException(self._response_error(resp))
+            if resp.status_code == 404:
+                # 404 not worth retrying.
+                raise QueryReturnedNotFoundException(self._response_error(resp))
+            raise ConnectionException(self._response_error(resp))
+
     @property
     def root_rhx_gis(self) -> Optional[str]:
         """rhx_gis string returned in the / query."""
@@ -549,19 +703,19 @@ class RateController:
 
     def __init__(self, context: InstaloaderContext):
         self._context = context
-        self._query_timestamps = dict()  # type: Dict[str, List[float]]
+        self._query_timestamps: Dict[str, List[float]] = dict()
         self._earliest_next_request_time = 0.0
+        self._iphone_earliest_next_request_time = 0.0
 
     def sleep(self, secs: float):
         """Wait given number of seconds."""
         # Not static, to allow for the behavior of this method to depend on context-inherent properties, such as
         # whether we are logged in.
-        # pylint:disable=no-self-use
         time.sleep(secs)
 
     def _dump_query_timestamps(self, current_time: float, failed_query_type: str):
         windows = [10, 11, 20, 22, 30, 60]
-        self._context.error("Requests within last {} minutes grouped by type:"
+        self._context.error("Number of requests within last {} minutes grouped by type:"
                             .format('/'.join(str(w) for w in windows)),
                             repeat_at_end=False)
         for query_type, times in self._query_timestamps.items():
@@ -573,11 +727,14 @@ class RateController:
             ), repeat_at_end=False)
 
     def count_per_sliding_window(self, query_type: str) -> int:
-        """Return how many requests can be done within the sliding window."""
+        """Return how many requests of the given type can be done within a sliding window of 11 minutes.
+
+        This is called by :meth:`RateController.query_waittime` and allows to simply customize wait times before queries
+        at query_type granularity. Consider overriding :meth:`RateController.query_waittime` directly if you need more
+        control."""
         # Not static, to allow for the count_per_sliding_window to depend on context-inherent properties, such as
         # whether we are logged in.
-        # pylint:disable=no-self-use
-        return 75 if query_type in ['iphone', 'other'] else 200
+        return 75 if query_type == 'other' else 200
 
     def _reqs_in_sliding_window(self, query_type: Optional[str], current_time: float, window: float) -> List[float]:
         if query_type is not None:
@@ -593,6 +750,7 @@ class RateController:
     def query_waittime(self, query_type: str, current_time: float, untracked_queries: bool = False) -> float:
         """Calculate time needed to wait before query can be executed."""
         per_type_sliding_window = 660
+        iphone_sliding_window = 1800
         if query_type not in self._query_timestamps:
             self._query_timestamps[query_type] = []
         self._query_timestamps[query_type] = list(filter(lambda t: t > current_time - 60 * 60,
@@ -618,25 +776,43 @@ class RateController:
 
         def untracked_next_request_time():
             if untracked_queries:
-                reqs_in_sliding_window = self._reqs_in_sliding_window(query_type, current_time, per_type_sliding_window)
-                self._earliest_next_request_time = min(reqs_in_sliding_window) + per_type_sliding_window + 6
-            return self._earliest_next_request_time
+                if query_type == "iphone":
+                    reqs_in_sliding_window = self._reqs_in_sliding_window(query_type, current_time,
+                                                                          iphone_sliding_window)
+                    self._iphone_earliest_next_request_time = min(reqs_in_sliding_window) + iphone_sliding_window + 18
+                else:
+                    reqs_in_sliding_window = self._reqs_in_sliding_window(query_type, current_time,
+                                                                          per_type_sliding_window)
+                    self._earliest_next_request_time = min(reqs_in_sliding_window) + per_type_sliding_window + 6
+            return max(self._iphone_earliest_next_request_time, self._earliest_next_request_time)
+
+        def iphone_next_request():
+            if query_type == "iphone":
+                reqs_in_sliding_window = self._reqs_in_sliding_window(query_type, current_time, iphone_sliding_window)
+                if len(reqs_in_sliding_window) >= 199:
+                    return min(reqs_in_sliding_window) + iphone_sliding_window + 18
+            return 0.0
 
         return max(0.0,
                    max(
                        per_type_next_request_time(),
                        gql_accumulated_next_request_time(),
                        untracked_next_request_time(),
+                       iphone_next_request(),
                    ) - current_time)
 
     def wait_before_query(self, query_type: str) -> None:
-        """This method is called before a query to Instagram. It calls :meth:`RateController.sleep` to wait
-        until the request can be made."""
+        """This method is called before a query to Instagram.
+
+        It calls :meth:`RateController.query_waittime` to determine the time needed to wait and then calls
+        :meth:`RateController.sleep` to wait until the request can be made."""
         waittime = self.query_waittime(query_type, time.monotonic(), False)
         assert waittime >= 0
         if waittime > 15:
-            self._context.log("\nToo many queries in the last time. Need to wait {} seconds, until {:%H:%M}."
-                              .format(round(waittime), datetime.now() + timedelta(seconds=waittime)))
+            formatted_waittime = ("{} seconds".format(round(waittime)) if waittime <= 666 else
+                                  "{} minutes".format(round(waittime / 60)))
+            self._context.log("\nToo many queries in the last time. Need to wait {}, until {:%H:%M}."
+                              .format(formatted_waittime, datetime.now() + timedelta(seconds=waittime)))
         if waittime > 0:
             self.sleep(waittime)
         if query_type not in self._query_timestamps:
@@ -645,8 +821,10 @@ class RateController:
             self._query_timestamps[query_type].append(time.monotonic())
 
     def handle_429(self, query_type: str) -> None:
-        """This method is called to handle a 429 Too Many Requests response. It calls :meth:`RateController.sleep` to
-         wait until we can repeat the same request."""
+        """This method is called to handle a 429 Too Many Requests response.
+
+        It calls :meth:`RateController.query_waittime` to determine the time needed to wait and then calls
+        :meth:`RateController.sleep` to wait until we can repeat the same request."""
         current_time = time.monotonic()
         waittime = self.query_waittime(query_type, current_time, True)
         assert waittime >= 0
@@ -656,8 +834,10 @@ class RateController:
                         "App while Instaloader is running.")
         self._context.error(textwrap.fill(text_for_429), repeat_at_end=False)
         if waittime > 1.5:
-            self._context.error("The request will be retried in {} seconds, at {:%H:%M}."
-                                .format(round(waittime), datetime.now() + timedelta(seconds=waittime)),
+            formatted_waittime = ("{} seconds".format(round(waittime)) if waittime <= 666 else
+                                  "{} minutes".format(round(waittime / 60)))
+            self._context.error("The request will be retried in {}, at {:%H:%M}."
+                                .format(formatted_waittime, datetime.now() + timedelta(seconds=waittime)),
                                 repeat_at_end=False)
         if waittime > 0:
             self.sleep(waittime)

instaloader/lateststamps.py

@@ -0,0 +1,117 @@
+import configparser
+from datetime import datetime, timezone
+from typing import Optional
+from os.path import dirname
+from os import makedirs
+
+
+class LatestStamps:
+    """LatestStamps class.
+
+    Convenience class for retrieving and storing data from the :option:`--latest-stamps` file.
+
+    :param latest_stamps_file: path to file.
+
+    .. versionadded:: 4.8"""
+    PROFILE_ID = 'profile-id'
+    PROFILE_PIC = 'profile-pic'
+    POST_TIMESTAMP = 'post-timestamp'
+    TAGGED_TIMESTAMP = 'tagged-timestamp'
+    IGTV_TIMESTAMP = 'igtv-timestamp'
+    STORY_TIMESTAMP = 'story-timestamp'
+    ISO_FORMAT = '%Y-%m-%dT%H:%M:%S.%f%z'
+
+    def __init__(self, latest_stamps_file):
+        self.file = latest_stamps_file
+        self.data = configparser.ConfigParser()
+        self.data.read(latest_stamps_file)
+
+    def _save(self):
+        if dn := dirname(self.file):
+            makedirs(dn, exist_ok=True)
+        with open(self.file, 'w') as f:
+            self.data.write(f)
+
+    def _ensure_section(self, section: str):
+        if not self.data.has_section(section):
+            self.data.add_section(section)
+
+    def get_profile_id(self, profile_name: str) -> Optional[int]:
+        """Returns stored ID of profile."""
+        try:
+            return self.data.getint(profile_name, self.PROFILE_ID)
+        except (configparser.Error, ValueError):
+            return None
+
+    def save_profile_id(self, profile_name: str, profile_id: int):
+        """Stores ID of profile."""
+        self._ensure_section(profile_name)
+        self.data.set(profile_name, self.PROFILE_ID, str(profile_id))
+        self._save()
+
+    def rename_profile(self, old_profile: str, new_profile: str):
+        """Renames a profile."""
+        self._ensure_section(new_profile)
+        for option in [self.PROFILE_ID, self.PROFILE_PIC, self.POST_TIMESTAMP,
+                       self.TAGGED_TIMESTAMP, self.IGTV_TIMESTAMP, self.STORY_TIMESTAMP]:
+            if self.data.has_option(old_profile, option):
+                value = self.data.get(old_profile, option)
+                self.data.set(new_profile, option, value)
+        self.data.remove_section(old_profile)
+        self._save()
+
+    def _get_timestamp(self, section: str, key: str) -> datetime:
+        try:
+            return datetime.strptime(self.data.get(section, key), self.ISO_FORMAT)
+        except (configparser.Error, ValueError):
+            return datetime.fromtimestamp(0, timezone.utc)
+
+    def _set_timestamp(self, section: str, key: str, timestamp: datetime):
+        self._ensure_section(section)
+        self.data.set(section, key, timestamp.strftime(self.ISO_FORMAT))
+        self._save()
+
+    def get_last_post_timestamp(self, profile_name: str) -> datetime:
+        """Returns timestamp of last download of a profile's posts."""
+        return self._get_timestamp(profile_name, self.POST_TIMESTAMP)
+
+    def set_last_post_timestamp(self, profile_name: str, timestamp: datetime):
+        """Sets timestamp of last download of a profile's posts."""
+        self._set_timestamp(profile_name, self.POST_TIMESTAMP, timestamp)
+
+    def get_last_tagged_timestamp(self, profile_name: str) -> datetime:
+        """Returns timestamp of last download of a profile's tagged posts."""
+        return self._get_timestamp(profile_name, self.TAGGED_TIMESTAMP)
+
+    def set_last_tagged_timestamp(self, profile_name: str, timestamp: datetime):
+        """Sets timestamp of last download of a profile's tagged posts."""
+        self._set_timestamp(profile_name, self.TAGGED_TIMESTAMP, timestamp)
+
+    def get_last_igtv_timestamp(self, profile_name: str) -> datetime:
+        """Returns timestamp of last download of a profile's igtv posts."""
+        return self._get_timestamp(profile_name, self.IGTV_TIMESTAMP)
+
+    def set_last_igtv_timestamp(self, profile_name: str, timestamp: datetime):
+        """Sets timestamp of last download of a profile's igtv posts."""
+        self._set_timestamp(profile_name, self.IGTV_TIMESTAMP, timestamp)
+
+    def get_last_story_timestamp(self, profile_name: str) -> datetime:
+        """Returns timestamp of last download of a profile's stories."""
+        return self._get_timestamp(profile_name, self.STORY_TIMESTAMP)
+
+    def set_last_story_timestamp(self, profile_name: str, timestamp: datetime):
+        """Sets timestamp of last download of a profile's stories."""
+        self._set_timestamp(profile_name, self.STORY_TIMESTAMP, timestamp)
+
+    def get_profile_pic(self, profile_name: str) -> str:
+        """Returns filename of profile's last downloaded profile pic."""
+        try:
+            return self.data.get(profile_name, self.PROFILE_PIC)
+        except configparser.Error:
+            return ""
+
+    def set_profile_pic(self, profile_name: str, profile_pic: str):
+        """Sets filename of profile's last downloaded profile pic."""
+        self._ensure_section(profile_name)
+        self.data.set(profile_name, self.PROFILE_PIC, profile_pic)
+        self._save()

instaloader/nodeiterator.py

@@ -5,21 +5,20 @@ import os
 from contextlib import contextmanager
 from datetime import datetime, timedelta
 from lzma import LZMAError
-from typing import Any, Callable, Dict, Iterator, NamedTuple, Optional, Tuple, TypeVar
+from typing import Any, Callable, Dict, Iterable, Iterator, NamedTuple, Optional, Tuple, TypeVar
 
-from .exceptions import InvalidArgumentException, QueryReturnedBadRequestException
+from .exceptions import AbortDownloadException, InvalidArgumentException, QueryReturnedBadRequestException
 from .instaloadercontext import InstaloaderContext
 
-FrozenNodeIterator = NamedTuple('FrozenNodeIterator',
-                                [('query_hash', str),
-                                 ('query_variables', Dict),
-                                 ('query_referer', Optional[str]),
-                                 ('context_username', Optional[str]),
-                                 ('total_index', int),
-                                 ('best_before', Optional[float]),
-                                 ('remaining_data', Optional[Dict])])
-FrozenNodeIterator.__doc__ = \
-    """A serializable representation of a :class:`NodeIterator` instance, saving its iteration state."""
+class FrozenNodeIterator(NamedTuple):
+    query_hash: str
+    query_variables: Dict
+    query_referer: Optional[str]
+    context_username: Optional[str]
+    total_index: int
+    best_before: Optional[float]
+    remaining_data: Optional[Dict]
+    first_node: Optional[Dict]
 FrozenNodeIterator.query_hash.__doc__ = """The GraphQL ``query_hash`` parameter."""
 FrozenNodeIterator.query_variables.__doc__ = """The GraphQL ``query_variables`` parameter."""
 FrozenNodeIterator.query_referer.__doc__ = """The HTTP referer used for the GraphQL query."""
@@ -28,7 +27,7 @@ FrozenNodeIterator.total_index.__doc__ = """Number of items that have already be
 FrozenNodeIterator.best_before.__doc__ = """Date when parts of the stored nodes might have expired."""
 FrozenNodeIterator.remaining_data.__doc__ = \
     """The already-retrieved, yet-unprocessed ``edges`` and the ``page_info`` at time of freezing."""
-
+FrozenNodeIterator.first_node.__doc__ = """Node data of the first item, if an item has been produced."""
 
 T = TypeVar('T')
 
@@ -55,6 +54,9 @@ class NodeIterator(Iterator[T]):
        post_iterator = profile.get_posts()
        post_iterator.thaw(load("resume_information.json"))
 
+    (an appropriate method to load and save the :class:`FrozenNodeIterator` is e.g.
+    :func:`load_structure_from_file` and :func:`save_structure_to_file`.)
+
     A :class:`FrozenNodeIterator` can only be thawn with a matching NodeIterator, i.e. a NodeIterator instance that has
     been constructed with the same parameters as the instance that is represented by the :class:`FrozenNodeIterator` in
     question. This is to ensure that an iteration cannot be resumed in a wrong, unmatching loop. As a quick way to
@@ -74,7 +76,8 @@ class NodeIterator(Iterator[T]):
                  node_wrapper: Callable[[Dict], T],
                  query_variables: Optional[Dict[str, Any]] = None,
                  query_referer: Optional[str] = None,
-                 first_data: Optional[Dict[str, Any]] = None):
+                 first_data: Optional[Dict[str, Any]] = None,
+                 is_first: Optional[Callable[[T, Optional[T]], bool]] = None):
         self._context = context
         self._query_hash = query_hash
         self._edge_extractor = edge_extractor
@@ -88,9 +91,11 @@ class NodeIterator(Iterator[T]):
             self._best_before = datetime.now() + NodeIterator._shelf_life
         else:
             self._data = self._query()
+        self._first_node: Optional[Dict] = None
+        self._is_first = is_first
 
     def _query(self, after: Optional[str] = None) -> Dict:
-        pagination_variables = {'first': NodeIterator._graphql_page_length}  # type: Dict[str, Any]
+        pagination_variables: Dict[str, Any] = {'first': NodeIterator._graphql_page_length}
         if after is not None:
             pagination_variables['after'] = after
         try:
@@ -124,17 +129,25 @@ class NodeIterator(Iterator[T]):
             except KeyboardInterrupt:
                 self._page_index, self._total_index = page_index, total_index
                 raise
-            return self._node_wrapper(node)
-        if self._data['page_info']['has_next_page']:
+            item = self._node_wrapper(node)
+            if self._is_first is not None:
+                if self._is_first(item, self.first_item):
+                    self._first_node = node
+            else:
+                if self._first_node is None:
+                    self._first_node = node
+            return item
+        if self._data.get('page_info', {}).get('has_next_page'):
             query_response = self._query(self._data['page_info']['end_cursor'])
-            page_index, data = self._page_index, self._data
-            try:
-                self._page_index = 0
-                self._data = query_response
-            except KeyboardInterrupt:
-                self._page_index, self._data = page_index, data
-                raise
-            return self.__next__()
+            if self._data['edges'] != query_response['edges'] and len(query_response['edges']) > 0:
+                page_index, data = self._page_index, self._data
+                try:
+                    self._page_index = 0
+                    self._data = query_response
+                except KeyboardInterrupt:
+                    self._page_index, self._data = page_index, data
+                    raise
+                return self.__next__()
         raise StopIteration()
 
     @property
@@ -150,16 +163,31 @@ class NodeIterator(Iterator[T]):
     @property
     def magic(self) -> str:
         """Magic string for easily identifying a matching iterator file for resuming (hash of some parameters)."""
-        if 'blake2b' not in hashlib.algorithms_available:
-            magic_hash = hashlib.new('sha224')
-        else:
-            # Use blake2b when possible, i.e. on Python >= 3.6.
-            magic_hash = hashlib.blake2b(digest_size=6)  # type:ignore  # pylint: disable=no-member
+        magic_hash = hashlib.blake2b(digest_size=6)
         magic_hash.update(json.dumps(
             [self._query_hash, self._query_variables, self._query_referer, self._context.username]
         ).encode())
         return base64.urlsafe_b64encode(magic_hash.digest()).decode()
 
+    @property
+    def first_item(self) -> Optional[T]:
+        """
+        If this iterator has produced any items, returns the first item produced.
+
+        It is possible to override what is considered the first item (for example, to consider the
+        newest item in case items are not in strict chronological order) by passing a callback
+        function as the `is_first` parameter when creating the class.
+
+        .. versionadded:: 4.8
+        .. versionchanged:: 4.9.2
+           What is considered the first item can be overridden.
+        """
+        return self._node_wrapper(self._first_node) if self._first_node is not None else None
+
+    @staticmethod
+    def page_length() -> int:
+        return NodeIterator._graphql_page_length
+
     def freeze(self) -> FrozenNodeIterator:
         """Freeze the iterator for later resuming."""
         remaining_data = None
@@ -174,6 +202,7 @@ class NodeIterator(Iterator[T]):
             total_index=max(self.total_index - 1, 0),
             best_before=self._best_before.timestamp() if self._best_before else None,
             remaining_data=remaining_data,
+            first_node=self._first_node,
         )
 
     def thaw(self, frozen: FrozenNodeIterator) -> None:
@@ -200,18 +229,21 @@ class NodeIterator(Iterator[T]):
         self._total_index = frozen.total_index
         self._best_before = datetime.fromtimestamp(frozen.best_before)
         self._data = frozen.remaining_data
+        if frozen.first_node is not None:
+            self._first_node = frozen.first_node
 
 
 @contextmanager
 def resumable_iteration(context: InstaloaderContext,
-                        iterator: Iterator,
+                        iterator: Iterable,
                         load: Callable[[InstaloaderContext, str], Any],
                         save: Callable[[FrozenNodeIterator, str], None],
                         format_path: Callable[[str], str],
                         check_bbd: bool = True,
                         enabled: bool = True) -> Iterator[Tuple[bool, int]]:
     """
-    High-level context manager to handle a resumable iteration that can be interrupted with a KeyboardInterrupt.
+    High-level context manager to handle a resumable iteration that can be interrupted
+    with a :class:`KeyboardInterrupt` or an :class:`AbortDownloadException`.
 
     It can be used as follows to automatically load a previously-saved state into the iterator, save the iterator's
     state when interrupted, and delete the resume file upon completion::
@@ -239,6 +271,9 @@ def resumable_iteration(context: InstaloaderContext,
     :param format_path: Returns the path to the resume file for the given magic.
     :param check_bbd: Whether to check the best before date and reject an expired FrozenNodeIterator.
     :param enabled: Set to False to disable all functionality and simply execute the inner body.
+
+    .. versionchanged:: 4.7
+       Also interrupt on :class:`AbortDownloadException`.
     """
     if not enabled or not isinstance(iterator, NodeIterator):
         yield False, 0
@@ -258,12 +293,13 @@ def resumable_iteration(context: InstaloaderContext,
             is_resuming = True
             start_index = iterator.total_index
             context.log("Resuming from {}.".format(resume_file_path))
-        except (InvalidArgumentException, LZMAError, json.decoder.JSONDecodeError) as exc:
+        except (InvalidArgumentException, LZMAError, json.decoder.JSONDecodeError, EOFError) as exc:
             context.error("Warning: Not resuming from {}: {}".format(resume_file_path, exc))
     try:
         yield is_resuming, start_index
-    except KeyboardInterrupt:
-        os.makedirs(os.path.dirname(resume_file_path), exist_ok=True)
+    except (KeyboardInterrupt, AbortDownloadException):
+        if os.path.dirname(resume_file_path):
+            os.makedirs(os.path.dirname(resume_file_path), exist_ok=True)
         save(iterator.freeze(), resume_file_path)
         context.log("\nSaved resume information to {}.".format(resume_file_path))
         raise

instaloader/sectioniterator.py

@@ -0,0 +1,46 @@
+from typing import Any, Callable, Dict, Iterator, Optional, TypeVar
+
+from .instaloadercontext import InstaloaderContext
+
+T = TypeVar('T')
+
+
+class SectionIterator(Iterator[T]):
+    """Iterator for the new 'sections'-style responses.
+
+    .. versionadded:: 4.9"""
+    def __init__(self,
+                 context: InstaloaderContext,
+                 sections_extractor: Callable[[Dict[str, Any]], Dict[str, Any]],
+                 media_wrapper: Callable[[Dict], T],
+                 query_path: str,
+                 first_data: Optional[Dict[str, Any]] = None):
+        self._context = context
+        self._sections_extractor = sections_extractor
+        self._media_wrapper = media_wrapper
+        self._query_path = query_path
+        self._data = first_data or self._query()
+        self._page_index = 0
+        self._section_index = 0
+
+    def __iter__(self):
+        return self
+
+    def _query(self, max_id: Optional[str] = None) -> Dict[str, Any]:
+        pagination_variables = {"max_id": max_id} if max_id is not None else {}
+        return self._sections_extractor(
+            self._context.get_json(self._query_path, params={"__a": 1, "__d": "dis", **pagination_variables})
+        )
+
+    def __next__(self) -> T:
+        if self._page_index < len(self._data['sections']):
+            media = self._data['sections'][self._page_index]['layout_content']['medias'][self._section_index]['media']
+            self._section_index += 1
+            if self._section_index >= len(self._data['sections'][self._page_index]['layout_content']['medias']):
+                self._section_index = 0
+                self._page_index += 1
+            return self._media_wrapper(media)
+        if self._data['more_available']:
+            self._page_index, self._section_index, self._data = 0, 0, self._query(self._data["next_max_id"])
+            return self.__next__()
+        raise StopIteration()

setup.py

@@ -3,7 +3,6 @@
 import re
 import sys
 import os
-import platform
 from setuptools import setup
 
 
@@ -19,13 +18,13 @@ def get_version():
     raise SystemExit("Could not find version string.")
 
 
-if sys.version_info < (3, 5):
-    sys.exit('Instaloader requires Python >= 3.5.')
+if sys.version_info < (3, 8):
+    sys.exit('Instaloader requires Python >= 3.8.')
 
 requirements = ['requests>=2.4']
-
-if platform.system() == 'Windows' and sys.version_info < (3, 6):
-    requirements.append('win_unicode_console')
+optional_requirements = {
+    'browser_cookie3': ['browser_cookie3>=0.19.1'],
+}
 
 keywords = (['instagram', 'instagram-scraper', 'instagram-client', 'instagram-feed', 'downloader', 'videos', 'photos',
              'pictures', 'instagram-user-photos', 'instagram-photos', 'instagram-metadata', 'instagram-downloader',
@@ -38,6 +37,7 @@ setup(
     name='instaloader',
     version=get_version(),
     packages=['instaloader'],
+    package_data={'instaloader': ['py.typed']},
     url='https://instaloader.github.io/',
     license='MIT',
     author='Alexander Graf, André Koch-Kramer',
@@ -46,9 +46,10 @@ setup(
                 'from Instagram.',
     long_description=open(os.path.join(SRC, 'README.rst')).read(),
     install_requires=requirements,
-    python_requires='>=3.5',
+    python_requires='>=3.8',
+    extras_require=optional_requirements,
     entry_points={'console_scripts': ['instaloader=instaloader.__main__:main']},
-    zip_safe=True,
+    zip_safe=False,
     keywords=keywords,
     classifiers=[
         'Development Status :: 5 - Production/Stable',
@@ -57,10 +58,11 @@ setup(
         'Intended Audience :: Developers',
         'License :: OSI Approved :: MIT License',
         'Operating System :: OS Independent',
-        'Programming Language :: Python :: 3.5',
-        'Programming Language :: Python :: 3.6',
-        'Programming Language :: Python :: 3.7',
         'Programming Language :: Python :: 3.8',
+        'Programming Language :: Python :: 3.9',
+        'Programming Language :: Python :: 3.10',
+        'Programming Language :: Python :: 3.11',
+        'Programming Language :: Python :: 3.12',
         'Programming Language :: Python :: 3 :: Only',
         'Topic :: Internet',
         'Topic :: Multimedia :: Graphics'

test/instaloader_unittests.py

@@ -23,7 +23,7 @@ PRIVATE_PROFILE_ID = 1706625676
 EMPTY_PROFILE = "not_public"
 EMPTY_PROFILE_ID = 1928659031
 
-ratecontroller = None  # type: Optional[instaloader.RateController]
+ratecontroller: Optional[instaloader.RateController] = None
 
 
 class TestInstaloaderAnonymously(unittest.TestCase):