attempt to fix 2FA

Fixes #1905.

.github/ISSUE_TEMPLATE/bug_report.md

@@ -24,4 +24,9 @@ If applicable, add error messages and tracebacks to help explain your problem.
 **Additional context**
 Add any other context about the problem here.
 
-<!-- please also see https://instaloader.github.io/contributing.html for how to report a bug -->
+<!--
+Also see https://instaloader.github.io/contributing.html for how to report a bug.
+
+Please understand that a bug issue that lacks the required information cannot be
+processed.
+-->

.github/ISSUE_TEMPLATE/feature_request.md

@@ -6,13 +6,15 @@ labels: "feature suggestion"
 ---
 
 **Provide us a use case of the feature**
-How could the user invoke the new function? Which problem would it solve? If new information is obtained, how would it be further processed?
+How could the user invoke the new function? Which problem would it solve?
+If new information is obtained, how would it be further processed?
 
 **Describe the solution you'd like**
 A clear and concise description of what you want to happen.
 
 **Describe alternatives you've considered**
 A clear and concise description of any alternative solutions or features you've considered.
+How do you achieve the here-described goal without the feature you are requesting?
 
 **If the feature request is accepted, would you be willing to submit a pull request?**
 Yes / No
@@ -21,4 +23,9 @@ Yes / No
 **Additional context**
 Add any other context about the feature request here.
 
-<!-- please also see https://instaloader.github.io/contributing.html for how to suggest a feature -->
+<!-- 
+Also see https://instaloader.github.io/contributing.html for how to suggest a feature.
+
+Please understand that a feature suggestion issue that lacks the required information cannot be
+processed.
+-->

.github/ISSUE_TEMPLATE/question.md

@@ -7,6 +7,12 @@ labels: question
 
 Your question here...
 
-<!-- Please see our documentation: https://instaloader.github.io/ -->
+<!--
+People will be better able and more motivated to provide help if your question
 
-<!-- Basic questions regarding Instaloader's usage might be more appropriate to ask on Stack Overflow -->
+ - has a concise and meaningful title,
+ - is not covered by the documentation: https://instaloader.github.io/,
+ - is actually related to Instaloader (this is not a Python help forum),
+ - has not already been answered here or on other sites such as Stack Overflow: https://stackoverflow.com/questions/tagged/instaloader,
+ - is written in a polite and welcoming tone.
+-->

.github/funding.yml

@@ -0,0 +1,2 @@
+github: [aandergr]
+custom: "https://instaloader.github.io/sponsors.html"

.github/pull_request_template.md

@@ -1,7 +1,8 @@
 
 
 <!--
-Please describe:
+Thanks for your willingness to contribute to Instaloader! Please briefly
+describe
 
 - A motivation for this change, e.g.
   - Fixes # .

.github/workflows/documentation.yml

@@ -0,0 +1,31 @@
+name: "Documentation"
+
+on: [push, pull_request]
+
+jobs:
+  docs:
+    name: "Documentation"
+    runs-on: ubuntu-latest
+    steps:
+      - name: "Checkout Instaloader Repository"
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0  # needed for building docs
+      - name: "Setup Python"
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.8
+      - name: "Install Dependencies"
+        run: |
+          python -m pip install pipenv==2023.2.4
+          pipenv --python `python --version | grep -Eo '3\.[0-9]+'` sync --dev
+      - name: "Build Documentation"
+        run: pipenv --python `python --version | grep -Eo '3\.[0-9]+'` run make -C docs html SPHINXOPTS="-W -n"
+      - name: "Deploy Documentation"
+        if: github.event_name == 'push' && github.ref == 'refs/heads/master'
+        uses: JamesIves/github-pages-deploy-action@v4.2.5
+        with:
+          branch: master
+          repository-name: instaloader/instaloader.github.io
+          folder: docs/_build/html
+          token: ${{ secrets.PAGES_DEPLOYMENT_TOKEN }}

.github/workflows/lint.yml

@@ -7,24 +7,21 @@ jobs:
     name: PyLint and MyPy
     runs-on: ubuntu-latest
     strategy:
+      fail-fast: false
       matrix:
-        python-version: [3.8]
+        python-version: ["3.8", "3.9", "3.10", "3.11"]
     steps:
       - name: Checkout Instaloader Repository
         uses: actions/checkout@v2
-        with:
-          fetch-depth: 0  # needed for building docs
       - name: Setup Python
         uses: actions/setup-python@v2
         with:
           python-version: ${{ matrix.python-version }}
       - name: Install Dependencies
         run: |
-          python -m pip install pipenv==2020.6.2
+          python -m pip install pipenv==2023.2.4
-          pipenv sync --dev
+          pipenv --python `python --version | grep -Eo '3\.[0-9]+'` sync --dev
       - name: PyLint
-        run: pipenv run pylint instaloader
+        run: pipenv --python `python --version | grep -Eo '3\.[0-9]+'` run pylint instaloader
       - name: MyPy
-        run: pipenv run mypy -m instaloader
+        run: pipenv --python `python --version | grep -Eo '3\.[0-9]+'` run mypy -m instaloader
-      - name: Build Documentation
-        run: pipenv run make -C docs html SPHINXOPTS="-W -n"

.github/workflows/packages.yml

@@ -0,0 +1,36 @@
+name: "Publish Packages"
+
+on:
+  push:
+    tags:
+      - v*
+
+jobs:
+  publish:
+    name: "Publish Package"
+    runs-on: ubuntu-latest
+    steps:
+      - name: "Checkout Repository"
+        uses: actions/checkout@v2
+      - name: "Setup Python"
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.8
+      - name: "Get the tagged version"
+        id: get_version
+        env:
+          GITHUB_REF: ${{ github.ref }}
+        run: echo ::set-output name=VERSION::${GITHUB_REF/refs\/tags\/v/}
+        shell: bash
+      - name: "Generate Python Package"
+        run: python setup.py sdist
+      - name: "Publish Distribution to PyPI"
+        uses: pypa/gh-action-pypi-publish@master
+        with:
+          password: ${{ secrets.PYPI_PASSWORD }}
+      - name: "Publish Distribution to AUR"
+        run: deploy/arch/deploy.sh $VERSION_TAG
+        env:
+          VERSION_TAG: v${{ steps.get_version.outputs.VERSION }}
+          AUR_KEY: ${{ secrets.AUR_KEY }}
+          AUR_IV: ${{ secrets.AUR_IV }}

.github/workflows/stale.yml

@@ -18,7 +18,7 @@ jobs:
         stale-issue-message: 'There has been no activity on this question for an extended period of time. This issue will be closed after further 14 days of inactivity.'
         stale-issue-label: 'stale'
         exempt-issue-labels: 'leave open'
-        days-before-stale: 15
+        days-before-stale: 21
         days-before-close: -1
         remove-stale-when-updated: false
     - uses: actions/stale@v1
@@ -30,5 +30,5 @@ jobs:
         stale-pr-label: 'stale'
         exempt-issue-label: 'leave open'
         exempt-pr-label: 'leave open'
-        days-before-stale: 90
+        days-before-stale: 189
         days-before-close: 14

.github/workflows/windows_exe.yml

@@ -10,9 +10,9 @@ jobs:
       - name: Checkout Instaloader repository
         uses: actions/checkout@v2
       - name: Setup Python
-        uses: actions/setup-python@v1
+        uses: actions/setup-python@v2
         with:
-          python-version: 3.8
+          python-version: "3.9"
           architecture: x64
       - name: Get the tagged version
         id: get_version

.travis.yml

@@ -1,36 +0,0 @@
-dist: xenial
-language: python
-python:
- - "3.5"
- - "3.6"
- - "3.7"
-install:
- - pipenv sync --dev
-script:
- - pylint instaloader
- - mypy -m instaloader
- - make -C docs html SPHINXOPTS="-W -n"
-addons:
-  ssh_known_hosts: aur.archlinux.org
-deploy:
-  - provider: pypi
-    user: aandergr
-    on:
-      tags: true
-      python: 3.6
-    password:
-      secure: KBsATtXNnEzqr0X2ALdyx2zaYj2EUfK8YjP4jEOilBELpJ4evHYXVgRkn9CKXGsD7LKbWmR3gz2rr8NZ6U42pHw0fpQHizz91MpJSKbNY7RZZh4UVWMDh81izeglchWAF+m3Wdpxpo/GBDbrZIbRQFhPgB+kjSzJo3uiCS+yVXtGG+6csJHzd0WkJjsVHz+XSdxyOiLy/qFF3nmY6ZrVrgw6WbX2YLbx9je6DxK3YRrnBqgMgH7Ui2xWWbgJCk3KKpQTMiMKnTa9mdeMJAd26YMCz1gG/X7ck6VNly/6bs5+Zq7/bU0dSolhPBhNthyccYdS0IeGPdPF3YqatblJ4fLGTfHn8oJs3/T11yIW9pvOT3UftEynPOUX+en2Qnb/3WfgwUfVts8i8DAYgEFLGEzbF5XMGy4gwAD/bZ0JW0pmgAG2PZsdOokAt+ydlNj/uxq1GFW2doZbul6k4t03p1mik6itYscvuhKoEm8Yw6wr1P9aDvMR6FFpSu4gDt2XrKmRmWkSXxpNeCfZujTjyAnISxLVQGSY7BG5K0Z2HbDtqqNz9nj4WgNGzZmngiRofCevWR+vPIDUDW7NCX03csfcFS9RRllBO/BrL7NVUV+Qp9dfTQYp+8SqYLWgHtDoXdgyMREjJaAmbx2u/TYTY8bADZ5F3vGUpv2j5hP3NTU=
-  - provider: pages
-    skip-cleanup: true
-    github-token: $GITHUB_TOKEN
-    local-dir: docs/_build/html
-    repo: instaloader/instaloader.github.io
-    target-branch: master
-    on:
-      branch: master
-      python: 3.6
-  - provider: script
-    script: deploy/arch/deploy.sh $TRAVIS_TAG
-    on:
-      tags: true
-      python: 3.6

Pipfile

@@ -11,6 +11,13 @@ pyinstaller = "*"
 pefile = "*"
 pywin32-ctypes = "*"
 psutil = "*"
+typing-extensions = "*"
+types-requests = "*"
+typed-ast = "*"
+jinja2 = "~=3.0.1"
+wrapt = "*"  # Needed for Python<3.11
+tomli = "*"  # Needed for Python<3.11
+dill = "*"  # Needed for Python<3.11
 
 [packages]
 requests = "*"

README.rst

@@ -2,11 +2,7 @@
 
 .. badges-start
 
-|travis| |pypi| |pyversion| |license| |aur| |contributors| |downloads|
+|pypi| |pyversion| |license| |aur| |contributors| |downloads|
-
-.. |travis| image:: https://img.shields.io/travis/instaloader/instaloader/master.svg
-   :alt: Travis-CI Build Status
-   :target: https://travis-ci.org/instaloader/instaloader
 
 .. |pypi| image:: https://img.shields.io/pypi/v/instaloader.svg
    :alt: Instaloader PyPI Project Page
@@ -29,7 +25,7 @@
 
 .. |downloads| image:: https://pepy.tech/badge/instaloader/month
    :alt: PyPI Download Count
-   :target: https://pepy.tech/project/instaloader/month
+   :target: https://pepy.tech/project/instaloader
 
 .. badges-end
 
@@ -84,7 +80,19 @@ To later **update your local copy** of that profiles, you may run
     instaloader --fast-update profile [profile ...]
 
 If ``--fast-update`` is given, Instaloader stops when arriving at the
-first already-downloaded picture. When updating profiles, Instaloader
+first already-downloaded picture.
+
+Alternatively, you can use ``--latest-stamps`` to have Instaloader store
+the time each profile was last downloaded and only download newer media:
+
+::
+
+    instaloader --latest-stamps -- profile [profile ...]
+
+With this option it's possible to move or delete downloaded media and still keep
+the archive updated.
+
+When updating profiles, Instaloader
 automatically **detects profile name changes** and renames the target directory
 accordingly.
 
@@ -102,6 +110,38 @@ already have a valid session cookie file.
 
 `Instaloader Documentation <https://instaloader.github.io/basic-usage.html>`__
 
+Contributing
+------------
+
+As an open source project, Instaloader heavily depends on the contributions from
+its community. See
+`contributing <https://instaloader.github.io/contributing.html>`__
+for how you may help Instaloader to become an even greater tool.
+
+Supporters
+----------
+
+.. current-sponsors-start
+
+| Instaloader is proudly sponsored by
+|  `@rocketapi-io <https://github.com/rocketapi-io>`__
+|  `@socialmethod <https://github.com/socialmethod>`__
+
+See `Alex' GitHub Sponsors <https://github.com/sponsors/aandergr>`__ page for
+how you can sponsor the development of Instaloader!
+
+.. current-sponsors-end
+
+It is a pleasure for us to share our Instaloader to the world, and we are proud
+to have attracted such an active and motivating community, with so many users
+who share their suggestions and ideas with us. Buying a community-sponsored beer
+or coffee from time to time is very likely to further raise our passion for the
+development of Instaloader.
+
+| For Donations, we provide GitHub Sponsors page, a PayPal.Me link and a Bitcoin address.
+|  GitHub Sponsors: `Sponsor @aandergr on GitHub Sponsors <https://github.com/sponsors/aandergr>`__
+|  PayPal: `PayPal.me/aandergr <https://www.paypal.me/aandergr>`__
+|  BTC: 1Nst4LoadeYzrKjJ1DX9CpbLXBYE9RKLwY
 
 Disclaimer
 ----------
@@ -114,21 +154,3 @@ subsidiaries. This is an independent and unofficial project. Use at your own ris
 Instaloader is licensed under an MIT license. Refer to ``LICENSE`` file for more information.
 
 .. disclaimer-end
-
-Contributing
-------------
-
-As an open source project, Instaloader heavily depends on the contributions from
-its community. See
-`contributing <https://instaloader.github.io/contributing.html>`__
-for how you may help Instaloader to become an even greater tool.
-
-It is a pleasure for us to share our Instaloader to the world, and we are proud
-to have attracted such an active and motivating community, with so many users
-who share their suggestions and ideas with us. Buying a community-sponsored beer
-or coffee from time to time is very likely to further raise our passion for the
-development of Instaloader.
-
-| For Donations, we provide a PayPal.Me link and a Bitcoin address.
-|  PayPal: `PayPal.me/aandergr <https://www.paypal.me/aandergr>`__
-|  BTC: 1Nst4LoadeYzrKjJ1DX9CpbLXBYE9RKLwY

deploy/arch/.SRCINFO.template

@@ -6,7 +6,7 @@ pkgbase = instaloader
 	arch = any
 	license = MIT
 	makedepends = python-setuptools
-	depends = python>=3.5
+	depends = python>=3.8
 	depends = python-requests>=2.4
 	options = !emptydirs
 	source = instaloader-{{version}}.tar.gz::https://codeload.github.com/instaloader/instaloader/tar.gz/v{{version}}

deploy/arch/PKGBUILD.template

@@ -8,7 +8,7 @@ arch=('any')
 url="https://instaloader.github.io/"
 license=('MIT')
 groups=()
-depends=('python>=3.5' 'python-requests>=2.4')
+depends=('python>=3.8' 'python-requests>=2.4')
 makedepends=('python-setuptools')
 options=('!emptydirs')
 source=($pkgname-$pkgver.tar.gz::https://codeload.github.com/instaloader/instaloader/tar.gz/v$pkgver)

deploy/arch/deploy.sh

@@ -8,10 +8,12 @@ echo $VERSION | grep -qe "[abc]" && exit 0
 cd $(dirname $0)
 
 # decrypt and add ssh key
-openssl aes-256-cbc -K $encrypted_354637631c28_key -iv $encrypted_354637631c28_iv -in id_rsa_AUR.enc -out /tmp/AUR_openssh -d
+openssl aes-256-cbc -K $AUR_KEY -iv $AUR_IV -in id_rsa_AUR.enc -out /tmp/AUR_openssh -d
 eval "$(ssh-agent -s)"
 chmod 600 /tmp/AUR_openssh
 ssh-add /tmp/AUR_openssh
+mkdir -p ~/.ssh
+ssh-keyscan -t rsa aur.archlinux.org >> ~/.ssh/known_hosts
 
 # clone and modify AUR repo
 git clone --depth 1 ssh://aur@aur.archlinux.org/instaloader.git

deploy/windows/create_exe.py

@@ -44,8 +44,8 @@ with open('__main__.py', 'w+') as f:
     f.writelines(lines)
 
 # install dependencies and invoke PyInstaller
-commands = ["pip install pipenv==2020.6.2",
+commands = ["pip install pipenv==2023.2.4",
-            "pipenv sync --dev",
+            f"pipenv --python {sys.version_info.major}.{sys.version_info.minor} sync --dev",
             "pipenv run pyinstaller --log-level=DEBUG instaloader.spec"]
 
 for command in commands:

docs/basic-usage.rst

@@ -39,6 +39,16 @@ To later **update your local copy** of that profiles, you may run
 If :option:`--fast-update` is given, Instaloader stops when arriving at the
 first already-downloaded picture.
 
+Alternatively, you can use :option:`--latest-stamps` to have Instaloader store
+the time each profile was last downloaded and only download newer media:
+
+::
+
+    instaloader --latest-stamps -- profile [profile ...]
+
+With this option it's possible to move or delete downloaded media and still keep
+the archive updated.
+
 When updating profiles, Instaloader
 automatically **detects profile name changes** and renames the target directory
 accordingly.
@@ -149,27 +159,40 @@ target. The default is ``--dirname-pattern={target}``. In the dirname
 pattern, the token ``{target}`` is replaced by the target name, and
 ``{profile}`` is replaced by the owner of the post which is downloaded.
 
-:option:`--filename-pattern` configures the path of the post's files relative
+:option:`--filename-pattern` configures the path of the post and story's files relative
 to the target directory that is specified with :option:`--dirname-pattern`.
 The default is ``--filename-pattern={date_utc}_UTC``.
 The tokens ``{target}`` and ``{profile}`` are replaced like in the
-dirname pattern. The following tokens are defined for usage with
+dirname pattern.
-:option:`--filename-pattern`:
+
+:option:`--title-pattern` is similar to :option:`--filename-pattern`, but for profile
+pics, hashtag profile pics, and highlight covers. The default is
+``{date_utc}_UTC_{typename}`` if :option:`--dirname-pattern` contains ``{target}`` or
+``{profile}``, or ``{target}_{date_utc}_UTC_{typename}`` if it does not. Some tokens
+are not supported for this option, see below for details.
+
+The following tokens are defined for usage with
+:option:`--filename-pattern` and :option:`--title-pattern`:
 
 - ``{target}``
    Target name (as given in Instaloader command line)
 
 - ``{profile}`` (same as ``{owner_username}``)
-   Owner of the Post / StoryItem.
+   Owner of the Post / StoryItem / ProfilePic. For hashtag profile pics and
+   highlight covers, equivalent to ``{target}``.
 
 - ``{owner_id}``
-   Unique integer ID of owner profile.
+   Unique integer ID of owner profile. For hashtag profile pics, equivalent to
+   ``{target}``.
 
 - ``{shortcode}``
-   Shortcode (identifier string).
+   Shortcode (identifier string). Not available for :option:`--title-pattern`.
 
 - ``{mediaid}``
-   Integer representation of shortcode.
+   Integer representation of shortcode. Not available for :option:`--title-pattern`.
+
+- ``{filename}``
+   Instagram's internal filename.
 
 - ``{date_utc}`` (same as ``{date}``)
    Creation time in UTC timezone.
@@ -179,6 +202,10 @@ dirname pattern. The following tokens are defined for usage with
 
       {date_utc:%Y-%m-%d_%H-%M-%S}
 
+- ``{typename}``
+   Type of media being saved, such as GraphImage, GraphStoryVideo, profile_pic,
+   etc.
+
 For example, encode the poster's profile name in the filenames with::
 
     instaloader --filename-pattern={date_utc}_UTC_{profile} "#hashtag"

docs/cli-options.rst

@@ -82,6 +82,16 @@ What to Download of each Post
    Template to write in txt file for each StoryItem. See
    :ref:`metadata-text-files`.
 
+.. option:: --slide
+
+   Download only selected images of a sidecar. You can select single images using their
+   index in the sidecar starting with the leftmost or you can specify a range of images
+   with the following syntax: ``start_index-end_index``. Example:
+   ``--slide 1`` will select only the first image, ``--slide last`` only the last one and ``--slide 1-3`` will select only
+   the first three images.
+
+   .. versionadded:: 4.6
+
 .. option:: --no-metadata-json
 
    Do not create a JSON file containing the metadata of each post.
@@ -90,7 +100,6 @@ What to Download of each Post
 
    Do not xz compress JSON files, rather create pretty formatted JSONs.
 
-
 What to Download of each Profile
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -137,6 +146,23 @@ Which Posts to Download
    This flag is recommended when you use Instaloader to update your personal
    Instagram archive.
 
+.. option:: --latest-stamps [STAMPSFILE]
+
+   Works similarly to :option:`--fast-update`, but instead of relying on already
+   downloaded media, the time each profile was downloaded is stored, and only
+   media newer than the last download is fetched. This allows updating your
+   personal Instagram archive while emptying the target directories.
+
+   Only works for media associated with a specific profile, and that is returned
+   in chronological order: profile posts, profile stories, profile IGTV posts
+   and profile tagged posts.
+
+   By default, the information is stored in
+   ``~/.config/instaloader/latest-stamps.ini``, but you can specify an
+   alternative location.
+
+   .. versionadded:: 4.8
+
 .. option:: --post-filter filter, --only-if filter
 
    Expression that, if given, must evaluate to True for each post to be
@@ -154,8 +180,8 @@ Which Posts to Download
 
 .. option:: --count COUNT, -c
 
-   Do not attempt to download more than COUNT posts.  Applies only to
+   Do not attempt to download more than COUNT posts.  Applies to
-   ``#hashtag``, ``%location id``, and ``:feed``.
+   ``#hashtag``, ``%location_id``, ``:feed``, and ``:saved``.
 
 
 Login (Download Private Profiles)
@@ -196,12 +222,30 @@ How to Download
 
 .. option:: --filename-pattern FILENAME_PATTERN
 
-   Prefix of filenames, relative to the directory given with
+   Prefix of filenames for posts and stories, relative to the directory given with
    :option:`--dirname-pattern`. ``{profile}`` is replaced by the profile name,
    ``{target}`` is replaced by the target you specified, i.e.  either ``:feed``,
    ``#hashtag`` or the profile name. Defaults to ``{date_utc}_UTC``.
    See :ref:`filename-specification` for a list of supported tokens.
 
+.. option:: --title-pattern TITLE_PATTERN
+
+   Prefix of filenames for profile pics, hashtag profile pics, and highlight
+   covers, relative to the directory given with :option:`--dirname-pattern`.
+   Defaults to ``{date_utc}_UTC_{typename}`` if :option:`--dirname-pattern`
+   contains ``{target}`` or ``{profile}``, otherwise defaults to
+   ``{target}_{date_utc}_UTC_{typename}``.
+   See :ref:`filename-specification` for a list of supported tokens.
+
+   .. versionadded:: 4.8
+
+.. option:: --sanitize-paths
+
+   Force sanitization of paths so that the resulting file and directory names
+   are valid on both Windows and Unix.
+
+   .. versionadded:: 4.9
+
 .. option:: --resume-prefix prefix
 
    For many targets, Instaloader is capable of resuming a previously-aborted
@@ -210,12 +254,21 @@ How to Download
    used to save the information to resume an interrupted download.  The default
    prefix is ``iterator``.
 
-   Resuming an interrupted download is supported for most, but not all targets.
+   Resuming an interrupted download is supported for the following targets:
+    - Profile posts,
+    - Profile IGTV posts (:option:`--igtv`),
+    - Profile tagged posts (:option:`--tagged`),
+    - Saved posts (``:saved``),
+    - Hashtags.
+
+   This feature is enabled by default for targets where it is supported;
+   :option:`--resume-prefix` only changes the name of the iterator files.
+
+   To turn this feature off, use :option:`--no-resume`.
+
    JSON files with resume information are always compressed, regardless of
    :option:`--no-compress-json`.
 
-   This feature is turned off entirely with :option:`--no-resume`.
-
    .. versionadded:: 4.5
 
 .. option:: --no-resume
@@ -228,7 +281,7 @@ How to Download
 .. option:: --user-agent USER_AGENT
 
    User Agent to use for HTTP requests. Per default, Instaloader pretends being
-   Chrome/51.
+   Chrome/92 on Linux.
 
 .. option:: --max-connection-attempts N
 
@@ -238,10 +291,30 @@ How to Download
 
 .. option:: --request-timeout N
 
-   Seconds to wait before timing out a connection request.
+   Seconds to wait before timing out a connection request. Defaults to 300.
 
    .. versionadded:: 4.3
 
+   .. versionchanged:: 4.6
+      Enabled this option by default with a timeout of 300 seconds.
+
+.. option:: --abort-on STATUS_CODE_LIST
+
+   Comma-separated list of HTTP status codes that cause Instaloader to abort,
+   bypassing all retry logic.
+
+   For example, with ``--abort-on=302,400,429``, Instaloader will stop if a
+   request is responded with a 302 redirect, a Bad Request error, or a Too Many
+   Requests error.
+
+   .. versionadded:: 4.7
+
+.. option:: --no-iphone
+
+   Do not attempt to download iPhone version of images and videos.
+
+   .. versionadded:: 4.8
+
 Miscellaneous Options
 ^^^^^^^^^^^^^^^^^^^^^
 
@@ -256,12 +329,13 @@ Miscellaneous Options
    Read arguments from file `args.txt`, a shortcut to provide arguments from
    file rather than command-line. This provides a convenient way to hide login
    info from CLI, and can also be used to simplify management of long arguments.
+   You can provide more than one file at once, e.g.: ``+args1.txt +args2.txt``.
 
    .. note::
 
       Text file should separate arguments with line breaks.
 
-   args.txt example::
+   `args.txt` example::
 
       --login=MYUSERNAME
       --password=MYPASSWORD

docs/codesnippets/615_import_firefox_session.py

@@ -23,7 +23,7 @@ def get_cookiefile():
 
 def import_session(cookiefile, sessionfile):
     print("Using cookies from {}.".format(cookiefile))
-    conn = connect(cookiefile)
+    conn = connect(f"file:{cookiefile}?immutable=1", uri=True)
     try:
         cookie_data = conn.execute(
             "SELECT name, value FROM moz_cookies WHERE baseDomain='instagram.com'"

docs/conf.py

@@ -48,7 +48,7 @@ autodoc_default_flags = ['show-inheritance', 'members', 'undoc-members']
 autodoc_member_order = 'bysource'
 
 intersphinx_mapping = {'python': ('https://docs.python.org/3', None),
-                       'requests': ('https://requests.kennethreitz.org/en/master/', None)}
+                       'requests': ('https://requests.readthedocs.io/en/latest/', None)}
 
 nitpick_ignore = [('py:class', 'typing.Tuple')]
 
@@ -65,7 +65,7 @@ extlinks = {
         'Issue #'
     ),
     'example': (
-        'https://raw.githubusercontent.com/instaloader/instaloader/v' + current_release + '/docs/codesnippets/%s',
+        'https://raw.githubusercontent.com/instaloader/instaloader/master/docs/codesnippets/%s',
         'Example '
     ),
 }

docs/contributing.rst

@@ -11,6 +11,23 @@ Instaloader's development is organized on
 `GitHub <https://github.com/instaloader/instaloader>`__, where Issues and Pull
 Requests are discussed.
 
+Contributing to an open source project is an unmatchable opportunity to improve
+and manifest programming skills. For issues that are good starting points to get
+involved in Instaloader development, as they do not require deep insight into
+Instaloader's code base, have a look at our
+`Issues labelled 'good first issue' <https://github.com/instaloader/instaloader/contribute>`__.
+
+This document covers several ways of how you can contribute back to Instaloader.
+
+Answering Questions
+-------------------
+
+The easiest way to help out is to answer questions. If you are interested in
+answering questions regarding Instaloader, good places to begin are
+
+- `Questions tagged 'instaloader' on Stack Overflow <https://stackoverflow.com/questions/tagged/instaloader>`__,
+- `Instaloader Issues labeled 'question' <https://github.com/instaloader/instaloader/issues?q=is%3Aissue+is%3Aopen+label%3Aquestion>`__.
+
 Reporting Bugs
 --------------
 
@@ -41,22 +58,39 @@ reporting a problem, please keep the following in mind:
 Writing Code or Improving the Documentation
 -------------------------------------------
 
-Changes of the Instaloader source can be proposed as a
+Improvements of the Instaloader source or its documentation can be proposed as a
-`Pull Request <https://github.com/instaloader/instaloader/pulls>`__. There are only
+`Pull Request <https://github.com/instaloader/instaloader/pulls>`__.
-few things to consider:
 
-- Base your Pull Request on the ``master`` branch if it fixes a bug,
+- Please base your Pull Request on
-  or the ``upcoming/v4.X`` branch (if it exists at the moment of submitting the PR)
-  otherwise.
 
-- We use `Pylint <https://www.pylint.org/>`__ for error and syntax checking of
+  - ``master``, which will be released with the next minor release, if it is
-  the source and `MyPy <https://github.com/python/mypy>`__ for type checking.
-  Beware that sometimes it might be better to disable a warning rather than
-  adapting the code to a tool's desires.
 
-- The documentation source is located in the ``docs`` folder. The file
+    - a bug fix that does not require extensive testing,
-  ``cli-options.rst`` is merely an RST-formatted copy of ``instaloader --help``
+    - an improvement to the documentation,
-  output, of which the source is in ``instaloader/__main__.py``.
+
+  - ``upcoming/v4.X``, if it is
+
+    - a new feature,
+    - a bug fix that does require thorough testing before being released to a
+      final version.
+
+- All Pull Requests are analyzed by `Pylint <https://www.pylint.org/>`__ for
+  error and syntax checking of the source and
+  `Mypy <https://github.com/python/mypy>`__ for type checking. You can run them
+  locally with::
+
+     pylint instaloader
+     mypy -m instaloader
+
+- Improvements to the documentation are very welcome. The documentation is
+  created with `Sphinx <https://www.sphinx-doc.org/en/2.0/>`__, version 2,
+  and can be build locally using::
+
+     make -C docs html
+
+- Feel free to create an issue to make sure someone from the Instaloader team
+  agrees that the change might be an improvement, or if you want to discuss
+  your basic proposal, before working on a pull request.
 
 Proposing Features
 ------------------
@@ -98,21 +132,9 @@ details. If you feel the need to do so, please create a "feature suggestion".
   be implemented and the underlying problem could be solved. Also **describe
   alternatives** that you have considered.
 
-Donations
+Sponsoring
----------
+----------
 
-.. donations-start
+.. include:: sponsors.rst
-
+   :start-after: donations-start
-It is a pleasure for us to share our Instaloader to the world, and we are proud
+   :end-before: donations-end
-to have attracted such an active and motivating community, with so many users
-who share their suggestions and ideas with us. Buying a community-sponsored beer
-or coffee from time to time is very likely to further raise our passion for the
-development of Instaloader.
-
-| For donations, we provide a PayPal.Me link and a Bitcoin address.
-|  `PayPal.me/aandergr <https://www.paypal.me/aandergr>`__
-|  BTC: 1Nst4LoadeYzrKjJ1DX9CpbLXBYE9RKLwY
-
-.. donations-end
-
-.. (Discussion in :issue:`130`)

docs/index.rst

@@ -78,13 +78,6 @@ Useful Links
 - `Issue Tracker / Bug Tracker <https://github.com/instaloader/instaloader/issues>`__
 - `Version History <https://github.com/instaloader/instaloader/releases>`__
 
-Disclaimer
-----------
-
-.. include:: ../README.rst
-   :start-after: disclaimer-start
-   :end-before: disclaimer-end
-
 Contributing
 ------------
 
@@ -92,10 +85,20 @@ As an open source project, Instaloader heavily depends on the contributions from
 its community. See :ref:`contributing` for how you may help Instaloader to
 become an even greater tool.
 
-.. include:: contributing.rst
+Supporters
+----------
+
+.. include:: sponsors.rst
    :start-after: donations-start
    :end-before: donations-end
 
+Disclaimer
+----------
+
+.. include:: ../README.rst
+   :start-after: disclaimer-start
+   :end-before: disclaimer-end
+
 ..
    * :ref:`genindex`
    * :ref:`modindex`

docs/installation.rst

@@ -8,7 +8,7 @@ Install Instaloader
 To **install Instaloader**,
 
 #. Ensure that you have `Python <https://www.python.org/>`__, at least
-   version 3.5, and `pip <https://pypi.python.org/pypi/pip>`__
+   version 3.8, and `pip <https://pypi.python.org/pypi/pip>`__
    installed.
 
 #. Then, install Instaloader using::

docs/module/exceptions.rst

@@ -47,3 +47,5 @@ Exceptions
 
 
 .. autoexception:: TooManyRequestsException
+
+.. autoexception:: AbortDownloadException

docs/module/nodeiterator.rst

@@ -25,6 +25,13 @@ Iterator :class:`NodeIterator` and a context manager
 .. autoclass:: FrozenNodeIterator
    :no-show-inheritance:
 
+   A serializable representation of a :class:`NodeIterator` instance, saving
+   its iteration state.
+
+   It can be serialized and deserialized with :func:`save_structure_to_file`
+   and :func:`load_structure_from_file`, as well as with :mod:`json` and
+   :mod:`pickle` thanks to being a :class:`~typing.NamedTuple`.
+
 ``resumable_iteration``
 """""""""""""""""""""""
 

docs/module/structures.rst

@@ -77,6 +77,16 @@ Loading and Saving
 :class:`Post`, :class:`StoryItem`, :class:`Profile`, :class:`Hashtag` and
 :class:`FrozenNodeIterator` can be saved and loaded to/from JSON files.
 
+.. autofunction:: load_structure
+
 .. autofunction:: load_structure_from_file
 
+.. autofunction:: get_json_structure
+
 .. autofunction:: save_structure_to_file
+
+LatestStamps
+""""""""""""
+
+.. autoclass:: LatestStamps
+   :no-show-inheritance:

docs/sponsors.rst

@@ -0,0 +1,23 @@
+.. _sponsors:
+
+Sponsoring the development of Instaloader
+=========================================
+
+.. donations-start
+
+.. include:: ../README.rst
+   :start-after: current-sponsors-start
+   :end-before: current-sponsors-end
+
+It is a pleasure for us to share our Instaloader to the world, and we are proud
+to have attracted such an active and motivating community, with so many users
+who share their suggestions and ideas with us. Buying a community-sponsored beer
+or coffee from time to time is very likely to further raise our passion for the
+development of Instaloader.
+
+| For Donations, we provide GitHub Sponsors page, a PayPal.Me link and a Bitcoin address.
+|  GitHub Sponsors: `Sponsor @aandergr on GitHub Sponsors <https://github.com/sponsors/aandergr>`__
+|  PayPal: `PayPal.me/aandergr <https://www.paypal.me/aandergr>`__
+|  BTC: 1Nst4LoadeYzrKjJ1DX9CpbLXBYE9RKLwY
+
+.. donations-end

docs/troubleshooting.rst

@@ -62,7 +62,8 @@ Two-Factor-Authentication. It also supports handling the *checkpoint challenge*,
 issued when Instagram suspects authentication activity on your account, by
 pointing the user to an URL to be opened in a browser.
 
-Nevertheless, in :issue:`92` and :issue:`615` users reported problems with
+Nevertheless, in :issue:`92`, :issue:`615`, :issue:`1150` and :issue:`1217`,
+users reported problems with
 logging in. We recommend to always keep the session file which Instaloader
 creates when using :option:`--login`. If a session file is present,
 :option:`--login` does not make use of the failure-prone login procedure.

instaloader/__init__.py

@@ -1,7 +1,7 @@
 """Download pictures (or videos) along with their captions and other metadata from Instagram."""
 
 
-__version__ = '4.5.5'
+__version__ = '4.10'
 
 
 try:
@@ -15,6 +15,8 @@ else:
 from .exceptions import *
 from .instaloader import Instaloader
 from .instaloadercontext import InstaloaderContext, RateController
+from .lateststamps import LatestStamps
 from .nodeiterator import NodeIterator, FrozenNodeIterator, resumable_iteration
 from .structures import (Hashtag, Highlight, Post, PostSidecarNode, PostComment, PostCommentAnswer, PostLocation,
-                         Profile, Story, StoryItem, TopSearchResults, load_structure_from_file, save_structure_to_file)
+                         Profile, Story, StoryItem, TopSearchResults, load_structure_from_file, save_structure_to_file,
+                         load_structure, get_json_structure)

instaloader/__main__.py

@@ -5,14 +5,15 @@ import datetime
 import os
 import re
 import sys
-from argparse import ArgumentParser, SUPPRESS
+from argparse import ArgumentParser, ArgumentTypeError, SUPPRESS
 from typing import List, Optional
 
-from . import (Instaloader, InstaloaderException, InvalidArgumentException, Post, Profile, ProfileNotExistsException,
+from . import (AbortDownloadException, BadCredentialsException, Instaloader, InstaloaderException,
-               StoryItem, __version__, load_structure_from_file, TwoFactorAuthRequiredException,
+               InvalidArgumentException, Post, Profile, ProfileNotExistsException, StoryItem,
-               BadCredentialsException)
+               TwoFactorAuthRequiredException, __version__, load_structure_from_file)
-from .instaloader import get_default_session_filename
+from .instaloader import (get_default_session_filename, get_default_stamps_filename)
 from .instaloadercontext import default_user_agent
+from .lateststamps import LatestStamps
 
 
 def usage_string():
@@ -27,6 +28,14 @@ def usage_string():
 {0} --help""".format(argv0, len(argv0), '')
 
 
+def http_status_code_list(code_list_str: str) -> List[int]:
+    codes = [int(s) for s in code_list_str.split(',')]
+    for code in codes:
+        if not 100 <= code <= 599:
+            raise ArgumentTypeError("Invalid HTTP status code: {}".format(code))
+    return codes
+
+
 def filterstr_to_filterfunc(filter_str: str, item_type: type):
     """Takes an --post-filter=... or --storyitem-filter=... filter
      specification and makes a filter_func Callable out of it."""
@@ -36,7 +45,6 @@ def filterstr_to_filterfunc(filter_str: str, item_type: type):
 
     class TransformFilterAst(ast.NodeTransformer):
         def visit_Name(self, node: ast.Name):
-            # pylint:disable=no-self-use
             if not isinstance(node.ctx, ast.Load):
                 raise InvalidArgumentException("Invalid filter: Modifying variables ({}) not allowed.".format(node.id))
             if node.id == "datetime":
@@ -68,6 +76,7 @@ def _main(instaloader: Instaloader, targetlist: List[str],
           download_tagged: bool = False,
           download_igtv: bool = False,
           fast_update: bool = False,
+          latest_stamps_file: Optional[str] = None,
           max_count: Optional[int] = None, post_filter_str: Optional[str] = None,
           storyitem_filter_str: Optional[str] = None) -> None:
     """Download set of profiles, hashtags etc. and handle logging in and session files if desired."""
@@ -80,6 +89,10 @@ def _main(instaloader: Instaloader, targetlist: List[str],
     if storyitem_filter_str is not None:
         storyitem_filter = filterstr_to_filterfunc(storyitem_filter_str, StoryItem)
         instaloader.context.log('Only download storyitems with property "{}".'.format(storyitem_filter_str))
+    latest_stamps = None
+    if latest_stamps_file is not None:
+        latest_stamps = LatestStamps(latest_stamps_file)
+        instaloader.context.log(f"Using latest stamps from {latest_stamps_file}.")
     # Login, if desired
     if username is not None:
         if not re.match(r"^[A-Za-z0-9._]+$", username):
@@ -100,7 +113,8 @@ def _main(instaloader: Instaloader, targetlist: List[str],
                             code = input("Enter 2FA verification code: ")
                             instaloader.two_factor_login(code)
                             break
-                        except BadCredentialsException:
+                        except BadCredentialsException as err:
+                            print(err, file=sys.stderr)
                             pass
             else:
                 instaloader.interactive_login(username)
@@ -164,7 +178,7 @@ def _main(instaloader: Instaloader, targetlist: List[str],
                                                      post_filter=post_filter)
                 elif re.match(r"^[A-Za-z0-9._]+$", target):
                     try:
-                        profile = instaloader.check_profile_id(target)
+                        profile = instaloader.check_profile_id(target, latest_stamps)
                         if instaloader.context.is_logged_in and profile.has_blocked_viewer:
                             if download_profile_pic or ((download_posts or download_tagged or download_igtv)
                                                         and not profile.is_private):
@@ -183,7 +197,8 @@ def _main(instaloader: Instaloader, targetlist: List[str],
                             instaloader.context.log("Trying again anonymously, helps in case you are just blocked.")
                             with instaloader.anonymous_copy() as anonymous_loader:
                                 with instaloader.context.error_catcher():
-                                    anonymous_retry_profiles.add(anonymous_loader.check_profile_id(target))
+                                    anonymous_retry_profiles.add(anonymous_loader.check_profile_id(target,
+                                                                                                   latest_stamps))
                                     instaloader.context.error("Warning: {} will be downloaded anonymously (\"{}\")."
                                                               .format(target, err))
                         else:
@@ -198,21 +213,25 @@ def _main(instaloader: Instaloader, targetlist: List[str],
         if len(profiles) > 1:
             instaloader.context.log("Downloading {} profiles: {}".format(len(profiles),
                                                                          ' '.join([p.username for p in profiles])))
-        if profiles and (download_profile_pic or download_posts) and not instaloader.context.is_logged_in:
+        if instaloader.context.iphone_support and profiles and (download_profile_pic or download_posts) and \
-            instaloader.context.error("Warning: Use --login to download higher-quality versions of pictures.")
+           not instaloader.context.is_logged_in:
+            instaloader.context.log("Hint: Use --login to download higher-quality versions of pictures.")
         instaloader.download_profiles(profiles,
                                       download_profile_pic, download_posts, download_tagged, download_igtv,
                                       download_highlights, download_stories,
-                                      fast_update, post_filter, storyitem_filter)
+                                      fast_update, post_filter, storyitem_filter, latest_stamps=latest_stamps)
         if anonymous_retry_profiles:
             instaloader.context.log("Downloading anonymously: {}"
                                     .format(' '.join([p.username for p in anonymous_retry_profiles])))
             with instaloader.anonymous_copy() as anonymous_loader:
                 anonymous_loader.download_profiles(anonymous_retry_profiles,
                                                    download_profile_pic, download_posts, download_tagged, download_igtv,
-                                                   fast_update=fast_update, post_filter=post_filter)
+                                                   fast_update=fast_update, post_filter=post_filter,
+                                                   latest_stamps=latest_stamps)
     except KeyboardInterrupt:
         print("\nInterrupted by user.", file=sys.stderr)
+    except AbortDownloadException as exc:
+        print("\nDownload aborted: {}.".format(exc), file=sys.stderr)
     # Save session if it is useful
     if instaloader.context.is_logged_in:
         instaloader.save_session_to_file(sessionfile)
@@ -267,6 +286,8 @@ def main():
                         help="Do not download regular posts.")
     g_prof.add_argument('--no-profile-pic', action='store_true',
                         help='Do not download profile picture.')
+    g_post.add_argument('--slide', action='store',
+                        help='Set what image/interval of a sidecar you want to download.')
     g_post.add_argument('--no-pictures', action='store_true',
                         help='Do not download post pictures. Cannot be used together with --fast-update. '
                              'Implies --no-video-thumbnails, does not imply --no-videos.')
@@ -311,7 +332,10 @@ def main():
     g_cond.add_argument('-F', '--fast-update', action='store_true',
                         help='For each target, stop when encountering the first already-downloaded picture. This '
                              'flag is recommended when you use Instaloader to update your personal Instagram archive.')
-
+    g_cond.add_argument('--latest-stamps', nargs='?', metavar='STAMPSFILE', const=get_default_stamps_filename(),
+                        help='Store the timestamps of latest media scraped for each profile. This allows updating '
+                             'your personal Instagram archive even if you delete the destination directories. '
+                             'If STAMPSFILE is not provided, defaults to ' + get_default_stamps_filename())
     g_cond.add_argument('--post-filter', '--only-if', metavar='filter',
                         help='Expression that, if given, must evaluate to True for each post to be downloaded. Must be '
                              'a syntactically valid python expression. Variables are evaluated to '
@@ -323,7 +347,7 @@ def main():
 
     g_cond.add_argument('-c', '--count',
                         help='Do not attempt to download more than COUNT posts. '
-                             'Applies only to #hashtag and :feed.')
+                             'Applies to #hashtag, %%location_id, :feed, and :saved.')
 
     g_login = parser.add_argument_group('Login (Download Private Profiles)',
                                         'Instaloader can login to Instagram. This allows downloading private profiles. '
@@ -346,13 +370,21 @@ def main():
                             '{target} is replaced by the target you specified, i.e. either :feed, #hashtag or the '
                             'profile name. Defaults to \'{target}\'.')
     g_how.add_argument('--filename-pattern',
-                       help='Prefix of filenames, relative to the directory given with '
+                       help='Prefix of filenames for posts and stories, relative to the directory given with '
                             '--dirname-pattern. {profile} is replaced by the profile name,'
                             '{target} is replaced by the target you specified, i.e. either :feed'
                             '#hashtag or the profile name. Defaults to \'{date_utc}_UTC\'')
+    g_how.add_argument('--title-pattern',
+                       help='Prefix of filenames for profile pics, hashtag profile pics, and highlight covers. '
+                            'Defaults to \'{date_utc}_UTC_{typename}\' if --dirname-pattern contains \'{target}\' '
+                            'or \'{dirname}\', or if --dirname-pattern is not specified. Otherwise defaults to '
+                            '\'{target}_{date_utc}_UTC_{typename}\'.')
     g_how.add_argument('--resume-prefix', metavar='PREFIX',
                        help='Prefix for filenames that are used to save the information to resume an interrupted '
                             'download.')
+    g_how.add_argument('--sanitize-paths', action='store_true',
+                       help='Sanitize paths so that the resulting file and directory names are valid on both '
+                            'Windows and Unix.')
     g_how.add_argument('--no-resume', action='store_true',
                        help='Do not resume a previously-aborted download iteration, and do not save such information '
                             'when interrupted.')
@@ -365,8 +397,13 @@ def main():
                             'connection fails, it can be manually skipped by hitting CTRL+C. Set this to 0 to retry '
                             'infinitely.')
     g_how.add_argument('--commit-mode', action='store_true', help=SUPPRESS)
-    g_how.add_argument('--request-timeout', metavar='N', type=float,
+    g_how.add_argument('--request-timeout', metavar='N', type=float, default=300.0,
-                       help='seconds to wait before timing out a connection request')
+                       help='Seconds to wait before timing out a connection request. Defaults to 300.')
+    g_how.add_argument('--abort-on', type=http_status_code_list, metavar="STATUS_CODES",
+                       help='Comma-separated list of HTTP status codes that cause Instaloader to abort, bypassing all '
+                            'retry logic.')
+    g_how.add_argument('--no-iphone', action='store_true',
+                        help='Do not attempt to download iPhone version of images and videos.')
 
     g_misc = parser.add_argument_group('Miscellaneous Options')
     g_misc.add_argument('-q', '--quiet', action='store_true',
@@ -424,7 +461,12 @@ def main():
                              max_connection_attempts=args.max_connection_attempts,
                              request_timeout=args.request_timeout,
                              resume_prefix=resume_prefix,
-                             check_resume_bbd=not args.use_aged_resume_files)
+                             check_resume_bbd=not args.use_aged_resume_files,
+                             slide=args.slide,
+                             fatal_status_codes=args.abort_on,
+                             iphone_support=not args.no_iphone,
+                             title_pattern=args.title_pattern,
+                             sanitize_paths=args.sanitize_paths)
         _main(loader,
               args.profile,
               username=args.login.lower() if args.login is not None else None,
@@ -437,12 +479,13 @@ def main():
               download_tagged=args.tagged,
               download_igtv=args.igtv,
               fast_update=args.fast_update,
+              latest_stamps_file=args.latest_stamps,
               max_count=int(args.count) if args.count is not None else None,
               post_filter_str=args.post_filter,
               storyitem_filter_str=args.storyitem_filter)
         loader.close()
     except InstaloaderException as err:
-        raise SystemExit("Fatal error: %s" % err)
+        raise SystemExit("Fatal error: %s" % err) from err
 
 
 if __name__ == "__main__":

instaloader/exceptions.py

@@ -64,3 +64,17 @@ class QueryReturnedNotFoundException(ConnectionException):
 
 class TooManyRequestsException(ConnectionException):
     pass
+
+class IPhoneSupportDisabledException(InstaloaderException):
+    pass
+
+class AbortDownloadException(Exception):
+    """
+    Exception that is not catched in the error catchers inside the download loop and so aborts the
+    download loop.
+
+    This exception is not a subclass of ``InstaloaderException``.
+
+    .. versionadded:: 4.7
+    """
+    pass

instaloader/instaloadercontext.py

@@ -9,6 +9,7 @@ import sys
 import textwrap
 import time
 import urllib.parse
+import uuid
 from contextlib import contextmanager
 from datetime import datetime, timedelta
 from functools import partial
@@ -24,17 +25,44 @@ def copy_session(session: requests.Session, request_timeout: Optional[float] = N
     """Duplicates a requests.Session."""
     new = requests.Session()
     new.cookies = requests.utils.cookiejar_from_dict(requests.utils.dict_from_cookiejar(session.cookies))
-    new.headers = session.headers.copy()
+    new.headers = session.headers.copy()  # type: ignore
-    if request_timeout is not None:
+    # Override default timeout behavior.
-        # Override default timeout behavior.
+    # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
-        # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
+    new.request = partial(new.request, timeout=request_timeout)  # type: ignore
-        new.request = partial(new.request, timeout=request_timeout) # type: ignore
     return new
 
 
 def default_user_agent() -> str:
     return 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ' \
-           '(KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36'
+           '(KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36'
+
+
+def default_iphone_headers() -> Dict[str, Any]:
+    return {'User-Agent': 'Instagram 273.0.0.16.70 (iPad13,8; iOS 16_3; en_US; en-US; ' \
+                          'scale=2.00; 2048x2732; 452417278) AppleWebKit/420+',
+            'x-ads-opt-out': '1',
+            'x-bloks-is-panorama-enabled': 'true',
+            'x-bloks-version-id': '01507c21540f73e2216b6f62a11a5b5e51aa85491b72475c080da35b1228ddd6',
+            'x-fb-client-ip': 'True',
+            'x-fb-connection-type': 'wifi',
+            'x-fb-http-engine': 'Liger',
+            'x-fb-server-cluster': 'True',
+            'x-fb': '1',
+            'x-ig-abr-connection-speed-kbps': '2',
+            'x-ig-app-id': '124024574287414',
+            'x-ig-app-locale': 'en-US',
+            'x-ig-app-startup-country': 'US',
+            'x-ig-bandwidth-speed-kbps': '0.000',
+            'x-ig-capabilities': '36r/F/8=',
+            'x-ig-connection-speed': '{}kbps'.format(random.randint(1000, 20000)),
+            'x-ig-connection-type': 'WiFi',
+            'x-ig-device-locale': 'en-US',
+            'x-ig-mapped-locale': 'en-US',
+            'x-ig-timezone-offset': str((datetime.now().astimezone().utcoffset() or timedelta(seconds=0)).seconds),
+            'x-ig-www-claim': '0',
+            'x-pigeon-session-id': str(uuid.uuid4()),
+            'x-tigon-is-retry': 'False',
+            'x-whatsapp': '0'}
 
 
 class InstaloaderContext:
@@ -53,43 +81,57 @@ class InstaloaderContext:
     """
 
     def __init__(self, sleep: bool = True, quiet: bool = False, user_agent: Optional[str] = None,
-                 max_connection_attempts: int = 3, request_timeout: Optional[float] = None,
+                 max_connection_attempts: int = 3, request_timeout: float = 300.0,
-                 rate_controller: Optional[Callable[["InstaloaderContext"], "RateController"]] = None):
+                 rate_controller: Optional[Callable[["InstaloaderContext"], "RateController"]] = None,
+                 fatal_status_codes: Optional[List[int]] = None,
+                 iphone_support: bool = True):
 
         self.user_agent = user_agent if user_agent is not None else default_user_agent()
         self.request_timeout = request_timeout
         self._session = self.get_anonymous_session()
         self.username = None
+        self.user_id = None
         self.sleep = sleep
         self.quiet = quiet
         self.max_connection_attempts = max_connection_attempts
         self._graphql_page_length = 50
         self._root_rhx_gis = None
         self.two_factor_auth_pending = None
+        self.iphone_support = iphone_support
+        self.iphone_headers = default_iphone_headers()
 
         # error log, filled with error() and printed at the end of Instaloader.main()
-        self.error_log = []                      # type: List[str]
+        self.error_log: List[str] = []
 
         self._rate_controller = rate_controller(self) if rate_controller is not None else RateController(self)
 
         # Can be set to True for testing, disables supression of InstaloaderContext._error_catcher
         self.raise_all_errors = False
 
+        # HTTP status codes that should cause an AbortDownloadException
+        self.fatal_status_codes = fatal_status_codes or []
+
         # Cache profile from id (mapping from id to Profile)
-        self.profile_id_cache = dict()           # type: Dict[int, Any]
+        self.profile_id_cache: Dict[int, Any] = dict()
 
     @contextmanager
     def anonymous_copy(self):
         session = self._session
         username = self.username
+        user_id = self.user_id
+        iphone_headers = self.iphone_headers
         self._session = self.get_anonymous_session()
         self.username = None
+        self.user_id = None
+        self.iphone_headers = default_iphone_headers()
         try:
             yield self
         finally:
             self._session.close()
             self.username = username
             self._session = session
+            self.user_id = user_id
+            self.iphone_headers = iphone_headers
 
     @property
     def is_logged_in(self) -> bool:
@@ -149,7 +191,6 @@ class InstaloaderContext:
         if empty_session_only:
             del header['Host']
             del header['Origin']
-            del header['Referer']
             del header['X-Instagram-AJAX']
             del header['X-Requested-With']
         return header
@@ -161,28 +202,34 @@ class InstaloaderContext:
                                 'ig_vw': '1920', 'csrftoken': '',
                                 's_network': '', 'ds_user_id': ''})
         session.headers.update(self._default_http_header(empty_session_only=True))
-        if self.request_timeout is not None:
+        # Override default timeout behavior.
-            # Override default timeout behavior.
+        # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
-            # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
+        session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
-            session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
         return session
 
+    def save_session(self):
+        """Not meant to be used directly, use :meth:`Instaloader.save_session`."""
+        return requests.utils.dict_from_cookiejar(self._session.cookies)
+
+    def load_session(self, username, sessiondata):
+        """Not meant to be used directly, use :meth:`Instaloader.load_session`."""
+        session = requests.Session()
+        session.cookies = requests.utils.cookiejar_from_dict(sessiondata)
+        session.headers.update(self._default_http_header())
+        session.headers.update({'X-CSRFToken': session.cookies.get_dict()['csrftoken']})
+        # Override default timeout behavior.
+        # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
+        session.request = partial(session.request, timeout=self.request_timeout)  # type: ignore
+        self._session = session
+        self.username = username
+
     def save_session_to_file(self, sessionfile):
         """Not meant to be used directly, use :meth:`Instaloader.save_session_to_file`."""
-        pickle.dump(requests.utils.dict_from_cookiejar(self._session.cookies), sessionfile)
+        pickle.dump(self.save_session(), sessionfile)
 
     def load_session_from_file(self, username, sessionfile):
         """Not meant to be used directly, use :meth:`Instaloader.load_session_from_file`."""
-        session = requests.Session()
+        self.load_session(username, pickle.load(sessionfile))
-        session.cookies = requests.utils.cookiejar_from_dict(pickle.load(sessionfile))
-        session.headers.update(self._default_http_header())
-        session.headers.update({'X-CSRFToken': session.cookies.get_dict()['csrftoken']})
-        if self.request_timeout is not None:
-            # Override default timeout behavior.
-            # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
-            session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
-        self._session = session
-        self.username = username
 
     def test_login(self) -> Optional[str]:
         """Not meant to be used directly, use :meth:`Instaloader.test_login`."""
@@ -206,12 +253,13 @@ class InstaloaderContext:
                                 'ig_vw': '1920', 'ig_cb': '1', 'csrftoken': '',
                                 's_network': '', 'ds_user_id': ''})
         session.headers.update(self._default_http_header())
-        if self.request_timeout is not None:
+        # Getting mid cookie value
-            # Override default timeout behavior.
+        session.get("https://i.instagram.com/api/v1/web/login_page/")
-            # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
+        # Override default timeout behavior.
-            session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
+        # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
-        session.get('https://www.instagram.com/web/__mid/')
+        session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
-        csrf_token = session.cookies.get_dict()['csrftoken']
+        csrf_json = self.get_json('accounts/login/', {}, session=session)
+        csrf_token = csrf_json['config']['csrf_token']
         session.headers.update({'X-CSRFToken': csrf_token})
         # Not using self.get_json() here, because we need to access csrftoken cookie
         self.do_sleep()
@@ -222,8 +270,10 @@ class InstaloaderContext:
                              data={'enc_password': enc_password, 'username': user}, allow_redirects=True)
         try:
             resp_json = login.json()
-        except json.decoder.JSONDecodeError:
+        except json.decoder.JSONDecodeError as err:
-            raise ConnectionException("Login error: JSON decode fail, {} - {}.".format(login.status_code, login.reason))
+            raise ConnectionException(
+                "Login error: JSON decode fail, {} - {}.".format(login.status_code, login.reason)
+            ) from err
         if resp_json.get('two_factor_required'):
             two_factor_session = copy_session(session, self.request_timeout)
             two_factor_session.headers.update({'X-CSRFToken': csrf_token})
@@ -259,9 +309,11 @@ class InstaloaderContext:
                 # username is invalid.
                 raise InvalidArgumentException('Login error: User {} does not exist.'.format(user))
         # '{"authenticated": true, "user": true, "userId": ..., "oneTapPrompt": false, "status": "ok"}'
-        session.headers.update({'X-CSRFToken': login.cookies['csrftoken']})
+        if 'csrftoken' in login.cookies:
+            session.headers.update({'X-CSRFToken': login.cookies['csrftoken']})
         self._session = session
         self.username = user
+        self.user_id = resp_json['userId']
 
     def two_factor_login(self, two_factor_code):
         """Second step of login if 2FA is enabled.
@@ -275,16 +327,24 @@ class InstaloaderContext:
             raise InvalidArgumentException("No two-factor authentication pending.")
         (session, user, two_factor_id) = self.two_factor_auth_pending
 
-        login = session.post('https://www.instagram.com/accounts/login/ajax/two_factor/',
+        login = session.post('https://www.instagram.com/api/v1/web/accounts/login/ajax/two_factor/',
-                             data={'username': user, 'verificationCode': two_factor_code, 'identifier': two_factor_id},
+                             data={
+                                 'identifier': two_factor_id,
+                                 'queryParams': '{"next":"/"}',
+                                 'trust_signal': 'true',
+                                 'username': user,
+                                 'verification_method': '1',
+                                 'verificationCode': two_factor_code,
+                             },
                              allow_redirects=True)
         resp_json = login.json()
         if resp_json['status'] != 'ok':
             if 'message' in resp_json:
-                raise BadCredentialsException("Login error: {}".format(resp_json['message']))
+                raise BadCredentialsException("2FA error: {}".format(resp_json['message']))
             else:
-                raise BadCredentialsException("Login error: \"{}\" status.".format(resp_json['status']))
+                raise BadCredentialsException("2FA error: \"{}\" status.".format(resp_json['status']))
-        session.headers.update({'X-CSRFToken': login.cookies['csrftoken']})
+        if 'csrftoken' in login.cookies:
+            session.headers.update({'X-CSRFToken': login.cookies['csrftoken']})
         self._session = session
         self.username = user
         self.two_factor_auth_pending = None
@@ -292,10 +352,11 @@ class InstaloaderContext:
     def do_sleep(self):
         """Sleep a short time if self.sleep is set. Called before each request to instagram.com."""
         if self.sleep:
-            time.sleep(min(random.expovariate(0.7), 5.0))
+            time.sleep(min(random.expovariate(0.6), 15.0))
 
     def get_json(self, path: str, params: Dict[str, Any], host: str = 'www.instagram.com',
-                 session: Optional[requests.Session] = None, _attempt=1) -> Dict[str, Any]:
+                 session: Optional[requests.Session] = None, _attempt=1,
+                 response_headers: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
         """JSON request to Instagram.
 
         :param path: URL, relative to the given domain which defaults to www.instagram.com/
@@ -320,19 +381,31 @@ class InstaloaderContext:
             if is_other_query:
                 self._rate_controller.wait_before_query('other')
             resp = sess.get('https://{0}/{1}'.format(host, path), params=params, allow_redirects=False)
+            if resp.status_code in self.fatal_status_codes:
+                redirect = " redirect to {}".format(resp.headers['location']) if 'location' in resp.headers else ""
+                body = ""
+                if resp.headers['Content-Type'].startswith('application/json'):
+                    body = ': ' + resp.text[:500] + ('…' if len(resp.text) > 501 else '')
+                raise AbortDownloadException("Query to https://{}/{} responded with \"{} {}\"{}{}".format(
+                    host, path, resp.status_code, resp.reason, redirect, body
+                ))
             while resp.is_redirect:
                 redirect_url = resp.headers['location']
                 self.log('\nHTTP redirect from https://{0}/{1} to {2}'.format(host, path, redirect_url))
-                if redirect_url.startswith('https://www.instagram.com/accounts/login'):
+                if (redirect_url.startswith('https://www.instagram.com/accounts/login') or
+                    redirect_url.startswith('https://i.instagram.com/accounts/login')):
                     if not self.is_logged_in:
                         raise LoginRequiredException("Redirected to login page. Use --login.")
-                    # alternate rate limit exceeded behavior
+                    raise AbortDownloadException("Redirected to login page. You've been logged out, please wait " +
-                    raise TooManyRequestsException("Redirected to login")
+                                                 "some time, recreate the session and try again")
                 if redirect_url.startswith('https://{}/'.format(host)):
                     resp = sess.get(redirect_url if redirect_url.endswith('/') else redirect_url + '/',
                                     params=params, allow_redirects=False)
                 else:
                     break
+            if response_headers is not None:
+                response_headers.clear()
+                response_headers.update(resp.headers)
             if resp.status_code == 400:
                 raise QueryReturnedBadRequestException("400 Bad Request")
             if resp.status_code == 404:
@@ -343,20 +416,13 @@ class InstaloaderContext:
                 raise ConnectionException("HTTP error code {}.".format(resp.status_code))
             is_html_query = not is_graphql_query and not "__a" in params and host == "www.instagram.com"
             if is_html_query:
-                match = re.search(r'window\._sharedData = (.*);</script>', resp.text)
+                # Extract JSON from HTML response
+                match = re.search('(?<={"raw":").*?(?<!\\\\)(?=")', resp.text)
                 if match is None:
-                    raise QueryReturnedNotFoundException("Could not find \"window._sharedData\" in html response.")
+                    raise QueryReturnedNotFoundException("Could not find JSON data in html response.")
-                resp_json = json.loads(match.group(1))
+                # Unescape escaped JSON string
-                entry_data = resp_json.get('entry_data')
+                unescaped_string = match.group(0).encode("utf-8").decode("unicode_escape")
-                post_or_profile_page = list(entry_data.values())[0] if entry_data is not None else None
+                resp_json = json.loads(unescaped_string)
-                if post_or_profile_page is None:
-                    raise ConnectionException("\"window._sharedData\" does not contain required keys.")
-                # If GraphQL data is missing in `window._sharedData`, search for it in `__additionalDataLoaded`.
-                if 'graphql' not in post_or_profile_page[0]:
-                    match = re.search(r'window\.__additionalDataLoaded\(.*?({.*"graphql":.*})\);</script>',
-                                      resp.text)
-                    if match is not None:
-                        post_or_profile_page[0]['graphql'] = json.loads(match.group(1))['graphql']
                 return resp_json
             else:
                 resp_json = resp.json()
@@ -383,7 +449,8 @@ class InstaloaderContext:
                         self._rate_controller.handle_429('iphone')
                     if is_other_query:
                         self._rate_controller.handle_429('other')
-                return self.get_json(path=path, params=params, host=host, session=sess, _attempt=_attempt + 1)
+                return self.get_json(path=path, params=params, host=host, session=sess, _attempt=_attempt + 1,
+                                     response_headers=response_headers)
             except KeyboardInterrupt:
                 self.error("[skipped by user]", repeat_at_end=False)
                 raise ConnectionException(error_string) from err
@@ -473,11 +540,57 @@ class InstaloaderContext:
 
         .. versionadded:: 4.2.1"""
         with copy_session(self._session, self.request_timeout) as tempsession:
-            tempsession.headers['User-Agent'] = 'Instagram 123.1.0.26.115 (iPhone12,1; iOS 13_3; en_US; en-US; ' \
+            # Set headers to simulate an API request from iPad
-                                                'scale=2.00; 1656x3584; 190542906)'
+            tempsession.headers['ig-intended-user-id'] = str(self.user_id)
-            for header in ['Host', 'Origin', 'X-Instagram-AJAX', 'X-Requested-With']:
+            tempsession.headers['x-pigeon-rawclienttime'] = '{:.6f}'.format(time.time())
+
+            # Add headers obtained from previous iPad request
+            tempsession.headers.update(self.iphone_headers)
+
+            # Extract key information from cookies if we haven't got it already from a previous request
+            header_cookies_mapping = {'x-mid': 'mid',
+                                     'ig-u-ds-user-id': 'ds_user_id',
+                                     'x-ig-device-id': 'ig_did',
+                                     'x-ig-family-device-id': 'ig_did',
+                                     'family_device_id': 'ig_did'}
+
+            # Map the cookie value to the matching HTTP request header
+            cookies = tempsession.cookies.get_dict().copy()
+            for key, value in header_cookies_mapping.items():
+                if value in cookies:
+                    if key not in tempsession.headers:
+                        tempsession.headers[key] = cookies[value]
+                    else:
+                        # Remove the cookie value if it's already specified as a header
+                        tempsession.cookies.pop(value, None)
+
+            # Edge case for ig-u-rur header due to special string encoding in cookie
+            if 'rur' in cookies:
+                if 'ig-u-rur' not in tempsession.headers:
+                    tempsession.headers['ig-u-rur'] = cookies['rur'].strip('\"').encode('utf-8') \
+                                                                    .decode('unicode_escape')
+                else:
+                    tempsession.cookies.pop('rur', None)
+
+            # Remove headers specific to Desktop version
+            for header in ['Host', 'Origin', 'X-Instagram-AJAX', 'X-Requested-With', 'Referer']:
                 tempsession.headers.pop(header, None)
-            return self.get_json(path, params, 'i.instagram.com', tempsession)
+
+            # No need for cookies if we have a bearer token
+            if 'authorization' in tempsession.headers:
+                tempsession.cookies.clear()
+
+            response_headers = dict()    # type: Dict[str, Any]
+            response = self.get_json(path, params, 'i.instagram.com', tempsession, response_headers=response_headers)
+
+            # Extract the ig-set-* headers and use them in the next request
+            for key, value in response_headers.items():
+                if key.startswith('ig-set-'):
+                    self.iphone_headers[key.replace('ig-set-', '')] = value
+                elif key.startswith('x-ig-set-'):
+                    self.iphone_headers[key.replace('x-ig-set-', 'x-ig-')] = value
+
+            return response
 
     def write_raw(self, resp: Union[bytes, requests.Response], filename: str) -> None:
         """Write raw response data into a file.
@@ -521,6 +634,28 @@ class InstaloaderContext:
         :raises ConnectionException: When download repeatedly failed."""
         self.write_raw(self.get_raw(url), filename)
 
+    def head(self, url: str, allow_redirects: bool = False) -> requests.Response:
+        """HEAD a URL anonymously.
+
+        :raises QueryReturnedNotFoundException: When the server responds with a 404.
+        :raises QueryReturnedForbiddenException: When the server responds with a 403.
+        :raises ConnectionException: When request failed.
+
+        .. versionadded:: 4.7.6
+        """
+        with self.get_anonymous_session() as anonymous_session:
+            resp = anonymous_session.head(url, allow_redirects=allow_redirects)
+        if resp.status_code == 200:
+            return resp
+        else:
+            if resp.status_code == 403:
+                # suspected invalid URL signature
+                raise QueryReturnedForbiddenException("403 when accessing {}.".format(url))
+            if resp.status_code == 404:
+                # 404 not worth retrying.
+                raise QueryReturnedNotFoundException("404 when accessing {}.".format(url))
+            raise ConnectionException("HTTP error code {}.".format(resp.status_code))
+
     @property
     def root_rhx_gis(self) -> Optional[str]:
         """rhx_gis string returned in the / query."""
@@ -551,19 +686,19 @@ class RateController:
 
     def __init__(self, context: InstaloaderContext):
         self._context = context
-        self._query_timestamps = dict()  # type: Dict[str, List[float]]
+        self._query_timestamps: Dict[str, List[float]] = dict()
         self._earliest_next_request_time = 0.0
+        self._iphone_earliest_next_request_time = 0.0
 
     def sleep(self, secs: float):
         """Wait given number of seconds."""
         # Not static, to allow for the behavior of this method to depend on context-inherent properties, such as
         # whether we are logged in.
-        # pylint:disable=no-self-use
         time.sleep(secs)
 
     def _dump_query_timestamps(self, current_time: float, failed_query_type: str):
         windows = [10, 11, 20, 22, 30, 60]
-        self._context.error("Requests within last {} minutes grouped by type:"
+        self._context.error("Number of requests within last {} minutes grouped by type:"
                             .format('/'.join(str(w) for w in windows)),
                             repeat_at_end=False)
         for query_type, times in self._query_timestamps.items():
@@ -575,11 +710,14 @@ class RateController:
             ), repeat_at_end=False)
 
     def count_per_sliding_window(self, query_type: str) -> int:
-        """Return how many requests can be done within the sliding window."""
+        """Return how many requests of the given type can be done within a sliding window of 11 minutes.
+
+        This is called by :meth:`RateController.query_waittime` and allows to simply customize wait times before queries
+        at query_type granularity. Consider overriding :meth:`RateController.query_waittime` directly if you need more
+        control."""
         # Not static, to allow for the count_per_sliding_window to depend on context-inherent properties, such as
         # whether we are logged in.
-        # pylint:disable=no-self-use
+        return 75 if query_type == 'other' else 200
-        return 75 if query_type in ['iphone', 'other'] else 200
 
     def _reqs_in_sliding_window(self, query_type: Optional[str], current_time: float, window: float) -> List[float]:
         if query_type is not None:
@@ -595,6 +733,7 @@ class RateController:
     def query_waittime(self, query_type: str, current_time: float, untracked_queries: bool = False) -> float:
         """Calculate time needed to wait before query can be executed."""
         per_type_sliding_window = 660
+        iphone_sliding_window = 1800
         if query_type not in self._query_timestamps:
             self._query_timestamps[query_type] = []
         self._query_timestamps[query_type] = list(filter(lambda t: t > current_time - 60 * 60,
@@ -620,25 +759,43 @@ class RateController:
 
         def untracked_next_request_time():
             if untracked_queries:
-                reqs_in_sliding_window = self._reqs_in_sliding_window(query_type, current_time, per_type_sliding_window)
+                if query_type == "iphone":
-                self._earliest_next_request_time = min(reqs_in_sliding_window) + per_type_sliding_window + 6
+                    reqs_in_sliding_window = self._reqs_in_sliding_window(query_type, current_time,
-            return self._earliest_next_request_time
+                                                                          iphone_sliding_window)
+                    self._iphone_earliest_next_request_time = min(reqs_in_sliding_window) + iphone_sliding_window + 18
+                else:
+                    reqs_in_sliding_window = self._reqs_in_sliding_window(query_type, current_time,
+                                                                          per_type_sliding_window)
+                    self._earliest_next_request_time = min(reqs_in_sliding_window) + per_type_sliding_window + 6
+            return max(self._iphone_earliest_next_request_time, self._earliest_next_request_time)
+
+        def iphone_next_request():
+            if query_type == "iphone":
+                reqs_in_sliding_window = self._reqs_in_sliding_window(query_type, current_time, iphone_sliding_window)
+                if len(reqs_in_sliding_window) >= 199:
+                    return min(reqs_in_sliding_window) + iphone_sliding_window + 18
+            return 0.0
 
         return max(0.0,
                    max(
                        per_type_next_request_time(),
                        gql_accumulated_next_request_time(),
                        untracked_next_request_time(),
+                       iphone_next_request(),
                    ) - current_time)
 
     def wait_before_query(self, query_type: str) -> None:
-        """This method is called before a query to Instagram. It calls :meth:`RateController.sleep` to wait
+        """This method is called before a query to Instagram.
-        until the request can be made."""
+
+        It calls :meth:`RateController.query_waittime` to determine the time needed to wait and then calls
+        :meth:`RateController.sleep` to wait until the request can be made."""
         waittime = self.query_waittime(query_type, time.monotonic(), False)
         assert waittime >= 0
         if waittime > 15:
-            self._context.log("\nToo many queries in the last time. Need to wait {} seconds, until {:%H:%M}."
+            formatted_waittime = ("{} seconds".format(round(waittime)) if waittime <= 666 else
-                              .format(round(waittime), datetime.now() + timedelta(seconds=waittime)))
+                                  "{} minutes".format(round(waittime / 60)))
+            self._context.log("\nToo many queries in the last time. Need to wait {}, until {:%H:%M}."
+                              .format(formatted_waittime, datetime.now() + timedelta(seconds=waittime)))
         if waittime > 0:
             self.sleep(waittime)
         if query_type not in self._query_timestamps:
@@ -647,8 +804,10 @@ class RateController:
             self._query_timestamps[query_type].append(time.monotonic())
 
     def handle_429(self, query_type: str) -> None:
-        """This method is called to handle a 429 Too Many Requests response. It calls :meth:`RateController.sleep` to
+        """This method is called to handle a 429 Too Many Requests response.
-         wait until we can repeat the same request."""
+
+        It calls :meth:`RateController.query_waittime` to determine the time needed to wait and then calls
+        :meth:`RateController.sleep` to wait until we can repeat the same request."""
         current_time = time.monotonic()
         waittime = self.query_waittime(query_type, current_time, True)
         assert waittime >= 0
@@ -658,8 +817,10 @@ class RateController:
                         "App while Instaloader is running.")
         self._context.error(textwrap.fill(text_for_429), repeat_at_end=False)
         if waittime > 1.5:
-            self._context.error("The request will be retried in {} seconds, at {:%H:%M}."
+            formatted_waittime = ("{} seconds".format(round(waittime)) if waittime <= 666 else
-                                .format(round(waittime), datetime.now() + timedelta(seconds=waittime)),
+                                  "{} minutes".format(round(waittime / 60)))
+            self._context.error("The request will be retried in {}, at {:%H:%M}."
+                                .format(formatted_waittime, datetime.now() + timedelta(seconds=waittime)),
                                 repeat_at_end=False)
         if waittime > 0:
             self.sleep(waittime)

instaloader/lateststamps.py

@@ -0,0 +1,117 @@
+import configparser
+from datetime import datetime, timezone
+from typing import Optional
+from os.path import dirname
+from os import makedirs
+
+
+class LatestStamps:
+    """LatestStamps class.
+
+    Convenience class for retrieving and storing data from the :option:`--latest-stamps` file.
+
+    :param latest_stamps_file: path to file.
+
+    .. versionadded:: 4.8"""
+    PROFILE_ID = 'profile-id'
+    PROFILE_PIC = 'profile-pic'
+    POST_TIMESTAMP = 'post-timestamp'
+    TAGGED_TIMESTAMP = 'tagged-timestamp'
+    IGTV_TIMESTAMP = 'igtv-timestamp'
+    STORY_TIMESTAMP = 'story-timestamp'
+    ISO_FORMAT = '%Y-%m-%dT%H:%M:%S.%f%z'
+
+    def __init__(self, latest_stamps_file):
+        self.file = latest_stamps_file
+        self.data = configparser.ConfigParser()
+        self.data.read(latest_stamps_file)
+
+    def _save(self):
+        if dn := dirname(self.file):
+            makedirs(dn, exist_ok=True)
+        with open(self.file, 'w') as f:
+            self.data.write(f)
+
+    def _ensure_section(self, section: str):
+        if not self.data.has_section(section):
+            self.data.add_section(section)
+
+    def get_profile_id(self, profile_name: str) -> Optional[int]:
+        """Returns stored ID of profile."""
+        try:
+            return self.data.getint(profile_name, self.PROFILE_ID)
+        except (configparser.Error, ValueError):
+            return None
+
+    def save_profile_id(self, profile_name: str, profile_id: int):
+        """Stores ID of profile."""
+        self._ensure_section(profile_name)
+        self.data.set(profile_name, self.PROFILE_ID, str(profile_id))
+        self._save()
+
+    def rename_profile(self, old_profile: str, new_profile: str):
+        """Renames a profile."""
+        self._ensure_section(new_profile)
+        for option in [self.PROFILE_ID, self.PROFILE_PIC, self.POST_TIMESTAMP,
+                       self.TAGGED_TIMESTAMP, self.IGTV_TIMESTAMP, self.STORY_TIMESTAMP]:
+            if self.data.has_option(old_profile, option):
+                value = self.data.get(old_profile, option)
+                self.data.set(new_profile, option, value)
+        self.data.remove_section(old_profile)
+        self._save()
+
+    def _get_timestamp(self, section: str, key: str) -> datetime:
+        try:
+            return datetime.strptime(self.data.get(section, key), self.ISO_FORMAT)
+        except (configparser.Error, ValueError):
+            return datetime.fromtimestamp(0, timezone.utc)
+
+    def _set_timestamp(self, section: str, key: str, timestamp: datetime):
+        self._ensure_section(section)
+        self.data.set(section, key, timestamp.strftime(self.ISO_FORMAT))
+        self._save()
+
+    def get_last_post_timestamp(self, profile_name: str) -> datetime:
+        """Returns timestamp of last download of a profile's posts."""
+        return self._get_timestamp(profile_name, self.POST_TIMESTAMP)
+
+    def set_last_post_timestamp(self, profile_name: str, timestamp: datetime):
+        """Sets timestamp of last download of a profile's posts."""
+        self._set_timestamp(profile_name, self.POST_TIMESTAMP, timestamp)
+
+    def get_last_tagged_timestamp(self, profile_name: str) -> datetime:
+        """Returns timestamp of last download of a profile's tagged posts."""
+        return self._get_timestamp(profile_name, self.TAGGED_TIMESTAMP)
+
+    def set_last_tagged_timestamp(self, profile_name: str, timestamp: datetime):
+        """Sets timestamp of last download of a profile's tagged posts."""
+        self._set_timestamp(profile_name, self.TAGGED_TIMESTAMP, timestamp)
+
+    def get_last_igtv_timestamp(self, profile_name: str) -> datetime:
+        """Returns timestamp of last download of a profile's igtv posts."""
+        return self._get_timestamp(profile_name, self.IGTV_TIMESTAMP)
+
+    def set_last_igtv_timestamp(self, profile_name: str, timestamp: datetime):
+        """Sets timestamp of last download of a profile's igtv posts."""
+        self._set_timestamp(profile_name, self.IGTV_TIMESTAMP, timestamp)
+
+    def get_last_story_timestamp(self, profile_name: str) -> datetime:
+        """Returns timestamp of last download of a profile's stories."""
+        return self._get_timestamp(profile_name, self.STORY_TIMESTAMP)
+
+    def set_last_story_timestamp(self, profile_name: str, timestamp: datetime):
+        """Sets timestamp of last download of a profile's stories."""
+        self._set_timestamp(profile_name, self.STORY_TIMESTAMP, timestamp)
+
+    def get_profile_pic(self, profile_name: str) -> str:
+        """Returns filename of profile's last downloaded profile pic."""
+        try:
+            return self.data.get(profile_name, self.PROFILE_PIC)
+        except configparser.Error:
+            return ""
+
+    def set_profile_pic(self, profile_name: str, profile_pic: str):
+        """Sets filename of profile's last downloaded profile pic."""
+        self._ensure_section(profile_name)
+        self.data.set(profile_name, self.PROFILE_PIC, profile_pic)
+        self._save()

instaloader/nodeiterator.py

@@ -5,21 +5,20 @@ import os
 from contextlib import contextmanager
 from datetime import datetime, timedelta
 from lzma import LZMAError
-from typing import Any, Callable, Dict, Iterator, NamedTuple, Optional, Tuple, TypeVar
+from typing import Any, Callable, Dict, Iterable, Iterator, NamedTuple, Optional, Tuple, TypeVar
 
-from .exceptions import InvalidArgumentException, QueryReturnedBadRequestException
+from .exceptions import AbortDownloadException, InvalidArgumentException, QueryReturnedBadRequestException
 from .instaloadercontext import InstaloaderContext
 
-FrozenNodeIterator = NamedTuple('FrozenNodeIterator',
+class FrozenNodeIterator(NamedTuple):
-                                [('query_hash', str),
+    query_hash: str
-                                 ('query_variables', Dict),
+    query_variables: Dict
-                                 ('query_referer', Optional[str]),
+    query_referer: Optional[str]
-                                 ('context_username', Optional[str]),
+    context_username: Optional[str]
-                                 ('total_index', int),
+    total_index: int
-                                 ('best_before', Optional[float]),
+    best_before: Optional[float]
-                                 ('remaining_data', Optional[Dict])])
+    remaining_data: Optional[Dict]
-FrozenNodeIterator.__doc__ = \
+    first_node: Optional[Dict]
-    """A serializable representation of a :class:`NodeIterator` instance, saving its iteration state."""
 FrozenNodeIterator.query_hash.__doc__ = """The GraphQL ``query_hash`` parameter."""
 FrozenNodeIterator.query_variables.__doc__ = """The GraphQL ``query_variables`` parameter."""
 FrozenNodeIterator.query_referer.__doc__ = """The HTTP referer used for the GraphQL query."""
@@ -28,7 +27,7 @@ FrozenNodeIterator.total_index.__doc__ = """Number of items that have already be
 FrozenNodeIterator.best_before.__doc__ = """Date when parts of the stored nodes might have expired."""
 FrozenNodeIterator.remaining_data.__doc__ = \
     """The already-retrieved, yet-unprocessed ``edges`` and the ``page_info`` at time of freezing."""
-
+FrozenNodeIterator.first_node.__doc__ = """Node data of the first item, if an item has been produced."""
 
 T = TypeVar('T')
 
@@ -55,6 +54,9 @@ class NodeIterator(Iterator[T]):
        post_iterator = profile.get_posts()
        post_iterator.thaw(load("resume_information.json"))
 
+    (an appropriate method to load and save the :class:`FrozenNodeIterator` is e.g.
+    :func:`load_structure_from_file` and :func:`save_structure_to_file`.)
+
     A :class:`FrozenNodeIterator` can only be thawn with a matching NodeIterator, i.e. a NodeIterator instance that has
     been constructed with the same parameters as the instance that is represented by the :class:`FrozenNodeIterator` in
     question. This is to ensure that an iteration cannot be resumed in a wrong, unmatching loop. As a quick way to
@@ -74,7 +76,8 @@ class NodeIterator(Iterator[T]):
                  node_wrapper: Callable[[Dict], T],
                  query_variables: Optional[Dict[str, Any]] = None,
                  query_referer: Optional[str] = None,
-                 first_data: Optional[Dict[str, Any]] = None):
+                 first_data: Optional[Dict[str, Any]] = None,
+                 is_first: Optional[Callable[[T, Optional[T]], bool]] = None):
         self._context = context
         self._query_hash = query_hash
         self._edge_extractor = edge_extractor
@@ -88,9 +91,11 @@ class NodeIterator(Iterator[T]):
             self._best_before = datetime.now() + NodeIterator._shelf_life
         else:
             self._data = self._query()
+        self._first_node: Optional[Dict] = None
+        self._is_first = is_first
 
     def _query(self, after: Optional[str] = None) -> Dict:
-        pagination_variables = {'first': NodeIterator._graphql_page_length}  # type: Dict[str, Any]
+        pagination_variables: Dict[str, Any] = {'first': NodeIterator._graphql_page_length}
         if after is not None:
             pagination_variables['after'] = after
         try:
@@ -124,17 +129,25 @@ class NodeIterator(Iterator[T]):
             except KeyboardInterrupt:
                 self._page_index, self._total_index = page_index, total_index
                 raise
-            return self._node_wrapper(node)
+            item = self._node_wrapper(node)
-        if self._data['page_info']['has_next_page']:
+            if self._is_first is not None:
+                if self._is_first(item, self.first_item):
+                    self._first_node = node
+            else:
+                if self._first_node is None:
+                    self._first_node = node
+            return item
+        if self._data.get('page_info', {}).get('has_next_page'):
             query_response = self._query(self._data['page_info']['end_cursor'])
-            page_index, data = self._page_index, self._data
+            if self._data['edges'] != query_response['edges'] and len(query_response['edges']) > 0:
-            try:
+                page_index, data = self._page_index, self._data
-                self._page_index = 0
+                try:
-                self._data = query_response
+                    self._page_index = 0
-            except KeyboardInterrupt:
+                    self._data = query_response
-                self._page_index, self._data = page_index, data
+                except KeyboardInterrupt:
-                raise
+                    self._page_index, self._data = page_index, data
-            return self.__next__()
+                    raise
+                return self.__next__()
         raise StopIteration()
 
     @property
@@ -150,16 +163,27 @@ class NodeIterator(Iterator[T]):
     @property
     def magic(self) -> str:
         """Magic string for easily identifying a matching iterator file for resuming (hash of some parameters)."""
-        if 'blake2b' not in hashlib.algorithms_available:
+        magic_hash = hashlib.blake2b(digest_size=6)
-            magic_hash = hashlib.new('sha224')
-        else:
-            # Use blake2b when possible, i.e. on Python >= 3.6.
-            magic_hash = hashlib.blake2b(digest_size=6)  # type:ignore  # pylint: disable=no-member
         magic_hash.update(json.dumps(
             [self._query_hash, self._query_variables, self._query_referer, self._context.username]
         ).encode())
         return base64.urlsafe_b64encode(magic_hash.digest()).decode()
 
+    @property
+    def first_item(self) -> Optional[T]:
+        """
+        If this iterator has produced any items, returns the first item produced.
+
+        It is possible to override what is considered the first item (for example, to consider the
+        newest item in case items are not in strict chronological order) by passing a callback
+        function as the `is_first` parameter when creating the class.
+
+        .. versionadded:: 4.8
+        .. versionchanged:: 4.9.2
+           What is considered the first item can be overridden.
+        """
+        return self._node_wrapper(self._first_node) if self._first_node is not None else None
+
     def freeze(self) -> FrozenNodeIterator:
         """Freeze the iterator for later resuming."""
         remaining_data = None
@@ -174,6 +198,7 @@ class NodeIterator(Iterator[T]):
             total_index=max(self.total_index - 1, 0),
             best_before=self._best_before.timestamp() if self._best_before else None,
             remaining_data=remaining_data,
+            first_node=self._first_node,
         )
 
     def thaw(self, frozen: FrozenNodeIterator) -> None:
@@ -200,18 +225,21 @@ class NodeIterator(Iterator[T]):
         self._total_index = frozen.total_index
         self._best_before = datetime.fromtimestamp(frozen.best_before)
         self._data = frozen.remaining_data
+        if frozen.first_node is not None:
+            self._first_node = frozen.first_node
 
 
 @contextmanager
 def resumable_iteration(context: InstaloaderContext,
-                        iterator: Iterator,
+                        iterator: Iterable,
                         load: Callable[[InstaloaderContext, str], Any],
                         save: Callable[[FrozenNodeIterator, str], None],
                         format_path: Callable[[str], str],
                         check_bbd: bool = True,
                         enabled: bool = True) -> Iterator[Tuple[bool, int]]:
     """
-    High-level context manager to handle a resumable iteration that can be interrupted with a KeyboardInterrupt.
+    High-level context manager to handle a resumable iteration that can be interrupted
+    with a :class:`KeyboardInterrupt` or an :class:`AbortDownloadException`.
 
     It can be used as follows to automatically load a previously-saved state into the iterator, save the iterator's
     state when interrupted, and delete the resume file upon completion::
@@ -239,6 +267,9 @@ def resumable_iteration(context: InstaloaderContext,
     :param format_path: Returns the path to the resume file for the given magic.
     :param check_bbd: Whether to check the best before date and reject an expired FrozenNodeIterator.
     :param enabled: Set to False to disable all functionality and simply execute the inner body.
+
+    .. versionchanged:: 4.7
+       Also interrupt on :class:`AbortDownloadException`.
     """
     if not enabled or not isinstance(iterator, NodeIterator):
         yield False, 0
@@ -258,11 +289,11 @@ def resumable_iteration(context: InstaloaderContext,
             is_resuming = True
             start_index = iterator.total_index
             context.log("Resuming from {}.".format(resume_file_path))
-        except (InvalidArgumentException, LZMAError, json.decoder.JSONDecodeError) as exc:
+        except (InvalidArgumentException, LZMAError, json.decoder.JSONDecodeError, EOFError) as exc:
             context.error("Warning: Not resuming from {}: {}".format(resume_file_path, exc))
     try:
         yield is_resuming, start_index
-    except KeyboardInterrupt:
+    except (KeyboardInterrupt, AbortDownloadException):
         if os.path.dirname(resume_file_path):
             os.makedirs(os.path.dirname(resume_file_path), exist_ok=True)
         save(iterator.freeze(), resume_file_path)

instaloader/sectioniterator.py

@@ -0,0 +1,46 @@
+from typing import Any, Callable, Dict, Iterator, Optional, TypeVar
+
+from .instaloadercontext import InstaloaderContext
+
+T = TypeVar('T')
+
+
+class SectionIterator(Iterator[T]):
+    """Iterator for the new 'sections'-style responses.
+
+    .. versionadded:: 4.9"""
+    def __init__(self,
+                 context: InstaloaderContext,
+                 sections_extractor: Callable[[Dict[str, Any]], Dict[str, Any]],
+                 media_wrapper: Callable[[Dict], T],
+                 query_path: str,
+                 first_data: Optional[Dict[str, Any]] = None):
+        self._context = context
+        self._sections_extractor = sections_extractor
+        self._media_wrapper = media_wrapper
+        self._query_path = query_path
+        self._data = first_data or self._query()
+        self._page_index = 0
+        self._section_index = 0
+
+    def __iter__(self):
+        return self
+
+    def _query(self, max_id: Optional[str] = None) -> Dict[str, Any]:
+        pagination_variables = {"max_id": max_id} if max_id is not None else {}
+        return self._sections_extractor(
+            self._context.get_json(self._query_path, params={"__a": 1, "__d": "dis", **pagination_variables})
+        )
+
+    def __next__(self) -> T:
+        if self._page_index < len(self._data['sections']):
+            media = self._data['sections'][self._page_index]['layout_content']['medias'][self._section_index]['media']
+            self._section_index += 1
+            if self._section_index >= len(self._data['sections'][self._page_index]['layout_content']['medias']):
+                self._section_index = 0
+                self._page_index += 1
+            return self._media_wrapper(media)
+        if self._data['more_available']:
+            self._page_index, self._section_index, self._data = 0, 0, self._query(self._data["next_max_id"])
+            return self.__next__()
+        raise StopIteration()

setup.py

@@ -3,7 +3,6 @@
 import re
 import sys
 import os
-import platform
 from setuptools import setup
 
 
@@ -19,14 +18,11 @@ def get_version():
     raise SystemExit("Could not find version string.")
 
 
-if sys.version_info < (3, 5):
+if sys.version_info < (3, 8):
-    sys.exit('Instaloader requires Python >= 3.5.')
+    sys.exit('Instaloader requires Python >= 3.8.')
 
 requirements = ['requests>=2.4']
 
-if platform.system() == 'Windows' and sys.version_info < (3, 6):
-    requirements.append('win_unicode_console')
-
 keywords = (['instagram', 'instagram-scraper', 'instagram-client', 'instagram-feed', 'downloader', 'videos', 'photos',
              'pictures', 'instagram-user-photos', 'instagram-photos', 'instagram-metadata', 'instagram-downloader',
              'instagram-stories'])
@@ -38,6 +34,7 @@ setup(
     name='instaloader',
     version=get_version(),
     packages=['instaloader'],
+    package_data={'instaloader': ['py.typed']},
     url='https://instaloader.github.io/',
     license='MIT',
     author='Alexander Graf, André Koch-Kramer',
@@ -46,9 +43,9 @@ setup(
                 'from Instagram.',
     long_description=open(os.path.join(SRC, 'README.rst')).read(),
     install_requires=requirements,
-    python_requires='>=3.5',
+    python_requires='>=3.8',
     entry_points={'console_scripts': ['instaloader=instaloader.__main__:main']},
-    zip_safe=True,
+    zip_safe=False,
     keywords=keywords,
     classifiers=[
         'Development Status :: 5 - Production/Stable',
@@ -57,10 +54,10 @@ setup(
         'Intended Audience :: Developers',
         'License :: OSI Approved :: MIT License',
         'Operating System :: OS Independent',
-        'Programming Language :: Python :: 3.5',
-        'Programming Language :: Python :: 3.6',
-        'Programming Language :: Python :: 3.7',
         'Programming Language :: Python :: 3.8',
+        'Programming Language :: Python :: 3.9',
+        'Programming Language :: Python :: 3.10',
+        'Programming Language :: Python :: 3.11',
         'Programming Language :: Python :: 3 :: Only',
         'Topic :: Internet',
         'Topic :: Multimedia :: Graphics'

test/instaloader_unittests.py

@@ -23,7 +23,7 @@ PRIVATE_PROFILE_ID = 1706625676
 EMPTY_PROFILE = "not_public"
 EMPTY_PROFILE_ID = 1928659031
 
-ratecontroller = None  # type: Optional[instaloader.RateController]
+ratecontroller: Optional[instaloader.RateController] = None
 
 
 class TestInstaloaderAnonymously(unittest.TestCase):