Release of Version 4.9.6

Fixed downloading with location id
Detect redirects to https://i.instagram.com/accounts/login
2026-10-17 06:02:18 -07:00 · 2023-02-11 13:21:07 +01:00 · 2023-02-11 13:13:27 +01:00 · 2023-02-11 13:00:52 +01:00 · 2023-02-11 12:45:38 +01:00 · 2022-09-26 21:38:51 +02:00
38 changed files with 2544 additions and 1064 deletions
@@ -6,13 +6,15 @@ labels: "feature suggestion"
 ---

 **Provide us a use case of the feature**
-How could the user invoke the new function? Which problem would it solve? If new information is obtained, how would it be further processed?
+How could the user invoke the new function? Which problem would it solve?
+If new information is obtained, how would it be further processed?

 **Describe the solution you'd like**
 A clear and concise description of what you want to happen.

 **Describe alternatives you've considered**
 A clear and concise description of any alternative solutions or features you've considered.
+How do you achieve the here-described goal without the feature you are requesting?

 **If the feature request is accepted, would you be willing to submit a pull request?**
 Yes / No
@@ -21,4 +23,9 @@ Yes / No
 **Additional context**
 Add any other context about the feature request here.

-<!-- please also see https://instaloader.github.io/contributing.html for how to suggest a feature -->
+<!-- 
+Also see https://instaloader.github.io/contributing.html for how to suggest a feature.
+
+Please understand that a feature suggestion issue that lacks the required information cannot be
+processed.
+-->
@@ -1 +1,2 @@
-custom: "https://instaloader.github.io/contributing.html#donations"
+github: [aandergr]
+custom: "https://instaloader.github.io/sponsors.html"
@@ -0,0 +1,31 @@
+name: "Documentation"
+
+on: [push, pull_request]
+
+jobs:
+  docs:
+    name: "Documentation"
+    runs-on: ubuntu-latest
+    steps:
+      - name: "Checkout Instaloader Repository"
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0  # needed for building docs
+      - name: "Setup Python"
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.8
+      - name: "Install Dependencies"
+        run: |
+          python -m pip install pipenv==2023.2.4
+          pipenv --python `python --version | grep -Eo '3\.[0-9]+'` sync --dev
+      - name: "Build Documentation"
+        run: pipenv --python `python --version | grep -Eo '3\.[0-9]+'` run make -C docs html SPHINXOPTS="-W -n"
+      - name: "Deploy Documentation"
+        if: github.event_name == 'push' && github.ref == 'refs/heads/master'
+        uses: JamesIves/github-pages-deploy-action@v4.2.5
+        with:
+          branch: master
+          repository-name: instaloader/instaloader.github.io
+          folder: docs/_build/html
+          token: ${{ secrets.PAGES_DEPLOYMENT_TOKEN }}
@@ -7,24 +7,21 @@ jobs:
    name: PyLint and MyPy
    runs-on: ubuntu-latest
    strategy:
+      fail-fast: false
      matrix:
-        python-version: [3.8]
+        python-version: ["3.8", "3.9", "3.10", "3.11"]
    steps:
      - name: Checkout Instaloader Repository
        uses: actions/checkout@v2
-        with:
-          fetch-depth: 0  # needed for building docs
      - name: Setup Python
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install Dependencies
        run: |
-          python -m pip install pipenv==2020.6.2
-          pipenv sync --dev
+          python -m pip install pipenv==2023.2.4
+          pipenv --python `python --version | grep -Eo '3\.[0-9]+'` sync --dev
      - name: PyLint
-        run: pipenv run pylint instaloader
+        run: pipenv --python `python --version | grep -Eo '3\.[0-9]+'` run pylint instaloader
      - name: MyPy
-        run: pipenv run mypy -m instaloader
-      - name: Build Documentation
-        run: pipenv run make -C docs html SPHINXOPTS="-W -n"
+        run: pipenv --python `python --version | grep -Eo '3\.[0-9]+'` run mypy -m instaloader
@@ -0,0 +1,36 @@
+name: "Publish Packages"
+
+on:
+  push:
+    tags:
+      - v*
+
+jobs:
+  publish:
+    name: "Publish Package"
+    runs-on: ubuntu-latest
+    steps:
+      - name: "Checkout Repository"
+        uses: actions/checkout@v2
+      - name: "Setup Python"
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.8
+      - name: "Get the tagged version"
+        id: get_version
+        env:
+          GITHUB_REF: ${{ github.ref }}
+        run: echo ::set-output name=VERSION::${GITHUB_REF/refs\/tags\/v/}
+        shell: bash
+      - name: "Generate Python Package"
+        run: python setup.py sdist
+      - name: "Publish Distribution to PyPI"
+        uses: pypa/gh-action-pypi-publish@master
+        with:
+          password: ${{ secrets.PYPI_PASSWORD }}
+      - name: "Publish Distribution to AUR"
+        run: deploy/arch/deploy.sh $VERSION_TAG
+        env:
+          VERSION_TAG: v${{ steps.get_version.outputs.VERSION }}
+          AUR_KEY: ${{ secrets.AUR_KEY }}
+          AUR_IV: ${{ secrets.AUR_IV }}
@@ -18,7 +18,7 @@ jobs:
        stale-issue-message: 'There has been no activity on this question for an extended period of time. This issue will be closed after further 14 days of inactivity.'
        stale-issue-label: 'stale'
        exempt-issue-labels: 'leave open'
-        days-before-stale: 15
+        days-before-stale: 21
        days-before-close: -1
        remove-stale-when-updated: false
    - uses: actions/stale@v1
@@ -30,5 +30,5 @@ jobs:
        stale-pr-label: 'stale'
        exempt-issue-label: 'leave open'
        exempt-pr-label: 'leave open'
-        days-before-stale: 90
+        days-before-stale: 189
        days-before-close: 14
@@ -10,9 +10,9 @@ jobs:
      - name: Checkout Instaloader repository
        uses: actions/checkout@v2
      - name: Setup Python
-        uses: actions/setup-python@v1
+        uses: actions/setup-python@v2
        with:
-          python-version: 3.8
+          python-version: "3.9"
          architecture: x64
      - name: Get the tagged version
        id: get_version
@@ -1,36 +0,0 @@
-dist: xenial
-language: python
-python:
- - "3.5"
- - "3.6"
- - "3.7"
-install:
- - pipenv sync --dev
-script:
- - pylint instaloader
- - mypy -m instaloader
- - make -C docs html SPHINXOPTS="-W -n"
-addons:
-  ssh_known_hosts: aur.archlinux.org
-deploy:
-  - provider: pypi
-    user: aandergr
-    on:
-      tags: true
-      python: 3.6
-    password:
-      secure: KBsATtXNnEzqr0X2ALdyx2zaYj2EUfK8YjP4jEOilBELpJ4evHYXVgRkn9CKXGsD7LKbWmR3gz2rr8NZ6U42pHw0fpQHizz91MpJSKbNY7RZZh4UVWMDh81izeglchWAF+m3Wdpxpo/GBDbrZIbRQFhPgB+kjSzJo3uiCS+yVXtGG+6csJHzd0WkJjsVHz+XSdxyOiLy/qFF3nmY6ZrVrgw6WbX2YLbx9je6DxK3YRrnBqgMgH7Ui2xWWbgJCk3KKpQTMiMKnTa9mdeMJAd26YMCz1gG/X7ck6VNly/6bs5+Zq7/bU0dSolhPBhNthyccYdS0IeGPdPF3YqatblJ4fLGTfHn8oJs3/T11yIW9pvOT3UftEynPOUX+en2Qnb/3WfgwUfVts8i8DAYgEFLGEzbF5XMGy4gwAD/bZ0JW0pmgAG2PZsdOokAt+ydlNj/uxq1GFW2doZbul6k4t03p1mik6itYscvuhKoEm8Yw6wr1P9aDvMR6FFpSu4gDt2XrKmRmWkSXxpNeCfZujTjyAnISxLVQGSY7BG5K0Z2HbDtqqNz9nj4WgNGzZmngiRofCevWR+vPIDUDW7NCX03csfcFS9RRllBO/BrL7NVUV+Qp9dfTQYp+8SqYLWgHtDoXdgyMREjJaAmbx2u/TYTY8bADZ5F3vGUpv2j5hP3NTU=
-  - provider: pages
-    skip-cleanup: true
-    github-token: $GITHUB_TOKEN
-    local-dir: docs/_build/html
-    repo: instaloader/instaloader.github.io
-    target-branch: master
-    on:
-      branch: master
-      python: 3.6
-  - provider: script
-    script: deploy/arch/deploy.sh $TRAVIS_TAG
-    on:
-      tags: true
-      python: 3.6
@@ -11,6 +11,13 @@ pyinstaller = "*"
 pefile = "*"
 pywin32-ctypes = "*"
 psutil = "*"
+typing-extensions = "*"
+types-requests = "*"
+typed-ast = "*"
+jinja2 = "~=3.0.1"
+wrapt = "*"  # Needed for Python<3.11
+tomli = "*"  # Needed for Python<3.11
+dill = "*"  # Needed for Python<3.11

 [packages]
 requests = "*"
@@ -86,6 +86,16 @@ To later **update your local copy** of that profiles, you may run
 If ``--fast-update`` is given, Instaloader stops when arriving at the
 first already-downloaded picture.

+Alternatively, you can use ``--latest-stamps`` to have Instaloader store
+the time each profile was last downloaded and only download newer media:
+
+::
+
+    instaloader --latest-stamps -- profile [profile ...]
+
+With this option it's possible to move or delete downloaded media and still keep
+the archive updated.
+
 When updating profiles, Instaloader
 automatically **detects profile name changes** and renames the target directory
 accordingly.
@@ -104,6 +114,37 @@ already have a valid session cookie file.

 `Instaloader Documentation <https://instaloader.github.io/basic-usage.html>`__

+Contributing
+------------
+
+As an open source project, Instaloader heavily depends on the contributions from
+its community. See
+`contributing <https://instaloader.github.io/contributing.html>`__
+for how you may help Instaloader to become an even greater tool.
+
+Supporters
+----------
+
+.. current-sponsors-start
+
+| Instaloader is proudly sponsored by
+|  `@socialmethod <https://github.com/socialmethod>`__
+
+See `Alex' GitHub Sponsors <https://github.com/sponsors/aandergr>`__ page for
+how you can sponsor the development of Instaloader!
+
+.. current-sponsors-end
+
+It is a pleasure for us to share our Instaloader to the world, and we are proud
+to have attracted such an active and motivating community, with so many users
+who share their suggestions and ideas with us. Buying a community-sponsored beer
+or coffee from time to time is very likely to further raise our passion for the
+development of Instaloader.
+
+| For Donations, we provide GitHub Sponsors page, a PayPal.Me link and a Bitcoin address.
+|  GitHub Sponsors: `Sponsor @aandergr on GitHub Sponsors <https://github.com/sponsors/aandergr>`__
+|  PayPal: `PayPal.me/aandergr <https://www.paypal.me/aandergr>`__
+|  BTC: 1Nst4LoadeYzrKjJ1DX9CpbLXBYE9RKLwY

 Disclaimer
 ----------
@@ -116,21 +157,3 @@ subsidiaries. This is an independent and unofficial project. Use at your own ris
 Instaloader is licensed under an MIT license. Refer to ``LICENSE`` file for more information.

 .. disclaimer-end
-
-Contributing
------------
-
-As an open source project, Instaloader heavily depends on the contributions from
-its community. See
-`contributing <https://instaloader.github.io/contributing.html>`__
-for how you may help Instaloader to become an even greater tool.
-
-It is a pleasure for us to share our Instaloader to the world, and we are proud
-to have attracted such an active and motivating community, with so many users
-who share their suggestions and ideas with us. Buying a community-sponsored beer
-or coffee from time to time is very likely to further raise our passion for the
-development of Instaloader.
-
-| For Donations, we provide a PayPal.Me link and a Bitcoin address.
-|  PayPal: `PayPal.me/aandergr <https://www.paypal.me/aandergr>`__
-|  BTC: 1Nst4LoadeYzrKjJ1DX9CpbLXBYE9RKLwY
@@ -6,7 +6,7 @@ pkgbase = instaloader
 	arch = any
 	license = MIT
 	makedepends = python-setuptools
-	depends = python>=3.5
+	depends = python>=3.8
 	depends = python-requests>=2.4
 	options = !emptydirs
 	source = instaloader-{{version}}.tar.gz::https://codeload.github.com/instaloader/instaloader/tar.gz/v{{version}}
@@ -8,7 +8,7 @@ arch=('any')
 url="https://instaloader.github.io/"
 license=('MIT')
 groups=()
-depends=('python>=3.5' 'python-requests>=2.4')
+depends=('python>=3.8' 'python-requests>=2.4')
 makedepends=('python-setuptools')
 options=('!emptydirs')
 source=($pkgname-$pkgver.tar.gz::https://codeload.github.com/instaloader/instaloader/tar.gz/v$pkgver)
@@ -8,10 +8,12 @@ echo $VERSION | grep -qe "[abc]" && exit 0
 cd $(dirname $0)

 # decrypt and add ssh key
-openssl aes-256-cbc -K $encrypted_354637631c28_key -iv $encrypted_354637631c28_iv -in id_rsa_AUR.enc -out /tmp/AUR_openssh -d
+openssl aes-256-cbc -K $AUR_KEY -iv $AUR_IV -in id_rsa_AUR.enc -out /tmp/AUR_openssh -d
 eval "$(ssh-agent -s)"
 chmod 600 /tmp/AUR_openssh
 ssh-add /tmp/AUR_openssh
+mkdir -p ~/.ssh
+ssh-keyscan -t rsa aur.archlinux.org >> ~/.ssh/known_hosts

 # clone and modify AUR repo
 git clone --depth 1 ssh://aur@aur.archlinux.org/instaloader.git
@@ -44,8 +44,8 @@ with open('__main__.py', 'w+') as f:
    f.writelines(lines)

 # install dependencies and invoke PyInstaller
-commands = ["pip install pipenv==2020.6.2",
-            "pipenv sync --dev",
+commands = ["pip install pipenv==2023.2.4",
+            f"pipenv --python {sys.version_info.major}.{sys.version_info.minor} sync --dev",
            "pipenv run pyinstaller --log-level=DEBUG instaloader.spec"]

 for command in commands:
@@ -39,6 +39,16 @@ To later **update your local copy** of that profiles, you may run
 If :option:`--fast-update` is given, Instaloader stops when arriving at the
 first already-downloaded picture.

+Alternatively, you can use :option:`--latest-stamps` to have Instaloader store
+the time each profile was last downloaded and only download newer media:
+
+::
+
+    instaloader --latest-stamps -- profile [profile ...]
+
+With this option it's possible to move or delete downloaded media and still keep
+the archive updated.
+
 When updating profiles, Instaloader
 automatically **detects profile name changes** and renames the target directory
 accordingly.
@@ -149,27 +159,37 @@ target. The default is ``--dirname-pattern={target}``. In the dirname
 pattern, the token ``{target}`` is replaced by the target name, and
 ``{profile}`` is replaced by the owner of the post which is downloaded.

-:option:`--filename-pattern` configures the path of the post's files relative
+:option:`--filename-pattern` configures the path of the post and story's files relative
 to the target directory that is specified with :option:`--dirname-pattern`.
 The default is ``--filename-pattern={date_utc}_UTC``.
 The tokens ``{target}`` and ``{profile}`` are replaced like in the
-dirname pattern. The following tokens are defined for usage with
-:option:`--filename-pattern`:
+dirname pattern.
+
+:option:`--title-pattern` is similar to :option:`--filename-pattern`, but for profile
+pics, hashtag profile pics, and highlight covers. The default is
+``{date_utc}_UTC_{typename}`` if :option:`--dirname-pattern` contains ``{target}`` or
+``{profile}``, or ``{target}_{date_utc}_UTC_{typename}`` if it does not. Some tokens
+are not supported for this option, see below for details.
+
+The following tokens are defined for usage with
+:option:`--filename-pattern` and :option:`--title-pattern`:

 - ``{target}``
   Target name (as given in Instaloader command line)

 - ``{profile}`` (same as ``{owner_username}``)
-   Owner of the Post / StoryItem.
+   Owner of the Post / StoryItem / ProfilePic. For hashtag profile pics and
+   highlight covers, equivalent to ``{target}``.

 - ``{owner_id}``
-   Unique integer ID of owner profile.
+   Unique integer ID of owner profile. For hashtag profile pics, equivalent to
+   ``{target}``.

 - ``{shortcode}``
-   Shortcode (identifier string).
+   Shortcode (identifier string). Not available for :option:`--title-pattern`.

 - ``{mediaid}``
-   Integer representation of shortcode.
+   Integer representation of shortcode. Not available for :option:`--title-pattern`.

 - ``{filename}``
   Instagram's internal filename.
@@ -182,6 +202,10 @@ dirname pattern. The following tokens are defined for usage with

      {date_utc:%Y-%m-%d_%H-%M-%S}

+- ``{typename}``
+   Type of media being saved, such as GraphImage, GraphStoryVideo, profile_pic,
+   etc.
+
 For example, encode the poster's profile name in the filenames with::

    instaloader --filename-pattern={date_utc}_UTC_{profile} "#hashtag"
@@ -100,7 +100,6 @@ What to Download of each Post

   Do not xz compress JSON files, rather create pretty formatted JSONs.

-
 What to Download of each Profile
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

@@ -147,6 +146,23 @@ Which Posts to Download
   This flag is recommended when you use Instaloader to update your personal
   Instagram archive.

+.. option:: --latest-stamps [STAMPSFILE]
+
+   Works similarly to :option:`--fast-update`, but instead of relying on already
+   downloaded media, the time each profile was downloaded is stored, and only
+   media newer than the last download is fetched. This allows updating your
+   personal Instagram archive while emptying the target directories.
+
+   Only works for media associated with a specific profile, and that is returned
+   in chronological order: profile posts, profile stories, profile IGTV posts
+   and profile tagged posts.
+
+   By default, the information is stored in
+   ``~/.config/instaloader/latest-stamps.ini``, but you can specify an
+   alternative location.
+
+   .. versionadded:: 4.8
+
 .. option:: --post-filter filter, --only-if filter

   Expression that, if given, must evaluate to True for each post to be
@@ -164,8 +180,8 @@ Which Posts to Download

 .. option:: --count COUNT, -c

-   Do not attempt to download more than COUNT posts.  Applies only to
-   ``#hashtag``, ``%location id``, and ``:feed``.
+   Do not attempt to download more than COUNT posts.  Applies to
+   ``#hashtag``, ``%location_id``, ``:feed``, and ``:saved``.


 Login (Download Private Profiles)
@@ -206,12 +222,30 @@ How to Download

 .. option:: --filename-pattern FILENAME_PATTERN

-   Prefix of filenames, relative to the directory given with
+   Prefix of filenames for posts and stories, relative to the directory given with
   :option:`--dirname-pattern`. ``{profile}`` is replaced by the profile name,
   ``{target}`` is replaced by the target you specified, i.e.  either ``:feed``,
   ``#hashtag`` or the profile name. Defaults to ``{date_utc}_UTC``.
   See :ref:`filename-specification` for a list of supported tokens.

+.. option:: --title-pattern TITLE_PATTERN
+
+   Prefix of filenames for profile pics, hashtag profile pics, and highlight
+   covers, relative to the directory given with :option:`--dirname-pattern`.
+   Defaults to ``{date_utc}_UTC_{typename}`` if :option:`--dirname-pattern`
+   contains ``{target}`` or ``{profile}``, otherwise defaults to
+   ``{target}_{date_utc}_UTC_{typename}``.
+   See :ref:`filename-specification` for a list of supported tokens.
+
+   .. versionadded:: 4.8
+
+.. option:: --sanitize-paths
+
+   Force sanitization of paths so that the resulting file and directory names
+   are valid on both Windows and Unix.
+
+   .. versionadded:: 4.9
+
 .. option:: --resume-prefix prefix

   For many targets, Instaloader is capable of resuming a previously-aborted
@@ -224,7 +258,8 @@ How to Download
    - Profile posts,
    - Profile IGTV posts (:option:`--igtv`),
    - Profile tagged posts (:option:`--tagged`),
-    - Saved posts (``:saved``).
+    - Saved posts (``:saved``),
+    - Hashtags.

   This feature is enabled by default for targets where it is supported;
   :option:`--resume-prefix` only changes the name of the iterator files.
@@ -246,7 +281,7 @@ How to Download
 .. option:: --user-agent USER_AGENT

   User Agent to use for HTTP requests. Per default, Instaloader pretends being
-   Chrome/88 on Linux.
+   Chrome/92 on Linux.

 .. option:: --max-connection-attempts N

@@ -263,6 +298,23 @@ How to Download
   .. versionchanged:: 4.6
      Enabled this option by default with a timeout of 300 seconds.

+.. option:: --abort-on STATUS_CODE_LIST
+
+   Comma-separated list of HTTP status codes that cause Instaloader to abort,
+   bypassing all retry logic.
+
+   For example, with ``--abort-on=302,400,429``, Instaloader will stop if a
+   request is responded with a 302 redirect, a Bad Request error, or a Too Many
+   Requests error.
+
+   .. versionadded:: 4.7
+
+.. option:: --no-iphone
+
+   Do not attempt to download iPhone version of images and videos.
+
+   .. versionadded:: 4.8
+
 Miscellaneous Options
 ^^^^^^^^^^^^^^^^^^^^^

@@ -277,12 +329,13 @@ Miscellaneous Options
   Read arguments from file `args.txt`, a shortcut to provide arguments from
   file rather than command-line. This provides a convenient way to hide login
   info from CLI, and can also be used to simplify management of long arguments.
+   You can provide more than one file at once, e.g.: ``+args1.txt +args2.txt``.

   .. note::

      Text file should separate arguments with line breaks.

-   args.txt example::
+   `args.txt` example::

      --login=MYUSERNAME
      --password=MYPASSWORD
@@ -48,7 +48,7 @@ autodoc_default_flags = ['show-inheritance', 'members', 'undoc-members']
 autodoc_member_order = 'bysource'

 intersphinx_mapping = {'python': ('https://docs.python.org/3', None),
-                       'requests': ('https://requests.kennethreitz.org/en/master/', None)}
+                       'requests': ('https://requests.readthedocs.io/en/latest/', None)}

 nitpick_ignore = [('py:class', 'typing.Tuple')]

@@ -132,21 +132,9 @@ details. If you feel the need to do so, please create a "feature suggestion".
  be implemented and the underlying problem could be solved. Also **describe
  alternatives** that you have considered.

-Donations
---------
+Sponsoring
+----------

-.. donations-start
-
-It is a pleasure for us to share our Instaloader to the world, and we are proud
-to have attracted such an active and motivating community, with so many users
-who share their suggestions and ideas with us. Buying a community-sponsored beer
-or coffee from time to time is very likely to further raise our passion for the
-development of Instaloader.
-
-| For donations, we provide a PayPal.Me link and a Bitcoin address.
-|  `PayPal.me/aandergr <https://www.paypal.me/aandergr>`__
-|  BTC: 1Nst4LoadeYzrKjJ1DX9CpbLXBYE9RKLwY
-
-.. donations-end
-
-.. (Discussion in :issue:`130`)
+.. include:: sponsors.rst
+   :start-after: donations-start
+   :end-before: donations-end
@@ -78,13 +78,6 @@ Useful Links
 - `Issue Tracker / Bug Tracker <https://github.com/instaloader/instaloader/issues>`__
 - `Version History <https://github.com/instaloader/instaloader/releases>`__

-Disclaimer
----------
-
-.. include:: ../README.rst
-   :start-after: disclaimer-start
-   :end-before: disclaimer-end
-
 Contributing
 ------------

@@ -92,10 +85,20 @@ As an open source project, Instaloader heavily depends on the contributions from
 its community. See :ref:`contributing` for how you may help Instaloader to
 become an even greater tool.

-.. include:: contributing.rst
+Supporters
+----------
+
+.. include:: sponsors.rst
   :start-after: donations-start
   :end-before: donations-end

+Disclaimer
+----------
+
+.. include:: ../README.rst
+   :start-after: disclaimer-start
+   :end-before: disclaimer-end
+
 ..
   * :ref:`genindex`
   * :ref:`modindex`
@@ -8,7 +8,7 @@ Install Instaloader
 To **install Instaloader**,

 #. Ensure that you have `Python <https://www.python.org/>`__, at least
-   version 3.5, and `pip <https://pypi.python.org/pypi/pip>`__
+   version 3.8, and `pip <https://pypi.python.org/pypi/pip>`__
   installed.

 #. Then, install Instaloader using::
@@ -47,3 +47,5 @@ Exceptions


 .. autoexception:: TooManyRequestsException
+
+.. autoexception:: AbortDownloadException
@@ -25,6 +25,13 @@ Iterator :class:`NodeIterator` and a context manager
 .. autoclass:: FrozenNodeIterator
   :no-show-inheritance:

+   A serializable representation of a :class:`NodeIterator` instance, saving
+   its iteration state.
+
+   It can be serialized and deserialized with :func:`save_structure_to_file`
+   and :func:`load_structure_from_file`, as well as with :mod:`json` and
+   :mod:`pickle` thanks to being a :func:`~collections.namedtuple`.
+
 ``resumable_iteration``
 """""""""""""""""""""""

@@ -77,6 +77,16 @@ Loading and Saving
 :class:`Post`, :class:`StoryItem`, :class:`Profile`, :class:`Hashtag` and
 :class:`FrozenNodeIterator` can be saved and loaded to/from JSON files.

+.. autofunction:: load_structure
+
 .. autofunction:: load_structure_from_file

+.. autofunction:: get_json_structure
+
 .. autofunction:: save_structure_to_file
+
+LatestStamps
+""""""""""""
+
+.. autoclass:: LatestStamps
+   :no-show-inheritance:
@@ -0,0 +1,23 @@
+.. _sponsors:
+
+Sponsoring the development of Instaloader
+=========================================
+
+.. donations-start
+
+.. include:: ../README.rst
+   :start-after: current-sponsors-start
+   :end-before: current-sponsors-end
+
+It is a pleasure for us to share our Instaloader to the world, and we are proud
+to have attracted such an active and motivating community, with so many users
+who share their suggestions and ideas with us. Buying a community-sponsored beer
+or coffee from time to time is very likely to further raise our passion for the
+development of Instaloader.
+
+| For Donations, we provide GitHub Sponsors page, a PayPal.Me link and a Bitcoin address.
+|  GitHub Sponsors: `Sponsor @aandergr on GitHub Sponsors <https://github.com/sponsors/aandergr>`__
+|  PayPal: `PayPal.me/aandergr <https://www.paypal.me/aandergr>`__
+|  BTC: 1Nst4LoadeYzrKjJ1DX9CpbLXBYE9RKLwY
+
+.. donations-end
@@ -62,7 +62,8 @@ Two-Factor-Authentication. It also supports handling the *checkpoint challenge*,
 issued when Instagram suspects authentication activity on your account, by
 pointing the user to an URL to be opened in a browser.

-Nevertheless, in :issue:`92` and :issue:`615` users reported problems with
+Nevertheless, in :issue:`92`, :issue:`615`, :issue:`1150` and :issue:`1217`,
+users reported problems with
 logging in. We recommend to always keep the session file which Instaloader
 creates when using :option:`--login`. If a session file is present,
 :option:`--login` does not make use of the failure-prone login procedure.
@@ -1,7 +1,7 @@
 """Download pictures (or videos) along with their captions and other metadata from Instagram."""


-__version__ = '4.6.2'
+__version__ = '4.9.6'


 try:
@@ -15,6 +15,8 @@ else:
 from .exceptions import *
 from .instaloader import Instaloader
 from .instaloadercontext import InstaloaderContext, RateController
+from .lateststamps import LatestStamps
 from .nodeiterator import NodeIterator, FrozenNodeIterator, resumable_iteration
 from .structures import (Hashtag, Highlight, Post, PostSidecarNode, PostComment, PostCommentAnswer, PostLocation,
-                         Profile, Story, StoryItem, TopSearchResults, load_structure_from_file, save_structure_to_file)
+                         Profile, Story, StoryItem, TopSearchResults, load_structure_from_file, save_structure_to_file,
+                         load_structure, get_json_structure)
@@ -5,14 +5,15 @@ import datetime
 import os
 import re
 import sys
-from argparse import ArgumentParser, SUPPRESS
+from argparse import ArgumentParser, ArgumentTypeError, SUPPRESS
 from typing import List, Optional

-from . import (Instaloader, InstaloaderException, InvalidArgumentException, Post, Profile, ProfileNotExistsException,
-               StoryItem, __version__, load_structure_from_file, TwoFactorAuthRequiredException,
-               BadCredentialsException)
-from .instaloader import get_default_session_filename
+from . import (AbortDownloadException, BadCredentialsException, Instaloader, InstaloaderException,
+               InvalidArgumentException, Post, Profile, ProfileNotExistsException, StoryItem,
+               TwoFactorAuthRequiredException, __version__, load_structure_from_file)
+from .instaloader import (get_default_session_filename, get_default_stamps_filename)
 from .instaloadercontext import default_user_agent
+from .lateststamps import LatestStamps


 def usage_string():
@@ -27,6 +28,14 @@ def usage_string():
 {0} --help""".format(argv0, len(argv0), '')


+def http_status_code_list(code_list_str: str) -> List[int]:
+    codes = [int(s) for s in code_list_str.split(',')]
+    for code in codes:
+        if not 100 <= code <= 599:
+            raise ArgumentTypeError("Invalid HTTP status code: {}".format(code))
+    return codes
+
+
 def filterstr_to_filterfunc(filter_str: str, item_type: type):
    """Takes an --post-filter=... or --storyitem-filter=... filter
     specification and makes a filter_func Callable out of it."""
@@ -36,7 +45,6 @@ def filterstr_to_filterfunc(filter_str: str, item_type: type):

    class TransformFilterAst(ast.NodeTransformer):
        def visit_Name(self, node: ast.Name):
-            # pylint:disable=no-self-use
            if not isinstance(node.ctx, ast.Load):
                raise InvalidArgumentException("Invalid filter: Modifying variables ({}) not allowed.".format(node.id))
            if node.id == "datetime":
@@ -68,6 +76,7 @@ def _main(instaloader: Instaloader, targetlist: List[str],
          download_tagged: bool = False,
          download_igtv: bool = False,
          fast_update: bool = False,
+          latest_stamps_file: Optional[str] = None,
          max_count: Optional[int] = None, post_filter_str: Optional[str] = None,
          storyitem_filter_str: Optional[str] = None) -> None:
    """Download set of profiles, hashtags etc. and handle logging in and session files if desired."""
@@ -80,6 +89,10 @@ def _main(instaloader: Instaloader, targetlist: List[str],
    if storyitem_filter_str is not None:
        storyitem_filter = filterstr_to_filterfunc(storyitem_filter_str, StoryItem)
        instaloader.context.log('Only download storyitems with property "{}".'.format(storyitem_filter_str))
+    latest_stamps = None
+    if latest_stamps_file is not None:
+        latest_stamps = LatestStamps(latest_stamps_file)
+        instaloader.context.log(f"Using latest stamps from {latest_stamps_file}.")
    # Login, if desired
    if username is not None:
        if not re.match(r"^[A-Za-z0-9._]+$", username):
@@ -100,7 +113,8 @@ def _main(instaloader: Instaloader, targetlist: List[str],
                            code = input("Enter 2FA verification code: ")
                            instaloader.two_factor_login(code)
                            break
-                        except BadCredentialsException:
+                        except BadCredentialsException as err:
+                            print(err, file=sys.stderr)
                            pass
            else:
                instaloader.interactive_login(username)
@@ -164,7 +178,7 @@ def _main(instaloader: Instaloader, targetlist: List[str],
                                                     post_filter=post_filter)
                elif re.match(r"^[A-Za-z0-9._]+$", target):
                    try:
-                        profile = instaloader.check_profile_id(target)
+                        profile = instaloader.check_profile_id(target, latest_stamps)
                        if instaloader.context.is_logged_in and profile.has_blocked_viewer:
                            if download_profile_pic or ((download_posts or download_tagged or download_igtv)
                                                        and not profile.is_private):
@@ -183,7 +197,8 @@ def _main(instaloader: Instaloader, targetlist: List[str],
                            instaloader.context.log("Trying again anonymously, helps in case you are just blocked.")
                            with instaloader.anonymous_copy() as anonymous_loader:
                                with instaloader.context.error_catcher():
-                                    anonymous_retry_profiles.add(anonymous_loader.check_profile_id(target))
+                                    anonymous_retry_profiles.add(anonymous_loader.check_profile_id(target,
+                                                                                                   latest_stamps))
                                    instaloader.context.error("Warning: {} will be downloaded anonymously (\"{}\")."
                                                              .format(target, err))
                        else:
@@ -198,21 +213,25 @@ def _main(instaloader: Instaloader, targetlist: List[str],
        if len(profiles) > 1:
            instaloader.context.log("Downloading {} profiles: {}".format(len(profiles),
                                                                         ' '.join([p.username for p in profiles])))
-        if profiles and (download_profile_pic or download_posts) and not instaloader.context.is_logged_in:
-            instaloader.context.error("Warning: Use --login to download higher-quality versions of pictures.")
+        if instaloader.context.iphone_support and profiles and (download_profile_pic or download_posts) and \
+           not instaloader.context.is_logged_in:
+            instaloader.context.log("Hint: Use --login to download higher-quality versions of pictures.")
        instaloader.download_profiles(profiles,
                                      download_profile_pic, download_posts, download_tagged, download_igtv,
                                      download_highlights, download_stories,
-                                      fast_update, post_filter, storyitem_filter)
+                                      fast_update, post_filter, storyitem_filter, latest_stamps=latest_stamps)
        if anonymous_retry_profiles:
            instaloader.context.log("Downloading anonymously: {}"
                                    .format(' '.join([p.username for p in anonymous_retry_profiles])))
            with instaloader.anonymous_copy() as anonymous_loader:
                anonymous_loader.download_profiles(anonymous_retry_profiles,
                                                   download_profile_pic, download_posts, download_tagged, download_igtv,
-                                                   fast_update=fast_update, post_filter=post_filter)
+                                                   fast_update=fast_update, post_filter=post_filter,
+                                                   latest_stamps=latest_stamps)
    except KeyboardInterrupt:
        print("\nInterrupted by user.", file=sys.stderr)
+    except AbortDownloadException as exc:
+        print("\nDownload aborted: {}.".format(exc), file=sys.stderr)
    # Save session if it is useful
    if instaloader.context.is_logged_in:
        instaloader.save_session_to_file(sessionfile)
@@ -313,7 +332,10 @@ def main():
    g_cond.add_argument('-F', '--fast-update', action='store_true',
                        help='For each target, stop when encountering the first already-downloaded picture. This '
                             'flag is recommended when you use Instaloader to update your personal Instagram archive.')
-
+    g_cond.add_argument('--latest-stamps', nargs='?', metavar='STAMPSFILE', const=get_default_stamps_filename(),
+                        help='Store the timestamps of latest media scraped for each profile. This allows updating '
+                             'your personal Instagram archive even if you delete the destination directories. '
+                             'If STAMPSFILE is not provided, defaults to ' + get_default_stamps_filename())
    g_cond.add_argument('--post-filter', '--only-if', metavar='filter',
                        help='Expression that, if given, must evaluate to True for each post to be downloaded. Must be '
                             'a syntactically valid python expression. Variables are evaluated to '
@@ -325,7 +347,7 @@ def main():

    g_cond.add_argument('-c', '--count',
                        help='Do not attempt to download more than COUNT posts. '
-                             'Applies only to #hashtag and :feed.')
+                             'Applies to #hashtag, %%location_id, :feed, and :saved.')

    g_login = parser.add_argument_group('Login (Download Private Profiles)',
                                        'Instaloader can login to Instagram. This allows downloading private profiles. '
@@ -348,13 +370,21 @@ def main():
                            '{target} is replaced by the target you specified, i.e. either :feed, #hashtag or the '
                            'profile name. Defaults to \'{target}\'.')
    g_how.add_argument('--filename-pattern',
-                       help='Prefix of filenames, relative to the directory given with '
+                       help='Prefix of filenames for posts and stories, relative to the directory given with '
                            '--dirname-pattern. {profile} is replaced by the profile name,'
                            '{target} is replaced by the target you specified, i.e. either :feed'
                            '#hashtag or the profile name. Defaults to \'{date_utc}_UTC\'')
+    g_how.add_argument('--title-pattern',
+                       help='Prefix of filenames for profile pics, hashtag profile pics, and highlight covers. '
+                            'Defaults to \'{date_utc}_UTC_{typename}\' if --dirname-pattern contains \'{target}\' '
+                            'or \'{dirname}\', or if --dirname-pattern is not specified. Otherwise defaults to '
+                            '\'{target}_{date_utc}_UTC_{typename}\'.')
    g_how.add_argument('--resume-prefix', metavar='PREFIX',
                       help='Prefix for filenames that are used to save the information to resume an interrupted '
                            'download.')
+    g_how.add_argument('--sanitize-paths', action='store_true',
+                       help='Sanitize paths so that the resulting file and directory names are valid on both '
+                            'Windows and Unix.')
    g_how.add_argument('--no-resume', action='store_true',
                       help='Do not resume a previously-aborted download iteration, and do not save such information '
                            'when interrupted.')
@@ -369,6 +399,11 @@ def main():
    g_how.add_argument('--commit-mode', action='store_true', help=SUPPRESS)
    g_how.add_argument('--request-timeout', metavar='N', type=float, default=300.0,
                       help='Seconds to wait before timing out a connection request. Defaults to 300.')
+    g_how.add_argument('--abort-on', type=http_status_code_list, metavar="STATUS_CODES",
+                       help='Comma-separated list of HTTP status codes that cause Instaloader to abort, bypassing all '
+                            'retry logic.')
+    g_how.add_argument('--no-iphone', action='store_true',
+                        help='Do not attempt to download iPhone version of images and videos.')

    g_misc = parser.add_argument_group('Miscellaneous Options')
    g_misc.add_argument('-q', '--quiet', action='store_true',
@@ -427,7 +462,11 @@ def main():
                             request_timeout=args.request_timeout,
                             resume_prefix=resume_prefix,
                             check_resume_bbd=not args.use_aged_resume_files,
-                             slide=args.slide)
+                             slide=args.slide,
+                             fatal_status_codes=args.abort_on,
+                             iphone_support=not args.no_iphone,
+                             title_pattern=args.title_pattern,
+                             sanitize_paths=args.sanitize_paths)
        _main(loader,
              args.profile,
              username=args.login.lower() if args.login is not None else None,
@@ -440,12 +479,13 @@ def main():
              download_tagged=args.tagged,
              download_igtv=args.igtv,
              fast_update=args.fast_update,
+              latest_stamps_file=args.latest_stamps,
              max_count=int(args.count) if args.count is not None else None,
              post_filter_str=args.post_filter,
              storyitem_filter_str=args.storyitem_filter)
        loader.close()
    except InstaloaderException as err:
-        raise SystemExit("Fatal error: %s" % err)
+        raise SystemExit("Fatal error: %s" % err) from err


 if __name__ == "__main__":
@@ -64,3 +64,17 @@ class QueryReturnedNotFoundException(ConnectionException):

 class TooManyRequestsException(ConnectionException):
    pass
+
+class IPhoneSupportDisabledException(InstaloaderException):
+    pass
+
+class AbortDownloadException(Exception):
+    """
+    Exception that is not catched in the error catchers inside the download loop and so aborts the
+    download loop.
+
+    This exception is not a subclass of ``InstaloaderException``.
+
+    .. versionadded:: 4.7
+    """
+    pass
@@ -10,7 +10,6 @@ import tempfile
 from contextlib import contextmanager, suppress
 from datetime import datetime, timezone
 from functools import wraps
-from hashlib import md5
 from io import BytesIO
 from pathlib import Path
 from typing import Any, Callable, IO, Iterator, List, Optional, Set, Union, cast
@@ -21,23 +20,30 @@ import urllib3  # type: ignore

 from .exceptions import *
 from .instaloadercontext import InstaloaderContext, RateController
+from .lateststamps import LatestStamps
 from .nodeiterator import NodeIterator, resumable_iteration
+from .sectioniterator import SectionIterator
 from .structures import (Hashtag, Highlight, JsonExportable, Post, PostLocation, Profile, Story, StoryItem,
-                         load_structure_from_file, save_structure_to_file, PostSidecarNode)
+                         load_structure_from_file, save_structure_to_file, PostSidecarNode, TitlePic)
+
+
+def _get_config_dir() -> str:
+    if platform.system() == "Windows":
+        # on Windows, use %LOCALAPPDATA%\Instaloader
+        localappdata = os.getenv("LOCALAPPDATA")
+        if localappdata is not None:
+            return os.path.join(localappdata, "Instaloader")
+        # legacy fallback - store in temp dir if %LOCALAPPDATA% is not set
+        return os.path.join(tempfile.gettempdir(), ".instaloader-" + getpass.getuser())
+    # on Unix, use ~/.config/instaloader
+    return os.path.join(os.getenv("XDG_CONFIG_HOME", os.path.expanduser("~/.config")), "instaloader")


 def get_default_session_filename(username: str) -> str:
    """Returns default session filename for given username."""
+    configdir = _get_config_dir()
    sessionfilename = "session-{}".format(username)
-    if platform.system() == "Windows":
-        # on Windows, use %LOCALAPPDATA%\Instaloader\session-USERNAME
-        localappdata = os.getenv("LOCALAPPDATA")
-        if localappdata is not None:
-            return os.path.join(localappdata, "Instaloader", sessionfilename)
-        # legacy fallback - store in temp dir if %LOCALAPPDATA% is not set
-        return os.path.join(tempfile.gettempdir(), ".instaloader-" + getpass.getuser(), sessionfilename)
-    # on Unix, use ~/.config/instaloader/session-USERNAME
-    return os.path.join(os.getenv("XDG_CONFIG_HOME", os.path.expanduser("~/.config")), "instaloader", sessionfilename)
+    return os.path.join(configdir, sessionfilename)


 def get_legacy_session_filename(username: str) -> str:
@@ -47,6 +53,17 @@ def get_legacy_session_filename(username: str) -> str:
    return filename.lower()


+def get_default_stamps_filename() -> str:
+    """
+    Returns default filename for latest stamps database.
+
+    .. versionadded:: 4.8
+
+    """
+    configdir = _get_config_dir()
+    return os.path.join(configdir, "latest-stamps.ini")
+
+
 def format_string_contains_key(format_string: str, key: str) -> bool:
    # pylint:disable=unused-variable
    for literal_text, field_name, format_spec, conversion in string.Formatter().parse(format_string):
@@ -102,7 +119,7 @@ class _ArbitraryItemFormatter(string.Formatter):

    def get_value(self, key, args, kwargs):
        """Override to substitute {ATTRIBUTE} by attributes of our _item."""
-        if key == 'filename' and isinstance(self._item, (Post, StoryItem, PostSidecarNode)):
+        if key == 'filename' and isinstance(self._item, (Post, StoryItem, PostSidecarNode, TitlePic)):
            return "{filename}"
        if hasattr(self._item, key):
            return getattr(self._item, key)
@@ -120,20 +137,38 @@ class _ArbitraryItemFormatter(string.Formatter):


 class _PostPathFormatter(_ArbitraryItemFormatter):
+    RESERVED: set = {'CON', 'PRN', 'AUX', 'NUL',
+                     'COM1', 'COM2', 'COM3', 'COM4', 'COM5', 'COM6', 'COM7', 'COM8', 'COM9',
+                     'LPT1', 'LPT2', 'LPT3', 'LPT4', 'LPT5', 'LPT6', 'LPT7', 'LPT8', 'LPT9'}
+
+    def __init__(self, item: Any, force_windows_path: bool = False):
+        super().__init__(item)
+        self.force_windows_path = force_windows_path
+
    def get_value(self, key, args, kwargs):
        ret = super().get_value(key, args, kwargs)
        if not isinstance(ret, str):
            return ret
-        return self.sanitize_path(ret)
+        return self.sanitize_path(ret, self.force_windows_path)

    @staticmethod
-    def sanitize_path(ret: str) -> str:
+    def sanitize_path(ret: str, force_windows_path: bool = False) -> str:
        """Replaces '/' with similar looking Division Slash and some other illegal filename characters on Windows."""
        ret = ret.replace('/', '\u2215')
-        if platform.system() == 'Windows':
+
+        if ret.startswith('.'):
+            ret = ret.replace('.', '\u2024', 1)
+
+        if force_windows_path or platform.system() == 'Windows':
            ret = ret.replace(':', '\uff1a').replace('<', '\ufe64').replace('>', '\ufe65').replace('\"', '\uff02')
            ret = ret.replace('\\', '\ufe68').replace('|', '\uff5c').replace('?', '\ufe16').replace('*', '\uff0a')
            ret = ret.replace('\n', ' ').replace('\r', ' ')
+            root, ext = os.path.splitext(ret)
+            if root.upper() in _PostPathFormatter.RESERVED:
+                root += '_'
+            if ext == '.':
+                ext = '\u2024'
+            ret = root + ext
        return ret


@@ -144,6 +179,9 @@ class Instaloader:
    :param user_agent: :option:`--user-agent`
    :param dirname_pattern: :option:`--dirname-pattern`, default is ``{target}``
    :param filename_pattern: :option:`--filename-pattern`, default is ``{date_utc}_UTC``
+    :param title_pattern:
+       :option:`--title-pattern`, default is ``{date_utc}_UTC_{typename}`` if ``dirname_pattern`` contains
+       ``{target}`` or ``{profile}``, ``{target}_{date_utc}_UTC_{typename}`` otherwise.
    :param download_pictures: not :option:`--no-pictures`
    :param download_videos: not :option:`--no-videos`
    :param download_video_thumbnails: not :option:`--no-video-thumbnails`
@@ -161,6 +199,9 @@ class Instaloader:
    :param resume_prefix: :option:`--resume-prefix`, or None for :option:`--no-resume`.
    :param check_resume_bbd: Whether to check the date of expiry of resume files and reject them if expired.
    :param slide: :option:`--slide`
+    :param fatal_status_codes: :option:`--abort-on`
+    :param iphone_support: not :option:`--no-iphone`
+    :param sanitize_paths: :option:`--sanitize-paths`

    .. attribute:: context

@@ -176,25 +217,39 @@ class Instaloader:
                 download_pictures=True,
                 download_videos: bool = True,
                 download_video_thumbnails: bool = True,
-                 download_geotags: bool = True,
-                 download_comments: bool = True,
+                 download_geotags: bool = False,
+                 download_comments: bool = False,
                 save_metadata: bool = True,
                 compress_json: bool = True,
-                 post_metadata_txt_pattern: str = None,
-                 storyitem_metadata_txt_pattern: str = None,
+                 post_metadata_txt_pattern: Optional[str] = None,
+                 storyitem_metadata_txt_pattern: Optional[str] = None,
                 max_connection_attempts: int = 3,
                 request_timeout: float = 300.0,
                 rate_controller: Optional[Callable[[InstaloaderContext], RateController]] = None,
                 resume_prefix: Optional[str] = "iterator",
                 check_resume_bbd: bool = True,
-                 slide: Optional[str] = None):
+                 slide: Optional[str] = None,
+                 fatal_status_codes: Optional[List[int]] = None,
+                 iphone_support: bool = True,
+                 title_pattern: Optional[str] = None,
+                 sanitize_paths: bool = False):

        self.context = InstaloaderContext(sleep, quiet, user_agent, max_connection_attempts,
-                                          request_timeout, rate_controller)
+                                          request_timeout, rate_controller, fatal_status_codes,
+                                          iphone_support)

        # configuration parameters
        self.dirname_pattern = dirname_pattern or "{target}"
        self.filename_pattern = filename_pattern or "{date_utc}_UTC"
+        if title_pattern is not None:
+            self.title_pattern = title_pattern
+        else:
+            if (format_string_contains_key(self.dirname_pattern, 'profile') or
+                format_string_contains_key(self.dirname_pattern, 'target')):
+                self.title_pattern = '{date_utc}_UTC_{typename}'
+            else:
+                self.title_pattern = '{target}_{date_utc}_UTC_{typename}'
+        self.sanitize_paths = sanitize_paths
        self.download_pictures = download_pictures
        self.download_videos = download_videos
        self.download_video_thumbnails = download_video_thumbnails
@@ -256,7 +311,10 @@ class Instaloader:
            request_timeout=self.context.request_timeout,
            resume_prefix=self.resume_prefix,
            check_resume_bbd=self.check_resume_bbd,
-            slide=self.slide)
+            slide=self.slide,
+            fatal_status_codes=self.context.fatal_status_codes,
+            iphone_support=self.context.iphone_support,
+            sanitize_paths=self.sanitize_paths)
        yield new_loader
        self.context.error_log.extend(new_loader.context.error_log)
        new_loader.context.error_log = []  # avoid double-printing of errors
@@ -277,15 +335,25 @@ class Instaloader:
                     filename_suffix: Optional[str] = None, _attempt: int = 1) -> bool:
        """Downloads and saves picture with given url under given directory with given timestamp.
        Returns true, if file was actually downloaded, i.e. updated."""
-        urlmatch = re.search('\\.[a-z0-9]*\\?', url)
-        file_extension = url[-3:] if urlmatch is None else urlmatch.group(0)[1:-1]
        if filename_suffix is not None:
            filename += '_' + filename_suffix
-        filename += '.' + file_extension
-        if os.path.isfile(filename):
+        urlmatch = re.search('\\.[a-z0-9]*\\?', url)
+        file_extension = url[-3:] if urlmatch is None else urlmatch.group(0)[1:-1]
+        nominal_filename = filename + '.' + file_extension
+        if os.path.isfile(nominal_filename):
+            self.context.log(nominal_filename + ' exists', end=' ', flush=True)
+            return False
+        resp = self.context.get_raw(url)
+        if 'Content-Type' in resp.headers and resp.headers['Content-Type']:
+            header_extension = '.' + resp.headers['Content-Type'].split(';')[0].split('/')[-1]
+            header_extension = header_extension.lower().replace('jpeg', 'jpg')
+            filename += header_extension
+        else:
+            filename = nominal_filename
+        if filename != nominal_filename and os.path.isfile(filename):
            self.context.log(filename + ' exists', end=' ', flush=True)
            return False
-        self.context.get_and_write_raw(url, filename)
+        self.context.write_raw(resp, filename)
        os.utime(filename, (datetime.now().timestamp(), mtime.timestamp()))
        return True

@@ -332,25 +400,54 @@ class Instaloader:
                            combined_answers.extend(y['answers'])
                        unique_comments_list[-1]['answers'] = get_unique_comments(combined_answers)
            return unique_comments_list
+
+        def get_new_comments(new_comments, start):
+            for idx, comment in enumerate(new_comments, start=start+1):
+                if idx % 250 == 0:
+                    self.context.log('{}'.format(idx), end='…', flush=True)
+                yield comment
+
+        def save_comments(extended_comments):
+            unique_comments = get_unique_comments(extended_comments, combine_answers=True)
+            answer_ids = set(int(answer['id']) for comment in unique_comments for answer in comment.get('answers', []))
+            with open(filename, 'w') as file:
+                file.write(json.dumps(list(filter(lambda t: int(t['id']) not in answer_ids, unique_comments)),
+                                      indent=4))
+
+        base_filename = filename
        filename += '_comments.json'
        try:
            with open(filename) as fp:
                comments = json.load(fp)
        except (FileNotFoundError, json.decoder.JSONDecodeError):
            comments = list()
-        comments.extend(_postcomment_asdict(comment) for comment in post.get_comments())
+
+        comments_iterator = post.get_comments()
+        try:
+            with resumable_iteration(
+                    context=self.context,
+                    iterator=comments_iterator,
+                    load=load_structure_from_file,
+                    save=save_structure_to_file,
+                    format_path=lambda magic: "{}_{}_{}.json.xz".format(base_filename, self.resume_prefix, magic),
+                    check_bbd=self.check_resume_bbd,
+                    enabled=self.resume_prefix is not None
+            ) as (_is_resuming, start_index):
+                comments.extend(_postcomment_asdict(comment)
+                                for comment in get_new_comments(comments_iterator, start_index))
+        except (KeyboardInterrupt, AbortDownloadException):
+            if comments:
+                save_comments(comments)
+            raise
        if comments:
-            comments = get_unique_comments(comments, combine_answers=True)
-            answer_ids = set(int(answer['id']) for comment in comments for answer in comment.get('answers', []))
-            with open(filename, 'w') as file:
-                file.write(json.dumps(list(filter(lambda t: int(t['id']) not in answer_ids, comments)), indent=4))
+            save_comments(comments)
            self.context.log('comments', end=' ', flush=True)

    def save_caption(self, filename: str, mtime: datetime, caption: str) -> None:
        """Updates picture caption / Post metadata info"""
        def _elliptify(caption):
            pcaption = caption.replace('\n', ' ').strip()
-            return '[' + ((pcaption[:29] + u"\u2026") if len(pcaption) > 31 else pcaption) + ']'
+            return '[' + ((pcaption[:29] + "\u2026") if len(pcaption) > 31 else pcaption) + ']'
        filename += '.txt'
        caption += '\n'
        pcaption = _elliptify(caption)
@@ -381,17 +478,19 @@ class Instaloader:
            self.context.log(pcaption, end=' ', flush=True)
        except UnicodeEncodeError:
            self.context.log('txt', end=' ', flush=True)
-        with open(filename, 'wb') as text_file:
-            with BytesIO(bcaption) as bio:
-                shutil.copyfileobj(cast(IO, bio), text_file)
+        with open(filename, 'w', encoding='UTF-8') as fio:
+            fio.write(caption)
        os.utime(filename, (datetime.now().timestamp(), mtime.timestamp()))

    def save_location(self, filename: str, location: PostLocation, mtime: datetime) -> None:
        """Save post location name and Google Maps link."""
        filename += '_location.txt'
-        location_string = (location.name + "\n" +
-                           "https://maps.google.com/maps?q={0},{1}&ll={0},{1}\n".format(location.lat,
-                                                                                        location.lng))
+        if location.lat is not None and location.lng is not None:
+            location_string = (location.name + "\n" +
+                               "https://maps.google.com/maps?q={0},{1}&ll={0},{1}\n".format(location.lat,
+                                                                                            location.lng))
+        else:
+            location_string = location.name
        with open(filename, 'wb') as text_file:
            with BytesIO(location_string.encode()) as bio:
                shutil.copyfileobj(cast(IO, bio), text_file)
@@ -424,31 +523,52 @@ class Instaloader:

        .. versionadded:: 4.3"""

-        def _epoch_to_string(epoch: datetime) -> str:
-            return epoch.strftime('%Y-%m-%d_%H-%M-%S_UTC')
-
        http_response = self.context.get_raw(url)
        date_object = None  # type: Optional[datetime]
        if 'Last-Modified' in http_response.headers:
            date_object = datetime.strptime(http_response.headers["Last-Modified"], '%a, %d %b %Y %H:%M:%S GMT')
+            date_object = date_object.replace(tzinfo=timezone.utc)
            pic_bytes = None
-            pic_identifier = _epoch_to_string(date_object)
        else:
            pic_bytes = http_response.content
-            pic_identifier = md5(pic_bytes).hexdigest()[:16]
-        filename = self.format_filename_within_target_path(target, owner_profile, pic_identifier, name_suffix, 'jpg')
+        ig_filename = url.split('/')[-1].split('?')[0]
+        pic_data = TitlePic(owner_profile, target, name_suffix, ig_filename, date_object)
+        dirname = _PostPathFormatter(pic_data, self.sanitize_paths).format(self.dirname_pattern, target=target)
+        filename_template = os.path.join(
+                dirname,
+                _PostPathFormatter(pic_data, self.sanitize_paths).format(self.title_pattern, target=target))
+        filename = self.__prepare_filename(filename_template, lambda: url) + ".jpg"
        content_length = http_response.headers.get('Content-Length', None)
        if os.path.isfile(filename) and (not self.context.is_logged_in or
                                         (content_length is not None and
                                          os.path.getsize(filename) >= int(content_length))):
            self.context.log(filename + ' already exists')
-            return None
+            return
        os.makedirs(os.path.dirname(filename), exist_ok=True)
        self.context.write_raw(pic_bytes if pic_bytes else http_response, filename)
        if date_object:
            os.utime(filename, (datetime.now().timestamp(), date_object.timestamp()))
        self.context.log('')  # log output of _get_and_write_raw() does not produce \n

+    def download_profilepic_if_new(self, profile: Profile, latest_stamps: Optional[LatestStamps]) -> None:
+        """
+        Downloads and saves profile pic if it has not been downloaded before.
+
+        :param latest_stamps: Database with the last downloaded data. If not present,
+               the profile pic is downloaded unless it already exists
+
+        .. versionadded:: 4.8
+        """
+        if latest_stamps is None:
+            self.download_profilepic(profile)
+            return
+        profile_pic_basename = profile.profile_pic_url_no_iphone.split('/')[-1].split('?')[0]
+        saved_basename = latest_stamps.get_profile_pic(profile.username)
+        if saved_basename == profile_pic_basename:
+            return
+        self.download_profilepic(profile)
+        latest_stamps.set_profile_pic(profile.username, profile_pic_basename)
+
    def download_profilepic(self, profile: Profile) -> None:
        """Downloads and saves profile pic."""
        self.download_title_pic(profile.profile_pic_url, profile.username.lower(), 'profile_pic', profile)
@@ -536,11 +656,12 @@ class Instaloader:
        os.makedirs(os.path.dirname(filename), exist_ok=True)
        return filename

-    def format_filename(self, item: Union[Post, StoryItem, PostSidecarNode], target: Optional[Union[str, Path]] = None):
+    def format_filename(self, item: Union[Post, StoryItem, PostSidecarNode, TitlePic],
+                        target: Optional[Union[str, Path]] = None):
        """Format filename of a :class:`Post` or :class:`StoryItem` according to ``filename-pattern`` parameter.

        .. versionadded:: 4.1"""
-        return _PostPathFormatter(item).format(self.filename_pattern, target=target)
+        return _PostPathFormatter(item, self.sanitize_paths).format(self.filename_pattern, target=target)

    def download_post(self, post: Post, target: Union[str, Path]) -> bool:
        """
@@ -551,7 +672,28 @@ class Instaloader:
        :return: True if something was downloaded, False otherwise, i.e. file was already there
        """

-        dirname = _PostPathFormatter(post).format(self.dirname_pattern, target=target)
+        def _already_downloaded(path: str) -> bool:
+            if not os.path.isfile(path):
+                return False
+            else:
+                self.context.log(path + ' exists', end=' ', flush=True)
+                return True
+
+        def _all_already_downloaded(path_base, is_videos_enumerated) -> bool:
+            if '{filename}' in self.filename_pattern:
+                # full URL needed to evaluate actual filename, cannot determine at
+                # this point if all sidecar nodes were already downloaded.
+                return False
+            for idx, is_video in is_videos_enumerated:
+                if self.download_pictures and (not is_video or self.download_video_thumbnails):
+                    if not _already_downloaded("{0}_{1}.jpg".format(path_base, idx)):
+                        return False
+                if is_video and self.download_videos:
+                    if not _already_downloaded("{0}_{1}.mp4".format(path_base, idx)):
+                        return False
+            return True
+
+        dirname = _PostPathFormatter(post, self.sanitize_paths).format(self.dirname_pattern, target=target)
        filename_template = os.path.join(dirname, self.format_filename(post, target=target))
        filename = self.__prepare_filename(filename_template, lambda: post.url)

@@ -559,34 +701,47 @@ class Instaloader:
        downloaded = True
        if post.typename == 'GraphSidecar':
            if self.download_pictures or self.download_videos:
-                for edge_number, sidecar_node in enumerate(
-                        post.get_sidecar_nodes(self.slide_start, self.slide_end),
-                        start=post.mediacount if self.slide_start < 0 else self.slide_start + 1
+                if not _all_already_downloaded(
+                        filename_template, enumerate(
+                            (post.get_is_videos()[i]
+                             for i in range(self.slide_start % post.mediacount, self.slide_end % post.mediacount + 1)),
+                            start=self.slide_start % post.mediacount + 1
+                        )
                ):
-                    suffix = str(edge_number)  # type: Optional[str]
-                    if '{filename}' in self.filename_pattern:
-                        suffix = None
-                    if self.download_pictures and (not sidecar_node.is_video or self.download_video_thumbnails):
-                        # pylint:disable=cell-var-from-loop
-                        sidecar_filename = self.__prepare_filename(filename_template, lambda: sidecar_node.display_url)
-                        # Download sidecar picture or video thumbnail (--no-pictures implies --no-video-thumbnails)
-                        downloaded &= self.download_pic(filename=sidecar_filename, url=sidecar_node.display_url,
-                                                        mtime=post.date_local, filename_suffix=suffix)
-                    if sidecar_node.is_video and self.download_videos:
-                        # pylint:disable=cell-var-from-loop
-                        sidecar_filename = self.__prepare_filename(filename_template, lambda: sidecar_node.video_url)
-                        # Download sidecar video if desired
-                        downloaded &= self.download_pic(filename=sidecar_filename, url=sidecar_node.video_url,
-                                                        mtime=post.date_local, filename_suffix=suffix)
+                    for edge_number, sidecar_node in enumerate(
+                            post.get_sidecar_nodes(self.slide_start, self.slide_end),
+                            start=self.slide_start % post.mediacount + 1
+                    ):
+                        suffix = str(edge_number)  # type: Optional[str]
+                        if '{filename}' in self.filename_pattern:
+                            suffix = None
+                        if self.download_pictures and (not sidecar_node.is_video or self.download_video_thumbnails):
+                            # pylint:disable=cell-var-from-loop
+                            sidecar_filename = self.__prepare_filename(filename_template,
+                                                                       lambda: sidecar_node.display_url)
+                            # Download sidecar picture or video thumbnail (--no-pictures implies --no-video-thumbnails)
+                            downloaded &= self.download_pic(filename=sidecar_filename, url=sidecar_node.display_url,
+                                                            mtime=post.date_local, filename_suffix=suffix)
+                        if sidecar_node.is_video and self.download_videos:
+                            # pylint:disable=cell-var-from-loop
+                            sidecar_filename = self.__prepare_filename(filename_template,
+                                                                       lambda: sidecar_node.video_url)
+                            # Download sidecar video if desired
+                            downloaded &= self.download_pic(filename=sidecar_filename, url=sidecar_node.video_url,
+                                                            mtime=post.date_local, filename_suffix=suffix)
+                else:
+                    downloaded = False
        elif post.typename == 'GraphImage':
            # Download picture
            if self.download_pictures:
-                downloaded = self.download_pic(filename=filename, url=post.url, mtime=post.date_local)
+                downloaded = (not _already_downloaded(filename + ".jpg") and
+                              self.download_pic(filename=filename, url=post.url, mtime=post.date_local))
        elif post.typename == 'GraphVideo':
            # Download video thumbnail (--no-pictures implies --no-video-thumbnails)
            if self.download_pictures and self.download_video_thumbnails:
                with self.context.error_catcher("Video thumbnail of {}".format(post)):
-                    downloaded = self.download_pic(filename=filename, url=post.url, mtime=post.date_local)
+                    downloaded = (not _already_downloaded(filename + ".jpg") and
+                                  self.download_pic(filename=filename, url=post.url, mtime=post.date_local))
        else:
            self.context.error("Warning: {0} has unknown typename: {1}".format(post, post.typename))

@@ -597,7 +752,8 @@ class Instaloader:

        # Download video if desired
        if post.is_video and self.download_videos:
-            downloaded &= self.download_pic(filename=filename, url=post.video_url, mtime=post.date_local)
+            downloaded &= (not _already_downloaded(filename + ".mp4") and
+                           self.download_pic(filename=filename, url=post.video_url, mtime=post.date_local))

        # Download geotags if desired
        if self.download_geotags and post.location:
@@ -647,7 +803,8 @@ class Instaloader:
                         userids: Optional[List[Union[int, Profile]]] = None,
                         fast_update: bool = False,
                         filename_target: Optional[str] = ':stories',
-                         storyitem_filter: Optional[Callable[[StoryItem], bool]] = None) -> None:
+                         storyitem_filter: Optional[Callable[[StoryItem], bool]] = None,
+                         latest_stamps: Optional[LatestStamps] = None) -> None:
        """
        Download available stories from user followees or all stories of users whose ID are given.
        Does not mark stories as seen.
@@ -658,20 +815,37 @@ class Instaloader:
        :param filename_target: Replacement for {target} in dirname_pattern and filename_pattern
               or None if profile name should be used instead
        :param storyitem_filter: function(storyitem), which returns True if given StoryItem should be downloaded
+        :param latest_stamps: Database with the last times each user was scraped
        :raises LoginRequiredException: If called without being logged in.
+
+        .. versionchanged:: 4.8
+           Add `latest_stamps` parameter.
        """

        if not userids:
            self.context.log("Retrieving all visible stories...")
+            profile_count = None
        else:
            userids = [p if isinstance(p, int) else p.userid for p in userids]
+            profile_count = len(userids)

-        for user_story in self.get_stories(userids):
+        for i, user_story in enumerate(self.get_stories(userids), start=1):
            name = user_story.owner_username
-            self.context.log("Retrieving stories from profile {}.".format(name))
+            if profile_count is not None:
+                msg = "[{0:{w}d}/{1:{w}d}] Retrieving stories from profile {2}.".format(i, profile_count, name,
+                                                                                        w=len(str(profile_count)))
+            else:
+                msg = "[{:3d}] Retrieving stories from profile {}.".format(i, name)
+            self.context.log(msg)
            totalcount = user_story.itemcount
            count = 1
+            if latest_stamps is not None:
+                # pylint:disable=cell-var-from-loop
+                last_scraped = latest_stamps.get_last_story_timestamp(name)
+                scraped_timestamp = datetime.now().astimezone()
            for item in user_story.get_items():
+                if latest_stamps is not None and item.date_local <= last_scraped:
+                    break
                if storyitem_filter is not None and not storyitem_filter(item):
                    self.context.log("<{} skipped>".format(item), flush=True)
                    continue
@@ -681,6 +855,8 @@ class Instaloader:
                    downloaded = self.download_storyitem(item, filename_target if filename_target else name)
                    if fast_update and not downloaded:
                        break
+            if latest_stamps is not None:
+                latest_stamps.set_last_story_timestamp(name, scraped_timestamp)

    def download_storyitem(self, item: StoryItem, target: Union[str, Path]) -> bool:
        """Download one user story.
@@ -690,17 +866,30 @@ class Instaloader:
        :return: True if something was downloaded, False otherwise, i.e. file was already there
        """

+        def _already_downloaded(path: str) -> bool:
+            if not os.path.isfile(path):
+                return False
+            else:
+                self.context.log(path + ' exists', end=' ', flush=True)
+                return True
+
        date_local = item.date_local
-        dirname = _PostPathFormatter(item).format(self.dirname_pattern, target=target)
+        dirname = _PostPathFormatter(item, self.sanitize_paths).format(self.dirname_pattern, target=target)
        filename_template = os.path.join(dirname, self.format_filename(item, target=target))
        filename = self.__prepare_filename(filename_template, lambda: item.url)
        downloaded = False
-        if not item.is_video or self.download_video_thumbnails is True:
-            url = item.url
-            downloaded = self.download_pic(filename=filename, url=url, mtime=date_local)
+        video_url_fetch_failed = False
        if item.is_video and self.download_videos is True:
-            filename = self.__prepare_filename(filename_template, lambda: str(item.video_url))
-            downloaded |= self.download_pic(filename=filename, url=item.video_url, mtime=date_local)
+            video_url = item.video_url
+            if video_url:
+                filename = self.__prepare_filename(filename_template, lambda: str(video_url))
+                downloaded |= (not _already_downloaded(filename + ".mp4") and
+                               self.download_pic(filename=filename, url=video_url, mtime=date_local))
+            else:
+                video_url_fetch_failed = True
+        if video_url_fetch_failed or not item.is_video or self.download_video_thumbnails is True:
+            downloaded = (not _already_downloaded(filename + ".jpg") and
+                          self.download_pic(filename=filename, url=item.url, mtime=date_local))
        # Save caption if desired
        metadata_string = _ArbitraryItemFormatter(item).format(self.storyitem_metadata_txt_pattern).strip()
        if metadata_string:
@@ -758,8 +947,9 @@ class Instaloader:
            name = user_highlight.owner_username
            highlight_target = (filename_target
                                if filename_target
-                                else (Path(_PostPathFormatter.sanitize_path(name)) /
-                                      _PostPathFormatter.sanitize_path(user_highlight.title)))  # type: Union[str, Path]
+                                else (Path(_PostPathFormatter.sanitize_path(name, self.sanitize_paths)) /
+                                      _PostPathFormatter.sanitize_path(user_highlight.title,
+                                                                       self.sanitize_paths)))  # type: Union[str, Path]
            self.context.log("Retrieving highlights \"{}\" from profile {}".format(user_highlight.title, name))
            self.download_highlight_cover(user_highlight, highlight_target)
            totalcount = user_highlight.itemcount
@@ -783,7 +973,8 @@ class Instaloader:
                            post_filter: Optional[Callable[[Post], bool]] = None,
                            max_count: Optional[int] = None,
                            total_count: Optional[int] = None,
-                            owner_profile: Optional[Profile] = None) -> None:
+                            owner_profile: Optional[Profile] = None,
+                            takewhile: Optional[Callable[[Post], bool]] = None) -> None:
        """
        Download the Posts returned by given Post Iterator.

@@ -792,6 +983,9 @@ class Instaloader:
        .. versionchanged:: 4.5
           Transparently resume an aborted operation if `posts` is a :class:`NodeIterator`.

+        .. versionchanged:: 4.8
+           Add `takewhile` parameter.
+
        :param posts: Post Iterator to loop through.
        :param target: Target name.
        :param fast_update: :option:`--fast-update`.
@@ -799,12 +993,15 @@ class Instaloader:
        :param max_count: Maximum count of Posts to download (:option:`--count`).
        :param total_count: Total number of posts returned by given iterator.
        :param owner_profile: Associated profile, if any.
+        :param takewhile: Expression evaluated for each post. Once it returns false, downloading stops.
        """
        displayed_count = (max_count if total_count is None or max_count is not None and max_count < total_count
                           else total_count)
        sanitized_target = target
        if isinstance(target, str):
-            sanitized_target = _PostPathFormatter.sanitize_path(target)
+            sanitized_target = _PostPathFormatter.sanitize_path(target, self.sanitize_paths)
+        if takewhile is None:
+            takewhile = lambda _: True
        with resumable_iteration(
                context=self.context,
                iterator=posts,
@@ -817,7 +1014,10 @@ class Instaloader:
                enabled=self.resume_prefix is not None
        ) as (is_resuming, start_index):
            for number, post in enumerate(posts, start=start_index + 1):
-                if max_count is not None and number > max_count:
+                should_stop = not takewhile(post)
+                if should_stop and post.is_pinned:
+                    continue
+                if (max_count is not None and number > max_count) or should_stop:
                    break
                if displayed_count is not None:
                    self.context.log("[{0:{w}d}/{1:{w}d}] ".format(number, displayed_count,
@@ -849,7 +1049,7 @@ class Instaloader:
                        except PostChangedException:
                            post_changed = True
                            continue
-                    if fast_update and not downloaded and not post_changed:
+                    if fast_update and not downloaded and not post_changed and not post.is_pinned:
                        # disengage fast_update for first post when resuming
                        if not is_resuming or number > 0:
                            break
@@ -880,7 +1080,7 @@ class Instaloader:
                                               'has_stories': False})["data"]

    @_requires_login
-    def download_feed_posts(self, max_count: int = None, fast_update: bool = False,
+    def download_feed_posts(self, max_count: Optional[int] = None, fast_update: bool = False,
                            post_filter: Optional[Callable[[Post], bool]] = None) -> None:
        """
        Download pictures from the user's feed.
@@ -901,7 +1101,7 @@ class Instaloader:
        self.posts_download_loop(self.get_feed_posts(), ":feed", fast_update, post_filter, max_count=max_count)

    @_requires_login
-    def download_saved_posts(self, max_count: int = None, fast_update: bool = False,
+    def download_saved_posts(self, max_count: Optional[int] = None, fast_update: bool = False,
                             post_filter: Optional[Callable[[Post], bool]] = None) -> None:
        """Download user's saved pictures.

@@ -929,18 +1129,12 @@ class Instaloader:
        .. versionchanged:: 4.2.9
           Require being logged in (as required by Instagram)
        """
-        has_next_page = True
-        end_cursor = None
-        while has_next_page:
-            if end_cursor:
-                params = {'__a': 1, 'max_id': end_cursor}
-            else:
-                params = {'__a': 1}
-            location_data = self.context.get_json('explore/locations/{0}/'.format(location),
-                                                  params)['graphql']['location']['edge_location_to_media']
-            yield from (Post(self.context, edge['node']) for edge in location_data['edges'])
-            has_next_page = location_data['page_info']['has_next_page']
-            end_cursor = location_data['page_info']['end_cursor']
+        yield from SectionIterator(
+            self.context,
+            lambda d: d["native_location_data"]["recent"],
+            lambda m: Post.from_iphone_struct(self.context, m),
+            f"explore/locations/{location}/",
+        )

    @_requires_login
    def download_location(self, location: str,
@@ -989,8 +1183,8 @@ class Instaloader:
        """Get Posts associated with a #hashtag.

        .. deprecated:: 4.4
-           Use :meth:`Hashtag.get_posts`."""
-        return Hashtag.from_name(self.context, hashtag).get_posts()
+           Use :meth:`Hashtag.get_posts_resumable`."""
+        return Hashtag.from_name(self.context, hashtag).get_posts_resumable()

    def download_hashtag(self, hashtag: Union[Hashtag, str],
                         max_count: Optional[int] = None,
@@ -1026,7 +1220,7 @@ class Instaloader:
                self.download_hashtag_profilepic(hashtag)
        if posts:
            self.context.log("Retrieving pictures with hashtag #{}...".format(hashtag.name))
-            self.posts_download_loop(hashtag.get_all_posts(), target, fast_update, post_filter,
+            self.posts_download_loop(hashtag.get_posts_resumable(), target, fast_update, post_filter,
                                     max_count=max_count)
        if self.save_metadata:
            json_filename = '{0}/{1}'.format(self.dirname_pattern.format(profile=target,
@@ -1036,25 +1230,47 @@ class Instaloader:

    def download_tagged(self, profile: Profile, fast_update: bool = False,
                        target: Optional[str] = None,
-                        post_filter: Optional[Callable[[Post], bool]] = None) -> None:
+                        post_filter: Optional[Callable[[Post], bool]] = None,
+                        latest_stamps: Optional[LatestStamps] = None) -> None:
        """Download all posts where a profile is tagged.

-        .. versionadded:: 4.1"""
+        .. versionadded:: 4.1
+
+        .. versionchanged:: 4.8
+           Add `latest_stamps` parameter."""
        self.context.log("Retrieving tagged posts for profile {}.".format(profile.username))
-        self.posts_download_loop(profile.get_tagged_posts(),
+        posts_takewhile: Optional[Callable[[Post], bool]] = None
+        if latest_stamps is not None:
+            last_scraped = latest_stamps.get_last_tagged_timestamp(profile.username)
+            posts_takewhile = lambda p: p.date_local > last_scraped
+        tagged_posts = profile.get_tagged_posts()
+        self.posts_download_loop(tagged_posts,
                                 target if target
-                                 else (Path(_PostPathFormatter.sanitize_path(profile.username)) /
-                                       _PostPathFormatter.sanitize_path(':tagged')),
-                                 fast_update, post_filter)
+                                 else (Path(_PostPathFormatter.sanitize_path(profile.username, self.sanitize_paths)) /
+                                       _PostPathFormatter.sanitize_path(':tagged', self.sanitize_paths)),
+                                 fast_update, post_filter, takewhile=posts_takewhile)
+        if latest_stamps is not None and tagged_posts.first_item is not None:
+            latest_stamps.set_last_tagged_timestamp(profile.username, tagged_posts.first_item.date_local)

    def download_igtv(self, profile: Profile, fast_update: bool = False,
-                      post_filter: Optional[Callable[[Post], bool]] = None) -> None:
+                      post_filter: Optional[Callable[[Post], bool]] = None,
+                      latest_stamps: Optional[LatestStamps] = None) -> None:
        """Download IGTV videos of a profile.

-        .. versionadded:: 4.3"""
+        .. versionadded:: 4.3
+
+        .. versionchanged:: 4.8
+           Add `latest_stamps` parameter."""
        self.context.log("Retrieving IGTV videos for profile {}.".format(profile.username))
-        self.posts_download_loop(profile.get_igtv_posts(), profile.username, fast_update, post_filter,
-                                 total_count=profile.igtvcount, owner_profile=profile)
+        posts_takewhile: Optional[Callable[[Post], bool]] = None
+        if latest_stamps is not None:
+            last_scraped = latest_stamps.get_last_igtv_timestamp(profile.username)
+            posts_takewhile = lambda p: p.date_local > last_scraped
+        igtv_posts = profile.get_igtv_posts()
+        self.posts_download_loop(igtv_posts, profile.username, fast_update, post_filter,
+                                 total_count=profile.igtvcount, owner_profile=profile, takewhile=posts_takewhile)
+        if latest_stamps is not None and igtv_posts.first_item is not None:
+            latest_stamps.set_last_igtv_timestamp(profile.username, igtv_posts.first_item.date_local)

    def _get_id_filename(self, profile_name: str) -> str:
        if ((format_string_contains_key(self.dirname_pattern, 'profile') or
@@ -1066,9 +1282,22 @@ class Instaloader:
            return os.path.join(self.dirname_pattern.format(),
                                '{0}_id'.format(profile_name.lower()))

+    def load_profile_id(self, profile_name: str) -> Optional[int]:
+        """
+        Load ID of profile from profile directory.
+
+        .. versionadded:: 4.8
+        """
+        id_filename = self._get_id_filename(profile_name)
+        try:
+            with open(id_filename, 'rb') as id_file:
+                return int(id_file.read())
+        except (FileNotFoundError, ValueError):
+            return None
+
    def save_profile_id(self, profile: Profile):
        """
-        Store ID of profile locally.
+        Store ID of profile on profile directory.

        .. versionadded:: 4.0.6
        """
@@ -1078,13 +1307,18 @@ class Instaloader:
            text_file.write(str(profile.userid) + "\n")
            self.context.log("Stored ID {0} for profile {1}.".format(profile.userid, profile.username))

-    def check_profile_id(self, profile_name: str) -> Profile:
+    def check_profile_id(self, profile_name: str, latest_stamps: Optional[LatestStamps] = None) -> Profile:
        """
        Consult locally stored ID of profile with given name, check whether ID matches and whether name
        has changed and return current name of the profile, and store ID of profile.

        :param profile_name: Profile name
+        :param latest_stamps: Database of downloaded data. If present, IDs are retrieved from it,
+               otherwise from the target directory
        :return: Instance of current profile
+
+        .. versionchanged:: 4.8
+           Add `latest_stamps` parameter.
        """
        profile = None
        profile_name_not_exists_err = None
@@ -1092,10 +1326,11 @@ class Instaloader:
            profile = Profile.from_username(self.context, profile_name)
        except ProfileNotExistsException as err:
            profile_name_not_exists_err = err
-        id_filename = self._get_id_filename(profile_name)
-        try:
-            with open(id_filename, 'rb') as id_file:
-                profile_id = int(id_file.read())
+        if latest_stamps is None:
+            profile_id = self.load_profile_id(profile_name)
+        else:
+            profile_id = latest_stamps.get_profile_id(profile_name)
+        if profile_id is not None:
            if (profile is None) or \
                    (profile_id != profile.userid):
                if profile is not None:
@@ -1106,23 +1341,31 @@ class Instaloader:
                                                                                                  profile_id))
                profile_from_id = Profile.from_id(self.context, profile_id)
                newname = profile_from_id.username
-                self.context.log("Profile {0} has changed its name to {1}.".format(profile_name, newname))
-                if ((format_string_contains_key(self.dirname_pattern, 'profile') or
-                     format_string_contains_key(self.dirname_pattern, 'target'))):
-                    os.rename(self.dirname_pattern.format(profile=profile_name.lower(),
-                                                          target=profile_name.lower()),
-                              self.dirname_pattern.format(profile=newname.lower(),
-                                                          target=newname.lower()))
+                if profile_name == newname:
+                    self.context.error(
+                        f"Warning: Profile {profile_name} could not be retrieved by its name, but by its ID.")
+                    return profile_from_id
+                self.context.error("Profile {0} has changed its name to {1}.".format(profile_name, newname))
+                if latest_stamps is None:
+                    if ((format_string_contains_key(self.dirname_pattern, 'profile') or
+                         format_string_contains_key(self.dirname_pattern, 'target'))):
+                        os.rename(self.dirname_pattern.format(profile=profile_name.lower(),
+                                                              target=profile_name.lower()),
+                                  self.dirname_pattern.format(profile=newname.lower(),
+                                                              target=newname.lower()))
+                    else:
+                        os.rename('{0}/{1}_id'.format(self.dirname_pattern.format(), profile_name.lower()),
+                                  '{0}/{1}_id'.format(self.dirname_pattern.format(), newname.lower()))
                else:
-                    os.rename('{0}/{1}_id'.format(self.dirname_pattern.format(), profile_name.lower()),
-                              '{0}/{1}_id'.format(self.dirname_pattern.format(), newname.lower()))
+                    latest_stamps.rename_profile(profile_name, newname)
                return profile_from_id
            # profile exists and profile id matches saved id
            return profile
-        except (FileNotFoundError, ValueError):
-            pass
        if profile is not None:
-            self.save_profile_id(profile)
+            if latest_stamps is None:
+                self.save_profile_id(profile)
+            else:
+                latest_stamps.save_profile_id(profile.username, profile.userid)
            return profile
        if profile_name_not_exists_err:
            raise profile_name_not_exists_err
@@ -1137,7 +1380,8 @@ class Instaloader:
                          fast_update: bool = False,
                          post_filter: Optional[Callable[[Post], bool]] = None,
                          storyitem_filter: Optional[Callable[[Post], bool]] = None,
-                          raise_errors: bool = False):
+                          raise_errors: bool = False,
+                          latest_stamps: Optional[LatestStamps] = None):
        """High-level method to download set of profiles.

        :param profiles: Set of profiles to download.
@@ -1153,11 +1397,15 @@ class Instaloader:
        :param raise_errors:
           Whether :exc:`LoginRequiredException` and :exc:`PrivateProfileNotFollowedException` should be raised or
           catched and printed with :meth:`InstaloaderContext.error_catcher`.
+        :param latest_stamps: :option:`--latest-stamps`.

        .. versionadded:: 4.1

        .. versionchanged:: 4.3
           Add `igtv` parameter.
+
+        .. versionchanged:: 4.8
+           Add `latest_stamps` parameter.
        """

        @contextmanager
@@ -1167,14 +1415,16 @@ class Instaloader:
        # error_handler type is Callable[[Optional[str]], ContextManager[None]] (not supported with Python 3.5.0..3.5.3)
        error_handler = _error_raiser if raise_errors else self.context.error_catcher

-        for profile in profiles:
+        for i, profile in enumerate(profiles, start=1):
+            self.context.log("[{0:{w}d}/{1:{w}d}] Downloading profile {2}".format(i, len(profiles), profile.username,
+                                                                                  w=len(str(len(profiles)))))
            with error_handler(profile.username):  # type: ignore # (ignore type for Python 3.5 support)
                profile_name = profile.username

                # Download profile picture
                if profile_pic:
                    with self.context.error_catcher('Download profile picture of {}'.format(profile_name)):
-                        self.download_profilepic(profile)
+                        self.download_profilepic_if_new(profile, latest_stamps)

                # Save metadata as JSON if desired.
                if self.save_metadata:
@@ -1196,12 +1446,14 @@ class Instaloader:
                # Download tagged, if requested
                if tagged:
                    with self.context.error_catcher('Download tagged of {}'.format(profile_name)):
-                        self.download_tagged(profile, fast_update=fast_update, post_filter=post_filter)
+                        self.download_tagged(profile, fast_update=fast_update, post_filter=post_filter,
+                                             latest_stamps=latest_stamps)

                # Download IGTV, if requested
                if igtv:
                    with self.context.error_catcher('Download IGTV of {}'.format(profile_name)):
-                        self.download_igtv(profile, fast_update=fast_update, post_filter=post_filter)
+                        self.download_igtv(profile, fast_update=fast_update, post_filter=post_filter,
+                                           latest_stamps=latest_stamps)

                # Download highlights, if requested
                if highlights:
@@ -1211,14 +1463,24 @@ class Instaloader:
                # Iterate over pictures and download them
                if posts:
                    self.context.log("Retrieving posts from profile {}.".format(profile_name))
-                    self.posts_download_loop(profile.get_posts(), profile_name, fast_update, post_filter,
-                                             total_count=profile.mediacount, owner_profile=profile)
+                    posts_takewhile: Optional[Callable[[Post], bool]] = None
+                    if latest_stamps is not None:
+                        # pylint:disable=cell-var-from-loop
+                        last_scraped = latest_stamps.get_last_post_timestamp(profile_name)
+                        posts_takewhile = lambda p: p.date_local > last_scraped
+                    posts_to_download = profile.get_posts()
+                    self.posts_download_loop(posts_to_download, profile_name, fast_update, post_filter,
+                                             total_count=profile.mediacount, owner_profile=profile,
+                                             takewhile=posts_takewhile)
+                    if latest_stamps is not None and posts_to_download.first_item is not None:
+                        latest_stamps.set_last_post_timestamp(profile_name,
+                                                              posts_to_download.first_item.date_local)

        if stories and profiles:
            with self.context.error_catcher("Download stories"):
                self.context.log("Downloading stories")
                self.download_stories(userids=list(profiles), fast_update=fast_update, filename_target=None,
-                                      storyitem_filter=storyitem_filter)
+                                      storyitem_filter=storyitem_filter, latest_stamps=latest_stamps)

    def download_profile(self, profile_name: Union[str, Profile],
                         profile_pic: bool = True, profile_pic_only: bool = False,
@@ -1317,5 +1579,6 @@ class Instaloader:
                    code = input("Enter 2FA verification code: ")
                    self.two_factor_login(code)
                    break
-                except BadCredentialsException:
+                except BadCredentialsException as err:
+                    print(err, file=sys.stderr)
                    pass
@@ -24,16 +24,16 @@ def copy_session(session: requests.Session, request_timeout: Optional[float] = N
    """Duplicates a requests.Session."""
    new = requests.Session()
    new.cookies = requests.utils.cookiejar_from_dict(requests.utils.dict_from_cookiejar(session.cookies))
-    new.headers = session.headers.copy()
+    new.headers = session.headers.copy()  # type: ignore
    # Override default timeout behavior.
    # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
-    new.request = partial(new.request, timeout=request_timeout) # type: ignore
+    new.request = partial(new.request, timeout=request_timeout)  # type: ignore
    return new


 def default_user_agent() -> str:
    return 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ' \
-           '(KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36'
+           '(KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36'


 class InstaloaderContext:
@@ -53,7 +53,9 @@ class InstaloaderContext:

    def __init__(self, sleep: bool = True, quiet: bool = False, user_agent: Optional[str] = None,
                 max_connection_attempts: int = 3, request_timeout: float = 300.0,
-                 rate_controller: Optional[Callable[["InstaloaderContext"], "RateController"]] = None):
+                 rate_controller: Optional[Callable[["InstaloaderContext"], "RateController"]] = None,
+                 fatal_status_codes: Optional[List[int]] = None,
+                 iphone_support: bool = True):

        self.user_agent = user_agent if user_agent is not None else default_user_agent()
        self.request_timeout = request_timeout
@@ -65,6 +67,7 @@ class InstaloaderContext:
        self._graphql_page_length = 50
        self._root_rhx_gis = None
        self.two_factor_auth_pending = None
+        self.iphone_support = iphone_support

        # error log, filled with error() and printed at the end of Instaloader.main()
        self.error_log = []                      # type: List[str]
@@ -74,6 +77,9 @@ class InstaloaderContext:
        # Can be set to True for testing, disables supression of InstaloaderContext._error_catcher
        self.raise_all_errors = False

+        # HTTP status codes that should cause an AbortDownloadException
+        self.fatal_status_codes = fatal_status_codes or []
+
        # Cache profile from id (mapping from id to Profile)
        self.profile_id_cache = dict()           # type: Dict[int, Any]

@@ -148,7 +154,6 @@ class InstaloaderContext:
        if empty_session_only:
            del header['Host']
            del header['Origin']
-            del header['Referer']
            del header['X-Instagram-AJAX']
            del header['X-Requested-With']
        return header
@@ -206,8 +211,8 @@ class InstaloaderContext:
        # Override default timeout behavior.
        # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
        session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
-        session.get('https://www.instagram.com/web/__mid/')
-        csrf_token = session.cookies.get_dict()['csrftoken']
+        csrf_json = self.get_json('accounts/login/', {}, session=session)
+        csrf_token = csrf_json['config']['csrf_token']
        session.headers.update({'X-CSRFToken': csrf_token})
        # Not using self.get_json() here, because we need to access csrftoken cookie
        self.do_sleep()
@@ -218,8 +223,10 @@ class InstaloaderContext:
                             data={'enc_password': enc_password, 'username': user}, allow_redirects=True)
        try:
            resp_json = login.json()
-        except json.decoder.JSONDecodeError:
-            raise ConnectionException("Login error: JSON decode fail, {} - {}.".format(login.status_code, login.reason))
+        except json.decoder.JSONDecodeError as err:
+            raise ConnectionException(
+                "Login error: JSON decode fail, {} - {}.".format(login.status_code, login.reason)
+            ) from err
        if resp_json.get('two_factor_required'):
            two_factor_session = copy_session(session, self.request_timeout)
            two_factor_session.headers.update({'X-CSRFToken': csrf_token})
@@ -277,9 +284,9 @@ class InstaloaderContext:
        resp_json = login.json()
        if resp_json['status'] != 'ok':
            if 'message' in resp_json:
-                raise BadCredentialsException("Login error: {}".format(resp_json['message']))
+                raise BadCredentialsException("2FA error: {}".format(resp_json['message']))
            else:
-                raise BadCredentialsException("Login error: \"{}\" status.".format(resp_json['status']))
+                raise BadCredentialsException("2FA error: \"{}\" status.".format(resp_json['status']))
        session.headers.update({'X-CSRFToken': login.cookies['csrftoken']})
        self._session = session
        self.username = user
@@ -316,10 +323,16 @@ class InstaloaderContext:
            if is_other_query:
                self._rate_controller.wait_before_query('other')
            resp = sess.get('https://{0}/{1}'.format(host, path), params=params, allow_redirects=False)
+            if resp.status_code in self.fatal_status_codes:
+                redirect = " redirect to {}".format(resp.headers['location']) if 'location' in resp.headers else ""
+                raise AbortDownloadException("Query to https://{}/{} responded with \"{} {}\"{}".format(
+                    host, path, resp.status_code, resp.reason, redirect
+                ))
            while resp.is_redirect:
                redirect_url = resp.headers['location']
                self.log('\nHTTP redirect from https://{0}/{1} to {2}'.format(host, path, redirect_url))
-                if redirect_url.startswith('https://www.instagram.com/accounts/login'):
+                if (redirect_url.startswith('https://www.instagram.com/accounts/login') or
+                    redirect_url.startswith('https://i.instagram.com/accounts/login')):
                    if not self.is_logged_in:
                        raise LoginRequiredException("Redirected to login page. Use --login.")
                    # alternate rate limit exceeded behavior
@@ -517,6 +530,28 @@ class InstaloaderContext:
        :raises ConnectionException: When download repeatedly failed."""
        self.write_raw(self.get_raw(url), filename)

+    def head(self, url: str, allow_redirects: bool = False) -> requests.Response:
+        """HEAD a URL anonymously.
+
+        :raises QueryReturnedNotFoundException: When the server responds with a 404.
+        :raises QueryReturnedForbiddenException: When the server responds with a 403.
+        :raises ConnectionException: When request failed.
+
+        .. versionadded:: 4.7.6
+        """
+        with self.get_anonymous_session() as anonymous_session:
+            resp = anonymous_session.head(url, allow_redirects=allow_redirects)
+        if resp.status_code == 200:
+            return resp
+        else:
+            if resp.status_code == 403:
+                # suspected invalid URL signature
+                raise QueryReturnedForbiddenException("403 when accessing {}.".format(url))
+            if resp.status_code == 404:
+                # 404 not worth retrying.
+                raise QueryReturnedNotFoundException("404 when accessing {}.".format(url))
+            raise ConnectionException("HTTP error code {}.".format(resp.status_code))
+
    @property
    def root_rhx_gis(self) -> Optional[str]:
        """rhx_gis string returned in the / query."""
@@ -555,7 +590,6 @@ class RateController:
        """Wait given number of seconds."""
        # Not static, to allow for the behavior of this method to depend on context-inherent properties, such as
        # whether we are logged in.
-        # pylint:disable=no-self-use
        time.sleep(secs)

    def _dump_query_timestamps(self, current_time: float, failed_query_type: str):
@@ -579,7 +613,6 @@ class RateController:
        control."""
        # Not static, to allow for the count_per_sliding_window to depend on context-inherent properties, such as
        # whether we are logged in.
-        # pylint:disable=no-self-use
        return 75 if query_type == 'other' else 200

    def _reqs_in_sliding_window(self, query_type: Optional[str], current_time: float, window: float) -> List[float]:
@@ -0,0 +1,113 @@
+import configparser
+from datetime import datetime, timezone
+from typing import Optional
+
+
+class LatestStamps:
+    """LatestStamps class.
+
+    Convenience class for retrieving and storing data from the :option:`--latest-stamps` file.
+
+    :param latest_stamps_file: path to file.
+
+    .. versionadded:: 4.8"""
+    PROFILE_ID = 'profile-id'
+    PROFILE_PIC = 'profile-pic'
+    POST_TIMESTAMP = 'post-timestamp'
+    TAGGED_TIMESTAMP = 'tagged-timestamp'
+    IGTV_TIMESTAMP = 'igtv-timestamp'
+    STORY_TIMESTAMP = 'story-timestamp'
+    ISO_FORMAT = '%Y-%m-%dT%H:%M:%S.%f%z'
+
+    def __init__(self, latest_stamps_file):
+        self.file = latest_stamps_file
+        self.data = configparser.ConfigParser()
+        self.data.read(latest_stamps_file)
+
+    def _save(self):
+        with open(self.file, 'w') as f:
+            self.data.write(f)
+
+    def _ensure_section(self, section: str):
+        if not self.data.has_section(section):
+            self.data.add_section(section)
+
+    def get_profile_id(self, profile_name: str) -> Optional[int]:
+        """Returns stored ID of profile."""
+        try:
+            return self.data.getint(profile_name, self.PROFILE_ID)
+        except (configparser.Error, ValueError):
+            return None
+
+    def save_profile_id(self, profile_name: str, profile_id: int):
+        """Stores ID of profile."""
+        self._ensure_section(profile_name)
+        self.data.set(profile_name, self.PROFILE_ID, str(profile_id))
+        self._save()
+
+    def rename_profile(self, old_profile: str, new_profile: str):
+        """Renames a profile."""
+        self._ensure_section(new_profile)
+        for option in [self.PROFILE_ID, self.PROFILE_PIC, self.POST_TIMESTAMP,
+                       self.TAGGED_TIMESTAMP, self.IGTV_TIMESTAMP, self.STORY_TIMESTAMP]:
+            if self.data.has_option(old_profile, option):
+                value = self.data.get(old_profile, option)
+                self.data.set(new_profile, option, value)
+        self.data.remove_section(old_profile)
+        self._save()
+
+    def _get_timestamp(self, section: str, key: str) -> datetime:
+        try:
+            return datetime.strptime(self.data.get(section, key), self.ISO_FORMAT)
+        except (configparser.Error, ValueError):
+            return datetime.fromtimestamp(0, timezone.utc)
+
+    def _set_timestamp(self, section: str, key: str, timestamp: datetime):
+        self._ensure_section(section)
+        self.data.set(section, key, timestamp.strftime(self.ISO_FORMAT))
+        self._save()
+
+    def get_last_post_timestamp(self, profile_name: str) -> datetime:
+        """Returns timestamp of last download of a profile's posts."""
+        return self._get_timestamp(profile_name, self.POST_TIMESTAMP)
+
+    def set_last_post_timestamp(self, profile_name: str, timestamp: datetime):
+        """Sets timestamp of last download of a profile's posts."""
+        self._set_timestamp(profile_name, self.POST_TIMESTAMP, timestamp)
+
+    def get_last_tagged_timestamp(self, profile_name: str) -> datetime:
+        """Returns timestamp of last download of a profile's tagged posts."""
+        return self._get_timestamp(profile_name, self.TAGGED_TIMESTAMP)
+
+    def set_last_tagged_timestamp(self, profile_name: str, timestamp: datetime):
+        """Sets timestamp of last download of a profile's tagged posts."""
+        self._set_timestamp(profile_name, self.TAGGED_TIMESTAMP, timestamp)
+
+    def get_last_igtv_timestamp(self, profile_name: str) -> datetime:
+        """Returns timestamp of last download of a profile's igtv posts."""
+        return self._get_timestamp(profile_name, self.IGTV_TIMESTAMP)
+
+    def set_last_igtv_timestamp(self, profile_name: str, timestamp: datetime):
+        """Sets timestamp of last download of a profile's igtv posts."""
+        self._set_timestamp(profile_name, self.IGTV_TIMESTAMP, timestamp)
+
+    def get_last_story_timestamp(self, profile_name: str) -> datetime:
+        """Returns timestamp of last download of a profile's stories."""
+        return self._get_timestamp(profile_name, self.STORY_TIMESTAMP)
+
+    def set_last_story_timestamp(self, profile_name: str, timestamp: datetime):
+        """Sets timestamp of last download of a profile's stories."""
+        self._set_timestamp(profile_name, self.STORY_TIMESTAMP, timestamp)
+
+    def get_profile_pic(self, profile_name: str) -> str:
+        """Returns filename of profile's last downloaded profile pic."""
+        try:
+            return self.data.get(profile_name, self.PROFILE_PIC)
+        except configparser.Error:
+            return ""
+
+    def set_profile_pic(self, profile_name: str, profile_pic: str):
+        """Sets filename of profile's last downloaded profile pic."""
+        self._ensure_section(profile_name)
+        self.data.set(profile_name, self.PROFILE_PIC, profile_pic)
+        self._save()
@@ -5,9 +5,9 @@ import os
 from contextlib import contextmanager
 from datetime import datetime, timedelta
 from lzma import LZMAError
-from typing import Any, Callable, Dict, Iterator, NamedTuple, Optional, Tuple, TypeVar
+from typing import Any, Callable, Dict, Iterable, Iterator, NamedTuple, Optional, Tuple, TypeVar

-from .exceptions import InvalidArgumentException, QueryReturnedBadRequestException
+from .exceptions import AbortDownloadException, InvalidArgumentException, QueryReturnedBadRequestException
 from .instaloadercontext import InstaloaderContext

 FrozenNodeIterator = NamedTuple('FrozenNodeIterator',
@@ -17,9 +17,8 @@ FrozenNodeIterator = NamedTuple('FrozenNodeIterator',
                                 ('context_username', Optional[str]),
                                 ('total_index', int),
                                 ('best_before', Optional[float]),
-                                 ('remaining_data', Optional[Dict])])
-FrozenNodeIterator.__doc__ = \
-    """A serializable representation of a :class:`NodeIterator` instance, saving its iteration state."""
+                                 ('remaining_data', Optional[Dict]),
+                                 ('first_node', Optional[Dict])])
 FrozenNodeIterator.query_hash.__doc__ = """The GraphQL ``query_hash`` parameter."""
 FrozenNodeIterator.query_variables.__doc__ = """The GraphQL ``query_variables`` parameter."""
 FrozenNodeIterator.query_referer.__doc__ = """The HTTP referer used for the GraphQL query."""
@@ -28,7 +27,7 @@ FrozenNodeIterator.total_index.__doc__ = """Number of items that have already be
 FrozenNodeIterator.best_before.__doc__ = """Date when parts of the stored nodes might have expired."""
 FrozenNodeIterator.remaining_data.__doc__ = \
    """The already-retrieved, yet-unprocessed ``edges`` and the ``page_info`` at time of freezing."""
-
+FrozenNodeIterator.first_node.__doc__ = """Node data of the first item, if an item has been produced."""

 T = TypeVar('T')

@@ -55,6 +54,9 @@ class NodeIterator(Iterator[T]):
       post_iterator = profile.get_posts()
       post_iterator.thaw(load("resume_information.json"))

+    (an appropriate method to load and save the :class:`FrozenNodeIterator` is e.g.
+    :func:`load_structure_from_file` and :func:`save_structure_to_file`.)
+
    A :class:`FrozenNodeIterator` can only be thawn with a matching NodeIterator, i.e. a NodeIterator instance that has
    been constructed with the same parameters as the instance that is represented by the :class:`FrozenNodeIterator` in
    question. This is to ensure that an iteration cannot be resumed in a wrong, unmatching loop. As a quick way to
@@ -74,7 +76,8 @@ class NodeIterator(Iterator[T]):
                 node_wrapper: Callable[[Dict], T],
                 query_variables: Optional[Dict[str, Any]] = None,
                 query_referer: Optional[str] = None,
-                 first_data: Optional[Dict[str, Any]] = None):
+                 first_data: Optional[Dict[str, Any]] = None,
+                 is_first: Optional[Callable[[T, Optional[T]], bool]] = None):
        self._context = context
        self._query_hash = query_hash
        self._edge_extractor = edge_extractor
@@ -88,6 +91,8 @@ class NodeIterator(Iterator[T]):
            self._best_before = datetime.now() + NodeIterator._shelf_life
        else:
            self._data = self._query()
+        self._first_node: Optional[Dict] = None
+        self._is_first = is_first

    def _query(self, after: Optional[str] = None) -> Dict:
        pagination_variables = {'first': NodeIterator._graphql_page_length}  # type: Dict[str, Any]
@@ -124,17 +129,25 @@ class NodeIterator(Iterator[T]):
            except KeyboardInterrupt:
                self._page_index, self._total_index = page_index, total_index
                raise
-            return self._node_wrapper(node)
+            item = self._node_wrapper(node)
+            if self._is_first is not None:
+                if self._is_first(item, self.first_item):
+                    self._first_node = node
+            else:
+                if self._first_node is None:
+                    self._first_node = node
+            return item
        if self._data['page_info']['has_next_page']:
            query_response = self._query(self._data['page_info']['end_cursor'])
-            page_index, data = self._page_index, self._data
-            try:
-                self._page_index = 0
-                self._data = query_response
-            except KeyboardInterrupt:
-                self._page_index, self._data = page_index, data
-                raise
-            return self.__next__()
+            if self._data['edges'] != query_response['edges'] and len(query_response['edges']) > 0:
+                page_index, data = self._page_index, self._data
+                try:
+                    self._page_index = 0
+                    self._data = query_response
+                except KeyboardInterrupt:
+                    self._page_index, self._data = page_index, data
+                    raise
+                return self.__next__()
        raise StopIteration()

    @property
@@ -150,16 +163,27 @@ class NodeIterator(Iterator[T]):
    @property
    def magic(self) -> str:
        """Magic string for easily identifying a matching iterator file for resuming (hash of some parameters)."""
-        if 'blake2b' not in hashlib.algorithms_available:
-            magic_hash = hashlib.new('sha224')
-        else:
-            # Use blake2b when possible, i.e. on Python >= 3.6.
-            magic_hash = hashlib.blake2b(digest_size=6)  # type:ignore  # pylint: disable=no-member
+        magic_hash = hashlib.blake2b(digest_size=6)
        magic_hash.update(json.dumps(
            [self._query_hash, self._query_variables, self._query_referer, self._context.username]
        ).encode())
        return base64.urlsafe_b64encode(magic_hash.digest()).decode()

+    @property
+    def first_item(self) -> Optional[T]:
+        """
+        If this iterator has produced any items, returns the first item produced.
+
+        It is possible to override what is considered the first item (for example, to consider the
+        newest item in case items are not in strict chronological order) by passing a callback
+        function as the `is_first` parameter when creating the class.
+
+        .. versionadded:: 4.8
+        .. versionchanged:: 4.9.2
+           What is considered the first item can be overridden.
+        """
+        return self._node_wrapper(self._first_node) if self._first_node is not None else None
+
    def freeze(self) -> FrozenNodeIterator:
        """Freeze the iterator for later resuming."""
        remaining_data = None
@@ -174,6 +198,7 @@ class NodeIterator(Iterator[T]):
            total_index=max(self.total_index - 1, 0),
            best_before=self._best_before.timestamp() if self._best_before else None,
            remaining_data=remaining_data,
+            first_node=self._first_node,
        )

    def thaw(self, frozen: FrozenNodeIterator) -> None:
@@ -200,18 +225,21 @@ class NodeIterator(Iterator[T]):
        self._total_index = frozen.total_index
        self._best_before = datetime.fromtimestamp(frozen.best_before)
        self._data = frozen.remaining_data
+        if frozen.first_node is not None:
+            self._first_node = frozen.first_node


@contextmanager
 def resumable_iteration(context: InstaloaderContext,
-                        iterator: Iterator,
+                        iterator: Iterable,
                        load: Callable[[InstaloaderContext, str], Any],
                        save: Callable[[FrozenNodeIterator, str], None],
                        format_path: Callable[[str], str],
                        check_bbd: bool = True,
                        enabled: bool = True) -> Iterator[Tuple[bool, int]]:
    """
-    High-level context manager to handle a resumable iteration that can be interrupted with a KeyboardInterrupt.
+    High-level context manager to handle a resumable iteration that can be interrupted
+    with a :class:`KeyboardInterrupt` or an :class:`AbortDownloadException`.

    It can be used as follows to automatically load a previously-saved state into the iterator, save the iterator's
    state when interrupted, and delete the resume file upon completion::
@@ -239,6 +267,9 @@ def resumable_iteration(context: InstaloaderContext,
    :param format_path: Returns the path to the resume file for the given magic.
    :param check_bbd: Whether to check the best before date and reject an expired FrozenNodeIterator.
    :param enabled: Set to False to disable all functionality and simply execute the inner body.
+
+    .. versionchanged:: 4.7
+       Also interrupt on :class:`AbortDownloadException`.
    """
    if not enabled or not isinstance(iterator, NodeIterator):
        yield False, 0
@@ -262,7 +293,7 @@ def resumable_iteration(context: InstaloaderContext,
            context.error("Warning: Not resuming from {}: {}".format(resume_file_path, exc))
    try:
        yield is_resuming, start_index
-    except KeyboardInterrupt:
+    except (KeyboardInterrupt, AbortDownloadException):
        if os.path.dirname(resume_file_path):
            os.makedirs(os.path.dirname(resume_file_path), exist_ok=True)
        save(iterator.freeze(), resume_file_path)
@@ -0,0 +1,46 @@
+from typing import Any, Callable, Dict, Iterator, Optional, TypeVar
+
+from .instaloadercontext import InstaloaderContext
+
+T = TypeVar('T')
+
+
+class SectionIterator(Iterator[T]):
+    """Iterator for the new 'sections'-style responses.
+
+    .. versionadded:: 4.9"""
+    def __init__(self,
+                 context: InstaloaderContext,
+                 sections_extractor: Callable[[Dict[str, Any]], Dict[str, Any]],
+                 media_wrapper: Callable[[Dict], T],
+                 query_path: str,
+                 first_data: Optional[Dict[str, Any]] = None):
+        self._context = context
+        self._sections_extractor = sections_extractor
+        self._media_wrapper = media_wrapper
+        self._query_path = query_path
+        self._data = first_data or self._query()
+        self._page_index = 0
+        self._section_index = 0
+
+    def __iter__(self):
+        return self
+
+    def _query(self, max_id: Optional[str] = None) -> Dict[str, Any]:
+        pagination_variables = {"max_id": max_id} if max_id is not None else {}
+        return self._sections_extractor(
+            self._context.get_json(self._query_path, params={"__a": 1, "__d": "dis", **pagination_variables})
+        )
+
+    def __next__(self) -> T:
+        if self._page_index < len(self._data['sections']):
+            media = self._data['sections'][self._page_index]['layout_content']['medias'][self._section_index]['media']
+            self._section_index += 1
+            if self._section_index >= len(self._data['sections'][self._page_index]['layout_content']['medias']):
+                self._section_index = 0
+                self._page_index += 1
+            return self._media_wrapper(media)
+        if self._data['more_available']:
+            self._page_index, self._section_index, self._data = 0, 0, self._query(self._data["next_max_id"])
+            return self.__next__()
+        raise StopIteration()
@@ -3,13 +3,18 @@ import lzma
 import re
 from base64 import b64decode, b64encode
 from collections import namedtuple
+from contextlib import suppress
 from datetime import datetime
-from typing import Any, Dict, Iterator, List, Optional, Union
+from itertools import islice
+from pathlib import Path
+from typing import Any, Callable, Dict, Iterable, Iterator, List, Optional, Tuple, Union
+from unicodedata import normalize

 from . import __version__
 from .exceptions import *
 from .instaloadercontext import InstaloaderContext
 from .nodeiterator import FrozenNodeIterator, NodeIterator
+from .sectioniterator import SectionIterator

 PostSidecarNode = namedtuple('PostSidecarNode', ['is_video', 'display_url', 'video_url'])
 PostSidecarNode.__doc__ = "Item of a Sidecar Post."
@@ -34,8 +39,8 @@ PostLocation.id.__doc__ = "ID number of location."
 PostLocation.name.__doc__ = "Location name."
 PostLocation.slug.__doc__ = "URL friendly variant of location name."
 PostLocation.has_public_page.__doc__ = "Whether location has a public page."
-PostLocation.lat.__doc__ = "Latitude (:class:`float`)."
-PostLocation.lng.__doc__ = "Longitude (:class:`float`)."
+PostLocation.lat.__doc__ = "Latitude (:class:`float` or None)."
+PostLocation.lng.__doc__ = "Longitude (:class:`float` or None)."


 class Post:
@@ -88,6 +93,41 @@ class Post:
        """Create a post object from a given mediaid"""
        return cls.from_shortcode(context, Post.mediaid_to_shortcode(mediaid))

+    @classmethod
+    def from_iphone_struct(cls, context: InstaloaderContext, media: Dict[str, Any]):
+        """Create a post from a given iphone_struct.
+
+        .. versionadded:: 4.9"""
+        media_types = {
+            1: "GraphImage",
+            2: "GraphVideo",
+            8: "GraphSidecar",
+        }
+        fake_node = {
+            "shortcode": media["code"],
+            "id": media["pk"],
+            "__typename": media_types[media["media_type"]],
+            "is_video": media_types[media["media_type"]] == "GraphVideo",
+            "date": media["taken_at"],
+            "caption": media["caption"].get("text") if media.get("caption") is not None else None,
+            "title": media.get("title"),
+            "viewer_has_liked": media["has_liked"],
+            "edge_media_preview_like": {"count": media["like_count"]},
+            "iphone_struct": media,
+        }
+        with suppress(KeyError):
+            fake_node["display_url"] = media['image_versions2']['candidates'][0]['url']
+        with suppress(KeyError):
+            fake_node["video_url"] = media['video_versions'][-1]['url']
+            fake_node["video_duration"] = media["video_duration"]
+            fake_node["video_view_count"] = media["view_count"]
+        with suppress(KeyError):
+            fake_node["edge_sidecar_to_children"] = {"edges": [{"node": {
+                "display_url": node['image_versions2']['candidates'][0]['url'],
+                "is_video": media_types[node["media_type"]] == "GraphVideo",
+            }} for node in media["carousel_media"]]}
+        return cls(context, fake_node, Profile.from_iphone_struct(context, media["user"]) if "user" in media else None)
+
    @staticmethod
    def shortcode_to_mediaid(code: str) -> int:
        if len(code) > 11:
@@ -128,6 +168,14 @@ class Post:
        """The mediaid is a decimal representation of the media shortcode."""
        return int(self._node['id'])

+    @property
+    def title(self) -> Optional[str]:
+        """Title of post"""
+        try:
+            return self._field('title')
+        except KeyError:
+            return None
+
    def __repr__(self):
        return '<Post {}>'.format(self.shortcode)

@@ -160,6 +208,8 @@ class Post:

    @property
    def _iphone_struct(self) -> Dict[str, Any]:
+        if not self._context.iphone_support:
+            raise IPhoneSupportDisabledException("iPhone support is disabled.")
        if not self._context.is_logged_in:
            raise LoginRequiredException("--login required to access iPhone media info endpoint.")
        if not self._iphone_struct_:
@@ -213,17 +263,16 @@ class Post:

    @property
    def date_local(self) -> datetime:
-        """Timestamp when the post was created (local time zone)."""
-        return datetime.fromtimestamp(self._node["date"]
-                                      if "date" in self._node
-                                      else self._node["taken_at_timestamp"])
+        """Timestamp when the post was created (local time zone).
+
+        .. versionchanged:: 4.9
+           Return timezone aware datetime object."""
+        return datetime.fromtimestamp(self._get_timestamp_date_created()).astimezone()

    @property
    def date_utc(self) -> datetime:
        """Timestamp when the post was created (UTC)."""
-        return datetime.utcfromtimestamp(self._node["date"]
-                                         if "date" in self._node
-                                         else self._node["taken_at_timestamp"])
+        return datetime.utcfromtimestamp(self._get_timestamp_date_created())

    @property
    def date(self) -> datetime:
@@ -238,13 +287,13 @@ class Post:
    @property
    def url(self) -> str:
        """URL of the picture / video thumbnail of the post"""
-        if self.typename == "GraphImage" and self._context.is_logged_in:
+        if self.typename == "GraphImage" and self._context.iphone_support and self._context.is_logged_in:
            try:
                orig_url = self._iphone_struct['image_versions2']['candidates'][0]['url']
-                url = re.sub(r'&se=\d+(&?)', r'\1', orig_url)
+                url = re.sub(r'([?&])se=\d+&?', r'\1', orig_url).rstrip('&')
                return url
            except (InstaloaderException, KeyError, IndexError) as err:
-                self._context.error('{} Unable to fetch high quality image version of {}.'.format(err, self))
+                self._context.error(f"Unable to fetch high quality image version of {self}: {err}")
        return self._node["display_url"] if "display_url" in self._node else self._node["display_src"]

    @property
@@ -264,6 +313,23 @@ class Post:
            return len(edges)
        return 1

+    def _get_timestamp_date_created(self) -> float:
+        """Timestamp when the post was created"""
+        return (self._node["date"]
+                if "date" in self._node
+                else self._node["taken_at_timestamp"])
+
+    def get_is_videos(self) -> List[bool]:
+        """
+        Return a list containing the ``is_video`` property for each media in the post.
+
+        .. versionadded:: 4.7
+        """
+        if self.typename == 'GraphSidecar':
+            edges = self._field('edge_sidecar_to_children', 'edges')
+            return [edge['node']['is_video'] for edge in edges]
+        return [self.is_video]
+
    def get_sidecar_nodes(self, start=0, end=-1) -> Iterator[PostSidecarNode]:
        """
        Sidecar nodes of a Post with typename==GraphSidecar.
@@ -273,36 +339,41 @@ class Post:
        """
        if self.typename == 'GraphSidecar':
            edges = self._field('edge_sidecar_to_children', 'edges')
-            if any(edge['node']['is_video'] and 'video_url' not in edge['node'] for edge in edges):
-                # video_url is only present in full metadata, issue #558.
-                edges = self._full_metadata['edge_sidecar_to_children']['edges']
            if end < 0:
                end = len(edges)-1
            if start < 0:
                start = len(edges)-1
+            if any(edge['node']['is_video'] and 'video_url' not in edge['node'] for edge in edges[start:(end+1)]):
+                # video_url is only present in full metadata, issue #558.
+                edges = self._full_metadata['edge_sidecar_to_children']['edges']
            for idx, edge in enumerate(edges):
                if start <= idx <= end:
                    node = edge['node']
                    is_video = node['is_video']
                    display_url = node['display_url']
-                    if not is_video and self._context.is_logged_in:
+                    if not is_video and self._context.iphone_support and self._context.is_logged_in:
                        try:
                            carousel_media = self._iphone_struct['carousel_media']
                            orig_url = carousel_media[idx]['image_versions2']['candidates'][0]['url']
-                            display_url = re.sub(r'&se=\d+(&?)', r'\1', orig_url)
+                            display_url = re.sub(r'([?&])se=\d+&?', r'\1', orig_url).rstrip('&')
                        except (InstaloaderException, KeyError, IndexError) as err:
-                            self._context.error('{} Unable to fetch high quality image version of {}.'.format(
-                                err, self))
+                            self._context.error(f"Unable to fetch high quality image version of {self}: {err}")
                    yield PostSidecarNode(is_video=is_video, display_url=display_url,
                                          video_url=node['video_url'] if is_video else None)

    @property
    def caption(self) -> Optional[str]:
        """Caption."""
+        def _normalize(string: Optional[str]) -> Optional[str]:
+            if string is not None:
+                return normalize("NFC", string)
+            else:
+                return None
+
        if "edge_media_to_caption" in self._node and self._node["edge_media_to_caption"]["edges"]:
-            return self._node["edge_media_to_caption"]["edges"][0]["node"]["text"]
+            return _normalize(self._node["edge_media_to_caption"]["edges"][0]["node"]["text"])
        elif "caption" in self._node:
-            return self._node["caption"]
+            return _normalize(self._node["caption"])
        return None

    @property
@@ -320,9 +391,11 @@ class Post:
        """List of all lowercased profiles that are mentioned in the Post's caption, without preceeding @."""
        if not self.caption:
            return []
-        # This regular expression is from jStassen, adjusted to use Python's \w to support Unicode
+        # This regular expression is modified from jStassen, adjusted to use Python's \w to
+        # support Unicode and a word/beginning of string delimiter at the beginning to ensure
+        # that no email addresses join the list of mentions.
        # http://blog.jstassen.com/2016/03/code-regex-for-instagram-username-and-hashtags/
-        mention_regex = re.compile(r"(?:@)(\w(?:(?:\w|(?:\.(?!\.))){0,28}(?:\w))?)")
+        mention_regex = re.compile(r"(?:^|\W|_)(?:@)(\w(?:(?:\w|(?:\.(?!\.))){0,28}(?:\w))?)", re.ASCII)
        return re.findall(mention_regex, self.caption.lower())

    @property
@@ -332,9 +405,19 @@ class Post:
        .. versionadded:: 4.2.6"""
        def _elliptify(caption):
            pcaption = ' '.join([s.replace('/', '\u2215') for s in caption.splitlines() if s]).strip()
-            return (pcaption[:30] + u"\u2026") if len(pcaption) > 31 else pcaption
+            return (pcaption[:30] + "\u2026") if len(pcaption) > 31 else pcaption
        return _elliptify(self.caption) if self.caption else ''

+    @property
+    def accessibility_caption(self) -> Optional[str]:
+        """Accessibility caption of the post, if available.
+
+        .. versionadded:: 4.9"""
+        try:
+            return self._field("accessibility_caption")
+        except KeyError:
+            return None
+
    @property
    def tagged_users(self) -> List[str]:
        """List of all lowercased users that are tagged in the Post."""
@@ -353,13 +436,35 @@ class Post:
    def video_url(self) -> Optional[str]:
        """URL of the video, or None."""
        if self.is_video:
-            if self._context.is_logged_in:
+            version_urls = []
+            try:
+                version_urls.append(self._field('video_url'))
+            except (InstaloaderException, KeyError, IndexError) as err:
+                self._context.error(f"Warning: Unable to fetch video from graphql of {self}: {err}")
+            if self._context.iphone_support and self._context.is_logged_in:
                try:
-                    url = self._iphone_struct['video_versions'][0]['url']
-                    return url
+                    version_urls.extend(version['url'] for version in self._iphone_struct['video_versions'])
                except (InstaloaderException, KeyError, IndexError) as err:
-                    self._context.error('{} Unable to fetch high quality video version of {}.'.format(err, self))
-            return self._field('video_url')
+                    self._context.error(f"Unable to fetch high-quality video version of {self}: {err}")
+            version_urls = list(dict.fromkeys(version_urls))
+            if len(version_urls) == 0:
+                return None
+            if len(version_urls) == 1:
+                return version_urls[0]
+            url_candidates: List[Tuple[int, str]] = []
+            for idx, version_url in enumerate(version_urls):
+                try:
+                    url_candidates.append((
+                        int(self._context.head(version_url, allow_redirects=True).headers.get('Content-Length', 0)),
+                        version_url
+                    ))
+                except (InstaloaderException, KeyError, IndexError) as err:
+                    self._context.error(f"Video URL candidate {idx+1}/{len(version_urls)} for {self}: {err}")
+            if not url_candidates:
+                # All candidates fail: Fallback to default URL and handle errors later at the actual download attempt
+                return version_urls[0]
+            url_candidates.sort()
+            return url_candidates[-1][1]
        return None

    @property
@@ -407,12 +512,15 @@ class Post:
        except KeyError:
            return self._field('edge_media_to_comment', 'count')

-    def get_comments(self) -> Iterator[PostComment]:
+    def get_comments(self) -> Iterable[PostComment]:
        r"""Iterate over all comments of the post.

        Each comment is represented by a PostComment namedtuple with fields text (string), created_at (datetime),
        id (int), owner (:class:`Profile`) and answers (:class:`~typing.Iterator`\ [:class:`PostCommentAnswer`])
        if available.
+
+        .. versionchanged:: 4.7
+           Change return type to ``Iterable``.
        """
        def _postcommentanswer(node):
            return PostCommentAnswer(id=int(node['id']),
@@ -447,16 +555,15 @@ class Post:
                               answers=_postcommentanswers(node))
        if self.comments == 0:
            # Avoid doing additional requests if there are no comments
-            return
+            return []

        comment_edges = self._field('edge_media_to_comment', 'edges')
-        answers_count = sum([edge['node'].get('edge_threaded_comments', {}).get('count', 0) for edge in comment_edges])
+        answers_count = sum(edge['node'].get('edge_threaded_comments', {}).get('count', 0) for edge in comment_edges)

        if self.comments == len(comment_edges) + answers_count:
            # If the Post's metadata already contains all parent comments, don't do GraphQL requests to obtain them
-            yield from (_postcomment(comment['node']) for comment in comment_edges)
-            return
-        yield from NodeIterator(
+            return [_postcomment(comment['node']) for comment in comment_edges]
+        return NodeIterator(
            self._context,
            '97b41c52301f77ce508f55e66d17620e',
            lambda d: d['data']['shortcode_media']['edge_media_to_parent_comment'],
@@ -530,12 +637,19 @@ class Post:
            return None
        location_id = int(loc['id'])
        if any(k not in loc for k in ('name', 'slug', 'has_public_page', 'lat', 'lng')):
-            loc = self._context.get_json("explore/locations/{0}/".format(location_id),
-                                         params={'__a': 1})['graphql']['location']
+            loc.update(self._context.get_json("explore/locations/{0}/".format(location_id),
+                                              params={'__a': 1, '__d': 'dis'})['native_location_data']['location_info'])
        self._location = PostLocation(location_id, loc['name'], loc['slug'], loc['has_public_page'],
-                                      loc['lat'], loc['lng'])
+                                      loc.get('lat'), loc.get('lng'))
        return self._location

+    @property
+    def is_pinned(self) -> bool:
+        """True if this Post has been pinned by at least one user.
+
+        .. versionadded: 4.9.2"""
+        return 'pinned_for_users' in self._node and bool(self._node['pinned_for_users'])
+

 class Profile:
    """
@@ -617,6 +731,20 @@ class Profile:
        context.profile_id_cache[profile_id] = profile
        return profile

+    @classmethod
+    def from_iphone_struct(cls, context: InstaloaderContext, media: Dict[str, Any]):
+        """Create a profile from a given iphone_struct.
+
+        .. versionadded:: 4.9"""
+        return cls(context, {
+            "id": media["pk"],
+            "username": media["username"],
+            "is_private": media["is_private"],
+            "full_name": media["full_name"],
+            "profile_pic_url_hd": media["profile_pic_url"],
+            "iphone_struct": media,
+        })
+
    @classmethod
    def own_profile(cls, context: InstaloaderContext):
        """Return own profile if logged-in.
@@ -642,13 +770,19 @@ class Profile:
    def _obtain_metadata(self):
        try:
            if not self._has_full_metadata:
-                metadata = self._context.get_json('{}/feed/'.format(self.username), params={})
-                self._node = metadata['entry_data']['ProfilePage'][0]['graphql']['user']
+                metadata = self._context.get_iphone_json(f'api/v1/users/web_profile_info/?username={self.username}',
+                                                         params={})
+                if metadata['data']['user'] is None:
+                    raise ProfileNotExistsException('Profile {} does not exist.'.format(self.username))
+                self._node = metadata['data']['user']
                self._has_full_metadata = True
        except (QueryReturnedNotFoundException, KeyError) as err:
            top_search_results = TopSearchResults(self._context, self.username)
            similar_profiles = [profile.username for profile in top_search_results.get_profiles()]
            if similar_profiles:
+                if self.username in similar_profiles:
+                    raise ProfileNotExistsException(
+                        f"Profile {self.username} seems to exist, but could not be loaded.") from err
                raise ProfileNotExistsException('Profile {} does not exist.\nThe most similar profile{}: {}.'
                                                .format(self.username,
                                                        's are' if len(similar_profiles) > 1 else ' is',
@@ -670,6 +804,8 @@ class Profile:

    @property
    def _iphone_struct(self) -> Dict[str, Any]:
+        if not self._context.iphone_support:
+            raise IPhoneSupportDisabledException("iPhone support is disabled.")
        if not self._context.is_logged_in:
            raise LoginRequiredException("--login required to access iPhone profile info endpoint.")
        if not self._iphone_struct_:
@@ -770,14 +906,13 @@ class Profile:
    def has_public_story(self) -> bool:
        if not self._has_public_story:
            self._obtain_metadata()
-            # query not rate limited if invoked anonymously:
-            with self._context.anonymous_copy() as anonymous_context:
-                data = anonymous_context.graphql_query('9ca88e465c3f866a76f7adee3871bdd8',
-                                                       {'user_id': self.userid, 'include_chaining': False,
-                                                        'include_reel': False, 'include_suggested_users': False,
-                                                        'include_logged_out_extras': True,
-                                                        'include_highlight_reels': False},
-                                                       'https://www.instagram.com/{}/'.format(self.username))
+            # query rate might be limited:
+            data = self._context.graphql_query('9ca88e465c3f866a76f7adee3871bdd8',
+                                               {'user_id': self.userid, 'include_chaining': False,
+                                                'include_reel': False, 'include_suggested_users': False,
+                                                'include_logged_out_extras': True,
+                                                'include_highlight_reels': False},
+                                               'https://www.instagram.com/{}/'.format(self.username))
            self._has_public_story = data['data']['user']['has_public_story']
        assert self._has_public_story is not None
        return self._has_public_story
@@ -813,15 +948,22 @@ class Profile:

        .. versionchanged:: 4.2.1
           Require being logged in for HD version (as required by Instagram)."""
-        if self._context.is_logged_in:
+        if self._context.iphone_support and self._context.is_logged_in:
            try:
                return self._iphone_struct['hd_profile_pic_url_info']['url']
            except (InstaloaderException, KeyError) as err:
-                self._context.error('{} Unable to fetch high quality profile pic.'.format(err))
+                self._context.error(f"Unable to fetch high quality profile pic: {err}")
                return self._metadata("profile_pic_url_hd")
        else:
            return self._metadata("profile_pic_url_hd")

+    @property
+    def profile_pic_url_no_iphone(self) -> str:
+        """Return URL of lower-quality profile picture.
+
+        .. versionadded:: 4.9.3"""
+        return self._metadata("profile_pic_url_hd")
+
    def get_profile_pic_url(self) -> str:
        """.. deprecated:: 4.0.3

@@ -841,6 +983,7 @@ class Profile:
            {'id': self.userid},
            'https://www.instagram.com/{0}/'.format(self.username),
            self._metadata('edge_owner_to_timeline_media'),
+            Profile._make_is_newest_checker()
        )

    def get_saved_posts(self) -> NodeIterator[Post]:
@@ -874,6 +1017,7 @@ class Profile:
            lambda n: Post(self._context, n, self if int(n['owner']['id']) == self.userid else None),
            {'id': self.userid},
            'https://www.instagram.com/{0}/'.format(self.username),
+            is_first=Profile._make_is_newest_checker()
        )

    def get_igtv_posts(self) -> NodeIterator[Post]:
@@ -891,8 +1035,13 @@ class Profile:
            {'id': self.userid},
            'https://www.instagram.com/{0}/channel/'.format(self.username),
            self._metadata('edge_felix_video_timeline'),
+            Profile._make_is_newest_checker()
        )

+    @staticmethod
+    def _make_is_newest_checker() -> Callable[[Post, Optional[Post]], bool]:
+        return lambda post, first: first is None or post.date_local > first.date_local
+
    def get_followers(self) -> NodeIterator['Profile']:
        """
        Retrieve list of followers of given profile.
@@ -998,13 +1147,36 @@ class StoryItem:
    def __hash__(self) -> int:
        return hash(self.mediaid)

+    @classmethod
+    def from_mediaid(cls, context: InstaloaderContext, mediaid: int):
+        """Create a StoryItem object from a given mediaid.
+
+        .. versionadded:: 4.9
+        """
+        pic_json = context.graphql_query(
+            '2b0673e0dc4580674a88d426fe00ea90',
+            {'shortcode': Post.mediaid_to_shortcode(mediaid)}
+        )
+        shortcode_media = pic_json['data']['shortcode_media']
+        if shortcode_media is None:
+            raise BadResponseException("Fetching StoryItem metadata failed.")
+        return cls(context, shortcode_media)
+
    @property
    def _iphone_struct(self) -> Dict[str, Any]:
+        if not self._context.iphone_support:
+            raise IPhoneSupportDisabledException("iPhone support is disabled.")
        if not self._context.is_logged_in:
            raise LoginRequiredException("--login required to access iPhone media info endpoint.")
        if not self._iphone_struct_:
-            data = self._context.get_iphone_json(path='api/v1/media/{}/info/'.format(self.mediaid), params={})
-            self._iphone_struct_ = data['items'][0]
+            data = self._context.get_iphone_json(
+                path='api/v1/feed/reels_media/?reel_ids={}'.format(self.owner_id), params={}
+            )
+            self._iphone_struct_ = {}
+            for item in data['reels'][str(self.owner_id)]['items']:
+                if item['pk'] == self.mediaid:
+                    self._iphone_struct_ = item
+                    break
        return self._iphone_struct_

    @property
@@ -1027,8 +1199,11 @@ class StoryItem:

    @property
    def date_local(self) -> datetime:
-        """Timestamp when the StoryItem was created (local time zone)."""
-        return datetime.fromtimestamp(self._node['taken_at_timestamp'])
+        """Timestamp when the StoryItem was created (local time zone).
+
+        .. versionchanged:: 4.9
+           Return timezone aware datetime object."""
+        return datetime.fromtimestamp(self._node['taken_at_timestamp']).astimezone()

    @property
    def date_utc(self) -> datetime:
@@ -1058,13 +1233,14 @@ class StoryItem:
    @property
    def url(self) -> str:
        """URL of the picture / video thumbnail of the StoryItem"""
-        if self.typename == "GraphStoryImage" and self._context.is_logged_in:
+        if self.typename in ["GraphStoryImage", "StoryImage"] and \
+                self._context.iphone_support and self._context.is_logged_in:
            try:
                orig_url = self._iphone_struct['image_versions2']['candidates'][0]['url']
-                url = re.sub(r'&se=\d+(&?)', r'\1', orig_url)
+                url = re.sub(r'([?&])se=\d+&?', r'\1', orig_url).rstrip('&')
                return url
            except (InstaloaderException, KeyError, IndexError) as err:
-                self._context.error('{} Unable to fetch high quality image version of {}.'.format(err, self))
+                self._context.error(f"Unable to fetch high quality image version of {self}: {err}")
        return self._node['display_resources'][-1]['src']

    @property
@@ -1081,7 +1257,35 @@ class StoryItem:
    def video_url(self) -> Optional[str]:
        """URL of the video, or None."""
        if self.is_video:
-            return self._node['video_resources'][-1]['src']
+            version_urls = []
+            try:
+                version_urls.append(self._node['video_resources'][-1]['src'])
+            except (InstaloaderException, KeyError, IndexError) as err:
+                self._context.error(f"Warning: Unable to fetch video from graphql of {self}: {err}")
+            if self._context.iphone_support and self._context.is_logged_in:
+                try:
+                    version_urls.extend(version['url'] for version in self._iphone_struct['video_versions'])
+                except (InstaloaderException, KeyError, IndexError) as err:
+                    self._context.error(f"Unable to fetch high-quality video version of {self}: {err}")
+            version_urls = list(dict.fromkeys(version_urls))
+            if len(version_urls) == 0:
+                return None
+            if len(version_urls) == 1:
+                return version_urls[0]
+            url_candidates: List[Tuple[int, str]] = []
+            for idx, version_url in enumerate(version_urls):
+                try:
+                    url_candidates.append((
+                        int(self._context.head(version_url, allow_redirects=True).headers.get('Content-Length', 0)),
+                        version_url
+                    ))
+                except (InstaloaderException, KeyError, IndexError) as err:
+                    self._context.error(f"Video URL candidate {idx+1}/{len(version_urls)} for {self}: {err}")
+            if not url_candidates:
+                # All candidates fail: Fallback to default URL and handle errors later at the actual download attempt
+                return version_urls[0]
+            url_candidates.sort()
+            return url_candidates[-1][1]
        return None


@@ -1111,6 +1315,7 @@ class Story:
        self._node = node
        self._unique_id = None      # type: Optional[str]
        self._owner_profile = None  # type: Optional[Profile]
+        self._iphone_struct_ = None  # type: Optional[Dict[str, Any]]

    def __repr__(self):
        return '<Story by {} changed {:%Y-%m-%d_%H-%M-%S_UTC}>'.format(self.owner_username, self.latest_media_utc)
@@ -1181,9 +1386,23 @@ class Story:
        """The story owner's ID."""
        return self.owner_profile.userid

+    def _fetch_iphone_struct(self) -> None:
+        if self._context.iphone_support and self._context.is_logged_in and not self._iphone_struct_:
+            data = self._context.get_iphone_json(
+                path='api/v1/feed/reels_media/?reel_ids={}'.format(self.owner_id), params={}
+            )
+            self._iphone_struct_ = data['reels'][str(self.owner_id)]
+
    def get_items(self) -> Iterator[StoryItem]:
        """Retrieve all items from a story."""
-        yield from (StoryItem(self._context, item, self.owner_profile) for item in reversed(self._node['items']))
+        self._fetch_iphone_struct()
+        for item in reversed(self._node['items']):
+            if self._iphone_struct_ is not None:
+                for iphone_struct_item in self._iphone_struct_['items']:
+                    if iphone_struct_item['pk'] == int(item['id']):
+                        item['iphone_struct'] = iphone_struct_item
+                        break
+            yield StoryItem(self._context, item, self.owner_profile)


 class Highlight(Story):
@@ -1213,6 +1432,7 @@ class Highlight(Story):
        super().__init__(context, node)
        self._owner_profile = owner
        self._items = None  # type: Optional[List[Dict[str, Any]]]
+        self._iphone_struct_ = None  # type: Optional[Dict[str, Any]]

    def __repr__(self):
        return '<Highlight by {}: {}>'.format(self.owner_username, self.title)
@@ -1251,6 +1471,13 @@ class Highlight(Story):
                                                       "highlight_reel_ids": [str(self.unique_id)],
                                                       "precomposed_overlay": False})['data']['reels_media'][0]['items']

+    def _fetch_iphone_struct(self) -> None:
+        if self._context.iphone_support and self._context.is_logged_in and not self._iphone_struct_:
+            data = self._context.get_iphone_json(
+                path='api/v1/feed/reels_media/?reel_ids=highlight:{}'.format(self.unique_id), params={}
+            )
+            self._iphone_struct_ = data['reels']['highlight:{}'.format(self.unique_id)]
+
    @property
    def itemcount(self) -> int:
        """Count of items associated with the :class:`Highlight` instance."""
@@ -1261,8 +1488,15 @@ class Highlight(Story):
    def get_items(self) -> Iterator[StoryItem]:
        """Retrieve all associated highlight items."""
        self._fetch_items()
+        self._fetch_iphone_struct()
        assert self._items is not None
-        yield from (StoryItem(self._context, item, self.owner_profile) for item in self._items)
+        for item in self._items:
+            if self._iphone_struct_ is not None:
+                for iphone_struct_item in self._iphone_struct_['items']:
+                    if iphone_struct_item['pk'] == int(item['id']):
+                        item['iphone_struct'] = iphone_struct_item
+                        break
+            yield StoryItem(self._context, item, self.owner_profile)


 class Hashtag:
@@ -1280,6 +1514,9 @@ class Hashtag:
          L.download_post(post, target="#"+hashtag.name)

    Also, this class implements == and is hashable.
+
+    .. versionchanged:: 4.9
+       Removed ``get_related_tags()`` and ``is_top_media_only`` as these features were removed from Instagram.
    """
    def __init__(self, context: InstaloaderContext, node: Dict[str, Any]):
        assert "name" in node
@@ -1308,19 +1545,21 @@ class Hashtag:
        return self._node["name"].lower()

    def _query(self, params):
-        return self._context.get_json("explore/tags/{0}/".format(self.name),
-                                      params)["graphql"]["hashtag"]
+        json_response = self._context.get_json("explore/tags/{0}/".format(self.name), params)
+        return json_response["graphql"]["hashtag"] if "graphql" in json_response else json_response["data"]

    def _obtain_metadata(self):
        if not self._has_full_metadata:
-            self._node = self._query({"__a": 1})
+            self._node = self._query({"__a": 1, "__d": "dis"})
            self._has_full_metadata = True

    def _asdict(self):
        json_node = self._node.copy()
        # remove posts
        json_node.pop("edge_hashtag_to_top_posts", None)
+        json_node.pop("top", None)
        json_node.pop("edge_hashtag_to_media", None)
+        json_node.pop("recent", None)
        return json_node

    def __repr__(self):
@@ -1356,30 +1595,33 @@ class Hashtag:
        return self._metadata("profile_pic_url")

    @property
-    def description(self) -> str:
+    def description(self) -> Optional[str]:
        return self._metadata("description")

    @property
    def allow_following(self) -> bool:
-        return self._metadata("allow_following")
+        return bool(self._metadata("allow_following"))

    @property
    def is_following(self) -> bool:
-        return self._metadata("is_following")
-
-    @property
-    def is_top_media_only(self) -> bool:
-        return self._metadata("is_top_media_only")
-
-    def get_related_tags(self) -> Iterator["Hashtag"]:
-        """Yields similar hashtags."""
-        yield from (Hashtag(self._context, edge["node"])
-                    for edge in self._metadata("edge_hashtag_to_related_tags", "edges"))
+        try:
+            return self._metadata("is_following")
+        except KeyError:
+            return bool(self._metadata("following"))

    def get_top_posts(self) -> Iterator[Post]:
        """Yields the top posts of the hashtag."""
-        yield from (Post(self._context, edge["node"])
-                    for edge in self._metadata("edge_hashtag_to_top_posts", "edges"))
+        try:
+            yield from (Post(self._context, edge["node"])
+                        for edge in self._metadata("edge_hashtag_to_top_posts", "edges"))
+        except KeyError:
+            yield from SectionIterator(
+                self._context,
+                lambda d: d["data"]["top"],
+                lambda m: Post.from_iphone_struct(self._context, m),
+                f"explore/tags/{self.name}/",
+                self._metadata("top"),
+            )

    @property
    def mediacount(self) -> int:
@@ -1389,23 +1631,38 @@ class Hashtag:
        The number of posts with a certain hashtag may differ from the number of posts that can actually be accessed, as
        the hashtag count might include private posts
        """
-        return self._metadata("edge_hashtag_to_media", "count")
+        try:
+            return self._metadata("edge_hashtag_to_media", "count")
+        except KeyError:
+            return self._metadata("media_count")

    def get_posts(self) -> Iterator[Post]:
-        """Yields the posts associated with this hashtag."""
-        self._metadata("edge_hashtag_to_media", "edges")
-        self._metadata("edge_hashtag_to_media", "page_info")
-        conn = self._metadata("edge_hashtag_to_media")
-        yield from (Post(self._context, edge["node"]) for edge in conn["edges"])
-        while conn["page_info"]["has_next_page"]:
-            data = self._query({'__a': 1, 'max_id': conn["page_info"]["end_cursor"]})
-            conn = data["edge_hashtag_to_media"]
+        """Yields the recent posts associated with this hashtag.
+
+        .. deprecated:: 4.9
+           Use :meth:`Hashtag.get_posts_resumable` as this method may return incorrect results (:issue:`1457`)"""
+        try:
+            self._metadata("edge_hashtag_to_media", "edges")
+            self._metadata("edge_hashtag_to_media", "page_info")
+            conn = self._metadata("edge_hashtag_to_media")
            yield from (Post(self._context, edge["node"]) for edge in conn["edges"])
+            while conn["page_info"]["has_next_page"]:
+                data = self._query({'__a': 1, 'max_id': conn["page_info"]["end_cursor"]})
+                conn = data["edge_hashtag_to_media"]
+                yield from (Post(self._context, edge["node"]) for edge in conn["edges"])
+        except KeyError:
+            yield from SectionIterator(
+                self._context,
+                lambda d: d["data"]["recent"],
+                lambda m: Post.from_iphone_struct(self._context, m),
+                f"explore/tags/{self.name}/",
+                self._metadata("recent"),
+            )

    def get_all_posts(self) -> Iterator[Post]:
        """Yields all posts, i.e. all most recent posts and the top posts, in almost-chronological order."""
-        sorted_top_posts = iter(sorted(self.get_top_posts(), key=lambda p: p.date_utc, reverse=True))
-        other_posts = self.get_posts()
+        sorted_top_posts = iter(sorted(islice(self.get_top_posts(), 9), key=lambda p: p.date_utc, reverse=True))
+        other_posts = self.get_posts_resumable()
        next_top = next(sorted_top_posts, None)
        next_other = next(other_posts, None)
        while next_top is not None or next_other is not None:
@@ -1431,6 +1688,20 @@ class Hashtag:
                yield next_other
                next_other = next(other_posts, None)

+    def get_posts_resumable(self) -> NodeIterator[Post]:
+        """Get the recent posts of the hashtag in a resumable fashion.
+
+        :rtype: NodeIterator[Post]
+
+        .. versionadded:: 4.9"""
+        return NodeIterator(
+            self._context, "9b498c08113f1e09617a1703c22b2f32",
+            lambda d: d['data']['hashtag']['edge_hashtag_to_media'],
+            lambda n: Post(self._context, n),
+            {'tag_name': self.name},
+            f"https://www.instagram.com/explore/tags/{self.name}/"
+        )
+

 class TopSearchResults:
    """
@@ -1480,7 +1751,7 @@ class TopSearchResults:
            place = location.get('place', {})
            slug = place.get('slug')
            loc = place.get('location', {})
-            yield PostLocation(int(loc['pk']), loc['name'], slug, None, loc['lat'], loc['lng'])
+            yield PostLocation(int(loc['pk']), loc['name'], slug, None, loc.get('lat'), loc.get('lng'))

    def get_hashtag_strings(self) -> Iterator[str]:
        """
@@ -1510,12 +1781,72 @@ class TopSearchResults:
        return self._searchstring


+class TitlePic:
+    def __init__(self, profile: Optional[Profile], target: Union[str, Path], typename: str,
+                 filename: str, date_utc: Optional[datetime]):
+        self._profile = profile
+        self._target = target
+        self._typename = typename
+        self._filename = filename
+        self._date_utc = date_utc
+
+    @property
+    def profile(self) -> Union[str, Path]:
+        return self._profile.username.lower() if self._profile is not None else self._target
+
+    @property
+    def owner_username(self) -> Union[str, Path]:
+        return self.profile
+
+    @property
+    def owner_id(self) -> Union[str, Path]:
+        return str(self._profile.userid) if self._profile is not None else self._target
+
+    @property
+    def target(self) -> Union[str, Path]:
+        return self._target
+
+    @property
+    def typename(self) -> str:
+        return self._typename
+
+    @property
+    def filename(self) -> str:
+        return self._filename
+
+    @property
+    def date_utc(self) -> Optional[datetime]:
+        return self._date_utc
+
+    @property
+    def date(self) -> Optional[datetime]:
+        return self.date_utc
+
+    @property
+    def date_local(self) -> Optional[datetime]:
+        return self._date_utc.astimezone() if self._date_utc is not None else None
+
+
 JsonExportable = Union[Post, Profile, StoryItem, Hashtag, FrozenNodeIterator]


+def get_json_structure(structure: JsonExportable) -> dict:
+    """Returns Instaloader JSON structure for a :class:`Post`, :class:`Profile`, :class:`StoryItem`, :class:`Hashtag`
+     or :class:`FrozenNodeIterator` so that it can be loaded by :func:`load_structure`.
+
+    :param structure: :class:`Post`, :class:`Profile`, :class:`StoryItem` or :class:`Hashtag`
+
+    .. versionadded:: 4.8
+    """
+    return {
+        'node': structure._asdict(),
+        'instaloader': {'version': __version__, 'node_type': structure.__class__.__name__}
+    }
+
+
 def save_structure_to_file(structure: JsonExportable, filename: str) -> None:
-    """Saves a :class:`Post`, :class:`Profile`, :class:`StoryItem` or :class:`Hashtag` to a '.json' or '.json.xz' file
-    such that it can later be loaded by :func:`load_structure_from_file`.
+    """Saves a :class:`Post`, :class:`Profile`, :class:`StoryItem`, :class:`Hashtag` or :class:`FrozenNodeIterator` to a
+    '.json' or '.json.xz' file such that it can later be loaded by :func:`load_structure_from_file`.

    If the specified filename ends in '.xz', the file will be LZMA compressed. Otherwise, a pretty-printed JSON file
    will be created.
@@ -1523,8 +1854,7 @@ def save_structure_to_file(structure: JsonExportable, filename: str) -> None:
    :param structure: :class:`Post`, :class:`Profile`, :class:`StoryItem` or :class:`Hashtag`
    :param filename: Filename, ends in '.json' or '.json.xz'
    """
-    json_structure = {'node': structure._asdict(),
-                      'instaloader': {'version': __version__, 'node_type': structure.__class__.__name__}}
+    json_structure = get_json_structure(structure)
    compress = filename.endswith('.xz')
    if compress:
        with lzma.open(filename, 'wt', check=lzma.CHECK_NONE) as fp:
@@ -1534,20 +1864,15 @@ def save_structure_to_file(structure: JsonExportable, filename: str) -> None:
            json.dump(json_structure, fp=fp, indent=4, sort_keys=True)


-def load_structure_from_file(context: InstaloaderContext, filename: str) -> JsonExportable:
-    """Loads a :class:`Post`, :class:`Profile`, :class:`StoryItem` or :class:`Hashtag` from a '.json' or '.json.xz' file
-    that has been saved by :func:`save_structure_to_file`.
+def load_structure(context: InstaloaderContext, json_structure: dict) -> JsonExportable:
+    """Loads a :class:`Post`, :class:`Profile`, :class:`StoryItem`, :class:`Hashtag` or :class:`FrozenNodeIterator` from
+    a json structure.

    :param context: :attr:`Instaloader.context` linked to the new object, used for additional queries if neccessary.
-    :param filename: Filename, ends in '.json' or '.json.xz'
+    :param json_structure: Instaloader JSON structure
+
+    .. versionadded:: 4.8
    """
-    compressed = filename.endswith('.xz')
-    if compressed:
-        fp = lzma.open(filename, 'rt')
-    else:
-        fp = open(filename, 'rt')
-    json_structure = json.load(fp)
-    fp.close()
    if 'node' in json_structure and 'instaloader' in json_structure and \
            'node_type' in json_structure['instaloader']:
        node_type = json_structure['instaloader']['node_type']
@@ -1560,11 +1885,28 @@ def load_structure_from_file(context: InstaloaderContext, filename: str) -> Json
        elif node_type == "Hashtag":
            return Hashtag(context, json_structure['node'])
        elif node_type == "FrozenNodeIterator":
+            if not 'first_node' in json_structure['node']:
+                json_structure['node']['first_node'] = None
            return FrozenNodeIterator(**json_structure['node'])
-        else:
-            raise InvalidArgumentException("{}: Not an Instaloader JSON.".format(filename))
    elif 'shortcode' in json_structure:
        # Post JSON created with Instaloader v3
        return Post.from_shortcode(context, json_structure['shortcode'])
+    raise InvalidArgumentException("Passed json structure is not an Instaloader JSON")
+
+
+def load_structure_from_file(context: InstaloaderContext, filename: str) -> JsonExportable:
+    """Loads a :class:`Post`, :class:`Profile`, :class:`StoryItem`, :class:`Hashtag` or :class:`FrozenNodeIterator` from
+    a '.json' or '.json.xz' file that has been saved by :func:`save_structure_to_file`.
+
+    :param context: :attr:`Instaloader.context` linked to the new object, used for additional queries if neccessary.
+    :param filename: Filename, ends in '.json' or '.json.xz'
+    """
+    compressed = filename.endswith('.xz')
+    if compressed:
+        fp = lzma.open(filename, 'rt')
    else:
-        raise InvalidArgumentException("{}: Not an Instaloader JSON.".format(filename))
+        # pylint:disable=consider-using-with
+        fp = open(filename, 'rt')
+    json_structure = json.load(fp)
+    fp.close()
+    return load_structure(context, json_structure)
@@ -3,7 +3,6 @@
 import re
 import sys
 import os
-import platform
 from setuptools import setup


@@ -19,14 +18,11 @@ def get_version():
    raise SystemExit("Could not find version string.")


-if sys.version_info < (3, 5):
-    sys.exit('Instaloader requires Python >= 3.5.')
+if sys.version_info < (3, 8):
+    sys.exit('Instaloader requires Python >= 3.8.')

 requirements = ['requests>=2.4']

-if platform.system() == 'Windows' and sys.version_info < (3, 6):
-    requirements.append('win_unicode_console')
-
 keywords = (['instagram', 'instagram-scraper', 'instagram-client', 'instagram-feed', 'downloader', 'videos', 'photos',
             'pictures', 'instagram-user-photos', 'instagram-photos', 'instagram-metadata', 'instagram-downloader',
             'instagram-stories'])
@@ -47,7 +43,7 @@ setup(
                'from Instagram.',
    long_description=open(os.path.join(SRC, 'README.rst')).read(),
    install_requires=requirements,
-    python_requires='>=3.5',
+    python_requires='>=3.8',
    entry_points={'console_scripts': ['instaloader=instaloader.__main__:main']},
    zip_safe=False,
    keywords=keywords,
@@ -58,10 +54,10 @@ setup(
        'Intended Audience :: Developers',
        'License :: OSI Approved :: MIT License',
        'Operating System :: OS Independent',
-        'Programming Language :: Python :: 3.5',
-        'Programming Language :: Python :: 3.6',
-        'Programming Language :: Python :: 3.7',
        'Programming Language :: Python :: 3.8',
+        'Programming Language :: Python :: 3.9',
+        'Programming Language :: Python :: 3.10',
+        'Programming Language :: Python :: 3.11',
        'Programming Language :: Python :: 3 :: Only',
        'Topic :: Internet',
        'Topic :: Multimedia :: Graphics'
Author	SHA1	Message	Date
Alexander Graf	6a3d763cf3	Release of Version 4.9.6	2023-02-11 13:21:07 +01:00
SayanthD	e2d42807a2	Fixed downloading with location id	2023-02-11 13:13:27 +01:00
Eduardo M KALINOWSKI	d07bd6a096	Detect redirects to https://i.instagram.com/accounts/login	2023-02-11 13:00:52 +01:00
Alexander Graf	0462d5c34a	update CI dependencies, require Python>=3.8	2023-02-11 12:45:38 +01:00
Alexander Graf	3cc29a4ceb	Release of Version 4.9.5	2022-09-26 21:38:51 +02:00
ILikeToCode	ffb0aea5eb	Fixed downloading hashtags (#1749 )	2022-09-26 21:37:02 +02:00
Alexander Graf	84bb60c155	Release of Version 4.9.4	2022-09-16 20:58:52 +02:00
fireattack	7857909886	Handle empty video resource in some old stories in highlights (#1732 )	2022-09-16 20:56:01 +02:00
Alexander Graf	2206a7e5d7	Add new sponsor to README.rst We are proud to announce @socialmethod as our new sponsor.	2022-09-16 19:03:24 +02:00
Alexander Graf	f1b5e9cdf1	Increase stale period for pull requests and issues	2022-09-07 21:09:46 +02:00
Eduardo Kalinowski	8debcbb7ab	Fixes incorrect latest stamp when download is resumed (#1717 )	2022-09-07 21:01:48 +02:00
Alexander Graf	e97fd8a27c	Release of Version 4.9.3	2022-08-11 20:00:42 +02:00
fireattack	7f9d1618ad	Use api/v1/feed/reels_media endpoint to get story info (#1675 )	2022-08-11 19:59:02 +02:00
fireattack	6a42906697	don't try to call iPhone api when comparing profile_pic URL (#1681 )	2022-08-11 19:56:52 +02:00
fireattack	4bb32e3c5a	write caption in text mode (#1632 )	2022-07-14 21:00:24 +02:00
Alexander Graf	5e232fd4f5	Release of Version 4.9.2	2022-07-04 20:18:22 +02:00
Eduardo Kalinowski	da0dcb106c	Handle pinned posts with --fast-update and --latest-stamps (#1586 ) Instead of hard-coding a check for pinned items when storing the first item, allows the creater of NodeIterator to specify how the comparison should be made. This also allows storing the newest post (whatever it's status is), instead of simply storing the newst non-pinned post, which prevents redownloads in case a user adds only a pinned post between runs.	2022-07-04 20:02:41 +02:00
Alexander Graf	8649a10113	docs: fix link to requests intersphinx inventory	2022-06-23 20:50:05 +02:00
Ziemowit Zabawa	2d37e1dca2	Make docs explicit about importing multiple argument files (#1548 )	2022-06-16 10:25:43 +02:00
fireattack	37cbc5d72d	Handle StoryImage in StoryItem.url (#1558 )	2022-06-16 10:18:59 +02:00
Alexander Graf	853e860363	Release of Version 4.9.1	2022-06-08 18:13:38 +02:00
twitter-79	90cfeaf471	Fixed fetching a users profile information (#1561 ) * fixed fetching a users profile information switched to new api/v1/users/web_profile_info/?username=xy endpoint * fixed non existing users thanks to @Dibort for fixing this!	2022-06-08 18:12:07 +02:00
Nicolai Weitkemper	5a24064012	Fix incorrect error messages when certain requests fail (#1536 ) - handle case where `Profile.from_id` works, but `Profile.from_username` doesn't - Handle case where `QueryReturnedNotFoundException` lies and `similar_profiles` contains the exact username - Still raises `ProfileNotExistsException` for compatibility.	2022-05-30 21:29:34 +02:00
MiguelX413	0704602e50	Checks if caption is not None before normalization (#1475 )	2022-04-18 09:49:34 +02:00
Alexander Graf	218d88fc6d	Release of Version 4.9	2022-03-26 17:18:56 +01:00
Alexander Graf	385c6c8a35	Merge branch 'upcoming/v4.9'	2022-03-26 17:15:09 +01:00
Alexander Graf	c0e5d0475b	StoryItem.from_mediaid() Addresses #1238.	2022-03-26 17:03:57 +01:00
Alexander Graf	fbd7df2a90	Use resumable iteration to download hashtags Closes #1457.	2022-03-26 17:02:06 +01:00
JeremyKj87	569808f7a4	Fix for infinity loading if next page is empty (#1465 )	2022-03-26 15:57:45 +01:00
Alexander Graf	b808843efe	Actions: Optimize Publish Package workflow	2022-03-19 17:27:22 +01:00
Alexander Graf	f66cdd3e07	Release of Version 4.8.5	2022-03-19 17:15:52 +01:00
Alexander Graf	83dc5f749d	Attempt to fix Documentation deployment	2022-03-19 17:02:24 +01:00
Alexander Graf	41de9e5a7c	Attempt to fix AUR deployment	2022-03-19 16:53:55 +01:00
Alexander Graf	d938f3e2f5	Third Beta Release for Version 4.9	2022-03-19 16:35:53 +01:00
Alexander Graf	374d9d0d81	Merge branch 'fully-migrate-to-gh-actions' into upcoming/v4.9	2022-03-19 16:34:59 +01:00
Alexander Graf	3cbee1b318	Fully Migrate to GitHub Actions	2022-03-19 16:29:48 +01:00
Alexander Graf	dbc91cbeb1	Release of Version 4.8.5	2022-03-19 15:08:36 +01:00
fireattack	5a42a7537f	Use header to determine file extension (#1447 )	2022-03-19 15:04:42 +01:00
MiguelX413	d8d268f861	Mention regex will now be restricted to ASCII (#1456 )	2022-03-18 14:57:28 +01:00
MiguelX413	463ada519e	Add unicode normalization of captions (#1453 )	2022-03-17 16:06:26 +01:00
canh	604b107586	Add --sanitize-paths option (#1452 )	2022-03-17 15:27:36 +01:00
Alexander Graf	37a93ee59a	Second Beta Release for Version 4.9	2022-01-15 17:34:18 +01:00
Alexander Graf Misael Eduardo Kalinowski	2d4f3b8537	Make {Post,StoryItem}.date_local timezone aware This commit is made up of `21365ec668` `555c86633c` `d864ce08ff` Discussed in #1305, #1316, #1372. Co-Authored-By: Misael <misael.om@gmail.com> Co-Authored-By: Eduardo Kalinowski <eduardo@kalinowski.com.br>	2022-01-15 17:34:18 +01:00
Alexander Graf	8c752532e1	Merge branch 'master' into upcoming/v4.9	2022-01-15 17:17:19 +01:00
Alexander Graf	fdcf2e1367	Release of Version 4.8.4	2022-01-15 17:11:45 +01:00
Alexander Graf	ca78fee307	Revert date_utc and date_local being timezone aware Fixes #1379. Revert "Make {Post,StoryItem}.date_utc timezone aware" `d864ce08ff`. Revert "Remove unnecessary conversion to timezone aware timestamp (#1372)" `555c86633c`. Revert "Timezone option for datetime style formatting (#1316)" `21365ec668`.	2022-01-15 17:08:04 +01:00
Alexander Graf	d6e5e31005	Release of Version 4.8.3	2022-01-07 14:56:40 +01:00
Alexander Graf	d864ce08ff	Make {Post,StoryItem}.date_utc timezone aware Also simplify {Post,StoryItem}.date_local. Discussed in #1316.	2022-01-07 14:53:23 +01:00
Eduardo Kalinowski	555c86633c	Remove unnecessary conversion to timezone aware timestamp (#1372 ) Post.date_local now already returns a timestamp with the appropriate timezone.	2022-01-07 14:31:41 +01:00
Misael	21365ec668	Timezone option for datetime style formatting (#1316 ) Add timezone awareness for datetime objects returned by `date_local` and `date_utc`. Fixes #1305.	2021-12-31 16:50:16 +01:00
Alexander Graf	6343e9a69d	Release of Version 4.8.2	2021-11-27 16:25:12 +01:00
Alexander Graf	06574eb428	Fix KeyError on attempt to get incomplete location Fixes #1349.	2021-11-27 16:25:12 +01:00
JeremyKj87	815ba83d77	avoid comments endless loop - check if query_response is always the same (#1347 )	2021-11-27 15:47:32 +01:00
Alexander Graf	5bff9b5474	First Beta Release for Version 4.9	2021-11-12 21:44:08 +01:00
Alexander Graf	a23b832a69	Add Post.accessibility_caption property	2021-11-12 21:41:40 +01:00
Alexander Graf	5d18857695	Fix download of hashtags and locations Fixes #1080, fixes #1129, closes #1240.	2021-11-12 21:33:48 +01:00
Alexander Graf	d6fd4c560c	Merge branch 'master' into upcoming/v4.9	2021-11-12 20:28:19 +01:00
fireattack	470fc0d905	Fix KeyError for video_url, and better workflow (#1321 ) Fixes #1320.	2021-11-12 15:21:33 +01:00
Alexander Graf	0300d8a4f1	Fix CI for Python 3.10	2021-11-12 14:32:15 +01:00
Alexander Graf	f09bd51ed4	Update feature suggestion issue template	2021-09-25 10:32:36 +02:00
Arman Yeghiazaryan	7a32272b55	Fix the issue with folder names starting with dots (#1252 )	2021-08-14 16:27:14 +02:00
Alexander Graf	9eb2339028	Update default user agent string	2021-08-10 09:25:00 +02:00
Alexander Graf	c45b536be9	Update dependencies	2021-08-10 09:25:00 +02:00
Alexander Graf	2c116683c2	Release of Version 4.8.1	2021-08-10 09:15:11 +02:00
Alexander Graf	054fa47245	Fix rare KeyError: 'video_versions' error Closes #1251.	2021-08-10 09:13:42 +02:00
Alexander Graf	1f8ebe09b4	Release of Version 4.8	2021-08-04 18:27:45 +02:00
Alexander Graf	87dfdff335	Merge branch 'master' into upcoming/v4.8	2021-08-04 18:24:46 +02:00
Alexander Graf	631503d463	Log "Using latest stamps from ..." message Instaloader always logs which files are loaded and considered in the current run (such as session files or resume files). This commit adds a log message "Using latest stamps from ..." when using --latest-stamps, to be consistent.	2021-08-04 18:12:12 +02:00
Eduardo Kalinowski	43c9d32b22	Avoid uninitialized variable when called with :stories (#1242 )	2021-08-04 18:04:16 +02:00
Alexander Graf	1ad9111753	Release of Version 4.7.6	2021-08-04 17:46:06 +02:00
fireattackandAlexander Graf	327fcfd8e8	Download best-quality video (#1232 ) Co-authored-by: Alexander Graf <17130992+aandergr@users.noreply.github.com>	2021-08-04 17:42:14 +02:00
Arman Yeghiazaryan	ae39ab9893	Fix Post.location / --geotags (#1244 ) Closes #1109.	2021-08-04 17:28:56 +02:00
Alexander Graf	0372f7c8e8	Release of Version 4.7.5	2021-07-24 20:34:14 +02:00
Alexander Graf	aaeaaa0e22	Fourth Release Candidate for Version 4.8	2021-07-24 19:29:56 +02:00
Alexander Graf	35bd8357d1	Merge branch 'master' into upcoming/v4.8	2021-07-24 19:28:53 +02:00
Eduardo Kalinowski	9cdf679fc1	Fix interruped downloads with --latest-stamps (#1219 ) The most recent post is cached in NodeIterator (and saved to the disk), and its timestamp is used, instead of the timestamp instaloader was run. This way, even in later resuming runs the timestamp stored is the same that would have been stored if the first run. Fixes #1206.	2021-07-24 19:27:46 +02:00
Alexander Graf	94d599db61	List all login issues in docs/troubleshooting	2021-07-24 19:18:39 +02:00
fireattack	a0e3e8db0d	Do not delete referer header (#1237 ) since it causes IG's CDN to limit download speed	2021-07-24 16:37:07 +02:00
Alexander Graf	26c9f71132	Third Release Candidate for Version 4.8	2021-06-22 21:52:26 +02:00
Eduardo Kalinowski	210719af56	Avoid uninitialized variable (#1204 ) fixes #1203	2021-06-22 21:50:06 +02:00
Alexander Graf	8c1129e556	Add GitHub Sponsors link	2021-06-20 13:50:59 +02:00
Alexander Graf	c200031be6	Increase stale period for pull requests and issues	2021-06-20 13:50:59 +02:00
Alexander Graf	a0d2ec95b9	Release of Version 4.7.4	2021-06-18 15:19:51 +02:00
Alexander Graf	750dd3f47f	Second Release Candidate for Version 4.8	2021-06-18 15:13:01 +02:00
Alexander Graf	839b77eef0	First Release Candidate for Version 4.8	2021-06-16 23:17:04 +02:00
Alexander Graf	e346842f7b	Merge branch 'master' into upcoming/v4.8	2021-06-16 23:08:18 +02:00
Alexander Graf	9b3f809a9a	Release of Version 4.7.3	2021-06-16 23:00:18 +02:00
twitter-79	803440ac50	Fix Login Error (#1192 ) taking csrf token from login page as it is empty using the old url Fixes #1150.	2021-06-16 22:35:58 +02:00
Eduardo Kalinowski	455a757159	Make downloads using --latest-stamps resumable (#1193 ) Using itertools.takewhile() on a NodeIterator returns a plain Iterator, and so it's not resumable. The strategy has been altered to pass an extra argument to posts_download_loop, a lambda that is evaluated for each post, and causes the loop to stop when it returns false.	2021-06-16 22:19:16 +02:00
Alexander Graf	07de73bde1	Fix missing 2FA error message	2021-06-15 21:05:57 +02:00
spykard	bf83d406d3	Fix and Update has_public_story Functionality (#1183 )	2021-06-15 19:39:06 +02:00
Eduardo Kalinowski	ac5d6e312a	Report Profile renames even if --quiet is enabled (#1180 ) While this isn't really an error, it requires user action, so the message is printed to stderr even with --quiet, and repeated at the end when that option is not enabled. Resolves #1159.	2021-06-10 18:49:05 +02:00
Alexander Graf	3bb4a5b4b1	Release of Version 4.7.2	2021-06-06 11:18:48 +02:00
Kalle Westerling	9ffdf258a9	Fix Post.mentions including email addresses (#1167 ) Earlier, if someone had an email address in the caption, those were provided as mentions in the Post.mentions attribute. With this update to the regular expression, email addresses are no longer ingested. Fixes #1029.	2021-06-06 11:13:05 +02:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	d530a7a37f	Bump urllib3 from 1.26.4 to 1.26.5 (#1163 ) Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.4 to 1.26.5. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](https://github.com/urllib3/urllib3/compare/1.26.4...1.26.5) --- updated-dependencies: - dependency-name: urllib3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2021-06-04 15:12:01 +02:00
Eduardo Kalinowski	d142fb70b0	--latest-stamps, like --fast-update, but without the need to keep downloaded files (#1131 ) Adds the --latest-stamps command line option, pointing to a file where the latest time each profile was scraped. On the next run, only posts newer than that time are downloaded. Fixes #1122.	2021-05-27 18:22:33 +02:00
SAO	8a26943215	update documentation for --count option (#1139 )	2021-05-17 18:51:39 +02:00
Eduardo Kalinowski	5345470ebf	Add option --title-pattern for profile pics and covers (#1127 )	2021-05-15 16:25:36 +02:00
Eduardo Kalinowski	a2715cbe03	Show progress of profile scraping (#1114 )	2021-04-29 21:14:23 +02:00
anokhovd	fa19b95107	Add support for loading structures directly from python json objects (#1092 )	2021-04-22 20:12:42 +02:00
A Bored Pervert	3837b642b0	Make iPhone endpoint query optional (#1076 )	2021-04-22 20:06:35 +02:00
Alexander Graf	e67b9e6c37	Improve docs on how to load/save FrozenNodeIterator	2021-04-12 16:54:44 +02:00
Alexander Graf	2bf175a111	Release of Version 4.7.1	2021-03-27 20:47:28 +01:00
Alexander Graf	95d8534aed	Fix SRCINFO depends metadata in AUR	2021-03-27 20:46:53 +01:00
Alexander GrafandAndré Koch-Kramer	b583cd07e1	Get better-quality version of pictures if logged-in Fixes #1065 (also see #630). Co-Authored-By: André Koch-Kramer <koch-kramer@web.de>	2021-03-27 20:42:56 +01:00
Alexander Graf	52f258e89b	Release of Version 4.7	2021-03-27 15:19:12 +01:00
Alexander Graf	e9888e72fd	Second Release Candidate for Version 4.7	2021-03-20 17:44:04 +01:00
Alexander Graf	c72b2a8fc2	Default download_comments and download_geotags to False	2021-03-20 17:39:37 +01:00
Alexander Graf	e2cec0a0bf	Update dependencies; test on py3.9; drop python 3.5 support	2021-03-20 17:30:15 +01:00
Alexander Graf	08eaa7d474	always keep exception chains	2021-03-20 17:17:08 +01:00
Alexander Graf	f70355e803	Update default user agent	2021-03-20 17:15:05 +01:00
Alexander Graf	bba7d5131c	First Release Candidate for Version 4.7	2021-03-13 11:24:53 +01:00
Alexander GrafandAndré Koch-Kramer	f6731566cd	Support resuming of downloading comments Co-Authored-By: André Koch-Kramer <koch-kramer@web.de>	2021-03-13 11:17:56 +01:00
Alexander Graf	cd13211603	Minor optimization in get_sidecar_nodes()	2021-02-24 16:35:08 +01:00
Alexander Graf	8635a576f0	Fix line-too-long pylint complaint	2021-02-24 16:25:56 +01:00
Alexander Graf	8f594a3a30	Merge branch 'master' into upcoming/v4.7	2021-02-24 16:10:27 +01:00
Alexander Graf	a2d756b177	Flag --abort-on to abort on given status codes Closes #920.	2021-02-13 19:23:14 +01:00
Alexander Graf	e15d67c065	Change login hint from "Warning" to "Hint"	2021-02-13 19:12:13 +01:00
Alexander Graf	640066dd60	Merge branch 'master' into upcoming/v4.7	2021-01-30 17:07:21 +01:00
Alexander GrafandAndré Koch-Kramer	672b0b1840	Fix efficiency bug regarding re-downloading targets Remove unnecessary queries when a file in a target does not have to be re-downloaded. This significantly speeds up the download in cases where the majority of media is already present locally. Co-Authored-By: André Koch-Kramer <koch-kramer@web.de>	2021-01-30 16:54:35 +01:00
17it151tejaschauhan Tejas Chauhan Alexander Graf	43c521afa9	Add method to get title of post (#976 ) Co-authored-by: Tejas Chauhan <tejas.chauhan@kevit.io> Co-authored-by: Alexander Graf <17130992+aandergr@users.noreply.github.com>	2021-01-16 18:50:27 +01:00