Release of Version 4.5.3

Fixes wrong "Profile does not exist" error.

Co-authored-by: André Koch-Kramer <koch-kramer@web.de>
Co-authored-by: Alexander Graf <17130992+aandergr@users.noreply.github.com>

.github/ISSUE_TEMPLATE/bug_report.md

@@ -13,7 +13,7 @@ Steps to reproduce the behavior:
 (e.g. Instaloader command line)
 
 **Expected behavior**
-A clear and concise description of what you expected to happen (if not obvious).
+A clear and concise description of what you expected to happen (even if it seems obvious).
 
 **Error messages and tracebacks**
 If applicable, add error messages and tracebacks to help explain your problem.

README.rst

@@ -27,8 +27,9 @@
    :alt: Arch User Repository Package
    :target: https://aur.archlinux.org/packages/instaloader/
 
-.. |downloads| image:: https://img.shields.io/pypi/dm/instaloader.svg
+.. |downloads| image:: https://pepy.tech/badge/instaloader/month
    :alt: PyPI Download Count
+   :target: https://pepy.tech/project/instaloader/month
 
 .. badges-end
 

docs/_templates/layout.html

@@ -26,7 +26,7 @@
     </a>
     <ul class="navbar-nav mr-auto">
       <li class="nav-item ml-0 ml-sm-2">
-        <a target="_blank" class="nav-link" rel="noopener"
+        <a class="nav-link"
            href="https://github.com/instaloader/instaloader/releases/tag/v{{ current_release }}"
            title="Released on {{ current_release_date }}" data-toggle="tooltip">
           v<b>{{ current_release }}</b>
@@ -35,17 +35,17 @@
     </ul>
     <ul class="navbar-nav ml-auto d-none d-sm-flex">
       <li class="nav-item">
-        <a target="_blank" class="nav-link" rel="noopener" href="https://github.com/instaloader/instaloader/issues">
+        <a class="nav-link" href="https://github.com/instaloader/instaloader/issues">
           Issues
         </a>
       </li>
       <li class="nav-item">
-        <a target="_blank" class="nav-link" rel="noopener" href="https://github.com/instaloader/instaloader/releases">
+        <a class="nav-link" href="https://github.com/instaloader/instaloader/releases">
           Releases
         </a>
       </li>
       <li class="nav-item">
-        <a target="_blank" class="nav-link" rel="noopener" href="https://github.com/instaloader/instaloader">
+        <a class="nav-link" href="https://github.com/instaloader/instaloader">
           GitHub
         </a>
       </li>
@@ -63,9 +63,9 @@
 {% block content %}
   <div class="container-fluid">
     <div class="row">
-      <aside class="instaloaderdoc-sidebar collapse d-md-block col-md-3 col-xl-2 bg-light"
-             id="instaloaderdocSidebar">
-        <nav>
+      <div class="instaloaderdoc-sidebar collapse d-md-block col-md-3 col-xl-2 bg-light"
+           id="instaloaderdocSidebar">
+        <nav role="navigation" aria-label="Table of Contents">
           <form action="{{ pathto('search') }}" method="get" role="search" class="border-bottom mr-2 pb-2 mb-2 px-1">
             <label class="sr-only" for="searchQueryInput">Search</label>
             <input type="search" name="q" id="searchQueryInput" class="form-control my-2"
@@ -74,8 +74,8 @@
           <h4>Contents</h4>
           {{ toctree(maxdepth=-1) }}
         </nav>
-      </aside>
-      <main class="col-12 col-md-9 col-xl-10 pt-2">
+      </div>
+      <main class="col-12 col-md-9 col-xl-10 pt-2" role="main">
         {% block body %}{% endblock %}
       </main>
     </div>

docs/as-module.rst

@@ -13,7 +13,7 @@ Python Module :mod:`instaloader`
 .. highlight:: python
 
 Instaloader exposes its internally used methods and structures as a Python
-module, making it a **powerful and intuitive Python API for Instagram**,
+module, making it a powerful and intuitive Python API for Instagram,
 allowing to further customize obtaining media and metadata.
 
 Start with getting an instance of :class:`Instaloader`::
@@ -82,20 +82,20 @@ metadata of a Profile. :class:`Profile` instances can be created with:
 - :meth:`Profile.get_followers`
    Profiles that follow given user.
 
-- :attr:`Post.owner_profile`, :attr:`Story.owner_profile` and :attr:`StoryItem.owner_profile`
+- :attr:`Post.owner_profile`, :attr:`Story.owner_profile`, :attr:`StoryItem.owner_profile`
    Owner profile of particular object.
 
 - :meth:`Post.get_likes`
    Profiles that liked a given :class:`Post`
 
-- :attr:`PostComment.owner` attribute for comment in :meth:`Post.get_comments`
+- :attr:`PostComment.owner`
    Profile of a Post comment.
 
-For a list of real code examples using the Instaloader module for advanced
-tasks, see :ref:`codesnippets`.
+For a list of a few code examples that use the Instaloader module, see
+:ref:`codesnippets`.
 
-The reference of the many classes provided by the :mod:`instaloader` module is
-divided into the following subsections.
+The reference of the classes and functions provided by the :mod:`instaloader` module is
+divided into the following subsections:
 
 .. toctree::
    :maxdepth: 2

docs/basic-usage.rst

@@ -77,7 +77,7 @@ Instaloader supports the following targets:
       instructs Instaloader to also **download the user's stories**,
 
    - :option:`--highlights`
-      to **download highlights of each profile that is downloaded**,
+      to **download the highlights of that profile**,
 
    - :option:`--tagged`
       to **download posts where the user is tagged**, and
@@ -86,7 +86,7 @@ Instaloader supports the following targets:
       to **download IGTV videos**.
 
 - ``"#hashtag"``
-   Posts with a certain **hashtag** (the quotes are usually necessary),
+   Posts with a certain **hashtag** (the quotes are usually necessary).
 
 - ``%location id``
    Posts tagged with a given location; the location ID is the numerical ID
@@ -98,13 +98,13 @@ Instaloader supports the following targets:
 
 - ``:stories``
    The currently-visible **stories** of your followees (requires
-   :option:`--login`),
+   :option:`--login`).
 
 - ``:feed``
-   Your **feed** (requires :option:`--login`),
+   Your **feed** (requires :option:`--login`).
 
 - ``:saved``
-   Posts which are marked as **saved** (requires :option:`--login`),
+   Posts which are marked as **saved** (requires :option:`--login`).
 
 - ``@profile``
    All profiles that are followed by ``profile``, i.e. the *followees* of
@@ -196,7 +196,7 @@ Filter Posts
 .. py:currentmodule:: instaloader
 
 The options :option:`--post-filter` and :option:`--storyitem-filter`
-allows to specify criteria that posts or story items have to
+allow to specify criteria that posts or story items have to
 meet to be downloaded. If not given, all posts are downloaded.
 
 The filter string must be a
@@ -208,7 +208,7 @@ Id est, the following attributes can be used with both
 :option:`--post-filter` and :option:`--storyitem-filter`:
 
 - :attr:`~Post.owner_username` (str), :attr:`~Post.owner_id` (int)
-   Owner profile username / userid.
+   Owner profile username / user ID.
 
 - :attr:`~Post.date_utc` (datetime), :attr:`~Post.date_local` (datetime)
    Creation timestamp. Since :class:`~datetime.datetime` objects can be created
@@ -217,7 +217,7 @@ Id est, the following attributes can be used with both
       instaloader --post-filter="date_utc <= datetime(2018, 5, 31)" target
 
 - :attr:`~Post.is_video` (bool)
-   Post/StoryItem is a video. For example, you may skip videos::
+   Whether Post/StoryItem is a video. For example, you may skip videos::
 
       instaloader --post-filter="not is_video" target
 
@@ -234,8 +234,8 @@ As :option:`--post-filter`, the following attributes can be used additionally:
       instaloader --login=your_username --post-filter=viewer_has_liked :feed
 
 - :attr:`~Post.likes` (int), :attr:`~Post.comments` (int)
-   Likes count / Comments count. You might only want to download posts that
-   either you liked or were liked by many others**::
+   Likes count / comments count. You might only want to download posts that
+   were either liked by yourself or by many others::
 
       instaloader --login=your_username --post-filter="likes>100 or viewer_has_liked" profile
 
@@ -265,7 +265,7 @@ along with each post where the Post's caption is saved.
 You can customize what metadata to save for each Post or StoryItem with
 :option:`--post-metadata-txt` and :option:`--storyitem-metadata-txt`. The
 default is ``--post-metadata-txt={caption}`` and no storyitem metadata txt.
-These strings are formatted similar as in the :ref:`filename-specification` and
+These strings are formatted similar as the path patterns described in :ref:`filename-specification` and
 the result is saved in text files, unless it is empty.
 
 Specifying these options multiple times results in output having multiple lines,
@@ -299,10 +299,10 @@ Instaloader logged-in, create a session file::
 
    instaloader --login=your_username
 
-Then use the same parameter in your cronjob to load the session and download
+Then use the same username in your cronjob to load the session and download
 the given targets::
 
-   instaloader --login=your_username --quiet <target> [...]
+   instaloader --login=your_username --quiet target [...]
 
 Instaloader saves the session file to
 ``~/.config/instaloader/session-YOUR-USERNAME``. See
@@ -317,5 +317,6 @@ Instaloader exposes its internally used methods and structures, making it a
 powerful and intuitive Python API for Instagram, allowing to further customize
 obtaining media and metadata.
 
-Also see :ref:`codesnippets`, where we collect example scripts that use
-Instaloader to achieve more complex tasks.
+Also see :ref:`codesnippets`, where we collect a few example scripts that use
+Instaloader for simple tasks that cannot be done with the command line
+interface.

docs/cli-options.rst

@@ -59,7 +59,7 @@ What to Download of each Post
 
 .. option:: --geotags, -G
 
-   **Download geotags** when available. Geotags are stored as a text file with
+   Download geotags when available. Geotags are stored as a text file with
    the location's name and a Google Maps link. This requires an additional
    request to the Instagram server for each picture. Requires :option:`--login`.
 
@@ -106,12 +106,12 @@ What to Download of each Profile
 
 .. option:: --stories, -s
 
-   Also **download stories** of each profile that is downloaded. Requires
+   Also download stories of each profile that is downloaded. Requires
    :option:`--login`.
 
 .. option:: --highlights
 
-   Also **download highlights** of each profile that is downloaded. Requires
+   Also download highlights of each profile that is downloaded. Requires
    :option:`--login`.
 
    .. versionadded:: 4.1
@@ -161,7 +161,7 @@ Which Posts to Download
 Login (Download Private Profiles)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Instaloader can **login to Instagram**. This allows downloading private
+Instaloader can login to Instagram. This allows downloading private
 profiles. To login, pass the :option:`--login` option. Your session cookie (not your
 password!) will be saved to a local file to be reused next time you want
 Instaloader to login.
@@ -261,12 +261,12 @@ Miscellaneous Options
 
       Text file should separate arguments with line breaks.
 
-      args.txt example::
+   args.txt example::
 
-         --login=MYUSERNAME
-         --password=MYPASSWORD
-         --fast-update
-         profile1
-         profile2
+      --login=MYUSERNAME
+      --password=MYPASSWORD
+      --fast-update
+      profile1
+      profile2
 
    .. versionadded:: 4.1

docs/codesnippets.rst

@@ -42,7 +42,7 @@ The code example with :func:`~itertools.dropwhile` and
 :func:`~itertools.takewhile` makes the assumption that the post iterator returns
 posts in exact chronological order.  As discussed in :issue:`666`, the following
 approach fits for an **almost chronological order**, where up to *k* older posts
-are inserted into an otherwise chronological order, such as an Hashtag feed.
+are inserted into an otherwise chronological order, such as a Hashtag feed.
 
 .. literalinclude:: codesnippets/666_historical_hashtag_data.py
 
@@ -50,7 +50,7 @@ Likes of a Profile / Ghost Followers
 ------------------------------------
 
 To obtain a list of your inactive followers, i.e. followers that did not like
-any of your pictures, into a file you can use this approach.
+any of your pictures, you can use this approach.
 
 .. literalinclude:: codesnippets/120_ghost_followers.py
 
@@ -78,11 +78,11 @@ Discussed in :issue:`56`.
 Only one Post per User
 ----------------------
 
-To download only the one most recent post from each user, this snippet creates a
-:class:`set` that contains the users of which a post has already been
-downloaded. While iterating the posts, it checks whether the post's owner
-already is in the set. If not, the post is downloaded from Instagram and the
-user is added to that set.
+To download only the single most recent post per user within a hashtag feed,
+this snippet uses a :class:`set` that contains the users of whom a post has
+already been downloaded. For each post, it checks whether the post's creator is
+already contained in that set. If not, the post is downloaded from Instagram and
+the user is added to that set.
 
 .. literalinclude:: codesnippets/113_only_one_per_user.py
 
@@ -100,31 +100,6 @@ With Instaloader, it is easy to download the few most-liked pictures of a user.
 
 Discussed in :issue:`194`.
 
-Upgrade Images by Local Copies
-------------------------------
-
-The following script finds local versions of images fetched by Instaloader, in
-order to upgrade the downloaded images by locally-found versions with better
-quality. It uses image hashing to identify similar images.
-
-`updgrade-instaloader-images.py <https://gist.github.com/pavelkryukov/15f93d19a99428a284a8bcec27e0187b>`__ (external link to GitHub Gist)
-
-Discussed in :issue:`46`.
-
-Add Captions to Images
-----------------------
-
-Instaloader does not modify the downloaded JPEG file. However, one could combine
-it with an imaging library such as Pillow or PIL to render the caption on
-Instagram pictures.  The following shows an approach.
-
-.. literalinclude:: codesnippets/110_pil_captions.py
-
-See also :attr:`Post.caption`, :attr:`Post.url`, :meth:`Post.from_shortcode`,
-:func:`load_structure_from_file`.
-
-Discussed in :issue:`110`.
-
 Metadata JSON Files
 -------------------
 

docs/codesnippets/110_pil_captions.py

@@ -1,20 +0,0 @@
-from io import BytesIO
-
-from requests import get
-from PIL import Image, ImageDraw
-from instaloader import *
-
-L = Instaloader()
-
-# Load Post instance
-post = load_structure_from_file(L.context, '2017-10-01_18-53-03_UTC.json.xz')
-# or post = Post.from_shortcode(L.context, SHORTCODE)
-
-# Render caption
-image = Image.open(BytesIO(get(post.url).content))
-draw = ImageDraw.Draw(image)
-color = 'rgb(0, 0, 0)'  # black color
-draw.text((300,100), post.caption.encode('latin1', errors='ignore'), fill=color)
-
-# Save image
-image.save('test.jpg')

docs/codesnippets/120_ghost_followers.py

@@ -2,26 +2,26 @@ import instaloader
 
 L = instaloader.Instaloader()
 
-USER = 'your_account'
+USER = "your_account"
 PROFILE = USER
 
-# Your preferred way of logging in:
+# Load session previously saved with `instaloader -l USERNAME`:
 L.load_session_from_file(USER)
 
 profile = instaloader.Profile.from_username(L.context, PROFILE)
 
 likes = set()
-print('Fetching likes of all posts of profile {}.'.format(profile.username))
+print("Fetching likes of all posts of profile {}.".format(profile.username))
 for post in profile.get_posts():
     print(post)
     likes = likes | set(post.get_likes())
 
-print('Fetching followers of profile {}.'.format(profile.username))
+print("Fetching followers of profile {}.".format(profile.username))
 followers = set(profile.get_followers())
 
 ghosts = followers - likes
 
-print('Storing ghosts into file.')
-with open('/YOUR PATH/inactive-users.txt', 'w') as f:
+print("Storing ghosts into file.")
+with open("inactive-users.txt", 'w') as f:
     for ghost in ghosts:
         print(ghost.username, file=f)

docs/codesnippets/666_historical_hashtag_data.py

@@ -9,7 +9,7 @@ SINCE = datetime(2020, 5, 10)  # further from today, inclusive
 UNTIL = datetime(2020, 5, 11)  # closer to today, not inclusive
 
 k = 0  # initiate k
-k_list = []  # uncomment this to tune k
+#k_list = []  # uncomment this to tune k
 
 for post in posts:
     postdate = post.date
@@ -24,7 +24,8 @@ for post in posts:
             continue
     else:
         L.download_post(post, "#urbanphotography")
-        k = 0  # set k to 0
         # if you want to tune k, uncomment below to get your k max
         #k_list.append(k)
+        k = 0  # set k to 0
+
 #max(k_list)

docs/contributing.rst

@@ -32,7 +32,7 @@ reporting a problem, please keep the following in mind:
 
 - Include all **error messages and tracebacks** in the report.
 
-- If not obvious, describe **which behavior you expected**
+- Even if it seems obvious, describe **which behavior you expected**
   instead of what actually happened.
 
 - If we have closed an issue apparently inadvertently or inappropriately, please

docs/installation.rst

@@ -5,27 +5,27 @@ Install Instaloader
 
 .. highlight:: none
 
-**To install Instaloader**,
+To **install Instaloader**,
 
 #. Ensure that you have `Python <https://www.python.org/>`__, at least
-   version 3.5 and `pip <https://pypi.python.org/pypi/pip>`__
-   installed
+   version 3.5, and `pip <https://pypi.python.org/pypi/pip>`__
+   installed.
 
 #. Then, install Instaloader using::
 
       pip3 install instaloader
 
-**To upgrade Instaloader**, do::
+To **upgrade Instaloader** to its current version, do::
 
    pip3 install --upgrade instaloader
 
 
-**Alternative methods for installing Instaloader:**
+**Alternative methods** for installing Instaloader:
 
 - If you do not want to use pip, even though it is highly recommended,
   and prefer to **install Instaloader manually**,
   `Download the Source <https://github.com/instaloader/instaloader/releases/latest>`__,
-  extract the Zip or Tarball and execute ``instaloader.py`` from there.
+  extract the Zip or Tarball and run ``setup.py`` from there.
 
 - On **Arch Linux**, you may install Instaloader using the
   `Instaloader AUR package <https://aur.archlinux.org/packages/instaloader/>`__.
@@ -33,6 +33,6 @@ Install Instaloader
 - On **Windows 10**, you may download the standalone executable from the
   `current release page <https://github.com/instaloader/instaloader/releases/latest>`__.
 
-- To test the most current pre-release or development version of Instaloader::
+- To test the most current **pre-release** version of Instaloader::
 
      pip3 install --pre instaloader

docs/troubleshooting.rst

@@ -5,8 +5,8 @@ Troubleshooting
 
 .. highlight:: python
 
-429 - Too Many Requests
------------------------
+429 Too Many Requests
+---------------------
 
 Instaloader has a logic to keep track of its requests to Instagram and to obey
 their rate limits. Since they are nowhere documented, we try them out
@@ -14,17 +14,17 @@ experimentally. We have a daily cron job running to confirm that Instaloader
 still stays within the rate limits. Nevertheless, the rate control logic assumes
 that
 
-- at one time, Instaloader is the only application that consumes requests. I.e.
+- at one time, Instaloader is the only application that consumes requests, i.e.
   neither the Instagram browser interface, nor a mobile app, nor another
-  Instaloader instance is running in parallel,
+  Instaloader instance is running in parallel, and
 
 - no requests had been consumed when Instaloader starts.
 
 The latter one implies that restarting or reinstantiating Instaloader often
-within short time is prone to cause a 429. When a request is denied with a 429,
+within short time is prone to cause a 429. If a request is denied with a 429,
 Instaloader retries the request as soon as the temporary ban is assumed to be
 expired. In case the retry continuously fails for some reason, which should not
-happen in normal conditions, consider adjusting the
+happen under normal conditions, consider adjusting the
 :option:`--max-connection-attempts` option.
 
 There have been observations that services, that in their nature offer
@@ -46,7 +46,7 @@ Private but not followed
 You have to follow a private account to access most of its associated
 information.
 
-Login Error
+Login error
 -----------
 
 Instaloader's login *should* work fine, both with and without
@@ -55,13 +55,13 @@ issued when Instagram suspects authentication activity on your account, by
 pointing the user to an URL to be opened in a browser.
 
 Nevertheless, in :issue:`92` and :issue:`615` users reported problems with
-logging in. We recommend to always keep the sessionfile which Instaloader
-creates when using :option:`--login`. If a sessionfile is present,
+logging in. We recommend to always keep the session file which Instaloader
+creates when using :option:`--login`. If a session file is present,
 :option:`--login` does not make make use of the failure-prone login procedure.
 Also, session files usually do not expire and can be copied between different
 computers without any problems.
 
-If you do not have a sessionfile present, you may use the following script
+If you do not have a session file present, you may use the following script
 (:example:`615_import_firefox_session.py`) to workaround login problems by
 importing the session cookies from Firefox and bypassing Instaloader's login and
 so still use Instaloader's logged-in functionality.
@@ -80,5 +80,5 @@ To use this script,
 
 This script also supports specifying a cookie file path, which may be useful if
 you use multiple Firefox profiles or if your operating system has the directory
-structure differently set up. Also, you can specify an alternative sessionfile
+structure differently set up. Also, you can specify an alternative session file
 path.

instaloader/__init__.py

@@ -1,7 +1,7 @@
 """Download pictures (or videos) along with their captions and other metadata from Instagram."""
 
 
-__version__ = '4.5b1'
+__version__ = '4.5.3'
 
 
 try:

instaloader/instaloader.py

@@ -13,7 +13,7 @@ from functools import wraps
 from hashlib import md5
 from io import BytesIO
 from pathlib import Path
-from typing import Any, Callable, ContextManager, IO, Iterator, List, Optional, Set, Union, cast
+from typing import Any, Callable, IO, Iterator, List, Optional, Set, Union, cast
 
 import requests
 import urllib3  # type: ignore
@@ -513,26 +513,28 @@ class Instaloader:
 
         # Download the image(s) / video thumbnail and videos within sidecars if desired
         downloaded = True
-        if self.download_pictures:
-            if post.typename == 'GraphSidecar':
-                edge_number = 1
-                for sidecar_node in post.get_sidecar_nodes():
-                    # Download picture or video thumbnail
-                    if not sidecar_node.is_video or self.download_video_thumbnails is True:
+        if post.typename == 'GraphSidecar':
+            if self.download_pictures or self.download_videos:
+                for edge_number, sidecar_node in enumerate(post.get_sidecar_nodes(), start=1):
+                    if self.download_pictures and (not sidecar_node.is_video or self.download_video_thumbnails):
+                        # Download sidecar picture or video thumbnail (--no-pictures implies --no-video-thumbnails)
                         downloaded &= self.download_pic(filename=filename, url=sidecar_node.display_url,
                                                         mtime=post.date_local, filename_suffix=str(edge_number))
-                    # Additionally download video if available and desired
-                    if sidecar_node.is_video and self.download_videos is True:
+                    if sidecar_node.is_video and self.download_videos:
+                        # Download sidecar video if desired
                         downloaded &= self.download_pic(filename=filename, url=sidecar_node.video_url,
                                                         mtime=post.date_local, filename_suffix=str(edge_number))
-                    edge_number += 1
-            elif post.typename == 'GraphImage':
+        elif post.typename == 'GraphImage':
+            # Download picture
+            if self.download_pictures:
                 downloaded = self.download_pic(filename=filename, url=post.url, mtime=post.date_local)
-            elif post.typename == 'GraphVideo':
-                if self.download_video_thumbnails is True:
+        elif post.typename == 'GraphVideo':
+            # Download video thumbnail (--no-pictures implies --no-video-thumbnails)
+            if self.download_pictures and self.download_video_thumbnails:
+                with self.context.error_catcher("Video thumbnail of {}".format(post)):
                     downloaded = self.download_pic(filename=filename, url=post.url, mtime=post.date_local)
-            else:
-                self.context.error("Warning: {0} has unknown typename: {1}".format(post, post.typename))
+        else:
+            self.context.error("Warning: {0} has unknown typename: {1}".format(post, post.typename))
 
         # Save caption if desired
         metadata_string = _ArbitraryItemFormatter(post).format(self.post_metadata_txt_pattern).strip()
@@ -540,7 +542,7 @@ class Instaloader:
             self.save_caption(filename=filename, mtime=post.date_local, caption=metadata_string)
 
         # Download video if desired
-        if post.is_video and self.download_videos is True:
+        if post.is_video and self.download_videos:
             downloaded &= self.download_pic(filename=filename, url=post.video_url, mtime=post.date_local)
 
         # Download geotags if desired
@@ -548,11 +550,11 @@ class Instaloader:
             self.save_location(filename, post.location, post.date_local)
 
         # Update comments if desired
-        if self.download_comments is True:
+        if self.download_comments:
             self.update_comments(filename=filename, post=post)
 
         # Save metadata as JSON if desired.
-        if self.save_metadata is not False:
+        if self.save_metadata:
             self.save_metadata_json(filename, post)
 
         self.context.log()
@@ -758,17 +760,16 @@ class Instaloader:
                 ),
                 check_bbd=self.check_resume_bbd,
                 enabled=self.resume_prefix is not None
-        ) as resume_info:
-            is_resuming, start_index = resume_info
-            for number, post in enumerate(posts):
-                if max_count is not None and number + start_index >= max_count:
+        ) as (is_resuming, start_index):
+            for number, post in enumerate(posts, start=start_index + 1):
+                if max_count is not None and number > max_count:
                     break
                 if displayed_count is not None:
-                    self.context.log("[{0:{w}d}/{1:{w}d}] ".format(number + start_index + 1, displayed_count,
+                    self.context.log("[{0:{w}d}/{1:{w}d}] ".format(number, displayed_count,
                                                                    w=len(str(displayed_count))),
                                      end="", flush=True)
                 else:
-                    self.context.log("[{:3d}] ".format(number + start_index + 1), end="", flush=True)
+                    self.context.log("[{:3d}] ".format(number), end="", flush=True)
                 if post_filter is not None:
                     try:
                         if not post_filter(post):
@@ -856,7 +857,7 @@ class Instaloader:
         """
         self.context.log("Retrieving saved posts...")
         assert self.context.username is not None  # safe due to @_requires_login; required by typechecker
-        node_iterator = Profile.from_username(self.context, self.context.username).get_saved_posts()
+        node_iterator = Profile.own_profile(self.context).get_saved_posts()
         self.posts_download_loop(node_iterator, ":saved",
                                  fast_update, post_filter,
                                  max_count=max_count, total_count=node_iterator.count)
@@ -1108,11 +1109,11 @@ class Instaloader:
         def _error_raiser(_str):
             yield
 
-        error_handler = cast(Callable[[Optional[str]], ContextManager[None]],
-                             _error_raiser if raise_errors else self.context.error_catcher)
+        # error_handler type is Callable[[Optional[str]], ContextManager[None]] (not supported with Python 3.5.0..3.5.3)
+        error_handler = _error_raiser if raise_errors else self.context.error_catcher
 
         for profile in profiles:
-            with error_handler(profile.username):
+            with error_handler(profile.username):  # type: ignore # (ignore type for Python 3.5 support)
                 profile_name = profile.username
 
                 # Download profile picture

instaloader/instaloadercontext.py

@@ -351,10 +351,10 @@ class InstaloaderContext:
                     raise ConnectionException("\"window._sharedData\" does not contain required keys.")
                 # If GraphQL data is missing in `window._sharedData`, search for it in `__additionalDataLoaded`.
                 if 'graphql' not in post_or_profile_page[0]:
-                    match = re.search(r'window\.__additionalDataLoaded\([^{]+{"graphql":({.*})}\);</script>',
+                    match = re.search(r'window\.__additionalDataLoaded\(.*?({.*"graphql":.*})\);</script>',
                                       resp.text)
                     if match is not None:
-                        post_or_profile_page[0]['graphql'] = json.loads(match.group(1))
+                        post_or_profile_page[0]['graphql'] = json.loads(match.group(1))['graphql']
                 return resp_json
             else:
                 resp_json = resp.json()
@@ -487,7 +487,7 @@ class InstaloaderContext:
                 shutil.copyfileobj(resp.raw, file)
             else:
                 file.write(resp)
-        os.rename(filename + '.temp', filename)
+        os.replace(filename + '.temp', filename)
 
     def get_raw(self, url: str, _attempt=1) -> requests.Response:
         """Downloads a file anonymously.
@@ -549,8 +549,8 @@ class RateController:
 
     def __init__(self, context: InstaloaderContext):
         self._context = context
-        self._graphql_query_timestamps = dict()  # type: Dict[str, List[float]]
-        self._graphql_earliest_next_request_time = 0.0
+        self._query_timestamps = dict()  # type: Dict[str, List[float]]
+        self._earliest_next_request_time = 0.0
 
     def sleep(self, secs: float):
         """Wait given number of seconds."""
@@ -559,38 +559,75 @@ class RateController:
         # pylint:disable=no-self-use
         time.sleep(secs)
 
-    def _dump_query_timestamps(self, current_time: float):
-        windows = [10, 11, 15, 20, 30, 60]
-        print("GraphQL requests:", file=sys.stderr)
-        for query_hash, times in self._graphql_query_timestamps.items():
-            print("  {}".format(query_hash), file=sys.stderr)
-            for window in windows:
-                reqs_in_sliding_window = sum(t > current_time - window * 60 for t in times)
-                print("    last {} minutes: {} requests".format(window, reqs_in_sliding_window), file=sys.stderr)
+    def _dump_query_timestamps(self, current_time: float, failed_query_type: str):
+        windows = [10, 11, 20, 22, 30, 60]
+        self._context.error("Requests within last {} minutes grouped by type:"
+                            .format('/'.join(str(w) for w in windows)),
+                            repeat_at_end=False)
+        for query_type, times in self._query_timestamps.items():
+            reqs_in_sliding_window = [sum(t > current_time - w * 60 for t in times) for w in windows]
+            self._context.error(" {} {:>32}: {}".format(
+                "*" if query_type == failed_query_type else " ",
+                query_type,
+                " ".join("{:4}".format(reqs) for reqs in reqs_in_sliding_window)
+            ), repeat_at_end=False)
 
     def count_per_sliding_window(self, query_type: str) -> int:
-        """Return how many GraphQL requests can be done within the sliding window."""
+        """Return how many requests can be done within the sliding window."""
         # Not static, to allow for the count_per_sliding_window to depend on context-inherent properties, such as
         # whether we are logged in.
-        # pylint:disable=no-self-use,unused-argument
-        return 200
+        # pylint:disable=no-self-use
+        return 75 if query_type in ['iphone', 'other'] else 200
+
+    def _reqs_in_sliding_window(self, query_type: Optional[str], current_time: float, window: float) -> List[float]:
+        if query_type is not None:
+            # timestamps of type query_type
+            relevant_timestamps = self._query_timestamps[query_type]
+        else:
+            # all GraphQL queries, i.e. not 'iphone' or 'other'
+            graphql_query_timestamps = filter(lambda tp: tp[0] not in ['iphone', 'other'],
+                                              self._query_timestamps.items())
+            relevant_timestamps = [t for times in (tp[1] for tp in graphql_query_timestamps) for t in times]
+        return list(filter(lambda t: t > current_time - window, relevant_timestamps))
 
     def query_waittime(self, query_type: str, current_time: float, untracked_queries: bool = False) -> float:
-        """Calculate time needed to wait before GraphQL query can be executed."""
-        sliding_window = 660
-        if query_type not in self._graphql_query_timestamps:
-            self._graphql_query_timestamps[query_type] = []
-        self._graphql_query_timestamps[query_type] = list(filter(lambda t: t > current_time - 60 * 60,
-                                                                 self._graphql_query_timestamps[query_type]))
-        reqs_in_sliding_window = list(filter(lambda t: t > current_time - sliding_window,
-                                             self._graphql_query_timestamps[query_type]))
-        count_per_sliding_window = self.count_per_sliding_window(query_type)
-        if len(reqs_in_sliding_window) < count_per_sliding_window and not untracked_queries:
-            return max(0.0, self._graphql_earliest_next_request_time - current_time)
-        next_request_time = min(reqs_in_sliding_window) + sliding_window + 6
-        if untracked_queries:
-            self._graphql_earliest_next_request_time = next_request_time
-        return max(next_request_time, self._graphql_earliest_next_request_time) - current_time
+        """Calculate time needed to wait before query can be executed."""
+        per_type_sliding_window = 660
+        if query_type not in self._query_timestamps:
+            self._query_timestamps[query_type] = []
+        self._query_timestamps[query_type] = list(filter(lambda t: t > current_time - 60 * 60,
+                                                         self._query_timestamps[query_type]))
+
+        def per_type_next_request_time():
+            reqs_in_sliding_window = self._reqs_in_sliding_window(query_type, current_time, per_type_sliding_window)
+            if len(reqs_in_sliding_window) < self.count_per_sliding_window(query_type):
+                return 0.0
+            else:
+                return min(reqs_in_sliding_window) + per_type_sliding_window + 6
+
+        def gql_accumulated_next_request_time():
+            if query_type in ['iphone', 'other']:
+                return 0.0
+            gql_accumulated_sliding_window = 600
+            gql_accumulated_max_count = 275
+            reqs_in_sliding_window = self._reqs_in_sliding_window(None, current_time, gql_accumulated_sliding_window)
+            if len(reqs_in_sliding_window) < gql_accumulated_max_count:
+                return 0.0
+            else:
+                return min(reqs_in_sliding_window) + gql_accumulated_sliding_window
+
+        def untracked_next_request_time():
+            if untracked_queries:
+                reqs_in_sliding_window = self._reqs_in_sliding_window(query_type, current_time, per_type_sliding_window)
+                self._earliest_next_request_time = min(reqs_in_sliding_window) + per_type_sliding_window + 6
+            return self._earliest_next_request_time
+
+        return max(0.0,
+                   max(
+                       per_type_next_request_time(),
+                       gql_accumulated_next_request_time(),
+                       untracked_next_request_time(),
+                   ) - current_time)
 
     def wait_before_query(self, query_type: str) -> None:
         """This method is called before a query to Instagram. It calls :meth:`RateController.sleep` to wait
@@ -602,24 +639,25 @@ class RateController:
                               .format(round(waittime), datetime.now() + timedelta(seconds=waittime)))
         if waittime > 0:
             self.sleep(waittime)
-        if query_type not in self._graphql_query_timestamps:
-            self._graphql_query_timestamps[query_type] = [time.monotonic()]
+        if query_type not in self._query_timestamps:
+            self._query_timestamps[query_type] = [time.monotonic()]
         else:
-            self._graphql_query_timestamps[query_type].append(time.monotonic())
+            self._query_timestamps[query_type].append(time.monotonic())
 
     def handle_429(self, query_type: str) -> None:
         """This method is called to handle a 429 Too Many Requests response. It calls :meth:`RateController.sleep` to
          wait until we can repeat the same request."""
-        text_for_429 = ("HTTP error code 429 was returned because too many queries occurred in the last time. "
-                        "Please do not use Instagram in your browser or run multiple instances of Instaloader "
-                        "in parallel.")
-        print(textwrap.fill(text_for_429), file=sys.stderr)
         current_time = time.monotonic()
         waittime = self.query_waittime(query_type, current_time, True)
         assert waittime >= 0
-        if waittime > 15:
-            self._context.log("The request will be retried in {} seconds, at {:%H:%M}."
-                              .format(round(waittime), datetime.now() + timedelta(seconds=waittime)))
-        self._dump_query_timestamps(current_time)
+        self._dump_query_timestamps(current_time, query_type)
+        text_for_429 = ("Instagram responded with HTTP error \"429 - Too Many Requests\". Please do not run multiple "
+                        "instances of Instaloader in parallel or within short sequence. Also, do not use any Instagram "
+                        "App while Instaloader is running.")
+        self._context.error(textwrap.fill(text_for_429), repeat_at_end=False)
+        if waittime > 1.5:
+            self._context.error("The request will be retried in {} seconds, at {:%H:%M}."
+                                .format(round(waittime), datetime.now() + timedelta(seconds=waittime)),
+                                repeat_at_end=False)
         if waittime > 0:
             self.sleep(waittime)

instaloader/nodeiterator.py

@@ -81,11 +81,13 @@ class NodeIterator(Iterator[T]):
         self._node_wrapper = node_wrapper
         self._query_variables = query_variables if query_variables is not None else {}
         self._query_referer = query_referer
-        self._data = first_data
         self._page_index = 0
         self._total_index = 0
-        self._best_before = (None if first_data is None else
-                             datetime.now() + NodeIterator._shelf_life)
+        if first_data is not None:
+            self._data = first_data
+            self._best_before = datetime.now() + NodeIterator._shelf_life
+        else:
+            self._data = self._query()
 
     def _query(self, after: Optional[str] = None) -> Dict:
         pagination_variables = {'first': NodeIterator._graphql_page_length}  # type: Dict[str, Any]
@@ -113,8 +115,6 @@ class NodeIterator(Iterator[T]):
         return self
 
     def __next__(self) -> T:
-        if self._data is None:
-            self._data = self._query()
         if self._page_index < len(self._data['edges']):
             node = self._data['edges'][self._page_index]['node']
             page_index, total_index = self._page_index, self._total_index
@@ -193,8 +193,12 @@ class NodeIterator(Iterator[T]):
                 self._query_referer != frozen.query_referer or
                 self._context.username != frozen.context_username):
             raise InvalidArgumentException("Mismatching resume information.")
+        if not frozen.best_before:
+            raise InvalidArgumentException("\"best before\" date missing.")
+        if frozen.remaining_data is None:
+            raise InvalidArgumentException("\"remaining_data\" missing.")
         self._total_index = frozen.total_index
-        self._best_before = datetime.fromtimestamp(frozen.best_before) if frozen.best_before else None
+        self._best_before = datetime.fromtimestamp(frozen.best_before)
         self._data = frozen.remaining_data
 
 
@@ -219,8 +223,7 @@ def resumable_iteration(context: InstaloaderContext,
                load=lambda _, path: FrozenNodeIterator(**json.load(open(path))),
                save=lambda fni, path: json.dump(fni._asdict(), open(path, 'w')),
                format_path=lambda magic: "resume_info_{}.json".format(magic)
-       ) as resume_info:
-           is_resuming, start_index = resume_info
+       ) as (is_resuming, start_index):
            for post in post_iterator:
                do_something_with(post)
 

instaloader/structures.py

@@ -549,7 +549,7 @@ class Profile:
         """
         # pylint:disable=protected-access
         profile = cls(context, {'username': username.lower()})
-        profile._obtain_metadata()  # to raise ProfileNotExistException now in case username is invalid
+        profile._obtain_metadata()  # to raise ProfileNotExistsException now in case username is invalid
         return profile
 
     @classmethod
@@ -579,6 +579,17 @@ class Profile:
         context.profile_id_cache[profile_id] = profile
         return profile
 
+    @classmethod
+    def own_profile(cls, context: InstaloaderContext):
+        """Return own profile if logged-in.
+
+        :param context: :attr:`Instaloader.context`
+
+        .. versionadded:: 4.5.2"""
+        if not context.is_logged_in:
+            raise LoginRequiredException("--login required to access own profile.")
+        return cls(context, context.graphql_query("d6f4427fbe92d846298cf93df0b937d3", {})["data"]["user"])
+
     def _asdict(self):
         json_node = self._node.copy()
         # remove posts to avoid "Circular reference detected" exception
@@ -802,7 +813,6 @@ class Profile:
         if self.username != self._context.username:
             raise LoginRequiredException("--login={} required to get that profile's saved posts.".format(self.username))
 
-        self._obtain_metadata()
         return NodeIterator(
             self._context,
             'f883d95537fbcd400f466f63d42bd8a1',
@@ -810,7 +820,6 @@ class Profile:
             lambda n: Post(self._context, n),
             {'id': self.userid},
             'https://www.instagram.com/{0}/'.format(self.username),
-            self._metadata('edge_saved_media'),
         )
 
     def get_tagged_posts(self) -> NodeIterator[Post]:
@@ -1334,7 +1343,7 @@ class Hashtag:
             yield from (Post(self._context, edge["node"]) for edge in conn["edges"])
 
     def get_all_posts(self) -> Iterator[Post]:
-        """Yields all posts, i.e. all most recent posts and the top posts, in chronological order."""
+        """Yields all posts, i.e. all most recent posts and the top posts, in almost-chronological order."""
         sorted_top_posts = iter(sorted(self.get_top_posts(), key=lambda p: p.date_utc, reverse=True))
         other_posts = self.get_posts()
         next_top = next(sorted_top_posts, None)