Cache Profiles on creation and create one at test_login()

.github/ISSUE_TEMPLATE/bug_report.md

@@ -13,7 +13,7 @@ Steps to reproduce the behavior:
 (e.g. Instaloader command line)
 
 **Expected behavior**
-A clear and concise description of what you expected to happen (even if it seems obvious).
+A clear and concise description of what you expected to happen (if not obvious).
 
 **Error messages and tracebacks**
 If applicable, add error messages and tracebacks to help explain your problem.
@@ -24,9 +24,4 @@ If applicable, add error messages and tracebacks to help explain your problem.
 **Additional context**
 Add any other context about the problem here.
 
-<!--
-Also see https://instaloader.github.io/contributing.html for how to report a bug.
-
-Please understand that a bug issue that lacks the required information cannot be
-processed.
--->
+<!-- please also see https://instaloader.github.io/contributing.html for how to report a bug -->

.github/ISSUE_TEMPLATE/question.md

@@ -7,12 +7,6 @@ labels: question
 
 Your question here...
 
-<!--
-People will be better able and more motivated to provide help if your question
+<!-- Please see our documentation: https://instaloader.github.io/ -->
 
- - has a concise and meaningful title,
- - is not covered by the documentation: https://instaloader.github.io/,
- - is actually related to Instaloader (this is not a Python help forum),
- - has not already been answered here or on other sites such as Stack Overflow: https://stackoverflow.com/questions/tagged/instaloader,
- - is written in a polite and welcoming tone.
--->
+<!-- Basic questions regarding Instaloader's usage might be more appropriate to ask on Stack Overflow -->

.github/funding.yml

@@ -1 +0,0 @@
-custom: "https://instaloader.github.io/contributing.html#donations"

.github/pull_request_template.md

@@ -1,8 +1,7 @@
 
 
 <!--
-Thanks for your willingness to contribute to Instaloader! Please briefly
-describe
+Please describe:
 
 - A motivation for this change, e.g.
   - Fixes # .

AUTHORS.md

@@ -6,4 +6,3 @@ Instaloader is written by
 - Alexander Graf (@aandergr)
 - André Koch-Kramer (@Thammus)
 - Lars Lindqvist (@e5150)
-- ... and many more, see https://github.com/instaloader/instaloader/graphs/contributors

CODE_OF_CONDUCT.md

@@ -1,135 +0,0 @@
-
-# Contributor Covenant Code of Conduct
-
-## Our Pledge
-
-We as members, contributors, and leaders pledge to make participation in our
-community a harassment-free experience for everyone, regardless of age, body
-size, visible or invisible disability, ethnicity, sex characteristics, gender
-identity and expression, level of experience, education, socio-economic status,
-nationality, personal appearance, race, religion, or sexual identity
-and orientation.
-
-We pledge to act and interact in ways that contribute to an open, welcoming,
-diverse, inclusive, and healthy community.
-
-## Our Standards
-
-Examples of behavior that contributes to a positive environment for our
-community include:
-
-* Demonstrating empathy and kindness toward other people
-* Being respectful of differing opinions, viewpoints, and experiences
-* Giving and gracefully accepting constructive feedback
-* Accepting responsibility and apologizing to those affected by our mistakes,
-  and learning from the experience
-* Focusing on what is best not just for us as individuals, but for the
-  overall community
-
-Examples of unacceptable behavior include:
-
-* The use of sexualized language or imagery, and sexual attention or
-  advances of any kind
-* Trolling, insulting or derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or email
-  address, without their explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
-  professional setting
-
-## Enforcement Responsibilities
-
-Community leaders are responsible for clarifying and enforcing our standards of
-acceptable behavior and will take appropriate and fair corrective action in
-response to any behavior that they deem inappropriate, threatening, offensive,
-or harmful.
-
-Community leaders have the right and responsibility to remove, edit, or reject
-comments, commits, code, wiki edits, issues, and other contributions that are
-not aligned to this Code of Conduct, and will communicate reasons for moderation
-decisions when appropriate.
-
-## Scope
-
-This Code of Conduct applies within all community spaces, and also applies when
-an individual is officially representing the community in public spaces.
-Examples of representing our community include using an official e-mail address,
-posting via an official social media account, or acting as an appointed
-representative at an online or offline event.
-
-## Enforcement
-
-Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported to the community leaders responsible for enforcement by opening an
-issue or contacting one or more of the project maintainers.
-
-All complaints will be reviewed and investigated promptly and fairly.
-
-All community leaders are obligated to respect the privacy and security of the
-reporter of any incident.
-
-## Enforcement Guidelines
-
-Community leaders will follow these Community Impact Guidelines in determining
-the consequences for any action they deem in violation of this Code of Conduct:
-
-### 1. Correction
-
-**Community Impact**: Use of inappropriate language or other behavior deemed
-unprofessional or unwelcome in the community.
-
-**Consequence**: A private, written warning from community leaders, providing
-clarity around the nature of the violation and an explanation of why the
-behavior was inappropriate. A public apology may be requested.
-
-### 2. Warning
-
-**Community Impact**: A violation through a single incident or series
-of actions.
-
-**Consequence**: A warning with consequences for continued behavior. No
-interaction with the people involved, including unsolicited interaction with
-those enforcing the Code of Conduct, for a specified period of time. This
-includes avoiding interactions in community spaces as well as external channels
-like social media. Violating these terms may lead to a temporary or
-permanent ban.
-
-### 3. Temporary Ban
-
-**Community Impact**: A serious violation of community standards, including
-sustained inappropriate behavior.
-
-**Consequence**: A temporary ban from any sort of interaction or public
-communication with the community for a specified period of time. No public or
-private interaction with the people involved, including unsolicited interaction
-with those enforcing the Code of Conduct, is allowed during this period.
-Violating these terms may lead to a permanent ban.
-
-### 4. Permanent Ban
-
-**Community Impact**: Demonstrating a pattern of violation of community
-standards, including sustained inappropriate behavior,  harassment of an
-individual, or aggression toward or disparagement of classes of individuals.
-
-**Consequence**: A permanent ban from any sort of public interaction within
-the community.
-
-## Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage],
-version 2.0, available at
-[https://www.contributor-covenant.org/version/2/0/code_of_conduct.html][v2.0].
-
-Community Impact Guidelines were inspired by 
-[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
-
-For answers to common questions about this code of conduct, see the FAQ at
-[https://www.contributor-covenant.org/faq][FAQ]. Translations are available 
-at [https://www.contributor-covenant.org/translations][translations].
-
-[homepage]: https://www.contributor-covenant.org
-[v2.0]: https://www.contributor-covenant.org/version/2/0/code_of_conduct.html
-[Mozilla CoC]: https://github.com/mozilla/diversity
-[FAQ]: https://www.contributor-covenant.org/faq
-[translations]: https://www.contributor-covenant.org/translations
-

LICENSE

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2016-2020 Alexander Graf and André Koch-Kramer.
+Copyright (c) 2016-2019 Alexander Graf and André Koch-Kramer.
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.rst

@@ -29,7 +29,7 @@
 
 .. |downloads| image:: https://pepy.tech/badge/instaloader/month
    :alt: PyPI Download Count
-   :target: https://pepy.tech/project/instaloader
+   :target: https://pepy.tech/project/instaloader/month
 
 .. badges-end
 
@@ -84,9 +84,7 @@ To later **update your local copy** of that profiles, you may run
     instaloader --fast-update profile [profile ...]
 
 If ``--fast-update`` is given, Instaloader stops when arriving at the
-first already-downloaded picture.
-
-When updating profiles, Instaloader
+first already-downloaded picture. When updating profiles, Instaloader
 automatically **detects profile name changes** and renames the target directory
 accordingly.
 

docs/basic-usage.rst

@@ -171,9 +171,6 @@ dirname pattern. The following tokens are defined for usage with
 - ``{mediaid}``
    Integer representation of shortcode.
 
-- ``{filename}``
-   Instagram's internal filename.
-
 - ``{date_utc}`` (same as ``{date}``)
    Creation time in UTC timezone.
    `strftime()-style formatting options <https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior>`__

docs/cli-options.rst

@@ -82,16 +82,6 @@ What to Download of each Post
    Template to write in txt file for each StoryItem. See
    :ref:`metadata-text-files`.
 
-.. option:: --slide
-
-   Download only selected images of a sidecar. You can select single images using their
-   index in the sidecar starting with the leftmost or you can specify a range of images
-   with the following syntax: ``start_index-end_index``. Example:
-   ``--slide 1`` will select only the first image, ``--slide last`` only the last one and ``--slide 1-3`` will select only
-   the first three images.
-
-   .. versionadded:: 4.6
-
 .. option:: --no-metadata-json
 
    Do not create a JSON file containing the metadata of each post.
@@ -220,20 +210,12 @@ How to Download
    used to save the information to resume an interrupted download.  The default
    prefix is ``iterator``.
 
-   Resuming an interrupted download is supported for the following targets:
-    - Profile posts,
-    - Profile IGTV posts (:option:`--igtv`),
-    - Profile tagged posts (:option:`--tagged`),
-    - Saved posts (``:saved``).
-
-   This feature is enabled by default for targets where it is supported;
-   :option:`--resume-prefix` only changes the name of the iterator files.
-
-   To turn this feature off, use :option:`--no-resume`.
-
+   Resuming an interrupted download is supported for most, but not all targets.
    JSON files with resume information are always compressed, regardless of
    :option:`--no-compress-json`.
 
+   This feature is turned off entirely with :option:`--no-resume`.
+
    .. versionadded:: 4.5
 
 .. option:: --no-resume
@@ -246,7 +228,7 @@ How to Download
 .. option:: --user-agent USER_AGENT
 
    User Agent to use for HTTP requests. Per default, Instaloader pretends being
-   Chrome/88 on Linux.
+   Chrome/51.
 
 .. option:: --max-connection-attempts N
 
@@ -256,24 +238,10 @@ How to Download
 
 .. option:: --request-timeout N
 
-   Seconds to wait before timing out a connection request. Defaults to 300.
+   Seconds to wait before timing out a connection request.
 
    .. versionadded:: 4.3
 
-   .. versionchanged:: 4.6
-      Enabled this option by default with a timeout of 300 seconds.
-
-.. option:: --abort-on STATUS_CODE_LIST
-
-   Comma-separated list of HTTP status codes that cause Instaloader to abort,
-   bypassing all retry logic.
-
-   For example, with ``--abort-on=302,400,429``, Instaloader will stop if a
-   request is responded with a 302 redirect, a Bad Request error, or a Too Many
-   Requests error.
-
-   .. versionadded:: 4.7
-
 Miscellaneous Options
 ^^^^^^^^^^^^^^^^^^^^^
 

docs/codesnippets/615_import_firefox_session.py

@@ -23,7 +23,7 @@ def get_cookiefile():
 
 def import_session(cookiefile, sessionfile):
     print("Using cookies from {}.".format(cookiefile))
-    conn = connect(f"file:{cookiefile}?immutable=1", uri=True)
+    conn = connect(cookiefile)
     try:
         cookie_data = conn.execute(
             "SELECT name, value FROM moz_cookies WHERE baseDomain='instagram.com'"

docs/conf.py

@@ -65,7 +65,7 @@ extlinks = {
         'Issue #'
     ),
     'example': (
-        'https://raw.githubusercontent.com/instaloader/instaloader/master/docs/codesnippets/%s',
+        'https://raw.githubusercontent.com/instaloader/instaloader/v' + current_release + '/docs/codesnippets/%s',
         'Example '
     ),
 }

docs/contributing.rst

@@ -11,23 +11,6 @@ Instaloader's development is organized on
 `GitHub <https://github.com/instaloader/instaloader>`__, where Issues and Pull
 Requests are discussed.
 
-Contributing to an open source project is an unmatchable opportunity to improve
-and manifest programming skills. For issues that are good starting points to get
-involved in Instaloader development, as they do not require deep insight into
-Instaloader's code base, have a look at our
-`Issues labelled 'good first issue' <https://github.com/instaloader/instaloader/contribute>`__.
-
-This document covers several ways of how you can contribute back to Instaloader.
-
-Answering Questions
--------------------
-
-The easiest way to help out is to answer questions. If you are interested in
-answering questions regarding Instaloader, good places to begin are
-
-- `Questions tagged 'instaloader' on Stack Overflow <https://stackoverflow.com/questions/tagged/instaloader>`__,
-- `Instaloader Issues labeled 'question' <https://github.com/instaloader/instaloader/issues?q=is%3Aissue+is%3Aopen+label%3Aquestion>`__.
-
 Reporting Bugs
 --------------
 
@@ -49,7 +32,7 @@ reporting a problem, please keep the following in mind:
 
 - Include all **error messages and tracebacks** in the report.
 
-- Even if it seems obvious, describe **which behavior you expected**
+- If not obvious, describe **which behavior you expected**
   instead of what actually happened.
 
 - If we have closed an issue apparently inadvertently or inappropriately, please
@@ -58,39 +41,22 @@ reporting a problem, please keep the following in mind:
 Writing Code or Improving the Documentation
 -------------------------------------------
 
-Improvements of the Instaloader source or its documentation can be proposed as a
-`Pull Request <https://github.com/instaloader/instaloader/pulls>`__.
+Changes of the Instaloader source can be proposed as a
+`Pull Request <https://github.com/instaloader/instaloader/pulls>`__. There are only
+few things to consider:
 
-- Please base your Pull Request on
+- Base your Pull Request on the ``master`` branch if it fixes a bug,
+  or the ``upcoming/v4.X`` branch (if it exists at the moment of submitting the PR)
+  otherwise.
 
-  - ``master``, which will be released with the next minor release, if it is
+- We use `Pylint <https://www.pylint.org/>`__ for error and syntax checking of
+  the source and `MyPy <https://github.com/python/mypy>`__ for type checking.
+  Beware that sometimes it might be better to disable a warning rather than
+  adapting the code to a tool's desires.
 
-    - a bug fix that does not require extensive testing,
-    - an improvement to the documentation,
-
-  - ``upcoming/v4.X``, if it is
-
-    - a new feature,
-    - a bug fix that does require thorough testing before being released to a
-      final version.
-
-- All Pull Requests are analyzed by `Pylint <https://www.pylint.org/>`__ for
-  error and syntax checking of the source and
-  `Mypy <https://github.com/python/mypy>`__ for type checking. You can run them
-  locally with::
-
-     pylint instaloader
-     mypy -m instaloader
-
-- Improvements to the documentation are very welcome. The documentation is
-  created with `Sphinx <https://www.sphinx-doc.org/en/2.0/>`__, version 2,
-  and can be build locally using::
-
-     make -C docs html
-
-- Feel free to create an issue to make sure someone from the Instaloader team
-  agrees that the change might be an improvement, or if you want to discuss
-  your basic proposal, before working on a pull request.
+- The documentation source is located in the ``docs`` folder. The file
+  ``cli-options.rst`` is merely an RST-formatted copy of ``instaloader --help``
+  output, of which the source is in ``instaloader/__main__.py``.
 
 Proposing Features
 ------------------

docs/installation.rst

@@ -33,9 +33,6 @@ To **upgrade Instaloader** to its current version, do::
 - On **Windows 10**, you may download the standalone executable from the
   `current release page <https://github.com/instaloader/instaloader/releases/latest>`__.
 
-- On **Android**, you can use Instaloader with `Termux <https://play.google.com/store/apps/details?id=com.termux>`__
-  after typing ``pkg install python`` and ``pip3 install instaloader``.
-
 - To test the most current **pre-release** version of Instaloader::
 
      pip3 install --pre instaloader

docs/module/exceptions.rst

@@ -47,5 +47,3 @@ Exceptions
 
 
 .. autoexception:: TooManyRequestsException
-
-.. autoexception:: AbortDownloadException

docs/troubleshooting.rst

@@ -9,7 +9,10 @@ Troubleshooting
 ---------------------
 
 Instaloader has a logic to keep track of its requests to Instagram and to obey
-their rate limits. The rate controller assumes that
+their rate limits. Since they are nowhere documented, we try them out
+experimentally. We have a daily cron job running to confirm that Instaloader
+still stays within the rate limits. Nevertheless, the rate control logic assumes
+that
 
 - at one time, Instaloader is the only application that consumes requests, i.e.
   neither the Instagram browser interface, nor a mobile app, nor another
@@ -18,13 +21,7 @@ their rate limits. The rate controller assumes that
 - no requests had been consumed when Instaloader starts.
 
 The latter one implies that restarting or reinstantiating Instaloader often
-within short time is prone to cause a 429.
-
-Since the behavior of the rate controller might change between different
-versions of Instaloader, make sure to use the current version of Instaloader,
-especially when encountering many 429 errors.
-
-If a request is denied with a 429,
+within short time is prone to cause a 429. If a request is denied with a 429,
 Instaloader retries the request as soon as the temporary ban is assumed to be
 expired. In case the retry continuously fails for some reason, which should not
 happen under normal conditions, consider adjusting the
@@ -35,18 +32,13 @@ promiscuous IP addresses, such as cloud, VPN and public proxy services, might be
 subject to significantly stricter limits for anonymous access. However,
 logged-in accesses (see :option:`--login`) do not seem to be affected.
 
-Instaloader allows to adjust the rate controlling behavior by overriding
-:class:`instaloader.RateController`.
-
 Too many queries in the last time
 ---------------------------------
 
 **"Too many queries in the last time"** is not an error. It is a notice that the
 rate limit has almost been reached, according to Instaloader's own rate
-accounting mechanism.
-
-Instaloader allows to adjust the rate controlling behavior by overriding
-:class:`instaloader.RateController`.
+accounting mechanism. We regularly adjust this mechanism to match Instagram's
+current rate limiting.
 
 Private but not followed
 ------------------------
@@ -65,8 +57,9 @@ pointing the user to an URL to be opened in a browser.
 Nevertheless, in :issue:`92` and :issue:`615` users reported problems with
 logging in. We recommend to always keep the session file which Instaloader
 creates when using :option:`--login`. If a session file is present,
-:option:`--login` does not make use of the failure-prone login procedure.
-Also, session files usually do not expire.
+:option:`--login` does not make make use of the failure-prone login procedure.
+Also, session files usually do not expire and can be copied between different
+computers without any problems.
 
 If you do not have a session file present, you may use the following script
 (:example:`615_import_firefox_session.py`) to workaround login problems by

instaloader/__init__.py

@@ -1,7 +1,7 @@
 """Download pictures (or videos) along with their captions and other metadata from Instagram."""
 
 
-__version__ = '4.7rc1'
+__version__ = '4.5'
 
 
 try:

instaloader/__main__.py

@@ -5,12 +5,12 @@ import datetime
 import os
 import re
 import sys
-from argparse import ArgumentParser, ArgumentTypeError, SUPPRESS
+from argparse import ArgumentParser, SUPPRESS
 from typing import List, Optional
 
-from . import (AbortDownloadException, BadCredentialsException, Instaloader, InstaloaderException,
-               InvalidArgumentException, Post, Profile, ProfileNotExistsException, StoryItem,
-               TwoFactorAuthRequiredException, __version__, load_structure_from_file)
+from . import (Instaloader, InstaloaderException, InvalidArgumentException, Post, Profile, ProfileNotExistsException,
+               StoryItem, __version__, load_structure_from_file, TwoFactorAuthRequiredException,
+               BadCredentialsException)
 from .instaloader import get_default_session_filename
 from .instaloadercontext import default_user_agent
 
@@ -27,14 +27,6 @@ def usage_string():
 {0} --help""".format(argv0, len(argv0), '')
 
 
-def http_status_code_list(code_list_str: str) -> List[int]:
-    codes = [int(s) for s in code_list_str.split(',')]
-    for code in codes:
-        if not 100 <= code <= 599:
-            raise ArgumentTypeError("Invalid HTTP status code: {}".format(code))
-    return codes
-
-
 def filterstr_to_filterfunc(filter_str: str, item_type: type):
     """Takes an --post-filter=... or --storyitem-filter=... filter
      specification and makes a filter_func Callable out of it."""
@@ -207,7 +199,7 @@ def _main(instaloader: Instaloader, targetlist: List[str],
             instaloader.context.log("Downloading {} profiles: {}".format(len(profiles),
                                                                          ' '.join([p.username for p in profiles])))
         if profiles and (download_profile_pic or download_posts) and not instaloader.context.is_logged_in:
-            instaloader.context.log("Hint: Use --login to download higher-quality versions of pictures.")
+            instaloader.context.error("Warning: Use --login to download higher-quality versions of pictures.")
         instaloader.download_profiles(profiles,
                                       download_profile_pic, download_posts, download_tagged, download_igtv,
                                       download_highlights, download_stories,
@@ -221,8 +213,6 @@ def _main(instaloader: Instaloader, targetlist: List[str],
                                                    fast_update=fast_update, post_filter=post_filter)
     except KeyboardInterrupt:
         print("\nInterrupted by user.", file=sys.stderr)
-    except AbortDownloadException as exc:
-        print("\nDownload aborted: {}.".format(exc), file=sys.stderr)
     # Save session if it is useful
     if instaloader.context.is_logged_in:
         instaloader.save_session_to_file(sessionfile)
@@ -232,13 +222,14 @@ def _main(instaloader: Instaloader, targetlist: List[str],
             # Instaloader did at least save a session file
             instaloader.context.log("No targets were specified, thus nothing has been downloaded.")
         else:
-            # Instaloader did not do anything
+            # Instloader did not do anything
             instaloader.context.log("usage:" + usage_string())
 
 
 def main():
     parser = ArgumentParser(description=__doc__, add_help=False, usage=usage_string(),
-                            epilog="The complete documentation can be found at "
+                            epilog="Report issues at https://github.com/instaloader/instaloader/issues. "
+                                   "The complete documentation can be found at "
                                    "https://instaloader.github.io/.",
                             fromfile_prefix_chars='+')
 
@@ -277,8 +268,6 @@ def main():
                         help="Do not download regular posts.")
     g_prof.add_argument('--no-profile-pic', action='store_true',
                         help='Do not download profile picture.')
-    g_post.add_argument('--slide', action='store',
-                        help='Set what image/interval of a sidecar you want to download.')
     g_post.add_argument('--no-pictures', action='store_true',
                         help='Do not download post pictures. Cannot be used together with --fast-update. '
                              'Implies --no-video-thumbnails, does not imply --no-videos.')
@@ -377,11 +366,8 @@ def main():
                             'connection fails, it can be manually skipped by hitting CTRL+C. Set this to 0 to retry '
                             'infinitely.')
     g_how.add_argument('--commit-mode', action='store_true', help=SUPPRESS)
-    g_how.add_argument('--request-timeout', metavar='N', type=float, default=300.0,
-                       help='Seconds to wait before timing out a connection request. Defaults to 300.')
-    g_how.add_argument('--abort-on', type=http_status_code_list, metavar="STATUS_CODES",
-                       help='Comma-separated list of HTTP status codes that cause Instaloader to abort, bypassing all '
-                            'retry logic.')
+    g_how.add_argument('--request-timeout', metavar='N', type=float,
+                       help='seconds to wait before timing out a connection request')
 
     g_misc = parser.add_argument_group('Miscellaneous Options')
     g_misc.add_argument('-q', '--quiet', action='store_true',
@@ -439,9 +425,7 @@ def main():
                              max_connection_attempts=args.max_connection_attempts,
                              request_timeout=args.request_timeout,
                              resume_prefix=resume_prefix,
-                             check_resume_bbd=not args.use_aged_resume_files,
-                             slide=args.slide,
-                             fatal_status_codes=args.abort_on)
+                             check_resume_bbd=not args.use_aged_resume_files)
         _main(loader,
               args.profile,
               username=args.login.lower() if args.login is not None else None,

instaloader/exceptions.py

@@ -64,15 +64,3 @@ class QueryReturnedNotFoundException(ConnectionException):
 
 class TooManyRequestsException(ConnectionException):
     pass
-
-
-class AbortDownloadException(Exception):
-    """
-    Exception that is not catched in the error catchers inside the download loop and so aborts the
-    download loop.
-
-    This exception is not a subclass of ``InstaloaderException``.
-
-    .. versionadded:: 4.7
-    """
-    pass

instaloader/instaloader.py

@@ -13,8 +13,7 @@ from functools import wraps
 from hashlib import md5
 from io import BytesIO
 from pathlib import Path
-from typing import Any, Callable, IO, Iterator, List, Optional, Set, Union, cast
-from urllib.parse import urlparse
+from typing import Any, Callable, ContextManager, IO, Iterator, List, Optional, Set, Union, cast
 
 import requests
 import urllib3  # type: ignore
@@ -23,7 +22,7 @@ from .exceptions import *
 from .instaloadercontext import InstaloaderContext, RateController
 from .nodeiterator import NodeIterator, resumable_iteration
 from .structures import (Hashtag, Highlight, JsonExportable, Post, PostLocation, Profile, Story, StoryItem,
-                         load_structure_from_file, save_structure_to_file, PostSidecarNode)
+                         load_structure_from_file, save_structure_to_file)
 
 
 def get_default_session_filename(username: str) -> str:
@@ -102,8 +101,6 @@ class _ArbitraryItemFormatter(string.Formatter):
 
     def get_value(self, key, args, kwargs):
         """Override to substitute {ATTRIBUTE} by attributes of our _item."""
-        if key == 'filename' and isinstance(self._item, (Post, StoryItem, PostSidecarNode)):
-            return "{filename}"
         if hasattr(self._item, key):
             return getattr(self._item, key)
         return super().get_value(key, args, kwargs)
@@ -160,8 +157,6 @@ class Instaloader:
     :param rate_controller: Generator for a :class:`RateController` to override rate controlling behavior
     :param resume_prefix: :option:`--resume-prefix`, or None for :option:`--no-resume`.
     :param check_resume_bbd: Whether to check the date of expiry of resume files and reject them if expired.
-    :param slide: :option:`--slide`
-    :param fatal_status_codes: :option:`--abort-on`
 
     .. attribute:: context
 
@@ -184,15 +179,13 @@ class Instaloader:
                  post_metadata_txt_pattern: str = None,
                  storyitem_metadata_txt_pattern: str = None,
                  max_connection_attempts: int = 3,
-                 request_timeout: float = 300.0,
+                 request_timeout: Optional[float] = None,
                  rate_controller: Optional[Callable[[InstaloaderContext], RateController]] = None,
                  resume_prefix: Optional[str] = "iterator",
-                 check_resume_bbd: bool = True,
-                 slide: Optional[str] = None,
-                 fatal_status_codes: Optional[List[int]] = None):
+                 check_resume_bbd: bool = True):
 
         self.context = InstaloaderContext(sleep, quiet, user_agent, max_connection_attempts,
-                                          request_timeout, rate_controller, fatal_status_codes)
+                                          request_timeout, rate_controller)
 
         # configuration parameters
         self.dirname_pattern = dirname_pattern or "{target}"
@@ -211,31 +204,6 @@ class Instaloader:
         self.resume_prefix = resume_prefix
         self.check_resume_bbd = check_resume_bbd
 
-        self.slide = slide or ""
-        self.slide_start = 0
-        self.slide_end = -1
-        if self.slide != "":
-            splitted = self.slide.split('-')
-            if len(splitted) == 1:
-                if splitted[0] == 'last':
-                    # download only last image of a sidecar
-                    self.slide_start = -1
-                else:
-                    if int(splitted[0]) > 0:
-                        self.slide_start = self.slide_end = int(splitted[0])-1
-                    else:
-                        raise InvalidArgumentException("--slide parameter must be greater than 0.")
-            elif len(splitted) == 2:
-                if splitted[1] == 'last':
-                    self.slide_start = int(splitted[0])-1
-                elif 0 < int(splitted[0]) < int(splitted[1]):
-                    self.slide_start = int(splitted[0])-1
-                    self.slide_end = int(splitted[1])-1
-                else:
-                    raise InvalidArgumentException("Invalid data for --slide parameter.")
-            else:
-                raise InvalidArgumentException("Invalid data for --slide parameter.")
-
     @contextmanager
     def anonymous_copy(self):
         """Yield an anonymous, otherwise equally-configured copy of an Instaloader instance; Then copy its error log."""
@@ -257,9 +225,7 @@ class Instaloader:
             max_connection_attempts=self.context.max_connection_attempts,
             request_timeout=self.context.request_timeout,
             resume_prefix=self.resume_prefix,
-            check_resume_bbd=self.check_resume_bbd,
-            slide=self.slide,
-            fatal_status_codes=self.context.fatal_status_codes)
+            check_resume_bbd=self.check_resume_bbd)
         yield new_loader
         self.context.error_log.extend(new_loader.context.error_log)
         new_loader.context.error_log = []  # avoid double-printing of errors
@@ -335,47 +301,18 @@ class Instaloader:
                             combined_answers.extend(y['answers'])
                         unique_comments_list[-1]['answers'] = get_unique_comments(combined_answers)
             return unique_comments_list
-
-        def get_new_comments(new_comments, start):
-            for idx, comment in enumerate(new_comments, start=start+1):
-                if idx % 250 == 0:
-                    self.context.log('{}'.format(idx), end='…', flush=True)
-                yield comment
-
-        def save_comments(extended_comments):
-            unique_comments = get_unique_comments(extended_comments, combine_answers=True)
-            answer_ids = set(int(answer['id']) for comment in unique_comments for answer in comment.get('answers', []))
-            with open(filename, 'w') as file:
-                file.write(json.dumps(list(filter(lambda t: int(t['id']) not in answer_ids, unique_comments)),
-                                      indent=4))
-
-        base_filename = filename
         filename += '_comments.json'
         try:
             with open(filename) as fp:
                 comments = json.load(fp)
         except (FileNotFoundError, json.decoder.JSONDecodeError):
             comments = list()
-
-        comments_iterator = post.get_comments()
-        try:
-            with resumable_iteration(
-                    context=self.context,
-                    iterator=comments_iterator,
-                    load=load_structure_from_file,
-                    save=save_structure_to_file,
-                    format_path=lambda magic: "{}_{}_{}.json.xz".format(base_filename, self.resume_prefix, magic),
-                    check_bbd=self.check_resume_bbd,
-                    enabled=self.resume_prefix is not None
-            ) as (_is_resuming, start_index):
-                comments.extend(_postcomment_asdict(comment)
-                                for comment in get_new_comments(comments_iterator, start_index))
-        except (KeyboardInterrupt, AbortDownloadException):
-            if comments:
-                save_comments(comments)
-            raise
+        comments.extend(_postcomment_asdict(comment) for comment in post.get_comments())
         if comments:
-            save_comments(comments)
+            comments = get_unique_comments(comments, combine_answers=True)
+            answer_ids = set(int(answer['id']) for comment in comments for answer in comment.get('answers', []))
+            with open(filename, 'w') as file:
+                file.write(json.dumps(list(filter(lambda t: int(t['id']) not in answer_ids, comments)), indent=4))
             self.context.log('comments', end=' ', flush=True)
 
     def save_caption(self, filename: str, mtime: datetime, caption: str) -> None:
@@ -533,7 +470,13 @@ class Instaloader:
 
     def test_login(self) -> Optional[str]:
         """Returns the Instagram username to which given :class:`requests.Session` object belongs, or None."""
-        return self.context.test_login()
+        data = self.context.graphql_query("d6f4427fbe92d846298cf93df0b937d3", {}).get('data', {}).get('user')
+        if data is not None:
+            profile = Profile.from_node(self.context, {'id': data.get('id'),
+                                                       'username': data.get('username'),
+                                                       'profile_pic_url_hd': data.get('profile_pic_url')})
+            return profile.username
+        return None
 
     def login(self, user: str, passwd: str) -> None:
         """Log in to instagram with given username and password and internally store session object.
@@ -555,20 +498,7 @@ class Instaloader:
         .. versionadded:: 4.2"""
         self.context.two_factor_login(two_factor_code)
 
-    @staticmethod
-    def __prepare_filename(filename_template: str, url: Callable[[], str]) -> str:
-        """Replace filename token inside filename_template with url's filename and assure the directories exist.
-
-        .. versionadded:: 4.6"""
-        if "{filename}" in filename_template:
-            filename = filename_template.replace("{filename}",
-                                                 os.path.splitext(os.path.basename(urlparse(url()).path))[0])
-        else:
-            filename = filename_template
-        os.makedirs(os.path.dirname(filename), exist_ok=True)
-        return filename
-
-    def format_filename(self, item: Union[Post, StoryItem, PostSidecarNode], target: Optional[Union[str, Path]] = None):
+    def format_filename(self, item: Union[Post, StoryItem], target: Optional[Union[str, Path]] = None):
         """Format filename of a :class:`Post` or :class:`StoryItem` according to ``filename-pattern`` parameter.
 
         .. versionadded:: 4.1"""
@@ -583,78 +513,32 @@ class Instaloader:
         :return: True if something was downloaded, False otherwise, i.e. file was already there
         """
 
-        def _already_downloaded(path: str) -> bool:
-            if not os.path.isfile(path):
-                return False
-            else:
-                self.context.log(path + ' exists', end=' ', flush=True)
-                return True
-
-        def _all_already_downloaded(path_base, is_videos_enumerated) -> bool:
-            if '{filename}' in self.filename_pattern:
-                # full URL needed to evaluate actual filename, cannot determine at
-                # this point if all sidecar nodes were already downloaded.
-                return False
-            for idx, is_video in is_videos_enumerated:
-                if self.download_pictures and (not is_video or self.download_video_thumbnails):
-                    if not _already_downloaded("{0}_{1}.jpg".format(path_base, idx)):
-                        return False
-                if is_video and self.download_videos:
-                    if not _already_downloaded("{0}_{1}.mp4".format(path_base, idx)):
-                        return False
-            return True
-
         dirname = _PostPathFormatter(post).format(self.dirname_pattern, target=target)
-        filename_template = os.path.join(dirname, self.format_filename(post, target=target))
-        filename = self.__prepare_filename(filename_template, lambda: post.url)
+        filename = os.path.join(dirname, self.format_filename(post, target=target))
+        os.makedirs(os.path.dirname(filename), exist_ok=True)
 
         # Download the image(s) / video thumbnail and videos within sidecars if desired
         downloaded = True
-        if post.typename == 'GraphSidecar':
-            if self.download_pictures or self.download_videos:
-                if not _all_already_downloaded(
-                        filename_template, enumerate(
-                            (post.get_is_videos()[i]
-                             for i in range(self.slide_start % post.mediacount, self.slide_end % post.mediacount + 1)),
-                            start=self.slide_start % post.mediacount + 1
-                        )
-                ):
-                    for edge_number, sidecar_node in enumerate(
-                            post.get_sidecar_nodes(self.slide_start, self.slide_end),
-                            start=self.slide_start % post.mediacount + 1
-                    ):
-                        suffix = str(edge_number)  # type: Optional[str]
-                        if '{filename}' in self.filename_pattern:
-                            suffix = None
-                        if self.download_pictures and (not sidecar_node.is_video or self.download_video_thumbnails):
-                            # pylint:disable=cell-var-from-loop
-                            sidecar_filename = self.__prepare_filename(filename_template,
-                                                                       lambda: sidecar_node.display_url)
-                            # Download sidecar picture or video thumbnail (--no-pictures implies --no-video-thumbnails)
-                            downloaded &= self.download_pic(filename=sidecar_filename, url=sidecar_node.display_url,
-                                                            mtime=post.date_local, filename_suffix=suffix)
-                        if sidecar_node.is_video and self.download_videos:
-                            # pylint:disable=cell-var-from-loop
-                            sidecar_filename = self.__prepare_filename(filename_template,
-                                                                       lambda: sidecar_node.video_url)
-                            # Download sidecar video if desired
-                            downloaded &= self.download_pic(filename=sidecar_filename, url=sidecar_node.video_url,
-                                                            mtime=post.date_local, filename_suffix=suffix)
-                else:
-                    downloaded = False
-        elif post.typename == 'GraphImage':
-            # Download picture
-            if self.download_pictures:
-                downloaded = (not _already_downloaded(filename + ".jpg") and
-                              self.download_pic(filename=filename, url=post.url, mtime=post.date_local))
-        elif post.typename == 'GraphVideo':
-            # Download video thumbnail (--no-pictures implies --no-video-thumbnails)
-            if self.download_pictures and self.download_video_thumbnails:
-                with self.context.error_catcher("Video thumbnail of {}".format(post)):
-                    downloaded = (not _already_downloaded(filename + ".jpg") and
-                                  self.download_pic(filename=filename, url=post.url, mtime=post.date_local))
-        else:
-            self.context.error("Warning: {0} has unknown typename: {1}".format(post, post.typename))
+        if self.download_pictures:
+            if post.typename == 'GraphSidecar':
+                edge_number = 1
+                for sidecar_node in post.get_sidecar_nodes():
+                    # Download picture or video thumbnail
+                    if not sidecar_node.is_video or self.download_video_thumbnails is True:
+                        downloaded &= self.download_pic(filename=filename, url=sidecar_node.display_url,
+                                                        mtime=post.date_local, filename_suffix=str(edge_number))
+                    # Additionally download video if available and desired
+                    if sidecar_node.is_video and self.download_videos is True:
+                        downloaded &= self.download_pic(filename=filename, url=sidecar_node.video_url,
+                                                        mtime=post.date_local, filename_suffix=str(edge_number))
+                    edge_number += 1
+            elif post.typename == 'GraphImage':
+                downloaded = self.download_pic(filename=filename, url=post.url, mtime=post.date_local)
+            elif post.typename == 'GraphVideo':
+                if self.download_video_thumbnails is True:
+                    downloaded = self.download_pic(filename=filename, url=post.url, mtime=post.date_local)
+            else:
+                self.context.error("Warning: {0} has unknown typename: {1}".format(post, post.typename))
 
         # Save caption if desired
         metadata_string = _ArbitraryItemFormatter(post).format(self.post_metadata_txt_pattern).strip()
@@ -662,20 +546,19 @@ class Instaloader:
             self.save_caption(filename=filename, mtime=post.date_local, caption=metadata_string)
 
         # Download video if desired
-        if post.is_video and self.download_videos:
-            downloaded &= (not _already_downloaded(filename + ".mp4") and
-                           self.download_pic(filename=filename, url=post.video_url, mtime=post.date_local))
+        if post.is_video and self.download_videos is True:
+            downloaded &= self.download_pic(filename=filename, url=post.video_url, mtime=post.date_local)
 
         # Download geotags if desired
         if self.download_geotags and post.location:
             self.save_location(filename, post.location, post.date_local)
 
         # Update comments if desired
-        if self.download_comments:
+        if self.download_comments is True:
             self.update_comments(filename=filename, post=post)
 
         # Save metadata as JSON if desired.
-        if self.save_metadata:
+        if self.save_metadata is not False:
             self.save_metadata_json(filename, post)
 
         self.context.log()
@@ -700,12 +583,12 @@ class Instaloader:
 
         def _userid_chunks():
             assert userids is not None
-            userids_per_query = 50
+            userids_per_query = 100
             for i in range(0, len(userids), userids_per_query):
                 yield userids[i:i + userids_per_query]
 
         for userid_chunk in _userid_chunks():
-            stories = self.context.graphql_query("303a4ae99711322310f25250d988f3b7",
+            stories = self.context.graphql_query("bf41e22b1c4ba4c9f31b844ebb7d9056",
                                                  {"reel_ids": userid_chunk, "precomposed_overlay": False})["data"]
             yield from (Story(self.context, media) for media in stories['reels_media'])
 
@@ -757,25 +640,16 @@ class Instaloader:
         :return: True if something was downloaded, False otherwise, i.e. file was already there
         """
 
-        def _already_downloaded(path: str) -> bool:
-            if not os.path.isfile(path):
-                return False
-            else:
-                self.context.log(path + ' exists', end=' ', flush=True)
-                return True
-
         date_local = item.date_local
         dirname = _PostPathFormatter(item).format(self.dirname_pattern, target=target)
-        filename_template = os.path.join(dirname, self.format_filename(item, target=target))
-        filename = self.__prepare_filename(filename_template, lambda: item.url)
+        filename = os.path.join(dirname, self.format_filename(item, target=target))
+        os.makedirs(os.path.dirname(filename), exist_ok=True)
         downloaded = False
         if not item.is_video or self.download_video_thumbnails is True:
-            downloaded = (not _already_downloaded(filename + ".jpg") and
-                          self.download_pic(filename=filename, url=item.url, mtime=date_local))
+            url = item.url
+            downloaded = self.download_pic(filename=filename, url=url, mtime=date_local)
         if item.is_video and self.download_videos is True:
-            filename = self.__prepare_filename(filename_template, lambda: str(item.video_url))
-            downloaded |= (not _already_downloaded(filename + ".mp4") and
-                           self.download_pic(filename=filename, url=item.video_url, mtime=date_local))
+            downloaded |= self.download_pic(filename=filename, url=item.video_url, mtime=date_local)
         # Save caption if desired
         metadata_string = _ArbitraryItemFormatter(item).format(self.storyitem_metadata_txt_pattern).strip()
         if metadata_string:
@@ -987,7 +861,7 @@ class Instaloader:
         """
         self.context.log("Retrieving saved posts...")
         assert self.context.username is not None  # safe due to @_requires_login; required by typechecker
-        node_iterator = Profile.own_profile(self.context).get_saved_posts()
+        node_iterator = Profile.from_username(self.context, self.context.username).get_saved_posts()
         self.posts_download_loop(node_iterator, ":saved",
                                  fast_update, post_filter,
                                  max_count=max_count, total_count=node_iterator.count)
@@ -1239,11 +1113,11 @@ class Instaloader:
         def _error_raiser(_str):
             yield
 
-        # error_handler type is Callable[[Optional[str]], ContextManager[None]] (not supported with Python 3.5.0..3.5.3)
-        error_handler = _error_raiser if raise_errors else self.context.error_catcher
+        error_handler = cast(Callable[[Optional[str]], ContextManager[None]],
+                             _error_raiser if raise_errors else self.context.error_catcher)
 
         for profile in profiles:
-            with error_handler(profile.username):  # type: ignore # (ignore type for Python 3.5 support)
+            with error_handler(profile.username):
                 profile_name = profile.username
 
                 # Download profile picture
@@ -1259,13 +1133,10 @@ class Instaloader:
                     self.save_metadata_json(json_filename, profile)
 
                 # Catch some errors
-                if tagged or igtv or highlights or posts:
-                    if (not self.context.is_logged_in and
-                            profile.is_private):
+                if profile.is_private and (tagged or igtv or highlights or posts):
+                    if not self.context.is_logged_in:
                         raise LoginRequiredException("--login=USERNAME required.")
-                    if (self.context.username != profile.username and
-                            profile.is_private and
-                            not profile.followed_by_viewer):
+                    if not profile.followed_by_viewer and self.context.username != profile.username:
                         raise PrivateProfileNotFollowedException("Private but not followed.")
 
                 # Download tagged, if requested

instaloader/instaloadercontext.py

@@ -25,15 +25,16 @@ def copy_session(session: requests.Session, request_timeout: Optional[float] = N
     new = requests.Session()
     new.cookies = requests.utils.cookiejar_from_dict(requests.utils.dict_from_cookiejar(session.cookies))
     new.headers = session.headers.copy()
-    # Override default timeout behavior.
-    # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
-    new.request = partial(new.request, timeout=request_timeout) # type: ignore
+    if request_timeout is not None:
+        # Override default timeout behavior.
+        # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
+        new.request = partial(new.request, timeout=request_timeout) # type: ignore
     return new
 
 
 def default_user_agent() -> str:
     return 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ' \
-           '(KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36'
+           '(KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36'
 
 
 class InstaloaderContext:
@@ -52,9 +53,8 @@ class InstaloaderContext:
     """
 
     def __init__(self, sleep: bool = True, quiet: bool = False, user_agent: Optional[str] = None,
-                 max_connection_attempts: int = 3, request_timeout: float = 300.0,
-                 rate_controller: Optional[Callable[["InstaloaderContext"], "RateController"]] = None,
-                 fatal_status_codes: Optional[List[int]] = None):
+                 max_connection_attempts: int = 3, request_timeout: Optional[float] = None,
+                 rate_controller: Optional[Callable[["InstaloaderContext"], "RateController"]] = None):
 
         self.user_agent = user_agent if user_agent is not None else default_user_agent()
         self.request_timeout = request_timeout
@@ -75,12 +75,12 @@ class InstaloaderContext:
         # Can be set to True for testing, disables supression of InstaloaderContext._error_catcher
         self.raise_all_errors = False
 
-        # HTTP status codes that should cause an AbortDownloadException
-        self.fatal_status_codes = fatal_status_codes or []
-
         # Cache profile from id (mapping from id to Profile)
         self.profile_id_cache = dict()           # type: Dict[int, Any]
 
+        # Cache profile from name (mapping from username to Profile)
+        self.profile_name_cache = dict()         # type: Dict[str, Any]
+
     @contextmanager
     def anonymous_copy(self):
         session = self._session
@@ -164,9 +164,10 @@ class InstaloaderContext:
                                 'ig_vw': '1920', 'csrftoken': '',
                                 's_network': '', 'ds_user_id': ''})
         session.headers.update(self._default_http_header(empty_session_only=True))
-        # Override default timeout behavior.
-        # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
-        session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
+        if self.request_timeout is not None:
+            # Override default timeout behavior.
+            # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
+            session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
         return session
 
     def save_session_to_file(self, sessionfile):
@@ -179,14 +180,20 @@ class InstaloaderContext:
         session.cookies = requests.utils.cookiejar_from_dict(pickle.load(sessionfile))
         session.headers.update(self._default_http_header())
         session.headers.update({'X-CSRFToken': session.cookies.get_dict()['csrftoken']})
-        # Override default timeout behavior.
-        # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
-        session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
+        if self.request_timeout is not None:
+            # Override default timeout behavior.
+            # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
+            session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
         self._session = session
         self.username = username
 
     def test_login(self) -> Optional[str]:
-        """Not meant to be used directly, use :meth:`Instaloader.test_login`."""
+        """Not meant to be used directly, use :meth:`Instaloader.test_login`.
+
+           .. deprecated:: 4.5
+              :meth:`Instaloader.test_login` now calls :meth:`InstaloaderContext.graphql_query` directly
+              without using this function.
+        """
         data = self.graphql_query("d6f4427fbe92d846298cf93df0b937d3", {})
         return data["data"]["user"]["username"] if data["data"]["user"] is not None else None
 
@@ -207,9 +214,10 @@ class InstaloaderContext:
                                 'ig_vw': '1920', 'ig_cb': '1', 'csrftoken': '',
                                 's_network': '', 'ds_user_id': ''})
         session.headers.update(self._default_http_header())
-        # Override default timeout behavior.
-        # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
-        session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
+        if self.request_timeout is not None:
+            # Override default timeout behavior.
+            # Need to silence mypy bug for this. See: https://github.com/python/mypy/issues/2427
+            session.request = partial(session.request, timeout=self.request_timeout) # type: ignore
         session.get('https://www.instagram.com/web/__mid/')
         csrf_token = session.cookies.get_dict()['csrftoken']
         session.headers.update({'X-CSRFToken': csrf_token})
@@ -292,7 +300,7 @@ class InstaloaderContext:
     def do_sleep(self):
         """Sleep a short time if self.sleep is set. Called before each request to instagram.com."""
         if self.sleep:
-            time.sleep(min(random.expovariate(0.6), 15.0))
+            time.sleep(min(random.expovariate(0.7), 5.0))
 
     def get_json(self, path: str, params: Dict[str, Any], host: str = 'www.instagram.com',
                  session: Optional[requests.Session] = None, _attempt=1) -> Dict[str, Any]:
@@ -320,19 +328,12 @@ class InstaloaderContext:
             if is_other_query:
                 self._rate_controller.wait_before_query('other')
             resp = sess.get('https://{0}/{1}'.format(host, path), params=params, allow_redirects=False)
-            if resp.status_code in self.fatal_status_codes:
-                redirect = " redirect to {}".format(resp.headers['location']) if 'location' in resp.headers else ""
-                raise AbortDownloadException("Query to https://{}/{} responded with \"{} {}\"{}".format(
-                    host, path, resp.status_code, resp.reason, redirect
-                ))
             while resp.is_redirect:
                 redirect_url = resp.headers['location']
                 self.log('\nHTTP redirect from https://{0}/{1} to {2}'.format(host, path, redirect_url))
                 if redirect_url.startswith('https://www.instagram.com/accounts/login'):
-                    if not self.is_logged_in:
-                        raise LoginRequiredException("Redirected to login page. Use --login.")
                     # alternate rate limit exceeded behavior
-                    raise TooManyRequestsException("Redirected to login")
+                    raise TooManyRequestsException("429 Too Many Requests: redirected to login")
                 if redirect_url.startswith('https://{}/'.format(host)):
                     resp = sess.get(redirect_url if redirect_url.endswith('/') else redirect_url + '/',
                                     params=params, allow_redirects=False)
@@ -358,10 +359,10 @@ class InstaloaderContext:
                     raise ConnectionException("\"window._sharedData\" does not contain required keys.")
                 # If GraphQL data is missing in `window._sharedData`, search for it in `__additionalDataLoaded`.
                 if 'graphql' not in post_or_profile_page[0]:
-                    match = re.search(r'window\.__additionalDataLoaded\(.*?({.*"graphql":.*})\);</script>',
+                    match = re.search(r'window\.__additionalDataLoaded\([^{]+{"graphql":({.*})}\);</script>',
                                       resp.text)
                     if match is not None:
-                        post_or_profile_page[0]['graphql'] = json.loads(match.group(1))['graphql']
+                        post_or_profile_page[0]['graphql'] = json.loads(match.group(1))
                 return resp_json
             else:
                 resp_json = resp.json()
@@ -478,7 +479,7 @@ class InstaloaderContext:
 
         .. versionadded:: 4.2.1"""
         with copy_session(self._session, self.request_timeout) as tempsession:
-            tempsession.headers['User-Agent'] = 'Instagram 146.0.0.27.125 (iPhone12,1; iOS 13_3; en_US; en-US; ' \
+            tempsession.headers['User-Agent'] = 'Instagram 123.1.0.26.115 (iPhone12,1; iOS 13_3; en_US; en-US; ' \
                                                 'scale=2.00; 1656x3584; 190542906)'
             for header in ['Host', 'Origin', 'X-Instagram-AJAX', 'X-Requested-With']:
                 tempsession.headers.pop(header, None)
@@ -494,7 +495,7 @@ class InstaloaderContext:
                 shutil.copyfileobj(resp.raw, file)
             else:
                 file.write(resp)
-        os.replace(filename + '.temp', filename)
+        os.rename(filename + '.temp', filename)
 
     def get_raw(self, url: str, _attempt=1) -> requests.Response:
         """Downloads a file anonymously.
@@ -556,9 +557,8 @@ class RateController:
 
     def __init__(self, context: InstaloaderContext):
         self._context = context
-        self._query_timestamps = dict()  # type: Dict[str, List[float]]
-        self._earliest_next_request_time = 0.0
-        self._iphone_earliest_next_request_time = 0.0
+        self._graphql_query_timestamps = dict()  # type: Dict[str, List[float]]
+        self._graphql_earliest_next_request_time = 0.0
 
     def sleep(self, secs: float):
         """Wait given number of seconds."""
@@ -567,132 +567,67 @@ class RateController:
         # pylint:disable=no-self-use
         time.sleep(secs)
 
-    def _dump_query_timestamps(self, current_time: float, failed_query_type: str):
-        windows = [10, 11, 20, 22, 30, 60]
-        self._context.error("Number of requests within last {} minutes grouped by type:"
-                            .format('/'.join(str(w) for w in windows)),
-                            repeat_at_end=False)
-        for query_type, times in self._query_timestamps.items():
-            reqs_in_sliding_window = [sum(t > current_time - w * 60 for t in times) for w in windows]
-            self._context.error(" {} {:>32}: {}".format(
-                "*" if query_type == failed_query_type else " ",
-                query_type,
-                " ".join("{:4}".format(reqs) for reqs in reqs_in_sliding_window)
-            ), repeat_at_end=False)
+    def _dump_query_timestamps(self, current_time: float):
+        windows = [10, 11, 15, 20, 30, 60]
+        print("GraphQL requests:", file=sys.stderr)
+        for query_hash, times in self._graphql_query_timestamps.items():
+            print("  {}".format(query_hash), file=sys.stderr)
+            for window in windows:
+                reqs_in_sliding_window = sum(t > current_time - window * 60 for t in times)
+                print("    last {} minutes: {} requests".format(window, reqs_in_sliding_window), file=sys.stderr)
 
     def count_per_sliding_window(self, query_type: str) -> int:
-        """Return how many requests of the given type can be done within a sliding window of 11 minutes.
-
-        This is called by :meth:`RateController.query_waittime` and allows to simply customize wait times before queries
-        at query_type granularity. Consider overriding :meth:`RateController.query_waittime` directly if you need more
-        control."""
+        """Return how many GraphQL requests can be done within the sliding window."""
         # Not static, to allow for the count_per_sliding_window to depend on context-inherent properties, such as
         # whether we are logged in.
-        # pylint:disable=no-self-use
-        return 75 if query_type == 'other' else 200
-
-    def _reqs_in_sliding_window(self, query_type: Optional[str], current_time: float, window: float) -> List[float]:
-        if query_type is not None:
-            # timestamps of type query_type
-            relevant_timestamps = self._query_timestamps[query_type]
-        else:
-            # all GraphQL queries, i.e. not 'iphone' or 'other'
-            graphql_query_timestamps = filter(lambda tp: tp[0] not in ['iphone', 'other'],
-                                              self._query_timestamps.items())
-            relevant_timestamps = [t for times in (tp[1] for tp in graphql_query_timestamps) for t in times]
-        return list(filter(lambda t: t > current_time - window, relevant_timestamps))
+        # pylint:disable=no-self-use,unused-argument
+        return 200
 
     def query_waittime(self, query_type: str, current_time: float, untracked_queries: bool = False) -> float:
-        """Calculate time needed to wait before query can be executed."""
-        per_type_sliding_window = 660
-        iphone_sliding_window = 1800
-        if query_type not in self._query_timestamps:
-            self._query_timestamps[query_type] = []
-        self._query_timestamps[query_type] = list(filter(lambda t: t > current_time - 60 * 60,
-                                                         self._query_timestamps[query_type]))
-
-        def per_type_next_request_time():
-            reqs_in_sliding_window = self._reqs_in_sliding_window(query_type, current_time, per_type_sliding_window)
-            if len(reqs_in_sliding_window) < self.count_per_sliding_window(query_type):
-                return 0.0
-            else:
-                return min(reqs_in_sliding_window) + per_type_sliding_window + 6
-
-        def gql_accumulated_next_request_time():
-            if query_type in ['iphone', 'other']:
-                return 0.0
-            gql_accumulated_sliding_window = 600
-            gql_accumulated_max_count = 275
-            reqs_in_sliding_window = self._reqs_in_sliding_window(None, current_time, gql_accumulated_sliding_window)
-            if len(reqs_in_sliding_window) < gql_accumulated_max_count:
-                return 0.0
-            else:
-                return min(reqs_in_sliding_window) + gql_accumulated_sliding_window
-
-        def untracked_next_request_time():
-            if untracked_queries:
-                if query_type == "iphone":
-                    reqs_in_sliding_window = self._reqs_in_sliding_window(query_type, current_time,
-                                                                          iphone_sliding_window)
-                    self._iphone_earliest_next_request_time = min(reqs_in_sliding_window) + iphone_sliding_window + 18
-                else:
-                    reqs_in_sliding_window = self._reqs_in_sliding_window(query_type, current_time,
-                                                                          per_type_sliding_window)
-                    self._earliest_next_request_time = min(reqs_in_sliding_window) + per_type_sliding_window + 6
-            return max(self._iphone_earliest_next_request_time, self._earliest_next_request_time)
-
-        def iphone_next_request():
-            if query_type == "iphone":
-                reqs_in_sliding_window = self._reqs_in_sliding_window(query_type, current_time, iphone_sliding_window)
-                if len(reqs_in_sliding_window) >= 199:
-                    return min(reqs_in_sliding_window) + iphone_sliding_window + 18
-            return 0.0
-
-        return max(0.0,
-                   max(
-                       per_type_next_request_time(),
-                       gql_accumulated_next_request_time(),
-                       untracked_next_request_time(),
-                       iphone_next_request(),
-                   ) - current_time)
+        """Calculate time needed to wait before GraphQL query can be executed."""
+        sliding_window = 660
+        if query_type not in self._graphql_query_timestamps:
+            self._graphql_query_timestamps[query_type] = []
+        self._graphql_query_timestamps[query_type] = list(filter(lambda t: t > current_time - 60 * 60,
+                                                                 self._graphql_query_timestamps[query_type]))
+        reqs_in_sliding_window = list(filter(lambda t: t > current_time - sliding_window,
+                                             self._graphql_query_timestamps[query_type]))
+        count_per_sliding_window = self.count_per_sliding_window(query_type)
+        if len(reqs_in_sliding_window) < count_per_sliding_window and not untracked_queries:
+            return max(0.0, self._graphql_earliest_next_request_time - current_time)
+        next_request_time = min(reqs_in_sliding_window) + sliding_window + 6
+        if untracked_queries:
+            self._graphql_earliest_next_request_time = next_request_time
+        return max(next_request_time, self._graphql_earliest_next_request_time) - current_time
 
     def wait_before_query(self, query_type: str) -> None:
-        """This method is called before a query to Instagram.
-
-        It calls :meth:`RateController.query_waittime` to determine the time needed to wait and then calls
-        :meth:`RateController.sleep` to wait until the request can be made."""
+        """This method is called before a query to Instagram. It calls :meth:`RateController.sleep` to wait
+        until the request can be made."""
         waittime = self.query_waittime(query_type, time.monotonic(), False)
         assert waittime >= 0
         if waittime > 15:
-            formatted_waittime = ("{} seconds".format(round(waittime)) if waittime <= 666 else
-                                  "{} minutes".format(round(waittime / 60)))
-            self._context.log("\nToo many queries in the last time. Need to wait {}, until {:%H:%M}."
-                              .format(formatted_waittime, datetime.now() + timedelta(seconds=waittime)))
+            self._context.log("\nToo many queries in the last time. Need to wait {} seconds, until {:%H:%M}."
+                              .format(round(waittime), datetime.now() + timedelta(seconds=waittime)))
         if waittime > 0:
             self.sleep(waittime)
-        if query_type not in self._query_timestamps:
-            self._query_timestamps[query_type] = [time.monotonic()]
+        if query_type not in self._graphql_query_timestamps:
+            self._graphql_query_timestamps[query_type] = [time.monotonic()]
         else:
-            self._query_timestamps[query_type].append(time.monotonic())
+            self._graphql_query_timestamps[query_type].append(time.monotonic())
 
     def handle_429(self, query_type: str) -> None:
-        """This method is called to handle a 429 Too Many Requests response.
-
-        It calls :meth:`RateController.query_waittime` to determine the time needed to wait and then calls
-        :meth:`RateController.sleep` to wait until we can repeat the same request."""
+        """This method is called to handle a 429 Too Many Requests response. It calls :meth:`RateController.sleep` to
+         wait until we can repeat the same request."""
+        text_for_429 = ("HTTP error code 429 was returned because too many queries occurred in the last time. "
+                        "Please do not use Instagram in your browser or run multiple instances of Instaloader "
+                        "in parallel.")
+        print(textwrap.fill(text_for_429), file=sys.stderr)
         current_time = time.monotonic()
         waittime = self.query_waittime(query_type, current_time, True)
         assert waittime >= 0
-        self._dump_query_timestamps(current_time, query_type)
-        text_for_429 = ("Instagram responded with HTTP error \"429 - Too Many Requests\". Please do not run multiple "
-                        "instances of Instaloader in parallel or within short sequence. Also, do not use any Instagram "
-                        "App while Instaloader is running.")
-        self._context.error(textwrap.fill(text_for_429), repeat_at_end=False)
-        if waittime > 1.5:
-            formatted_waittime = ("{} seconds".format(round(waittime)) if waittime <= 666 else
-                                  "{} minutes".format(round(waittime / 60)))
-            self._context.error("The request will be retried in {}, at {:%H:%M}."
-                                .format(formatted_waittime, datetime.now() + timedelta(seconds=waittime)),
-                                repeat_at_end=False)
+        if waittime > 15:
+            self._context.log("The request will be retried in {} seconds, at {:%H:%M}."
+                              .format(round(waittime), datetime.now() + timedelta(seconds=waittime)))
+        self._dump_query_timestamps(current_time)
         if waittime > 0:
             self.sleep(waittime)

instaloader/nodeiterator.py

@@ -5,9 +5,9 @@ import os
 from contextlib import contextmanager
 from datetime import datetime, timedelta
 from lzma import LZMAError
-from typing import Any, Callable, Dict, Iterable, Iterator, NamedTuple, Optional, Tuple, TypeVar
+from typing import Any, Callable, Dict, Iterator, NamedTuple, Optional, Tuple, TypeVar
 
-from .exceptions import AbortDownloadException, InvalidArgumentException, QueryReturnedBadRequestException
+from .exceptions import InvalidArgumentException, QueryReturnedBadRequestException
 from .instaloadercontext import InstaloaderContext
 
 FrozenNodeIterator = NamedTuple('FrozenNodeIterator',
@@ -81,13 +81,11 @@ class NodeIterator(Iterator[T]):
         self._node_wrapper = node_wrapper
         self._query_variables = query_variables if query_variables is not None else {}
         self._query_referer = query_referer
+        self._data = first_data
         self._page_index = 0
         self._total_index = 0
-        if first_data is not None:
-            self._data = first_data
-            self._best_before = datetime.now() + NodeIterator._shelf_life
-        else:
-            self._data = self._query()
+        self._best_before = (None if first_data is None else
+                             datetime.now() + NodeIterator._shelf_life)
 
     def _query(self, after: Optional[str] = None) -> Dict:
         pagination_variables = {'first': NodeIterator._graphql_page_length}  # type: Dict[str, Any]
@@ -115,6 +113,8 @@ class NodeIterator(Iterator[T]):
         return self
 
     def __next__(self) -> T:
+        if self._data is None:
+            self._data = self._query()
         if self._page_index < len(self._data['edges']):
             node = self._data['edges'][self._page_index]['node']
             page_index, total_index = self._page_index, self._total_index
@@ -193,26 +193,21 @@ class NodeIterator(Iterator[T]):
                 self._query_referer != frozen.query_referer or
                 self._context.username != frozen.context_username):
             raise InvalidArgumentException("Mismatching resume information.")
-        if not frozen.best_before:
-            raise InvalidArgumentException("\"best before\" date missing.")
-        if frozen.remaining_data is None:
-            raise InvalidArgumentException("\"remaining_data\" missing.")
         self._total_index = frozen.total_index
-        self._best_before = datetime.fromtimestamp(frozen.best_before)
+        self._best_before = datetime.fromtimestamp(frozen.best_before) if frozen.best_before else None
         self._data = frozen.remaining_data
 
 
 @contextmanager
 def resumable_iteration(context: InstaloaderContext,
-                        iterator: Iterable,
+                        iterator: Iterator,
                         load: Callable[[InstaloaderContext, str], Any],
                         save: Callable[[FrozenNodeIterator, str], None],
                         format_path: Callable[[str], str],
                         check_bbd: bool = True,
                         enabled: bool = True) -> Iterator[Tuple[bool, int]]:
     """
-    High-level context manager to handle a resumable iteration that can be interrupted
-    with a :class:`KeyboardInterrupt` or an :class:`AbortDownloadException`.
+    High-level context manager to handle a resumable iteration that can be interrupted with a KeyboardInterrupt.
 
     It can be used as follows to automatically load a previously-saved state into the iterator, save the iterator's
     state when interrupted, and delete the resume file upon completion::
@@ -240,9 +235,6 @@ def resumable_iteration(context: InstaloaderContext,
     :param format_path: Returns the path to the resume file for the given magic.
     :param check_bbd: Whether to check the best before date and reject an expired FrozenNodeIterator.
     :param enabled: Set to False to disable all functionality and simply execute the inner body.
-
-    .. versionchanged:: 4.7
-       Also interrupt on :class:`AbortDownloadException`.
     """
     if not enabled or not isinstance(iterator, NodeIterator):
         yield False, 0
@@ -266,9 +258,8 @@ def resumable_iteration(context: InstaloaderContext,
             context.error("Warning: Not resuming from {}: {}".format(resume_file_path, exc))
     try:
         yield is_resuming, start_index
-    except (KeyboardInterrupt, AbortDownloadException):
-        if os.path.dirname(resume_file_path):
-            os.makedirs(os.path.dirname(resume_file_path), exist_ok=True)
+    except KeyboardInterrupt:
+        os.makedirs(os.path.dirname(resume_file_path), exist_ok=True)
         save(iterator.freeze(), resume_file_path)
         context.log("\nSaved resume information to {}.".format(resume_file_path))
         raise

instaloader/structures.py

@@ -4,7 +4,7 @@ import re
 from base64 import b64decode, b64encode
 from collections import namedtuple
 from datetime import datetime
-from typing import Any, Dict, Iterable, Iterator, List, Optional, Union
+from typing import Any, Dict, Iterator, List, Optional, Union
 
 from . import __version__
 from .exceptions import *
@@ -128,14 +128,6 @@ class Post:
         """The mediaid is a decimal representation of the media shortcode."""
         return int(self._node['id'])
 
-    @property
-    def title(self) -> Optional[str]:
-        """Title of post"""
-        try:
-            return self._field('title')
-        except KeyError:
-            return None
-
     def __repr__(self):
         return '<Post {}>'.format(self.shortcode)
 
@@ -155,6 +147,10 @@ class Post:
             )
             self._full_metadata_dict = pic_json['data']['shortcode_media']
             if self._full_metadata_dict is None:
+                # issue #449
+                self._context.error("Fetching Post metadata failed (issue #449). "
+                                    "The following data has been returned:\n"
+                                    + json.dumps(pic_json['entry_data'], indent=2))
                 raise BadResponseException("Fetching Post metadata failed.")
             if self.shortcode != self._full_metadata_dict['shortcode']:
                 self._node.update(self._full_metadata_dict)
@@ -200,7 +196,7 @@ class Post:
                 # is contained.
                 # Note that we cannot use Profile.from_id() here since that would lead us into a recursion.
                 owner_struct = self._full_metadata['owner']
-            self._owner_profile = Profile(self._context, owner_struct)
+            self._owner_profile = Profile.from_node(self._context, owner_struct)
         return self._owner_profile
 
     @property
@@ -211,13 +207,7 @@ class Post:
     @property
     def owner_id(self) -> int:
         """The ID of the Post's owner."""
-        # The ID may already be available, e.g. if the post instance was created
-        # from an `hashtag.get_posts()` iterator, so no need to make another
-        # http request.
-        if 'owner' in self._node and 'id' in self._node['owner']:
-            return self._node['owner']['id']
-        else:
-            return self.owner_profile.userid
+        return self.owner_profile.userid
 
     @property
     def date_local(self) -> datetime:
@@ -260,60 +250,26 @@ class Post:
         """Type of post, GraphImage, GraphVideo or GraphSidecar"""
         return self._field('__typename')
 
-    @property
-    def mediacount(self) -> int:
-        """
-        The number of media in a sidecar Post, or 1 if the Post it not a sidecar.
-
-        .. versionadded:: 4.6
-        """
+    def get_sidecar_nodes(self) -> Iterator[PostSidecarNode]:
+        """Sidecar nodes of a Post with typename==GraphSidecar."""
         if self.typename == 'GraphSidecar':
             edges = self._field('edge_sidecar_to_children', 'edges')
-            return len(edges)
-        return 1
-
-    def get_is_videos(self) -> List[bool]:
-        """
-        Return a list containing the ``is_video`` property for each media in the post.
-
-        .. versionadded:: 4.7
-        """
-        if self.typename == 'GraphSidecar':
-            edges = self._field('edge_sidecar_to_children', 'edges')
-            return [edge['node']['is_video'] for edge in edges]
-        return [self.is_video]
-
-    def get_sidecar_nodes(self, start=0, end=-1) -> Iterator[PostSidecarNode]:
-        """
-        Sidecar nodes of a Post with typename==GraphSidecar.
-
-        .. versionchanged:: 4.6
-           Added parameters *start* and *end* to specify a slice of sidecar media.
-        """
-        if self.typename == 'GraphSidecar':
-            edges = self._field('edge_sidecar_to_children', 'edges')
-            if end < 0:
-                end = len(edges)-1
-            if start < 0:
-                start = len(edges)-1
-            if any(edge['node']['is_video'] and 'video_url' not in edge['node'] for edge in edges[start:(end+1)]):
+            if any(edge['node']['is_video'] for edge in edges):
                 # video_url is only present in full metadata, issue #558.
                 edges = self._full_metadata['edge_sidecar_to_children']['edges']
             for idx, edge in enumerate(edges):
-                if start <= idx <= end:
-                    node = edge['node']
-                    is_video = node['is_video']
-                    display_url = node['display_url']
-                    if not is_video and self._context.is_logged_in:
-                        try:
-                            carousel_media = self._iphone_struct['carousel_media']
-                            orig_url = carousel_media[idx]['image_versions2']['candidates'][0]['url']
-                            display_url = re.sub(r'&se=\d+(&?)', r'\1', orig_url)
-                        except (InstaloaderException, KeyError, IndexError) as err:
-                            self._context.error('{} Unable to fetch high quality image version of {}.'.format(
-                                err, self))
-                    yield PostSidecarNode(is_video=is_video, display_url=display_url,
-                                          video_url=node['video_url'] if is_video else None)
+                node = edge['node']
+                is_video = node['is_video']
+                display_url = node['display_url']
+                if not is_video and self._context.is_logged_in:
+                    try:
+                        carousel_media = self._iphone_struct['carousel_media']
+                        orig_url = carousel_media[idx]['image_versions2']['candidates'][0]['url']
+                        display_url = re.sub(r'&se=\d+(&?)', r'\1', orig_url)
+                    except (InstaloaderException, KeyError, IndexError) as err:
+                        self._context.error('{} Unable to fetch high quality image version of {}.'.format(err, self))
+                yield PostSidecarNode(is_video=is_video, display_url=display_url,
+                                      video_url=node['video_url'] if is_video else None)
 
     @property
     def caption(self) -> Optional[str]:
@@ -372,12 +328,6 @@ class Post:
     def video_url(self) -> Optional[str]:
         """URL of the video, or None."""
         if self.is_video:
-            if self._context.is_logged_in:
-                try:
-                    url = self._iphone_struct['video_versions'][0]['url']
-                    return url
-                except (InstaloaderException, KeyError, IndexError) as err:
-                    self._context.error('{} Unable to fetch high quality video version of {}.'.format(err, self))
             return self._field('video_url')
         return None
 
@@ -426,21 +376,18 @@ class Post:
         except KeyError:
             return self._field('edge_media_to_comment', 'count')
 
-    def get_comments(self) -> Iterable[PostComment]:
+    def get_comments(self) -> Iterator[PostComment]:
         r"""Iterate over all comments of the post.
 
         Each comment is represented by a PostComment namedtuple with fields text (string), created_at (datetime),
         id (int), owner (:class:`Profile`) and answers (:class:`~typing.Iterator`\ [:class:`PostCommentAnswer`])
         if available.
-
-        .. versionchanged:: 4.7
-           Change return type to ``Iterable``.
         """
         def _postcommentanswer(node):
             return PostCommentAnswer(id=int(node['id']),
                                      created_at_utc=datetime.utcfromtimestamp(node['created_at']),
                                      text=node['text'],
-                                     owner=Profile(self._context, node['owner']),
+                                     owner=Profile.from_node(self._context, node['owner']),
                                      likes_count=node.get('edge_liked_by', {}).get('count', 0))
 
         def _postcommentanswers(node):
@@ -469,15 +416,16 @@ class Post:
                                answers=_postcommentanswers(node))
         if self.comments == 0:
             # Avoid doing additional requests if there are no comments
-            return []
+            return
 
         comment_edges = self._field('edge_media_to_comment', 'edges')
         answers_count = sum([edge['node'].get('edge_threaded_comments', {}).get('count', 0) for edge in comment_edges])
 
         if self.comments == len(comment_edges) + answers_count:
             # If the Post's metadata already contains all parent comments, don't do GraphQL requests to obtain them
-            return [_postcomment(comment['node']) for comment in comment_edges]
-        return NodeIterator(
+            yield from (_postcomment(comment['node']) for comment in comment_edges)
+            return
+        yield from NodeIterator(
             self._context,
             '97b41c52301f77ce508f55e66d17620e',
             lambda d: d['data']['shortcode_media']['edge_media_to_parent_comment'],
@@ -487,27 +435,20 @@ class Post:
         )
 
     def get_likes(self) -> Iterator['Profile']:
-        """
-        Iterate over all likes of the post. A :class:`Profile` instance of each likee is yielded.
-
-        .. versionchanged:: 4.5.4
-           Require being logged in (as required by Instagram).
-        """
-        if not self._context.is_logged_in:
-            raise LoginRequiredException("--login required to access likes of a post.")
+        """Iterate over all likes of the post. A :class:`Profile` instance of each likee is yielded."""
         if self.likes == 0:
             # Avoid doing additional requests if there are no comments
             return
         likes_edges = self._field('edge_media_preview_like', 'edges')
         if self.likes == len(likes_edges):
             # If the Post's metadata already contains all likes, don't do GraphQL requests to obtain them
-            yield from (Profile(self._context, like['node']) for like in likes_edges)
+            yield from (Profile.from_node(self._context, like['node']) for like in likes_edges)
             return
         yield from NodeIterator(
             self._context,
             '1cb6ec562846122743b61e492c85999f',
             lambda d: d['data']['shortcode_media']['edge_liked_by'],
-            lambda n: Profile(self._context, n),
+            lambda n: Profile.from_node(self._context, n),
             {'shortcode': self.shortcode},
             'https://www.instagram.com/p/{0}/'.format(self.shortcode),
         )
@@ -533,7 +474,7 @@ class Post:
         .. versionadded:: 4.4
         """
         return ([] if not self.is_sponsored else
-                [Profile(self._context, edge['node']['sponsor']) for edge in
+                [Profile.from_node(self._context, edge['node']['sponsor']) for edge in
                  self._field('edge_media_to_sponsor_user', 'edges')])
 
     @property
@@ -596,6 +537,17 @@ class Profile:
             # if loaded from JSON with load_structure_from_file()
             self._iphone_struct_ = node['iphone_struct']
 
+    @classmethod
+    def from_node(cls, context: InstaloaderContext, node: Dict[str, Any]) -> 'Profile':
+        assert 'username' in node
+        if node['username'] in context.profile_name_cache:
+            return context.profile_name_cache[node['username']]
+        profile = cls(context, node)
+        if 'id' in node:
+            context.profile_id_cache[node['id']] = profile
+        context.profile_name_cache[profile.username] = profile
+        return profile
+
     @classmethod
     def from_username(cls, context: InstaloaderContext, username: str):
         """Create a Profile instance from a given username, raise exception if it does not exist.
@@ -607,8 +559,11 @@ class Profile:
         :raises: :class:`ProfileNotExistsException`
         """
         # pylint:disable=protected-access
-        profile = cls(context, {'username': username.lower()})
-        profile._obtain_metadata()  # to raise ProfileNotExistsException now in case username is invalid
+        if username.lower() in context.profile_name_cache:
+            return context.profile_name_cache[username.lower()]
+        profile = cls.from_node(context, {'username': username.lower()})
+        profile._obtain_metadata()  # to raise ProfileNotExistException now in case username is invalid
+        context.profile_id_cache[profile.userid] = profile
         return profile
 
     @classmethod
@@ -631,23 +586,10 @@ class Profile:
                                       'include_highlight_reels': False},
                                      rhx_gis=context.root_rhx_gis)['data']['user']
         if data:
-            profile = cls(context, data['reel']['owner'])
+            return cls.from_node(context, data['reel']['owner'])
         else:
             raise ProfileNotExistsException("No profile found, the user may have blocked you (ID: " +
                                             str(profile_id) + ").")
-        context.profile_id_cache[profile_id] = profile
-        return profile
-
-    @classmethod
-    def own_profile(cls, context: InstaloaderContext):
-        """Return own profile if logged-in.
-
-        :param context: :attr:`Instaloader.context`
-
-        .. versionadded:: 4.5.2"""
-        if not context.is_logged_in:
-            raise LoginRequiredException("--login required to access own profile.")
-        return cls(context, context.graphql_query("d6f4427fbe92d846298cf93df0b937d3", {})["data"]["user"])
 
     def _asdict(self):
         json_node = self._node.copy()
@@ -856,7 +798,7 @@ class Profile:
         self._obtain_metadata()
         return NodeIterator(
             self._context,
-            '003056d32c2554def87228bc3fd9668a',
+            '472f257a40c653c64c666ce877d59d2b',
             lambda d: d['data']['user']['edge_owner_to_timeline_media'],
             lambda n: Post(self._context, n, self),
             {'id': self.userid},
@@ -872,6 +814,7 @@ class Profile:
         if self.username != self._context.username:
             raise LoginRequiredException("--login={} required to get that profile's saved posts.".format(self.username))
 
+        self._obtain_metadata()
         return NodeIterator(
             self._context,
             'f883d95537fbcd400f466f63d42bd8a1',
@@ -879,6 +822,7 @@ class Profile:
             lambda n: Post(self._context, n),
             {'id': self.userid},
             'https://www.instagram.com/{0}/'.format(self.username),
+            self._metadata('edge_saved_media'),
         )
 
     def get_tagged_posts(self) -> NodeIterator[Post]:
@@ -928,7 +872,7 @@ class Profile:
             self._context,
             '37479f2b8209594dde7facb0d904896a',
             lambda d: d['data']['user']['edge_followed_by'],
-            lambda n: Profile(self._context, n),
+            lambda n: Profile.from_node(self._context, n),
             {'id': str(self.userid)},
             'https://www.instagram.com/{0}/'.format(self.username),
         )
@@ -947,7 +891,7 @@ class Profile:
             self._context,
             '58712303d941c6855d4e888c5f0cd22f',
             lambda d: d['data']['user']['edge_follow'],
-            lambda n: Profile(self._context, n),
+            lambda n: Profile.from_node(self._context, n),
             {'id': str(self.userid)},
             'https://www.instagram.com/{0}/'.format(self.username),
         )
@@ -962,7 +906,7 @@ class Profile:
         if not self._context.is_logged_in:
             raise LoginRequiredException("--login required to get a profile's similar accounts.")
         self._obtain_metadata()
-        yield from (Profile(self._context, edge["node"]) for edge in
+        yield from (Profile.from_node(self._context, edge["node"]) for edge in
                     self._context.graphql_query("ad99dd9d3646cc3c0dda65debcd266a7",
                                                 {"user_id": str(self.userid), "include_chaining": True},
                                                 "https://www.instagram.com/{0}/"
@@ -984,17 +928,11 @@ class StoryItem:
         self._context = context
         self._node = node
         self._owner_profile = owner_profile
-        self._iphone_struct_ = None
-        if 'iphone_struct' in node:
-            # if loaded from JSON with load_structure_from_file()
-            self._iphone_struct_ = node['iphone_struct']
 
     def _asdict(self):
         node = self._node
         if self._owner_profile:
             node['owner'] = self._owner_profile._asdict()
-        if self._iphone_struct_:
-            node['iphone_struct'] = self._iphone_struct_
         return node
 
     @property
@@ -1019,15 +957,6 @@ class StoryItem:
     def __hash__(self) -> int:
         return hash(self.mediaid)
 
-    @property
-    def _iphone_struct(self) -> Dict[str, Any]:
-        if not self._context.is_logged_in:
-            raise LoginRequiredException("--login required to access iPhone media info endpoint.")
-        if not self._iphone_struct_:
-            data = self._context.get_iphone_json(path='api/v1/media/{}/info/'.format(self.mediaid), params={})
-            self._iphone_struct_ = data['items'][0]
-        return self._iphone_struct_
-
     @property
     def owner_profile(self) -> Profile:
         """:class:`Profile` instance of the story item's owner."""
@@ -1079,13 +1008,6 @@ class StoryItem:
     @property
     def url(self) -> str:
         """URL of the picture / video thumbnail of the StoryItem"""
-        if self.typename == "GraphStoryImage" and self._context.is_logged_in:
-            try:
-                orig_url = self._iphone_struct['image_versions2']['candidates'][0]['url']
-                url = re.sub(r'&se=\d+(&?)', r'\1', orig_url)
-                return url
-            except (InstaloaderException, KeyError, IndexError) as err:
-                self._context.error('{} Unable to fetch high quality image version of {}.'.format(err, self))
         return self._node['display_resources'][-1]['src']
 
     @property
@@ -1158,14 +1080,14 @@ class Story:
 
     @property
     def last_seen_local(self) -> Optional[datetime]:
-        """Timestamp of the most recent StoryItem that has been watched or None (local time zone)."""
+        """Timestamp when the story has last been watched or None (local time zone)."""
         if self._node['seen']:
             return datetime.fromtimestamp(self._node['seen'])
         return None
 
     @property
     def last_seen_utc(self) -> Optional[datetime]:
-        """Timestamp of the most recent StoryItem that has been watched or None (UTC)."""
+        """Timestamp when the story has last been watched or None (UTC)."""
         if self._node['seen']:
             return datetime.utcfromtimestamp(self._node['seen'])
         return None
@@ -1189,7 +1111,7 @@ class Story:
     def owner_profile(self) -> Profile:
         """:class:`Profile` instance of the story owner."""
         if not self._owner_profile:
-            self._owner_profile = Profile(self._context, self._node['user'])
+            self._owner_profile = Profile.from_node(self._context, self._node['user'])
         return self._owner_profile
 
     @property
@@ -1247,7 +1169,7 @@ class Highlight(Story):
     def owner_profile(self) -> Profile:
         """:class:`Profile` instance of the highlights' owner."""
         if not self._owner_profile:
-            self._owner_profile = Profile(self._context, self._node['owner'])
+            self._owner_profile = Profile.from_node(self._context, self._node['owner'])
         return self._owner_profile
 
     @property
@@ -1431,13 +1353,9 @@ class Hashtag:
         next_other = next(other_posts, None)
         while next_top is not None or next_other is not None:
             if next_other is None:
-                assert next_top is not None
-                yield next_top
                 yield from sorted_top_posts
                 break
             if next_top is None:
-                assert next_other is not None
-                yield next_other
                 yield from other_posts
                 break
             if next_top == next_other:
@@ -1482,7 +1400,7 @@ class TopSearchResults:
             user_node = user['user']
             if 'pk' in user_node:
                 user_node['id'] = user_node['pk']
-            yield Profile(self._context, user_node)
+            yield Profile.from_node(self._context, user_node)
 
     def get_prefixed_usernames(self) -> Iterator[str]:
         """
@@ -1575,7 +1493,7 @@ def load_structure_from_file(context: InstaloaderContext, filename: str) -> Json
         if node_type == "Post":
             return Post(context, json_structure['node'])
         elif node_type == "Profile":
-            return Profile(context, json_structure['node'])
+            return Profile.from_node(context, json_structure['node'])
         elif node_type == "StoryItem":
             return StoryItem(context, json_structure['node'])
         elif node_type == "Hashtag":

setup.py

@@ -38,7 +38,6 @@ setup(
     name='instaloader',
     version=get_version(),
     packages=['instaloader'],
-    package_data={'instaloader': ['py.typed']},
     url='https://instaloader.github.io/',
     license='MIT',
     author='Alexander Graf, André Koch-Kramer',
@@ -49,7 +48,7 @@ setup(
     install_requires=requirements,
     python_requires='>=3.5',
     entry_points={'console_scripts': ['instaloader=instaloader.__main__:main']},
-    zip_safe=False,
+    zip_safe=True,
     keywords=keywords,
     classifiers=[
         'Development Status :: 5 - Production/Stable',