Release of version 3.2.2

Fixes #75 as suggested by @e5150.

.travis.yml

@@ -9,7 +9,7 @@ python:
 install:
  - pip install pylint requests
 script:
- - python3 -m pylint -r n -d bad-whitespace,missing-docstring,too-many-arguments,locally-disabled,line-too-long,too-many-public-methods,too-many-lines,too-many-instance-attributes,too-many-locals,too-many-branches,too-many-statements instaloader
+ - python3 -m pylint -r n -d bad-whitespace,missing-docstring,too-many-arguments,locally-disabled,line-too-long,too-many-public-methods,too-many-lines,too-many-instance-attributes,too-many-locals,too-many-branches,too-many-statements,inconsistent-return-statements instaloader
 deploy:
   provider: pypi
   user: aandergr

README.rst

@@ -1,39 +1,37 @@
 Instaloader
 ===========
 
-Installation
-------------
+::
 
-.. installation-start
+    $ pip3 install instaloader
 
-Instaloader requires `Python <https://www.python.org/>`__, at least
-version 3.5.  If you have `pip <https://pypi.python.org/pypi/pip>`__
-installed, you may install Instaloader using
+    $ instaloader profile [profile ...]
+
+**Instaloader**
+
+- downloads **public and private profiles, hashtags, user stories and
+  feeds**,
+
+- downloads **comments, geotags and captions** of each post,
+
+- automatically **detects profile name changes** and renames the target
+  directory accordingly,
+
+- allows **fine-grained customization** of filters and where to store
+  downloaded media.
 
 ::
 
-    pip3 install instaloader
+    instaloader [--comments] [--geotags] [--stories]
+                [--login YOUR-USERNAME] [--fast-update]
+                profile | "#hashtag" | :stories | :feed
 
-Alternatively, to get the most current version of Instaloader from our
-`Git repository <https://github.com/Thammus/instaloader>`__:
+`Instaloader Documentation <https://instaloader.readthedocs.io/>`__
 
-::
-
-    pip3 install git+https://github.com/Thammus/instaloader
-
-(pass ``--upgrade`` to upgrade if Instaloader is already installed)
-
-Instaloader requires
-`requests <http://python-requests.org/>`__, which
-will be installed automatically, if it is not already installed.
-
-.. installation-end
 
 How to Automatically Download Pictures from Instagram
 -----------------------------------------------------
 
-.. basic-usage-start
-
 To **download all pictures and videos of a profile**, as well as the
 **profile picture**, do
 
@@ -52,8 +50,8 @@ To later **update your local copy** of that profiles, you may run
 
 If ``--fast-update`` is given, Instaloader stops when arriving at the
 first already-downloaded picture. When updating profiles, Instaloader
-automatically **detects profile name changes** and renames the target
-directory accordingly.
+automatically **detects profile name changes** and renames the target directory
+accordingly.
 
 Instaloader can also be used to **download private profiles**. To do so,
 invoke it with
@@ -62,299 +60,13 @@ invoke it with
 
     instaloader --login=your_username profile [profile ...]
 
-When logging in, Instaloader **stores the session cookies** in a file in
-your temporary directory, which will be reused later the next time
-``--login`` is given.  So you can download private profiles
-**non-interactively** when you already have a valid session cookie file.
+When logging in, Instaloader **stores the session cookies** in a file in your
+temporary directory, which will be reused later the next time ``--login``
+is given.  So you can download private profiles **non-interactively** when you
+already have a valid session cookie file.
 
-What to Download
-^^^^^^^^^^^^^^^^
+`Instaloader Documentation <https://instaloader.readthedocs.io/basic-usage.html>`__
 
-Instaloader does not only download media by-profile. More generally, you
-may specify the following targets:
-
-- ``profile``: Public profile, or private profile with ``--login``,
-
-- ``"#hashtag"``: Posts with a certain **hashtag** (the quotes are
-  usually neccessary),
-
-- ``:stories``: The currently-visible **stories** of your followees
-  (requires ``--login``),
-
-- ``:feed``: Your **feed** (requires ``--login``),
-
-- ``@profile``: All profiles which are followed by ``profile``, i.e. the
-  *followees* of ``profile`` (requires ``--login``).
-
-Instaloader goes through all media matching the specified targets and
-downloads the pictures and videos and their captions. You can specify
-``--comments`` to also **download comments** of each post and
-``--geotags`` to **download geotags** of each post and save them as
-Google Maps link.  For each profile you download, ``--stories``
-instructs Instaloader to **download the user's stories**.
-
-Filename Specification
-^^^^^^^^^^^^^^^^^^^^^^
-
-For each target, Instaloader creates a directory named after the target,
-i.e. ``profile``, ``#hashtag``, ``:feed``, etc. and therein saves the
-posts in files named after the post's timestamp.
-
-``--dirname-pattern`` allows to configure the directory name of each
-target. The default is ``--dirname-pattern={target}``. In the dirname
-pattern, the token ``{target}`` is replaced by the target name, and
-``{profile}`` is replaced by the owner of the post which is downloaded.
-
-``--filename-pattern`` configures the path of the post's files relative
-to the target directory. The default is ``--filename-pattern={date}``.
-The tokens ``{target}`` and ``{profile}`` are replaced like in the
-dirname pattern. Further, the tokens ``{date}`` and ``{shortcode}`` are
-defined.
-
-For example, encode the poster's profile name in the filenames with:
-
-::
-
-    instaloader --filename-pattern={date}_{profile} "#hashtag"
-
-The pattern string is formatted with Python's string formatter. This
-gives additional flexibilty for pattern specification. For example,
-`strftime-style formatting options <https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior>`__
-are supported for the post's
-timestamp. The default for ``{date}`` is ``{date:%Y-%m-%d_%H-%M-%S}``.
-
-Filter Posts
-^^^^^^^^^^^^
-
-The ``--only-if`` option allows to specify criterias that posts have to
-meet to be downloaded. If not given, all posts are downloaded. It must
-be a boolean Python expression where the variables ``likes``,
-``comments``, ``viewer_has_liked``, ``is_video``, and many
-more are defined.
-
-A few examples:
-
-To **download the pictures from your feed that you have liked**:
-
-::
-
-    instaloader --login=your_username --only-if=viewer_has_liked :feed
-
-Or you might only want to download **posts that either you liked or were
-liked by many others**:
-
-::
-
-    instaloader --login=your_username --only-if="likes>100 or viewer_has_liked" profile
-
-Or you may **skip videos**:
-
-::
-
-    instaloader --only-if="not is_video" target
-
-Or you may filter by hashtags that occur in the Post's caption. For
-example, to download posts of kittens that are cute: ::
-
-    instaloader --only-if="'cute' in caption_hashtags" "#kitten"
-
-.. basic-usage-end
-
-(For a more complete description of the ``-only-if`` option, refer to
-the `Instaloader Documentation <https://instaloader.readthedocs.io/basic-usage.html#filter-posts>`__)
-
-
-Advanced Options
-----------------
-
-.. cli-options-start
-
-The following flags can be given to Instaloader to specify how profiles should
-be downloaded.
-
-To get a list of all flags, their abbreviations and their descriptions, you may
-run ``instaloader --help``.
-
-What to Download
-^^^^^^^^^^^^^^^^
-
-Specify a list of profiles or #hashtags. For each of these, Instaloader
-creates a folder and downloads all posts along with the pictures's
-captions and the current **profile picture**. If an already-downloaded profile
-has been renamed, Instaloader automatically **finds it by its unique ID** and
-renames the folder likewise.
-
-Instead of a *profile* or a *#hashtag*, the special targets
-``:feed`` (pictures from your feed) and
-``:stories`` (stories of your followees) can be specified.
-
---profile-pic-only         Only download profile picture.
---no-videos                Do not download videos.
---geotags                  **Download geotags** when available. Geotags are stored as
-                           a text file with the location's name and a Google Maps
-                           link. This requires an additional request to the
-                           Instagram server for each picture, which is why it is
-                           disabled by default.
---no-geotags               Do not store geotags, even if they can be obtained
-                           without any additional request.
---comments                 Download and update comments for each post. This
-                           requires an additional request to the Instagram server
-                           for each post, which is why it is disabled by default.
---no-captions              Do not store media captions, although no additional
-                           request is needed to obtain them.
---stories                  Also **download stories** of each profile that is
-                           downloaded. Requires ``--login``.
---stories-only             Rather than downloading regular posts of each
-                           specified profile, only download stories.
-                           Requires ``--login``.
---only-if filter           Expression that, if given, must evaluate to True for each post to
-                           be downloaded. Must be a syntactically valid python
-                           expression. Variables are evaluated to
-                           ``instaloader.Post`` attributes.
-                           Example: ``--only-if=viewer_has_liked``.
-
-
-When to Stop Downloading
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-If none of these options are given, Instaloader goes through all pictures
-matching the specified targets.
-
---fast-update              For each target, stop when encountering the first
-                           already-downloaded picture. This flag is recommended
-                           when you use Instaloader to update your personal
-                           Instagram archive.
---count COUNT              Do not attempt to download more than COUNT posts.
-                           Applies only to ``#hashtag``, ``:feed-all`` and ``:feed-liked``.
-
-
-Login (Download Private Profiles)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Instaloader can **login to Instagram**. This allows downloading private
-profiles. To login, pass the ``--login`` option. Your session cookie (not your
-password!) will be saved to a local file to be reused next time you want
-Instaloader to login.
-
---login YOUR-USERNAME      Login name (profile name) for your Instagram account.
---sessionfile SESSIONFILE  Path for loading and storing session key file.
-                           Defaults to a path
-                           within your temporary directory, encoding your local
-                           username and your Instagram profile name.
---password YOUR-PASSWORD   Password for your Instagram account. Without this
-                           option, you'll be prompted for your password
-                           interactively if there is not yet a valid session
-                           file.
-
-How to Download
-^^^^^^^^^^^^^^^
-
---dirname-pattern DIRNAME_PATTERN
-                           Name of directory where to store posts. ``{profile}``
-                           is replaced by the profile name, ``{target}`` is replaced
-                           by the target you specified, i.e. either ``:feed``,
-                           ``#hashtag`` or the profile name. Defaults to ``{target}``.
---filename-pattern FILENAME_PATTERN
-                           Prefix of filenames. Posts are stored in the
-                           directory whose pattern is given with ``--dirname-pattern``.
-                           ``{profile}`` is replaced by the profile name,
-                           ``{target}`` is replaced by the target you specified, i.e.
-                           either ``:feed``, ``#hashtag`` or the profile name. Also, the
-                           fields ``{date}`` and ``{shortcode}`` can be specified.
-                           Defaults to ``{date:%Y-%m-%d_%H-%M-%S}``.
---user-agent USER_AGENT    User Agent to use for HTTP requests. Per default,
-                           Instaloader pretends being Chrome/51.
-
-Miscellaneous Options
-^^^^^^^^^^^^^^^^^^^^^
-
---quiet                    Disable user interaction, i.e. do not print messages
-                           (except errors) and fail if login credentials are
-                           needed but not given. This makes Instaloader
-                           **suitable as a cron job**.
-
-.. cli-options-end
-
-Usage as Python module
-----------------------
-
-.. as-module-intro-start
-
-You may also use parts of Instaloader as library to do other interesting
-things.
-
-For example, to get a list of all followees and a list of all followers of a profile, do
-
-.. code:: python
-
-    import instaloader
-
-    # Get instance
-    loader = instaloader.Instaloader()
-
-    # Login
-    loader.interactive_login(USERNAME)
-
-    # Print followees
-    print(PROFILE + " follows these profiles:")
-    for f in loader.get_followees(PROFILE):
-        print("\t%s\t%s" % (f['username'], f['full_name']))
-
-    # Print followers
-    print("Followers of " + PROFILE + ":")
-    for f in loader.get_followers(PROFILE):
-        print("\t%s\t%s" % (f['username'], f['full_name']))
-
-Then, you may download all pictures of all followees with
-
-.. code:: python
-
-    for f in loader.get_followees(PROFILE):
-        loader.download_profile(f['username'])
-
-You could also download your last 20 liked pics with
-
-.. code:: python
-
-    loader.download_feed_posts(max_count=20, fast_update=True,
-                               filter_func=lambda post: post.viewer_has_liked)
-
-To download the last 20 pictures with hashtag #cat, do
-
-.. code:: python
-
-    loader.download_hashtag('cat', max_count=20)
-
-Generally, Instaloader provides methods to iterate over the Posts from
-a certain source.
-
-.. code:: python
-
-    for post in loader.get_hashtag_posts('cat'):
-        # post is an instance of instaloader.Post
-        self.download_post(post, target='#cat')
-
-Each Instagram profile has its own unique ID which stays unmodified even
-if a user changes his/her username. To get said ID, given the profile's
-name, you may call
-
-.. code:: python
-
-    loader.get_id_by_username(PROFILE_NAME)
-
-``get_followees()`` also returns unique IDs for all loaded followees. To
-get the current username of a profile, given this unique ID
-``get_username_by_id()`` can be used. For example:
-
-.. code:: python
-
-    loader.get_username_by_id(loader.get_followees()[0]['id'])
-
-.. as-module-intro-end
-
-Refer to the
-`Instaloader Documentation <https://instaloader.readthedocs.io/as-module.html>`__ for
-more information.
 
 Disclaimer
 ----------

docs/README.md

@@ -0,0 +1,15 @@
+# Instaloader Documentation
+
+This directory contains sources for the Instaloader documentation.
+
+The documentation is hosted at:
+https://instaloader.readthedocs.io/
+
+## Build
+
+The documentation is created with [Sphinx](http://www.sphinx-doc.org/). To build it, use
+
+```
+pip3 install -r requirements.txt
+make html
+```

docs/_templates/caption.html

@@ -1 +1,2 @@
-<h2>Instaloader</h2>
+<h2><a href="index.html">Instaloader</a></h2>
+<p>Download pictures (or videos) along with their captions and other metadata from Instagram.</p>

docs/_templates/layout.html

@@ -1,4 +1,5 @@
 {% extends "!layout.html" %}
 {% block extrahead %}
+<meta name="google-site-verification" content="0esgS4FkYROvPe-I9MYvC1tTYtfASPpqxBdpTHEBIcQ" />
 <script async defer src="https://buttons.github.io/buttons.js"></script>
 {% endblock %}

docs/_templates/links.html

@@ -4,6 +4,11 @@
   <li class="toctree-l1"><a class="reference internal" href="{{ next.link|e }}">{{ next.title }}</a></li>
 </ul>
 {% endif %}
+<h3>Current Release</h3>
+<ul>
+  <li class="toctree-l1"><a href="https://github.com/Thammus/instaloader/releases/tag/v{{current_release}}">Version <b>{{current_release}}</b></a></li>
+  <li>Released on <b>{{current_release_date}}</b>.</li>
+</ul>
 <h3>Links</h3>
 <ul>
   <li><a href="https://github.com/Thammus/instaloader/issues">Issue Tracker</a></li>

docs/_templates/page.html

@@ -0,0 +1,10 @@
+{% extends "!page.html" %}
+{% block body %}
+{{ super() }}
+{% if next and prev %}
+<h2>Next Section</h2>
+<p>
+  <a class="reference internal" href="{{ next.link|e }}">{{ next.title }}</a>
+</p>
+{% endif %}
+{% endblock %}

docs/as-module.rst

@@ -1,26 +1,116 @@
-Usage as Python Module
-----------------------
+.. meta::
+   :description:
+      Instaloader can also be used as a powerful and easy-to-use
+      Python API for Instagram, allowing to download media and metadata.
 
-Introduction
-^^^^^^^^^^^^
+Python Module :mod:`instaloader`
+--------------------------------
 
-.. include:: ../README.rst
-   :start-after: as-module-intro-start
-   :end-before: as-module-intro-end
+.. module:: instaloader
 
+.. highlight:: python
+
+Instaloader exposes its internally used methods as a Python module, making it a
+**powerful and easy-to-use Python API for Instagram**, allowing to further
+customize obtaining media and metadata.
+
+Start with getting an instance of :class:`Instaloader`::
+
+    import instaloader
+
+    # Get instance
+    L = instaloader.Instaloader()
+
+    # Optionally, login or load session
+    L.login(USER, PASSWORD)        # (login)
+    L.interactive_login(USER)      # (ask password on terminal)
+    L.load_session_from_file(USER) # (load session created w/
+                                   #  `instaloader -l USERNAME`)
+
+:mod:`instaloader` provides the :class:`Post` structure, which represents a
+picture, video or sidecar (set of multiple pictures/videos) posted in a user's
+profile. :class:`Instaloader` provides methods to iterate over Posts from a
+certain source::
+
+    for post in L.get_hashtag_posts('cat'):
+        # post is an instance of instaloader.Post
+        L.download_post(post, target='#cat')
+
+Besides :func:`Instaloader.get_hashtag_posts`, there is
+:func:`Instaloader.get_feed_posts` and :func:`Instaloader.get_profile_posts`.
+Also, :class:`Post` instances can be created with :func:`Post.from_shortcode`
+and :func:`Post.from_mediaid`.
+
+Further, information about profiles can be easily obtained. For example, you may
+print a list of all followees and followers of a profile with::
+
+    # Print followees
+    print(PROFILE + " follows these profiles:")
+    for f in L.get_followees(PROFILE):
+        print("\t%s\t%s" % (f['username'], f['full_name']))
+
+    # Print followers
+    print("Followers of " + PROFILE + ":")
+    for f in L.get_followers(PROFILE):
+        print("\t%s\t%s" % (f['username'], f['full_name']))
+
+Then, you may download all pictures of all followees with::
+
+    for f in L.get_followees(PROFILE):
+        L.download_profile(f['username'])
+
+Each Instagram profile has its own unique ID which stays unmodified even if a
+user changes his/her username. To get said ID, given the profile's name, you may
+call::
+
+    L.get_id_by_username(PROFILE_NAME)
+
+A reference of the many methods provided by the :mod:`instaloader` module is
+provided in the remainder of this document. Feel free to direct any issue or
+contribution to
+`Instaloader on Github <https://github.com/Thammus/instaloader>`__.
 
 ``Instaloader`` (Main Class)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-.. autoclass:: instaloader.Instaloader
-   :members:
-   :undoc-members:
-
-.. _post-class:
+.. autoclass:: Instaloader
+   :no-show-inheritance:
 
 ``Post`` Class
 ^^^^^^^^^^^^^^
 
-.. autoclass:: instaloader.Post
-   :members:
-   :undoc-members:
+.. autoclass:: Post
+   :no-show-inheritance:
+
+Miscellaneous Functions
+^^^^^^^^^^^^^^^^^^^^^^^
+
+.. autofunction:: shortcode_to_mediaid
+
+.. autofunction:: mediaid_to_shortcode
+
+.. autoclass:: Tristate
+
+Exceptions
+^^^^^^^^^^
+
+.. autoexception:: InstaloaderException
+   :no-show-inheritance:
+
+.. autoexception:: QueryReturnedNotFoundException
+
+.. autoexception:: ProfileNotExistsException
+
+.. autoexception:: ProfileHasNoPicsException
+
+.. autoexception:: PrivateProfileNotFollowedException
+
+.. autoexception:: LoginRequiredException
+
+.. autoexception:: InvalidArgumentException
+
+.. autoexception:: BadResponseException
+
+.. autoexception:: BadCredentialsException
+
+.. autoexception:: ConnectionException

docs/basic-usage.rst

@@ -1,15 +1,152 @@
 Download Pictures from Instagram
 ---------------------------------
 
+.. highlight:: none
+
+.. NOTE that Section "Basic Usage" is duplicated in README.rst.
+
 Basic Usage
 ^^^^^^^^^^^
 
-.. include:: ../README.rst
-   :start-after: basic-usage-start
-   :end-before: basic-usage-end
+To **download all pictures and videos of a profile**, as well as the
+**profile picture**, do
 
-.. (continuation of --only-if explanation)
+::
+
+    instaloader profile [profile ...]
+
+where ``profile`` is the name of a profile you want to download. Instead
+of only one profile, you may also specify a list of profiles.
+
+To later **update your local copy** of that profiles, you may run
+
+::
+
+    instaloader --fast-update profile [profile ...]
+
+If :option:`--fast-update` is given, Instaloader stops when arriving at the
+first already-downloaded picture. When updating profiles, Instaloader
+automatically **detects profile name changes** and renames the target directory
+accordingly.
+
+Instaloader can also be used to **download private profiles**. To do so,
+invoke it with
+
+::
+
+    instaloader --login=your_username profile [profile ...]
+
+When logging in, Instaloader **stores the session cookies** in a file in your
+temporary directory, which will be reused later the next time :option:`--login`
+is given.  So you can download private profiles **non-interactively** when you
+already have a valid session cookie file.
+
+.. _what-to-download:
+
+What to Download
+^^^^^^^^^^^^^^^^
+
+Instaloader supports the following targets:
+
+``profile``
+   Public profile, or private profile with :option:`--login`. For each profile
+   you download, :option:`--stories` instructs Instaloader to also
+   **download the user's stories**.
+
+``"#hashtag"``
+   Posts with a certain **hashtag** (the quotes are usually neccessary),
+
+``:stories``
+   The currently-visible **stories** of your followees (requires
+   :option:`--login`),
+
+``:feed``
+   Your **feed** (requires :option:`--login`),
+
+``@profile``
+   All profiles that are followed by ``profile``, i.e. the *followees* of
+   ``profile`` (requires :option:`--login`).
+
+Instaloader goes through all media matching the specified targets and
+downloads the pictures and videos and their captions. You can specify
+
+- :option:`--comments`, to also **download comments** of each post,
+
+- :option:`--geotags`, to **download geotags** of each post and save them as
+  Google Maps link,
+
+- :option:`--metadata-json`, to store further post metadata in a separate JSON
+  file.
+
+.. _filename-specification:
+
+Filename Specification
+^^^^^^^^^^^^^^^^^^^^^^
+
+For each target, Instaloader creates a directory named after the target,
+i.e. ``profile``, ``#hashtag``, ``:feed``, etc. and therein saves the
+posts in files named after the post's timestamp.
+
+:option:`--dirname-pattern` allows to configure the directory name of each
+target. The default is ``--dirname-pattern={target}``. In the dirname
+pattern, the token ``{target}`` is replaced by the target name, and
+``{profile}`` is replaced by the owner of the post which is downloaded.
+
+:option:`--filename-pattern` configures the path of the post's files relative
+to the target directory. The default is ``--filename-pattern={date}``.
+The tokens ``{target}`` and ``{profile}`` are replaced like in the
+dirname pattern. Further, the tokens ``{date}``, ``{date_utc}`` and ``{shortcode}`` are
+defined. Additionally, in case of not downloading stories, the attributes of
+:class:`.Post` can be used, e.g. ``{post.owner_id}`` or ``{post.mediaid}``.
+
+For example, encode the poster's profile name in the filenames with:
+
+::
+
+    instaloader --filename-pattern={date}_{profile} "#hashtag"
+
+The pattern string is formatted with Python's string formatter. This
+gives additional flexibilty for pattern specification. For example,
+`strftime-style formatting options <https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior>`__
+are supported for the post's
+timestamp. The default for ``{date}`` is ``{date:%Y-%m-%d_%H-%M-%S}``.
+
+.. _filter-posts:
+
+Filter Posts
+^^^^^^^^^^^^
+
+The :option:`--only-if` option allows to specify criterias that posts have to
+meet to be downloaded. If not given, all posts are downloaded. It must be a
+boolean Python expression where the variables :attr:`.likes`, :attr:`.comments`,
+:attr:`.viewer_has_liked`, :attr:`.is_video`, and many more are defined.
+
+A few examples:
+
+To **download the pictures from your feed that you have liked**:
+
+::
+
+    instaloader --login=your_username --only-if=viewer_has_liked :feed
+
+Or you might only want to download **posts that either you liked or were
+liked by many others**:
+
+::
+
+    instaloader --login=your_username --only-if="likes>100 or viewer_has_liked" profile
+
+Or you may **skip videos**:
+
+::
+
+    instaloader --only-if="not is_video" target
+
+Or you may filter by hashtags that occur in the Post's caption. For
+example, to download posts of kittens that are cute: ::
+
+    instaloader --only-if="'cute' in caption_hashtags" "#kitten"
 
 The given string is evaluated as a
 `Python boolean expression <https://docs.python.org/3/reference/expressions.html#boolean-operations>`__,
-where all occuring variables are properties of the :ref:`post-class`.
+where all occuring variables are attributes of the :class:`.Post` class.

docs/cli-options.rst

@@ -1,6 +1,172 @@
 Command Line Options
 ====================
 
-.. include:: ../README.rst
-   :start-after: cli-options-start
-   :end-before: cli-options-end
+Instaloader is invoked with::
+
+   $ instaloader [options] target [target ...]
+
+where ``target`` is a ``profile``, a ``"#hashtag"``, ``@profile`` (all profiles
+that *profile* is following), or if logged in ``:feed`` (pictures from your
+feed) or ``:stories`` (stories of your followees).
+
+Here we explain the additional options that can be given to Instaloader to
+customize its behavior.  To get a list of all flags, their abbreviations and
+their descriptions, you may also run ``instaloader --help``.
+
+What to Download
+^^^^^^^^^^^^^^^^
+
+Specify a list of targets (profiles, #hashtags, ``:feed`` or ``:stories``). For
+each of these, Instaloader creates a folder and stores all posts along with the
+pictures's captions and the current **profile picture** there. If an
+already-downloaded profile has been renamed, Instaloader automatically **finds
+it by its unique ID** and renames the folder likewise.
+
+.. option:: --profile-pic-only
+
+   Only download profile picture.
+
+.. option:: --no-profile-pic
+
+   Do not download profile picture.
+
+.. option:: --no-videos
+
+   Do not download videos.
+
+.. option:: --no-video-thumbnails
+
+   Do not download thumbnails of videos.
+
+.. option:: --geotags
+
+   **Download geotags** when available. Geotags are stored as a text file with
+   the location's name and a Google Maps link. This requires an additional
+   request to the Instagram server for each picture, which is why it is disabled
+   by default.
+
+.. option:: --no-geotags
+
+   Do not store geotags, even if they can be obtained without any additional
+   request.
+
+.. option:: --comments
+
+   Download and update comments for each post. This requires an additional
+   request to the Instagram server for each post, which is why it is disabled by
+   default.
+
+.. option:: --no-captions
+
+   Do not store media captions, although no additional request is needed to
+   obtain them.
+
+.. option:: --stories
+
+   Also **download stories** of each profile that is downloaded. Requires
+   :option:`--login`.
+
+.. option:: --metadata-json
+
+   Create a JSON file containing the metadata of each post. This does not
+   include comments (see :option:`--comments`) nor geotags (see
+   :option:`--geotags`). The JSON files contain the properties of
+   :class:`instaloader.Post`.
+
+.. option:: --stories-only
+
+   Rather than downloading regular posts of each specified profile, only
+   download stories.  Requires :option:`--login`. Does not imply
+   :option:`--no-profile-pic`.
+
+.. option:: --only-if filter
+
+   Expression that, if given, must evaluate to True for each post to be
+   downloaded.  Must be a syntactically valid Python expression. Variables are
+   evaluated to :class:`instaloader.Post` attributes.  Example:
+   ``--only-if=viewer_has_liked``. See :ref:`filter-posts` for more
+   examples.
+
+
+When to Stop Downloading
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+If none of these options are given, Instaloader goes through all pictures
+matching the specified targets.
+
+.. option:: --fast-update
+
+   For each target, stop when encountering the first already-downloaded picture.
+   This flag is recommended when you use Instaloader to update your personal
+   Instagram archive.
+
+.. option:: --count COUNT
+
+   Do not attempt to download more than COUNT posts.  Applies only to
+   ``#hashtag`` and ``:feed``.
+
+
+Login (Download Private Profiles)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Instaloader can **login to Instagram**. This allows downloading private
+profiles. To login, pass the :option:`--login` option. Your session cookie (not your
+password!) will be saved to a local file to be reused next time you want
+Instaloader to login.
+
+.. option:: --login YOUR-USERNAME
+
+   Login name (profile name) for your Instagram account.
+
+.. option:: --sessionfile SESSIONFILE
+
+   Path for loading and storing session key file.  Defaults to a path within
+   your temporary directory, encoding your local username and your Instagram
+   profile name.
+
+.. option:: --password YOUR-PASSWORD
+
+   Password for your Instagram account.  Without this option, you'll be prompted
+   for your password interactively if there is not yet a valid session file.
+
+How to Download
+^^^^^^^^^^^^^^^
+
+.. option:: --dirname-pattern DIRNAME_PATTERN
+
+   Name of directory where to store posts. ``{profile}`` is replaced by the
+   profile name, ``{target}`` is replaced by the target you specified, i.e.
+   either ``:feed``, ``#hashtag`` or the profile name. Defaults to ``{target}``.
+   See :ref:`filename-specification`.
+
+.. option:: --filename-pattern FILENAME_PATTERN
+
+   Prefix of filenames. Posts are stored in the directory whose pattern is given
+   with ``--dirname-pattern``.  ``{profile}`` is replaced by the profile name,
+   ``{target}`` is replaced by the target you specified, i.e.  either ``:feed``,
+   ``#hashtag`` or the profile name.  Also, the fields ``{date}``, ``{date_utc}`` and
+   ``{shortcode}`` can be specified.  In case of not downloading stories, the
+   attributes of the :class:`.Post` class can be used in addition, e.g.
+   ``{post.owner_id}`` or ``{post.mediaid}``.
+   Defaults to
+   ``{date:%Y-%m-%d_%H-%M-%S}``.  See :ref:`filename-specification`.
+
+.. option:: --user-agent USER_AGENT
+
+   User Agent to use for HTTP requests. Per default, Instaloader pretends being
+   Chrome/51.
+
+.. option:: --max-connection-attempts N
+
+   Maximum number of connection attempts until a request is aborted. Defaults
+   to ``3``. If a connection fails, it can be manually skipped by hitting
+   :kbd:`Control-c`. Set this to ``0`` to retry infinitely.
+
+Miscellaneous Options
+^^^^^^^^^^^^^^^^^^^^^
+
+.. option:: --quiet
+
+   Disable user interaction, i.e. do not print messages (except errors) and fail
+   if login credentials are needed but not given. This makes Instaloader
+   **suitable as a cron job**.

docs/conf.py

@@ -19,6 +19,9 @@
 #
 import os
 import sys
+import re
+import requests
+from datetime import datetime
 sys.path.insert(0, os.path.abspath('..'))
 
 # -- General configuration ------------------------------------------------
@@ -34,8 +37,16 @@ extensions = [
     'sphinx.ext.autodoc',
     'sphinx_autodoc_typehints',
     'sphinx.ext.githubpages',
+    'sphinx.ext.intersphinx'
 ]
 
+autodoc_default_flags = ['show-inheritance', 'members', 'undoc-members']
+
+autodoc_member_order = 'bysource'
+
+intersphinx_mapping = {'python': ('https://docs.python.org/3', None),
+                       'requests': ('http://docs.python-requests.org/en/master', None)}
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 
@@ -344,3 +355,15 @@ texinfo_documents = [
 # If true, do not generate a @detailmenu in the "Top" node's menu.
 #
 # texinfo_no_detailmenu = False
+
+
+def get_latest_tag(repo):
+    tags = requests.get("https://api.github.com/repos/{}/git/refs/tags".format(repo)).json()
+    latest_tag = requests.get(tags[-1]['object']['url']).json()
+    version_string = latest_tag['tag']
+    version_date = datetime.strptime(latest_tag['tagger']['date'], "%Y-%m-%dT%H:%M:%SZ")
+    return version_string[1:], re.sub(r'\b0+(\d)', r'\1', "{:%d %b %Y}".format(version_date))
+
+current_release, current_release_date = get_latest_tag('Thammus/instaloader')
+
+html_context = {'current_release': current_release, 'current_release_date': current_release_date}

docs/contributing.rst

@@ -0,0 +1,83 @@
+Contributing to Instaloader
+===========================
+
+As an open source project, Instaloader heavily depends on the contributions from
+its community.  In this document, we advise on how you may help Instaloader to
+become an even greater tool.
+
+Instaloader's development is organized on 
+`GitHub <https://github.com/Thammus/instaloader>`__, where Issues and Pull
+Requests are discussed.
+
+Reporting Bugs
+--------------
+
+If you encounter a bug, do not hesitate to report it in our
+`Issue Tracker <https://github.com/Thammus/instaloader/issues>`__. Since
+Instaloader is actively developed, the majority of bugs is fixed within only
+**4 days** after being reported. When reporting a problem, please keep the
+following in mind:
+
+- Ensure you **use the latest version** of Instaloader. The currently-installed
+  version can be found out with ``instaloader --version``.
+
+- Briefly **check whether the bug has already been reported**. If you find an
+  issue reporting the same bug you encountered, comment there rather than
+  opening a new issue. However, if unsure, it is advised to create a new issue.
+
+- State **how the bug can be reproduced**, i.e. how Instaloader was invoked
+  when the problem occured (of course, you may anonymize profile names etc.).
+
+- Include all **error messages and tracebacks** in the report.
+
+- If not obvious, describe **which behavior you expected**
+  instead of what actually happened.
+
+- If you think an issue has been closed accidentally or inappropriately, feel
+  free to reopen it.
+
+Writing Code or Improving the Documentation
+-------------------------------------------
+
+Changes of the Instaloader source can be proposed as a
+`Pull Request <https://github.com/Thammus/instaloader/pulls>`__. There are only
+few things to consider:
+
+- Sometimes, the most current code is not in the ``master`` branch. Check that
+  you forked from the most recent branch.
+
+- We use `Pylint <https://www.pylint.org/>`__ for error and syntax checking of
+  ``instaloader.py``. The file ``.travis.yml`` in the project's root directory
+  shows how it is invoked. Note that sometimes it might be better to disable a
+  warning rather than adapting the code to Pylint's desires.
+
+- The documentation source is located in the ``docs`` folder. The file
+  ``cli-options.rst`` is merely an RST-formatted copy of ``ìnstaloader --help``
+  output, of which the source is near the bottom of ``instaloader.py``.
+
+- Feel free to contact us, even if you "only" have Proof-of-Concepts or
+  not-fully integrated changes. They already might be an advance for the
+  project.
+
+Suggesting Features
+-------------------
+
+If you have an idea of how Instaloader should be enhanced, but do not want to
+implement the feature yourself, feel free to open a ticket in our 
+`Issue Tracker <https://github.com/Thammus/instaloader/issues>`__.
+Please consider the following:
+
+- Instaloader already has plenty of features. **Check the documentation**
+  beforehand to ensure your desired suggestion is not already implemented.
+
+- Briefly **ensure that your idea has not already been suggested**. If you find
+  an issue suggesting the same or a similar feature, share your thoughts in a
+  comment there, instead of opening a new issue.
+
+- If possible, provide us a **use case of the feature**: How could the user
+  invoke the new function? Which problem would it solve? If new information is
+  obtained, how would it be further processed?
+
+- **Be patient**. Naturally, bugs and pull requests have a higher priority than
+  feature suggestions. Keep in mind that this is a free software project, and
+  unfortunately we only have limited time to work on it.

docs/index.rst

@@ -1,12 +1,18 @@
+.. meta::
+   :description:
+      Command line tool to download pictures (and videos) from Instagram.
+      Instaloader downloads public and private profiles, hashtags, user stories,
+      feeds, comments, geotags, captions and other metadata of each post.
+
 Instaloader
 ===========
 
+.. highlight:: none
+
 **Instaloader** is a tool to download pictures (or videos) along with
 their captions and other metadata from Instagram.
 
-With `Python <https://www.python.org/>`__ installed, do:
-
-::
+With `Python <https://www.python.org/>`__ installed, do::
 
     $ pip3 install instaloader
 
@@ -16,18 +22,24 @@ With `Python <https://www.python.org/>`__ installed, do:
 
 - downloads **public and private profiles, hashtags, user stories and
   feeds**,
-  
+
 - downloads **comments, geotags and captions** of each post,
 
 - automatically **detects profile name changes** and renames the target
   directory accordingly,
-  
+
 - allows **fine-grained customization** of filters and where to store
   downloaded media,
 
-- is free `Open Source <https://github.com/Thammus/instaloader>`__
+- is free `open source <https://github.com/Thammus/instaloader>`__
   software written in Python.
 
+::
+
+    instaloader [--comments] [--geotags] [--stories]
+                [--login YOUR-USERNAME] [--fast-update]
+                profile | "#hashtag" | :stories | :feed
+
 
 Table of Contents
 -----------------
@@ -39,6 +51,7 @@ Table of Contents
    basic-usage
    cli-options
    as-module
+   contributing
 
 Useful Links
 ------------

docs/installation.rst

@@ -1,11 +1,38 @@
 Installation
 ============
 
-.. include:: ../README.rst
-   :start-after: installation-start
-   :end-before: installation-end
+.. highlight:: none
 
-If you do not want to use pip, even though it is highly recommended,
-and prefer installing Instaloader manually,
-`Download the Source <https://github.com/Thammus/instaloader/releases>`__,
-extract the Zip or Tarball and execute ``instaloader.py`` from there.
+Instaloader requires `Python <https://www.python.org/>`__, at least
+version 3.5.  If you have `pip <https://pypi.python.org/pypi/pip>`__
+installed, you may install Instaloader using
+
+::
+
+    pip3 install instaloader
+
+Alternatively, to get the most current version of Instaloader from our
+`Git repository <https://github.com/Thammus/instaloader>`__:
+
+::
+
+    pip3 install git+https://github.com/Thammus/instaloader
+
+(pass ``--upgrade`` to upgrade if Instaloader is already installed)
+
+Instaloader requires
+`requests <http://python-requests.org/>`__, which
+will be installed automatically, if it is not already installed.
+
+**Alternative methods for installing Instaloader:**
+
+- If you do not want to use pip, even though it is highly recommended,
+  and prefer to **install Instaloader manually**,
+  `Download the Source <https://github.com/Thammus/instaloader/releases>`__,
+  extract the Zip or Tarball and execute ``instaloader.py`` from there.
+
+- On **Arch Linux**, you may install Instaloader using the
+  `Instaloader AUR package <https://aur.archlinux.org/packages/instaloader/>`__.
+
+- On **Gentoo Linux**, you may install Instaloader using the
+  `Instaloader Ebuild <https://bugs.gentoo.org/642100>`__.

docs/requirements.txt

@@ -1 +1,3 @@
+requests
+sphinx
 sphinx-autodoc-typehints

instaloader.py

@@ -29,7 +29,7 @@ import requests.utils
 import urllib3
 
 
-__version__ = '3.0rc1'
+__version__ = '3.2.2'
 
 
 try:
@@ -43,7 +43,9 @@ else:
 
 
 class InstaloaderException(Exception):
-    """Base exception for this script"""
+    """Base exception for this script.
+
+    :note: This exception should not be raised directly."""
     pass
 
 
@@ -51,6 +53,10 @@ class QueryReturnedNotFoundException(InstaloaderException):
     pass
 
 
+class QueryReturnedForbiddenException(InstaloaderException):
+    pass
+
+
 class ProfileNotExistsException(InstaloaderException):
     pass
 
@@ -124,7 +130,7 @@ def mediaid_to_shortcode(mediaid: int) -> str:
 def format_string_contains_key(format_string: str, key: str) -> bool:
     # pylint:disable=unused-variable
     for literal_text, field_name, format_spec, conversion in string.Formatter().parse(format_string):
-        if field_name == key:
+        if field_name == key or field_name.startswith(key + '.'):
             return True
     return False
 
@@ -163,23 +169,29 @@ class Post:
     """
     Structure containing information about an Instagram post.
 
-    Created by Instaloader methods get_profile_posts(), get_hashtag_posts(), get_feed_posts(). Posts are linked to
-    an Instaloader instance which is used for error logging and obtaining of additional metadata, if required.
-    This class unifies access to the properties associated with a post. It implements == and is hashable.
+    Created by Instaloader methods :meth:`.get_profile_posts`, :meth:`.get_hashtag_posts`, :meth:`.get_feed_posts`.
+    Posts are linked to an :class:`Instaloader` instance which is used for error logging and obtaining of additional
+    metadata, if required. This class unifies access to the properties associated with a post. It implements == and is
+    hashable.
+
+    The properties defined here are accessible by the filter expressions specified with the :option:`--only-if`
+    parameter and exported into JSON files with :option:`--metadata-json`.
     """
 
     LOGIN_REQUIRING_PROPERTIES = ["viewer_has_liked"]
 
-    def __init__(self, instaloader: 'Instaloader', node: Dict[str, Any], profile: Optional[str] = None):
+    def __init__(self, instaloader: 'Instaloader', node: Dict[str, Any],
+                 profile: Optional[str] = None, profile_id: Optional[int] = None):
         """Create a Post instance from a node structure as returned by Instagram.
 
-        :param instaloader: Instaloader instance used for additional queries if neccessary.
+        :param instaloader: :class:`Instaloader` instance used for additional queries if neccessary.
         :param node: Node structure.
         :param profile: The name of the owner, if already known at creation.
         """
         self._instaloader = instaloader
         self._node = node
         self._profile = profile
+        self._profile_id = profile_id
         self._full_metadata_dict = None
 
     @classmethod
@@ -200,6 +212,11 @@ class Post:
         """Media shortcode. URL of the post is instagram.com/p/<shortcode>/."""
         return self._node['shortcode'] if 'shortcode' in self._node else self._node['code']
 
+    @property
+    def mediaid(self) -> int:
+        """The mediaid is a decimal representation of the media shortcode."""
+        return int(self._node['id'])
+
     def __repr__(self):
         return '<Post {}>'.format(self.shortcode)
 
@@ -247,10 +264,22 @@ class Post:
             return 'UNKNOWN'
 
     @property
-    def date(self) -> datetime:
-        """Timestamp when the post was created."""
+    def owner_id(self) -> int:
+        """The ID of the Post's owner."""
+        if self._profile_id:
+            return self._profile_id
+        return int(self._field('owner', 'id'))
+
+    @property
+    def date_local(self) -> datetime:
+        """Timestamp when the post was created (local time zone)."""
         return datetime.fromtimestamp(self._node["date"] if "date" in self._node else self._node["taken_at_timestamp"])
 
+    @property
+    def date_utc(self) -> datetime:
+        """Timestamp when the post was created (UTC)."""
+        return datetime.utcfromtimestamp(self._node["date"] if "date" in self._node else self._node["taken_at_timestamp"])
+
     @property
     def url(self) -> str:
         """URL of the picture / video thumbnail of the post"""
@@ -277,13 +306,32 @@ class Post:
 
     @property
     def caption_hashtags(self) -> List[str]:
-        """List of all hashtags (without preceeding #) which occur in the Post's caption."""
+        """List of all lowercased hashtags (without preceeding #) that occur in the Post's caption."""
         if not self.caption:
             return []
         # This regular expression is from jStassen, adjusted to use Python's \w to support Unicode
         # http://blog.jstassen.com/2016/03/code-regex-for-instagram-username-and-hashtags/
         hashtag_regex = re.compile(r"(?:#)(\w(?:(?:\w|(?:\.(?!\.))){0,28}(?:\w))?)")
-        return re.findall(hashtag_regex, self.caption)
+        return re.findall(hashtag_regex, self.caption.lower())
+
+    @property
+    def caption_mentions(self) -> List[str]:
+        """List of all lowercased profiles that are mentioned in the Post's caption, without preceeding @."""
+        if not self.caption:
+            return []
+        # This regular expression is from jStassen, adjusted to use Python's \w to support Unicode
+        # http://blog.jstassen.com/2016/03/code-regex-for-instagram-username-and-hashtags/
+        mention_regex = re.compile(r"(?:@)(\w(?:(?:\w|(?:\.(?!\.))){0,28}(?:\w))?)")
+        return re.findall(mention_regex, self.caption.lower())
+
+    @property
+    def tagged_users(self) -> List[str]:
+        """List of all lowercased users that are tagged in the Post."""
+        try:
+            return [edge['node']['user']['username' ].lower() for edge in self._field('edge_media_to_tagged_user',
+                                                                                      'edges')]
+        except KeyError:
+            return []
 
     @property
     def is_video(self) -> bool:
@@ -336,13 +384,34 @@ class Post:
                                                        params={'__a': 1})
             return location_json["location"]
 
+    @staticmethod
+    def json_encoder(obj) -> Dict[str, Any]:
+        """Convert instance of :class:`Post` to a JSON-serializable dictionary."""
+        if not isinstance(obj, Post):
+            raise TypeError("Object of type {} is not a Post object.".format(obj.__class__.__name__))
+        jsondict = {}
+        for prop in dir(Post):
+            if prop[0].isupper() or prop[0] == '_':
+                # skip uppercase and private properties
+                continue
+            val = obj.__getattribute__(prop)
+            if val is True or val is False or isinstance(val, (str, int, float, list)):
+                jsondict[prop] = val
+            elif isinstance(val, datetime):
+                jsondict[prop] = val.isoformat()
+        return jsondict
 
 class Tristate(Enum):
     """Tri-state to encode whether we should save certain information, i.e. videos, captions, comments or geotags.
 
-    never: Do not save, even if the information is available without any additional request,
-    no_extra_query: Save if and only if available without doing additional queries,
-    always: Save (and query, if neccessary).
+    :attr:`never`
+        Do not save, even if the information is available without any additional request,
+
+    :attr:`no_extra_query`
+        Save if and only if available without doing additional queries,
+
+    :attr:`always`
+        Save (and query, if neccessary).
     """
     never = 0
     no_extra_query = 1
@@ -356,9 +425,12 @@ class Instaloader:
                  dirname_pattern: Optional[str] = None,
                  filename_pattern: Optional[str] = None,
                  download_videos: Tristate = Tristate.always,
+                 download_video_thumbnails: Tristate = Tristate.always,
                  download_geotags: Tristate = Tristate.no_extra_query,
-                 download_captions: Tristate = Tristate.no_extra_query,
-                 download_comments: Tristate = Tristate.no_extra_query):
+                 save_captions: Tristate = Tristate.no_extra_query,
+                 download_comments: Tristate = Tristate.no_extra_query,
+                 save_metadata: Tristate = Tristate.never,
+                 max_connection_attempts: int = 3):
 
         # configuration parameters
         self.user_agent = user_agent if user_agent is not None else default_user_agent()
@@ -367,20 +439,30 @@ class Instaloader:
         self.sleep = sleep
         self.quiet = quiet
         self.dirname_pattern = dirname_pattern if dirname_pattern is not None else '{target}'
-        self.filename_pattern = filename_pattern.replace('{date}', '{date:%Y-%m-%d_%H-%M-%S}') \
-            if filename_pattern is not None else '{date:%Y-%m-%d_%H-%M-%S}'
+        if filename_pattern is not None:
+            filename_pattern = re.sub(r"({(?:post\.)?date)([:}])", r"\1_utc\2", filename_pattern)
+            self.filename_pattern_old = filename_pattern.replace('{date_utc}', '{date_utc:%Y-%m-%d_%H-%M-%S}')
+            self.filename_pattern_old = re.sub(r"(?i)({(?:post\.)?date_utc:[^}]*?)_UTC",
+                                               r"\1", self.filename_pattern_old)
+            filename_pattern = re.sub(r"(?i)({(date_utc|post\.date_utc):(?![^}]*UTC[^}]*).*?)}",
+                                      r"\1_UTC}", filename_pattern)
+            self.filename_pattern = filename_pattern.replace('{date_utc}', '{date_utc:%Y-%m-%d_%H-%M-%S_UTC}')
+        else:
+            self.filename_pattern = '{date_utc:%Y-%m-%d_%H-%M-%S_UTC}'
+            self.filename_pattern_old = '{date_utc:%Y-%m-%d_%H-%M-%S}'
         self.download_videos = download_videos
+        self.download_video_thumbnails = download_video_thumbnails
         self.download_geotags = download_geotags
-        self.download_captions = download_captions
+        self.save_captions = save_captions
         self.download_comments = download_comments
-        self.previous_queries = dict()
+        self.save_metadata = save_metadata
+        self.max_connection_attempts = max_connection_attempts
 
         # error log, filled with error() and printed at the end of Instaloader.main()
         self.error_log = []
 
         # For the adaption of sleep intervals (rate control)
-        self.request_count = 0
-        self.last_request_time = 0
+        self.previous_queries = dict()
 
     @property
     def is_logged_in(self) -> bool:
@@ -392,8 +474,11 @@ class Instaloader:
         """Yield an anonymous, otherwise equally-configured copy of an Instaloader instance; Then copy its error log."""
         new_loader = Instaloader(self.sleep, self.quiet, self.user_agent,
                                  self.dirname_pattern, self.filename_pattern,
-                                 self.download_videos, self.download_geotags,
-                                 self.download_captions, self.download_comments)
+                                 self.download_videos,
+                                 self.download_video_thumbnails,
+                                 self.download_geotags,
+                                 self.save_captions, self.download_comments,
+                                 self.save_metadata, self.max_connection_attempts)
         new_loader.previous_queries = self.previous_queries
         yield new_loader
         self.error_log.extend(new_loader.error_log)
@@ -431,7 +516,7 @@ class Instaloader:
         if self.sleep:
             time.sleep(random.uniform(0.5, 3))
 
-    def _get_and_write_raw(self, url: str, filename: str, tries: int = 3) -> None:
+    def _get_and_write_raw(self, url: str, filename: str, _attempt = 1) -> None:
         """Downloads raw data.
 
         :raises QueryReturnedNotFoundException: When the server responds with a 404.
@@ -444,27 +529,32 @@ class Instaloader:
                     resp.raw.decode_content = True
                     shutil.copyfileobj(resp.raw, file)
             else:
+                if resp.status_code == 403:
+                    # suspected invalid URL signature
+                    raise QueryReturnedForbiddenException("403 when accessing {}.".format(url))
                 if resp.status_code == 404:
                     # 404 not worth retrying.
                     raise QueryReturnedNotFoundException("404 when accessing {}.".format(url))
                 raise ConnectionException("HTTP error code {}.".format(resp.status_code))
         except (urllib3.exceptions.HTTPError, requests.exceptions.RequestException, ConnectionException) as err:
             error_string = "URL {}: {}".format(url, err)
-            if tries <= 1:
+            if _attempt == self.max_connection_attempts:
+                raise ConnectionException(error_string)
+            self.error(error_string + " [retrying; skip with ^C]", repeat_at_end=False)
+            try:
+                self._sleep()
+                self._get_and_write_raw(url, filename, _attempt + 1)
+            except KeyboardInterrupt:
+                self.error("[skipped by user]", repeat_at_end=False)
                 raise ConnectionException(error_string)
-            else:
-                self.error(error_string + " [retrying]", repeat_at_end=False)
-            self._sleep()
-            self._get_and_write_raw(url, filename, tries - 1)
 
     def get_json(self, url: str, params: Dict[str, Any],
-                 session: Optional[requests.Session] = None, tries: int = 3) -> Dict[str, Any]:
+                 session: Optional[requests.Session] = None, _attempt = 1) -> Dict[str, Any]:
         """JSON request to Instagram.
 
-        :param url: URL, relative to https://www.instagram.com/
+        :param url: URL, relative to www.instagram.com/
         :param params: GET parameters
         :param session: Session to use, or None to use self.session
-        :param tries: Maximum number of attempts until an exception is raised
         :return: Decoded response dictionary
         :raises QueryReturnedNotFoundException: When the server responds with a 404.
         :raises ConnectionException: When query repeatedly failed.
@@ -512,21 +602,25 @@ class Instaloader:
             return resp_json
         except (ConnectionException, json.decoder.JSONDecodeError, requests.exceptions.RequestException) as err:
             error_string = "JSON Query to {}: {}".format(url, err)
-            if tries <= 1:
+            if _attempt == self.max_connection_attempts:
+                raise ConnectionException(error_string)
+            self.error(error_string + " [retrying; skip with ^C]", repeat_at_end=False)
+            text_for_429 = ("HTTP error code 429 was returned because too many queries occured in the last time. "
+                            "Please do not use Instagram in your browser or run multiple instances of Instaloader "
+                            "in parallel.")
+            try:
+                if isinstance(err, TooManyRequests):
+                    print(textwrap.fill(text_for_429), file=sys.stderr)
+                    if is_graphql_query:
+                        waittime = graphql_query_waittime(query_id=params['query_id'], untracked_queries=True)
+                        if waittime > 0:
+                            self._log('The request will be retried in {} seconds.'.format(waittime))
+                            time.sleep(waittime)
+                self._sleep()
+                return self.get_json(url, params, sess, _attempt + 1)
+            except KeyboardInterrupt:
+                self.error("[skipped by user]", repeat_at_end=False)
                 raise ConnectionException(error_string)
-            self.error(error_string + " [retrying]", repeat_at_end=False)
-            if isinstance(err, TooManyRequests):
-                text_for_429 = ("HTTP error code 429 was returned because too many queries occured in the last time. "
-                                "Please do not use Instagram in your browser or run multiple instances of Instaloader "
-                                "in parallel.")
-                print(textwrap.fill(text_for_429), file=sys.stderr)
-                if is_graphql_query:
-                    waittime = graphql_query_waittime(query_id=params['query_id'], untracked_queries=True)
-                    if waittime > 0:
-                        self._log('The request will be retried in {} seconds.'.format(waittime))
-                        time.sleep(waittime)
-            self._sleep()
-            return self.get_json(url, params, sess, tries - 1)
 
     def _default_http_header(self, empty_session_only: bool = False) -> Dict[str, str]:
         """Returns default HTTP header we use for requests."""
@@ -607,7 +701,7 @@ class Instaloader:
     def graphql_node_list(self, query_id: int, query_variables: Dict[str, Any], query_referer: Optional[str],
                           edge_extractor: Callable[[Dict[str, Any]], Dict[str, Any]]) -> Iterator[Dict[str, Any]]:
         """Retrieve a list of GraphQL nodes."""
-        query_variables['first'] = 500
+        query_variables['first'] = 200
         data = self.graphql_query(query_id, query_variables, query_referer)
         while True:
             edge_struct = edge_extractor(data)
@@ -643,30 +737,48 @@ class Instaloader:
                                           lambda d: d['data']['user']['edge_follow'])
 
     def download_pic(self, filename: str, url: str, mtime: datetime,
-                     filename_suffix: Optional[str] = None) -> bool:
+                     filename_alt: Optional[str] = None, filename_suffix: Optional[str] = None) -> bool:
         """Downloads and saves picture with given url under given directory with given timestamp.
         Returns true, if file was actually downloaded, i.e. updated."""
-        urlmatch = re.search('\\.[a-z]*\\?', url)
+        urlmatch = re.search('\\.[a-z0-9]*\\?', url)
         file_extension = url[-3:] if urlmatch is None else urlmatch.group(0)[1:-1]
         if filename_suffix is not None:
             filename += '_' + filename_suffix
+            if filename_alt is not None:
+                filename_alt += '_' + filename_suffix
         filename += '.' + file_extension
         if os.path.isfile(filename):
             self._log(filename + ' exists', end=' ', flush=True)
             return False
+        if filename_alt is not None:
+            filename_alt += '.' + file_extension
+            if os.path.isfile(filename_alt):
+                self._log(filename_alt + 'exists', end=' ', flush=True)
+                return False
         self._get_and_write_raw(url, filename)
         os.utime(filename, (datetime.now().timestamp(), mtime.timestamp()))
         return True
 
-    def update_comments(self, filename: str, post: Post) -> None:
-        filename += '_comments.json'
+    def save_metadata_json(self, filename: str, post: Post) -> None:
+        """Saves metadata JSON file of a :class:`Post`."""
+        filename += '.json'
+        json.dump(post, fp=open(filename, 'w'), indent=4, default=Post.json_encoder)
+        self._log('json', end=' ', flush=True)
+
+    def update_comments(self, filename: str, post: Post, filename_alt: Optional[str] = None) -> None:
         try:
-            comments = json.load(open(filename))
+            filename_current = filename + '_comments.json'
+            comments = json.load(open(filename_current))
         except FileNotFoundError:
-            comments = list()
+            try:
+                filename_current = filename_alt + '_comments.json'
+                comments = json.load(open(filename_current))
+            except (FileNotFoundError, TypeError):
+                filename_current = filename + '_comments.json'
+                comments = list()
         comments.extend(post.get_comments())
         if comments:
-            with open(filename, 'w') as file:
+            with open(filename_current, 'w') as file:
                 comments_list = sorted(sorted(list(comments), key=lambda t: t['id']),
                                        key=lambda t: t['created_at'], reverse=True)
                 unique_comments_list = [comments_list[0]]
@@ -679,17 +791,25 @@ class Instaloader:
                     if x['id'] != y['id']:
                         unique_comments_list.append(y)
                 file.write(json.dumps(unique_comments_list, indent=4))
+            os.rename(filename_current, filename + '_comments.json')
             self._log('comments', end=' ', flush=True)
 
-    def save_caption(self, filename: str, mtime: datetime, caption: str) -> None:
+    def save_caption(self, filename: str, mtime: datetime, caption: str, filename_alt: Optional[str] = None) -> None:
         """Updates picture caption"""
         filename += '.txt'
+        if filename_alt is not None:
+            filename_alt += '.txt'
         pcaption = caption.replace('\n', ' ').strip()
         caption = caption.encode("UTF-8")
         pcaption = '[' + ((pcaption[:29] + u"\u2026") if len(pcaption) > 31 else pcaption) + ']'
         with suppress(FileNotFoundError):
-            with open(filename, 'rb') as file:
-                file_caption = file.read()
+            try:
+                with open(filename, 'rb') as file:
+                    file_caption = file.read()
+            except FileNotFoundError:
+                if filename_alt is not None:
+                    with open(filename_alt, 'rb') as file:
+                        file_caption = file.read()
             if file_caption.replace(b'\r\n', b'\n') == caption.replace(b'\r\n', b'\n'):
                 try:
                     self._log(pcaption + ' unchanged', end=' ', flush=True)
@@ -697,15 +817,22 @@ class Instaloader:
                     self._log('txt unchanged', end=' ', flush=True)
                 return None
             else:
-                def get_filename(index):
-                    return filename if index == 0 else (filename[:-4] + '_old_' +
-                                                        (str(0) if index < 10 else str()) + str(index) + filename[-4:])
+                def get_filename(file, index):
+                    return file if index == 0 else (file[:-4] + '_old_' +
+                                                    (str(0) if index < 10 else str()) + str(index) + file[-4:])
 
                 i = 0
-                while os.path.isfile(get_filename(i)):
+                file_exists_list = []
+                while True:
+                    file_exists_list.append(1 if os.path.isfile(get_filename(filename, i)) else 0)
+                    if not file_exists_list[i] and filename_alt is not None:
+                        file_exists_list[i] = 2 if os.path.isfile(get_filename(filename_alt, i)) else 0
+                    if not file_exists_list[i]:
+                        break
                     i = i + 1
                 for index in range(i, 0, -1):
-                    os.rename(get_filename(index - 1), get_filename(index))
+                    os.rename(get_filename(filename if file_exists_list[index - 1] % 2 else filename_alt, index - 1),
+                              get_filename(filename, index))
                 try:
                     self._log(pcaption + ' updated', end=' ', flush=True)
                 except UnicodeEncodeError:
@@ -728,8 +855,11 @@ class Instaloader:
         os.utime(filename, (datetime.now().timestamp(), mtime.timestamp()))
         self._log('geo', end=' ', flush=True)
 
-    def download_profilepic(self, name: str, url: str) -> None:
-        """Downloads and saves profile pic with given url."""
+    def download_profilepic(self, name: str, profile_metadata: Dict[str, Any]) -> None:
+        """Downloads and saves profile pic."""
+
+        url = profile_metadata["user"]["profile_pic_url_hd"] if "profile_pic_url_hd" in profile_metadata["user"] \
+            else profile_metadata["user"]["profile_pic_url"]
 
         def _epoch_to_string(epoch: datetime) -> str:
             return epoch.strftime('%Y-%m-%d_%H-%M-%S')
@@ -747,13 +877,18 @@ class Instaloader:
         if os.path.isfile(filename):
             self._log(filename + ' already exists')
             return None
-        url = re.sub(r'/s([1-9][0-9]{2})x\1/', '/s2048x2048/', url)
-        self._get_and_write_raw(url, filename)
+        url_best = re.sub(r'/s([1-9][0-9]{2})x\1/', '/s2048x2048/', url)
+        url_best = re.sub(r'/vp/[a-f0-9]{32}/[A-F0-9]{8}/', '/', url_best)      # remove signature
+        try:
+            self._get_and_write_raw(url_best, filename)
+        except (QueryReturnedForbiddenException, QueryReturnedNotFoundException) as err:
+            self.error('{} Retrying with lower quality version.'.format(err))
+            self._get_and_write_raw(url, filename)
         os.utime(filename, (datetime.now().timestamp(), date_object.timestamp()))
         self._log('') # log output of _get_and_write_raw() does not produce \n
 
     def save_session_to_file(self, filename: Optional[str] = None) -> None:
-        """Saves requests.Session object."""
+        """Saves internally stored :class:`requests.Session` object."""
         if filename is None:
             filename = get_default_session_filename(self.username)
         dirname = os.path.dirname(filename)
@@ -766,11 +901,11 @@ class Instaloader:
             self._log("Saved session to %s." % filename)
 
     def load_session_from_file(self, username: str, filename: Optional[str] = None) -> None:
-        """Internally stores requests.Session object loaded from file.
+        """Internally stores :class:`requests.Session` object loaded from file.
 
         If filename is None, the file with the default session path is loaded.
 
-        :raises FileNotFoundError; If the file does not exist.
+        :raises FileNotFoundError: If the file does not exist.
         """
         if filename is None:
             filename = get_default_session_filename(username)
@@ -784,7 +919,7 @@ class Instaloader:
             self.username = username
 
     def test_login(self, session: Optional[requests.Session]) -> Optional[str]:
-        """Returns the Instagram username to which given requests.Session object belongs, or None."""
+        """Returns the Instagram username to which given :class:`requests.Session` object belongs, or None."""
         if session:
             data = self.get_json('', params={'__a': 1}, session=session)
             return data['graphql']['user']['username'] if 'graphql' in data else None
@@ -828,52 +963,73 @@ class Instaloader:
         profilename = post.owner_username if needs_profilename else None
         dirname = self.dirname_pattern.format(profile=profilename, target=target.lower())
         filename = dirname + '/' + self.filename_pattern.format(profile=profilename, target=target.lower(),
-                                                                date=post.date, shortcode=post.shortcode)
+                                                                date_utc=post.date_utc,
+                                                                shortcode=post.shortcode,
+                                                                post=post)
+        filename_old = dirname + '/' + self.filename_pattern_old.replace("{post.date_utc", "{date_utc") \
+                                                                .format(profile=profilename, target=target.lower(),
+                                                                        date_utc=post.date_local,
+                                                                        shortcode=post.shortcode,
+                                                                        post=post)
         os.makedirs(os.path.dirname(filename), exist_ok=True)
 
         # Download the image(s) / video thumbnail and videos within sidecars if desired
+        downloaded = False
         if post.typename == 'GraphSidecar':
             edge_number = 1
-            downloaded = True
             for edge in post.get_sidecar_edges():
-                edge_downloaded = self.download_pic(filename=filename,
+                # Download picture or video thumbnail
+                if not edge['node']['is_video'] or self.download_video_thumbnails is Tristate.always:
+                    downloaded |= self.download_pic(filename=filename,
+                                                    filename_alt=filename_old,
                                                     url=edge['node']['display_url'],
-                                                    mtime=post.date,
+                                                    mtime=post.date_local,
                                                     filename_suffix=str(edge_number))
                 # Additionally download video if available and desired
                 if edge['node']['is_video'] and self.download_videos is Tristate.always:
-                    self.download_pic(filename=filename,
-                                      url=edge['node']['video_url'],
-                                      mtime=post.date,
-                                      filename_suffix=str(edge_number))
-                downloaded = downloaded and edge_downloaded
+                    downloaded |= self.download_pic(filename=filename,
+                                                    filename_alt=filename_old,
+                                                    url=edge['node']['video_url'],
+                                                    mtime=post.date_local,
+                                                    filename_suffix=str(edge_number))
                 edge_number += 1
-        elif post.typename in ['GraphImage', 'GraphVideo']:
-            downloaded = self.download_pic(filename=filename, url=post.url, mtime=post.date)
+        elif post.typename == 'GraphImage':
+            downloaded = self.download_pic(filename=filename, filename_alt=filename_old,
+                                           url=post.url, mtime=post.date_local)
+        elif post.typename == 'GraphVideo':
+            if self.download_video_thumbnails is Tristate.always:
+                downloaded = self.download_pic(filename=filename, filename_alt=filename_old,
+                                               url=post.url, mtime=post.date_local)
         else:
             self.error("Warning: {0} has unknown typename: {1}".format(post, post.typename))
-            downloaded = False
 
         # Save caption if desired
-        if self.download_captions is not Tristate.never:
+        if self.save_captions is not Tristate.never:
             if post.caption:
-                self.save_caption(filename, post.date, post.caption)
+                self.save_caption(filename=filename, filename_alt=filename_old,
+                                  mtime=post.date_local, caption=post.caption)
             else:
                 self._log("<no caption>", end=' ', flush=True)
 
         # Download video if desired
         if post.is_video and self.download_videos is Tristate.always:
-            self.download_pic(filename=filename, url=post.video_url, mtime=post.date)
+            downloaded |= self.download_pic(filename=filename, filename_alt=filename_old,
+                                            url=post.video_url, mtime=post.date_local)
 
         # Download geotags if desired
         if self.download_geotags is Tristate.always:
             location = post.get_location()
             if location:
-                self.save_location(filename, location, post.date)
+                self.save_location(filename, location, post.date_local)
 
         # Update comments if desired
         if self.download_comments is Tristate.always:
-            self.update_comments(filename, post)
+            self.update_comments(filename=filename, filename_alt=filename_old, post=post)
+
+        # Save metadata as JSON if desired.  It might require an extra query, depending on which information has been
+        # already obtained.  Regarding Tristate interpretation, we always assume that it requires an extra query.
+        if self.save_metadata is Tristate.always:
+            self.save_metadata_json(filename, post)
 
         self._log()
         return downloaded
@@ -927,6 +1083,10 @@ class Instaloader:
         :param filename_target: Replacement for {target} in dirname_pattern and filename_pattern
         """
 
+        if format_string_contains_key(self.filename_pattern, 'post'):
+            raise InvalidArgumentException("The \"post\" keyword is not supported in the filename pattern when "
+                                           "downloading stories.")
+
         if not self.is_logged_in:
             raise LoginRequiredException('Login required to download stories')
 
@@ -940,44 +1100,61 @@ class Instaloader:
             for item in user_stories["items"]:
                 self._log("[%3i/%3i] " % (count, totalcount), end="", flush=True)
                 count += 1
-
-                shortcode = item["code"] if "code" in item else "no_code"
-                date = datetime.fromtimestamp(item["taken_at"])
-
-                dirname = self.dirname_pattern.format(profile=name, target=filename_target)
-                filename = dirname + '/' + self.filename_pattern.format(profile=name, target=filename_target,
-                                                                        date=date,
-                                                                        shortcode=shortcode)
-                os.makedirs(os.path.dirname(filename), exist_ok=True)
-                with self._error_catcher('Download story {} from user {}'.format(shortcode, name)):
-                    if "image_versions2" in item:
-                        url = item["image_versions2"]["candidates"][0]["url"]
-                        downloaded = self.download_pic(filename=filename,
-                                                       url=url,
-                                                       mtime=date)
-                    else:
-                        self._log("Warning: Unable to find story image.")
-                        downloaded = False
-                    if "caption" in item and item["caption"] is not None and \
-                                    self.download_captions is not Tristate.never:
-                        caption = item["caption"]
-                        if isinstance(caption, dict) and "text" in caption:
-                            caption = caption["text"]
-                        self.save_caption(filename, date, caption)
-                    else:
-                        self._log("<no caption>", end=' ', flush=True)
-                    if "video_versions" in item and self.download_videos is Tristate.always:
-                        downloaded = self.download_pic(filename=filename,
-                                                       url=item["video_versions"][0]["url"],
-                                                       mtime=date)
-                    if item["story_locations"] and self.download_geotags is not Tristate.never:
-                        location = item["story_locations"][0]["location"]
-                        if location:
-                            self.save_location(filename, location, date)
-                    self._log()
+                with self._error_catcher('Download story from user {}'.format(name)):
+                    downloaded = self.download_story(item, filename_target, name)
                     if fast_update and not downloaded:
                         break
 
+    def download_story(self, item: Dict[str, Any], target: str, profile: str) -> bool:
+        """Download one user story.
+
+        :param item: Story item, as in story['items'] for story in :meth:`get_stories`
+        :param target: Replacement for {target} in dirname_pattern and filename_pattern
+        :param profile: Owner profile name
+        :return: True if something was downloaded, False otherwise, i.e. file was already there
+        """
+
+        shortcode = item["code"] if "code" in item else "no_code"
+        date_local = datetime.fromtimestamp(item["taken_at"])
+        date_utc = datetime.utcfromtimestamp(item["taken_at"])
+        dirname = self.dirname_pattern.format(profile=profile, target=target)
+        filename = dirname + '/' + self.filename_pattern.format(profile=profile, target=target,
+                                                                date_utc=date_utc,
+                                                                shortcode=shortcode)
+        filename_old = dirname + '/' + self.filename_pattern_old.format(profile=profile, target=target,
+                                                                        date_utc=date_local,
+                                                                        shortcode=shortcode)
+        os.makedirs(os.path.dirname(filename), exist_ok=True)
+        downloaded = False
+        if "image_versions2" in item:
+            if "video_versions" not in item or self.download_video_thumbnails is Tristate.always:
+                url = item["image_versions2"]["candidates"][0]["url"]
+                downloaded = self.download_pic(filename=filename,
+                                               filename_alt=filename_old,
+                                               url=url,
+                                               mtime=date_local)
+        else:
+            self._log("Warning: Unable to find story image.")
+        if "caption" in item and item["caption"] is not None and \
+                self.save_captions is not Tristate.never:
+            caption = item["caption"]
+            if isinstance(caption, dict) and "text" in caption:
+                caption = caption["text"]
+            self.save_caption(filename=filename, filename_alt=filename_old, mtime=date_local, caption=caption)
+        else:
+            self._log("<no caption>", end=' ', flush=True)
+        if "video_versions" in item and self.download_videos is Tristate.always:
+            downloaded |= self.download_pic(filename=filename,
+                                            filename_alt=filename_old,
+                                            url=item["video_versions"][0]["url"],
+                                            mtime=date_local)
+        if item["story_locations"] and self.download_geotags is not Tristate.never:
+            location = item["story_locations"][0]["location"]
+            if location:
+                self.save_location(filename, location, date_local)
+        self._log()
+        return downloaded
+
     def get_feed_posts(self) -> Iterator[Post]:
         """Get Posts of the user's feed."""
 
@@ -1011,7 +1188,7 @@ class Instaloader:
         """
         Download pictures from the user's feed.
 
-        Example to download up to the 20 pics the user last liked: ::
+        Example to download up to the 20 pics the user last liked::
 
             loader = Instaloader()
             loader.load_session_from_file('USER')
@@ -1050,7 +1227,7 @@ class Instaloader:
                          fast_update: bool = False) -> None:
         """Download pictures of one hashtag.
 
-        To download the last 30 pictures with hashtag #cat, do ::
+        To download the last 30 pictures with hashtag #cat, do::
 
             loader = Instaloader()
             loader.download_hashtag('cat', max_count=30)
@@ -1081,7 +1258,8 @@ class Instaloader:
         has changed and return current name of the profile, and store ID of profile.
 
         :param profile: Profile name
-        :param profile_metadata: The profile's metadata (get_profile_metadata()), or None if the profile was not found
+        :param profile_metadata:
+            The profile's metadata (:meth:`get_profile_metadata`), or None if the profile was not found
         :return: current profile name, profile id
         """
         profile_exists = profile_metadata is not None
@@ -1126,7 +1304,7 @@ class Instaloader:
         raise ProfileNotExistsException("Profile {0} does not exist.".format(profile))
 
     def get_profile_metadata(self, profile_name: str) -> Dict[str, Any]:
-        """Retrieves a profile's metadata, for use with e.g. get_profile_posts() and check_profile_id()."""
+        """Retrieves a profile's metadata, for use with e.g. :meth:`get_profile_posts` and :meth:`check_profile_id`."""
         try:
             return self.get_json('{}/'.format(profile_name), params={'__a': 1})
         except QueryReturnedNotFoundException:
@@ -1135,23 +1313,27 @@ class Instaloader:
     def get_profile_posts(self, profile_metadata: Dict[str, Any]) -> Iterator[Post]:
         """Retrieve all posts from a profile."""
         profile_name = profile_metadata['user']['username']
-        yield from (Post(self, node, profile=profile_name) for node in profile_metadata['user']['media']['nodes'])
+        profile_id = int(profile_metadata['user']['id'])
+        yield from (Post(self, node, profile=profile_name, profile_id=profile_id)
+                    for node in profile_metadata['user']['media']['nodes'])
         has_next_page = profile_metadata['user']['media']['page_info']['has_next_page']
         end_cursor = profile_metadata['user']['media']['page_info']['end_cursor']
         while has_next_page:
             # We do not use self.graphql_node_list() here, because profile_metadata
             # lets us obtain the first 12 nodes 'for free'
             data = self.graphql_query(17888483320059182, {'id': profile_metadata['user']['id'],
-                                                          'first': 500,
+                                                          'first': 200,
                                                           'after': end_cursor},
                                       'https://www.instagram.com/{0}/'.format(profile_name))
             media = data['data']['user']['edge_owner_to_timeline_media']
-            yield from (Post(self, edge['node'], profile=profile_name) for edge in media['edges'])
+            yield from (Post(self, edge['node'], profile=profile_name, profile_id=profile_id)
+                        for edge in media['edges'])
             has_next_page = media['page_info']['has_next_page']
             end_cursor = media['page_info']['end_cursor']
 
     def download_profile(self, name: str,
-                         profile_pic_only: bool = False, fast_update: bool = False,
+                         profile_pic: bool = True, profile_pic_only: bool = False,
+                         fast_update: bool = False,
                          download_stories: bool = False, download_stories_only: bool = False,
                          filter_func: Optional[Callable[[Post], bool]] = None) -> None:
         """Download one profile"""
@@ -1172,8 +1354,9 @@ class Instaloader:
             profile_metadata = self.get_profile_metadata(name)
 
         # Download profile picture
-        with self._error_catcher('Download profile picture of {}'.format(name)):
-            self.download_profilepic(name, profile_metadata["user"]["profile_pic_url"])
+        if profile_pic or profile_pic_only:
+            with self._error_catcher('Download profile picture of {}'.format(name)):
+                self.download_profilepic(name, profile_metadata)
         if profile_pic_only:
             return
 
@@ -1231,7 +1414,8 @@ class Instaloader:
 
     def main(self, profilelist: List[str], username: Optional[str] = None, password: Optional[str] = None,
              sessionfile: Optional[str] = None, max_count: Optional[int] = None,
-             profile_pic_only: bool = False, fast_update: bool = False,
+             profile_pic: bool = True, profile_pic_only: bool = False,
+             fast_update: bool = False,
              stories: bool = False, stories_only: bool = False,
              filter_str: Optional[str] = None) -> None:
         """Download set of profiles, hashtags etc. and handle logging in and session files if desired."""
@@ -1295,7 +1479,7 @@ class Instaloader:
             for target in targets:
                 with self._error_catcher():
                     try:
-                        self.download_profile(target, profile_pic_only, fast_update, stories, stories_only,
+                        self.download_profile(target, profile_pic, profile_pic_only, fast_update, stories, stories_only,
                                               filter_func=filter_func)
                     except ProfileNotExistsException as err:
                         if username is not None:
@@ -1303,8 +1487,8 @@ class Instaloader:
                             self._log("Trying again anonymously, helps in case you are just blocked.")
                             with self.anonymous_copy() as anonymous_loader:
                                 with self._error_catcher():
-                                    anonymous_loader.download_profile(target, profile_pic_only, fast_update,
-                                                                      filter_func=filter_func)
+                                    anonymous_loader.download_profile(target, profile_pic, profile_pic_only,
+                                                                      fast_update, filter_func=filter_func)
                         else:
                             raise err
         except KeyboardInterrupt:
@@ -1339,8 +1523,12 @@ def main():
                              'followees.')
     g_what.add_argument('-P', '--profile-pic-only', action='store_true',
                         help='Only download profile picture.')
+    g_what.add_argument('--no-profile-pic', action='store_true',
+                        help='Do not download profile picture.')
     g_what.add_argument('-V', '--no-videos', action='store_true',
                         help='Do not download videos.')
+    g_what.add_argument('--no-video-thumbnails', action='store_true',
+                        help='Do not download thumbnails of videos.')
     g_what.add_argument('-G', '--geotags', action='store_true',
                         help='Download geotags when available. Geotags are stored as a '
                              'text file with the location\'s name and a Google Maps link. '
@@ -1354,11 +1542,14 @@ def main():
                              'server for each post, which is why it is disabled by default.')
     g_what.add_argument('--no-captions', action='store_true',
                         help='Do not store media captions, although no additional request is needed to obtain them.')
+    g_what.add_argument('--metadata-json', action='store_true',
+                        help='Create a JSON file containing the metadata of each post. This does not include comments '
+                             'nor geotags.')
     g_what.add_argument('-s', '--stories', action='store_true',
                         help='Also download stories of each profile that is downloaded. Requires --login.')
     g_what.add_argument('--stories-only', action='store_true',
                         help='Rather than downloading regular posts of each specified profile, only download '
-                             'stories. Requires --login.')
+                             'stories. Requires --login. Does not imply --no-profile-pic.')
     g_what.add_argument('--only-if', metavar='filter',
                         help='Expression that, if given, must evaluate to True for each post to be downloaded. Must be '
                              'a syntactically valid python expression. Variables are evaluated to '
@@ -1372,7 +1563,7 @@ def main():
                              'flag is recommended when you use Instaloader to update your personal Instagram archive.')
     g_stop.add_argument('-c', '--count',
                         help='Do not attempt to download more than COUNT posts. '
-                             'Applies only to #hashtag, :feed-all and :feed-liked.')
+                             'Applies only to #hashtag and :feed.')
 
     g_login = parser.add_argument_group('Login (Download Private Profiles)',
                                         'Instaloader can login to Instagram. This allows downloading private profiles. '
@@ -1398,11 +1589,16 @@ def main():
                        help='Prefix of filenames. Posts are stored in the directory whose pattern is given with '
                             '--dirname-pattern. {profile} is replaced by the profile name, '
                             '{target} is replaced by the target you specified, i.e. either :feed, #hashtag or the '
-                            'profile name. Also, the fields date and shortcode can be specified. Defaults to '
-                            '\'{date:%%Y-%%m-%%d_%%H-%%M-%%S}\'.')
+                            'profile name. Also, the fields {date} and {shortcode} can be specified. In case of not '
+                            'downloading stories, the attributes of the Post class can be used in addition, e.g. '
+                            '{post.owner_id} or {post.mediaid}. Defaults to \'{date:%%Y-%%m-%%d_%%H-%%M-%%S}\'.')
     g_how.add_argument('--user-agent',
                        help='User Agent to use for HTTP requests. Defaults to \'{}\'.'.format(default_user_agent()))
     g_how.add_argument('-S', '--no-sleep', action='store_true', help=SUPPRESS)
+    g_how.add_argument('--max-connection-attempts', metavar='N', type=int, default=3,
+                       help='Maximum number of connection attempts until a request is aborted. Defaults to 3. If a '
+                            'connection fails, it can be manually skipped by hitting CTRL+C. Set this to 0 to retry '
+                            'infinitely.')
 
     g_misc = parser.add_argument_group('Miscellaneous Options')
     g_misc.add_argument('-q', '--quiet', action='store_true',
@@ -1426,8 +1622,10 @@ def main():
                              "eventually --only-if=viewer_has_liked.")
 
         download_videos = Tristate.always if not args.no_videos else Tristate.no_extra_query
+        download_video_thumbnails = Tristate.always if not args.no_video_thumbnails else Tristate.never
         download_comments = Tristate.always if args.comments else Tristate.no_extra_query
-        download_captions = Tristate.no_extra_query if not args.no_captions else Tristate.never
+        save_captions = Tristate.no_extra_query if not args.no_captions else Tristate.never
+        save_metadata = Tristate.always if args.metadata_json else Tristate.never
 
         if args.geotags and args.no_geotags:
             raise SystemExit("--geotags and --no-geotags given. I am confused and refuse to work.")
@@ -1441,12 +1639,21 @@ def main():
         loader = Instaloader(sleep=not args.no_sleep, quiet=args.quiet,
                              user_agent=args.user_agent,
                              dirname_pattern=args.dirname_pattern, filename_pattern=args.filename_pattern,
-                             download_videos=download_videos, download_geotags=download_geotags,
-                             download_captions=download_captions, download_comments=download_comments)
-        loader.main(args.profile, args.login.lower() if args.login is not None else None, args.password,
-                    args.sessionfile,
-                    int(args.count) if args.count is not None else None,
-                    args.profile_pic_only, args.fast_update, args.stories, args.stories_only, args.only_if)
+                             download_videos=download_videos, download_video_thumbnails=download_video_thumbnails,
+                             download_geotags=download_geotags,
+                             save_captions=save_captions, download_comments=download_comments,
+                             save_metadata=save_metadata, max_connection_attempts=args.max_connection_attempts)
+        loader.main(args.profile,
+                    username=args.login.lower() if args.login is not None else None,
+                    password=args.password,
+                    sessionfile=args.sessionfile,
+                    max_count=int(args.count) if args.count is not None else None,
+                    profile_pic=not args.no_profile_pic,
+                    profile_pic_only=args.profile_pic_only,
+                    fast_update=args.fast_update,
+                    stories=args.stories,
+                    stories_only=args.stories_only,
+                    filter_str=args.only_if)
     except InstaloaderException as err:
         raise SystemExit("Fatal error: %s" % err)