Release of version 3.2.1

Fixes #59.

README.rst

@@ -1,39 +1,37 @@
 Instaloader
 ===========
 
-Installation
-------------
+::
 
-.. installation-start
+    $ pip3 install instaloader
 
-Instaloader requires `Python <https://www.python.org/>`__, at least
-version 3.5.  If you have `pip <https://pypi.python.org/pypi/pip>`__
-installed, you may install Instaloader using
+    $ instaloader profile [profile ...]
+
+**Instaloader**
+
+- downloads **public and private profiles, hashtags, user stories and
+  feeds**,
+
+- downloads **comments, geotags and captions** of each post,
+
+- automatically **detects profile name changes** and renames the target
+  directory accordingly,
+
+- allows **fine-grained customization** of filters and where to store
+  downloaded media.
 
 ::
 
-    pip3 install instaloader
+    instaloader [--comments] [--geotags] [--stories]
+                [--login YOUR-USERNAME] [--fast-update]
+                profile | "#hashtag" | :stories | :feed
 
-Alternatively, to get the most current version of Instaloader from our
-`Git repository <https://github.com/Thammus/instaloader>`__:
+`Instaloader Documentation <https://instaloader.readthedocs.io/>`__
 
-::
-
-    pip3 install git+https://github.com/Thammus/instaloader
-
-(pass ``--upgrade`` to upgrade if Instaloader is already installed)
-
-Instaloader requires
-`requests <http://python-requests.org/>`__, which
-will be installed automatically, if it is not already installed.
-
-.. installation-end
 
 How to Automatically Download Pictures from Instagram
 -----------------------------------------------------
 
-.. basic-usage-start
-
 To **download all pictures and videos of a profile**, as well as the
 **profile picture**, do
 
@@ -52,8 +50,8 @@ To later **update your local copy** of that profiles, you may run
 
 If ``--fast-update`` is given, Instaloader stops when arriving at the
 first already-downloaded picture. When updating profiles, Instaloader
-automatically **detects profile name changes** and renames the target
-directory accordingly.
+automatically **detects profile name changes** and renames the target directory
+accordingly.
 
 Instaloader can also be used to **download private profiles**. To do so,
 invoke it with
@@ -62,291 +60,13 @@ invoke it with
 
     instaloader --login=your_username profile [profile ...]
 
-When logging in, Instaloader **stores the session cookies** in a file in
-your temporary directory, which will be reused later the next time
-``--login`` is given.  So you can download private profiles
-**non-interactively** when you already have a valid session cookie file.
+When logging in, Instaloader **stores the session cookies** in a file in your
+temporary directory, which will be reused later the next time ``--login``
+is given.  So you can download private profiles **non-interactively** when you
+already have a valid session cookie file.
 
-What to Download
-^^^^^^^^^^^^^^^^
+`Instaloader Documentation <https://instaloader.readthedocs.io/basic-usage.html>`__
 
-Instaloader does not only download media by-profile. More generally, you
-may specify the following targets:
-
-- ``profile``: Public profile, or private profile with ``--login``,
-
-- ``"#hashtag"``: Posts with a certain **hashtag** (the quotes are
-  usually neccessary),
-
-- ``:stories``: The currently-visible **stories** of your followees
-  (requires ``--login``),
-
-- ``:feed``: Your **feed** (requires ``--login``),
-
-- ``@profile``: All profiles that are followed by ``profile``, i.e. the
-  *followees* of ``profile`` (requires ``--login``).
-
-Instaloader goes through all media matching the specified targets and
-downloads the pictures and videos and their captions. You can specify
-``--comments`` to also **download comments** of each post and
-``--geotags`` to **download geotags** of each post and save them as
-Google Maps link.  For each profile you download, ``--stories``
-instructs Instaloader to **download the user's stories**.
-
-Filename Specification
-^^^^^^^^^^^^^^^^^^^^^^
-
-For each target, Instaloader creates a directory named after the target,
-i.e. ``profile``, ``#hashtag``, ``:feed``, etc. and therein saves the
-posts in files named after the post's timestamp.
-
-``--dirname-pattern`` allows to configure the directory name of each
-target. The default is ``--dirname-pattern={target}``. In the dirname
-pattern, the token ``{target}`` is replaced by the target name, and
-``{profile}`` is replaced by the owner of the post which is downloaded.
-
-``--filename-pattern`` configures the path of the post's files relative
-to the target directory. The default is ``--filename-pattern={date}``.
-The tokens ``{target}`` and ``{profile}`` are replaced like in the
-dirname pattern. Further, the tokens ``{date}`` and ``{shortcode}`` are
-defined.
-
-For example, encode the poster's profile name in the filenames with:
-
-::
-
-    instaloader --filename-pattern={date}_{profile} "#hashtag"
-
-The pattern string is formatted with Python's string formatter. This
-gives additional flexibilty for pattern specification. For example,
-`strftime-style formatting options <https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior>`__
-are supported for the post's
-timestamp. The default for ``{date}`` is ``{date:%Y-%m-%d_%H-%M-%S}``.
-
-Filter Posts
-^^^^^^^^^^^^
-
-The ``--only-if`` option allows to specify criterias that posts have to
-meet to be downloaded. If not given, all posts are downloaded. It must
-be a boolean Python expression where the variables ``likes``,
-``comments``, ``viewer_has_liked``, ``is_video``, and many
-more are defined.
-
-A few examples:
-
-To **download the pictures from your feed that you have liked**:
-
-::
-
-    instaloader --login=your_username --only-if=viewer_has_liked :feed
-
-Or you might only want to download **posts that either you liked or were
-liked by many others**:
-
-::
-
-    instaloader --login=your_username --only-if="likes>100 or viewer_has_liked" profile
-
-Or you may **skip videos**:
-
-::
-
-    instaloader --only-if="not is_video" target
-
-Or you may filter by hashtags that occur in the Post's caption. For
-example, to download posts of kittens that are cute: ::
-
-    instaloader --only-if="'cute' in caption_hashtags" "#kitten"
-
-.. basic-usage-end
-
-(For a more complete description of the ``-only-if`` option, refer to
-the `Instaloader Documentation <https://instaloader.readthedocs.io/basic-usage.html#filter-posts>`__)
-
-
-Advanced Options
-----------------
-
-.. cli-options-start
-
-The following flags can be given to Instaloader to specify how profiles should
-be downloaded.
-
-To get a list of all flags, their abbreviations and their descriptions, you may
-run ``instaloader --help``.
-
-What to Download
-^^^^^^^^^^^^^^^^
-
-Specify a list of profiles or #hashtags. For each of these, Instaloader
-creates a folder and downloads all posts along with the pictures's
-captions and the current **profile picture**. If an already-downloaded profile
-has been renamed, Instaloader automatically **finds it by its unique ID** and
-renames the folder likewise.
-
-Instead of a *profile* or a *#hashtag*, the special targets
-``:feed`` (pictures from your feed) and
-``:stories`` (stories of your followees) can be specified.
-
---profile-pic-only         Only download profile picture.
---no-videos                Do not download videos.
---geotags                  **Download geotags** when available. Geotags are stored as
-                           a text file with the location's name and a Google Maps
-                           link. This requires an additional request to the
-                           Instagram server for each picture, which is why it is
-                           disabled by default.
---no-geotags               Do not store geotags, even if they can be obtained
-                           without any additional request.
---comments                 Download and update comments for each post. This
-                           requires an additional request to the Instagram server
-                           for each post, which is why it is disabled by default.
---no-captions              Do not store media captions, although no additional
-                           request is needed to obtain them.
---stories                  Also **download stories** of each profile that is
-                           downloaded. Requires ``--login``.
---stories-only             Rather than downloading regular posts of each
-                           specified profile, only download stories.
-                           Requires ``--login``.
---only-if filter           Expression that, if given, must evaluate to True for each post to
-                           be downloaded. Must be a syntactically valid python
-                           expression. Variables are evaluated to
-                           ``instaloader.Post`` attributes.
-                           Example: ``--only-if=viewer_has_liked``.
-
-
-When to Stop Downloading
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-If none of these options are given, Instaloader goes through all pictures
-matching the specified targets.
-
---fast-update              For each target, stop when encountering the first
-                           already-downloaded picture. This flag is recommended
-                           when you use Instaloader to update your personal
-                           Instagram archive.
---count COUNT              Do not attempt to download more than COUNT posts.
-                           Applies only to ``#hashtag``, ``:feed-all`` and ``:feed-liked``.
-
-
-Login (Download Private Profiles)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Instaloader can **login to Instagram**. This allows downloading private
-profiles. To login, pass the ``--login`` option. Your session cookie (not your
-password!) will be saved to a local file to be reused next time you want
-Instaloader to login.
-
---login YOUR-USERNAME      Login name (profile name) for your Instagram account.
---sessionfile SESSIONFILE  Path for loading and storing session key file.
-                           Defaults to a path
-                           within your temporary directory, encoding your local
-                           username and your Instagram profile name.
---password YOUR-PASSWORD   Password for your Instagram account. Without this
-                           option, you'll be prompted for your password
-                           interactively if there is not yet a valid session
-                           file.
-
-How to Download
-^^^^^^^^^^^^^^^
-
---dirname-pattern DIRNAME_PATTERN
-                           Name of directory where to store posts. ``{profile}``
-                           is replaced by the profile name, ``{target}`` is replaced
-                           by the target you specified, i.e. either ``:feed``,
-                           ``#hashtag`` or the profile name. Defaults to ``{target}``.
---filename-pattern FILENAME_PATTERN
-                           Prefix of filenames. Posts are stored in the
-                           directory whose pattern is given with ``--dirname-pattern``.
-                           ``{profile}`` is replaced by the profile name,
-                           ``{target}`` is replaced by the target you specified, i.e.
-                           either ``:feed``, ``#hashtag`` or the profile name. Also, the
-                           fields ``{date}`` and ``{shortcode}`` can be specified.
-                           Defaults to ``{date:%Y-%m-%d_%H-%M-%S}``.
---user-agent USER_AGENT    User Agent to use for HTTP requests. Per default,
-                           Instaloader pretends being Chrome/51.
-
-Miscellaneous Options
-^^^^^^^^^^^^^^^^^^^^^
-
---quiet                    Disable user interaction, i.e. do not print messages
-                           (except errors) and fail if login credentials are
-                           needed but not given. This makes Instaloader
-                           **suitable as a cron job**.
-
-.. cli-options-end
-
-Usage as Python module
-----------------------
-
-.. as-module-intro-start
-
-You may also use parts of Instaloader as library to do other interesting
-things.
-
-For example, to get a list of all followees and a list of all followers of a profile, do
-
-.. code:: python
-
-    import instaloader
-
-    # Get instance
-    loader = instaloader.Instaloader()
-
-    # Login
-    loader.interactive_login(USERNAME)
-
-    # Print followees
-    print(PROFILE + " follows these profiles:")
-    for f in loader.get_followees(PROFILE):
-        print("\t%s\t%s" % (f['username'], f['full_name']))
-
-    # Print followers
-    print("Followers of " + PROFILE + ":")
-    for f in loader.get_followers(PROFILE):
-        print("\t%s\t%s" % (f['username'], f['full_name']))
-
-Then, you may download all pictures of all followees with
-
-.. code:: python
-
-    for f in loader.get_followees(PROFILE):
-        loader.download_profile(f['username'])
-
-You could also download your last 20 liked pics with
-
-.. code:: python
-
-    loader.download_feed_posts(max_count=20, fast_update=True,
-                               filter_func=lambda post: post.viewer_has_liked)
-
-To download the last 20 pictures with hashtag #cat, do
-
-.. code:: python
-
-    loader.download_hashtag('cat', max_count=20)
-
-Generally, Instaloader provides methods to iterate over the Posts from
-a certain source.
-
-.. code:: python
-
-    for post in loader.get_hashtag_posts('cat'):
-        # post is an instance of instaloader.Post
-        loader.download_post(post, target='#cat')
-
-Each Instagram profile has its own unique ID which stays unmodified even
-if a user changes his/her username. To get said ID, given the profile's
-name, you may call
-
-.. code:: python
-
-    loader.get_id_by_username(PROFILE_NAME)
-
-.. as-module-intro-end
-
-Refer to the
-`Instaloader Documentation <https://instaloader.readthedocs.io/as-module.html>`__ for
-more information.
 
 Disclaimer
 ----------

docs/README.md

@@ -0,0 +1,15 @@
+# Instaloader Documentation
+
+This directory contains sources for the Instaloader documentation.
+
+The documentation is hosted at:
+https://instaloader.readthedocs.io/
+
+## Build
+
+The documentation is created with [Sphinx](http://www.sphinx-doc.org/). To build it, use
+
+```
+pip3 install sphinx sphinx-autodoc-typehints
+make html
+```

docs/_templates/caption.html

@@ -1 +1,2 @@
-<h2>Instaloader</h2>
+<h2><a href="index.html">Instaloader</a></h2>
+<p>Download pictures (or videos) along with their captions and other metadata from Instagram.</p>

docs/_templates/layout.html

@@ -1,4 +1,5 @@
 {% extends "!layout.html" %}
 {% block extrahead %}
+<meta name="google-site-verification" content="0esgS4FkYROvPe-I9MYvC1tTYtfASPpqxBdpTHEBIcQ" />
 <script async defer src="https://buttons.github.io/buttons.js"></script>
 {% endblock %}

docs/as-module.rst

@@ -1,26 +1,116 @@
-Usage as Python Module
-----------------------
+.. meta::
+   :description:
+      Instaloader can also be used as a powerful and easy-to-use
+      Python API for Instagram, allowing to download media and metadata.
 
-Introduction
-^^^^^^^^^^^^
+Python Module :mod:`instaloader`
+--------------------------------
 
-.. include:: ../README.rst
-   :start-after: as-module-intro-start
-   :end-before: as-module-intro-end
+.. module:: instaloader
 
+.. highlight:: python
+
+Instaloader exposes its internally used methods as a Python module, making it a
+**powerful and easy-to-use Python API for Instagram**, allowing to further
+customize obtaining media and metadata.
+
+Start with getting an instance of :class:`Instaloader`::
+
+    import instaloader
+
+    # Get instance
+    L = instaloader.Instaloader()
+
+    # Optionally, login or load session
+    L.login(USER, PASSWORD)        # (login)
+    L.interactive_login(USER)      # (ask password on terminal)
+    L.load_session_from_file(USER) # (load session created w/
+                                   #  `instaloader -l USERNAME`)
+
+:mod:`instaloader` provides the :class:`Post` structure, which represents a
+picture, video or sidecar (set of multiple pictures/videos) posted in a user's
+profile. :class:`Instaloader` provides methods to iterate over Posts from a
+certain source::
+
+    for post in L.get_hashtag_posts('cat'):
+        # post is an instance of instaloader.Post
+        L.download_post(post, target='#cat')
+
+Besides :func:`Instaloader.get_hashtag_posts`, there is
+:func:`Instaloader.get_feed_posts` and :func:`Instaloader.get_profile_posts`.
+Also, :class:`Post` instances can be created with :func:`Post.from_shortcode`
+and :func:`Post.from_mediaid`.
+
+Further, information about profiles can be easily obtained. For example, you may
+print a list of all followees and followers of a profile with::
+
+    # Print followees
+    print(PROFILE + " follows these profiles:")
+    for f in L.get_followees(PROFILE):
+        print("\t%s\t%s" % (f['username'], f['full_name']))
+
+    # Print followers
+    print("Followers of " + PROFILE + ":")
+    for f in L.get_followers(PROFILE):
+        print("\t%s\t%s" % (f['username'], f['full_name']))
+
+Then, you may download all pictures of all followees with::
+
+    for f in L.get_followees(PROFILE):
+        L.download_profile(f['username'])
+
+Each Instagram profile has its own unique ID which stays unmodified even if a
+user changes his/her username. To get said ID, given the profile's name, you may
+call::
+
+    L.get_id_by_username(PROFILE_NAME)
+
+A reference of the many methods provided by the :mod:`instaloader` module is
+provided in the remainder of this document. Feel free to direct any issue or
+contribution to
+`Instaloader on Github <https://github.com/Thammus/instaloader>`__.
 
 ``Instaloader`` (Main Class)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-.. autoclass:: instaloader.Instaloader
-   :members:
-   :undoc-members:
-
-.. _post-class:
+.. autoclass:: Instaloader
+   :no-show-inheritance:
 
 ``Post`` Class
 ^^^^^^^^^^^^^^
 
-.. autoclass:: instaloader.Post
-   :members:
-   :undoc-members:
+.. autoclass:: Post
+   :no-show-inheritance:
+
+Miscellaneous Functions
+^^^^^^^^^^^^^^^^^^^^^^^
+
+.. autofunction:: shortcode_to_mediaid
+
+.. autofunction:: mediaid_to_shortcode
+
+.. autoclass:: Tristate
+
+Exceptions
+^^^^^^^^^^
+
+.. autoexception:: InstaloaderException
+   :no-show-inheritance:
+
+.. autoexception:: QueryReturnedNotFoundException
+
+.. autoexception:: ProfileNotExistsException
+
+.. autoexception:: ProfileHasNoPicsException
+
+.. autoexception:: PrivateProfileNotFollowedException
+
+.. autoexception:: LoginRequiredException
+
+.. autoexception:: InvalidArgumentException
+
+.. autoexception:: BadResponseException
+
+.. autoexception:: BadCredentialsException
+
+.. autoexception:: ConnectionException

docs/basic-usage.rst

@@ -1,15 +1,152 @@
 Download Pictures from Instagram
 ---------------------------------
 
+.. highlight:: none
+
+.. NOTE that Section "Basic Usage" is duplicated in README.rst.
+
 Basic Usage
 ^^^^^^^^^^^
 
-.. include:: ../README.rst
-   :start-after: basic-usage-start
-   :end-before: basic-usage-end
+To **download all pictures and videos of a profile**, as well as the
+**profile picture**, do
 
-.. (continuation of --only-if explanation)
+::
+
+    instaloader profile [profile ...]
+
+where ``profile`` is the name of a profile you want to download. Instead
+of only one profile, you may also specify a list of profiles.
+
+To later **update your local copy** of that profiles, you may run
+
+::
+
+    instaloader --fast-update profile [profile ...]
+
+If :option:`--fast-update` is given, Instaloader stops when arriving at the
+first already-downloaded picture. When updating profiles, Instaloader
+automatically **detects profile name changes** and renames the target directory
+accordingly.
+
+Instaloader can also be used to **download private profiles**. To do so,
+invoke it with
+
+::
+
+    instaloader --login=your_username profile [profile ...]
+
+When logging in, Instaloader **stores the session cookies** in a file in your
+temporary directory, which will be reused later the next time :option:`--login`
+is given.  So you can download private profiles **non-interactively** when you
+already have a valid session cookie file.
+
+.. _what-to-download:
+
+What to Download
+^^^^^^^^^^^^^^^^
+
+Instaloader supports the following targets:
+
+``profile``
+   Public profile, or private profile with :option:`--login`. For each profile
+   you download, :option:`--stories` instructs Instaloader to also
+   **download the user's stories**.
+
+``"#hashtag"``
+   Posts with a certain **hashtag** (the quotes are usually neccessary),
+
+``:stories``
+   The currently-visible **stories** of your followees (requires
+   :option:`--login`),
+
+``:feed``
+   Your **feed** (requires :option:`--login`),
+
+``@profile``
+   All profiles that are followed by ``profile``, i.e. the *followees* of
+   ``profile`` (requires :option:`--login`).
+
+Instaloader goes through all media matching the specified targets and
+downloads the pictures and videos and their captions. You can specify
+
+- :option:`--comments`, to also **download comments** of each post,
+
+- :option:`--geotags`, to **download geotags** of each post and save them as
+  Google Maps link,
+
+- :option:`--metadata-json`, to store further post metadata in a separate JSON
+  file.
+
+.. _filename-specification:
+
+Filename Specification
+^^^^^^^^^^^^^^^^^^^^^^
+
+For each target, Instaloader creates a directory named after the target,
+i.e. ``profile``, ``#hashtag``, ``:feed``, etc. and therein saves the
+posts in files named after the post's timestamp.
+
+:option:`--dirname-pattern` allows to configure the directory name of each
+target. The default is ``--dirname-pattern={target}``. In the dirname
+pattern, the token ``{target}`` is replaced by the target name, and
+``{profile}`` is replaced by the owner of the post which is downloaded.
+
+:option:`--filename-pattern` configures the path of the post's files relative
+to the target directory. The default is ``--filename-pattern={date}``.
+The tokens ``{target}`` and ``{profile}`` are replaced like in the
+dirname pattern. Further, the tokens ``{date}`` and ``{shortcode}`` are
+defined. Additionally, in case of not downloading stories, the attributes of
+:class:`.Post` can be used, e.g. ``{post.owner_id}`` or ``{post.mediaid}``.
+
+For example, encode the poster's profile name in the filenames with:
+
+::
+
+    instaloader --filename-pattern={date}_{profile} "#hashtag"
+
+The pattern string is formatted with Python's string formatter. This
+gives additional flexibilty for pattern specification. For example,
+`strftime-style formatting options <https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior>`__
+are supported for the post's
+timestamp. The default for ``{date}`` is ``{date:%Y-%m-%d_%H-%M-%S}``.
+
+.. _filter-posts:
+
+Filter Posts
+^^^^^^^^^^^^
+
+The :option:`--only-if` option allows to specify criterias that posts have to
+meet to be downloaded. If not given, all posts are downloaded. It must be a
+boolean Python expression where the variables :attr:`.likes`, :attr:`.comments`,
+:attr:`.viewer_has_liked`, :attr:`.is_video`, and many more are defined.
+
+A few examples:
+
+To **download the pictures from your feed that you have liked**:
+
+::
+
+    instaloader --login=your_username --only-if=viewer_has_liked :feed
+
+Or you might only want to download **posts that either you liked or were
+liked by many others**:
+
+::
+
+    instaloader --login=your_username --only-if="likes>100 or viewer_has_liked" profile
+
+Or you may **skip videos**:
+
+::
+
+    instaloader --only-if="not is_video" target
+
+Or you may filter by hashtags that occur in the Post's caption. For
+example, to download posts of kittens that are cute: ::
+
+    instaloader --only-if="'cute' in caption_hashtags" "#kitten"
 
 The given string is evaluated as a
 `Python boolean expression <https://docs.python.org/3/reference/expressions.html#boolean-operations>`__,
-where all occuring variables are properties of the :ref:`post-class`.
+where all occuring variables are attributes of the :class:`.Post` class.

docs/cli-options.rst

@@ -1,6 +1,163 @@
 Command Line Options
 ====================
 
-.. include:: ../README.rst
-   :start-after: cli-options-start
-   :end-before: cli-options-end
+Instaloader is invoked with::
+
+   $ instaloader [options] target [target ...]
+
+where ``target`` is a ``profile``, a ``"#hashtag"``, ``@profile`` (all profiles
+that *profile* is following), or if logged in ``:feed`` (pictures from your
+feed) or ``:stories`` (stories of your followees).
+
+Here we explain the additional options that can be given to Instaloader to
+customize its behavior.  To get a list of all flags, their abbreviations and
+their descriptions, you may also run ``instaloader --help``.
+
+What to Download
+^^^^^^^^^^^^^^^^
+
+Specify a list of targets (profiles, #hashtags, ``:feed`` or ``:stories``). For
+each of these, Instaloader creates a folder and stores all posts along with the
+pictures's captions and the current **profile picture** there. If an
+already-downloaded profile has been renamed, Instaloader automatically **finds
+it by its unique ID** and renames the folder likewise.
+
+.. option:: --profile-pic-only
+
+   Only download profile picture.
+
+.. option:: --no-videos
+
+   Do not download videos.
+
+.. option:: --geotags
+
+   **Download geotags** when available. Geotags are stored as a text file with
+   the location's name and a Google Maps link. This requires an additional
+   request to the Instagram server for each picture, which is why it is disabled
+   by default.
+
+.. option:: --no-geotags
+
+   Do not store geotags, even if they can be obtained without any additional
+   request.
+
+.. option:: --comments
+
+   Download and update comments for each post. This requires an additional
+   request to the Instagram server for each post, which is why it is disabled by
+   default.
+
+.. option:: --no-captions
+
+   Do not store media captions, although no additional request is needed to
+   obtain them.
+
+.. option:: --stories
+
+   Also **download stories** of each profile that is downloaded. Requires
+   :option:`--login`.
+
+.. option:: --metadata-json
+
+   Create a JSON file containing the metadata of each post. This does not
+   include comments (see :option:`--comments`) nor geotags (see
+   :option:`--geotags`). The JSON files contain the properties of
+   :class:`instaloader.Post`.
+
+.. option:: --stories-only
+
+   Rather than downloading regular posts of each specified profile, only
+   download stories.  Requires :option:`--login`.
+
+.. option:: --only-if filter
+
+   Expression that, if given, must evaluate to True for each post to be
+   downloaded.  Must be a syntactically valid Python expression. Variables are
+   evaluated to :class:`instaloader.Post` attributes.  Example:
+   ``--only-if=viewer_has_liked``. See :ref:`filter-posts` for more
+   examples.
+
+
+When to Stop Downloading
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+If none of these options are given, Instaloader goes through all pictures
+matching the specified targets.
+
+.. option:: --fast-update
+
+   For each target, stop when encountering the first already-downloaded picture.
+   This flag is recommended when you use Instaloader to update your personal
+   Instagram archive.
+
+.. option:: --count COUNT
+
+   Do not attempt to download more than COUNT posts.  Applies only to
+   ``#hashtag`` and ``:feed``.
+
+
+Login (Download Private Profiles)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Instaloader can **login to Instagram**. This allows downloading private
+profiles. To login, pass the :option:`--login` option. Your session cookie (not your
+password!) will be saved to a local file to be reused next time you want
+Instaloader to login.
+
+.. option:: --login YOUR-USERNAME
+
+   Login name (profile name) for your Instagram account.
+
+.. option:: --sessionfile SESSIONFILE
+
+   Path for loading and storing session key file.  Defaults to a path within
+   your temporary directory, encoding your local username and your Instagram
+   profile name.
+
+.. option:: --password YOUR-PASSWORD
+
+   Password for your Instagram account.  Without this option, you'll be prompted
+   for your password interactively if there is not yet a valid session file.
+
+How to Download
+^^^^^^^^^^^^^^^
+
+.. option:: --dirname-pattern DIRNAME_PATTERN
+
+   Name of directory where to store posts. ``{profile}`` is replaced by the
+   profile name, ``{target}`` is replaced by the target you specified, i.e.
+   either ``:feed``, ``#hashtag`` or the profile name. Defaults to ``{target}``.
+   See :ref:`filename-specification`.
+
+.. option:: --filename-pattern FILENAME_PATTERN
+
+   Prefix of filenames. Posts are stored in the directory whose pattern is given
+   with ``--dirname-pattern``.  ``{profile}`` is replaced by the profile name,
+   ``{target}`` is replaced by the target you specified, i.e.  either ``:feed``,
+   ``#hashtag`` or the profile name.  Also, the fields ``{date}`` and
+   ``{shortcode}`` can be specified.  In case of not downloading stories, the
+   attributes of the :class:`.Post` class can be used in addition, e.g.
+   ``{post.owner_id}`` or ``{post.mediaid}``.
+   Defaults to
+   ``{date:%Y-%m-%d_%H-%M-%S}``.  See :ref:`filename-specification`.
+
+.. option:: --user-agent USER_AGENT
+
+   User Agent to use for HTTP requests. Per default, Instaloader pretends being
+   Chrome/51.
+
+.. option:: --max-connection-attempts N
+
+   Maximum number of connection attempts until a request is aborted. Defaults
+   to ``3``. If a connection fails, it can be manually skipped by hitting
+   :kbd:`Control-c`. Set this to ``0`` to retry infinitely.
+
+Miscellaneous Options
+^^^^^^^^^^^^^^^^^^^^^
+
+.. option:: --quiet
+
+   Disable user interaction, i.e. do not print messages (except errors) and fail
+   if login credentials are needed but not given. This makes Instaloader
+   **suitable as a cron job**.

docs/conf.py

@@ -34,8 +34,16 @@ extensions = [
     'sphinx.ext.autodoc',
     'sphinx_autodoc_typehints',
     'sphinx.ext.githubpages',
+    'sphinx.ext.intersphinx'
 ]
 
+autodoc_default_flags = ['show-inheritance', 'members', 'undoc-members']
+
+autodoc_member_order = 'bysource'
+
+intersphinx_mapping = {'python': ('https://docs.python.org/3', None),
+                       'requests': ('http://docs.python-requests.org/en/master', None)}
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 

docs/index.rst

@@ -1,12 +1,18 @@
+.. meta::
+   :description:
+      Command line tool to download pictures (and videos) from Instagram.
+      Instaloader downloads public and private profiles, hashtags, user stories,
+      feeds, comments, geotags, captions and other metadata of each post.
+
 Instaloader
 ===========
 
+.. highlight:: none
+
 **Instaloader** is a tool to download pictures (or videos) along with
 their captions and other metadata from Instagram.
 
-With `Python <https://www.python.org/>`__ installed, do:
-
-::
+With `Python <https://www.python.org/>`__ installed, do::
 
     $ pip3 install instaloader
 
@@ -16,18 +22,24 @@ With `Python <https://www.python.org/>`__ installed, do:
 
 - downloads **public and private profiles, hashtags, user stories and
   feeds**,
-  
+
 - downloads **comments, geotags and captions** of each post,
 
 - automatically **detects profile name changes** and renames the target
   directory accordingly,
-  
+
 - allows **fine-grained customization** of filters and where to store
   downloaded media,
 
-- is free `Open Source <https://github.com/Thammus/instaloader>`__
+- is free `open source <https://github.com/Thammus/instaloader>`__
   software written in Python.
 
+::
+
+    instaloader [--comments] [--geotags] [--stories]
+                [--login YOUR-USERNAME] [--fast-update]
+                profile | "#hashtag" | :stories | :feed
+
 
 Table of Contents
 -----------------

docs/installation.rst

@@ -1,9 +1,28 @@
 Installation
 ============
 
-.. include:: ../README.rst
-   :start-after: installation-start
-   :end-before: installation-end
+.. highlight:: none
+
+Instaloader requires `Python <https://www.python.org/>`__, at least
+version 3.5.  If you have `pip <https://pypi.python.org/pypi/pip>`__
+installed, you may install Instaloader using
+
+::
+
+    pip3 install instaloader
+
+Alternatively, to get the most current version of Instaloader from our
+`Git repository <https://github.com/Thammus/instaloader>`__:
+
+::
+
+    pip3 install git+https://github.com/Thammus/instaloader
+
+(pass ``--upgrade`` to upgrade if Instaloader is already installed)
+
+Instaloader requires
+`requests <http://python-requests.org/>`__, which
+will be installed automatically, if it is not already installed.
 
 If you do not want to use pip, even though it is highly recommended,
 and prefer installing Instaloader manually,

instaloader.py

@@ -29,7 +29,7 @@ import requests.utils
 import urllib3
 
 
-__version__ = '3.0'
+__version__ = '3.2.1'
 
 
 try:
@@ -43,7 +43,9 @@ else:
 
 
 class InstaloaderException(Exception):
-    """Base exception for this script"""
+    """Base exception for this script.
+
+    :note: This exception should not be raised directly."""
     pass
 
 
@@ -124,7 +126,7 @@ def mediaid_to_shortcode(mediaid: int) -> str:
 def format_string_contains_key(format_string: str, key: str) -> bool:
     # pylint:disable=unused-variable
     for literal_text, field_name, format_spec, conversion in string.Formatter().parse(format_string):
-        if field_name == key:
+        if field_name == key or field_name.startswith(key + '.'):
             return True
     return False
 
@@ -163,23 +165,29 @@ class Post:
     """
     Structure containing information about an Instagram post.
 
-    Created by Instaloader methods get_profile_posts(), get_hashtag_posts(), get_feed_posts(). Posts are linked to
-    an Instaloader instance which is used for error logging and obtaining of additional metadata, if required.
-    This class unifies access to the properties associated with a post. It implements == and is hashable.
+    Created by Instaloader methods :meth:`.get_profile_posts`, :meth:`.get_hashtag_posts`, :meth:`.get_feed_posts`.
+    Posts are linked to an :class:`Instaloader` instance which is used for error logging and obtaining of additional
+    metadata, if required. This class unifies access to the properties associated with a post. It implements == and is
+    hashable.
+
+    The properties defined here are accessible by the filter expressions specified with the :option:`--only-if`
+    parameter and exported into JSON files with :option:`--metadata-json`.
     """
 
     LOGIN_REQUIRING_PROPERTIES = ["viewer_has_liked"]
 
-    def __init__(self, instaloader: 'Instaloader', node: Dict[str, Any], profile: Optional[str] = None):
+    def __init__(self, instaloader: 'Instaloader', node: Dict[str, Any],
+                 profile: Optional[str] = None, profile_id: Optional[int] = None):
         """Create a Post instance from a node structure as returned by Instagram.
 
-        :param instaloader: Instaloader instance used for additional queries if neccessary.
+        :param instaloader: :class:`Instaloader` instance used for additional queries if neccessary.
         :param node: Node structure.
         :param profile: The name of the owner, if already known at creation.
         """
         self._instaloader = instaloader
         self._node = node
         self._profile = profile
+        self._profile_id = profile_id
         self._full_metadata_dict = None
 
     @classmethod
@@ -200,6 +208,11 @@ class Post:
         """Media shortcode. URL of the post is instagram.com/p/<shortcode>/."""
         return self._node['shortcode'] if 'shortcode' in self._node else self._node['code']
 
+    @property
+    def mediaid(self) -> int:
+        """The mediaid is a decimal representation of the media shortcode."""
+        return int(self._node['id'])
+
     def __repr__(self):
         return '<Post {}>'.format(self.shortcode)
 
@@ -246,6 +259,13 @@ class Post:
             self._instaloader.error("Get owner name of {}: {} -- using \'UNKNOWN\'.".format(self, err))
             return 'UNKNOWN'
 
+    @property
+    def owner_id(self) -> int:
+        """The ID of the Post's owner."""
+        if self._profile_id:
+            return self._profile_id
+        return int(self._field('owner', 'id'))
+
     @property
     def date(self) -> datetime:
         """Timestamp when the post was created."""
@@ -285,6 +305,25 @@ class Post:
         hashtag_regex = re.compile(r"(?:#)(\w(?:(?:\w|(?:\.(?!\.))){0,28}(?:\w))?)")
         return re.findall(hashtag_regex, self.caption.lower())
 
+    @property
+    def caption_mentions(self) -> List[str]:
+        """List of all lowercased profiles that are mentioned in the Post's caption, without preceeding @."""
+        if not self.caption:
+            return []
+        # This regular expression is from jStassen, adjusted to use Python's \w to support Unicode
+        # http://blog.jstassen.com/2016/03/code-regex-for-instagram-username-and-hashtags/
+        mention_regex = re.compile(r"(?:@)(\w(?:(?:\w|(?:\.(?!\.))){0,28}(?:\w))?)")
+        return re.findall(mention_regex, self.caption.lower())
+
+    @property
+    def tagged_users(self) -> List[str]:
+        """List of all lowercased users that are tagged in the Post."""
+        try:
+            return [edge['node']['user']['username' ].lower() for edge in self._field('edge_media_to_tagged_user',
+                                                                                      'edges')]
+        except KeyError:
+            return []
+
     @property
     def is_video(self) -> bool:
         """True if the Post is a video."""
@@ -336,13 +375,34 @@ class Post:
                                                        params={'__a': 1})
             return location_json["location"]
 
+    @staticmethod
+    def json_encoder(obj) -> Dict[str, Any]:
+        """Convert instance of :class:`Post` to a JSON-serializable dictionary."""
+        if not isinstance(obj, Post):
+            raise TypeError("Object of type {} is not a Post object.".format(obj.__class__.__name__))
+        jsondict = {}
+        for prop in dir(Post):
+            if prop[0].isupper() or prop[0] == '_':
+                # skip uppercase and private properties
+                continue
+            val = obj.__getattribute__(prop)
+            if val is True or val is False or isinstance(val, (str, int, float, list)):
+                jsondict[prop] = val
+            elif isinstance(val, datetime):
+                jsondict[prop] = val.isoformat()
+        return jsondict
 
 class Tristate(Enum):
     """Tri-state to encode whether we should save certain information, i.e. videos, captions, comments or geotags.
 
-    never: Do not save, even if the information is available without any additional request,
-    no_extra_query: Save if and only if available without doing additional queries,
-    always: Save (and query, if neccessary).
+    :attr:`never`
+        Do not save, even if the information is available without any additional request,
+
+    :attr:`no_extra_query`
+        Save if and only if available without doing additional queries,
+
+    :attr:`always`
+        Save (and query, if neccessary).
     """
     never = 0
     no_extra_query = 1
@@ -357,8 +417,10 @@ class Instaloader:
                  filename_pattern: Optional[str] = None,
                  download_videos: Tristate = Tristate.always,
                  download_geotags: Tristate = Tristate.no_extra_query,
-                 download_captions: Tristate = Tristate.no_extra_query,
-                 download_comments: Tristate = Tristate.no_extra_query):
+                 save_captions: Tristate = Tristate.no_extra_query,
+                 download_comments: Tristate = Tristate.no_extra_query,
+                 save_metadata: Tristate = Tristate.never,
+                 max_connection_attempts: int = 3):
 
         # configuration parameters
         self.user_agent = user_agent if user_agent is not None else default_user_agent()
@@ -371,16 +433,16 @@ class Instaloader:
             if filename_pattern is not None else '{date:%Y-%m-%d_%H-%M-%S}'
         self.download_videos = download_videos
         self.download_geotags = download_geotags
-        self.download_captions = download_captions
+        self.save_captions = save_captions
         self.download_comments = download_comments
-        self.previous_queries = dict()
+        self.save_metadata = save_metadata
+        self.max_connection_attempts = max_connection_attempts
 
         # error log, filled with error() and printed at the end of Instaloader.main()
         self.error_log = []
 
         # For the adaption of sleep intervals (rate control)
-        self.request_count = 0
-        self.last_request_time = 0
+        self.previous_queries = dict()
 
     @property
     def is_logged_in(self) -> bool:
@@ -393,7 +455,8 @@ class Instaloader:
         new_loader = Instaloader(self.sleep, self.quiet, self.user_agent,
                                  self.dirname_pattern, self.filename_pattern,
                                  self.download_videos, self.download_geotags,
-                                 self.download_captions, self.download_comments)
+                                 self.save_captions, self.download_comments,
+                                 self.save_metadata, self.max_connection_attempts)
         new_loader.previous_queries = self.previous_queries
         yield new_loader
         self.error_log.extend(new_loader.error_log)
@@ -431,7 +494,7 @@ class Instaloader:
         if self.sleep:
             time.sleep(random.uniform(0.5, 3))
 
-    def _get_and_write_raw(self, url: str, filename: str, tries: int = 3) -> None:
+    def _get_and_write_raw(self, url: str, filename: str, _attempt = 1) -> None:
         """Downloads raw data.
 
         :raises QueryReturnedNotFoundException: When the server responds with a 404.
@@ -450,21 +513,23 @@ class Instaloader:
                 raise ConnectionException("HTTP error code {}.".format(resp.status_code))
         except (urllib3.exceptions.HTTPError, requests.exceptions.RequestException, ConnectionException) as err:
             error_string = "URL {}: {}".format(url, err)
-            if tries <= 1:
+            if _attempt == self.max_connection_attempts:
+                raise ConnectionException(error_string)
+            self.error(error_string + " [retrying; skip with ^C]", repeat_at_end=False)
+            try:
+                self._sleep()
+                self._get_and_write_raw(url, filename, _attempt + 1)
+            except KeyboardInterrupt:
+                self.error("[skipped by user]", repeat_at_end=False)
                 raise ConnectionException(error_string)
-            else:
-                self.error(error_string + " [retrying]", repeat_at_end=False)
-            self._sleep()
-            self._get_and_write_raw(url, filename, tries - 1)
 
     def get_json(self, url: str, params: Dict[str, Any],
-                 session: Optional[requests.Session] = None, tries: int = 3) -> Dict[str, Any]:
+                 session: Optional[requests.Session] = None, _attempt = 1) -> Dict[str, Any]:
         """JSON request to Instagram.
 
         :param url: URL, relative to www.instagram.com/
         :param params: GET parameters
         :param session: Session to use, or None to use self.session
-        :param tries: Maximum number of attempts until an exception is raised
         :return: Decoded response dictionary
         :raises QueryReturnedNotFoundException: When the server responds with a 404.
         :raises ConnectionException: When query repeatedly failed.
@@ -512,21 +577,25 @@ class Instaloader:
             return resp_json
         except (ConnectionException, json.decoder.JSONDecodeError, requests.exceptions.RequestException) as err:
             error_string = "JSON Query to {}: {}".format(url, err)
-            if tries <= 1:
+            if _attempt == self.max_connection_attempts:
+                raise ConnectionException(error_string)
+            self.error(error_string + " [retrying; skip with ^C]", repeat_at_end=False)
+            text_for_429 = ("HTTP error code 429 was returned because too many queries occured in the last time. "
+                            "Please do not use Instagram in your browser or run multiple instances of Instaloader "
+                            "in parallel.")
+            try:
+                if isinstance(err, TooManyRequests):
+                    print(textwrap.fill(text_for_429), file=sys.stderr)
+                    if is_graphql_query:
+                        waittime = graphql_query_waittime(query_id=params['query_id'], untracked_queries=True)
+                        if waittime > 0:
+                            self._log('The request will be retried in {} seconds.'.format(waittime))
+                            time.sleep(waittime)
+                self._sleep()
+                return self.get_json(url, params, sess, _attempt + 1)
+            except KeyboardInterrupt:
+                self.error("[skipped by user]", repeat_at_end=False)
                 raise ConnectionException(error_string)
-            self.error(error_string + " [retrying]", repeat_at_end=False)
-            if isinstance(err, TooManyRequests):
-                text_for_429 = ("HTTP error code 429 was returned because too many queries occured in the last time. "
-                                "Please do not use Instagram in your browser or run multiple instances of Instaloader "
-                                "in parallel.")
-                print(textwrap.fill(text_for_429), file=sys.stderr)
-                if is_graphql_query:
-                    waittime = graphql_query_waittime(query_id=params['query_id'], untracked_queries=True)
-                    if waittime > 0:
-                        self._log('The request will be retried in {} seconds.'.format(waittime))
-                        time.sleep(waittime)
-            self._sleep()
-            return self.get_json(url, params, sess, tries - 1)
 
     def _default_http_header(self, empty_session_only: bool = False) -> Dict[str, str]:
         """Returns default HTTP header we use for requests."""
@@ -607,7 +676,7 @@ class Instaloader:
     def graphql_node_list(self, query_id: int, query_variables: Dict[str, Any], query_referer: Optional[str],
                           edge_extractor: Callable[[Dict[str, Any]], Dict[str, Any]]) -> Iterator[Dict[str, Any]]:
         """Retrieve a list of GraphQL nodes."""
-        query_variables['first'] = 500
+        query_variables['first'] = 200
         data = self.graphql_query(query_id, query_variables, query_referer)
         while True:
             edge_struct = edge_extractor(data)
@@ -646,7 +715,7 @@ class Instaloader:
                      filename_suffix: Optional[str] = None) -> bool:
         """Downloads and saves picture with given url under given directory with given timestamp.
         Returns true, if file was actually downloaded, i.e. updated."""
-        urlmatch = re.search('\\.[a-z]*\\?', url)
+        urlmatch = re.search('\\.[a-z0-9]*\\?', url)
         file_extension = url[-3:] if urlmatch is None else urlmatch.group(0)[1:-1]
         if filename_suffix is not None:
             filename += '_' + filename_suffix
@@ -658,6 +727,12 @@ class Instaloader:
         os.utime(filename, (datetime.now().timestamp(), mtime.timestamp()))
         return True
 
+    def save_metadata_json(self, filename: str, post: Post) -> None:
+        """Saves metadata JSON file of a :class:`Post`."""
+        filename += '.json'
+        json.dump(post, fp=open(filename, 'w'), indent=4, default=Post.json_encoder)
+        self._log('json', end=' ', flush=True)
+
     def update_comments(self, filename: str, post: Post) -> None:
         filename += '_comments.json'
         try:
@@ -753,7 +828,7 @@ class Instaloader:
         self._log('') # log output of _get_and_write_raw() does not produce \n
 
     def save_session_to_file(self, filename: Optional[str] = None) -> None:
-        """Saves requests.Session object."""
+        """Saves internally stored :class:`requests.Session` object."""
         if filename is None:
             filename = get_default_session_filename(self.username)
         dirname = os.path.dirname(filename)
@@ -766,7 +841,7 @@ class Instaloader:
             self._log("Saved session to %s." % filename)
 
     def load_session_from_file(self, username: str, filename: Optional[str] = None) -> None:
-        """Internally stores requests.Session object loaded from file.
+        """Internally stores :class:`requests.Session` object loaded from file.
 
         If filename is None, the file with the default session path is loaded.
 
@@ -784,7 +859,7 @@ class Instaloader:
             self.username = username
 
     def test_login(self, session: Optional[requests.Session]) -> Optional[str]:
-        """Returns the Instagram username to which given requests.Session object belongs, or None."""
+        """Returns the Instagram username to which given :class:`requests.Session` object belongs, or None."""
         if session:
             data = self.get_json('', params={'__a': 1}, session=session)
             return data['graphql']['user']['username'] if 'graphql' in data else None
@@ -828,7 +903,8 @@ class Instaloader:
         profilename = post.owner_username if needs_profilename else None
         dirname = self.dirname_pattern.format(profile=profilename, target=target.lower())
         filename = dirname + '/' + self.filename_pattern.format(profile=profilename, target=target.lower(),
-                                                                date=post.date, shortcode=post.shortcode)
+                                                                date=post.date, shortcode=post.shortcode,
+                                                                post=post)
         os.makedirs(os.path.dirname(filename), exist_ok=True)
 
         # Download the image(s) / video thumbnail and videos within sidecars if desired
@@ -855,7 +931,7 @@ class Instaloader:
             downloaded = False
 
         # Save caption if desired
-        if self.download_captions is not Tristate.never:
+        if self.save_captions is not Tristate.never:
             if post.caption:
                 self.save_caption(filename, post.date, post.caption)
             else:
@@ -875,6 +951,11 @@ class Instaloader:
         if self.download_comments is Tristate.always:
             self.update_comments(filename, post)
 
+        # Save metadata as JSON if desired.  It might require an extra query, depending on which information has been
+        # already obtained.  Regarding Tristate interpretation, we always assume that it requires an extra query.
+        if self.save_metadata is Tristate.always:
+            self.save_metadata_json(filename, post)
+
         self._log()
         return downloaded
 
@@ -927,6 +1008,10 @@ class Instaloader:
         :param filename_target: Replacement for {target} in dirname_pattern and filename_pattern
         """
 
+        if format_string_contains_key(self.filename_pattern, 'post'):
+            raise InvalidArgumentException("The \"post\" keyword is not supported in the filename pattern when "
+                                           "downloading stories.")
+
         if not self.is_logged_in:
             raise LoginRequiredException('Login required to download stories')
 
@@ -959,7 +1044,7 @@ class Instaloader:
                         self._log("Warning: Unable to find story image.")
                         downloaded = False
                     if "caption" in item and item["caption"] is not None and \
-                                    self.download_captions is not Tristate.never:
+                                    self.save_captions is not Tristate.never:
                         caption = item["caption"]
                         if isinstance(caption, dict) and "text" in caption:
                             caption = caption["text"]
@@ -1011,7 +1096,7 @@ class Instaloader:
         """
         Download pictures from the user's feed.
 
-        Example to download up to the 20 pics the user last liked: ::
+        Example to download up to the 20 pics the user last liked::
 
             loader = Instaloader()
             loader.load_session_from_file('USER')
@@ -1050,7 +1135,7 @@ class Instaloader:
                          fast_update: bool = False) -> None:
         """Download pictures of one hashtag.
 
-        To download the last 30 pictures with hashtag #cat, do ::
+        To download the last 30 pictures with hashtag #cat, do::
 
             loader = Instaloader()
             loader.download_hashtag('cat', max_count=30)
@@ -1081,7 +1166,8 @@ class Instaloader:
         has changed and return current name of the profile, and store ID of profile.
 
         :param profile: Profile name
-        :param profile_metadata: The profile's metadata (get_profile_metadata()), or None if the profile was not found
+        :param profile_metadata:
+            The profile's metadata (:meth:`get_profile_metadata`), or None if the profile was not found
         :return: current profile name, profile id
         """
         profile_exists = profile_metadata is not None
@@ -1126,7 +1212,7 @@ class Instaloader:
         raise ProfileNotExistsException("Profile {0} does not exist.".format(profile))
 
     def get_profile_metadata(self, profile_name: str) -> Dict[str, Any]:
-        """Retrieves a profile's metadata, for use with e.g. get_profile_posts() and check_profile_id()."""
+        """Retrieves a profile's metadata, for use with e.g. :meth:`get_profile_posts` and :meth:`check_profile_id`."""
         try:
             return self.get_json('{}/'.format(profile_name), params={'__a': 1})
         except QueryReturnedNotFoundException:
@@ -1135,18 +1221,21 @@ class Instaloader:
     def get_profile_posts(self, profile_metadata: Dict[str, Any]) -> Iterator[Post]:
         """Retrieve all posts from a profile."""
         profile_name = profile_metadata['user']['username']
-        yield from (Post(self, node, profile=profile_name) for node in profile_metadata['user']['media']['nodes'])
+        profile_id = int(profile_metadata['user']['id'])
+        yield from (Post(self, node, profile=profile_name, profile_id=profile_id)
+                    for node in profile_metadata['user']['media']['nodes'])
         has_next_page = profile_metadata['user']['media']['page_info']['has_next_page']
         end_cursor = profile_metadata['user']['media']['page_info']['end_cursor']
         while has_next_page:
             # We do not use self.graphql_node_list() here, because profile_metadata
             # lets us obtain the first 12 nodes 'for free'
             data = self.graphql_query(17888483320059182, {'id': profile_metadata['user']['id'],
-                                                          'first': 500,
+                                                          'first': 200,
                                                           'after': end_cursor},
                                       'https://www.instagram.com/{0}/'.format(profile_name))
             media = data['data']['user']['edge_owner_to_timeline_media']
-            yield from (Post(self, edge['node'], profile=profile_name) for edge in media['edges'])
+            yield from (Post(self, edge['node'], profile=profile_name, profile_id=profile_id)
+                        for edge in media['edges'])
             has_next_page = media['page_info']['has_next_page']
             end_cursor = media['page_info']['end_cursor']
 
@@ -1354,6 +1443,9 @@ def main():
                              'server for each post, which is why it is disabled by default.')
     g_what.add_argument('--no-captions', action='store_true',
                         help='Do not store media captions, although no additional request is needed to obtain them.')
+    g_what.add_argument('--metadata-json', action='store_true',
+                        help='Create a JSON file containing the metadata of each post. This does not include comments '
+                             'nor geotags.')
     g_what.add_argument('-s', '--stories', action='store_true',
                         help='Also download stories of each profile that is downloaded. Requires --login.')
     g_what.add_argument('--stories-only', action='store_true',
@@ -1372,7 +1464,7 @@ def main():
                              'flag is recommended when you use Instaloader to update your personal Instagram archive.')
     g_stop.add_argument('-c', '--count',
                         help='Do not attempt to download more than COUNT posts. '
-                             'Applies only to #hashtag, :feed-all and :feed-liked.')
+                             'Applies only to #hashtag and :feed.')
 
     g_login = parser.add_argument_group('Login (Download Private Profiles)',
                                         'Instaloader can login to Instagram. This allows downloading private profiles. '
@@ -1398,11 +1490,16 @@ def main():
                        help='Prefix of filenames. Posts are stored in the directory whose pattern is given with '
                             '--dirname-pattern. {profile} is replaced by the profile name, '
                             '{target} is replaced by the target you specified, i.e. either :feed, #hashtag or the '
-                            'profile name. Also, the fields date and shortcode can be specified. Defaults to '
-                            '\'{date:%%Y-%%m-%%d_%%H-%%M-%%S}\'.')
+                            'profile name. Also, the fields {date} and {shortcode} can be specified. In case of not '
+                            'downloading stories, the attributes of the Post class can be used in addition, e.g. '
+                            '{post.owner_id} or {post.mediaid}. Defaults to \'{date:%%Y-%%m-%%d_%%H-%%M-%%S}\'.')
     g_how.add_argument('--user-agent',
                        help='User Agent to use for HTTP requests. Defaults to \'{}\'.'.format(default_user_agent()))
     g_how.add_argument('-S', '--no-sleep', action='store_true', help=SUPPRESS)
+    g_how.add_argument('--max-connection-attempts', metavar='N', type=int, default=3,
+                       help='Maximum number of connection attempts until a request is aborted. Defaults to 3. If a '
+                            'connection fails, it can be manually skipped by hitting CTRL+C. Set this to 0 to retry '
+                            'infinitely.')
 
     g_misc = parser.add_argument_group('Miscellaneous Options')
     g_misc.add_argument('-q', '--quiet', action='store_true',
@@ -1427,7 +1524,8 @@ def main():
 
         download_videos = Tristate.always if not args.no_videos else Tristate.no_extra_query
         download_comments = Tristate.always if args.comments else Tristate.no_extra_query
-        download_captions = Tristate.no_extra_query if not args.no_captions else Tristate.never
+        save_captions = Tristate.no_extra_query if not args.no_captions else Tristate.never
+        save_metadata = Tristate.always if args.metadata_json else Tristate.never
 
         if args.geotags and args.no_geotags:
             raise SystemExit("--geotags and --no-geotags given. I am confused and refuse to work.")
@@ -1442,7 +1540,8 @@ def main():
                              user_agent=args.user_agent,
                              dirname_pattern=args.dirname_pattern, filename_pattern=args.filename_pattern,
                              download_videos=download_videos, download_geotags=download_geotags,
-                             download_captions=download_captions, download_comments=download_comments)
+                             save_captions=save_captions, download_comments=download_comments,
+                             save_metadata=save_metadata, max_connection_attempts=args.max_connection_attempts)
         loader.main(args.profile, args.login.lower() if args.login is not None else None, args.password,
                     args.sessionfile,
                     int(args.count) if args.count is not None else None,