Release of version 3.2.1

Fixes #59.

README.rst

@@ -20,6 +20,12 @@ Instaloader
 - allows **fine-grained customization** of filters and where to store
   downloaded media.
 
+::
+
+    instaloader [--comments] [--geotags] [--stories]
+                [--login YOUR-USERNAME] [--fast-update]
+                profile | "#hashtag" | :stories | :feed
+
 `Instaloader Documentation <https://instaloader.readthedocs.io/>`__
 
 

docs/README.md

@@ -0,0 +1,15 @@
+# Instaloader Documentation
+
+This directory contains sources for the Instaloader documentation.
+
+The documentation is hosted at:
+https://instaloader.readthedocs.io/
+
+## Build
+
+The documentation is created with [Sphinx](http://www.sphinx-doc.org/). To build it, use
+
+```
+pip3 install sphinx sphinx-autodoc-typehints
+make html
+```

docs/_templates/caption.html

@@ -1 +1,2 @@
-<h2>Instaloader</h2>
+<h2><a href="index.html">Instaloader</a></h2>
+<p>Download pictures (or videos) along with their captions and other metadata from Instagram.</p>

docs/_templates/layout.html

@@ -1,4 +1,5 @@
 {% extends "!layout.html" %}
 {% block extrahead %}
+<meta name="google-site-verification" content="0esgS4FkYROvPe-I9MYvC1tTYtfASPpqxBdpTHEBIcQ" />
 <script async defer src="https://buttons.github.io/buttons.js"></script>
 {% endblock %}

docs/as-module.rst

@@ -1,3 +1,8 @@
+.. meta::
+   :description:
+      Instaloader can also be used as a powerful and easy-to-use
+      Python API for Instagram, allowing to download media and metadata.
+
 Python Module :mod:`instaloader`
 --------------------------------
 
@@ -5,68 +10,65 @@ Python Module :mod:`instaloader`
 
 .. highlight:: python
 
-You may also use parts of Instaloader as library to do other interesting
-things.
+Instaloader exposes its internally used methods as a Python module, making it a
+**powerful and easy-to-use Python API for Instagram**, allowing to further
+customize obtaining media and metadata.
 
-For example, to get a list of all followees and a list of all followers of a profile, do
-
-.. code:: python
+Start with getting an instance of :class:`Instaloader`::
 
     import instaloader
 
     # Get instance
-    loader = instaloader.Instaloader()
+    L = instaloader.Instaloader()
 
-    # Login
-    loader.interactive_login(USERNAME)
+    # Optionally, login or load session
+    L.login(USER, PASSWORD)        # (login)
+    L.interactive_login(USER)      # (ask password on terminal)
+    L.load_session_from_file(USER) # (load session created w/
+                                   #  `instaloader -l USERNAME`)
+
+:mod:`instaloader` provides the :class:`Post` structure, which represents a
+picture, video or sidecar (set of multiple pictures/videos) posted in a user's
+profile. :class:`Instaloader` provides methods to iterate over Posts from a
+certain source::
+
+    for post in L.get_hashtag_posts('cat'):
+        # post is an instance of instaloader.Post
+        L.download_post(post, target='#cat')
+
+Besides :func:`Instaloader.get_hashtag_posts`, there is
+:func:`Instaloader.get_feed_posts` and :func:`Instaloader.get_profile_posts`.
+Also, :class:`Post` instances can be created with :func:`Post.from_shortcode`
+and :func:`Post.from_mediaid`.
+
+Further, information about profiles can be easily obtained. For example, you may
+print a list of all followees and followers of a profile with::
 
     # Print followees
     print(PROFILE + " follows these profiles:")
-    for f in loader.get_followees(PROFILE):
+    for f in L.get_followees(PROFILE):
         print("\t%s\t%s" % (f['username'], f['full_name']))
 
     # Print followers
     print("Followers of " + PROFILE + ":")
-    for f in loader.get_followers(PROFILE):
+    for f in L.get_followers(PROFILE):
         print("\t%s\t%s" % (f['username'], f['full_name']))
 
-Then, you may download all pictures of all followees with
+Then, you may download all pictures of all followees with::
 
-.. code:: python
+    for f in L.get_followees(PROFILE):
+        L.download_profile(f['username'])
 
-    for f in loader.get_followees(PROFILE):
-        loader.download_profile(f['username'])
+Each Instagram profile has its own unique ID which stays unmodified even if a
+user changes his/her username. To get said ID, given the profile's name, you may
+call::
 
-You could also download your last 20 liked pics with
-
-.. code:: python
-
-    loader.download_feed_posts(max_count=20, fast_update=True,
-                               filter_func=lambda post: post.viewer_has_liked)
-
-To download the last 20 pictures with hashtag #cat, do
-
-.. code:: python
-
-    loader.download_hashtag('cat', max_count=20)
-
-Generally, :class:`Instaloader` provides methods to iterate over the Posts from
-a certain source.
-
-.. code:: python
-
-    for post in loader.get_hashtag_posts('cat'):
-        # post is an instance of instaloader.Post
-        loader.download_post(post, target='#cat')
-
-Each Instagram profile has its own unique ID which stays unmodified even
-if a user changes his/her username. To get said ID, given the profile's
-name, you may call
-
-.. code:: python
-
-    loader.get_id_by_username(PROFILE_NAME)
+    L.get_id_by_username(PROFILE_NAME)
 
+A reference of the many methods provided by the :mod:`instaloader` module is
+provided in the remainder of this document. Feel free to direct any issue or
+contribution to
+`Instaloader on Github <https://github.com/Thammus/instaloader>`__.
 
 ``Instaloader`` (Main Class)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

docs/basic-usage.rst

@@ -96,7 +96,8 @@ pattern, the token ``{target}`` is replaced by the target name, and
 to the target directory. The default is ``--filename-pattern={date}``.
 The tokens ``{target}`` and ``{profile}`` are replaced like in the
 dirname pattern. Further, the tokens ``{date}`` and ``{shortcode}`` are
-defined.
+defined. Additionally, in case of not downloading stories, the attributes of
+:class:`.Post` can be used, e.g. ``{post.owner_id}`` or ``{post.mediaid}``.
 
 For example, encode the poster's profile name in the filenames with:
 

docs/cli-options.rst

@@ -1,24 +1,26 @@
 Command Line Options
 ====================
 
-The following flags can be given to Instaloader to specify how profiles should
-be downloaded.
+Instaloader is invoked with::
 
-To get a list of all flags, their abbreviations and their descriptions,
-run ``instaloader --help``.
+   $ instaloader [options] target [target ...]
+
+where ``target`` is a ``profile``, a ``"#hashtag"``, ``@profile`` (all profiles
+that *profile* is following), or if logged in ``:feed`` (pictures from your
+feed) or ``:stories`` (stories of your followees).
+
+Here we explain the additional options that can be given to Instaloader to
+customize its behavior.  To get a list of all flags, their abbreviations and
+their descriptions, you may also run ``instaloader --help``.
 
 What to Download
 ^^^^^^^^^^^^^^^^
 
-Specify a list of profiles or #hashtags. For each of these, Instaloader
-creates a folder and downloads all posts along with the pictures's
-captions and the current **profile picture**. If an already-downloaded profile
-has been renamed, Instaloader automatically **finds it by its unique ID** and
-renames the folder likewise.
-
-Instead of a *profile* or a *#hashtag*, the special targets
-``:feed`` (pictures from your feed) and
-``:stories`` (stories of your followees) can be specified.
+Specify a list of targets (profiles, #hashtags, ``:feed`` or ``:stories``). For
+each of these, Instaloader creates a folder and stores all posts along with the
+pictures's captions and the current **profile picture** there. If an
+already-downloaded profile has been renamed, Instaloader automatically **finds
+it by its unique ID** and renames the folder likewise.
 
 .. option:: --profile-pic-only
 
@@ -92,7 +94,7 @@ matching the specified targets.
 .. option:: --count COUNT
 
    Do not attempt to download more than COUNT posts.  Applies only to
-   ``#hashtag``, ``:feed-all`` and ``:feed-liked``.
+   ``#hashtag`` and ``:feed``.
 
 
 Login (Download Private Profiles)
@@ -134,8 +136,11 @@ How to Download
    with ``--dirname-pattern``.  ``{profile}`` is replaced by the profile name,
    ``{target}`` is replaced by the target you specified, i.e.  either ``:feed``,
    ``#hashtag`` or the profile name.  Also, the fields ``{date}`` and
-   ``{shortcode}`` can be specified.  Defaults to ``{date:%Y-%m-%d_%H-%M-%S}``.
-   See :ref:`filename-specification`.
+   ``{shortcode}`` can be specified.  In case of not downloading stories, the
+   attributes of the :class:`.Post` class can be used in addition, e.g.
+   ``{post.owner_id}`` or ``{post.mediaid}``.
+   Defaults to
+   ``{date:%Y-%m-%d_%H-%M-%S}``.  See :ref:`filename-specification`.
 
 .. option:: --user-agent USER_AGENT
 
@@ -145,7 +150,7 @@ How to Download
 .. option:: --max-connection-attempts N
 
    Maximum number of connection attempts until a request is aborted. Defaults
-   to ``3``. If a connection fails, it can by manually skipped by hitting
+   to ``3``. If a connection fails, it can be manually skipped by hitting
    :kbd:`Control-c`. Set this to ``0`` to retry infinitely.
 
 Miscellaneous Options

docs/index.rst

@@ -1,6 +1,14 @@
+.. meta::
+   :description:
+      Command line tool to download pictures (and videos) from Instagram.
+      Instaloader downloads public and private profiles, hashtags, user stories,
+      feeds, comments, geotags, captions and other metadata of each post.
+
 Instaloader
 ===========
 
+.. highlight:: none
+
 **Instaloader** is a tool to download pictures (or videos) along with
 their captions and other metadata from Instagram.
 
@@ -26,6 +34,12 @@ With `Python <https://www.python.org/>`__ installed, do::
 - is free `open source <https://github.com/Thammus/instaloader>`__
   software written in Python.
 
+::
+
+    instaloader [--comments] [--geotags] [--stories]
+                [--login YOUR-USERNAME] [--fast-update]
+                profile | "#hashtag" | :stories | :feed
+
 
 Table of Contents
 -----------------

instaloader.py

@@ -29,7 +29,7 @@ import requests.utils
 import urllib3
 
 
-__version__ = '3.1'
+__version__ = '3.2.1'
 
 
 try:
@@ -126,7 +126,7 @@ def mediaid_to_shortcode(mediaid: int) -> str:
 def format_string_contains_key(format_string: str, key: str) -> bool:
     # pylint:disable=unused-variable
     for literal_text, field_name, format_spec, conversion in string.Formatter().parse(format_string):
-        if field_name == key:
+        if field_name == key or field_name.startswith(key + '.'):
             return True
     return False
 
@@ -176,7 +176,8 @@ class Post:
 
     LOGIN_REQUIRING_PROPERTIES = ["viewer_has_liked"]
 
-    def __init__(self, instaloader: 'Instaloader', node: Dict[str, Any], profile: Optional[str] = None):
+    def __init__(self, instaloader: 'Instaloader', node: Dict[str, Any],
+                 profile: Optional[str] = None, profile_id: Optional[int] = None):
         """Create a Post instance from a node structure as returned by Instagram.
 
         :param instaloader: :class:`Instaloader` instance used for additional queries if neccessary.
@@ -186,6 +187,7 @@ class Post:
         self._instaloader = instaloader
         self._node = node
         self._profile = profile
+        self._profile_id = profile_id
         self._full_metadata_dict = None
 
     @classmethod
@@ -206,6 +208,11 @@ class Post:
         """Media shortcode. URL of the post is instagram.com/p/<shortcode>/."""
         return self._node['shortcode'] if 'shortcode' in self._node else self._node['code']
 
+    @property
+    def mediaid(self) -> int:
+        """The mediaid is a decimal representation of the media shortcode."""
+        return int(self._node['id'])
+
     def __repr__(self):
         return '<Post {}>'.format(self.shortcode)
 
@@ -252,6 +259,13 @@ class Post:
             self._instaloader.error("Get owner name of {}: {} -- using \'UNKNOWN\'.".format(self, err))
             return 'UNKNOWN'
 
+    @property
+    def owner_id(self) -> int:
+        """The ID of the Post's owner."""
+        if self._profile_id:
+            return self._profile_id
+        return int(self._field('owner', 'id'))
+
     @property
     def date(self) -> datetime:
         """Timestamp when the post was created."""
@@ -662,7 +676,7 @@ class Instaloader:
     def graphql_node_list(self, query_id: int, query_variables: Dict[str, Any], query_referer: Optional[str],
                           edge_extractor: Callable[[Dict[str, Any]], Dict[str, Any]]) -> Iterator[Dict[str, Any]]:
         """Retrieve a list of GraphQL nodes."""
-        query_variables['first'] = 500
+        query_variables['first'] = 200
         data = self.graphql_query(query_id, query_variables, query_referer)
         while True:
             edge_struct = edge_extractor(data)
@@ -701,7 +715,7 @@ class Instaloader:
                      filename_suffix: Optional[str] = None) -> bool:
         """Downloads and saves picture with given url under given directory with given timestamp.
         Returns true, if file was actually downloaded, i.e. updated."""
-        urlmatch = re.search('\\.[a-z]*\\?', url)
+        urlmatch = re.search('\\.[a-z0-9]*\\?', url)
         file_extension = url[-3:] if urlmatch is None else urlmatch.group(0)[1:-1]
         if filename_suffix is not None:
             filename += '_' + filename_suffix
@@ -889,7 +903,8 @@ class Instaloader:
         profilename = post.owner_username if needs_profilename else None
         dirname = self.dirname_pattern.format(profile=profilename, target=target.lower())
         filename = dirname + '/' + self.filename_pattern.format(profile=profilename, target=target.lower(),
-                                                                date=post.date, shortcode=post.shortcode)
+                                                                date=post.date, shortcode=post.shortcode,
+                                                                post=post)
         os.makedirs(os.path.dirname(filename), exist_ok=True)
 
         # Download the image(s) / video thumbnail and videos within sidecars if desired
@@ -993,6 +1008,10 @@ class Instaloader:
         :param filename_target: Replacement for {target} in dirname_pattern and filename_pattern
         """
 
+        if format_string_contains_key(self.filename_pattern, 'post'):
+            raise InvalidArgumentException("The \"post\" keyword is not supported in the filename pattern when "
+                                           "downloading stories.")
+
         if not self.is_logged_in:
             raise LoginRequiredException('Login required to download stories')
 
@@ -1202,18 +1221,21 @@ class Instaloader:
     def get_profile_posts(self, profile_metadata: Dict[str, Any]) -> Iterator[Post]:
         """Retrieve all posts from a profile."""
         profile_name = profile_metadata['user']['username']
-        yield from (Post(self, node, profile=profile_name) for node in profile_metadata['user']['media']['nodes'])
+        profile_id = int(profile_metadata['user']['id'])
+        yield from (Post(self, node, profile=profile_name, profile_id=profile_id)
+                    for node in profile_metadata['user']['media']['nodes'])
         has_next_page = profile_metadata['user']['media']['page_info']['has_next_page']
         end_cursor = profile_metadata['user']['media']['page_info']['end_cursor']
         while has_next_page:
             # We do not use self.graphql_node_list() here, because profile_metadata
             # lets us obtain the first 12 nodes 'for free'
             data = self.graphql_query(17888483320059182, {'id': profile_metadata['user']['id'],
-                                                          'first': 500,
+                                                          'first': 200,
                                                           'after': end_cursor},
                                       'https://www.instagram.com/{0}/'.format(profile_name))
             media = data['data']['user']['edge_owner_to_timeline_media']
-            yield from (Post(self, edge['node'], profile=profile_name) for edge in media['edges'])
+            yield from (Post(self, edge['node'], profile=profile_name, profile_id=profile_id)
+                        for edge in media['edges'])
             has_next_page = media['page_info']['has_next_page']
             end_cursor = media['page_info']['end_cursor']
 
@@ -1442,7 +1464,7 @@ def main():
                              'flag is recommended when you use Instaloader to update your personal Instagram archive.')
     g_stop.add_argument('-c', '--count',
                         help='Do not attempt to download more than COUNT posts. '
-                             'Applies only to #hashtag, :feed-all and :feed-liked.')
+                             'Applies only to #hashtag and :feed.')
 
     g_login = parser.add_argument_group('Login (Download Private Profiles)',
                                         'Instaloader can login to Instagram. This allows downloading private profiles. '
@@ -1468,14 +1490,15 @@ def main():
                        help='Prefix of filenames. Posts are stored in the directory whose pattern is given with '
                             '--dirname-pattern. {profile} is replaced by the profile name, '
                             '{target} is replaced by the target you specified, i.e. either :feed, #hashtag or the '
-                            'profile name. Also, the fields date and shortcode can be specified. Defaults to '
-                            '\'{date:%%Y-%%m-%%d_%%H-%%M-%%S}\'.')
+                            'profile name. Also, the fields {date} and {shortcode} can be specified. In case of not '
+                            'downloading stories, the attributes of the Post class can be used in addition, e.g. '
+                            '{post.owner_id} or {post.mediaid}. Defaults to \'{date:%%Y-%%m-%%d_%%H-%%M-%%S}\'.')
     g_how.add_argument('--user-agent',
                        help='User Agent to use for HTTP requests. Defaults to \'{}\'.'.format(default_user_agent()))
     g_how.add_argument('-S', '--no-sleep', action='store_true', help=SUPPRESS)
     g_how.add_argument('--max-connection-attempts', metavar='N', type=int, default=3,
                        help='Maximum number of connection attempts until a request is aborted. Defaults to 3. If a '
-                            'connection fails, it can by manually skipped by hitting CTRL+C. Set this to 0 to retry '
+                            'connection fails, it can be manually skipped by hitting CTRL+C. Set this to 0 to retry '
                             'infinitely.')
 
     g_misc = parser.add_argument_group('Miscellaneous Options')